Decoding method, encoding method, decoding device, and encoding device
By decoding motion information and prediction mode indicators from the bitstream and adopting double-single hybrid prediction technology, the challenges of improving encoding efficiency and image quality in existing video encoding technologies are solved, and resource optimization and processing speed are improved.
Patent Information
- Application Number
- CN202380033084.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-13
- Filing Date
- 2023-04-03
- Publication Date
- 2024-11-26
AI Technical Summary
Existing video encoding technologies have challenges in improving coding efficiency, image quality and reducing processing volume and circuit scale, especially in the selection of motion vectors and reference images, resulting in complex processing and waste of resources.
A decoding method is proposed to decode the motion information and prediction mode indicator of the image block from the bitstream, and to decide whether to use double-single mixed prediction based on the indicator, generate the prediction image of the image block by combining single prediction and double prediction, and control the prediction example to reduce processing volume and complexity.
It improves coding efficiency and image quality, reduces processing volume and circuit scale, enhances processing speed, and appropriately selects motion vectors and reference images, suppressing processing complexity and resource waste.
Smart Images

Figure CN119032560A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a decoding method, an encoding method, a decoding device, and an encoding device. Background Art
[0002] Video coding technology is progressing from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). With this progress, there is a constant need to provide improvements and optimizations in video coding technology in order to handle the ever-increasing amount of digital video data in a variety of applications. The present disclosure relates to further advancements, improvements, and optimizations in video coding.
[0003] In addition, Non-Patent Document 1 relates to an example of a conventional standard related to the above-mentioned video encoding technology. In addition, Non-Patent Document 2 relates to a new technical proposal related to the video encoding technology.
[0004] Prior art literature
[0005] Non-patent literature
[0006] Non-patent document 1: H.265 (ISO / IEC 23008-2HEVC) / HEVC (High Efficiency Video Coding)
[0007] Non-patent document 2: Yi-Wen Chen et al., “AHG12: Enhanced bi-directional motion compensation”, JVET-Y0125, Joint Video Experts Team (JVET) of ITU-T SG16WP3 and ISO / IEC JTC 1 / SC29, 25th Meeting, teleconference, January 12-21, 2022 Summary of the invention
[0008] Problems to be solved by the invention
[0009] Regarding the encoding method as described above, it is hoped to propose a new method in order to improve the encoding efficiency, improve the image quality, reduce the processing amount, reduce the circuit scale, or appropriately select elements or actions such as filters, blocks, sizes, motion vectors, reference pictures or reference blocks.
[0010] The present disclosure provides, for example, structures or methods that can contribute to one or more of the following benefits: improvement in coding efficiency, improvement in image quality, reduction in processing volume, reduction in circuit size, improvement in processing speed, and appropriate selection of elements or actions. In addition, the present disclosure may include structures or methods that can contribute to benefits other than those described above.
[0011] Means used to solve problems
[0012] For example, a decoding method of a technical solution disclosed in the present invention is a decoding method for decoding a current block of an image from a bitstream, decoding a prediction mode indicator indicating whether a dual-single mixed prediction should be performed on the current block from the bitstream; decoding two instances of motion information for the current block from the bitstream; when the prediction mode indicator indicates that a dual-single mixed prediction should be performed on the current block, (i) using one of the two instances to perform a first prediction as a single prediction on a first partition that is a part of the current block, and (ii) using both of the two instances to perform a second prediction as a dual prediction on a second partition that is a part of the current block.
[0013] In video coding technology, new technical solutions are desired to improve coding efficiency, improve image quality, reduce circuit size, and the like.
[0014] Each technical solution of the present disclosure or a part of its structure or method can achieve at least one of, for example, improvement in coding efficiency, improvement in image quality, reduction in the amount of processing for coding / decoding, reduction in circuit scale, or improvement in the processing speed of coding / decoding. Alternatively, each technical solution of the present disclosure or a part of its structure or method can perform appropriate selection of components / actions such as filters, blocks, sizes, motion vectors, reference pictures, reference blocks, etc. in coding and decoding. In addition, the present disclosure also includes the disclosure of structures or methods that can provide benefits other than the above. For example, a structure or method that improves coding efficiency while suppressing an increase in the amount of processing.
[0015] More advantages and effects of a technical solution of the present disclosure are more clearly understood from the specification and the accompanying drawings. The advantages and / or effects are obtained from several technical solutions and the features described in the specification and the accompanying drawings, but all of them are not necessarily provided in order to obtain one or more advantages and / or effects.
[0016] In addition, these global or specific technical solutions may also be implemented by systems, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, and recording media.
[0017] Effects of the Invention
[0018] The structure or method of a technical solution of the present disclosure can contribute to one or more of, for example, improvement of coding efficiency, improvement of image quality, reduction of processing volume, reduction of circuit scale, improvement of processing speed, and appropriate selection of elements or actions. In addition, the structure or method of a technical solution of the present disclosure can also contribute to benefits other than the above. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a schematic diagram showing an example of the configuration of a transmission system according to an embodiment.
[0020] Figure 2 This is a diagram showing an example of the hierarchical structure of data in a stream.
[0021] Figure 3 This is a diagram showing an example of the structure of a slice.
[0022] Figure 4 This is a diagram showing an example of a tile structure.
[0023] Figure 5 This is a diagram showing an example of a coding structure in the case of scalable coding.
[0024] Figure 6 This is a diagram showing an example of a coding structure in the case of scalable coding.
[0025] Figure 7 This is a block diagram showing an example of the functional structure of the encoding device according to the embodiment.
[0026] Figure 8 This is a block diagram showing an implementation example of a coding device.
[0027] Fig. 9 This is a flowchart showing an example of the overall encoding process of the encoding device.
[0028] Fig.10 This is a diagram showing an example of block division.
[0029] Fig.11 This is a diagram showing an example of the functional structure of the dividing unit.
[0030] Fig.12 It is a diagram showing an example of a division pattern.
[0031] Fig.13A This is a diagram showing an example of a syntax tree of a segmentation pattern.
[0032] Fig. 13B This is a diagram showing another example of a syntax tree of a segmentation pattern.
[0033] Fig.14This is a table showing the transformation basis functions corresponding to each transformation type.
[0034] Fig.15 This is a diagram showing an example of SVT.
[0035] Fig.16 This is a flowchart showing an example of the processing of the conversion unit.
[0036] Fig.17 This is a flowchart showing another example of the processing of the conversion unit.
[0037] Fig.18 This is a block diagram showing an example of the functional configuration of a quantization unit.
[0038] Fig.19 This is a flowchart showing an example of quantization by the quantization unit.
[0039] Fig. 20 This is a block diagram showing an example of the functional structure of the entropy coding unit.
[0040] Fig.21 This is a diagram showing the CABAC process in the entropy coding unit.
[0041] Fig. 22 This is a block diagram showing an example of the functional configuration of a loop filter unit.
[0042] Fig.23A This is a diagram showing an example of the shape of a filter used in an ALF (adaptive loop filter).
[0043] Fig. 23B This is a diagram showing another example of the shape of the filter used in ALF.
[0044] Fig.23C This is a diagram showing another example of the shape of the filter used in ALF.
[0045] Fig.23D This is a diagram showing an example in which a Y sample (first component) is used in CCALF of Cb and CCALF of Cr (a plurality of components different from the first component).
[0046] Fig.23E is a diagram showing a diamond-shaped filter.
[0047] Fig.23F It is a diagram showing an example of JC-CCALF.
[0048] Figure 23G This is a diagram showing an example of weight_index candidates of JC-CCALF.
[0049] Fig.24This is a block diagram showing an example of a detailed configuration of a loop filter unit that functions as a DBF.
[0050] Fig.25 This is a diagram showing an example of a deblocking filter having a filter characteristic that is symmetric with respect to a block boundary.
[0051] Fig.26 This is a diagram for explaining an example of a block boundary on which a deblocking filtering process is performed.
[0052] Fig. 27 This is a diagram showing an example of the Bs value.
[0053] Fig.28 This is a flowchart showing an example of processing performed by the prediction unit of the encoding device.
[0054] Fig.29 This is a flowchart showing another example of the processing performed by the prediction unit of the encoding device.
[0055] Fig.30 This is a flowchart showing another example of the processing performed by the prediction unit of the encoding device.
[0056] Fig.31 This is a diagram showing an example of 67 intra prediction modes of intra prediction.
[0057] Fig.32 This is a flowchart showing an example of processing of the intra prediction unit.
[0058] Fig.33 A diagram showing an example of each reference picture.
[0059] Fig.34 This is a conceptual diagram showing an example of a reference picture list.
[0060] Fig.35 This is a flowchart showing the flow of basic processing of inter-frame prediction.
[0061] Fig.36 This is a flowchart showing an example of MV derivation.
[0062] Fig.37 This is a flowchart showing another example of MV derivation.
[0063] Fig.38A This is a diagram showing an example of classification of each mode derived from MV.
[0064] Fig.38B This is a diagram showing an example of classification of each mode derived from MV.
[0065] Fig.39 This is a flowchart showing an example of inter prediction based on the normal inter mode.
[0066] Fig.40 is a flowchart showing an example of inter-frame prediction based on the normal merge mode.
[0067] Fig.41 This is a diagram used to illustrate an example of MV export processing based on the normal merge mode.
[0068] Fig.42 This is a diagram for explaining an example of MV derivation processing based on the HMVP mode.
[0069] Fig.43 This is a flowchart showing an example of FRUC (frame rate up conversion).
[0070] Fig.44 This is a diagram for explaining an example of pattern matching (bilateral matching) between two blocks along a motion trajectory.
[0071] Fig.45 This is a diagram for explaining an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture.
[0072] Fig.46A This is a diagram for explaining an example of derivation of MV in sub-block units in an affine mode using two control points.
[0073] Fig.46B This is a diagram for explaining an example of derivation of MV in sub-block units in the affine mode using three control points.
[0074] Fig.47A This is a conceptual diagram for explaining an example of MV derivation of control points in the affine mode.
[0075] Fig.47B This is a conceptual diagram for explaining an example of MV derivation of control points in the affine mode.
[0076] Fig.47C This is a conceptual diagram for explaining an example of MV derivation of control points in the affine mode.
[0077] Fig.48A is a diagram used to illustrate an affine pattern with 2 control points.
[0078] Fig.48B is a diagram used to illustrate an affine pattern with three control points.
[0079] Fig.49AThis is a conceptual diagram for explaining an example of a method of deriving an MV of a control point when the number of control points in a coded block and a current block is different.
[0080] Fig.49B This is a conceptual diagram for explaining another example of a method for deriving an MV of a control point when the number of control points in a coded block and a current block is different.
[0081] Fig.50 This is a flowchart showing an example of processing in the affine merge mode.
[0082] Fig.51 This is a flowchart showing an example of processing in the affine inter mode.
[0083] Fig.52A This is a diagram used to explain the generation of predicted images of two triangles.
[0084] Fig.52B This is a conceptual diagram showing examples of the first part of the first partition and the first sample set and the second sample set.
[0085] Fig.52C This is a conceptual diagram showing the first part of the first partition.
[0086] Fig.53 This is a flowchart showing an example of the triangle mode.
[0087] Fig.54 This is a diagram showing an example of the ATMVP mode for deriving MV in sub-block units.
[0088] Fig.55 The diagram shows the relationship between the merge mode and DMVR (dynamic motion vector refreshing).
[0089] Fig.56 This is a conceptual diagram for explaining an example of DMVR.
[0090] Fig.57 This is a conceptual diagram for explaining another example of DMVR for determining MV.
[0091] Fig.58A This is a diagram showing an example of motion search in DMVR.
[0092] Fig.58B This is a flowchart showing an example of motion search in DMVR.
[0093] Fig.59 This is a flowchart showing an example of generating a predicted image.
[0094] Fig.60 This is a flowchart showing another example of generating a predicted image.
[0095] Fig.61 This is a flowchart for explaining an example of predicted image correction processing based on OBMC (overlapped block motion compensation).
[0096] Fig.62 This is a conceptual diagram for explaining an example of predicted image correction processing based on OBMC.
[0097] Fig.63 This is a diagram used to explain a model that assumes uniform linear motion.
[0098] Fig.64 This is a flowchart showing an example of inter-frame prediction based on BIO.
[0099] Fig.65 This is a diagram showing an example of the functional structure of an inter-frame prediction unit that performs inter-frame prediction based on BIO.
[0100] Fig.66A This is a diagram for explaining an example of a method of generating a predicted image using a brightness correction process based on LIC (local illumination compensation).
[0101] Fig.66B This is a flowchart showing an example of a method for generating a predicted image using a brightness correction process based on LIC.
[0102] Fig.67 It is a block diagram showing the functional structure of a decoding device according to an embodiment.
[0103] Fig.68 This is a block diagram showing an implementation example of a decoding device.
[0104] Fig.69 This is a flowchart showing an example of the overall decoding process of the decoding device.
[0105] Fig.70 This is a diagram showing the relationship between the division determination unit and other components.
[0106] Fig.71 This is a block diagram showing an example of the functional structure of an entropy decoding unit.
[0107] Fig.72 This is a diagram showing the CABAC process in the entropy decoding unit.
[0108] Fig.73This is a block diagram showing an example of the functional structure of the inverse quantization unit.
[0109] Fig.74 This is a flowchart showing an example of inverse quantization performed by the inverse quantization unit.
[0110] Fig.75 This is a flowchart showing an example of processing of the inverse transformation unit.
[0111] Fig.76 This is a flowchart showing another example of the processing of the inverse transformation unit.
[0112] Fig.77 This is a block diagram showing an example of the functional configuration of a loop filter unit.
[0113] Fig.78 This is a flowchart showing an example of processing performed by the prediction unit of the decoding device.
[0114] Fig.79 This is a flowchart showing another example of the processing performed by the prediction unit of the decoding device.
[0115] Fig.80A This is a flowchart showing a part of another example of the processing performed by the prediction unit of the decoding device.
[0116] Fig.80B This is a flowchart showing the remaining part of another example of the processing performed by the prediction unit of the decoding device.
[0117] Fig.81 This is a diagram showing an example of processing performed by the intra prediction unit of the decoding device.
[0118] Fig.82 This is a flowchart showing an example of MV derivation in the decoding device.
[0119] Fig.83 This is a flowchart showing another example of MV derivation in the decoding device.
[0120] Fig.84 This is a flowchart showing an example of inter prediction based on the normal inter mode in a decoding device.
[0121] Fig.85 This is a flowchart showing an example of inter-frame prediction based on the normal merge mode in a decoding device.
[0122] Fig.86 This is a flowchart showing an example of inter-frame prediction based on the FRUC mode in the decoding device.
[0123] Fig.87 This is a flowchart showing an example of inter-frame prediction based on the affine merge mode in a decoding device.
[0124] Fig.88 This is a flowchart showing an example of inter prediction based on the affine inter mode in a decoding device.
[0125] Fig.89 This is a flowchart showing an example of inter-frame prediction based on the triangular mode in a decoding device.
[0126] Fig.90 This is a flowchart showing an example of a DMVR-based motion search in a decoding device.
[0127] Fig.91 This is a flowchart showing a detailed example of DMVR-based motion search in a decoding device.
[0128] Fig.92 This is a flowchart showing an example of generation of a predicted image in a decoding device.
[0129] Fig.93 This is a flowchart showing another example of generation of a predicted image in a decoding device.
[0130] Fig.94 This is a flowchart showing an example of correction of a predicted image based on OBMC in a decoding device.
[0131] Fig.95 This is a flowchart showing an example of correction of a predicted image based on BIO in a decoding device.
[0132] Fig.96 This is a flowchart showing an example of correction of a predicted image based on LIC in a decoding device.
[0133] Fig.97 This is a conceptual diagram showing dual-single hybrid prediction when a reference block is partially located outside a reference picture.
[0134] Fig.98 This is a conceptual diagram showing multiple examples of geometric segmentation patterns.
[0135] Fig.99A This is a conceptual diagram showing an example in which uni-prediction is applied to both of two partitions determined according to a geometric partitioning pattern.
[0136] Fig.99B This is a conceptual diagram showing an example in which uniprediction is applied to one of two partitions and biprediction is applied to the other.
[0137] Fig.99C This is a conceptual diagram showing another example in which uniprediction is applied to one of two partitions and biprediction is applied to the other.
[0138] Fig.100AThis is a flowchart showing a portion of the operation of determining parameters of the geometric segmentation mode.
[0139] Fig.100B is a flowchart showing the remainder of the operations for determining the parameters of the geometric segmentation pattern.
[0140] Fig.101A This is a flowchart showing a part of the operation of generating a predicted image according to the parameters of the geometric segmentation pattern.
[0141] Fig.101B This is a flowchart showing the rest of the operation of generating a predicted image according to the parameters of the geometric segmentation pattern.
[0142] Fig.102 This is a conceptual diagram showing multiple examples of dual-single mixed prediction in a new partition partitioning mode different from the geometric partitioning mode.
[0143] Fig.103A This is a flowchart showing a portion of the operation of determining parameters for partition division.
[0144] Fig.103B is a flowchart showing the remainder of the operations for determining the parameters for partition splitting.
[0145] Fig.104A This is a conceptual diagram showing an example of weights applied to each pixel in two partitions determined by vertical division.
[0146] Fig.104B This is a conceptual diagram showing another example of weights applied to pixels in two partitions determined by vertical division.
[0147] Fig.104C This is a conceptual diagram showing still another example of the weight applied to each pixel of two partitions determined by vertical division.
[0148] Fig.104D This is a conceptual diagram showing still another example of the weight applied to each pixel of two partitions determined by vertical division.
[0149] Fig.105 This is a conceptual diagram showing an example in which bi-prediction is allowed only for the larger partition out of two partitions.
[0150] Fig.106 This is a flowchart showing the actions performed by the encoding device according to the embodiment.
[0151] Fig.107 This is a flowchart showing the operations performed by the decoding device according to the embodiment.
[0152] Fig.108 It is an overall structural diagram of the content supply system that realizes content distribution services.
[0153] Fig.109 This is a diagram showing an example of a display screen of a web page.
[0154] Fig.110 This is a diagram showing an example of a display screen of a web page.
[0155] Fig.111 The diagram shows an example of a smart phone.
[0156] Fig.112 This is a block diagram showing an example of the configuration of a smart phone. DETAILED DESCRIPTION
[0157] [Introduction]
[0158] In the encoding of images, the Geometric Partitioning Mode is sometimes used. The Geometric Partitioning Mode is also called GPM or GPM Mode. In the Geometric Partitioning Mode, two partitions of a block are determined by a partition line. And, a prediction for the block is generated by combining a prediction for one partition with a prediction for another partition. As a result, the prediction accuracy may be improved.
[0159] In the geometric partitioning mode, a single prediction is used for the prediction of the partition. By using a dual prediction for the prediction of the partition, the prediction accuracy may be further improved. However, by using a dual prediction, the number of instances of reference images and motion vectors used in the prediction increases, the processing becomes complicated, and the processing amount may increase.
[0160] Therefore, the decoding method of Example 1 is a decoding method for decoding a current block of an image from a bitstream, decoding a prediction mode indicator indicating whether dual-single mixed prediction should be performed on the current block from the bitstream; decoding two instances of motion information for the current block from the bitstream; and when the prediction mode indicator indicates that the dual-single mixed prediction should be performed on the current block, (i) using one of the two instances, performing a first prediction as a single prediction on a first partition as a part of the current block, and (ii) using both of the two instances, performing a second prediction as a dual prediction on a second partition as a part of the current block.
[0161] Thus, it is possible to generate a predicted image of the current block by combining a bi-predicted partition with a uni-predicted partition. Therefore, it is possible to improve the accuracy of the predicted image of the current block. In addition, it is possible to suppress the increase in the number of instances used in the prediction by combining a uni-prediction using one of the two instances with a bi-prediction using both. Therefore, it is possible to suppress the complication of the processing and the increase in the processing amount.
[0162] Furthermore, there are cases where the execution of dual-single hybrid prediction can be reliably controlled according to the prediction mode indicator indicating whether dual-single hybrid prediction should be performed on the current block. Therefore, there are cases where the predicted image of the current block can be reliably generated.
[0163] In addition, the decoding method of Example 2 may be that in the decoding method of Example 1, the two partitions, namely the first partition and the second partition, in the current block are determined according to a geometric partitioning pattern; the current block is divided into the two partitions along a boundary; and the boundary is a line separating the two partitions.
[0164] Thus, in the geometric partitioning mode, it is possible to combine a bi-predicted partition with a uni-predicted partition to generate a predicted image of the current block. Therefore, in the geometric partitioning mode, it is possible to apply bi-prediction to the prediction of the partition while suppressing the complexity of the processing and the increase in the processing amount.
[0165] In addition, the decoding method of Example 3 may be that, in the decoding method of Example 2, a partition indicator is also decoded from the above-mentioned bit stream, and the partition indicator indicates which of the above-mentioned two partitions is the above-mentioned first partition and which of the above-mentioned two partitions is at least one of the above-mentioned second partition.
[0166] As a result, there are cases where bi-predicted partitions and uni-predicted partitions can be reliably specified according to the partition indicator in the bitstream.
[0167] In addition, the decoding method of Example 4 may be that, in the decoding method of Example 1 or Example 2, a partition indicator is further decoded from the bit stream, the partition indicator indicating (i) the position and direction of a boundary, the boundary being the boundary between two partitions, namely the first partition and the second partition in the current block, and being the line separating the two partitions, and (ii) which of the two instances is used for the prediction of the first partition.
[0168] Thus, the configuration of the first partition and the second partition may be reliably specified according to the partition indicator in the bitstream. Also, the instance used for uni-prediction of the two instances may be reliably specified according to the partition indicator in the bitstream.
[0169] Furthermore, the decoding method of Example 5 may be that, in the decoding method of Example 4, the partition indicator further indicates which of the two partitions is the first partition and which of the two partitions is at least one of the second partition.
[0170] As a result, there are cases where bi-predicted partitions and uni-predicted partitions can be reliably specified according to the partition indicator in the bitstream.
[0171] In addition, the decoding method of Example 6 may be, in the decoding method of Example 2 or Example 4, further setting the above-mentioned two partitions as the above-mentioned first partition or the above-mentioned second partition based on at least one of the size and shape of at least one of the above-mentioned two partitions.
[0172] This makes it possible to adaptively specify bi-predictive partitions and uni-predictive partitions according to the characteristics of the partitions.
[0173] In addition, the decoding method of Example 7 may be that in the decoding method of any one of Examples 1 to 6, in the above-mentioned first prediction and the above-mentioned second prediction, one of the above-mentioned two instances is used to perform the above-mentioned first prediction of the above-mentioned current block; the other of the above-mentioned two instances is used to perform the above-mentioned third prediction as a single prediction of the above-mentioned current block; a weighted linear combination of the above-mentioned first prediction and the above-mentioned third prediction is performed; for at least a part of the above-mentioned first partition, the ratio of the weights of the above-mentioned first prediction is set to 1, and the ratio of the weights of the above-mentioned third prediction is set to 0; for at least a part of the above-mentioned second partition, the ratio of the weights of the above-mentioned third prediction is set to the maximum in the above-mentioned current block and less than 1, and the ratio of the weights of the above-mentioned first prediction is set to the difference between the ratio of the weights of the above-mentioned third prediction and 1.
[0174] This makes it possible to reliably control bi-prediction and uni-prediction according to the setting of the weight ratio, thereby suppressing the complication of processing and the increase in the amount of processing.
[0175] In addition, the decoding method of Example 8 may be that, in the decoding method of Example 7, for the at least a portion of the second partition, the ratio of the weight of the third prediction is set to 0.5, and the ratio of the weight of the first prediction is set to 0.5.
[0176] As a result, in at least a part of the second partition, the first prediction and the third prediction may be combined in the same proportion with good balance.
[0177] In addition, the decoding method of Example 9 may be that in the decoding method of Example 7 or Example 8, for a boundary area, the ratio of the weight of the first prediction and the ratio of the weight of the third prediction are respectively set in a manner that gradually changes from the ratio set for the at least a part of the first partition to the ratio set for the at least a part of the second partition, and the boundary area is an area between the at least a part of the first partition and the at least a part of the second partition, and is an area including the boundary between the first partition and the second partition.
[0178] This makes it possible to switch between bi-prediction and uni-prediction smoothly in space. Therefore, it is possible to generate a natural predicted image without any discomfort. This makes it possible to improve the accuracy of prediction.
[0179] In addition, the decoding method of Example 10 may be, in the decoding method of any one of Examples 1 to 9, further, for the first sub-block belonging to at least a part of the above-mentioned first partition, an instance of one of the above-mentioned two instances used in the prediction of the above-mentioned first partition is stored; for the second sub-block belonging to at least one of the boundary area and the above-mentioned at least a part of the above-mentioned second partition, both of the above-mentioned two instances are stored, and the above-mentioned boundary area is an area between the above-mentioned at least a part of the above-mentioned first partition and at least a part of the above-mentioned second partition, and is an area including the boundary between the above-mentioned first partition and the above-mentioned second partition.
[0180] Thus, two instances for bi-prediction can be associated with sub-blocks belonging to a region to which bi-prediction is applied. Also, one instance for uni-prediction can be associated with sub-blocks belonging to a region to which uni-prediction is applied. Thus, reliable motion information corresponding to each region can be stored for reference.
[0181] In addition, the decoding method of Example 11 may be, in the decoding method of any one of Examples 1 to 10, the dual-single mixed prediction may not be performed when at least one of the first partition and the second partition has a prescribed shape.
[0182] This may allow control not to apply dual-single mixed prediction to a predetermined shape that is not suitable for dual-single mixed prediction. Therefore, inefficient operation may be suppressed.
[0183] In addition, the decoding method of Example 12 may be that, in the decoding method of Example 11, the above-mentioned prescribed shape is a shape that can be realized by a normal block division different from the partition division in the above-mentioned dual-single mixed prediction.
[0184] This makes it possible to perform control without applying dual-single hybrid prediction to a predetermined shape obtained by normal block division. Therefore, it is possible to suppress inefficient operation.
[0185] In addition, the encoding method of Example 13 is an encoding method for encoding a current block of an image into a bit stream. In this encoding method, when a dual-single mixed prediction should be performed on the above-mentioned current block, (i) one of the two instances of motion information used for the above-mentioned current block is used to perform a first prediction as a single prediction on the first partition as a part of the above-mentioned current block, and (ii) both of the above-mentioned two instances are used to perform a second prediction as a dual prediction on the second partition as a part of the above-mentioned current block; a prediction mode indicator indicating whether the above-mentioned dual-single mixed prediction should be performed on the above-mentioned current block is encoded into the above-mentioned bit stream; and the above-mentioned two instances are encoded into the above-mentioned bit stream.
[0186] Thus, it is possible to generate a predicted image of the current block by combining a bi-predicted partition with a uni-predicted partition. Therefore, it is possible to improve the accuracy of the predicted image of the current block. In addition, it is possible to suppress the increase in the number of instances used in the prediction by combining a uni-prediction using one of the two instances with a bi-prediction using both. Therefore, it is possible to suppress the complication of the processing and the increase in the processing amount.
[0187] Furthermore, there are cases where the execution of dual-single hybrid prediction can be reliably controlled according to the prediction mode indicator indicating whether dual-single hybrid prediction should be performed on the current block. Therefore, there are cases where the predicted image of the current block can be reliably generated.
[0188] In addition, the encoding method of Example 14 may be that in the encoding method of Example 13, the two partitions, namely the first partition and the second partition, in the current block are determined according to a geometric partitioning pattern; the current block is divided into the two partitions along a boundary; and the boundary is a line separating the two partitions.
[0189] Thus, in the geometric partitioning mode, it is possible to combine a bi-predicted partition with a uni-predicted partition to generate a predicted image of the current block. Therefore, in the geometric partitioning mode, it is possible to apply bi-prediction to the prediction of the partition while suppressing the complexity of the processing and the increase in the processing amount.
[0190] In addition, the encoding method of Example 15 may be that, in the encoding method of Example 14, a partition indicator is further encoded into the above-mentioned bit stream, and the above-mentioned partition indicator indicates which of the above-mentioned two partitions is the above-mentioned first partition and which of the above-mentioned two partitions is at least one of the above-mentioned second partition.
[0191] As a result, there are cases where bi-predicted partitions and uni-predicted partitions can be reliably specified according to the partition indicator in the bitstream.
[0192] In addition, the encoding method of Example 16 may be that, in the encoding method of Example 13 or Example 14, a partition indicator is further encoded into the above-mentioned bit stream, wherein the above-mentioned partition indicator indicates (i) the position and direction of the boundary, the above-mentioned boundary is the boundary between the above-mentioned first partition and the above-mentioned second partition in the above-mentioned current block, and is the line separating the above-mentioned two partitions, and (ii) which of the above-mentioned two instances is used for the prediction of the above-mentioned first partition.
[0193] Thus, the configuration of the first partition and the second partition may be reliably specified according to the partition indicator in the bitstream. Also, the instance used for uni-prediction of the two instances may be reliably specified according to the partition indicator in the bitstream.
[0194] Furthermore, the encoding method of Example 17 may be that, in the encoding method of Example 16, the partition indicator further indicates which of the two partitions is the first partition and which of the two partitions is at least one of the second partition.
[0195] As a result, there are cases where bi-predicted partitions and uni-predicted partitions can be reliably specified according to the partition indicator in the bitstream.
[0196] In addition, the encoding method of Example 18 may be that, in the encoding method of Example 14 or Example 16, the two partitions are respectively set as the first partition or the second partition based on at least one of the size and shape of at least one of the two partitions.
[0197] This makes it possible to adaptively specify bi-predictive partitions and uni-predictive partitions according to the characteristics of the partitions.
[0198] In addition, the encoding method of Example 19 can be that in the encoding method of any one of Examples 13 to 18, in the above-mentioned first prediction and the above-mentioned second prediction, one of the above-mentioned two instances is used to perform the above-mentioned first prediction of the above-mentioned current block; the other of the above-mentioned two instances is used to perform the above-mentioned third prediction as a single prediction of the above-mentioned current block; a weighted linear combination of the above-mentioned first prediction and the above-mentioned third prediction is performed; for at least a part of the above-mentioned first partition, the ratio of the weight of the above-mentioned first prediction is set to 1, and the ratio of the weight of the above-mentioned third prediction is set to 0; for at least a part of the above-mentioned second partition, the ratio of the weight of the above-mentioned third prediction is set to the maximum in the above-mentioned current block and less than 1, and the ratio of the weight of the above-mentioned first prediction is set to the difference between the ratio of the weight of the above-mentioned third prediction and 1.
[0199] This makes it possible to reliably control bi-prediction and uni-prediction according to the setting of the weight ratio, thereby suppressing the complication of processing and the increase in the amount of processing.
[0200] In addition, the encoding method of Example 20 may be that, in the encoding method of Example 19, for the above-mentioned at least a part of the above-mentioned second partition, the ratio of the weight of the above-mentioned third prediction is set to 0.5, and the ratio of the weight of the above-mentioned first prediction is set to 0.5.
[0201] As a result, in at least a part of the second partition, the first prediction and the third prediction may be combined in the same proportion with good balance.
[0202] In addition, the encoding method of Example 21 may be that in the encoding method of Example 19 or Example 20, for a boundary area, the ratio of the weight of the first prediction and the ratio of the weight of the third prediction are respectively set in a manner that gradually changes from the ratio set for the at least a part of the first partition to the ratio set for the at least a part of the second partition, and the boundary area is an area between the at least a part of the first partition and the at least a part of the second partition, and is an area including the boundary between the first partition and the second partition.
[0203] This makes it possible to switch between bi-prediction and uni-prediction smoothly in space. Therefore, it is possible to generate a natural predicted image without any discomfort. This makes it possible to improve the accuracy of prediction.
[0204] In addition, the encoding method of Example 22 may be, in the encoding methods of Examples 13 to 21, further, for the first sub-block belonging to at least a part of the above-mentioned first partition, an instance of one of the above-mentioned two instances used in the prediction of the above-mentioned first partition is stored; for the second sub-block belonging to at least one of the boundary area and the above-mentioned at least a part of the above-mentioned second partition, both of the above-mentioned two instances are stored, and the above-mentioned boundary area is an area between the above-mentioned at least a part of the above-mentioned first partition and at least a part of the above-mentioned second partition, and is an area including the boundary between the above-mentioned first partition and the above-mentioned second partition.
[0205] Thus, two instances for bi-prediction can be associated with sub-blocks belonging to a region to which bi-prediction is applied. Also, one instance for uni-prediction can be associated with sub-blocks belonging to a region to which uni-prediction is applied. Thus, reliable motion information corresponding to each region can be stored for reference.
[0206] In addition, the encoding method of Example 23 may be that in the encoding method of any one of Examples 13 to 22, when at least one of the first partition and the second partition has a prescribed shape, the dual-single mixed prediction is not performed.
[0207] This may allow control not to apply dual-single mixed prediction to a predetermined shape that is not suitable for dual-single mixed prediction. Therefore, inefficient operation may be suppressed.
[0208] In addition, the encoding method of Example 24 may be that, in the encoding method of Example 23, the above-mentioned prescribed shape is a shape that can be realized by a normal block division different from the partition division in the above-mentioned dual-single mixed prediction.
[0209] This makes it possible to perform control without applying dual-single hybrid prediction to a predetermined shape obtained by normal block division. Therefore, it is possible to suppress inefficient operation.
[0210] In addition, the non-temporary computer-readable recording medium of Example 25 is a non-temporary computer-readable recording medium for a computer, wherein a bit stream is stored for causing the computer to perform a decoding process for decoding a current block using a motion vector; in the above decoding process, a prediction mode indicator indicating whether a dual-single mixed prediction should be performed on the above current block is decoded from the above bit stream; two instances of motion information for the above current block are decoded from the above bit stream; when the above prediction mode indicator indicates that the above dual-single mixed prediction should be performed on the above current block, (i) one of the above two instances is used to perform a first prediction as a single prediction on the first partition as a part of the above current block, and (ii) both of the above two instances are used to perform a second prediction as a dual prediction on the second partition as a part of the above current block.
[0211] Therefore, there is a case where the above-mentioned decoding method can be executed via the recording medium.
[0212] In addition, the decoding device of Example 26 is a decoding device that decodes a current block of an image from a bit stream, comprising: a circuit; and a memory connected to the above-mentioned circuit; when the above-mentioned circuit is in operation, it decodes a prediction mode indicator indicating whether a dual-single mixed prediction should be performed on the above-mentioned current block from the above-mentioned bit stream; decodes two instances of motion information used for the above-mentioned current block from the above-mentioned bit stream; when the above-mentioned prediction mode indicator indicates that the above-mentioned dual-single mixed prediction should be performed on the above-mentioned current block, (i) using one of the above-mentioned two instances, a first prediction as a single prediction is performed on the first partition as a part of the above-mentioned current block, and (ii) using both of the above-mentioned two instances, a second prediction as a dual prediction is performed on the second partition as a part of the above-mentioned current block.
[0213] As a result, there are cases where the above-mentioned decoding method can be executed by the decoding device.
[0214] In addition, the encoding device of Example 27 is an encoding device that encodes a current block of an image into a bit stream, comprising: a circuit; and a memory connected to the above-mentioned circuit; when the above-mentioned circuit is in operation, when the above-mentioned current block should be subjected to dual-single mixed prediction, (i) using one of the two instances of motion information used for the above-mentioned current block, a first prediction as a single prediction is performed on the first partition as a part of the above-mentioned current block, and (ii) using both of the above-mentioned two instances, a second prediction as a dual prediction is performed on the second partition as a part of the above-mentioned current block; a prediction mode indicator indicating whether the above-mentioned dual-single mixed prediction should be performed on the above-mentioned current block is encoded into the above-mentioned bit stream; and the above-mentioned two instances are encoded into the above-mentioned bit stream.
[0215] Therefore, there is a case where the above-mentioned encoding method can be executed by the encoding device.
[0216] Furthermore, the decoding device of Example 28 includes an input unit, an entropy decoding unit, an inverse quantization unit, an inverse transformation unit, an intra-frame prediction unit, an inter-frame prediction unit, a loop filter unit, and an output unit.
[0217] The input unit inputs a coded bit stream. The entropy decoding unit applies variable length decoding to the coded bit stream to derive quantized coefficients. The inverse quantization unit inversely quantizes the quantized coefficients to derive transform coefficients. The inverse transform unit inversely transforms the transform coefficients to derive prediction errors.
[0218] The intra prediction unit generates a prediction signal for a current block in the current picture using reference pixels in the current picture. The inter prediction unit generates a prediction signal for a current block in the current picture using reference blocks in a reference picture different from the current picture.
[0219] The loop filter unit applies filtering to a reconstructed block of a current block included in the current picture. And, the output unit outputs the current picture.
[0220] Furthermore, the entropy decoding unit is a decoding method for decoding a current block of an image from a bitstream in action, decoding a prediction mode indicator indicating whether a dual-single mixed prediction should be performed on the current block from the bitstream; decoding two instances of motion information for the current block from the bitstream; and when the prediction mode indicator indicates that the dual-single mixed prediction should be performed on the current block, (i) using one of the two instances to perform a first prediction as a single prediction on a first partition as a part of the current block, and (ii) using both of the two instances to perform a second prediction as a dual prediction on a second partition as a part of the current block.
[0221] Furthermore, the encoding device of Example 29 includes an input unit, a segmentation unit, an intra-frame prediction unit, an inter-frame prediction unit, a loop filter unit, a transformation unit, a quantization unit, an entropy encoding unit, and an output unit.
[0222] The input unit inputs a current picture. The dividing unit divides the current picture into a plurality of blocks.
[0223] The intra prediction unit generates a prediction signal of a current block included in the current picture using reference pixels included in the current picture. The inter prediction unit generates a prediction signal of a current block included in the current picture using reference blocks included in a reference picture different from the current picture. The loop filter unit applies filtering to a reconstructed block of the current block included in the current picture.
[0224] The transform unit transforms the prediction error between the original signal of the current block included in the current picture and the prediction signal generated by the intra prediction unit or the inter prediction unit to generate a transform coefficient. The quantization unit quantizes the transform coefficient to generate a quantized coefficient. The entropy coding unit applies variable length coding to the quantized coefficient to generate a coded bit stream. Furthermore, the output unit outputs the coded bit stream including the quantized coefficient to which the variable length coding is applied and the control information.
[0225] Furthermore, the entropy coding unit, during operation, when dual-single mixed prediction should be performed on the current block, (i) uses one of the two instances of motion information for the current block to perform a first prediction as a single prediction on the first partition as a part of the current block, and (ii) uses both of the two instances to perform a second prediction as a dual prediction on the second partition as a part of the current block; encodes a prediction mode indicator indicating whether the dual-single mixed prediction should be performed on the current block into the bitstream; and encodes the two instances into the bitstream.
[0226] Furthermore, these general or specific technical solutions may also be implemented by systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.
[0227] [Definition of terms]
[0228] Each term may be defined as follows as an example.
[0229] (1) Image
[0230] A unit of data composed of a collection of pixels, including a picture or a block smaller than a picture, and including still images in addition to moving images.
[0231] (2) Pictures
[0232] A processing unit of an image composed of a collection of pixels, and may also be called a frame or a field.
[0233] (3) Block
[0234] It is a processing unit that includes a set of a specific number of pixels, as exemplified in the following examples, regardless of the name. In addition, regardless of the shape, for example, it includes not only a rectangle composed of M×N pixels, a square composed of M×M pixels, but also a triangle, a circle, and other shapes.
[0235] (Block example)
[0236] Slice / tile / brick
[0237] CTU / super block / basic partition unit
[0238] VPDU / Hardware Processing Division Unit
[0239] CU / Processing block unit / Prediction block unit (PU) / Orthogonal transform block unit (TU) / Unit
[0240] ·Sub-block
[0241] (4) Pixels / Sample
[0242] It is the smallest unit of a point that constitutes an image, and includes not only pixels at integer positions but also pixels at decimal positions generated based on pixels at integer positions.
[0243] (5) Pixel value / sample value
[0244] It is an inherent value of a pixel, including not only brightness value, color difference value, RGB grayscale (gradation), but also depth value (depth value) or binary value of 0 and 1.
[0245] (6) Logo
[0246] In addition to 1 bit, multiple bits are also included, for example, a parameter or index of 2 bits or more may be used. In addition, not only binary values but also multiple values including numbers in other bases may be used.
[0247] (7) Signal
[0248] It is marked and encoded for the purpose of transmitting information. In addition to discrete digital signals, it also includes analog signals that take continuous values.
[0249] (8) Stream / Bitstream
[0250] Refers to a data string of digital data or a stream of digital data. In addition to a single stream, a stream / bit stream can also be divided into multiple layers and consist of multiple streams. In addition, in addition to the case of transmission via serial communication via a single transmission path, it also includes the case of transmission via packet communication via multiple transmission paths.
[0251] (9) Difference / Differential
[0252] In the case of scalars, in addition to simple difference (x-y), any operation involving difference is sufficient, including the absolute value of the difference (|x-y|), square difference (x^2-y^2), square root of the difference (√(x-y)), weighted difference (ax-by: a, b are constants), and offset difference (x-y+a: a is the offset).
[0253] (10) and
[0254] In the case of scalars, in addition to the simple sum (x+y), any operation involving the sum is sufficient, including the absolute value of the sum (|x+y|), the square sum (x^2+y^2), the square root of the sum (√(x+y)), the weighted sum (ax+by: a, b are constants), and the offset sum (x+y+a: a is the offset).
[0255] (11) based on
[0256] This includes the case where factors other than the factors to be based on are taken into consideration. In addition, in addition to the case where a direct result is obtained, the case where a result is obtained via an intermediate result is also included.
[0257] (12) Use (used, using)
[0258] This includes the case where factors other than the factors to be used are taken into consideration. Furthermore, in addition to the case where a direct result is obtained, the case where a result is obtained via an intermediate result is also included.
[0259] (13) prohibit (prohibit, forbid)
[0260] It can be called not allowed. In addition, not being prohibited or allowed does not necessarily mean an obligation.
[0261] (14) Limit (limit, restriction / restrict / restricted)
[0262] It can be called "not allowed". In addition, not being prohibited or being allowed does not necessarily mean an obligation. Moreover, it is sufficient to prohibit part of it in terms of quantity or quality, and this also includes the case of a complete prohibition.
[0263] (15) Chromatic aberration (chroma)
[0264] An adjective denoted by the symbols Cb and Cr that specifies that a sample array or a single sample represents one of two color difference signals associated with a primary color. The term chrominance may be used instead of the term chroma.
[0265] (16) Brightness (luma)
[0266] An adjective that specifies that a sample array or a single sample represents a monochromatic signal associated with a primary color and is represented by a mark or subscript Y or L. The term luminance may be used instead of the term luma.
[0267] [Explanation of the Record]
[0268] In the drawings, the same reference numerals denote the same or similar components. In addition, the sizes and relative positions of the components in the drawings are not necessarily drawn to scale.
[0269] The following embodiments are specifically described with reference to the accompanying drawings. In addition, the embodiments described below are all used to represent general or specific examples. The numerical values, shapes, materials, components, configuration positions and connection forms of components, steps, relationships and sequences of steps, etc. shown in the following embodiments are used as examples and are not intended to limit the claims.
[0270] The following describes implementations of the encoding device and the decoding device. The implementations are examples of encoding devices and decoding devices to which the processing and / or structures described in each technical solution of the present disclosure can be applied. The processing and / or structure can also be implemented in encoding devices and decoding devices different from the implementations. For example, with respect to the processing and / or structure applied to the implementations, for example, one of the following can also be implemented.
[0271] (1) Any of the plurality of components of the encoding device or decoding device of the embodiment described in each technical aspect of the present disclosure may be replaced with or combined with another component described in any of the technical aspects of the present disclosure.
[0272] (2) In the coding device or decoding device of the embodiment, any changes such as addition, replacement, deletion, etc. of functions or processes performed by some of the multiple components of the coding device or decoding device may be made. For example, a certain function or process may be replaced by another function or process described in any of the technical solutions of the present disclosure or combined with the other function or process.
[0273] (3) In the method implemented by the encoding device or decoding device of the embodiment, any changes such as addition, replacement, and deletion may be made to a part of the multiple processes included in the method. For example, a certain process in the method may be replaced by another process described in one of the technical solutions of the present disclosure or combined with the other process.
[0274] (4) Some of the multiple components constituting the encoding device or decoding device of the implementation method may be combined with the components described in any of the technical solutions of the present disclosure, or may be combined with some of the components having the functions described in any of the technical solutions of the present disclosure, or may be combined with some of the components implementing the processing implemented by the components described in the technical solutions of the present disclosure.
[0275] (5) A constituent element having a part of the functions of the encoding device or decoding device of an embodiment, or a constituent element that implements a part of the processing of the encoding device or decoding device of an embodiment may be combined with or replaced with a constituent element described in any of the technical solutions of the present disclosure, a constituent element having a part of the functions described in any of the technical solutions of the present disclosure, or a constituent element that implements a part of the processing described in any of the technical solutions of the present disclosure.
[0276] (6) In the method implemented by the encoding device or decoding device of the embodiment, one of the multiple processes included in the method may be replaced by a process described in any of the technical solutions of the present disclosure or a similar process, or a combination thereof.
[0277] (7) Some of the multiple processes included in the method implemented by the encoding device or the decoding device of the embodiment may be combined with the process described in any of the technical aspects of the present disclosure.
[0278] (8) The implementation of the processing and / or structure described in each technical solution of the present disclosure is not limited to the encoding device or decoding device of the implementation. For example, the processing and / or structure may also be implemented in a device used for a purpose different from the motion picture encoding or motion picture decoding disclosed in the implementation.
[0279] [System Structure]
[0280] Figure 1 It is a schematic diagram showing an example of the configuration of the transmission system according to the present embodiment.
[0281] The transmission system Trs is a system that transmits a stream generated by encoding an image and decodes the transmitted stream. Such a transmission system Trs is, for example, Figure 1 As shown, it includes an encoding device 100, a network Nw and a decoding device 200.
[0282] An image is input to the encoding device 100. The encoding device 100 generates a stream by encoding the input image, and outputs the stream to the network Nw. The stream includes, for example, the encoded image and control information for decoding the encoded image. The image is compressed by the encoding.
[0283] In addition, the original image before being encoded that is input to the encoding device 100 is also called the original image, the original signal, or the original sample. In addition, the image can be a moving image or a still image. In addition, the image is a higher-level concept of a sequence, a picture, a block, etc., and is not limited by spatial and temporal regions unless otherwise specified. In addition, an image is composed of an arrangement of pixels or pixel values, and the signal or pixel value representing the image is also called a sample. In addition, the stream can also be referred to as a bit stream, a coded bit stream, a compressed bit stream, or a coded signal. Furthermore, the encoding device can also be referred to as an image encoding device or a moving image encoding device, and the encoding method of the encoding device 100 can also be referred to as a coding method, an image coding method, or a moving image coding method.
[0284] The network Nw transmits the stream generated by the encoding device 100 to the decoding device 200. The network Nw may be the Internet, a wide area network (WAN), a small-scale network (LAN), or a combination thereof. The network Nw is not necessarily limited to a two-way communication network, and may also be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. In addition, the network Nw may also be replaced by a storage medium such as a DVD (Digital Versatile Disc) or a BD (Blu-Ray Disc (registered trademark)) that records the stream.
[0285] The decoding device 200 generates a decoded image, for example, which is an uncompressed image, by decoding the stream transmitted by the network Nw. For example, the decoding device decodes the stream according to a decoding method corresponding to the encoding method of the encoding device 100 .
[0286] In addition, the decoding device may also be referred to as an image decoding device or a moving image decoding device, and the decoding method of the decoding device 200 may also be referred to as a decoding method, an image decoding method, or a moving image decoding method.
[0287] [Data structure]
[0288] Figure 2 is a diagram showing an example of a hierarchical structure of data in a stream. The stream includes, for example, a video sequence. The video sequence is, for example, Figure 2 As shown in (a), it includes VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), SEI (Supplemental Enhancement Information) and multiple pictures.
[0289] In a moving picture composed of a plurality of layers, the VPS includes coding parameters common to the plurality of layers and coding parameters associated with the plurality of layers or each layer included in the moving picture.
[0290] The SPS includes parameters used for the sequence, that is, coding parameters that the decoding device 200 refers to in order to decode the sequence. For example, the coding parameters may represent the width or height of the picture. In addition, there may be multiple SPSs.
[0291] The PPS includes parameters used for the picture, that is, coding parameters that the decoding device 200 refers to in order to decode each picture in the sequence. For example, the coding parameters may include a reference value of a quantization width used in decoding the picture and a flag indicating the application of weighted prediction. In addition, there may be multiple PPSs. In addition, there are cases where the SPS and PPS are simply referred to as parameter sets.
[0292] like Figure 2 As shown in (b) of FIG. 1 , a picture may include a picture header and one or more slices. The picture header includes coding parameters that the decoding device 200 refers to in order to decode the one or more slices.
[0293] like Figure 2 As shown in (c) of FIG. 1 , a slice includes a slice header and one or more tiles. The slice header includes coding parameters that the decoding device 200 refers to in order to decode the one or more tiles.
[0294] like Figure 2 As shown in (d), a brick includes one or more CTUs (Coding Tree Unit).
[0295] In addition, a picture may not include slices, but may include tile groups instead of the slices. In this case, the tile group includes one or more tiles. In addition, a tile may also include a slice.
[0296] CTU is also called super block or basic partition unit. Figure 2 As shown in (e) of FIG. 1 , a CTU header and one or more CUs (Coding Units) are included. The CTU header includes coding parameters that the decoding device 200 refers to in order to decode the one or more CUs.
[0297] A CU can be split into multiple smaller CUs. Figure 2 As shown in (f), the CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information used to predict the CU, and the residual coefficient information is information representing the prediction residual described later. In addition, the CU is basically the same as the PU (Prediction Unit) and the TU (Transform Unit), but, for example, in the SBT described later, multiple TUs smaller than the CU may be included. In addition, the CU can be processed according to each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. The VPDU is a fixed unit that can be processed in one stage when pipeline processing is performed in hardware, for example.
[0298] Alternatively, a stream may not have Figure 2The layer of a part of the layers shown. In addition, the order of these layers may be changed, and a layer may be replaced by another layer. In addition, a picture that is the object of processing by a device such as the encoding device 100 or the decoding device 200 at the current time point is called a current picture. If the processing is encoding, the current picture is synonymous with the encoding object picture, and if the processing is decoding, the current picture is synonymous with the decoding object picture. In addition, a block such as a CU or a CU that is the object of processing by a device such as the encoding device 100 or the decoding device 200 at the current time point is called a current block. If the processing is encoding, the current block is synonymous with the encoding object block, and if the processing is decoding, the current block is synonymous with the decoding object block.
[0299] [Composition of slices / tiles of the image]
[0300] In order to decode pictures in parallel, pictures may be constructed in slice units or tile units.
[0301] A slice is a basic coding unit constituting a picture. A picture is composed of, for example, one or more slices. In addition, a slice is composed of one or more continuous CTUs.
[0302] Figure 3 This is a diagram showing an example of the structure of a slice. For example, a picture contains 11×8 CTUs and is divided into 4 slices (slices 1 to 4). Slice 1 is composed of 16 CTUs, slice 2 is composed of 21 CTUs, slice 3 is composed of 29 CTUs, and slice 4 is composed of 22 CTUs. Here, each CTU in the picture belongs to a certain slice. The shape of the slice is the form in which the picture is divided in the horizontal direction. The boundary of the slice does not need to be the end of the picture, and it can be anywhere in the boundary of the CTU in the picture. The processing order (encoding order or decoding order) of the CTU in the slice is, for example, a raster scan order. In addition, a slice includes a slice header and encoded data. The slice header can record the characteristics of the slice, such as the CTU address at the beginning of the slice and the slice type.
[0303] A tile is a unit of a rectangular area constituting a picture, and a number called TileId can be assigned to each tile in a raster scan order.
[0304] Figure 4is a diagram showing an example of a tile structure. For example, a picture includes 11×8 CTUs and is divided into four rectangular area tiles (tiles 1-4). When tiles are used, the processing order of CTUs is changed compared to when tiles are not used. When tiles are not used, multiple CTUs in a picture are processed, for example, in raster scan order. When tiles are used, at least one CTU in each of the multiple tiles is processed, for example, in raster scan order. For example, Figure 4 As shown, the processing order of multiple CTUs included in tile 1 is from the left end of the 1st column of tile 1 to the right end of the 1st column of tile 1, and then from the left end of the 2nd column of tile 1 to the right end of the 2nd column of tile 1.
[0305] In addition, one tile may include one or more slices, and one slice may include one or more tiles.
[0306] In addition, a picture may also be composed of tile set units. A tile set may include one or more tile groups, or may include one or more tiles. A picture may also be composed of only one of a tile set, a tile group, and a tile. For example, the order in which a plurality of tiles are scanned in raster order in each tile set is set as the basic encoding order of the tiles. A collection of one or more tiles in a continuous basic encoding order in each tile set is set as a tile group. Such a picture may also be obtained by the segmentation unit 102 described later (see Figure 7 )constitute.
[0307] [Scalable Coding]
[0308] Figure 5 and Figure 6 This is a diagram showing an example of the structure of a scalable stream.
[0309] The encoding device 100 may be as follows Figure 5As shown, by dividing a plurality of pictures into a plurality of layers and encoding them, a temporally / spatially scalable stream is generated. For example, the encoding device 100 realizes scalability in which an enhancement layer exists above the base layer by encoding the pictures according to each layer. Such encoding of each picture is called scalable encoding. Thus, the decoding device 200 can switch the image quality of the image displayed by decoding the stream. That is, the decoding device 200 determines which layer to decode according to internal factors such as its own performance and external factors such as the state of the communication band. As a result, the decoding device 200 can freely switch the same content to low-resolution content and high-resolution content for decoding. For example, the user of the stream uses a smart phone to watch the moving image of the stream halfway while on the move, and uses an Internet TV or other device to watch the subsequent part of the moving image after returning home. In addition, each of the above-mentioned smart phones and devices is equipped with a decoding device 200 with the same or different performance. In this case, if the device decodes to the upper layer in the stream, the user can watch high-quality moving images after returning home. This eliminates the need for the encoding device 100 to generate a plurality of streams having the same content but different image qualities, thereby reducing the processing load.
[0310] Furthermore, the enhancement layer may include meta information such as statistical information based on the image. The decoding device 200 can generate a high-definition moving image by making the picture of the base layer super-resolution based on the meta information. Super-resolution can be either an increase in the SN ratio or an increase in resolution at the same resolution. The meta information may include information for determining linear or nonlinear filter coefficients used in super-resolution processing, or information for determining parameter values in filtering processing, machine learning, or least squares calculation used in super-resolution processing.
[0311] Alternatively, the picture may be divided into tiles according to the meaning of each object in the picture. In this case, the decoding device 200 can decode only a part of the area in the picture by selecting the tile to be decoded. In addition, the attributes of the object (people, cars, balls, etc.) and the position in the picture (coordinate position in the same picture, etc.) may be saved as meta information. In this case, the decoding device 200 can determine the position of the desired object based on the meta information and determine the tile containing the object. For example, Figure 6 As shown, the meta information is stored using a data storage structure different from the image data, such as SEI in HEVC. The meta information indicates, for example, the position, size, or color of the main object.
[0312] Alternatively, the meta information may be stored in units consisting of a plurality of pictures, such as a stream, a sequence, or a random access unit. Thus, the decoding device 200 can obtain the time when a specific person appears in a moving image, and by using the time and information of the picture unit, can identify the picture where the object exists and the position of the object in the picture.
[0313] [Encoding device]
[0314] Next, the encoding device 100 according to the embodiment will be described. Figure 7 1 is a block diagram showing an example of a functional configuration of the encoding device 100 according to the embodiment. The encoding device 100 encodes an image in units of blocks.
[0315] like Figure 7 As shown, the encoding device 100 is a device for encoding an image in units of blocks, and includes a segmentation unit 102, a subtraction unit 104, a transformation unit 106, a quantization unit 108, an entropy coding unit 110, an inverse quantization unit 112, an inverse transformation unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. In addition, the intra-frame prediction unit 124 and the inter-frame prediction unit 126 are respectively configured as a part of a prediction processing unit.
[0316] [Implementation Example of Encoding Device]
[0317] Figure 8 1 is a block diagram showing an implementation example of the coding device 100. The coding device 100 includes a processor a1 and a memory a2. Figure 7 The multiple components of the encoding device 100 shown are Figure 8 It is implemented by the processor a1 and the memory a2 shown.
[0318] Processor a1 is a circuit that performs information processing and is able to access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit that encodes an image. Processor a1 may be a processor such as a CPU. In addition, processor a1 may be a collection of multiple electronic circuits. In addition, for example, processor a1 may play a role in Figure 7 The functions of the multiple components of the encoding device 100 other than the components for storing information are shown.
[0319] The memory a2 is a dedicated or general-purpose memory for storing information used by the processor a1 to encode an image. The memory a2 may be an electronic circuit and may be connected to the processor a1. In addition, the memory a2 may be included in the processor a1. In addition, the memory a2 may be a collection of multiple electronic circuits. In addition, the memory a2 may be a magnetic disk or an optical disk, or may be a storage device or a recording medium. In addition, the memory a2 may be a non-volatile memory or a volatile memory.
[0320] For example, the memory a2 may store a coded image or a stream corresponding to the coded image. In addition, the memory a2 may store a program for the processor a1 to encode an image.
[0321] In addition, for example, memory a2 can serve as Figure 7 The memory a2 may be used to store information among the multiple components of the encoding device 100 shown in FIG. Figure 7 The functions of the block memory 118 and the frame memory 122 are shown. More specifically, the memory a2 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).
[0322] In addition, in the encoding device 100, it is not necessary to implement Figure 7 All of the plurality of constituent elements shown may not perform all of the plurality of processes described above. Figure 7 Some of the multiple components shown may be included in other devices, and some of the multiple processes described above may be executed by other devices.
[0323] Hereinafter, after describing the overall processing flow of the encoding device 100 , each component included in the encoding device 100 will be described.
[0324] [Overall flow of encoding processing]
[0325] Fig. 9 1 is a flowchart showing an example of the overall encoding process of the encoding device 100 .
[0326] First, the segmentation unit 102 of the encoding device 100 segments the picture included in the original image into a plurality of fixed-size blocks (128×128 pixels) (step Sa_1). Next, the segmentation unit 102 selects a segmentation mode for the fixed-size block (step Sa_2). That is, the segmentation unit 102 further segments the fixed-size block into a plurality of blocks constituting the selected segmentation mode. Next, the encoding device 100 performs steps Sa_3 to Sa_9 on each of the plurality of blocks.
[0327] The prediction processing unit composed of the intra prediction unit 124 and the inter prediction unit 126, and the prediction control unit 128 generate a prediction image of the current block (step Sa_3). In addition, the prediction image is also called a prediction signal, a prediction block, or a prediction sample.
[0328] Next, the subtraction unit 104 generates a difference between the current block and the predicted image as a prediction residual (step Sa_4). The prediction residual is also called a prediction error.
[0329] Next, the transform unit 106 and the quantization unit 108 transform and quantize the predicted image to generate a plurality of quantization coefficients (step Sa_5).
[0330] Next, the entropy coding unit 110 encodes the plurality of quantized coefficients and prediction parameters related to generation of a predicted image (specifically, performs entropy coding) to generate a stream (step Sa_6).
[0331] Next, the inverse quantization unit 112 and the inverse transformation unit 114 perform inverse quantization and inverse transformation on the plurality of quantized coefficients, thereby restoring the prediction residual (step Sa_7).
[0332] Next, the adding unit 116 adds the predicted image to the restored prediction residual to reconstruct the current block (step Sa_8). Thus, a reconstructed image is generated. In addition, a reconstructed image is also called a reconstructed block, and in particular, a reconstructed image generated by the encoding device 100 is also called a local decoded block or a local decoded image.
[0333] When the reconstructed image is generated, the loop filter unit 120 filters the reconstructed image as necessary (step Sa_9).
[0334] Next, the encoding device 100 determines whether the encoding of the entire picture has been completed (step Sa_10 ). If it is determined that the encoding has not been completed (“No” in step Sa_10 ), the processing from step Sa_2 is repeatedly executed.
[0335] In addition, in the above example, the encoding device 100 selects one partition mode for a fixed-size block and encodes each block according to the partition mode, but each block may be encoded according to a plurality of partition modes. In this case, the encoding device 100 may evaluate the cost of each of the plurality of partition modes, for example, and selects a stream obtained by encoding according to the partition mode with the minimum cost as the final output stream.
[0336] Furthermore, the encoding device 100 may perform the processes of steps Sa_1 to Sa_10 sequentially, or may perform a plurality of some of the processes in parallel, or may change the order.
[0337] The encoding process of the encoding device 100 is a hybrid encoding using predictive encoding and transform encoding. In addition, predictive encoding is performed by an encoding loop, which is composed of a subtraction unit 104, a transform unit 106, a quantization unit 108, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a loop filter unit 120, a block memory 118, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, and a prediction control unit 128. That is, the prediction processing unit composed of the intra-frame prediction unit 124 and the inter-frame prediction unit 126 constitutes a part of the encoding loop.
[0338] [Division]
[0339] The segmentation unit 102 divides each picture included in the original image into a plurality of blocks, and outputs each block to the subtraction unit 104. For example, the segmentation unit 102 first divides the picture into blocks of a fixed size (e.g., 128×128 pixels). The fixed-size blocks are sometimes referred to as coding tree units (CTUs). Next, the segmentation unit 102 divides the fixed-size blocks into blocks of variable sizes (e.g., less than 64×64 pixels), for example based on recursive quadtree and / or binary tree block segmentation. That is, the segmentation unit 102 selects a segmentation mode. The variable-size blocks are sometimes referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). In addition, in various implementation examples, it is not necessary to distinguish between CUs, PUs, and TUs, and part or all of the blocks in the picture may become processing units of CUs, PUs, or TUs.
[0340] Fig.10 FIG. 1 is a diagram showing an example of block division in an implementation manner. Fig.10 In FIG, the solid line represents the block boundary based on the quadtree block partition, and the dotted line represents the block boundary based on the binary tree block partition.
[0341] Here, the block 10 is a square block of 128×128 pixels. The block 10 is first partitioned into four square blocks of 64×64 pixels (quadtree block partitioning).
[0342] The upper left square block of 64×64 pixels is further vertically divided into two rectangular blocks of 32×64 pixels each, and the left rectangular block of 32×64 pixels is further vertically divided into two rectangular blocks of 16×64 pixels each (binary tree block division). As a result, the upper left square block of 64×64 pixels is divided into two rectangular blocks 11 and 12 of 16×64 pixels and a rectangular block 13 of 32×64 pixels.
[0343] The upper right square block of 64×64 pixels is horizontally divided into two rectangular blocks 14 and 15 each consisting of 64×32 pixels (binary tree block division).
[0344] The lower left square block of 64×64 pixels is divided into 4 square blocks of 32×32 pixels each (quadtree block division). The upper left block and the lower right block of the 4 square blocks of 32×32 pixels are further divided. The upper left square block of 32×32 pixels is vertically divided into 2 rectangular blocks of 16×32 pixels each, and the right rectangular block of 16×32 pixels is further horizontally divided into 2 square blocks of 16×16 pixels each (binary tree block division). The lower right square block of 32×32 pixels is horizontally divided into 2 rectangular blocks of 32×16 pixels each (binary tree block division). As a result, the lower left square block of 64×64 pixels is divided into a rectangular block 16 of 16×32 pixels, two square blocks 17 and 18 of 16×16 pixels respectively, two square blocks 19 and 20 of 32×32 pixels respectively, and two rectangular blocks 21 and 22 of 32×16 pixels respectively.
[0345] The lower right block 23 consisting of 64×64 pixels is not divided.
[0346] As above, in Fig.10 In FIG. 1 , based on recursive quad-tree and binary tree block partitioning, the block 10 is partitioned into 13 blocks 11 to 23 of variable sizes. Such a partitioning is sometimes referred to as QTBT (quad-tree plus binary tree) partitioning.
[0347] In addition, Fig.10 In the example, one block is divided into four or two blocks (quadtree or binary tree block division), but the division is not limited to these. For example, one block can also be divided into three blocks (ternary tree block division). The division including such ternary tree block division is sometimes called MBT (multi type tree) division.
[0348] Fig.11 1 is a diagram showing an example of the functional structure of the segmentation unit 102. Fig.11 As shown, the division unit 102 may include a block division determination unit 102a. As an example, the block division determination unit 102a may perform the following processing.
[0349] The block division determination unit 102a collects block information from the block memory 118 or the frame memory 122, and determines the above-mentioned division mode based on the block information. The division unit 102 divides the original image according to the division mode and outputs one or more blocks obtained by the division to the subtraction unit 104.
[0350] In addition, the block division determination unit 102a outputs, for example, the parameter indicating the above-mentioned division mode to the transform unit 106, the inverse transform unit 114, the intra prediction unit 124, the inter prediction unit 126, and the entropy coding unit 110. The transform unit 106 can transform the prediction residual based on the parameter, and the intra prediction unit 124 and the inter prediction unit 126 can generate a predicted image based on the parameter. In addition, the entropy coding unit 110 can entropy code the parameter.
[0351] As an example, parameters related to the segmentation mode may be written into the stream as follows.
[0352] Fig.12 The figure shows an example of the division mode. The division modes include, for example, 4 divisions (QT) in which the block is divided into 2 in the horizontal direction and 2 in the vertical direction, 3 divisions (HT or VT) in which the block is divided in the same direction at a ratio of 1:2:1, 2 divisions (HB or VB) in which the block is divided in the same direction at a ratio of 1:1, and no division (NS).
[0353] In the case of 4-division and no-division, the division mode does not have the block division direction, and in the case of 2-division and 3-division, the division mode has the division direction information.
[0354] Fig.13A and Fig. 13B is a diagram showing an example of a syntax tree of a segmentation pattern. Fig.13A In the example, first, there is information indicating whether to perform a split (S: Split flag), then there is information indicating whether to perform a 4-part split (QT: QT flag). Then there is information indicating whether to perform a 3-part split or a 2-part split (TT: TT flag or BT: BT flag), and finally there is information indicating the split direction (Ver: Vertical flag or Hor: Horizontal flag). In addition, for one or more blocks obtained by splitting based on such a splitting pattern, the splitting can be further repeatedly applied with the same processing. That is, as an example, it is possible to recursively implement the judgment of whether to perform a split, whether to perform a 4-part split, whether the splitting method is horizontal or vertical, and whether to perform a 3-part split or a 2-part split, and the implemented judgment results can be calculated according to the conditions. Fig.13A The encoding order disclosed in the syntax tree shown is encoded into the stream.
[0355] In addition, Fig.13A In the syntax tree shown, this information is arranged in the order of S, QT, TT, Ver, but it can also be arranged in the order of S, QT, Ver, BT. Fig. 13BIn the example, first, there is information indicating whether to split (S: Split flag), then there is information indicating whether to split into 4 (QT: QT flag). Then there is information indicating the split direction (Ver: Vertical flag or Hor: Horizontal flag), and finally there is information indicating whether to split into 2 or 3 (BT: BT flag or TT: TT flag).
[0356] In addition, the division pattern described here is an example, and a pattern other than the division pattern described may be used, or only a part of the division pattern described may be used.
[0357] [Subtraction Department]
[0358] The subtracting unit 104 subtracts the prediction image (the prediction image input from the prediction control unit 128) from the original image in units of blocks input from the dividing unit 102 and divided by the dividing unit 102. That is, the subtracting unit 104 calculates the prediction residual of the current block. Then, the subtracting unit 104 outputs the calculated prediction residual to the transforming unit 106.
[0359] The original image is an input signal of the encoding device 100, and is, for example, a signal representing an image of each picture constituting a moving image (for example, a luminance (luma) signal and two color difference (chroma) signals).
[0360] [Conversion Department]
[0361] The transform unit 106 transforms the prediction residual in the spatial domain into transform coefficients in the frequency domain, and outputs the transform coefficients to the quantization unit 108. Specifically, the transform unit 106 performs a preset discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction residual in the spatial domain, for example.
[0362] In addition, the transform unit 106 can adaptively select a transform type from a plurality of transform types, and transform the prediction residual into a transform coefficient using a transform basis function corresponding to the selected transform type. Such a transform is sometimes referred to as EMT (explicit multiple core transform) or AMT (adaptive multiple transform). In addition, the transform basis function is sometimes referred to as a basis.
[0363] The plurality of transform types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. In addition, these transform types may be expressed as DCT2, DCT5, DCT8, DST1, and DST7, respectively. Fig.14is a table showing the transformation basis functions corresponding to each transformation type. Fig.14 In , N represents the number of input pixels. The selection of a transform type from these multiple transform types may depend on, for example, the type of prediction (intra-frame prediction, inter-frame prediction, etc.), or may depend on the intra-frame prediction mode.
[0364] Information indicating whether such EMT or AMT is applied (e.g., called an EMT flag or an AMT flag) and information indicating the selected transform type are usually signaled at the CU level. In addition, the signaling of such information does not need to be limited to the CU level, but may also be at other levels (e.g., sequence level, picture level, slice level, brick level, or CTU level).
[0365] In addition, the transformation unit 106 may also re-transform the transformation coefficient (i.e., the transformation result). Such re-transformation is sometimes referred to as AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transformation unit 106 re-transforms each sub-block (e.g., a sub-block of 4×4 pixels) contained in the block of transformation coefficients corresponding to the intra-frame prediction residual. Information indicating whether NSST is applied and information related to the transformation matrix used for NSST are usually signaled at the CU level. In addition, the signaling of this information does not need to be limited to the CU level, but may also be other levels (e.g., sequence level, picture level, slice level, brick level, or CTU level).
[0366] The transform unit 106 may apply separable transforms and non-separable transforms. Separable transforms are a method of performing multiple transforms by separating the input dimension in each direction, and non-separable transforms are a method of treating two or more dimensions as one dimension and performing the transforms together when the input is multi-dimensional.
[0367] For example, as an example of an indivisible transformation, when a block of 4×4 pixels is input, it is regarded as an array of 16 elements and a transformation process is performed on the array using a 16×16 transformation matrix.
[0368] In another example of the inseparable transformation, after treating a 4×4 pixel input block as an array of 16 elements, a transformation (Hypercube Givens Transform) may be performed on the array by performing a plurality of Givens rotations.
[0369] In the transformation performed by the transformation unit 106 , the transformation type of the transformation basis function for transforming to the frequency domain can be switched according to the region within the CU. As an example, there is SVT (Spatially Varying Transform).
[0370] Fig.15 This is a diagram showing an example of SVT.
[0371] In SVT, Fig.15 As shown, the CU is divided into two equal parts in the horizontal direction or the vertical direction, and only one of the regions is transformed into the frequency region. The transformation type can be set for each region, for example, using DST7 and DCT8. For example, in the two regions obtained by dividing the CU into two equal parts in the vertical direction, DST7 and DCT8 can be used for the region at position 0. Alternatively, in these two regions, DST7 can be used for the region at position 1. Similarly, in the two regions obtained by dividing the CU into two equal parts in the horizontal direction, DST7 and DCT8 can be used for the region at position 0. Alternatively, in these two regions, DST7 can be used for the region at position 1. In such a Fig.15 In the example shown, only one of the two regions in the CU is transformed, and the other is not transformed, but the two regions can also be transformed separately. In addition, the division method is not only 2-way division, but also 4-way division. In addition, it is also possible to encode the information indicating the division method and signal it in the same way as the CU division to make it more flexible. In addition, SVT is sometimes called SBT (Sub-block Transform).
[0372] The above-mentioned AMT and EMT can also be called MTS (Multiple Transform Selection). When MTS is applied, a transform type such as DST7 or DCT8 can be selected, and information indicating the selected transform type can be encoded as index information for each CU. On the other hand, as a process of selecting a transform type based on the shape of the CU and using the index information for orthogonal transform without encoding, there is a process called IMTS (Implicit MTS). When IMTS is applied, for example, if the shape of the CU is a rectangle, the short side of the rectangle uses DST7 and the long side uses DCT2 to perform orthogonal transforms respectively. In addition, for example, when the shape of the CU is a square, if MTS is valid in the sequence, DCT2 is used, and if MTS is invalid, DST7 is used for orthogonal transform. DCT2 and DST7 are examples, and other transform types can also be used, and it can also be set to a combination that makes the combination of transform types used different. IMTS can be used only for intra-frame predicted blocks, or it can be used for both intra-frame predicted blocks and inter-frame predicted blocks.
[0373] In the above, three processes of MTS, SBT and IMTS are described as the selection process for selectively switching the transform type used in the orthogonal transform. All of the three selection processes can be made effective, or only a part of the selection processes can be selectively made effective. Whether each selection process is made effective can be identified by flag information in a header such as SPS. For example, if all three selection processes are effective, one of the three selection processes can be selected for orthogonal transform in units of CUs. In addition, regarding the selection process for selectively switching the transform type, if at least one of the following four functions [1] to [4] can be implemented, a selection process different from the above three selection processes can be used, or the above three selection processes can be replaced by other processes respectively. Function [1] is a function of performing an orthogonal transform on the entire range within a CU and encoding information indicating the transform type used in the transform. Function [2] is a function of performing an orthogonal transform on the entire range of a CU and determining the transform type based on a prescribed rule without encoding the information indicating the transform type. Function [3] is a function of performing an orthogonal transform on a part of a CU and encoding information indicating the transform type used in the transform. Function [4] is a function of performing an orthogonal transform on a part of the CU area, determining the transform type based on a prescribed rule without encoding information indicating the transform type used in the transform, etc.
[0374] In addition, whether or not to apply each of MTS, IMTS, and SBT may be determined for each processing unit, for example, for each sequence unit, picture unit, tile unit, slice unit, CTU unit, or CU unit.
[0375] In addition, the tool for selectively switching the transformation type in the present disclosure can be renamed as a method for adaptively selecting a basis used in the transformation process, a selection process, or a process of selecting a basis. In addition, the tool for selectively switching the transformation type can also be renamed as a mode for adaptively selecting the transformation type.
[0376] Fig.16 This is a flowchart showing an example of the processing of the conversion unit 106.
[0377] For example, the transform unit 106 determines whether to perform an orthogonal transform (step St_1). Here, if the transform unit 106 determines to perform an orthogonal transform ("Yes" in step St_1), the transform type used in the orthogonal transform is selected from a plurality of transform types (step St_2). Next, the transform unit 106 applies the selected transform type to the prediction residual of the current block to perform an orthogonal transform (step St_3). Next, the transform unit 106 outputs information indicating the selected transform type to the entropy coding unit 110, thereby encoding the information (step St_4). On the other hand, if the transform unit 106 determines not to perform an orthogonal transform ("No" in step St_1), the information indicating that the orthogonal transform is not performed is output to the entropy coding unit 110, thereby encoding the information (step St_5). In addition, the determination of whether to perform an orthogonal transform in step St_1 can be determined, for example, based on the size of the transform block, the prediction mode applied to the CU, etc. In addition, the information indicating the transform type used in the orthogonal transform may not be encoded, but an orthogonal transform may be performed using a predetermined transform type.
[0378] Fig.17 : is a flowchart showing another example of the processing of the conversion unit 106. Fig.17 The example shown is similar to Fig.16 The example shown is also an example of orthogonal transform in which a method of selectively switching the transform type used in the orthogonal transform is applied.
[0379] As an example, the first transform type group may include DCT2, DST7, and DCT8. Also, as an example, the second transform type group may include DCT2. Also, the transform types included in the first transform type group and the second transform type group may overlap in part or may be completely different transform types.
[0380] Specifically, the transform unit 106 determines whether the transform size is less than or equal to a specified value (step Su_1). Here, if it is determined to be less than the specified value ("Yes" in step Su_1), the transform unit 106 uses the transform type included in the first transform type group to perform an orthogonal transform on the prediction residual of the current block (step Su_2). Next, the transform unit 106 outputs information indicating which transform type is used among the one or more transform types included in the first transform type group to the entropy coding unit 110, thereby encoding the information (step Su_3). On the other hand, if the transform unit 106 determines that the transform size is not less than or equal to the specified value ("No" in step Su_1), it uses the second transform type group to perform an orthogonal transform on the prediction residual of the current block (step Su_4).
[0381] In step Su_3, the information indicating the transform type used in the orthogonal transform may be information indicating a combination of a transform type applied in the vertical direction and a transform type applied in the horizontal direction of the current block. In addition, the first transform type group may include only one transform type, and the information indicating the transform type used in the orthogonal transform may not be encoded. The second transform type group may include multiple transform types, and the information indicating the transform type used in the orthogonal transform among one or more transform types included in the second transform type group may be encoded.
[0382] Furthermore, the transform type may be determined based on only the transform size. Furthermore, as long as the transform type used in the orthogonal transform is determined based on the transform size, the process is not limited to the determination of whether the transform size is equal to or less than a predetermined value.
[0383] [Quantitative Department]
[0384] The quantization unit 108 quantizes the transform coefficients output from the transform unit 106. Specifically, the quantization unit 108 scans a plurality of transform coefficients of the current block in a predetermined scanning order, and quantizes the transform coefficients based on a quantization parameter (QP) corresponding to the scanned transform coefficients. Next, the quantization unit 108 outputs the quantized plurality of transform coefficients of the current block (hereinafter referred to as quantized coefficients) to the entropy coding unit 110 and the inverse quantization unit 112.
[0385] The prescribed scanning order is the order for quantization / inverse quantization of transform coefficients. For example, the prescribed scanning order is defined according to the ascending order (order from low frequency to high frequency) or descending order (order from high frequency to low frequency) of frequency.
[0386] The quantization parameter (QP) is a parameter that defines the quantization step size (quantization width). For example, if the value of the quantization parameter increases, the quantization step size also increases. That is, if the value of the quantization parameter increases, the error of the quantization coefficient (quantization error) increases.
[0387] In addition, there are cases where quantization matrices are used in quantization. For example, there are cases where multiple quantization matrices are used corresponding to frequency transform sizes such as 4×4 and 8×8, prediction modes such as intra-frame prediction and inter-frame prediction, and pixel components such as brightness and color difference. In addition, quantization refers to digitizing the values obtained by sampling at a predetermined interval and establishing a correspondence with a predetermined level. In this technical field, there are also cases where expressions such as rounding, rounding, or scaling are used.
[0388] As a method of using a quantization matrix, there are a method of using a quantization matrix directly set on the encoding device 100 side and a method of using a default quantization matrix (default matrix). On the encoding device 100 side, by directly setting the quantization matrix, it is possible to set a quantization matrix corresponding to the characteristics of the image. However, in this case, there is a disadvantage that the amount of code increases by encoding the quantization matrix. In addition, instead of directly using the default quantization matrix or the encoded quantization matrix, the quantization matrix used in the quantization of the current block can be generated based on the default quantization matrix or the encoded quantization matrix.
[0389] On the other hand, there is also a method of quantizing the coefficients of high-frequency components and low-frequency components in the same manner without using a quantization matrix. This method is equivalent to a method of using a quantization matrix (flat matrix) in which all coefficients have the same value.
[0390] The quantization matrix may be encoded, for example, at a sequence level, a picture level, a slice level, a tile level, or a CTU level.
[0391] The quantization unit 108, when using a quantization matrix, for example, performs scaling using the value of the quantization matrix, such as a quantization width calculated based on a quantization parameter, for each transform coefficient. The quantization process performed without using a quantization matrix may be a process of quantizing the transform coefficient based on the quantization width calculated based on the quantization parameter, etc. In addition, in the quantization process performed without using a quantization matrix, the quantization width may be multiplied by a predetermined value common to all transform coefficients in the block.
[0392] Fig.18 : is a block diagram showing an example of the functional configuration of the quantization unit 108 .
[0393] The quantization unit 108 includes, for example, a difference quantization parameter generation unit 108 a , a predicted quantization parameter generation unit 108 b , a quantization parameter generation unit 108 c , a quantization parameter storage unit 108 d , and a quantization processing unit 108 e .
[0394] Fig.19 This is a flowchart showing an example of quantization performed by the quantization unit 108 .
[0395] As an example, the quantization unit 108 may be based on Fig.19 The flowchart shown implements quantization for each CU. Specifically, the quantization parameter generation unit 108c determines whether to perform quantization (step Sv_1). Here, if it is determined to perform quantization ("Yes" in step Sv_1), the quantization parameter generation unit 108c generates a quantization parameter for the current block (step Sv_2) and stores the quantization parameter in the quantization parameter storage unit 108d (step Sv_3).
[0396] Next, the quantization processing unit 108e quantizes the transform coefficients of the current block using the quantization parameter generated in step Sv_2 (step Sv_4). Next, the predicted quantization parameter generation unit 108b obtains the quantization parameter of the processing unit different from the current block from the quantization parameter storage unit 108d (step Sv_5). The predicted quantization parameter generation unit 108b generates the predicted quantization parameter of the current block based on the obtained quantization parameter (step Sv_6). The differential quantization parameter generation unit 108a calculates the difference between the quantization parameter of the current block generated by the quantization parameter generation unit 108c and the predicted quantization parameter of the current block generated by the predicted quantization parameter generation unit 108b (step Sv_7). By calculating the difference, a differential quantization parameter is generated. The differential quantization parameter generation unit 108a outputs the differential quantization parameter to the entropy coding unit 110, so that the differential quantization parameter is encoded (step Sv_8).
[0397] In addition, the differential quantization parameter may be encoded at a sequence level, a picture level, a slice level, a brick level, or a CTU level. In addition, the initial value of the quantization parameter may be encoded at a sequence level, a picture level, a slice level, a brick level, or a CTU level. At this time, the quantization parameter may be generated using the initial value of the quantization parameter and the differential quantization parameter.
[0398] Furthermore, the quantization unit 108 may include a plurality of quantizers, and may apply dependent quantization for quantizing the transform coefficients using a quantization method selected from a plurality of quantization methods.
[0399] [Entropy coding unit]
[0400] Fig. 20 2 is a block diagram showing an example of the functional structure of the entropy coding unit 110 .
[0401] The entropy coding unit 110 generates a stream by entropy coding the quantization coefficients input from the quantization unit 108 and the prediction parameters input from the prediction parameter generation unit 130. In this entropy coding, CABAC (Context-based Adaptive Binary Arithmetic Coding) is used, for example. Specifically, the entropy coding unit 110 includes, for example, a binarization unit 110a, a context control unit 110b, and a binary arithmetic coding unit 110c. The binarization unit 110a performs binarization to convert multi-value signals such as quantization coefficients and prediction parameters into binary signals. Examples of binarization methods include Truncated Rice Binarization, Exponential Golomb codes, and Fixed Length Binarization. The context control unit 110b derives the context value corresponding to the characteristics of the syntactic element or the surrounding conditions, that is, the probability of occurrence of the binary signal. Methods for deriving the context value include, for example, bypass, syntax element reference, upper / left neighboring block reference, hierarchy information reference, etc. The binary arithmetic coding unit 110c performs arithmetic coding on the binary signal using the derived context value.
[0402] Fig.21 3 is a diagram showing the flow of CABAC in the entropy coding unit 110 .
[0403] First, in CABAC in the entropy coding unit 110, initialization is performed. In this initialization, initialization in the binary arithmetic coding unit 110c and setting of the initial context value are performed. In addition, the binarization unit 110a and the binary arithmetic coding unit 110c perform binarization and arithmetic coding on multiple quantized coefficients of the CTU in sequence, for example. At this time, the context control unit 110b updates the context value every time arithmetic coding is performed. Next, the context control unit 110b saves the context value as post-processing. The saved context value is used, for example, as the initial value of the context value of the next CTU.
[0404] [Inverse Quantization Unit]
[0405] The inverse quantization unit 112 inversely quantizes the quantized coefficients input from the quantization unit 108. Specifically, the inverse quantization unit 112 inversely quantizes the quantized coefficients of the current block in a predetermined scanning order. Furthermore, the inverse quantization unit 112 outputs the inversely quantized transform coefficients of the current block to the inverse transform unit 114.
[0406] [Inverse transformation unit]
[0407] The inverse transform unit 114 inversely transforms the transform coefficients input from the inverse quantization unit 112 to restore the prediction residual. Specifically, the inverse transform unit 114 performs an inverse transform corresponding to the transform performed by the transform unit 106 on the transform coefficients to restore the prediction residual of the current block. Next, the inverse transform unit 114 outputs the restored prediction residual to the addition unit 116.
[0408] In addition, the restored prediction residual generally loses information due to quantization, and therefore does not match the prediction error calculated by the subtraction unit 104. That is, the restored prediction residual generally includes a quantization error.
[0409] [Addition Department]
[0410] The adder 116 reconstructs the current block by adding the prediction residual input from the inverse transform unit 114 and the prediction image input from the prediction control unit 128 . As a result, a reconstructed image is generated. Next, the adder 116 outputs the reconstructed image to the block memory 118 and the loop filter unit 120 .
[0411] [Block Memory]
[0412] The block memory 118 is a storage unit for storing blocks in the current picture that are referenced in, for example, intra-frame prediction. Specifically, the block memory 118 stores the reconstructed image output from the adder 116 .
[0413] [Frame Memory]
[0414] The frame memory 122 is a storage unit for storing reference pictures used in, for example, inter-frame prediction, and is sometimes referred to as a frame buffer. Specifically, the frame memory 122 stores a reconstructed image filtered by the loop filter unit 120 .
[0415] [Loop filter section]
[0416] The loop filter unit 120 performs loop filtering on the reconstructed image output from the adder 116, and outputs the filtered reconstructed image to the frame memory 122. The so-called loop filtering is filtering used within the encoding loop (in-loop filtering), and includes, for example, adaptive loop filtering (ALF), deblocking filtering (DF or DBF), and sample adaptive offset (SAO).
[0417] Fig. 22 3 is a block diagram showing an example of the functional configuration of the loop filter unit 120 .
[0418] For example, Fig. 22As shown, the loop filter unit 120 includes a deblocking filter processing unit 120a, an SAO processing unit 120b, and an ALF processing unit 120c. The deblocking filter processing unit 120a performs the above-mentioned deblocking filter processing on the reconstructed image. The SAO processing unit 120b performs the above-mentioned SAO processing on the reconstructed image after the deblocking filter processing. In addition, the ALF processing unit 120c applies the above-mentioned ALF processing to the reconstructed image after the SAO processing. Details about the ALF and the deblocking filter will be described later. The SAO processing is a processing to improve the image quality by reducing ringing (a distortion phenomenon such as fluctuation of pixel values around edges) and correcting the deviation of pixel values. The SAO processing includes, for example, edge offset processing and band offset processing. In addition, the loop filter unit 120 may not have Fig. 22 All processing units disclosed in the above may only include a part of the processing units. In addition, the loop filter unit 120 may be a Fig. 22 A structure in which each of the above-mentioned processes is performed in an order different from the processing order disclosed in.
[0419] [Loop Filter Section> Adaptive Loop Filter]
[0420] In ALF, a least square error filter is applied to remove coding distortion. For example, a filter selected from a plurality of filters based on the direction and activity of a local gradient is applied to each sub-block of 2×2 pixels in the current block.
[0421] Specifically, first, a sub-block (e.g., a 2×2 pixel sub-block) is classified into a plurality of classes (e.g., 15 or 25 classes). The classification of the sub-block is performed, for example, based on the direction and activity of the gradient. In a specific example, the classification value C (e.g., C=5D+A) is calculated using the direction value D of the gradient (e.g., 0 to 2 or 0 to 4) and the activity value A of the gradient (e.g., 0 to 4). And, based on the classification value C, the sub-block is classified into a plurality of classes.
[0422] The direction value D of the gradient is derived, for example, by comparing the gradients in multiple directions (eg, horizontal, vertical, and two diagonal directions). In addition, the activity value A of the gradient is derived, for example, by adding the gradients in multiple directions and quantizing the addition result.
[0423] Based on the result of such classification, a filter to be used for a sub-block is determined from a plurality of filters.
[0424] As the shape of the filter used in the ALF, for example, a circularly symmetric shape is used. Figures 23A to 23C FIG. 1 is a diagram showing a plurality of examples of the shape of filters used in ALF. Fig.23A represents a 5×5 diamond-shaped filter, Fig. 23B represents a 7×7 diamond shape filter, Fig.23C Represents a 9×9 diamond shape filter. The information representing the shape of the filter is usually signaled at the picture level. In addition, the signaling of the information representing the shape of the filter does not need to be limited to the picture level, and can also be other levels (for example, sequence level, slice level, brick level, CTU level or CU level).
[0425] The on / off of ALF can be determined, for example, at the picture level or the CU level. For example, regarding brightness, whether to apply ALF can be determined at the CU level, and regarding color difference, whether to apply ALF can be determined at the picture level. Information indicating whether ALF is on / off is usually signaled at the picture level or the CU level. In addition, the signaling of information indicating whether ALF is on / off does not need to be limited to the picture level or the CU level, and can also be other levels (for example, sequence level, slice level, brick level, or CTU level).
[0426] In addition, as described above, one filter is selected from a plurality of filters to perform ALF processing on a sub-block. For each of the plurality of filters (e.g., up to 15 or 25 filters), a coefficient set consisting of a plurality of coefficients used in the filter is usually signaled at a picture level. In addition, the signaling of the coefficient set does not need to be limited to the picture level, and may be other levels (e.g., sequence level, slice level, brick level, CTU level, CU level, or sub-block level).
[0427] [Loop Filter > Cross Component Adaptive Loop Filter]
[0428] Fig.23D This is a diagram showing an example in which a Y sample (first component) is used in CCALF of Cb and CCALF of Cr (a plurality of components different from the first component). Fig.23E is a diagram showing a diamond-shaped filter.
[0429] An example of CC-ALF is to use a linear diamond filter ( Fig.23D , Fig.23E) is applied to the luminance channel of each color difference component. For example, the filter coefficients are sent in APS, scaled by a factor of 2^10, and rounded for fixed decimal point representation. The application of the filter can be controlled based on a variable block size, which is notified by a context-encoded flag received for each block of samples. The block size and CC-ALF activation flag are received at the slice level for each color difference component. The syntax and semantics of CC-ALF are provided in the Appendix. In this document, block sizes of 16×16, 32×32, 64×64, and 128×128 are supported (in color difference samples).
[0430] [Loop Filter > Joint Chroma CrossComponent Adaptive Loop Filter]
[0431] Fig.23F It is a diagram showing an example of JC-CCALF. Figure 23G This is a diagram showing an example of weight_index candidates of JC-CCALF.
[0432] One example of JC-CCALF uses only one CCALF filter, generates one CCALF filter output as the color difference adjustment signal for only one color component, and applies appropriately weighted versions of the same color difference adjustment signal to the other color components. This reduces the complexity of the existing CCALF to about half.
[0433] The weight value is encoded in the sign flag and the weight index. The weight index (denoted as weight_index) is encoded as 3 bits and specifies the size of the JC-CCALF weight JcCcWeight. It cannot be set equal to 0. The size of JcCcWeight is determined as follows.
[0434] When weight_index is 4 or less, JcCcWeight is equal to weight_index>>2.
[0435] In other cases, JcCcWeight is equal to 4 / (weight_index-4).
[0436] The block level on / off control for the ALF filter is separate for Cb and Cr. This is the same as CCALF, where 2 separate sets of block level on / off control flags are encoded. Here, unlike CCALF, the on / off control block size for Cb and Cr is the same, so only 1 block size variable is encoded.
[0437] [Loop Filter Section> Deblocking Filter]
[0438] In the deblocking filtering process, the loop filter unit 120 performs filtering on the block boundaries of the reconstructed image to reduce distortion generated at the block boundaries.
[0439] Fig.24 It is a block diagram showing an example of a detailed structure of the deblocking filter processing unit 120a.
[0440] The deblocking filter processing unit 120 a includes, for example, a boundary determination unit 1201 , a filter determination unit 1203 , a filter processing unit 1205 , a processing determination unit 1208 , a filter characteristic determination unit 1207 , and switches 1202 , 1204 , and 1206 .
[0441] The boundary determination unit 1201 determines whether or not there is a pixel to be processed by the deblocking filter (ie, a target pixel) near a block boundary. Next, the boundary determination unit 1201 outputs the determination result to the switch 1202 and the processing determination unit 1208 .
[0442] When the boundary determination unit 1201 determines that the target pixel exists near the block boundary, the switch 1202 outputs the image before filtering to the switch 1204. On the contrary, when the boundary determination unit 1201 determines that the target pixel does not exist near the block boundary, the switch 1202 outputs the image before filtering to the switch 1206. The image before filtering is an image including the target pixel and at least one peripheral pixel surrounding the target pixel.
[0443] The filter determination unit 1203 determines whether to perform deblocking filtering on the target pixel based on the pixel value of at least one surrounding pixel around the target pixel. Next, the filter determination unit 1203 outputs the determination result to the switch 1204 and the processing determination unit 1208 .
[0444] When the filter determination unit 1203 determines that the deblocking filter process is to be performed on the target pixel, the switch 1204 outputs the image before the filtering process obtained through the switch 1202 to the filtering process unit 1205. On the contrary, when the filter determination unit 1203 determines that the deblocking filter process is not to be performed on the target pixel, the switch 1204 outputs the image before the filtering process obtained through the switch 1202 to the switch 1206.
[0445] When the filter processing unit 1205 obtains the image before filtering via the switches 1202 and 1204 , it performs deblocking filtering on the target pixel using the filter characteristics determined by the filter characteristic determination unit 1207 . Then, the filter processing unit 1205 outputs the filtered pixel to the switch 1206 .
[0446] The switch 1206 selectively outputs pixels that have not been processed by the deblocking filter and pixels that have been processed by the deblocking filter by the filter processing unit 1205 , according to the control of the processing determination unit 1208 .
[0447] The processing determination unit 1208 controls the switch 1206 based on the respective determination results of the boundary determination unit 1201 and the filtering determination unit 1203. That is, when the boundary determination unit 1201 determines that the object pixel exists near the block boundary and the filtering determination unit 1203 determines that the object pixel is to be subjected to deblocking filtering, the processing determination unit 1208 causes the pixel subjected to deblocking filtering to be output from the switch 1206. In addition, other than the above-mentioned case, the processing determination unit 1208 causes the pixel that has not been subjected to deblocking filtering to be output from the switch 1206. By repeatedly outputting such pixels, the filtered image is output from the switch 1206. In addition, Fig.24 The illustrated structure is an example of the structure of the deblocking filter processing unit 120a, and the deblocking filter processing unit 120a may have other structures.
[0448] Fig.25 This is a diagram showing an example of a deblocking filter having a filter characteristic that is symmetric with respect to a block boundary.
[0449] In the deblocking filter process, for example, pixel values and quantization parameters are used to select one of two deblocking filters with different characteristics, namely, a strong filter and a weak filter. Fig.25 As shown, when pixels p0 to p2 and pixels q0 to q2 exist across a block boundary, the pixel values of pixels q0 to q2 are changed to pixel values q'0 to q'2 by performing calculations shown in the following equations.
[0450] q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8
[0451] q'1=(p0+q0+q1+q2+2) / 4
[0452] q'2=(p0+q0+q1+3×q2+2×q3+4) / 8
[0453] In the above equations, p0 to p2 and q0 to q2 are pixel values of pixels p0 to p2 and pixels q0 to q2, respectively. In addition, q3 is the pixel value of pixel q3 adjacent to pixel q2 on the opposite side of the block boundary. In addition, on the right side of each of the above equations, the coefficient by which the pixel value of each pixel used in the deblocking filter process is multiplied is the filter coefficient.
[0454] Furthermore, in the deblocking filter process, clipping can also be performed so that the pixel value after the operation does not exceed the threshold value. In the clipping process, the pixel value after the operation based on the above formula is clipped to "pixel value before operation ± 2 × threshold value" using a threshold value determined by a quantization parameter. This can prevent excessive smoothing.
[0455] Fig.26 This is a diagram for explaining an example of a block boundary on which a deblocking filtering process is performed. Fig. 27 This is a diagram showing an example of BS value.
[0456] The block boundary to be deblocked is, for example, Fig.26 The deblocking filter is performed in units of, for example, 4 rows or 4 columns. First, for Fig.26 The blocks P and Q shown are as follows: Fig. 27 That determines the Bs (Boundary Strength) value.
[0457] according to Fig. 27 The Bs value can determine whether to perform deblocking filtering with different strengths even at the block boundaries of the same image. Deblocking filtering for color difference signals is performed when the Bs value is 2. Deblocking filtering for luminance signals is performed when the Bs value is 1 or more and the specified conditions are met. In addition, the determination condition of the Bs value is not limited to Fig. 27 The situation shown can also be determined based on other parameters.
[0458] [Prediction unit (intra-frame prediction unit / inter-frame prediction unit / prediction control unit)]
[0459] Fig.28 1 is a flowchart showing an example of processing performed by the prediction unit of the encoding device 100. In addition, as an example, the prediction unit is composed of all or part of the components of the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128. The prediction processing unit includes the intra prediction unit 124 and the inter prediction unit 126, for example.
[0460] The prediction unit generates a prediction image of the current block (step Sb_1). The prediction image includes, for example, an intra-frame prediction image (intra-frame prediction signal) or an inter-frame prediction image (inter-frame prediction signal). Specifically, the prediction unit generates a prediction image of the current block using a reconstructed image obtained by generating prediction images for other blocks, generating prediction residuals, generating quantization coefficients, restoring prediction residuals, and adding prediction images.
[0461] The reconstructed image may be, for example, an image of a reference picture or an image including the current block, that is, an image of an encoded block in the current picture (that is, the other blocks mentioned above). The encoded block in the current picture may be, for example, an adjacent block of the current block.
[0462] Fig.29 This is a flowchart showing another example of the processing performed by the prediction unit of the encoding device 100.
[0463] The prediction unit generates a prediction image by the first method (step Sc_1a), generates a prediction image by the second method (step Sc_1b), and generates a prediction image by the third method (step Sc_1c). The first method, the second method, and the third method are different methods for generating prediction images, and may be, for example, an inter-frame prediction method, an intra-frame prediction method, and prediction methods other than these. In these prediction methods, the above-mentioned reconstructed image can be used.
[0464] Next, the prediction unit evaluates the predicted images generated by steps Sc_1a, Sc_1b, and Sc_1c, respectively (step Sc_2). For example, the prediction unit calculates the cost C for the predicted images generated by steps Sc_1a, Sc_1b, and Sc_1c, respectively, and evaluates the predicted images by comparing the costs C of these predicted images. In addition, the cost C is calculated using a formula of the R-D optimization model, such as C=D+λ×R. In this formula, D is the coding distortion of the predicted image, for example, represented by the absolute value of the difference between the pixel value of the current block and the pixel value of the predicted image. In addition, R is the bit rate of the stream. In addition, λ is, for example, an undetermined multiplier of Lagrangian.
[0465] Next, the prediction unit selects one of the prediction images generated by steps Sc_1a, Sc_1b, and Sc_1c, respectively (step Sc_3). That is, the prediction unit selects a method or mode for obtaining the final prediction image. For example, the prediction unit selects a prediction image with the smallest cost C based on the cost C calculated for these prediction images. Alternatively, the evaluation in step Sc_2 and the selection of the prediction image in step Sc_3 may also be performed based on the parameters used in the encoding process. The encoding device 100 may signal information used to determine the selected prediction image, method, or mode into the stream. The information may be, for example, a flag. Thus, the decoding device 200 can generate a prediction image in the method or mode selected in the encoding device 100 based on the information. In addition, in Fig.29 In the example shown, the prediction unit selects a certain prediction image after generating the prediction images in various ways. However, the prediction unit may select a method or mode based on the parameters used in the above-mentioned encoding process before generating these prediction images, and generate the prediction images according to the method or mode.
[0466] For example, the first method and the second method may be intra prediction and inter prediction, respectively, and the prediction unit may select a final prediction image for the current block from prediction images generated by these prediction methods.
[0467] Fig.30 This is a flowchart showing another example of the processing performed by the prediction unit of the encoding device 100.
[0468] First, the prediction unit generates a prediction image by intra prediction (step Sd_1a), and generates a prediction image by inter prediction (step Sd_1b). In addition, the prediction image generated by intra prediction is also called intra prediction image, and the prediction image generated by inter prediction is also called inter prediction image.
[0469] Next, the prediction unit evaluates the intra-frame prediction image and the inter-frame prediction image respectively (step Sd_2). The above-mentioned cost C can be used in this evaluation. In addition, the prediction unit can select the prediction image with the smallest cost C calculated from the intra-frame prediction image and the inter-frame prediction image as the final prediction image of the current block (step Sd_3). That is, the prediction method or mode used to generate the prediction image of the current block is selected.
[0470] [Intra-frame prediction unit]
[0471] The intra prediction unit 124 performs intra prediction (also called intra-screen prediction) of the current block with reference to the blocks in the current picture stored in the block memory 118, thereby generating a predicted image of the current block (i.e., intra-prediction image). Specifically, the intra prediction unit 124 performs intra prediction with reference to the pixel values (e.g., brightness values, color difference values) of the blocks adjacent to the current block, thereby generating an intra-prediction image, and outputs the intra-prediction image to the prediction control unit 128.
[0472] For example, the intra prediction unit 124 performs intra prediction using one of a plurality of predetermined intra prediction modes. The plurality of intra prediction modes generally include one or more non-directional prediction modes and a plurality of directional prediction modes.
[0473] The one or more non-directional prediction modes include, for example, a Planar prediction mode and a DC prediction mode defined in the H.265 / HEVC standard.
[0474] The plurality of directional prediction modes include, for example, 33 directional prediction modes defined by the H.265 / HEVC standard. In addition, the plurality of directional prediction modes may include 32 directional prediction modes in addition to the 33 directional prediction modes (a total of 65 directional prediction modes). Fig.31This is a diagram showing all 67 intra prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in intra prediction. The solid arrows indicate the 33 directions specified by the H.265 / HEVC standard, and the dotted arrows indicate the additional 32 directions (2 non-directional prediction modes in Fig.31 (not shown in the figure).
[0475] In various implementation examples, the luminance block can be referenced in the intra prediction of the chrominance block. That is, the chrominance component of the current block can be predicted based on the luminance component of the current block. There is a case where such intra prediction is called CCLM (cross-component linear model) prediction. Such an intra prediction mode of the chrominance block with reference to the luminance block (for example, called CCLM mode) can be added as one of the intra prediction modes of the chrominance block.
[0476] The intra prediction unit 124 can correct the pixel value after intra prediction based on the gradient of the reference pixel in the horizontal / vertical direction. Intra prediction accompanied by such correction is sometimes called PDPC (position dependent intraprediction combination). Information indicating whether PDPC is applied (for example, called a PDPC flag) is usually signaled at the CU level. In addition, the signaling of this information does not need to be limited to the CU level, but can also be other levels (for example, sequence level, picture level, slice level, tile level or CTU level).
[0477] Fig.32 This is a flowchart showing an example of the processing of the intra prediction unit 124 .
[0478] The intra prediction unit 124 selects one intra prediction mode from a plurality of intra prediction modes (step Sw_1). Next, the intra prediction unit 124 generates a predicted image according to the selected intra prediction mode (step Sw_2). Next, the intra prediction unit 124 determines the MPM (Most Probable Modes) (step Sw_3). The MPM is composed of, for example, 6 intra prediction modes. Two of the 6 intra prediction modes may be a Planar prediction mode and a DC prediction mode, and the remaining 4 modes may be directional prediction modes. Furthermore, the intra prediction unit 124 determines whether the intra prediction mode selected by step Sw_1 is included in the MPM (step Sw_4).
[0479] Here, if it is determined that the selected intra prediction mode is included in the MPM ("Yes" in step Sw_4), the intra prediction unit 124 sets the MPM flag to 1 (step Sw_5), and generates information indicating the selected intra prediction mode in the MPM (step Sw_6). In addition, the MPM flag set to 1 and the information indicating the intra prediction mode are encoded by the entropy encoding unit 110 as prediction parameters.
[0480] On the other hand, if it is determined that the selected intra-frame prediction mode is not included in the MPM ("No" in step Sw_4), the intra-frame prediction unit 124 sets the MPM flag to 0 (step Sw_7). Alternatively, the intra-frame prediction unit 124 does not set the MPM flag. Next, the intra-frame prediction unit 124 generates information indicating the selected intra-frame prediction mode among one or more intra-frame prediction modes not included in the MPM (step Sw_8). In addition, the MPM flag set to 0 and the information indicating the intra-frame prediction mode are respectively encoded as prediction parameters by the entropy coding unit 110. The information indicating the intra-frame prediction mode shows, for example, a value between 0 and 60.
[0481] [Inter-frame prediction unit]
[0482] The inter-frame prediction unit 126 performs inter-frame prediction (also called inter-screen prediction) of the current block with reference to a reference picture different from the current picture stored in the frame memory 122, thereby generating a predicted image (inter-frame prediction image). Inter-frame prediction is performed in units of the current block or the current sub-block within the current block. The sub-block is contained in the block and is a unit smaller than the block. The size of the sub-block can be 4×4 pixels, 8×8 pixels, or a size other than these. The size of the sub-block can also be switched by units such as slices, bricks, or pictures.
[0483] For example, the inter-frame prediction unit 126 performs motion search (motion estimation) in the reference picture for the current block or current sub-block, and finds the reference block or sub-block that is most consistent with the current block or current sub-block. Next, the inter-frame prediction unit 126 obtains motion information (such as a motion vector) for compensating for the motion or change from the reference block or sub-block to the current block or sub-block. The inter-frame prediction unit 126 performs motion compensation (or motion prediction) based on the motion information and generates an inter-frame prediction image of the current block or sub-block. The inter-frame prediction unit 126 outputs the generated inter-frame prediction image to the prediction control unit 128.
[0484] The motion information used in motion compensation can be signaled as an inter-frame prediction image in various forms. For example, a motion vector can be signaled. As another example, the difference between a motion vector and a predicted motion vector (motion vector predictor) can be signaled.
[0485] [See picture list]
[0486] Fig.33 is a diagram showing an example of each reference picture, Fig.34 1 is a conceptual diagram showing an example of a reference picture list. The reference picture list is a list showing one or more reference pictures stored in the frame memory 122. Fig.33 In the figure, rectangles represent pictures, arrows represent reference relationships between pictures, the horizontal axis represents time, I, P, and B in the rectangles represent intra-frame prediction pictures, single-prediction pictures, and dual-prediction pictures, respectively, and the numbers in the rectangles represent the decoding order. Fig.33 As shown in , the decoding order of each picture is I0, P1, B2, B3, B4, and the display order of each picture is I0, B3, B2, B4, P1. Fig.34 As shown in FIG, a reference picture list is a list of candidates for reference pictures. For example, a picture (or slice) can have more than one reference picture list. For example, if the current picture is a single-prediction picture, one reference picture list is used. If the current picture is a double-prediction picture, two reference picture lists are used. Fig.33 and Fig.34 In the example, picture B3, which is the current picture currPic, has two reference picture lists, namely the L0 list and the L1 list. When the current picture currPic is picture B3, the candidates for the reference pictures of the current picture currPic are I0, P1 and B2, and each reference picture list (i.e., the L0 list and the L1 list) represents these pictures. The inter-frame prediction unit 126 or the prediction control unit 128 specifies which picture in each reference picture list is actually referenced through the reference picture index refIdxLx. Fig.34 , reference pictures P1 and B2 are specified by reference picture indices refIdxL0 and refIdxL1.
[0487] Such a reference picture list may be generated in sequence units, picture units, slice units, brick units, CTU units, or CU units. Furthermore, a reference picture index indicating a reference picture referenced in inter prediction among reference pictures shown in the reference picture list may be encoded at a sequence level, a picture level, a slice level, a brick level, a CTU level, or a CU level. Furthermore, a common reference picture list may be used in a plurality of inter prediction modes.
[0488] [Basic process of inter-frame prediction]
[0489] Fig.35 This is a flowchart showing the basic process of inter-frame prediction.
[0490] The inter prediction unit 126 first generates a predicted image (steps Se_1 to Se_3). Next, the subtraction unit 104 generates a difference between the current block and the predicted image as a prediction residual (step Se_4).
[0491] Here, in the generation of the predicted image, the inter-frame prediction unit 126, for example, determines the motion vector (MV) of the current block (steps Se_1 and Se_2) and performs motion compensation (step Se_3) to generate the predicted image. In addition, in the determination of the MV, the inter-frame prediction unit 126, for example, performs the selection of candidate motion vectors (candidate MVs) (step Se_1) and the derivation of MVs (step Se_2) to determine the MV. For example, the inter-frame prediction unit 126 generates a candidate MV list, selects at least one candidate MV from the candidate MV list, and selects the candidate MV. In addition, for the candidate MV list, the MV derived in the past can be added as a candidate MV. In addition, in the derivation of the MV, the inter-frame prediction unit 126 can select at least one candidate MV from at least one candidate MV, and determine the selected at least one candidate MV as the MV of the current block. Alternatively, the inter-frame prediction unit 126 can search the area of the reference picture indicated by the candidate MV for the selected at least one candidate MV, and determine the MV of the current block. In addition, searching the area of the reference picture can be called motion estimation.
[0492] In the above example, steps Se_1 to Se_3 are performed by the inter-frame prediction unit 126 , but processing such as step Se_1 or step Se_2 may be performed by other components included in the encoding device 100 .
[0493] In addition, a candidate MV list may be prepared for each process in each inter-frame prediction mode, or a common candidate MV list may be used in multiple inter-frame prediction modes. Fig. 9 The processing of steps Sa_3 and Sa_4 shown in FIG. 1 is equivalent to the processing of step Se_3. Fig.30 The processing of step Sd_1b.
[0494] [MV export process]
[0495] Fig.36 This is a flowchart showing an example of MV derivation.
[0496] The inter-frame prediction unit 126 may derive the MV of the current block in a mode of encoding motion information (eg, MV). In this case, for example, the motion information may be encoded and signaled as a prediction parameter. That is, the encoded motion information is included in the stream.
[0497] Alternatively, the inter prediction unit 126 may derive the MV in a mode in which the motion information is not encoded. In this case, the motion information is not included in the stream.
[0498] Here, the modes for deriving MV include the normal inter mode, normal merge mode, FRUC mode, and affine mode described later. Among these modes, the modes for encoding motion information include the normal inter mode, normal merge mode, and affine mode (specifically, affine inter mode and affine merge mode). In addition, the motion information may include not only the MV but also the predicted MV selection information described later. In addition, the modes for not encoding motion information include the FRUC mode. The inter prediction unit 126 selects a mode for deriving the MV of the current block from these multiple modes, and derives the MV of the current block using the selected mode.
[0499] Fig.37 This is a flowchart showing another example of MV derivation.
[0500] The inter-frame prediction unit 126 can derive the MV of the current block in a mode of encoding the differential MV. In this case, for example, the differential MV is encoded and signaled as a prediction parameter. That is, the encoded differential MV is included in the stream. The differential MV is the difference between the MV of the current block and the predicted MV. In addition, the predicted MV is a predicted motion vector.
[0501] Alternatively, the inter prediction unit 126 may derive the MV in a mode in which the difference MV is not encoded. In this case, the encoded difference MV is not included in the stream.
[0502] Here, as described above, the modes for deriving the MV include the normal inter mode, normal merge mode, FRUC mode, and affine mode described later. Among these modes, the modes for encoding the differential MV include the normal inter mode and affine mode (specifically, affine inter mode). In addition, the modes for not encoding the differential MV include the FRUC mode, normal merge mode, and affine mode (specifically, affine merge mode). The inter prediction unit 126 selects a mode for deriving the MV of the current block from these multiple modes, and derives the MV of the current block using the selected mode.
[0503] [MV export mode]
[0504] Fig.38A and Fig.38B is a diagram showing an example of the classification of each mode derived from MV. Fig.38AAs shown in FIG. 1 , the MV derivation modes are roughly classified into three modes, depending on whether the motion information is encoded and whether the differential MV is encoded. The three modes are inter-frame mode, merge mode, and FRUC (frame rate up-conversion) mode. The inter-frame mode is a mode for performing motion search and is a mode for encoding motion information and differential MV. For example, Fig.38B As shown in FIG. 1 , the inter-frame mode includes the affine inter-frame mode and the normal inter-frame mode. The merge mode is a mode in which motion search is not performed. It is a mode in which MV is selected from the surrounding coded blocks and the MV of the current block is derived using the MV. The merge mode is basically a mode in which motion information is encoded and differential MV is not encoded. For example, Fig.38B As shown, the merge mode includes a normal merge mode (also called a normal merge mode or a regular merge mode), an MMVD (Merge with Motion Vector Difference) mode, a CIIP (Combined inter merge / intra prediction) mode, a triangle mode, an ATMVP mode, and an affine merge mode. Here, in the MMVD mode among the modes included in the merge mode, the differential MV is encoded as an exception. In addition, the above-mentioned affine merge mode and affine inter mode are modes included in the affine mode. The affine mode is a mode in which the MVs of the multiple sub-blocks constituting the current block are derived as the MV of the current block by assuming an affine transformation. The FRUC mode is a mode in which the MV of the current block is derived by searching between the encoded areas, and is a mode in which neither the motion information nor the differential MV is encoded. In addition, the details of each of these modes will be described later.
[0505] in addition, Fig.38A and Fig.38B The classification of each mode shown is an example and is not limited to this. For example, when the difference MV is encoded in the CIIP mode, the CIIP mode is classified as the inter-frame mode.
[0506] [MV Export > Normal Interframe Mode]
[0507] Normal inter mode is an inter prediction mode for deriving the MV of the current block by finding a block similar to the image of the current block from the area of the reference picture indicated by the candidate MV. In addition, in this normal inter mode, the differential MV is encoded.
[0508] Fig.39 is a flowchart showing an example of inter prediction based on normal inter mode.
[0509] The inter prediction unit 126 first obtains a plurality of candidate MVs for the current block based on information such as MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sg_1). That is, the inter prediction unit 126 creates a candidate MV list.
[0510] Next, the inter-frame prediction unit 126 extracts N (N is an integer greater than or equal to 2) candidate MVs from the plurality of candidate MVs obtained in step Sg_1 as prediction MV candidates according to a predetermined priority order (step Sg_2). The priority order is set in advance for each of the N candidate MVs.
[0511] Next, the inter-frame prediction unit 126 selects one prediction MV candidate from the N prediction MV candidates as the prediction MV of the current block (step Sg_3). At this time, the inter-frame prediction unit 126 encodes the prediction MV selection information used to identify the selected prediction MV into the stream. That is, the inter-frame prediction unit 126 outputs the prediction MV selection information as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.
[0512] Next, the inter-frame prediction unit 126 refers to the coded reference picture and derives the MV of the current block (step Sg_4). At this time, the inter-frame prediction unit 126 also encodes the difference value between the derived MV and the predicted MV as a difference MV into the stream. That is, the inter-frame prediction unit 126 outputs the difference MV as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130. In addition, the coded reference picture is a picture composed of a plurality of blocks reconstructed after encoding.
[0513] Finally, the inter prediction unit 126 performs motion compensation on the current block using the derived MV and the encoded reference picture, thereby generating a predicted image of the current block (step Sg_5). The processing of steps Sg_1 to Sg_5 is performed on each block. For example, if the processing of steps Sg_1 to Sg_5 is performed on all blocks included in the slice, the inter prediction using the normal inter mode for the slice is completed. In addition, if the processing of steps Sg_1 to Sg_5 is performed on all blocks included in the picture, the inter prediction using the normal inter mode for the picture is completed. In addition, instead of performing the processing of steps Sg_1 to Sg_5 on all blocks included in the slice, if the processing is performed on a part of the blocks, the inter prediction using the normal inter mode for the slice is completed. Similarly, if the processing of steps Sg_1 to Sg_5 is performed on a part of the blocks included in the picture, the inter prediction using the normal inter mode for the picture is completed.
[0514] The predicted image is the above-mentioned inter prediction signal. In addition, information indicating the inter prediction mode (normal inter mode in the above-mentioned example) used to generate the predicted image contained in the coded signal is coded as a prediction parameter, for example.
[0515] In addition, the candidate MV list can be common to the list used in other modes. In addition, the processing related to the candidate MV list can be applied to the processing related to the list used in other modes. The processing related to the candidate MV list is, for example, extraction or selection of candidate MVs from the candidate MV list, rearrangement of candidate MVs, or deletion of candidate MVs.
[0516] [MV Export > Normal Merge Mode]
[0517] The normal merge mode is an inter-frame prediction mode that selects a candidate MV from a candidate MV list as the MV of the current block to derive the MV. In addition, the normal merge mode is a merge mode in a narrow sense, and is also referred to as the merge mode for short. In this embodiment, the normal merge mode is distinguished from the merge mode, and the merge mode is used in a broad sense.
[0518] Fig.40 is a flowchart showing an example of inter-frame prediction based on normal merge mode.
[0519] The inter prediction unit 126 first obtains a plurality of candidate MVs for the current block based on information such as MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sh_1 ). That is, the inter prediction unit 126 creates a candidate MV list.
[0520] Next, the inter-frame prediction unit 126 selects one candidate MV from the multiple candidate MVs obtained in step Sh_1 to derive the MV of the current block (step Sh_2). At this time, the inter-frame prediction unit 126 encodes MV selection information for identifying the selected candidate MV into the stream. That is, the inter-frame prediction unit 126 outputs the MV selection information as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.
[0521] Finally, the inter prediction unit 126 uses the derived MV and the encoded reference picture to perform motion compensation on the current block, thereby generating a predicted image of the current block (step Sh_3). The processing of steps Sh_1 to Sh_3 is performed on each block, for example. For example, if the processing of steps Sh_1 to Sh_3 is performed on all blocks included in the slice, the inter prediction using the normal merge mode for the slice is completed. In addition, if the processing of steps Sh_1 to Sh_3 is performed on all blocks included in the picture, the inter prediction using the normal merge mode for the picture is completed. In addition, the processing of steps Sh_1 to Sh_3 may not be performed on all blocks included in the slice, but if it is performed on a part of the blocks, the inter prediction using the normal merge mode for the slice is completed. Similarly, if the processing of steps Sh_1 to Sh_3 is performed on a part of the blocks included in the picture, the inter prediction using the normal merge mode for the picture is completed.
[0522] Furthermore, information indicating the inter prediction mode (normal merge mode in the above example) used for generating the predicted image contained in the stream is encoded as a prediction parameter, for example.
[0523] Fig.41 This is a diagram used to illustrate an example of MV export processing of the current image based on the normal merge mode.
[0524] First, the inter-frame prediction unit 126 generates a candidate MV list in which candidate MVs are registered. The candidate MVs include MVs of multiple coded blocks spatially located around the current block, that is, spatial neighboring candidate MVs, MVs of blocks near the position where the current block is projected in the coded reference picture, that is, temporal neighboring candidate MVs, MVs generated by combining the MV values of the spatial neighboring candidate MV and the temporal neighboring candidate MV, that is, combined candidate MVs, and MVs with a value of zero, that is, zero candidate MVs.
[0525] Next, the inter prediction unit 126 selects one candidate MV from among the plurality of candidate MVs registered in the candidate MV list, and determines the one candidate MV as the MV of the current block.
[0526] Furthermore, the entropy coding unit 110 describes merge_idx, which is a signal indicating which candidate MV is selected, in the stream and encodes it.
[0527] In addition, Fig.41 The candidate MVs registered in the candidate MV list described in the figure are just examples, and may be a number different from the number in the figure, or a structure that does not include some types of candidate MVs in the figure, or a structure that adds candidate MVs other than the types of candidate MVs in the figure.
[0528] The final MV can be determined by using the MV of the current block derived by the normal merge mode and performing DMVR (dynamic motion vector refreshing) described later. In addition, the differential MV is not encoded in the normal merge mode, but is encoded in the MMVD mode. The MMVD mode selects one candidate MV from the candidate MV list in the same way as the normal merge mode, but encodes the differential MV. Fig.38B As shown, such MMVD can be classified as a merge mode together with the normal merge mode. In addition, the differential MV in the MMVD mode can be different from the differential MV used in the inter-frame mode. For example, the derivation of the differential MV in the MMVD mode can be a process with less processing than the derivation of the differential MV in the inter-frame mode.
[0529] Furthermore, a CIIP (Combined inter merge / intra prediction) mode may be performed in which a prediction image generated by inter prediction and a prediction image generated by intra prediction are superimposed to generate a prediction image of the current block.
[0530] In addition, the candidate MV list may also be referred to as a candidate list. In addition, merge_idx is MV selection information.
[0531] [MV Export > HMVP Mode]
[0532] Fig.42 This is a diagram for explaining an example of MV derivation processing of the current picture based on the HMVP mode.
[0533] In normal merge mode, the MV of the current block (e.g., CU) is determined by selecting one candidate MV from a candidate MV list generated by referring to an encoded block (e.g., CU). Here, other candidate MVs may also be registered in the candidate MV list. Such a mode of registering other candidate MVs is called HMVP mode.
[0534] In the HMVP mode, unlike the candidate MV list in the normal merge mode, a FIFO (First-In First-Out) buffer for HMVP is used to manage candidate MVs.
[0535] The FIFO cache stores motion information such as the MV of previously processed blocks in order from the newest to the oldest. In the management of the FIFO cache, each time a block is processed, the MV of the latest block (i.e., the CU that was processed just before) is saved in the FIFO cache, and the MV of the oldest CU in the FIFO cache (i.e., the first CU that was processed) is deleted from the FIFO cache. Fig.42In the example shown, HMVP1 is the MV of the newest block, and HMVP5 is the MV of the oldest block.
[0536] Furthermore, for example, the inter-frame prediction unit 126 checks, for each MV managed in the FIFO buffer, from HMVP1 in sequence whether the MV is different from all the candidate MVs registered in the candidate MV list of the normal merge mode. Furthermore, if the inter-frame prediction unit 126 determines that the MV is different from all the candidate MVs, it can add the MV managed in the FIFO buffer as a candidate MV to the candidate MV list of the normal merge mode. At this time, the candidate MVs registered from the FIFO buffer can be one or more.
[0537] By using the HMVP mode in this way, not only the MVs of blocks that are spatially or temporally adjacent to the current block, but also the MVs of blocks that have been processed in the past can be added to the candidates. As a result, the variation of candidate MVs in the normal merge mode is expanded, which increases the possibility of improving coding efficiency.
[0538] In addition, the above-mentioned MV may also be motion information. That is, the information stored in the candidate MV list and the FIFO buffer may include not only the value of the MV, but also information of the referenced picture, information indicating the direction and number of references, etc. In addition, the above-mentioned block is, for example, a CU.
[0539] in addition, Fig.42 The candidate MV list and FIFO buffer are one example. The candidate MV list and FIFO buffer can also be the same as Fig.42 Lists or buffers of different sizes, or in the same Fig.42 Structure of registering candidate MVs in different orders. The processing described here is common to both the encoding device 100 and the decoding device 200 .
[0540] In addition, the HMVP mode can also be applied to modes other than the normal merge mode. For example, in the FIFO buffer, the motion information such as the MV of the blocks processed in the past in the affine mode is stored in order from new to old and used as candidate MVs. The mode in which the HMVP mode is applied to the affine mode can be called the historical affine mode.
[0541] [MV Export > FRUC Mode]
[0542] The motion information may be derived on the decoding device 200 side without being signaled from the encoding device 100 side. For example, the motion information may be derived by performing a motion search on the decoding device 200 side. In this case, the motion search is performed on the decoding device 200 side without using the pixel value of the current block. Such modes of performing motion search on the decoding device 200 side include a FRUC (frame rate up-conversion) mode or a PMMVD (pattern matched motion vector derivation) mode, etc.
[0543] exist Fig.43 An example of FRUC processing is shown in . First, with reference to the MVs of each encoded block that is spatially or temporally adjacent to the current block, a list representing these MVs as candidate MVs is generated (i.e., a candidate MV list, which can be common to the candidate MV list of the normal merge mode) (step Si_1). Next, the optimal candidate MV is selected from the multiple candidate MVs registered in the candidate MV list (step Si_2). For example, an evaluation value of each candidate MV included in the candidate MV list is calculated, and based on the evaluation value, one candidate MV is selected as the optimal candidate MV. Next, based on the selected optimal candidate MV, the MV for the current block is derived (step Si_4). Specifically, for example, the selected optimal candidate MV is directly derived as the MV for the current block. In addition, for example, the MV for the current block can be derived by performing pattern matching in the surrounding area of the position in the reference picture corresponding to the selected optimal candidate MV. That is, the area around the optimal candidate MV may be searched using pattern matching and evaluation values in the reference picture, and when there is an MV with a better evaluation value, the optimal candidate MV may be updated to that MV and used as the final MV of the current block. The update to the MV with a better evaluation value may not be performed.
[0544] Finally, the inter-frame prediction unit 126 uses the derived MV and the encoded reference picture to perform motion compensation on the current block, thereby generating a predicted image of the current block (step Si_5). The processing of steps Si_1 to Si_5 is performed, for example, on each block. For example, if the processing of steps Si_1 to Si_5 is performed on all the blocks included in the slice, the inter-frame prediction using the FRUC mode for the slice is completed. In addition, if the processing of steps Si_1 to Si_5 is performed on all the blocks included in the picture, the inter-frame prediction using the FRUC mode for the picture is completed. In addition, the processing of steps Si_1 to Si_5 may not be performed on all the blocks included in the slice, but if it is performed on a part of the blocks, the inter-frame prediction using the FRUC mode for the slice is completed. Similarly, if the processing of steps Si_1 to Si_5 is performed on a part of the blocks included in the picture, the inter-frame prediction using the FRUC mode for the picture is completed.
[0545] The sub-block unit can also be processed in the same way as the above-mentioned block unit.
[0546] The evaluation value can be calculated by various methods. For example, the reconstructed image of the area in the reference picture corresponding to the MV is compared with the reconstructed image of a predetermined area (for example, the area can be an area of another reference picture or an area of an adjacent block of the current picture as shown below). In addition, the difference in pixel values of the two reconstructed images can be calculated and used for the evaluation value of the MV. In addition, the evaluation value can be calculated using information other than the difference value.
[0547] Next, pattern matching is described in detail. First, a candidate MV included in the candidate MV list (also called merge list) is selected as the starting point of the search based on pattern matching. As pattern matching, the first pattern matching or the second pattern matching can be used. There are cases where the first pattern matching and the second pattern matching are respectively called bilateral matching and template matching.
[0548] [MV Export > FRUC > Bilateral Matching]
[0549] In the first pattern matching, pattern matching is performed between two blocks in two different reference pictures along the motion trajectory of the current block. Therefore, in the first pattern matching, as the prescribed area used for calculating the evaluation value of the candidate MV, an area in another reference picture along the motion trajectory of the current block is used.
[0550] Fig.441 is a diagram for explaining an example of the first pattern matching (bilateral matching) between two blocks in two reference pictures along a motion trajectory. Fig.44 As shown, in the first mode matching, the best matching pair is searched among the pairs of two blocks in two different reference pictures (Ref0, Ref1) along the motion trajectory of the current block (Cur block) to derive two MVs (MV0, MV1). Specifically, for the current block, the difference between the reconstructed image at a specified position in the first encoded reference picture (Ref0) specified by the candidate MV and the reconstructed image at a specified position in the second encoded reference picture (Ref1) specified by the symmetric MV that scales the candidate MV by the display time interval is derived, and the evaluation value is calculated using the obtained difference value. The candidate MV with the best evaluation value can be selected as the best candidate MV among multiple candidate MVs.
[0551] Under the assumption of a continuous motion trajectory, the MVs (MV0, MV1) indicating the two reference blocks are proportional to the temporal distances (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, when the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the two reference pictures are equal, in the first pattern matching, mirror-symmetric bidirectional MVs are derived.
[0552] [MV Export > FRUC > Template Matching]
[0553] In the second pattern matching (template matching), pattern matching is performed between a template in the current picture (a block adjacent to the current block in the current picture (e.g., an upper and / or left adjacent block)) and a block in the reference picture. Therefore, in the second pattern matching, a block adjacent to the current block in the current picture is used as a predetermined area for calculating the evaluation value of the candidate MV.
[0554] Fig.45 is a diagram for explaining an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture. Fig.45 As shown, in the second mode matching, the MV of the current block is derived by searching the reference picture (Ref0) for the block that best matches the block adjacent to the current block (Cur block) in the current picture (Cur Pic). Specifically, for the current block, the difference between the reconstructed image of the coded area of both or one of the left and upper adjacent areas and the reconstructed image at the same position in the coded reference picture (Ref0) specified by the candidate MV is derived, and the evaluation value is calculated using the obtained difference value. The candidate MV with the best evaluation value can be selected as the best candidate MV from multiple candidate MVs.
[0555] Information indicating whether such a FRUC mode is applied (e.g., called a FRUC flag) can be signaled at the CU level. In addition, when the FRUC mode is applied (e.g., when the FRUC flag is true), information indicating the method of pattern matching that can be applied (first pattern matching or second pattern matching) can be signaled at the CU level. In addition, the signaling of this information does not need to be limited to the CU level, but can also be other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or sub-block level).
[0556] [MV Export > Affine Mode]
[0557] The affine mode is a mode for generating MVs using affine transformation, and for example, MVs can be derived in sub-block units based on MVs of a plurality of adjacent blocks. This mode is sometimes referred to as an affine motion compensation prediction mode.
[0558] Fig.46A FIG. 1 is a diagram for explaining an example of deriving an MV in sub-block units based on MVs of a plurality of adjacent blocks. Fig.46A In the example, the current block includes 16 sub-blocks of 4×4 pixels. Here, based on the MV of the adjacent block, the motion vector v of the upper left corner control point of the current block is derived. 0 Similarly, based on the MV of the adjacent sub-block, the motion vector v of the upper right corner control point of the current block is derived 1 Next, the two motion vectors v are converted into 0 and v 1 Projection, derive the motion vectors (v x , v y ).
[0559] [Formula 1]
[0560]
[0561] Here, x and y represent the horizontal position and vertical position of the sub-block, respectively, and w represents a preset weight coefficient.
[0562] Information indicating such an affine mode (e.g., called an affine flag) may be signaled at the CU level. In addition, the signaling of the information indicating the affine mode need not be limited to the CU level, but may be other levels (e.g., sequence level, picture level, slice level, tile level, CTU level, or sub-block level).
[0563] In addition, such an affine mode may include several modes in which the MVs of the upper left and upper right control points are derived in different ways. For example, the affine mode includes two modes: an affine inter-frame (also called an affine normal inter-frame) mode and an affine merge mode.
[0564] Fig.46B FIG. 1 is a diagram for explaining an example of deriving a sub-block-based MV in an affine mode using three control points. Fig.46B In the example, the current block includes 16 sub-blocks of 4×4 pixels. Here, based on the MV of the adjacent block, the motion vector v of the upper left corner control point of the current block is derived. 0 Similarly, based on the MV of the adjacent block, the motion vector v of the upper right corner control point of the current block is derived 1 , based on the MV of the adjacent block, derive the motion vector v of the lower left corner control point of the current block 2 And, through the following formula (1B), the three motion vectors v 0 、v 1 and v 2 Projection, derive the motion vectors (v x , v y ).
[0565] [Formula 2]
[0566]
[0567] Here, x and y represent the horizontal position and vertical position of the center of the sub-block, respectively, and w and h represent pre-set weight coefficients. Alternatively, w represents the width of the current block, and h represents the height of the current block.
[0568] Affine modes using different numbers of control points (e.g., 2 and 3) can be switched and signaled at the CU level. In addition, information indicating the number of control points of the affine mode used at the CU level can be signaled at other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or sub-block level).
[0569] In addition, such an affine mode with three control points may include several modes with different methods of deriving the MVs of the upper left, upper right, and lower left control points. For example, similar to the above-mentioned affine mode with two control points, the affine mode with three control points includes two modes, namely, the affine inter-frame mode and the affine merge mode.
[0570] In addition, in the affine mode, the size of each sub-block included in the current block is not limited to 4×4 pixels, but may be other sizes. For example, the size of each sub-block may be 8×8 pixels.
[0571] [MV Export > Affine Mode > Control Points]
[0572] Fig.47A , Fig.47B and Fig.47C This is a conceptual diagram for explaining an example of MV derivation of control points in the affine mode.
[0573] In affine mode, such as Fig.47A As shown, for example, based on a plurality of MVs corresponding to blocks encoded in affine mode among the encoded blocks A (left), B (upper), C (upper right), D (lower left), and E (upper left) adjacent to the current block, the predicted MVs of the control points of the current block are calculated. Specifically, the encoded blocks A (left), B (upper), C (upper right), D (lower left), and E (upper left) are checked in this order to determine the first valid block encoded in affine mode. Based on a plurality of MVs corresponding to the determined blocks, the MVs of the control points of the current block are calculated.
[0574] For example, Fig.47B As shown in the figure, when the block A adjacent to the left of the current block is encoded in an affine mode with two control points, the motion vector v projected to the upper left corner and the upper right corner of the encoded block including the block A is derived. 3 and v 4 Next, according to the derived motion vector v 3 and v 4 , calculate the motion vector v of the upper left corner control point of the current block 0 and the motion vector v of the upper right control point 1 .
[0575] For example, Fig.47C As shown in the figure, when the block A adjacent to the left of the current block is encoded in an affine mode with three control points, the motion vector v projected to the upper left corner, upper right corner and lower left corner of the encoded block including the block A is derived. 3 、v 4 and v 5 Next, according to the derived motion vector v 3 、v 4 and v 5 , calculate the motion vector v of the upper left corner control point of the current block 0 , the motion vector v of the upper right control point 1 and the motion vector v of the lower left control point 2 .
[0576] in addition, Figures 47A to 47C The MV derivation method shown can be used in the following Fig.50 The derivation of the MV of each control point of the current block in step Sk_1 shown in FIG. 1 can also be used for the following Fig.51The derivation of the predicted MV of each control point of the current block in step Sj_1 is shown.
[0577] Fig.48A and Fig.48B This is a conceptual diagram for explaining another example of derivation of a control point MV in the affine mode.
[0578] Fig.48A is a diagram used to illustrate an affine pattern with 2 control points.
[0579] In this affine model, if Fig.48A As shown in FIG. 1 , the MV selected from the MVs of the coded blocks A, B, and C adjacent to the current block is used as the motion vector v of the upper left corner control point of the current block. 0 Similarly, the MV selected from the MVs of the coded blocks D and E adjacent to the current block is used as the motion vector v of the upper right corner control point of the current block. 1 .
[0580] Fig.48B is a diagram used to illustrate an affine pattern with three control points.
[0581] In this affine model, if Fig.48B As shown in FIG. 1 , the MV selected from the MVs of the coded blocks A, B, and C adjacent to the current block is used as the motion vector v of the upper left corner control point of the current block. 0 Similarly, the MV selected from the MVs of the coded blocks D and E adjacent to the current block is used as the motion vector v of the upper right corner control point of the current block. 1 Furthermore, the MV selected from the MVs of the coded blocks F and G adjacent to the current block is used as the motion vector v of the lower left corner control point of the current block. 2 .
[0582] in addition, Fig.48A and Fig.48B The MV derivation method shown can be used in the following Fig.50 The derivation of the MV of each control point of the current block in step Sk_1 shown in FIG. 1 can also be used for the following Fig.51 The predicted MV of each control point of the current block is derived in step Sj_1.
[0583] Here, for example, when affine modes with different numbers of control points (for example, 2 and 3) are switched at the CU level and signaled, the number of control points in the coded block and the current block may be different.
[0584] Fig.49A and Fig.49BThis is a conceptual diagram for explaining an example of a method for deriving an MV of a control point when the number of control points in a coded block and a current block is different.
[0585] For example, Fig.49A As shown in FIG. 1 , the current block has three control points at the upper left corner, the upper right corner, and the lower left corner, and the block A adjacent to the left of the current block is encoded in an affine mode with two control points. In this case, the motion vector v projected to the upper left corner and the upper right corner of the encoded block including the block A is derived. 3 and v 4 And, according to the derived motion vector v 3 and v 4 , calculate the motion vector v of the upper left corner control point of the current block 0 and the motion vector v of the upper right control point 1 Then, according to the derived motion vector v 0 and v 1 , calculate the motion vector v of the lower left control point 2 .
[0586] For example, Fig.49B As shown in FIG. 1 , the current block has two control points at the upper left corner and the upper right corner, and the block A adjacent to the left of the current block is encoded in an affine mode with three control points. In this case, the motion vector v projected to the upper left corner, the upper right corner, and the lower left corner of the encoded block including the block A is derived. 3 、v 4 and v 5 Next, according to the derived motion vector v 3 、v 4 and v 5 , calculate the motion vector v of the upper left corner control point of the current block 0 and the motion vector v of the upper right control point 1 .
[0587] in addition, Fig.49A and Fig.49B The MV derivation method shown can be used in the following Fig.50 The derivation of the MV of each control point of the current block in step Sk_1 shown in FIG. 1 can also be used for the following Fig.51 The predicted MV of each control point of the current block is derived in step Sj_1.
[0588] [MV Export > Affine Mode > Affine Merge Mode]
[0589] Fig.50 This is a flowchart showing an example of the affine merge mode.
[0590] In the affine merge mode, first, the inter-frame prediction unit 126 derives the MV of each control point of the current block (step Sk_1). Fig.46A The top left and top right corners of the current block are shown, or Fig.46B The points shown are the upper left corner, upper right corner, and lower left corner of the current block. At this time, the inter prediction unit 126 may encode MV selection information for identifying the derived two or three MVs into the stream.
[0591] For example, when using Figures 47A to 47C In the case of the MV derivation method shown, Fig.47A As shown, the inter-frame prediction unit 126 checks the encoded blocks A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) in this order to determine the first valid block after being encoded in the affine mode.
[0592] The inter-frame prediction unit 126 uses the first valid block that is determined to be encoded in the affine mode to derive the MV of the control point. For example, when block A is determined and block A has two control points, Fig.47B As shown, the inter-frame prediction unit 126 predicts the upper left corner and the upper right corner of the coded block including the block A based on the motion vector v 3 and v 4 , calculate the motion vector v of the upper left corner control point of the current block 0 and the motion vector v of the upper right control point 1 For example, the inter-frame prediction unit 126 calculates the motion vectors v at the upper left corner and the upper right corner of the coded block. 3 and v 4 Project it into the current block and calculate the motion vector v of the upper left corner control point of the current block 0 and the motion vector v of the upper right control point 1 .
[0593] Alternatively, if block A is determined and has 3 control points, Fig.47C As shown, the inter-frame prediction unit 126 generates the motion vectors v at the upper left corner, upper right corner, and lower left corner of the coded blocks including block A. 3 , v 4 and v 5 , calculate the motion vector v of the upper left corner control point of the current block 0 , the motion vector v of the upper right control point 1 and the motion vector v of the lower left control point 2 For example, the inter-frame prediction unit 126 calculates the motion vectors v at the upper left corner, upper right corner, and lower left corner of the coded block. 3 、v 4 and v 5Project it into the current block and calculate the motion vector v of the upper left corner control point of the current block 0 , the motion vector v of the upper right control point 1 and the motion vector v of the lower left control point 2 .
[0594] In addition, as mentioned above Fig.49A As shown, when block A is determined and block A has 2 control points, the MV of 3 control points can also be calculated, as described above. Fig.49B As shown, when block A is determined and block A has 3 control points, MVs of 2 control points can also be calculated.
[0595] Next, the inter-frame prediction unit 126 performs motion compensation on each of the multiple sub-blocks included in the current block. That is, the inter-frame prediction unit 126 uses two motion vectors v for each of the multiple sub-blocks. 0 and v 1 and the above equation (1A), or using three motion vectors v 0 、v 1 and v 2 and the above-mentioned formula (1B), the MV of the sub-block is calculated as the affine MV (step Sk_2). Next, the inter-frame prediction unit 126 uses these affine MVs and the encoded reference picture to perform motion compensation on the sub-block (step Sk_3). If the processing of steps Sk_2 and Sk_3 is performed on all sub-blocks included in the current block, the processing of generating the predicted image using the affine merge mode for the current block is completed. That is, motion compensation is performed on the current block to generate the predicted image of the current block.
[0596] In addition, in step Sk_1, the candidate MV list mentioned above may also be generated. The candidate MV list may be, for example, a list containing candidate MVs derived using a plurality of MV derivation methods for each control point. The plurality of MV derivation methods may be Figures 47A to 47C The MV export method shown in Fig.48A and Fig.48B The MV export method shown in Fig.49A and Fig.49B The MV derivation method shown, and any combination of other MV derivation methods.
[0597] In addition, the candidate MV list may include candidate MVs of a mode other than the affine mode and a mode for performing prediction in sub-block units.
[0598] In addition, as a candidate MV list, for example, a candidate MV list including candidate MVs of an affine merge mode with 2 control points and a candidate MV list including candidate MVs of an affine merge mode with 3 control points may be generated. Alternatively, a candidate MV list including candidate MVs of an affine merge mode with 2 control points and a candidate MV list including candidate MVs of an affine merge mode with 3 control points may be generated separately. Alternatively, a candidate MV list including candidate MVs of one of an affine merge mode with 2 control points and an affine merge mode with 3 control points may be generated. The candidate MVs may be, for example, MVs of encoded blocks A (left), B (top), C (top right), D (bottom left) and E (top left), or MVs of valid blocks among these blocks.
[0599] In addition, as the MV selection information, an index indicating which candidate MV in the candidate MV list it is may be transmitted.
[0600] [MV Export > Affine Mode > Affine Inter Mode]
[0601] Fig.51 This is a flowchart showing an example of the affine inter mode.
[0602] In the affine inter mode, first, the inter prediction unit 126 derives the prediction MV (v 0 , v 1 ) or (v 0 , v 1 , v 2 )(Step Sj_1). Control points are as follows Fig.46A or Fig.46B As shown, it is the point at the upper left corner, upper right corner or lower left corner of the current block.
[0603] For example, when using Fig.48A and Fig.48B In the case of the MV derivation method shown in FIG. 1 , the inter-frame prediction unit 126 selects Fig.48A or Fig.48B The MV of a block in the coded blocks near each control point of the current block shown is used to derive the predicted MV (v 0 , v 1 ) or (v 0 , v 1 , v 2 ). At this time, the inter-frame prediction unit 126 encodes the prediction MV selection information used to identify the selected 2 or 3 prediction MVs into the stream.
[0604] For example, the inter-frame prediction unit 126 may determine which block's MV to select as the predicted MV of the control point from the coded blocks adjacent to the current block by using cost evaluation, and may describe a flag indicating which predicted MV is selected in the bitstream. That is, the inter-frame prediction unit 126 outputs the predicted MV selection information such as the flag as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.
[0605] Next, the inter-frame prediction unit 126 updates the prediction MVs selected or derived in step Sj_1 respectively (step Sj_2) while performing motion search (steps Sj_3 and Sj_4). That is, the inter-frame prediction unit 126 uses the MVs of each sub-block corresponding to the updated prediction MV as affine MVs and calculates using the above-mentioned formula (1A) or formula (1B) (step Sj_3). Next, the inter-frame prediction unit 126 uses these affine MVs and the encoded reference picture to perform motion compensation on each sub-block (step Sj_4). Whenever the prediction MV is updated in step Sj_2, the processing of steps Sj_3 and Sj_4 is performed on all blocks in the current block. As a result, the inter-frame prediction unit 126 determines the prediction MV that obtains the minimum cost as the MV of the control point in the motion search loop (step Sj_5). At this time, the inter-frame prediction unit 126 further encodes the difference value between the determined MV and the prediction MV as the difference MV into the stream. That is, the inter prediction unit 126 outputs the difference MV as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130 .
[0606] Finally, the inter-frame prediction unit 126 performs motion compensation on the current block by using the determined MV and the encoded reference picture, thereby generating a predicted image of the current block (step Sj_6).
[0607] In addition, in step Sj_1, the candidate MV list mentioned above may also be generated. The candidate MV list may be, for example, a list of candidate MVs derived using a plurality of MV derivation methods for each control point. The plurality of MV derivation methods may be Figures 47A to 47C The MV export method shown in Fig.48A and Fig.48B The MV export method shown in Fig.49A and Fig.49B The MV derivation method shown, and any combination of other MV derivation methods.
[0608] Furthermore, the candidate MV list may include candidate MVs of a mode other than the affine mode that performs prediction in sub-block units.
[0609] In addition, as a candidate MV list, a candidate MV list including candidate MVs of an affine inter mode with 2 control points and candidate MVs of an affine inter mode with 3 control points may be generated. Alternatively, a candidate MV list including candidate MVs of an affine inter mode with 2 control points and a candidate MV list including candidate MVs of an affine inter mode with 3 control points may be generated separately. Alternatively, a candidate MV list including candidate MVs of an affine inter mode with 2 control points and one of affine inter modes with 3 control points may be generated. The candidate MVs may be, for example, MVs of the encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left), or MVs of valid blocks among these blocks.
[0610] In addition, as the predicted MV selection information, an index indicating which candidate MV in the candidate MV list it is may be transmitted.
[0611] [MV Export > Triangle Mode]
[0612] In the above example, the inter-frame prediction unit 126 generates a rectangular prediction image for the rectangular current block. However, the inter-frame prediction unit 126 may generate a plurality of prediction images of shapes different from the rectangle for the rectangular current block, and combine these plurality of prediction images to generate a final rectangular prediction image. The shape different from the rectangle may be, for example, a triangle.
[0613] Fig.52A This is a diagram used to explain the generation of predicted images of two triangles.
[0614] The inter-frame prediction unit 126 performs motion compensation on the first partition of the triangle in the current block using the first MV of the first partition, thereby generating a predicted image of the triangle. Similarly, the inter-frame prediction unit 126 performs motion compensation on the second partition of the triangle in the current block using the second MV of the second partition, thereby generating a predicted image of the triangle. Furthermore, the inter-frame prediction unit 126 combines these predicted images to generate a predicted image of the same rectangle as the current block.
[0615] In addition, as the prediction image of the first partition, the first rectangular prediction image corresponding to the current block can be generated using the first MV. In addition, as the prediction image of the second partition, the second rectangular prediction image corresponding to the current block can be generated using the second MV. The prediction image of the current block can be generated by weighted addition of the first prediction image and the second prediction image. In addition, the weighted addition can be performed only on a portion of the boundary between the first partition and the second partition.
[0616] Fig.52B1 is a conceptual diagram showing an example of a first portion of a first partition overlapping a second partition and a first sample set and a second sample set that can be weighted as part of a modification process. The first portion can be, for example, one-fourth of the width or height of the first partition. In other examples, the first portion can also have a width corresponding to N samples adjacent to an edge of the first partition. Here, N is an integer greater than zero, for example, N can be an integer of 2. Fig.52B A rectangular partition is represented as a rectangular portion having a width that is one-fourth of the width of the first partition. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples inside the first portion. Fig.52B The central example of represents a rectangular partition having a rectangular portion having a height that is one-fourth of the height of the first partition. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples inside the first portion. Fig.52B The example on the right of shows a triangular partition of a polygonal part having a height corresponding to two samples. Here, the first sample set includes samples outside the first part and samples inside the first part, and the second sample set includes samples inside the first part.
[0617] The first portion may be a portion of the first partition that overlaps with an adjacent partition. Fig.52C This is a conceptual diagram showing a portion of the first partition that overlaps a portion of an adjacent partition, that is, a first portion of the first partition. For simplicity of description, a rectangular partition having a portion that overlaps a spatially adjacent rectangular partition is shown. Partitions having other shapes such as triangular partitions may also be used, and the overlapping portion may overlap a spatially or temporally adjacent partition.
[0618] Furthermore, an example is shown in which predicted images are generated for each of two partitions using inter-frame prediction, but a predicted image may be generated for at least one partition using intra-frame prediction.
[0619] Fig.53 This is a flowchart showing an example of the triangle mode.
[0620] In the triangular mode, first, the inter-frame prediction unit 126 divides the current block into the first partition and the second partition (step Sx_1). At this time, the inter-frame prediction unit 126 can encode the partition information related to the division into each partition as a prediction parameter into the stream. That is, the inter-frame prediction unit 126 can output the partition information as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.
[0621] Next, the inter prediction unit 126 first obtains a plurality of candidate MVs for the current block based on information such as MVs of a plurality of coded blocks temporally or spatially surrounding the current block (step Sx_2). That is, the inter prediction unit 126 creates a candidate MV list.
[0622] Next, the inter-frame prediction unit 126 selects the candidate MV of the first partition and the candidate MV of the second partition as the first MV and the second MV, respectively, from the plurality of candidate MVs obtained in step Sx_2 (step Sx_3). At this time, the inter-frame prediction unit 126 may encode the MV selection information for identifying the selected candidate MV as a prediction parameter into the stream. That is, the inter-frame prediction unit 126 may output the MV selection information as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.
[0623] Next, the inter-frame prediction unit 126 generates a first predicted image by performing motion compensation using the selected first MV and the encoded reference picture (step Sx_4). Similarly, the inter-frame prediction unit 126 generates a second predicted image by performing motion compensation using the selected second MV and the encoded reference picture (step Sx_5).
[0624] Finally, the inter prediction unit 126 generates a prediction image of the current block by performing weighted addition of the first prediction image and the second prediction image (step Sx_6).
[0625] In addition, Fig.52A In the example shown, the first partition and the second partition are each a triangle, but they may also be a trapezoid or may be different shapes from each other. Fig.52A In the example shown, the current block is composed of 2 partitions, but it can also be composed of 3 or more partitions.
[0626] In addition, the first partition and the second partition may overlap. That is, the first partition and the second partition may include the same pixel region. In this case, the predicted image in the first partition and the predicted image in the second partition may be used to generate the predicted image of the current block.
[0627] Furthermore, although this example shows an example in which predicted images are generated by inter-frame prediction for both of the two partitions, a predicted image may be generated by intra-frame prediction for at least one partition.
[0628] In addition, the candidate MV list used to select the first MV and the candidate MV list used to select the second MV may be different or the same candidate MV list.
[0629] In addition, the partition information may include at least an index indicating a direction of partitioning the current block into a plurality of partitions. The MV selection information may include an index indicating a selected first MV and an index indicating a selected second MV. One index may also indicate a plurality of information. For example, one index indicating a part or the whole of the partition information and a part or the whole of the MV selection information may be encoded.
[0630] [MV Export > ATMVP Mode]
[0631] Fig.54 This is a diagram showing an example of the ATMVP mode for deriving MV in sub-block units.
[0632] The ATMVP mode is a mode classified as a merge mode. For example, in the ATMVP mode, candidate MVs in sub-block units are registered in a candidate MV list for a normal merge mode.
[0633] Specifically, in the ATMVP model, first, Fig.54 As shown, in the coded reference picture specified by the MV (MV0) of the block adjacent to the lower left of the current block, a temporal MV reference block corresponding to the current block is determined. Next, for each sub-block in the current block, the MV used when encoding the area corresponding to the sub-block in the temporal MV reference block is determined. The MV determined in this way is included in the candidate MV list as a candidate MV of the sub-block of the current block. When such a candidate MV of each sub-block is selected from the candidate MV list, motion compensation using the candidate MV as the MV of the sub-block is performed on the sub-block. In this way, a predicted image of each sub-block is generated.
[0634] In addition, Fig.54 In the example shown, the block adjacent to the lower left of the current block is used as the surrounding MV reference block, but other blocks may be used. In addition, the size of the sub-block may be 4×4 pixels, 8×8 pixels, or other sizes. The size of the sub-block may also be switched in units such as slices, bricks, or pictures.
[0635] [Sports Search>DMVR]
[0636] Fig.55 This is a diagram showing the relationship between the merge mode and DMVR.
[0637] The inter-frame prediction unit 126 derives the MV of the current block in the merge mode (step S1_1). Next, the inter-frame prediction unit 126 determines whether to perform a search for the MV, that is, a motion search (step S1_2). Here, if the inter-frame prediction unit 126 determines not to perform a motion search ("No" in step S1_2), the MV derived in step S1_1 is determined as the final MV for the current block (step S1_4). That is, in this case, the MV of the current block is determined in the merge mode.
[0638] On the other hand, if it is determined in step S1_1 that motion search is to be performed ("Yes" in step S1_2), the inter-frame prediction unit 126 searches the surrounding area of the reference picture represented by the MV derived in step S1_1 to derive the final MV for the current block (step S1_3). That is, in this case, the MV of the current block is determined by DMVR.
[0639] Fig.56 This is a conceptual diagram for explaining an example of DMVR for determining MV.
[0640] First, for example, in merge mode, candidate MVs (L0 and L1) are selected for the current block. Next, according to candidate MV (L0), reference pixels are determined from the first reference picture (L0) of the coded picture in the L0 list. Similarly, according to candidate MV (L1), reference pixels are determined from the second reference picture (L1) of the coded picture in the L1 list. The template is generated by taking the average value of these reference pixels.
[0641] Next, the template is used to search the surrounding areas of the candidate MVs of the first reference picture (L0) and the second reference picture (L1), and the MV with the smallest cost is determined as the final MV of the current block. In addition, the cost can be calculated using, for example, the difference between the pixel values of the template and the pixel values of the search area and the candidate MV values.
[0642] Even if it is not the process described here, any process may be used as long as it is a process that can search the vicinity of the candidate MV and derive the final MV.
[0643] Fig.57 This is a conceptual diagram for explaining another example of DMVR for determining MV. Fig.57 This example is shown with Fig.56 The example of DMVR shown is different in that the cost is calculated without generating a template.
[0644] First, the inter prediction unit 126 searches for the reference block periphery included in the reference pictures of the L0 list and the L1 list based on the initial MV as the candidate MV obtained from the candidate MV list. Fig.57As shown, the initial MV corresponding to the reference block of the L0 list is InitMV_L0, and the initial MV corresponding to the reference block of the L1 list is InitMV_L1. The inter-frame prediction unit 126 first sets the search position for the reference picture of the L0 list in the motion search. The differential vector representing the set search position, specifically the differential vector from the position represented by the initial MV (i.e., InitMV_L0) to the search position is MVd_L0. In addition, the inter-frame prediction unit 126 determines the search position in the reference picture of the L1 list. The search position is represented by the differential vector from the position represented by the initial MV (i.e., InitMV_L1) to the search position. Specifically, the inter-frame prediction unit 126 determines the differential vector as MVd_L1 by mirroring MVd_L0. That is, in the reference pictures of the L0 list and the L1 list, the inter-frame prediction unit 126 uses the position symmetrical to the position represented by the initial MV as the search position. The inter prediction unit 126 calculates the sum of absolute differences (SAD) of pixel values in the block at each search position as a cost, and finds a search position at which the cost is minimized.
[0645] Fig.58A is a diagram showing an example of motion search in DMVR. Fig.58B : is a flowchart showing an example of the motion search.
[0646] First, the inter-frame prediction unit 126 calculates the cost of the search position (also called the starting point) represented by the initial MV and the eight search positions around it in step 1. Then, the inter-frame prediction unit 126 determines whether the cost of the search position other than the starting point is the smallest. Here, if the inter-frame prediction unit 126 determines that the cost of the search position other than the starting point is the smallest, it moves to the search position with the smallest cost and performs the process of step 2. On the other hand, if the cost of the starting point is the smallest, the inter-frame prediction unit 126 skips the process of step 2 and performs the process of step 3.
[0647] In step 2, the inter-frame prediction unit 126 uses the search position moved according to the processing result of step 1 as a new starting point, and performs the same search as the processing of step 1. In addition, the inter-frame prediction unit 126 determines whether the cost of the search position other than the starting point is the smallest. Here, if the cost of the search position other than the starting point is the smallest, the inter-frame prediction unit 126 performs the processing of step 4. On the other hand, if the cost of the starting point is the smallest, the inter-frame prediction unit 126 performs the processing of step 3.
[0648] In step 4, the inter prediction unit 126 treats the search position of the starting point as the final search position, and determines the difference between the position indicated by the initial MV and the final search position as a difference vector.
[0649] In step 3, the inter-frame prediction unit 126 determines the pixel position with the minimum cost based on the cost of the four points above, below, left, and right of the starting point of step 1 or step 2, and uses the pixel position as the final search position. The pixel position with the minimum cost is determined by weighted addition of the vectors of the four points above, below, left, and right ((0, 1), (0, -1), (-1, 0), (1, 0)) using the cost at the search position of each of the four points as the weight. In addition, the inter-frame prediction unit 126 determines the difference between the position represented by the initial MV and the final search position as a difference vector.
[0650] [Motion Compensation > BIO / OBMC / LIC]
[0651] In motion compensation, there are modes for generating a predicted image and correcting the predicted image. Examples of such modes are BIO, OBMC, and LIC, which will be described later.
[0652] Fig.59 This is a flowchart showing an example of generating a predicted image.
[0653] The inter prediction unit 126 generates a predicted image (step Sm_1 ), and corrects the predicted image using any of the above-mentioned modes (step Sm_2 ).
[0654] Fig.60 This is a flowchart showing another example of generating a predicted image.
[0655] The inter-frame prediction unit 126 derives the MV of the current block (step Sn_1). Next, the inter-frame prediction unit 126 generates a predicted image using the MV (step Sn_2) and determines whether to perform a correction process (step Sn_3). Here, if the inter-frame prediction unit 126 determines to perform a correction process ("Yes" in step Sn_3), it generates a final predicted image by correcting the predicted image (step Sn_4). In addition, in the LIC described later, brightness and color difference can be corrected in step Sn_4. On the other hand, if the inter-frame prediction unit 126 determines not to perform a correction process ("No" in step Sn_3), it outputs the predicted image as the final predicted image without correction (step Sn_5).
[0656] [Motion Compensation > OBMC]
[0657] An inter-frame prediction image can be generated using not only the motion information of the current block obtained by motion search but also the motion information of the adjacent blocks. Specifically, a prediction image based on the motion information obtained by motion search (in the reference picture) and a prediction image based on the motion information of the adjacent blocks (in the current picture) can be weighted and added to generate an inter-frame prediction image in sub-block units in the current block. Such inter-frame prediction (motion compensation) is sometimes called OBMC (ovulerlapped block motion compensation) or OBMC mode.
[0658] In the OBMC mode, information indicating the size of the sub-block used for OBMC (e.g., OBMC block size) may be signaled at the sequence level. Furthermore, information indicating whether the OBMC mode is applied (e.g., OBMC flag) may be signaled at the CU level. In addition, the signaling level of this information does not need to be limited to the sequence level and the CU level, and may be other levels (e.g., picture level, slice level, tile level, CTU level, or sub-block level).
[0659] The OBMC mode will be described in more detail. Fig.61 and Fig.62 It is a flowchart and a conceptual diagram for explaining the outline of the predicted image correction processing based on OBMC.
[0660] First, if Fig.62 As shown, the MV assigned to the current block is used to obtain a predicted image (Pred) based on conventional motion compensation. Fig.62 In the figure, the arrow "MV" points to the reference picture, indicating what the current block of the current picture refers to in order to obtain the predicted image.
[0661] Next, the MV (MV_L) derived for the coded left adjacent block is applied (reused) to the current block to obtain a predicted image (Pred_L). The MV (MV_L) is represented by an arrow "MV_L" pointing from the current block to the reference picture. Then, the first correction of the predicted image is performed by superimposing the two predicted images Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.
[0662] Similarly, the MV (MV_U) derived for the encoded upper adjacent block is applied (reused) to the current block to obtain a predicted image (Pred_U). The MV (MV_U) is represented by an arrow "MV_U" pointing from the current block to the reference picture. And, by superimposing the predicted image Pred_U on the predicted image (for example, Pred and Pred_L) after the first correction, the second correction of the predicted image is performed. This has the effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is the final predicted image of the current block after blending (smoothing) the boundaries with the adjacent blocks.
[0663] In addition, the above example is a correction method using two paths of blocks adjacent to the left and above, but the correction method may also be a correction method using three paths or more paths of blocks adjacent to the right and / or below.
[0664] Furthermore, the area to be superimposed may not be the pixel area of the entire block, but may be only a partial area near the block boundary.
[0665] In addition, the prediction image correction processing of OBMC for obtaining one prediction image Pred by superimposing the additional prediction images Pred_L and Pred_U from one reference image is described here. However, when the prediction image is corrected based on a plurality of reference images, the same processing may be applied to each of the plurality of reference images. In such a case, after obtaining the corrected prediction image from each reference image by performing image correction of OBMC based on the plurality of reference images, the final prediction image is obtained by further superimposing the obtained plurality of corrected prediction images.
[0666] In addition, in OBMC, the unit of the current block may be a PU unit, or a sub-block unit obtained by further dividing the PU.
[0667] As a method for determining whether OBMC is applied, there is a method using, for example, obmc_flag, a signal indicating whether OBMC is applied. As a specific example, the encoding device 100 can determine whether the current block belongs to a region with complex motion. In the case of a region with complex motion, the encoding device 100 sets the value 1 as obmc_flag and applies OBMC for encoding. In the case of a region with complex motion, the encoding device 100 sets the value 0 as obmc_flag and does not apply OBMC for encoding the block. On the other hand, in the decoding device 200, obmc_flag described in the stream is decoded, and decoding is performed by switching whether OBMC is applied according to the value.
[0668] [Motion Compensation > BIO]
[0669] Next, the method of deriving MV is described. First, a mode of deriving MV based on a model assuming uniform linear motion is described. This mode is sometimes called a BIO (bidirectional optical flow) mode. In addition, the bidirectional optical flow can also be expressed as BDOF instead of BIO.
[0670] Fig.63 This is a diagram used to illustrate a model that assumes uniform linear motion. Fig.63 In , (vx, vy) represents the velocity vector, τ0 and τ1 represent the temporal distance between the current picture (Cur Pic) and the two reference pictures (Ref0 and Ref1), (MVx0, MVy0) represents the MV corresponding to the reference picture Ref0, and (MVx1, MVy1) represents the MV corresponding to the reference picture Ref1.
[0671] At this time, under the assumption of uniform linear motion of the velocity vector (vx, vy), (MVx0, MVy0) and (MVx1, MVy1) are respectively expressed as (vxτ0, vyτ0) and (-vxτ1, -vyτ1), and the following optical flow equation (2) holds.
[0672] [Formula 3]
[0673]
[0674] Here, I(k) represents the brightness value of the reference image k (k=0, 1) after motion compensation. The optical flow equation indicates that the sum of (i) the temporal differential of the brightness value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is equal to zero. Based on the combination of the optical flow equation and Hermite interpolation, the motion vector of the block unit obtained from the candidate MV list, etc. can be corrected in pixel units.
[0675] Alternatively, the decoding device 200 may derive the MV by a method different from the derivation of the motion vector based on the model assuming uniform linear motion. For example, the motion vector may be derived in sub-block units based on the MVs of a plurality of adjacent blocks.
[0676] Fig.64 is a flowchart showing an example of inter-frame prediction in accordance with BIO. Fig.65 2 is a diagram showing an example of the functional structure of the inter-frame prediction unit 126 that performs the inter-frame prediction in accordance with the BIO.
[0677] like Fig.65As shown, the inter-frame prediction unit 126 includes, for example, a memory 126a, an interpolation image derivation unit 126b, a gradient image derivation unit 126c, an optical flow derivation unit 126d, a correction value derivation unit 126e, and a predicted image correction unit 126f.
[0678] The inter prediction unit 126 derives two motion vectors (M0, M1) using two reference pictures (Ref0, Ref1) different from the picture (Cur Pic) containing the current block. Furthermore, the inter prediction unit 126 derives a predicted image of the current block using these two motion vectors (M0, M1) (step Sy_1). In addition, the motion vector M0 is a motion vector (MVx0, MVy0) corresponding to the reference picture Ref0, and the motion vector M1 is a motion vector (MVx1, MVy1) corresponding to the reference picture Ref1.
[0679] Next, the interpolation image deriving unit 126b refers to the memory 126a and uses the motion vector M0 and the reference picture L0 to derive the interpolation image I of the current block. 0 In addition, the interpolation image deriving unit 126b refers to the memory 126a and uses the motion vector M1 and the reference picture L1 to derive the interpolation image I of the current block. 1 (Step Sy_2). Here, the interpolation image I 0 is the image contained in the reference picture Ref0 derived for the current block, and the interpolated image I 1 The interpolated image I is the image included in the reference picture Ref1 derived for the current block. 0 And the interpolated image I 1 Alternatively, in order to properly derive the gradient image described later, the interpolated image I 0 And the interpolated image I 1 They can also be images larger than the current block. 0 and I 1 It may also include a predicted image derived by applying the motion vector (M0, M1) and the reference picture (L0, L1) and the motion compensation filter.
[0680] In addition, the gradient image deriving unit 126c obtains the interpolation image I 0 And the interpolated image I 1 Export the gradient image of the current block (Ix 0 , Ix 1 , Iy 0 , Iy 1 )(Step Sy_3). In addition, the gradient image in the horizontal direction is (Ix 0 , Ix 1 ), the vertical gradient image is (Iy 0, Iy 1 ). The gradient image deriving unit 126c may derive the gradient image by, for example, applying a gradient filter to the interpolation image. The gradient image only needs to represent the spatial variation of the pixel value along the horizontal direction or the vertical direction.
[0681] Next, the optical flow derivation unit 126d uses the interpolated image (I 0 , I 1 ) and gradient image (Ix 0 , Ix 1 , Iy 0 , Iy 1 ), and derive the optical flow (vx, vy) as the above-mentioned velocity vector (step Sy_4). The optical flow is a coefficient that corrects the spatial movement of the pixel, and can also be called a local motion estimation value, a corrected motion vector, or a corrected weight vector. As an example, the sub-block can be a sub-CU of 4×4 pixels. In addition, the optical flow can also be derived not in sub-block units, but in other units such as pixel units.
[0682] Next, the inter-frame prediction unit 126 corrects the predicted image of the current block using the optical flow (vx, vy). For example, the correction value deriving unit 126e derives the correction value of the value of the pixel included in the current block using the optical flow (vx, vy) (step Sy_5). Furthermore, the predicted image correction unit 126f can correct the predicted image of the current block using the correction value (step Sy_6). In addition, the correction value can be derived in units of each pixel, or in units of multiple pixels or sub-blocks.
[0683] In addition, the BIO processing flow is not limited to Fig.64 The processing disclosed in . It is also possible to implement only Fig.64 Part of the processing disclosed in the process may be processed, different processing may be added or replaced, or the processing order may be different.
[0684] [Motion Compensation > LIC]
[0685] Next, an example of a mode for generating a predicted image (prediction) using LIC (local illumination compensation) will be described.
[0686] Fig.66A : is a diagram for explaining an example of a method for generating a predicted image using brightness correction processing based on LIC. Fig.66B This is a flowchart showing an example of a method for generating a predicted image using the LIC.
[0687] First, the inter prediction unit 126 derives an MV from an already encoded reference picture, and obtains a reference picture corresponding to the current block (step Sz_1 ).
[0688] Next, the inter-frame prediction unit 126 extracts information indicating how the brightness value has changed in the reference picture and the current picture for the current block (step Sz_2). This extraction is performed based on the brightness pixel values of the encoded left adjacent reference area (peripheral reference area) and the encoded upper adjacent reference area (peripheral reference area) in the current picture, and the brightness pixel values at the same position in the reference picture specified by the derived MV. Next, the inter-frame prediction unit 126 calculates the brightness correction parameter using the information indicating how the brightness value has changed (step Sz_3).
[0689] The inter-frame prediction unit 126 generates a predicted image for the current block by applying the brightness correction parameter to the reference image in the reference picture specified by the MV (step Sz_4). That is, the reference image in the reference picture specified by the MV, that is, the predicted image, is corrected based on the brightness correction parameter. In this correction, the brightness correction may be performed or the color difference correction may be performed. That is, the correction parameter of the color difference may be calculated using information indicating how the color difference has changed, and the color difference correction process may be performed.
[0690] in addition, Fig.66A The shape of the peripheral reference area in is just an example, and shapes other than this may also be used.
[0691] In addition, the process of generating a predicted image from one reference picture is described here, but the case of generating a predicted image from multiple reference pictures is also the same. The predicted image can be generated by performing brightness correction processing on the reference images obtained from each reference picture in the same way as described above.
[0692] As a method for determining whether to apply LIC, for example, there is a method using a lic_flag, which is a signal indicating whether to apply LIC. As a specific example, in the encoding device 100, it is determined whether the current block belongs to an area where brightness changes, and if it belongs to an area where brightness changes, a value of 1 is set as lic_flag, and LIC is applied to perform encoding, and if it does not belong to an area where brightness changes, a value of 0 is set as lic_flag, and LIC is not applied to perform encoding. On the other hand, in the decoding device 200, lic_flag described in the stream is decoded, and whether to apply LIC is switched according to the value, and decoding is performed.
[0693] As another method for determining whether to apply LIC, for example, there is a method of determining based on whether LIC is applied to the surrounding blocks. As a specific example, when the current block is processed in the merge mode, the inter-frame prediction unit 126 determines whether the surrounding coded blocks selected when deriving the MV in the merge mode are coded using LIC. The inter-frame prediction unit 126 switches whether to apply LIC based on the result and performs coding. In addition, in the case of this example, the same process is also applied to the process on the decoding device 200 side.
[0694] use Fig.66A and Fig.66B LIC (luminance correction processing) has been described above, and its details are described below.
[0695] First, the inter prediction unit 126 derives an MV for acquiring a reference image corresponding to the current block from a reference picture that is an encoded picture.
[0696] Next, the inter-frame prediction unit 126 uses the brightness pixel values of the left adjacent and upper adjacent encoded surrounding reference areas and the brightness pixel values at the same position in the reference picture specified by MV to extract information indicating how the brightness values in the reference picture and the current picture have changed, and calculates the brightness correction parameter. For example, let the brightness pixel value of a certain pixel in the surrounding reference area in the current picture be p0, and let the brightness pixel value of the pixel in the surrounding reference area in the reference picture at the same position as the pixel be p1. The inter-frame prediction unit 126 calculates the coefficients A and B that optimize A×p1+B=p0 as the brightness correction parameter for a plurality of pixels in the surrounding reference area.
[0697] Next, the inter-frame prediction unit 126 generates a predicted image for the current block by performing brightness correction processing on the reference image in the reference picture specified by the MV using the brightness correction parameter. For example, let the brightness pixel value in the reference image be p2, and let the brightness pixel value of the predicted image after the brightness correction processing be p3. The inter-frame prediction unit 126 calculates A×p2+B=p3 for each pixel in the reference image to generate a predicted image after the brightness correction processing.
[0698] Alternatively, you can use Fig.66A For example, a region including a predetermined number of pixels after thinning out from the upper adjacent pixels and the left adjacent pixels may be used as the surrounding reference region. In addition, the surrounding reference region is not limited to the region adjacent to the current block, and may also be a region not adjacent to the current block. Fig.66AIn the example shown, the peripheral reference area in the reference picture is an area specified by the MV of the current picture from the peripheral reference area in the current picture, but it may also be an area specified by another MV. For example, the other MV may also be the MV of the peripheral reference area in the current picture.
[0699] In addition, although the operation in the encoding device 100 is described here, the operation in the decoding device 200 is also the same.
[0700] In addition, LIC can be applied not only to brightness but also to color difference. In this case, correction parameters can be derived separately for Y, Cb, and Cr, or common correction parameters can be used for some of them.
[0701] Furthermore, the LIC process may be applied in sub-block units. For example, the modification parameters may be derived using the surrounding reference area of the current sub-block and the surrounding reference area of the reference sub-block in the reference picture specified by the MV of the current sub-block.
[0702] [Prediction Control Department]
[0703] The prediction control unit 128 selects either an intra-frame prediction image (an image or signal output from the intra-frame prediction unit 124 ) or an inter-frame prediction image (an image or signal output from the inter-frame prediction unit 126 ), and outputs the selected prediction image to the subtraction unit 104 and the addition unit 116 .
[0704] [Prediction parameter generation unit]
[0705] The prediction parameter generation unit 130 may output information related to intra prediction, inter prediction, and selection of a prediction image in the prediction control unit 128 to the entropy coding unit 110 as a prediction parameter. The entropy coding unit 110 may generate a stream based on the prediction parameter input from the prediction parameter generation unit 130 and the quantization coefficient input from the quantization unit 108. The prediction parameter may also be used in the decoding device 200. The decoding device 200 may receive the stream and decode it, and perform the same process as the prediction process performed in the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128. The prediction parameter may include a selection prediction signal (for example, MV, prediction type, or prediction mode used by the intra prediction unit 124 or the inter prediction unit 126), or an arbitrary index, flag, or value based on or indicating the prediction process performed in the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128.
[0706] [Decoding device]
[0707] Next, a decoding device 200 that can decode the stream output from the above-described encoding device 100 will be described. Fig.672 is a block diagram showing an example of a functional configuration of a decoding device 200 according to an embodiment. The decoding device 200 is a device that decodes a stream that is an encoded image in units of blocks.
[0708] like Fig.67 As shown, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transformation unit 206, an addition unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a partition determination unit 224. In addition, the intra prediction unit 216 and the inter prediction unit 218 are each configured as a part of a prediction processing unit.
[0709] [Implementation Example of Decoding Device]
[0710] Fig.68 2 is a block diagram showing an implementation example of the decoding device 200. The decoding device 200 includes a processor b1 and a memory b2. Fig.67 The multiple components of the decoding device 200 shown are Fig.68 It is implemented by the processor b1 and the memory b2 shown.
[0711] Processor b1 is a circuit that performs information processing and is able to access memory b2. For example, processor b1 is a dedicated or general-purpose electronic circuit that decodes a stream. Processor b1 may be a processor such as a CPU. In addition, processor b1 may be a collection of multiple electronic circuits. In addition, for example, processor b1 may function as Fig.67 The functions of the multiple components of the decoding device 200 other than the components for storing information are described above.
[0712] The memory b2 is a dedicated or general memory for storing information used by the processor b1 to decode the stream. The memory b2 may be an electronic circuit and may be connected to the processor b1. In addition, the memory b2 may also be included in the processor b1. In addition, the memory b2 may be a collection of multiple electronic circuits. In addition, the memory b2 may be a magnetic disk or an optical disk, etc., or may be expressed as a storage device or a recording medium, etc. In addition, the memory b2 may be a non-volatile memory or a volatile memory.
[0713] For example, the memory b2 may store images or streams. In addition, the memory b2 may store a program used by the processor b1 to decode the streams.
[0714] In addition, for example, memory b2 can function as Fig.67The memory b2 can be used as a component for storing information among the multiple components of the decoding device 200 shown in FIG. Fig.67 The functions of the block memory 210 and the frame memory 214 are shown. More specifically, the memory b2 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).
[0715] In addition, in the decoding device 200, it is also possible not to implement Fig.67 All of the multiple constituent elements shown above may not perform all of the multiple processes described above. Fig.67 Part of the plurality of components shown in the figure may be included in other devices, and part of the plurality of processes described above may be executed by other devices.
[0716] Hereinafter, after describing the overall processing flow of the decoding device 200, each component included in the decoding device 200 will be described. In addition, among the components included in the decoding device 200, a detailed description of the components that perform the same processing as the components included in the encoding device 100 will be omitted. For example, the inverse quantization unit 204, the inverse transformation unit 206, the addition unit 208, the block memory 210, the frame memory 214, the intra-frame prediction unit 216, the inter-frame prediction unit 218, the prediction control unit 220, and the loop filter unit 212 included in the decoding device 200 perform the same processing as the inverse quantization unit 112, the inverse transformation unit 114, the addition unit 116, the block memory 118, the frame memory 122, the intra-frame prediction unit 124, the inter-frame prediction unit 126, the prediction control unit 128, and the loop filter unit 120 included in the encoding device 100, respectively.
[0717] [Overall flow of decoding processing]
[0718] Fig.69 : is a flowchart showing an example of the overall decoding process of the decoding device 200.
[0719] First, the partition determination unit 224 of the decoding device 200 determines the partition mode of each of the plurality of fixed-size blocks (128×128 pixels) included in the picture based on the parameters input from the entropy decoding unit 202 (step Sp_1). This partition mode is the partition mode selected by the encoding device 100. Next, the decoding device 200 performs the processing of steps Sp_2 to Sp_6 on each of the plurality of blocks constituting the partition mode.
[0720] The entropy decoding unit 202 decodes (specifically, performs entropy decoding) the encoded quantization coefficients and prediction parameters of the current block (step Sp_2).
[0721] Next, the inverse quantization unit 204 and the inverse transformation unit 206 restore the prediction residual of the current block by performing inverse quantization and inverse transformation on the plurality of quantized coefficients (step Sp_3).
[0722] Next, the prediction processing unit composed of the intra-frame prediction unit 216, the inter-frame prediction unit 218 and the prediction control unit 220 generates a predicted image of the current block (step Sp_4).
[0723] Next, the adding unit 208 reconstructs the current block into a reconstructed image (also referred to as a decoded image block) by adding the predicted image to the prediction residual (step Sp_5).
[0724] Next, when the reconstructed image is generated, the loop filter unit 212 filters the reconstructed image (step Sp_6).
[0725] Next, the decoding device 200 determines whether the decoding of the entire picture is completed (step Sp_7 ). If it is determined that the decoding is not completed (“No” in step Sp_7 ), the processing from step Sp_1 is repeatedly executed.
[0726] In addition, the processing of these steps Sp_1 to Sp_7 may be performed sequentially by the decoding device 200, or a plurality of processing of some of these processes may be performed in parallel, or the order may be reversed.
[0727] [Division Decision Section]
[0728] Fig.70 2 is a diagram showing the relationship between the division determination unit 224 and other components. As an example, the division determination unit 224 can perform the following processing.
[0729] The partition decision unit 224 collects block information from the block memory 210 or the frame memory 214, and further obtains parameters from the entropy decoding unit 202. The partition decision unit 224 may determine the partition mode of the fixed-size block based on the block information and the parameters. The partition decision unit 224 may output information indicating the partition mode determined to the inverse transform unit 206, the intra-frame prediction unit 216, and the inter-frame prediction unit 218. The inverse transform unit 206 may perform inverse transformation on the transform coefficients based on the partition mode indicated by the information from the partition decision unit 224. The intra-frame prediction unit 216 and the inter-frame prediction unit 218 may generate a predicted image based on the partition mode indicated by the information from the partition decision unit 224.
[0730] [Entropy decoding unit]
[0731] Fig.71 It is a block diagram showing an example of the functional structure of the entropy decoding unit 202.
[0732] The entropy decoding unit 202 generates parameters related to quantization coefficients, prediction parameters, and segmentation modes by entropy decoding the stream. In this entropy decoding, CABAC is used, for example. Specifically, the entropy decoding unit 202 includes, for example, a binary arithmetic decoding unit 202a, a context control unit 202b, and a multi-valued unit 202c. The binary arithmetic decoding unit 202a uses the context value derived by the context control unit 202b to arithmetically decode the stream into a binary signal. The context control unit 202b, like the context control unit 110b of the encoding device 100, derives the context value corresponding to the feature of the syntactic element or the surrounding conditions, that is, the probability of occurrence of the binary signal. The multi-valued unit 202c performs a multi-valued conversion (debinarization) of the binary signal output from the binary arithmetic decoding unit 202a into a multi-valued signal representing the above-mentioned quantization coefficients, etc. The multi-valued conversion is performed in the manner of the above-mentioned binarization.
[0733] The entropy decoding unit 202 outputs the quantized coefficients in units of blocks to the inverse quantization unit 204. The entropy decoding unit 202 may output the quantized coefficients contained in the stream (see Figure 1 The intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220 can perform the same prediction processing as that performed by the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 on the encoding device 100 side.
[0734] [Entropy decoding unit]
[0735] Fig.72 3 is a diagram showing the flow of CABAC in the entropy decoding unit 202.
[0736] First, in the CABAC of the entropy decoding unit 202, initialization is performed. In this initialization, the binary arithmetic decoding unit 202a is initialized and the initial context value is set. In addition, the binary arithmetic decoding unit 202a and the multi-valued unit 202c perform arithmetic decoding and multi-valued conversion on the coded data of, for example, CTU. At this time, the context control unit 202b updates the context value every time arithmetic decoding is performed. In addition, as a post-processing, the context control unit 202b saves the context value. The saved context value is used, for example, as the initial value of the context value for the next CTU.
[0737] [Inverse Quantization Unit]
[0738] The inverse quantization unit 204 inversely quantizes the quantized coefficients of the current block as input from the entropy decoding unit 202. Specifically, the inverse quantization unit 204 inversely quantizes each quantized coefficient of the current block based on a quantization parameter corresponding to the quantized coefficient. Furthermore, the inverse quantization unit 204 outputs the inversely quantized quantized coefficients (i.e., transform coefficients) of the current block to the inverse transform unit 206.
[0739] Fig.73 This is a block diagram showing an example of the functional configuration of the inverse quantization unit 204 .
[0740] The inverse quantization unit 204 includes, for example, a quantization parameter generation unit 204 a , a predicted quantization parameter generation unit 204 b , a quantization parameter storage unit 204 d , and an inverse quantization processing unit 204 e .
[0741] Fig.74 This is a flowchart showing an example of inverse quantization performed by the inverse quantization unit 204 .
[0742] As an example, the inverse quantization unit 204 may be based on Fig.74 In the process shown in FIG. 1 , inverse quantization is performed for each CU. Specifically, the quantization parameter generation unit 204a determines whether to perform inverse quantization (step Sv_11). Here, if it is determined to perform inverse quantization (“Yes” in step Sv_11), the quantization parameter generation unit 204a obtains the differential quantization parameter of the current block from the entropy decoding unit 202 (step Sv_12).
[0743] Next, the predicted quantization parameter generator 204b obtains a quantization parameter of a processing unit different from the current block from the quantization parameter storage unit 204d (step Sv_13). The predicted quantization parameter generator 204b generates a predicted quantization parameter of the current block based on the obtained quantization parameter (step Sv_14).
[0744] Next, the quantization parameter generator 204a adds the differential quantization parameter of the current block obtained from the entropy decoding unit 202 to the predicted quantization parameter of the current block generated by the predicted quantization parameter generator 204b (step Sv_15). By this addition, the quantization parameter of the current block is generated. In addition, the quantization parameter generator 204a stores the quantization parameter of the current block in the quantization parameter storage unit 204d (step Sv_16).
[0745] Next, the inverse quantization processing unit 204e inversely quantizes the quantized coefficients of the current block into transform coefficients using the quantization parameters generated in step Sv_15 (step Sv_17).
[0746] In addition, the differential quantization parameter may be decoded at a bit sequence level, a picture level, a slice level, a brick level, or a CTU level. In addition, the initial value of the quantization parameter may be decoded at a sequence level, a picture level, a slice level, a brick level, or a CTU level. At this time, the quantization parameter may also be generated using the initial value of the quantization parameter and the differential quantization parameter.
[0747] In addition, the inverse quantization unit 204 may include a plurality of inverse quantizers, and may inversely quantize the quantized coefficients using an inverse quantization method selected from a plurality of inverse quantization methods.
[0748] [Inverse transformation unit]
[0749] The inverse transform unit 206 restores the prediction residual by inversely transforming the transform coefficients that are input from the inverse quantization unit 204 .
[0750] For example, when the information decoded from the stream indicates that EMT or AMT is applied (for example, the AMT flag is true), the inverse transform unit 206 inversely transforms the transform coefficients of the current block based on the information indicating the decoded transform type.
[0751] Furthermore, for example, when the information decoded from the stream interpreter indicates that NSST is to be applied, the inverse transform unit 206 applies an inverse re-transform to the transform coefficients.
[0752] Fig.75 This is a flowchart showing an example of processing of the inverse transformation unit 206.
[0753] For example, the inverse transform unit 206 determines whether there is information indicating that orthogonal transform is not performed in the stream (step St_11). Here, if it is determined that the information does not exist ("No" in step St_11), the inverse transform unit 206 obtains the information indicating the transform type decoded by the entropy decoding unit 202 (step St_12). Next, the inverse transform unit 206 determines the transform type used in the orthogonal transform of the encoding device 100 based on the information (step St_13). Next, the inverse transform unit 206 performs inverse orthogonal transform using the determined transform type (step St_14).
[0754] Fig.76 This is a flowchart showing another example of the processing of the inverse transformation unit 206.
[0755] For example, the inverse transform unit 206 determines whether the transform size is less than or equal to a predetermined value (step Su_11). Here, if it is determined to be less than or equal to the predetermined value ("Yes" in step Su_11), the inverse transform unit 206 obtains information indicating which transform type among the one or more transform types included in the first transform type group is used by the encoding device 100 from the entropy decoding unit 202 (step Su_12). In addition, such information is decoded by the entropy decoding unit 202 and output to the inverse transform unit 206.
[0756] Based on this information, the inverse transform unit 206 determines the transform type used in the orthogonal transform in the encoding device 100 (step Su_13). Next, the inverse transform unit 206 performs an inverse orthogonal transform on the transform coefficients of the current block using the determined transform type (step Su_14). On the other hand, if the inverse transform unit 206 determines in step Su_11 that the transform size is not less than the specified value ("No" in step Su_11), it performs an inverse orthogonal transform on the transform coefficients of the current block using the second transform type group (step Su_15).
[0757] In addition, as an example, the inverse orthogonal transform of the inverse transform unit 206 may be performed according to each TU. Fig.75 or Fig.76 The process shown is implemented. In addition, it is also possible to perform inverse orthogonal transform using a predetermined transform type without decoding the information indicating the transform type used in the orthogonal transform. In addition, the transform type is specifically DST7 or DCT8, etc., and in the inverse orthogonal transform, an inverse transform basis function corresponding to the transform type is used.
[0758] [Addition Department]
[0759] The adding unit 208 reconstructs the current block by adding the prediction residual as input from the inverse transform unit 206 and the prediction image as input from the prediction control unit 220. That is, the reconstructed image of the current block is generated. Then, the adding unit 208 outputs the reconstructed image of the current block to the block memory 210 and the loop filter unit 212.
[0760] [Block Memory]
[0761] The block memory 210 is a storage unit for storing blocks in the current picture that are referenced in intra prediction. Specifically, the block memory 210 stores the reconstructed image output from the adder 208 .
[0762] [Loop filter section]
[0763] The loop filter unit 212 performs loop filtering on the reconstructed image generated by the adder 208 , and outputs the filtered reconstructed image to the frame memory 214 , a display device, and the like.
[0764] When the information indicating the on / off of the ALF decoded from the stream indicates that the ALF is on, one filter is selected from a plurality of filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed image.
[0765] Fig.77 2 is a block diagram showing an example of a functional configuration of the loop filter unit 212. The loop filter unit 212 has the same configuration as the loop filter unit 120 of the encoding device 100.
[0766] The loop filter unit 212 is, for example, Fig.77 As shown in FIG. 1 , the loop filter unit 212 includes a deblocking filter processing unit 212a, a SAO processing unit 212b, and an ALF processing unit 212c. The deblocking filter processing unit 212a performs the above-mentioned deblocking filter processing on the reconstructed image. The SAO processing unit 212b performs the above-mentioned SAO processing on the reconstructed image after the deblocking filter processing. In addition, the ALF processing unit 212c applies the above-mentioned ALF processing to the reconstructed image after the SAO processing. In addition, the loop filter unit 212 may not include Fig.77 Alternatively, the loop filter unit 212 may be configured in the same manner as in Fig.77 A structure in which the above-mentioned processes are performed in an order different from the process order disclosed in.
[0767] [Frame Memory]
[0768] The frame memory 214 is a storage unit for storing reference pictures used in inter-frame prediction, and is sometimes referred to as a frame buffer. Specifically, the frame memory 214 stores the reconstructed image filtered by the loop filter unit 212 .
[0769] [Prediction unit (intra-frame prediction unit / inter-frame prediction unit / prediction control unit)]
[0770] Fig.78 2 is a flowchart showing an example of processing performed by the prediction unit of the decoding device 200. In addition, as an example, the prediction unit is composed of all or part of the components of the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220. The prediction processing unit includes the intra prediction unit 216 and the inter prediction unit 218, for example.
[0771] The prediction unit generates a prediction image of the current block (step Sq_1). The prediction image is also called a prediction signal or a prediction block. In addition, the prediction signal includes, for example, an intra-frame prediction signal or an inter-frame prediction signal. Specifically, the prediction unit generates a prediction image of the current block using a reconstructed image that has been obtained by generating prediction images for other blocks, restoring prediction residuals, and adding prediction images. The prediction unit of the decoding device 200 generates the same prediction image as the prediction image generated by the prediction unit of the encoding device 100. That is, the prediction image generation methods used in these prediction units are common or corresponding to each other.
[0772] The reconstructed image may be, for example, an image of a reference picture or an image including the current block, that is, an image of a decoded block in the current picture (that is, the other blocks mentioned above). The decoded block in the current picture may be, for example, an adjacent block of the current block.
[0773] Fig.79 This is a flowchart showing another example of the processing performed by the prediction unit of the decoding device 200.
[0774] The prediction unit determines a method or mode for generating a prediction image (step Sr_1). For example, the method or mode can be determined based on prediction parameters or the like.
[0775] When the prediction unit determines the first mode as the mode for generating the prediction image, the prediction unit generates the prediction image according to the first mode (step Sr_2a). In addition, when the prediction unit determines the second mode as the mode for generating the prediction image, the prediction unit generates the prediction image according to the second mode (step Sr_2b). In addition, when the prediction unit determines the third mode as the mode for generating the prediction image, the prediction unit generates the prediction image according to the third mode (step Sr_2c).
[0776] The first method, the second method, and the third method are different methods for generating a predicted image, and may be, for example, an inter-frame prediction method, an intra-frame prediction method, or other prediction methods. In these prediction methods, the above-mentioned reconstructed image can be used.
[0777] Fig.80A and Fig.80B This is a flowchart showing another example of the processing performed by the prediction unit of the decoding device 200.
[0778] As an example, the forecasting unit can Fig.80A and Fig.80B The process shown in the figure is used for prediction processing. Fig.80A and Fig.80BThe intra block copy shown is a mode belonging to inter-frame prediction, and is a mode in which a block included in the current picture is used as a reference image or reference block. That is, in the intra block copy, a picture different from the current picture is not referenced. In addition, Fig.80A The PCM mode shown is a mode belonging to intra-frame prediction, and is a mode in which neither transformation nor quantization is performed.
[0779] [Intra-frame prediction unit]
[0780] The intra prediction unit 216 performs intra prediction based on the intra prediction mode decoded from the stream, with reference to the block in the current picture stored in the block memory 210, thereby generating a predicted image (i.e., intra prediction image) of the current block. Specifically, the intra prediction unit 216 performs intra prediction with reference to the pixel values (e.g., brightness values, color difference values) of the blocks adjacent to the current block to generate the intra prediction image, and outputs the intra prediction image to the prediction control unit 220.
[0781] Furthermore, when an intra prediction mode that refers to a luminance block is selected in intra prediction of a chrominance block, the intra prediction unit 216 may predict the chrominance component of the current block based on the luminance component of the current block.
[0782] Furthermore, when the information decoded from the stream indicates application of PDPC, the intra prediction unit 216 corrects the pixel value after intra prediction based on the gradient of the reference pixel in the horizontal and vertical directions.
[0783] Fig.81 This is a diagram showing an example of processing by the intra prediction unit 216 of the decoding device 200 .
[0784] The intra prediction unit 216 first determines whether there is an MPM flag indicating 1 in the stream (step Sw_11). Here, if it is determined that there is an MPM flag indicating 1 ("Yes" in step Sw_11), the intra prediction unit 216 obtains information indicating the intra prediction mode selected in the encoding device 100 in the MPM from the entropy decoding unit 202 (step Sw_12). In addition, the information is decoded by the entropy decoding unit 202 and output to the intra prediction unit 216. Next, the intra prediction unit 216 determines the MPM (step Sw_13). The MPM is composed of, for example, 6 intra prediction modes. Next, the intra prediction unit 216 determines the intra prediction mode represented by the information obtained in step Sw_12 from among the multiple intra prediction modes included in the MPM (step Sw_14).
[0785] On the other hand, if the intra prediction unit 216 determines in step Sw_11 that there is no MPM flag indicating 1 in the stream ("No" in step Sw_11), it obtains information indicating the intra prediction mode selected in the encoding device 100 (step Sw_15). That is, the intra prediction unit 216 obtains from the entropy decoding unit 202 information indicating the intra prediction mode selected in the encoding device 100 among one or more intra prediction modes not included in the MPM. In addition, the information is decoded by the entropy decoding unit 202 and output to the intra prediction unit 216. Next, the intra prediction unit 216 determines the intra prediction mode indicated by the information obtained in step Sw_15 from among the one or more intra prediction modes not included in the MPM (step Sw_17).
[0786] The intra prediction unit 216 generates a predicted image according to the intra prediction mode determined in step Sw_14 or step Sw_17 (step Sw_18 ).
[0787] [Inter-frame prediction unit]
[0788] The inter-frame prediction unit 218 predicts the current block with reference to the reference picture stored in the frame memory 214. The prediction is performed in units of the current block or sub-blocks in the current block. In addition, the sub-block is included in the block and is a unit smaller than the block. The size of the sub-block can be 4×4 pixels, 8×8 pixels, or other sizes. The size of the sub-block can also be switched in units such as slices, bricks, or pictures.
[0789] For example, the inter-frame prediction unit 218 uses motion information (e.g., MV) interpreted from the stream (e.g., prediction parameters output from the entropy decoding unit 202) to perform motion compensation, thereby generating an inter-frame prediction image of the current block or sub-block, and outputs the inter-frame prediction image to the prediction control unit 220.
[0790] When the information decoded from the stream decoder indicates that the OBMC mode is applied, the inter prediction unit 218 generates an inter prediction image using not only the motion information of the current block obtained by the motion search but also the motion information of the neighboring blocks.
[0791] Furthermore, when the information obtained from the stream decoding indicates that the FRUC mode is applied, the inter-frame prediction unit 218 performs motion search according to the pattern matching method (bilateral matching or template matching) obtained from the stream decoding to derive motion information. Next, the inter-frame prediction unit 218 performs motion compensation (prediction) using the derived motion information.
[0792] When the BIO mode is applied, the inter prediction unit 218 derives the MV based on a model assuming uniform linear motion. When the information decoded from the stream decoder indicates that the affine mode is applied, the inter prediction unit 218 derives the MV in sub-block units based on the MVs of a plurality of adjacent blocks.
[0793] [MV export process]
[0794] Fig.82 This is a flowchart showing an example of MV derivation in the decoding device 200.
[0795] The inter-frame prediction unit 218 determines, for example, whether to decode motion information (e.g., MV). For example, the inter-frame prediction unit 218 can make a determination based on a prediction mode included in the stream, or can make a determination based on other information included in the stream. Here, if the inter-frame prediction unit 218 determines to decode the motion information, the MV of the current block is derived in a mode in which the motion information is decoded. On the other hand, if the inter-frame prediction unit 218 determines not to decode the motion information, the MV is derived in a mode in which the motion information is not decoded.
[0796] Here, the modes for deriving MV include the normal inter mode, normal merge mode, FRUC mode, and affine mode described later. Among these modes, the modes for decoding motion information include the normal inter mode, normal merge mode, and affine mode (specifically, affine inter mode and affine merge mode). In addition, the motion information includes not only MV but also predicted MV selection information described later. In addition, the modes for not decoding motion information include the FRUC mode. The inter prediction unit 218 selects a mode for deriving the MV of the current block from these multiple modes, and derives the MV of the current block using the selected mode.
[0797] Fig.83 This is a flowchart showing another example of MV derivation in the decoding device 200.
[0798] The inter-frame prediction unit 218 determines whether to decode the differential MV, for example, and the inter-frame prediction unit 218 can determine based on the prediction mode included in the stream, or can determine based on other information included in the stream. Here, if the inter-frame prediction unit 218 determines to decode the differential MV, it can derive the MV of the current block in a mode of decoding the differential MV. In this case, for example, the differential MV included in the stream is decoded as a prediction parameter.
[0799] On the other hand, if the inter prediction unit 218 determines not to decode the difference MV, it derives the MV in a mode in which the difference MV is not decoded. In this case, the encoded difference MV is not included in the stream.
[0800] Here, as described above, the modes for deriving the MV include the normal inter mode, normal merge mode, FRUC mode, and affine mode described later. Among these modes, the modes for encoding the differential MV include the normal inter mode and affine mode (specifically, the affine inter mode). In addition, the modes for not encoding the differential MV include the FRUC mode, normal merge mode, and affine mode (specifically, the affine merge mode). The inter prediction unit 218 selects a mode for deriving the MV of the current block from these multiple modes, and derives the MV of the current block using the selected mode.
[0801] [MV Export > Normal Interframe Mode]
[0802] For example, when the information extracted from the stream decoder indicates that the normal inter mode is applied, the inter prediction unit 218 derives an MV in the normal merge mode based on the information extracted from the stream decoder, and performs motion compensation (prediction) using the MV.
[0803] Fig.84 This is a flowchart showing an example of inter prediction based on the normal inter mode in the decoding apparatus 200 .
[0804] The inter-frame prediction unit 218 of the decoding device 200 performs motion compensation on each block. At this time, the inter-frame prediction unit 218 first obtains multiple candidate MVs for the current block based on information such as MVs of multiple decoded blocks that are temporally or spatially around the current block (step Sg_11). That is, the inter-frame prediction unit 218 creates a candidate MV list.
[0805] Next, the inter-frame prediction unit 218 extracts N (N is an integer greater than or equal to 2) candidate MVs from the plurality of candidate MVs obtained in step Sg_11 according to a predetermined priority order as prediction motion vector candidates (also referred to as prediction MV candidates) (step Sg_12). In addition, the priority order is pre-set for each of the N prediction MV candidates.
[0806] Next, the inter-frame prediction unit 218 decodes the prediction MV selection information from the input stream, and uses the decoded prediction MV selection information to select one prediction MV candidate from the N prediction MV candidates as the prediction MV of the current block (step Sg_13).
[0807] Next, the inter prediction unit 218 decodes the difference MV from the input stream, adds the difference value of the decoded difference MV to the selected prediction MV, and thereby derives the MV of the current block (step Sg_14).
[0808] Finally, the inter prediction unit 218 performs motion compensation on the current block using the derived MV and the decoded reference picture, and generates a predicted image of the current block (step Sg_15). The processing of steps Sg_11 to Sg_15 is performed on each block. For example, if the processing of steps Sg_11 to Sg_15 is performed on all the blocks included in the slice, the inter prediction using the normal inter mode for the slice is completed. In addition, if the processing of steps Sg_11 to Sg_15 is performed on all the blocks included in the picture, the inter prediction using the normal inter mode for the picture is completed. In addition, the processing of steps Sg_11 to Sg_15 may not be performed on all the blocks included in the slice, but if it is performed on a part of the blocks, the inter prediction using the normal inter mode for the slice is completed. Similarly, if the processing of steps Sg_11 to Sg_15 is performed on a part of the blocks included in the picture, the inter prediction using the normal inter mode for the picture is completed.
[0809] [MV Export > Normal Merge Mode]
[0810] For example, when the information decoded from the stream indicates application of the normal merge mode, the inter prediction unit 218 derives an MV in the normal merge mode, and performs motion compensation (prediction) using the MV.
[0811] Fig.85 3 is a flowchart showing an example of inter-frame prediction based on the normal merge mode in the decoding device 200.
[0812] The inter-frame prediction unit 218 first obtains a plurality of candidate MVs for the current block based on information such as MVs of a plurality of decoded blocks temporally or spatially surrounding the current block (step Sh_11). That is, the inter-frame prediction unit 218 creates a candidate MV list.
[0813] Next, the inter-frame prediction unit 218 selects one candidate MV from the multiple candidate MVs obtained in step Sh_11 to derive the MV of the current block (step Sh_12). Specifically, the inter-frame prediction unit 218 obtains MV selection information included as a prediction parameter in the stream, for example, and selects the candidate MV identified by the MV selection information as the MV of the current block.
[0814] Finally, the inter prediction unit 218 performs motion compensation on the current block using the derived MV and the decoded reference picture, and generates a predicted image of the current block (step Sh_13). The processing of steps Sh_11 to Sh_13 is performed on each block, for example. For example, if the processing of steps Sh_11 to Sh_13 is performed on all the blocks included in the slice, the inter prediction using the normal merge mode for the slice is completed. In addition, if the processing of steps Sh_11 to Sh_13 is performed on all the blocks included in the picture, the inter prediction using the normal merge mode for the picture is completed. In addition, the processing of steps Sh_11 to Sh_13 may not be performed on all the blocks included in the slice, but if it is performed on a part of the blocks, the inter prediction using the normal merge mode for the slice is completed. Similarly, if the processing of steps Sh_11 to Sh_13 is performed on a part of the blocks included in the picture, the inter prediction using the normal merge mode for the picture is completed.
[0815] [MV Export > FRUC Mode]
[0816] For example, when the information decoded from the stream decoder indicates the application of the FRUC mode, the inter-frame prediction unit 218 derives the MV in the FRUC mode and uses the MV to perform motion compensation (prediction). In this case, the motion information is not signaled from the encoding device 100 side, but is derived from the decoding device 200 side. For example, the decoding device 200 can derive the motion information by performing a motion search. In this case, the decoding device 200 performs a motion search without using the pixel value of the current block.
[0817] Fig.86 This is a flowchart showing an example of inter-frame prediction based on the FRUC mode in the decoding device 200 .
[0818] First, the inter-frame prediction unit 218 refers to the MVs of each decoded block that is spatially or temporally adjacent to the current block, and generates a list representing these MVs as candidate MVs (that is, a candidate MV list, which can be common to the candidate MV list of the normal merge mode) (step Si_11). Next, the inter-frame prediction unit 218 selects the optimal candidate MV from the multiple candidate MVs registered in the candidate MV list (step Si_12). For example, the inter-frame prediction unit 218 calculates the evaluation value of each candidate MV included in the candidate MV list, and selects one candidate MV as the optimal candidate MV based on the evaluation value. Next, the inter-frame prediction unit 218 derives the MV for the current block based on the selected optimal candidate MV (step Si_14). Specifically, for example, the selected optimal candidate MV is directly derived as the MV for the current block. In addition, for example, it is also possible to derive the MV for the current block by performing pattern matching in the surrounding area of the position in the reference picture corresponding to the selected optimal candidate MV. That is, it is also possible to perform pattern matching in the reference picture and search using the evaluation value for the area around the optimal candidate MV, and then, if there is an MV with a good evaluation value, update the optimal candidate MV to that MV and use it as the final MV of the current block. It is also possible not to perform the update to the MV with a better evaluation value.
[0819] Finally, the inter-frame prediction unit 218 performs motion compensation on the current block by using the derived MV and the decoded reference picture, and generates a predicted image of the current block (step Si_15). The processing of steps Si_11 to Si_15 is performed on each block, for example. For example, if the processing of steps Si_11 to Si_15 is performed on all the blocks included in the slice, the inter-frame prediction using the FRUC mode for the slice is completed. In addition, if the processing of steps Si_11 to Si_15 is performed on all the blocks included in the picture, the inter-frame prediction using the FRUC mode for the picture is completed. The sub-block unit can also be processed in the same way as the above-mentioned block unit.
[0820] [MV Export > Affine Merge Mode]
[0821] For example, when the information decoded from the stream indicates application of the affine merge mode, the inter prediction unit 218 derives an MV in the affine merge mode, and performs motion compensation (prediction) using the MV.
[0822] Fig.87 This is a flowchart showing an example of inter-frame prediction based on the affine merge mode in the decoding device 200.
[0823] In the affine merge mode, first, the inter-frame prediction unit 218 derives the MV of each control point of the current block (step Sk_11). Fig.46AAs shown, these are the top left and top right corners of the current block, or as Fig.46B As shown are the points of the upper left corner, upper right corner and lower left corner of the current block.
[0824] For example, when using Figures 47A to 47C In the case of the MV derivation method shown in FIG. Fig.47A As shown, the decoded blocks A (left), B (top), C (top right), D (bottom left) and E (top left) are checked in order to determine the first valid block after being decoded in affine mode.
[0825] The inter-frame prediction unit 218 uses the first valid block that is determined and decoded in the affine mode to derive the MV of the control point. For example, when block A is determined and block A has two control points, Fig.47B As shown, the inter-frame prediction unit 218 calculates the motion vectors v at the upper left corner and the upper right corner of the decoded block including block A. 3 and v 4 Project it into the current block and calculate the motion vector v of the upper left corner control point of the current block 0 and the motion vector v of the upper right control point 1 . Thus, the MV of each control point is derived.
[0826] In addition, if Fig.49A As shown, when block A is determined and block A has 2 control points, the MV of 3 control points can also be calculated, such as Fig.49B As shown, when block A is determined and block A has 3 control points, MVs of 2 control points can also be calculated.
[0827] Furthermore, when the stream includes MV selection information as a prediction parameter, the inter prediction unit 218 can derive the MV of each control point of the current block using the MV selection information.
[0828] Next, the inter-frame prediction unit 218 performs motion compensation on each of the multiple sub-blocks included in the current block. That is, the inter-frame prediction unit 218 uses two motion vectors v for each of the multiple sub-blocks. 0 and v 1 and the above equation (1A), or using three motion vectors v 0 、v 1 and v 2and the above-mentioned formula (1B), the MV of the sub-block is calculated as the affine MV (step Sk_12). Next, the inter-frame prediction unit 218 uses these affine MVs and the decoded reference picture to perform motion compensation on the sub-block (step Sk_13). If the processing of steps Sk_12 and Sk_13 is performed on all sub-blocks included in the current block, the inter-frame prediction using the affine merge mode for the current block is completed. That is, motion compensation is performed on the current block to generate a predicted image of the current block.
[0829] In addition, in step Sk_11, the candidate MV list mentioned above may also be generated. The candidate MV list may be, for example, a list of candidate MVs derived using a plurality of MV derivation methods for each control point. The plurality of MV derivation methods may be Figures 47A to 47C The MV export method shown in Fig.48A and Fig.48B The MV export method shown in Fig.49A and Fig.49B The MV derivation method shown, and any combination of other MV derivation methods.
[0830] Furthermore, the candidate MV list may include candidate MVs of a mode other than the affine mode that performs prediction in sub-block units.
[0831] In addition, as a candidate MV list, for example, a candidate MV list including candidate MVs of an affine merge mode with 2 control points and a candidate MV list including candidate MVs of an affine merge mode with 3 control points may be generated. Alternatively, a candidate MV list including candidate MVs of an affine merge mode with 2 control points and a candidate MV list including candidate MVs of an affine merge mode with 3 control points may be generated separately. Alternatively, a candidate MV list including candidate MVs of one of an affine merge mode with 2 control points and an affine merge mode with 3 control points may be generated.
[0832] [MV Export > Affine Inter-frame Mode]
[0833] For example, when the information decoded from the stream indicates application of the affine inter mode, the inter prediction unit 218 derives an MV in the affine inter mode, and performs motion compensation (prediction) using the MV.
[0834] Fig.88 : is a flowchart showing an example of inter prediction based on the affine inter mode in the decoding device 200 .
[0835] In the affine inter mode, first, the inter prediction unit 218 derives the prediction MV (v 0 , v 1 ) or (v 0 , v1 , v 2 )(Step Sj_11). The control point is, for example, Fig.46A or Fig.46B As shown, it is the point at the upper left corner, upper right corner, or lower left corner of the current block.
[0836] The inter-frame prediction unit 218 obtains the prediction MV selection information included in the stream as a prediction parameter, and uses the MV identified by the prediction MV selection information to derive the prediction MV of each control point of the current block. Fig.48A and Fig.48B In the case of the MV derivation method shown in FIG. Fig.48A or Fig.48B The predicted MV (v 0 , v 1 ) or (v 0 , v 1 , v 2 ).
[0837] Next, the inter-frame prediction unit 218 obtains each differential MV included as a prediction parameter in the stream, and adds the predicted MV of each control point of the current block to the differential MV corresponding to the predicted MV (step Sj_12). Thus, the MV of each control point of the current block is derived.
[0838] Next, the inter-frame prediction unit 218 performs motion compensation on each of the multiple sub-blocks included in the current block. That is, the inter-frame prediction unit 218 uses two motion vectors v for each of the multiple sub-blocks. 0 and v 1 and the above equation (1A), or using three motion vectors v 0 、v 1 and v 2 and the above-mentioned formula (1B), the MV of the sub-block is calculated as the affine MV (step Sj_13). And, the inter-frame prediction unit 218 uses these affine MVs and the decoded reference picture to perform motion compensation on the sub-block (step Sj_14). If the processing of steps Sj_13 and Sj_14 is performed on all sub-blocks included in the current block, the inter-frame prediction using the affine merge mode for the current block is completed. That is, motion compensation is performed on the current block to generate a predicted image of the current block.
[0839] In addition, in step Sj_11, similarly to step Sk_11, the above-mentioned candidate MV list can also be generated.
[0840] [MV Export > Triangle Mode]
[0841] For example, when the information decoded from the stream indicates application of the triangular mode, the inter prediction unit 218 derives an MV in the triangular mode and performs motion compensation (prediction) using the MV.
[0842] Fig.89 : is a flowchart showing an example of inter-frame prediction based on the triangular mode in the decoding device 200 .
[0843] In the triangular mode, first, the inter-frame prediction unit 218 divides the current block into the first partition and the second partition (step Sx_11). At this time, the inter-frame prediction unit 218 can obtain information related to the division into each partition, that is, partition information, from the stream as a prediction parameter. And, the inter-frame prediction unit 218 can divide the current block into the first partition and the second partition based on the partition information.
[0844] Next, the inter-frame prediction unit 218 first obtains a plurality of candidate MVs for the current block based on information such as MVs of a plurality of decoded blocks temporally or spatially surrounding the current block (step Sx_12). That is, the inter-frame prediction unit 218 creates a candidate MV list.
[0845] Next, the inter-frame prediction unit 218 selects the candidate MV of the first partition and the candidate MV of the second partition from the multiple candidate MVs obtained in step Sx_11 as the first MV and the second MV, respectively (step Sx_13). At this time, the inter-frame prediction unit 218 can obtain MV selection information for identifying the selected candidate MV from the stream as a prediction parameter. And, the inter-frame prediction unit 218 can select the first MV and the second MV based on the MV selection information.
[0846] Next, the inter-frame prediction unit 218 generates a first predicted image by performing motion compensation using the selected first MV and the decoded reference picture (step Sx_14). Similarly, the inter-frame prediction unit 218 generates a second predicted image by performing motion compensation using the selected second MV and the decoded reference picture (step Sx_15).
[0847] Finally, the inter prediction unit 218 generates a prediction image of the current block by performing weighted addition of the first prediction image and the second prediction image (step Sx_16).
[0848] [Sports Search>DMVR]
[0849] For example, when the information decoded from the stream indicates the application of DMVR, the inter prediction unit 218 performs motion search using DMVR.
[0850] Fig.90 : is a flowchart showing an example of a motion search based on DMVR in the decoding device 200 .
[0851] The inter-frame prediction unit 218 first derives the MV of the current block in the merge mode (step S1_11). Next, the inter-frame prediction unit 218 derives the final MV for the current block by searching the surrounding area of the reference picture represented by the MV derived in step S1_11 (step S1_12). That is, the MV of the current block is determined by DMVR.
[0852] Fig.91 This is a flowchart showing a detailed example of the motion search based on DMVR in the decoding device 200.
[0853] First, the inter-frame prediction unit 218 Fig.58A In step 1 shown in FIG. 1 , the cost of the search position (also called the starting point) represented by the initial MV and the eight surrounding search positions is calculated. Next, the inter-frame prediction unit 218 determines whether the cost of the search position other than the starting point is the smallest. Here, if the inter-frame prediction unit 218 determines that the cost of the search position other than the starting point is the smallest, it moves to the search position with the smallest cost and performs Fig.58A On the other hand, if the cost of the starting point is the smallest, the inter-frame prediction unit 218 will Fig.58A The process of step 2 shown is skipped and the process of step 3 is performed.
[0854] exist Fig.58A In step 2 shown, the inter-frame prediction unit 218 uses the search position moved according to the processing result of step 1 as a new starting point, and performs the same search as the processing of step 1. In addition, the inter-frame prediction unit 218 determines whether the cost of the search position other than the starting point is the smallest. Here, if the cost of the search position other than the starting point is the smallest, the inter-frame prediction unit 218 performs the processing of step 4. On the other hand, if the cost of the starting point is the smallest, the inter-frame prediction unit 218 performs the processing of step 3.
[0855] In step 4, the inter-frame prediction unit 218 treats the search position of the starting point as the final search position, and determines the difference between the position indicated by the initial MV and the final search position as a difference vector.
[0856] exist Fig.58A In step 3 shown, the inter-frame prediction unit 218 determines the pixel position with the minimum cost based on the cost of the four points above, below, left and right of the starting point of step 1 or step 2, and sets the pixel position as the final search position. The pixel position with the minimum cost is determined by weighted addition of the vectors ((0, 1)(0, -1)(-1, 0)(1, 0)) of the four points above, below, left and right using the cost at the search position of each of the four points as the weight. In addition, the inter-frame prediction unit 218 determines the difference between the position represented by the initial MV and the final search position as a difference vector.
[0857] [Motion Compensation > BIO / OBMC / LIC]
[0858] For example, when the information decoded from the stream indicates application of correction of the predicted image, the inter prediction unit 218 corrects the predicted image according to the correction mode after generating the predicted image. The correction mode is, for example, the above-mentioned BIO, OBMC, and LIC.
[0859] Fig.92 This is a flowchart showing an example of generation of a predicted image in the decoding device 200 .
[0860] The inter prediction unit 218 generates a predicted image (step Sm_11 ), and corrects the predicted image using any of the above-mentioned modes (step Sm_12 ).
[0861] Fig.93 This is a flowchart showing another example of generation of a predicted image in the decoding device 200 .
[0862] The inter-frame prediction unit 218 derives the MV of the current block (step Sn_11). Next, the inter-frame prediction unit 218 generates a predicted image using the MV (step Sn_12), and determines whether to perform a correction process (step Sn_13). For example, the inter-frame prediction unit 218 obtains a prediction parameter included in the stream, and determines whether to perform a correction process based on the prediction parameter. The prediction parameter is, for example, a flag indicating whether the above-mentioned modes are applied. Here, if the inter-frame prediction unit 218 determines to perform a correction process ("Yes" in step Sn_13), it generates a final predicted image by correcting the predicted image (step Sn_14). In addition, in LIC, the brightness and color difference of the predicted image may also be corrected in step Sn_14. On the other hand, if the inter-frame prediction unit 218 determines not to perform a correction process ("No" in step Sn_13), it outputs the predicted image as the final predicted image without correction (step Sn_15).
[0863] [Motion Compensation > OBMC]
[0864] For example, when the information decoded from the stream indicates the application of OBMC, the inter prediction unit 218 corrects the predicted image in accordance with OBMC when generating the predicted image.
[0865] Fig.94 : is a flowchart showing an example of correction of a predicted image based on OBMC in the decoding device 200. Fig.94 The flowchart shows the use of Fig.62 The process of correcting the predicted image of the current picture and the reference picture is shown.
[0866] First, the inter-frame prediction unit 218 Fig.62As shown, a predicted image (Pred) based on normal motion compensation is obtained using the MV assigned to the current block.
[0867] Next, the inter-frame prediction unit 218 applies (reuses) the MV (MV_L) derived for the decoded left adjacent block to the current block to obtain a predicted image (Pred_L). Next, the inter-frame prediction unit 218 performs the first correction of the predicted image by superimposing the two predicted images Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.
[0868] Similarly, the inter-frame prediction unit 218 applies (reuses) the MV (MV_U) derived for the decoded upper adjacent block to the current block to obtain a predicted image (Pred_U). Next, the inter-frame prediction unit 218 performs a second correction of the predicted image by superimposing the predicted image Pred_U on the predicted image (e.g., Pred and Pred_L) after the first correction. This has the effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is the final predicted image of the current block after blending (smoothing) the boundaries with the adjacent blocks.
[0869] [Motion Compensation > BIO]
[0870] For example, when the information decoded from the stream indicates application of BIO, the inter prediction unit 218 corrects the predicted image according to the BIO when generating the predicted image.
[0871] Fig.95 : is a flowchart showing an example of correction of a predicted image based on BIO in the decoding device 200 .
[0872] The inter-frame prediction unit 218 is as follows Fig.63 As shown, two reference pictures (Ref0, Ref1) different from the picture (Cur Pic) containing the current block are used to derive two motion vectors (M0, M1). Next, the inter prediction unit 218 uses these two motion vectors (M0, M1) to derive a predicted image of the current block (step Sy_11). In addition, the motion vector M0 is the motion vector (MVx0, MVy0) corresponding to the reference picture Ref0, and the motion vector M1 is the motion vector (MVx1, MVy1) corresponding to the reference picture Ref1.
[0873] Next, the inter-frame prediction unit 218 uses the motion vector M0 and the reference picture L0 to derive an interpolated image I of the current block. 0 In addition, the inter-frame prediction unit 218 uses the motion vector M1 and the reference picture L1 to derive the interpolated image I of the current block. 1 (Step Sy_12). Here, the interpolation image I 0is the image contained in the reference picture Ref0 derived for the current block, the interpolated image I 1 It is an image derived for the current block and included in the reference picture Ref1. 0 And the interpolated image I 1 Alternatively, in order to properly derive the gradient image described later, the interpolated image I 0 And the interpolated image I 1 They can also be images larger than the current block. 0 and I 1 It may include a predicted image derived by applying a motion vector (M0, M1) and a reference picture (L0, L1) and a motion compensation filter.
[0874] In addition, the inter-frame prediction unit 218 obtains the interpolation image I 0 And the interpolated image I 1 , export the gradient image of the current block (Ix 0 , Ix 1 , Iy 0 , Iy 1 )(Step Sy_13). In addition, the gradient image in the horizontal direction is (Ix 0 , Ix 1 ), the vertical gradient image is (Iy 0 , Iy 1 ). The inter-frame prediction unit 218 may derive the gradient image by, for example, applying a gradient filter to the interpolation image. The gradient image may be any image that represents the spatial variation of pixel values along the horizontal direction or the vertical direction.
[0875] Next, the inter-frame prediction unit 218 uses the interpolated image (I 0 , I 1 ) and gradient image (Ix 0 , Ix 1 , Iy 0 , Iy 1 ) derives the optical flow (vx, vy) as the velocity vector mentioned above (step Sy_14). As an example, the sub-block may be a sub-CU of 4x4 pixels.
[0876] Next, the inter-frame prediction unit 218 corrects the predicted image of the current block using the optical flow (vx, vy). For example, the inter-frame prediction unit 218 uses the optical flow (vx, vy) to derive a correction value of the value of the pixel included in the current block (step Sy_15). Next, the inter-frame prediction unit 218 can correct the predicted image of the current block using the correction value (step Sy_16). In addition, the correction value can be derived in units of each pixel, or in units of multiple pixels or sub-blocks.
[0877] In addition, the BIO process is not limited to Fig.95 The processing disclosed in . Fig.95 Part of the processing disclosed in the process may be added or replaced by different processing, or may be executed in a different processing order.
[0878] [Motion Compensation > LIC]
[0879] For example, when the information decoded from the stream indicates application of LIC, the inter prediction unit 218 corrects the predicted image according to LIC when generating the predicted image.
[0880] Fig.96 : is a flowchart showing an example of modification of a predicted image based on LIC in the decoding device 200.
[0881] First, the inter prediction unit 218 obtains a reference image corresponding to the current block from a decoded reference picture using MV (step Sz_11).
[0882] Next, the inter-frame prediction unit 218 extracts information indicating how the brightness value of the current block has changed between the reference picture and the current picture (step Sz_12). Fig.66A As shown in FIG. 1 , the inter-frame prediction unit 218 calculates the brightness correction parameter based on the brightness pixel values of the decoded left adjacent reference area (peripheral reference area) and the decoded upper adjacent reference area (peripheral reference area) in the current picture and the brightness pixel values at the same position in the reference picture specified by the derived MV. Next, the inter-frame prediction unit 218 calculates the brightness correction parameter using information indicating how the brightness value has changed (step Sz_13).
[0883] The inter-frame prediction unit 218 generates a predicted image for the current block by applying the brightness correction parameter to the reference image in the reference picture specified by the MV (step Sz_14). That is, the reference image in the reference picture specified by the MV, i.e., the predicted image, is corrected based on the brightness correction parameter. In this correction, brightness correction may be performed or color difference correction may be performed.
[0884] [Prediction Control Department]
[0885] The prediction control unit 220 selects either the intra prediction image or the inter prediction image, and outputs the selected prediction image to the addition unit 208. Overall, the structures, functions, and processes of the prediction control unit 220, the intra prediction unit 216, and the inter prediction unit 218 on the decoding device 200 side may correspond to those of the prediction control unit 128, the intra prediction unit 124, and the inter prediction unit 126 on the encoding device 100 side.
[0886] [Bidirectional prediction and geometric partitioning mode (GPM)]
[0887] The present disclosure provides an extension of motion compensation for video coding by enabling the mixing of uni-predicted and bi-predicted sub-blocks within a coding block using only two motion vectors for bi-directional motion compensation. The concept can also be applied to geometric partitioning mode (GPM).
[0888] Specifically, for one of the two partitions obtained by GPM, unidirectional prediction is applied using one of the two motion candidates that are signaled, as in the prediction in GPM. For the other partition, bidirectional prediction is applied using the two motion candidates that are signaled. In addition, the partition obtained by GPM can also be expressed as a GPM partition.
[0889] The encoding device 100 and the decoding device 200 can predict a data block in an image using an already encoded picture and data obtained from the signaled motion information. The motion information may include a reference picture index pointing to an already encoded / decoded picture (reference picture) in a list of already encoded / decoded pictures together with a motion vector pointing to an associated block (reference block) in an already encoded / decoded picture.
[0890] In many video coding standards, bi-prediction is possible, and two instances of motion information are signaled for one block. One instance of motion information can point to a reference picture that is past the current picture in the display order, and another instance of motion information can point to another reference picture that is future than the current picture in the display order. Here, the display order corresponds to the output order or the input order. The instances of motion information can include motion vectors and reference picture indices, etc.
[0891] Alternatively, two instances of motion information may be used to apply prediction to a block, where the two instances refer to reference pictures that are located before or after the current picture in the display order.
[0892] Prediction using two reference pictures is not limited to one reference picture being located before the current picture and the other reference picture being located after the current picture in the display order. Therefore, inter-frame prediction using one reference block of one reference picture is called uni-prediction, and inter-frame prediction using two reference blocks of two reference pictures is called bi-prediction.
[0893] When performing bi-prediction of the current block, two uni-predictions can be performed using one of the two instances of motion information. Next, the two uni-predictions can be combined and the weight w(x, y) represented by the following equation (3) can be applied to perform bi-prediction of the current block.
[0894] [Formula 4]
[0895] pred(x,y)=w(x,y))*pA(x,y)+(1-w(x,y))*pB(x,y) (3)
[0896] In formula (3), pred(x, y) represents the value of the dual prediction of the pixel located at row y and column x of the current block. The dual prediction pred(x, y) is a combination of two single predictions pA(x, y) and pB(x, y) combined using weight w(x, y). In formula (3), the minimum value of weight w(x, y) is 0, and the maximum value of weight w(x, y) is 1.
[0897] For example, in the case of bi-prediction as defined in VVC, the weight w(x, y) can be set to 1 / 2 for all pixels of the current block. The weight w(x, y) is not limited to 1 / 2, and there are other ways to apply a weight different from 1 / 2. However, typically, the weight is fixed in the entire block. That is, the weight w(x, y) is not necessarily a function of x and y.
[0898] In the bi-prediction of the current block, there is a case where the reference block represented by the motion information is not completely included in the reference picture. In other words, there is a case where the reference block is partially located outside the reference picture. For example, there is a case where the reference block is partially located inside the reference picture and partially located in the padding region surrounding the reference picture. Such a situation is described in Non-Patent Document 2.
[0899] In non-patent document 2, the following scheme is proposed: bi-prediction is performed only on the portion of the current block whose corresponding portion of the reference block is located in the reference picture, and uni-prediction is performed on the portion whose corresponding portion of the reference block is not located in the reference picture. In this case, motion information pointing to the reference block in the reference picture is used in uni-prediction. That is, uni-prediction and bi-prediction are mixed in the same block to predict the block.
[0900] More specifically, the prediction described in Non-Patent Document 2 is used only when the motion vector points to a position outside the reference picture boundary. Furthermore, for the portion of the current block that corresponds to the outside of the picture boundary by one of the two motion vectors, single prediction is used instead of bi-prediction. For the other portions of the current block, bi-prediction is used.
[0901] Fig.97 : is a conceptual diagram showing a dual-single hybrid prediction (hybrid dual / single prediction) when a reference block is partially located outside a reference picture. Fig.97 The lower part of represents the current picture and the current block in the current picture.
[0902] As in Fig.97As shown by the arrow pointing to the reference block of the reference picture in the upper left portion of , th...
Claims
1. A decoding method for decoding a current block of an image from a bit stream, characterized in that: Decoding a prediction mode indicator indicating whether dual-single hybrid prediction should be performed on the current block from the bitstream; decoding two instances of motion information for the current block from the bitstream; When the prediction mode indicator indicates that the dual and single mixed prediction should be performed on the current block, (i) one of the two instances is used to perform the first prediction as a single prediction on the first partition as part of the current block, and (ii) both of the two instances are used to perform the second prediction as a dual prediction on the second partition as part of the current block.
2. The decoding method according to claim 1, characterized in that: The first partition and the second partition in the current block are determined according to a geometric partitioning mode; The current block is divided into the two partitions along the boundary; The boundary is the line that separates the two partitions.
3. The decoding method according to claim 2, characterized in that: A partition indicator indicating which of the two partitions is the first partition and which of the two partitions is at least one of the second partition is also decoded from the bit stream.
4. The decoding method according to claim 1 or 2, characterized in that: A partition indicator is also decoded from the bitstream, which indicates (i) the position and direction of a boundary, where the boundary is the boundary between two partitions, the first partition and the second partition in the current block, and is the line separating the two partitions, and (ii) which of the two instances is used for the prediction of the first partition.
5. The decoding method according to claim 4, characterized in that: The partition indicator further indicates at least one of which of the two partitions is the first partition and which of the two partitions is the second partition.
6. The decoding method according to claim 2, characterized in that: Furthermore, based on at least one of the size and shape of at least one of the two partitions, the two partitions are respectively set as the first partition or the second partition.
7. The decoding method according to claim 1 or 2, characterized in that: In the above-mentioned prediction 1 and prediction 2, Using one of the two examples to perform the first prediction of the current block; Using the other of the above two examples to perform a third prediction of the above current block as a single prediction; Performing a weighted linear combination of the first prediction and the third prediction; For at least a portion of the first partition, the weight ratio of the first prediction is set to 1, and the weight ratio of the third prediction is set to 0; For at least a part of the second partition, the ratio of the third prediction weights is set to be the maximum in the current block and smaller than 1, and the ratio of the first prediction weights is set to the difference between the ratio of the third prediction weights and 1.
8. The decoding method according to claim 7, characterized in that: For the at least a portion of the second partition, the weight ratio of the third prediction is set to 0.5, and the weight ratio of the first prediction is set to 0.
5.
9. The decoding method according to claim 7, characterized in that: For the boundary area, the ratio of the weight of the above-mentioned first prediction and the ratio of the weight of the above-mentioned third prediction are set in a manner that gradually changes from the ratio set for the above-mentioned at least a part of the above-mentioned first partition to the ratio set for the above-mentioned at least a part of the above-mentioned second partition, and the above-mentioned boundary area is an area between the above-mentioned at least a part of the above-mentioned first partition and the above-mentioned at least a part of the above-mentioned second partition, and is an area including the boundary between the above-mentioned first partition and the above-mentioned second partition.
10. The decoding method according to claim 1 or 2, characterized in that: and then, storing, for a first sub-block belonging to at least a part of the first partition, an instance of one of the two instances used in prediction of the first partition; For the second sub-block belonging to at least one of the boundary area and the at least one part of the above-mentioned second partition, both sides of the above-mentioned two instances are stored, and the above-mentioned boundary area is the area between the above-mentioned at least one part of the above-mentioned first partition and at least one part of the above-mentioned second partition, and is the area including the boundary between the above-mentioned first partition and the above-mentioned second partition.
11. The decoding method according to claim 1 or 2, characterized in that: When at least one of the first partition and the second partition has a predetermined shape, the dual-single mixed prediction is not performed.
12. The decoding method according to claim 11, characterized in that: The above-mentioned predetermined shape is a shape that can be realized by a normal block division different from the partition division in the above-mentioned dual-single mixed prediction.
13. A method for encoding a current block of an image into a bit stream, characterized in that: In a case where a hybrid dual-single prediction should be performed on the current block, (i) a first prediction as a uni-prediction is performed on a first partition as part of the current block using one of two instances of motion information for the current block, and (ii) a second prediction as a bi-prediction is performed on a second partition as part of the current block using both of the two instances; encoding a prediction mode indicator indicating whether the dual-single hybrid prediction should be performed on the current block into the bitstream; The above 2 instances are encoded into the above bitstream.
14. The encoding method according to claim 13, characterized in that: The first partition and the second partition in the current block are determined according to a geometric partitioning mode; The current block is divided into the two partitions along the boundary; The boundary is the line that separates the two partitions.
15. The encoding method according to claim 14, characterized in that: A partition indicator is also encoded in the bit stream, the partition indicator indicating which of the two partitions is the first partition and which of the two partitions is at least one of the second partition.
16. The encoding method according to claim 13 or 14, characterized in that: A partition indicator is also encoded into the bitstream, the partition indicator indicating (i) the position and direction of a boundary, the boundary being the boundary between two partitions, the first partition and the second partition in the current block, and being the line separating the two partitions, and (ii) which of the two instances is used for the prediction of the first partition.
17. The encoding method according to claim 16, characterized in that: The partition indicator further indicates at least one of which of the two partitions is the first partition and which of the two partitions is the second partition.
18. The encoding method according to claim 14, characterized in that: Furthermore, based on at least one of the size and shape of at least one of the two partitions, the two partitions are respectively set as the first partition or the second partition.
19. The encoding method according to claim 13 or 14, characterized in that: In the above-mentioned prediction 1 and prediction 2, Using one of the two examples to perform the first prediction of the current block; Using the other of the above two examples to perform a third prediction of the above current block as a single prediction; Performing a weighted linear combination of the first prediction and the third prediction; For at least a portion of the first partition, the weight ratio of the first prediction is set to 1, and the weight ratio of the third prediction is set to 0; For at least a part of the second partition, the ratio of the third prediction weights is set to be the maximum in the current block and smaller than 1, and the ratio of the first prediction weights is set to the difference between the ratio of the third prediction weights and 1.
20. The encoding method according to claim 19, characterized in that For the at least a portion of the second partition, the weight ratio of the third prediction is set to 0.5, and the weight ratio of the first prediction is set to 0.
5.
21. The encoding method according to claim 19, characterized in that For the boundary area, the ratio of the weight of the above-mentioned first prediction and the ratio of the weight of the above-mentioned third prediction are set in a manner that gradually changes from the ratio set for the above-mentioned at least a part of the above-mentioned first partition to the ratio set for the above-mentioned at least a part of the above-mentioned second partition, and the above-mentioned boundary area is an area between the above-mentioned at least a part of the above-mentioned first partition and the above-mentioned at least a part of the above-mentioned second partition, and is an area including the boundary between the above-mentioned first partition and the above-mentioned second partition.
22. The encoding method according to claim 13 or 14, characterized in that: and then, storing, for a first sub-block belonging to at least a part of the first partition, an instance of one of the two instances used in prediction of the first partition; For the second sub-block belonging to at least one of the boundary area and the at least one part of the above-mentioned second partition, both sides of the above-mentioned two instances are stored, and the above-mentioned boundary area is the area between the above-mentioned at least one part of the above-mentioned first partition and at least one part of the above-mentioned second partition, and is the area including the boundary between the above-mentioned first partition and the above-mentioned second partition.
23. The encoding method according to claim 13 or 14, characterized in that: When at least one of the first partition and the second partition has a predetermined shape, the dual-single mixed prediction is not performed.
24. The encoding method according to claim 23, characterized in that The above-mentioned predetermined shape is a shape that can be realized by a normal block division different from the partition division in the above-mentioned dual-single mixed prediction.
25. A non-transitory computer-readable recording medium, which is a non-transitory computer-readable recording medium for a computer, characterized in that: storing a bit stream that causes a computer to perform a decoding process for decoding a current block using a motion vector; In the above decoding process, Decoding a prediction mode indicator indicating whether dual-single hybrid prediction should be performed on the current block from the bitstream; decoding two instances of motion information for the current block from the bitstream; When the prediction mode indicator indicates that the dual and single mixed prediction should be performed on the current block, (i) one of the two instances is used to perform the first prediction as a single prediction on the first partition as part of the current block, and (ii) both of the two instances are used to perform the second prediction as a dual prediction on the second partition as part of the current block.
26. A decoding device for decoding a current block of an image from a bit stream, characterized in that: have: Circuits; and A memory connected to the above circuit; The above circuit is in action. Decoding a prediction mode indicator indicating whether dual-single hybrid prediction should be performed on the current block from the bitstream; decoding two instances of motion information for the current block from the bitstream; When the prediction mode indicator indicates that the dual and single mixed prediction should be performed on the current block, (i) one of the two instances is used to perform the first prediction as a single prediction on the first partition as part of the current block, and (ii) both of the two instances are used to perform the second prediction as a dual prediction on the second partition as part of the current block.
27. An encoding device for encoding a current block of an image into a bit stream, characterized in that have: Circuits; and A memory connected to the above circuit; The above circuit is in action. In a case where a hybrid dual-single prediction should be performed on the current block, (i) a first prediction as a uni-prediction is performed on a first partition as part of the current block using one of two instances of motion information for the current block, and (ii) a second prediction as a bi-prediction is performed on a second partition as part of the current block using both of the two instances; encoding a prediction mode indicator indicating whether the dual-single hybrid prediction should be performed on the current block into the bitstream; The above 2 instances are encoded into the above bitstream.
