Encoding device, decoding device, encoding method, and decoding method

Abstract

Include the appropriate information in the bitstream. [Solution] The encoding device 100 comprises a circuit and a memory, and the circuit includes in a bitstream a value indicating the DPB Output Delay in a first case in which multiple temporal layers are to be decoded, and outputs the bitstream to a network, wherein n-1 upper pictures are placed between adjacent lower pictures in display order, n indicates the frame rate ratio between the first case and the second case in which the first temporal layer and below are to be decoded, and when n is 2, n-1 upper pictures are placed between adjacent lower pictures in coding order, the value indicating the DPB Output Delay of the lower picture in the first case is equal to the value indicating the DPB Output Delay of the lower picture in the second case, and when n is 4, n-1 upper pictures are not placed between adjacent lower pictures in coding order.

Description

[Technical Field]

[0001] This disclosure relates to video coding, and more particularly to systems, components, and methods for encoding and decoding moving images. [Background technology]

[0002] Video coding technology has advanced from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). With this advancement, there is a constant need to provide improvements and optimizations to video coding technology to handle the ever-increasing volume of digital video data across various applications. This disclosure relates to further advancements, improvements, and optimizations in video coding.

[0003] Non-patent document 1 relates to an example of a conventional standard concerning the video coding technology described above. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] H.265(ISO / IEC 23008-2 HEVC) / HEVC(High Efficiency Video Coding) [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] Regarding the encoding methods described above, there is a need for proposals for new methods to improve encoding efficiency, image quality, processing load, circuit size, or to appropriately select elements or actions such as filters, blocks, size, motion vectors, reference pictures, or reference blocks.

[0006] The present disclosure provides a configuration or method that can contribute to, for example, one or more of improvement in encoding efficiency, improvement in image quality, reduction in processing amount, reduction in circuit scale, improvement in processing speed, and appropriate selection of elements or operations. Note that the present disclosure may include a configuration or method that can contribute to benefits other than those described above.

Means for Solving the Problems

[0007] For example, an encoding apparatus according to one aspect of the present disclosure includes a circuit and a memory connected to the circuit. In operation, the circuit encodes a plurality of pictures according to a time scalability that defines a plurality of temporal layers, includes a value indicating the DPB (Decoded Picture Buffer) Output Delay of each of the plurality of pictures in a first case where the plurality of temporal layers are decoding targets in a bit stream, outputs the bit stream to a network, and sets pictures included in temporal layers below a first temporal layer among the plurality of temporal layers as lower pictures, and pictures included in temporal layers larger than the first temporal layer among the plurality of temporal layers as upper pictures. When at least one upper picture is arranged between two adjacent lower pictures in the display order, the number of the at least one upper picture is n - 1, n is an integer greater than 1, and n indicates a ratio between a frame rate in a second case where temporal layers below the first temporal layer are decoding targets and the frame rate in the first case. When n is 2, n - 1 upper pictures are arranged between two adjacent lower pictures in the encoding order, a value indicating the DPB Output Delay of the lower picture in the first case is equal to a value indicating the DPB Output Delay of the lower picture in the second case, and when n is 4, n - 1 upper pictures are not arranged between two adjacent lower pictures in the encoding order.

[0008] Each embodiment in the present disclosure, or each of some of its configurations or methods, enables at least one of, for example, improvement in encoding efficiency, improvement in image quality, reduction in the amount of encoding / decoding processing, reduction in circuit scale, or improvement in the processing speed of encoding / decoding. Alternatively, each embodiment in the present disclosure, or each of some of its configurations or methods, enables an appropriate selection of components / operations such as filters, blocks, sizes, motion vectors, reference pictures, reference blocks, etc. in encoding and decoding. Note that the present disclosure also includes disclosure of configurations or methods that can provide benefits other than those described above. For example, a configuration or method that improves encoding efficiency while suppressing an increase in the amount of processing.

[0009] Further advantages and effects in one aspect of the present disclosure will become apparent from the specification and the drawings. Such advantages and / or effects can be obtained respectively by some embodiments and the features described in the specification and the drawings, but not all of them are necessarily provided in order to obtain one or more advantages and / or effects.

[0010] Note that these general or specific aspects may be implemented by a system, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be implemented by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

Advantages of the Invention

[0011] A configuration or method according to one aspect of the present disclosure can contribute to one or more of, for example, improvement in encoding efficiency, improvement in image quality, reduction in processing amount, reduction in circuit scale, improvement in processing speed, and appropriate selection of elements or operations. Note that a configuration or method according to one aspect of the present disclosure may contribute to benefits other than those described above.

Brief Description of the Drawings

[0012] [Figure 1] FIG. 1 is a schematic diagram showing an example of the configuration of a transmission system according to an embodiment. [Figure 2] Figure 2 shows an example of a data hierarchical structure in a stream. [Figure 3] Figure 3 shows an example of the slice configuration. [Figure 4] Figure 4 shows an example of a tile configuration. [Figure 5] Figure 5 shows an example of an encoding structure during scalable encoding. [Figure 6] Figure 6 shows an example of an encoding structure during scalable encoding. [Figure 7] Figure 7 is a block diagram showing an example of the configuration of an encoding device according to an embodiment. [Figure 8] Figure 8 is a block diagram showing an example implementation of an encoding device. [Figure 9] Figure 9 is a flowchart showing an example of the overall encoding process by the encoding device. [Figure 10] Figure 10 shows an example of block division. [Figure 11] Figure 11 shows an example of the configuration of the divided section. [Figure 12] Figure 12 shows an example of a division pattern. [Figure 13A] Figure 13A shows an example of a syntax tree for a partitioning pattern. [Figure 13B] Figure 13B shows another example of a syntax tree for a partitioning pattern. [Figure 14] Figure 14 is a table showing the transformation basis functions corresponding to each transformation type. [Figure 15] Figure 15 shows an example of an SVT. [Figure 16] Figure 16 is a flowchart showing an example of processing by the conversion unit. [Figure 17] Figure 17 is a flowchart showing another example of processing by the conversion unit. [Figure 18] Figure 18 is a block diagram showing an example of the configuration of the quantization unit. [Figure 19]Figure 19 is a flowchart showing an example of quantization by the quantization unit. [Figure 20] Figure 20 is a block diagram showing an example of the configuration of the entropy coding unit. [Figure 21] Figure 21 shows the CABAC flow in the entropy coding section. [Figure 22] Figure 22 is a block diagram showing an example of the configuration of the loop filter section. [Figure 23A] Figure 23A shows an example of the filter shape used in an adaptive loop filter (ALF). [Figure 23B] Figure 23B shows another example of the filter shape used in ALF. [Figure 23C] Figure 23C shows another example of the filter shape used in ALF. [Figure 23D] Figure 23D shows an example where the Y sample (first component) is used in Cb CCALF and Cr CCALF (multiple components different from the first component). [Figure 23E] Figure 23E shows a diamond-shaped filter. [Figure 23F] Figure 23F shows an example of JC-CCALF. [Figure 23G] Figure 23G shows an example of a candidate weight_index for JC-CCALF. [Figure 24] Figure 24 is a block diagram showing an example of a detailed configuration of the loop filter section that functions as a DBF. [Figure 25] Figure 25 shows an example of a deblocking filter with symmetrical filter characteristics with respect to block boundaries. [Figure 26] Figure 26 illustrates an example of a block boundary where deblocking filtering is performed. [Figure 27] Figure 27 shows an example of a Bs value. [Figure 28] Figure 28 is a flowchart showing an example of the processing performed in the prediction unit of the encoding device. [Figure 29] Figure 29 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device. [Figure 30] Figure 30 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device. [Figure 31] Figure 31 shows an example of 67 intra-prediction modes in intra-prediction. [Figure 32] Figure 32 is a flowchart showing an example of processing by the intra-prediction unit. [Figure 33] Figure 33 shows an example of each reference picture. [Figure 34] Figure 34 is a conceptual diagram showing an example of a reference picture list. [Figure 35] Figure 35 is a flowchart showing the basic processing flow of interpretation prediction. [Figure 36] Figure 36 is a flowchart showing an example of MV derivation. [Figure 37] Figure 37 is a flowchart showing another example of MV derivation. [Figure 38A] Figure 38A shows an example of the classification of each mode in MV derivation. [Figure 38B] Figure 38B shows an example of the classification of each mode in MV derivation. [Figure 39] Figure 39 is a flowchart showing an example of inter-mode prediction. [Figure 40] Figure 40 is a flowchart showing an example of inter prediction using normal merge mode. [Figure 41] Figure 41 is a diagram illustrating an example of MV derivation processing using normal merge mode. [Figure 42] Figure 42 illustrates an example of MV derivation processing using HMVP (History-based Motion Vector Prediction / Predictor) mode. [Figure 43]Figure 43 is a flowchart showing an example of FRUC (frame rate up conversion). [Figure 44] Figure 44 illustrates an example of pattern matching (bilateral matching) between two blocks along a motion trajectory. [Figure 45] Figure 45 illustrates an example of pattern matching (template matching) between a template in the current picture and a block in a referenced picture. [Figure 46A] Figure 46A illustrates an example of deriving the subblock unit MV in affine mode using two control points. [Figure 46B] Figure 46B illustrates an example of deriving the subblock unit MV in affine mode using three control points. [Figure 47A] Figure 47A is a conceptual diagram illustrating an example of the MV derivation of the control point in affine mode. [Figure 47B] Figure 47B is a conceptual diagram illustrating an example of the MV derivation of the control point in affine mode. [Figure 47C] Figure 47C is a conceptual diagram illustrating an example of the MV derivation of the control point in affine mode. [Figure 48A] Figure 48A is a diagram illustrating an affine mode with two control points. [Figure 48B] Figure 48B is a diagram illustrating an affine mode with three control points. [Figure 49A] Figure 49A is a conceptual diagram illustrating an example of a method for deriving the control point MV when the number of control points differs between the encoded block and the current block. [Figure 49B] Figure 49B is a conceptual diagram illustrating another example of how to derive the control point MV when the number of control points differs between the encoded block and the current block. [Figure 50]Figure 50 is a flowchart showing an example of processing in affine merge mode. [Figure 51] Figure 51 is a flowchart showing an example of affine intermode processing. [Figure 52A] Figure 52A is a diagram illustrating the generation of two triangular prediction images. [Figure 52B] Figure 52B is a conceptual diagram showing the first part of the first partition, as well as examples of the first and second sample sets. [Figure 52C] Figure 52C is a conceptual diagram showing the first part of the first partition. [Figure 53] Figure 53 is a flowchart showing an example of the triangle mode. [Figure 54] Figure 54 shows an example of the ATMVP (Advanced Temporal Motion Vector Prediction / Predictor) mode in which MV is derived at the subblock level. [Figure 55] Figure 55 shows the relationship between merge mode and DMVR (dynamic motion vector refreshing). [Figure 56] Figure 56 is a conceptual diagram illustrating an example of a DMVR. [Figure 57] Figure 57 is a conceptual diagram illustrating another example of DMVR for determining MV. [Figure 58A] Figure 58A shows an example of motion search in DMVR. [Figure 58B] Figure 58B is a flowchart showing an example of motion search in a DMVR. [Figure 59] Figure 59 is a flowchart showing an example of predictive image generation. [Figure 60] Figure 60 is a flowchart showing another example of predictive image generation. [Figure 61]Figure 61 is a flowchart illustrating an example of predictive image correction processing using OBMC (overlapped block motion compensation). [Figure 62] Figure 62 is a conceptual diagram illustrating an example of predictive image correction processing using OBMC. [Figure 63] Figure 63 is a diagram illustrating a model that assumes uniform linear motion. [Figure 64] Figure 64 is a flowchart showing an example of an inter-prediction method according to BIO. [Figure 65] Figure 65 shows an example of the configuration of the inter prediction unit that performs inter prediction according to BIO. [Figure 66A] Figure 66A illustrates an example of a predictive image generation method using luminance correction processing by LIC (local illumination compensation). [Figure 66B] Figure 66B is a flowchart showing an example of a predictive image generation method using luminance correction processing by LIC. [Figure 67] Figure 67 is a block diagram showing the configuration of a decoding device according to an embodiment. [Figure 68] Figure 68 is a block diagram showing an example of a decoding device implementation. [Figure 69] Figure 69 is a flowchart showing an example of the overall decoding process by the decoding device. [Figure 70] Figure 70 shows the relationship between the division decision unit and other components. [Figure 71] Figure 71 is a block diagram showing an example of the configuration of the entropy decoding unit. [Figure 72] Figure 72 shows the CABAC flow in the entropy decoding section. [Figure 73] Figure 73 is a block diagram showing an example of the configuration of the inverse quantization unit. [Figure 74] Figure 74 is a flowchart showing an example of inverse quantization by an inverse quantization unit. [Figure 75]Figure 75 is a flowchart showing an example of processing by the inverse transform unit. [Figure 76] Figure 76 is a flowchart showing another example of processing by the inverse transform unit. [Figure 77] Figure 77 is a block diagram showing an example of the configuration of the loop filter section. [Figure 78] Figure 78 is a flowchart showing an example of the processing performed in the prediction unit of the decoding device. [Figure 79] Figure 79 is a flowchart showing another example of the processing performed in the prediction unit of the decoding device. [Figure 80A] Figure 80A is a flowchart showing some other examples of the processing performed in the prediction unit of the decoding device. [Figure 80B] Figure 80B is a flowchart showing the remainder of other examples of processing performed in the prediction section of the decoding device. [Figure 81] Figure 81 shows an example of processing performed by the intra-prediction unit of the decoding device. [Figure 82] Figure 82 is a flowchart showing an example of MV derivation in a decoding device. [Figure 83] Figure 83 is a flowchart showing another example of MV derivation in a decoding device. [Figure 84] Figure 84 is a flowchart showing an example of inter-mode prediction in a decoding device. [Figure 85] Figure 85 is a flowchart showing an example of inter prediction using normal merge mode in a decoding device. [Figure 86] Figure 86 is a flowchart showing an example of interpretation using FRUC mode in a decoding device. [Figure 87] Figure 87 is a flowchart showing an example of interpretation using affine merge mode in a decoding device. [Figure 88] Figure 88 is a flowchart showing an example of inter-prediction using affine intermode in a decoding device. [Figure 89]Figure 89 is a flowchart showing an example of inter prediction using triangle mode in a decoding device. [Figure 90] Figure 90 is a flowchart showing an example of motion detection using a DMVR in a decoding device. [Figure 91] Figure 91 is a flowchart showing a detailed example of motion detection by DMVR in a decoding device. [Figure 92] Figure 92 is a flowchart showing an example of predictive image generation in a decoding device. [Figure 93] Figure 93 is a flowchart showing another example of predictive image generation in a decoding device. [Figure 94] Figure 94 is a flowchart showing an example of image correction by OBMC in a decoding device. [Figure 95] Figure 95 is a flowchart showing an example of BIO-based correction of predicted images in a decoding device. [Figure 96] Figure 96 is a flowchart showing an example of predictive image correction by LIC in a decoding device. [Figure 97] Figure 97 shows an example of the syntax of a buffering period SEI message (BP SEI). [Figure 98] Figure 98 shows an example of the syntax for a picture timing SEI message (PT SEI). [Figure 99] Figure 99 shows an example of DPB (Decoded Picture Buffer) parameter syntax. [Figure 100] Figure 100 shows an example of an interpredictive reference picture structure having time scalability according to the first embodiment. [Figure 101] Figure 101 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure of Figure 100. [Figure 102]Figure 102 shows the relationship between the decoding or display timing of the 120fps layer and the sub-layers (60fps layer and 30fps layer) in the interpredictive reference picture structure of Figure 100. [Figure 103] Figure 103 shows an example of parameters related to the decoding or display timing of each AU in the interpredictive reference picture structure of Figure 100. [Figure 104] Figure 104 shows an example of an interpredictive reference picture structure with time scalability according to the first embodiment. [Figure 105] Figure 105 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure of Figure 104. [Figure 106] Figure 106 shows an example of the parameters related to the decoding / display timing of each AU in the interpredictive reference picture structure of Figure 104. [Figure 107] Figure 107 shows an example of an interpredictive reference picture structure with time scalability according to the first embodiment. [Figure 108] Figure 108 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure of Figure 107. [Figure 109] Figure 109 shows an example of parameters related to the decoding or display timing of each AU in the interpredictive reference picture structure of Figure 107. [Figure 110] Figure 110 shows an example of an interpredictive reference picture structure having time scalability according to the first embodiment. [Figure 111] Figure 111 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure of Figure 110. [Figure 112] Figure 112 shows an example of the parameters related to the decoding / display timing of each AU in the interpredictive reference picture structure of Figure 110. [Figure 113]Figure 113 shows an example of the parameters related to the decoding / display timing of each AU in the interpredictive reference picture structure of Figure 110. [Figure 114] Figure 114 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure of Figure 113. [Figure 115] Figure 115 shows an example of the parameters related to the decoding / display timing of each AU in the interpredictive reference picture structure of Figure 113. [Figure 116] Figure 116 shows an example of an interpredictive reference picture structure with time scalability according to the second embodiment. [Figure 117] Figure 117 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure of Figure 116. [Figure 118] Figure 118 shows the relationship between the decoding or display timing of the 120fps layer and the sub-layers (60fps layer and 30fps layer) in the interpredictive reference picture structure of Figure 116. [Figure 119] Figure 119 shows an example of parameters related to the decoding or display timing of each AU in the interpredictive reference picture structure of Figure 116. [Figure 120] Figure 120 shows an example of an interpredictive reference picture structure with time scalability according to the second embodiment. [Figure 121] Figure 121 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure of Figure 120. [Figure 122] Figure 122 shows an example of the parameters related to the decoding / display timing of each AU in the interpredictive reference picture structure of Figure 120. [Figure 123] Figure 123 shows an example of an interpredictive reference picture structure with time scalability according to the second embodiment. [Figure 124]Figure 124 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure of Figure 123. [Figure 125] Figure 125 shows an example of the parameters related to the decoding / display timing of each AU in the interpredictive reference picture structure of Figure 123. [Figure 126] Figure 126 shows an example of an interpredictive reference picture structure with time scalability according to the second embodiment. [Figure 127] Figure 127 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure shown in Figure 126. [Figure 128] Figure 128 shows an example of the parameters related to the decoding / display timing of each AU in the interpredictive reference picture structure of Figure 126. [Figure 129] Figure 129 shows an example of an interpredictive reference picture structure with time scalability according to the second embodiment. [Figure 130] Figure 130 shows an example of DPB parameters included in a bitstream having the interpredictive reference picture structure shown in Figure 129. [Figure 131] Figure 131 shows an example of the parameters related to the decoding / display timing of each AU in the interpredictive reference picture structure of Figure 129. [Figure 132] Figure 132 shows an example of a third-mode interpredictive reference picture structure, which is formed by linking together multiple first-mode interpredictive reference picture structures having multiple independent time scalability. [Figure 133] Figure 133 shows an example of a third-mode interpredictive reference picture structure, which is formed by linking together multiple first-mode interpredictive reference picture structures having multiple independent time scalability. [Figure 134] Figure 134 shows an example of a third-mode interpredictive reference picture structure, which is formed by linking together multiple second-mode interpredictive reference picture structures having multiple independent time scalability. [Figure 135]Figure 135 shows an example of a third-mode interpredictive reference picture structure, which is formed by linking together multiple second-mode interpredictive reference picture structures having multiple independent time scalability. [Figure 136] Figure 136 is a flowchart showing the operations performed by the encoding device. [Figure 137] Figure 137 is a flowchart showing the operations performed by the decryption device. [Figure 138] Figure 138 is an overall diagram of the content supply system that enables the content distribution service. [Figure 139] Figure 139 shows an example of how a web page is displayed. [Figure 140] Figure 140 shows an example of how a web page is displayed. [Figure 141] Figure 141 shows an example of a smartphone. [Figure 142] Figure 142 is a block diagram showing an example of a smartphone configuration. [Modes for carrying out the invention]

[0013] [Introduction] In the field of video encoding, there is a conventional technique called time scalability coding, which defines multiple temporal layers. With this time scalability coding, the frame rate can be changed by changing the temporal layer being encoded.

[0014] Furthermore, in the field of video encoding, DPB (Decoded Picture Buffer) Output Timing is known for controlling the display time of pictures.

[0015] The inventors of this application have found that if the ratio of high frame rates to low frame rates achieved by time scalability coding and the arrangement of multiple pictures in coding order or decoding order satisfy a predetermined relationship, then the DPB Output Timing of a given picture at a high frame rate may match the DPB Output Timing of a given picture at a low frame rate. In other words, the inventors of this application have found that if the predetermined relationship is not satisfied, then the DPB Output Timing of a given picture at a high frame rate may not match the DPB Output Timing of a given picture at a low frame rate.

[0016] Therefore, an encoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, encodes a plurality of pictures in encoding order according to a time scalability defining a plurality of temporal layers, and includes in the bitstream information indicating whether the DPB (Decoded Picture Buffer) Output Timings of a plurality of lower pictures included in the at least one first temporal layer in the first case, where the decoding target is at least one first temporal layer below a lower temporal layer among the plurality of temporal layers, matches the DPB Output Timings of the plurality of lower pictures in the second case, where the decoding target is a plurality of second temporal layers below the upper temporal layer, and at least one upper picture included in the upper temporal layer is provided between two adjacent lower pictures among the plurality of lower pictures in the encoding order, the number of the at least one upper picture is n-1, where n is an integer greater than 1, and represents the ratio of the second frame rate in the second case to the first frame rate in the first case.

[0017] As a result, for example, when n, which represents the ratio, is equal to 2, and multiple upper-level pictures and multiple lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may coincide. In this case, the encoding device according to one aspect of the present disclosure can notify the decoding device that there is no need to adjust the DPB Output Timings, whether for the first or second frame rate, by including information indicating the coincidence in the bitstream. Also, for example, even when n, which represents the ratio, is equal to 2, and upper-level pictures and lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may not coincide. For example, if the DPB Output Timings in the first case are delayed more than the DPB Output Timings in the second case in order to adjust the display time of the decoded picture, the DPB Output Timings in the first case and the DPB Output Timings in the second case will not coincide. In this case, an encoding device according to one aspect of the present disclosure may notify a decoding device that it is necessary to adjust the DPB Output Timings in either the first or second case by including information indicating a mismatch in the bitstream. The adjustment of DPB Output Timings is required when the decoding device switches the frame rate.

[0018] Furthermore, for example, if n is equal to 2, then in the encoding order, the multiple upper pictures included in the upper temporal layer and the multiple lower pictures are arranged alternately.

[0019] As a result, an encoding device according to one aspect of the present disclosure can notify a decoding device that, when n, which represents the ratio, is equal to 2, and multiple upper-level pictures and multiple lower-level pictures are arranged alternately, there is no need to adjust the DPB Output Timings, whether for a first frame rate or a second frame rate.

[0020] Furthermore, for example, if n is equal to 4, the plurality of lower pictures are placed in the m-th position, which is a non-negative integer, in the coding order, and the plurality of upper pictures that are coded only in the second case are placed in the m+1th, m+2nd, and m+3rd positions in the coding order.

[0021] As a result, even with such an arrangement, the encoding device according to one aspect of the present disclosure can notify the decoding device whether or not it is necessary to adjust the DPB Output Timing by including the above information in the bitstream, since the DPB Output Timing in the first case and the DPB Output Timing in the second case may or may not match.

[0022] Furthermore, for example, the plurality of pictures include a plurality of second pictures that are decoded in common in the third case, the first case, and the second case, where the decoding target is at least one third temporal layer other than the upper temporal layer and the lower temporal layer, the information indicates that the DPB Output Timings of the plurality of second pictures in the first case match the DPB Output Timings of the plurality of second pictures in the second case, and the circuit includes second information in the bitstream indicating whether the DPB Output Timings of the plurality of second pictures in the first or second case match the DPB Output Timings of the plurality of second pictures in the third case.

[0023] As a result, even if the DPB Output Timings in the first case and the DPB Output Timings in the second case match, the limitations on picture placement make it highly likely that the DPB Output Timings in the third case will not match them. Therefore, an encoding device according to one aspect of the present disclosure can notify a decoding device whether or not it is necessary to adjust the DPB Output Timings by including the second information in the bitstream.

[0024] Furthermore, for example, the third temporal layer is a second lower temporal layer lower than the lower temporal layer, and the circuit includes in the bitstream the information indicating that the DPB Output Timings of the plurality of second pictures in the first case match the DPB Output Timings of the plurality of second pictures in the second case, and includes in the bitstream the second information indicating that the DPB Output Timings of the plurality of second pictures in the first or second case do not match the DPB Output Timings of the plurality of second pictures in the third case.

[0025] This allows the encoding device 100 to notify the decoding device that it needs to adjust the DPB Output Timings by including the second information in the bitstream, for example, if the DPB Output Timings in the first case and the DPB Output Timings in the second case match, but the DPB Output Timings in the third case do not match them due to picture placement restrictions.

[0026] Furthermore, for example, the information includes first delay information indicating whether or not there is a first delay in the DPB Output Timing of the upper temporal layer relative to the DPB Output Timing of the top-level temporal layer, and second delay information indicating whether or not there is a second delay in the DPB Output Timing of the lower temporal layer relative to the DPB Output Timing of the top-level temporal layer.

[0027] As a result, an encoding device according to one aspect of the present disclosure can notify a decoding device that, for example, there is no first delay and no second delay, there is no need to adjust the DPB Output Timing.

[0028] Furthermore, for example, the top-level temporal layer and the upper-level temporal layer are the same, and the first delay information is not included in the information.

[0029] As a result, when the top-level temporal layer and the upper temporal layer are the same temporal layer, the encoding device can notify the decoding device whether or not it is necessary to adjust the DPB Output Timing simply by including second delay information indicating the presence or absence of a second delay in the lower temporal layer's DPB Output Timing in the above information.

[0030] Furthermore, a decoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, obtains information from a bitstream indicating whether the DPB (Decoded Picture Buffer) Output Timings of a plurality of lower pictures included in the at least one first temporal layer, in the first case where the decoding target is at least one first temporal layer below a lower temporal layer among a plurality of temporal layers, and the DPB Output Timings of the plurality of lower pictures, in the second case where the decoding target is a plurality of second temporal layers below the upper temporal layer, match, and decodes the plurality of pictures in decoding order according to the time scalability that defines the plurality of temporal layers based on the information, wherein at least one upper picture to be included in the upper temporal layer is provided between two adjacent lower pictures among the plurality of lower pictures in the decoding order, the number of the at least one upper picture is n-1, where n is an integer greater than 1, and represents the ratio of the second frame rate in the second case to the first frame rate in the first case.

[0031] As a result, for example, if n, which represents the ratio, is equal to 2, and multiple upper-level pictures and multiple lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may coincide. In this case, the decoding device according to one aspect of the present disclosure can obtain information indicating the coincidence from the bitstream and know that there is no need to adjust the DPB Output Timings in either the first or second frame rate. Also, for example, even if n, which represents the ratio, is equal to 2, and upper-level pictures and lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may not coincide. For example, if the DPB Output Timings in the first case are delayed more than the DPB Output Timings in the second case in order to adjust the display time of the decoded picture, the DPB Output Timings in the first case and the DPB Output Timings in the second case will not coincide. In this case, a decoding device according to one aspect of the present disclosure can obtain information from the bitstream indicating a mismatch, thereby knowing that it is necessary to adjust the DPB Output Timings in either the first or second case. Therefore, a decoding device according to one aspect of the present disclosure can appropriately adjust the DPB Output Timing by obtaining the above information. Note that adjustment of DPB Output Timings is necessary when the decoding device switches the frame rate.

[0032] Furthermore, for example, if n is equal to 2, then in the decoding order, the multiple upper pictures included in the upper temporal layer and the multiple lower pictures are arranged alternately.

[0033] As a result, the decoding device according to one aspect of the present disclosure does not adjust the DPB Output Timings, whether at the first frame rate or the second frame rate, when n, which represents the ratio, is equal to 2, and multiple upper pictures and multiple lower pictures are arranged alternately.

[0034] Furthermore, for example, if n is equal to 4, in the decoding order, the multiple lower pictures are placed in the m-th position which is a non-negative integer, and in the decoding order, the multiple upper pictures that are decoded only in the second case are placed in the m+1th, m+2nd, and m+3rd positions.

[0035] As a result, even with such an arrangement, the decoding device according to one aspect of the present disclosure can obtain the above information from the bitstream and switch whether or not to adjust the DPB Output Timing based on the above information, since the DPB Output Timing in the first case and the DPB Output Timing in the second case may or may not match.

[0036] Furthermore, for example, the plurality of pictures include a plurality of second pictures that are decoded in common in the third case, the first case, and the second case, where the decoding target is at least one third temporal layer other than the upper temporal layer and the lower temporal layer, the information indicates that the DPB Output Timings of the plurality of second pictures in the first case match the DPB Output Timings of the plurality of second pictures in the second case, and the circuit includes second information in the bitstream indicating whether the DPB Output Timings of the plurality of second pictures in the first or second case match the DPB Output Timings of the plurality of second pictures in the third case.

[0037] As a result, even if the DPB Output Timings in the first case and the DPB Output Timings in the second case match, the limitations on picture placement make it highly likely that the DPB Output Timings in the third case will not match them. Therefore, a decoding device according to one aspect of this disclosure can switch whether or not to adjust the DPB Output Timing based on the above information by obtaining second information from the bitstream.

[0038] Furthermore, for example, the third temporal layer is a second lower temporal layer lower than the lower temporal layer, and the circuit includes in the bitstream the information indicating that the DPB Output Timings of the plurality of second pictures in the first case match the DPB Output Timings of the plurality of second pictures in the second case, and includes in the bitstream the second information indicating that the DPB Output Timings of the plurality of second pictures in the first or second case do not match the DPB Output Timings of the plurality of second pictures in the third case.

[0039] As a result, a decoding device according to one aspect of the present disclosure can adjust the DPB Output Timings by obtaining second information from the bitstream, for example, if the DPB Output Timings in the first case and the DPB Output Timings in the second case match, but the DPB Output Timings in the third case do not match them due to picture placement restrictions.

[0040] Furthermore, for example, the information includes first delay information indicating whether or not there is a first delay in the DPB Output Timing of the upper temporal layer relative to the DPB Output Timing of the top-level temporal layer, and second delay information indicating whether or not there is a second delay in the DPB Output Timing of the lower temporal layer relative to the DPB Output Timing of the top-level temporal layer.

[0041] As a result, the decoding device according to one aspect of the present disclosure does not adjust the DPB Output Timing if it obtains information from the bitstream indicating, for example, that there is no first delay and no second delay.

[0042] Furthermore, for example, the top-level temporal layer and the upper-level temporal layer are the same, and the first delay information is not included in the information.

[0043] As a result, in one aspect of the present disclosure, when the top-level temporal layer and the upper temporal layer are the same temporal layer, the decoding device can switch whether or not to adjust the DPB Output Timing simply by obtaining the above information from the bitstream, because the above information includes second delay information indicating whether or not there is a second delay in the lower temporal layer's DPB Output Timing.

[0044] Furthermore, an encoding method according to one aspect of the present disclosure encodes a plurality of pictures in encoding order according to a time scalability defining a plurality of temporal layers, and includes in the bitstream information indicating whether the DPB (Decoded Picture Buffer) Output Timings of a plurality of lower pictures included in the at least one first temporal layer in the first case, where the decoding target is at least one first temporal layer below a lower temporal layer among the plurality of temporal layers, matches the DPB Output Timings of the plurality of lower pictures in the second case, where the decoding target is a plurality of second temporal layers below the upper temporal layer, and provides in the encoding order at least one upper picture included in the upper temporal layer between two adjacent lower pictures among the plurality of lower pictures, the number of the at least one upper picture is n-1, where n is an integer greater than 1, and represents the ratio of the second frame rate in the second case to the first frame rate in the first case.

[0045] As a result, for example, when n, which represents the ratio, is equal to 2, and multiple upper-level pictures and multiple lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may coincide. In this case, an apparatus performing an encoding method according to one aspect of the present disclosure can include information indicating the coincidence in the bitstream to notify the decoding apparatus that there is no need to adjust the DPB Output Timings, whether for the first or second frame rate. Also, for example, even when n, which represents the ratio, is equal to 2, and upper-level pictures and lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may not coincide. For example, if the DPB Output Timings in the first case are delayed more than the DPB Output Timings in the second case in order to adjust the display time of the decoded picture, the DPB Output Timings in the first case and the DPB Output Timings in the second case will not coincide. In this case, an apparatus performing an encoding method according to one aspect of the present disclosure may notify a decoding device that it is necessary to adjust the DPB Output Timings in the first or second case by including information indicating a mismatch in the bitstream. The adjustment of DPB Output Timings is necessary when the decoding device switches the frame rate.

[0046] Furthermore, a decoding method according to one aspect of the present disclosure involves obtaining information from a bitstream indicating whether the DPB (Decoded Picture Buffer) Output Timings of a plurality of lower pictures included in the at least one first temporal layer, in the first case where the decoding target is at least one first temporal layer lower than a higher temporal layer among a plurality of temporal layers, and the DPB Output Timings of the plurality of lower pictures, in the second case where the decoding target is a plurality of second temporal layers lower than the higher temporal layer, match; decoding the plurality of pictures in the decoding order according to the time scalability that defines the plurality of temporal layers based on the information; providing at least one upper picture to be included in the upper temporal layer between two adjacent lower pictures among the plurality of lower pictures in the decoding order, the number of the at least one upper picture being n-1, where n is an integer greater than 1, and represents the ratio of the second frame rate in the second case to the first frame rate in the first case.

[0047] As a result, for example, when n, which represents the ratio, is equal to 2, and multiple upper-level pictures and multiple lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may coincide. In this case, an apparatus performing a decoding method according to one aspect of the present disclosure can obtain information indicating the coincidence from the bitstream and know that it is not necessary to adjust the DPB Output Timings in either the first or second frame rate. Also, for example, even when n, which represents the ratio, is equal to 2, and upper-level pictures and lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may not coincide. For example, if the DPB Output Timings in the first case are delayed more than the DPB Output Timings in the second case in order to adjust the display time of the decoded picture, the DPB Output Timings in the first case and the DPB Output Timings in the second case will not coincide. In this case, an apparatus performing a decoding method according to one aspect of the present disclosure can learn that it is necessary to adjust the DPB Output Timings in either the first or second case by obtaining information from the bitstream indicating a mismatch. Therefore, an apparatus performing a decoding method according to one aspect of the present disclosure can appropriately adjust the DPB Output Timing by obtaining the above information. Note that adjustment of DPB Output Timings is necessary when the decoding device switches the frame rate.

[0048] Furthermore, a non-temporary recording medium according to one aspect of the present disclosure is a non-temporary storage medium for storing a bitstream, wherein the bitstream includes information for causing a decoding device that receives the bitstream to perform a decoding process, and the information is for decoding a plurality of pictures in a decoding order according to a time scalability that defines a plurality of temporal layers, and the information indicates whether the DPB (Decoded Picture Buffer) Output Timings of a plurality of lower pictures included in the at least one first temporal layer, in the first case where the target of decoding is at least one first temporal layer lower than a higher temporal layer, and whether the DPB Output Timings of the plurality of lower pictures match in the second case where the target of decoding is a plurality of second temporal layers lower than the higher temporal layer.

[0049] As a result, for example, when n, which represents the ratio, is equal to 2, and multiple upper-level pictures and multiple lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may coincide. In this case, a non-temporary recording medium according to one aspect of the present disclosure can include information indicating the coincidence in the bitstream to notify the decoding device that there is no need to adjust the DPB Output Timings in either the first or second frame rate. Also, for example, even when n, which represents the ratio, is equal to 2, and upper-level pictures and lower-level pictures are arranged alternately, the DPB Output Timings in the first case and the DPB Output Timings in the second case may not coincide. For example, if the DPB Output Timings in the first case are delayed more than the DPB Output Timings in the second case in order to adjust the display time of the decoded picture, the DPB Output Timings in the first case and the DPB Output Timings in the second case will not coincide. In this case, a non-temporary recording medium according to one aspect of the present disclosure can notify the decoding device that it is necessary to adjust the DPB Output Timings in either the first or second case by including information indicating a mismatch in the bitstream. The adjustment of DPB Output Timings is required when the decoding device switches the frame rate.

[0050] The non-temporary recording medium described above may be provided in the device. For example, a decoding device according to one aspect of this disclosure includes the non-temporary recording medium. A bitstream storage device according to one aspect of this disclosure also includes the non-temporary recording medium. A bitstream transmission device according to one aspect of this disclosure includes the non-temporary recording medium and a transmission unit that transmits a bitstream read from the non-temporary recording medium. Furthermore, one aspect of this disclosure is a bitstream storage method or a bitstream transmission method.

[0051] Furthermore, for example, an encoding device according to one aspect of this disclosure comprises an input unit, a division unit, an intra-prediction unit, an inter-prediction unit, a loop filter unit, a conversion unit, a quantization unit, an entropy encoding unit, and an output unit.

[0052] The current picture is input to the input unit. The division unit divides the current picture into multiple blocks.

[0053] The intra prediction unit generates a prediction signal for the current block contained in the current picture using a reference image contained in the current picture. The inter prediction unit generates a prediction signal for the current block contained in the current picture using a reference image contained in a reference picture different from the current picture. The loop filter unit applies a filter to the reconstructed block of the current block contained in the current picture.

[0054] The conversion unit converts the prediction error between the original signal of the current block included in the current picture and the predicted signal generated by the intra-prediction unit or the inter-prediction unit to generate conversion coefficients. The quantization unit quantizes the conversion coefficients to generate quantization coefficients. The entropy coding unit applies variable-length coding to the quantization coefficients to generate an encoded bitstream. The encoded bitstream, which includes the quantization coefficients to which variable-length coding has been applied and control information, is output from the output unit.

[0055] Furthermore, for example, the entropy encoding unit, in its operation, encodes a plurality of pictures in encoding order according to the time scalability that defines a plurality of temporal layers, and includes in the bitstream information indicating whether the DPB (Decoded Picture Buffer) Output Timings of a plurality of lower pictures included in at least one first temporal layer in the first case, which targets at least one first temporal layer below a lower temporal layer among the plurality of temporal layers, and the DPB Output Timings of a plurality of lower pictures in the second case, which targets a plurality of second temporal layers below a higher temporal layer, match. In the encoding order, at least one upper picture is provided between two adjacent lower pictures among the plurality of lower pictures, and the number of at least one upper picture is n-1, where n is an integer greater than 1, and represents the ratio of the second frame rate in the second case to the first frame rate in the first case.

[0056] Furthermore, for example, a decoding device according to one aspect of the present disclosure comprises an input unit, an entropy decoding unit, an inverse quantization unit, an inverse transform unit, an intra-prediction unit, an inter-prediction unit, a loop filter unit, and an output unit.

[0057] The input unit receives an encoded bitstream. The entropy decoding unit applies variable-length decoding to the encoded bitstream to derive quantization coefficients. The inverse quantization unit dequantizes the quantization coefficients to derive conversion coefficients. The inverse transformation unit inversely transforms the conversion coefficients to derive prediction errors.

[0058] The intra prediction unit generates a prediction signal for the current block contained in the current picture using a reference image contained in the current picture. The inter prediction unit generates a prediction signal for the current block contained in the current picture using a reference image contained in a reference picture different from the current picture.

[0059] The loop filter unit applies a filter to the reconstructed block of the current block included in the current picture. Then, the current picture is output from the output unit.

[0060] Furthermore, for example, the entropy decoding unit, in its operation, obtains information from the bitstream indicating whether the DPB (Decoded Picture Buffer) Output Timings of multiple lower pictures included in at least one first temporal layer in the first case, which targets at least one first temporal layer lower than the upper temporal layer among the multiple temporal layers, and the DPB Output Timings of multiple lower pictures in the second case, which targets multiple second temporal layers lower than the upper temporal layer, match. Based on this information, it decodes the multiple pictures in the decoding order according to the time scalability that defines the multiple temporal layers, and in the decoding order, at least one upper picture to be included in the upper temporal layer is provided between two adjacent lower pictures among the multiple lower pictures, the number of at least one upper picture is n-1, where n is an integer greater than 1, and represents the ratio of the second frame rate in the second case to the first frame rate in the first case.

[0061] Furthermore, these comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0062] [Definition of Terms] The following definitions may be used as examples for each term.

[0063] (1) Image A unit of data composed of a collection of pixels, consisting of pictures and smaller blocks, and includes both still images and videos.

[0064] (2) Picture A processing unit of an image, composed of a collection of pixels, is sometimes called a frame or field.

[0065] (3) Block A set containing a specific number of pixels is a processing unit, and its name is not restricted, as shown in the examples below. Its shape is also not restricted; for example, it includes not only rectangles made of M×N pixels and squares made of M×M pixels, but also triangles, circles, and other shapes.

[0066] (Example of a block) Slice / tile / brick ·CTU / Superblock / Basic division unit • VPDU / Hardware processing partitioning unit ·CU / Processing Block Unit / Prediction Block Unit (PU) / Orthogonal Transform Block Unit (TU) / Unit • Subblock

[0067] (4) Pixels / Samples The smallest unit of an image is a point, which includes not only pixels at integer positions but also pixels at decimal positions that are generated based on pixels at integer positions.

[0068] (5) Pixel value / Sample value These are the unique values ​​that each pixel possesses, including not only luminance values, color difference values, and RGB gradations, but also depth values, or binary values ​​of 0 or 1.

[0069] (6) Flag This includes not only single-bit values ​​but also multi-bit values, such as parameters or indices of two or more bits. Furthermore, it is not limited to binary values, but also includes multi-value values ​​using other number systems.

[0070] (7) Signal This refers to a system of symbols and encoding used to transmit information, and includes not only discretized digital signals but also analog signals that take continuous values.

[0071] (8) Stream / Bitstream A stream / bitstream refers to a sequence of digital data or a flow of digital data. A stream / bitstream may consist of a single stream or multiple streams divided into multiple layers. Furthermore, it includes cases where data is transmitted via serial communication over a single transmission path, as well as cases where data is transmitted via packet communication over multiple transmission paths.

[0072] (9) Difference / difference For scalar quantities, it is sufficient to include not only the simple difference (xy), but also any difference operation, including the absolute value of the difference (|xy|), the squared difference (x^2-y^2), the square root of the difference (√(xy)), the weighted difference (ax-by: a, b is a constant), and the offset difference (x-y+a: a is the offset).

[0073] (10) sum For scalar quantities, it is sufficient to include not only simple sums (x+y) but also summation operations, including the absolute value of the sum (|x+y|), sum of squares (x^2+y^2), square root of the sum (√(x+y)), weighted sums (ax+by: a, b is a constant), and offset sums (x+y+a: a is the offset).

[0074] (11) based on This includes cases where factors other than the underlying subject are taken into consideration. Furthermore, it includes cases where the result is obtained not only through direct calculation but also through intermediate steps.

[0075] (12) using This includes cases where factors other than the target element are taken into consideration. Furthermore, it includes cases where the result is obtained not only through direct calculation but also through intermediate steps.

[0076] (13) prohibit, forbid This can be rephrased as "not permitted." Furthermore, not prohibiting something or allowing it does not necessarily mean it is an obligation.

[0077] (14) To limit (limit, restriction / restrict / restricted) This can be rephrased as "not permitted." Furthermore, not prohibiting something or allowing it does not necessarily mean it is an obligation. Additionally, it may suffice for something to be prohibited quantitatively or qualitatively in part, and it may also include cases of complete prohibition.

[0078] (15) Chroma These are adjectives, represented by the symbols Cb and Cr, that specify that a sample sequence or a single sample represents one of two color difference signals related to a primary color. The term chrominance can also be used instead of chroma.

[0079] (16) Luminance (luma) This is an adjective, represented by a symbol or a subscript Y or L, that specifies that a sample sequence or a single sample represents a monochrome signal related to a primary color. The term luminance can also be used instead of luma.

[0080] [Explanation regarding the description] In drawings, the same reference number indicates the same or similar component. Furthermore, the size and relative position of components in the drawings are not necessarily depicted to a consistent scale.

[0081] The embodiments will be described in detail below with reference to the drawings. Note that the embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement and connection forms of components, steps, relationships and sequences of steps shown in the following embodiments are examples only and are not intended to limit the scope of the claims.

[0082] Embodiments of encoding and decoding devices are described below. These embodiments are examples of encoding and decoding devices to which the processes and / or configurations described in each aspect of this disclosure can be applied. The processes and / or configurations can also be implemented in encoding and decoding devices different from those in the embodiments. For example, with respect to the processes and / or configurations applicable to the embodiments, one of the following may be implemented:

[0083] (1) Any of the multiple components of the encoding or decoding device of the embodiments described in each aspect of the present disclosure may be replaced or combined with other components described in any of the aspects of the present disclosure.

[0084] (2) In the encoding or decoding device of the embodiment, any modifications such as addition, replacement, or deletion of functions or processes performed by some of the multiple components of the encoding or decoding device may be made. For example, any of the functions or processes may be replaced or combined with other functions or processes described in any of the embodiments of this disclosure.

[0085] (3) In the methods performed by the encoding or decoding apparatus of the embodiment, any modifications, such as additions, replacements, and deletions, may be made to some of the processes included in the method. For example, any of the processes in the method may be replaced with or combined with other processes described in any of the embodiments of this disclosure.

[0086] (4) Some of the multiple components constituting the encoding or decoding device of the embodiment may be combined with components described in any of the embodiments of this disclosure, or with components that have some of the functions described in any of the embodiments of this disclosure, or with components that perform some of the processing performed by the components described in any of the embodiments of this disclosure.

[0087] (5) Components that provide some of the functions of the encoding or decoding device of the embodiment, or components that perform some of the processing of the encoding or decoding device of the embodiment, may be combined with or replaced with components described in any of the aspects of the disclosure and components that provide some of the functions described in any of the aspects of the disclosure, or components that perform some of the processing described in any of the aspects of the disclosure.

[0088] (6) In a method performed by an encoding or decoding device of an embodiment, any of the processes included in the method may be replaced or combined with any of the processes described in any of the embodiments of the present disclosure.

[0089] (7) Some of the processes included in the methods performed by the encoding or decoding device of the embodiment may be combined with the processes described in any of the embodiments of this disclosure.

[0090] (8) The methods of carrying out the processes and / or configurations described in each aspect of the present disclosure are not limited to the encoding or decoding devices of the embodiments. For example, the processes and / or configurations may be carried out in devices used for purposes other than the video encoding or video decoding disclosed in the embodiments.

[0091] [System Configuration] Figure 1 is a schematic diagram showing an example of the configuration of the transmission system according to this embodiment.

[0092] The transmission system Trs is a system that transmits a stream generated by encoding an image and decodes the transmitted stream. Such a transmission system Trs includes, for example, an encoding device 100, a network Nw, and a decoding device 200, as shown in Figure 1.

[0093] An image is input to the encoding device 100. The encoding device 100 generates a stream by encoding the input image and outputs the stream to the network Nw. The stream includes, for example, the encoded image and control information for decoding the encoded image. The image is compressed by this encoding process.

[0094] The original image input to the encoding device 100 before encoding is also called the original image, original signal, or original sample. The image may be a moving image or a still image. The image is a higher-level concept than sequences, pictures, and blocks, and is not limited by spatial and temporal domains unless otherwise specified. The image consists of a sequence of pixels or pixel values, and the signal or pixel value representing the image is also called a sample. The stream may be called a bitstream, encoded bitstream, compressed bitstream, or encoded signal. Furthermore, the encoding device may be called an image encoding device or a moving image encoding device, and the encoding method by the encoding device 100 may be called an encoding method, an image encoding method, or a moving image encoding method.

[0095] The network Nw transmits the stream generated by the encoding device 100 to the decoding device 200. The network Nw may be the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network Nw is not necessarily limited to a bidirectional communication network; it may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, the network Nw may be replaced by a storage medium that records the stream, such as a DVD (Digital Versatile Disc) or a BD (Blu-Ray Disc®).

[0096] The decoding device 200 generates a decoded image, which is, for example, an uncompressed image, by decoding the stream transmitted by the network Nw. For example, the decoding device decodes the stream according to a decoding method that corresponds to the encoding method by the encoding device 100.

[0097] The decoding device may also be called an image decoding device or a video decoding device, and the decoding method performed by the decoding device 200 may be called a decoding method, an image decoding method, or a video decoding method.

[0098] [Data structure] Figure 2 shows an example of a data hierarchy in a stream. The stream includes, for example, a video sequence. This video sequence includes, for example, a VPS (Video Parameter Set), an SPS (Sequence Parameter Set), a PPS (Picture Parameter Set), an SEI (Supplemental Enhancement Information), and multiple pictures, as shown in Figure 2(a).

[0099] VPS includes encoding parameters common to multiple layers in a video composed of multiple layers, and encoding parameters related to the multiple layers included in the video, or to individual layers.

[0100] The SPS includes parameters used for the sequence, i.e., encoding parameters that the decoding device 200 references to decode the sequence. For example, these encoding parameters may represent the width or height of the picture. Multiple SPSs may exist.

[0101] The PPS includes parameters used for the picture, i.e., encoding parameters that the decoding device 200 references to decode each picture in the sequence. For example, these encoding parameters may include a reference value for the quantization width used to decode the picture and a flag indicating the application of weighted prediction. Multiple PPSs may exist. The SPS and PPS are sometimes simply referred to as parameter sets.

[0102] The picture may include a picture header and one or more slices, as shown in Figure 2(b). The picture header includes encoding parameters that the decoding device 200 references to decode the one or more slices.

[0103] A slice includes a slice header and one or more bricks, as shown in Figure 2(c). The slice header includes encoding parameters that the decoding device 200 references to decode the one or more bricks.

[0104] A brick contains one or more Coding Tree Units (CTUs), as shown in Figure 2(d).

[0105] Note that a picture may not contain slices, but instead contain tile groups. In this case, a tile group contains one or more tiles. A brick may also contain slices.

[0106] A CTU is also called a superblock or basic division unit. Such a CTU includes a CTU header and one or more CUs (Coding Units), as shown in Figure 2(e). The CTU header contains coding parameters that the decoding device 200 refers to in order to decode one or more CUs.

[0107] A CU may be divided into multiple smaller CUs. Furthermore, as shown in Figure 2(f), a CU includes a CU header, prediction information, and residual coefficient information. The prediction information is for predicting the CU, and the residual coefficient information indicates the predicted residual, which will be described later. While a CU is basically identical to a PU (Prediction Unit) and a TU (Transform Unit), in SBT (Science-Based Testing), for example, it may include multiple TUs smaller than the CU. Additionally, a CU may be processed for each VPDU (Virtual Pipeline Decoding Unit) that constitutes it. A VPDU is a fixed unit that can be processed in one stage when performing pipeline processing in hardware, for example.

[0108] Note that a stream does not necessarily have to have some of the layers shown in Figure 2. Also, the order of these layers may be changed, and any layer may be replaced by another layer. The picture that is currently being processed by a device such as the encoding device 100 or the decoding device 200 is called the current picture. If the processing is encoding, the current picture is synonymous with the picture to be encoded, and if the processing is decoding, the current picture is synonymous with the picture to be decoded. Also, a block, such as CU or CU, that is currently being processed by a device such as the encoding device 100 or the decoding device 200 is called the current block. If the processing is encoding, the current block is synonymous with the block to be encoded, and if the processing is decoding, the current block is synonymous with the block to be decoded.

[0109] [Picture composition: slice / tile] To decode pictures in parallel, they may be composed of slices or tiles.

[0110] A slice is the basic encoding unit that makes up a picture. A picture is composed of, for example, one or more slices. A slice, in turn, consists of one or more consecutive CTUs.

[0111] Figure 3 shows an example of a slice configuration. For example, a picture contains 11 × 8 CTUs and is divided into four slices (slice 1-4). Slice 1 consists of, for example, 16 CTUs, slice 2 consists of, for example, 21 CTUs, slice 3 consists of, for example, 29 CTUs, and slice 4 consists of, for example, 22 CTUs. Here, each CTU in the picture belongs to one of the slices. The shape of the slice is the horizontal division of the picture. The boundaries of the slice do not have to be at the edges of the screen, but can be anywhere among the boundaries of the CTUs within the screen. The processing order (encoding order or decoding order) of the CTUs in a slice is, for example, the raster scan order. A slice also includes a slice header and encoded data. The slice header may describe the characteristics of the slice, such as the CTU address of the beginning of the slice and the slice type.

[0112] A tile is a rectangular area that makes up a picture. Each tile may be assigned a number called a TileId in the order of the raster scan.

[0113] Figure 4 shows an example of a tile configuration. For example, a picture contains 11 × 8 CTUs and is divided into four rectangular tiles (tiles 1-4). When tiles are used, the processing order of CTUs is changed compared to when tiles are not used. When tiles are not used, multiple CTUs in a picture are processed, for example, in raster scan order. When tiles are used, in each of the multiple tiles, at least one CTU is processed, for example, in raster scan order. For example, as shown in Figure 4, the processing order of multiple CTUs contained in tile 1 is from the left end of the first column of tile 1 to the right end of the first column of tile 1, and then from the left end of the second column of tile 1 to the right end of the second column of tile 1.

[0114] Note that one tile may contain one or more slices, and one slice may contain one or more tiles.

[0115] A picture may be composed of tilesets. A tileset may contain one or more tile groups, or one or more tiles. A picture may consist of only one of a tileset, a tile group, or a tile. For example, the order in which multiple tiles are scanned in raster order for each tileset is defined as the basic coding order of the tiles. Within each tileset, a collection of one or more tiles that follow the same basic coding order is defined as a tile group. Such a picture may be composed of the division unit 102 (see Figure 7) described later.

[0116] [Scalable encoding] Figures 5 and 6 show an example of a scalable stream configuration.

[0117] As shown in Figure 5, the encoding device 100 may generate a temporally / spatially scalable stream by encoding each of the multiple pictures into one of the multiple layers. For example, the encoding device 100 achieves scalability by encoding each picture layer by layer, with an enhancement layer existing above the base layer. This type of encoding of each picture is called scalable encoding. This allows the decoding device 200 to switch the image quality displayed by decoding the stream. In other words, the decoding device 200 decides which layers to decode based on internal factors such as its own performance and external factors such as the state of the communication bandwidth. As a result, the decoding device 200 can freely switch between decoding the same content into low-resolution content and high-resolution content. For example, a user of the stream might watch part of the video on their smartphone while on the go, and then watch the rest of the video on a device such as an internet TV after returning home. Each of the aforementioned smartphones and devices incorporates a decoding device 200 with the same or different performance. In this case, if the device decodes up to the upper layers of the stream, the user can view high-definition video after returning home. This eliminates the need for the encoding device 100 to generate multiple streams with the same content but different image quality, thereby reducing the processing load.

[0118] Furthermore, the enhancement layer may include metadata based on image statistics. The decoding device 200 may generate a high-resolution video by super-resolution the picture of the base layer based on the metadata. Super-resolution may be either an improvement in the signal-to-noise ratio (SN) at the same resolution, or an increase in resolution. The metadata may include information for identifying linear or nonlinear filter coefficients used in the super-resolution process, or information for identifying parameter values ​​in the filtering process, machine learning, or least-squares operation used in the super-resolution process.

[0119] Alternatively, the picture may be divided into tiles or the like, depending on the meaning of each object within the picture. In this case, the decoding device 200 may decode only a portion of the picture by selecting the tiles to be decoded. Furthermore, the attributes of the objects (people, cars, balls, etc.) and their positions within the picture (coordinate positions within the same picture, etc.) may be stored as metadata. In this case, the decoding device 200 can identify the position of a desired object based on the metadata and determine the tile containing that object. For example, as shown in Figure 6, the metadata is stored using a data storage structure different from the image data, such as SEI in HEVC. This metadata indicates, for example, the position, size, or color of the main object.

[0120] Furthermore, metadata may be stored in units consisting of multiple pictures, such as streams, sequences, or random access units. This allows the decoding device 200 to obtain information such as the time when a specific person appears in the video, and by using that time and the information in the picture units, it can identify the picture in which the object exists and the position of the object within that picture.

[0121] [Encoding device] Next, the encoding device 100 according to the embodiment will be described. Figure 7 is a block diagram showing an example of the configuration of the encoding device 100 according to the embodiment. The encoding device 100 encodes images in block units.

[0122] As shown in Figure 7, the encoding device 100 is a device that encodes an image in block units and comprises a division unit 102, a subtraction unit 104, a transformation unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transformation unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. The intra prediction unit 124 and the inter prediction unit 126 are each configured as part of the prediction processing unit.

[0123] [Example of an encoding device implementation] Figure 8 is a block diagram showing an example implementation of the encoding device 100. The encoding device 100 includes a processor a1 and memory a2. For example, the multiple components of the encoding device 100 shown in Figure 7 are implemented by the processor a1 and memory a2 shown in Figure 8.

[0124] Processor a1 is a circuit that performs information processing and is a circuit that can access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit for encoding images. Processor a1 may be a processor such as a CPU. Alternatively, processor a1 may be a collection of multiple electronic circuits. Furthermore, for example, processor a1 may play the role of multiple components of the encoding device 100 shown in Figure 7, excluding the component for storing information.

[0125] Memory a2 is a dedicated or general-purpose memory in which information for the processor a1 to encode an image is stored. Memory a2 may be an electronic circuit and may be connected to the processor a1. Memory a2 may also be included in the processor a1. Memory a2 may also be a collection of multiple electronic circuits. Memory a2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory a2 may also be a non-volatile memory or a volatile memory.

[0126] For example, memory a2 may store the image to be encoded, or it may store a stream corresponding to the encoded image. Alternatively, memory a2 may store a program for processor a1 to encode the image.

[0127] Furthermore, for example, memory a2 may play the role of an information storage component among the multiple components of the encoding device 100 shown in Figure 7. Specifically, memory a2 may play the role of the block memory 118 and frame memory 122 shown in Figure 7. More specifically, memory a2 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0128] Furthermore, the encoding device 100 does not necessarily have to implement all of the components shown in Figure 7, nor does it have to perform all of the processes described above. Some of the components shown in Figure 7 may be included in other devices, and some of the processes described above may be performed by other devices.

[0129] The following describes the overall processing flow of the encoding device 100, followed by a description of each component included in the encoding device 100.

[0130] [Overall flow of the encoding process] Figure 9 is a flowchart showing an example of the overall encoding process performed by the encoding device 100.

[0131] First, the division unit 102 of the encoding device 100 divides the picture contained in the original image into multiple fixed-size blocks (128 x 128 pixels) (step Sa_1). Then, the division unit 102 selects a division pattern for each of these fixed-size blocks (step Sa_2). In other words, the division unit 102 further divides the fixed-size blocks into multiple blocks that constitute the selected division pattern. Then, the encoding device 100 performs steps Sa_3 to Sa_9 for each of these multiple blocks.

[0132] The prediction processing unit, consisting of the intra-prediction unit 124 and the inter-prediction unit 126, and the prediction control unit 128 generate a predicted image of the current block (step Sa_3). The predicted image is also called a predicted signal, predicted block, or predicted sample.

[0133] Next, the subtraction unit 104 generates the difference between the current block and the predicted image as the predicted residual (step Sa_4). The predicted residual is also called the prediction error.

[0134] Next, the transformation unit 106 and the quantization unit 108 generate multiple quantization coefficients by performing transformation and quantization on the predicted image (step Sa_5).

[0135] Next, the entropy coding unit 110 generates a stream by coding (specifically, entropy coding) its multiple quantization coefficients and prediction parameters related to the generation of the predicted image (step Sa_6).

[0136] Next, the inverse quantization unit 112 and the inverse transformation unit 114 restore the predicted residuals by performing inverse quantization and inverse transformation on a plurality of quantization coefficients (step Sa_7).

[0137] Next, the summing unit 116 reconstructs the current block by adding the predicted image to the recovered predicted residual (step Sa_8). This generates a reconstructed image. The reconstructed image is also called a reconstructed block, and the reconstructed image generated by the encoding device 100 is also called a local decoded block or local decoded image.

[0138] Once this reconstructed image is generated, the loop filter unit 120 performs filtering on the reconstructed image as needed (step Sa_9).

[0139] Then, the encoding device 100 determines whether or not the encoding of the entire picture is complete (step Sa_10). If it determines that it is not complete (No. in step Sa_10), it repeats the process from step Sa_2.

[0140] In the example described above, the encoding device 100 selects one partitioning pattern for a fixed-size block and encodes each block according to that partitioning pattern. However, it may also encode each block according to multiple partitioning patterns. In this case, the encoding device 100 may evaluate the cost of each of the multiple partitioning patterns and, for example, select the stream obtained by encoding according to the partitioning pattern with the smallest cost as the final output stream.

[0141] Furthermore, the processes in steps Sa_1 to Sa_10 may be performed sequentially by the encoding device 100, some of these processes may be performed in parallel, and the order may be changed.

[0142] The encoding process performed by such an encoding device 100 is a hybrid encoding using predictive encoding and transformative encoding. Furthermore, predictive encoding is performed by an encoding loop consisting of a subtraction unit 104, a transformer unit 106, a quantization unit 108, an inverse quantization unit 112, an inverse transformer unit 114, an addition unit 116, a loop filter unit 120, a block memory 118, a frame memory 122, an intra-prediction unit 124, an inter-prediction unit 126, and a prediction control unit 128. In other words, the prediction processing unit consisting of the intra-prediction unit 124 and the inter-prediction unit 126 constitutes a part of the encoding loop.

[0143] [Divided part] The splitting unit 102 divides each picture contained in the original image into multiple blocks and outputs each block to the subtraction unit 104. For example, the splitting unit 102 first divides the picture into blocks of a fixed size (e.g., 128x128 pixels). These fixed-size blocks are sometimes called coding tree units (CTUs). Then, the splitting unit 102 divides each of the fixed-size blocks into blocks of a variable size (e.g., 64x64 pixels or less) based on, for example, a recursive quadtree and / or binary tree block partitioning. In other words, the splitting unit 102 selects a partitioning pattern. These variable-size blocks are sometimes called coding units (CUs), prediction units (PUs), or transformation units (TUs). In various implementation examples, CUs, PUs, and TUs do not need to be distinguished, and some or all of the blocks in the picture may be processing units of CUs, PUs, or TUs.

[0144] Figure 10 shows an example of block partitioning in the embodiment. In Figure 10, solid lines represent block boundaries due to quadtree block partitioning, and dashed lines represent block boundaries due to binary tree block partitioning.

[0145] Here, block 10 is a 128x128 pixel square block. This block 10 is first divided into four 64x64 pixel square blocks (quadtree block partitioning).

[0146] The top-left 64x64 pixel square block is further divided vertically into two rectangular blocks, each consisting of 32x64 pixels, and the left 32x64 pixel rectangular block is further divided vertically into two rectangular blocks, each consisting of 16x64 pixels (binary tree block partitioning). As a result, the top-left 64x64 pixel square block is divided into two 16x64 pixel rectangular blocks 11 and 12 and a 32x64 pixel rectangular block 13.

[0147] The 64x64 pixel square block in the upper right is horizontally divided into two rectangular blocks 14 and 15, each consisting of 64x32 pixels (binary tree block partitioning).

[0148] The 64x64 pixel square block in the bottom left is divided into four 32x32 pixel square blocks (quadrutree block partitioning). Of these four 32x32 pixel square blocks, the top left and bottom right blocks are further divided. The top left 32x32 pixel square block is vertically divided into two 16x32 pixel rectangular blocks, and the right 16x32 pixel rectangular block is further horizontally divided into two 16x16 pixel square blocks (binary tree block partitioning). The bottom right 32x32 pixel square block is horizontally divided into two 32x16 pixel rectangular blocks (binary tree block partitioning). As a result, the 64x64 pixel square block in the lower left is divided into a 16x32 pixel rectangular block 16, two 16x16 pixel square blocks 17 and 18, two 32x32 pixel square blocks 19 and 20, and two 32x16 pixel rectangular blocks 21 and 22.

[0149] Block 23, consisting of 64x64 pixels in the lower right corner, will not be divided.

[0150] As described above, in Figure 10, block 10 is divided into 13 variable-sized blocks 11-23 based on a recursive quad-tree and binary tree block partition. Such a partition is sometimes called a QTBT (quad-tree plus binary tree) partition.

[0151] In Figure 10, one block was divided into four or two blocks (quadrutree or binary tree block partitioning), but partitioning is not limited to these. For example, one block may be divided into three blocks (ternary tree block partitioning). Partitioning that includes such ternary tree block partitioning is sometimes called MBT (multi-type tree) partitioning.

[0152] FIG. 11 is a diagram showing an example of the configuration of the division unit 102. As shown in FIG. 11, the division unit 102 may include a block division determination unit 102a. The block division determination unit 102a may perform the following processing as an example.

[0153] The block division determination unit 102a collects block information from, for example, the block memory 118 or the frame memory 122, and determines the above-described division pattern based on the block information. The division unit 102 divides the original image according to the division pattern, and outputs one or more blocks obtained by the division to the subtraction unit 104.

[0154] In addition, the block division determination unit 102a outputs, for example, parameters indicating the above-described division pattern to the conversion unit 106, the inverse conversion unit 114, the intra prediction unit 124, the inter prediction unit 126, and the entropy encoding unit 110. The conversion unit 106 may convert the prediction residual based on the parameters, and the intra prediction unit 124 and the inter prediction unit 126 may generate a prediction image based on the parameters. Further, the entropy encoding unit 110 may perform entropy encoding on the parameters.

[0155] The parameters regarding the division pattern may be written into the stream as follows as an example.

[0156] FIG. 12 is a diagram showing an example of a division pattern. The division pattern includes, for example, a four-way division (QT) that divides a block into two in each of the horizontal and vertical directions, a three-way division (HT or VT) that divides a block in the same direction at a ratio of 1:2:1, a two-way division (HB or VB) that divides a block in the same direction at a ratio of 1:1, and no division (NS).

[0157] Note that in the case of four-way division and no division, the division pattern does not have a block division direction, and in the case of two-way division and three-way division, the division pattern has division direction information.

[0158] Figures 13A and 13B are diagrams showing an example of a syntax tree of a splitting pattern. In the example of Figure 13A, first, information (S: Split flag) indicating whether to perform splitting exists. Next, information (QT: QT flag) indicating whether to perform four-way splitting exists. Next, information (TT: TT flag or BT: BT flag) indicating whether to perform three-way splitting or two-way splitting exists, and finally, information (Ver: Vertical flag or Hor: Horizontal flag) indicating the splitting direction exists. Note that for each of one or more blocks obtained by splitting according to such a splitting pattern, splitting may be repeatedly applied in the same manner. That is, as an example, the determination of whether to perform splitting, whether to perform four-way splitting, whether the splitting method is horizontal or vertical, and whether to perform three-way splitting or two-way splitting may be recursively performed, and the obtained determination results may be encoded into a stream according to the encoding order disclosed in the syntax tree shown in Figure 13A.

[0159] Also, in the syntax tree shown in Figure 13A, the information is arranged in the order of S, QT, TT, Ver, but they may be arranged in the order of S, QT, Ver, BT. That is, in the example of Figure 13B, first, information (S: Split flag) indicating whether to perform splitting exists. Next, information (QT: QT flag) indicating whether to perform four-way splitting exists. Next, information (Ver: Vertical flag or Hor: Horizontal flag) indicating the splitting direction exists, and finally, information (BT: BT flag or TT: TT flag) indicating whether to perform two-way splitting or three-way splitting exists.

[0160] Note that the splitting pattern described here is an example, and other patterns than the described splitting pattern may be used, or only a part of the described splitting pattern may be used.

[0161] [Subtraction unit] The subtraction unit 104 subtracts the predicted image (the predicted image input from the prediction control unit 128) from the original image in block units that are input from the division unit 102 and divided by the division unit 102. In other words, the subtraction unit 104 calculates the predicted residual of the current block. The subtraction unit 104 then outputs the calculated predicted residual to the conversion unit 106.

[0162] The original image is the input signal to the encoding device 100, and is, for example, a signal representing the image of each picture that makes up the video (e.g., a luminance (luma) signal and two chroma (chroma) signals).

[0163] [Conversion section] The conversion unit 106 converts the predicted residual in the spatial domain into conversion coefficients in the frequency domain and outputs the conversion coefficients to the quantization unit 108. Specifically, the conversion unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the predicted residual in the spatial domain, for example.

[0164] The transformation unit 106 may adaptively select a transformation type from among several transformation types and use a transformation basis function corresponding to the selected transformation type to convert the predicted residuals into transformation coefficients. Such a transformation is sometimes called an EMT (explicit multiple core transform) or an AMT (adaptive multiple transform). The transformation basis function is sometimes simply called a basis.

[0165] The multiple transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. These transformation types may also be denoted as DCT2, DCT5, DCT8, DST1, and DST7, respectively. Figure 14 is a table showing the transformation basis functions corresponding to each transformation type. In Figure 14, N represents the number of input pixels. The selection of a transformation type from among these multiple transformation types may depend, for example, on the type of prediction (intra-prediction and inter-prediction, etc.) or on the intra-prediction mode.

[0166] Information indicating whether or not to apply such EMT or AMT (e.g., called an EMT flag or AMT flag) and information indicating the selected conversion type are typically signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (e.g., sequence level, picture level, slice level, brick level, or CTU level).

[0167] Furthermore, the transformation unit 106 may retransform the transformation coefficients (i.e., the transformation result). Such retransformation is sometimes called AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transformation unit 106 performs retransformation for each subblock (e.g., a 4x4 pixel subblock) contained in the block of transformation coefficients corresponding to the intra-predicted residual. Information indicating whether or not to apply NSST and information regarding the transformation matrix used for NSST are usually signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (e.g., sequence level, picture level, slice level, brick level, or CTU level).

[0168] The transformation unit 106 may be subjected to either a separable transformation or a non-separable transformation. A separable transformation is a method in which the input is separated into directions equal to the number of dimensions and transformed multiple times, while a non-separable transformation is a method in which, when the input is multidimensional, two or more dimensions are treated as one dimension and transformed together.

[0169] For example, one example of a non-separable transformation is to treat a 4x4 pixel block as a single array with 16 elements and then perform a transformation on that array using a 16x16 transformation matrix.

[0170] Another example of a non-separable transformation is a Hypercube Givens Transform, which treats a 4x4 pixel input block as a single array with 16 elements and then performs multiple Givens rotations on that array.

[0171] In the conversion unit 106, the conversion type of the conversion basis function used to convert to the frequency domain can be switched depending on the region within the CU. One example is SVT (Spatially Varying Transform).

[0172] Figure 15 shows an example of an SVT.

[0173] In SVT, as shown in Figure 15, the CU is divided into two equal parts horizontally or vertically, and only one of the regions is converted to the frequency domain. The conversion type may be set for each region, for example, DST7 and DCT8 can be used. For example, of the two regions obtained by dividing the CU vertically, DST7 and DCT8 may be used for the region at position 0. Or, of those two regions, DST7 is used for the region at position 1. Similarly, of the two regions obtained by dividing the CU horizontally, DST7 and DCT8 are used for the region at position 0. Or, of those two regions, DST7 is used for the region at position 1. In this example shown in Figure 15, only one of the two regions in the CU is converted, and the other is not, but it is also possible to convert both regions. Furthermore, the division method may be not only bisecting but also quartet-dividing. It is also possible to make it more flexible by encoding information indicating the division method and signaling it in the same way as CU division. Note that SVT is sometimes also called SBT (Sub-block Transform).

[0174] The aforementioned AMT and EMT may also be called MTS (Multiple Transform Selection). When applying MTS, a transformation type such as DST7 or DCT8 can be selected, and information indicating the selected transformation type may be encoded as index information for each CU. On the other hand, there is a process called IMTS (Implicit MTS) that selects the transformation type to be used for orthogonal transformation based on the shape of the CU without encoding index information. When applying IMTS, for example, if the shape of the CU is rectangular, the shorter side of the rectangle is orthogonally transformed using DST7, and the longer side is orthogonally transformed using DCT2. Also, for example, if the shape of the CU is square, if MTS is enabled in the sequence, DCT2 is used, and if MTS is disabled, DST7 is used to perform the orthogonal transformation. DCT2 and DST7 are just examples, and other transformation types may be used, and different combinations of transformation types can be used. IMTS may be usable only in intra-prediction blocks, or it may be usable in both intra-prediction blocks and inter-prediction blocks.

[0175] The above describes three selection processes, MTS, SBT, and IMTS, which selectively switch the transformation type used for orthogonal transformations. All three selection processes may be enabled, or only some of them may be selectively enabled. Whether or not an individual selection process is enabled can be identified by flag information in the header, such as SPS. For example, if all three selection processes are enabled, one of the three selection processes is selected for each CU to perform the orthogonal transformation. Note that the selection process for selectively switching the transformation type may be different from the three selection processes described above, as long as at least one of the following four functions [1] to [4] can be implemented, and each of the three selection processes may be replaced with another process. Function [1] is a function that orthogonally transforms the entire range within the CU and encodes information indicating the transformation type used for the transformation. Function [2] is a function that orthogonally transforms the entire range within the CU and determines the transformation type based on predetermined rules without encoding information indicating the transformation type. Function [3] is a function that orthogonally transforms a portion of the CU and encodes information indicating the transformation type used for the transformation. Functions [4] include the ability to orthogonally transform a portion of the CU and determine the transformation type based on predetermined rules without encoding information indicating the transformation type used.

[0176] Furthermore, the application of MTS, IMTS, and SBT may be determined on a per-processing-unit basis. For example, the application may be determined on a per-sequence, per-picture, per-brick, per-slice, per-CTU, or per-CU basis.

[0177] Furthermore, the tool for selectively switching the conversion type in this disclosure may be rephrased as a method for adaptively selecting a base to be used in the conversion process, a selection process, or a process for selecting a base. Alternatively, the tool for selectively switching the conversion type may be rephrased as a mode for adaptively selecting the conversion type.

[0178] Figure 16 is a flowchart showing an example of processing by the conversion unit 106.

[0179] For example, the conversion unit 106 determines whether to perform an orthogonal conversion (step St_1). Here, when the conversion unit 106 determines to perform an orthogonal conversion (Yes in step St_1), it selects a conversion type to be used for the orthogonal conversion from a plurality of conversion types (step St_2). Next, the conversion unit 106 performs an orthogonal conversion by applying the selected conversion type to the prediction residual of the current block (step St_3). Then, the conversion unit 106 causes the entropy encoding unit 110 to encode the information indicating the selected conversion type by outputting the information to the entropy encoding unit 110 (step St_4). On the other hand, when the conversion unit 106 determines not to perform an orthogonal conversion (No in step St_1), it causes the entropy encoding unit 110 to encode the information indicating that no orthogonal conversion is performed by outputting the information to the entropy encoding unit 110 (step St_5). Note that the determination of whether to perform an orthogonal conversion in step St_1 may be made based on, for example, the size of the conversion block, the prediction mode applied to the CU, and the like. Also, the information indicating the conversion type to be used for the orthogonal conversion may not be encoded, and the orthogonal conversion may be performed using a predefined conversion type. <XXX

[0180] FIG. 17 is a flowchart showing another example of the processing by the conversion unit 106. Note that the example shown in FIG. 17 is an example of orthogonal conversion when the method of selectively switching the conversion type to be used for orthogonal conversion is applied, similar to the example shown in FIG. 16.

[0181] As an example, the first conversion type group may include DCT2, DST7, and DCT8. Also, as an example, the second conversion type group may include DCT2. Further, the conversion types included in the first conversion type group and the second conversion type group may partially overlap or may be all different conversion types.

[0182] Please note that there seems to be an error in the original text for the tag where it says "<XXX

[0180] " which might be a formatting or input error. I've translated it as "<XXX

[0180] " as it is, but it should be checked for accuracy.Specifically, the transformation unit 106 determines whether the transformation size is less than or equal to a predetermined value (step Su_1). If it determines that it is less than or equal to the predetermined value (Yes in step Su_1), the transformation unit 106 orthogonally transforms the predicted residual of the current block using the transformation types included in the first transformation type group (step Su_2). Next, the transformation unit 106 outputs information to the entropy encoding unit 110 indicating which of the one or more transformation types included in the first transformation type group will be used, thereby encoding that information (step Su_3). On the other hand, if the transformation unit 106 determines that the transformation size is not less than or equal to a predetermined value (No in step Su_1), it orthogonally transforms the predicted residual of the current block using the second transformation type group (step Su_4).

[0183] In step Su_3, the information indicating the transformation type used for the orthogonal transformation may be information indicating a combination of transformation types applied vertically and horizontally to the current block. Also, the first group of transformation types may contain only one transformation type, and the information indicating the transformation type used for the orthogonal transformation does not need to be encoded. The second group of transformation types may contain multiple transformation types, and the information indicating the transformation type used for the orthogonal transformation from one or more transformation types included in the second group of transformation types may be encoded.

[0184] Furthermore, the conversion type may be determined solely based on the conversion size. Note that if the process determines the conversion type used for orthogonal transformations based on the conversion size, it is not limited to determining whether the conversion size is less than or equal to a predetermined value.

[0185] [Quantization section] The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization unit 108 scans a plurality of conversion coefficients in the current block in a predetermined scanning order and quantizes the conversion coefficients based on the quantization parameter (QP) corresponding to the scanned conversion coefficients. The quantization unit 108 then outputs the plurality of quantized conversion coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy coding unit 110 and the inverse quantization unit 112.

[0186] A predetermined scan order is the order for quantization / inverse quantization of the transformation coefficients. For example, a predetermined scan order is defined as ascending frequency (from low to high frequency) or descending frequency (from high to low frequency).

[0187] The quantization parameter (QP) is a parameter that defines the quantization step (quantization width). For example, if the value of the quantization parameter increases, the quantization step also increases. In other words, if the value of the quantization parameter increases, the error in the quantization coefficient (quantization error) increases.

[0188] Furthermore, quantization matrices may be used for quantization. For example, several types of quantization matrices may be used corresponding to frequency conversion sizes such as 4x4 and 8x8, prediction modes such as intra-prediction and inter-prediction, and pixel components such as luminance and chrominance. Quantization refers to the process of digitizing values ​​sampled at predetermined intervals and associating them with predetermined levels. In this field, terms such as rounding, scaling, or scaling may also be used.

[0189] There are two methods for using the quantization matrix: using a quantization matrix directly set on the encoding device 100, and using a default quantization matrix (default matrix). On the encoding device 100, by directly setting the quantization matrix, a quantization matrix corresponding to the image features can be set. However, in this case, there is a disadvantage that the amount of code increases due to the encoding of the quantization matrix. Alternatively, instead of using the default quantization matrix or the encoded quantization matrix as is, a quantization matrix to be used for quantizing the current block may be generated based on the default quantization matrix or the encoded quantization matrix.

[0190] On the other hand, there is also a method that does not use a quantization matrix, and quantizes both the high-frequency and low-frequency components in the same way. This method is equivalent to using a quantization matrix where all coefficients have the same value (a flat matrix).

[0191] The quantization matrix may be encoded, for example, at the sequence level, picture level, slice level, brick level, or CTU level.

[0192] When a quantization matrix is ​​used, the quantization unit 108 scales the quantization width, which is obtained from quantization parameters, for each conversion coefficient using the values ​​of the quantization matrix. A quantization process performed without a quantization matrix may be a process that quantizes the conversion coefficients based on the quantization width obtained from quantization parameters. In a quantization process performed without a quantization matrix, the quantization width may be multiplied by a predetermined value common to all conversion coefficients within a block.

[0193] Figure 18 is a block diagram showing an example of the configuration of the quantization unit 108.

[0194] The quantization unit 108 includes, for example, a differential quantization parameter generation unit 108a, a predictive quantization parameter generation unit 108b, a quantization parameter generation unit 108c, a quantization parameter storage unit 108d, and a quantization processing unit 108e.

[0195] Figure 19 is a flowchart showing an example of quantization by the quantization unit 108.

[0196] For example, the quantization unit 108 may perform quantization for each CU based on the flowchart shown in Figure 19. Specifically, the quantization parameter generation unit 108c determines whether or not to perform quantization (step Sv_1). If it is determined that quantization should be performed (Yes in step Sv_1), the quantization parameter generation unit 108c generates quantization parameters for the current block (step Sv_2) and stores these quantization parameters in the quantization parameter storage unit 108d (step Sv_3).

[0197] Next, the quantization processing unit 108e quantizes the conversion coefficients of the current block using the quantization parameters generated in step Sv_2 (step Sv_4). Then, the prediction quantization parameter generation unit 108b obtains quantization parameters for a processing unit different from the current block from the quantization parameter storage unit 108d (step Sv_5). Based on the obtained quantization parameters, the prediction quantization parameter generation unit 108b generates prediction quantization parameters for the current block (step Sv_6). The difference quantization parameter generation unit 108a calculates the difference between the quantization parameters of the current block generated by the quantization parameter generation unit 108c and the prediction quantization parameters of the current block generated by the prediction quantization parameter generation unit 108b (step Sv_7). The difference quantization parameters are generated by calculating this difference. The differential quantization parameter generation unit 108a outputs the differential quantization parameter to the entropy encoding unit 110, thereby encoding the differential quantization parameter (step Sv_8).

[0198] The differential quantization parameters may be encoded at the sequence level, picture level, slice level, brick level, or CTU level. The initial values ​​of the quantization parameters may also be encoded at the sequence level, picture level, slice level, brick level, or CTU level. In this case, the quantization parameters may be generated using the initial values ​​of the quantization parameters and the differential quantization parameters.

[0199] The quantization unit 108 may be equipped with multiple quantizers, and may also apply dependent quantization, which quantizes the conversion coefficients using a quantization method selected from multiple quantization methods.

[0200] [Entropy coding unit] Figure 20 is a block diagram showing an example of the configuration of the entropy coding unit 110.

[0201] The entropy coding unit 110 generates a stream by performing entropy coding on the quantization coefficients input from the quantization unit 108 and the prediction parameters input from the prediction parameter generation unit 130. For example, CABAC (Context-based Adaptive Binary Arithmetic Coding) is used for this entropy coding. Specifically, the entropy coding unit 110 comprises, for example, a binarization unit 110a, a context control unit 110b, and a binary arithmetic coding unit 110c. The binarization unit 110a performs binarization, converting multi-level signals such as quantization coefficients and prediction parameters into binary signals. Examples of binarization methods include Truncated Rice Binarization, Exponential Golomb codes, and Fixed Length Binarization. The context control unit 110b derives context values, i.e., the probability of a binary signal occurring, according to the characteristics of the syntax elements or the surrounding circumstances. Methods for deriving this context value include, for example, bypass, syntax element referencing, upper / left adjacent block referencing, hierarchical information referencing, and others. The binary arithmetic coding unit 110c performs arithmetic coding on the binarized signal using the derived context value.

[0202] Figure 21 shows the CABAC flow in the entropy coding unit 110.

[0203] First, the CABAC in the entropy coding unit 110 is initialized. This initialization involves initializing the binary arithmetic coding unit 110c and setting an initial context value. Then, the binarization unit 110a and the binary arithmetic coding unit 110c sequentially perform binarization and arithmetic coding for each of the multiple quantization coefficients of the CTU, for example. At this time, the context control unit 110b updates the context value each time arithmetic coding is performed. Then, as a post-processing step, the context control unit 110b saves the context value. This saved context value is used, for example, as the initial value of the context value for the next CTU.

[0204] [Dequantization section] The inverse quantization unit 112 inversely quantizes the quantization coefficients input from the quantization unit 108. Specifically, the inverse quantization unit 112 inversely quantizes the quantization coefficients of the current block in a predetermined scanning order. Then, the inverse quantization unit 112 outputs the inversely quantized conversion coefficients of the current block to the inverse conversion unit 114.

[0205] [Inverse Transformation Section] The inverse transform unit 114 restores the predicted residual by performing an inverse transform on the transformation coefficients input from the inverse quantization unit 112. Specifically, the inverse transform unit 114 restores the predicted residual of the current block by performing an inverse transform on the transformation coefficients corresponding to the transformation by the transformation unit 106. The inverse transform unit 114 then outputs the restored predicted residual to the summation unit 116.

[0206] Furthermore, the recovered prediction residuals usually do not match the prediction error calculated by the subtraction unit 104 because information is typically lost due to quantization. In other words, the recovered prediction residuals usually contain quantization errors.

[0207] [Addition section] The adder 116 reconstructs the current block by adding the predicted residual input from the inverse transformer 114 and the predicted image input from the prediction control unit 128. As a result, a reconstructed image is generated. The adder 116 then outputs the reconstructed image to the block memory 118 and the loop filter unit 120.

[0208] [Block memory] The block memory 118 is a storage unit for storing blocks within the current picture, such as blocks referenced in intra prediction. Specifically, the block memory 118 stores the reconstructed image output from the adder 116.

[0209] [Frame memory] The frame memory 122 is a storage unit for storing reference pictures used, for example, in interpretation, and is sometimes called a frame buffer. Specifically, the frame memory 122 stores the reconstructed image filtered by the loop filter unit 120.

[0210] [Loop Filter Section] The loop filter unit 120 applies loop filtering to the reconstructed image output from the summing unit 116 and outputs the filtered reconstructed image to the frame memory 122. A loop filter is a filter used within the encoding loop (in-loop filter), and includes, for example, adaptive loop filters (ALF), deblocking filters (DF or DBF), and sample adaptive offsets (SAO).

[0211] Figure 22 is a block diagram showing an example of the configuration of the loop filter section 120.

[0212] The loop filter unit 120 includes, for example, a deblocking filter processing unit 120a, an SAO processing unit 120b, and an ALF processing unit 120c, as shown in Figure 22. The deblocking filter processing unit 120a applies the above-described deblocking filter processing to the reconstructed image. The SAO processing unit 120b applies the above-described SAO processing to the reconstructed image after the deblocking filter processing. The ALF processing unit 120c applies the above-described ALF processing to the reconstructed image after the SAO processing. Details of ALF and the deblocking filter will be described later. SAO processing is a process that improves image quality by reducing ringing (a phenomenon in which pixel values ​​are distorted in a wave-like manner around edges) and correcting the shift in pixel values. Examples of SAO processing include edge offset processing and band offset processing. Note that the loop filter unit 120 does not have to include all of the processing units disclosed in Figure 22, and may include only some of the processing units. Furthermore, the loop filter unit 120 may be configured to perform the above-mentioned processing in an order different from the processing order disclosed in Figure 22.

[0213] [Loop Filter Section > Adaptive Loop Filter] In ALF, a least-squares error filter is applied to remove coding distortion. For example, for each 2x2 pixel subblock within the current block, one filter selected from several filters is applied based on the direction and activity of the local gradient.

[0214] Specifically, first, subblocks (for example, 2x2 pixel subblocks) are classified into multiple classes (for example, 15 or 25 classes). The classification of subblocks is done, for example, based on the direction of the gradient and the activity level. In a specific example, a classification value C (for example, C = 5D + A) is calculated using the gradient direction value D (for example, 0 to 2 or 0 to 4) and the gradient activity value A (for example, 0 to 4). Then, based on the classification value C, the subblocks are classified into multiple classes.

[0215] The gradient direction value D is derived, for example, by comparing gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). The gradient activation value A is derived, for example, by adding the gradients in multiple directions and quantizing the sum.

[0216] Based on the results of this classification, a filter for the subblock is determined from among multiple filters.

[0217] For example, a circularly symmetric shape is used for filters in ALF. Figures 23A to 23C show several examples of filter shapes used in ALF. Figure 23A shows a 5x5 diamond-shaped filter, Figure 23B shows a 7x7 diamond-shaped filter, and Figure 23C shows a 9x9 diamond-shaped filter. Information indicating the filter shape is usually signaled at the picture level. However, the signaling of information indicating the filter shape is not limited to the picture level and may be at other levels (e.g., sequence level, slice level, brick level, CTU level, or CU level).

[0218] The on / off status of ALF may be determined, for example, at the picture level or CU level. For example, the decision to apply ALF may be made at the CU level for luminance, and at the picture level for color difference. Information indicating whether ALF is on or off is usually signaled at the picture level or CU level. However, the signaling of information indicating whether ALF is on or off is not limited to the picture level or CU level, and may be at other levels (e.g., sequence level, slice level, brick level, or CTU level).

[0219] Furthermore, as described above, one filter is selected from among several filters and the ALF process is applied to the subblock. For each of these filters (for example, up to 15 or 25 filters), the set of coefficients used in that filter is usually signaled at the picture level. However, the signaling of the coefficient set is not limited to the picture level; it may be at other levels (for example, sequence level, slice level, brick level, CTU level, CU level, or subblock level).

[0220] [Loop Filters > Cross Component Adaptive Loop Filter] Figure 23D shows an example where the Y sample (first component) is used in Cb CCALF and Cr CCALF (multiple components different from the first component). Figure 23E shows a diamond-shaped filter.

[0221] One example of CC-ALF operates by applying a linear diamond-shaped filter (Figures 23D, 23E) to the luminance channel of each chrominance component. For example, the filter coefficients are transmitted in APS, scaled by a factor of 2^10, and rounded for fixed-point representation. Filter application is controlled by a variable block size and notified by context-encoded flags received for each block of samples. The block size and CC-ALF enable flags are received at the slice level of each chrominance component. The syntax and semantics of CC-ALF are provided in the Appendix. The contributions support block sizes of 16x16, 32x32, 64x64, and 128x128 (for chrominance samples).

[0222] [Loop Filters > Joint Chroma Cross Component Adaptive Loop Filter] Figure 23F shows an example of JC-CCALF. Figure 23G shows an example of a candidate weight_index for JC-CCALF.

[0223] One example of JC-CCALF uses only one CCALF filter to generate a single CCALF filter output as a color difference adjustment signal for only one color component, and then applies appropriately weighted versions of the same color difference adjustment signal to the other color components. In this way, the complexity of existing CCALF is roughly halved.

[0224] The weight values ​​are encoded into a sign flag and a weight index. The weight index (indicated as weight_index) is encoded in 3 bits and specifies the magnitude of the JC-CCALF weight JcCcWeight. It cannot be equal to 0. The magnitude of JcCcWeight is determined as follows:

[0225] • If weight_index is 4 or less, JcCcWeight is equal to weight_index>>2.

[0226] Otherwise, JcCcWeight is equal to 4 / (weight_index-4).

[0227] The block-level on / off control for ALF filtering of Cb and Cr is separate. This is the same as CCALF, where two separate sets of block-level on / off control flags are encoded. Here, unlike CCALF, the on / off control block sizes of Cb and Cr are the same, so only one block size variable is encoded.

[0228] [Loop Filter Section > Deblocking Filter] In the deblocking filtering process, the loop filter unit 120 reduces distortion occurring at the block boundaries of the reconstructed image by applying a filter to those block boundaries.

[0229] Figure 24 is a block diagram showing an example of a detailed configuration of the deblocking filter processing unit 120a.

[0230] The deblocking filter processing unit 120a includes, for example, a boundary determination unit 1201, a filter determination unit 1203, a filter processing unit 1205, a processing determination unit 1208, a filter characteristic determination unit 1207, and switches 1202, 1204, and 1206.

[0231] The boundary determination unit 1201 determines whether or not a pixel to be deblocked and filtered (i.e., a target pixel) is located near a block boundary. The boundary determination unit 1201 then outputs the determination result to the switch 1202 and the processing determination unit 1208.

[0232] If the boundary determination unit 1201 determines that the target pixel is located near a block boundary, switch 1202 outputs the image before filtering to switch 1204. Conversely, if the boundary determination unit 1201 determines that the target pixel is not located near a block boundary, switch 1202 outputs the image before filtering to switch 1206. The image before filtering consists of the target pixel and at least one surrounding pixel located around that target pixel.

[0233] The filter determination unit 1203 determines whether or not to perform a deblocking filter on the target pixel based on the pixel values ​​of at least one surrounding pixel located around the target pixel. The filter determination unit 1203 then outputs the determination result to the switch 1204 and the processing determination unit 1208.

[0234] If the filter determination unit 1203 determines that deblocking filtering should be performed on the target pixel, switch 1204 outputs the pre-filtered image acquired via switch 1202 to the filter processing unit 1205. Conversely, if the filter determination unit 1203 determines that deblocking filtering should not be performed on the target pixel, switch 1204 outputs the pre-filtered image acquired via switch 1202 to switch 1206.

[0235] When the filter processing unit 1205 acquires an image before filtering via switches 1202 and 1204, it performs a deblocking filter process on the target pixel, using the filter characteristics determined by the filter characteristic determination unit 1207. The filter processing unit 1205 then outputs the filtered pixel to switch 1206.

[0236] Switch 1206 selectively outputs pixels that have not undergone deblocking and filtering, and pixels that have undergone deblocking and filtering by the filtering processing unit 1205, in accordance with the control by the processing determination unit 1208.

[0237] The processing determination unit 1208 controls the switch 1206 based on the determination results of the boundary determination unit 1201 and the filter determination unit 1203. Specifically, if the boundary determination unit 1201 determines that a target pixel is near a block boundary, and the filter determination unit 1203 determines that the target pixel should undergo deblocking and filtering, the processing determination unit 1208 outputs the deblocked and filtered pixel from the switch 1206. In other cases, the processing determination unit 1208 outputs the unfiltered pixel from the switch 1206. This output of pixels is repeated, resulting in the filtered image being output from the switch 1206. Note that the configuration shown in Figure 24 is just one example of the configuration of the deblocking and filtering processing unit 120a, and the deblocking and filtering processing unit 120a may have other configurations.

[0238] Figure 25 shows an example of a deblocking filter with symmetrical filter characteristics with respect to block boundaries.

[0239] In deblocking filtering, for example, one of two deblocking filters with different characteristics, namely a strong filter and a weak filter, is selected using pixel values ​​and quantization parameters. In the strong filter, as shown in Figure 25, if pixels p0~p2 and pixels q0~q2 exist on either side of a block boundary, the respective pixel values ​​of pixels q0~q2 are changed to pixel values ​​q'0~q'2 by performing the operation shown in the following equation.

[0240] q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8 q'1=(p0+q0+q1+q2+2) / 4 q'2=(p0+q0+q1+3×q2+2×q3+4) / 8

[0241] In the above equations, p0~p2 and q0~q2 are the pixel values ​​of pixels p0~p2 and pixels q0~q2, respectively. Also, q3 is the pixel value of pixel q3, which is adjacent to pixel q2 on the opposite side of the block boundary. Furthermore, the coefficient multiplied by the pixel value of each pixel used in the deblocking filter process on the right-hand side of each of the above equations is the filter coefficient.

[0242] Furthermore, in the deblocking filter process, clipping may be performed to ensure that the calculated pixel values ​​do not exceed a threshold. In this clipping process, the calculated pixel values ​​according to the above formula are clipped to "pre-calculation pixel value ± 2 × threshold" using a threshold determined from the quantization parameters. This prevents excessive smoothing.

[0243] Figure 26 illustrates an example of a block boundary where deblocking filtering is performed. Figure 27 shows an example of a BS value.

[0244] The block boundaries on which deblocking filtering is performed are, for example, the CU, PU, ​​or TU boundaries of an 8x8 pixel block as shown in Figure 26. Deblocking filtering is performed, for example, in units of 4 rows or 4 columns. First, for blocks P and Q shown in Figure 26, the Bs (Boundary Strength) value is determined as shown in Figure 27.

[0245] According to the Bs value in Figure 27, it may be determined whether or not to perform deblocking filtering of different strengths on block boundaries belonging to the same image. Deblocking filtering is performed on the chrominance signal when the Bs value is 2. Deblocking filtering is performed on the luminance signal when the Bs value is 1 or greater and predetermined conditions are met. Note that the criteria for determining the Bs value are not limited to those shown in Figure 27 and may be determined based on other parameters.

[0246] [Prediction Unit (Intra Prediction Unit, Inter Prediction Unit, Prediction Control Unit)] Figure 28 is a flowchart showing an example of processing performed in the prediction unit of the encoding device 100. For example, the prediction unit consists of all or some of the components of the intra-prediction unit 124, the inter-prediction unit 126, and the prediction control unit 128. The prediction processing unit includes, for example, the intra-prediction unit 124 and the inter-prediction unit 126.

[0247] The prediction unit generates a predicted image of the current block (step Sb_1). The predicted image may be, for example, an intra-prediction image (intra-prediction signal) or an inter-prediction image (inter-prediction signal). Specifically, the prediction unit generates a predicted image of the current block using a reconstructed image already obtained by generating predicted images for other blocks, generating prediction residuals, generating quantization coefficients, restoring the prediction residuals, and adding the predicted images.

[0248] The reconstructed image may be, for example, the image of the reference picture, or it may be the image of an encoded block (i.e., the other block mentioned above) within the current picture, which is the picture containing the current block. An encoded block within the current picture is, for example, an adjacent block to the current block.

[0249] Figure 29 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device 100.

[0250] The prediction unit generates a predicted image using a first method (step Sc_1a), a second method (step Sc_1b), and a third method (step Sc_1c). The first, second, and third methods are different methods for generating predicted images, and may be, for example, an interpretation method, an intraprediction method, and other prediction methods. These prediction methods may use the reconstructed images described above.

[0251] Next, the prediction unit evaluates the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_2). For example, the prediction unit calculates a cost C for each of the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c, and evaluates the predicted images by comparing their costs C. The cost C is calculated using the formula of the RD optimization model, for example, C = D + λ × R. In this formula, D is the coding distortion of the predicted image, and can be expressed as, for example, the sum of the absolute differences between the pixel values ​​of the current block and the pixel values ​​of the predicted image. R is the bitrate of the stream. λ is, for example, the Lagrange multiplier.

[0252] Next, the prediction unit selects one of the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_3). In other words, the prediction unit selects a method or mode to obtain the final predicted image. For example, the prediction unit selects the predicted image with the smallest cost C based on the cost C calculated for those predicted images. Alternatively, the evaluation in step Sc_2 and the selection of the predicted image in step Sc_3 may be based on parameters used in the coding process. The coding device 100 may signal information to identify the selected predicted image, method, or mode into a stream. This information may be, for example, a flag. This allows the decoding device 200 to generate a predicted image according to the method or mode selected by the coding device 100 based on this information. In the example shown in Figure 29, the prediction unit selects one of the predicted images after generating a predicted image for each method. However, the prediction unit may select a method or mode based on the parameters used in the coding process described above before generating those predicted images, and then generate the predicted image according to that method or mode.

[0253] For example, the first method and the second method are intra-prediction and inter-prediction, respectively, and the prediction unit may select the final predicted image for the current block from the predicted images generated according to these prediction methods.

[0254] Figure 30 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device 100.

[0255] First, the prediction unit generates a predicted image by intra-prediction (step Sd_1a) and then generates a predicted image by inter-prediction (step Sd_1b). The predicted image generated by intra-prediction is also called the intra-prediction image, and the predicted image generated by inter-prediction is also called the inter-prediction image.

[0256] Next, the prediction unit evaluates both the intra-predicted image and the inter-predicted image (step Sd_2). The cost C mentioned above may be used for this evaluation. The prediction unit may then select the prediction image with the smallest cost C from the intra-predicted image and the inter-predicted image as the final prediction image for the current block (step Sd_3). In other words, a prediction method or mode for generating the prediction image for the current block is selected.

[0257] [Intra Prediction Unit] The intra-prediction unit 124 generates a predicted image (i.e., an intra-predicted image) of the current block by performing intra-prediction (also called in-screen prediction) of the current block by referring to the block in the current picture stored in the block memory 118. Specifically, the intra-prediction unit 124 generates an intra-predicted image by performing intra-prediction by referring to the pixel values ​​(e.g., luminance value, chrominance value) of blocks adjacent to the current block, and outputs the intra-predicted image to the prediction control unit 128.

[0258] For example, the intra-prediction unit 124 performs intra-prediction using one of a predetermined set of intra-prediction modes. The set of intra-prediction modes typically includes one or more non-directional prediction modes and multiple directional prediction modes.

[0259] One or more non-directional prediction modes include, for example, the Planar prediction mode and DC prediction mode as defined in the H.265 / HEVC standard.

[0260] Multiple directional prediction modes include, for example, the 33 directional prediction modes defined in the H.265 / HEVC standard. Note that multiple directional prediction modes may also include 32 additional directional prediction modes (a total of 65 directional prediction modes). Figure 31 shows all 67 intra-prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in intra-prediction. Solid arrows represent the 33 directions defined in the H.265 / HEVC standard, and dashed arrows represent the additional 32 directions (the 2 non-directional prediction modes are not shown in Figure 31).

[0261] In various implementations, luminance blocks may be referenced in the intra-prediction of chrominance blocks. That is, the chrominance components of the current block may be predicted based on the luminance components of the current block. Such intra-prediction is sometimes called CCLM (cross-component linear model) prediction. Such an intra-prediction mode for chrominance blocks that references luminance blocks (e.g., called CCLM mode) may be added as one of the intra-prediction modes for chrominance blocks.

[0262] The intra-prediction unit 124 may correct the pixel values ​​after intra-prediction based on the gradient of the horizontal / vertical reference pixels. Intra-prediction with such correction is sometimes called PDPC (position dependent intra-prediction combination). Information indicating whether or not PDPC is applied (e.g., called a PDPC flag) is usually signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (e.g., sequence level, picture level, slice level, brick level, or CTU level).

[0263] Figure 32 is a flowchart showing an example of processing by the intra prediction unit 124.

[0264] The intra-prediction unit 124 selects one intra-prediction mode from a plurality of intra-prediction modes (step Sw_1). Then, the intra-prediction unit 124 generates a predicted image according to the selected intra-prediction mode (step Sw_2). Next, the intra-prediction unit 124 determines the MPM (Most Probable Modes) (step Sw_3). The MPM consists of, for example, six intra-prediction modes. Two of these six intra-prediction modes may be Planar prediction modes and DC prediction modes, and the remaining four modes may be directional prediction modes. Then, the intra-prediction unit 124 determines whether the intra-prediction mode selected in step Sw_1 is included in the MPM (step Sw_4).

[0265] If the intra-prediction unit 124 determines that the selected intra-prediction mode is included in the MPM (Yes in step Sw_4), it sets the MPM flag to 1 (step Sw_5) and generates information from the MPM indicating the selected intra-prediction mode (step Sw_6). The MPM flag set to 1 and the information indicating that intra-prediction mode are each encoded as prediction parameters by the entropy encoding unit 110.

[0266] On the other hand, if the intra-prediction unit 124 determines that the selected intra-prediction mode is not included in the MPM (No in step Sw_4), it sets the MPM flag to 0 (step Sw_7). Alternatively, the intra-prediction unit 124 does not set the MPM flag. Then, the intra-prediction unit 124 generates information indicating the selected intra-prediction mode from among one or more intra-prediction modes not included in the MPM (step Sw_8). The MPM flag set to 0 and the information indicating that intra-prediction mode are each encoded as prediction parameters by the entropy coding unit 110. The information indicating that intra-prediction mode represents a value between 0 and 60, for example.

[0267] [International Prediction Department] The inter-prediction unit 126 generates a predicted image (inter-predicted image) by performing inter-prediction (also called inter-frame prediction) of the current block by referring to a reference picture stored in the frame memory 122 that is different from the current picture. Inter-prediction is performed on a unit of the current block or the current subblock within the current block. A subblock is a unit contained within a block and is smaller than a block. The size of a subblock may be 4x4 pixels, 8x8 pixels, or other sizes. The size of a subblock may be switched on a unit such as a slice, brick, or picture.

[0268] For example, the interpretation unit 126 performs motion estimation within a reference picture for the current block or current subblock and finds the reference block or subblock that best matches the current block or current subblock. The interpretation unit 126 then obtains motion information (e.g., motion vectors) that compensates for the movement or change from the reference block or subblock to the current block or subblock. Based on this motion information, the interpretation unit 126 performs motion compensation (or motion prediction) and generates an interpretation prediction image of the current block or subblock. The interpretation unit 126 outputs the generated interpretation prediction image to the prediction control unit 128.

[0269] The motion information used for motion compensation may be signaled as an interprediction image in various forms. For example, the motion vector may be signaled. As another example, the difference between the motion vector and the predicted motion vector (motion vector predictor) may be signaled.

[0270] [Reference Picture List] Figure 33 is a diagram showing an example of each reference picture, and Figure 34 is a conceptual diagram showing an example of a reference picture list. The reference picture list is a list showing one or more reference pictures stored in the frame memory 122. In Figure 33, rectangles represent pictures, arrows indicate the reference relationships between pictures, the horizontal axis represents time, I, P, and B in the rectangles represent intra-prediction pictures, single-prediction pictures, and double-prediction pictures, respectively, and the numbers in the rectangles indicate the decoding order. As shown in Figure 33, the decoding order of each picture is I0, P1, B2, B3, B4, and the display order of each picture is I0, B3, B2, B4, P1. As shown in Figure 34, the reference picture list is a list representing candidates for reference pictures, and for example, one picture (or slice) may have one or more reference picture lists. For example, if the current picture is a single-prediction picture, one reference picture list is used, and if the current picture is a double-prediction picture, two reference picture lists are used. In the examples in Figures 33 and 34, picture B3, which is the current picture currPic, has two reference picture lists, the L0 list and the L1 list. When the current picture currPic is picture B3, the candidate reference pictures for that current picture currPic are I0, P1, and B2, and each reference picture list (i.e., the L0 list and the L1 list) indicates these pictures. The interpretation unit 126 or the prediction control unit 128 specifies which picture in each reference picture list to actually reference using the reference picture index refIdxLx. In Figure 34, reference pictures P1 and B2 are specified by the reference picture indices refIdxL0 and refIdxL1.

[0271] Such reference picture lists may be generated on a sequence, picture, slice, brick, CTU, or CU basis. Furthermore, the reference picture index indicating the reference picture referenced in interpretation among the reference pictures shown in the reference picture list may be encoded at the sequence, picture, slice, brick, CTU, or CU level. Additionally, a common reference picture list may be used across multiple interpretation modes.

[0272] [Basic flow of interpretation] Figure 35 is a flowchart showing the basic flow of interpretation prediction.

[0273] The interpretation unit 126 first generates a predicted image (steps Se_1 to Se_3). Next, the subtraction unit 104 generates the difference between the current block and the predicted image as the predicted residual (step Se_4).

[0274] Here, the interpretation unit 126 generates the predicted image by, for example, determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and performing motion compensation (step Se_3). In determining the MV, the interpretation unit 126 determines the MV by, for example, selecting candidate motion vectors (candidate MVs) (step Se_1) and deriving the MV (step Se_2). The selection of candidate MVs is performed, for example, by the interpretation unit 126 generating a candidate MV list and selecting at least one candidate MV from the candidate MV list. Note that previously derived MVs may be added as candidate MVs to the candidate MV list. Furthermore, in the derivation of the MV, the interpretation unit 126 may determine the selected at least one candidate MV as the MV of the current block by further selecting at least one candidate MV from at least one candidate MV. Alternatively, the interpretation unit 126 may determine the MV of the current block by searching the region of the reference picture indicated by each of the selected candidate MVs. This search of the reference picture region may be called motion estimation.

[0275] Furthermore, in the example described above, steps Se_1 to Se_3 are performed by the interpretation unit 126, but processing such as step Se_1 or step Se_2 may be performed by other components included in the encoding device 100.

[0276] Furthermore, a candidate MV list may be created for each process in each interpretation mode, or a common candidate MV list may be used across multiple interpretation modes. Also, the processes in steps Se_3 and Se_4 correspond to the processes in steps Sa_3 and Sa_4 shown in Figure 9, respectively. In addition, the process in step Se_3 corresponds to the process in step Sd_1b in Figure 30.

[0277] [MV Derivation Flow] Figure 36 is a flowchart showing an example of MV derivation.

[0278] The interpretation unit 126 may derive the MV of the current block in a mode that encodes motion information (e.g., MV). In this case, for example, the motion information may be encoded as prediction parameters and signaled. That is, the encoded motion information is included in the stream.

[0279] Alternatively, the interpretation unit 126 may derive MV in a mode that does not encode motion information. In this case, motion information is not included in the stream.

[0280] Here, the modes for MV derivation include the normal inter mode, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that encode motion information include the normal inter mode, normal merge mode, and affine mode (specifically, the affine inter mode and affine merge mode). Note that the motion information may include not only the MV but also the predicted MV selection information, which will be described later. Modes that do not encode motion information include the FRUC mode, etc. The inter prediction unit 126 selects a mode from these multiple modes for deriving the MV of the current block and derives the MV of the current block using the selected mode.

[0281] Figure 37 is a flowchart showing another example of MV derivation.

[0282] The interpretation unit 126 may derive the MV of the current block in a mode that encodes the differential MV. In this case, for example, the differential MV is encoded as a prediction parameter and signaled. That is, the encoded differential MV is included in the stream. This differential MV is the difference between the MV of the current block and its predicted MV. The predicted MV is the predicted motion vector.

[0283] Alternatively, the interpretation unit 126 may derive the MV in a mode that does not encode the differential MV. In this case, the encoded differential MV is not included in the stream.

[0284] As mentioned above, the modes for MV derivation include the normal inter, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that encode differential MV include the normal inter mode and affine mode (specifically, affine inter mode). Modes that do not encode differential MV include the FRUC mode, normal merge mode, and affine mode (specifically, affine merge mode). The inter prediction unit 126 selects a mode from these multiple modes to derive the MV of the current block, and uses the selected mode to derive the MV of the current block.

[0285] [Modes for MV derivation] Figures 38A and 38B illustrate an example of the classification of each mode of MV derivation. For example, as shown in Figure 38A, the modes of MV derivation can be broadly classified into three modes depending on whether motion information is encoded and whether differential MVs are encoded. The three modes are intermode, merge mode, and FRUC (frame rate up-conversion) mode. Intermode is a mode that performs motion search and encodes motion information and differential MVs. For example, as shown in Figure 38B, intermode includes affine intermode and normal intermode. Merge mode is a mode that does not perform motion search and selects an MV from a surrounding encoded block and uses that MV to derive the MV of the current block. This merge mode basically encodes motion information but does not encode differential MVs. For example, as shown in Figure 38B, the merge modes include the normal merge mode (sometimes called the regular merge mode or normal merge mode), the MMVD (Merge with Motion Vector Difference) mode, the CIIP (Combined inter merge / intra prediction) mode, the triangle mode, the ATMVP mode, and the affine merge mode. Here, in the MMVD mode, one of the modes included in the merge modes, the differential MV is encoded as an exception. The affine merge mode and the affine inter mode mentioned above are modes included in the affine mode. The affine mode is a mode that assumes an affine transformation and derives the MV of each of the multiple subblocks that make up the current block as the MV of the current block. The FRUC mode is a mode that derives the MV of the current block by performing a search between encoded regions, and does not encode either motion information or differential MV. Details of each of these modes will be described later.

[0286] Note that the classification of each mode shown in Figures 38A and 38B is just an example and is not limited to this classification. For example, if a differential MV is encoded in CIIP mode, that CIIP mode is classified as an intermode.

[0287] [MV Derivation > Normal Intermode] The normal intermode is an interpretation mode that derives the MV of the current block by finding blocks similar to the image of the current block from the region of the reference picture indicated by the candidate MV. In this normal intermode, the differential MV is also encoded.

[0288] Figure 39 is a flowchart showing an example of inter-mode prediction.

[0289] The interpretation unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks surrounding the current block in time or space (step Sg_1). In other words, the interpretation unit 126 creates a candidate MV list.

[0290] Next, the interpretation unit 126 extracts N candidate MVs (where N is an integer greater than or equal to 2) from the multiple candidate MVs obtained in step Sg_1 as predicted MV candidates, according to a predetermined priority order (step Sg_2). The priority order is predetermined for each of the N candidate MVs.

[0291] Next, the interpretation unit 126 selects one predicted MV candidate from the N predicted MV candidates as the predicted MV for the current block (step Sg_3). At this time, the interpretation unit 126 encodes predicted MV selection information into a stream to identify the selected predicted MV. In other words, the interpretation unit 126 outputs the predicted MV selection information as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0292] Next, the interpretation unit 126 refers to the encoded reference picture and derives the MV of the current block (step Sg_4). At this time, the interpretation unit 126 further encodes the difference between the derived MV and the predicted MV as the difference MV into the stream. In other words, the interpretation unit 126 outputs the difference MV as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130. The encoded reference picture is a picture consisting of multiple blocks that have been reconstructed after encoding.

[0293] Finally, the inter-prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Sg_5). The processes in steps Sg_1 to Sg_5 are performed for each block. For example, when the processes in steps Sg_1 to Sg_5 are performed for each block contained in a slice, the inter-prediction using the normal inter-mode for that slice is completed. Similarly, when the processes in steps Sg_1 to Sg_5 are performed for each block contained in a picture, the inter-prediction using the normal inter-mode for that picture is completed. Note that the processes in steps Sg_1 to Sg_5 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using the normal inter-mode for that slice is completed. Likewise, when the processes in steps Sg_1 to Sg_5 are performed for some blocks contained in a picture, the inter-prediction using the normal inter-mode for that picture is completed.

[0294] The predicted image is the interprediction signal described above. Furthermore, information indicating the interprediction mode used to generate the predicted image (normal intermode in the example above), which is included in the encoded signal, is encoded, for example, as a prediction parameter.

[0295] The candidate MV list may be the same as the list used in other modes. Furthermore, processing related to the candidate MV list may be applied to processing related to lists used in other modes. This processing related to the candidate MV list may include, for example, extracting or selecting candidate MVs from the candidate MV list, rearranging candidate MVs, or deleting candidate MVs.

[0296] [MV Derivation > Normal Merge Mode] The normal merge mode is an interpretation mode that derives an MV by selecting a candidate MV from a list of candidate MVs as the MV of the current block. Note that the normal merge mode is a merge mode in the narrow sense and is sometimes simply called the merge mode. In this embodiment, a distinction is made between the normal merge mode and the merge mode, and the term merge mode is used in the broad sense.

[0297] Figure 40 is a flowchart showing an example of inter prediction using normal merge mode.

[0298] The interpretation unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks surrounding the current block in time or space (step Sh_1). In other words, the interpretation unit 126 creates a candidate MV list.

[0299] Next, the interpretation unit 126 derives the MV of the current block by selecting one candidate MV from among the multiple candidate MVs obtained in step Sh_1 (step Sh_2). At this time, the interpretation unit 126 encodes MV selection information to identify the selected candidate MV into a stream. In other words, the interpretation unit 126 outputs the MV selection information as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0300] Finally, the inter prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Sh_3). The processes in steps Sh_1 to Sh_3 are performed for each block, for example. For example, when the processes in steps Sh_1 to Sh_3 are performed for each of the blocks contained in a slice, the inter prediction using normal merge mode for that slice is completed. Similarly, when the processes in steps Sh_1 to Sh_3 are performed for each of the blocks contained in a picture, the inter prediction using normal merge mode for that picture is completed. Note that the processes in steps Sh_1 to Sh_3 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter prediction using normal merge mode for that slice is completed. Likewise, when the processes in steps Sh_1 to Sh_3 are performed for some of the blocks contained in a picture, the inter prediction using normal merge mode for that picture is completed.

[0301] Furthermore, information indicating the inter-prediction mode used to generate the predicted image (normal merge mode in the example above), which is included in the stream, is encoded, for example, as prediction parameters.

[0302] Figure 41 is a diagram illustrating an example of the MV derivation process for the current picture using normal merge mode.

[0303] First, the interpretation unit 126 generates a candidate MV list in which candidate MVs are registered. Candidate MVs include spatially adjacent candidate MVs, which are MVs of multiple encoded blocks located spatially around the current block; temporally adjacent candidate MVs, which are MVs of nearby blocks projected onto the current block's position in the encoded reference picture; combined candidate MVs, which are MVs generated by combining the MV values ​​of spatially adjacent candidate MVs and temporally adjacent candidate MVs; and zero candidate MVs, which are MVs with a value of zero.

[0304] Next, the interpretation unit 126 selects one candidate MV from among the multiple candidate MVs registered in the candidate MV list, thereby determining that one candidate MV as the MV of the current block.

[0305] Furthermore, the entropy coding unit 110 writes and encodes merge_idx, a signal indicating which candidate MV was selected, into a stream.

[0306] Note that the candidate MVs registered in the candidate MV list explained in Figure 41 are just examples, and the number of candidates may differ from the number shown in the figure, the configuration may not include some of the types of candidate MVs shown in the figure, or it may include candidate MVs other than those shown in the figure.

[0307] The final MV may be determined by performing DMVR (dynamic motion vector refreshing), described later, using the MV of the current block derived in normal merge mode. Note that in normal merge mode, the differential MV is not encoded, but in MMVD mode, the differential MV is encoded. In MMVD mode, one candidate MV is selected from the candidate MV list, similar to normal merge mode, but the differential MV is encoded. Such MMVD may be classified as a merge mode together with normal merge mode, as shown in Figure 38B. Note that the differential MV in MMVD mode does not have to be the same as the differential MV used in inter-mode; for example, the derivation of the differential MV in MMVD mode may be a less computationally intensive process than the derivation of the differential MV in inter-mode.

[0308] Alternatively, a CIIP (Combined inter merge / intra prediction) mode may be used to generate a prediction image for the current block by overlaying the prediction image generated by inter prediction with the prediction image generated by intra prediction.

[0309] The candidate MV list may also be referred to simply as the candidate list. Furthermore, merge_idx is the MV selection information.

[0310] [MV Derivation > HMVP Mode] Figure 42 is a diagram illustrating an example of the MV derivation process for the current picture using HMVP mode.

[0311] In normal merge mode, the MV of the current block (e.g., CU) is determined by selecting one candidate MV from a list of candidate MVs generated by referencing the encoded block (e.g., CU). Other candidate MVs may be added to this list. This mode, where other candidate MVs are added, is called HMVP mode.

[0312] In HMVP mode, candidate MVs are managed using a separate FIFO (First-In First-Out) buffer for HMVP, distinct from the candidate MV list used in normal merge mode.

[0313] The FIFO buffer stores motion information such as MVs of blocks that have been processed in the past, in reverse chronological order. In the management of this FIFO buffer, each time a block is processed, the MV of the most recent block (i.e., the most recently processed CU) is stored in the FIFO buffer, and in its place, the MV of the oldest CU (i.e., the most recently processed CU) in the FIFO buffer is removed. In the example shown in Figure 42, HMVP1 is the MV of the most recent block, and HMVP5 is the MV of the oldest block.

[0314] For example, the interpretation unit 126 checks each MV managed in the FIFO buffer, starting with HMVP1, whether that MV is different from all the candidate MVs already registered in the candidate MV list for normal merge mode. If the interpretation unit 126 determines that it is different from all the candidate MVs, it may add the MV managed in the FIFO buffer as a candidate MV to the candidate MV list for normal merge mode. At this time, one or more candidate MVs may be registered from the FIFO buffer.

[0315] By using HMVP mode in this way, it becomes possible to include not only MVs of spatially or temporally adjacent blocks to the current block, but also MVs of previously processed blocks as candidates. As a result, the variety of candidate MVs in normal merge mode is broadened, which increases the likelihood of improving encoding efficiency.

[0316] Furthermore, the aforementioned MV may also be motion information. In other words, the information stored in the candidate MV list and FIFO buffer may include not only the MV value, but also information indicating the referenced picture, the direction and number of references, etc. Also, the aforementioned block is, for example, a CU.

[0317] Note that the candidate MV list and FIFO buffer in Figure 42 are just examples, and the candidate MV list and FIFO buffer may be lists or buffers of different sizes than those in Figure 42, or the candidate MVs may be registered in a different order than those in Figure 42. Also, the processing described here is common to both the encoding device 100 and the decoding device 200.

[0318] Furthermore, HMVP mode can be applied to modes other than normal merge mode. For example, motion information such as MV of blocks previously processed in affine mode can be stored in the FIFO buffer in chronological order from newest to oldest and used as candidate MV. A mode in which HMVP mode is applied to affine mode may be called history affine mode.

[0319] [MV Derivation > FRUC Mode] Motion information may be derived at the decoding device 200 without being signaled at the encoding device 100. For example, motion information may be derived at the decoding device 200 by performing a motion search. In this case, the decoding device 200 performs the motion search without using the pixel values ​​of the current block. Modes for performing such a motion search at the decoding device 200 include FRUC (frame rate up-conversion) mode and PMMVD (pattern matched motion vector derivation) mode.

[0320] An example of FRUC processing is shown in Figure 43. First, a list is generated (i.e., a candidate MV list, which may be the same as the candidate MV list for normal merge mode) that refers to the MVs of each encoded block spatially or temporally adjacent to the current block and indicates those MVs as candidate MVs (step Si_1). Next, the best candidate MV is selected from among the multiple candidate MVs registered in the candidate MV list (step Si_2). For example, an evaluation value is calculated for each candidate MV included in the candidate MV list, and one candidate MV is selected as the best candidate MV based on that evaluation value. Then, the MV for the current block is derived based on the selected best candidate MV (step Si_4). Specifically, for example, the selected best candidate MV is directly derived as the MV for the current block. Alternatively, for example, the MV for the current block may be derived by performing pattern matching in the area surrounding the position in the reference picture corresponding to the selected best candidate MV. In other words, a search is performed in the area surrounding the best candidate MV using pattern matching and evaluation values ​​in the reference picture. If an MV with a better evaluation value is found, the best candidate MV may be updated to that MV and made the final MV of the current block. It is not necessary to update to an MV with a better evaluation value.

[0321] Finally, the inter-prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Si_5). The processes in steps Si_1 to Si_5 are performed for each block, for example. For example, when the processes in steps Si_1 to Si_5 are performed for each of the blocks contained in a slice, the inter-prediction using FRUC mode for that slice is completed. Similarly, when the processes in steps Si_1 to Si_5 are performed for each of the blocks contained in a picture, the inter-prediction using FRUC mode for that picture is completed. Note that the processes in steps Si_1 to Si_5 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using FRUC mode for that slice is completed. Likewise, when the processes in steps Si_1 to Si_5 are performed for some of the blocks contained in a picture, the inter-prediction using FRUC mode for that picture is completed.

[0322] Subblocks may be processed in the same way as blocks as described above.

[0323] The evaluation value may be calculated by various methods. For example, the reconstructed image of a region in the reference picture corresponding to the MV may be compared with the reconstructed image of a predetermined region (which may be, for example, a region in another reference picture or a region in an adjacent block of the current picture, as shown below). The difference in pixel values ​​between the two reconstructed images may then be calculated and used as the evaluation value for the MV. In addition to the difference value, other information may also be used to calculate the evaluation value.

[0324] Next, we will explain pattern matching in detail. First, one candidate MV included in the candidate MV list (also called the merge list) is selected as the starting point for the search using pattern matching. For pattern matching, either first-order pattern matching or second-order pattern matching may be used. First-order pattern matching and second-order pattern matching are sometimes called bilateral matching and template matching, respectively.

[0325] [MV Derivation > FRUC > Bilateral Matching] In the first pattern matching, pattern matching is performed between two blocks in two different reference pictures that are aligned with the motion trajectory of the current block. Therefore, in the first pattern matching, a region in another reference picture aligned with the motion trajectory of the current block is used as a predetermined region for calculating the evaluation value of the candidate MV described above.

[0326] Figure 44 illustrates an example of first-order pattern matching (bilateral matching) between two blocks in two reference pictures along a motion trajectory. As shown in Figure 44, in first-order pattern matching, two MVs (MV0, MV1) are derived by searching for the best-matching pair of blocks in two different reference pictures (Ref0, Ref1) that are along the motion trajectory of the current block. Specifically, for the current block, the difference is derived between the reconstructed image at a specified position in the first encoded reference picture (Ref0) specified by the candidate MV and the reconstructed image at a specified position in the second encoded reference picture (Ref1) specified by a symmetric MV scaled by the display time interval of the candidate MV. An evaluation value is then calculated using the obtained difference value. The candidate MV with the best evaluation value among multiple candidate MVs should be selected as the best candidate MV.

[0327] Under the assumption of a continuous motion trajectory, the MV(MV0, MV1) pointing to two reference blocks is proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, if the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the two reference pictures are equal, then the first pattern matching derives a mirror-symmetric bidirectional MV.

[0328] [MV Derivation > FRUC > Template Matching] In the second pattern matching (template matching), pattern matching is performed between the template in the current picture (blocks adjacent to the current block in the current picture (e.g., blocks above and / or to the left)) and the blocks in the reference picture. Therefore, in the second pattern matching, the blocks adjacent to the current block in the current picture are used as a predetermined area for calculating the evaluation value of the candidate MV described above.

[0329] Figure 45 illustrates an example of pattern matching (template matching) between a template in the current picture and a block in the reference picture. As shown in Figure 45, in the second pattern matching, the MV of the current block is derived by searching in the reference picture (Ref0) for the block that best matches the block adjacent to the current block (Cur block) in the current picture (Cur Pic). Specifically, for the current block, the difference between the reconstructed image of the encoded region of both or either of the left adjacent and upper adjacent regions and the reconstructed image at the equivalent position in the encoded reference picture (Ref0) specified by the candidate MV is derived, and an evaluation value is calculated using the obtained difference value. The candidate MV with the best evaluation value among multiple candidate MVs should be selected as the best candidate MV.

[0330] Information indicating whether or not to apply such a FRUC mode (e.g., called a FRUC flag) may be signaled at the CU level. Furthermore, if FRUC mode is applied (e.g., the FRUC flag is true), information indicating the applicable pattern matching method (first pattern matching or second pattern matching) may be signaled at the CU level. Note that the signaling of this information is not limited to the CU level, but may be at other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or subblock level).

[0331] [MV Derivation > Affine Mode] The affine mode is a mode in which the MV is generated using an affine transform. For example, the MV may be derived on a subblock basis based on the MVs of multiple adjacent blocks. This mode is sometimes called the affine motion compensation prediction mode.

[0332] Figure 46A illustrates an example of deriving the MV of a subblock based on the MV of multiple adjacent blocks. In Figure 46A, the current block contains, for example, 16 subblocks consisting of 4x4 pixels. Here, the motion vector v0 of the upper left corner control point of the current block is derived based on the MV of the adjacent blocks, and similarly, the motion vector v1 of the upper right corner control point of the current block is derived based on the MV of the adjacent subblock. Then, by projecting the two motion vectors v0 and v1 using the following equation (1A), the motion vector (v) of each subblock within the current block is obtained. x ,v y ) is derived.

[0333]

number

[0334] Here, x and y represent the horizontal and vertical positions of the subblock, respectively, and w represents a predetermined weighting coefficient.

[0335] Information indicating such affine modes (e.g., called an affine flag) may be signaled at the CU level. However, the signaling of this information indicating affine modes is not limited to the CU level, but may be at other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or subblock level).

[0336] Furthermore, such affine modes may include several modes with different methods for deriving the MV of the upper-left and upper-right corner control points. For example, there are two affine modes: the affine inter (also called the affine normal inter) mode and the affine merge mode.

[0337] Figure 46B illustrates an example of deriving the motion vector (MV) for a subblock unit in affine mode using three control points. In Figure 46B, the current block includes, for example, a subblock consisting of 16 4x4 pixels. Here, the motion vector v0 of the upper left corner control point of the current block is derived based on the MV of the adjacent block. Similarly, the motion vector v1 of the upper right corner control point of the current block is derived based on the MV of the adjacent block, and the motion vector v2 of the lower left corner control point of the current block is derived based on the MV of the adjacent block. Then, by projecting the three motion vectors v0, v1, and v2 using the following equation (1B), the motion vector (v) of each subblock within the current block is obtained. x ,v y ) is derived.

[0338]

number

[0339] Here, x and y represent the horizontal and vertical positions of the subblock center, respectively, and w and h represent predetermined weighting coefficients. w may represent the width of the current block, and h may represent the height of the current block.

[0340] Affine modes using different numbers of control points (e.g., two and three) may be switched and signaled at the CU level. Information indicating the number of control points for the affine modes used at the CU level may also be signaled at other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or subblock level).

[0341] Furthermore, an affine mode having three control points may include several modes in which the MV of the upper left, upper right, and lower left corner control points is derived in different ways. For example, an affine mode having three control points has two modes, similar to the affine mode having two control points described above: an affine intermode and an affine merge mode.

[0342] In affine mode, the size of each subblock included in the current block is not limited to 4x4 pixels; it can be any other size. For example, each subblock may be 8x8 pixels.

[0343] [MV Derivation > Affine Mode > Control Point] Figures 47A, 47B, and 47C are conceptual diagrams illustrating an example of MV derivation of control points in affine mode.

[0344] In affine mode, as shown in Figure 47A, the predicted MV for each control point of the current block is calculated based on multiple MVs corresponding to blocks encoded in affine mode among the encoded blocks adjacent to the current block: Block A (left), Block B (top), Block C (upper right), Block D (lower left), and Block E (upper left). Specifically, these blocks are examined in the order of encoded Block A (left), Block B (top), Block C (upper right), Block D (lower left), and Block E (upper left), and the first valid block encoded in affine mode is identified. Based on the multiple MVs corresponding to this identified block, the MV for the control point of the current block is calculated.

[0345] For example, as shown in Figure 47B, if block A, which is adjacent to the left of the current block, is encoded in affine mode with two control points, then motion vectors v3 and v4 are derived, projected onto the upper-left and upper-right corners of the encoded block containing block A. Then, from the derived motion vectors v3 and v4, the motion vector v0 of the upper-left corner control point and the motion vector v1 of the upper-right corner control point of the current block are calculated.

[0346] For example, as shown in Figure 47C, if block A adjacent to the left of the current block is encoded in affine mode with three control points, motion vectors v3, v4, and v5 are derived projected onto the upper-left, upper-right, and lower-left corners of the encoded block containing block A. Then, from the derived motion vectors v3, v4, and v5, the motion vector v0 of the upper-left corner control point, the motion vector v1 of the upper-right corner control point, and the motion vector v2 of the lower-left corner control point of the current block are calculated.

[0347] The MV derivation method shown in Figures 47A to 47C may be used to derive the MV of each control point in the current block in step Sk_1 shown in Figure 50 later, or it may be used to derive the predicted MV of each control point in the current block in step Sj_1 shown in Figure 51 later.

[0348] Figures 48A and 48B are conceptual diagrams illustrating another example of the derivation of the control point MV in affine mode.

[0349] Figure 48A is a diagram illustrating an affine mode with two control points.

[0350] In this affine mode, as shown in Figure 48A, the MV selected from the respective MVs of the encoded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the upper left corner control point of the current block. Similarly, the MV selected from the respective MVs of the encoded blocks D and E adjacent to the current block is used as the motion vector v1 of the upper right corner control point of the current block.

[0351] Figure 48B is a diagram illustrating an affine mode with three control points.

[0352] In this affine mode, as shown in Figure 48B, the MV selected from each of the MVs of the encoded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the upper left corner control point of the current block. Similarly, the MV selected from each of the MVs of the encoded blocks D and E adjacent to the current block is used as the motion vector v1 of the upper right corner control point of the current block. Furthermore, the MV selected from each of the MVs of the encoded blocks F and G adjacent to the current block is used as the motion vector v2 of the lower left corner control point of the current block.

[0353] The MV derivation methods shown in Figures 48A and 48B may be used to derive the MV of each control point in the current block in step Sk_1 shown in Figure 50, which will be described later, or they may be used to derive the predicted MV of each control point in the current block in step Sj_1 shown in Figure 51, which will be described later.

[0354] Here, for example, when switching between affine modes with different numbers of control points (e.g., two and three) at the CU level to generate signals, the number of control points may differ between the encoded block and the current block.

[0355] Figures 49A and 49B are conceptual diagrams illustrating an example of a method for deriving the control point MV when the number of control points differs between the encoded block and the current block.

[0356] For example, as shown in Figure 49A, the current block has three control points at the upper left corner, upper right corner, and lower left corner, and is encoded in affine mode with block A adjacent to the left of the current block having two control points. In this case, motion vectors v3 and v4 are derived by projecting them onto the upper left and upper right corner positions of the encoded block containing block A. Then, from the derived motion vectors v3 and v4, the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block are calculated. Furthermore, from the derived motion vectors v0 and v1, the motion vector v2 of the lower left corner control point is calculated.

[0357] For example, as shown in Figure 49B, the current block is encoded in affine mode, having two control points at the upper left and upper right corners, and block A, adjacent to the left of the current block, has three control points. In this case, motion vectors v3, v4, and v5 are derived by projecting them onto the upper left, upper right, and lower left corners of the encoded block containing block A. Then, from the derived motion vectors v3, v4, and v5, the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block are calculated.

[0358] The MV derivation methods shown in Figures 49A and 49B may be used to derive the MV of each control point in the current block in step Sk_1 shown in Figure 50, which will be described later, or they may be used to derive the predicted MV of each control point in the current block in step Sj_1 shown in Figure 51, which will be described later.

[0359] [MV Derivation > Affine Mode > Affine Merge Mode] Figure 50 is a flowchart showing an example of the affine merge mode.

[0360] In affine merge mode, the interpretation unit 126 first derives the MVs of each control point in the current block (step Sk_1). The control points are the upper left and upper right corners of the current block, as shown in Figure 46A, or the upper left, upper right, and lower left corners of the current block, as shown in Figure 46B. At this time, the interpretation unit 126 may encode MV selection information into a stream to identify two or three of the derived MVs.

[0361] For example, when using the MV derivation method shown in Figures 47A to 47C, the interpretation unit 126 examines the encoded blocks in the order of A (left), B (top), C (upper right), D (lower left), and E (upper left), as shown in Figure 47A, and identifies the first valid block encoded in affine mode.

[0362] The interpretation unit 126 derives the motion vectors (MV) of the control points using the first valid block encoded in the identified affine mode. For example, if block A is identified and block A has two control points, as shown in Figure 47B, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block from the motion vectors v3 and v4 of the upper left and upper right corners of the encoded block containing block A. For example, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block by projecting the motion vectors v3 and v4 of the upper left and upper right corners of the encoded block onto the current block.

[0363] Alternatively, if block A is identified and block A has three control points, as shown in Figure 47C, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block from the motion vectors v3, v4, and v5 of the upper left, upper right, and lower left corners of the encoded block containing block A. For example, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block by projecting the motion vectors v3, v4, and v5 of the upper left, upper right, and lower left corners of the encoded block onto the current block.

[0364] Furthermore, as shown in Figure 49A above, if block A is identified and block A has two control points, the MV of three control points may be calculated. Alternatively, as shown in Figure 49B above, if block A is identified and block A has three control points, the MV of two control points may be calculated.

[0365] Next, the interpretation unit 126 performs motion compensation for each of the multiple subblocks contained in the current block. That is, for each of the multiple subblocks, the interpretation unit 126 calculates the MV of that subblock as an affine MV using two motion vectors v0 and v1 and equation (1A) described above, or using three motion vectors v0, v1 and v2 and equation (1B) described above (step Sk_2). Then, the interpretation unit 126 performs motion compensation for that subblock using those affine MVs and the encoded reference picture (step Sk_3). Once steps Sk_2 and Sk_3 have been executed for each of the subblocks contained in the current block, the process of generating a predicted image using the affine merge mode for that current block is completed. In other words, motion compensation is performed on the current block, and a predicted image for that current block is generated.

[0366] In step Sk_1, the above-mentioned candidate MV list may be generated. The candidate MV list may be a list containing candidate MVs derived for each control point using multiple MV derivation methods. The multiple MV derivation methods may be any combination of the MV derivation methods shown in Figures 47A to 47C, the MV derivation methods shown in Figures 48A and 48B, the MV derivation methods shown in Figures 49A and 49B, and other MV derivation methods.

[0367] The candidate MV list may also include candidate MVs from modes other than affine mode, where prediction is performed on a sub-block basis.

[0368] Furthermore, the candidate MV list may include, for example, candidate MVs for affine merge modes with two control points and candidate MVs for affine merge modes with three control points. Alternatively, a candidate MV list containing candidate MVs for affine merge modes with two control points and a candidate MV list containing candidate MVs for affine merge modes with three control points may be generated separately. Alternatively, a candidate MV list may be generated containing candidate MVs for one of the modes: affine merge mode with two control points and affine merge mode with three control points. The candidate MVs may be, for example, the MVs of encoded blocks A (left), B (top), C (upper right), D (lower left), and E (upper left), or they may be the MVs of valid blocks among those blocks.

[0369] Alternatively, you may send an index indicating which candidate MV from the candidate MV list is being selected as MV selection information.

[0370] [MV Derivation > Affine Mode > Affine Intermode] Figure 51 is a flowchart showing an example of an affine intermode.

[0371] In affine intermode, the interpretation unit 126 first derives the predicted MV(v0,v1) or (v0,v1,v2) for each of two or three control points of the current block (step Sj_1). The control points are the upper left corner, upper right corner, or lower left corner of the current block, as shown in Figure 46A or Figure 46B.

[0372] For example, when using the MV derivation method shown in Figures 48A and 48B, the interpretation unit 126 derives the predicted MV (v0,v1) or (v0,v1,v2) of the control point of the current block by selecting the MV of one of the encoded blocks near each control point of the current block shown in Figure 48A or Figure 48B. At this time, the interpretation unit 126 encodes predicted MV selection information into a stream to identify the two or three selected predicted MVs.

[0373] For example, the interpretation unit 126 may determine which block's MV from the encoded blocks adjacent to the current block to select as the predicted MV for the control point using cost evaluation or the like, and write a flag indicating which predicted MV was selected to the bitstream. In other words, the interpretation unit 126 outputs the predicted MV selection information, such as a flag, as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0374] Next, the interpretation unit 126 performs motion search (steps Sj_3 and Sj_4) while updating the predicted MVs selected or derived in step Sj_1 (step Sj_2). That is, the interpretation unit 126 calculates the MV of each subblock corresponding to the updated predicted MV as an affine MV using the above-mentioned equation (1A) or equation (1B) (step Sj_3). Then, the interpretation unit 126 performs motion compensation for each subblock using these affine MVs and encoded reference pictures (step Sj_4). The processing in steps Sj_3 and Sj_4 is performed for all blocks in the current block each time the predicted MV is updated in step Sj_2. As a result, the interpretation unit 126 determines, for example, the predicted MV that yields the smallest cost in the motion search loop as the MV of the control point (step Sj_5). At this time, the interpretation unit 126 further encodes the difference between the determined MV and the predicted MV as the difference MV into a stream. In other words, the interpretation unit 126 outputs the difference MV as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0375] Finally, the interpretation unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the determined MV and the encoded reference picture (step Sj_6).

[0376] In step Sj_1, the above-mentioned candidate MV list may be generated. The candidate MV list may be a list containing candidate MVs derived for each control point using multiple MV derivation methods. The multiple MV derivation methods may be any combination of the MV derivation methods shown in Figures 47A to 47C, the MV derivation methods shown in Figures 48A and 48B, the MV derivation methods shown in Figures 49A and 49B, and other MV derivation methods.

[0377] The candidate MV list may also include candidate MVs from modes other than affine mode, where prediction is performed on a sub-block basis.

[0378] Furthermore, a candidate MV list may be generated that includes candidate MVs for affine intermodes with two control points and candidate MVs for affine intermodes with three control points. Alternatively, a candidate MV list containing candidate MVs for affine intermodes with two control points and a candidate MV list containing candidate MVs for affine intermodes with three control points may be generated separately. Alternatively, a candidate MV list may be generated that includes candidate MVs for one of the modes: affine intermodes with two control points and affine intermodes with three control points. The candidate MVs may be, for example, the MVs of encoded blocks A (left), B (top), C (upper right), D (lower left), and E (upper left), or they may be the MVs of valid blocks among those blocks.

[0379] Additionally, as predicted MV selection information, you may send an index indicating which candidate MV from the candidate MV list it is.

[0380] [MV Derivation > Triangle Mode] In the example described above, the interpretation unit 126 generates one rectangular prediction image for the current rectangular block. However, the interpretation unit 126 may generate multiple prediction images of shapes different from rectangles for the current rectangular block, and then combine these multiple prediction images to generate the final rectangular prediction image. The shapes different from rectangles may be, for example, triangles.

[0381] Figure 52A is a diagram illustrating the generation of two triangular prediction images.

[0382] The interpretation unit 126 generates a predicted triangular image by performing motion compensation on the first triangular partition within the current block using the first MV of that first partition. Similarly, the interpretation unit 126 generates a predicted triangular image by performing motion compensation on the second triangular partition within the current block using the second MV of that second partition. Then, the interpretation unit 126 combines these predicted images to generate a predicted rectangular image identical to that of the current block.

[0383] Furthermore, a first rectangular prediction image corresponding to the current block may be generated using the first MV as the prediction image for the first partition. Similarly, a second rectangular prediction image corresponding to the current block may be generated using the second MV as the prediction image for the second partition. The prediction image for the current block may also be generated by weighting and adding the first and second prediction images. Note that the weighting and addition may be applied only to a portion of the region straddling the boundary between the first and second partitions.

[0384] Figure 52B is a conceptual diagram showing an example of a first portion of a first partition overlapping with a second partition, as well as a first and second sample set that may be weighted as part of a correction process. The first portion may be, for example, one-quarter of the width or height of the first partition. In another example, the first portion may have a width corresponding to N samples adjacent to the edge of the first partition, where N is an integer greater than zero, for example, N may be the integer 2. Figure 52B shows a rectangular partition having a rectangular portion with a width of one-quarter of the width of the first partition. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples within the first portion. The central example in Figure 52B shows a rectangular partition having a rectangular portion with a height of one-quarter of the height of the first partition. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples within the first portion. The example on the right in Figure 52B shows a triangular partition having polygonal sections of height corresponding to two samples. Here, the first sample set includes samples outside the first section and samples inside the first section, and the second sample set includes samples within the first section.

[0385] The first portion may be the portion of the first partition that overlaps with an adjacent partition. Figure 52C is a conceptual diagram showing the first portion of the first partition, which is the portion of the first partition that overlaps with a portion of an adjacent partition. For simplicity of explanation, a rectangular partition is shown that has a portion that overlaps with a spatially adjacent rectangular partition. Partitions of other shapes, such as triangular partitions, may be used, and the overlapping portion may overlap with a spatially or temporally adjacent partition.

[0386] Furthermore, while an example is shown in which prediction images are generated for each of the two partitions using interpretation, prediction images may also be generated for at least one partition using intrapretation.

[0387] Figure 53 is a flowchart showing an example of the triangle mode.

[0388] In triangle mode, the interpretation unit 126 first divides the current block into a first partition and a second partition (step Sx_1). At this time, the interpretation unit 126 may encode partition information, which is information about the division into each partition, into the stream as prediction parameters. In other words, the interpretation unit 126 may output partition information as prediction parameters to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0389] Next, the interpretation unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks surrounding the current block in time or space (step Sx_2). In other words, the interpretation unit 126 creates a candidate MV list.

[0390] Then, the interpretation unit 126 selects the candidate MVs for the first partition and the candidate MVs for the second partition from among the multiple candidate MVs obtained in step Sx_2 as the first MV and the second MV, respectively (step Sx_3). At this time, the interpretation unit 126 may encode MV selection information for identifying the selected candidate MVs into the stream as prediction parameters. In other words, the interpretation unit 126 may output the MV selection information as prediction parameters to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0391] Next, the interpretation unit 126 generates a first predicted image by performing motion compensation using the selected first MV and the encoded reference picture (step Sx_4). Similarly, the interpretation unit 126 generates a second predicted image by performing motion compensation using the selected second MV and the encoded reference picture (step Sx_5).

[0392] Finally, the interpretation unit 126 generates a predicted image of the current block by weighting and adding the first predicted image and the second predicted image (step Sx_6).

[0393] In the example shown in Figure 52A, the first and second partitions are triangular, but they may also be trapezoidal, or they may have different shapes from each other. Furthermore, in the example shown in Figure 52A, the current block is composed of two partitions, but it may be composed of three or more partitions.

[0394] Furthermore, the first and second partitions may overlap. That is, the first and second partitions may contain the same pixel region. In this case, the predicted image of the current block may be generated using the predicted image in the first partition and the predicted image in the second partition.

[0395] Furthermore, while this example shows that predicted images are generated by interpretation for both partitions, predicted images may also be generated by intrapretation for at least one partition.

[0396] Note that the candidate MV list for selecting the first MV and the candidate MV list for selecting the second MV may be different, or they may be the same.

[0397] The partition information may include an index indicating the partitioning direction for dividing the current block into multiple partitions. The MV selection information may include an index indicating the first selected MV and an index indicating the second selected MV. A single index may represent multiple pieces of information. For example, a single index may be encoded that represents part or all of the partition information and part or all of the MV selection information together.

[0398] [MV Derivation > ATMVP Mode] Figure 54 shows an example of an ATMVP mode in which MV is derived at the subblock level.

[0399] ATMVP mode is a mode classified as a merge mode. For example, in ATMVP mode, candidate MVs at the subblock level are registered in the candidate MV list used in normal merge mode.

[0400] Specifically, in ATMVP mode, first, as shown in Figure 54, the time MV reference block associated with the current block is identified in the encoded reference picture specified by the MV (MV0) of the block adjacent to the lower left of the current block. Next, for each subblock within the current block, the MV used when encoding the region corresponding to that subblock within the time MV reference block is identified. The MVs identified in this way are included in the candidate MV list as candidate MVs for the subblocks of the current block. When a candidate MV for each subblock is selected from the candidate MV list, motion compensation using that candidate MV as the subblock's MV is performed for that subblock. This generates a predicted image for each subblock.

[0401] In the example shown in Figure 54, the block adjacent to the lower left of the current block was used as the peripheral MV reference block, but other blocks may be used. Also, the size of the subblock may be 4x4 pixels, 8x8 pixels, or other sizes. The size of the subblock may be switched in units such as slices, bricks, or pictures.

[0402] [Motion Detection > DMVR] Figure 55 shows the relationship between merge mode and DMVR.

[0403] The interpretation unit 126 derives the MV of the current block in merge mode (step Sl_1). Next, the interpretation unit 126 determines whether or not to perform an MV search, i.e., a motion search (step Sl_2). If the interpretation unit 126 determines not to perform a motion search (No in step Sl_2), it determines the MV derived in step Sl_1 as the final MV for the current block (step Sl_4). In other words, in this case, the MV of the current block is determined in merge mode.

[0404] On the other hand, if it is determined in step Sl_1 to perform a motion search (Yes in step Sl_2), the interpretation unit 126 derives the final MV for the current block by searching the surrounding region of the reference picture indicated by the MV derived in step Sl_1 (step Sl_3). In other words, in this case, the MV for the current block is determined by the DMVR.

[0405] Figure 56 is a conceptual diagram illustrating an example of DMVR for determining MV.

[0406] First, for example in merge mode, candidate MVs (L0 and L1) are selected for the current block. Then, according to candidate MV (L0), reference pixels are identified from the first reference picture (L0), which is an encoded picture in the L0 list. Similarly, according to candidate MV (L1), reference pixels are identified from the second reference picture (L1), which is an encoded picture in the L1 list. A template is generated by taking the average of these reference pixels.

[0407] Next, using the template, the surrounding regions of candidate MVs for the first reference picture (L0) and the second reference picture (L1) are searched, and the MV with the minimum cost is determined as the final MV for the current block. The cost may be calculated, for example, using the difference between each pixel value in the template and each pixel value in the search region, as well as the candidate MV value.

[0408] Any process that can explore the vicinity of a candidate MV and derive the final MV is acceptable, even if it's not the exact process described here.

[0409] Figure 57 is a conceptual diagram illustrating another example of a DMVR for determining MV. Unlike the example of a DMVR shown in Figure 56, this example in Figure 57 calculates costs without generating a template.

[0410] First, the interpretation unit 126 searches around the reference blocks contained in the reference pictures of the L0 list and L1 list, respectively, based on the initial MV, which is a candidate MV obtained from the candidate MV list. For example, as shown in Figure 57, the initial MV corresponding to the reference block in the L0 list is InitMV_L0, and the initial MV corresponding to the reference block in the L1 list is InitMV_L1. In motion search, the interpretation unit 126 first sets the search position for the reference picture in the L0 list. The difference vector indicating the set search position, specifically the difference vector from the position indicated by the initial MV (i.e., InitMV_L0) to that search position, is MVd_L0. Then, the interpretation unit 126 determines the search position in the reference picture of the L1 list. This search position is indicated by the difference vector from the position indicated by the initial MV (i.e., InitMV_L1) to that search position. Specifically, the interpretation unit 126 determines this difference vector as MVd_L1 by mirroring MVd_L0. In other words, the interpretation unit 126 searches for a position that is symmetrical to the position indicated by the initial MV in the reference pictures of the L0 list and the L1 list. For each search position, the interpretation unit 126 calculates a cost such as the sum of the absolute differences in pixel values ​​within the block at that search position (SAD), and finds the search position that minimizes this cost.

[0411] Figure 58A is a diagram showing an example of motion search in a DMVR, and Figure 58B is a flowchart showing an example of that motion search.

[0412] First, in Step 1, the interpretation unit 126 calculates the cost of the search position indicated by the initial MV (also called the starting point) and the eight surrounding search positions. Then, the interpretation unit 126 determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 126 determines that the cost of the search positions other than the starting point is the minimum, it moves to the search position with the minimum cost and proceeds to Step 2. On the other hand, if the cost of the starting point is the minimum, the interpretation unit 126 skips Step 2 and proceeds to Step 3.

[0413] In Step 2, the interpretation unit 126 uses the search position moved according to the processing result of Step 1 as a new starting point and performs the same search as in Step 1. The interpretation unit 126 then determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 126 determines that the cost of the search positions other than the starting point is the minimum, it proceeds to Step 4. On the other hand, if the interpretation unit 126 determines that the cost of the starting point is the minimum, it proceeds to Step 3.

[0414] In Step 4, the interpretation unit 126 treats the starting position as the final search position and determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0415] In Step 3, the interpretation unit 126 determines the pixel position with the minimum cost based on the costs at the four points above, below, to the left and right of the starting point in Step 1 or Step 2, and sets that pixel position as the final search position. This decimal-precision pixel position is determined by weighting the vector of the four points above, below, to the left and right ((0,1),(0,-1),(-1,0),(1,0)) with the cost at each of the four search positions as the weight. The interpretation unit 126 then determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0416] [Motion compensation > BIO / OBMC / LIC] Motion compensation includes modes that generate a predictive image and then correct that predictive image. These modes include, for example, BIO, OBMC, and LIC, which will be described later.

[0417] Figure 59 is a flowchart showing an example of predictive image generation.

[0418] The interpretation unit 126 generates a predicted image (step Sm_1) and corrects the predicted image using one of the modes described above (step Sm_2).

[0419] Figure 60 is a flowchart showing another example of predictive image generation.

[0420] The interpretation unit 126 derives the MV of the current block (step Sn_1). Next, the interpretation unit 126 generates a predicted image using the MV (step Sn_2) and determines whether or not to perform correction processing (step Sn_3). If the interpretation unit 126 determines that correction processing should be performed (Yes in step Sn_3), it generates the final predicted image by correcting the predicted image (step Sn_4). In the LIC described later, brightness and color difference may also be corrected in step Sn_4. On the other hand, if the interpretation unit 126 determines that no correction processing should be performed (No in step Sn_3), it outputs the predicted image as the final predicted image without correction (step Sn_5).

[0421] [Motion Compensation > OBMC] Interpretation images may be generated using not only the motion information of the current block obtained through motion search, but also the motion information of adjacent blocks. Specifically, interpretation images may be generated at the sub-block level within the current block by weighted addition of a prediction image based on motion information obtained through motion search (in the reference picture) and a prediction image based on the motion information of adjacent blocks (in the current picture). Such interpretation (motion compensation) is sometimes called OBMC (oval-wrapped block motion compensation) or OBMC mode.

[0422] In OBMC mode, information indicating the size of the subblock for OBMC (e.g., called the OBMC block size) may be signaled at the sequence level. Furthermore, information indicating whether or not to apply OBMC mode (e.g., called the OBMC flag) may be signaled at the CU level. Note that the signaling levels of this information are not limited to the sequence level and CU level, but may be other levels (e.g., picture level, slice level, brick level, CTU level, or subblock level).

[0423] The OBMC mode will be explained in more detail. Figures 61 and 62 are flowcharts and conceptual diagrams illustrating the overview of the predictive image correction process using OBMC.

[0424] First, as shown in Figure 62, a predicted image (Pred) is obtained using normal motion compensation with the MV assigned to the current block. In Figure 62, the arrow "MV" points to the reference picture, indicating what the current block of the current picture is referencing in order to obtain the predicted image.

[0425] Next, the previously derived MV (MV_L) for the encoded left-adjacent block is applied (reused) to the current block to obtain the predicted image (Pred_L). The MV (MV_L) is indicated by an arrow “MV_L” pointing from the current block to the reference picture. Then, the first correction of the predicted image is performed by superimposing the two predicted images, Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.

[0426] Similarly, the previously derived MV (MV_U) for the encoded upper adjacent block is applied (reused) to the current block to obtain the predicted image (Pred_U). The MV (MV_U) is indicated by an arrow “MV_U” pointing from the current block to the reference picture. Then, the predicted image Pred_U is superimposed on the predicted image that has undergone the first correction (e.g., Pred and Pred_L) to perform a second correction of the predicted image. This has the effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is the final predicted image of the current block, with the boundaries with adjacent blocks blended (smoothed).

[0427] The above example is a two-pass correction method using left-adjacent and top-adjacent blocks, but the correction method may also be a three-pass or more-pass correction method using right-adjacent and / or bottom-adjacent blocks.

[0428] Furthermore, the area to be superimposed does not have to be the entire pixel area of ​​the block, but rather only a portion of the area near the block boundary.

[0429] Here, we have described the OBMC predictive image correction process for obtaining a single predictive image Pred by superimposing additional predictive images Pred_L and Pred_U onto a single reference picture. However, if the predictive image is corrected based on multiple reference images, the same process may be applied to each of the multiple reference pictures. In such cases, by performing OBMC image correction based on multiple reference pictures, a corrected predictive image is obtained from each reference picture, and then the final predictive image is obtained by further superimposing these multiple corrected predictive images.

[0430] In OBMC, the unit of a current block may be a PU unit, or it may be a subblock unit obtained by further dividing a PU.

[0431] One method for determining whether to apply OBMC is to use an obmc_flag signal, which indicates whether or not to apply OBMC. For example, the encoding device 100 may determine whether the current block belongs to a region with complex motion. If the encoding device 100 belongs to a region with complex motion, it sets the obmc_flag to a value of 1 and applies OBMC to perform encoding. If the block does not belong to a region with complex motion, it sets the obmc_flag to a value of 0 and encodes the block without applying OBMC. On the other hand, the decoding device 200 decodes the obmc_flag written in the stream and switches whether or not to apply OBMC depending on its value to perform decoding.

[0432] [Motion Compensation > BIO] Next, we will explain how to derive MV. First, we will describe the mode of deriving MV based on a model that assumes uniform linear motion. This mode is sometimes called the BIO (bi-directional optical flow) mode. Also, this bi-directional optical flow may be written as BDOF instead of BIO.

[0433] Figure 63 is a diagram illustrating a model assuming uniform linear motion. In Figure 63, (vx, vy) represents the velocity vector, and τ0 and τ1 represent the temporal distance between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1), respectively. (MVx0, MVy0) represents the MV corresponding to reference picture Ref0, and (MVx1, MVy1) represents the MV corresponding to reference picture Ref1.

[0434] Under the assumption of uniform linear motion of the velocity vector (vx, vy), (MVx0, MVy0) and (MVx1, MVy1) can be expressed as (vxτ0, vyτ0) and (-vxτ1, -vyτ1), respectively, and the following optical flow equality (2) holds.

[0435]

number

[0436] Here, I(k) represents the luminance value of the reference image k (k=0,1) after motion compensation. This optical flow equation shows that the sum of (i) the time derivative of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is equal to zero. Based on this optical flow equation and Hermite interpolation, block-level motion vectors obtained from a candidate MV list, etc., may be corrected on a pixel-by-pixel basis.

[0437] Furthermore, the motion vector (MV) may be derived on the decoding device 200 side using a method different from the derivation of the motion vector based on a model that assumes uniform linear motion. For example, the motion vector may be derived on a sub-block basis based on the MVs of multiple adjacent blocks.

[0438] Figure 64 is a flowchart illustrating an example of inter prediction according to BIO. Figure 65 is a diagram showing an example of the configuration of the inter prediction unit 126 that performs the inter prediction according to BIO.

[0439] As shown in FIG. 65, for example, the inter prediction unit 126 includes a memory 126a, an interpolation image derivation unit 126b, a gradient image derivation unit 126c, an optical flow derivation unit 126d, a correction value derivation unit 126e, and a prediction image correction unit 126f. Note that the memory 126a may be the frame memory 122.

[0440] The inter prediction unit 126 derives two motion vectors (M0, M1) using two reference pictures (Ref0, Ref1) different from the picture (Cur Pic) including the current block. Then, the inter prediction unit 126 derives a prediction image of the current block using the two motion vectors (M0, M1) (step Sy_1). Note that the motion vector M0 is a motion vector (MVx0, MVy0) corresponding to the reference picture Ref0, and the motion vector M1 is a motion vector (MVx1, MVy1) corresponding to the reference picture Ref1.

[0441] Next, the interpolation image derivation unit 126b refers to the memory 126a and derives an interpolation image I 0 of the current block using the motion vector M0 and the reference picture L0. Also, the interpolation image derivation unit 126b refers to the memory 126a and derives an interpolation image I 1 of the current block using the motion vector M1 and the reference picture L1 (step Sy_2). Here, the interpolation image I 0 is an image included in the reference picture Ref0 derived for the current block, and the interpolation image I 1 is an image included in the reference picture Ref1 derived for the current block. The interpolation image I 0 and the interpolation image I 1 may each have the same size as the current block. Alternatively, the interpolation image I 0 and the interpolation image I 1 may each be an image larger than the current block in order to appropriately derive a gradient image described later. Further, the interpolation images I 0 and I 1This may include a motion vector (M0, M1) and a reference picture (L0, L1), and a predicted image derived by applying a motion compensation filter.

[0442] Furthermore, the gradient image derivation unit 126c generates an interpolated image I 0 and interpolated image I 1 From, the gradient image of the current block (Ix 0 ,Ix 1 Iy 0 Iy 1 ) is derived (Step Sy_3). Note that the horizontal gradient image is (Ix 0 ,Ix 1 ) and the vertical gradient image is (Iy 0 Iy 1 The gradient image derivation unit 126c may derive the gradient image by, for example, applying a gradient filter to the interpolated image. The gradient image only needs to show the amount of spatial change in pixel values ​​along the horizontal or vertical direction.

[0443] Next, the optical flow derivation unit 126d interpolates the image (I) in units of multiple subblocks that constitute the current block. 0 ,I 1 ) and gradient image (Ix 0 ,Ix 1 Iy 0 Iy 1 Using the above-mentioned velocity vector, optical flow (vx, vy) is derived (step Sy_4). Optical flow is a coefficient that corrects the amount of spatial movement of pixels, and may also be called local motion estimate, corrected motion vector, or corrected weight vector. As an example, the subblock may be a 4x4 pixel subCU. Note that the derivation of optical flow may be performed in units other than subblocks, such as pixels.

[0444] Next, the interpretation unit 126 corrects the predicted image of the current block using optical flow (vx, vy). For example, the correction value derivation unit 126e derives correction values ​​for the pixel values ​​included in the current block using optical flow (vx, vy) (step Sy_5). Then, the predicted image correction unit 126f may correct the predicted image of the current block using the correction values ​​(step Sy_6). Note that the correction values ​​may be derived for each pixel, or for multiple pixels or subblocks.

[0445] Note that the BIO processing flow is not limited to the processing disclosed in Figure 64. Only some of the processing disclosed in Figure 64 may be performed, different processing may be added or replaced, or the processing may be executed in a different order.

[0446] [Motion Compensation > LIC] Next, we will describe an example of a mode that generates a predicted image (prediction) using LIC (local illumination compensation).

[0447] Figure 66A is a diagram illustrating an example of a predictive image generation method using luminance correction processing by LIC. Figure 66B is a flowchart illustrating the same predictive image generation method using LIC.

[0448] First, the interpretation unit 126 derives the MV from the encoded reference picture and obtains the reference image corresponding to the current block (step Sz_1).

[0449] Next, the interpretation unit 126 extracts information for the current block indicating how the luminance values ​​have changed between the reference picture and the current picture (step Sz_2). This extraction is performed based on the luminance pixel values ​​of the encoded left adjacent reference region (surrounding reference region) and encoded upper adjacent reference region (surrounding reference region) in the current picture, and the luminance pixel values ​​at the equivalent positions in the reference picture specified by the derived MV. Then, the interpretation unit 126 calculates luminance correction parameters using the information indicating how the luminance values ​​have changed (step Sz_3).

[0450] The interpretation unit 126 generates a predicted image for the current block by performing a brightness correction process on the reference image in the reference picture specified in MV, applying its brightness correction parameter (step Sz_4). In other words, the predicted image, which is the reference image in the reference picture specified in MV, is corrected based on the brightness correction parameter. This correction may involve brightness correction or color difference correction. Specifically, color difference correction parameters may be calculated using information indicating how the color difference has changed, and color difference correction processing may be performed.

[0451] Note that the shape of the peripheral reference region in Figure 66A is just one example, and other shapes may be used.

[0452] Furthermore, although this explanation describes the process of generating a predicted image from a single reference picture, the process is similar when generating predicted images from multiple reference pictures. Alternatively, the brightness correction process may be applied to each reference picture obtained from the reference picture in the same manner as described above before generating the predicted image.

[0453] One method for determining whether to apply LIC is to use a signal called lic_flag, which indicates whether or not to apply LIC. For example, in the encoding device 100, it is determined whether the current block belongs to a region where a brightness change is occurring. If it belongs to a region where a brightness change is occurring, the value of lic_flag is set to 1 and LIC is applied and encoding is performed. If it does not belong to a region where a brightness change is occurring, the value of lic_flag is set to 0 and encoding is performed without applying LIC. On the other hand, in the decoding device 200, the lic_flag written in the stream may be decoded, and the applicability of LIC may be switched according to its value before decoding.

[0454] Another method for determining whether to apply LIC is, for example, to determine whether LIC has been applied to the surrounding blocks. Specifically, if the current block is being processed in merge mode, the interpretation unit 126 determines whether the surrounding encoded blocks selected during MV derivation in merge mode were encoded with LIC. The interpretation unit 126 then switches whether to apply LIC and performs encoding based on the result. In this example, the same process is applied to the decoding device 200.

[0455] The LIC (Luminance Correction Processing) was explained using Figures 66A and 66B, but the details will be explained below.

[0456] First, the interpretation unit 126 derives an MV for obtaining the reference image corresponding to the current block from the reference picture, which is an encoded picture.

[0457] Next, the interpretation unit 126 extracts information indicating how the luminance values ​​have changed between the reference picture and the current picture, using the luminance pixel values ​​of the left-adjacent and upper-adjacent encoded peripheral reference regions and the luminance pixel values ​​at equivalent positions in the reference picture specified by MV, and calculates luminance correction parameters. For example, let p0 be the luminance pixel value of a pixel in the peripheral reference region of the current picture, and p1 be the luminance pixel value of a pixel in the peripheral reference region of the reference picture at an equivalent position to that pixel. The interpretation unit 126 calculates coefficients A and B as luminance correction parameters to optimize A×p1+B=p0 for multiple pixels in the peripheral reference region.

[0458] Next, the interpretation unit 126 generates a predicted image for the current block by performing a brightness correction process on the reference image in the reference picture specified by MV using a brightness correction parameter. For example, let p2 be the brightness pixel value in the reference image, and p3 be the brightness pixel value of the predicted image after brightness correction processing. The interpretation unit 126 generates the predicted image after brightness correction processing by calculating A × p2 + B = p3 for each pixel in the reference image.

[0459] Furthermore, a portion of the peripheral reference area shown in Figure 66A may be used. For example, a region containing a predetermined number of pixels obtained by thinning out the upper adjacent pixels and the left adjacent pixels may be used as the peripheral reference area. Also, the peripheral reference area is not limited to the region adjacent to the current block, but may be a region not adjacent to the current block. In the example shown in Figure 66A, the peripheral reference area in the reference picture is the region specified by the MV of the current picture, but it may be a region specified by another MV. For example, the other MV may be the MV of the peripheral reference area in the current picture.

[0460] Although the operation of the encoding device 100 has been described here, the operation of the decoding device 200 is similar.

[0461] Furthermore, LIC may be applied not only to luminance but also to chrominance. In this case, correction parameters may be derived individually for each of Y, Cb, and Cr, or a common correction parameter may be used for any of them.

[0462] Furthermore, LIC processing may be applied on a subblock basis. For example, correction parameters may be derived using the surrounding reference region of the current subblock and the surrounding reference region of the reference subblock within the reference picture specified by the MV of the current subblock.

[0463] [Prediction Control Unit] The prediction control unit 128 selects either an intra-prediction image (an image or signal output from the intra-prediction unit 124) or an inter-prediction image (an image or signal output from the inter-prediction unit 126), and outputs the selected prediction image to the subtraction unit 104 and the addition unit 116.

[0464] [Prediction parameter generation unit] The prediction parameter generation unit 130 may output information regarding intra-prediction, inter-prediction, and the selection of predicted images in the prediction control unit 128 as prediction parameters to the entropy coding unit 110. The entropy coding unit 110 may generate a stream based on the prediction parameters input from the prediction parameter generation unit 130 and the quantization coefficients input from the quantization unit 108. The prediction parameters may be used by the decoding device 200. The decoding device 200 may receive and decode the stream and perform the same processing as the prediction processing performed in the intra-prediction unit 124, inter-prediction unit 126, and prediction control unit 128. The prediction parameters may include a selected prediction signal (e.g., MV, prediction type, or prediction mode used in the intra-prediction unit 124 or inter-prediction unit 126), or arbitrary indices, flags, or values ​​that are based on or indicate the prediction processing performed in the intra-prediction unit 124, inter-prediction unit 126, and prediction control unit 128.

[0465] [Decoding device] Next, a decoding device 200 capable of decoding the stream output from the above-described encoding device 100 will be explained. Figure 67 is a block diagram showing an example of the configuration of the decoding device 200 according to the embodiment. The decoding device 200 is a device that decodes the encoded image stream in block units.

[0466] As shown in Figure 67, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a division decision unit 224. The intra prediction unit 216 and the inter prediction unit 218 are each configured as part of the prediction processing unit.

[0467] [Example of a decryption device implementation] Figure 68 is a block diagram showing an example implementation of the decoding device 200. The decoding device 200 comprises a processor b1 and memory b2. For example, several components of the decoding device 200 shown in Figure 67 are implemented by the processor b1 and memory b2 shown in Figure 68.

[0468] Processor b1 is a circuit that performs information processing and is a circuit that can access memory b2. For example, processor b1 is a dedicated or general-purpose electronic circuit for decoding streams. Processor b1 may be a processor such as a CPU. Alternatively, processor b1 may be a collection of multiple electronic circuits. Furthermore, for example, processor b1 may play the role of multiple components of the decoding device 200 shown in Figure 67, etc., excluding the component for storing information.

[0469] Memory b2 is a dedicated or general-purpose memory that stores information for processor b1 to decode the stream. Memory b2 may be an electronic circuit and may be connected to processor b1. Memory b2 may also be included in processor b1. Memory b2 may also be a collection of multiple electronic circuits. Memory b2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory b2 may also be non-volatile memory or volatile memory.

[0470] For example, memory b2 may store an image or a stream. Alternatively, memory b2 may store a program for processor b1 to decode the stream.

[0471] Furthermore, for example, memory b2 may play the role of an information storage component among the multiple components of the decoding device 200 shown in Figure 67, etc. Specifically, memory b2 may play the role of the block memory 210 and frame memory 214 shown in Figure 67. More specifically, reconstructed images (specifically, reconstructed blocks or reconstructed pictures, etc.) may be stored in memory b2.

[0472] Furthermore, the decoding device 200 does not necessarily have to implement all of the components shown in Figure 67, etc., nor does it have to perform all of the processes described above. Some of the components shown in Figure 67, etc., may be included in other devices, and some of the processes described above may be performed by other devices.

[0473] The following describes the overall processing flow of the decoding device 200, followed by a description of each component included in the decoding device 200. Note that detailed explanations of components included in the decoding device 200 that perform the same processing as those included in the encoding device 100 will be omitted. For example, the inverse quantization unit 204, inverse transform unit 206, adder unit 208, block memory 210, frame memory 214, intra prediction unit 216, inter prediction unit 218, prediction control unit 220, and loop filter unit 212 included in the decoding device 200 perform the same processing as the inverse quantization unit 112, inverse transform unit 114, adder unit 116, block memory 118, frame memory 122, intra prediction unit 124, inter prediction unit 126, prediction control unit 128, and loop filter unit 120 included in the encoding device 100.

[0474] [Overall flow of the decryption process] Figure 69 is a flowchart showing an example of the overall decoding process by the decoding device 200.

[0475] First, the partitioning determination unit 224 of the decoding device 200 determines the partitioning pattern for each of the multiple fixed-size blocks (128 × 128 pixels) contained in the picture based on the parameters input from the entropy decoding unit 202 (step Sp_1). This partitioning pattern is the partitioning pattern selected by the encoding device 100. Then, the decoding device 200 performs steps Sp_2 to Sp_6 for each of the multiple blocks that constitute that partitioning pattern.

[0476] The entropy decoding unit 202 decodes (specifically, entropy decoding) the encoded quantization coefficients and prediction parameters of the current block (step Sp_2).

[0477] Next, the inverse quantization unit 204 and the inverse transformation unit 206 reconstruct the predicted residual of the current block by performing inverse quantization and inverse transformation on multiple quantization coefficients (step Sp_3).

[0478] Next, the prediction processing unit, consisting of the intra-prediction unit 216, the inter-prediction unit 218, and the prediction control unit 220, generates a prediction image of the current block (step Sp_4).

[0479] Next, the summing unit 208 reconstructs the current block into a reconstructed image (also called a decoded image block) by adding the predicted image to the predicted residual (step Sp_5).

[0480] Then, once this reconstructed image is generated, the loop filter unit 212 performs filtering on the reconstructed image (step Sp_6).

[0481] The decryption device 200 then determines whether or not the entire picture has been decrypted (step Sp_7). If it determines that it has not been completed (No. in step Sp_7), it repeats the process from step Sp_1.

[0482] The processes in steps Sp_1 to Sp_7 may be performed sequentially by the decoding device 200, some of these processes may be performed in parallel, and the order may be changed.

[0483] [Partitioning decision section] Figure 70 shows the relationship between the division decision unit 224 and other components. The division decision unit 224 may perform the following processing as an example.

[0484] The division decision unit 224 collects block information from, for example, the block memory 210 or the frame memory 214, and further obtains parameters from the entropy decoding unit 202. The division decision unit 224 may then determine a division pattern for fixed-size blocks based on the block information and parameters. The division decision unit 224 may then output information indicating the determined division pattern to the inverse transform unit 206, the intra prediction unit 216, and the inter prediction unit 218. The inverse transform unit 206 may perform an inverse transform on the transformation coefficients based on the division pattern indicated by the information from the division decision unit 224. The intra prediction unit 216 and the inter prediction unit 218 may generate a predicted image based on the division pattern indicated by the information from the division decision unit 224.

[0485] [Entropy Decoder] Figure 71 is a block diagram showing an example of the configuration of the entropy decoding unit 202.

[0486] The entropy decoding unit 202 generates quantization coefficients, prediction parameters, and parameters related to the partition pattern by entropy decoding the stream. For example, CABAC is used for this entropy decoding. Specifically, the entropy decoding unit 202 comprises, for example, a binary arithmetic decoding unit 202a, a context control unit 202b, and a multileveling unit 202c. The binary arithmetic decoding unit 202a arithmetically decodes the stream into a binary signal using the context value derived by the context control unit 202b. Similar to the context control unit 110b of the encoding device 100, the context control unit 202b derives context values, i.e., the probability of a binary signal occurring, according to the characteristics of the syntax elements or the surrounding circumstances. The multileveling unit 202c performs debinarization, converting the binary signal output from the binary arithmetic decoding unit 202a into a multilevel signal showing the quantization coefficients and the like. This multileveling is performed according to the binaryization method described above.

[0487] The entropy decoding unit 202 outputs the quantization coefficients to the inverse quantization unit 204 in block units. The entropy decoding unit 202 may also output the prediction parameters included in the stream (see Figure 1) to the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220. The intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220 can perform the same prediction processing as the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 on the encoding device 100 side.

[0488] [Entropy Decoder] Figure 72 shows the CABAC flow in the entropy decoding unit 202.

[0489] First, the CABAC in the entropy decoding unit 202 is initialized. This initialization includes the initialization of the binary arithmetic decoding unit 202a and the setting of an initial context value. Then, the binary arithmetic decoding unit 202a and the multi-leveling unit 202c perform arithmetic decoding and multi-leveling on the encoded data of, for example, the CTU. At this time, the context control unit 202b updates the context value each time arithmetic decoding is performed. Then, as a post-processing step, the context control unit 202b saves the context value. This saved context value is used, for example, as the initial value of the context value for the next CTU.

[0490] [Dequantization section] The inverse quantization unit 204 inversely quantizes the quantization coefficients of the current block, which are input from the entropy decoding unit 202. Specifically, for each quantization coefficient of the current block, the inverse quantization unit 204 inversely quantizes the quantization coefficient based on the quantization parameter corresponding to that quantization coefficient. The inverse quantization unit 204 then outputs the inversely quantized quantization coefficients (i.e., transformation coefficients) of the current block to the inverse transformation unit 206.

[0491] Figure 73 is a block diagram showing an example of the configuration of the inverse quantization unit 204.

[0492] The inverse quantization unit 204 includes, for example, a quantization parameter generation unit 204a, a prediction quantization parameter generation unit 204b, a quantization parameter storage unit 204d, and an inverse quantization processing unit 204e.

[0493] Figure 74 is a flowchart showing an example of inverse quantization by the inverse quantization unit 204.

[0494] The inverse quantization unit 204 may, for example, perform inverse quantization processing for each CU based on the flow shown in Figure 74. Specifically, the quantization parameter generation unit 204a determines whether or not to perform inverse quantization (step Sv_11). If it is determined that inverse quantization should be performed (Yes in step Sv_11), the quantization parameter generation unit 204a obtains the difference quantization parameters of the current block from the entropy decoding unit 202 (step Sv_12).

[0495] Next, the predictive quantization parameter generation unit 204b obtains quantization parameters for a processing unit different from the current block from the quantization parameter storage unit 204d (step Sv_13). Based on the obtained quantization parameters, the predictive quantization parameter generation unit 204b generates predictive quantization parameters for the current block (step Sv_14).

[0496] Then, the quantization parameter generation unit 204a adds the difference quantization parameter of the current block obtained from the entropy decoding unit 202 and the predicted quantization parameter of the current block generated by the predicted quantization parameter generation unit 204b (step Sv_15). This addition generates the quantization parameter of the current block. The quantization parameter generation unit 204a then stores the quantization parameter of the current block in the quantization parameter storage unit 204d (step Sv_16).

[0497] Next, the inverse quantization processing unit 204e inversely quantizes the quantization coefficients of the current block into transformation coefficients using the quantization parameters generated in step Sv_15 (step Sv_17).

[0498] The differential quantization parameters may be decoded at the bit sequence level, picture level, slice level, brick level, or CTU level. The initial values ​​of the quantization parameters may also be decoded at the sequence level, picture level, slice level, brick level, or CTU level. In this case, the quantization parameters may be generated using the initial values ​​of the quantization parameters and the differential quantization parameters.

[0499] The inverse quantization unit 204 may be equipped with multiple inverse quantizers, and the quantization coefficients may be inversely quantized using an inverse quantization method selected from multiple inverse quantization methods.

[0500] [Inverse Transformation Section] The inverse transform unit 206 recovers the predicted residual by inversely transforming the transformation coefficients, which are input from the inverse quantization unit 204.

[0501] For example, if the information read from the stream indicates that EMT or AMT should be applied (e.g., the AMT flag is true), the inverse transformer 206 inversely transforms the transformation coefficients of the current block based on the information indicating the type of transformation read.

[0502] For example, if the information read from the stream indicates that NSST should be applied, the inverse conversion unit 206 applies inverse re-conversion to the conversion coefficients.

[0503] Figure 75 is a flowchart showing an example of processing by the inverse transform unit 206.

[0504] For example, the inverse transform unit 206 determines whether or not information indicating that an orthogonal transform is not performed exists in the stream (step St_11). If it determines that such information does not exist (No in step St_11), the inverse transform unit 206 obtains information indicating the transform type, which has been decoded by the entropy decoding unit 202 (step St_12). Next, the inverse transform unit 206 determines the transform type used for the orthogonal transform of the encoding device 100 based on that information (step St_13). Then, the inverse transform unit 206 performs an inverse orthogonal transform using the determined transform type (step St_14).

[0505] Figure 76 is a flowchart showing another example of processing by the inverse transform unit 206.

[0506] For example, the inverse transform unit 206 determines whether the transformation size is less than or equal to a predetermined value (step Su_11). If it determines that it is less than or equal to the predetermined value (Yes in step Su_11), the inverse transform unit 206 obtains information from the entropy decoding unit 202 indicating which of the one or more transformation types included in the first transformation type group was used by the encoding device 100 (step Su_12). This information is decoded by the entropy decoding unit 202 and output to the inverse transform unit 206.

[0507] Based on this information, the inverse transform unit 206 determines the type of transformation used in the orthogonal transformation in the encoding device 100 (step Su_13). Then, the inverse transform unit 206 performs an inverse orthogonal transformation on the transformation coefficients of the current block using the determined transformation type (step Su_14). On the other hand, if the inverse transform unit 206 determines in step Su_11 that the transformation size is not less than or equal to a predetermined value (No. in step Su_11), it performs an inverse orthogonal transformation on the transformation coefficients of the current block using a second group of transformation types (step Su_15).

[0508] The inverse orthogonal transformation by the inverse transformation unit 206 may, for example, be performed for each TU according to the flow shown in Figure 75 or Figure 76. Alternatively, the inverse orthogonal transformation may be performed using a predetermined transformation type without decoding the information indicating the transformation type used in the orthogonal transformation. Specifically, the transformation type may be DST7 or DCT8, and the inverse transformation basis function corresponding to that transformation type is used in the inverse orthogonal transformation.

[0509] [Addition section] The summing unit 208 reconstructs the current block by adding the predicted residual, which is the input from the inverse transform unit 206, and the predicted image, which is the input from the prediction control unit 220. In other words, a reconstructed image of the current block is generated. The summing unit 208 then outputs the reconstructed image of the current block to the block memory 210 and the loop filter unit 212.

[0510] [Block memory] The block memory 210 is a block referenced in intra prediction and is a storage unit for storing blocks within the current picture. Specifically, the block memory 210 stores the reconstructed image output from the adder 208.

[0511] [Loop Filter Section] The loop filter unit 212 applies a loop filter to the reconstructed image generated by the adder unit 208, and outputs the filtered reconstructed image to the frame memory 214 and a display device, etc.

[0512] If the information interpreted from the stream indicating ALF on / off indicates ALF is on, one filter is selected from several filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed image.

[0513] Figure 77 is a block diagram showing an example of the configuration of the loop filter unit 212. Note that the loop filter unit 212 has the same configuration as the loop filter unit 120 of the encoding device 100.

[0514] The loop filter unit 212 includes, for example, a deblocking filter processing unit 212a, an SAO processing unit 212b, and an ALF processing unit 212c, as shown in Figure 77. The deblocking filter processing unit 212a applies the above-described deblocking filter processing to the reconstructed image. The SAO processing unit 212b applies the above-described SAO processing to the reconstructed image after the deblocking filter processing. The ALF processing unit 212c applies the above-described ALF processing to the reconstructed image after the SAO processing. Note that the loop filter unit 212 does not have to include all the processing units disclosed in Figure 77, and may include only some of the processing units. Also, the loop filter unit 212 may be configured to perform the above-described processing in an order different from the processing order disclosed in Figure 77.

[0515] [Frame memory] The frame memory 214 is a memory unit for storing reference pictures used in interpretation, and is sometimes called a frame buffer. Specifically, the frame memory 214 stores the reconstructed image that has been filtered by the loop filter unit 212.

[0516] [Prediction Unit (Intra Prediction Unit, Inter Prediction Unit, Prediction Control Unit)] Figure 78 is a flowchart showing an example of processing performed in the prediction unit of the decoding device 200. For example, the prediction unit consists of all or some of the components of the intra-prediction unit 216, the inter-prediction unit 218, and the prediction control unit 220. The prediction processing unit includes, for example, the intra-prediction unit 216 and the inter-prediction unit 218.

[0517] The prediction unit generates a predicted image of the current block (step Sq_1). This predicted image is also called a predicted signal or predicted block. The predicted signal may include, for example, an intra-prediction signal or an inter-prediction signal. Specifically, the prediction unit generates the predicted image of the current block using a reconstructed image already obtained by generating predicted images for other blocks, restoring prediction residuals, and adding predicted images. The prediction unit of the decoder 200 generates the same predicted image as the prediction unit of the encoder 100. In other words, the methods for generating predicted images used by these prediction units are common to or correspond to each other.

[0518] The reconstructed image may be, for example, the image of the reference picture, or it may be the image of the decoded block (i.e., the other block mentioned above) within the current picture, which is the picture containing the current block. The decoded block within the current picture is, for example, the adjacent block to the current block.

[0519] Figure 79 is a flowchart showing another example of the processing performed in the prediction unit of the decoding device 200.

[0520] The prediction unit determines a method or mode for generating the predicted image (step Sr_1). For example, this method or mode may be determined based on prediction parameters, for example.

[0521] If the prediction unit determines a first method as the mode for generating the predicted image, it generates the predicted image according to that first method (step Sr_2a). If the prediction unit determines a second method as the mode for generating the predicted image, it generates the predicted image according to that second method (step Sr_2b). If the prediction unit determines a third method as the mode for generating the predicted image, it generates the predicted image according to that third method (step Sr_2c).

[0522] The first, second, and third methods are different methods for generating predictive images, and may be, for example, an interpretation method, an intrapretation method, and other prediction methods, respectively. These prediction methods may use the reconstructed images described above.

[0523] Figures 80A and 80B are flowcharts showing other examples of processing performed in the prediction unit of the decoding device 200.

[0524] The prediction unit may perform prediction processing according to the flow shown in Figures 80A and 80B as an example. The intrablock copy shown in Figures 80A and 80B is one mode belonging to interprediction, in which blocks included in the current picture are referenced as reference images or reference blocks. In other words, in intrablock copy, pictures different from the current picture are not referenced. The PCM mode shown in Figure 80A is also one mode belonging to intraprediction, in which no transformation or quantization is performed.

[0525] [Intra Prediction Unit] The intra-prediction unit 216 generates a predicted image of the current block (i.e., an intra-predicted image) by performing intra-prediction based on the intra-prediction mode decoded from the stream, by referring to the blocks in the current picture stored in the block memory 210. Specifically, the intra-prediction unit 216 generates an intra-predicted image by performing intra-prediction by referring to the pixel values ​​(e.g., luminance values, chrominance values) of blocks adjacent to the current block, and outputs the intra-predicted image to the prediction control unit 220.

[0526] Furthermore, if an intra-prediction mode that references a luminance block is selected in the intra-prediction of a color difference block, the intra-prediction unit 216 may predict the color difference component of the current block based on the luminance component of the current block.

[0527] Furthermore, if the information interpreted from the stream indicates the application of PDPC, the intra-prediction unit 216 corrects the pixel value after intra-prediction based on the gradient of the reference pixels in the horizontal / vertical directions.

[0528] Figure 81 shows an example of processing by the intra-prediction unit 216 of the decoding device 200.

[0529] First, the intra-prediction unit 216 determines whether or not an MPM flag indicating 1 exists in the stream (step Sw_11). If it determines that an MPM flag indicating 1 exists (Yes in step Sw_11), the intra-prediction unit 216 obtains information from the entropy decoding unit 202 indicating the intra-prediction mode selected by the encoding device 100 from the MPM (step Sw_12). This information is decoded by the entropy decoding unit 202 and output to the intra-prediction unit 216. Next, the intra-prediction unit 216 determines the MPM (step Sw_13). The MPM consists of, for example, six intra-prediction modes. Then, the intra-prediction unit 216 determines the intra-prediction mode indicated by the information obtained in step Sw_12 from among the multiple intra-prediction modes included in the MPM (step Sw_14).

[0530] Meanwhile, if the intra-prediction unit 216 determines in step Sw_11 that there is no MPM flag indicating 1 in the stream (No. in step Sw_11), it obtains information indicating the intra-prediction mode selected by the encoding device 100 (step Sw_15). In other words, the intra-prediction unit 216 obtains information from the entropy decoding unit 202 indicating the intra-prediction mode selected by the encoding device 100 from among one or more intra-prediction modes not included in the MPM. This information is decoded by the entropy decoding unit 202 and output to the intra-prediction unit 216. The intra-prediction unit 216 then determines the intra-prediction mode indicated by the information obtained in step Sw_15 from among the one or more intra-prediction modes not included in the MPM (step Sw_17).

[0531] The intra-prediction unit 216 generates a predicted image according to the intra-prediction mode determined in step Sw_14 or step Sw_17 (step Sw_18).

[0532] [International Prediction Department] The interpretation unit 218 predicts the current block by referring to a reference picture stored in the frame memory 214. Prediction is performed in units of the current block or subblocks within the current block. A subblock is a unit contained within a block and is smaller than a block. The size of a subblock may be 4x4 pixels, 8x8 pixels, or other sizes. The size of a subblock may be switched to units such as slices, bricks, or pictures.

[0533] For example, the interpretation unit 218 generates an interpretation image of the current block or subblock by performing motion compensation using motion information (e.g., MV) decoded from the stream (e.g., prediction parameters output from the entropy decoding unit 202), and outputs the interpretation image to the prediction control unit 220.

[0534] If the information read from the stream indicates that OBMC mode should be applied, the interpretation unit 218 generates an interpretation prediction image using not only the motion information of the current block obtained by motion search, but also the motion information of adjacent blocks.

[0535] Furthermore, if the information deciphered from the stream indicates that FRUC mode should be applied, the interpretation unit 218 derives motion information by performing a motion search according to the pattern matching method (bilateral matching or template matching) deciphered from the stream. Then, the interpretation unit 218 performs motion compensation (prediction) using the derived motion information.

[0536] Furthermore, when the BIO mode is applied, the interpretation unit 218 derives the MV based on a model that assumes uniform linear motion. Also, if the information read from the stream indicates that the affine mode is applied, the interpretation unit 218 derives the MV on a sub-block basis based on the MVs of multiple adjacent blocks.

[0537] [MV Derivation Flow] Figure 82 is a flowchart showing an example of MV derivation in the decoding device 200.

[0538] The interpretation unit 218 determines, for example, whether or not to decode motion information (e.g., MV). For example, the interpretation unit 218 may make this determination according to the prediction mode included in the stream, or it may make this determination based on other information included in the stream. Here, if the interpretation unit 218 determines to decode the motion information, it derives the MV of the current block in the mode for decoding the motion information. On the other hand, if the interpretation unit 218 determines not to decode the motion information, it derives the MV in the mode for not decoding the motion information.

[0539] Here, the modes for MV derivation include the normal inter mode, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that decode motion information include the normal inter mode, normal merge mode, and affine mode (specifically, affine inter mode and affine merge mode). Note that the motion information may include not only MV but also predicted MV selection information, which will be described later. Modes that do not decode motion information include the FRUC mode, etc. The inter prediction unit 218 selects a mode from these multiple modes for deriving the MV of the current block and derives the MV of the current block using the selected mode.

[0540] Figure 83 is a flowchart showing another example of MV derivation in the decoding device 200.

[0541] The interpretation unit 218 determines, for example, whether or not to decode the differential MV. For example, the interpretation unit 218 may make this determination according to the prediction mode included in the stream, or it may make this determination based on other information included in the stream. Here, if the interpretation unit 218 determines to decode the differential MV, it may derive the MV of the current block in the mode for decoding the differential MV. In this case, for example, the differential MV included in the stream is decoded as a prediction parameter.

[0542] On the other hand, if the interpretation unit 218 determines that it does not decode the differential MV, it derives the MV in a mode that does not decode the differential MV. In this case, the encoded differential MV is not included in the stream.

[0543] As mentioned above, the modes for MV derivation include the normal inter, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that encode differential MV include the normal inter mode and affine mode (specifically, affine inter mode). Modes that do not encode differential MV include the FRUC mode, normal merge mode, and affine mode (specifically, affine merge mode). The inter prediction unit 218 selects a mode from these multiple modes to derive the MV of the current block, and uses the selected mode to derive the MV of the current block.

[0544] [MV Derivation > Normal Intermode] For example, if the information interpreted from the stream indicates that the normal intermode should be applied, the interpretation unit 218 derives the MV in normal merge mode based on the information interpreted from the stream and uses that MV to perform motion compensation (prediction).

[0545] Figure 84 is a flowchart showing an example of inter-mode prediction in the decoding device 200.

[0546] The interpretation unit 218 of the decoding device 200 performs motion compensation for each block. In this process, the interpretation unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks surrounding the current block in time or space (step Sg_11). In other words, the interpretation unit 218 creates a candidate MV list.

[0547] Next, the interpretation unit 218 extracts N candidate MVs (where N is an integer greater than or equal to 2) from the multiple candidate MVs obtained in step Sg_11 as predicted motion vector candidates (also called predicted MV candidates) according to a predetermined priority order (step Sg_12). The priority order is predetermined for each of the N predicted MV candidates.

[0548] Next, the interpretation unit 218 decodes the predicted MV selection information from the input stream and uses the decoded predicted MV selection information to select one predicted MV candidate from the N predicted MV candidates as the predicted MV for the current block (step Sg_13).

[0549] Next, the interpretation unit 218 decodes the differential MV from the input stream and derives the MV of the current block by adding the decoded differential MV (the difference value) to the selected predicted MV (step Sg_14).

[0550] Finally, the inter-prediction unit 218 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Sg_15). The processes in steps Sg_11 to Sg_15 are performed for each block. For example, when the processes in steps Sg_11 to Sg_15 are performed for each block contained in a slice, the inter-prediction using the normal inter-mode for that slice is completed. Similarly, when the processes in steps Sg_11 to Sg_15 are performed for each block contained in a picture, the inter-prediction using the normal inter-mode for that picture is completed. Note that the processes in steps Sg_11 to Sg_15 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using the normal inter-mode for that slice is completed. Likewise, when the processes in steps Sg_11 to Sg_15 are performed for some blocks contained in a picture, the inter-prediction using the normal inter-mode for that picture is completed.

[0551] [MV Derivation > Normal Merge Mode] For example, if the information read from the stream indicates the application of normal merge mode, the interpretation unit 218 derives MV in normal merge mode and uses that MV to perform motion compensation (prediction).

[0552] Figure 85 is a flowchart showing an example of inter prediction using normal merge mode in the decoding device 200.

[0553] The interpretation unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks surrounding the current block in terms of time or space (step Sh_11). In other words, the interpretation unit 218 creates a candidate MV list.

[0554] Next, the interpretation unit 218 derives the MV of the current block by selecting one candidate MV from among the multiple candidate MVs obtained in step Sh_11 (step Sh_12). Specifically, the interpretation unit 218 obtains, for example, MV selection information included in the stream as prediction parameters, and selects the candidate MV identified by that MV selection information as the MV of the current block.

[0555] Finally, the inter-prediction unit 218 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Sh_13). The processes in steps Sh_11 to Sh_13 are performed for each block, for example. For example, when the processes in steps Sh_11 to Sh_13 are performed for each of the blocks contained in a slice, the inter-prediction using normal merge mode for that slice is completed. Similarly, when the processes in steps Sh_11 to Sh_13 are performed for each of the blocks contained in a picture, the inter-prediction using normal merge mode for that picture is completed. Note that the processes in steps Sh_11 to Sh_13 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using normal merge mode for that slice is completed. Likewise, when the processes in steps Sh_11 to Sh_13 are performed for some of the blocks contained in a picture, the inter-prediction using normal merge mode for that picture is completed.

[0556] [MV Derivation > FRUC Mode] For example, if the information decoded from the stream indicates the application of FRUC mode, the interpretation unit 218 derives the MV in FRUC mode and uses that MV to perform motion compensation (prediction). In this case, the motion information is not signaled from the encoding device 100 but is derived from the decoding device 200. For example, the decoding device 200 may derive the motion information by performing a motion search. In this case, the decoding device 200 performs a motion search without using the pixel values ​​of the current block.

[0557] Figure 86 is a flowchart showing an example of interpretation using FRUC mode in the decoding device 200.

[0558] First, the interpretation unit 218 refers to the MVs of each decoded block that is spatially or temporally adjacent to the current block and generates a list of candidate MVs (i.e., a candidate MV list, which may be the same as the candidate MV list for normal merge mode) (step Si_11). Next, the interpretation unit 218 selects the best candidate MV from among the multiple candidate MVs registered in the candidate MV list (step Si_12). For example, the interpretation unit 218 calculates an evaluation value for each candidate MV included in the candidate MV list and selects one candidate MV as the best candidate MV based on that evaluation value. Then, the interpretation unit 218 derives the MV for the current block based on the selected best candidate MV (step Si_14). Specifically, for example, the selected best candidate MV is directly derived as the MV for the current block. Alternatively, for example, the MV for the current block may be derived by performing pattern matching in the area surrounding the position in the reference picture corresponding to the selected best candidate MV. In other words, a search is performed in the area surrounding the best candidate MV using pattern matching and evaluation values ​​in the reference picture. If an MV with a better evaluation value is found, the best candidate MV may be updated to that MV and made the final MV of the current block. It is not necessary to update to an MV with a better evaluation value.

[0559] Finally, the inter-prediction unit 218 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Si_15). The processes in steps Si_11 to Si_15 are performed for each block, for example. For example, when the processes in steps Si_11 to Si_15 are performed for each of the blocks contained in a slice, the inter-prediction using FRUC mode for that slice is completed. Similarly, when the processes in steps Si_11 to Si_15 are performed for each of the blocks contained in a picture, the inter-prediction using FRUC mode for that picture is completed. Sub-blocks may also be processed in the same way as the block units described above.

[0560] [MV Derivation > Affine Merge Mode] For example, if the information deciphered from the stream indicates the application of affine merge mode, the interpretation unit 218 derives the motion video (MV) in affine merge mode and uses that MV to perform motion compensation (prediction).

[0561] Figure 87 is a flowchart showing an example of interpretation using the affine merge mode in the decoding device 200.

[0562] In affine merge mode, the interpretation unit 218 first derives the MV of each control point of the current block (step Sk_11). The control points are the upper left and upper right corners of the current block, as shown in Figure 46A, or the upper left, upper right, and lower left corners of the current block, as shown in Figure 46B.

[0563] For example, when using the MV derivation method shown in Figures 47A to 47C, the interpretation unit 218 examines the decoded blocks in the order of Block A (left), Block B (top), Block C (upper right), Block D (lower left), and Block E (upper left), as shown in Figure 47A, and identifies the first valid block decoded in affine mode.

[0564] The interpretation unit 218 derives the motion vectors (MV) of the control points using the first valid block decoded in the identified affine mode. For example, if block A is identified and block A has two control points, as shown in Figure 47B, the interpretation unit 218 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block by projecting the motion vectors v3 and v4 of the upper left and upper right corners of the decoded block containing block A onto the current block. This derives the MV of each control point.

[0565] Furthermore, as shown in Figure 49A, if block A is identified and block A has two control points, the MV of three control points may be calculated. Alternatively, as shown in Figure 49B, if block A is identified and block A has three control points, the MV of two control points may be calculated.

[0566] Furthermore, if the stream includes MV selection information as a prediction parameter, the interpretation unit 218 may use that MV selection information to derive the MV of each control point in the current block.

[0567] Next, the interpretation unit 218 performs motion compensation for each of the multiple subblocks contained in the current block. That is, for each of the multiple subblocks, the interpretation unit 218 calculates the MV of that subblock as an affine MV using two motion vectors v0 and v1 and equation (1A) described above, or using three motion vectors v0, v1 and v2 and equation (1B) described above (step Sk_12). Then, the interpretation unit 218 performs motion compensation for that subblock using those affine MVs and the decoded reference picture (step Sk_13). Once steps Sk_12 and Sk_13 have been executed for each of the subblocks contained in the current block, the interpretation using the affine merge mode for that current block is completed. In other words, motion compensation is performed for the current block, and a predicted image of that current block is generated.

[0568] In step Sk_11, the above-mentioned candidate MV list may be generated. The candidate MV list may be a list containing candidate MVs derived for each control point using multiple MV derivation methods. The multiple MV derivation methods may be any combination of the MV derivation methods shown in Figures 47A to 47C, the MV derivation methods shown in Figures 48A and 48B, the MV derivation methods shown in Figures 49A and 49B, and other MV derivation methods.

[0569] The candidate MV list may also include candidate MVs from modes other than affine mode, where prediction is performed on a sub-block basis.

[0570] Furthermore, the candidate MV list may include, for example, candidate MVs for an affine merge mode with two control points and candidate MVs for an affine merge mode with three control points. Alternatively, a candidate MV list containing candidate MVs for an affine merge mode with two control points and a candidate MV list containing candidate MVs for an affine merge mode with three control points may be generated separately. Alternatively, a candidate MV list may be generated containing candidate MVs for one of the modes: an affine merge mode with two control points or an affine merge mode with three control points.

[0571] [MV Derivation > Affine Intermode] For example, if the information interpreted from the stream indicates the application of affine intermode, the interpretation unit 218 derives the motion video (MV) using affine intermode and performs motion compensation (prediction) using that MV.

[0572] Figure 88 is a flowchart showing an example of inter-prediction using affine intermode in the decoding device 200.

[0573] In affine intermode, the interpretation unit 218 first derives the predicted MV(v0,v1) or (v0,v1,v2) for each of two or three control points of the current block (step Sj_11). The control points are, for example, the upper left corner, upper right corner, or lower left corner of the current block, as shown in Figure 46A or Figure 46B.

[0574] The interpretation unit 218 acquires predicted MV selection information included in the stream as prediction parameters, and uses the MVs identified by the predicted MV selection information to derive the predicted MV for each control point in the current block. For example, when using the MV derivation method shown in Figures 48A and 48B, the interpretation unit 218 derives the predicted MV (v0,v1) or (v0,v1,v2) for the control point of the current block by selecting the MV of the block identified by the predicted MV selection information from among the decoded blocks near each control point in the current block shown in Figure 48A or Figure 48B.

[0575] Next, the interpretation unit 218 obtains, for example, each differential MV included in the stream as a prediction parameter, and adds the predicted MV of each control point in the current block to the differential MV corresponding to that predicted MV (step Sj_12). This derives the MV of each control point in the current block.

[0576] Next, the interpretation unit 218 performs motion compensation for each of the multiple subblocks contained in the current block. That is, for each of the multiple subblocks, the interpretation unit 218 calculates the MV of that subblock as an affine MV using two motion vectors v0 and v1 and equation (1A) described above, or using three motion vectors v0, v1 and v2 and equation (1B) described above (step Sj_13). Then, the interpretation unit 218 performs motion compensation for that subblock using those affine MVs and the decoded reference picture (step Sj_14). Once steps Sj_13 and Sj_14 have been executed for each of the subblocks contained in the current block, the interpretation using the affine merge mode for that current block is completed. In other words, motion compensation is performed for the current block, and a predicted image of that current block is generated.

[0577] In step Sj_11, the candidate MV list described above may be generated, similar to step Sk_11.

[0578] [MV Derivation > Triangle Mode] For example, if the information interpreted from the stream indicates the application of triangle mode, the interpretation unit 218 derives MV in triangle mode and uses that MV to perform motion compensation (prediction).

[0579] Figure 89 is a flowchart showing an example of inter prediction using triangle mode in the decoding device 200.

[0580] In triangle mode, the interpretation unit 218 first divides the current block into a first partition and a second partition (step Sx_11). At this time, the interpretation unit 218 may obtain partition information, which is information about the division into each partition, from the stream as prediction parameters. Then, the interpretation unit 218 may divide the current block into a first partition and a second partition according to that partition information.

[0581] Next, the interpretation unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks surrounding the current block in time or space (step Sx_12). In other words, the interpretation unit 218 creates a candidate MV list.

[0582] Then, the interpretation unit 218 selects the candidate MV for the first partition and the candidate MV for the second partition from among the multiple candidate MVs obtained in step Sx_11 as the first MV and the second MV, respectively (step Sx_13). At this time, the interpretation unit 218 may obtain MV selection information to identify the selected candidate MVs from the stream as prediction parameters. Then, the interpretation unit 218 may select the first MV and the second MV according to that MV selection information.

[0583] Next, the interpretation unit 218 generates a first predicted image by performing motion compensation using the selected first MV and the decoded reference picture (step Sx_14). Similarly, the interpretation unit 218 generates a second predicted image by performing motion compensation using the selected second MV and the decoded reference picture (step Sx_15).

[0584] Finally, the interpretation unit 218 generates a predicted image of the current block by weighting and adding the first predicted image and the second predicted image (step Sx_16).

[0585] [Motion Detection > DMVR] For example, if the information deciphered from the stream indicates the application of DMVR, the interpretation unit 218 performs motion search using DMVR.

[0586] Figure 90 is a flowchart showing an example of motion detection using DMVR in the decoding device 200.

[0587] The interpretation unit 218 first derives the MV of the current block in merge mode (step Sl_11). Next, the interpretation unit 218 derives the final MV for the current block by searching the surrounding region of the reference picture indicated by the MV derived in step Sl_11 (step Sl_12). In other words, the MV of the current block is determined by the DMVR.

[0588] Figure 91 is a flowchart showing a detailed example of motion detection by the DMVR in the decoding device 200.

[0589] First, in Step 1 shown in Figure 58A, the interpretation unit 218 calculates the cost of the search position indicated by the initial MV (also called the starting point) and the eight surrounding search positions. Then, the interpretation unit 218 determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 218 determines that the cost of the search positions other than the starting point is the minimum, it moves to the search position with the minimum cost and performs the process shown in Figure 58A in Step 2. On the other hand, if the cost of the starting point is the minimum, the interpretation unit 218 skips the process shown in Figure 58A in Step 2 and proceeds to the process shown in Step 3.

[0590] In Step 2, shown in Figure 58A, the interpretation unit 218 uses the search position moved according to the processing result of Step 1 as a new starting point and performs a search similar to the process in Step 1. The interpretation unit 218 then determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 218 determines that the cost of the search positions other than the starting point is the minimum, it proceeds to Step 4. On the other hand, if the interpretation unit 218 determines that the cost of the starting point is the minimum, it proceeds to Step 3.

[0591] In Step 4, the interpretation unit 218 treats the starting position as the final search position and determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0592] In Step 3 shown in Figure 58A, the interpretation unit 218 determines the pixel position with the minimum cost based on the costs of the four points above, below, to the left and right of the starting point in Step 1 or Step 2, and sets that pixel position as the final search position. This decimal-precision pixel position is determined by weighting the vector of the four points above, below, to the left and right ((0,1),(0,-1),(-1,0),(1,0)) with the cost of each of the four search positions as the weight. The interpretation unit 218 then determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0593] [Motion compensation > BIO / OBMC / LIC] For example, if the information read from the stream indicates the application of correction to the predicted image, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to the correction mode. These modes include, for example, the BIO, OBMC, and LIC mentioned above.

[0594] Figure 92 is a flowchart showing an example of predictive image generation in the decoding device 200.

[0595] The interpretation unit 218 generates a predicted image (step Sm_11) and corrects the predicted image using one of the modes described above (step Sm_12).

[0596] Figure 93 is a flowchart showing another example of predictive image generation in the decoding device 200.

[0597] The interpretation unit 218 derives the MV of the current block (step Sn_11). Next, the interpretation unit 218 generates a predicted image using the MV (step Sn_12) and determines whether or not to perform correction processing (step Sn_13). For example, the interpretation unit 218 obtains prediction parameters included in the stream and determines whether or not to perform correction processing based on these prediction parameters. These prediction parameters are, for example, flags indicating whether or not to apply each of the above modes. If the interpretation unit 218 determines that correction processing should be performed (Yes in step Sn_13), it generates the final predicted image by correcting the predicted image (step Sn_14). In LIC, the brightness and color difference of the predicted image may be corrected in step Sn_14. On the other hand, if the interpretation unit 218 determines that no correction processing should be performed (No in step Sn_13), it outputs the predicted image as the final predicted image without correction (step Sn_15).

[0598] [Motion Compensation > OBMC] For example, if the information interpreted from the stream indicates the application of OBMC, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to OBMC.

[0599] Figure 94 is a flowchart showing an example of predictive image correction by OBMC in the decoding device 200. The flowchart in Figure 94 shows the process of predictive image correction using the current picture and reference picture shown in Figure 62.

[0600] First, the interpretation unit 218 acquires a predicted image (Pred) using normal motion compensation with the MV assigned to the current block, as shown in Figure 62.

[0601] Next, the interpretation unit 218 applies (reuses) the already derived MV (MV_L) for the decoded left adjacent block to the current block to obtain a predicted image (Pred_L). Then, the interpretation unit 218 performs the first correction of the predicted image by superimposing the two predicted images, Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.

[0602] Similarly, the interpretation unit 218 applies (reuses) the already derived MV (MV_U) for the decoded upper adjacent block to the current block to obtain a predicted image (Pred_U). Then, the interpretation unit 218 performs a second correction of the predicted image by superimposing the predicted image Pred_U onto the first corrected predicted image (e.g., Pred and Pred_L). This has the effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is the final predicted image of the current block, with the boundaries with adjacent blocks blended (smoothed).

[0603] [Motion Compensation > BIO] For example, if the information interpreted from the stream indicates the application of BIO, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to BIO.

[0604] Figure 95 is a flowchart showing an example of BIO correction of the predicted image in the decoding device 200.

[0605] As shown in Figure 63, the interpretation unit 218 derives two motion vectors (M0, M1) using two reference pictures (Ref0, Ref1) that are different from the picture (Cur Pic) containing the current block. Then, the interpretation unit 218 derives a predicted image of the current block using these two motion vectors (M0, M1) (step Sy_11). Motion vector M0 is the motion vector (MVx0, MVy0) corresponding to reference picture Ref0, and motion vector M1 is the motion vector (MVx1, MVy1) corresponding to reference picture Ref1.

[0606] Next, the interpretation unit 218 interpolates the current block image I using the motion vector M0 and the reference picture L0. 0 The following is derived. Furthermore, the interpretation unit 218 uses the motion vector M1 and the reference picture L1 to interpolate the current block image I 1 Derive the following (step Sy_12). Here, interpolated image I 0 This is the image contained in the reference picture Ref0, which is derived for the current block, and is the interpolated image I 1 This is the image contained in reference picture Ref1, which is derived for the current block. Interpolated image I 0 and interpolated image I 1 Each of these may be the same size as the current block. Alternatively, interpolated image I 0 and interpolated image I 1 Each of these may be a larger image than the current block in order to properly derive the gradient image described later. Furthermore, interpolated image I 0 and I 1 This may include a motion vector (M0, M1) and a reference picture (L0, L1), and a predicted image derived by applying a motion compensation filter.

[0607] Furthermore, the interpolation prediction unit 218 generates interpolated image I 0 and interpolated image I 1 From, the gradient image of the current block (Ix 0 ,Ix 1 Iy 0 Iy 1 ) is derived (step Sy_13). Note that the horizontal gradient image is (Ix 0 ,Ix 1 ) and the vertical gradient image is (Iy 0 Iy 1 The interpretation unit 218 may, for example, derive a gradient image by applying a gradient filter to the interpolated image. The gradient image only needs to show the amount of spatial change in pixel values ​​along the horizontal or vertical direction.

[0608] Next, the interpretation unit 218 interpolates the image (I) in units of multiple subblocks that constitute the current block. 0 ,I 1 ) and gradient image (Ix 0 ,Ix 1 Iy 0 Iy 1 The optical flow (vx, vy), which is the velocity vector described above, is derived using (step Sy_14). As an example, the subblock may be a 4x4 pixel subCU.

[0609] Next, the interpretation unit 218 corrects the predicted image of the current block using optical flow (vx, vy). For example, the interpretation unit 218 derives correction values ​​for the pixel values ​​included in the current block using optical flow (vx, vy) (step Sy_15). The interpretation unit 218 may then correct the predicted image of the current block using the correction values ​​(step Sy_16). The correction values ​​may be derived for each pixel, or for multiple pixels or subblocks.

[0610] Note that the BIO processing flow is not limited to the processing disclosed in Figure 95. Only a portion of the processing disclosed in Figure 95 may be performed, different processing may be added or replaced, or the processing may be executed in a different order.

[0611] [Motion Compensation > LIC] For example, if the information interpreted from the stream indicates the application of LIC, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to LIC.

[0612] Figure 96 is a flowchart showing an example of image correction by LIC in the decoding device 200.

[0613] First, the interpretation unit 218 uses MV to obtain the reference image corresponding to the current block from the decoded reference picture (step Sz_11).

[0614] Next, the interpretation unit 218 extracts information for the current block indicating how the luminance values ​​have changed between the reference picture and the current picture (step Sz_12). This extraction is performed based on the luminance pixel values ​​of the decoded left adjacent reference region (surrounding reference region) and the decoded upper adjacent reference region (surrounding reference region) in the current picture, as shown in Figure 66A, and the luminance pixel values ​​at the equivalent positions in the reference picture specified by the derived MV. Then, the interpretation unit 218 calculates luminance correction parameters using the information indicating how the luminance values ​​have changed (step Sz_13).

[0615] The interpretation unit 218 generates a predicted image for the current block by performing a brightness correction process on the reference image in the reference picture specified in MV, applying its brightness correction parameter (step Sz_14). In other words, the predicted image, which is the reference image in the reference picture specified in MV, is corrected based on the brightness correction parameter. This correction may involve brightness correction or color difference correction.

[0616] [Prediction Control Unit] The prediction control unit 220 selects either an intra-predicted image or an inter-predicted image and outputs the selected prediction image to the summing unit 208. Overall, the configuration, functions, and processing of the prediction control unit 220, intra-prediction unit 216, and inter-prediction unit 218 on the decoding device 200 side may correspond to the configuration, functions, and processing of the prediction control unit 128, intra-prediction unit 124, and inter-prediction unit 126 on the encoding device 100 side.

[0617] [CPB Removal Time, DPB Output time] First, let's explain the CPB (Coded Picture Buffer) Removal time and the DPB (Decoded Picture Buffer) Output time.

[0618] Figure 97 shows an example of the syntax of a buffering period SEI (Supplemental Enhancement Information) message (BP SEI). The buffering period SEI message notifies the reference time used to determine the decryption time of an access unit (AU) (also called CPB Removal Time, which is the time when the AU's data is extracted from the CPB) or the display time of an AU (also called DPB Output Time, which is the time when the AU's data is output from the DPB). For example, the reference time is determined for each sublayer by a predetermined procedure using bp_nal_initial_cpb_removal_delay or bp_nal_initial_cpb_removal_offset in Figure 97.

[0619] As shown in the example in Figure 97, the buffering period SEI message may also notify parameters (such as bp_cpb_removal_delay_delta_val or bp_dpb_output_tid_offset) used to determine the decoding and display times of multiple AUs in the bitstream.

[0620] Figure 98 shows an example of the syntax of a picture timing SEI message (PT SEI). The picture timing SEI message notifies information (such as pt_cpb_removal_delay_minus1 or pt_dpb_output_delay) regarding the relative time (CPB Removal Delay, DPB Output Delay) from a reference time determined based on the corresponding buffering period SEI message to the corresponding AU's decoding time or display time.

[0621] As shown in the example in Figure 98, the picture timing SEI message may also notify multiple AUs in the bitstream of parameters (such as pt_cpb_removal_delay_delta_idx) used to determine the CPB Removal Delay or DPB Output Delay.

[0622] For example, bp_cpb_removal_delay_delta_val may be selected according to pt_cpb_removal_delay_delta_idx, and the CPB Removal Delay of the corresponding AU may be determined by a predetermined method. Alternatively, the DPB Output Delay of each sublayer of the corresponding AU may be determined by a predetermined method based on pt_dpb_output_delay and bp_dpb_output_tid_offset.

[0623] [DPB parameters] Next, we will explain the parameters related to DPB (DPB). Figure 99 shows an example of the DPB parameter syntax. The DPB parameter syntax notifies each sublayer of information regarding the DPB size (max_dec_pic_buffering_minus1[i]), information regarding the maximum number of picture reorders (max_num_reorder_pics[i]), and information regarding the maximum delay (max_latency_increase_plus1[i]).

[0624] max_dec_pic_buffering_minus1[i]+1 is the maximum size of the DPB required for the decoding process, expressed as the number of pictures, with one picture representing one unit. max_num_reorder_pics[i] represents the maximum number of pictures that precede a given picture in the decoding order but follow it in the output order (in other words, the display order).

[0625] The maximum delay (MaxLatencyPictures[i]) is the maximum number of pictures that precede a given picture in output order (i.e., display order) but follow it in decoding order, and is determined by the following formula.

[0626] MaxLatencyPictures[i]=max_num_reorder_pics[i]+max_latency_increase_plus1[i]-1

[0627] [First aspect] First, let me explain the first aspect.

[0628] The encoding device 100 encodes multiple pictures in encoding order according to the time scalability that defines the multiple temporal layers, and includes in the bitstream information indicating whether the DPB (Decoded Picture Buffer) Output Timings of multiple lower pictures included in at least one first temporal layer in the first case, which targets at least one first temporal layer below a lower temporal layer among the multiple temporal layers, and the DPB Output Timings of multiple lower pictures in the second case, which targets multiple second temporal layers below the upper temporal layer, match. In the encoding order, at least one upper picture is provided between two adjacent lower pictures among the multiple lower pictures, which are included in the upper temporal layer, and the number of at least one upper picture is n-1, where n is an integer greater than 1, and represents the ratio of the second frame rate in the second case to the first frame rate in the first case.

[0629] The decoding device 200 obtains information from the bitstream indicating whether the DPB (Decoded Picture Buffer) Output Timings of multiple lower pictures included in at least one first temporal layer in the first case, which targets at least one first temporal layer below a lower temporal layer among the multiple temporal layers, match the DPB Output Timings of multiple lower pictures in the second case, which targets multiple second temporal layers below the upper temporal layer. Based on this information, the device decodes the multiple pictures in the decoding order according to the time scalability that defines the multiple temporal layers. In the decoding order, at least one upper picture is provided between two adjacent lower pictures among the multiple lower pictures, and the number of at least one upper picture is n-1, where n is an integer greater than 1 and represents the ratio of the second frame rate in the second case to the first frame rate in the first case.

[0630] Figure 100 shows an example of an interpredictive reference picture structure with temporal scalability according to the first embodiment. In this example, when the decoder 200 decodes up to the top temporal layer (here, temporal layer 4), for example, video is played back at 120 fps (frames / second), and the frame rate of the played video is halved as the number of layers decoded decreases by one. For example, when the decoder 200 decodes up to the upper temporal layer (here, temporal layer 3), video is played back at 60 fps. Also, for example, when the decoder 200 decodes at least one first temporal layer (here, temporal layers 0-2) below the lower temporal layer (here, temporal layer 2) (i.e., in the first case above), video is played back at 30 fps. The numbers in the square boxes in Figure 100 indicate the decoding order, but the encoding order is the same as the decoding order.

[0631] Furthermore, in the bitstream, the picture of the top temporal layer (the unshaded rectangle in the figure) and the pictures other than the top temporal layer (the shaded rectangle in the figure) are arranged alternately in encoding order. For example, in the case of a 120fps bitstream, in encoding order, at least one upper picture included in the top temporal layer (temporal layer 4) is placed between two adjacent lower pictures (for example, shaded rectangles 0 and 2 in the figure) of the multiple lower pictures (shaded rectangles in the figure) (n-1) (in the example of Figure 100, there is one, which is shaded rectangle 1). Therefore, in the case of a 120fps bitstream, n is equal to 2, and in encoding order, multiple upper pictures included in the upper temporal layer (the topmost temporal layer in the case of 120fps) and multiple lower pictures included in at least one first temporal layer (temporal layers 0-3) below the lower temporal layer (the uppermost temporal layer 3 in the case of 120fps) are arranged alternately.

[0632] Therefore, in a decoding device 200 that has the capability to decode a 60fps bitstream but not a 120fps bitstream, 60fps video can be played back without exceeding the capabilities of the decoding device 200 (for example, decoding processing speed, bitstream receive buffer (CPB) size, or decoded picture buffer (DPB) size) by extracting and decoding pictures other than the top temporal layer (the shaded rectangular frame in the figure). Because the bitstream has the above layer configuration, for example, when a decoding device that has the capability to decode a 60fps bitstream acquires a bitstream having the above configuration, the timing at which the decoding device decodes the 60fps lay...

Claims

1. Circuits and, The circuit comprises a memory connected to the aforementioned circuit, In operation, the aforementioned circuit Encode multiple pictures according to a temporal scalability that defines multiple temporal layers, In the first case where the plurality of temporal layers are to be decoded, the bitstream includes a value indicating the DPB (Decoded Picture Buffer) Output Delay for each of the plurality of pictures, The bitstream is output to the network, When, among the plurality of temporal layers, the pictures contained in the first temporal layer or lower are defined as lower-level pictures, and the pictures contained in the temporal layers larger than the first temporal layer are defined as upper-level pictures, In the display order, at least one of the upper-level pictures is placed between two adjacent lower-level pictures. The number of the aforementioned at least one upper picture is n-1, n is an integer greater than 1 and represents the ratio between the frame rate in the second case, where the temporal layers below the first temporal layer are to be decoded, and the frame rate in the first case. When n is 2, in the encoding order, n-1 upper-level pictures are arranged between two adjacent lower-level pictures, and the value indicating the DPB Output Delay of the lower-level picture in the first case is equal to the value indicating the DPB Output Delay of the lower-level picture in the second case. When n is 4, in the encoding order, n-1 upper-level pictures are not placed between two adjacent lower-level pictures. Encoding device.

2. The aforementioned network is a communication network. The encoding device according to claim 1.

3. The aforementioned network is the internet. The encoding device according to claim 1.

4. Circuits and, The circuit comprises a memory connected to the aforementioned circuit, In operation, the aforementioned circuit Receive a bitstream from the network, In the first case where multiple temporal layers are to be decoded, a value indicating the DPB (Decoded Picture Buffer) Output Delay for each of the multiple pictures is obtained from the bitstream, The plurality of pictures are decoded in the order of decoding according to the time scalability that defines the plurality of temporal layers, based on the DPB Output Delay. When, among the plurality of temporal layers, the pictures contained in the first temporal layer or lower are defined as lower-level pictures, and the pictures contained in the temporal layers larger than the first temporal layer are defined as upper-level pictures, In the display order, at least one of the upper-level pictures is placed between two adjacent lower-level pictures. The number of the aforementioned at least one upper picture is n-1, n is an integer greater than 1 and represents the ratio between the frame rate in the second case, where the temporal layers below the first temporal layer are to be decoded, and the frame rate in the first case. When n is 2, in the decoding order, n-1 upper-level pictures are arranged between two adjacent lower-level pictures, and the value indicating the DPB Output Delay of the lower-level picture in the first case is equal to the value indicating the DPB Output Delay of the lower-level picture in the second case. When n is 4, in the decoding order, n-1 upper-level pictures are not placed between two adjacent lower-level pictures. Decoding device.

5. The aforementioned network is a communication network. The decoding device according to claim 4.

6. The aforementioned network is the internet. The decoding device according to claim 4.

7. Encode multiple pictures according to a temporal scalability that defines multiple temporal layers, In the first case where the plurality of temporal layers are to be decoded, the bitstream includes a value indicating the DPB (Decoded Picture Buffer) Output Delay for each of the plurality of pictures, The bitstream is output to the network, When, among the plurality of temporal layers, the pictures contained in the first temporal layer or lower are defined as lower-level pictures, and the pictures contained in the temporal layers larger than the first temporal layer are defined as upper-level pictures, In the display order, at least one of the upper-level pictures is placed between two adjacent lower-level pictures. The number of the aforementioned at least one upper picture is n-1, n is an integer greater than 1 and represents the ratio between the frame rate in the second case, where the temporal layers below the first temporal layer are to be decoded, and the frame rate in the first case. When n is 2, in the encoding order, n-1 upper-level pictures are arranged between two adjacent lower-level pictures, and the value indicating the DPB Output Delay of the lower-level picture in the first case is equal to the value indicating the DPB Output Delay of the lower-level picture in the second case. When n is 4, in the encoding order, n-1 upper-level pictures are not placed between two adjacent lower-level pictures. Encoding method.

8. The aforementioned network is a communication network. The encoding method according to claim 7.

9. The aforementioned network is the internet. The encoding method according to claim 7.

10. Receive a bitstream from the network, In the first case where multiple temporal layers are to be decoded, a value indicating the DPB (Decoded Picture Buffer) Output Delay for each of the multiple pictures is obtained from the bitstream, The plurality of pictures are decoded in the order of decoding according to the time scalability that defines the plurality of temporal layers, based on the DPB Output Delay. When, among the plurality of temporal layers, the pictures contained in the first temporal layer or lower are defined as lower-level pictures, and the pictures contained in the temporal layers larger than the first temporal layer are defined as upper-level pictures, In the display order, at least one of the upper-level pictures is placed between two adjacent lower-level pictures. The number of the aforementioned at least one upper picture is n-1, n is an integer greater than 1 and represents the ratio between the frame rate in the second case, where the temporal layers below the first temporal layer are to be decoded, and the frame rate in the first case. When n is 2, in the decoding order, n-1 upper-level pictures are arranged between two adjacent lower-level pictures, and the value indicating the DPB Output Delay of the lower-level picture in the first case is equal to the value indicating the DPB Output Delay of the lower-level picture in the second case. When n is 4, in the decoding order, n-1 upper-level pictures are not placed between two adjacent lower-level pictures. Decryption method.

11. The aforementioned network is a communication network. The decoding method according to claim 10.

12. The aforementioned network is the internet. The decoding method according to claim 10.

Images