Encoding device, decoding device, encoding method, and decoding method

Abstract

An encoding device (100) comprises a circuit and a memory that is connected to the circuit. The circuit, in operation, controls whether or not the resolution of a picture is to be changed from the resolution of a preceding picture which precedes the picture in display order or encoding order, in accordance with the restriction that the change is allowed only when the picture is one of one or more random access pictures, and if the resolution of a reference picture used for encoding a inter prediction picture differs from the resolution of the inter prediction picture, the circuit resamples a reference image of the reference picture in accordance with the difference between the resolution of the reference picture and the resolution of the inter prediction picture, and uses the resampled reference image to encode an image of the inter prediction picture.

Description

[Technical Field]

[0001] This disclosure relates to an encoding device, a decoding device, an encoding method, and a decoding method. [Background technology]

[0002] Video coding technology has advanced from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). With this advancement, there is a constant need to provide improvements and optimizations to video coding technology to handle the ever-increasing volume of digital video data across various applications. This disclosure relates to further advancements, improvements, and optimizations in video coding.

[0003] Non-patent document 1 relates to an example of a conventional standard concerning the video coding technology described above. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] H.265(ISO / IEC 23008-2 HEVC) / HEVC(High Efficiency Video Coding) [Overview of the project] [Problems that the invention aims to solve]

[0005] Regarding the encoding methods described above, there is a need for proposals for new methods to improve encoding efficiency, image quality, processing load, circuit size, or to appropriately select elements or actions such as filters, blocks, size, motion vectors, reference pictures, or reference blocks.

[0006] The present disclosure provides a configuration or method that can contribute to, for example, improvement of encoding efficiency, improvement of image quality, reduction of processing amount, reduction of circuit scale, improvement of processing speed, and appropriate selection of elements or operations. Note that the present disclosure may include a configuration or method that can contribute to benefits other than those described above.

Means for Solving the Problem

[0007] For example, an encoding apparatus according to one aspect of the present disclosure includes a circuit and a memory connected to the circuit. In operation, the circuit controls whether to change the resolution of a picture from the resolution of a previous picture before the picture in one of the display order and the encoding order, only when the picture is any one of one or more random access pictures, according to a constraint that the change is allowed. When the resolution of a reference picture used for encoding an inter prediction picture is different from the resolution of the inter prediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the inter prediction picture, and the image of the inter prediction picture is encoded using the resampled reference image The system generates encoded data and transmits the encoded data, and the constraints allow the resolution of the picture to be changed only if the picture is one of the one or more random access pictures for each k-th random access picture, where k is an integer greater than 1. .

[0008] Each embodiment in the present disclosure, or each part of its configuration or method, enables, for example, at least one of improvement of encoding efficiency, improvement of image quality, reduction of encoding / decoding processing amount, reduction of circuit scale, or improvement of encoding / decoding processing speed. Alternatively, each embodiment in the present disclosure, or each part of its configuration or method, enables appropriate selection of components / operations such as filters, blocks, sizes, motion vectors, reference pictures, reference blocks, etc. in encoding and decoding. Note that the present disclosure also includes disclosure of a configuration or method that can provide benefits other than those described above. For example, a configuration or method for improving encoding efficiency while suppressing an increase in processing amount.

[0009] Further advantages and effects of one aspect of this disclosure will be made apparent from the specification and drawings. Such advantages and / or effects may be obtained by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided to obtain one or more advantages and / or effects.

[0010] These general or specific embodiments may be implemented as a system, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of system, method, integrated circuit, computer program, and recording medium. [Effects of the Invention]

[0011] A configuration or method relating to one aspect of this disclosure may contribute to one or more of the following: improved encoding efficiency, improved image quality, reduced processing load, reduced circuit size, improved processing speed, and appropriate selection of elements or operations. A configuration or method relating to one aspect of this disclosure may also contribute to other benefits not mentioned above. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a schematic diagram showing an example of the configuration of a transmission system according to an embodiment. [Figure 2] Figure 2 shows an example of a data hierarchical structure in a stream. [Figure 3] Figure 3 shows an example of the slice configuration. [Figure 4] Figure 4 shows an example of a tile configuration. [Figure 5] Figure 5 shows an example of an encoding structure during scalable encoding. [Figure 6] Figure 6 shows an example of an encoding structure during scalable encoding. [Figure 7] Figure 7 is a block diagram showing an example of the configuration of an encoding device according to an embodiment. [Figure 8]Figure 8 is a block diagram showing an example implementation of an encoding device. [Figure 9] Figure 9 is a flowchart showing an example of the overall encoding process by the encoding device. [Figure 10] Figure 10 shows an example of block division. [Figure 11] Figure 11 shows an example of the configuration of the divided section. [Figure 12] Figure 12 shows an example of a division pattern. [Figure 13A] Figure 13A shows an example of a syntax tree for a partitioning pattern. [Figure 13B] Figure 13B shows another example of a syntax tree for a partitioning pattern. [Figure 14] Figure 14 is a table showing the transformation basis functions corresponding to each transformation type. [Figure 15] Figure 15 shows an example of an SVT. [Figure 16] Figure 16 is a flowchart showing an example of processing by the conversion unit. [Figure 17] Figure 17 is a flowchart showing another example of processing by the conversion unit. [Figure 18] Figure 18 is a block diagram showing an example of the configuration of the quantization unit. [Figure 19] Figure 19 is a flowchart showing an example of quantization by the quantization unit. [Figure 20] Figure 20 is a block diagram showing an example of the configuration of the entropy coding unit. [Figure 21] Figure 21 shows the CABAC flow in the entropy coding section. [Figure 22] Figure 22 is a block diagram showing an example of the configuration of the loop filter section. [Figure 23A] Figure 23A shows an example of the filter shape used in an adaptive loop filter (ALF). [Figure 23B] Figure 23B shows another example of the filter shape used in ALF. [Figure 23C] Figure 23C shows another example of the filter shape used in ALF. [Figure 23D] Figure 23D shows an example where the Y sample (first component) is used in Cb CCALF and Cr CCALF (multiple components different from the first component). [Figure 23E] Figure 23E shows a diamond-shaped filter. [Figure 23F] Figure 23F shows an example of JC-CCALF. [Figure 23G] Figure 23G shows an example of a candidate weight_index for JC-CCALF. [Figure 24] Figure 24 is a block diagram showing an example of a detailed configuration of the loop filter section that functions as a DBF. [Figure 25] Figure 25 shows an example of a deblocking filter with symmetrical filter characteristics with respect to block boundaries. [Figure 26] Figure 26 illustrates an example of a block boundary where deblocking filtering is performed. [Figure 27] Figure 27 shows an example of a Bs value. [Figure 28] Figure 28 is a flowchart showing an example of the processing performed in the prediction unit of the encoding device. [Figure 29] Figure 29 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device. [Figure 30] Figure 30 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device. [Figure 31] Figure 31 shows an example of 67 intra-prediction modes in intra-prediction. [Figure 32] Figure 32 is a flowchart showing an example of processing by the intra-prediction unit. [Figure 33] Figure 33 shows an example of each reference picture. [Figure 34]Figure 34 is a conceptual diagram showing an example of a reference picture list. [Figure 35] Figure 35 is a flowchart showing the basic processing flow of interpretation prediction. [Figure 36] Figure 36 is a flowchart showing an example of MV derivation. [Figure 37] Figure 37 is a flowchart showing another example of MV derivation. [Figure 38A] Figure 38A shows an example of the classification of each mode in MV derivation. [Figure 38B] Figure 38B shows an example of the classification of each mode in MV derivation. [Figure 39] Figure 39 is a flowchart showing an example of inter-mode prediction. [Figure 40] Figure 40 is a flowchart showing an example of inter prediction using normal merge mode. [Figure 41] Figure 41 is a diagram illustrating an example of MV derivation processing using normal merge mode. [Figure 42] Figure 42 is a diagram illustrating an example of MV derivation processing using HMVP (History-based Motion Vector Prediction / Predictor) mode. [Figure 43] Figure 43 is a flowchart showing an example of FRUC (frame rate up conversion). [Figure 44] Figure 44 illustrates an example of pattern matching (bilateral matching) between two blocks along a motion trajectory. [Figure 45] Figure 45 illustrates an example of pattern matching (template matching) between a template in the current picture and a block in a referenced picture. [Figure 46A] Figure 46A illustrates an example of deriving the subblock unit MV in affine mode using two control points. [Figure 46B]Figure 46B illustrates an example of deriving the subblock unit MV in affine mode using three control points. [Figure 47A] Figure 47A is a conceptual diagram illustrating an example of the MV derivation of the control point in affine mode. [Figure 47B] Figure 47B is a conceptual diagram illustrating an example of the MV derivation of the control point in affine mode. [Figure 47C] Figure 47C is a conceptual diagram illustrating an example of the MV derivation of the control point in affine mode. [Figure 48A] Figure 48A is a diagram illustrating an affine mode with two control points. [Figure 48B] Figure 48B is a diagram illustrating an affine mode with three control points. [Figure 49A] Figure 49A is a conceptual diagram illustrating an example of a method for deriving the control point MV when the number of control points differs between the encoded block and the current block. [Figure 49B] Figure 49B is a conceptual diagram illustrating another example of how to derive the control point MV when the number of control points differs between the encoded block and the current block. [Figure 50] Figure 50 is a flowchart showing an example of processing in affine merge mode. [Figure 51] Figure 51 is a flowchart showing an example of affine intermode processing. [Figure 52A] Figure 52A is a diagram illustrating the generation of two triangular prediction images. [Figure 52B] Figure 52B is a conceptual diagram showing the first part of the first partition, as well as examples of the first and second sample sets. [Figure 52C] Figure 52C is a conceptual diagram showing the first part of the first partition. [Figure 53] Figure 53 is a flowchart showing an example of the triangle mode. [Figure 54] Figure 54 shows an example of the ATMVP (Advanced Temporal Motion Vector Prediction / Predictor) mode in which MV is derived at the subblock level. [Figure 55] Figure 55 shows the relationship between merge mode and DMVR (dynamic motion vector refreshing). [Figure 56] Figure 56 is a conceptual diagram illustrating an example of a DMVR. [Figure 57] Figure 57 is a conceptual diagram illustrating another example of DMVR for determining MV. [Figure 58A] Figure 58A shows an example of motion search in DMVR. [Figure 58B] Figure 58B is a flowchart showing an example of motion search in a DMVR. [Figure 59] Figure 59 is a flowchart showing an example of predictive image generation. [Figure 60] Figure 60 is a flowchart showing another example of predictive image generation. [Figure 61] Figure 61 is a flowchart illustrating an example of predictive image correction processing using OBMC (overlapped block motion compensation). [Figure 62] Figure 62 is a conceptual diagram illustrating an example of predictive image correction processing using OBMC. [Figure 63] Figure 63 is a diagram illustrating a model that assumes uniform linear motion. [Figure 64] Figure 64 is a flowchart showing an example of an inter-prediction method according to BIO. [Figure 65] Figure 65 shows an example of the configuration of the inter prediction unit that performs inter prediction according to BIO. [Figure 66A]Figure 66A illustrates an example of a predictive image generation method using luminance correction processing by LIC (local illumination compensation). [Figure 66B] Figure 66B is a flowchart showing an example of a predictive image generation method using luminance correction processing by LIC. [Figure 67] Figure 67 is a block diagram showing the configuration of a decoding device according to an embodiment. [Figure 68] Figure 68 is a block diagram showing an example of a decoding device implementation. [Figure 69] Figure 69 is a flowchart showing an example of the overall decoding process by the decoding device. [Figure 70] Figure 70 shows the relationship between the division decision unit and other components. [Figure 71] Figure 71 is a block diagram showing an example of the configuration of the entropy decoding unit. [Figure 72] Figure 72 shows the CABAC flow in the entropy decoding section. [Figure 73] Figure 73 is a block diagram showing an example of the configuration of the inverse quantization unit. [Figure 74] Figure 74 is a flowchart showing an example of inverse quantization by an inverse quantization unit. [Figure 75] Figure 75 is a flowchart showing an example of processing by the inverse transform unit. [Figure 76] Figure 76 is a flowchart showing another example of processing by the inverse transform unit. [Figure 77] Figure 77 is a block diagram showing an example of the configuration of the loop filter section. [Figure 78] Figure 78 is a flowchart showing an example of the processing performed in the prediction unit of the decoding device. [Figure 79] Figure 79 is a flowchart showing another example of the processing performed in the prediction unit of the decoding device. [Figure 80A] Figure 80A is a flowchart showing some other examples of the processing performed in the prediction unit of the decoding device. [Figure 80B] Figure 80B is a flowchart showing the remainder of other examples of processing performed in the prediction section of the decoding device. [Figure 81] Figure 81 shows an example of processing performed by the intra-prediction unit of the decoding device. [Figure 82] Figure 82 is a flowchart showing an example of MV derivation in a decoding device. [Figure 83] Figure 83 is a flowchart showing another example of MV derivation in a decoding device. [Figure 84] Figure 84 is a flowchart showing an example of inter-mode prediction in a decoding device. [Figure 85] Figure 85 is a flowchart showing an example of inter prediction using normal merge mode in a decoding device. [Figure 86] Figure 86 is a flowchart showing an example of interpretation using FRUC mode in a decoding device. [Figure 87] Figure 87 is a flowchart showing an example of interpretation using affine merge mode in a decoding device. [Figure 88] Figure 88 is a flowchart showing an example of inter-prediction using affine intermode in a decoding device. [Figure 89] Figure 89 is a flowchart showing an example of inter prediction using triangle mode in a decoding device. [Figure 90] Figure 90 is a flowchart showing an example of motion detection using a DMVR in a decoding device. [Figure 91] Figure 91 is a flowchart showing a detailed example of motion detection by DMVR in a decoding device. [Figure 92] Figure 92 is a flowchart showing an example of predictive image generation in a decoding device. [Figure 93] Figure 93 is a flowchart showing another example of predictive image generation in a decoding device. [Figure 94]Figure 94 is a flowchart showing an example of image correction by OBMC in a decoding device. [Figure 95] Figure 95 is a flowchart showing an example of BIO-based correction of predicted images in a decoding device. [Figure 96] Figure 96 is a flowchart showing an example of predictive image correction by LIC in a decoding device. [Figure 97] Figure 97 is a schematic diagram showing the change in resolution from high resolution to low resolution. [Figure 98A] Figure 98A is a schematic diagram showing an example of a resolution change process. [Figure 98B] Figure 98B is a schematic diagram showing another example of the resolution change process. [Figure 98C] Figure 98C is a schematic diagram showing yet another example of the resolution change process. [Figure 99] Figure 99 is a schematic diagram showing the relationships between multiple parameters. [Figure 100A] Figure 100A is a schematic diagram showing an example of a video stream that fits the constraint k>1. [Figure 100B] Figure 100B is a schematic diagram illustrating an example of a video stream that does not conform to the constraint k>1. [Figure 101] Figure 101 is a flowchart showing examples of picture encoding and picture decoding methods. [Figure 102] Figure 102 is a flowchart showing an example of an encoding method for encoding a picture sequence into a bitstream. [Figure 103] Figure 103 is a conceptual diagram showing a server device and a receiver device that comply with the DASH protocol. [Figure 104A] Figure 104A is a flowchart showing some examples of encoding methods that use rate control. [Figure 104B] Figure 104B is a flowchart showing the remainder of an example of an encoding method using rate control. [Figure 105] Figure 105 is a flowchart showing an example of a decoding method for a broadcast receiving device. [Figure 106A] Figure 106A is a flowchart showing some examples of encoding methods using a set of predefined resolutions. [Figure 106B] Figure 106B is a flowchart showing the remainder of an example of an encoding method using a specified resolution set. [Figure 107A] Figure 107A is a flowchart showing some examples of encoding methods using rate control and a set of defined resolutions. [Figure 107B] Figure 107B is a flowchart showing the remainder of an example of an encoding method using rate control and a set of defined resolutions. [Figure 108] Figure 108 is a flowchart showing the operations performed by the encoding device according to the embodiment. [Figure 109] Figure 109 is a flowchart showing the operations performed by the decoding device according to the embodiment. [Figure 110] Figure 110 is an overall diagram of the content supply system that realizes the content distribution service. [Figure 111] Figure 111 shows an example of how a web page is displayed. [Figure 112] Figure 112 shows an example of how a web page is displayed. [Figure 113] Figure 113 shows an example of a smartphone. [Figure 114] Figure 114 is a block diagram showing an example of a smartphone configuration. [Modes for carrying out the invention]

[0013] [Introduction] When multiple pictures constituting a video are encoded sequentially, the resolution of a picture may be changed from the resolution of a preceding picture that is earlier in the encoding or display order than the picture in question. If the resolution of the reference picture used to encode the interprediction picture is different from the resolution of the interprediction picture, the reference image of the reference picture may be resampled, and the image of the interprediction picture may be encoded using the resampled reference image.

[0014] This makes it possible to use interpretation even when the resolution of the reference picture used to encode the interpretation picture differs from the resolution of the interpretation picture.

[0015] However, frequent changes in resolution increase the consumption of computing and memory resources, placing a heavy burden on the encoding and decoding devices. Furthermore, it is difficult to constantly secure the computing and memory resources necessary to handle resolution changes. Additionally, testing the encoding and decoding devices is not easy when resolution changes occur at arbitrary times.

[0016] For example, an encoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit controls, in operation, whether to change the resolution of a picture from the resolution of a preceding picture prior to the picture in either the display order or the encoding order, according to a constraint that the change is permitted only if the picture is one or more random access pictures, and if the resolution of a reference picture used to encode an interprediction picture is different from the resolution of the interprediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interprediction picture, and the image of the interprediction picture is encoded using the resampled reference image.

[0017] This may allow the encoding device to suppress frequent resolution changes. Consequently, the encoding device may be able to reduce the consumption of computational and memory resources, thereby reducing its burden. Furthermore, the encoding device may be able to suppress the need to constantly reserve computational and memory resources to accommodate resolution changes. In addition, the encoding device may be able to reduce the difficulty of testing.

[0018] Furthermore, for example, the constraint allows changing the resolution of the picture only if the picture is one of the one or more random access pictures for each k-th random access picture, where k is an integer greater than 1.

[0019] This may allow the encoding device to further suppress changes in resolution. Consequently, the encoding device may be able to further reduce the consumption of computing and memory resources, and thus further reduce its burden.

[0020] Furthermore, for example, the circuit sequentially encodes multiple pictures into a bitstream, monitors the bit amount of the bitstream, and controls the change in the resolution of the pictures according to the monitored bit amount.

[0021] This may allow the encoding device to change the resolution according to the bit amount of the bitstream, and to adjust the bit amount of the bitstream in conjunction with the change in resolution.

[0022] Furthermore, for example, the circuit encodes at least one first trailing picture belonging to a first picture group, then encodes a random access picture included in one or more random access pictures that belongs to a second picture group that appears later in display order than the first picture group, then encodes at least one leading picture belonging to the first picture group, then encodes at least one second trailing picture belonging to the second picture group, wherein the resolution of the at least one leading picture is not permitted to be different from the resolution of the at least one first trailing picture, the resolution of the random access picture is not permitted to be different from the resolution of the at least one leading picture, and the resolution of the at least one second trailing picture is not permitted to be different from the resolution of the random access picture.

[0023] This allows the encoding device to change the resolution using only random access pictures in the display order. Therefore, the encoding device may be able to smoothly change the resolution according to the display order.

[0024] Furthermore, for example, the circuit encodes at least one first trailing picture belonging to a first picture group, then encodes a random access picture included in one or more random access pictures that belongs to a second picture group that appears later in display order than the first picture group, then encodes at least one leading picture belonging to the first picture group, then encodes at least one second trailing picture belonging to the second picture group, wherein the resolution of the random access picture may differ from the resolution of the at least one first trailing picture, the resolution of the at least one leading picture may not differ from the resolution of the random access picture, and the resolution of the at least one second trailing picture may not differ from the resolution of the at least one leading picture.

[0025] This allows the encoding device to change the resolution using only random access pictures in the encoding order. Therefore, the encoding device may be able to smoothly change the resolution in accordance with the encoding order.

[0026] Furthermore, for example, the aforementioned constraint does not permit changing the resolution of the picture within each of the multiple segments defined in the DASH (Dynamic Adaptive Streaming over HTTP) protocol.

[0027] This may allow the encoding device to limit resolution changes to segment by segment. Therefore, this may allow the encoding device to suppress frequent resolution changes.

[0028] Furthermore, for example, the aforementioned constraint does not permit changing the resolution of the picture at intervals shorter than the threshold in each of the multiple representations defined in the DASH protocol.

[0029] This may allow the encoding device to suppress frequent resolution changes within the same representation. Therefore, this may further suppress frequent resolution changes by the encoding device.

[0030] Furthermore, for example, the aforementioned constraint limits the resolution of the picture to one of several resolution candidates.

[0031] This may allow the encoding device to limit the resolution to one of several resolution candidates. Therefore, the encoding device may be able to reduce the complexity of the process. Furthermore, the encoding device may be able to reduce the difficulty of testing.

[0032] Furthermore, for example, the plurality of resolution candidates include a plurality of resolution candidates determined by scaling the resolution of the preceding picture by a plurality of fixed ratios.

[0033] This allows the encoding device to smoothly change the resolution from the original resolution using a fixed ratio. Furthermore, it may be possible to smoothly resample the reference image using a fixed ratio. Therefore, the encoding device may be able to suppress the complexity of the processing.

[0034] Furthermore, for example, the plurality of fixed ratios include at least one of 2x, 3 / 2x, 4 / 3x, 3 / 4x, 2 / 3x, and 1 / 2x.

[0035] This allows the encoding device to change the resolution at a ratio that makes resampling the reference image easier. Therefore, the encoding device may be able to reduce the complexity of the processing.

[0036] Furthermore, for example, the aforementioned multiple resolution candidates include multiple resolution candidates defined as 7680×4320 pixels, 5120×2880 pixels, 3840×2160 pixels, 2560×1440 pixels, 1920×1080 pixels, 1280×720 pixels, and 960×540 pixels.

[0037] This allows encoding devices to utilize general-purpose resolutions and maintain versatility.

[0038] Furthermore, for example, the plurality of resolution candidates include a plurality of resolution candidates determined by applying the addition or subtraction of a plurality of fixed difference values ​​to the resolution of the preceding picture.

[0039] This may allow the encoding device to appropriately limit the modified resolution according to the original resolution and a fixed difference value, thereby potentially suppressing the complexity of the process.

[0040] Furthermore, for example, the constraint limits the maximum resolution specified as an upper limit in the sequence parameter set to be equal to the resolution specified as the resolution of at least one picture in the picture parameter set applied to at least one of the multiple pictures to which the sequence parameter set applies, and the circuit encodes the maximum resolution determined according to the constraint into the sequence parameter set and encodes the resolution specified as the resolution of at least one picture into the picture parameter set.

[0041] This may allow the encoding device to prevent the specification of an excessively large maximum resolution in the sequence parameter set. Consequently, the encoding device may prevent the allocation of excessive computational and memory resources.

[0042] Furthermore, for example, a decoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit controls, in operation, whether to change the resolution of a picture from the resolution of a preceding picture prior to the picture in either the display order or the decoding order, subject to the constraint that such change is permitted only if the picture is one or more random access pictures, and if the resolution of a reference picture used for decoding an interprediction picture is different from the resolution of the interprediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interprediction picture, and the image of the interprediction picture is decoded using the resampled reference image.

[0043] This may allow the decoding device to suppress frequent resolution changes. Consequently, the decoding device may be able to reduce the consumption of computing and memory resources, thereby reducing its burden. Furthermore, the decoding device may be able to suppress the need to constantly reserve computing and memory resources to accommodate resolution changes. In addition, the decoding device may be able to reduce the difficulty of testing.

[0044] Furthermore, for example, the circuit allocates at least one of the computational resources and memory resources for post-processing to display the decoded image, in accordance with the constraints.

[0045] This may allow the decoding device to prepare the necessary and sufficient computing or memory resources for post-processing to display the decoded image, in accordance with the constraints.

[0046] Furthermore, for example, the constraint allows changing the resolution of the picture only if the picture is one of the one or more random access pictures for each k-th random access picture, where k is an integer greater than 1.

[0047] This may allow the decoding device to further suppress changes in resolution. Consequently, the decoding device may be able to further reduce the consumption of computing and memory resources, and thus further reduce its burden.

[0048] Furthermore, for example, the circuit sequentially decodes multiple pictures from a bitstream, monitors the bit amount of the bitstream, and controls the resolution of the pictures according to the monitored bit amount.

[0049] This may allow the decoding device to change the resolution according to the bit amount of the bitstream, and to adjust the bit amount of the bitstream in conjunction with the change in resolution.

[0050] Furthermore, for example, the circuit decodes at least one first trailing picture belonging to a first picture group, then decodes a random access picture included in one or more random access pictures that belongs to a second picture group that appears later in display order than the first picture group, then decodes at least one leading picture belonging to the first picture group, then decodes at least one second trailing picture belonging to the second picture group, and the constraints are such that the resolution of the at least one leading picture may not be different from the resolution of the at least one first trailing picture, the resolution of the random access picture may not be different from the resolution of the at least one leading picture, and the resolution of the at least one second trailing picture may not be different from the resolution of the random access picture.

[0051] This allows the decoding device to change the resolution using only random access pictures in the display order. Therefore, the decoding device may be able to smoothly change the resolution according to the display order.

[0052] Furthermore, for example, the circuit decodes at least one first trailing picture belonging to a first picture group, then decodes a random access picture included in the one or more random access pictures that belongs to a second picture group that appears later in display order than the first picture group, then decodes at least one leading picture belonging to the first picture group, then decodes at least one second trailing picture belonging to the second picture group, wherein the resolution of the random access picture may differ from the resolution of the at least one first trailing picture, the resolution of the at least one leading picture may not differ from the resolution of the random access picture, and the resolution of the at least one second trailing picture may not differ from the resolution of the at least one leading picture.

[0053] This allows the decoding device to change the resolution using only random access pictures in the decoding order. Therefore, the decoding device may be able to smoothly change the resolution in accordance with the decoding order.

[0054] Furthermore, for example, the aforementioned constraint does not permit changing the resolution of the picture within each of the multiple segments defined in the DASH (Dynamic Adaptive Streaming over HTTP) protocol.

[0055] This may allow the decoder to limit resolution changes to segment by segment. Therefore, this may allow the decoder to suppress frequent resolution changes.

[0056] Furthermore, for example, the aforementioned constraint does not permit changing the resolution of the picture at intervals shorter than the threshold in each of the multiple representations defined in the DASH protocol.

[0057] This may allow the decoder to suppress changing the resolution at short intervals within the same representation. Therefore, this may further suppress frequent changes in resolution by the decoder.

[0058] Furthermore, for example, the aforementioned constraint limits the resolution of the picture to one of several resolution candidates.

[0059] This may allow the decoding device to limit the resolution to one of several resolution candidates. Therefore, the decoding device may be able to reduce the complexity of the process. Furthermore, the decoding device may be able to reduce the difficulty of testing.

[0060] Furthermore, for example, the plurality of resolution candidates include a plurality of resolution candidates determined by scaling the resolution of the preceding picture by a plurality of fixed ratios.

[0061] This may allow the decoding device to smoothly change the resolution from the original resolution using a fixed ratio. Furthermore, it may allow the decoding device to smoothly resample the reference image using a fixed ratio. Therefore, the decoding device may be able to suppress the complexity of the processing.

[0062] Furthermore, for example, the plurality of fixed ratios include at least one of 2x, 3 / 2x, 4 / 3x, 3 / 4x, 2 / 3x, and 1 / 2x.

[0063] This may allow the decoding device to change the resolution at a ratio that makes resampling the reference image easier. Therefore, the decoding device may be able to reduce the complexity of the process.

[0064] Furthermore, for example, the aforementioned multiple resolution candidates include multiple resolution candidates defined as 7680×4320 pixels, 5120×2880 pixels, 3840×2160 pixels, 2560×1440 pixels, 1920×1080 pixels, 1280×720 pixels, and 960×540 pixels.

[0065] This may allow the decoding device to utilize a general-purpose resolution, thus maintaining its versatility.

[0066] Furthermore, for example, the plurality of resolution candidates include a plurality of resolution candidates determined by applying the addition or subtraction of a plurality of fixed difference values ​​to the resolution of the preceding picture.

[0067] This may allow the decoding device to appropriately limit the modified resolution according to the original resolution and a fixed difference value, thereby potentially suppressing the complexity of the process.

[0068] Furthermore, for example, the constraint limits the maximum resolution specified as an upper limit in the sequence parameter set to be equal to the resolution specified as the resolution of at least one picture in the picture parameter set applied to at least one of the multiple pictures to which the sequence parameter set applies, and the circuit decodes the maximum resolution determined according to the constraint from the sequence parameter set and decodes the resolution specified as the resolution of at least one picture from the picture parameter set.

[0069] This may allow the decoding device to prevent the specification of an excessively large maximum resolution in the sequence parameter set. Consequently, the decoding device may be able to prevent the allocation of excessive computational and memory resources.

[0070] Furthermore, for example, an encoding method according to one aspect of the present disclosure controls whether the resolution of a picture is changed from the resolution of a preceding picture prior to the picture in either the display order or the encoding order, according to a constraint that the change is permitted only if the picture is one or more random access pictures, and if the resolution of a reference picture used to encode an interprediction picture is different from the resolution of the interprediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interprediction picture, and the image of the interprediction picture is encoded using the resampled reference image.

[0071] This may make it possible to reduce the frequency of resolution changes. Consequently, it may be possible to reduce the consumption of computing and memory resources, thereby alleviating the burden. It may also be possible to reduce the need to constantly reserve computing and memory resources to accommodate resolution changes. Furthermore, it may be possible to reduce the difficulty of testing.

[0072] Furthermore, for example, a decoding method according to one aspect of the present disclosure controls whether the resolution of a picture is changed from the resolution of a preceding picture prior to the picture in either the display order or the decoding order, according to a constraint that the change is permitted only if the picture is one or more random access pictures. If the resolution of a reference picture used to decode an interprediction picture is different from the resolution of the interprediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interprediction picture, and the image of the interprediction picture is decoded using the resampled reference image.

[0073] This may make it possible to reduce the frequency of resolution changes. Consequently, it may be possible to reduce the consumption of computing and memory resources, thereby alleviating the burden. It may also be possible to reduce the need to constantly reserve computing and memory resources to accommodate resolution changes. Furthermore, it may be possible to reduce the difficulty of testing.

[0074] Furthermore, for example, a non-temporary computer-readable recording medium according to one aspect of the present disclosure is a non-temporary computer-readable recording medium that stores a bitstream, the bitstream including a plurality of pictures and a plurality of parameters for performing a decoding process to decode the plurality of pictures from the bitstream, the decoding process controlling whether to change the resolution of a picture from the resolution of a preceding picture prior to the picture in either the display order or the decoding order, in an operation according to the plurality of parameters, subject to the constraint that the change is permitted only if the picture is one or more random access pictures, and if the resolution of a reference picture used to decode an interprediction picture is different from the resolution of the interprediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interprediction picture, and the image of the interprediction picture is decoded using the resampled reference image.

[0075] This may make it possible to reduce the frequency of resolution changes. Consequently, it may be possible to reduce the consumption of computing and memory resources, thereby alleviating the burden. It may also be possible to reduce the need to constantly reserve computing and memory resources to accommodate resolution changes. Furthermore, it may be possible to reduce the difficulty of testing.

[0076] Furthermore, for example, the plurality of parameters include a first parameter indicating whether or not it is permissible to change the resolution of one or more of the plurality of pictures, and a second parameter indicating the resolution of the picture, wherein if it is permissible to change the resolution of one or more of the pictures according to the first parameter, and the picture is included in the one or more pictures, and it is permissible to change the resolution of the picture according to the constraint, the resolution of the picture is changed according to the second parameter.

[0077] This may allow for resolution changes while suppressing frequent changes in resolution, in accordance with constraints and multiple parameters related to resolution changes.

[0078] Furthermore, for example, an encoding device according to one aspect of this disclosure comprises an input unit, a division unit, an intra-prediction unit, an inter-prediction unit, a loop filter unit, a conversion unit, a quantization unit, an entropy encoding unit, and an output unit.

[0079] The current picture is input to the input unit. The division unit divides the current picture into multiple blocks.

[0080] The intra prediction unit generates a prediction signal for the current block contained in the current picture using a reference image contained in the current picture. The inter prediction unit generates a prediction signal for the current block contained in the current picture using a reference image contained in a reference picture different from the current picture. The loop filter unit applies a filter to the reconstructed block of the current block contained in the current picture.

[0081] The conversion unit converts the prediction error between the original signal of the current block included in the current picture and the predicted signal generated by the intra-prediction unit or the inter-prediction unit to generate conversion coefficients. The quantization unit quantizes the conversion coefficients to generate quantization coefficients. The entropy coding unit applies variable-length coding to the quantization coefficients to generate an encoded bitstream. The encoded bitstream, which includes the quantization coefficients to which variable-length coding has been applied and control information, is output from the output unit.

[0082] Furthermore, for example, the interpretation unit controls whether to change the resolution of a picture from the resolution of a preceding picture prior to the picture in either the display order or the encoding order, according to a constraint that such change is only permitted if the picture is one or more random access pictures. If the resolution of the reference picture used to encode the interpretation picture is different from the resolution of the interpretation picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interpretation picture, and the image of the interpretation picture is encoded using the resampled reference image.

[0083] Furthermore, for example, a decoding device according to one aspect of the present disclosure comprises an input unit, an entropy decoding unit, an inverse quantization unit, an inverse transform unit, an intra-prediction unit, an inter-prediction unit, a loop filter unit, and an output unit.

[0084] The input unit receives an encoded bitstream. The entropy decoding unit applies variable-length decoding to the encoded bitstream to derive quantization coefficients. The inverse quantization unit dequantizes the quantization coefficients to derive conversion coefficients. The inverse transformation unit inversely transforms the conversion coefficients to derive prediction errors.

[0085] The intra prediction unit generates a prediction signal for the current block contained in the current picture using a reference image contained in the current picture. The inter prediction unit generates a prediction signal for the current block contained in the current picture using a reference image contained in a reference picture different from the current picture.

[0086] The loop filter unit applies a filter to the reconstructed block of the current block included in the current picture. Then, the current picture is output from the output unit.

[0087] Furthermore, for example, the interpretation unit controls whether to change the resolution of a picture from the resolution of a preceding picture that precedes the picture in either the display order or the decoding order, according to a constraint that such change is only permitted if the picture is one or more random access pictures. If the resolution of the reference picture used to decode the interpretation picture is different from the resolution of the interpretation picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interpretation picture, and the image of the interpretation picture is decoded using the resampled reference image.

[0088] Furthermore, these comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0089] [Definition of Terms] The following definitions may be used as examples for each term.

[0090] (1) Image A unit of data composed of a collection of pixels, consisting of pictures and smaller blocks, and includes both still images and videos.

[0091] (2) Picture A processing unit of an image, composed of a collection of pixels, is sometimes called a frame or field.

[0092] (3) Block A set containing a specific number of pixels is a processing unit, and its name is not restricted, as shown in the examples below. Its shape is also not restricted; for example, it includes not only rectangles made of M×N pixels and squares made of M×M pixels, but also triangles, circles, and other shapes.

[0093] (Example of a block) Slice / tile / brick ·CTU / Superblock / Basic division unit • VPDU / Hardware processing partitioning unit ·CU / Processing Block Unit / Prediction Block Unit (PU) / Orthogonal Transformation Block Unit (TU) / Unit • Subblock

[0094] (4) Pixels / Samples The smallest unit of an image is a point, which includes not only pixels at integer positions but also pixels at decimal positions that are generated based on pixels at integer positions.

[0095] (5) Pixel value / Sample value These are the unique values ​​that each pixel possesses, including not only luminance values, color difference values, and RGB gradations, but also depth values, or binary values ​​of 0 or 1.

[0096] (6) Flag This includes not only single-bit values ​​but also multi-bit values, such as parameters or indices of two or more bits. Furthermore, it is not limited to binary values, but also includes multi-value values ​​using other number systems.

[0097] (7) Signal This refers to a system of symbols and encoding used to transmit information, and includes not only discretized digital signals but also analog signals that take continuous values.

[0098] (8) Stream / Bitstream A stream / bitstream refers to a sequence of digital data or a flow of digital data. A stream / bitstream may consist of a single stream or multiple streams divided into multiple layers. Furthermore, it includes cases where data is transmitted via serial communication over a single transmission path, as well as cases where data is transmitted via packet communication over multiple transmission paths.

[0099] (9) Difference / difference For scalar quantities, it is sufficient to include not only the simple difference (xy), but also any difference operation, including the absolute value of the difference (|xy|), the squared difference (x^2-y^2), the square root of the difference (√(xy)), the weighted difference (ax-by: a, b is a constant), and the offset difference (x-y+a: a is the offset).

[0100] (10) sum For scalar quantities, it is sufficient to include not only simple sums (x+y) but also summation operations, including the absolute value of the sum (|x+y|), sum of squares (x^2+y^2), square root of the sum (√(x+y)), weighted sums (ax+by: a, b is a constant), and offset sums (x+y+a: a is the offset).

[0101] (11) based on This includes cases where factors other than the underlying subject are taken into consideration. Furthermore, it includes cases where the result is obtained not only through direct calculation but also through intermediate steps.

[0102] (12) using This includes cases where factors other than the target element are taken into consideration. Furthermore, it includes cases where the result is obtained not only through direct calculation but also through intermediate steps.

[0103] (13) To prohibit, to forbid This can be rephrased as "not permitted." Furthermore, not prohibiting something or allowing it does not necessarily mean it is an obligation.

[0104] (14) To limit (limit, restriction / restrict / restricted) This can be rephrased as "not permitted." Furthermore, not prohibiting something or allowing it does not necessarily mean it is an obligation. Additionally, it may suffice for something to be partially prohibited quantitatively or qualitatively, and it may also include cases of complete prohibition.

[0105] (15) Chroma The symbols Cb and Cr are adjectives that specify that a sample sequence or a single sample represents one of two color difference signals related to a primary color. The term chrominance can also be used instead of chroma.

[0106] (16) Luminance (luma) This is an adjective, represented by a symbol or a subscript Y or L, that specifies that a sample sequence or a single sample represents a monochrome signal related to a primary color. The term luminance can also be used instead of luma.

[0107] [Explanation regarding the description] In drawings, the same reference number indicates the same or similar component. Furthermore, the size and relative position of components in the drawings are not necessarily depicted to a consistent scale.

[0108] The embodiments will be described in detail below with reference to the drawings. Note that the embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement and connection forms of components, steps, relationships and sequences of steps shown in the following embodiments are examples only and are not intended to limit the scope of the claims.

[0109] Embodiments of encoding and decoding devices are described below. These embodiments are examples of encoding and decoding devices to which the processes and / or configurations described in each aspect of this disclosure can be applied. The processes and / or configurations can also be implemented in encoding and decoding devices different from those in the embodiments. For example, with respect to the processes and / or configurations applicable to the embodiments, one of the following may be implemented:

[0110] (1) Any of the multiple components of the encoding or decoding device of the embodiments described in each aspect of the present disclosure may be replaced or combined with other components described in any of the aspects of the present disclosure.

[0111] (2) In the encoding or decoding device of the embodiment, any modifications such as addition, replacement, or deletion of functions or processes performed by some of the multiple components of the encoding or decoding device may be made. For example, any of the functions or processes may be replaced or combined with other functions or processes described in any of the embodiments of this disclosure.

[0112] (3) In the methods performed by the encoding or decoding apparatus of the embodiment, any modifications, such as additions, replacements, and deletions, may be made to some of the processes included in the method. For example, any of the processes in the method may be replaced with or combined with other processes described in any of the embodiments of this disclosure.

[0113] (4) Some of the multiple components constituting the encoding or decoding device of the embodiment may be combined with components described in any of the embodiments of this disclosure, or with components that have some of the functions described in any of the embodiments of this disclosure, or with components that perform some of the processing performed by the components described in any of the embodiments of this disclosure.

[0114] (5) Components that provide some of the functions of the encoding or decoding device of the embodiment, or components that perform some of the processing of the encoding or decoding device of the embodiment, may be combined with or replaced with components described in any of the aspects of the disclosure and components that provide some of the functions described in any of the aspects of the disclosure, or components that perform some of the processing described in any of the aspects of the disclosure.

[0115] (6) In a method performed by an encoding or decoding device of an embodiment, any of the processes included in the method may be replaced or combined with any of the processes described in any of the embodiments of the present disclosure.

[0116] (7) Some of the processes included in the methods performed by the encoding or decoding device of the embodiment may be combined with the processes described in any of the embodiments of this disclosure.

[0117] (8) The methods of carrying out the processes and / or configurations described in each aspect of the present disclosure are not limited to the encoding or decoding devices of the embodiments. For example, the processes and / or configurations may be carried out in devices used for purposes other than the video encoding or video decoding disclosed in the embodiments.

[0118] [System Configuration] Figure 1 is a schematic diagram showing an example of the configuration of the transmission system according to this embodiment.

[0119] The transmission system Trs is a system that transmits a stream generated by encoding an image and decodes the transmitted stream. Such a transmission system Trs includes, for example, an encoding device 100, a network Nw, and a decoding device 200, as shown in Figure 1.

[0120] An image is input to the encoding device 100. The encoding device 100 generates a stream by encoding the input image and outputs the stream to the network Nw. The stream includes, for example, the encoded image and control information for decoding the encoded image. The image is compressed by this encoding process.

[0121] The original image input to the encoding device 100 before encoding is also called the original image, original signal, or original sample. The image may be a moving image or a still image. The image is a higher-level concept than sequences, pictures, and blocks, and is not limited by spatial and temporal domains unless otherwise specified. The image consists of a sequence of pixels or pixel values, and the signal or pixel value representing the image is also called a sample. The stream may be called a bitstream, encoded bitstream, compressed bitstream, or encoded signal. Furthermore, the encoding device may be called an image encoding device or a moving image encoding device, and the encoding method by the encoding device 100 may be called an encoding method, an image encoding method, or a moving image encoding method.

[0122] The network Nw transmits the stream generated by the encoding device 100 to the decoding device 200. The network Nw may be the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network Nw is not necessarily limited to a bidirectional communication network; it may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, the network Nw may be replaced by a storage medium that records the stream, such as a DVD (Digital Versatile Disc) or a BD (Blu-Ray Disc®).

[0123] The decoding device 200 generates a decoded image, which is, for example, an uncompressed image, by decoding the stream transmitted by the network Nw. For example, the decoding device decodes the stream according to a decoding method that corresponds to the encoding method by the encoding device 100.

[0124] The decoding device may also be called an image decoding device or a video decoding device, and the decoding method performed by the decoding device 200 may be called a decoding method, an image decoding method, or a video decoding method.

[0125] [Data structure] Figure 2 shows an example of a data hierarchy in a stream. The stream includes, for example, a video sequence. This video sequence includes, for example, a VPS (Video Parameter Set), an SPS (Sequence Parameter Set), a PPS (Picture Parameter Set), an SEI (Supplemental Enhancement Information), and multiple pictures, as shown in Figure 2(a).

[0126] VPS includes encoding parameters common to multiple layers in a video composed of multiple layers, and encoding parameters related to the multiple layers included in the video, or to individual layers.

[0127] The SPS includes parameters used for the sequence, i.e., encoding parameters that the decoding device 200 references to decode the sequence. For example, these encoding parameters may represent the width or height of the picture. Multiple SPSs may exist.

[0128] The PPS includes parameters used for the picture, i.e., encoding parameters that the decoding device 200 references to decode each picture in the sequence. For example, these encoding parameters may include a reference value for the quantization width used to decode the picture and a flag indicating the application of weighted prediction. Multiple PPSs may exist. The SPS and PPS are sometimes simply referred to as parameter sets.

[0129] The picture may include a picture header and one or more slices, as shown in Figure 2(b). The picture header includes encoding parameters that the decoding device 200 references to decode the one or more slices.

[0130] A slice includes a slice header and one or more bricks, as shown in Figure 2(c). The slice header includes encoding parameters that the decoding device 200 references to decode the one or more bricks.

[0131] A brick contains one or more Coding Tree Units (CTUs), as shown in Figure 2(d).

[0132] Note that a picture may not contain slices, but instead contain tile groups. In this case, a tile group contains one or more tiles. A brick may also contain slices.

[0133] A CTU is also called a superblock or basic division unit. Such a CTU includes a CTU header and one or more CUs (Coding Units), as shown in Figure 2(e). The CTU header contains coding parameters that the decoding device 200 refers to in order to decode one or more CUs.

[0134] A CU may be divided into multiple smaller CUs. Furthermore, as shown in Figure 2(f), a CU includes a CU header, prediction information, and residual coefficient information. The prediction information is for predicting the CU, and the residual coefficient information indicates the predicted residual, which will be described later. While a CU is basically identical to a PU (Prediction Unit) and a TU (Transform Unit), in SBT (Science-Based Testing), for example, it may include multiple TUs smaller than the CU. Additionally, a CU may be processed for each VPDU (Virtual Pipeline Decoding Unit) that constitutes it. A VPDU is a fixed unit that can be processed in one stage when performing pipeline processing in hardware, for example.

[0135] Note that a stream does not necessarily have to have some of the layers shown in Figure 2. Also, the order of these layers may be changed, and any layer may be replaced by another layer. The picture that is currently being processed by a device such as the encoding device 100 or the decoding device 200 is called the current picture. If the processing is encoding, the current picture is synonymous with the picture to be encoded, and if the processing is decoding, the current picture is synonymous with the picture to be decoded. Also, a block, such as CU or CU, that is currently being processed by a device such as the encoding device 100 or the decoding device 200 is called the current block. If the processing is encoding, the current block is synonymous with the block to be encoded, and if the processing is decoding, the current block is synonymous with the block to be decoded.

[0136] [Picture composition: slice / tile] To decode pictures in parallel, they may be composed of slices or tiles.

[0137] A slice is the basic encoding unit that makes up a picture. A picture is composed of, for example, one or more slices. A slice, in turn, consists of one or more consecutive CTUs.

[0138] Figure 3 shows an example of a slice configuration. For example, a picture contains 11 × 8 CTUs and is divided into four slices (slice 1-4). Slice 1 consists of, for example, 16 CTUs, slice 2 consists of, for example, 21 CTUs, slice 3 consists of, for example, 29 CTUs, and slice 4 consists of, for example, 22 CTUs. Here, each CTU in the picture belongs to one of the slices. The shape of the slice is the horizontal division of the picture. The boundaries of the slice do not have to be at the edges of the screen, but can be anywhere among the boundaries of the CTUs within the screen. The processing order (encoding order or decoding order) of the CTUs in a slice is, for example, the raster scan order. A slice also includes a slice header and encoded data. The slice header may describe the characteristics of the slice, such as the CTU address of the beginning of the slice and the slice type.

[0139] A tile is a rectangular area that makes up a picture. Each tile may be assigned a number called a TileId in the order of the raster scan.

[0140] Figure 4 shows an example of a tile configuration. For example, a picture contains 11 × 8 CTUs and is divided into four rectangular tiles (tiles 1-4). When tiles are used, the processing order of CTUs is changed compared to when tiles are not used. When tiles are not used, multiple CTUs in a picture are processed, for example, in raster scan order. When tiles are used, in each of the multiple tiles, at least one CTU is processed, for example, in raster scan order. For example, as shown in Figure 4, the processing order of multiple CTUs contained in tile 1 is from the left end of the first column of tile 1 to the right end of the first column of tile 1, and then from the left end of the second column of tile 1 to the right end of the second column of tile 1.

[0141] Note that one tile may contain one or more slices, and one slice may contain one or more tiles.

[0142] A picture may be composed of tilesets. A tileset may contain one or more tile groups, or one or more tiles. A picture may consist of only one of a tileset, a tile group, or a tile. For example, the order in which multiple tiles are scanned in raster order for each tileset is defined as the basic coding order of the tiles. Within each tileset, a collection of one or more tiles that follow the same basic coding order is defined as a tile group. Such a picture may be composed of the division unit 102 (see Figure 7) described later.

[0143] [Scalable encoding] Figures 5 and 6 show an example of a scalable stream configuration.

[0144] As shown in Figure 5, the encoding device 100 may generate a temporally / spatially scalable stream by encoding each of the multiple pictures into one of the multiple layers. For example, the encoding device 100 achieves scalability by encoding each picture layer by layer, with an enhancement layer existing above the base layer. This type of encoding of each picture is called scalable encoding. This allows the decoding device 200 to switch the image quality displayed by decoding the stream. In other words, the decoding device 200 decides which layers to decode based on internal factors such as its own performance and external factors such as the state of the communication bandwidth. As a result, the decoding device 200 can freely switch between decoding the same content into low-resolution content and high-resolution content. For example, a user of the stream might watch part of the video on their smartphone while on the go, and then watch the rest of the video on a device such as an internet TV after returning home. Each of the aforementioned smartphones and devices incorporates a decoding device 200 with the same or different performance. In this case, if the device decodes up to the upper layers of the stream, the user can view high-definition video after returning home. This eliminates the need for the encoding device 100 to generate multiple streams with the same content but different image quality, thereby reducing the processing load.

[0145] Furthermore, the enhancement layer may include metadata based on image statistics. The decoding device 200 may generate a high-resolution video by super-resolution the picture of the base layer based on the metadata. Super-resolution may be either an improvement in the signal-to-noise ratio (SN) at the same resolution, or an increase in resolution. The metadata may include information for identifying linear or nonlinear filter coefficients used in the super-resolution process, or information for identifying parameter values ​​in the filtering process, machine learning, or least-squares operation used in the super-resolution process.

[0146] Alternatively, the picture may be divided into tiles or the like, depending on the meaning of each object within the picture. In this case, the decoding device 200 may decode only a portion of the picture by selecting the tiles to be decoded. Furthermore, the attributes of the objects (people, cars, balls, etc.) and their positions within the picture (coordinate positions within the same picture, etc.) may be stored as metadata. In this case, the decoding device 200 can identify the position of a desired object based on the metadata and determine the tile containing that object. For example, as shown in Figure 6, the metadata is stored using a data storage structure different from the image data, such as SEI in HEVC. This metadata indicates, for example, the position, size, or color of the main object.

[0147] Furthermore, metadata may be stored in units consisting of multiple pictures, such as streams, sequences, or random access units. This allows the decoding device 200 to obtain information such as the time when a specific person appears in the video, and by using that time and the information in the picture units, it can identify the picture in which the object exists and the position of the object within that picture.

[0148] [Encoding device] Next, the encoding device 100 according to the embodiment will be described. Figure 7 is a block diagram showing an example of the configuration of the encoding device 100 according to the embodiment. The encoding device 100 encodes images in block units.

[0149] As shown in Figure 7, the encoding device 100 is a device that encodes an image in block units and comprises a division unit 102, a subtraction unit 104, a transformation unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transformation unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. The intra prediction unit 124 and the inter prediction unit 126 are each configured as part of the prediction processing unit.

[0150] [Example of an encoding device implementation] Figure 8 is a block diagram showing an example implementation of the encoding device 100. The encoding device 100 includes a processor a1 and memory a2. For example, the multiple components of the encoding device 100 shown in Figure 7 are implemented by the processor a1 and memory a2 shown in Figure 8.

[0151] Processor a1 is a circuit that performs information processing and is a circuit that can access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit for encoding images. Processor a1 may be a processor such as a CPU. Alternatively, processor a1 may be a collection of multiple electronic circuits. Furthermore, for example, processor a1 may play the role of multiple components of the encoding device 100 shown in Figure 7, excluding the component for storing information.

[0152] Memory a2 is a dedicated or general-purpose memory in which information for the processor a1 to encode an image is stored. Memory a2 may be an electronic circuit and may be connected to the processor a1. Memory a2 may also be included in the processor a1. Memory a2 may also be a collection of multiple electronic circuits. Memory a2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory a2 may also be a non-volatile memory or a volatile memory.

[0153] For example, memory a2 may store the image to be encoded, or it may store a stream corresponding to the encoded image. Alternatively, memory a2 may store a program for processor a1 to encode the image.

[0154] Furthermore, for example, memory a2 may play the role of an information storage component among the multiple components of the encoding device 100 shown in Figure 7. Specifically, memory a2 may play the role of the block memory 118 and frame memory 122 shown in Figure 7. More specifically, memory a2 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0155] Furthermore, the encoding device 100 does not necessarily have to implement all of the components shown in Figure 7, nor does it have to perform all of the processes described above. Some of the components shown in Figure 7 may be included in other devices, and some of the processes described above may be performed by other devices.

[0156] The following describes the overall processing flow of the encoding device 100, followed by a description of each component included in the encoding device 100.

[0157] [Overall flow of the encoding process] Figure 9 is a flowchart showing an example of the overall encoding process performed by the encoding device 100.

[0158] First, the division unit 102 of the encoding device 100 divides the picture contained in the original image into multiple fixed-size blocks (128 x 128 pixels) (step Sa_1). Then, the division unit 102 selects a division pattern for each of these fixed-size blocks (step Sa_2). In other words, the division unit 102 further divides the fixed-size blocks into multiple blocks that constitute the selected division pattern. Then, the encoding device 100 performs steps Sa_3 to Sa_9 for each of these multiple blocks.

[0159] The prediction processing unit, consisting of the intra-prediction unit 124 and the inter-prediction unit 126, and the prediction control unit 128 generate a predicted image of the current block (step Sa_3). The predicted image is also called a predicted signal, predicted block, or predicted sample.

[0160] Next, the subtraction unit 104 generates the difference between the current block and the predicted image as the predicted residual (step Sa_4). The predicted residual is also called the prediction error.

[0161] Next, the transformation unit 106 and the quantization unit 108 generate multiple quantization coefficients by performing transformation and quantization on the predicted image (step Sa_5).

[0162] Next, the entropy coding unit 110 generates a stream by coding (specifically, entropy coding) its multiple quantization coefficients and prediction parameters related to the generation of the predicted image (step Sa_6).

[0163] Next, the inverse quantization unit 112 and the inverse transformation unit 114 restore the predicted residuals by performing inverse quantization and inverse transformation on a plurality of quantization coefficients (step Sa_7).

[0164] Next, the summing unit 116 reconstructs the current block by adding the predicted image to the recovered predicted residual (step Sa_8). This generates a reconstructed image. The reconstructed image is also called a reconstructed block, and the reconstructed image generated by the encoding device 100 is also called a local decoded block or local decoded image.

[0165] Once this reconstructed image is generated, the loop filter unit 120 performs filtering on the reconstructed image as needed (step Sa_9).

[0166] Then, the encoding device 100 determines whether or not the encoding of the entire picture is complete (step Sa_10). If it determines that it is not complete (No. in step Sa_10), it repeats the process from step Sa_2.

[0167] In the example described above, the encoding device 100 selects one partitioning pattern for a fixed-size block and encodes each block according to that partitioning pattern. However, it may also encode each block according to multiple partitioning patterns. In this case, the encoding device 100 may evaluate the cost of each of the multiple partitioning patterns and, for example, select the stream obtained by encoding according to the partitioning pattern with the smallest cost as the final output stream.

[0168] Furthermore, the processes in steps Sa_1 to Sa_10 may be performed sequentially by the encoding device 100, some of these processes may be performed in parallel, and the order may be changed.

[0169] The encoding process performed by such an encoding device 100 is a hybrid encoding using predictive encoding and transformative encoding. Furthermore, predictive encoding is performed by an encoding loop consisting of a subtraction unit 104, a transformer unit 106, a quantization unit 108, an inverse quantization unit 112, an inverse transformer unit 114, an addition unit 116, a loop filter unit 120, a block memory 118, a frame memory 122, an intra-prediction unit 124, an inter-prediction unit 126, and a prediction control unit 128. In other words, the prediction processing unit consisting of the intra-prediction unit 124 and the inter-prediction unit 126 constitutes a part of the encoding loop.

[0170] [Divided part] The splitting unit 102 splits each picture included in the original image into a plurality of blocks, and outputs each block to the subtraction unit 104. For example, the splitting unit 102 first splits the picture into blocks of a fixed size (e.g., 128x128 pixels). These blocks of fixed size are sometimes called coding tree units (CTUs). Then, the splitting unit 102 splits each of the fixed-size blocks into blocks of a variable size (e.g., 64x64 pixels or less) based on, for example, recursive quadtree and / or binary tree block splitting. That is, the splitting unit 102 selects a splitting pattern. These variable-size blocks are sometimes called coding units (CUs), prediction units (PUs), or transform units (TUs). Note that in various implementation examples, it is not necessary to distinguish between CUs, PUs, and TUs, and some or all of the blocks in the picture may be processing units of CUs, PUs, or TUs.

[0171] FIG. 10 is a diagram showing an example of block splitting in the embodiment. In FIG. 10, the solid lines represent block boundaries by quadtree block splitting, and the dashed lines represent block boundaries by binary tree block splitting.

[0172] Here, the block 10 is a square block of 128x128 pixels. This block 10 is first split into four square blocks of 64x64 pixels (quadtree block splitting).

[0173] The upper-left square block of 64x64 pixels is further vertically split into two rectangular blocks each consisting of 32x64 pixels, and the left rectangular block of 32x64 pixels is further vertically split into two rectangular blocks each consisting of 16x64 pixels (binary tree block splitting). As a result, the upper-left square block of 64x64 pixels is split into two rectangular blocks 11 and 12 of 16x64 pixels and a rectangular block 13 of 32x64 pixels.

[0174] The 64x64 pixel square block in the upper right is horizontally divided into two rectangular blocks 14 and 15, each consisting of 64x32 pixels (binary tree block partitioning).

[0175] The 64x64 pixel square block in the bottom left is divided into four 32x32 pixel square blocks (quadrutree block partitioning). Of these four 32x32 pixel square blocks, the top left and bottom right blocks are further divided. The top left 32x32 pixel square block is vertically divided into two 16x32 pixel rectangular blocks, and the right 16x32 pixel rectangular block is further horizontally divided into two 16x16 pixel square blocks (binary tree block partitioning). The bottom right 32x32 pixel square block is horizontally divided into two 32x16 pixel rectangular blocks (binary tree block partitioning). As a result, the 64x64 pixel square block in the lower left is divided into a 16x32 pixel rectangular block 16, two 16x16 pixel square blocks 17 and 18, two 32x32 pixel square blocks 19 and 20, and two 32x16 pixel rectangular blocks 21 and 22.

[0176] Block 23, consisting of 64x64 pixels in the lower right corner, will not be divided.

[0177] As described above, in Figure 10, block 10 is divided into 13 variable-sized blocks 11-23 based on a recursive quad-tree and binary tree block partition. Such a partition is sometimes called a QTBT (quad-tree plus binary tree) partition.

[0178] In Figure 10, one block was divided into four or two blocks (quadrutree or binary tree block partitioning), but partitioning is not limited to these. For example, one block may be divided into three blocks (ternary tree block partitioning). Partitioning that includes such ternary tree block partitioning is sometimes called MBT (multi-type tree) partitioning.

[0179] Figure 11 shows an example of the configuration of the division unit 102. As shown in Figure 11, the division unit 102 may include a block division determination unit 102a. The block division determination unit 102a may perform the following processing as an example.

[0180] The block division determination unit 102a collects block information from, for example, the block memory 118 or the frame memory 122, and determines the division pattern based on that block information. The division unit 102 divides the original image according to the division pattern and outputs one or more blocks obtained by the division to the subtraction unit 104.

[0181] Furthermore, the block division determination unit 102a outputs parameters indicating the division pattern described above to the conversion unit 106, the inverse conversion unit 114, the intra prediction unit 124, the inter prediction unit 126, and the entropy coding unit 110. The conversion unit 106 may convert the prediction residuals based on these parameters, and the intra prediction unit 124 and the inter prediction unit 126 may generate a prediction image based on these parameters. The entropy coding unit 110 may also perform entropy coding on these parameters.

[0182] The parameters related to the splitting pattern may be written to the stream as follows, for example.

[0183] Figure 12 shows examples of division patterns. Division patterns include, for example, quad division (QT), where the block is divided into two horizontally and two vertically; 3 divisions (HT or VT), where the block is divided in the same direction in a 1:2:1 ratio; 2 divisions (HB or VB), where the block is divided in the same direction in a 1:1 ratio; and no division (NS).

[0184] Note that in the case of 4 divisions and no division, the division pattern does not have a block division direction, while in the case of 2 divisions and 3 divisions, the division pattern has division direction information.

[0185] Figures 13A and 13B show examples of syntax trees for splitting patterns. In the example in Figure 13A, first there is information indicating whether or not to split (S: Split flag), then there is information indicating whether or not to split into four (QT: QT flag), then there is information indicating whether or not to split into three or two (TT: TT flag or BT: BT flag), and finally there is information indicating the direction of splitting (Ver: Vertical flag or Hor: Horizontal flag). Note that the same splitting process may be repeatedly applied to each of the one or more blocks obtained by splitting using such a splitting pattern. That is, as an example, the determination of whether or not to split, whether or not to split into four, whether the splitting method is horizontal or vertical, and whether or not to split into three or two may be performed recursively, and the results of the determinations performed may be encoded into a stream according to the encoding order disclosed in the syntax tree shown in Figure 13A.

[0186] Furthermore, in the syntax tree shown in Figure 13A, the information is arranged in the order of S, QT, TT, and Ver, but it may also be arranged in the order of S, QT, Ver, and BT. In other words, in the example in Figure 13B, first there is information indicating whether or not to perform a split (S: Split flag), then there is information indicating whether or not to perform a four-way split (QT: QT flag). Next there is information indicating the direction of the split (Ver: Vertical flag or Hor: Horizontal flag), and finally there is information indicating whether to perform a two-way split or a three-way split (BT: BT flag or TT: TT flag).

[0187] Note that the division patterns described here are just examples; you may use other division patterns, or only a part of the division patterns described.

[0188] [Subtraction Unit] The subtraction unit 104 subtracts the predicted image (the predicted image input from the prediction control unit 128) from the original image in block units that are input from the division unit 102 and divided by the division unit 102. In other words, the subtraction unit 104 calculates the predicted residual of the current block. The subtraction unit 104 then outputs the calculated predicted residual to the conversion unit 106.

[0189] The original image is the input signal to the encoding device 100, and is, for example, a signal representing the image of each picture that makes up the video (e.g., a luminance (luma) signal and two chroma (chroma) signals).

[0190] [Conversion section] The conversion unit 106 converts the predicted residual in the spatial domain into conversion coefficients in the frequency domain and outputs the conversion coefficients to the quantization unit 108. Specifically, the conversion unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the predicted residual in the spatial domain, for example.

[0191] The transformation unit 106 may adaptively select a transformation type from among several transformation types and convert the predicted residuals into transformation coefficients using a transformation basis function corresponding to the selected transformation type. Such a transformation is sometimes called an EMT (explicit multiple core transform) or an AMT (adaptive multiple transform). The transformation basis function is sometimes simply called a basis.

[0192] Multiple transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. These transformation types may also be denoted as DCT2, DCT5, DCT8, DST1, and DST7, respectively. Figure 14 is a table showing the transformation basis functions corresponding to each transformation type. In Figure 14, N represents the number of input pixels. The selection of a transformation type from among these multiple transformation types may depend, for example, on the type of prediction (intra-prediction and inter-prediction, etc.) or on the intra-prediction mode.

[0193] Information indicating whether to apply such EMT or AMT (for example, called an EMT flag or an AMT flag) and information indicating the selected conversion type are usually signaled at the CU level. Note that the signaling of this information does not have to be limited to the CU level and may be at other levels (for example, sequence level, picture level, slice level, block level, or CTU level).

[0194] Also, the conversion unit 106 may re-convert the conversion coefficients (that is, the conversion results). Such re-conversion is sometimes called AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the conversion unit 106 performs re-conversion for each sub-block (for example, a 4x4 pixel sub-block) included in the block of conversion coefficients corresponding to the intra prediction residual. Information indicating whether to apply NSST and information regarding the conversion matrix used for NSST are usually signaled at the CU level. Note that the signaling of this information does not have to be limited to the CU level and may be at other levels (for example, sequence level, picture level, slice level, block level, or CTU level).

[0195] Both separable conversion and non-separable conversion may be applied to the conversion unit 106. Separable conversion is a method in which conversion is performed multiple times by separating for each direction by the number of dimensions of the input, and non-separable conversion is a method in which when the input is multi-dimensional, two or more dimensions are regarded as one dimension and conversion is performed together.

[0196] For example, as an example of non-separable conversion, when the input is a 4×4 pixel block, it is regarded as an array having 16 elements, and a conversion process is performed on the array with a 16×16 conversion matrix.

[0197] Another example of a non-separable transformation is a transformation (Hypercube Givens Transform) in which a 4x4 pixel input block is treated as a single array with 16 elements, and then multiple Givens rotations are performed on that array.

[0198] In the conversion unit 106, the conversion type of the conversion basis function used to convert to the frequency domain can be switched depending on the region within the CU. One example is SVT (Spatially Varying Transform).

[0199] Figure 15 shows an example of an SVT.

[0200] In SVT, as shown in Figure 15, the CU is divided into two equal parts horizontally or vertically, and only one of the regions is converted to the frequency domain. The conversion type may be set for each region, for example, DST7 and DCT8 can be used. For example, of the two regions obtained by dividing the CU vertically, DST7 and DCT8 may be used for the region at position 0. Or, of those two regions, DST7 is used for the region at position 1. Similarly, of the two regions obtained by dividing the CU horizontally, DST7 and DCT8 are used for the region at position 0. Or, of those two regions, DST7 is used for the region at position 1. In this example shown in Figure 15, only one of the two regions in the CU is converted, and the other is not, but it is also possible to convert both regions. Furthermore, the division method may be not only bisecting but also quartet-dividing. It is also possible to make it more flexible by encoding information indicating the division method and signaling it in the same way as CU division. Note that SVT is sometimes also called SBT (Sub-block Transform).

[0201] The aforementioned AMT and EMT may also be called MTS (Multiple Transform Selection). When applying MTS, a transformation type such as DST7 or DCT8 can be selected, and information indicating the selected transformation type may be encoded as index information for each CU. On the other hand, there is a process called IMTS (Implicit MTS) that selects the transformation type to be used for orthogonal transformation based on the shape of the CU without encoding index information. When applying IMTS, for example, if the shape of the CU is rectangular, the shorter side of the rectangle is orthogonally transformed using DST7, and the longer side is orthogonally transformed using DCT2. Also, for example, if the shape of the CU is square, if MTS is enabled in the sequence, DCT2 is used, and if MTS is disabled, DST7 is used to perform the orthogonal transformation. DCT2 and DST7 are just examples, and other transformation types may be used, and different combinations of transformation types can be used. IMTS may be usable only in intra-prediction blocks, or it may be usable in both intra-prediction blocks and inter-prediction blocks.

[0202] The above describes three selection processes, MTS, SBT, and IMTS, which selectively switch the transformation type used for orthogonal transformations. All three selection processes may be enabled, or only some of them may be selectively enabled. Whether or not an individual selection process is enabled can be identified by flag information in the header, such as SPS. For example, if all three selection processes are enabled, one of the three selection processes is selected for each CU to perform the orthogonal transformation. Note that the selection process for selectively switching the transformation type may be different from the three selection processes described above, as long as at least one of the following four functions [1] to [4] can be implemented, and each of the three selection processes may be replaced with another process. Function [1] is a function that orthogonally transforms the entire range within the CU and encodes information indicating the transformation type used for the transformation. Function [2] is a function that orthogonally transforms the entire range within the CU and determines the transformation type based on predetermined rules without encoding information indicating the transformation type. Function [3] is a function that orthogonally transforms a portion of the CU and encodes information indicating the transformation type used for the transformation. Functions [4] include the ability to orthogonally transform a portion of the CU and determine the transformation type based on predetermined rules without encoding information indicating the transformation type used.

[0203] Furthermore, the application of MTS, IMTS, and SBT may be determined on a per-processing-unit basis. For example, the application may be determined on a per-sequence, per-picture, per-brick, per-slice, per-CTU, or per-CU basis.

[0204] Furthermore, the tool for selectively switching the conversion type in this disclosure may be rephrased as a method for adaptively selecting a base to be used in the conversion process, a selection process, or a process for selecting a base. Alternatively, the tool for selectively switching the conversion type may be rephrased as a mode for adaptively selecting the conversion type.

[0205] Figure 16 is a flowchart showing an example of processing by the conversion unit 106.

[0206] For example, the transformation unit 106 determines whether or not to perform an orthogonal transformation (step St_1). If the transformation unit 106 determines to perform an orthogonal transformation (Yes in step St_1), it selects a transformation type to be used for the orthogonal transformation from among multiple transformation types (step St_2). Next, the transformation unit 106 performs the orthogonal transformation by applying the selected transformation type to the predicted residual of the current block (step St_3). Then, the transformation unit 106 outputs information indicating the selected transformation type to the entropy encoding unit 110, thereby encoding that information (step St_4). On the other hand, if the transformation unit 106 determines not to perform an orthogonal transformation (No in step St_1), it outputs information indicating that an orthogonal transformation will not be performed to the entropy encoding unit 110, thereby encoding that information (step St_5). Note that the determination of whether or not to perform an orthogonal transformation in step St_1 may be based on, for example, the size of the transformation block, the prediction mode applied to the CU, etc. Furthermore, information indicating the type of transformation used for the orthogonal transformation is not encoded, and the orthogonal transformation may be performed using a predetermined transformation type.

[0207] Figure 17 is a flowchart showing another example of processing by the transformation unit 106. Note that the example shown in Figure 17, like the example shown in Figure 16, is an example of an orthogonal transformation where a method of selectively switching the transformation type used for the orthogonal transformation is applied.

[0208] For example, the first group of conversion types may include DCT2, DST7, and DCT8. Also, for example, the second group of conversion types may include DCT2. Furthermore, the conversion types included in the first group of conversion types and the second group of conversion types may overlap in some respects, or they may all be different conversion types.

[0209] Specifically, the transformation unit 106 determines whether the transformation size is less than or equal to a predetermined value (step Su_1). If it determines that it is less than or equal to the predetermined value (Yes in step Su_1), the transformation unit 106 orthogonally transforms the predicted residual of the current block using the transformation types included in the first transformation type group (step Su_2). Next, the transformation unit 106 outputs information to the entropy encoding unit 110 indicating which of the one or more transformation types included in the first transformation type group will be used, thereby encoding that information (step Su_3). On the other hand, if the transformation unit 106 determines that the transformation size is not less than or equal to a predetermined value (No in step Su_1), it orthogonally transforms the predicted residual of the current block using the second transformation type group (step Su_4).

[0210] In step Su_3, the information indicating the transformation type used for the orthogonal transformation may be information indicating a combination of transformation types applied vertically and horizontally to the current block. Also, the first group of transformation types may contain only one transformation type, and the information indicating the transformation type used for the orthogonal transformation does not need to be encoded. The second group of transformation types may contain multiple transformation types, and the information indicating the transformation type used for the orthogonal transformation from one or more transformation types included in the second group of transformation types may be encoded.

[0211] Furthermore, the conversion type may be determined solely based on the conversion size. Note that if the process determines the conversion type used for orthogonal transformations based on the conversion size, it is not limited to determining whether the conversion size is less than or equal to a predetermined value.

[0212] [Quantization section] The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization unit 108 scans a plurality of conversion coefficients in the current block in a predetermined scanning order and quantizes the conversion coefficients based on the quantization parameter (QP) corresponding to the scanned conversion coefficients. The quantization unit 108 then outputs the plurality of quantized conversion coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy coding unit 110 and the inverse quantization unit 112.

[0213] A predetermined scan order is the order for quantization / inverse quantization of the transformation coefficients. For example, a predetermined scan order is defined as ascending frequency (from low to high frequency) or descending frequency (from high to low frequency).

[0214] The quantization parameter (QP) is a parameter that defines the quantization step (quantization width). For example, if the value of the quantization parameter increases, the quantization step also increases. In other words, if the value of the quantization parameter increases, the error in the quantization coefficient (quantization error) increases.

[0215] Furthermore, quantization matrices may be used for quantization. For example, several types of quantization matrices may be used corresponding to frequency conversion sizes such as 4x4 and 8x8, prediction modes such as intra-prediction and inter-prediction, and pixel components such as luminance and chrominance. Quantization refers to the process of digitizing values ​​sampled at predetermined intervals and associating them with predetermined levels. In this field, terms such as rounding, scaling, or scaling may also be used.

[0216] There are two methods for using the quantization matrix: using a quantization matrix directly set on the encoding device 100, and using a default quantization matrix (default matrix). On the encoding device 100, by directly setting the quantization matrix, a quantization matrix corresponding to the image features can be set. However, in this case, there is a disadvantage that the amount of code increases due to the encoding of the quantization matrix. Alternatively, instead of using the default quantization matrix or the encoded quantization matrix as is, a quantization matrix to be used for quantizing the current block may be generated based on the default quantization matrix or the encoded quantization matrix.

[0217] On the other hand, there is also a method that does not use a quantization matrix, and quantizes both the high-frequency and low-frequency components in the same way. This method is equivalent to using a quantization matrix where all coefficients have the same value (a flat matrix).

[0218] The quantization matrix may be encoded, for example, at the sequence level, picture level, slice level, brick level, or CTU level.

[0219] When a quantization matrix is ​​used, the quantization unit 108 scales the quantization width, which is obtained from quantization parameters, for each conversion coefficient using the values ​​of the quantization matrix. A quantization process performed without a quantization matrix may be a process that quantizes the conversion coefficients based on the quantization width obtained from quantization parameters. In a quantization process performed without a quantization matrix, the quantization width may be multiplied by a predetermined value common to all conversion coefficients within a block.

[0220] Figure 18 is a block diagram showing an example of the configuration of the quantization unit 108.

[0221] The quantization unit 108 includes, for example, a differential quantization parameter generation unit 108a, a predictive quantization parameter generation unit 108b, a quantization parameter generation unit 108c, a quantization parameter storage unit 108d, and a quantization processing unit 108e.

[0222] Figure 19 is a flowchart showing an example of quantization by the quantization unit 108.

[0223] For example, the quantization unit 108 may perform quantization for each CU based on the flowchart shown in Figure 19. Specifically, the quantization parameter generation unit 108c determines whether or not to perform quantization (step Sv_1). If it is determined that quantization should be performed (Yes in step Sv_1), the quantization parameter generation unit 108c generates quantization parameters for the current block (step Sv_2) and stores these quantization parameters in the quantization parameter storage unit 108d (step Sv_3).

[0224] Next, the quantization processing unit 108e quantizes the conversion coefficients of the current block using the quantization parameters generated in step Sv_2 (step Sv_4). Then, the prediction quantization parameter generation unit 108b obtains quantization parameters for a processing unit different from the current block from the quantization parameter storage unit 108d (step Sv_5). Based on the obtained quantization parameters, the prediction quantization parameter generation unit 108b generates prediction quantization parameters for the current block (step Sv_6). The difference quantization parameter generation unit 108a calculates the difference between the quantization parameters of the current block generated by the quantization parameter generation unit 108c and the prediction quantization parameters of the current block generated by the prediction quantization parameter generation unit 108b (step Sv_7). The difference quantization parameters are generated by calculating this difference. The differential quantization parameter generation unit 108a outputs the differential quantization parameter to the entropy encoding unit 110, thereby encoding the differential quantization parameter (step Sv_8).

[0225] The differential quantization parameters may be encoded at the sequence level, picture level, slice level, brick level, or CTU level. The initial values ​​of the quantization parameters may also be encoded at the sequence level, picture level, slice level, brick level, or CTU level. In this case, the quantization parameters may be generated using the initial values ​​of the quantization parameters and the differential quantization parameters.

[0226] The quantization unit 108 may be equipped with multiple quantizers, and may also apply dependent quantization, which quantizes the conversion coefficients using a quantization method selected from multiple quantization methods.

[0227] [Entropy coding unit] Figure 20 is a block diagram showing an example of the configuration of the entropy coding unit 110.

[0228] The entropy coding unit 110 generates a stream by performing entropy coding on the quantization coefficients input from the quantization unit 108 and the prediction parameters input from the prediction parameter generation unit 130. For example, CABAC (Context-based Adaptive Binary Arithmetic Coding) is used for this entropy coding. Specifically, the entropy coding unit 110 comprises, for example, a binarization unit 110a, a context control unit 110b, and a binary arithmetic coding unit 110c. The binarization unit 110a performs binarization, converting multi-level signals such as quantization coefficients and prediction parameters into binary signals. Examples of binarization methods include Truncated Rice Binarization, Exponential Golomb codes, and Fixed Length Binarization. The context control unit 110b derives context values, i.e., the probability of a binary signal occurring, according to the characteristics of the syntax elements or the surrounding circumstances. Methods for deriving this context value include, for example, bypass, syntax element referencing, upper / left adjacent block referencing, hierarchical information referencing, and others. The binary arithmetic coding unit 110c performs arithmetic coding on the binarized signal using the derived context value.

[0229] Figure 21 shows the CABAC flow in the entropy coding unit 110.

[0230] First, the CABAC in the entropy coding unit 110 is initialized. This initialization involves initializing the binary arithmetic coding unit 110c and setting an initial context value. Then, the binarization unit 110a and the binary arithmetic coding unit 110c sequentially perform binarization and arithmetic coding for each of the multiple quantization coefficients of the CTU, for example. At this time, the context control unit 110b updates the context value each time arithmetic coding is performed. Then, as a post-processing step, the context control unit 110b saves the context value. This saved context value is used, for example, as the initial value of the context value for the next CTU.

[0231] [Dequantization section] The inverse quantization unit 112 inversely quantizes the quantization coefficients input from the quantization unit 108. Specifically, the inverse quantization unit 112 inversely quantizes the quantization coefficients of the current block in a predetermined scanning order. Then, the inverse quantization unit 112 outputs the inversely quantized conversion coefficients of the current block to the inverse conversion unit 114.

[0232] [Inverse Transformation Section] The inverse transform unit 114 restores the predicted residual by performing an inverse transform on the transformation coefficients input from the inverse quantization unit 112. Specifically, the inverse transform unit 114 restores the predicted residual of the current block by performing an inverse transform on the transformation coefficients corresponding to the transformation by the transformation unit 106. The inverse transform unit 114 then outputs the restored predicted residual to the summation unit 116.

[0233] Furthermore, the recovered prediction residuals usually do not match the prediction error calculated by the subtraction unit 104 because information is typically lost due to quantization. In other words, the recovered prediction residuals usually contain quantization errors.

[0234] [Addition section] The adder 116 reconstructs the current block by adding the predicted residual input from the inverse transformer 114 and the predicted image input from the prediction control unit 128. As a result, a reconstructed image is generated. The adder 116 then outputs the reconstructed image to the block memory 118 and the loop filter unit 120.

[0235] [Block memory] The block memory 118 is a storage unit for storing blocks within the current picture, such as blocks referenced in intra prediction. Specifically, the block memory 118 stores the reconstructed image output from the adder 116.

[0236] [Frame memory] The frame memory 122 is a storage unit for storing reference pictures used, for example, in interpretation, and is sometimes called a frame buffer. Specifically, the frame memory 122 stores the reconstructed image filtered by the loop filter unit 120.

[0237] [Loop Filter Section] The loop filter unit 120 applies loop filtering to the reconstructed image output from the summing unit 116 and outputs the filtered reconstructed image to the frame memory 122. A loop filter is a filter used within the encoding loop (in-loop filter), and includes, for example, adaptive loop filters (ALF), deblocking filters (DF or DBF), and sample adaptive offsets (SAO).

[0238] Figure 22 is a block diagram showing an example of the configuration of the loop filter section 120.

[0239] The loop filter unit 120 includes, for example, a deblocking filter processing unit 120a, an SAO processing unit 120b, and an ALF processing unit 120c, as shown in Figure 22. The deblocking filter processing unit 120a applies the above-described deblocking filter processing to the reconstructed image. The SAO processing unit 120b applies the above-described SAO processing to the reconstructed image after the deblocking filter processing. The ALF processing unit 120c applies the above-described ALF processing to the reconstructed image after the SAO processing. Details of ALF and the deblocking filter will be described later. SAO processing is a process that improves image quality by reducing ringing (a phenomenon in which pixel values ​​are distorted in a wave-like manner around edges) and correcting the shift in pixel values. Examples of SAO processing include edge offset processing and band offset processing. Note that the loop filter unit 120 does not have to include all of the processing units disclosed in Figure 22, and may include only some of the processing units. Furthermore, the loop filter unit 120 may be configured to perform the above-mentioned processing in an order different from the processing order disclosed in Figure 22.

[0240] [Loop Filter Section > Adaptive Loop Filter] In ALF, a least-squares error filter is applied to remove coding distortion. For example, for each 2x2 pixel subblock within the current block, one filter selected from several filters is applied based on the direction and activity of the local gradient.

[0241] Specifically, first, subblocks (for example, 2x2 pixel subblocks) are classified into multiple classes (for example, 15 or 25 classes). The classification of subblocks is done, for example, based on the direction of the gradient and the activity level. In a specific example, a classification value C (for example, C = 5D + A) is calculated using the gradient direction value D (for example, 0 to 2 or 0 to 4) and the gradient activity value A (for example, 0 to 4). Then, based on the classification value C, the subblocks are classified into multiple classes.

[0242] The gradient direction value D is derived, for example, by comparing gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). The gradient activation value A is derived, for example, by adding the gradients in multiple directions and quantizing the sum.

[0243] Based on the results of this classification, a filter for the subblock is determined from among multiple filters.

[0244] For example, a circularly symmetric shape is used for filters in ALF. Figures 23A to 23C show several examples of filter shapes used in ALF. Figure 23A shows a 5x5 diamond-shaped filter, Figure 23B shows a 7x7 diamond-shaped filter, and Figure 23C shows a 9x9 diamond-shaped filter. Information indicating the filter shape is usually signaled at the picture level. However, the signaling of information indicating the filter shape is not limited to the picture level and may be at other levels (e.g., sequence level, slice level, brick level, CTU level, or CU level).

[0245] The on / off status of ALF may be determined, for example, at the picture level or CU level. For example, the decision to apply ALF may be made at the CU level for luminance, and at the picture level for color difference. Information indicating whether ALF is on or off is usually signaled at the picture level or CU level. However, the signaling of information indicating whether ALF is on or off is not limited to the picture level or CU level, and may be at other levels (e.g., sequence level, slice level, brick level, or CTU level).

[0246] Furthermore, as described above, one filter is selected from among several filters and the ALF process is applied to the subblock. For each of these filters (for example, up to 15 or 25 filters), the set of coefficients used in that filter is usually signaled at the picture level. However, the signaling of the coefficient set is not limited to the picture level; it may be at other levels (for example, sequence level, slice level, brick level, CTU level, CU level, or subblock level).

[0247] [Loop Filters > Cross Component Adaptive Loop Filter] Figure 23D shows an example where the Y sample (first component) is used in Cb CCALF and Cr CCALF (multiple components different from the first component). Figure 23E shows a diamond-shaped filter.

[0248] One example of CC-ALF operates by applying a linear diamond-shaped filter (Figures 23D and 23E) to the luminance channel of each chrominance component. For example, the filter coefficients are transmitted in APS, scaled by a factor of 2^10, and rounded for fixed-point representation. Filter application is controlled by a variable block size and notified by context-encoded flags received for each block of samples. The block size and CC-ALF enable flags are received at the slice level of each chrominance component. The syntax and semantics of CC-ALF are provided in the Appendix. The documentation supports block sizes of 16x16, 32x32, 64x64, and 128x128 (for chrominance samples).

[0249] [Loop Filters > Joint Chroma Cross Component Adaptive Loop Filter] Figure 23F shows an example of JC-CCALF. Figure 23G shows an example of a candidate weight_index for JC-CCALF.

[0250] One example of JC-CCALF uses only one CCALF filter to generate a single CCALF filter output as a color difference adjustment signal for only one color component, and then applies appropriately weighted versions of the same color difference adjustment signal to the other color components. In this way, the complexity of existing CCALF is roughly halved.

[0251] The weight values ​​are encoded into a sign flag and a weight index. The weight index (indicated as weight_index) is encoded in 3 bits and specifies the magnitude of the JC-CCALF weight JcCcWeight. It cannot be equal to 0. The magnitude of JcCcWeight is determined as follows:

[0252] • If weight_index is 4 or less, JcCcWeight is equal to weight_index>>2.

[0253] Otherwise, JcCcWeight is equal to 4 / (weight_index-4).

[0254] The block-level on / off control for ALF filtering of Cb and Cr is separate. This is the same as CCALF, where two separate sets of block-level on / off control flags are encoded. Here, unlike CCALF, the on / off control block sizes of Cb and Cr are the same, so only one block size variable is encoded.

[0255] [Loop Filter Section > Deblocking Filter] In the deblocking filtering process, the loop filter unit 120 reduces distortion occurring at the block boundaries of the reconstructed image by applying a filter to those block boundaries.

[0256] Figure 24 is a block diagram showing an example of a detailed configuration of the deblocking filter processing unit 120a.

[0257] The deblocking filter processing unit 120a includes, for example, a boundary determination unit 1201, a filter determination unit 1203, a filter processing unit 1205, a processing determination unit 1208, a filter characteristic determination unit 1207, and switches 1202, 1204, and 1206.

[0258] The boundary determination unit 1201 determines whether or not a pixel to be deblocked and filtered (i.e., a target pixel) is located near a block boundary. The boundary determination unit 1201 then outputs the determination result to the switch 1202 and the processing determination unit 1208.

[0259] If the boundary determination unit 1201 determines that the target pixel is located near a block boundary, switch 1202 outputs the image before filtering to switch 1204. Conversely, if the boundary determination unit 1201 determines that the target pixel is not located near a block boundary, switch 1202 outputs the image before filtering to switch 1206. The image before filtering consists of the target pixel and at least one surrounding pixel located around that target pixel.

[0260] The filter determination unit 1203 determines whether or not to perform a deblocking filter on the target pixel based on the pixel values ​​of at least one surrounding pixel located around the target pixel. The filter determination unit 1203 then outputs the determination result to the switch 1204 and the processing determination unit 1208.

[0261] If the filter determination unit 1203 determines that deblocking filtering should be performed on the target pixel, switch 1204 outputs the pre-filtered image acquired via switch 1202 to the filter processing unit 1205. Conversely, if the filter determination unit 1203 determines that deblocking filtering should not be performed on the target pixel, switch 1204 outputs the pre-filtered image acquired via switch 1202 to switch 1206.

[0262] When the filter processing unit 1205 acquires an image before filtering via switches 1202 and 1204, it performs a deblocking filter process on the target pixel, using the filter characteristics determined by the filter characteristic determination unit 1207. The filter processing unit 1205 then outputs the filtered pixel to switch 1206.

[0263] Switch 1206 selectively outputs pixels that have not undergone deblocking and filtering, and pixels that have undergone deblocking and filtering by the filtering processing unit 1205, in accordance with the control by the processing determination unit 1208.

[0264] The processing determination unit 1208 controls the switch 1206 based on the determination results of the boundary determination unit 1201 and the filter determination unit 1203. Specifically, if the boundary determination unit 1201 determines that a target pixel is near a block boundary, and the filter determination unit 1203 determines that the target pixel should undergo deblocking and filtering, the processing determination unit 1208 outputs the deblocked and filtered pixel from the switch 1206. In other cases, the processing determination unit 1208 outputs the unfiltered pixel from the switch 1206. This output of pixels is repeated, resulting in the filtered image being output from the switch 1206. Note that the configuration shown in Figure 24 is just one example of the configuration of the deblocking and filtering processing unit 120a, and the deblocking and filtering processing unit 120a may have other configurations.

[0265] Figure 25 shows an example of a deblocking filter with symmetrical filter characteristics with respect to block boundaries.

[0266] In deblocking filtering, for example, one of two deblocking filters with different characteristics, namely a strong filter and a weak filter, is selected using pixel values ​​and quantization parameters. In the strong filter, as shown in Figure 25, if pixels p0~p2 and pixels q0~q2 exist on either side of a block boundary, the respective pixel values ​​of pixels q0~q2 are changed to pixel values ​​q'0~q'2 by performing the operation shown in the following equation.

[0267] q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8 q'1=(p0+q0+q1+q2+2) / 4 q'2=(p0+q0+q1+3×q2+2×q3+4) / 8

[0268] In the above equations, p0~p2 and q0~q2 are the pixel values ​​of pixels p0~p2 and pixels q0~q2, respectively. Also, q3 is the pixel value of pixel q3, which is adjacent to pixel q2 on the opposite side of the block boundary. Furthermore, the coefficient multiplied by the pixel value of each pixel used in the deblocking filter process on the right-hand side of each of the above equations is the filter coefficient.

[0269] Furthermore, in the deblocking filter process, clipping may be performed to ensure that the calculated pixel values ​​do not exceed a threshold. In this clipping process, the calculated pixel values ​​according to the above formula are clipped to "pre-calculation pixel value ± 2 × threshold" using a threshold determined from the quantization parameters. This prevents excessive smoothing.

[0270] Figure 26 illustrates an example of a block boundary where deblocking filtering is performed. Figure 27 shows an example of a BS value.

[0271] The block boundaries on which deblocking filtering is performed are, for example, the CU, PU, ​​or TU boundaries of an 8x8 pixel block as shown in Figure 26. Deblocking filtering is performed, for example, in units of 4 rows or 4 columns. First, for blocks P and Q shown in Figure 26, the Bs (Boundary Strength) value is determined as shown in Figure 27.

[0272] According to the Bs value in Figure 27, it may be determined whether or not to perform deblocking filtering of different strengths on block boundaries belonging to the same image. Deblocking filtering is performed on the chrominance signal when the Bs value is 2. Deblocking filtering is performed on the luminance signal when the Bs value is 1 or greater and predetermined conditions are met. Note that the criteria for determining the Bs value are not limited to those shown in Figure 27 and may be determined based on other parameters.

[0273] [Prediction Unit (Intra Prediction Unit, Inter Prediction Unit, Prediction Control Unit)] Figure 28 is a flowchart showing an example of processing performed in the prediction unit of the encoding device 100. For example, the prediction unit consists of all or some of the components of the intra-prediction unit 124, the inter-prediction unit 126, and the prediction control unit 128. The prediction processing unit includes, for example, the intra-prediction unit 124 and the inter-prediction unit 126.

[0274] The prediction unit generates a predicted image of the current block (step Sb_1). The predicted image may be, for example, an intra-prediction image (intra-prediction signal) or an inter-prediction image (inter-prediction signal). Specifically, the prediction unit generates a predicted image of the current block using a reconstructed image already obtained by generating predicted images for other blocks, generating prediction residuals, generating quantization coefficients, restoring the prediction residuals, and adding the predicted images.

[0275] The reconstructed image may be, for example, the image of the reference picture, or it may be the image of an encoded block (i.e., the other block mentioned above) within the current picture, which is the picture containing the current block. An encoded block within the current picture is, for example, an adjacent block to the current block.

[0276] Figure 29 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device 100.

[0277] The prediction unit generates a predicted image using a first method (step Sc_1a), a second method (step Sc_1b), and a third method (step Sc_1c). The first, second, and third methods are different methods for generating predicted images, and may be, for example, an interpretation method, an intraprediction method, and other prediction methods. These prediction methods may use the reconstructed images described above.

[0278] Next, the prediction unit evaluates the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_2). For example, the prediction unit calculates a cost C for each of the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c, and evaluates the predicted images by comparing their costs C. The cost C is calculated using the formula of the RD optimization model, for example, C = D + λ × R. In this formula, D is the coding distortion of the predicted image, and can be expressed as, for example, the sum of the absolute differences between the pixel values ​​of the current block and the pixel values ​​of the predicted image. R is the bitrate of the stream. λ is, for example, the Lagrange multiplier.

[0279] Next, the prediction unit selects one of the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_3). In other words, the prediction unit selects a method or mode to obtain the final predicted image. For example, the prediction unit selects the predicted image with the smallest cost C based on the cost C calculated for those predicted images. Alternatively, the evaluation in step Sc_2 and the selection of the predicted image in step Sc_3 may be based on parameters used in the coding process. The coding device 100 may signal information to identify the selected predicted image, method, or mode into a stream. This information may be, for example, a flag. This allows the decoding device 200 to generate a predicted image according to the method or mode selected by the coding device 100 based on this information. In the example shown in Figure 29, the prediction unit selects one of the predicted images after generating a predicted image for each method. However, the prediction unit may select a method or mode based on the parameters used in the coding process described above before generating those predicted images, and then generate the predicted image according to that method or mode.

[0280] For example, the first method and the second method are intra-prediction and inter-prediction, respectively, and the prediction unit may select the final predicted image for the current block from the predicted images generated according to these prediction methods.

[0281] Figure 30 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device 100.

[0282] First, the prediction unit generates a predicted image by intra-prediction (step Sd_1a) and then generates a predicted image by inter-prediction (step Sd_1b). The predicted image generated by intra-prediction is also called the intra-prediction image, and the predicted image generated by inter-prediction is also called the inter-prediction image.

[0283] Next, the prediction unit evaluates both the intra-predicted image and the inter-predicted image (step Sd_2). The cost C mentioned above may be used for this evaluation. The prediction unit may then select the prediction image with the smallest cost C from the intra-predicted image and the inter-predicted image as the final prediction image for the current block (step Sd_3). In other words, a prediction method or mode for generating the prediction image for the current block is selected.

[0284] [Intra Prediction Unit] The intra-prediction unit 124 generates a predicted image (i.e., an intra-predicted image) of the current block by performing intra-prediction (also called in-screen prediction) of the current block by referring to the block in the current picture stored in the block memory 118. Specifically, the intra-prediction unit 124 generates an intra-predicted image by performing intra-prediction by referring to the pixel values ​​(e.g., luminance value, chrominance value) of blocks adjacent to the current block, and outputs the intra-predicted image to the prediction control unit 128.

[0285] For example, the intra-prediction unit 124 performs intra-prediction using one of a predetermined set of intra-prediction modes. The set of intra-prediction modes typically includes one or more non-directional prediction modes and multiple directional prediction modes.

[0286] One or more non-directional prediction modes include, for example, the Planar prediction mode and DC prediction mode as defined in the H.265 / HEVC standard.

[0287] Multiple directional prediction modes include, for example, the 33 directional prediction modes defined in the H.265 / HEVC standard. Note that multiple directional prediction modes may also include 32 additional directional prediction modes (a total of 65 directional prediction modes). Figure 31 shows all 67 intra-prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in intra-prediction. Solid arrows represent the 33 directions defined in the H.265 / HEVC standard, and dashed arrows represent the additional 32 directions (the 2 non-directional prediction modes are not shown in Figure 31).

[0288] In various implementations, luminance blocks may be referenced in the intra-prediction of chrominance blocks. That is, the chrominance components of the current block may be predicted based on the luminance components of the current block. Such intra-prediction is sometimes called CCLM (cross-component linear model) prediction. Such an intra-prediction mode for chrominance blocks that references luminance blocks (e.g., called CCLM mode) may be added as one of the intra-prediction modes for chrominance blocks.

[0289] The intra-prediction unit 124 may correct the pixel values ​​after intra-prediction based on the gradient of the horizontal / vertical reference pixels. Intra-prediction with such correction is sometimes called PDPC (position dependent intra-prediction combination). Information indicating whether or not PDPC is applied (e.g., called a PDPC flag) is usually signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (e.g., sequence level, picture level, slice level, brick level, or CTU level).

[0290] Figure 32 is a flowchart showing an example of processing by the intra prediction unit 124.

[0291] The intra-prediction unit 124 selects one intra-prediction mode from a plurality of intra-prediction modes (step Sw_1). Then, the intra-prediction unit 124 generates a predicted image according to the selected intra-prediction mode (step Sw_2). Next, the intra-prediction unit 124 determines the MPM (Most Probable Modes) (step Sw_3). The MPM consists of, for example, six intra-prediction modes. Two of these six intra-prediction modes may be Planar prediction modes and DC prediction modes, and the remaining four modes may be directional prediction modes. Then, the intra-prediction unit 124 determines whether the intra-prediction mode selected in step Sw_1 is included in the MPM (step Sw_4).

[0292] If the intra-prediction unit 124 determines that the selected intra-prediction mode is included in the MPM (Yes in step Sw_4), it sets the MPM flag to 1 (step Sw_5) and generates information from the MPM indicating the selected intra-prediction mode (step Sw_6). The MPM flag set to 1 and the information indicating that intra-prediction mode are each encoded as prediction parameters by the entropy encoding unit 110.

[0293] On the other hand, if the intra-prediction unit 124 determines that the selected intra-prediction mode is not included in the MPM (No in step Sw_4), it sets the MPM flag to 0 (step Sw_7). Alternatively, the intra-prediction unit 124 does not set the MPM flag. Then, the intra-prediction unit 124 generates information indicating the selected intra-prediction mode from among one or more intra-prediction modes not included in the MPM (step Sw_8). The MPM flag set to 0 and the information indicating that intra-prediction mode are each encoded as prediction parameters by the entropy coding unit 110. The information indicating that intra-prediction mode represents a value between 0 and 60, for example.

[0294] [International Prediction Department] The inter-prediction unit 126 generates a predicted image (inter-predicted image) by performing inter-prediction (also called inter-frame prediction) of the current block by referring to a reference picture stored in the frame memory 122 that is different from the current picture. Inter-prediction is performed on a unit of the current block or the current subblock within the current block. A subblock is a unit contained within a block and is smaller than a block. The size of a subblock may be 4x4 pixels, 8x8 pixels, or other sizes. The size of a subblock may be switched on a unit such as a slice, brick, or picture.

[0295] For example, the interpretation unit 126 performs motion estimation within a reference picture for the current block or current subblock and finds the reference block or subblock that best matches the current block or current subblock. The interpretation unit 126 then obtains motion information (e.g., motion vectors) that compensates for the movement or change from the reference block or subblock to the current block or subblock. Based on this motion information, the interpretation unit 126 performs motion compensation (or motion prediction) and generates an interpretation prediction image of the current block or subblock. The interpretation unit 126 outputs the generated interpretation prediction image to the prediction control unit 128.

[0296] The motion information used for motion compensation may be signaled as an interprediction image in various forms. For example, the motion vector may be signaled. As another example, the difference between the motion vector and the predicted motion vector (motion vector predictor) may be signaled.

[0297] [Reference Picture List] Figure 33 is a diagram showing an example of each reference picture, and Figure 34 is a conceptual diagram showing an example of a reference picture list. The reference picture list is a list showing one or more reference pictures stored in the frame memory 122. In Figure 33, rectangles represent pictures, arrows indicate the reference relationships between pictures, the horizontal axis represents time, I, P, and B in the rectangles represent intra-prediction pictures, single-prediction pictures, and double-prediction pictures, respectively, and the numbers in the rectangles indicate the decoding order. As shown in Figure 33, the decoding order of each picture is I0, P1, B2, B3, B4, and the display order of each picture is I0, B3, B2, B4, P1. As shown in Figure 34, the reference picture list is a list representing candidates for reference pictures, and for example, one picture (or slice) may have one or more reference picture lists. For example, if the current picture is a single-prediction picture, one reference picture list is used, and if the current picture is a double-prediction picture, two reference picture lists are used. In the examples in Figures 33 and 34, picture B3, which is the current picture currPic, has two reference picture lists, the L0 list and the L1 list. When the current picture currPic is picture B3, the candidate reference pictures for that current picture currPic are I0, P1, and B2, and each reference picture list (i.e., the L0 list and the L1 list) indicates these pictures. The interpretation unit 126 or the prediction control unit 128 specifies which picture in each reference picture list to actually reference using the reference picture index refIdxLx. In Figure 34, reference pictures P1 and B2 are specified by the reference picture indices refIdxL0 and refIdxL1.

[0298] Such reference picture lists may be generated on a sequence, picture, slice, brick, CTU, or CU basis. Furthermore, the reference picture index indicating the reference picture referenced in interpretation among the reference pictures shown in the reference picture list may be encoded at the sequence, picture, slice, brick, CTU, or CU level. Additionally, a common reference picture list may be used across multiple interpretation modes.

[0299] [Basic flow of interpretation] Figure 35 is a flowchart showing the basic flow of interpretation prediction.

[0300] The interpretation unit 126 first generates a predicted image (steps Se_1 to Se_3). Next, the subtraction unit 104 generates the difference between the current block and the predicted image as the predicted residual (step Se_4).

[0301] Here, the interpretation unit 126 generates the predicted image by, for example, determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and performing motion compensation (step Se_3). In determining the MV, the interpretation unit 126 determines the MV by, for example, selecting candidate motion vectors (candidate MVs) (step Se_1) and deriving the MV (step Se_2). The selection of candidate MVs is performed, for example, by the interpretation unit 126 generating a candidate MV list and selecting at least one candidate MV from the candidate MV list. Note that previously derived MVs may be added as candidate MVs to the candidate MV list. Furthermore, in the derivation of the MV, the interpretation unit 126 may determine the selected at least one candidate MV as the MV of the current block by further selecting at least one candidate MV from at least one candidate MV. Alternatively, the interpretation unit 126 may determine the MV of the current block by searching the region of the reference picture indicated by each of the selected candidate MVs. This search of the reference picture region may be called motion estimation.

[0302] Furthermore, in the example described above, steps Se_1 to Se_3 are performed by the interpretation unit 126, but processing such as step Se_1 or step Se_2 may be performed by other components included in the encoding device 100.

[0303] Furthermore, a candidate MV list may be created for each process in each interpretation mode, or a common candidate MV list may be used across multiple interpretation modes. Also, the processes in steps Se_3 and Se_4 correspond to the processes in steps Sa_3 and Sa_4 shown in Figure 9, respectively. In addition, the process in step Se_3 corresponds to the process in step Sd_1b in Figure 30.

[0304] [MV Derivation Flow] Figure 36 is a flowchart showing an example of MV derivation.

[0305] The interpretation unit 126 may derive the MV of the current block in a mode that encodes motion information (e.g., MV). In this case, for example, the motion information may be encoded as prediction parameters and signaled. That is, the encoded motion information is included in the stream.

[0306] Alternatively, the interpretation unit 126 may derive MV in a mode that does not encode motion information. In this case, motion information is not included in the stream.

[0307] Here, the modes for MV derivation include the normal inter mode, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that encode motion information include the normal inter mode, normal merge mode, and affine mode (specifically, the affine inter mode and affine merge mode). Note that the motion information may include not only the MV but also the predicted MV selection information, which will be described later. Modes that do not encode motion information include the FRUC mode, etc. The inter prediction unit 126 selects a mode from these multiple modes for deriving the MV of the current block and derives the MV of the current block using the selected mode.

[0308] Figure 37 is a flowchart showing another example of MV derivation.

[0309] The interpretation unit 126 may derive the MV of the current block in a mode that encodes the differential MV. In this case, for example, the differential MV is encoded as a prediction parameter and signaled. That is, the encoded differential MV is included in the stream. This differential MV is the difference between the MV of the current block and its predicted MV. The predicted MV is the predicted motion vector.

[0310] Alternatively, the interpretation unit 126 may derive the MV in a mode that does not encode the differential MV. In this case, the encoded differential MV is not included in the stream.

[0311] As mentioned above, the modes for MV derivation include the normal inter, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that encode differential MV include the normal inter mode and affine mode (specifically, affine inter mode). Modes that do not encode differential MV include the FRUC mode, normal merge mode, and affine mode (specifically, affine merge mode). The inter prediction unit 126 selects a mode from these multiple modes to derive the MV of the current block, and uses the selected mode to derive the MV of the current block.

[0312] [Modes for MV derivation] Figures 38A and 38B illustrate an example of the classification of each mode of MV derivation. For example, as shown in Figure 38A, the modes of MV derivation can be broadly classified into three modes depending on whether motion information is encoded and whether differential MVs are encoded. The three modes are intermode, merge mode, and FRUC (frame rate up-conversion) mode. Intermode is a mode that performs motion search and encodes motion information and differential MVs. For example, as shown in Figure 38B, intermode includes affine intermode and normal intermode. Merge mode is a mode that does not perform motion search and selects an MV from a surrounding encoded block and uses that MV to derive the MV of the current block. This merge mode basically encodes motion information but does not encode differential MVs. For example, as shown in Figure 38B, the merge modes include the normal merge mode (sometimes called the regular merge mode or normal merge mode), the MMVD (Merge with Motion Vector Difference) mode, the CIIP (Combined inter merge / intra prediction) mode, the triangle mode, the ATMVP mode, and the affine merge mode. Here, in the MMVD mode, one of the modes included in the merge modes, the differential MV is encoded as an exception. The affine merge mode and the affine inter mode mentioned above are modes included in the affine mode. The affine mode is a mode that assumes an affine transformation and derives the MV of each of the multiple subblocks that make up the current block as the MV of the current block. The FRUC mode is a mode that derives the MV of the current block by performing a search between encoded regions, and does not encode either motion information or differential MV. Details of each of these modes will be described later.

[0313] Note that the classification of each mode shown in Figures 38A and 38B is just an example and is not limited to this classification. For example, if a differential MV is encoded in CIIP mode, that CIIP mode is classified as an intermode.

[0314] [MV Derivation > Normal Intermode] The normal intermode is an interpretation mode that derives the MV of the current block by finding blocks similar to the image of the current block from the region of the reference picture indicated by the candidate MV. In this normal intermode, the differential MV is also encoded.

[0315] Figure 39 is a flowchart showing an example of inter-mode prediction.

[0316] The interpretation unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks surrounding the current block in time or space (step Sg_1). In other words, the interpretation unit 126 creates a candidate MV list.

[0317] Next, the interpretation unit 126 extracts N candidate MVs (where N is an integer greater than or equal to 2) from the multiple candidate MVs obtained in step Sg_1 as predicted MV candidates, according to a predetermined priority order (step Sg_2). The priority order is predetermined for each of the N candidate MVs.

[0318] Next, the interpretation unit 126 selects one predicted MV candidate from the N predicted MV candidates as the predicted MV for the current block (step Sg_3). At this time, the interpretation unit 126 encodes predicted MV selection information into a stream to identify the selected predicted MV. In other words, the interpretation unit 126 outputs the predicted MV selection information as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0319] Next, the interpretation unit 126 refers to the encoded reference picture and derives the MV of the current block (step Sg_4). At this time, the interpretation unit 126 further encodes the difference between the derived MV and the predicted MV as the difference MV into the stream. In other words, the interpretation unit 126 outputs the difference MV as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130. The encoded reference picture is a picture consisting of multiple blocks that have been reconstructed after encoding.

[0320] Finally, the inter-prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Sg_5). The processes in steps Sg_1 to Sg_5 are performed for each block. For example, when the processes in steps Sg_1 to Sg_5 are performed for each block contained in a slice, the inter-prediction using the normal inter-mode for that slice is completed. Similarly, when the processes in steps Sg_1 to Sg_5 are performed for each block contained in a picture, the inter-prediction using the normal inter-mode for that picture is completed. Note that the processes in steps Sg_1 to Sg_5 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using the normal inter-mode for that slice is completed. Likewise, when the processes in steps Sg_1 to Sg_5 are performed for some blocks contained in a picture, the inter-prediction using the normal inter-mode for that picture is completed.

[0321] The predicted image is the interprediction signal described above. Furthermore, information indicating the interprediction mode used to generate the predicted image (normal intermode in the example above), which is included in the encoded signal, is encoded, for example, as a prediction parameter.

[0322] The candidate MV list may be the same as the list used in other modes. Furthermore, processing related to the candidate MV list may be applied to processing related to lists used in other modes. This processing related to the candidate MV list may include, for example, extracting or selecting candidate MVs from the candidate MV list, rearranging candidate MVs, or deleting candidate MVs.

[0323] [MV Derivation > Normal Merge Mode] The normal merge mode is an interpretation mode that derives an MV by selecting a candidate MV from a list of candidate MVs as the MV of the current block. Note that the normal merge mode is a merge mode in the narrow sense and is sometimes simply called the merge mode. In this embodiment, a distinction is made between the normal merge mode and the merge mode, and the term merge mode is used in the broad sense.

[0324] Figure 40 is a flowchart showing an example of inter prediction using normal merge mode.

[0325] The interpretation unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks surrounding the current block in time or space (step Sh_1). In other words, the interpretation unit 126 creates a candidate MV list.

[0326] Next, the interpretation unit 126 derives the MV of the current block by selecting one candidate MV from among the multiple candidate MVs obtained in step Sh_1 (step Sh_2). At this time, the interpretation unit 126 encodes MV selection information to identify the selected candidate MV into a stream. In other words, the interpretation unit 126 outputs the MV selection information as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0327] Finally, the inter prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Sh_3). The processes in steps Sh_1 to Sh_3 are performed for each block, for example. For example, when the processes in steps Sh_1 to Sh_3 are performed for each of the blocks contained in a slice, the inter prediction using normal merge mode for that slice is completed. Similarly, when the processes in steps Sh_1 to Sh_3 are performed for each of the blocks contained in a picture, the inter prediction using normal merge mode for that picture is completed. Note that the processes in steps Sh_1 to Sh_3 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter prediction using normal merge mode for that slice is completed. Likewise, when the processes in steps Sh_1 to Sh_3 are performed for some of the blocks contained in a picture, the inter prediction using normal merge mode for that picture is completed.

[0328] Furthermore, information indicating the inter-prediction mode used to generate the predicted image (normal merge mode in the example above), which is included in the stream, is encoded, for example, as prediction parameters.

[0329] Figure 41 is a diagram illustrating an example of the MV derivation process for the current picture using normal merge mode.

[0330] First, the interpretation unit 126 generates a candidate MV list in which candidate MVs are registered. Candidate MVs include spatially adjacent candidate MVs, which are MVs of multiple encoded blocks located spatially around the current block; temporally adjacent candidate MVs, which are MVs of nearby blocks projected onto the current block's position in the encoded reference picture; combined candidate MVs, which are MVs generated by combining the MV values ​​of spatially adjacent candidate MVs and temporally adjacent candidate MVs; and zero candidate MVs, which are MVs with a value of zero.

[0331] Next, the interpretation unit 126 selects one candidate MV from among the multiple candidate MVs registered in the candidate MV list, thereby determining that one candidate MV as the MV of the current block.

[0332] Furthermore, the entropy coding unit 110 writes and encodes merge_idx, a signal indicating which candidate MV was selected, into a stream.

[0333] Note that the candidate MVs registered in the candidate MV list explained in Figure 41 are just examples, and the number of candidates may differ from the number shown in the figure, the configuration may not include some of the types of candidate MVs shown in the figure, or it may include candidate MVs other than those shown in the figure.

[0334] The final MV may be determined by performing DMVR (dynamic motion vector refreshing), described later, using the MV of the current block derived in normal merge mode. Note that in normal merge mode, the differential MV is not encoded, but in MMVD mode, the differential MV is encoded. In MMVD mode, one candidate MV is selected from the candidate MV list, similar to normal merge mode, but the differential MV is encoded. Such MMVD may be classified as a merge mode together with normal merge mode, as shown in Figure 38B. Note that the differential MV in MMVD mode does not have to be the same as the differential MV used in inter-mode; for example, the derivation of the differential MV in MMVD mode may be a less computationally intensive process than the derivation of the differential MV in inter-mode.

[0335] Alternatively, a CIIP (Combined inter merge / intra prediction) mode may be used to generate a prediction image for the current block by overlaying the prediction image generated by inter prediction with the prediction image generated by intra prediction.

[0336] The candidate MV list may also be referred to simply as the candidate list. Furthermore, merge_idx is the MV selection information.

[0337] [MV Derivation > HMVP Mode] Figure 42 is a diagram illustrating an example of the MV derivation process for the current picture using HMVP mode.

[0338] In normal merge mode, the MV of the current block (e.g., CU) is determined by selecting one candidate MV from a list of candidate MVs generated by referencing the encoded block (e.g., CU). Other candidate MVs may be added to this list. This mode, where other candidate MVs are added, is called HMVP mode.

[0339] In HMVP mode, candidate MVs are managed using a separate FIFO (First-In First-Out) buffer for HMVP, distinct from the candidate MV list used in normal merge mode.

[0340] The FIFO buffer stores motion information such as MVs of blocks that have been processed in the past, in reverse chronological order. In the management of this FIFO buffer, each time a block is processed, the MV of the most recent block (i.e., the most recently processed CU) is stored in the FIFO buffer, and in its place, the MV of the oldest CU (i.e., the most recently processed CU) in the FIFO buffer is removed. In the example shown in Figure 42, HMVP1 is the MV of the most recent block, and HMVP5 is the MV of the oldest block.

[0341] For example, the interpretation unit 126 checks each MV managed in the FIFO buffer, starting with HMVP1, whether that MV is different from all the candidate MVs already registered in the candidate MV list for normal merge mode. If the interpretation unit 126 determines that it is different from all the candidate MVs, it may add the MV managed in the FIFO buffer as a candidate MV to the candidate MV list for normal merge mode. At this time, one or more candidate MVs may be registered from the FIFO buffer.

[0342] By using HMVP mode in this way, it becomes possible to include not only MVs of spatially or temporally adjacent blocks to the current block, but also MVs of previously processed blocks as candidates. As a result, the variety of candidate MVs in normal merge mode is broadened, which increases the likelihood of improving encoding efficiency.

[0343] Furthermore, the aforementioned MV may also be motion information. In other words, the information stored in the candidate MV list and FIFO buffer may include not only the MV value, but also information indicating the referenced picture, the direction and number of references, etc. Also, the aforementioned block is, for example, a CU.

[0344] Note that the candidate MV list and FIFO buffer in Figure 42 are just examples, and the candidate MV list and FIFO buffer may be lists or buffers of different sizes than those in Figure 42, or the candidate MVs may be registered in a different order than those in Figure 42. Also, the processing described here is common to both the encoding device 100 and the decoding device 200.

[0345] Furthermore, HMVP mode can be applied to modes other than normal merge mode. For example, motion information such as MV of blocks previously processed in affine mode can be stored in the FIFO buffer in chronological order from newest to oldest and used as candidate MV. A mode in which HMVP mode is applied to affine mode may be called history affine mode.

[0346] [MV Derivation > FRUC Mode] Motion information may be derived at the decoding device 200 without being signaled at the encoding device 100. For example, motion information may be derived at the decoding device 200 by performing a motion search. In this case, the decoding device 200 performs the motion search without using the pixel values ​​of the current block. Modes for performing such a motion search at the decoding device 200 include FRUC (frame rate up-conversion) mode and PMMVD (pattern matched motion vector derivation) mode.

[0347] An example of FRUC processing is shown in Figure 43. First, a list is generated (i.e., a candidate MV list, which may be the same as the candidate MV list for normal merge mode) that refers to the MVs of each encoded block spatially or temporally adjacent to the current block and indicates those MVs as candidate MVs (step Si_1). Next, the best candidate MV is selected from among the multiple candidate MVs registered in the candidate MV list (step Si_2). For example, an evaluation value is calculated for each candidate MV included in the candidate MV list, and one candidate MV is selected as the best candidate MV based on that evaluation value. Then, the MV for the current block is derived based on the selected best candidate MV (step Si_4). Specifically, for example, the selected best candidate MV is directly derived as the MV for the current block. Alternatively, for example, the MV for the current block may be derived by performing pattern matching in the area surrounding the position in the reference picture corresponding to the selected best candidate MV. In other words, a search is performed in the area surrounding the best candidate MV using pattern matching and evaluation values ​​in the reference picture. If an MV with a better evaluation value is found, the best candidate MV may be updated to that MV and made the final MV of the current block. It is not necessary to update to an MV with a better evaluation value.

[0348] Finally, the inter-prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Si_5). The processes in steps Si_1 to Si_5 are performed for each block, for example. For example, when the processes in steps Si_1 to Si_5 are performed for each of the blocks contained in a slice, the inter-prediction using FRUC mode for that slice is completed. Similarly, when the processes in steps Si_1 to Si_5 are performed for each of the blocks contained in a picture, the inter-prediction using FRUC mode for that picture is completed. Note that the processes in steps Si_1 to Si_5 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using FRUC mode for that slice is completed. Likewise, when the processes in steps Si_1 to Si_5 are performed for some of the blocks contained in a picture, the inter-prediction using FRUC mode for that picture is completed.

[0349] Subblocks may be processed in the same way as blocks as described above.

[0350] The evaluation value may be calculated by various methods. For example, the reconstructed image of a region in the reference picture corresponding to the MV may be compared with the reconstructed image of a predetermined region (which may be, for example, a region in another reference picture or a region in an adjacent block of the current picture, as shown below). The difference in pixel values ​​between the two reconstructed images may then be calculated and used as the evaluation value for the MV. In addition to the difference value, other information may also be used to calculate the evaluation value.

[0351] Next, we will explain pattern matching in detail. First, one candidate MV included in the candidate MV list (also called the merge list) is selected as the starting point for the search using pattern matching. For pattern matching, either first-order pattern matching or second-order pattern matching may be used. First-order pattern matching and second-order pattern matching are sometimes called bilateral matching and template matching, respectively.

[0352] [MV Derivation > FRUC > Bilateral Matching] In the first pattern matching, pattern matching is performed between two blocks in two different reference pictures that are aligned with the motion trajectory of the current block. Therefore, in the first pattern matching, a region in another reference picture aligned with the motion trajectory of the current block is used as a predetermined region for calculating the evaluation value of the candidate MV described above.

[0353] Figure 44 illustrates an example of first-order pattern matching (bilateral matching) between two blocks in two reference pictures along a motion trajectory. As shown in Figure 44, in first-order pattern matching, two MVs (MV0, MV1) are derived by searching for the best-matching pair of blocks in two different reference pictures (Ref0, Ref1) that are along the motion trajectory of the current block. Specifically, for the current block, the difference is derived between the reconstructed image at a specified position in the first encoded reference picture (Ref0) specified by the candidate MV and the reconstructed image at a specified position in the second encoded reference picture (Ref1) specified by a symmetric MV scaled by the display time interval of the candidate MV. An evaluation value is then calculated using the obtained difference value. The candidate MV with the best evaluation value among multiple candidate MVs should be selected as the best candidate MV.

[0354] Under the assumption of a continuous motion trajectory, the MV(MV0, MV1) pointing to two reference blocks is proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, if the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the two reference pictures are equal, then the first pattern matching derives a mirror-symmetric bidirectional MV.

[0355] [MV Derivation > FRUC > Template Matching] In the second pattern matching (template matching), pattern matching is performed between the template in the current picture (blocks adjacent to the current block in the current picture (e.g., blocks above and / or to the left)) and the blocks in the reference picture. Therefore, in the second pattern matching, the blocks adjacent to the current block in the current picture are used as a predetermined area for calculating the evaluation value of the candidate MV described above.

[0356] Figure 45 illustrates an example of pattern matching (template matching) between a template in the current picture and a block in the reference picture. As shown in Figure 45, in the second pattern matching, the MV of the current block is derived by searching in the reference picture (Ref0) for the block that best matches the block adjacent to the current block (Cur block) in the current picture (Cur Pic). Specifically, for the current block, the difference between the reconstructed image of the encoded region of both or either of the left adjacent and upper adjacent regions and the reconstructed image at the equivalent position in the encoded reference picture (Ref0) specified by the candidate MV is derived, and an evaluation value is calculated using the obtained difference value. The candidate MV with the best evaluation value among multiple candidate MVs should be selected as the best candidate MV.

[0357] Information indicating whether or not to apply such a FRUC mode (e.g., called a FRUC flag) may be signaled at the CU level. Furthermore, if FRUC mode is applied (e.g., the FRUC flag is true), information indicating the applicable pattern matching method (first pattern matching or second pattern matching) may be signaled at the CU level. Note that the signaling of this information is not limited to the CU level, but may be at other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or subblock level).

[0358] [MV Derivation > Affine Mode] The affine mode is a mode in which the MV is generated using an affine transform. For example, the MV may be derived on a subblock basis based on the MVs of multiple adjacent blocks. This mode is sometimes called the affine motion compensation prediction mode.

[0359] Figure 46A illustrates an example of deriving the MV of a subblock based on the MV of multiple adjacent blocks. In Figure 46A, the current block contains, for example, 16 subblocks consisting of 4x4 pixels. Here, the motion vector v0 of the upper left corner control point of the current block is derived based on the MV of the adjacent blocks, and similarly, the motion vector v1 of the upper right corner control point of the current block is derived based on the MV of the adjacent subblock. Then, by projecting the two motion vectors v0 and v1 using the following equation (1A), the motion vector (v) of each subblock within the current block is obtained. x ,v y ) is derived.

[0360]

number

[0361] Here, x and y represent the horizontal and vertical positions of the subblock, respectively, and w represents a predetermined weighting coefficient.

[0362] Information indicating such affine modes (e.g., called an affine flag) may be signaled at the CU level. However, the signaling of this information indicating affine modes is not limited to the CU level, but may be at other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or subblock level).

[0363] Furthermore, such affine modes may include several modes with different methods for deriving the MV of the upper-left and upper-right corner control points. For example, there are two affine modes: the affine inter (also called the affine normal inter) mode and the affine merge mode.

[0364] Figure 46B illustrates an example of deriving the motion vector (MV) for a subblock unit in affine mode using three control points. In Figure 46B, the current block includes, for example, a subblock consisting of 16 4x4 pixels. Here, the motion vector v0 of the upper left corner control point of the current block is derived based on the MV of the adjacent block. Similarly, the motion vector v1 of the upper right corner control point of the current block is derived based on the MV of the adjacent block, and the motion vector v2 of the lower left corner control point of the current block is derived based on the MV of the adjacent block. Then, by projecting the three motion vectors v0, v1, and v2 using the following equation (1B), the motion vector (v) of each subblock within the current block is obtained. x ,v y ) is derived.

[0365]

number

[0366] Here, x and y represent the horizontal and vertical positions of the subblock center, respectively, and w and h represent predetermined weighting coefficients. w may represent the width of the current block, and h may represent the height of the current block.

[0367] Affine modes using different numbers of control points (e.g., two and three) may be switched and signaled at the CU level. Information indicating the number of control points for the affine modes used at the CU level may also be signaled at other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or subblock level).

[0368] Furthermore, an affine mode having three control points may include several modes in which the MV of the upper left, upper right, and lower left corner control points is derived in different ways. For example, an affine mode having three control points has two modes, similar to the affine mode having two control points described above: an affine intermode and an affine merge mode.

[0369] In affine mode, the size of each subblock included in the current block is not limited to 4x4 pixels; it can be any other size. For example, each subblock may be 8x8 pixels.

[0370] [MV Derivation > Affine Mode > Control Point] Figures 47A, 47B, and 47C are conceptual diagrams illustrating an example of MV derivation of control points in affine mode.

[0371] In affine mode, as shown in Figure 47A, the predicted MV for each control point of the current block is calculated based on multiple MVs corresponding to blocks encoded in affine mode among the encoded blocks adjacent to the current block: Block A (left), Block B (top), Block C (upper right), Block D (lower left), and Block E (upper left). Specifically, these blocks are examined in the order of encoded Block A (left), Block B (top), Block C (upper right), Block D (lower left), and Block E (upper left), and the first valid block encoded in affine mode is identified. Based on the multiple MVs corresponding to this identified block, the MV for the control point of the current block is calculated.

[0372] For example, as shown in Figure 47B, if block A, which is adjacent to the left of the current block, is encoded in affine mode with two control points, then motion vectors v3 and v4 are derived, projected onto the upper-left and upper-right corners of the encoded block containing block A. Then, from the derived motion vectors v3 and v4, the motion vector v0 of the upper-left corner control point and the motion vector v1 of the upper-right corner control point of the current block are calculated.

[0373] For example, as shown in Figure 47C, if block A adjacent to the left of the current block is encoded in affine mode with three control points, motion vectors v3, v4, and v5 are derived projected onto the upper-left, upper-right, and lower-left corners of the encoded block containing block A. Then, from the derived motion vectors v3, v4, and v5, the motion vector v0 of the upper-left corner control point, the motion vector v1 of the upper-right corner control point, and the motion vector v2 of the lower-left corner control point of the current block are calculated.

[0374] The MV derivation method shown in Figures 47A to 47C may be used to derive the MV of each control point in the current block in step Sk_1 shown in Figure 50 later, or it may be used to derive the predicted MV of each control point in the current block in step Sj_1 shown in Figure 51 later.

[0375] Figures 48A and 48B are conceptual diagrams illustrating another example of the derivation of the control point MV in affine mode.

[0376] Figure 48A is a diagram illustrating an affine mode with two control points.

[0377] In this affine mode, as shown in Figure 48A, the MV selected from the respective MVs of the encoded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the upper left corner control point of the current block. Similarly, the MV selected from the respective MVs of the encoded blocks D and E adjacent to the current block is used as the motion vector v1 of the upper right corner control point of the current block.

[0378] Figure 48B is a diagram illustrating an affine mode with three control points.

[0379] In this affine mode, as shown in Figure 48B, the MV selected from each of the MVs of the encoded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the upper left corner control point of the current block. Similarly, the MV selected from each of the MVs of the encoded blocks D and E adjacent to the current block is used as the motion vector v1 of the upper right corner control point of the current block. Furthermore, the MV selected from each of the MVs of the encoded blocks F and G adjacent to the current block is used as the motion vector v2 of the lower left corner control point of the current block.

[0380] The MV derivation methods shown in Figures 48A and 48B may be used to derive the MV of each control point in the current block in step Sk_1 shown in Figure 50, which will be described later, or they may be used to derive the predicted MV of each control point in the current block in step Sj_1 shown in Figure 51, which will be described later.

[0381] Here, for example, when switching between affine modes with different numbers of control points (e.g., two and three) at the CU level to generate signals, the number of control points may differ between the encoded block and the current block.

[0382] Figures 49A and 49B are conceptual diagrams illustrating an example of a method for deriving the control point MV when the number of control points differs between the encoded block and the current block.

[0383] For example, as shown in Figure 49A, the current block has three control points at the upper left corner, upper right corner, and lower left corner, and is encoded in affine mode with block A adjacent to the left of the current block having two control points. In this case, motion vectors v3 and v4 are derived by projecting them onto the upper left and upper right corner positions of the encoded block containing block A. Then, from the derived motion vectors v3 and v4, the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block are calculated. Furthermore, from the derived motion vectors v0 and v1, the motion vector v2 of the lower left corner control point is calculated.

[0384] For example, as shown in Figure 49B, the current block is encoded in affine mode, having two control points at the upper left and upper right corners, and block A, adjacent to the left of the current block, has three control points. In this case, motion vectors v3, v4, and v5 are derived by projecting them onto the upper left, upper right, and lower left corners of the encoded block containing block A. Then, from the derived motion vectors v3, v4, and v5, the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block are calculated.

[0385] The MV derivation methods shown in Figures 49A and 49B may be used to derive the MV of each control point in the current block in step Sk_1 shown in Figure 50, which will be described later, or they may be used to derive the predicted MV of each control point in the current block in step Sj_1 shown in Figure 51, which will be described later.

[0386] [MV Derivation > Affine Mode > Affine Merge Mode] Figure 50 is a flowchart showing an example of the affine merge mode.

[0387] In affine merge mode, the interpretation unit 126 first derives the MVs of each control point in the current block (step Sk_1). The control points are the upper left and upper right corners of the current block, as shown in Figure 46A, or the upper left, upper right, and lower left corners of the current block, as shown in Figure 46B. At this time, the interpretation unit 126 may encode MV selection information into a stream to identify two or three of the derived MVs.

[0388] For example, when using the MV derivation method shown in Figures 47A to 47C, the interpretation unit 126 examines the encoded blocks in the order of A (left), B (top), C (upper right), D (lower left), and E (upper left), as shown in Figure 47A, and identifies the first valid block encoded in affine mode.

[0389] The interpretation unit 126 derives the motion vectors (MV) of the control points using the first valid block encoded in the identified affine mode. For example, if block A is identified and block A has two control points, as shown in Figure 47B, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block from the motion vectors v3 and v4 of the upper left and upper right corners of the encoded block containing block A. For example, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block by projecting the motion vectors v3 and v4 of the upper left and upper right corners of the encoded block onto the current block.

[0390] Alternatively, if block A is identified and block A has three control points, as shown in Figure 47C, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block from the motion vectors v3, v4, and v5 of the upper left, upper right, and lower left corners of the encoded block containing block A. For example, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block by projecting the motion vectors v3, v4, and v5 of the upper left, upper right, and lower left corners of the encoded block onto the current block.

[0391] Furthermore, as shown in Figure 49A above, if block A is identified and block A has two control points, the MV of three control points may be calculated. Alternatively, as shown in Figure 49B above, if block A is identified and block A has three control points, the MV of two control points may be calculated.

[0392] Next, the interpretation unit 126 performs motion compensation for each of the multiple subblocks contained in the current block. That is, for each of the multiple subblocks, the interpretation unit 126 calculates the MV of that subblock as an affine MV using two motion vectors v0 and v1 and equation (1A) described above, or using three motion vectors v0, v1 and v2 and equation (1B) described above (step Sk_2). Then, the interpretation unit 126 performs motion compensation for that subblock using those affine MVs and the encoded reference picture (step Sk_3). Once steps Sk_2 and Sk_3 have been executed for each of the subblocks contained in the current block, the process of generating a predicted image using the affine merge mode for that current block is completed. In other words, motion compensation is performed on the current block, and a predicted image for that current block is generated.

[0393] In step Sk_1, the above-mentioned candidate MV list may be generated. The candidate MV list may be a list containing candidate MVs derived for each control point using multiple MV derivation methods. The multiple MV derivation methods may be any combination of the MV derivation methods shown in Figures 47A to 47C, the MV derivation methods shown in Figures 48A and 48B, the MV derivation methods shown in Figures 49A and 49B, and other MV derivation methods.

[0394] The candidate MV list may also include candidate MVs from modes other than affine mode, where prediction is performed on a sub-block basis.

[0395] Furthermore, the candidate MV list may include, for example, candidate MVs for affine merge modes with two control points and candidate MVs for affine merge modes with three control points. Alternatively, a candidate MV list containing candidate MVs for affine merge modes with two control points and a candidate MV list containing candidate MVs for affine merge modes with three control points may be generated separately. Alternatively, a candidate MV list may be generated containing candidate MVs for one of the modes: affine merge mode with two control points and affine merge mode with three control points. The candidate MVs may be, for example, the MVs of encoded blocks A (left), B (top), C (upper right), D (lower left), and E (upper left), or they may be the MVs of valid blocks among those blocks.

[0396] Alternatively, you may send an index indicating which candidate MV from the candidate MV list is being selected as MV selection information.

[0397] [MV Derivation > Affine Mode > Affine Intermode] Figure 51 is a flowchart showing an example of an affine intermode.

[0398] In affine intermode, the interpretation unit 126 first derives the predicted MV(v0,v1) or (v0,v1,v2) for each of two or three control points of the current block (step Sj_1). The control points are the upper left corner, upper right corner, or lower left corner of the current block, as shown in Figure 46A or Figure 46B.

[0399] For example, when using the MV derivation method shown in Figures 48A and 48B, the interpretation unit 126 derives the predicted MV (v0,v1) or (v0,v1,v2) of the control point of the current block by selecting the MV of one of the encoded blocks near each control point of the current block shown in Figure 48A or Figure 48B. At this time, the interpretation unit 126 encodes predicted MV selection information into a stream to identify the two or three selected predicted MVs.

[0400] For example, the interpretation unit 126 may determine which block's MV from the encoded blocks adjacent to the current block to select as the predicted MV for the control point using cost evaluation or the like, and write a flag indicating which predicted MV was selected to the bitstream. In other words, the interpretation unit 126 outputs the predicted MV selection information, such as a flag, as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0401] Next, the interpretation unit 126 performs motion search (steps Sj_3 and Sj_4) while updating the predicted MVs selected or derived in step Sj_1 (step Sj_2). That is, the interpretation unit 126 calculates the MV of each subblock corresponding to the updated predicted MV as an affine MV using the above-mentioned equation (1A) or equation (1B) (step Sj_3). Then, the interpretation unit 126 performs motion compensation for each subblock using these affine MVs and encoded reference pictures (step Sj_4). The processing in steps Sj_3 and Sj_4 is performed for all blocks in the current block each time the predicted MV is updated in step Sj_2. As a result, the interpretation unit 126 determines, for example, the predicted MV that yields the smallest cost in the motion search loop as the MV of the control point (step Sj_5). At this time, the interpretation unit 126 further encodes the difference between the determined MV and the predicted MV as the difference MV into a stream. In other words, the interpretation unit 126 outputs the difference MV as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0402] Finally, the interpretation unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the determined MV and the encoded reference picture (step Sj_6).

[0403] In step Sj_1, the above-mentioned candidate MV list may be generated. The candidate MV list may be a list containing candidate MVs derived for each control point using multiple MV derivation methods. The multiple MV derivation methods may be any combination of the MV derivation methods shown in Figures 47A to 47C, the MV derivation methods shown in Figures 48A and 48B, the MV derivation methods shown in Figures 49A and 49B, and other MV derivation methods.

[0404] The candidate MV list may also include candidate MVs from modes other than affine mode, where prediction is performed on a sub-block basis.

[0405] Furthermore, a candidate MV list may be generated that includes candidate MVs for affine intermodes with two control points and candidate MVs for affine intermodes with three control points. Alternatively, a candidate MV list containing candidate MVs for affine intermodes with two control points and a candidate MV list containing candidate MVs for affine intermodes with three control points may be generated separately. Alternatively, a candidate MV list may be generated that includes candidate MVs for one of the modes: affine intermodes with two control points and affine intermodes with three control points. The candidate MVs may be, for example, the MVs of encoded blocks A (left), B (top), C (upper right), D (lower left), and E (upper left), or they may be the MVs of valid blocks among those blocks.

[0406] Additionally, as predicted MV selection information, you may send an index indicating which candidate MV from the candidate MV list it is.

[0407] [MV Derivation > Triangle Mode] In the example described above, the interpretation unit 126 generates one rectangular prediction image for the current rectangular block. However, the interpretation unit 126 may generate multiple prediction images of shapes different from rectangles for the current rectangular block, and then combine these multiple prediction images to generate the final rectangular prediction image. The shapes different from rectangles may be, for example, triangles.

[0408] Figure 52A is a diagram illustrating the generation of two triangular prediction images.

[0409] The interpretation unit 126 generates a predicted triangular image by performing motion compensation on the first triangular partition within the current block using the first MV of that first partition. Similarly, the interpretation unit 126 generates a predicted triangular image by performing motion compensation on the second triangular partition within the current block using the second MV of that second partition. Then, the interpretation unit 126 combines these predicted images to generate a predicted rectangular image identical to that of the current block.

[0410] Furthermore, a first rectangular prediction image corresponding to the current block may be generated using the first MV as the prediction image for the first partition. Similarly, a second rectangular prediction image corresponding to the current block may be generated using the second MV as the prediction image for the second partition. The prediction image for the current block may also be generated by weighting and adding the first and second prediction images. Note that the weighting and addition may be applied only to a portion of the region straddling the boundary between the first and second partitions.

[0411] Figure 52B is a conceptual diagram showing an example of a first portion of a first partition overlapping with a second partition, as well as a first and second sample set that may be weighted as part of a correction process. The first portion may be, for example, one-quarter of the width or height of the first partition. In another example, the first portion may have a width corresponding to N samples adjacent to the edge of the first partition, where N is an integer greater than zero, for example, N may be the integer 2. Figure 52B shows a rectangular partition having a rectangular portion with a width of one-quarter of the width of the first partition. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples within the first portion. The central example in Figure 52B shows a rectangular partition having a rectangular portion with a height of one-quarter of the height of the first partition. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples within the first portion. The example on the right in Figure 52B shows a triangular partition with polygonal sections of height corresponding to two samples. Here, the first sample set includes samples outside the first section and samples inside the first section, and the second sample set includes samples within the first section.

[0412] The first portion may be the portion of the first partition that overlaps with an adjacent partition. Figure 52C is a conceptual diagram showing the first portion of the first partition, which is the portion of the first partition that overlaps with a portion of an adjacent partition. For simplicity of explanation, a rectangular partition is shown that has a portion that overlaps with a spatially adjacent rectangular partition. Partitions of other shapes, such as triangular partitions, may be used, and the overlapping portion may overlap with a spatially or temporally adjacent partition.

[0413] Furthermore, while an example is shown in which prediction images are generated for each of the two partitions using interpretation, prediction images may also be generated for at least one partition using intrapretation.

[0414] Figure 53 is a flowchart showing an example of the triangle mode.

[0415] In triangle mode, the interpretation unit 126 first divides the current block into a first partition and a second partition (step Sx_1). At this time, the interpretation unit 126 may encode partition information, which is information about the division into each partition, into the stream as prediction parameters. In other words, the interpretation unit 126 may output partition information as prediction parameters to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0416] Next, the interpretation unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks surrounding the current block in time or space (step Sx_2). In other words, the interpretation unit 126 creates a candidate MV list.

[0417] Then, the interpretation unit 126 selects the candidate MVs for the first partition and the candidate MVs for the second partition from among the multiple candidate MVs obtained in step Sx_2 as the first MV and the second MV, respectively (step Sx_3). At this time, the interpretation unit 126 may encode MV selection information for identifying the selected candidate MVs into the stream as prediction parameters. In other words, the interpretation unit 126 may output the MV selection information as prediction parameters to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0418] Next, the interpretation unit 126 generates a first predicted image by performing motion compensation using the selected first MV and the encoded reference picture (step Sx_4). Similarly, the interpretation unit 126 generates a second predicted image by performing motion compensation using the selected second MV and the encoded reference picture (step Sx_5).

[0419] Finally, the interpretation unit 126 generates a predicted image of the current block by weighting and adding the first predicted image and the second predicted image (step Sx_6).

[0420] In the example shown in Figure 52A, the first and second partitions are triangular, but they may also be trapezoidal, or they may have different shapes from each other. Furthermore, in the example shown in Figure 52A, the current block is composed of two partitions, but it may be composed of three or more partitions.

[0421] Furthermore, the first and second partitions may overlap. That is, the first and second partitions may contain the same pixel region. In this case, the predicted image of the current block may be generated using the predicted image in the first partition and the predicted image in the second partition.

[0422] Furthermore, while this example shows that predicted images are generated by interpretation for both partitions, predicted images may also be generated by intrapretation for at least one partition.

[0423] Note that the candidate MV list for selecting the first MV and the candidate MV list for selecting the second MV may be different, or they may be the same.

[0424] The partition information may include an index indicating the partitioning direction for dividing the current block into multiple partitions. The MV selection information may include an index indicating the first selected MV and an index indicating the second selected MV. A single index may represent multiple pieces of information. For example, a single index may be encoded that represents part or all of the partition information and part or all of the MV selection information together.

[0425] [MV Derivation > ATMVP Mode] Figure 54 shows an example of an ATMVP mode in which MV is derived at the subblock level.

[0426] ATMVP mode is a mode classified as a merge mode. For example, in ATMVP mode, candidate MVs at the subblock level are registered in the candidate MV list used in normal merge mode.

[0427] Specifically, in ATMVP mode, first, as shown in Figure 54, the time MV reference block associated with the current block is identified in the encoded reference picture specified by the MV (MV0) of the block adjacent to the lower left of the current block. Next, for each subblock within the current block, the MV used when encoding the region corresponding to that subblock within the time MV reference block is identified. The MVs identified in this way are included in the candidate MV list as candidate MVs for the subblocks of the current block. When a candidate MV for each subblock is selected from the candidate MV list, motion compensation using that candidate MV as the subblock's MV is performed for that subblock. This generates a predicted image for each subblock.

[0428] In the example shown in Figure 54, the block adjacent to the lower left of the current block was used as the peripheral MV reference block, but other blocks may be used. Also, the size of the subblock may be 4x4 pixels, 8x8 pixels, or other sizes. The size of the subblock may be switched in units such as slices, bricks, or pictures.

[0429] [Motion Detection > DMVR] Figure 55 shows the relationship between merge mode and DMVR.

[0430] The interpretation unit 126 derives the MV of the current block in merge mode (step Sl_1). Next, the interpretation unit 126 determines whether or not to perform an MV search, i.e., a motion search (step Sl_2). If the interpretation unit 126 determines not to perform a motion search (No in step Sl_2), it determines the MV derived in step Sl_1 as the final MV for the current block (step Sl_4). In other words, in this case, the MV of the current block is determined in merge mode.

[0431] On the other hand, if it is determined in step Sl_1 to perform a motion search (Yes in step Sl_2), the interpretation unit 126 derives the final MV for the current block by searching the surrounding region of the reference picture indicated by the MV derived in step Sl_1 (step Sl_3). In other words, in this case, the MV for the current block is determined by the DMVR.

[0432] Figure 56 is a conceptual diagram illustrating an example of DMVR for determining MV.

[0433] First, for example in merge mode, candidate MVs (L0 and L1) are selected for the current block. Then, according to candidate MV (L0), reference pixels are identified from the first reference picture (L0), which is an encoded picture in the L0 list. Similarly, according to candidate MV (L1), reference pixels are identified from the second reference picture (L1), which is an encoded picture in the L1 list. A template is generated by taking the average of these reference pixels.

[0434] Next, using the template, the surrounding regions of candidate MVs for the first reference picture (L0) and the second reference picture (L1) are searched, and the MV with the minimum cost is determined as the final MV for the current block. The cost may be calculated, for example, using the difference between each pixel value in the template and each pixel value in the search region, as well as the candidate MV value.

[0435] Any process that can explore the vicinity of a candidate MV and derive the final MV is acceptable, even if it's not the exact process described here.

[0436] Figure 57 is a conceptual diagram illustrating another example of a DMVR for determining MV. Unlike the DMVR example shown in Figure 56, this example in Figure 57 calculates the cost without generating a template.

[0437] First, the interpretation unit 126 searches around the reference blocks contained in the reference pictures of the L0 list and L1 list, respectively, based on the initial MV, which is a candidate MV obtained from the candidate MV list. For example, as shown in Figure 57, the initial MV corresponding to the reference block in the L0 list is InitMV_L0, and the initial MV corresponding to the reference block in the L1 list is InitMV_L1. In motion search, the interpretation unit 126 first sets the search position for the reference picture in the L0 list. The difference vector indicating the set search position, specifically the difference vector from the position indicated by the initial MV (i.e., InitMV_L0) to that search position, is MVd_L0. Then, the interpretation unit 126 determines the search position in the reference picture of the L1 list. This search position is indicated by the difference vector from the position indicated by the initial MV (i.e., InitMV_L1) to that search position. Specifically, the interpretation unit 126 determines this difference vector as MVd_L1 by mirroring MVd_L0. In other words, the interpretation unit 126 searches for a position that is symmetrical to the position indicated by the initial MV in the reference pictures of the L0 list and the L1 list. For each search position, the interpretation unit 126 calculates a cost such as the sum of the absolute differences in pixel values ​​within the block at that search position (SAD), and finds the search position that minimizes this cost.

[0438] Figure 58A is a diagram showing an example of motion search in a DMVR, and Figure 58B is a flowchart showing an example of that motion search.

[0439] First, in Step 1, the interpretation unit 126 calculates the cost of the search position indicated by the initial MV (also called the starting point) and the eight surrounding search positions. Then, the interpretation unit 126 determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 126 determines that the cost of the search positions other than the starting point is the minimum, it moves to the search position with the minimum cost and proceeds to Step 2. On the other hand, if the cost of the starting point is the minimum, the interpretation unit 126 skips Step 2 and proceeds to Step 3.

[0440] In Step 2, the interpretation unit 126 uses the search position moved according to the processing result of Step 1 as a new starting point and performs the same search as in Step 1. The interpretation unit 126 then determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 126 determines that the cost of the search positions other than the starting point is the minimum, it proceeds to Step 4. On the other hand, if the interpretation unit 126 determines that the cost of the starting point is the minimum, it proceeds to Step 3.

[0441] In Step 4, the interpretation unit 126 treats the starting position as the final search position and determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0442] In Step 3, the interpretation unit 126 determines the pixel position with the minimum cost based on the costs at the four points above, below, to the left and right of the starting point in Step 1 or Step 2, and sets that pixel position as the final search position. This decimal-precision pixel position is determined by weighting the vector of the four points above, below, to the left and right ((0,1),(0,-1),(-1,0),(1,0)) with the cost at each of the four search positions as the weight. The interpretation unit 126 then determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0443] [Motion compensation > BIO / OBMC / LIC] Motion compensation includes modes that generate a predictive image and then correct that predictive image. These modes include, for example, BIO, OBMC, and LIC, which will be described later.

[0444] Figure 59 is a flowchart showing an example of predictive image generation.

[0445] The interpretation unit 126 generates a predicted image (step Sm_1) and corrects the predicted image using one of the modes described above (step Sm_2).

[0446] Figure 60 is a flowchart showing another example of predictive image generation.

[0447] The interpretation unit 126 derives the MV of the current block (step Sn_1). Next, the interpretation unit 126 generates a predicted image using the MV (step Sn_2) and determines whether or not to perform correction processing (step Sn_3). If the interpretation unit 126 determines that correction processing should be performed (Yes in step Sn_3), it generates the final predicted image by correcting the predicted image (step Sn_4). In the LIC described later, brightness and color difference may also be corrected in step Sn_4. On the other hand, if the interpretation unit 126 determines that no correction processing should be performed (No in step Sn_3), it outputs the predicted image as the final predicted image without correction (step Sn_5).

[0448] [Motion Compensation > OBMC] Interpretation images may be generated using not only the motion information of the current block obtained through motion search, but also the motion information of adjacent blocks. Specifically, interpretation images may be generated at the sub-block level within the current block by weighted addition of a prediction image based on motion information obtained through motion search (in the reference picture) and a prediction image based on the motion information of adjacent blocks (in the current picture). Such interpretation (motion compensation) is sometimes called OBMC (oval-wrapped block motion compensation) or OBMC mode.

[0449] In OBMC mode, information indicating the size of the subblock for OBMC (e.g., called the OBMC block size) may be signaled at the sequence level. Furthermore, information indicating whether or not to apply OBMC mode (e.g., called the OBMC flag) may be signaled at the CU level. Note that the signaling levels of this information are not limited to the sequence level and CU level, but may be other levels (e.g., picture level, slice level, brick level, CTU level, or subblock level).

[0450] The OBMC mode will be explained in more detail. Figures 61 and 62 are flowcharts and conceptual diagrams illustrating the overview of the predictive image correction process using OBMC.

[0451] First, as shown in Figure 62, a predicted image (Pred) is obtained using normal motion compensation with the MV assigned to the current block. In Figure 62, the arrow "MV" points to the reference picture, indicating what the current block of the current picture is referencing in order to obtain the predicted image.

[0452] Next, the previously derived MV (MV_L) for the encoded left-adjacent block is applied (reused) to the current block to obtain the predicted image (Pred_L). The MV (MV_L) is indicated by an arrow “MV_L” pointing from the current block to the reference picture. Then, the first correction of the predicted image is performed by superimposing the two predicted images, Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.

[0453] Similarly, the previously derived MV (MV_U) for the encoded upper adjacent block is applied (reused) to the current block to obtain the predicted image (Pred_U). The MV (MV_U) is indicated by an arrow “MV_U” pointing from the current block to the reference picture. Then, the predicted image Pred_U is superimposed on the predicted image that has undergone the first correction (e.g., Pred and Pred_L) to perform a second correction of the predicted image. This has the effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is the final predicted image of the current block, with the boundaries with adjacent blocks blended (smoothed).

[0454] The above example is a two-pass correction method using left-adjacent and top-adjacent blocks, but the correction method may also be a three-pass or more-pass correction method using right-adjacent and / or bottom-adjacent blocks.

[0455] Furthermore, the area to be superimposed does not have to be the entire pixel area of ​​the block, but rather only a portion of the area near the block boundary.

[0456] Here, we have described the OBMC predictive image correction process for obtaining a single predictive image Pred by superimposing additional predictive images Pred_L and Pred_U onto a single reference picture. However, if the predictive image is corrected based on multiple reference images, the same process may be applied to each of the multiple reference pictures. In such cases, by performing OBMC image correction based on multiple reference pictures, a corrected predictive image is obtained from each reference picture, and then the final predictive image is obtained by further superimposing these multiple corrected predictive images.

[0457] In OBMC, the unit of a current block may be a PU unit, or it may be a subblock unit obtained by further dividing a PU.

[0458] One method for determining whether to apply OBMC is to use an obmc_flag signal, which indicates whether or not to apply OBMC. For example, the encoding device 100 may determine whether the current block belongs to a region with complex motion. If the encoding device 100 belongs to a region with complex motion, it sets the obmc_flag to a value of 1 and applies OBMC to perform encoding. If the block does not belong to a region with complex motion, it sets the obmc_flag to a value of 0 and encodes the block without applying OBMC. On the other hand, the decoding device 200 decodes the obmc_flag written in the stream and switches whether or not to apply OBMC depending on its value to perform decoding.

[0459] [Motion Compensation > BIO] Next, we will explain how to derive MV. First, we will describe the mode of deriving MV based on a model that assumes uniform linear motion. This mode is sometimes called the BIO (bi-directional optical flow) mode. Also, this bi-directional optical flow may be written as BDOF instead of BIO.

[0460] Figure 63 is a diagram illustrating a model assuming uniform linear motion. In Figure 63, (vx, vy) represents the velocity vector, and τ0 and τ1 represent the temporal distance between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1), respectively. (MVx0, MVy0) represents the MV corresponding to reference picture Ref0, and (MVx1, MVy1) represents the MV corresponding to reference picture Ref1.

[0461] Under the assumption of uniform linear motion of the velocity vector (vx, vy), (MVx0, MVy0) and (MVx1, MVy1) can be expressed as (vxτ0, vyτ0) and (-vxτ1, -vyτ1), respectively, and the following optical flow equality (2) holds.

[0462]

number

[0463] Here, I(k) represents the luminance value of the reference image k (k=0,1) after motion compensation. This optical flow equation shows that the sum of (i) the time derivative of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is equal to zero. Based on this optical flow equation and Hermite interpolation, block-level motion vectors obtained from a candidate MV list, etc., may be corrected on a pixel-by-pixel basis.

[0464] Furthermore, the motion vector (MV) may be derived on the decoding device 200 side using a method different from the derivation of the motion vector based on a model that assumes uniform linear motion. For example, the motion vector may be derived on a sub-block basis based on the MVs of multiple adjacent blocks.

[0465] Figure 64 is a flowchart illustrating an example of inter prediction according to BIO. Figure 65 is a diagram showing an example of the configuration of the inter prediction unit 126 that performs the inter prediction according to BIO.

[0466] As shown in FIG. 65, for example, the inter prediction unit 126 includes a memory 126a, an interpolation image derivation unit 126b, a gradient image derivation unit 126c, an optical flow derivation unit 126d, a correction value derivation unit 126e, and a prediction image correction unit 126f. Note that the memory 126a may be the frame memory 122.

[0467] The inter prediction unit 126 derives two motion vectors (M0, M1) using two reference pictures (Ref0, Ref1) different from the picture (Cur Pic) including the current block. Then, the inter prediction unit 126 derives a prediction image of the current block using the two motion vectors (M0, M1) (step Sy_1). Note that the motion vector M0 is a motion vector (MVx0, MVy0) corresponding to the reference picture Ref0, and the motion vector M1 is a motion vector (MVx1, MVy1) corresponding to the reference picture Ref1.

[0468] Next, the interpolation image derivation unit 126b refers to the memory 126a and derives an interpolation image I 0 of the current block using the motion vector M0 and the reference picture L0. Also, the interpolation image derivation unit 126b refers to the memory 126a and derives an interpolation image I 1 of the current block using the motion vector M1 and the reference picture L1 (step Sy_2). Here, the interpolation image I 0 is an image included in the reference picture Ref0 derived for the current block, and the interpolation image I 1 is an image included in the reference picture Ref1 derived for the current block. The interpolation image I 0 and the interpolation image I 1 may each have the same size as the current block. Alternatively, the interpolation image I 0 and the interpolation image I 1 may each be an image larger than the current block in order to appropriately derive a gradient image described later. Further, the interpolation images I 0 and I 1This may include a motion vector (M0, M1) and a reference picture (L0, L1), and a predicted image derived by applying a motion compensation filter.

[0469] Furthermore, the gradient image derivation unit 126c generates an interpolated image I 0 and interpolated image I 1 From there, the gradient image of the current block (Ix 0 ,Ix 1 Iy 0 Iy 1 ) is derived (Step Sy_3). Note that the horizontal gradient image is (Ix 0 ,Ix 1 ) and the vertical gradient image is (Iy 0 Iy 1 The gradient image derivation unit 126c may derive the gradient image by, for example, applying a gradient filter to the interpolated image. The gradient image only needs to show the amount of spatial change in pixel values ​​along the horizontal or vertical direction.

[0470] Next, the optical flow derivation unit 126d interpolates the image (I) in units of multiple subblocks that constitute the current block. 0 ,I 1 ) and gradient image (Ix 0 ,Ix 1 Iy 0 Iy 1 Using the above-mentioned velocity vector, optical flow (vx, vy) is derived (step Sy_4). Optical flow is a coefficient that corrects the amount of spatial movement of pixels, and may also be called local motion estimate, corrected motion vector, or corrected weight vector. As an example, the subblock may be a 4x4 pixel subCU. Note that the derivation of optical flow may be performed in units other than subblocks, such as pixels.

[0471] Next, the interpretation unit 126 corrects the predicted image of the current block using optical flow (vx, vy). For example, the correction value derivation unit 126e derives correction values ​​for the pixel values ​​included in the current block using optical flow (vx, vy) (step Sy_5). Then, the predicted image correction unit 126f may correct the predicted image of the current block using the correction values ​​(step Sy_6). Note that the correction values ​​may be derived for each pixel, or for multiple pixels or subblocks.

[0472] Note that the BIO processing flow is not limited to the processing disclosed in Figure 64. Only some of the processing disclosed in Figure 64 may be performed, different processing may be added or replaced, or the processing may be executed in a different order.

[0473] [Motion Compensation > LIC] Next, we will describe an example of a mode that generates a predicted image (prediction) using LIC (local illumination compensation).

[0474] Figure 66A is a diagram illustrating an example of a predictive image generation method using luminance correction processing by LIC. Figure 66B is a flowchart illustrating an example of the predictive image generation method using the LIC.

[0475] First, the interpretation unit 126 derives the MV from the encoded reference picture and obtains the reference image corresponding to the current block (step Sz_1).

[0476] Next, the interpretation unit 126 extracts information for the current block indicating how the luminance values ​​have changed between the reference picture and the current picture (step Sz_2). This extraction is performed based on the luminance pixel values ​​of the encoded left adjacent reference region (surrounding reference region) and encoded upper adjacent reference region (surrounding reference region) in the current picture, and the luminance pixel values ​​at the equivalent positions in the reference picture specified by the derived MV. Then, the interpretation unit 126 calculates luminance correction parameters using the information indicating how the luminance values ​​have changed (step Sz_3).

[0477] The interpretation unit 126 generates a predicted image for the current block by performing a brightness correction process on the reference image in the reference picture specified in MV, applying its brightness correction parameter (step Sz_4). In other words, the predicted image, which is the reference image in the reference picture specified in MV, is corrected based on the brightness correction parameter. This correction may involve brightness correction or color difference correction. Specifically, color difference correction parameters may be calculated using information indicating how the color difference has changed, and color difference correction processing may be performed.

[0478] Note that the shape of the peripheral reference region in Figure 66A is just one example, and other shapes may be used.

[0479] Furthermore, although this explanation describes the process of generating a predicted image from a single reference picture, the process is similar when generating predicted images from multiple reference pictures. Alternatively, the brightness correction process may be applied to each reference picture obtained from the reference picture in the same manner as described above before generating the predicted image.

[0480] One method for determining whether to apply LIC is to use a signal called lic_flag, which indicates whether or not to apply LIC. For example, in the encoding device 100, it is determined whether the current block belongs to a region where a brightness change is occurring. If it belongs to a region where a brightness change is occurring, the value of lic_flag is set to 1 and LIC is applied and encoding is performed. If it does not belong to a region where a brightness change is occurring, the value of lic_flag is set to 0 and encoding is performed without applying LIC. On the other hand, in the decoding device 200, the lic_flag written in the stream may be decoded, and the applicability of LIC may be switched according to its value before decoding.

[0481] Another method for determining whether to apply LIC is, for example, to determine whether LIC has been applied to the surrounding blocks. Specifically, if the current block is being processed in merge mode, the interpretation unit 126 determines whether the surrounding encoded blocks selected during MV derivation in merge mode were encoded with LIC. The interpretation unit 126 then switches whether to apply LIC and performs encoding based on the result. In this example, the same process is applied to the decoding device 200.

[0482] The LIC (Luminance Correction Processing) was explained using Figures 66A and 66B, but the details will be explained below.

[0483] First, the interpretation unit 126 derives an MV for obtaining the reference image corresponding to the current block from the reference picture, which is an encoded picture.

[0484] Next, the interpretation unit 126 extracts information indicating how the luminance values ​​have changed between the reference picture and the current picture, using the luminance pixel values ​​of the left-adjacent and upper-adjacent encoded peripheral reference regions and the luminance pixel values ​​at equivalent positions in the reference picture specified by MV, and calculates luminance correction parameters. For example, let p0 be the luminance pixel value of a pixel in the peripheral reference region of the current picture, and p1 be the luminance pixel value of a pixel in the peripheral reference region of the reference picture at an equivalent position to that pixel. The interpretation unit 126 calculates coefficients A and B as luminance correction parameters to optimize A×p1+B=p0 for multiple pixels in the peripheral reference region.

[0485] Next, the interpretation unit 126 generates a predicted image for the current block by performing a brightness correction process on the reference image in the reference picture specified by MV using a brightness correction parameter. For example, let p2 be the brightness pixel value in the reference image, and p3 be the brightness pixel value of the predicted image after brightness correction processing. The interpretation unit 126 generates the predicted image after brightness correction processing by calculating A × p2 + B = p3 for each pixel in the reference image.

[0486] Furthermore, a portion of the peripheral reference area shown in Figure 66A may be used. For example, a region containing a predetermined number of pixels obtained by thinning out the upper adjacent pixels and the left adjacent pixels may be used as the peripheral reference area. Also, the peripheral reference area is not limited to the region adjacent to the current block, but may be a region not adjacent to the current block. In the example shown in Figure 66A, the peripheral reference area in the reference picture is the region specified by the MV of the current picture, but it may be a region specified by another MV. For example, the other MV may be the MV of the peripheral reference area in the current picture.

[0487] Although the operation of the encoding device 100 has been described here, the operation of the decoding device 200 is similar.

[0488] Furthermore, LIC may be applied not only to luminance but also to chrominance. In this case, correction parameters may be derived individually for each of Y, Cb, and Cr, or a common correction parameter may be used for any of them.

[0489] Furthermore, LIC processing may be applied on a subblock basis. For example, correction parameters may be derived using the surrounding reference region of the current subblock and the surrounding reference region of the reference subblock within the reference picture specified by the MV of the current subblock.

[0490] [Prediction Control Unit] The prediction control unit 128 selects either an intra-prediction image (an image or signal output from the intra-prediction unit 124) or an inter-prediction image (an image or signal output from the inter-prediction unit 126), and outputs the selected prediction image to the subtraction unit 104 and the addition unit 116.

[0491] [Prediction parameter generation unit] The prediction parameter generation unit 130 may output information regarding intra-prediction, inter-prediction, and the selection of predicted images in the prediction control unit 128 as prediction parameters to the entropy coding unit 110. The entropy coding unit 110 may generate a stream based on the prediction parameters input from the prediction parameter generation unit 130 and the quantization coefficients input from the quantization unit 108. The prediction parameters may be used by the decoding device 200. The decoding device 200 may receive and decode the stream and perform the same processing as the prediction processing performed in the intra-prediction unit 124, inter-prediction unit 126, and prediction control unit 128. The prediction parameters may include a selected prediction signal (e.g., MV, prediction type, or prediction mode used in the intra-prediction unit 124 or inter-prediction unit 126), or arbitrary indices, flags, or values ​​that are based on or indicate the prediction processing performed in the intra-prediction unit 124, inter-prediction unit 126, and prediction control unit 128.

[0492] [Decoding device] Next, a decoding device 200 capable of decoding the stream output from the above-described encoding device 100 will be explained. Figure 67 is a block diagram showing an example of the configuration of the decoding device 200 according to the embodiment. The decoding device 200 is a device that decodes the encoded image stream in block units.

[0493] As shown in Figure 67, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a division decision unit 224. The intra prediction unit 216 and the inter prediction unit 218 are each configured as part of the prediction processing unit.

[0494] [Example of a decryption device implementation] Figure 68 is a block diagram showing an example implementation of the decoding device 200. The decoding device 200 comprises a processor b1 and memory b2. For example, several components of the decoding device 200 shown in Figure 67 are implemented by the processor b1 and memory b2 shown in Figure 68.

[0495] Processor b1 is a circuit that performs information processing and is a circuit that can access memory b2. For example, processor b1 is a dedicated or general-purpose electronic circuit for decoding streams. Processor b1 may be a processor such as a CPU. Alternatively, processor b1 may be a collection of multiple electronic circuits. Furthermore, for example, processor b1 may play the role of multiple components of the decoding device 200 shown in Figure 67, etc., excluding the component for storing information.

[0496] Memory b2 is a dedicated or general-purpose memory that stores information for processor b1 to decode the stream. Memory b2 may be an electronic circuit and may be connected to processor b1. Memory b2 may also be included in processor b1. Memory b2 may also be a collection of multiple electronic circuits. Memory b2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory b2 may also be non-volatile memory or volatile memory.

[0497] For example, memory b2 may store an image or a stream. Alternatively, memory b2 may store a program for processor b1 to decode the stream.

[0498] Furthermore, for example, memory b2 may play the role of an information storage component among the multiple components of the decoding device 200 shown in Figure 67, etc. Specifically, memory b2 may play the role of the block memory 210 and frame memory 214 shown in Figure 67. More specifically, reconstructed images (specifically, reconstructed blocks or reconstructed pictures, etc.) may be stored in memory b2.

[0499] Furthermore, the decoding device 200 does not necessarily have to implement all of the components shown in Figure 67, etc., nor does it have to perform all of the processes described above. Some of the components shown in Figure 67, etc., may be included in other devices, and some of the processes described above may be performed by other devices.

[0500] The following describes the overall processing flow of the decoding device 200, followed by a description of each component included in the decoding device 200. Note that detailed explanations of components included in the decoding device 200 that perform the same processing as those included in the encoding device 100 will be omitted. For example, the inverse quantization unit 204, inverse transform unit 206, adder unit 208, block memory 210, frame memory 214, intra prediction unit 216, inter prediction unit 218, prediction control unit 220, and loop filter unit 212 included in the decoding device 200 perform the same processing as the inverse quantization unit 112, inverse transform unit 114, adder unit 116, block memory 118, frame memory 122, intra prediction unit 124, inter prediction unit 126, prediction control unit 128, and loop filter unit 120 included in the encoding device 100.

[0501] [Overall flow of the decryption process] Figure 69 is a flowchart showing an example of the overall decoding process by the decoding device 200.

[0502] First, the partitioning determination unit 224 of the decoding device 200 determines the partitioning pattern for each of the multiple fixed-size blocks (128 × 128 pixels) contained in the picture based on the parameters input from the entropy decoding unit 202 (step Sp_1). This partitioning pattern is the partitioning pattern selected by the encoding device 100. Then, the decoding device 200 performs steps Sp_2 to Sp_6 for each of the multiple blocks that constitute that partitioning pattern.

[0503] The entropy decoding unit 202 decodes (specifically, performs entropy decoding) the encoded quantization coefficients and prediction parameters of the current block (step Sp_2).

[0504] Next, the inverse quantization unit 204 and the inverse transformation unit 206 reconstruct the predicted residual of the current block by performing inverse quantization and inverse transformation on multiple quantization coefficients (step Sp_3).

[0505] Next, the prediction processing unit, consisting of the intra-prediction unit 216, the inter-prediction unit 218, and the prediction control unit 220, generates a prediction image of the current block (step Sp_4).

[0506] Next, the summing unit 208 reconstructs the current block into a reconstructed image (also called a decoded image block) by adding the predicted image to the predicted residual (step Sp_5).

[0507] Then, once this reconstructed image is generated, the loop filter unit 212 performs filtering on the reconstructed image (step Sp_6).

[0508] The decryption device 200 then determines whether or not the entire picture has been decrypted (step Sp_7). If it determines that it has not been completed (No. in step Sp_7), it repeats the process from step Sp_1.

[0509] The processes in steps Sp_1 to Sp_7 may be performed sequentially by the decoding device 200, some of these processes may be performed in parallel, and the order may be changed.

[0510] [Partitioning decision section] Figure 70 shows the relationship between the division decision unit 224 and other components. The division decision unit 224 may perform the following processing as an example.

[0511] The division decision unit 224 collects block information from, for example, the block memory 210 or the frame memory 214, and further obtains parameters from the entropy decoding unit 202. The division decision unit 224 may then determine a division pattern for fixed-size blocks based on the block information and parameters. The division decision unit 224 may then output information indicating the determined division pattern to the inverse transform unit 206, the intra prediction unit 216, and the inter prediction unit 218. The inverse transform unit 206 may perform an inverse transform on the transformation coefficients based on the division pattern indicated by the information from the division decision unit 224. The intra prediction unit 216 and the inter prediction unit 218 may generate a predicted image based on the division pattern indicated by the information from the division decision unit 224.

[0512] [Entropy Decoder] Figure 71 is a block diagram showing an example of the configuration of the entropy decoding unit 202.

[0513] The entropy decoding unit 202 generates quantization coefficients, prediction parameters, and parameters related to the partition pattern by entropy decoding the stream. For example, CABAC is used for this entropy decoding. Specifically, the entropy decoding unit 202 comprises, for example, a binary arithmetic decoding unit 202a, a context control unit 202b, and a multileveling unit 202c. The binary arithmetic decoding unit 202a arithmetically decodes the stream into a binary signal using the context value derived by the context control unit 202b. Similar to the context control unit 110b of the encoding device 100, the context control unit 202b derives context values, i.e., the probability of a binary signal occurring, according to the characteristics of the syntax elements or the surrounding circumstances. The multileveling unit 202c performs debinarization, converting the binary signal output from the binary arithmetic decoding unit 202a into a multilevel signal showing the quantization coefficients and the like. This multileveling is performed according to the binaryization method described above.

[0514] The entropy decoding unit 202 outputs the quantization coefficients to the inverse quantization unit 204 in block units. The entropy decoding unit 202 may also output the prediction parameters included in the stream (see Figure 1) to the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220. The intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220 can perform the same prediction processing as the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 on the encoding device 100 side.

[0515] [Entropy Decoder] Figure 72 shows the CABAC flow in the entropy decoding unit 202.

[0516] First, the CABAC in the entropy decoding unit 202 is initialized. This initialization includes the initialization of the binary arithmetic decoding unit 202a and the setting of an initial context value. Then, the binary arithmetic decoding unit 202a and the multi-leveling unit 202c perform arithmetic decoding and multi-leveling on the encoded data of, for example, the CTU. At this time, the context control unit 202b updates the context value each time arithmetic decoding is performed. Then, as a post-processing step, the context control unit 202b saves the context value. This saved context value is used, for example, as the initial value of the context value for the next CTU.

[0517] [Dequantization section] The inverse quantization unit 204 inversely quantizes the quantization coefficients of the current block, which are input from the entropy decoding unit 202. Specifically, for each quantization coefficient of the current block, the inverse quantization unit 204 inversely quantizes the quantization coefficient based on the quantization parameter corresponding to that quantization coefficient. The inverse quantization unit 204 then outputs the inversely quantized quantization coefficients (i.e., transformation coefficients) of the current block to the inverse transformation unit 206.

[0518] Figure 73 is a block diagram showing an example of the configuration of the inverse quantization unit 204.

[0519] The inverse quantization unit 204 includes, for example, a quantization parameter generation unit 204a, a prediction quantization parameter generation unit 204b, a quantization parameter storage unit 204d, and an inverse quantization processing unit 204e.

[0520] Figure 74 is a flowchart showing an example of inverse quantization by the inverse quantization unit 204.

[0521] The inverse quantization unit 204 may, for example, perform inverse quantization processing for each CU based on the flow shown in Figure 74. Specifically, the quantization parameter generation unit 204a determines whether or not to perform inverse quantization (step Sv_11). If it is determined that inverse quantization should be performed (Yes in step Sv_11), the quantization parameter generation unit 204a obtains the difference quantization parameters of the current block from the entropy decoding unit 202 (step Sv_12).

[0522] Next, the predictive quantization parameter generation unit 204b obtains quantization parameters for a processing unit different from the current block from the quantization parameter storage unit 204d (step Sv_13). Based on the obtained quantization parameters, the predictive quantization parameter generation unit 204b generates predictive quantization parameters for the current block (step Sv_14).

[0523] Then, the quantization parameter generation unit 204a adds the difference quantization parameter of the current block obtained from the entropy decoding unit 202 and the predicted quantization parameter of the current block generated by the predicted quantization parameter generation unit 204b (step Sv_15). This addition generates the quantization parameter of the current block. The quantization parameter generation unit 204a then stores the quantization parameter of the current block in the quantization parameter storage unit 204d (step Sv_16).

[0524] Next, the inverse quantization processing unit 204e inversely quantizes the quantization coefficients of the current block into transformation coefficients using the quantization parameters generated in step Sv_15 (step Sv_17).

[0525] The differential quantization parameters may be decoded at the bit sequence level, picture level, slice level, brick level, or CTU level. The initial values ​​of the quantization parameters may also be decoded at the sequence level, picture level, slice level, brick level, or CTU level. In this case, the quantization parameters may be generated using the initial values ​​of the quantization parameters and the differential quantization parameters.

[0526] The inverse quantization unit 204 may be equipped with multiple inverse quantizers, and the quantization coefficients may be inversely quantized using an inverse quantization method selected from multiple inverse quantization methods.

[0527] [Inverse Transformation Section] The inverse transform unit 206 recovers the predicted residual by inversely transforming the transformation coefficients, which are input from the inverse quantization unit 204.

[0528] For example, if the information read from the stream indicates that EMT or AMT should be applied (e.g., the AMT flag is true), the inverse transform unit 206 inversely transforms the transformation coefficients of the current block based on the information indicating the type of transformation read.

[0529] For example, if the information read from the stream indicates that NSST should be applied, the inverse conversion unit 206 applies inverse re-conversion to the conversion coefficients.

[0530] Figure 75 is a flowchart showing an example of processing by the inverse transform unit 206.

[0531] For example, the inverse transform unit 206 determines whether or not information indicating that an orthogonal transform is not performed exists in the stream (step St_11). If it determines that such information does not exist (No in step St_11), the inverse transform unit 206 obtains information indicating the transform type, which has been decoded by the entropy decoding unit 202 (step St_12). Next, the inverse transform unit 206 determines the transform type used for the orthogonal transform of the encoding device 100 based on that information (step St_13). Then, the inverse transform unit 206 performs an inverse orthogonal transform using the determined transform type (step St_14).

[0532] Figure 76 is a flowchart showing another example of processing by the inverse transform unit 206.

[0533] For example, the inverse transform unit 206 determines whether the transformation size is less than or equal to a predetermined value (step Su_11). If it determines that it is less than or equal to the predetermined value (Yes in step Su_11), the inverse transform unit 206 obtains information from the entropy decoding unit 202 indicating which of the one or more transformation types included in the first transformation type group was used by the encoding device 100 (step Su_12). This information is decoded by the entropy decoding unit 202 and output to the inverse transform unit 206.

[0534] Based on this information, the inverse transform unit 206 determines the type of transformation used in the orthogonal transformation in the encoding device 100 (step Su_13). Then, the inverse transform unit 206 performs an inverse orthogonal transformation on the transformation coefficients of the current block using the determined transformation type (step Su_14). On the other hand, if the inverse transform unit 206 determines in step Su_11 that the transformation size is not less than or equal to a predetermined value (No. in step Su_11), it performs an inverse orthogonal transformation on the transformation coefficients of the current block using a second group of transformation types (step Su_15).

[0535] The inverse orthogonal transformation by the inverse transformation unit 206 may, for example, be performed for each TU according to the flow shown in Figure 75 or Figure 76. Alternatively, the inverse orthogonal transformation may be performed using a predetermined transformation type without decoding the information indicating the transformation type used in the orthogonal transformation. Specifically, the transformation type may be DST7 or DCT8, and the inverse transformation basis function corresponding to that transformation type is used in the inverse orthogonal transformation.

[0536] [Addition section] The summing unit 208 reconstructs the current block by adding the predicted residual, which is the input from the inverse transform unit 206, and the predicted image, which is the input from the prediction control unit 220. In other words, a reconstructed image of the current block is generated. The summing unit 208 then outputs the reconstructed image of the current block to the block memory 210 and the loop filter unit 212.

[0537] [Block memory] The block memory 210 is a block referenced in intra prediction and is a storage unit for storing blocks within the current picture. Specifically, the block memory 210 stores the reconstructed image output from the adder 208.

[0538] [Loop Filter Section] The loop filter unit 212 applies a loop filter to the reconstructed image generated by the adder unit 208, and outputs the filtered reconstructed image to the frame memory 214 and a display device, etc.

[0539] If the information interpreted from the stream indicating ALF on / off indicates ALF is on, one filter is selected from several filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed image.

[0540] Figure 77 is a block diagram showing an example of the configuration of the loop filter unit 212. Note that the loop filter unit 212 has the same configuration as the loop filter unit 120 of the encoding device 100.

[0541] The loop filter unit 212 includes, for example, a deblocking filter processing unit 212a, an SAO processing unit 212b, and an ALF processing unit 212c, as shown in Figure 77. The deblocking filter processing unit 212a applies the above-described deblocking filter processing to the reconstructed image. The SAO processing unit 212b applies the above-described SAO processing to the reconstructed image after the deblocking filter processing. The ALF processing unit 212c applies the above-described ALF processing to the reconstructed image after the SAO processing. Note that the loop filter unit 212 does not have to include all the processing units disclosed in Figure 77, and may include only some of the processing units. Also, the loop filter unit 212 may be configured to perform the above-described processing in an order different from the processing order disclosed in Figure 77.

[0542] [Frame memory] The frame memory 214 is a memory unit for storing reference pictures used in interpretation, and is sometimes called a frame buffer. Specifically, the frame memory 214 stores the reconstructed image that has been filtered by the loop filter unit 212.

[0543] [Prediction Unit (Intra Prediction Unit, Inter Prediction Unit, Prediction Control Unit)] Figure 78 is a flowchart showing an example of processing performed in the prediction unit of the decoding device 200. For example, the prediction unit consists of all or some of the components of the intra-prediction unit 216, the inter-prediction unit 218, and the prediction control unit 220. The prediction processing unit includes, for example, the intra-prediction unit 216 and the inter-prediction unit 218.

[0544] The prediction unit generates a predicted image of the current block (step Sq_1). This predicted image is also called a predicted signal or predicted block. The predicted signal may include, for example, an intra-prediction signal or an inter-prediction signal. Specifically, the prediction unit generates the predicted image of the current block using a reconstructed image already obtained by generating predicted images for other blocks, restoring prediction residuals, and adding predicted images. The prediction unit of the decoder 200 generates the same predicted image as the prediction unit of the encoder 100. In other words, the methods for generating predicted images used by these prediction units are common to or correspond to each other.

[0545] The reconstructed image may be, for example, the image of the reference picture, or it may be the image of the decoded block (i.e., the other block mentioned above) within the current picture, which is the picture containing the current block. The decoded block within the current picture is, for example, the adjacent block to the current block.

[0546] Figure 79 is a flowchart showing another example of the processing performed in the prediction unit of the decoding device 200.

[0547] The prediction unit determines a method or mode for generating the predicted image (step Sr_1). For example, this method or mode may be determined based on, for example, prediction parameters.

[0548] If the prediction unit determines a first method as the mode for generating the predicted image, it generates the predicted image according to that first method (step Sr_2a). If the prediction unit determines a second method as the mode for generating the predicted image, it generates the predicted image according to that second method (step Sr_2b). If the prediction unit determines a third method as the mode for generating the predicted image, it generates the predicted image according to that third method (step Sr_2c).

[0549] The first, second, and third methods are different methods for generating predictive images, and may be, for example, an interpretation method, an intrapretation method, and other prediction methods, respectively. These prediction methods may use the reconstructed images described above.

[0550] Figures 80A and 80B are flowcharts showing other examples of processing performed in the prediction unit of the decoding device 200.

[0551] The prediction unit may perform prediction processing according to the flow shown in Figures 80A and 80B as an example. The intrablock copy shown in Figures 80A and 80B is one mode belonging to interprediction, in which blocks included in the current picture are referenced as reference images or reference blocks. In other words, in intrablock copy, pictures different from the current picture are not referenced. The PCM mode shown in Figure 80A is also one mode belonging to intraprediction, in which no transformation or quantization is performed.

[0552] [Intra Prediction Unit] The intra-prediction unit 216 generates a predicted image of the current block (i.e., an intra-predicted image) by performing intra-prediction based on the intra-prediction mode decoded from the stream, by referring to the blocks in the current picture stored in the block memory 210. Specifically, the intra-prediction unit 216 generates an intra-predicted image by performing intra-prediction by referring to the pixel values ​​(e.g., luminance values, chrominance values) of blocks adjacent to the current block, and outputs the intra-predicted image to the prediction control unit 220.

[0553] Furthermore, if an intra-prediction mode that references a luminance block is selected in the intra-prediction of a color difference block, the intra-prediction unit 216 may predict the color difference component of the current block based on the luminance component of the current block.

[0554] Furthermore, if the information interpreted from the stream indicates the application of PDPC, the intra-prediction unit 216 corrects the pixel value after intra-prediction based on the gradient of the reference pixels in the horizontal / vertical directions.

[0555] Figure 81 shows an example of processing by the intra-prediction unit 216 of the decoding device 200.

[0556] First, the intra-prediction unit 216 determines whether or not an MPM flag indicating 1 exists in the stream (step Sw_11). If it determines that an MPM flag indicating 1 exists (Yes in step Sw_11), the intra-prediction unit 216 obtains information from the entropy decoding unit 202 indicating the intra-prediction mode selected by the encoding device 100 from the MPM (step Sw_12). This information is decoded by the entropy decoding unit 202 and output to the intra-prediction unit 216. Next, the intra-prediction unit 216 determines the MPM (step Sw_13). The MPM consists of, for example, six intra-prediction modes. Then, the intra-prediction unit 216 determines the intra-prediction mode indicated by the information obtained in step Sw_12 from among the multiple intra-prediction modes included in the MPM (step Sw_14).

[0557] Meanwhile, if the intra-prediction unit 216 determines in step Sw_11 that there is no MPM flag indicating 1 in the stream (No. in step Sw_11), it obtains information indicating the intra-prediction mode selected by the encoding device 100 (step Sw_15). In other words, the intra-prediction unit 216 obtains information from the entropy decoding unit 202 indicating the intra-prediction mode selected by the encoding device 100 from among one or more intra-prediction modes not included in the MPM. This information is decoded by the entropy decoding unit 202 and output to the intra-prediction unit 216. The intra-prediction unit 216 then determines the intra-prediction mode indicated by the information obtained in step Sw_15 from among the one or more intra-prediction modes not included in the MPM (step Sw_17).

[0558] The intra-prediction unit 216 generates a predicted image according to the intra-prediction mode determined in step Sw_14 or step Sw_17 (step Sw_18).

[0559] [International Prediction Department] The interpretation unit 218 predicts the current block by referring to a reference picture stored in the frame memory 214. Prediction is performed in units of the current block or subblocks within the current block. A subblock is a unit contained within a block and is smaller than a block. The size of a subblock may be 4x4 pixels, 8x8 pixels, or other sizes. The size of a subblock may be switched to units such as slices, bricks, or pictures.

[0560] For example, the interpretation unit 218 generates an interpretation image of the current block or subblock by performing motion compensation using motion information (e.g., MV) decoded from the stream (e.g., prediction parameters output from the entropy decoding unit 202), and outputs the interpretation image to the prediction control unit 220.

[0561] If the information read from the stream indicates that OBMC mode should be applied, the interpretation unit 218 generates an interpretation prediction image using not only the motion information of the current block obtained by motion search, but also the motion information of adjacent blocks.

[0562] Furthermore, if the information deciphered from the stream indicates that FRUC mode should be applied, the interpretation unit 218 derives motion information by performing a motion search according to the pattern matching method (bilateral matching or template matching) deciphered from the stream. Then, the interpretation unit 218 performs motion compensation (prediction) using the derived motion information.

[0563] Furthermore, when the BIO mode is applied, the interpretation unit 218 derives the MV based on a model that assumes uniform linear motion. Also, if the information read from the stream indicates that the affine mode is applied, the interpretation unit 218 derives the MV on a sub-block basis based on the MVs of multiple adjacent blocks.

[0564] [MV Derivation Flow] Figure 82 is a flowchart showing an example of MV derivation in the decoding device 200.

[0565] The interpretation unit 218 determines, for example, whether or not to decode motion information (e.g., MV). For example, the interpretation unit 218 may make this determination according to the prediction mode included in the stream, or it may make this determination based on other information included in the stream. Here, if the interpretation unit 218 determines to decode the motion information, it derives the MV of the current block in the mode for decoding the motion information. On the other hand, if the interpretation unit 218 determines not to decode the motion information, it derives the MV in the mode for not decoding the motion information.

[0566] Here, the modes for MV derivation include the normal inter mode, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that decode motion information include the normal inter mode, normal merge mode, and affine mode (specifically, affine inter mode and affine merge mode). Note that the motion information may include not only MV but also predicted MV selection information, which will be described later. Modes that do not decode motion information include the FRUC mode, etc. The inter prediction unit 218 selects a mode from these multiple modes for deriving the MV of the current block and derives the MV of the current block using the selected mode.

[0567] Figure 83 is a flowchart showing another example of MV derivation in the decoding device 200.

[0568] The interpretation unit 218 determines, for example, whether or not to decode the differential MV. For example, the interpretation unit 218 may make this determination according to the prediction mode included in the stream, or it may make this determination based on other information included in the stream. Here, if the interpretation unit 218 determines to decode the differential MV, it may derive the MV of the current block in the mode for decoding the differential MV. In this case, for example, the differential MV included in the stream is decoded as a prediction parameter.

[0569] On the other hand, if the interpretation unit 218 determines that it does not decode the differential MV, it derives the MV in a mode that does not decode the differential MV. In this case, the encoded differential MV is not included in the stream.

[0570] As mentioned above, the modes for MV derivation include the normal inter, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that encode differential MV include the normal inter mode and affine mode (specifically, affine inter mode). Modes that do not encode differential MV include the FRUC mode, normal merge mode, and affine mode (specifically, affine merge mode). The inter prediction unit 218 selects a mode from these multiple modes to derive the MV of the current block, and uses the selected mode to derive the MV of the current block.

[0571] [MV Derivation > Normal Intermode] For example, if the information interpreted from the stream indicates that the normal intermode should be applied, the interpretation unit 218 derives the MV in normal merge mode based on the information interpreted from the stream and uses that MV to perform motion compensation (prediction).

[0572] Figure 84 is a flowchart showing an example of inter-mode prediction in the decoding device 200.

[0573] The interpretation unit 218 of the decoding device 200 performs motion compensation for each block. In this process, the interpretation unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks surrounding the current block in time or space (step Sg_11). In other words, the interpretation unit 218 creates a candidate MV list.

[0574] Next, the interpretation unit 218 extracts N candidate MVs (where N is an integer greater than or equal to 2) from the multiple candidate MVs obtained in step Sg_11 as predicted motion vector candidates (also called predicted MV candidates) according to a predetermined priority order (step Sg_12). The priority order is predetermined for each of the N predicted MV candidates.

[0575] Next, the interpretation unit 218 decodes the predicted MV selection information from the input stream and uses the decoded predicted MV selection information to select one predicted MV candidate from the N predicted MV candidates as the predicted MV for the current block (step Sg_13).

[0576] Next, the interpretation unit 218 decodes the differential MV from the input stream and derives the MV of the current block by adding the decoded differential MV (the difference value) to the selected predicted MV (step Sg_14).

[0577] Finally, the inter-prediction unit 218 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Sg_15). The processes in steps Sg_11 to Sg_15 are performed for each block. For example, when the processes in steps Sg_11 to Sg_15 are performed for each block contained in a slice, the inter-prediction using the normal inter-mode for that slice is completed. Similarly, when the processes in steps Sg_11 to Sg_15 are performed for each block contained in a picture, the inter-prediction using the normal inter-mode for that picture is completed. Note that the processes in steps Sg_11 to Sg_15 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using the normal inter-mode for that slice is completed. Likewise, when the processes in steps Sg_11 to Sg_15 are performed for some blocks contained in a picture, the inter-prediction using the normal inter-mode for that picture is completed.

[0578] [MV Derivation > Normal Merge Mode] For example, if the information read from the stream indicates the application of normal merge mode, the interpretation unit 218 derives MV in normal merge mode and uses that MV to perform motion compensation (prediction).

[0579] Figure 85 is a flowchart showing an example of inter prediction using normal merge mode in the decoding device 200.

[0580] The interpretation unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks surrounding the current block in terms of time or space (step Sh_11). In other words, the interpretation unit 218 creates a candidate MV list.

[0581] Next, the interpretation unit 218 derives the MV of the current block by selecting one candidate MV from among the multiple candidate MVs obtained in step Sh_11 (step Sh_12). Specifically, the interpretation unit 218 obtains, for example, MV selection information included in the stream as prediction parameters, and selects the candidate MV identified by that MV selection information as the MV of the current block.

[0582] Finally, the inter-prediction unit 218 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Sh_13). The processes in steps Sh_11 to Sh_13 are performed for each block, for example. For example, when the processes in steps Sh_11 to Sh_13 are performed for each of the blocks contained in a slice, the inter-prediction using normal merge mode for that slice is completed. Similarly, when the processes in steps Sh_11 to Sh_13 are performed for each of the blocks contained in a picture, the inter-prediction using normal merge mode for that picture is completed. Note that the processes in steps Sh_11 to Sh_13 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using normal merge mode for that slice is completed. Likewise, when the processes in steps Sh_11 to Sh_13 are performed for some of the blocks contained in a picture, the inter-prediction using normal merge mode for that picture is completed.

[0583] [MV Derivation > FRUC Mode] For example, if the information decoded from the stream indicates the application of FRUC mode, the interpretation unit 218 derives the MV in FRUC mode and uses that MV to perform motion compensation (prediction). In this case, the motion information is not signaled from the encoding device 100 but is derived from the decoding device 200. For example, the decoding device 200 may derive the motion information by performing a motion search. In this case, the decoding device 200 performs a motion search without using the pixel values ​​of the current block.

[0584] Figure 86 is a flowchart showing an example of interpretation using FRUC mode in the decoding device 200.

[0585] First, the interpretation unit 218 refers to the MVs of each decoded block that is spatially or temporally adjacent to the current block and generates a list of candidate MVs (i.e., a candidate MV list, which may be the same as the candidate MV list for normal merge mode) (step Si_11). Next, the interpretation unit 218 selects the best candidate MV from among the multiple candidate MVs registered in the candidate MV list (step Si_12). For example, the interpretation unit 218 calculates an evaluation value for each candidate MV included in the candidate MV list and selects one candidate MV as the best candidate MV based on that evaluation value. Then, the interpretation unit 218 derives the MV for the current block based on the selected best candidate MV (step Si_14). Specifically, for example, the selected best candidate MV is directly derived as the MV for the current block. Alternatively, for example, the MV for the current block may be derived by performing pattern matching in the area surrounding the position in the reference picture corresponding to the selected best candidate MV. In other words, a search is performed in the area surrounding the best candidate MV using pattern matching and evaluation values ​​in the reference picture. If an MV with a better evaluation value is found, the best candidate MV may be updated to that MV and made the final MV of the current block. It is not necessary to update to an MV with a better evaluation value.

[0586] Finally, the inter-prediction unit 218 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Si_15). The processes in steps Si_11 to Si_15 are performed for each block, for example. For example, when the processes in steps Si_11 to Si_15 are performed for each of the blocks contained in a slice, the inter-prediction using FRUC mode for that slice is completed. Similarly, when the processes in steps Si_11 to Si_15 are performed for each of the blocks contained in a picture, the inter-prediction using FRUC mode for that picture is completed. Sub-blocks may also be processed in the same way as the block units described above.

[0587] [MV Derivation > Affine Merge Mode] For example, if the information deciphered from the stream indicates the application of affine merge mode, the interpretation unit 218 derives the motion video (MV) in affine merge mode and uses that MV to perform motion compensation (prediction).

[0588] Figure 87 is a flowchart showing an example of interpretation using the affine merge mode in the decoding device 200.

[0589] In affine merge mode, the interpretation unit 218 first derives the MV of each control point of the current block (step Sk_11). The control points are the upper left and upper right corners of the current block, as shown in Figure 46A, or the upper left, upper right, and lower left corners of the current block, as shown in Figure 46B.

[0590] For example, when using the MV derivation method shown in Figures 47A to 47C, the interpretation unit 218 examines the decoded blocks in the order of Block A (left), Block B (top), Block C (upper right), Block D (lower left), and Block E (upper left), as shown in Figure 47A, and identifies the first valid block decoded in affine mode.

[0591] The interpretation unit 218 derives the motion vectors (MV) of the control points using the first valid block decoded in the identified affine mode. For example, if block A is identified and block A has two control points, as shown in Figure 47B, the interpretation unit 218 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block by projecting the motion vectors v3 and v4 of the upper left and upper right corners of the decoded block containing block A onto the current block. This derives the MV of each control point.

[0592] Furthermore, as shown in Figure 49A, if block A is identified and block A has two control points, the MV of three control points may be calculated. Alternatively, as shown in Figure 49B, if block A is identified and block A has three control points, the MV of two control points may be calculated.

[0593] Furthermore, if the stream includes MV selection information as a prediction parameter, the interpretation unit 218 may use that MV selection information to derive the MV of each control point in the current block.

[0594] Next, the interpretation unit 218 performs motion compensation for each of the multiple subblocks contained in the current block. That is, for each of the multiple subblocks, the interpretation unit 218 calculates the MV of that subblock as an affine MV using two motion vectors v0 and v1 and equation (1A) described above, or using three motion vectors v0, v1 and v2 and equation (1B) described above (step Sk_12). Then, the interpretation unit 218 performs motion compensation for that subblock using those affine MVs and the decoded reference picture (step Sk_13). Once steps Sk_12 and Sk_13 have been executed for each of the subblocks contained in the current block, the interpretation using the affine merge mode for that current block is completed. In other words, motion compensation is performed for the current block, and a predicted image of that current block is generated.

[0595] In step Sk_11, the above-mentioned candidate MV list may be generated. The candidate MV list may be a list containing candidate MVs derived for each control point using multiple MV derivation methods. The multiple MV derivation methods may be any combination of the MV derivation methods shown in Figures 47A to 47C, the MV derivation methods shown in Figures 48A and 48B, the MV derivation methods shown in Figures 49A and 49B, and other MV derivation methods.

[0596] The candidate MV list may also include candidate MVs from modes other than affine mode, where prediction is performed on a sub-block basis.

[0597] Furthermore, the candidate MV list may include, for example, candidate MVs for an affine merge mode with two control points and candidate MVs for an affine merge mode with three control points. Alternatively, a candidate MV list containing candidate MVs for an affine merge mode with two control points and a candidate MV list containing candidate MVs for an affine merge mode with three control points may be generated separately. Alternatively, a candidate MV list may be generated containing candidate MVs for one of the modes: an affine merge mode with two control points or an affine merge mode with three control points.

[0598] [MV Derivation > Affine Intermode] For example, if the information deciphered from the stream indicates the application of affine intermode, the interpretation unit 218 derives the motion video (MV) using affine intermode and performs motion compensation (prediction) using that MV.

[0599] Figure 88 is a flowchart showing an example of inter-prediction using affine intermode in the decoding device 200.

[0600] In affine intermode, the interpretation unit 218 first derives the predicted MV(v0,v1) or (v0,v1,v2) for each of two or three control points of the current block (step Sj_11). The control points are, for example, the upper left corner, upper right corner, or lower left corner of the current block, as shown in Figure 46A or Figure 46B.

[0601] The interpretation unit 218 acquires predicted MV selection information included in the stream as prediction parameters, and uses the MVs identified by the predicted MV selection information to derive the predicted MV for each control point in the current block. For example, when using the MV derivation method shown in Figures 48A and 48B, the interpretation unit 218 derives the predicted MV (v0,v1) or (v0,v1,v2) for the control point of the current block by selecting the MV of the block identified by the predicted MV selection information from among the decoded blocks near each control point in the current block shown in Figure 48A or Figure 48B.

[0602] Next, the interpretation unit 218 obtains, for example, each differential MV included in the stream as a prediction parameter, and adds the predicted MV of each control point in the current block to the differential MV corresponding to that predicted MV (step Sj_12). This derives the MV of each control point in the current block.

[0603] Next, the interpretation unit 218 performs motion compensation for each of the multiple subblocks contained in the current block. That is, for each of the multiple subblocks, the interpretation unit 218 calculates the MV of that subblock as an affine MV using two motion vectors v0 and v1 and equation (1A) described above, or using three motion vectors v0, v1 and v2 and equation (1B) described above (step Sj_13). Then, the interpretation unit 218 performs motion compensation for that subblock using those affine MVs and the decoded reference picture (step Sj_14). Once steps Sj_13 and Sj_14 have been executed for each of the subblocks contained in the current block, the interpretation using the affine merge mode for that current block is completed. In other words, motion compensation is performed for the current block, and a predicted image of that current block is generated.

[0604] In step Sj_11, the candidate MV list described above may be generated, similar to step Sk_11.

[0605] [MV Derivation > Triangle Mode] For example, if the information interpreted from the stream indicates the application of triangle mode, the interpretation unit 218 derives MV in triangle mode and uses that MV to perform motion compensation (prediction).

[0606] Figure 89 is a flowchart showing an example of inter prediction using triangle mode in the decoding device 200.

[0607] In triangle mode, the interpretation unit 218 first divides the current block into a first partition and a second partition (step Sx_11). At this time, the interpretation unit 218 may obtain partition information, which is information about the division into each partition, from the stream as prediction parameters. Then, the interpretation unit 218 may divide the current block into a first partition and a second partition according to that partition information.

[0608] Next, the interpretation unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks surrounding the current block in time or space (step Sx_12). In other words, the interpretation unit 218 creates a candidate MV list.

[0609] Then, the interpretation unit 218 selects the candidate MV for the first partition and the candidate MV for the second partition from among the multiple candidate MVs obtained in step Sx_11 as the first MV and the second MV, respectively (step Sx_13). At this time, the interpretation unit 218 may obtain MV selection information to identify the selected candidate MVs from the stream as prediction parameters. Then, the interpretation unit 218 may select the first MV and the second MV according to that MV selection information.

[0610] Next, the interpretation unit 218 generates a first predicted image by performing motion compensation using the selected first MV and the decoded reference picture (step Sx_14). Similarly, the interpretation unit 218 generates a second predicted image by performing motion compensation using the selected second MV and the decoded reference picture (step Sx_15).

[0611] Finally, the interpretation unit 218 generates a predicted image of the current block by weighting and adding the first predicted image and the second predicted image (step Sx_16).

[0612] [Motion Detection > DMVR] For example, if the information deciphered from the stream indicates the application of DMVR, the interpretation unit 218 performs motion search using DMVR.

[0613] Figure 90 is a flowchart showing an example of motion detection using DMVR in the decoding device 200.

[0614] The interpretation unit 218 first derives the MV of the current block in merge mode (step Sl_11). Next, the interpretation unit 218 derives the final MV for the current block by searching the surrounding region of the reference picture indicated by the MV derived in step Sl_11 (step Sl_12). In other words, the MV of the current block is determined by the DMVR.

[0615] Figure 91 is a flowchart showing a detailed example of motion detection by the DMVR in the decoding device 200.

[0616] First, in Step 1 shown in Figure 58A, the interpretation unit 218 calculates the cost of the search position indicated by the initial MV (also called the starting point) and the eight surrounding search positions. Then, the interpretation unit 218 determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 218 determines that the cost of the search positions other than the starting point is the minimum, it moves to the search position with the minimum cost and performs the process shown in Figure 58A in Step 2. On the other hand, if the cost of the starting point is the minimum, the interpretation unit 218 skips the process shown in Figure 58A in Step 2 and proceeds to the process shown in Step 3.

[0617] In Step 2, shown in Figure 58A, the interpretation unit 218 uses the search position moved according to the processing result of Step 1 as a new starting point and performs a search similar to the process in Step 1. The interpretation unit 218 then determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 218 determines that the cost of the search positions other than the starting point is the minimum, it proceeds to Step 4. On the other hand, if the interpretation unit 218 determines that the cost of the starting point is the minimum, it proceeds to Step 3.

[0618] In Step 4, the interpretation unit 218 treats the starting position as the final search position and determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0619] In Step 3 shown in Figure 58A, the interpretation unit 218 determines the pixel position with the minimum cost based on the costs of the four points above, below, to the left and right of the starting point in Step 1 or Step 2, and sets that pixel position as the final search position. This decimal-precision pixel position is determined by weighting the vector of the four points above, below, to the left and right ((0,1),(0,-1),(-1,0),(1,0)) with the cost of each of the four search positions as the weight. The interpretation unit 218 then determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0620] [Motion compensation > BIO / OBMC / LIC] For example, if the information read from the stream indicates the application of correction to the predicted image, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to the correction mode. These modes include, for example, the BIO, OBMC, and LIC mentioned above.

[0621] Figure 92 is a flowchart showing an example of predictive image generation in the decoding device 200.

[0622] The interpretation unit 218 generates a predicted image (step Sm_11) and corrects the predicted image using one of the modes described above (step Sm_12).

[0623] Figure 93 is a flowchart showing another example of predictive image generation in the decoding device 200.

[0624] The interpretation unit 218 derives the MV of the current block (step Sn_11). Next, the interpretation unit 218 generates a predicted image using the MV (step Sn_12) and determines whether or not to perform correction processing (step Sn_13). For example, the interpretation unit 218 obtains prediction parameters included in the stream and determines whether or not to perform correction processing based on these prediction parameters. These prediction parameters are, for example, flags indicating whether or not to apply each of the above modes. If the interpretation unit 218 determines that correction processing should be performed (Yes in step Sn_13), it generates the final predicted image by correcting the predicted image (step Sn_14). In LIC, the brightness and color difference of the predicted image may be corrected in step Sn_14. On the other hand, if the interpretation unit 218 determines that no correction processing should be performed (No in step Sn_13), it outputs the predicted image as the final predicted image without correction (step Sn_15).

[0625] [Motion Compensation > OBMC] For example, if the information interpreted from the stream indicates the application of OBMC, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to OBMC.

[0626] Figure 94 is a flowchart showing an example of predictive image correction by OBMC in the decoding device 200. The flowchart in Figure 94 shows the process of predictive image correction using the current picture and reference picture shown in Figure 62.

[0627] First, as shown in Figure 62, the interpretation unit 218 acquires a predicted image (Pred) using normal motion compensation with the MV assigned to the current block.

[0628] Next, the interpretation unit 218 applies (reuses) the already derived MV (MV_L) for the decoded left adjacent block to the current block to obtain a predicted image (Pred_L). Then, the interpretation unit 218 performs the first correction of the predicted image by superimposing the two predicted images, Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.

[0629] Similarly, the interpretation unit 218 applies (reuses) the already derived MV (MV_U) for the decoded upper adjacent block to the current block to obtain a predicted image (Pred_U). Then, the interpretation unit 218 performs a second correction of the predicted image by superimposing the predicted image Pred_U onto the first corrected predicted image (e.g., Pred and Pred_L). This has the effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is the final predicted image of the current block, with the boundaries with adjacent blocks blended (smoothed).

[0630] [Motion Compensation > BIO] For example, if the information interpreted from the stream indicates the application of BIO, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to BIO.

[0631] Figure 95 is a flowchart showing an example of BIO correction of the predicted image in the decoding device 200.

[0632] As shown in Figure 63, the interpretation unit 218 derives two motion vectors (M0, M1) using two reference pictures (Ref0, Ref1) that are different from the picture (Cur Pic) containing the current block. Then, the interpretation unit 218 derives a predicted image of the current block using these two motion vectors (M0, M1) (step Sy_11). Motion vector M0 is the motion vector (MVx0, MVy0) corresponding to reference picture Ref0, and motion vector M1 is the motion vector (MVx1, MVy1) corresponding to reference picture Ref1.

[0633] Next, the interpretation unit 218 interpolates the current block image I using the motion vector M0 and the reference picture L0. 0 The following is derived. Furthermore, the interpretation unit 218 uses the motion vector M1 and the reference picture L1 to interpolate the current block image I 1 Derive the following (step Sy_12). Here, interpolated image I 0 This is the image contained in the reference picture Ref0, which is derived for the current block, and is the interpolated image I 1 This is the image contained in reference picture Ref1, which is derived for the current block. Interpolated image I 0 and interpolated image I 1 Each of these may be the same size as the current block. Alternatively, interpolated image I 0 and interpolated image I 1 Each of these may be a larger image than the current block in order to properly derive the gradient image described later. Furthermore, interpolated image I 0 and I 1 This may include a motion vector (M0, M1) and a reference picture (L0, L1), and a predicted image derived by applying a motion compensation filter.

[0634] Furthermore, the interpretation unit 218 generates interpolated image I 0 and interpolated image I 1 From there, the gradient image of the current block (Ix 0 ,Ix 1 Iy 0 Iy 1 ) is derived (step Sy_13). Note that the horizontal gradient image is (Ix 0 ,Ix 1 ) and the vertical gradient image is (Iy 0 Iy 1 The interpretation unit 218 may derive the gradient image by, for example, applying a gradient filter to the interpolated image. The gradient image only needs to show the amount of spatial change in pixel values ​​along the horizontal or vertical direction.

[0635] Next, the interpretation unit 218 interpolates the image (I) in units of multiple subblocks that make up the current block. 0 ,I 1 ) and gradient image (Ix 0 ,Ix 1 Iy 0 Iy 1 The optical flow (vx, vy), which is the velocity vector described above, is derived using (step Sy_14). As an example, the subblock may be a 4x4 pixel subCU.

[0636] Next, the interpretation unit 218 corrects the predicted image of the current block using optical flow (vx, vy). For example, the interpretation unit 218 derives correction values ​​for the pixel values ​​included in the current block using optical flow (vx, vy) (step Sy_15). The interpretation unit 218 may then correct the predicted image of the current block using the correction values ​​(step Sy_16). The correction values ​​may be derived for each pixel, or for multiple pixels or subblocks.

[0637] Note that the BIO processing flow is not limited to the processing disclosed in Figure 95. Only a portion of the processing disclosed in Figure 95 may be performed, different processing may be added or replaced, or the processing may be executed in a different order.

[0638] [Motion Compensation > LIC] For example, if the information interpreted from the stream indicates the application of LIC, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to LIC.

[0639] Figure 96 is a flowchart showing an example of image correction by LIC in the decoding device 200.

[0640] First, the interpretation unit 218 uses MV to obtain the reference image corresponding to the current block from the decoded reference picture (step Sz_11).

[0641] Next, the interpretation unit 218 extracts information for the current block indicating how the luminance values ​​have changed between the reference picture and the current picture (step Sz_12). This extraction is performed based on the luminance pixel values ​​of the decoded left adjacent reference region (surrounding reference region) and the decoded upper adjacent reference region (surrounding reference region) in the current picture, as shown in Figure 66A, and the luminance pixel values ​​at the equivalent positions in the reference picture specified by the derived MV. Then, the interpretation unit 218 calculates luminance correction parameters using the information indicating how the luminance values ​​have changed (step Sz_13).

[0642] The interpretation unit 218 generates a predicted image for the current block by performing a brightness correction process on the reference image in the reference picture specified in MV, applying its brightness correction parameter (step Sz_14). In other words, the predicted image, which is the reference image in the reference picture specified in MV, is corrected based on the brightness correction parameter. This correction may involve brightness correction or color difference correction.

[0643] [Prediction Control Unit] The prediction control unit 220 selects either an intra-predicted image or an inter-predicted image and outputs the selected prediction image to the summing unit 208. Overall, the configuration, functions, and processing of the prediction control unit 220, intra-prediction unit 216, and inter-prediction unit 218 on the decoding device 200 side may correspond to the configuration, functions, and processing of the prediction control unit 128, intra-prediction unit 124, and inter-prediction unit 126 on the encoding device 100 side.

[0644] [Resolution constraints] The constraints on resolution will be explained below using several embodiments and examples.

[0645] (First aspect) Frequent RPR (Reference Picture Resampling) can lead to frequent size changes in subsequent processing, including display, potentially complicating control. Furthermore, frequent RPR can also lead to frequent changes in memory access methods (memory map switching) for the current picture and reference picture, potentially complicating control.

[0646] In the first embodiment, an example is shown where resolution changes are allowed only for random access pictures or multiples thereof (i.e., the kth random access picture) in order to reduce the frequency of switching. This simplifies control compared to when there are no constraints.

[0647] (Second aspect) If there are many RPR switching patterns, a corresponding number of matching size patterns for subsequent processing, including display, will be required, complicating the implementation. Furthermore, if there are many RPR switching patterns, a corresponding number of patterns for switching memory access methods (memory map switching) for the current picture and referenced picture will be required, also complicating the implementation.

[0648] In the second aspect, an example is shown where resolution changes (e.g., RPR) are enabled only within a specified set of resolutions in order to reduce the number of switching patterns. This simplifies the implementation compared to when there are no constraints.

[0649] (Third aspect) If a large size that is not actually used is described in the SPS (Small Picture Size) for a GOP (Group of Pictures), the decoding device 200 must allocate memory resources and other resources according to the size of the SPS, thus unnecessarily allocating a large amount of resources.

[0650] In the third aspect, an example is shown of more consistent signaling of SPS and PPS parameters related to picture resolution in order to enable more accurate resource scheduling. For example, the resolution signaled in SPS is not allowed to be greater than the maximum resolution signaled in the picture parameter set.

[0651] The three embodiments described above may be used to more efficiently utilize resources for encoding / decoding picture sequences. These three embodiments may be used separately. However, they (any two or all three) may be combined. Several specific examples of this disclosure address these embodiments, as will be explained in detail below. In particular, some specific examples relate to restricting the bitstream that uses reference picture resampling in order to reduce the processing load on the receiving device (decoder 200). A picture sequence is a collective term for multiple pictures.

[0652] Furthermore, some examples relate to encoding and / or decoding a picture sequence while applying constraints that picture resolution changes are permitted for each k-th (where k is a positive integer) random access point picture, but prohibited for pictures that are not k-th random access point pictures. A picture resolution change is a change from the first resolution of a picture prior to a random access point picture (intercoded picture) to the second resolution of a random access point picture (intracoded picture) and a picture after a random access point picture (intercoded picture).

[0653] [Change picture resolution] Versatile Video Coding (VVC) enables the possibility of changing the picture resolution of a picture sequence in any picture within a sequence that is not necessarily a random access intra-predicted picture. To enable this feature, it allows any picture to have reference pictures with different resolutions, as shown in Figure 97.

[0654] Figure 97 shows reference picture resampling between interpredicted pictures. In particular, Figure 97 shows a portion of picture sequence 300, which contains six pictures. Picture 310 has a first resolution, and picture 320 has a second resolution lower than the first resolution. Both picture 310 and picture 320 are interpredicted (i.e., predicted in a prediction mode that includes time prediction) and arranged in order of time dependency.

[0655] In this example, the arrow from the first picture to the second picture means that the second picture uses the first picture as a reference (and accordingly, the second picture is predicted by the first picture). As can be seen from Figure 97, the resolution change occurs in the first of the three pictures 320 after the three pictures 310.

[0656] Such resolution changes between interpredicted pictures require resampling the reference picture, for example, by using the interpolation filter of motion compensation processing as a resampling filter. VVC provides such a tool. This tool, called RPR (Reference Picture Resampling), was introduced to facilitate picture resolution changes in applications such as video conferencing, where random access pictures are not typically used much, and where there may be various speakers with different video stream resolutions depending on the equipment, for example. This means that the resolution may change frequently depending on who is actively speaking.

[0657] While the VVC RPR concept was developed for video conferencing applications, it can also be used in broadcast or streaming applications that typically use random access pictures periodically (usually every few seconds).

[0658] Some specific broadcast applications employ a so-called open GOP (open Group of Pictures) structure. In this case, multiple pictures that are coded after a random access picture but displayed before it may be predicted using multiple pictures belonging to the preceding GOP. Thus, a GOP is defined as multiple pictures located between two random access pictures in display order. However, a GOP includes the random access picture that is the beginning of the GOP. Another definition of a GOP is that it is multiple pictures located between two random access pictures in coded order.

[0659] Following a random access picture in the display order, all subsequent pictures within the same GOP are predicted using only the random access picture or pictures that come after it. This enables random access at that point.

[0660] Regarding terminology, a random-access intra-predicted picture is a picture that is spatially predicted (intra-predicted), and its decoding, and the decoding of subsequent pictures in the display order, does not depend on the decoding performed earlier for other pictures in the bitstream, thus enabling random access. Typically, a random-access intra-predicted picture does not contain any inter-prediction portion, and entropy coding / decoding restarts at the beginning of the picture.

[0661] Random access intra-prediction pictures are sometimes referred to as IRAP (Intra Random Access Point) pictures. They may also be referred to as random access point pictures, random access pictures, random access points, or IRAP. GOP (Group of Parallels) corresponds to a random access unit.

[0662] The term "display order" refers to the order in which the pictures decoded from the bitstream are displayed for playback. In more general terms, the display order corresponds to the order in which the pictures are input to the encoding device 100 for picture encoding (input order), and the order in which the pictures are output from the decoding device 200 (output order).

[0663] The encoding order is the order in which pictures are encoded, and the decoding order is the order in which pictures are decoded. Basically, the encoding order is equal to the decoding order. On the other hand, the encoding order is often different from the display order.

[0664] [Constraints on the receiving device regarding reference picture resampling (First aspect)] While RPR (Resolution-to-Resolution) is a useful feature, frequent RPR can cause several implementation problems, particularly on the receiving end. To address these issues, several constraints related to resolution changes are introduced below.

[0665] [Changing the resolution at random access points] According to one aspect of the present disclosure, an apparatus (encoding apparatus 100) is provided for encoding a picture sequence into a bitstream. Specifically, the apparatus comprises a processing circuit configured to encode a picture sequence into a bitstream. The processing circuit is also configured to apply constraints (in encoding). Here, picture resolution changes that conform to the constraints are permitted for each k-th (where k is a positive integer) random access point, but prohibited for pictures that do not conform to each k-th random access point.

[0666] The phrase "for each k-th random access point" means, for example, each random access point for the 1st random access point, each random access point for the 2nd random access point, and each random access point for the 3rd random access point.

[0667] In other words, "for each k-th random access point" corresponds to a random access point identified by having k-1 random access points in between, and corresponds to one random access point out of each of the k random access points.

[0668] For example, "for each k-th random access point" corresponds to each random access point when k=1. Similarly, "for each k-th random access point" corresponds to one of the two random access points when k=2, and to one of the three random access points when k=3.

[0669] An initial offset may be provided. This means that the first random access point where a resolution change is permitted is not necessarily the k-th random access point from the beginning of the sequence; in fact, it may actually be a random access point corresponding to an order greater than or less than k.

[0670] Changing picture resolution means changing the resolution of a picture from the resolution of the previous picture. "Previous" refers to the earlier picture in the encoding order and / or input order. A specific example of changing picture resolution is changing from the first resolution of (one or more) pictures that precede the random access point picture in the encoding order and / or input order to the second resolution of the random access point picture and (one or more) pictures that follow the random access point picture in the encoding order and / or input order.

[0671] The above configuration allows reference picture resampling to be enabled even in specific cases, such as broadcast and broadband distribution applications that frequently use random access pictures and open GOP (Group of Pictures) coding structures. A specific constraint is that picture resolution changes are permitted only for random access pictures. This change may be performed between typical broadcast video resolutions, as shown in the three implementation examples in Figures 98A, 98B, and 98C. Figures 98A, 98B, and 98C each illustrate the resolution change from HD to 4K resolution, with each rectangle representing a picture.

[0672] The term "4K resolution" typically refers to a picture resolution of approximately 8 megapixels, corresponding to a vertical or horizontal size of approximately 4000 pixels (e.g., luminance samples). The specific vertical and horizontal sizes of a 4K picture may vary; for example, a 4K picture may have various aspect ratios such as 4096×2160, 3996×2160, 3996×1716, 3840×2160, or 3840×2400. These numbers specify the number of columns × the number of rows, i.e., the horizontal × vertical resolution in pixels (samples).

[0673] The terms “HD resolution” or “Full HD resolution” typically refer to a picture resolution of 1920 × 1080 pixels, which results in approximately 2 megapixels per picture. However, this term may also be applied to other aspect ratios with approximately 2 megapixels. In the examples shown in Figures 98A, 98B, and 98C, HD resolution may be defined as 1920 × 1080 pixels, and 4K resolution may be defined as 3840 × 2160 pixels. However, this disclosure is not limited to any particular resolution and may apply to any resolution change.

[0674] Figures 98A, 98B, and 98C illustrate three scenarios. Each scenario shows pictures arranged in display order (the display order corresponds to the input order to the encoding device 100 and the output order to the decoding device 200). In this example, the pictures are encoded in four temporal sublayers (tId: 0-3) to support temporal scalability. Temporal scalability may be implemented, for example, within a VVC. The term "tId" in Figures 98A, 98B, and 98C represents the temporal ID, i.e., the ID of the temporal sublayer. However, temporal sublayer encoding is shown only as an example. This disclosure is applicable to streams with or without temporal scalability.

[0675] In the implementation example, encoding is performed using the VVC codec, and resolution changes are indicated in the bitstream by either or both of the following (1) and (2).

[0676] (1) pps_pic_width_in_luma_samples and / or pps_pic_height_in_luma_samples (2) One or more of the following: pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset For example, in a VVC-based implementation, both the `sps_ref_pic_resampling_enabled_flag` and `sps_res_change_in_clvs_allowed_flag` syntax elements of the Sequence Parameter Set (SPS) are set to equal 1. This makes it possible to use reference picture resampling within the same VVC layer. The parameter `sps_ref_pic_resampling_enabled_flag` can take one of two values, indicating whether reference picture resampling is enabled or disabled for sequences that reference the SPS, which is the carrier of this parameter.

[0677] A flag value equal to 1 for sps_ref_pic_resampling_enabled_flag indicates that reference picture resampling is enabled. In this case, the current picture referencing the SPS may have a slice that references a reference picture that has one or more parameters from the following seven parameters (1) to (7) that are different from the parameters of the current picture.

[0678] (1)pps_pic_width_in_luma_samples (2)pps_pic_height_in_luma_samples (3)pps_scaling_win_left_offset (4)pps_scaling_win_right_offset (5) pps_scaling_win_top_offset (6)pps_scaling_win_bottom_offset (7) sps_num_subpics_minus1

[0679] A flag value of 0 for sps_ref_pic_resampling_enabled_flag indicates that reference picture resampling is disabled. In this case, the current picture referencing the SPS will not have a slice that references a reference picture having one or more parameters from the seven parameters listed above that are different from the parameters of the current picture.

[0680] The parameter sps_res_change_in_clvs_allowed_flag indicates whether reference picture resampling is enabled or disabled for a specific CLVS (Coded Layer Video Sequence) that references an SPS.

[0681] In particular, if sps_res_change_in_clvs_allowed_flag is equal to 1, the picture spatial resolution may change within a CLVS that references an SPS. This specifies that resolution changes within a single-layer bitstream are enabled. If sps_res_change_in_clvs_allowed_flag is equal to 0, the picture spatial resolution does not change within a CLVS that references an SPS, but it may change between layers. This specifies that spatial scalability between different layers is enabled.

[0682] The resolution change itself is signaled, for example, by ...

Claims

1. Circuits and, The circuit comprises a memory connected to the aforementioned circuit, In operation, the aforementioned circuit Whether to change the resolution of a picture from the resolution of a preceding picture in either the display order or the encoding order is controlled according to a constraint that the change is only permitted if the picture is one or more random access pictures. If the resolution of the reference picture used to encode the interprediction picture differs from the resolution of the interprediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interprediction picture, and the image of the interprediction picture is encoded using the resampled reference image to generate encoded data. The encoded data is transmitted, Under the above constraint, the resolution of the picture is permitted to be changed only if the picture is one of the one or more random access pictures for each k-th random access picture. The aforementioned k is an integer greater than 1. Encoding device.

2. Circuits and, The circuit comprises a memory connected to the aforementioned circuit, In operation, the aforementioned circuit Receive encoded data, Whether to change the resolution of a picture from the resolution of a preceding picture in either the display order or the decoding order is controlled according to a constraint that the change is only permitted if the picture is one or more random access pictures. If the resolution of the reference picture used to decode the interprediction picture differs from the resolution of the interprediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interprediction picture, and the image of the interprediction picture is decoded from the encoded data using the resampled reference image. Under the above constraint, the resolution of the picture is permitted to be changed only if the picture is one of the one or more random access pictures for each k-th random access picture. The aforementioned k is an integer greater than 1. Decoding device.

3. Whether to change the resolution of a picture from the resolution of a preceding picture in either the display order or the encoding order is controlled according to a constraint that the change is only permitted if the picture is one or more random access pictures. If the resolution of the reference picture used to encode the interprediction picture differs from the resolution of the interprediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interprediction picture, and the image of the interprediction picture is encoded using the resampled reference image to generate encoded data. The encoded data is transmitted, Under the above constraint, the resolution of the picture is permitted to be changed only if the picture is one of the one or more random access pictures for each k-th random access picture. The aforementioned k is an integer greater than 1. Encoding method.

4. Receiving encoded data, Whether to change the resolution of a picture from the resolution of a preceding picture in either the display order or the decoding order is controlled according to a constraint that the change is only permitted if the picture is one or more random access pictures. If the resolution of the reference picture used to decode the interprediction picture differs from the resolution of the interprediction picture, the reference image of the reference picture is resampled according to the difference between the resolution of the reference picture and the resolution of the interprediction picture, and the image of the interprediction picture is decoded from the encoded data using the resampled reference image. Under the above constraint, the resolution of the picture is permitted to be changed only if the picture is one of the one or more random access pictures for each k-th random access picture. The aforementioned k is an integer greater than 1. Decryption method.

Images