Coding apparatus, coding method, bitstream transmission apparatus, and storage medium

Abstract

The present disclosure relates to a coding apparatus, a coding method, a bitstream transmission apparatus, and a storage medium. The coding apparatus includes a circuit and a memory connected to the circuit. In operation, the circuit encodes configuration information indicating configurations of a plurality of sub-pictures included in a picture, and encodes each of the plurality of sub-pictures. In operation, the circuit designs each of the plurality of sub-pictures such that at least one pixel included in each sub-picture is included in a same consistent cropping window in the picture. A condition for designing the plurality of sub-pictures includes that a left end of each of the plurality of sub-pictures is located at a position leftward of a right end of the consistent cropping window.

Description

[0001] This application is a divisional application of the invention patent application filed on March 18, 2021, with application number 202180020832.8 and entitled "Encoding / Decoding Apparatus, Encoding / Decoding Method, Transmitting Apparatus, and Storage Medium". Technical Field

[0002] This disclosure relates to video coding and image processing, and particularly to systems, components, and methods for encoding and decoding moving images and image processing. Background Technology

[0003] Video coding technologies have progressed from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). Along with this progress, improvements and optimizations to video coding technologies are constantly needed to handle the ever-increasing volume of digital video data for various applications. This disclosure relates to further advancements, improvements, and optimizations in video coding.

[0004] Furthermore, Non-Patent Document 1 relates to an example of an existing standard related to the aforementioned video coding technology.

[0005] Existing technical documents Non-patent literature Non-patent literature 1: H.265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency VideoCoding) Summary of the Invention

[0006] The problem that the invention aims to solve Regarding the aforementioned encoding methods, new approaches are expected to be proposed to improve encoding efficiency, image quality, reduce processing load, reduce circuit size, or to make appropriate selections of elements or actions such as filters, blocks, dimensions, motion vectors, reference images, or reference blocks.

[0007] This disclosure provides a structure or method capable of contributing to one or more of the following: improved encoding efficiency, improved image quality, reduced processing power, reduced circuit size, improved processing speed, and appropriate selection of elements or actions. Furthermore, this disclosure may include structures or methods that contribute to benefits other than those described above.

[0008] Methods for solving problems For example, one aspect of the encoding apparatus disclosed herein includes a circuit and a memory connected to the circuit. In operation, the circuit designs each of a plurality of sub-images contained in an image, encodes configuration information representing the respective configurations of the plurality of sub-images contained in the image, and encodes each of the plurality of sub-images. In operation, the circuit designs each of the plurality of sub-images such that at least one pixel contained in each sub-image is included in the same consistent clipping window in the image. The conditions for designing the plurality of sub-images include that the left end of each of the plurality of sub-images is located to the left of the right end of the consistent clipping window.

[0009] One aspect of the decoding apparatus disclosed herein includes a circuit and a memory connected to the circuit. During operation, the circuit decodes configuration information representing the configuration of multiple sub-images contained in an image. The multiple sub-images are determined according to conditions of the multiple sub-images. Each of the multiple sub-images is decoded, wherein at least one pixel contained in each of the multiple sub-images is contained within the same consistent clipping window in the image, which serves as a display object area. The conditions include that the left end of each of the multiple sub-images is located to the left of the right end of the consistent clipping window.

[0010] The present disclosure discloses an encoding method in which each of a plurality of sub-images contained in an image is designed, configuration information representing the respective configuration of the plurality of sub-images in the image is encoded, and each of the plurality of sub-images is encoded such that at least one pixel contained in each sub-image is contained in the same consistent clipping window in the image, and the conditions for designing the plurality of sub-images include that the left end of each of the plurality of sub-images is located to the left of the right end of the consistent clipping window.

[0011] The present disclosure discloses a decoding method in which configuration information representing the configuration of multiple sub-images contained in an image is decoded, the multiple sub-images being determined according to conditions of the multiple sub-images, and each of the multiple sub-images is decoded, wherein the condition is that at least one pixel contained in each of the multiple sub-images is contained in the same consistent clipping window in the image, which is a display object area, and the condition includes that the left end of each of the multiple sub-images is located to the left of the right end of the consistent clipping window.

[0012] One aspect of the bitstream transmitting apparatus disclosed herein includes a memory and circuitry connected to the memory. In operation, the circuitry generates a first parameter representing the configuration of a plurality of sub-images contained in an image. In operation, the circuitry transmits a bitstream containing the first parameter, wherein the first parameter indicates that each of the plurality of sub-images in the image is configured such that at least one pixel contained in each sub-image is included in the same consistent clipping window in the image, and the left end of each of the plurality of sub-images is located to the left of the right end of the consistent clipping window.

[0013] One aspect of this disclosure is a non-transitory storage medium for storing a computer program and a bitstream, wherein, when the computer program is executed by a processor, the encoding method of claim 3 is implemented to generate the bitstream, the bitstream including a first parameter, the first parameter representing the respective configuration of a plurality of sub-images contained in an image, the first parameter representing that each of the plurality of sub-images in the image is configured such that at least one pixel contained in each sub-image is contained within the same consistent clipping window in the image, and the left end of each of the plurality of sub-images is located to the left of the right end of the consistent clipping window.

[0014] The structures or methods of each embodiment or part thereof in this disclosure can achieve at least one of the following: improved encoding efficiency, improved image quality, reduced encoding / decoding processing load, reduced circuit size, or improved encoding / decoding processing speed. Alternatively, the structures or methods of each embodiment or part thereof in this disclosure can appropriately select constituent elements / actions such as filters, blocks, sizes, motion vectors, reference images, and reference blocks during encoding and decoding. Furthermore, this disclosure also includes disclosures of structures or methods that can provide benefits beyond those described above. For example, structures or methods that improve encoding efficiency while suppressing an increase in processing load.

[0015] Further advantages and effects of one embodiment of this disclosure are clarified based on the specification and accompanying drawings. These advantages and / or effects are obtained through several embodiments and features described in the specification and accompanying drawings, but it is not necessary to provide all of them in order to obtain one or more advantages and / or effects.

[0016] Furthermore, these general or specific forms can be realized through systems, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0017] Invention Effects The structure or method of this disclosure can contribute to one or more of the following: improved encoding efficiency, improved image quality, reduced processing load, reduced circuit size, improved processing speed, and appropriate selection of elements or actions. Furthermore, the structure or method of this disclosure can also contribute to benefits other than those described above. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating an example of the structure of a transmission system according to an implementation method.

[0019] Figure 2 This is a diagram representing an example of the hierarchical structure of data in a stream.

[0020] Figure 3 This is a diagram illustrating an example of the structure of a slice.

[0021] Figure 4 This is a diagram illustrating an example of the structure of a tile.

[0022] Figure 5 This is a diagram illustrating an example of a scalable coding structure.

[0023] Figure 6 This is a diagram illustrating an example of a scalable coding structure.

[0024] Figure 7 This is a block diagram illustrating an example of the structure of the encoding device in an embodiment.

[0025] Figure 8 This is a block diagram showing an example of the installation of an encoding device.

[0026] Figure 9 This is a flowchart illustrating an example of the overall encoding process performed by the encoding device.

[0027] Figure 10 This is a diagram representing an example of block partitioning.

[0028] Figure 11 This is a diagram illustrating an example of the structure of a segment.

[0029] Figure 12 This is a diagram showing an example of a segmentation style.

[0030] Figure 13A This is a diagram representing an example of a syntax tree for a segmentation pattern.

[0031] Figure 13B This is another example of a syntax tree representing a segmentation pattern.

[0032] Figure 14 It is a table representing the transformation basis functions corresponding to each transformation type.

[0033] Figure 15 This is a diagram representing an example of SVT.

[0034] Figure 16 This is a flowchart illustrating an example of the processing performed by the transformation unit.

[0035] Figure 17 This is a flowchart illustrating another example of the processing performed by the transformation unit.

[0036] Figure 18 This is a block diagram illustrating an example of the structure of the quantization unit.

[0037] Figure 19 This is a flowchart illustrating an example of quantization performed by the quantization department.

[0038] Figure 20 This is a block diagram illustrating an example of the structure of the entropy coding unit.

[0039] Figure 21 This is a diagram showing the flow of CABAC in the entropy coding section.

[0040] Figure 22 This is a block diagram illustrating an example of the structure of a cyclic filter section.

[0041] Figure 23A This is a diagram illustrating an example of the shape of a filter used in an ALF (adaptive loop filter).

[0042] Figure 23B This is another example of the shape of the filter used in ALF.

[0043] Figure 23C This is another example of the shape of the filter used in ALF.

[0044] Figure 23D This is a diagram showing an example of the Y sample (component 1) using CCALF for Cb and CCALF for Cr (multiple components different from component 1).

[0045] Figure 23E This is a diagram representing a diamond-shaped filter.

[0046] Figure 23F This is a diagram representing an example of JC-CCALF.

[0047] Figure 23G This is a graph representing examples of JC-CCALF weight_index candidates.

[0048] Figure 24 This is a block diagram illustrating an example of the detailed structure of the cyclic filtering section that functions as a DBF.

[0049] Figure 25 This is a diagram illustrating an example of deblocking filtering that has filtering characteristics symmetrical with respect to block boundaries.

[0050] Figure 26 This is a diagram illustrating an example of block boundaries used in deblocking filtering.

[0051] Figure 27 This is a graph representing an example of the Bs value.

[0052] Figure 28 This is a flowchart illustrating an example of the processing performed by the prediction unit of the encoding device.

[0053] Figure 29 This is a flowchart illustrating another example of the processing performed by the prediction unit of the encoding device.

[0054] Figure 30 This is a flowchart illustrating another example of the processing performed by the prediction unit of the encoding device.

[0055] Figure 31 This is a diagram representing one example of the 67 intra-prediction modes in intra-frame prediction.

[0056] Figure 32 This is a flowchart illustrating an example of the processing performed by the intra-frame prediction unit.

[0057] Figure 33 This is a diagram representing one example of each reference image.

[0058] Figure 34 This is a concept diagram representing an example of a list of reference images.

[0059] Figure 35 This is a flowchart representing the basic processing flow of inter-frame prediction.

[0060] Figure 36 This is a flowchart representing an example of MV export.

[0061] Figure 37 This is a flowchart representing another example of MV export.

[0062] Figure 38A This is a diagram illustrating an example of the classification of modes exported from MV.

[0063] Figure 38B This is a diagram illustrating an example of the classification of modes exported by MV.

[0064] Figure 39 This is a flowchart illustrating an example of inter-frame prediction based on a normal inter-frame pattern.

[0065] Figure 40This is a flowchart illustrating an example of inter-frame prediction based on a normal merging pattern.

[0066] Figure 41 This is a diagram used to illustrate an example of MV export processing based on the normal merge mode.

[0067] Figure 42 This is a diagram illustrating an example of MV derivation processing based on the HMVP (History-based Motion Vector Prediction / Predictor) model.

[0068] Figure 43 This is a flowchart illustrating an example of FRUC (frame rate up conversion).

[0069] Figure 44 This is a diagram illustrating an example of style matching (bidirectional matching) between two blocks along a motion track.

[0070] Figure 45 This is an example of style matching (template matching) between a template in the current image and a block in a reference image.

[0071] Figure 46A This is a diagram illustrating an example of the MV derivation of a sub-block unit in an affine mode using two control points.

[0072] Figure 46B This is a diagram illustrating an example of MV derivation for a sub-block unit in an affine pattern using 3 control points.

[0073] Figure 47A This is a conceptual diagram used to illustrate an example of MV derivation of control points in affine mode.

[0074] Figure 47B This is a conceptual diagram used to illustrate an example of MV derivation of control points in affine mode.

[0075] Figure 47C This is a conceptual diagram used to illustrate an example of MV derivation of control points in affine mode.

[0076] Figure 48A This is a diagram used to illustrate an affine pattern with two control points.

[0077] Figure 48B This is a diagram used to illustrate an affine pattern with three control points.

[0078] Figure 49AThis is a conceptual diagram illustrating an example of a method for deriving the MV of control points when the number of control points differs between an encoded block and the current block.

[0079] Figure 49B This is a conceptual diagram illustrating another example of the MV derivation method for control points when the number of control points differs between the encoded block and the current block.

[0080] Figure 50 This is a flowchart illustrating an example of affine merge mode processing.

[0081] Figure 51 This is a flowchart illustrating an example of affine inter-frame mode processing.

[0082] Figure 52A This is a diagram used to illustrate the generation of predicted images for two triangles.

[0083] Figure 52B This is a conceptual diagram representing the first part of the first partition and examples of the first and second sample sets.

[0084] Figure 52C This is a conceptual diagram representing the first part of the first partition.

[0085] Figure 53 This is a flowchart representing an example of a triangular pattern.

[0086] Figure 54 This is a diagram illustrating an example of the ATMVP (Advanced Temporal Motion Vector Prediction / Predictor) mode, which derives MV at the sub-block level.

[0087] Figure 55 This is a diagram showing the relationship between the merging mode and DMVR (dynamic motion vector refreshing).

[0088] Figure 56 This is a conceptual diagram used to illustrate an example of DMVR.

[0089] Figure 57 This is a conceptual diagram used to illustrate another example of DMVR used to determine MV.

[0090] Figure 58A This is a diagram illustrating an example of motion search in DMVR.

[0091] Figure 58B This is a flowchart illustrating an example of motion search in DMVR.

[0092] Figure 59 This is a flowchart illustrating an example of generating a predicted image.

[0093] Figure 60 This is a flowchart illustrating another example of the generation of a predicted image.

[0094] Figure 61 This is a flowchart illustrating an example of predictive image correction processing based on OBMC (overlapped block motion compensation).

[0095] Figure 62 This is a conceptual diagram used to illustrate an example of OBMC-based predictive image correction processing.

[0096] Figure 63 It is a diagram used to illustrate a model that assumes constant linear motion.

[0097] Figure 64 This is a flowchart illustrating an example of inter-frame prediction according to BIO.

[0098] Figure 65 This is a diagram illustrating an example of the structure of an inter-frame prediction unit that performs inter-frame prediction according to BIO.

[0099] Figure 66A This is a diagram illustrating an example of a predictive image generation method that uses LIC (local illumination compensation)-based brightness correction processing.

[0100] Figure 66B This is a flowchart illustrating an example of a predictive image generation method that uses LIC-based brightness correction processing.

[0101] Figure 67 This is a block diagram illustrating the structure of the decoding device in the implementation method.

[0102] Figure 68 This is a block diagram illustrating an example of the installation of a decoding device.

[0103] Figure 69 This is a flowchart illustrating an example of the overall decoding process performed by the decoding device.

[0104] Figure 70 It is a diagram that shows the relationship between the dividing part and other constituent elements.

[0105] Figure 71 This is a block diagram illustrating an example of the structure of the entropy decoding unit.

[0106] Figure 72 This is a diagram illustrating the CABAC flow in the entropy decoding section.

[0107] Figure 73 This is a block diagram illustrating an example of the structure of the inverse quantization unit.

[0108] Figure 74 This is a flowchart illustrating an example of inverse quantization performed by the inverse quantization unit.

[0109] Figure 75 This is a flowchart illustrating an example of the processing performed by the inverse transformation unit.

[0110] Figure 76 This is a flowchart illustrating another example of the processing performed by the inverse transformation unit.

[0111] Figure 77 This is a block diagram illustrating an example of the structure of a cyclic filter section.

[0112] Figure 78 This is a flowchart illustrating an example of the processing performed by the prediction unit of the decoding device.

[0113] Figure 79 This is a flowchart illustrating another example of the processing performed by the prediction unit of the decoding device.

[0114] Figure 80A This is a flowchart illustrating another example of the processing performed by the prediction unit of the decoding device.

[0115] Figure 80B This is a flowchart representing the remainder of another example of the processing performed by the prediction unit of the decoding device.

[0116] Figure 81 This diagram illustrates an example of the processing performed by the intra-frame prediction unit of the decoding device.

[0117] Figure 82 This is a flowchart representing an example of MV export in a decoding device.

[0118] Figure 83 This is a flowchart representing another example of MV export in a decoding device.

[0119] Figure 84 This is a flowchart illustrating an example of inter-frame prediction based on a normal inter-frame mode in a decoding device.

[0120] Figure 85 This is a flowchart illustrating an example of inter-frame prediction based on a normal merging mode in a decoding device.

[0121] Figure 86 This is a flowchart illustrating an example of inter-frame prediction based on FRUC mode in a decoding device.

[0122] Figure 87This is a flowchart illustrating an example of inter-frame prediction based on affine merging mode in a decoding device.

[0123] Figure 88 This is a flowchart illustrating an example of inter-frame prediction based on affine inter-frame patterns in a decoding device.

[0124] Figure 89 This is a flowchart illustrating an example of inter-frame prediction based on triangular patterns in a decoding device.

[0125] Figure 90 This is a flowchart illustrating an example of motion search based on DMVR in a decoding device.

[0126] Figure 91 This is a flowchart illustrating a detailed example of motion search based on DMVR in a decoding device.

[0127] Figure 92 This is a flowchart illustrating an example of the generation of a predicted image in a decoding device.

[0128] Figure 93 This is a flowchart illustrating another example of the generation of a predicted image in a decoding device.

[0129] Figure 94 This is a flowchart illustrating an example of OBMC-based correction of a predicted image in a decoding device.

[0130] Figure 95 This is a flowchart illustrating an example of BIO-based correction of a predicted image in a decoding device.

[0131] Figure 96 This is a flowchart illustrating an example of LIC-based correction of a predicted image in a decoding device.

[0132] Figure 97 It is a block diagram representing the structure of an image processing device.

[0133] Figure 98 It is a graph representing the syntax associated with the sub-picture.

[0134] Figure 99 This is a diagram illustrating an example of sub-image boundaries and consistent clipping windows within an image.

[0135] Figure 100 This is another example of a diagram showing the boundaries of sub-images and consistent clipping windows within an image.

[0136] Figure 101 This is a flowchart illustrating the process of rewriting the offset of the consistent cropping window within SPS and PPS during the extraction and processing of sub-images.

[0137] Figure 102This is a flowchart illustrating the conditional judgment process in the sub-image extraction process, specifically the process of rewriting the offset of the consistent cropping window within SPS and PPS.

[0138] Figure 103 This is a flowchart illustrating an example of the operation of the encoding device in the implementation method.

[0139] Figure 104 This is a flowchart illustrating an example of the operation of the decoding device in the implementation method.

[0140] Figure 105 This is a flowchart illustrating an example of the operation of the image processing apparatus in the embodiment.

[0141] Figure 106 This is a block diagram illustrating the bit stream transmission device in the implementation method.

[0142] Figure 107 This is a block diagram illustrating the bitstream storage device in the implementation method.

[0143] Figure 108 This is a diagram showing the overall structure of a content supply system that enables content distribution services.

[0144] Figure 109 This is an example of a web page display.

[0145] Figure 110 This is an example of a web page display.

[0146] Figure 111 This is a diagram illustrating an example of a smartphone.

[0147] Figure 112 This is a block diagram representing a structural example of a smartphone. Detailed Implementation

[0148] [Introduction] One aspect of the encoding apparatus disclosed herein includes a circuit and a memory connected to the circuit. During operation, the circuit designs each of a plurality of sub-images constituting an image such that at least one pixel contained in the sub-image is included in a consistent clipping window, encodes configuration information representing the respective configurations of the plurality of sub-images, and encodes each of the plurality of sub-images.

[0149] Therefore, one aspect of the encoding apparatus disclosed herein is capable of encoding each sub-image constituting an image as a valid sub-image including at least a portion of a consistent cropping window.

[0150] Furthermore, for example, in one embodiment of the encoding apparatus of this disclosure, the condition may include: for each of the plurality of sub-images, the left end of the sub-image is located to the left of the right end of the consistency clipping window.

[0151] Therefore, one aspect of the encoding device disclosed herein is capable of encoding a sub-image configured to not deviate from the right side of a consistent cropping window.

[0152] Furthermore, for example, in one embodiment of the encoding apparatus disclosed herein, the condition may include: for each of the plurality of sub-images, the right end of the sub-image is located to the right of the left end of the consistency clipping window.

[0153] Therefore, one aspect of the encoding device disclosed herein is capable of encoding a sub-image configured to not deviate from the left side of a consistent cropping window.

[0154] Furthermore, for example, in one embodiment of the encoding apparatus disclosed herein, the condition may include: for each of the plurality of sub-images, the upper end of the sub-image is located above the lower end of the uniform clipping window.

[0155] Therefore, one aspect of the encoding device disclosed herein is capable of encoding sub-images configured to not deviate from the lower side of a consistent clipping window.

[0156] Furthermore, for example, in one embodiment of the encoding apparatus disclosed herein, the condition may include: for each of the plurality of sub-images, the lower end of the sub-image is located below the upper end of the consistent clipping window.

[0157] Therefore, one aspect of the encoding device disclosed herein is capable of encoding sub-images configured to not deviate from the upper side of a consistent clipping window.

[0158] Furthermore, one aspect of the decoding apparatus disclosed herein includes a circuit and a memory connected to the circuit. During operation, the circuit decodes configuration information representing the configuration of each of the plurality of sub-images constituting an image, determined according to conditions. The circuit decodes each of the plurality of sub-images, wherein the condition is that for each of the plurality of sub-images, at least one pixel contained in the sub-image is included in a consistent clipping window that is a display object area.

[0159] Therefore, one aspect of the decoding apparatus disclosed herein is capable of decoding each sub-image constituting an image as a valid sub-image including at least a portion of a consistent cropping window.

[0160] Furthermore, for example, in one embodiment of the decoding apparatus disclosed herein, the condition may include: for each of the plurality of sub-images, the left end of the sub-image is located to the left of the right end of the consistent clipping window.

[0161] Therefore, one aspect of the decoding apparatus disclosed herein is capable of decoding a sub-image configured to not deviate from the right side of a consistent cropping window.

[0162] Furthermore, for example, in one embodiment of the decoding apparatus disclosed herein, the condition may include: for each of the plurality of sub-images, the right end of the sub-image is located to the right of the left end of the consistent clipping window.

[0163] Therefore, one aspect of the decoding apparatus disclosed herein is capable of decoding a sub-image configured to not deviate from the left side of a consistent cropping window.

[0164] Furthermore, for example, in one embodiment of the decoding apparatus disclosed herein, the condition may include: for each of the plurality of sub-images, the upper end of the sub-image is located above the lower end of the consistent clipping window.

[0165] Therefore, one aspect of the decoding apparatus disclosed herein is capable of decoding sub-images configured to not deviate from the lower side of a consistent cropping window.

[0166] Furthermore, for example, in one embodiment of the decoding apparatus disclosed herein, the condition may include: for each of the plurality of sub-images, the lower end of the sub-image is located below the upper end of the consistent clipping window.

[0167] Therefore, one aspect of the decoding apparatus disclosed herein is capable of decoding sub-images configured to not deviate from the upper side of a consistent cropping window.

[0168] Furthermore, one aspect of the image processing apparatus disclosed herein includes a circuit and a memory connected to the circuit. During operation, the circuit derives a new consistent clipping window as a display object area for a sub-bitstream of a sub-image constituting an image, extracts a sub-bitstream from the image's bitstream, and during the deriving of the new consistent clipping window, based on the configuration of the sub-images in the image, switches whether the offset value for the new consistent clipping window for the sub-bitstream applies the original offset value for the consistent clipping window for the bitstream, or applies zero.

[0169] Therefore, one aspect of the image processing apparatus of this disclosure can switch whether to apply the original offset value of the consistency clipping window to a new consistency clipping window based on the configuration of the sub-images in the image. Thus, one aspect of the image processing apparatus of this disclosure can derive a new consistency clipping window based on the configuration of the sub-images in the image and the original offset value.

[0170] Alternatively, for example, in one embodiment of the image processing apparatus of this disclosure, the circuit may describe information representing the new consistent clipping window in the parameter set of the sub-bit stream.

[0171] Thus, one aspect of the image processing apparatus disclosed herein is capable of generating sub-bitstreams that apply a new consistent clipping window.

[0172] Alternatively, for example, in one embodiment of the image processing apparatus disclosed herein, the parameter set may be a sequence parameter set of the sub-bit stream.

[0173] Thus, one aspect of the image processing apparatus disclosed herein is capable of generating a sub-bitstream having a sequence parameter set with a new uniform clipping window applied.

[0174] Alternatively, for example, in one embodiment of the image processing apparatus of this disclosure, when the sub-image is located at the left end of the image, the circuit applies the original offset value of the left side of the consistent clipping window for the bitstream to the offset value of the left side of the new consistent clipping window for the sub-bitstream.

[0175] Therefore, one aspect of the image processing apparatus disclosed herein can align the left end of the consistency clipping window for the sub-bitstream of a sub-image located at the left end of the image with the left end of the original consistency clipping window.

[0176] Alternatively, for example, in one embodiment of the image processing apparatus of this disclosure, the circuit may apply zero to the offset value on the left side of the new consistent cropping window for the sub-bitstream when the sub-image is not located at the left end of the image.

[0177] Therefore, one aspect of the image processing apparatus disclosed herein is capable of aligning the left end of the consistent cropping window for a sub-bitstream of a sub-image that is not located at the left end of the image with the left end of the sub-image.

[0178] Alternatively, for example, in one embodiment of the image processing apparatus of this disclosure, when the sub-image is located at the right end of the image, the circuit applies the original offset value to the right side of the new consistent cropping window for the sub-bitstream to the offset value to the right side of the consistent cropping window for the bitstream.

[0179] Therefore, one aspect of the image processing apparatus disclosed herein is capable of aligning the right end of the uniform clipping window for the sub-bitstream of a sub-image located at the right end of the image with the right end of the original uniform clipping window.

[0180] Alternatively, for example, in one embodiment of the image processing apparatus of this disclosure, the circuit may apply zero to the offset value on the right side of the new consistent cropping window for the sub-bitstream when the sub-image is not located at the right end of the image.

[0181] Therefore, one aspect of the image processing apparatus disclosed herein is capable of aligning the right end of the uniform cropping window for a sub-bitstream of a sub-image that is not located at the right end of the image with the right end of the sub-image.

[0182] Alternatively, for example, in one embodiment of the image processing apparatus of this disclosure, when the sub-image is located at the top of the image, the circuit applies the original offset value of the upper side of the consistent clipping window for the bitstream to the offset value of the upper side of the new consistent clipping window for the sub-bitstream.

[0183] Therefore, one aspect of the image processing apparatus disclosed herein can align the upper edge of the uniform clipping window for the sub-bitstream of the sub-image located at the upper edge of the image with the upper edge of the original uniform clipping window.

[0184] Alternatively, for example, in one embodiment of the image processing apparatus of this disclosure, the circuit may apply zero to the offset value above the new consistent cropping window for the sub-bitstream when the sub-image is not located at the top of the image.

[0185] Therefore, one aspect of the image processing apparatus disclosed herein is capable of aligning the top edge of the uniform cropping window for a sub-bitstream of a sub-image that is not located at the top edge of the image with the top edge of the sub-image.

[0186] Alternatively, for example, in one embodiment of the image processing apparatus of this disclosure, when the sub-image is located at the lower end of the image, the circuit applies the original offset value of the lower side of the consistent clipping window for the bitstream to the offset value of the lower side of the new consistent clipping window for the sub-bitstream.

[0187] Therefore, one aspect of the image processing apparatus disclosed herein is capable of aligning the lower end of the uniform clipping window for the sub-bitstream of a sub-image located at the lower end of the image with the lower end of the original uniform clipping window.

[0188] Alternatively, for example, in one embodiment of the image processing apparatus of this disclosure, the circuit may apply zero to the offset value at the bottom of the new consistent cropping window for the sub-bitstream when the sub-image is not located at the bottom of the image.

[0189] Therefore, one aspect of the image processing apparatus disclosed herein is capable of aligning the lower end of the uniform clipping window for a sub-bitstream of a sub-image that is not located at the lower end of the image with the lower end of the sub-image.

[0190] Furthermore, one form of the encoding method disclosed herein encodes configuration information representing the configuration of each of the plurality of sub-images constituting an image, determined according to conditions, and encodes each of the plurality of sub-images, wherein the condition is: for each of the plurality of sub-images, at least one pixel contained in the sub-image is contained in a consistent clipping window that is a display object area.

[0191] Therefore, the encoding method of this disclosure can achieve the same effect as the encoding device described above.

[0192] Furthermore, one form of the decoding method disclosed herein decodes configuration information representing the configuration of each of the plurality of sub-images constituting an image, determined according to conditions, and decodes each of the plurality of sub-images, wherein the condition is: for each of the plurality of sub-images, at least one pixel contained in the sub-image is contained in a consistent clipping window that serves as a display object area.

[0193] Therefore, the decoding method of this disclosure can achieve the same effect as the decoding device described above.

[0194] Furthermore, in one aspect of the image processing method disclosed herein, for a sub-bitstream of a sub-image constituting an image, a new consistent clipping window is exported as a display object region. The sub-bitstream is extracted from the bitstream of the image, and a new consistent clipping window is exported as a display object region for the sub-bitstream. In the export of the new consistent clipping window, based on the configuration of the sub-image in the image, the offset value of the new consistent clipping window for the sub-bitstream is switched, either applying the original offset value of the consistent clipping window for the bitstream or applying zero.

[0195] Therefore, the image processing method of this disclosure can achieve the same effect as the image processing apparatus described above.

[0196] Additionally, one embodiment of the bitstream transmitting apparatus disclosed herein includes a memory and circuitry connected to the memory. During operation, the circuitry generates a first parameter representing the configuration of each of a plurality of sub-images constituting an image, and transmits a bitstream containing the first parameter. The first parameter indicates that each of the plurality of sub-images constituting the image is configured such that at least one pixel contained in that sub-image is included in a consistent clipping window.

[0197] Therefore, one embodiment of the bitstream transmitting apparatus disclosed herein is capable of transmitting a bitstream in which each sub-image constituting an image is a valid sub-image containing at least a portion of a consistent clipping window.

[0198] Furthermore, one aspect of the bitstream storage device disclosed herein includes a storage device for storing the bitstream and circuitry connected to the storage device, the circuitry deriving a first parameter from the bitstream, the first parameter indicating that each of a plurality of sub-images constituting an image is configured such that at least one pixel contained in the sub-image is included in a consistent clipping window.

[0199] Therefore, one form of bitstream storage device disclosed herein is capable of storing bitstreams in which each sub-image constituting an image is a valid sub-image containing at least a portion of a consistent clipping window.

[0200] One aspect of the non-transitory storage medium disclosed herein is a non-transitory storage medium for storing a bitstream, the bitstream including a first parameter, the first parameter indicating the respective configuration of a plurality of sub-images constituting an image, the first parameter indicating that each of the plurality of sub-images constituting the image is configured such that at least one pixel contained in the sub-image is included in a consistent clipping window.

[0201] Thus, a non-transitory storage medium can store a bit stream in which each sub-image constituting an image is a valid sub-image containing at least a portion of a consistent clipping window.

[0202] Furthermore, for example, one embodiment of the coding apparatus disclosed herein includes an input unit, a segmentation unit, an intra-frame prediction unit, an inter-frame prediction unit, a cyclic filtering unit, a transform unit, a quantization unit, an entropy coding unit, and an output unit.

[0203] The input unit receives the current image. The segmentation unit divides the current image into multiple blocks.

[0204] The intra-frame prediction unit uses a reference image contained in the current image to generate a prediction signal for the current block contained in the current image. The inter-frame prediction unit uses a reference image contained in a different reference image to generate a prediction signal for the current block contained in the current image. The cyclic filtering unit applies a filter to the reconstructed block of the current block contained in the current image.

[0205] The transform unit transforms the prediction error between the original signal of the current block contained in the current image and the prediction signal generated by the intra-frame prediction unit or the inter-frame prediction unit to generate transform coefficients. The quantization unit quantizes the transform coefficients to generate quantization coefficients. The entropy coding unit generates a coded bitstream by applying variable-length coding to the quantization coefficients. Then, the output unit outputs the coded bitstream containing the quantization coefficients with applied variable-length coding and control information.

[0206] Additionally, for example, the entropy encoding unit, during operation, encodes configuration information representing the configurations of the plurality of sub-images constituting the image, determined according to conditions, and encodes each of the plurality of sub-images. The condition is that for each of the plurality of sub-images, at least one pixel contained in that sub-image is included in a consistent clipping window that serves as the display object area.

[0207] Furthermore, for example, one aspect of the decoding apparatus disclosed herein includes an input unit, an entropy decoding unit, an inverse quantization unit, an inverse transform unit, an intra-frame prediction unit, an inter-frame prediction unit, a cyclic filtering unit, and an output unit.

[0208] The input unit receives an encoded bitstream. The entropy decoding unit applies variable-length decoding to the encoded bitstream to derive quantization coefficients. The inverse quantization unit performs inverse quantization on the quantization coefficients to derive transform coefficients. The inverse transform unit performs inverse transform on the transform coefficients to derive the prediction error.

[0209] The intra-frame prediction unit uses a reference image contained in the current image to generate a prediction signal for the current block contained in the current image. The inter-frame prediction unit uses a reference image contained in a reference image different from the current image to generate a prediction signal for the current block contained in the current image.

[0210] The cyclic filtering unit applies a filter to the reconstructed block of the current block contained in the current image. Then, the current image is output from the output unit.

[0211] Additionally, for example, the entropy decoding unit decodes configuration information representing the configurations of multiple sub-images constituting an image, determined according to conditions, and decodes each of the multiple sub-images. The condition is that for each of the multiple sub-images, at least one pixel contained in that sub-image is included in a consistent clipping window that serves as the display object area.

[0212] Moreover, these inclusive or specific forms can be realized by non-temporary recording media such as systems, devices, methods, integrated circuits, computer programs, or computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0213] [Definition of the term] As an example, the terms can be defined as follows.

[0214] (1) Image It is a unit of data consisting of a collection of pixels, which consists of images or blocks smaller than images, including still images in addition to moving images.

[0215] (2) Pictures A frame is a unit of image processing consisting of a collection of pixels, sometimes referred to as a frame or field.

[0216] (3) blocks A processing unit is a collection of a specific number of pixels, as illustrated in the examples below, and the name is not limited. Furthermore, there are no restrictions on shape; for example, it includes rectangles composed of M×N pixels, squares composed of M×M pixels, as well as triangles, circles, and other shapes.

[0217] (Example of a block) • Slices / tiles / bricks • CTU / Superblock / Basic Unit of Division •VPDU / Hardware processing unit • CU / Processing Block Unit / Prediction Block Unit (PU) / Orthogonal Transformation Block Unit (TU) / Unit ·Sub-block (4) Pixels / Sample A point is the smallest unit that makes up an image. It includes not only pixels at integer positions but also pixels at fractional positions generated based on pixels at integer positions.

[0218] (5) Pixel value / Sample value These are the inherent values ​​of a pixel, including brightness, color difference, RGB grayscale, depth, or binary values ​​of 0 and 1.

[0219] (6) Sign Besides 1 bit, it also includes cases with multiple bits, such as parameters or indices of 2 bits or more. Furthermore, it can be not only a binary number but also a multi-valued number using other bases.

[0220] (7) Signal In order to transmit information, symbols and codes are used, including not only discretized digital signals but also analog signals that take continuous values.

[0221] (8) Stream / Bitstream This refers to a data string or stream of digital data. A stream / bit stream can be a single, hierarchical structure composed of multiple streams. Furthermore, besides transmission via serial communication over a single path, it also includes transmission via data packet communication over multiple paths.

[0222] (9) Difference / Difference In the case of scalars, any operation involving difference is acceptable, except for the simple difference (xy), including the absolute value of the difference (|xy|), the difference of squares (x^2-y^2), the square root of the difference (√(xy)), the weighted difference (ax-by: a and b are constants), and the offset difference (x-y+a: a is the offset).

[0223] (10) and In the case of scalars, any operation involving summation is acceptable, except for the simple sum (x+y), including the absolute value of the sum (|x+y|), the sum of squares (x^2+y^2), the square root of the sum (√(x+y)), the weighted sum (ax+by: a and b are constants), and the offset sum (x+y+a: a is the offset).

[0224] (11) based on This also includes cases where elements other than those of the object on which the result is based are included. In addition, besides cases where a direct result is obtained, cases where a result is obtained through intermediate results are also included.

[0225] (12) Use (used, using) This also includes cases where elements other than those used as the object are included. In addition, besides cases where a direct result is obtained, cases where a result is obtained through intermediate results are also included.

[0226] (13) Prohibit (prohibit, forbid) It can also be described as not being allowed. In addition, not prohibiting or allowing does not necessarily mean an obligation.

[0227] (14) Limit (limit, restriction / restrict / restricted) It can also be described as not allowing. Furthermore, not prohibiting or allowing does not necessarily imply an obligation. And it can be prohibited only partially in terms of quantity or quality, or it can be a complete prohibition.

[0228] (15) Color difference (chroma) It is an adjective, represented by the symbols Cb and Cr, that specifies one of the two color difference signals associated with the primary color, either in a sample arrangement or as a single sample representation. Furthermore, the term chrominance can be used instead of chroma.

[0229] (16) Luminance Luma is an adjective, indicated by the symbol or subscript Y or L, that specifies a monochrome signal associated with a primary color, whether representing an arrangement or a single sample. The term luminance can also be used instead of luma.

[0230] [Explanations related to the records] In the accompanying drawings, the same reference numerals denote the same or similar constituent elements. Furthermore, the dimensions and relative positions of the constituent elements in the drawings are not necessarily depicted to a fixed scale.

[0231] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below are inclusive or specific examples. The numerical values, shapes, materials, constituent elements, the arrangement and connection of constituent elements, steps, relationships and sequences of steps, etc., shown in the following embodiments are examples and are not intended to limit the scope of the claims.

[0232] The following describes embodiments of the encoding and decoding apparatus. These embodiments are examples of encoding and decoding apparatuses to which the processing and / or structures described in the various aspects of this disclosure can be applied. The processing and / or structures can also be implemented in encoding and decoding apparatuses different from the embodiments. For example, regarding the processing and / or structures applied to the embodiments, one of the following may also be performed.

[0233] (1) One of the multiple components of the encoding or decoding apparatus in the embodiments described in the various aspects of this disclosure may be replaced with other components described in the various aspects of this disclosure, or they may be combined. (2) In the encoding or decoding apparatus of the embodiment, the functions or processes performed by a portion of the multiple components of the encoding or decoding apparatus may be arbitrarily changed, such as by adding, replacing, or deleting functions or processes. For example, any function or process may be replaced with another function or process described in the various forms of this disclosure, or they may be combined; (3) In the method implemented by the encoding or decoding apparatus of the embodiment, any changes may be made to a portion of the multiple processes included in the method, such as adding, replacing, or deleting. For example, any process in the method may be replaced with another process described in the various aspects of this disclosure, or they may be combined; (4) Some of the constituent elements of the encoding or decoding apparatus of the embodiment may be combined with a constituent element described in each aspect of the present disclosure, or may be combined with a constituent element having a function described in each aspect of the present disclosure, or may be combined with a constituent element that performs a process performed by the constituent elements described in each aspect of the present disclosure. (5) A component having a part of the function of the encoding or decoding device of the embodiment, or a component having a part of the processing of the encoding or decoding device of the embodiment, may be combined or replaced with a component described in each aspect of the present disclosure, a component having a part of the function described in each aspect of the present disclosure, or a component having a part of the processing described in each aspect of the present disclosure. (6) In the method implemented by the encoding or decoding apparatus of the embodiment, one of the multiple processes included in the method is replaced by a process described in each aspect of this disclosure or the same process, or a combination thereof; (7) A portion of the processing included in the method implemented by the encoding or decoding apparatus of the embodiment may also be combined with the processing described in any of the various forms of this disclosure.

[0234] (8) The implementation of the processing and / or structure described in the various embodiments of this disclosure is not limited to the encoding or decoding apparatus of the embodiments. For example, the processing and / or structure may also be implemented in an apparatus used for a different purpose than the motion picture encoding or motion picture decoding disclosed in the embodiments.

[0235] [System Architecture] Figure 1 This is a schematic diagram illustrating an example of the structure of the transmission system according to this embodiment.

[0236] A transport system (Trs) is a system that transmits a stream generated by encoding an image and decodes the transmitted stream. Examples of such transport systems (Trs) include... Figure 1 As shown, it includes an encoding device 100, a network Nw, and a decoding device 200.

[0237] An image is input to the encoding device 100. The encoding device 100 generates a stream by encoding the input image and outputs the stream to the network Nw. The stream contains, for example, the encoded image and control information for decoding the encoded image. The image is compressed through this encoding.

[0238] Furthermore, the original image input to the encoding device 100 before encoding is also referred to as the original image, original signal, or original sample. Additionally, the image can be a moving image or a still image. Furthermore, an image is a broader concept than sequences, pictures, and blocks, and is not limited by spatial or temporal regions unless otherwise specified. Furthermore, an image is composed of an arrangement of pixels or pixel values, and the signal or pixel values ​​representing that image are also called samples. Additionally, the stream can be referred to as a bitstream, a coded bitstream, a compressed bitstream, or a coded signal. Moreover, the encoding device can also be called an image encoding device or a moving image encoding device, and the encoding method of the encoding device 100 can also be called an encoding method, an image encoding method, or a moving image encoding method.

[0239] Network Nw transmits the stream generated by encoding device 100 to decoding device 200. Network Nw can be the Internet, Wide Area Network (WAN), Local Area Network (LAN), or a combination thereof. Network Nw is not necessarily limited to a two-way communication network; it can also be a one-way communication network that transmits broadcast waves, such as terrestrial digital broadcasting or satellite broadcasting. Alternatively, network Nw can be replaced by a storage medium that records streams such as DVD (Digital Versatile Disc) or Blu-ray Disc (registered trademark).

[0240] The decoding device 200 decodes the stream transmitted by the network Nw to generate a decoded image, such as an uncompressed image. For example, the decoding device decodes the stream according to a decoding method corresponding to the encoding method of the encoding device 100.

[0241] In addition, the decoding device can also be called an image decoding device or a moving image decoding device, and the decoding method of the decoding device 200 can also be called a decoding method, an image decoding method, or a moving image decoding method.

[0242] [Data Structures] Figure 2 This is a diagram representing an example of the hierarchical structure of data in a stream. A stream, for example, includes video sequences. The video sequence is, for example, as shown below. Figure 2As shown in (a), it includes VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), SEI (Supplemental Enhancement Information), and multiple pictures.

[0243] VPS contains common encoding parameters for multiple layers in a moving image composed of multiple layers, and encoding parameters associated with multiple layers or individual layers contained in the moving image.

[0244] The SPS contains parameters used for the sequence, that is, encoding parameters that the decoding device 200 refers to in order to decode the sequence. For example, the encoding parameters may also represent the width or height of the image. Furthermore, multiple SPSs may exist.

[0245] The PPS contains parameters used for the images; that is, encoding parameters referenced by the decoding device 200 for decoding each image in the sequence. For example, these encoding parameters may also include a reference value for the quantization width used in image decoding and a flag indicating the application of weighted prediction. Furthermore, multiple PPSs may exist. Additionally, SPS and PPS are sometimes simply referred to as parameter sets.

[0246] like Figure 2 As shown in (b), the image may include an image header and one or more slices. The image header includes encoding parameters referenced by the decoding device 200 for decoding the one or more slices.

[0247] like Figure 2 As shown in (c), the slice includes a slice header and one or more bricks. The slice header includes encoding parameters referenced by the decoding device 200 for decoding the one or more bricks.

[0248] like Figure 2 As shown in (d), the brick contains more than one CTU (Coding Tree Unit).

[0249] Alternatively, the image may not contain a slice, but rather a set of tiles instead. In this case, the set of tiles contains more than one tile. Alternatively, a slice may be included within a brick.

[0250] CTU is also known as a superblock or basic partitioning unit. For example... Figure 2As shown in (e), such a CTU includes a CTU header and one or more CUs (Coding Units). The CTU header includes encoding parameters referenced by the decoding device 200 for decoding one or more CUs.

[0251] A CU can also be divided into multiple smaller CUs. Furthermore, as... Figure 2 As shown in (f), the CU includes a CU header, prediction information, and residual coefficient information. The prediction information is used to predict the CU, and the residual coefficient information represents the prediction residuals, which will be discussed later. Furthermore, the CU is essentially the same as the PU (Prediction Unit) and TU (Transform Unit), but, for example, in the SBT discussed later, it may also contain multiple TUs smaller than the CU. Additionally, the CU can process each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. A VPDU is, for example, a fixed unit that can be processed in one stage during pipeline processing in hardware.

[0252] Furthermore, a flow may not have Figure 2 The hierarchy shown refers to any part of the hierarchy. Furthermore, the order of these hierarchy levels can be interchanged, and any hierarchy can be replaced with another hierarchy. Additionally, the image of the object being processed by the encoding device 100 or decoding device 200 at the current time point is called the current image. If the processing is encoding, the current image is synonymous with the encoded object image; if the processing is decoding, the current image is synonymous with the decoded object image. Furthermore, a block of an object, such as a CU or CU block, being processed by the encoding device 100 or decoding device 200 at the current time point is called the current block. If the processing is encoding, the current block is synonymous with the encoded object block; if the processing is decoding, the current block is synonymous with the decoded object block.

[0253] [Structural slices / tiles of the image] To decode images in parallel, images are sometimes composed of slice units or tile units.

[0254] A slice is the basic encoding unit that makes up an image. An image is composed of one or more slices. In addition, a slice is composed of one or more consecutive CTUs.

[0255] Figure 3This is a diagram illustrating an example of a slice's structure. For instance, an image contains 11×8 CTUs and is divided into four slices (slices 1-4). Slice 1, for example, consists of 16 CTUs, slice 2 consists of 21 CTUs, slice 3 consists of 29 CTUs, and slice 4 consists of 22 CTUs. Here, each CTU within the image belongs to a specific slice. The shape of the slice is the shape that divides the image horizontally. The boundaries of the slices do not need to be at the edge of the image; they can be anywhere within the boundaries of the CTUs in the image. The processing order (encoding or decoding order) of the CTUs in a slice is, for example, a raster scan order. Additionally, a slice contains a slice header and encoded data. The slice header may also describe the characteristics of the slice, such as the CTU addresses at the beginning of the slice and the slice type.

[0256] A tile is a unit that makes up a rectangular area of ​​an image. Tiles can also be assigned a number called TileId according to the raster scan sequence.

[0257] Figure 4 This is an example diagram illustrating the structure of tiles. For instance, an image contains 11×8 CTUs and is divided into four rectangular tile regions (tiles 1-4). When using tiles, the processing order of the CTUs is changed compared to when tiles are not used. Without tiles, multiple CTUs within the image are processed, for example, in raster scan order. With tiles, in each of the multiple tiles, at least one CTU is processed, for example, in raster scan order. For example, as... Figure 4 As shown, the processing order of the multiple CTUs contained in tile 1 is from the left end of the first column of tile 1 to the right end of the first column of tile 1, and then from the left end of the second column of tile 1 to the right end of the second column of tile 1.

[0258] In addition, sometimes a tile contains more than one slice, and sometimes a slice contains more than one tile.

[0259] Furthermore, an image can also be composed of tile sets. A tile set can contain more than one tile group, or more than one tile. An image can consist only of a tile set, a tile group, or a single tile. For example, the order in which multiple tiles are scanned in raster order for each tile set is defined as the basic encoding order of the tiles. A set of more than one tile with consecutive basic encoding orders within each tile set is defined as a tile group. Such an image can also be composed of the segmentation unit 102 described later (see...). Figure 7 )constitute.

[0260] [Hyper-level coding] Figure 5 and Figure 6This is a diagram illustrating an example of a hierarchical flow structure.

[0261] like Figure 5 As shown, the encoding device 100 can generate a temporally / spatially scalable stream by encoding multiple images into one of multiple layers. For example, by encoding images layer by layer, the encoding device 100 achieves a scalability where the enhancement layer exists above the base layer. Such encoding of each image is called scalable encoding. Therefore, the decoding device 200 can switch the image quality displayed by decoding the stream. That is, the decoding device 200 determines which layer to decode based on internal factors such as its own performance and external factors such as the state of the communication band. As a result, the decoding device 200 can freely switch between decoding the same content as low-resolution content and high-resolution content. For example, a user of the stream might watch the moving images halfway through using a smartphone while on the move, and then watch the rest of the moving images at home using a device such as an internet TV. Furthermore, the smartphone and the device described above are each equipped with a decoding device 200 with similar or different performance characteristics. In this case, if the device decodes to the higher layer in the stream, the user can watch high-quality moving images at home. Therefore, the encoding device 100 does not need to generate multiple streams with the same content but different image quality, which can reduce the processing load.

[0262] Furthermore, the enhancement layer may also contain metadata such as image-based statistical information. Alternatively, the decoding device 200 may generate a high-quality motion image by performing super-resolution on the image of the base layer based on the metadata. Super-resolution may involve either improving the signal-to-noise ratio (SNR) at the same resolution or increasing the resolution. The metadata includes information used to determine whether linear or nonlinear filtering coefficients are used in the super-resolution process, or information determining the parameter values ​​in filtering, machine learning, or least-squares operations used in the super-resolution process.

[0263] Alternatively, the image can be segmented into tiles based on the meaning of each object within it. In this case, the decoding device 200 can decode only a portion of the image by selecting tiles as the objects to be decoded. Furthermore, the attributes of the objects (people, cars, balls, etc.) and their positions within the image (coordinates within the same image, etc.) can be stored as metadata. In this case, the decoding device 200 can determine the desired target's location based on the metadata and decide which tiles contain that target. For example, such as... Figure 6 As shown, metadata can also be stored using data storage structures different from image data, such as SEI in HEVC. This metadata can represent, for example, the location, size, or color of the main target.

[0264] Alternatively, metadata can be stored in units consisting of multiple images, such as streams, sequences, or random access units. Thus, the decoding device 200 can obtain information such as the time when a specific person appears in a moving image, and by using this time and image unit information, it can determine the image in which the target exists and the target's position within that image.

[0265] [Encoding device] Next, the encoding device 100 of the embodiment will be described. Figure 7 This is a block diagram illustrating an example of the structure of the encoding apparatus 100 according to an embodiment. The encoding apparatus 100 encodes images in block units.

[0266] like Figure 7 As shown, the encoding apparatus 100 is an apparatus for encoding images in block units, and includes a segmentation unit 102, a subtraction unit 104, a transform unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a block memory 118, a cyclic filtering unit 120, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. Furthermore, the intra-frame prediction unit 124 and the inter-frame prediction unit 126 are each configured as part of the prediction processing unit.

[0267] [Installation example of the encoding device] Figure 8 This is a block diagram showing an example of the installation of the encoding device 100. The encoding device 100 includes a processor a1 and a memory a2. For example, Figure 7 The multiple components of the encoding device 100 shown are composed of Figure 8 The processor a1 and memory a2 shown are installed and implemented.

[0268] Processor a1 is a circuit that processes information and has access to memory a2. For example, processor a1 may be a dedicated or general-purpose electronic circuit that encodes images. Processor a1 can also be a processor like a CPU. Alternatively, processor a1 can be a collection of multiple electronic circuits. Furthermore, for example, processor a1 can also function as… Figure 7 The functions of the multiple components of the encoding device 100 shown, excluding the component for storing information.

[0269] Memory a2 is a dedicated or general-purpose memory used by processor a1 to encode images. Memory a2 can be an electronic circuit or connected to processor a1. Alternatively, memory a2 can be included within processor a1. Memory a2 can also be a collection of multiple electronic circuits. Memory a2 can be a disk or optical disc, or it can be a storage device or recording medium. Memory a2 can be either non-volatile or volatile memory.

[0270] For example, memory a2 can store the encoded image, or it can store the stream corresponding to the encoded image. Additionally, memory a2 can also store the program used by processor a1 to encode the image.

[0271] Additionally, for example, memory a2 can also serve as Figure 7 The encoding device 100 shown has multiple components, including the component for storing information. Specifically, the memory a2 can serve as... Figure 7 The block memory 118 and frame memory 122 shown have the following functions. More specifically, reconstructed images (specifically, reconstructed blocks and reconstructed pictures, etc.) can be stored in memory a2.

[0272] Alternatively, the encoding device 100 may not require installation. Figure 7 All of the multiple constituent elements shown can also be processed without the aforementioned multiple processing steps. Figure 7 A portion of the multiple components shown may be included in other devices, or a portion of the multiple processes described above may be performed by other devices.

[0273] The following describes the overall processing flow of the encoding device 100, followed by an explanation of the constituent elements included in the encoding device 100.

[0274] [Overall Encoding Process] Figure 9 This is a flowchart illustrating an example of the overall encoding process performed by the encoding device 100.

[0275] First, the segmentation unit 102 of the encoding device 100 segments the image contained in the original image into multiple fixed-size blocks (128×128 pixels) (step Sa_1). Then, the segmentation unit 102 selects a segmentation pattern for each fixed-size block (step Sa_2). That is, the segmentation unit 102 further segments the fixed-size blocks into multiple blocks constituting the selected segmentation pattern. Then, the encoding device 100 performs the processing steps Sa_3 to Sa_9 for each of these multiple blocks.

[0276] The prediction processing unit, consisting of the intra-frame prediction unit 124 and the inter-frame prediction unit 126, and the prediction control unit 128 generate the prediction image of the current block (step Sa_3). In addition, the prediction image is also called the prediction signal, the prediction block, or the prediction sample.

[0277] Next, the subtraction unit 104 generates the difference between the current block and the predicted image as the prediction residual (step Sa_4). The prediction residual is also called the prediction error.

[0278] Next, the transformation unit 106 and the quantization unit 108 generate multiple quantization coefficients by transforming and quantizing the predicted image (step Sa_5).

[0279] Next, the entropy coding unit 110 generates a stream (step Sa_6) by encoding the plurality of quantization coefficients and prediction parameters related to the generation of the prediction image (specifically, entropy coding).

[0280] Next, the inverse quantization unit 112 and the inverse transformation unit 114 restore the prediction residual by performing inverse quantization and inverse transformation on multiple quantization coefficients (step Sa_7).

[0281] Next, the addition unit 116 reconstructs the current block by adding the prediction image to the restored prediction residual (step Sa_8). This generates a reconstructed image. Furthermore, the reconstructed image is also called a reconstructed block; specifically, the reconstructed image generated by the encoding device 100 is also called a local decoding block or a local decoding image.

[0282] When the reconstructed image is generated, the cyclic filtering unit 120 filters the reconstructed image as needed (step Sa_9).

[0283] Then, the encoding device 100 determines whether the encoding of the entire image has been completed (step Sa_10), and if it determines that the encoding has not been completed (no in step Sa_10), it repeatedly performs the processing that started from step Sa_2.

[0284] Furthermore, in the example described above, the encoding device 100 selects one segmentation pattern for blocks of a fixed size and encodes each block according to that segmentation pattern. However, it can also encode each block according to each of multiple segmentation patterns. In this case, the encoding device 100 can evaluate the cost for each of the multiple segmentation patterns and, for example, select the stream obtained by encoding according to the segmentation pattern with the lowest cost as the final output stream.

[0285] Furthermore, these steps Sa_1 to Sa_10 can be performed sequentially by the encoding device 100, and some of these processes can be performed in parallel or in reverse order.

[0286] The encoding process of this encoding device 100 uses a hybrid encoding method combining predictive coding and transform coding. Furthermore, predictive coding is performed through an encoding cycle, which consists of a subtraction unit 104, a transform unit 106, a quantization unit 108, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a cyclic filtering unit 120, a block memory 118, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, and a prediction control unit 128. That is, the prediction processing unit, composed of the intra-frame prediction unit 124 and the inter-frame prediction unit 126, constitutes part of the encoding cycle.

[0287] [Divider] The segmentation unit 102 divides each image contained in the original image into multiple blocks and outputs each block to the subtraction unit 104. For example, the segmentation unit 102 first segments the image into blocks of fixed size (e.g., 128×128 pixels). These fixed-size blocks may be called coding tree units (CTUs). Furthermore, the segmentation unit 102, for example, divides the fixed-size blocks into blocks of variable size (e.g., 64×64 pixels or less) based on recursive quadtree and / or binary tree block segmentation. That is, the segmentation unit 102 selects a segmentation style. These variable-size blocks may be called coding units (CUs), prediction units (PUs), or transform units (TUs). In addition, in various installation examples, it is not necessary to distinguish between CUs, PUs, and TUs, and some or all of the blocks in the image may be processed as CUs, PUs, or TUs.

[0288] Figure 10 This is a diagram illustrating an example of block segmentation in an implementation method. In Figure 10 In the diagram, solid lines represent block boundaries based on quadtree block partitioning, and dashed lines represent block boundaries based on binary tree block partitioning.

[0289] Here, block 10 is a 128×128 pixel square block. Block 10 is first divided into four 64×64 pixel square blocks (quadtree block partitioning).

[0290] The 64×64 pixel square block in the upper left corner is further vertically divided into two rectangular blocks each consisting of 32×64 pixels. The 32×64 pixel rectangular block on the left is further vertically divided into two rectangular blocks each consisting of 16×64 pixels (binary tree block partitioning). As a result, the 64×64 pixel square block in the upper left corner is divided into two 16×64 pixel rectangular blocks 11 and 12, and a 32×64 pixel rectangular block 13.

[0291] The 64×64 pixel square block in the upper right corner is horizontally divided into two rectangular blocks 14 and 15, each consisting of 64×32 pixels (binary tree block division).

[0292] The 64×64 pixel square in the lower left corner is divided into four 32×32 pixel squares (quadtree block partitioning). The upper left and lower right squares within these four 32×32 pixel squares are further partitioned. The upper left 32×32 pixel square is vertically divided into two 16×32 pixel rectangular squares, and the rightmost 16×32 pixel rectangular square is further horizontally divided into two 16×16 pixel squares (binary tree block partitioning). The lower right 32×32 pixel square is horizontally divided into two 32×16 pixel rectangular squares (binary tree block partitioning). As a result, the 64×64 pixel square block in the lower left corner was divided into a 16×32 pixel rectangular block 16, two 16×16 pixel square blocks 17 and 18, two 32×32 pixel square blocks 19 and 20, and two 32×16 pixel rectangular blocks 21 and 22.

[0293] The lower right block 23, consisting of 64×64 pixels, is not divided.

[0294] As described above, in Figure 10 In the example, block 10 is divided into 13 variable-size blocks 11 to 23 based on recursive quadtree and binary tree block partitioning. Such partitioning is sometimes referred to as QTBT (quad-tree plus binary tree) partitioning.

[0295] In addition, Figure 10 In this context, a block can be divided into 2 or 4 blocks (quadtree or binary tree block partitioning), but the partitioning is not limited to these. For example, a block can also be divided into 3 blocks (ternary tree partitioning). Partitioning including such ternary tree partitioning is sometimes referred to as MBT (multi-type tree) partitioning.

[0296] Figure 11 This is a diagram illustrating an example of the structure of the segment 102. (See diagram below.) Figure 11 As shown, the segmentation unit 102 may also include a block segmentation determination unit 102a. As an example, the block segmentation determination unit 102a may also perform the following processing.

[0297] The block segmentation determination unit 102a collects block information from the block memory 118 or the frame memory 122, and determines the segmentation pattern based on the block information. The segmentation unit 102 segments the original image according to the segmentation pattern and outputs one or more blocks obtained by the segmentation to the subtraction unit 104.

[0298] Furthermore, the block segmentation determination unit 102a outputs parameters representing the segmentation pattern described above to the transform unit 106, the inverse transform unit 114, the intra-frame prediction unit 124, the inter-frame prediction unit 126, and the entropy coding unit 110, for example. The transform unit 106 can transform the prediction residual based on these parameters, and the intra-frame prediction unit 124 and the inter-frame prediction unit 126 can generate prediction images based on these parameters. In addition, the entropy coding unit 110 can also entropy code these parameters.

[0299] As an example, parameters related to the splitting style can also be written to the stream as follows.

[0300] Figure 12 This is a diagram showing examples of segmentation styles. Segmentation styles include: four-part segmentation (QT), which divides the block into two parts horizontally and vertically; three-part segmentation (HT or VT), which divides the block in the same direction at a ratio of 1:2:1; two-part segmentation (HB or VB), which divides the block in the same direction at a ratio of 1:1; and no segmentation (NS).

[0301] In addition, in the case of four-segmentation and no segmentation, the segmentation style does not have block segmentation direction, while in the case of two-segmentation and three-segmentation, the segmentation style has segmentation direction information.

[0302] Figure 13A and Figure 13B This is a diagram representing an example of a syntax tree for a segmentation pattern. In Figure 13A In the example, first, there is information indicating whether to split (S: Split flag), then information indicating whether to perform a four-way split (QT: QT flag). Next, there is information indicating whether to perform a three-way split or a two-way split (TT: TT flag or BT: BT flag), and finally, there is information indicating the splitting direction (Ver: Vertical flag or Hor: Horizontal flag). Alternatively, for each of the more than one blocks obtained through such splitting pattern-based splitting, the same processing can be repeatedly applied to the splitting. That is, as an example, the determination of whether to split, whether to perform a four-way split, whether the splitting method is horizontal or vertical, and whether to perform a three-way split or a two-way split can be recursively implemented, and the results of the implemented determinations can be distributed according to... Figure 13A The encoding order exposed by the syntax tree shown is encoded into the stream.

[0303] In addition, Figure 13A In the syntax tree shown, this information is configured in the order of S, QT, TT, Ver, but it can also be configured in the order of S, QT, Ver, BT. That is, in Figure 13BIn the example, first, there is information indicating whether to perform a split (S: Split flag), then there is information indicating whether to perform a four-part split (QT: QT flag). Next, there is information indicating the splitting direction (Ver: Vertical flag or Hor: Horizontal flag), and finally there is information indicating whether to perform a two-part split or a three-part split (BT: BT flag or TT: TT flag).

[0304] Additionally, the segmentation style described here is an example; you may use segmentation styles other than those described, or you may use only a portion of the described segmentation styles.

[0305] [Subtraction Section] The subtraction unit 104 subtracts the prediction image (the prediction image input from the prediction control unit 128) from the original image in block units, which are input from the segmentation unit 102 and segmented by the segmentation unit 102. That is, the subtraction unit 104 calculates the prediction residual of the current block. Furthermore, the subtraction unit 104 outputs the calculated prediction residual to the transformation unit 106.

[0306] The original image is the input signal of the encoding device 100, such as a signal representing the image of each picture that constitutes the moving image (e.g., a luminance signal and two chroma signals).

[0307] [Transformation Section] The transformation unit 106 transforms the prediction residual in the spatial domain into transformation coefficients in the frequency domain, and outputs the transformation coefficients to the vectorization unit 108. Specifically, the transformation unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction residual in the spatial domain, for example.

[0308] Alternatively, the transform unit 106 can adaptively select a transform type from multiple transform types and use the transform basis function corresponding to the selected transform type to transform the predicted residuals into transform coefficients. Such a transform is sometimes called EMT (explicit multiple core transform) or AMT (adaptive multiple transform). Furthermore, the transform basis function is sometimes simply referred to as the basis.

[0309] Several transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Furthermore, these transformation types can be denoted as DCT2, DCT5, DCT8, DST1, and DST7, respectively. Figure 14 This is a table representing the transformation basis functions corresponding to each transformation type. Figure 14In this context, N represents the number of input pixels. The choice of transformation type from these multiple transformation types can depend on either the type of prediction (intra-frame prediction and inter-frame prediction, etc.) or the intra-frame prediction mode.

[0310] Information indicating whether such EMT or AMT is applied (e.g., referred to as EMT flags or AMT flags) and information indicating the selected transform type are typically signaled at the CU level. However, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, picture level, slice level, brick level, or CTU level).

[0311] Furthermore, the transform unit 106 can also perform a re-transformation on the transform coefficients (i.e., the transform result). Such a re-transformation is sometimes referred to as AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transform unit 106 performs a re-transformation on each sub-block (e.g., a 4×4 pixel sub-block) contained in the block of transform coefficients corresponding to the intra-frame prediction residual. Information indicating whether NSST is applied and information related to the transform matrix used in NSST are typically signaled at the CU level. However, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, image level, slice level, brick level, or CTU level).

[0312] Separable and non-separable transformations can also be applied in the transformation unit 106. A separable transformation refers to a method of performing multiple transformations by separating the input in each direction, which is equivalent to the number of dimensions of the input. A non-separable transformation refers to a method of treating two or more dimensions together as one dimension and transforming them together when the input is multidimensional.

[0313] For example, as a non-separable transformation, one could consider treating a 4×4 pixel block as a permutation of 16 elements and transforming that permutation with a 16×16 transformation matrix.

[0314] Furthermore, in a further example of the non-separable transformation, the 4×4 pixel input block can be viewed as an arrangement of 16 elements, and then a transformation involving multiple Givens rotations of that arrangement can be performed (HypercubeGivens Transform).

[0315] In the transformation within the transformation unit 106, the transformation type of the transformation basis function to be transformed into the frequency domain can be switched according to the region within the CU. As an example, there is SVT (Spatially Varying Transform).

[0316] Figure 15 This is a diagram representing an example of SVT.

[0317] In SVT, such as Figure 15 As shown, the CU is bisected horizontally or vertically, and only one region is transformed to the frequency domain. The transformation type can be set for each region, for example, using DST7 and DCT8. For example, DST7 and DCT8 can be used for the region at position 0 in the two regions obtained by bisecting the CU vertically. Alternatively, DST7 can be used for the region at position 1 in the two regions. Similarly, DST7 and DCT8 can be used for the region at position 0 in the two regions obtained by bisecting the CU horizontally. Alternatively, DST7 can be used for the region at position 1 in the two regions. In such a... Figure 15 In the example shown, only one of the two regions within the CU is transformed, while the other remains untransformed. However, it's also possible to transform each region separately. Furthermore, the segmentation method isn't limited to bisection; it can also be quarter-division. Moreover, it can be more flexible, encoding information representing the segmentation method and performing signaling similarly to CU segmentation. Additionally, SVT is sometimes referred to as SBT (Sub-block Transform).

[0318] The aforementioned AMT and EMT can also be referred to as MTS (Multiple Transform Selection). When applying MTS, transform types such as DST7 or DCT8 can be selected, indicating that the selected transform type information can be encoded into the index information of each CU. On the other hand, as a process that selects the transform type used in orthogonal transformation based on the shape of the CU without encoding the index information, there is a process called IMTS (Implicit MTS). When applying IMTS, for example, if the shape of the CU is rectangular, DST7 is used on the shorter side and DCT2 on the longer side, performing orthogonal transformations respectively. Alternatively, for example, if the shape of the CU is square, DCT2 is used for orthogonal transformation if MTS is valid within the sequence, and DST7 is used if MTS is invalid. DCT2 and DST7 are examples; other transform types can be used, and different combinations of transform types can be set. IMTS can be used only in intra-frame prediction blocks, or it can be used together in both intra-frame prediction blocks and inter-frame prediction blocks.

[0319] The above describes the three processes MTS, SBT, and IMTS as selection processes for selectively switching the transformation type used in orthogonal transformations. However, all three selection processes can be effective, or only a portion of the selection processes can be selectively effective. Whether each selection process is effective can be identified by the flag information in the header such as SPS. For example, if all three selection processes are effective, one of the three selection processes is selected for orthogonal transformation on a CU unit basis. In addition, as long as the selection process for selectively switching the transformation type can achieve at least one of the following four functions [1] to [4], a selection process different from the above three selection processes can be used, or the above three selection processes can be replaced with other processes respectively. Function [1] is to perform orthogonal transformation on the entire range within the CU and encode the information indicating the transformation type used in the transformation. Function [2] is to perform orthogonal transformation on the entire range of the CU and determine the transformation type based on the prescribed rules without encoding the information indicating the transformation type. Function [3] is to perform orthogonal transformation on a part of the CU and encode the information indicating the transformation type used in the transformation. Function [4] is to perform orthogonal transformation on a part of the CU, and to indicate that the information of the transformation type used in the transformation is not encoded but the transformation type is determined based on the prescribed rules.

[0320] Furthermore, the presence or absence of MTS, IMTS, and SBT can also be determined on a per-processing-unit basis. For example, the presence or absence of these applications can be determined by sequence unit, image unit, brick unit, slice unit, CTU unit, or CU unit.

[0321] Furthermore, the tool for selectively switching the transformation type in this disclosure can also be referred to as a method, selection process, or basis selection procedure that adaptively selects the basis used in the transformation process. Additionally, the tool for selectively switching the transformation type can also be referred to as a mode that adaptively selects the transformation type.

[0322] Figure 16 This is a flowchart illustrating an example of the processing performed by the transformation unit 106.

[0323] For example, the transformation unit 106 determines whether to perform an orthogonal transformation (step St_1). Here, when it is determined that an orthogonal transformation should be performed (yes in step St_1), the transformation unit 106 selects the transformation type used for orthogonal transformation from multiple transformation types (step St_2). Next, the transformation unit 106 performs orthogonal transformation by applying the selected transformation type to the prediction residual of the current block (step St_3). Then, the transformation unit 106 outputs information indicating the selected transformation type to the entropy encoding unit 110, thereby encoding the information (step St_4). On the other hand, when it is determined that an orthogonal transformation should not be performed (no in step St_1), the transformation unit 106 outputs information indicating that an orthogonal transformation should not be performed to the entropy encoding unit 110, thereby encoding the information (step St_5). Furthermore, the determination of whether to perform an orthogonal transformation in step St_1 can be based on, for example, the size of the transformation block, the prediction mode applied to the CU, etc. Additionally, the information indicating the transformation type used for orthogonal transformation is not encoded, or a predefined transformation type can be used for orthogonal transformation.

[0324] Figure 17 This is a flowchart illustrating another example of the processing performed by the transformation unit 106. Additionally, Figure 17 The example shown is similar to Figure 16 The example shown is also an example of an orthogonal transformation in which the method of selectively switching the type of transformation used by the orthogonal transformation is applied.

[0325] As an example, the first transformation type group can include DCT2, DST7, and DCT8. Similarly, as an example, the second transformation type group can include DCT2. Furthermore, the transformation types contained in the first and second transformation type groups can either partially overlap or be entirely different.

[0326] Specifically, the transformation unit 106 determines whether the transformation size is below a predetermined value (step Su_1). If it is determined to be below the predetermined value (yes in step Su_1), the transformation unit 106 performs an orthogonal transformation on the prediction residual of the current block using the transformation types included in the first transformation type group (step Su_2). Next, the transformation unit 106 encodes this information by outputting information indicating which transformation type from the one or more transformation types included in the first transformation type group is used to the entropy encoding unit 110 (step Su_3). On the other hand, if it is determined that the transformation size is not below the predetermined value (no in step Su_1), the transformation unit 106 performs an orthogonal transformation on the prediction residual of the current block using the second transformation type group (step Su_4).

[0327] In step Su_3, the information indicating the transformation type used in the orthogonal transformation can be a combination of the transformation type applied to the vertical direction and the transformation type applied to the horizontal direction of the current block. Alternatively, the first transformation type group may contain only one transformation type, and the information indicating the transformation type used in the orthogonal transformation may not be encoded. The second transformation type group may contain multiple transformation types, or the information indicating the transformation type used in the orthogonal transformation among the more than one transformation type included in the second transformation type group may be encoded.

[0328] Alternatively, the transformation type can be determined solely based on the transformation size. Furthermore, if the transformation type used for orthogonal transformation is determined based on the transformation size, it is not limited to determining whether the transformation size is below a specified value.

[0329] [Quantitative Department] The quantization unit 108 quantizes the transform coefficients output from the transform unit 106. Specifically, the quantization unit 108 scans multiple transform coefficients of the current block in a predetermined scan order and quantizes the transform coefficients based on the quantization parameters (QP) corresponding to the scanned transform coefficients. Furthermore, the quantization unit 108 outputs the quantized multiple transform coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy encoding unit 110 and the inverse quantization unit 112.

[0330] The specified scan order is the order in which the transform coefficients are quantized / inverse quantized. For example, the specified scan order is defined by ascending frequency (from low frequency to high frequency) or descending frequency (from high frequency to low frequency).

[0331] The quantization parameter (QP) is a parameter that defines the quantization step size (quantization width). For example, if the value of the quantization parameter increases, the quantization step size also increases. That is, if the value of the quantization parameter increases, the error of the quantization coefficients (quantization error) increases.

[0332] In addition, quantization matrices are sometimes used in quantization. For example, multiple quantization matrices are sometimes used corresponding to frequency-shifted sizes such as 4×4 and 8×8, prediction modes such as intra-frame prediction and inter-frame prediction, and pixel components such as luminance and chromatic aberration. Furthermore, quantization refers to digitizing values ​​sampled at predetermined intervals and mapping them to predetermined levels. In this technical field, rounding, scaling, or other similar techniques are sometimes used.

[0333] There are two methods for using quantization matrices: one is to use a quantization matrix directly set on the encoding device 100 side, and the other is to use a default quantization matrix (default matrix). On the encoding device 100 side, by directly setting the quantization matrix, a quantization matrix corresponding to the image features can be set. However, in this case, there is a drawback that the encoding amount increases due to the encoding of the quantization matrix. Alternatively, instead of directly using the default quantization matrix or the encoded quantization matrix, a quantization matrix used in the quantization of the current block can be generated based on the default quantization matrix or the encoded quantization matrix.

[0334] On the other hand, there are also methods that quantize without using a quantization matrix, where the coefficients of high-frequency components and low-frequency components are all the same. Furthermore, this method is equivalent to using a quantization matrix (a flat matrix) where all coefficients have the same value.

[0335] Quantization matrices can be encoded at the sequence level, image level, slice level, brick level, or CTU level, for example.

[0336] When using a quantization matrix, the quantization unit 108 scales the quantization width, calculated from quantization parameters, etc., for each transform coefficient using the values ​​of the quantization matrix. Quantization without using a quantization matrix can also be performed by quantizing the transform coefficients based on the quantization width calculated from quantization parameters, etc. Furthermore, in quantization without using a quantization matrix, the quantization width can be multiplied by a commonly defined value for all transform coefficients within the block.

[0337] Figure 18 This is a block diagram illustrating an example of the structure of the quantization unit 108.

[0338] The quantization unit 108 includes, for example, a differential quantization parameter generation unit 108a, a predictive quantization parameter generation unit 108b, a quantization parameter generation unit 108c, a quantization parameter storage unit 108d, and a quantization processing unit 108e.

[0339] Figure 19 This is a flowchart illustrating an example of quantization performed by the quantization unit 108.

[0340] As an example, the quantization unit 108 can be based on Figure 19 The flowchart shown illustrates quantization performed for each CU. Specifically, the quantization parameter generation unit 108c determines whether quantization should be performed (step Sv_1). Here, when it is determined that quantization should be performed (yes in step Sv_1), the quantization parameter generation unit 108c generates the quantization parameters for the current block (step Sv_2) and saves the quantization parameters to the quantization parameter storage unit 108d (step Sv_3).

[0341] Next, the quantization processing unit 108e quantizes the transform coefficients of the current block using the quantization parameters generated in step Sv_2 (step Sv_4). Then, the prediction quantization parameter generation unit 108b obtains quantization parameters for a different processing unit than the current block from the quantization parameter storage unit 108d (step Sv_5). Based on the obtained quantization parameters, the prediction quantization parameter generation unit 108b generates prediction quantization parameters for the current block (step Sv_6). The differential quantization parameter generation unit 108a calculates the difference between the quantization parameters of the current block generated by the quantization parameter generation unit 108c and the prediction quantization parameters of the current block generated by the prediction quantization parameter generation unit 108b (step Sv_7). By calculating this difference, differential quantization parameters are generated. The differential quantization parameter generation unit 108a outputs the differential quantization parameters to the entropy encoding unit 110, thereby encoding the differential quantization parameters (step Sv_8).

[0342] Furthermore, differential quantization parameters can be encoded at the sequence level, image level, slice level, brick level, or CTU level. Additionally, the initial values ​​of the quantization parameters can be encoded at the sequence level, image level, slice level, brick level, or CTU level. In this case, the quantization parameters can be generated using the initial values ​​of the quantization parameters and the differential quantization parameters.

[0343] Furthermore, the quantization unit 108 may have multiple quantizers, and may also apply dependent quantization to quantize the transform coefficients using a quantization method selected from multiple quantization methods.

[0344] [Entropy Coding Department] Figure 20 This is a block diagram illustrating an example of the structure of the entropy coding unit 110.

[0345] The entropy coding unit 110 performs entropy coding on the quantization coefficients input from the quantization unit 108 and the prediction parameters input from the prediction parameter generation unit 130, thereby generating a stream. In this entropy coding, for example, CABAC (Context-based Adaptive Binary Arithmetic Coding) is used. Specifically, the entropy coding unit 110 includes, for example, a binarization unit 110a, a context control unit 110b, and a binary arithmetic coding unit 110c. The binarization unit 110a performs binarization, transforming multi-valued signals such as quantization coefficients and prediction parameters into binary signals. Binarization methods include, for example, Truncated Rice Binarization, Exponential Golomb codes, Fixed Length Binarization, etc. The context control unit 110b derives context values ​​corresponding to the features of syntactic elements or the surrounding conditions, i.e., the probability of occurrence of the binary signal. Methods for deriving this context value include, for example, bypassing, syntactic element reference, upper / left adjacent block reference, hierarchical information reference, and others. The binary arithmetic coding unit 110c uses the derived context value to perform arithmetic coding on the binarized signal.

[0346] Figure 21 This is a diagram showing the flow of CABAC in the entropy coding unit 110.

[0347] First, initialization is performed in the CABAC within the entropy coding unit 110. This initialization includes initialization in the binary arithmetic coding unit 110c and setting of the initial context value. Then, the binarization unit 110a and the binary arithmetic coding unit 110c sequentially perform binarization and arithmetic coding on, for example, multiple quantization coefficients of the CTU. During this time, the context control unit 110b updates the context value each time arithmetic coding is performed. Then, the context control unit 110b performs context value backoff as a post-processing step. This backoffed context value is used, for example, as the initial value for the context value of the next CTU.

[0348] [Inverse Quantization Department] The inverse quantization unit 112 performs inverse quantization on the quantization coefficients input from the quantization unit 108. Specifically, the inverse quantization unit 112 performs inverse quantization on the quantization coefficients of the current block in a predetermined scan order. Furthermore, the inverse quantization unit 112 outputs the inverse quantized transform coefficients of the current block to the inverse transform unit 114.

[0349] [Inverse Transformation Section] The inverse transform unit 114 restores the prediction residual by performing an inverse transform on the transform coefficients input from the inverse quantization unit 112. Specifically, the inverse transform unit 114 restores the prediction residual of the current block by performing an inverse transform on the transform coefficients corresponding to the transform of the transform unit 106. Furthermore, the inverse transform unit 114 outputs the restored prediction residual to the addition unit 116.

[0350] Furthermore, the restored prediction residual is usually inconsistent with the prediction error calculated by the subtraction unit 104 because information was lost during quantization. That is, the restored prediction residual usually contains quantization error.

[0351] [Addition Department] The addition unit 116 reconstructs the current block by adding the prediction residual input from the inverse transform unit 114 to the prediction image input from the prediction control unit 128. As a result, a reconstructed image is generated. Furthermore, the addition unit 116 outputs the reconstructed image to the block memory 118 and the cyclic filtering unit 120.

[0352] [Block Memory] Block memory 118 is, for example, a storage unit for storing blocks referenced in intra-frame prediction, and blocks within the current image. Specifically, block memory 118 stores the reconstructed image output from addition unit 116.

[0353] [Frame Memory] The frame memory 122 is, for example, a storage unit used to store reference images used in inter-frame prediction, and is also referred to as a frame buffer. Specifically, the frame memory 122 stores the reconstructed image filtered by the cyclic filtering unit 120.

[0354] [Loop Filtering Section] The cyclic filtering unit 120 applies cyclic filtering processing to the reconstructed image output by the addition unit 116, and outputs the filtered reconstructed image to the frame memory 122. Cyclic filtering refers to filtering used within the encoding loop (in-loop filtering), such as adaptive cyclic filtering (ALF), deblocking filtering (DF or DBF), and sample adaptive offset (SAO).

[0355] Figure 22 This is a block diagram illustrating an example of the structure of the cyclic filter section 120.

[0356] For example Figure 22As shown, the cyclic filtering unit 120 includes a deblocking filtering processing unit 120a, a SAO processing unit 120b, and an ALF processing unit 120c. The deblocking filtering processing unit 120a applies the aforementioned deblocking filtering processing to the reconstructed image. The SAO processing unit 120b applies the aforementioned SAO processing to the reconstructed image after deblocking filtering. Additionally, the ALF processing unit 120c applies the aforementioned ALF processing to the reconstructed image after SAO processing. Details regarding ALF and deblocking filtering will be described later. SAO processing improves image quality by reducing ringing (a phenomenon where pixel values ​​fluctuate around edges) and correcting pixel value deviations. This SAO processing includes, for example, edge offset processing and band offset processing. Furthermore, the cyclic filtering unit 120 may not include... Figure 22 The disclosed processing unit may also include only a portion of the processing units. Furthermore, the cyclic filtering unit 120 may be configured according to... Figure 22 The structures that perform the above processes in different orders as disclosed in the text.

[0357] [Loop Filtering Section > Adaptive Loop Filter] In ALF, a least-squares error filter is used to remove coding distortion. For example, for each 2×2 pixel sub-block within the current block, one filter is selected from multiple filters based on the direction of the gradient and the activity of the locality.

[0358] Specifically, sub-blocks (e.g., 2×2 pixel sub-blocks) are first classified into multiple classes (e.g., 15 or 25 classes). Sub-block classification is performed, for example, based on gradient direction and activity. In a specific example, a classification value C (e.g., C = 5D + A) is calculated using the gradient direction value D (e.g., 0–2 or 0–4) and the gradient activity value A (e.g., 0–4). Then, based on the classification value C, the sub-blocks are classified into multiple classes.

[0359] The gradient direction value D is derived, for example, by comparing gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). Furthermore, the gradient activity value A is derived, for example, by summing the gradients in multiple directions and quantizing the sum.

[0360] Based on the results of this classification, the filter used for the sub-block is determined from among multiple filters.

[0361] The shape of the filter used in ALF can be, for example, a circular symmetrical shape. Figures 23A-23C This is a diagram showing several examples of the shapes of filters used in ALF. Figure 23A This represents a 5×5 rhombus-shaped filter. Figure 23BThis represents a 7×7 rhombus-shaped filter. Figure 23C This represents a 9×9 diamond-shaped filter. Information representing the filter's shape is typically signaled at the picture level. However, the signaling of filter shape information is not limited to the picture level; it can also be at other levels (e.g., sequence level, slice level, brick level, CTU level, or CU level).

[0362] The on / off state of ALF can also be determined at the picture level or the CU level. For example, regarding luminance, the decision to use ALF can be made at the CU level, while regarding chromatic aberration, it can be made at the picture level. Information indicating the on / off state of ALF is typically signaled at the picture level or the CU level. However, the signaling of information indicating the on / off state of ALF is not limited to the picture level or the CU level; it can also be at other levels (e.g., sequence level, slice level, brick level, or CTU level).

[0363] Additionally, as described above, one filter is selected from a plurality of filters to apply ALF processing to the sub-block. For each of these plurality of filters (e.g., up to 15 or 25 filters), the set of coefficients consisting of the plurality of coefficients used in that filter is typically signaled at the picture level. Furthermore, the signaling of the coefficient set is not limited to the picture level and can also be at other levels (e.g., sequence level, slice level, brick level, CTU level, CU level, or sub-block level).

[0364] [Loop Filtering > Cross Component Adaptive Loop Filter] Figure 23D This is a graph showing an example of Y sample (component 1) being used for CCALF of Cb and CCALF of Cr (multiple components different from component 1). Figure 23E This is a diagram representing a diamond-shaped filter.

[0365] One example of CC-ALF is achieved by using a linear diamond-shaped filter ( Figure 23D , Figure 23E The filter is applied to the luminance channel of each chromatic aberration component to perform the action. For example, filter coefficients are sent in APS, scaled by a factor of 2^10, and rounded to a fixed decimal point. The application of filters is controlled to a variable block size and is notified via context-encoded flags received by the block for each sample. The block size and CC-ALF validation flags are received at the slice level for each chromatic aberration component. The syntax and semantics of CC-ALF are provided in the Appendix. In this paper, block sizes of 16x16, 32x32, 64x64, and 128x128 are supported (in chromatic aberration samples).

[0366] [Loop Filtering > Combined with Joint Chroma Cross Component Adaptive Loop Filter] Figure 23F This is a diagram representing an example of JC-CCALF. Figure 23G This is a graph representing examples of JC-CCALF weight_index candidates.

[0367] One example of JC-CCALF uses only one CCALF filter to generate a single CCALF filter output as a color difference adjustment signal for only one color component. An appropriately weighted version of the same color difference adjustment signal is then applied to the other color components. This roughly halves the complexity of existing CCALF methods.

[0368] The weight values ​​are encoded as a sign flag and a weight index. The weight index (represented as weight_index) is encoded as 3 bits and specifies the size of the JC-CCALF weight JcCcWeight. It cannot be the same as 0. The size of JcCcWeight is determined as follows.

[0369] When weight_index is below 4, JcCcWeight is equal to weight_index >> 2.

[0370] In all other cases, JcCcWeight is equal to 4 / (weight_index-4).

[0371] The block-level on / off control for Cb and Cr ALF filters is separate. This is similar to CCALF, where two separate sets of block-level on / off control flags are encoded. Here, unlike CCALF, the on / off control block size for Cb and Cr is the same, so only one block size variable is encoded.

[0372] [Loop Filtering Section > Deblocking Filter] In the deblocking filtering process, the cyclic filtering unit 120 reduces the distortion generated at the block boundaries by filtering the block boundaries of the reconstructed image.

[0373] Figure 24 This is a block diagram illustrating an example of the detailed structure of the deblocking filter processing unit 120a.

[0374] The deblocking filtering processing unit 120a includes, for example, a boundary determination unit 1201, a filtering determination unit 1203, a filtering processing unit 1205, a processing determination unit 1208, a filtering characteristic determination unit 1207, and switches 1202, 1204, and 1206.

[0375] The boundary determination unit 1201 determines whether there are pixels (i.e., object pixels) that have undergone deblocking filtering near the block boundary. Then, the boundary determination unit 1201 outputs its determination result to the switch 1202 and the processing determination unit 1208.

[0376] If the boundary determination unit 1201 determines that the object pixel exists near the block boundary, the switch 1202 outputs the image before filtering to the switch 1204. Conversely, if the boundary determination unit 1201 determines that the object pixel does not exist near the block boundary, the switch 1202 outputs the image before filtering to the switch 1206. Furthermore, the image before filtering is an image composed of the object pixel and at least one surrounding pixel located around the object pixel.

[0377] The filtering determination unit 1203 determines whether to perform deblocking filtering on the target pixel based on the pixel values ​​of at least one surrounding pixel located around the target pixel. Then, the filtering determination unit 1203 outputs the determination result to the switch 1204 and the processing determination unit 1208.

[0378] If the filtering determination unit 1203 determines that deblocking filtering processing should be performed on the target pixels, the switch 1204 outputs the unfiltered image obtained via the switch 1202 to the filtering processing unit 1205. Conversely, if the filtering determination unit 1203 determines that deblocking filtering processing should not be performed on the target pixels, the switch 1204 outputs the unfiltered image obtained via the switch 1202 to the switch 1206.

[0379] Having obtained the image before filtering via switches 1202 and 1204, the filtering unit 1205 performs deblocking filtering on the target pixels, with filtering characteristics determined by the filtering characteristic determination unit 1207. Then, the filtering unit 1205 outputs the filtered pixels to switch 1206.

[0380] Under the control of the processing determination unit 1208, the switch 1206 selectively outputs pixels that have not been deblocked and pixels that have been deblocked by the filtering processing unit 1205.

[0381] The processing determination unit 1208 controls the switch 1206 based on the respective determination results of the boundary determination unit 1201 and the filtering determination unit 1203. That is, when the boundary determination unit 1201 determines that the object pixel exists near a block boundary and the filtering determination unit 1203 determines that deblocking filtering should be performed on the object pixel, the processing determination unit 1208 outputs the deblocked pixel from the switch 1206. Otherwise, the processing determination unit 1208 outputs pixels that have not undergone deblocking / filtering from the switch 1206. By repeatedly outputting such pixels, a filtered image is output from the switch 1206. Figure 24 The structure shown is an example of the structure in the deblocking filter processing unit 120a, but the deblocking filter processing unit 120a may also have other structures.

[0382] Figure 25 This is a diagram illustrating an example of deblocking filtering with filtering characteristics symmetrical about the block boundary.

[0383] In deblocking filtering, for example, using pixel values ​​and quantization parameters, two deblocking filters with different characteristics are selected: a strong filter and a weak filter. In the strong filter, such as... Figure 25 As shown, when there are pixels p0 to p2 and pixels q0 to q2 separated by a block boundary, the pixel values ​​of pixels q0 to q2 are changed to pixel values ​​q'0 to q'2 by performing the operation shown in the following formula.

[0384] q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8 q'1 = (p0 + q0 + q1 + q2 + 2) / 4 q'2=(p0+q0+q1+3×q2+2×q3+4) / 8 Furthermore, in the above formulas, p0~p2 and q0~q2 are the pixel values ​​of pixels p0~p2 and q0~q2 respectively. Additionally, q3 is the pixel value of pixel q3, which is adjacent to pixel q2 on the side opposite to the block boundary. Furthermore, the coefficients multiplied by the pixel values ​​of each pixel used in the deblocking filtering process on the right-hand side of the above formulas are the filtering coefficients.

[0385] Furthermore, in the deblocking filtering process, limiting can also be performed in a way that prevents the calculated pixel value from changing even if it exceeds a threshold. In this limiting process, a threshold determined by the quantization parameters is used to limit the calculated pixel value based on the above formula to "the pixel value before calculation ± 2 × the threshold". This prevents excessive smoothing.

[0386] Figure 26 This is a diagram illustrating an example of block boundaries used in deblocking filtering. Figure 27This is a graph representing an example of BS values.

[0387] The block boundaries for deblocking filtering are, for example, Figure 26 The boundaries of the CU, PU, ​​or TU of the 8×8 pixel block are shown. Deblocking filtering is performed, for example, in units of 4 rows or 4 columns. First, for Figure 26 Blocks P and Q are shown, as follows Figure 27 That determines the Bs (Boundary Strength) value.

[0388] according to Figure 27 The Bs value determines whether deblocking filtering of different intensities is applied even to block boundaries belonging to the same image. A Bs value of 2 performs deblocking filtering on the chrominance signal. A Bs value of 1 or higher, meeting specified conditions, performs deblocking filtering on the luminance signal. Furthermore, the criteria for determining the Bs value are not limited to... Figure 27 The conditions shown can also be determined based on other parameters.

[0389] [Prediction Unit (Intra-frame Prediction Unit / Inter-frame Prediction Unit / Prediction Control Unit)] Figure 28 This is a flowchart illustrating an example of the processing performed by the prediction unit of the coding apparatus 100. Furthermore, as an example, the prediction unit is composed of all or part of the components of the intra-frame prediction unit 124, the inter-frame prediction unit 126, and the prediction control unit 128. The prediction processing unit includes, for example, the intra-frame prediction unit 124 and the inter-frame prediction unit 126.

[0390] The prediction unit generates a prediction image for the current block (step Sb_1). Furthermore, the prediction image may include, for example, an intra-frame prediction image (intra-frame prediction signal) or an inter-frame prediction image (inter-frame prediction signal). Specifically, the prediction unit generates a prediction image for the current block using a reconstructed image obtained by generating prediction images for other blocks, generating prediction residuals, generating quantization coefficients, restoring prediction residuals, and adding prediction images.

[0391] The reconstructed image can be, for example, an image of a reference image, or an image containing the current block, i.e., an image of the encoded blocks within the current image (i.e., the other blocks mentioned above). The encoded blocks within the current image can be, for example, adjacent blocks of the current block.

[0392] Figure 29 This is a flowchart illustrating another example of the processing performed by the prediction unit of the encoding device 100.

[0393] The prediction unit generates a prediction image using a first method (step Sc_1a), a second method (step Sc_1b), and a third method (step Sc_1c). The first, second, and third methods are different methods used to generate the prediction image, and can be, for example, inter-frame prediction, intra-frame prediction, and other prediction methods. In such prediction methods, the reconstructed image described above can also be used.

[0394] Next, the prediction unit evaluates the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_2). For example, the prediction unit calculates the cost C for the predicted images generated in each of steps Sc_1a, Sc_1b, and Sc_1c, and compares the costs C of these predicted images to evaluate them. The cost C is calculated using an RD optimization model formula, such as C = D + λ × R. In this formula, D is the coding distortion of the predicted image, and is represented, for example, by the sum of the absolute differences between the pixel values ​​of the current block and the pixel values ​​of the predicted image. Furthermore, R is the bit rate of the stream. Additionally, λ is, for example, an indeterminate Lagrange multiplier.

[0395] Next, the prediction unit selects one of the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_3). That is, the prediction unit selects a mode or method for obtaining the final predicted image. For example, the prediction unit selects the predicted image with the lowest cost C based on the cost C calculated for these predicted images. Alternatively, the evaluation in step Sc_2 and the selection of the predicted image in step Sc_3 can also be based on parameters used in the encoding process. The encoding device 100 can signal the information used to determine the selected predicted image, mode, or method into a stream. This information may be, for example, a flag. Thus, the decoding device 200 can generate a predicted image based on this information, according to the mode or method selected in the encoding device 100. Furthermore, in Figure 29 In the example shown, after generating prediction images through various methods, the prediction unit selects any one of the prediction images. However, before generating these prediction images, the prediction unit can select a method or mode based on the parameters used for the above encoding process, and can generate prediction images according to that method or mode.

[0396] For example, the first method and the second method are intra-frame prediction and inter-frame prediction, respectively, and the prediction unit can select the final prediction image for the current block from the prediction images generated according to these prediction methods.

[0397] Figure 30 This is a flowchart illustrating another example of the processing performed by the prediction unit of the encoding device 100.

[0398] First, the prediction unit generates a prediction image through intra-frame prediction (step Sd_1a) and an inter-frame prediction (step Sd_1b). Furthermore, the prediction image generated through intra-frame prediction is also referred to as the intra-frame prediction image, and the prediction image generated through inter-frame prediction is also referred to as the inter-frame prediction image.

[0399] Next, the prediction unit evaluates each of the intra-frame predicted image and the inter-frame predicted image (step Sd_2). The cost C described above can also be used in this evaluation. Then, the prediction unit selects the predicted image with the lowest calculated cost C from the intra-frame and inter-frame predicted images as the final predicted image for the current block (step Sd_3). That is, the prediction method or mode used to generate the predicted image for the current block is selected.

[0400] Intra-frame prediction unit The intra-frame prediction unit 124 performs intra-frame prediction (also called intra-picture prediction) of the current block by referring to the blocks in the current image stored in the block memory 118, thereby generating a predicted image of the current block (i.e., an intra-frame prediction image). Specifically, the intra-frame prediction unit 124 generates an intra-frame prediction image by performing intra-frame prediction by referring to the pixel values ​​(e.g., brightness values, color difference values) of blocks adjacent to the current block, and outputs the intra-frame prediction image to the prediction control unit 128.

[0401] For example, the intra-prediction unit 124 performs intra-prediction using one of a plurality of predefined intra-prediction modes. The plurality of intra-prediction modes typically include one or more non-directional prediction modes and multiple directional prediction modes.

[0402] More than one non-directional prediction mode, such as the Planar prediction mode and DC prediction mode as specified in the H.265 / HEVC specification.

[0403] Multiple directional prediction modes may include, for example, the 33-directional prediction modes specified in the H.265 / HEVC specification. Alternatively, multiple directional prediction modes may also include 32 additional-directional prediction modes (totaling 65 directional prediction modes). Figure 31 This diagram represents all 67 intra-prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in intra-frame prediction. Solid arrows represent the 33 directions specified by the H.265 / HEVC specification, and dashed arrows represent the additional 32 directions (2 non-directional prediction modes in...). Figure 31 (Not shown in the image).

[0404] In various installation examples, the luma block can also be referenced in the intra-frame prediction of the chroma block. That is, the chroma component of the current block can be predicted based on the luma component of the current block. Such intra-frame prediction is sometimes referred to as CCLM (cross-component linear model) prediction. This intra-frame prediction mode of the chroma block, which references the luma block (e.g., called CCLM mode), can also be added as one of the intra-frame prediction modes for the chroma block.

[0405] The intra-prediction unit 124 can also correct the intra-predicted pixel values ​​based on the gradient of the reference pixels in the horizontal / vertical directions. Intra-prediction accompanied by such correction is sometimes referred to as PDPC (position-dependent intraprediction combination). Information indicating whether PDPC has been used (e.g., a PDPC flag) is typically signaled at the CU level. However, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, image level, slice level, brick level, or CTU level).

[0406] Figure 32 This is a flowchart illustrating an example of the processing performed by the intra-frame prediction unit 124.

[0407] The intra-prediction unit 124 selects one intra-prediction mode from multiple intra-prediction modes (step Sw_1). Then, the intra-prediction unit 124 generates a predicted image according to the selected intra-prediction mode (step Sw_2). Next, the intra-prediction unit 124 determines the MPM (Most Probable Modes) (step Sw_3). The MPM consists of, for example, six intra-prediction modes. Two of these six intra-prediction modes can be Planar prediction modes and DC prediction modes, and the remaining four modes can be directional prediction modes. Then, the intra-prediction unit 124 determines whether the intra-prediction mode selected in step Sw_1 is included in the MPM (step Sw_4).

[0408] Here, when it is determined that the selected intra-prediction mode is included in the MPM (yes in step Sw_4), the intra-prediction unit 124 sets the MPM flag to 1 (step Sw_5) and generates information indicating the selected intra-prediction mode in the MPM (step Sw_6). In addition, the MPM flag set to 1 and the information indicating the intra-prediction mode are encoded as prediction parameters by the entropy coding unit 110.

[0409] On the other hand, when it is determined that the selected intra-prediction mode is not included in the MPM (no in step Sw_4), the intra-prediction unit 124 sets the MPM flag to 0 (step Sw_7). Alternatively, the intra-prediction unit 124 does not set the MPM flag. Then, the intra-prediction unit 124 generates information indicating the selected intra-prediction mode among one or more intra-prediction modes not included in the MPM (step Sw_8). In addition, the MPM flag set to 0 and the information indicating the intra-prediction mode are encoded as prediction parameters by the entropy coding unit 110. The information indicating the intra-prediction mode represents, for example, any value from 0 to 60.

[0410] [Inter-frame prediction unit] The inter-frame prediction unit 126 performs inter-frame prediction (also called inter-picture prediction) of the current block by referring to a reference image stored in the frame memory 122 that is different from the current image, thereby generating a predicted image (inter-frame prediction image). Inter-frame prediction is performed on a unit of the current block or the current sub-block within the current block. A sub-block is contained within a block and is a unit smaller than the block. The size of a sub-block can be 4x4 pixels, 8x8 pixels, or other sizes. The size of the sub-block can also be switched in units such as slices, bricks, or images.

[0411] For example, the inter-frame prediction unit 126 performs motion estimation within a reference image for the current block or sub-block to find the reference block or sub-block that best matches the current block or sub-block. Furthermore, the inter-frame prediction unit 126 acquires motion information (e.g., motion vectors) to compensate for motion or changes from the reference block or sub-block to the current block or sub-block. Based on this motion information, the inter-frame prediction unit 126 performs motion compensation (or motion prediction) to generate an inter-frame prediction image for the current block or sub-block. Finally, the inter-frame prediction unit 126 outputs the generated inter-frame prediction image to the prediction control unit 128.

[0412] Motion information used in motion compensation is signaled in various forms as inter-frame predicted images. For example, motion vectors can also be signaled. As another example, the difference between the motion vector and the predicted motion vector can also be signaled.

[0413] [List of reference images] Figure 33 This is a diagram representing one example from each reference image. Figure 34 This is a conceptual diagram representing an example of a reference image list. The reference image list represents a list of one or more reference images stored in the frame memory 122. Additionally, in Figure 33In the diagram, rectangles represent images, arrows indicate the reference relationships between images, the horizontal axis represents time, and the I, P, and B symbols within the rectangles represent intra-frame predicted images, single-predicted images, and double-predicted images, respectively. The numbers within the rectangles indicate the decoding order. For example... Figure 33 As shown, the decoding order of each image is I0, P1, B2, B3, B4, and the display order of each image is I0, B3, B2, B4, P1. Figure 34 As shown, the reference image list is a list of candidate reference images. For example, one image (or slice) can have more than one reference image list. For instance, if the current image is a single-prediction image, one reference image list is used; if the current image is a double-prediction image, two reference image lists are used. Figure 33 and Figure 34 In the example, image B3, which is the current image currPic, has two reference image lists: L0 and L1. When the current image currPic is image B3, the candidate reference images for currPic are I0, P1, and B2, and each reference image list (i.e., L0 and L1 lists) represents these images. The inter-frame prediction unit 126 or the prediction control unit 128 specifies which image in each reference image list to actually reference using the reference image index refidxLx. Figure 34 In the text, reference images P1 and B2 are specified by referring to image indices refIdxL0 and refIdxL1.

[0414] Such a reference image list can be generated at the sequence unit, picture unit, slice unit, brick unit, CTU unit, or CU unit. Alternatively, the reference image index representing the reference image in the reference image list that is referenced in inter-frame prediction can be encoded at the sequence level, picture level, slice level, brick level, CTU level, or CU level. Furthermore, a common reference image list can be used across multiple inter-frame prediction modes.

[0415] [Basic process of inter-frame prediction] Figure 35 This is a flowchart representing the basic process of inter-frame prediction.

[0416] The inter-frame prediction unit 126 first generates a prediction image (steps Se_1 to Se_3). Next, the subtraction unit 104 generates the difference between the current block and the prediction image as the prediction residual (step Se_4).

[0417] Here, in generating the predicted image, the inter-frame prediction unit 126 generates the predicted image, for example, by determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and performing motion compensation (step Se_3). Furthermore, in determining the MV, the inter-frame prediction unit 126 determines the MV, for example, by selecting candidate motion vectors (candidate MVs) (step Se_1) and deriving the MV (step Se_2). The selection of candidate MVs is performed, for example, by the inter-frame prediction unit 126 generating a candidate MV list and selecting at least one candidate MV from the candidate MV list. Additionally, previously derived MVs can be added to the candidate MV list. Furthermore, in MV derivation, the inter-frame prediction unit 126 can also determine the MV of the current block by further selecting at least one candidate MV from the at least one candidate MV. Alternatively, the inter-frame prediction unit 126 can determine the MV of the current block by searching the region of the reference image indicated by each of the at least one selected candidate MV. Alternatively, the action of searching for the region of the reference image can also be called motion estimation.

[0418] Furthermore, in the above example, steps Se_1 to Se_3 are performed by the inter-frame prediction unit 126, but the processing of steps Se_1 or Se_2, for example, can also be performed by other components included in the encoding device 100.

[0419] Additionally, a candidate MV list can be created for each process in each inter-frame prediction mode, or a common candidate MV list can be used across multiple inter-frame prediction modes. Furthermore, the processes in steps Se_3 and Se_4 are respectively equivalent to... Figure 9 The processes shown are Sa_3 and Sa_4. Additionally, the process in step Se_3 is equivalent to... Figure 30 The processing of step Sd_1b.

[0420] [MV export process] Figure 36 This is a flowchart representing an example of MV export.

[0421] The inter-frame prediction unit 126 can derive the MV of the current block in a mode that encodes motion information (e.g., MV). In this case, for example, the motion information can be encoded as prediction parameters and signaled. That is, the encoded motion information is included in the stream.

[0422] Alternatively, the inter-frame prediction unit 126 can derive the MV in a mode where motion information is not encoded. In this case, the motion information is not included in the stream.

[0423] Here, the modes for MV derivation include the ordinary inter-frame mode, ordinary merging mode, FRUC mode, and affine mode, which will be described later. Among these modes, those that encode motion information include the ordinary inter-frame mode, ordinary merging mode, and affine mode (specifically, affine inter-frame mode and affine merging mode). Furthermore, the motion information can include not only MV but also the predicted MV selection information, which will be described later. In addition, modes that do not encode motion information include the FRUC mode, etc. The inter-frame prediction unit 126 selects a mode from these multiple modes for deriving the MV of the current block and derives the MV of the current block using the selected mode.

[0424] Figure 37 This is a flowchart representing another example of MV export.

[0425] The inter-frame prediction unit 126 can derive the MV of the current block in a mode that encodes the differential MV. In this case, for example, the differential MV is encoded as prediction parameters and signaled. That is, the encoded differential MV is contained in the stream. This differential MV is the difference between the MV of the current block and its predicted MV. Furthermore, the predicted MV is a predicted motion vector.

[0426] Alternatively, the inter-frame prediction unit 126 can derive the MV in a mode where the differential MV is not encoded. In this case, the encoded differential MV is not included in the stream.

[0427] Here, as described above, the MV derivation modes include the normal inter-frame mode, normal merging mode, FRUC mode, and affine mode, which will be described later. Among these modes, the modes that encode the differential MV include the normal inter-frame mode and the affine mode (specifically, the affine inter-frame mode). Furthermore, the modes that do not encode the differential MV include the FRUC mode, the normal merging mode, and the affine mode (specifically, the affine merging mode). The inter-frame prediction unit 126 selects a mode from these multiple modes for deriving the MV of the current block, and uses the selected mode to derive the MV of the current block.

[0428] [MV export mode] Figure 38A and Figure 38B This is a diagram illustrating an example of the classification of modes derived from MV. For example, as... Figure 38A As shown, based on whether motion information and differential MV are encoded, the MV-derived modes are classified into three main modes: inter-frame mode, merge mode, and FRUC (frame rate up-conversion) mode. Inter-frame mode is the mode for motion search, which encodes both motion information and differential MV. For example, as... Figure 38BAs shown, inter-frame modes include affine inter-frame mode and normal inter-frame mode. Merging mode is a mode that does not perform motion search; it selects the motion video (MV) from neighboring encoded blocks and uses that MV to derive the MV of the current block. This merging mode essentially encodes motion information without encoding the differential MV. For example, as... Figure 38B As shown, the merging modes include ordinary merging mode (sometimes also called normal merging mode or regular merging mode), MMVD (Merge with Motion Vector Difference) mode, CIIP (Combined inter merge / intraprediction) mode, triangular mode, ATMVP mode, and affine merging mode. Here, in the MMVD mode among the various merging modes, differential MVs are exceptionally encoded. Furthermore, the aforementioned affine merging mode and affine inter-frame mode are modes included in the affine mode. The affine mode assumes an affine transformation, deriving the MVs of multiple sub-blocks constituting the current block as the MV of the current block. The FRUC mode derives the MV of the current block by searching among the encoded regions; it is a mode in which neither motion information nor differential MVs are encoded. Further details about these modes will be described later.

[0429] in addition, Figure 38A and Figure 38B The classifications of the modes shown are examples, and are not limited to these. For instance, when differential MV is encoded in CIIP mode, the CIIP mode is classified as an inter-frame mode.

[0430] [MV Export > Normal Inter-Frame Mode] The normal inter-frame mode is an inter-frame prediction mode based on finding blocks similar to the current block's image from regions of a reference image represented by candidate MVs to derive the MV of the current block. Furthermore, in this normal inter-frame mode, differential MVs are encoded.

[0431] Figure 39 This is a flowchart illustrating an example of inter-frame prediction based on a normal inter-frame pattern.

[0432] First, the inter-frame prediction unit 126 obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks located around the current block in time or space (step Sg_1). That is, the inter-frame prediction unit 126 creates a candidate MV list.

[0433] Next, the inter-frame prediction unit 126 extracts N (N being an integer greater than 2) candidate MVs from the plurality of candidate MVs obtained in step Sg_1, respectively, as prediction MV candidates, according to a predetermined priority order (step Sg_2). Furthermore, this priority order is predetermined for each of the N candidate MVs.

[0434] Next, the inter-frame prediction unit 126 selects one prediction MV candidate from the N prediction MV candidates as the prediction MV of the current block (step Sg_3). At this time, the inter-frame prediction unit 126 encodes the prediction MV selection information used to identify the selected prediction MV into the stream. That is, the inter-frame prediction unit 126 outputs the prediction MV selection information as prediction parameters to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0435] Next, the inter-frame prediction unit 126, referring to the encoded reference image, derives the MV of the current block (step Sg_4). At this time, the inter-frame prediction unit 126 also encodes the difference between the derived MV and the predicted MV as a differential MV into the stream. That is, the inter-frame prediction unit 126 outputs the differential MV as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130. Furthermore, the encoded reference image is an image composed of multiple blocks reconstructed after encoding.

[0436] Finally, the inter-frame prediction unit 126 generates a predicted image for the current block by performing motion compensation on the current block using the derived MV and the encoded reference image (step Sg_5). The processing of steps Sg_1 to Sg_5 is performed on each block. For example, when the processing of steps Sg_1 to Sg_5 is performed on all blocks contained in a slice, the inter-frame prediction using the normal inter-frame mode for that slice ends. Similarly, when the processing of steps Sg_1 to Sg_5 is performed on all blocks contained in an image, the inter-frame prediction using the normal inter-frame mode for that image ends. Alternatively, the processing of steps Sg_1 to Sg_5 may be performed on only a portion of the blocks contained in a slice, at which point the inter-frame prediction using the normal inter-frame mode for that slice ends. Likewise, the processing of steps Sg_1 to Sg_5 may be performed on only a portion of the blocks contained in an image, at which point the inter-frame prediction using the normal inter-frame mode for that image ends.

[0437] Furthermore, the predicted image is the inter-frame prediction signal described above. Additionally, information contained in the encoded signal representing the inter-frame prediction mode used in the generation of the predicted image (in the example above, the ordinary inter-frame mode) is encoded as, for example, prediction parameters.

[0438] Furthermore, the candidate MV list can be used interchangeably with lists used in other patterns. Additionally, processing related to the candidate MV list can be applied to processing related to lists used in other patterns. Processes related to the candidate MV list include, for example, extracting or selecting candidate MVs, rearranging candidate MVs, or deleting candidate MVs.

[0439] [MV Export > Normal Merge Mode] The normal merging mode derives the inter-frame prediction mode of a MV by selecting a candidate MV from a list of candidate MVs as the MV of the current block. However, the normal merging mode is a narrower definition of the merging mode and is sometimes simply referred to as the merging mode. In this embodiment, the normal merging mode and the merging mode are distinguished, and the merging mode is used in a broader sense.

[0440] Figure 40 This is a flowchart illustrating an example of inter-frame prediction based on a normal merging pattern.

[0441] First, the inter-frame prediction unit 126 obtains multiple candidate MVs for the current block based on information such as multiple encoded block MVs located around the current block in time or space (step Sh_1). That is, the inter-frame prediction unit 126 creates a candidate MV list.

[0442] Next, the inter-frame prediction unit 126 selects one candidate MV from the multiple candidate MVs obtained in step Sh_1 and derives the MV of the current block (step Sh_2). At this time, the inter-frame prediction unit 126 encodes the MV selection information used to identify the selected candidate MV into the stream. That is, the inter-frame prediction unit 126 outputs the MV selection information as prediction parameters to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0443] Finally, the inter-frame prediction unit 126 generates a predicted image for the current block by performing motion compensation on the current block using the derived MV and the encoded reference image (step Sh_3). The processing of steps Sh_1 to Sh_3 is performed on each block, for example. For instance, when the processing of steps Sh_1 to Sh_3 is performed on all blocks contained in a slice, the inter-frame prediction using the normal merging mode for that slice ends. Similarly, when the processing of steps Sh_1 to Sh_3 is performed on all blocks contained in an image, the inter-frame prediction using the normal merging mode for that image ends. Furthermore, the processing of steps Sh_1 to Sh_3 can also be performed on a portion of the blocks instead of all blocks contained in a slice, at which point the inter-frame prediction using the normal merging mode for that slice ends. Likewise, the processing of steps Sh_1 to Sh_3 can also be performed on a portion of the blocks contained in an image, at which point the inter-frame prediction using the normal merging mode for that image ends.

[0444] Furthermore, information contained in the stream representing the inter-frame prediction mode used in the generation of the predicted image (in the example above, the normal merging mode) is encoded as, for example, prediction parameters.

[0445] Figure 41 This is a diagram illustrating an example of MV export processing for the current image based on the normal merging mode.

[0446] First, the inter-frame prediction unit 126 generates a list of candidate MVs that have been registered as candidate MVs. The candidate MVs include: spatially adjacent candidate MVs, which are MVs possessed by multiple encoded blocks located spatially adjacent to the current block; temporally adjacent candidate MVs, which are MVs possessed by nearby blocks whose positions in the encoded reference image have been projected onto the current block; combined candidate MVs, which are MVs generated by combining the MV values ​​of spatially adjacent candidate MVs and temporally adjacent candidate MVs; and zero candidate MVs, which are MVs with a value of zero, etc.

[0447] Next, the inter-frame prediction unit 126 selects one candidate MV from the multiple candidate MVs registered in the candidate MV list and determines that one candidate MV as the MV of the current block.

[0448] Furthermore, in the entropy coding unit 110, the signal indicating which candidate MV was selected, namely merge_idx, is encoded in the stream.

[0449] In addition, Figure 41 The candidate MVs registered in the candidate MV list described in the figure are one example. They may also be a different number than the number shown in the figure, or a structure that does not include a portion of the candidate MVs in the figure, or a structure that adds candidate MVs other than the candidate MVs in the figure.

[0450] Alternatively, the MV of the current block derived through normal merge mode can be used for DMVR (dynamic motion vector refreshing), as described later, to determine the final MV. Furthermore, in normal merge mode, the differential MV is not encoded, but in MMVD mode, the differential MV is encoded. MMVD mode selects one candidate MV from the candidate MV list, just like normal merge mode, but it encodes the differential MV. For example... Figure 38B As shown, such MMVD can also be classified as a merge mode along with the normal merge mode. Furthermore, the differential MV in MMVD mode may not be the same as the differential MV used in inter-frame mode; for example, the derivation of differential MV in MMVD mode can be a process with less processing power than the derivation of differential MV in inter-frame mode.

[0451] Alternatively, the prediction image generated in inter-frame prediction can be made to overlap with the prediction image generated in intra-frame prediction, thus generating the prediction image for the current block in CIIP (Combined inter merge / intra prediction) mode.

[0452] Alternatively, the candidate MV list can also be called the candidate list. Furthermore, merge_idx contains MV selection information.

[0453] [MV Export > HMVP Mode] Figure 42 This is a diagram illustrating an example of MV export processing for the current image based on the HMVP pattern.

[0454] In the normal merge mode, one candidate MV is selected from a list of candidate MVs generated by referencing an already encoded block (e.g., a CU), thus determining the MV of the current block, such as the CU. Alternatively, other candidate MVs can be registered in this list. The mode that registers such other candidate MVs is called the HMVP mode.

[0455] In HMVP mode, candidate MVs are managed using a FIFO (First-In First-Out) buffer, separate from the candidate MV list in normal merge mode.

[0456] In the FIFO buffer, motion information such as the MV of previously processed blocks is sequentially stored from the new FIFO buffer. In the management of this FIFO buffer, whenever a block is processed, the MV of the latest block (i.e., the immediately preceding CU) is stored in the FIFO buffer, and instead, the MV of the oldest CU (i.e., the first CU processed) is deleted from the FIFO buffer. Figure 42 In the example shown, HMVP1 is the MV of the latest block, and HMVP5 is the MV of the earliest block.

[0457] Then, for example, the inter-frame prediction unit 126 checks, starting from HMVP1, each MV managed in the FIFO buffer to see if it is different from all candidate MVs already registered in the candidate MV list of the normal merging mode. Furthermore, if it determines that the MV managed in the FIFO buffer is different from all candidate MVs, the inter-frame prediction unit 126 may add the MV managed in the FIFO buffer as a candidate MV to the candidate MV list of the normal merging mode. At this time, there may be one or more candidate MVs registered from the FIFO buffer.

[0458] In this way, by using the HMVP pattern, not only can the MVs of spatially or temporally adjacent blocks of the current block be added to the candidates, but also the MVs of previously processed blocks can be added to the candidates. As a result, by expanding the variation of candidate MVs in the ordinary merge pattern, the possibility of improving coding efficiency increases.

[0459] Furthermore, the aforementioned MV can also be motion information. That is, the information stored in the candidate MV list and the FIFO buffer can include not only the MV value, but also information indicating the referenced image, the reference direction, and the number of images, etc. Additionally, the aforementioned block is, for example, a CU.

[0460] in addition, Figure 42 The candidate MV list and FIFO buffer are one example; the candidate MV list and FIFO buffer can also be... Figure 42 Lists or buffers of different sizes, or those with... Figure 42 The structure of candidate MVs is registered in different orders. Furthermore, the processing described here is common to both the encoding device 100 and the decoding device 200.

[0461] Furthermore, the HMVP pattern can also be applied to patterns other than the normal merge pattern. For example, motion information such as the MV of blocks previously processed in affine mode can be sequentially saved from a new FIFO buffer and used as candidate MVs. The pattern that applies the HMVP pattern within affine mode can also be called the historical affine pattern.

[0462] [MV Export > FRUC Mode] Motion information can also be derived from the decoding device 200 instead of being signaled from the encoding device 100 side. For example, motion information can be derived by performing motion search on the decoding device 200 side. In this case, motion search is performed on the decoding device 200 side without using the pixel values ​​of the current block. Such modes of performing motion search on the decoding device 200 side include FRUC (frame rate up-conversion) mode or PMMVD (pattern matched motion vector derivation) mode, etc.

[0463] Figure 43The diagram illustrates an example of FRUC processing. First, referencing the MVs of each coded block spatially or temporally adjacent to the current block, a list is generated to represent these MVs as candidate MVs (i.e., a candidate MV list, which can also be common to the candidate MV list of the normal merge mode) (step Si_1). Next, the best candidate MV is selected from the multiple candidate MVs registered in the candidate MV list (step Si_2). For example, the evaluation value of each candidate MV included in the candidate MV list is calculated, and one candidate is selected as the best candidate MV based on the evaluation value. Then, based on the selected best candidate MV, the MV for the current block is derived (step Si_4). Specifically, for example, the selected best candidate MV is derived as is as the MV for the current block. Alternatively, for example, the MV for the current block can be derived by performing style matching in the surrounding region of the position in the reference image corresponding to the selected best candidate MV. That is, the surrounding region of the best candidate MV can also be searched using style matching in the reference image and the evaluation value, and if there is an MV with a better evaluation value, the best candidate MV is updated to that MV, and it is used as the final MV for the current block. Alternatively, updating to the MV with a better evaluation value may not be implemented.

[0464] Finally, the inter-frame prediction unit 126 generates a predicted image for the current block by performing motion compensation on the current block using the derived MV and the encoded reference image (step Si_5). The processing of steps Si_1 to Si_5 is performed on each block, for example. For instance, when the processing of steps Si_1 to Si_5 is performed on all blocks contained in a slice, the inter-frame prediction using the FRUC mode for that slice ends. Similarly, when the processing of steps Si_1 to Si_5 is performed on all blocks contained in an image, the inter-frame prediction using the FRUC mode for that image ends. Alternatively, the processing of steps Si_1 to Si_5 may also be performed on a subset of blocks instead of all blocks contained in a slice, at which point the inter-frame prediction using the FRUC mode for that slice ends. Likewise, the processing of steps Si_1 to Si_5 may also be performed on a subset of blocks contained in an image, at which point the inter-frame prediction using the FRUC mode for that image ends.

[0465] The same processing as that for block units can be performed when processing is done in sub-block units.

[0466] The evaluation value can also be calculated using various methods. For example, a reconstructed image of a region within a reference image corresponding to the MV can be compared with a reconstructed image of a specified region (e.g., as shown below, this region could be a region of another reference image or a region of a neighboring block in the current image). Then, the difference in pixel values ​​between the two reconstructed images can be calculated as the evaluation value for the MV. Alternatively, other information besides the difference value can be used to calculate the evaluation value.

[0467] Next, we will explain style matching in detail. First, select one candidate MV from the candidate MV list (also known as the merge list) as the starting point for the style matching-based search. Style matching can be either first-order style matching or second-order style matching. First-order style matching and second-order style matching can be referred to as bilateral matching and template matching, respectively.

[0468] [MV Export > FRUC > Two-way Matching] In the first style matching, style matching is performed between two blocks within two different reference images, along the motion trajectory of the current block. Therefore, in the first style matching, the region within other reference images along the motion trajectory of the current block is used as the defined region for calculating the evaluation value of the candidate MV.

[0469] Figure 44 This is an example of a first-style matching (bidirectional matching) between two blocks in two reference images along a motion trajectory. For example... Figure 44 As shown, in the first style matching, two MVs (MV0, MV1) are derived by searching for the best matching pair among two blocks within two different reference images (Ref0, Ref1) along the motion trajectory of the current block. Specifically, for the current block, the difference between the reconstructed image at a specified position within the first encoded reference image (Ref0) specified by the candidate MV and the reconstructed image at a specified position within the second encoded reference image (Ref1) specified by the symmetrical MV scaled by the display time interval is derived, and the resulting difference value is used to calculate an evaluation value. The candidate MV with the best evaluation value can be selected as the best candidate MV from among multiple candidate MVs.

[0470] Under the assumption of continuous motion trajectories, the MV (MV0, MV1) indicating two reference blocks is proportional to the temporal distance (TD0, TD1) between the current image (Cur Pic) and the two reference images (Ref0, Ref1). For example, in the case where the current image is located between the two reference images in time and the temporal distance from the current image to the two reference images is equal, a mirror-symmetric bidirectional MV is derived in the first style match.

[0471] [MV Export > FRUC > Template Matching] In the second style matching (template matching), style matching is performed between the template in the current image (the block adjacent to the current block in the current image (e.g., the top and / or left adjacent block)) and the block in the reference image. Therefore, in the second style matching, the block adjacent to the current block in the current image is used as the defined area for calculating the evaluation value of the candidate MV.

[0472] Figure 45 This is an example of style matching (template matching) between a template in the current image and a block in a reference image. For example... Figure 45 As shown, in the second style matching, the MV of the current block is derived by searching within the reference image (Ref0) for the block that best matches the block adjacent to the current block (Cur block) within the current image (Cur Pic). Specifically, for the current block, the difference between the reconstructed images of the encoded regions of the left and top adjacent regions or one of them and the reconstructed image at the same position within the encoded reference image (Ref0) specified by the candidate MV is derived, and the obtained difference value is used to calculate the evaluation value. The candidate MV with the best evaluation value among multiple candidate MVs can be selected as the best candidate MV.

[0473] Information indicating whether FRUC mode is used (e.g., referred to as the FRUC flag) is signaled at the CU level. Furthermore, when FRUC mode is used (e.g., when the FRUC flag is true), information indicating the available pattern matching method (first pattern matching or second pattern matching) is signaled at the CU level. Additionally, the signaling of this information is not limited to the CU level and can also be at other levels (e.g., sequence level, image level, slice level, brick level, CTU level, or sub-block level).

[0474] [MV Export > Affine Mode] Affine mode is a mode that uses affine transformations to generate motion representations (MVs). For example, MVs can also be derived on a sub-block basis based on the MVs of multiple adjacent blocks. This mode is sometimes referred to as affine motion compensation prediction mode.

[0475] Figure 46A This is a diagram illustrating an example of deriving the MV from sub-block units based on the MV of multiple adjacent blocks. Figure 46A In this context, the current block may consist of 16 sub-blocks, each composed of 4×4 pixels. Here, the motion vector v0 of the top-left control point of the current block is derived based on the MV of adjacent blocks, and similarly, the motion vector v1 of the top-right control point of the current block is derived based on the MV of adjacent sub-blocks. Then, according to the following equation (1A), the two motion vectors v0 and v1 are projected, and the motion vectors (v0, v1, v1) of each sub-block within the current block are derived. x v y ).

[0476]

Formula 1

[0477] Information representing this affine pattern (e.g., referred to as an affine flag) can be signaled at the CU level. Furthermore, the signaling of information representing this affine pattern is not limited to the CU level and can be at other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or sub-block level).

[0478] Furthermore, such affine modes can also include several modes with different methods for deriving the MV of the upper left and upper right control points. For example, in affine modes, there are two modes: affine inter-frame (also known as affine normal inter-frame) mode and affine merge mode.

[0479] Figure 46B This is a diagram illustrating an example of MV derivation for a sub-block unit in an affine pattern using 3 control points. Figure 46B In this context, the current block comprises 16 sub-blocks of 4×4 pixels each. Here, the motion vector v0 of the top-left control point of the current block is derived based on the MV of neighboring blocks. Similarly, the motion vector v1 of the top-right control point of the current block is derived based on the MV of neighboring blocks, and the motion vector v2 of the bottom-left control point of the current block is derived based on the MV of neighboring blocks. Then, according to the following equation (1B), the three motion vectors v0, v1, and v2 are projected to derive the motion vector (v0) of each sub-block within the current block. x v y ).

[0480]

Formula 2

[0481] Affine patterns using different numbers of control points (e.g., 2 and 3) can also be signaled at the CU level for switching. Alternatively, information indicating the number of control points for the affine pattern used at the CU level can be signaled at other levels (e.g., sequence level, picture level, slice level, brick level, CTU level, or sub-block level).

[0482] Furthermore, in such an affine mode with three control points, there can also be several modes with different methods for deriving the MV of the upper left, upper right, and lower left control points. For example, in the affine mode with three control points, similar to the affine mode with two control points, there are two modes: affine inter-frame mode and affine merge mode.

[0483] Additionally, in affine mode, the size of each sub-block contained within the current block is not limited to 4x4 pixels; it can also be other sizes. For example, the size of each sub-block can also be 8x8 pixels.

[0484] [MV Export > Affine Mode > Control Points] Figure 47A , Figure 47B and Figure 47C This is a conceptual diagram used to illustrate an example of MV derivation of control points in affine mode.

[0485] In affine mode, such as Figure 47A As shown, for example, based on multiple MVs corresponding to the affine-coded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) adjacent to the current block, the predicted MV of each of the control points in the current block is calculated. Specifically, these blocks are examined in the order of the encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) to determine the initial valid blocks encoded in affine mode. The MV of the control points of the current block is calculated based on the multiple MVs corresponding to the determined blocks.

[0486] For example, such as Figure 47B As shown, when encoding block A, which is adjacent to the left side of the current block, in an affine pattern with two control points, motion vectors v3 and v4 are derived, projected onto the upper left and upper right corners of the encoded block containing block A. Then, based on the derived motion vectors v3 and v4, the motion vector v0 of the upper left control point and the motion vector v1 of the upper right control point of the current block are calculated.

[0487] For example, such as Figure 47CAs shown, when encoding block A, which is adjacent to the left side of the current block, in affine mode with 3 control points, motion vectors v3, v4, and v5 are derived, projected onto the positions of the top-left, top-right, and bottom-left corners of the encoded block containing block A. Then, based on the derived motion vectors v3, v4, and v5, the motion vector v0 of the top-left control point, the motion vector v1 of the top-right control point, and the motion vector v2 of the bottom-left control point of the current block are calculated.

[0488] in addition, Figures 47A to 47C The method for exporting the MV shown can be used for the purposes described later. Figure 50 The deriving of the MV of each control point of the current block in step Sk_1 shown can also be used in the following discussion. Figure 51 The step Sj_1 shows the derivation of the predicted MV for each control point of the current block.

[0489] Figure 48A and Figure 48B This is a conceptual diagram used to illustrate another example of deriving the control point MV in an affine mode.

[0490] Figure 48A This is a diagram used to illustrate an affine pattern with two control points.

[0491] In this affine mode, such as Figure 48A As shown, the MV selected from the MVs of the coded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the upper left control point of the current block. Similarly, the MV selected from the MVs of the coded blocks D and E adjacent to the current block is used as the motion vector v1 of the upper right control point of the current block.

[0492] Figure 48B This is a diagram used to illustrate an affine pattern with three control points.

[0493] In this affine mode, such as Figure 48B As shown, the MV selected from the MVs of the previously encoded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the top-left control point of the current block. Similarly, the MV selected from the MVs of the previously encoded blocks D and E adjacent to the current block is used as the motion vector v1 of the top-right control point of the current block. Furthermore, the MV selected from the MVs of the previously encoded blocks F and G adjacent to the current block is used as the motion vector v2 of the bottom-left control point of the current block.

[0494] in addition, Figure 48A and Figure 48B The method for exporting the MV shown can be used in the following sections. Figure 50 The deriving of the MV of each control point of the current block in step Sk_1 shown can also be used for the following description. Figure 51The predicted MV of each control point in the current block is derived in step Sj_1.

[0495] Here, for example, in the case of signaling in affine modes where different numbers of control points (e.g., 2 and 3) are switched at the CU level, the number of control points may vary depending on the coded block and the current block.

[0496] Figure 49A and Figure 49B This is a conceptual diagram illustrating an example of a method for deriving the MV of control points when the number of control points differs between an encoded block and the current block.

[0497] For example, such as Figure 49A As shown, the current block has three control points: the top-left corner, the top-right corner, and the bottom-left corner. Block A, adjacent to the left of the current block, is encoded in an affine pattern with two control points. In this case, motion vectors v3 and v4 are derived, projected onto the top-left and top-right corners of the encoded block containing block A. Then, based on the derived motion vectors v3 and v4, the motion vector v0 for the top-left control point and the motion vector v1 for the top-right control point of the current block are calculated. Furthermore, based on the derived motion vectors v0 and v1, the motion vector v2 for the bottom-left control point is calculated.

[0498] For example, such as Figure 49B As shown, the current block has two control points, the top left and the top right. Block A, which is adjacent to the left of the current block, is encoded in an affine pattern with three control points. In this case, motion vectors v3, v4, and v5 are derived, projected onto the top left, top right, and bottom left corners of the encoded block containing block A. Then, based on the derived motion vectors v3, v4, and v5, the motion vector v0 of the top left control point and the motion vector v1 of the top right control point of the current block are calculated.

[0499] in addition, Figure 49A and Figure 49B The method for exporting the MV shown can be used in the following sections. Figure 50 The deriving of the MV of each control point of the current block in step Sk_1 shown can also be used for the following description. Figure 51 The predicted MV of each control point in the current block is derived in step Sj_1.

[0500] [MV Export > Affine Mode > Affine Merge Mode] Figure 50 This is a flowchart representing an example of an affine merge pattern.

[0501] In affine merging mode, firstly, the inter-frame prediction unit 126 derives the MV of each control point for the current block (step Sk_1). The control points are as follows: Figure 46AAs shown, these are the top left and top right corners of the current block, or as... Figure 46B The diagram shows the top-left, top-right, and bottom-left corners of the current block. At this time, the inter-frame prediction unit 126 can also encode MV selection information used to identify the derived two or three MVs into the stream.

[0502] For example, in use Figures 47A to 47C In the case of the MV export method shown, such as Figure 47A As shown, the inter-frame prediction unit 126 checks these blocks in the order of the encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) and determines the initial valid blocks encoded in affine mode.

[0503] The inter-frame prediction unit 126 uses the first valid block encoded in the determined affine mode to derive the MV of the control points. For example, when block A is determined and block A has 2 control points, as follows... Figure 47B As shown, the inter-frame prediction unit 126 calculates the motion vector v0 of the top-left control point and the motion vector v1 of the top-right control point of the current block based on the motion vectors v3 and v4 of the top-left and top-right corners of the encoded block containing block A. For example, by projecting the motion vectors v3 and v4 of the top-left and top-right corners of the encoded block onto the current block, the inter-frame prediction unit 126 calculates the motion vector v0 of the top-left control point and the motion vector v1 of the top-right control point of the current block.

[0504] Alternatively, if block A is determined and block A has 3 control points, such as Figure 47C As shown, the inter-frame prediction unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block based on the motion vectors v3, v4, and v5 of the upper left, upper right, and lower left corners of the encoded block containing block A. For example, by projecting the motion vectors v3, v4, and v5 of the upper left, upper right, and lower left corners of the encoded block onto the current block, the inter-frame prediction unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block.

[0505] Alternatively, it can be as described above. Figure 49A As shown, block A is determined, and the MV of the three control points is calculated when block A has two control points. This can also be done as described above. Figure 49B As shown, block A is determined, and if block A has 3 control points, the MV of 2 control points is calculated.

[0506] Next, the inter-frame prediction unit 126 performs motion compensation on each of the multiple sub-blocks contained in the current block. That is, for each of the multiple sub-blocks, the inter-frame prediction unit 126 calculates the MV of the sub-block as an affine MV using two motion vectors v0 and v1 and the above-described equation (1A), or using three motion vectors v0, v1, and v2 and the above-described equation (1B) (step Sk_2). Then, the inter-frame prediction unit 126 performs motion compensation on the sub-block using the affine MV and the encoded reference image (step Sk_3). When steps Sk_2 and Sk_3 are performed on all sub-blocks contained in the current block respectively, the process of generating the prediction image of the current block using the affine merging mode ends. That is, motion compensation is performed on the current block, and the prediction image of the current block is generated.

[0507] Furthermore, in step Sk_1, the aforementioned candidate MV list can also be generated. The candidate MV list could, for example, be a list containing candidate MVs exported using multiple MV export methods for each control point. Multiple MV export methods could be... Figures 47A to 47C The method for exporting the MV shown. Figure 48A and Figure 48B The method for exporting the MV shown. Figure 49A and Figure 49B The export method of the MV shown, as well as any combination of other MV export methods.

[0508] In addition, the candidate MV list can also include candidate MVs for patterns other than affine patterns that are predicted in sub-block units.

[0509] Alternatively, as a candidate MV list, for example, a candidate MV list containing candidate MVs with 2 control points and a candidate MV with 3 control points can be generated. Alternatively, a candidate MV list containing candidate MVs with 2 control points and a candidate MV list containing candidate MVs with 3 control points can be generated separately. Alternatively, a candidate MV list containing candidate MVs of either the 2-control-point affine merging pattern or the 3-control-point affine merging pattern can be generated. Candidate MVs can be, for example, the MVs of encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left), or the MVs of valid blocks among these blocks.

[0510] In addition, as MV selection information, an index indicating which candidate MV is in the candidate MV list can also be sent.

[0511] [MV Export > Affine Mode > Affine Inter-Frame Mode] Figure 51 This is a flowchart representing an example of an affine inter-frame mode.

[0512] In affine inter-frame mode, firstly, the inter-frame prediction unit 126 derives the prediction MV(v0, v1) or (v0, v1, v2) for each of the two or three control points of the current block (step Sj_1). For example... Figure 46A or Figure 46B As shown, the control point is the top left, top right, or bottom left corner of the current block.

[0513] For example, in use Figure 48A and Figure 48B In the case of the MV export method shown, the inter-frame prediction unit 126 selects... Figure 48A or Figure 48B The MV of a certain block among the coded blocks near each control point of the current block is shown, and the predicted MV (v0, v1) or (v0, v1, v2) of the control point of the current block is derived. At this time, the inter-frame prediction unit 126 encodes the predicted MV selection information used to identify the selected 2 or 3 predicted MVs into the stream.

[0514] For example, the inter-frame prediction unit 126 can determine which block's MV to select as the control point's prediction MV from the encoded blocks adjacent to the current block by using cost evaluation or the like, and can record a flag indicating which prediction MV was selected in the bitstream. That is, the inter-frame prediction unit 126 outputs the prediction MV selection information, such as the flag, as prediction parameters to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0515] Next, the inter-frame prediction unit 126 performs motion search (steps Sj_3 and Sj_4) while updating the predicted MV selected or derived in step Sj_1 (step Sj_2). That is, the inter-frame prediction unit 126 uses the MV of each sub-block corresponding to the predicted MV to be updated as an affine MV and calculates it using the above-described equation (1A) or equation (1B) (step Sj_3). Then, the inter-frame prediction unit 126 uses these affine MVs and the encoded reference image to perform motion compensation on each sub-block (step Sj_4). Whenever the predicted MV is updated in step Sj_2, the processing of steps Sj_3 and Sj_4 is performed on all blocks within the current block. As a result, in the motion search loop, the inter-frame prediction unit 126, for example, determines the predicted MV that can obtain the minimum cost as the MV of the control point (step Sj_5). At this time, the inter-frame prediction unit 126 also encodes the difference between the determined MV and the predicted MV as a differential MV into a stream. That is, the inter-frame prediction unit 126 outputs the difference MV as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0516] Finally, the inter-frame prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the determined MV and the encoded reference image (step Sj_6).

[0517] Alternatively, in step Sj_1, the aforementioned candidate MV list can also be generated. The candidate MV list could, for example, be a list containing candidate MVs exported using multiple MV export methods for each control point. Multiple MV export methods could be... Figures 47A to 47C The method for exporting the MV shown. Figure 48A and Figure 48B The method for exporting the MV shown. Figure 49A and Figure 49B The export method of the MV shown, as well as any combination of other MV export methods.

[0518] In addition, the candidate MV list can also include candidate MVs for patterns other than affine patterns that are predicted in sub-block units.

[0519] Alternatively, as a candidate MV list, a candidate MV list can be generated containing candidate MVs with 2 control points and candidate MVs with 3 control points. Alternatively, a candidate MV list containing candidate MVs with 2 control points and a candidate MV list containing candidate MVs with 3 control points can be generated separately. Alternatively, a candidate MV list can be generated containing either a candidate MV with 2 control points or a candidate MV with 3 control points. Candidate MVs can be, for example, the MVs of encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left), or the MVs of valid blocks among these blocks.

[0520] In addition, as information for predicting MV selection, an index indicating which candidate MV in the candidate MV list can also be sent.

[0521] [MV Export > Triangle Mode] In the example above, the inter-frame prediction unit 126 generates a single rectangular prediction image for the current block of the rectangle. However, the inter-frame prediction unit 126 can generate multiple prediction images of different shapes for the current block of the rectangle, and generate the final rectangular prediction image by combining these multiple prediction images. The shape different from a rectangle could, for example, be a triangle.

[0522] Figure 52A This is a diagram used to illustrate the generation of predicted images for two triangles.

[0523] The inter-frame prediction unit 126 performs motion compensation on the first partition of the triangle within the current block using the first MV of the first partition, thereby generating a predicted image of the triangle. Similarly, the inter-frame prediction unit 126 performs motion compensation on the second partition of the triangle within the current block using the second MV of the second partition, thereby generating a predicted image of the triangle. Furthermore, the inter-frame prediction unit 126 combines these predicted images to generate a predicted image of a rectangle identical to the current block.

[0524] Alternatively, the predicted image for the first partition can be generated using the first MV to create a rectangular first predicted image corresponding to the current block. Similarly, the predicted image for the second partition can be generated using the second MV to create a rectangular second predicted image corresponding to the current block. The predicted image for the current block can also be generated by weighted summing of the first and second predicted images. Furthermore, the weighted summing can be applied to only a portion of the area bordering the first and second partitions.

[0525] Figure 52B This is a conceptual diagram representing an example of the first portion of the first partition that overlaps with the second partition, and the first and second sample sets obtained by weighting them as part of a correction process. The first portion can be, for example, one-quarter of the width or height of the first partition. In another example, the first portion can have a width corresponding to the N samples adjacent to the edge of the first partition. Here, N is a positive integer, for example, N can be an integer 2. Figure 52B A rectangular partition represents a rectangular portion with a width that is one-quarter the width of the first partition. Here, the first sample set contains the samples on the outer side of the first partition and the samples on the inner side of the first partition, and the second sample set contains the samples within the first partition. Figure 52B The central example represents a rectangular partition with a height that is one-quarter of the height of the first partition. Here, the first sample set contains the samples on the outer side of the first partition and the samples on the inner side of the first partition, and the second sample set contains the samples within the first partition. Figure 52B The right example represents a triangular partition with polygonal portions corresponding to the heights of the two samples. Here, the first sample set contains the samples on the outer side of the first part and the samples on the inner side of the first part, and the second sample set contains the samples within the first part.

[0526] Part 1 can be a portion of the first partition that overlaps with the adjacent partition. Figure 52C This is a conceptual diagram representing the first part of the first partition, which overlaps with a portion of the first partition. For simplicity, a rectangular partition with a portion overlapping a spatially adjacent rectangular partition is shown. Partitions with other shapes, such as triangular partitions, can be used, and the overlapping portion can also overlap with spatially or temporally adjacent partitions.

[0527] Additionally, an example is shown of generating predicted images for two partitions separately using inter-frame prediction, but it is also possible to generate predicted images for at least one partition using intra-frame prediction.

[0528] Figure 53 This is a flowchart representing an example of a triangular pattern.

[0529] In the triangular mode, firstly, the inter-frame prediction unit 126 divides the current block into a first partition and a second partition (step Sx_1). At this time, the inter-frame prediction unit 126 can encode the partition information, which is related to the partitioning of each partition, into the stream as prediction parameters. That is, the inter-frame prediction unit 126 can output the partition information as prediction parameters to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0530] Next, the inter-frame prediction unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks located in the temporal or spatial domains surrounding the current block (step Sx_2). That is, the inter-frame prediction unit 126 creates a candidate MV list.

[0531] Then, the inter-frame prediction unit 126 selects the candidate MV of the first partition and the candidate MV of the second partition from the multiple candidate MVs obtained in step Sx_2 as the first MV and the second MV, respectively (step Sx_3). At this time, the inter-frame prediction unit 126 can also encode the MV selection information used to identify the selected candidate MV as prediction parameters into the stream. That is, the inter-frame prediction unit 126 can output the MV selection information as prediction parameters to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0532] Next, the inter-frame prediction unit 126 uses the selected first MV and the encoded reference image to perform motion compensation, thereby generating a first predicted image (step Sx_4). Similarly, the inter-frame prediction unit 126 uses the selected second MV and the encoded reference image to perform motion compensation, thereby generating a second predicted image (step Sx_5).

[0533] Finally, the inter-frame prediction unit 126 performs a weighted summation of the first and second prediction images to generate the prediction image for the current block (step Sx_6).

[0534] In addition, Figure 52A In the example shown, the first and second sections are triangles, but they could also be trapezoids, or they could be different shapes. Furthermore, in... Figure 52A In the example shown, the current block consists of 2 partitions, but it can also consist of more than 3 partitions.

[0535] Additionally, partitions 1 and 2 can be duplicated. That is, partitions 1 and 2 can contain the same pixel region. In this case, the predicted image for the current block can also be generated using the predicted image from partition 1 and the predicted image from partition 2.

[0536] In addition, this example shows the generation of predicted images in both partitions via inter-frame prediction, but it is also possible to generate predicted images for at least one partition via intra-frame prediction.

[0537] In addition, the candidate MV list used to select the 1st MV and the candidate MV list used to select the 2nd MV can be different or the same.

[0538] Furthermore, partitioning information may include indices indicating the partitioning direction that divides the current block into at least multiple partitions. MV selection information may also include indices indicating the selected 1st MV and the selected 2nd MV. A single index may also represent multiple pieces of information. For example, a single index summarizing a portion or all of the partitioning information and a portion or all of the MV selection information may also be encoded.

[0539] [MV Export > ATMVP Mode] Figure 54 This is a diagram illustrating an example of the ATMVP pattern, which derives MVs in sub-block units.

[0540] The ATMVP pattern is a pattern that is classified as a merge pattern. For example, in the ATMVP pattern, candidate MVs for sub-block units are registered in the candidate MV list used for the normal merge pattern.

[0541] Specifically, in the ATMVP model, firstly, as Figure 54 As shown, in the encoded reference image specified by the MV (MV0) of the block adjacent to the lower left of the current block, a reference block with a corresponding temporal MV is determined. Next, for each sub-block within the current block, the MV used for encoding the region corresponding to that sub-block within the temporal MV reference block is determined. The MVs thus determined are included in the candidate MV list as candidate MVs for the sub-blocks of the current block. When selecting candidate MVs for each sub-block from the candidate MV list, motion compensation is performed on that sub-block, using the candidate MV as the MV for that sub-block. This generates a predicted image for each sub-block.

[0542] In addition, Figure 54In the example shown, the block adjacent to the bottom left of the current block is used as the reference block for the surrounding MV, but other blocks can also be used. Furthermore, the size of the child block can be 4x4 pixels, 8x8 pixels, or any other size. The size of the child block can also be toggled in units such as slices, bricks, or images.

[0543] [Motion Search > DMVR] Figure 55 This is a diagram representing the merging patterns and the relationship between DMVR.

[0544] The inter-frame prediction unit 126 derives the MV of the current block in merge mode (step S1_1). Next, the inter-frame prediction unit 126 determines whether to perform an MV search, i.e., a motion search (step S1_2). Here, when it is determined that no motion search should be performed (no in step S1_2), the inter-frame prediction unit 126 determines the MV derived in step S1_1 as the final MV for the current block (step S1_4). That is, in this case, the MV of the current block is determined in merge mode.

[0545] On the other hand, when it is determined in step Sl_1 that motion search is to be performed (yes in step Sl_2), the inter-frame prediction unit 126 derives the final MV for the current block by searching the surrounding region of the reference image represented by the MV derived in step Sl_1 (step Sl_3). That is, in this case, the MV of the current block is determined by the DMVR.

[0546] Figure 56 This is a conceptual diagram used to illustrate an example of the DMVR used to determine the MV.

[0547] First, for example in merge mode, candidate MVs (L0 and L1) are selected for the current block. Then, based on the candidate MV (L0), reference pixels are determined according to the encoded images in the L0 list, i.e., the first reference image (L0). Similarly, based on the candidate MV (L1), reference pixels are determined according to the encoded images in the L1 list, i.e., the second reference image (L1). The template is generated by averaging these reference pixels.

[0548] Next, using this template, the surrounding areas of the candidate MVs in the first reference image (L0) and the second reference image (L1) are searched respectively, and the MV with the lowest cost is determined as the final MV of the current block. In addition, the cost can also be calculated using, for example, the difference between the pixel values ​​of the template and the pixel values ​​of the search area, as well as the candidate MV values.

[0549] Even if it's not the process described here, any process that can search for the surrounding elements of candidate music videos and derive the final music video can be used.

[0550] Figure 57 This is a conceptual diagram used to illustrate another example of DMVR used to determine MV. Figure 57 The example shown is similar to Figure 56 Unlike the example of DMVR shown, the cost is calculated without generating a template.

[0551] First, the inter-frame prediction unit 126 searches for the vicinity of reference blocks contained in the reference images of both the L0 and L1 lists, based on the candidate MVs obtained from the candidate MV list, i.e., the initial MV. For example, ... Figure 57 As shown, the initial MV corresponding to the reference block in the L0 list is InitMV_L0, and the initial MV corresponding to the reference block in the L1 list is InitMV_L1. During motion search, the inter-frame prediction unit 126 first sets the search position for the reference image in the L0 list. The difference vector representing this set search position is, specifically, the difference vector from the position represented by the initial MV (i.e., InitMV_L0) to this search position is MVd_L0. Then, the inter-frame prediction unit 126 determines the search position in the reference image of the L1 list. This search position is represented by the difference vector from the position shown by the initial MV (i.e., InitMV_L1) to this search position. Specifically, the inter-frame prediction unit 126 determines this difference vector as MVd_L1 by mirroring MVd_L0. That is, the inter-frame prediction unit 126 sets the position symmetrical to the position represented by the initial MV in the reference images of both the L0 and L1 lists as the search position. The inter-frame prediction unit 126 calculates the sum of absolute differences (SAD) of pixel values ​​within the block at each search location as a cost, and finds the search location with the minimum cost.

[0552] Figure 58A This is a diagram illustrating an example of motion search in DMVR. Figure 58B This is a flowchart representing an example of the motion search.

[0553] First, in Step 1, the inter-frame prediction unit 126 calculates the search position (also called the start point) represented by the initial MV and the costs of the eight search positions surrounding it. Then, the inter-frame prediction unit 126 determines whether the cost of the search positions other than the start point is the minimum. Here, if it is determined that the cost of the search positions other than the start point is the minimum, the inter-frame prediction unit 126 moves to the search position with the minimum cost and performs the processing in Step 2. On the other hand, if the cost of the start point is the minimum, the inter-frame prediction unit 126 skips the processing in Step 2 and performs the processing in Step 3.

[0554] In Step 2, the inter-frame prediction unit 126 uses the search position moved according to the processing result of Step 1 as the new starting point and performs the same search as in Step 1. Furthermore, the inter-frame prediction unit 126 determines whether the cost of the search position other than the starting point is the minimum. If the cost of the search position other than the starting point is the minimum, the inter-frame prediction unit 126 proceeds to Step 4. On the other hand, if the cost of the starting point is the minimum, the inter-frame prediction unit 126 proceeds to Step 3.

[0555] In Step 4, the inter-frame prediction unit 126 treats the search position of the starting point as the final search position and determines the difference between the position represented by the initial MV and the final search position as the difference vector.

[0556] In Step 3, the inter-frame prediction unit 126 determines the pixel position with the lowest cost (decimal precision) based on the cost of four points located above, below, left, and right of the starting point in Step 1 or Step 2, and sets this pixel position as the final search position. This decimal precision pixel position is determined by weighted summing of the vectors ((0, 1), (0, -1), (-1, 0), (1, 0)) located at the four points, using the cost of each of the four search positions as weights. Then, the inter-frame prediction unit 126 determines the difference between the position represented by the initial MV and this final search position as a difference vector.

[0557] [Motion Compensation > BIO / OBMC / LIC] In motion compensation, there are modes that generate a predicted image and then correct that predicted image. Examples of such modes include BIO, OBMC, and LIC, which will be discussed later.

[0558] Figure 59 This is a flowchart illustrating an example of generating a predicted image.

[0559] The inter-frame prediction unit 126 generates a prediction image (step Sm_1) and corrects the prediction image using any of the above modes (step Sm_2).

[0560] Figure 60 This is a flowchart illustrating another example of generating a predicted image.

[0561] The inter-frame prediction unit 126 derives the MV of the current block (step Sn_1). Next, the inter-frame prediction unit 126 uses the MV to generate a prediction image (step Sn_2) and determines whether to perform correction processing (step Sn_3). Here, when it is determined that correction processing is required (yes in step Sn_3), the inter-frame prediction unit 126 generates the final prediction image by correcting the prediction image (step Sn_4). In addition, in the LIC described later, luminance and chromatic aberration can also be corrected in step Sn_4. On the other hand, when it is determined that no correction processing is required (no in step Sn_3), the inter-frame prediction unit 126 outputs the prediction image as the final prediction image without correcting the prediction image (step Sn_5).

[0562] [Motion compensation > OBMC] Not only can motion information from the current block obtained through motion search be used, but motion information from neighboring blocks can also be used to generate inter-frame prediction images. Specifically, inter-frame prediction images can also be generated by weighted summing of prediction images based on motion information obtained through motion search (within the reference image) and prediction images based on motion information from neighboring blocks (within the current image), using sub-block units within the current block. Such inter-frame prediction (motion compensation) is sometimes referred to as OBMC (overlapped block motion compensation) or OBMC mode.

[0563] In OBMC mode, information indicating the size of sub-blocks used for OBMC (e.g., OBMC block size) can also be signaled at the sequence level. Furthermore, information indicating whether OBMC mode is applied (e.g., OBMC flags) can also be signaled at the CU level. Additionally, the signaling level for this information is not limited to sequence and CU levels; it can also be other levels (e.g., image level, slice level, brick level, CTU level, or sub-block level).

[0564] A more detailed explanation of the OBMC model is provided. Figure 61 and Figure 62 These are flowcharts and concept diagrams used to illustrate the outline of OBMC-based predictive image correction processing.

[0565] First, such as Figure 62 As shown, the predicted image (Pred) based on the usual motion compensation is obtained using the MV assigned to the current block. Figure 62 In the image, the arrow "MV" points to the reference image and indicates which block the current block of the current image references to obtain the predicted image.

[0566] Next, the MV (MV_L) already derived from the encoded left adjacent block is applied (reused) to the current block to obtain the predicted image (Pred_L). MV (MV_L) is represented by the arrow "MV_L" pointing from the current block to the reference image. Then, the first correction of the predicted image is performed by overlapping the two predicted images Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.

[0567] Similarly, the MV (MV_U) derived from the encoded upper neighboring block is applied (reused) to the current block to obtain the predicted image (Pred_U). MV (MV_U) is represented by the arrow "MV_U" pointing from the current block to the reference image. Then, a second correction of the predicted image is performed by aligning the predicted image Pred_U with the predicted images (e.g., Pred and Pred_L) that have undergone the first correction. This has the effect of blending the boundaries between neighboring blocks. The predicted image obtained through the second correction is the final predicted image of the current block, with the boundaries of neighboring blocks blended (smoothed).

[0568] Furthermore, the above example uses a two-path correction method that employs the left and top adjacent blocks, but this correction method could also be a three-path or higher correction method that employs the right and / or bottom adjacent blocks.

[0569] In addition, the overlapping area may not be the entire pixel area of ​​the block, but only a part of the area near the block boundary.

[0570] Furthermore, the OBMC prediction image correction process is described here, which involves obtaining a prediction image Pred by overlaying a reference image with the additional prediction images Pred_L and Pred_U. However, when correcting prediction images based on multiple reference images, the same process can be applied to each of the multiple reference images separately. In this case, by performing OBMC image correction based on multiple reference images, after obtaining the corrected prediction images from each reference image, the final prediction image is obtained by further overlaying the obtained multiple corrected prediction images.

[0571] In addition, in OBMC, the unit of the current block can be a PU unit or a sub-block unit after further subdivision of the PU.

[0572] One method for determining whether to apply OBMC is to use a signal, obmc_flag, that indicates whether OBMC is applied. As a specific example, the encoding device 100 can also determine whether the current block belongs to a motion-complex region. If the block belongs to a motion-complex region, the encoding device 100 sets obmc_flag to 1 and applies OBMC for encoding; if the block does not belong to a motion-complex region, it sets obmc_flag to 0 and does not apply OBMC. On the other hand, in the decoding device 200, the obmc_flag recorded in the stream is decoded, and the application of OBMC for decoding is switched based on this value.

[0573] [Motion Compensation > BIO] Next, the method for deriving MV will be explained. First, the mode for deriving MV based on a model assuming constant linear motion will be explained. This mode is sometimes called the BIO (bi-directional optical flow) mode. Alternatively, this bi-directional optical flow can also be expressed as BDOF instead of BIO.

[0574] Figure 63 This diagram is used to illustrate a model that assumes uniform linear motion. Figure 63 In the middle, (v x v y Let represent the velocity vector, and τ0 and τ1 represent the temporal distances between the current image (Cur Pic) and the two reference images (Ref0, Ref1), respectively. (MVx0, MVy0) represents the MV corresponding to reference image Ref0, and (MVx1, MVy1) represents the MV corresponding to reference image Ref1.

[0575] At this time, in the velocity vector (v x v y Under the assumption of constant linear motion of MVx0, MVy0 and MVx1, MVy1 are expressed as (vxτ0, vyτ0) and (-vxτ1, -vyτ1) respectively, and the following optical flow equation (2) holds.

[0576]

Formula 3

[0577] Alternatively, the motion vector (MV) can be derived on the decoding device 200 side using a different method than deriving the motion vector based on a model assuming constant linear motion. For example, the motion vector can also be derived on a sub-block basis based on the MV of multiple adjacent blocks.

[0578] Figure 64 This is a flowchart illustrating an example of inter-frame prediction according to BIO. Additionally, Figure 65 This is a diagram illustrating an example of the structure of the inter-frame prediction unit 126 that performs inter-frame prediction according to BIO.

[0579] like Figure 65 As shown, the inter-frame prediction unit 126 includes, for example, a memory 126a, an interpolated image export unit 126b, a gradient image export unit 126c, an optical flow export unit 126d, a correction value export unit 126e, and a prediction image correction unit 126f. Furthermore, the memory 126a may also be a frame memory 122.

[0580] The inter-frame prediction unit 126 uses two reference images (Ref0, Ref1) that are different from the image containing the current block (Cur Pic) to derive two motion vectors (M0, M1). Then, the inter-frame prediction unit 126 uses these two motion vectors (M0, M1) to derive the predicted image of the current block (step Sy_1). In addition, motion vector M0 is the motion vector (MVx0, MVy0) corresponding to reference image Ref0, and motion vector M1 is the motion vector (MVx1, MVy1) corresponding to reference image Ref1.

[0581] Next, the interpolation image export unit 126b, referring to the memory 126a, uses the motion vector M0 and the reference image L0 to export the interpolation image I of the current block. 0 Additionally, the interpolation image export unit 126b, referencing the memory 126a, exports the interpolation image I of the current block using the motion vector M1 and the reference image L1. 1 (Step Sy_2). Here, the interpolated image I 0 It is the interpolated image I, which is the image contained in the reference image Ref0 exported from the current block. 1This refers to the image exported from the current block, contained in reference image Ref1. Interpolated image I 0 and interpolated image I 1 Each can be the same size as the current block. Alternatively, to properly derive the gradient image described later, the interpolated image I... 0 and interpolated image I 1 Each of these can be an image larger than the current block. Furthermore, the interpolated image I... 0 and I 1 It can include a predicted image derived by applying motion vectors (M0, M1) and reference images (L0, L1), as well as a motion compensation filter.

[0582] In addition, the gradient image export unit 126c outputs the image based on the interpolated image I. 0 and interpolated image I 1 Export the gradient image of the current block (Ix) 0 , Ix 1 ,Iy 0 ,Iy 1 (Step Sy_3). Furthermore, the gradient image in the horizontal direction is (Ix 0 , Ix 1 The gradient image in the vertical direction is (Iy) 0 ,Iy 1 The gradient image extraction unit 126c can also derive the gradient image, for example, by applying a gradient filter to the interpolated image. The gradient image only needs to represent the spatial variation of pixel values ​​along the horizontal or vertical direction.

[0583] Next, the optical flow output unit 126d uses interpolated images (I) to construct multiple sub-block units that make up the current block. 0 I 1 ) and gradient image (Ix 0 , Ix 1 ,Iy 0 ,Iy 1 The optical flow (vx, vy) is derived as the aforementioned velocity vectors (step Sy_4). Optical flow is a coefficient that corrects the spatial movement of pixels; it can also be called a local motion estimate, a corrected motion vector, or a corrected weight vector. As an example, a sub-block can be a 4x4 pixel sub-CU. Alternatively, the optical flow can be derived not in sub-block units, but in pixel units or other units.

[0584] Next, the inter-frame prediction unit 126 uses optical flow (vx, vy) to correct the predicted image of the current block. For example, the correction value derivation unit 126e uses optical flow (vx, vy) to derive correction values ​​for the pixel values ​​contained in the current block (step Sy_5). Furthermore, the prediction image correction unit 126f can also use the correction values ​​to correct the predicted image of the current block (step Sy_6). Additionally, the correction values ​​can be derived per pixel unit, or per multiple pixel units, or per sub-block unit.

[0585] Furthermore, the BIO processing flow is not limited to Figure 64 The publicly disclosed processing method can be implemented by simply performing... Figure 64 The disclosed processing can be supplemented or replaced with different processing, or executed in different processing orders.

[0586] [Motion Compensation > LIC] Next, an example of a pattern for generating a predicted image (prediction) using LIC (local illumination compensation) will be explained.

[0587] Figure 66A This is a diagram illustrating an example of a predictive image generation method using LIC-based brightness correction processing. Additionally, Figure 66B This is a flowchart illustrating an example of a predictive image generation method using this LIC.

[0588] First, the inter-frame prediction unit 126 derives the MV from the encoded reference image and obtains the reference image corresponding to the current block (step Sz_1).

[0589] Next, the inter-frame prediction unit 126 extracts information indicating how the brightness values ​​change in the reference image and the current image for the current block (step Sz_2). This extraction is based on the brightness pixel values ​​of the encoded left adjacent reference region (peripheral reference region) and the encoded upper adjacent reference region (peripheral reference region) in the current image, and the brightness pixel values ​​at the same position in the reference image specified by the derived MV. Then, the inter-frame prediction unit 126 uses the information indicating how the brightness values ​​change to calculate the brightness correction parameters (step Sz_3).

[0590] The inter-frame prediction unit 126 applies the brightness correction parameter to perform brightness correction processing on the reference image within the reference image specified by MV, and generates a prediction image for the current block (step Sz_4). That is, the prediction image, which serves as the reference image within the reference image specified by MV, is corrected based on the brightness correction parameter. This correction can correct both brightness and chromatic aberration. Specifically, information indicating how chromatic aberration changes can be used to calculate the chromatic aberration correction parameter, and chromatic aberration correction processing can be performed.

[0591] in addition, Figure 66A The shape of the surrounding reference area in the example is one example; other shapes can also be used.

[0592] Furthermore, the process of generating a prediction image based on a single reference image has been described here. However, the same applies when generating a prediction image based on multiple reference images. A prediction image can also be generated after performing brightness correction processing on the reference images obtained from each reference image in the same way as described above.

[0593] One method for determining whether to apply LIC is to use a lic_flag as a signal indicating whether LIC is applied. Specifically, in the encoding device 100, it is determined whether the current block belongs to a region where a brightness change has occurred. If it does, the lic_flag is set to 1, and LIC is applied for encoding. If it does not belong to a region where a brightness change has occurred, the lic_flag is set to 0, and LIC is not applied for encoding. On the other hand, in the decoding device 200, decoding can also be performed by decoding the lic_flag recorded in the stream and switching whether LIC is applied based on its value.

[0594] Other methods for determining whether to apply LIC include determining whether LIC has been applied to surrounding blocks. As a specific example, when the current block is processed in merge mode, the inter-frame prediction unit 126 determines whether the surrounding encoded blocks selected during the MV derivation in merge mode have been encoded using LIC. Based on this result, the inter-frame prediction unit 126 switches between applying LIC and encoding. Furthermore, in this example, the same processing also applies to the decoding device 200.

[0595] use Figure 66A and Figure 66B The LIC (Limited Light Correction Processing) has been explained; its details are explained below.

[0596] First, the inter-frame prediction unit 126 derives the MV from the reference image, which is an encoded image, to obtain the reference image corresponding to the current block.

[0597] Next, the inter-frame prediction unit 126 uses the luminance pixel values ​​of the coded peripheral reference regions (left and top adjacent) and the luminance pixel values ​​at the same position in the reference image specified by MV to extract information indicating how the luminance values ​​change in the reference image and the current image, and calculates luminance correction parameters for the current block. For example, the luminance pixel value of a pixel in the peripheral reference region of the current image is set to p0, and the luminance pixel value of a pixel in the peripheral reference region of the reference image at the same position is set to p1. The inter-frame prediction unit 126 calculates coefficients A and B for optimizing A×p1+B=p0 as luminance correction parameters for multiple pixels in the peripheral reference region.

[0598] Next, the inter-frame prediction unit 126 performs brightness correction processing on the reference image within the reference image specified by MV using brightness correction parameters, and generates a prediction image for the current block. For example, the brightness pixel value in the reference image is set as p2, and the brightness pixel value of the brightness-corrected prediction image is set as p3. The inter-frame prediction unit 126 generates the brightness-corrected prediction image by calculating A×p2+B=p3 for each pixel in the reference image.

[0599] In addition, it can also be used Figure 66A A portion of the surrounding reference region shown. For example, a region containing a predetermined number of pixels, each spaced out from the upper adjacent pixel and the left adjacent pixel, can also be used as the surrounding reference region. Furthermore, the surrounding reference region is not limited to the region adjacent to the current block; it can also be a region not adjacent to the current block. Additionally, in Figure 66A In the example shown, the surrounding reference region within the reference image is the region specified by the current image's MV (Modified Image) from the surrounding reference region within the current image, but it can also be a region specified by another MV. For example, this other MV could also be the MV of the surrounding reference region within the current image.

[0600] Furthermore, the operation in the encoding device 100 has been described here, but the operation in the decoding device 200 is the same.

[0601] Furthermore, LIC can be applied not only to luminance but also to chromatic aberration. In this case, correction parameters can be derived individually for each of Y, Cb, and Cr, or a common correction parameter can be used for any of them.

[0602] Furthermore, LIC can also be applied at the sub-block level. For example, the correction parameters can be derived using the surrounding reference region of the current sub-block and the surrounding reference region of the reference sub-block within the reference image specified by the MV of the current sub-block.

[0603] [Predictive Control Department] The prediction control unit 128 selects one of the intra-frame prediction image (pixels or signals output from the intra-frame prediction unit 124) and the inter-frame prediction image (pixels or signals output from the inter-frame prediction unit 126), and outputs the selected prediction image to the subtraction unit 104 and the addition unit 116.

[0604] [Prediction Parameter Generation Department] The prediction parameter generation unit 130 can output information related to the selection of prediction images in intra-frame prediction, inter-frame prediction, and prediction control unit 128 as prediction parameters to the entropy coding unit 110. The entropy coding unit 110 can generate a stream based on the prediction parameters input from the prediction parameter generation unit 130 and the quantization coefficients input from the quantization unit 108. The prediction parameters can also be used in the decoding device 200. The decoding device 200 can also receive the stream and decode it, performing the same processing as the prediction processing performed in the intra-frame prediction unit 124, inter-frame prediction unit 126, and prediction control unit 128. The prediction parameters can include selecting a prediction signal (e.g., MV, prediction type, or prediction mode used by the intra-frame prediction unit 124 or inter-frame prediction unit 126), or any index, flag, or value representing the prediction processing performed in the intra-frame prediction unit 124, inter-frame prediction unit 126, and prediction control unit 128.

[0605] [Decoding device] Next, the decoding device 200, which is capable of decoding the stream output from the encoding device 100 described above, will be described. Figure 67 This is a block diagram illustrating an example of the structure of the decoding apparatus 200 according to an embodiment. The decoding apparatus 200 is a device that decodes the encoded image, i.e., the stream, in block units.

[0606] like Figure 67 As shown, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a cyclic filtering unit 212, a frame memory 214, an intra-frame prediction unit 216, an inter-frame prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a segmentation determination unit 224. Furthermore, the intra-frame prediction unit 216 and the inter-frame prediction unit 218 are each configured as part of the prediction processing unit.

[0607] [Installation example of the decoding device] Figure 68 This is a block diagram illustrating an installation example of the decoding device 200. The decoding device 200 includes a processor b1 and a memory b2. For example, Figure 67 The multiple components of the decoding device 200 shown are composed of Figure 68 The processor b1 and memory b2 shown are installed and implemented.

[0608] Processor b1 is a circuit that processes information and has access to memory b2. For example, processor b1 may be a dedicated or general-purpose electronic circuit for decoding streams. Processor b1 can also be a processor like a CPU. Alternatively, processor b1 can be an assembly of multiple electronic circuits. Furthermore, for example, processor b1 can also function as… Figure 67 The function of the multiple components of the decoding device 200 shown, excluding the component for storing information.

[0609] Memory b2 is a dedicated or general-purpose memory used by processor b1 to decode the stream. Memory b2 can be an electronic circuit or connected to processor b1. Alternatively, memory b2 can be included within processor b1. Alternatively, memory b2 can be a collection of multiple electronic circuits. Furthermore, memory b2 can be a disk or optical disc, or it can be a storage medium or recording medium. Furthermore, memory b2 can be either non-volatile or volatile memory.

[0610] For example, memory b2 can store images or streams. Additionally, memory b2 can also store programs for processor b1 to decode the streams.

[0611] Additionally, for example, memory b2 can also serve as Figure 67 The decoding device 200 shown above has multiple components, including the component for storing information. Specifically, the memory b2 can serve as... Figure 67 The block memory 210 and frame memory 214 shown below illustrate their functions. More specifically, reconstructed images (specifically, reconstructed blocks or reconstructed pictures, etc.) can be stored in memory b2.

[0612] Alternatively, it is not necessary to install in the decoding device 200. Figure 67 All of the multiple constituent elements shown above can also be processed without the aforementioned multiple processing steps. Figure 67 A portion of the multiple constituent elements shown may be included in other devices, or a portion of the multiple processes described above may be performed by other devices.

[0613] The following describes the overall processing flow of the decoding apparatus 200, followed by an explanation of each component included in the decoding apparatus 200. Furthermore, detailed descriptions are omitted for components in the decoding apparatus 200 that perform the same processing as those in the encoding apparatus 100. For example, the inverse quantization unit 204, inverse transform unit 206, adder 208, block memory 210, frame memory 214, intra-frame prediction unit 216, inter-frame prediction unit 218, prediction control unit 220, and cyclic filtering unit 212 included in the decoding apparatus 200 perform the same processing as those included in the encoding apparatus 100: inverse quantization unit 112, inverse transform unit 114, adder 116, block memory 118, frame memory 122, intra-frame prediction unit 124, inter-frame prediction unit 126, prediction control unit 128, and cyclic filtering unit 120.

[0614] [Overall Decoding Process] Figure 69 This is a flowchart illustrating an example of the overall decoding process performed by the decoding device 200.

[0615] First, the segmentation determination unit 224 of the decoding device 200 determines the segmentation style (step Sp_1) of each of the multiple fixed-size blocks (128×128 pixels) contained in the image based on the parameters input from the entropy decoding unit 202. This segmentation style is selected by the encoding device 100. Then, the decoding device 200 performs steps Sp_2 to Sp_6 on each of the multiple blocks constituting the segmentation style.

[0616] The entropy decoding unit 202 decodes the encoded quantization coefficients and prediction parameters of the current block (specifically, entropy decoding) (step Sp_2).

[0617] Next, the inverse quantization unit 204 and the inverse transformation unit 206 restore the prediction residual of the current block by performing inverse quantization and inverse transformation on multiple quantization coefficients (step Sp_3).

[0618] Next, the prediction processing unit, consisting of the intra-frame prediction unit 216, the inter-frame prediction unit 218, and the prediction control unit 220, generates the prediction image of the current block (step Sp_4).

[0619] Next, the addition unit 208 reconstructs the current block into a reconstructed image (also known as a decoded image block) by adding the predicted image to the predicted residual (step Sp_5).

[0620] Furthermore, when the reconstructed image is generated, the cyclic filtering unit 212 filters the reconstructed image (step Sp_6).

[0621] Then, the decoding device 200 determines whether the overall decoding of the image has been completed (step Sp_7). If it is determined that the decoding has not been completed (no in step Sp_7), the processing starting from step Sp_1 is repeatedly executed.

[0622] Furthermore, these steps Sp_1 to Sp_7 can be performed sequentially by the decoding device 200, and multiple processes of a portion of these processes can be performed in parallel or in reverse order.

[0623] [Division Decision Department] Figure 70 This is a diagram showing the relationship between the segmentation determination unit 224 and other constituent elements. As an example, the segmentation determination unit 224 can also undergo the following processing.

[0624] The segmentation decision unit 224 collects block information from, for example, block memory 210 or frame memory 214, and further obtains parameters from entropy decoding unit 202. Furthermore, the segmentation decision unit 224 can determine the segmentation pattern of fixed-size blocks based on this block information and parameters. The segmentation decision unit 224 can also output information representing the determined segmentation pattern to inverse transform unit 206, intra-frame prediction unit 216, and inter-frame prediction unit 218. The inverse transform unit 206 can also perform an inverse transform on the transform coefficients based on the segmentation pattern represented by the information from the segmentation decision unit 224. The intra-frame prediction unit 216 and the inter-frame prediction unit 218 can generate a predicted image based on the segmentation pattern represented by the information from the segmentation decision unit 224.

[0625] [Entropy Decoding Department] Figure 71 This is a block diagram illustrating an example of the structure of the entropy decoding unit 202.

[0626] The entropy decoding unit 202 performs entropy decoding on the stream to generate quantization coefficients, prediction parameters, and parameters related to the segmentation style. CABAC is used, for example, in this entropy decoding. Specifically, the entropy decoding unit 202 includes, for example, a binary arithmetic decoding unit 202a, a context control unit 202b, and a debinarization unit 202c. The binary arithmetic decoding unit 202a uses the context values ​​derived by the context control unit 202b to perform arithmetic decoding on the stream as a binary signal. Similar to the context control unit 110b of the encoding device 100, the context control unit 202b derives context values ​​corresponding to the characteristics of the syntactic elements or the surrounding conditions, i.e., the probability of occurrence of the binary signal. The debinarization unit 202c performs debinarization, transforming the binary signal output from the binary arithmetic decoding unit 202a into a debinarized signal representing the aforementioned quantization coefficients, etc. This debinarization is performed in the manner described in the binary decoding.

[0627] The entropy decoding unit 202 outputs quantization coefficients to the inverse quantization unit 204 in block units. The entropy decoding unit 202 can also output streams to the intra-frame prediction unit 216, the inter-frame prediction unit 218, and the prediction control unit 220 (see reference). Figure 1 The prediction parameters included in the prediction unit 216, the inter-frame prediction unit 218, and the prediction control unit 220 are capable of performing the same prediction processing as that performed by the intra-frame prediction unit 124, the inter-frame prediction unit 126, and the prediction control unit 128 on the coding device 100 side.

[0628] [Entropy Decoding Department] Figure 72 This is a diagram showing the CABAC flow in the entropy decoding unit 202.

[0629] First, initialization is performed in the CABAC within the entropy decoding unit 202. This initialization includes initialization in the binary arithmetic decoding unit 202c and setting of the initial context value. Then, the binary arithmetic decoding unit 202c and the multi-valuerization unit 202c perform arithmetic decoding and multi-valuerization on, for example, the encoded data of the CTU. During this time, the context control unit 202b updates the context value each time arithmetic decoding is performed. Then, the context control unit 202b causes the context value to be backed up as a post-processing step. This backed-up context value is used, for example, as the initial value for the context value of the next CTU.

[0630] [Inverse Quantization Department] The inverse quantization unit 204 performs inverse quantization on the quantization coefficients of the current block, which are input from the entropy decoding unit 202. Specifically, the inverse quantization unit 204 performs inverse quantization on each quantization coefficient of the current block based on the quantization parameters corresponding to that coefficient. Furthermore, the inverse quantization unit 204 outputs the inverse quantization coefficients (i.e., transform coefficients) of the current block to the inverse transform unit 206.

[0631] Figure 73 This is a block diagram illustrating an example of the structure of the inverse quantization unit 204.

[0632] The inverse quantization unit 204 includes, for example, a quantization parameter generation unit 204a, a predictive quantization parameter generation unit 204b, a quantization parameter storage unit 204d, and an inverse quantization processing unit 204e.

[0633] Figure 74 This is a flowchart illustrating an example of inverse quantization performed by the inverse quantization unit 204.

[0634] As an example, the inverse quantization unit 204 can be based on Figure 74The process shown performs inverse quantization for each CU. Specifically, the quantization parameter generation unit 204a determines whether to perform inverse quantization (step Sv_11). Here, when it is determined that inverse quantization should be performed (yes in step Sv_11), the quantization parameter generation unit 204a obtains the differential quantization parameters of the current block from the entropy decoding unit 202 (step Sv_12).

[0635] Next, the prediction quantization parameter generation unit 204b obtains quantization parameters of a different processing unit from the quantization parameter storage unit 204d (step Sv_13). Based on the obtained quantization parameters, the prediction quantization parameter generation unit 204b generates prediction quantization parameters for the current block (step Sv_14).

[0636] Then, the quantization parameter generation unit 204a adds the differential quantization parameters of the current block obtained from the entropy decoding unit 202 to the predicted quantization parameters of the current block generated by the predicted quantization parameter generation unit 204b (step Sv_15). This addition generates the quantization parameters of the current block. Furthermore, the quantization parameter generation unit 204a stores the quantization parameters of the current block in the quantization parameter storage unit 204d (step Sv_16).

[0637] Next, the inverse quantization processing unit 204e uses the quantization parameters generated in step Sv_15 to inverse quantize the quantization coefficients of the current block into transform coefficients (step Sv_17).

[0638] Furthermore, differential quantization parameters can be decoded at the bit sequence level, image level, slice level, brick level, or CTU level. Alternatively, the initial values ​​of the quantization parameters can be decoded at the sequence level, image level, slice level, brick level, or CTU level. In this case, the quantization parameters can be generated using the initial values ​​of the quantization parameters and the differential quantization parameters.

[0639] Furthermore, the inverse quantization unit 204 may have multiple inverse quantizers, and may also use an inverse quantization method selected from multiple inverse quantization methods to inverse quantize the quantization coefficients.

[0640] [Inverse Transformation Section] The inverse transform unit 206 restores the prediction residual by performing an inverse transform on the transform coefficients, which are inputs from the inverse quantization unit 204.

[0641] For example, if the information read from the stream represents the application of EMT or AMT (e.g., the AMT flag is true), the inverse transform unit 206 performs an inverse transform on the transform coefficients of the current block based on the information read representing the transform type.

[0642] Furthermore, for example, when NSST is applied to the information read from the stream, the inverse transform unit 206 applies an inverse re-transformation to the transform coefficients.

[0643] Figure 75 This is a flowchart illustrating an example of the processing performed by the inverse transformation unit 206.

[0644] For example, the inverse transform unit 206 determines whether information indicating that no orthogonal transform is performed exists in the stream (step St_11). Here, when it is determined that such information does not exist (no in step St_11), the inverse transform unit 206 obtains information indicating the transform type that has been decoded by the entropy decoding unit 202 (step St_12). Next, based on this information, the inverse transform unit 206 determines the transform type to be used in the orthogonal transform of the encoding device 100 (step St_13). Then, the inverse transform unit 206 performs an inverse orthogonal transform using the determined transform type (step St_14).

[0645] Figure 76 This is a flowchart illustrating another example of the processing performed by the inverse transformation unit 206.

[0646] For example, the inverse transform unit 206 determines whether the transform size is below a predetermined value (step Su_11). Here, when it is determined that it is below the predetermined value (yes in step Su_11), the inverse transform unit 206 obtains information from the entropy decoding unit 202 indicating which transform type among the more than one transform type included in the first transform type group is used by the encoding device 100 (step Su_12). Furthermore, such information is decoded by the entropy decoding unit 202 and output to the inverse transform unit 206.

[0647] Based on this information, the inverse transform unit 206 determines the transform type to be used in the orthogonal transform in the encoding device 100 (step Su_13). Then, the inverse transform unit 206 performs an inverse orthogonal transform on the transform coefficients of the current block using the determined transform type (step Su_14). On the other hand, when it is determined in step Su_11 that the transform size is not below a predetermined value (no in step Su_11), the inverse transform unit 206 performs an inverse orthogonal transform on the transform coefficients of the current block using the second transform type group (step Su_15).

[0648] Furthermore, as an example, the inverse orthogonal transformation performed by the inverse transformation unit 206 can be performed according to each TU. Figure 75 or Figure 76 The process shown is followed for implementation. Alternatively, instead of decoding the information indicating the transformation type used in the orthogonal transformation, an inverse orthogonal transformation can be performed using a predefined transformation type. Specifically, the transformation type is DST7 or DCT8, etc., and in the inverse orthogonal transformation, the inverse transformation basis function corresponding to that transformation type is used.

[0649] [Addition Department] The adder 208 reconstructs the current block by adding the prediction residual, which is input from the inverse transform 206, to the prediction image, which is input from the prediction control 220. That is, it generates a reconstructed image of the current block. Furthermore, the adder 208 outputs the reconstructed image of the current block to the block memory 210 and the cyclic filtering 212.

[0650] [Block Memory] Block memory 210 is a storage unit used to save blocks within the current image that serve as references in intra-frame prediction. Specifically, block memory 210 saves the reconstructed image output from adder 208.

[0651] [Loop Filtering Section] The cyclic filtering unit 212 applies cyclic filtering to the reconstructed image generated by the addition unit 208 and outputs the filtered reconstructed image to the frame memory 214 and a display device.

[0652] Given that the information indicating the on / off state of the ALF is read from the stream, one filter is selected from multiple filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed image.

[0653] Figure 77 This is a block diagram showing an example of the structure of the loop filter unit 212. Furthermore, the loop filter unit 212 has the same structure as the loop filter unit 120 of the encoding device 100.

[0654] For example, the cyclic filter unit 212 Figure 77 As shown, the image includes a deblocking filter processing unit 212a, a SAO processing unit 212b, and an ALF processing unit 212c. The deblocking filter processing unit 212a applies the aforementioned deblocking filter processing to the reconstructed image. The SAO processing unit 212b applies the aforementioned SAO processing to the reconstructed image after deblocking filter processing. Furthermore, the ALF processing unit 212c applies the aforementioned ALF processing to the reconstructed image after SAO processing. Additionally, a cyclic filtering unit 212 may not be included. Figure 77 The disclosed processing unit may also include only a portion of the processing units. Furthermore, the cyclic filtering unit 212 may be configured according to... Figure 77 The structures that perform the above processes in different orders as disclosed in the text.

[0655] [Frame Memory] The frame memory 214 is a storage unit used to store reference images used in inter-frame prediction; it is also sometimes called a frame buffer. Specifically, the frame memory 214 stores the reconstructed image after filtering by the cyclic filtering unit 212.

[0656] [Prediction Unit (Intra-frame Prediction Unit / Inter-frame Prediction Unit / Prediction Control Unit)] Figure 78 This is a flowchart illustrating an example of the processing performed by the prediction unit of the decoding apparatus 200. Furthermore, as an example, the prediction unit is composed of all or part of the components of an intra-frame prediction unit 216, an inter-frame prediction unit 218, and a prediction control unit 220. The prediction processing unit includes, for example, an intra-frame prediction unit 216 and an inter-frame prediction unit 218.

[0657] The prediction unit generates a prediction image for the current block (step Sq_1). This prediction image is also called a prediction signal or a prediction block. Additionally, the prediction signal may include, for example, an intra-frame prediction signal or an inter-frame prediction signal. Specifically, the prediction unit generates a prediction image for the current block using a reconstructed image obtained by generating prediction images for other blocks, restoring prediction residuals, and adding the prediction images. The prediction unit of the decoding apparatus 200 generates the same prediction image as the prediction image generated by the prediction unit of the encoding apparatus 100. That is, the prediction image generation methods used by these prediction units are common to or correspond to each other.

[0658] The reconstructed image can be, for example, an image of a reference image, or an image containing the current block, i.e., an image of a decoded block within the current image (i.e., the other blocks mentioned above). A decoded block within the current image can be, for example, a neighboring block of the current block.

[0659] Figure 79 This is a flowchart illustrating another example of the processing performed by the prediction unit of the decoding device 200.

[0660] The prediction unit determines the method or mode for generating the predicted image (step Sr_1). For example, the method or mode can be determined based on, for example, prediction parameters.

[0661] If the first mode is determined to be the mode for generating the prediction image, the prediction unit generates the prediction image according to the first mode (step Sr_2a). Furthermore, if the second mode is determined to be the mode for generating the prediction image, the prediction unit generates the prediction image according to the second mode (step Sr_2b). Furthermore, if the third mode is determined to be the mode for generating the prediction image, the prediction unit generates the prediction image according to the third mode (step Sr_2c).

[0662] Methods 1, 2, and 3 are different methods for generating the predicted image, and can be, for example, inter-frame prediction, intra-frame prediction, and other prediction methods. The reconstructed image described above can also be used in such prediction methods.

[0663] Figure 80A and Figure 80B This is a flowchart illustrating another example of the processing performed in the prediction section of the decoding device 200.

[0664] As an example, the forecasting department can also follow... Figure 80A and Figure 80B The process shown is used for predictive processing. Additionally, Figure 80A and Figure 80B The intra-frame block copying shown is a mode of inter-frame prediction, where blocks within the current image are referenced as reference images or reference blocks. That is, intra-frame block copying does not reference images different from the current image. Furthermore, Figure 80A The PCM mode shown is an intra-frame prediction mode that does not undergo transformation or quantization.

[0665] Intra-frame prediction unit The intra-prediction unit 216 performs intra-prediction based on the intra-prediction pattern read from the stream, referring to blocks within the current image stored in the block memory 210, thereby generating a predicted image of the current block (i.e., an intra-prediction image). Specifically, the intra-prediction unit 216 performs intra-prediction by referring to the pixel values ​​(e.g., luminance values, chromaticity values) of blocks adjacent to the current block, thereby generating an intra-prediction image, and outputs the intra-prediction image to the prediction control unit 220.

[0666] In addition, if the intra-prediction mode of the reference luma block is selected in the intra-prediction of the chromatic difference block, the intra-prediction unit 216 can also predict the chromatic difference component of the current block based on the luma component of the current block.

[0667] Furthermore, in the case of the application of PDPC, where the information read from the stream represents the pixel, the intra-prediction unit 216 corrects the pixel value after intra-prediction based on the gradient of the reference pixel in the horizontal / vertical direction.

[0668] Figure 81 This diagram illustrates an example of the processing performed by the intra-frame prediction unit 216 of the decoding device 200.

[0669] The intra-prediction unit 216 first determines whether the MPM flag representing 1 exists in the stream (step Sw_11). Here, when it is determined that the MPM flag representing 1 exists (yes in step Sw_11), the intra-prediction unit 216 obtains information from the entropy decoding unit 202 indicating the intra-prediction mode selected in the coding apparatus 100 within the MPM (step Sw_12). Furthermore, this information is decoded by the entropy decoding unit 202 and output to the intra-prediction unit 216. Next, the intra-prediction unit 216 determines the MPM (step Sw_13). The MPM, for example, consists of six intra-prediction modes. Then, the intra-prediction unit 216 determines the intra-prediction mode indicated by the information obtained in step Sw_12 from among the multiple intra-prediction modes included in the MPM (step Sw_14).

[0670] On the other hand, when it is determined in step Sw_11 that the MPM flag representing 1 does not exist in the stream (no in step Sw_11), the intra-prediction unit 216 obtains information representing the intra-prediction mode selected in the coding device 100 (step Sw_15). That is, the intra-prediction unit 216 obtains information from the entropy decoding unit 202 representing the intra-prediction mode selected in the coding device 100 from one or more intra-prediction modes not included in the MPM. Furthermore, this information is decoded by the entropy decoding unit 202 and output to the intra-prediction unit 216. Then, the intra-prediction unit 216 determines the intra-prediction mode represented by the information obtained in step Sw_15 from one or more intra-prediction modes not included in the MPM (step Sw_17).

[0671] The intra-frame prediction unit 216 generates a prediction image according to the intra-frame prediction mode determined in step Sw_14 or step Sw_17 (step Sw_18).

[0672] [Inter-frame prediction unit] The inter-frame prediction unit 218 predicts the current block by referring to a reference image stored in the frame memory 214. Prediction is performed on a unit scale of the current block or sub-blocks within the current block. Sub-blocks are contained within blocks and are smaller than blocks. The size of a sub-block can be 4x4 pixels, 8x8 pixels, or any other size. The size of the sub-block can also be switched between units such as slices, bricks, or images.

[0673] For example, the inter-frame prediction unit 218 uses motion information (e.g., MV) read from the stream (e.g., prediction parameters output from the entropy decoding unit 202) to perform motion compensation, thereby generating an inter-frame prediction image for the current block or sub-block, and outputs the inter-frame prediction image to the prediction control unit 220.

[0674] When the information read from the stream is represented in OBMC mode, the inter-frame prediction unit 218 uses not only the motion information of the current block obtained through motion search, but also the motion information of neighboring blocks to generate an inter-frame prediction image.

[0675] Furthermore, when the information read from the stream represents the application of FRUC mode, the inter-frame prediction unit 218 performs motion search according to the pattern matching method (bidirectional matching or template matching) read from the stream, thereby deriving motion information. Then, the inter-frame prediction unit 218 uses the derived motion information to perform motion compensation (prediction).

[0676] Furthermore, when applying BIO mode, the inter-frame prediction unit 218 derives the MV based on a model assuming constant linear motion. Additionally, when applying affine mode to the information representation read from the stream, the inter-frame prediction unit 218 derives the MV on a sub-block basis based on the MVs of multiple adjacent blocks.

[0677] [MV export process] Figure 82 This is a flowchart illustrating an example of MV export in the decoding device 200.

[0678] The inter-frame prediction unit 218 determines, for example, whether to decode motion information (e.g., motion video). For example, the inter-frame prediction unit 218 can make this determination based on the prediction mode contained in the stream, or based on other information contained in the stream. Here, when it is determined that motion information should be decoded, the inter-frame prediction unit 218 derives the motion video of the current block in the mode that decodes the motion information. On the other hand, when it is determined that motion information should not be decoded, the inter-frame prediction unit 218 derives the motion video in the mode that does not decode the motion information.

[0679] Here, the modes for MV derivation include the normal inter-frame mode, normal merging mode, FRUC mode, and affine mode, which will be described later. Among these modes, the modes that decode motion information include the normal inter-frame mode, normal merging mode, and affine mode (specifically, affine inter-frame mode and affine merging mode). Furthermore, the motion information can include not only MV but also the predicted MV selection information, which will be described later. In addition, modes that do not decode motion information include the FRUC mode, etc. The inter-frame prediction unit 218 selects a mode from these multiple modes for deriving the MV of the current block and derives the MV of the current block using the selected mode.

[0680] Figure 83 This is a flowchart illustrating another example of MV export in the decoding device 200.

[0681] The inter-frame prediction unit 218 determines, for example, whether to decode the differential MV. This determination can be based on the prediction mode contained in the stream or on other information contained in the stream. Here, when it is determined that the differential MV should be decoded, the inter-frame prediction unit 218 can derive the MV of the current block in the mode for decoding the differential MV. In this case, for example, the differential MV contained in the stream is decoded into prediction parameters.

[0682] On the other hand, when it is determined that the differential MV will not be decoded, the inter-frame prediction unit 218 derives the MV in the mode of not decoding the differential MV. In this case, the encoded differential MV is not included in the stream.

[0683] Here, as described above, the MV derivation modes include the normal inter-frame mode, normal merging mode, FRUC mode, and affine mode, which will be described later. Among these modes, the modes that encode the differential MV include the normal inter-frame mode and the affine mode (specifically, the affine inter-frame mode). In addition, the modes that do not encode the differential MV include the FRUC mode, the normal merging mode, and the affine mode (specifically, the affine merging mode). The inter-frame prediction unit 218 selects a mode from these multiple modes for deriving the MV of the current block, and derives the MV of the current block using the selected mode.

[0684] [MV Export > Normal Inter-Frame Mode] For example, when the information read from the stream represents the application of the normal inter-frame mode, the inter-frame prediction unit 218 derives the MV based on the information read from the stream in the normal merging mode, and uses the MV to perform motion compensation (prediction).

[0685] Figure 84 This is a flowchart illustrating an example of inter-frame prediction performed in the decoding device 200 using a normal inter-frame mode.

[0686] The inter-frame prediction unit 218 of the decoding device 200 performs motion compensation for each block. At this time, the inter-frame prediction unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks that are located in the vicinity of the current block in time or space (step Sg_11). That is, the inter-frame prediction unit 218 creates a candidate MV list.

[0687] Next, the inter-frame prediction unit 218 extracts N (N being an integer greater than 2) candidate MVs from the plurality of candidate MVs obtained in step Sg_11 as prediction motion vector candidates (also called prediction MV candidates) according to a predetermined priority order (step Sg_12). Alternatively, this priority order can also be predetermined for each of the N prediction MV candidates.

[0688] Next, the inter-frame prediction unit 218 decodes the prediction MV selection information from the input stream, and uses the decoded prediction MV selection information to select one prediction MV candidate from the N prediction MV candidates as the prediction MV of the current block (step Sg_13).

[0689] Next, the inter-frame prediction unit 218 decodes the differential MV from the input stream and derives the MV of the current block by adding the difference value of the decoded differential MV to the selected prediction MV (step Sg_14).

[0690] Finally, the inter-frame prediction unit 218 generates a predicted image for the current block by performing motion compensation on the current block using the derived MV and the decoded reference image (step Sg_15). The processing of steps Sg_11 to Sg_15 is performed on each block. For example, when the processing of steps Sg_11 to Sg_15 is performed on all blocks contained in a slice, the inter-frame prediction using the normal inter-frame mode for that slice ends. Similarly, when the processing of steps Sg_11 to Sg_15 is performed on all blocks contained in an image, the inter-frame prediction using the normal inter-frame mode for that image ends. Furthermore, the processing of steps Sg_11 to Sg_15 can also be performed on a portion of the blocks instead of all blocks contained in a slice, at which point the inter-frame prediction using the normal inter-frame mode for that slice ends. Likewise, when the processing of steps Sg_11 to Sg_15 is performed on a portion of the blocks contained in an image, the inter-frame prediction using the normal inter-frame mode for that image can also end.

[0691] [MV Export > Normal Merge Mode] For example, in the case where the information read from the stream represents the application of normal merging mode, the inter-frame prediction unit 218 derives the MV in normal merging mode and uses the MV for motion compensation (prediction).

[0692] Figure 85 This is a flowchart illustrating an example of inter-frame prediction based on a normal merging mode in the decoding device 200.

[0693] The inter-frame prediction unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks that are located in the vicinity of the current block in time or space (step Sh_11). That is, the inter-frame prediction unit 218 creates a candidate MV list.

[0694] Next, the inter-frame prediction unit 218 selects one candidate MV from the multiple candidate MVs obtained in step Sh_11 and derives the MV of the current block (step Sh_12). Specifically, the inter-frame prediction unit 218 obtains, for example, MV selection information contained in the stream as prediction parameters, and selects the candidate MV identified by the MV selection information as the MV of the current block.

[0695] Finally, the inter-frame prediction unit 218 performs motion compensation on the current block using the derived MV and the decoded reference image, generating a prediction image for the current block (step Sh_13). The processing of steps Sh_11 to Sh_13 is performed on each block, for example. For instance, when steps Sh_11 to Sh_13 are performed on all blocks contained in a slice, inter-frame prediction using the normal merging mode for that slice ends. Similarly, when steps Sh_11 to Sh_13 are performed on all blocks contained in an image, inter-frame prediction using the normal merging mode for that image ends. Furthermore, the processing of steps Sh_11 to Sh_13 can also be performed on a subset of blocks instead of all blocks contained in a slice, at which point inter-frame prediction using the normal merging mode for that slice ends. Likewise, the processing of steps Sh_11 to Sh_13 can be performed on a subset of blocks contained in an image, at which point inter-frame prediction using the normal merging mode for that image ends.

[0696] [MV Export > FRUC Mode] For example, in the case where the information read from the stream represents the FRUC mode, the inter-frame prediction unit 218 derives the motion video (MV) in the FRUC mode and uses the MV for motion compensation (prediction). In this case, the motion information is not signaled from the encoding device 100 side, but is derived from the decoding device 200 side. For example, the decoding device 200 can also derive the motion information by performing a motion search. In this case, the decoding device 200 does not use the pixel values ​​of the current block to perform the motion search.

[0697] Figure 86 This is a flowchart illustrating an example of inter-frame prediction based on FRUC mode in the decoding device 200.

[0698] First, the inter-frame prediction unit 218 generates a list of candidate MVs (i.e., a candidate MV list, which can also be common to the candidate MV list of the normal merging mode) by referring to the MVs of each decoded block that are spatially or temporally adjacent in the current block (step Si_11). Next, the inter-frame prediction unit 218 selects the best candidate MV from the multiple candidate MVs registered in the candidate MV list (step Si_12). For example, the inter-frame prediction unit 218 calculates the evaluation value of each candidate MV included in the candidate MV list and selects one candidate MV as the best candidate MV based on the evaluation value. Then, the inter-frame prediction unit 218 derives the MV for the current block based on the selected best candidate MV (step Si_14). Specifically, for example, the selected best candidate MV is directly derived as the MV for the current block. Alternatively, for example, the MV for the current block can also be derived by performing pattern matching in the surrounding region of the position in the reference image corresponding to the selected best candidate MV. That is, for the area surrounding the best candidate MV, a search is performed using style matching and evaluation values ​​from the reference image. Therefore, if an MV with a good evaluation value exists, the best candidate MV can be updated to that MV, and it becomes the final MV for the current block. Alternatively, updating to an MV with a better evaluation value may not be performed.

[0699] Finally, the inter-frame prediction unit 218 performs motion compensation on the current block using the derived MV and the decoded reference image, generating a predicted image for the current block (step Si_15). The processing of steps Si_11 to Si_15 is performed on each block, for example. For instance, when steps Si_11 to Si_15 are performed on all blocks contained in a slice, inter-frame prediction using the FRUC mode for that slice ends. Similarly, when steps Si_11 to Si_15 are performed on all blocks contained in an image, inter-frame prediction using the FRUC mode for that image ends. Processing can also be performed on a sub-block basis in the same way as on a block basis.

[0700] [MV Export > Affine Merge Mode] For example, in the case where the information read from the stream represents the application of affine merging mode, the inter-frame prediction unit 218 derives the MV in affine merging mode and uses the MV for motion compensation (prediction).

[0701] Figure 87 This is a flowchart illustrating an example of inter-frame prediction based on affine merging mode in the decoding device 200.

[0702] In affine merging mode, the inter-frame prediction unit 218 first derives the MV of each control point of the current block (step Sk_11). For example... Figure 46AAs shown, the control points are the top-left and top-right corners of the current block, or as... Figure 46B The image shows the top left, top right, and bottom left corners of the current block.

[0703] For example, in use Figures 47A to 47C In the case of the MV export method shown, such as Figure 47A As shown, the inter-frame prediction unit 218 checks these blocks in the order of decoded block A (left), block B (top), block C (top right), block D (bottom left), and block E (top left) to determine the first valid block decoded in affine mode.

[0704] The inter-frame prediction unit 218 uses the first valid block decoded in the determined affine mode to derive the MV of the control points. For example, if block A is determined and block A has 2 control points, then... Figure 47B As shown, the inter-frame prediction unit 218 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block by projecting the motion vectors v3 and v4 of the upper left and upper right corners of the decoded block containing block A onto the current block. Thus, the MV of each control point is derived.

[0705] In addition, such as Figure 49A As shown, given that block A has 2 control points, the MV of 3 control points can also be calculated, or as follows: Figure 49B Define block A as shown. If block A has 3 control points, calculate the MV of 2 control points.

[0706] In addition, when the stream contains MV selection information as a prediction parameter, the inter-frame prediction unit 218 can also use the MV selection information to derive the MV of each control point of the current block.

[0707] Next, the inter-frame prediction unit 218 performs motion compensation on each of the multiple sub-blocks contained in the current block. That is, for each of the multiple sub-blocks, the inter-frame prediction unit 218 uses two motion vectors v0 and v1 and the above-described equation (1A), or uses three motion vectors v0, v1, and v2 and the above-described equation (1B), to calculate the MV of the sub-block as an affine MV (step Sk_12). Then, the inter-frame prediction unit 218 uses these affine MVs and the decoded reference image to perform motion compensation on the sub-block (step Sk_13). When steps Sk_12 and Sk_13 are performed on all the sub-blocks contained in the current block, the inter-frame prediction using the affine merging mode for the current block ends. That is, motion compensation is performed on the current block, and a predicted image of the current block is generated.

[0708] Furthermore, in step Sk_11, the aforementioned candidate MV list can also be generated. The candidate MV list could, for example, be a list containing candidate MVs exported using multiple MV export methods for each control point. Multiple MV export methods could be... Figures 47A to 47C The method for exporting the MV shown. Figure 48A and Figure 48B The method for exporting the MV shown. Figure 49A and Figure 49B The export method of the MV shown, as well as any combination of other MV export methods.

[0709] In addition, the candidate MV list can also include candidate MVs for patterns other than affine patterns that are predicted in sub-block units.

[0710] Alternatively, as a candidate MV list, for example, a candidate MV list can be generated containing candidate MVs with affine merging patterns having 2 control points and candidate MVs with affine merging patterns having 3 control points. Alternatively, a candidate MV list can be generated separately containing candidate MVs with affine merging patterns having 2 control points and candidate MVs with affine merging patterns having 3 control points. Alternatively, a candidate MV list can be generated containing patterns that are either affine merging patterns with 2 control points or affine merging patterns with 3 control points.

[0711] [MV Export > Affine Inter-Frame Mode] For example, in the case where the information read from the stream represents an affine inter-frame pattern, the inter-frame prediction unit 218 derives the MV in the affine inter-frame pattern and uses the MV for motion compensation (prediction).

[0712] Figure 88 This is a flowchart illustrating an example of inter-frame prediction based on affine inter-frame patterns in the decoding device 200.

[0713] In affine inter-frame mode, firstly, the inter-frame prediction unit 218 derives the prediction MV(v0, v1) or (v0, v1, v2) for each of the two or three control points of the current block (step Sj_11). The control points are, for example, as follows: Figure 46A or Figure 46B The image shows the top left, top right, or bottom left corner of the current block.

[0714] The inter-frame prediction unit 218 acquires prediction MV selection information included in the stream as prediction parameters, and uses the MV identified by the prediction MV selection information to derive the prediction MV for each control point of the current block. For example, when using... Figure 48A and Figure 48B In the case of the MV export method shown, the inter-frame prediction unit 218 selects... Figure 48A or Figure 48BThe MV of the block identified by the prediction MV selection information in the decoded blocks near each control point of the current block is shown, and the prediction MV (v0, v1) or (v0, v1, v2) of the control point of the current block is derived.

[0715] Next, the inter-frame prediction unit 218, for example, obtains the differential MVs included in the stream as prediction parameters, and adds the predicted MVs of each control point of the current block to the differential MVs corresponding to those predicted MVs (step Sj_12). Thus, the MVs of each control point of the current block are derived.

[0716] Next, the inter-frame prediction unit 218 performs motion compensation on each of the multiple sub-blocks contained in the current block. That is, for each of the multiple sub-blocks, the inter-frame prediction unit 218 uses two motion vectors v0 and v1 and the above-described equation (1A), or uses three motion vectors v0, v1, and v2 and the above-described equation (1B), to calculate the MV of the sub-block as an affine MV (step Sj_13). Then, the inter-frame prediction unit 218 uses these affine MVs and the decoded reference image to perform motion compensation on the sub-block (step Sj_14). When steps Sj_13 and Sj_14 are performed on all the sub-blocks contained in the current block, the inter-frame prediction using the affine merging mode for the current block ends. That is, motion compensation is performed on the current block, and a predicted image of the current block is generated.

[0717] In addition, in step Sj_11, the candidate MV list described above can also be generated in the same way as in step Sk_11.

[0718] [MV Export > Triangle Mode] For example, in the case of using a triangular pattern to represent information read from a stream, the inter-frame prediction unit 218 derives the MV in the triangular pattern and uses the MV for motion compensation (prediction).

[0719] Figure 89 This is a flowchart illustrating an example of inter-frame prediction based on triangular patterns in the decoding device 200.

[0720] In the triangular mode, firstly, the inter-frame prediction unit 218 divides the current block into a first partition and a second partition (step Sx_11). At this time, the inter-frame prediction unit 218 can obtain information related to the division into each partition, i.e., partition information, from the stream as prediction parameters. Moreover, the inter-frame prediction unit 218 can divide the current block into a first partition and a second partition based on this partition information.

[0721] Next, the inter-frame prediction unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks located in the vicinity of the current block in time or space (step Sx_12). That is, the inter-frame prediction unit 218 creates a candidate MV list.

[0722] Then, the inter-frame prediction unit 218 selects the candidate MV of the first partition and the candidate MV of the second partition from the multiple candidate MVs obtained in step Sx_11 as the first MV and the second MV, respectively (step Sx_13). At this time, the inter-frame prediction unit 218 can also obtain MV selection information from the stream for identifying the selected candidate MVs as prediction parameters. Then, the inter-frame prediction unit 218 can select the first MV and the second MV according to the MV selection information.

[0723] Next, the inter-frame prediction unit 218 performs motion compensation using the selected first MV and the decoded reference image to generate a first predicted image (step Sx_14). Similarly, the inter-frame prediction unit 218 performs motion compensation using the selected second MV and the decoded reference image to generate a second predicted image (step Sx_15).

[0724] Finally, the inter-frame prediction unit 218 generates the prediction image for the current block by weighted summing of the first prediction image and the second prediction image (step Sx_16).

[0725] [Sports Search > DMVR] For example, in the case of an application where the information read from the stream represents the DMVR, the inter-frame prediction unit 218 performs motion search via the DMVR.

[0726] Figure 90 This is a flowchart illustrating an example of motion search based on DMVR in the decoding device 200.

[0727] The inter-frame prediction unit 218 first derives the MV of the current block in merge mode (step S1_11). Next, the inter-frame prediction unit 218 derives the final MV for the current block by searching the surrounding region of the reference image represented by the MV derived in step S1_11 (step S1_12). That is, the MV of the current block is determined by the DMVR.

[0728] Figure 91 This is a flowchart illustrating a detailed example of motion search based on DMVR in the decoding device 200.

[0729] First, the inter-frame prediction unit 218 in Figure 58A In Step 1, the costs of the initial MV-represented search position (also called the starting point) and the eight surrounding search positions are calculated. The inter-frame prediction unit 218 then determines whether the cost of any search position other than the starting point is the minimum. If the cost of any search position other than the starting point is determined to be the minimum, the inter-frame prediction unit 218 moves to the search position with the minimum cost and performs... Figure 58A The processing in Step 2 is shown. On the other hand, if the cost at the starting point is minimized, the inter-frame prediction unit 218 skips. Figure 58A The process shown in Step 2 will proceed to Step 3.

[0730] exist Figure 58A In Step 2, the inter-frame prediction unit 218 uses the search position moved according to the processing result of Step 1 as a new starting point and performs the same search as in Step 1. Furthermore, the inter-frame prediction unit 218 determines whether the cost of the search position other than the starting point is the minimum. If the cost of the search position other than the starting point is the minimum, the inter-frame prediction unit 218 proceeds to Step 4. On the other hand, if the cost of the starting point is the minimum, the inter-frame prediction unit 218 proceeds to Step 3.

[0731] In Step 4, the inter-frame prediction unit 218 treats the search position of the starting point as the final search position and determines the difference between the position shown by the initial MV and the final search position as the difference vector.

[0732] exist Figure 58A In Step 3, the inter-frame prediction unit 218 determines the pixel position with the lowest cost (decimal precision) based on the cost of four points (up, down, left, right) at the starting point of Step 1 or Step 2, and uses this pixel position as the final search position. This decimal precision pixel position is determined by weighted summing of the vectors ((0, 1), (0, -1), (-1, 0), (1, 0)) located at the four points (up, down, left, right) with the cost of their respective search positions as weights. Then, the inter-frame prediction unit 218 determines the difference between the position shown in the initial MV and the final search position as a difference vector.

[0733] [Motion Compensation > BIO / OBMC / LIC] For example, in applications where the information read from the stream represents a correction of the predicted image, when generating the predicted image, the inter-frame prediction unit 218 corrects the predicted image according to the correction mode. This mode could be, for example, BIO, OBMC, or LIC as described above.

[0734] Figure 92 This is a flowchart illustrating an example of the generation of a predicted image in the decoding device 200.

[0735] The inter-frame prediction unit 218 generates a prediction image (step Sm_11) and corrects the prediction image using any of the above modes (step Sm_12).

[0736] Figure 93 This is a flowchart illustrating another example of the generation of a predicted image in the decoding device 200.

[0737] The inter-frame prediction unit 218 derives the MV of the current block (step Sn_11). Next, the inter-frame prediction unit 218 uses the MV to generate a prediction image (step Sn_12) and determines whether correction processing should be performed (step Sn_13). For example, the inter-frame prediction unit 218 obtains prediction parameters contained in the stream and determines whether correction processing should be performed based on these prediction parameters. These prediction parameters may be, for example, flags indicating whether the aforementioned modes are applied. Here, when it is determined that correction processing should be performed (yes in step Sn_13), the inter-frame prediction unit 218 generates the final prediction image by correcting the prediction image (step Sn_14). Furthermore, in the LIC, the brightness and chromatic aberration of the prediction image can be corrected in step Sn_14. On the other hand, when it is determined that correction processing should not be performed (no in step Sn_13), the inter-frame prediction unit 218 does not correct the prediction image and outputs it as the final prediction image (step Sn_15).

[0738] [Motion Compensation > OBMC] For example, in the case of an application where the information read from the stream represents the OBMC, when generating a prediction image, the inter-frame prediction unit 218 corrects the prediction image according to the OBMC.

[0739] Figure 94 This is a flowchart illustrating an example of OBMC-based correction of the predicted image in the decoding device 200. Additionally, Figure 94 The flowchart shows that it uses Figure 62 The process of correcting the current image and the predicted image based on the reference image is shown.

[0740] First, such as Figure 62 As shown, the inter-frame prediction unit 218 uses the MV allocated to the current block to obtain a predicted image (Pred) based on normal motion compensation.

[0741] Next, the inter-frame prediction unit 218 applies (reuses) the MV (MV_L) already derived from the decoded left adjacent block to the current block to obtain a prediction image (Pred_L). Then, the inter-frame prediction unit 218 performs the first correction of the prediction image by making the two prediction images Pred and Pred_L coincide. This has the effect of blending the boundaries between adjacent blocks.

[0742] Similarly, the inter-frame prediction unit 218 applies (reuses) the MV (MV_U) already derived for the decoded previous adjacent block to the current block to obtain a prediction image (Pred_U). Then, the inter-frame prediction unit 218 performs a second correction to the prediction image by aligning the prediction image Pred_U with the prediction image (e.g., Pred and Pred_L) that has undergone the first correction. This has the effect of blending the boundaries between adjacent blocks. The prediction image obtained through the second correction is the final prediction image of the current block, blended (smoothed) with the boundaries of adjacent blocks.

[0743] [Motion Compensation > BIO] For example, in the case of an application where the information read from the stream represents BIO, when generating a prediction image, the inter-frame prediction unit 218 corrects the prediction image according to BIO.

[0744] Figure 95 This is a flowchart illustrating an example of BIO-based correction of a predicted image in the decoding device 200.

[0745] like Figure 63 As shown, the inter-frame prediction unit 218 uses two reference images (Ref0, Ref1) that are different from the image (CurPic) containing the current block to derive two motion vectors (M0, M1). Then, the inter-frame prediction unit 218 uses these two motion vectors (M0, M1) to derive the prediction image of the current block (step Sy_11). In addition, motion vector M0 is the motion vector (MVx0, MVy0) corresponding to reference image Ref0, and motion vector M1 is the motion vector (MVx1, MVy1) corresponding to reference image Ref1.

[0746] Next, the inter-frame prediction unit 218 uses the motion vector M0 and the reference image L0 to derive the interpolated image I of the current block. 0 Additionally, the inter-frame prediction unit 218 uses motion vector M1 and reference image L1 to derive the interpolated image I of the current block. 1 (Step Sy_12). Here, the interpolated image I 0 It is the interpolated image I, which is the image contained in the reference image Ref0 exported from the current block. 1 This refers to the image exported from the current block, contained in reference image Ref1. Interpolated image I 0 and interpolated image I 1 They can be of the same size as the current block. Alternatively, to properly derive the gradient image described later, the interpolated image I... 0 and interpolated image I 1 It can also be an image larger than the current block. Furthermore, the interpolated image I... 0 and I 1It can include a predicted image derived by applying motion vectors (M0, M1) and reference images (L0, L1), as well as a motion compensation filter.

[0747] Additionally, the inter-frame prediction unit 218 obtains data from the interpolated image I. 0 and interpolated image I 1 Export the gradient image of the current block (Ix) 0 , Ix 1 ,Iy 0 ,Iy 1 (Step Sy_13). Furthermore, the gradient image in the horizontal direction is (Ix 0 , Ix 1 The gradient image in the vertical direction is (Ix) 0 , Ix 1 The inter-frame prediction unit 218 can also derive the gradient image, for example, by applying a gradient filter to the interpolated image. The gradient image can be any image that represents the spatial variation of pixel values ​​along the horizontal or vertical direction.

[0748] Next, the inter-frame prediction unit 218 uses interpolated images (I1, I2, I3, to form multiple sub-block units constituting the current block.) 0 I 1 ) and gradient image (Ix 0 , Ix 1 ,Iy 0 ,Iy 1 Derive the optical flow (vx, vy) as the velocity vector described above (step Sy_14). As an example, a sub-block can be a 4x4 pixel sub-CU.

[0749] Next, the inter-frame prediction unit 218 uses optical flow (vx, vy) to correct the predicted image of the current block. For example, the inter-frame prediction unit 218 uses optical flow (vx, vy) to derive correction values ​​for the pixel values ​​contained in the current block (step Sy_15). Then, the inter-frame prediction unit 218 can use the correction values ​​to correct the predicted image of the current block (step Sy_16). Furthermore, the correction values ​​can be derived per pixel unit, or per multiple pixel units, or per sub-block unit.

[0750] Furthermore, the BIO processing flow is not limited to Figure 95 The publicly disclosed processing method can be implemented by simply performing... Figure 95 The disclosed processing can be supplemented or replaced with different processing, or executed in different processing orders.

[0751] [Motion Compensation > LIC] For example, in the case of the application of LIC (Literally Indicative Language) for information read from the stream, when generating a prediction image, the inter-frame prediction unit 218 corrects the prediction image according to the LIC.

[0752] Figure 96 This is a flowchart illustrating an example of LIC-based correction of a predicted image in the decoding device 200.

[0753] First, the inter-frame prediction unit 218 uses MV to obtain a reference image corresponding to the current block from the decoded reference image (step Sz_11).

[0754] Next, the inter-frame prediction unit 218 extracts information for the current block indicating how the brightness values ​​change in the reference image and the current image (step Sz_12). For example... Figure 66A As shown, this extraction is based on the brightness pixel values ​​of the decoded left adjacent reference region (peripheral reference region) and the decoded upper adjacent reference region (peripheral reference region) in the current image, and the brightness pixel values ​​at the same position in the reference image specified by the derived MV. Then, the inter-frame prediction unit 218 uses information indicating how the brightness values ​​change to calculate the brightness correction parameters (step Sz_13).

[0755] The inter-frame prediction unit 218 generates a prediction image for the current block by applying brightness correction parameters to a reference image within the reference image specified by the MV. That is, the prediction image, which serves as a reference image within the reference image specified by the MV, is corrected based on the brightness correction parameters. This correction can correct both brightness and chromatic aberration.

[0756] [Forecasting and Control Department] The prediction control unit 220 selects one of the intra-frame prediction image and the inter-frame prediction image, and outputs the selected prediction image to the addition unit 208. Generally, the structure, function, and processing of the prediction control unit 220, intra-frame prediction unit 216, and inter-frame prediction unit 218 on the decoding device 200 side can correspond to the structure, function, and processing of the prediction control unit 128, intra-frame prediction unit 124, and inter-frame prediction unit 126 on the encoding device 100 side.

[0757] [Structure of an image processing device] First, the structure of the image processing device will be explained. Figure 97 This is a block diagram illustrating the structure of an image processing apparatus 300. The image processing apparatus 300 includes a circuit 300a and a memory 300b. The image processing apparatus 300 is an apparatus for processing images. Specifically, the image processing apparatus 300 extracts a sub-bitstream of a sub-image from a bitstream of an image. The image processing apparatus 300 can be an apparatus that functions as an extractor.

[0758] Circuit 300a is a circuit that performs information processing and can access memory 300b. For example, circuit 300a is a dedicated or general-purpose electronic circuit for processing images. Circuit 300a can be a processor such as a CPU. Alternatively, circuit 300a can be an assembly of multiple electronic circuits. The basic operation of the image processing device 300 can also be performed by circuit 300a.

[0759] Memory 300b is a dedicated or general-purpose memory used to store information about images processed by circuit 300a. Memory 300b can be an electronic circuit or can be connected to circuit 300a. Alternatively, memory 300b can be included within circuit 300a. Furthermore, memory 300b can also be an assembly of multiple electronic circuits.

[0760] Furthermore, the memory 300b can be a disk, optical disk, etc., or it can be a storage device, recording medium, etc. Additionally, the memory 300b can be either non-volatile or volatile memory.

[0761] For example, memory 300b can store the image to be processed, or it can store the bit stream corresponding to the processed image. Additionally, memory 300b can also store the program used by circuit 300a to process the image.

[0762] [Internal structure of the sub-image extraction section] Figure 98 This is a graph representing the syntax associated with sub-pictures. Specifically, Figure 98 The syntax of the Sequence Parameter Set (SPS) in the bitstream is shown.

[0763] To form a decodable sub-bitstream, the sub-picture is designed to contain a rectangular region enclosed by the encoded picture that can be easily extracted from the bitstream. Therefore, when extracting a sub-picture, the values ​​of the rewritten Sequence Parameter Set (SPS) and / or Picture Parameter Set (PPS) of the bitstream need to be saved to the SPS and / or PPS of the sub-bitstream.

[0764] like Figure 98 As shown in the syntax, by encoding the coordinates of the top-left CTU (Coding Tree Unit) of a sub-image and the size of the sub-image, the region of each sub-image is indicated by the original SPS contained in the bitstream. In terms of luminance, the size of the CTU is the same as the size of the CTB (Coding Tree Block). Therefore, CTU can be read as CTB. Furthermore, CTB refers to the block containing the luminance or chrominance signals (values) that constitute the CTU; the CTU is composed of the luminance CTB and the chrominance CTB.

[0765] The coordinates of the top-left CTU of the sub-image include both the horizontal and vertical coordinates of the CTU. For example, the coordinates of the CTU could also be the coordinates of the top-left corner of the CTU. Alternatively, the coordinates of the CTU could also be the coordinates of the top-right corner of the CTU or the coordinates of the center of the CTU.

[0766] Specifically, Figure 98 In this context, `subpic_ctu_top_left_x[i]` represents the coordinates of the CTU in the horizontal direction, expressed in CTB units. Additionally, Figure 98 In this context, subpic_ctu_top_left_y[i] represents the coordinates of the CTU in the vertical direction, expressed in CTB units.

[0767] More specifically, `subpic_ctu_top_left_x[i]` represents the horizontal position of the top-left CTU of the i-th subpicture in CTB dimensions. Furthermore, CTB dimensions can be converted to CTU dimensions.

[0768] That is, subpic_ctu_top_left_x[i] represents the position of the top-left CTU of the i-th subpicture from the left edge of the image. Here, the leftmost CTU is the 0th CTU from the left.

[0769] Additionally, `subpic_ctu_top_left_y[i]` represents the vertical position of the top-left CTU of the i-th sub-image in CTB units. That is, `subpic_ctu_top_left_y[i]` indicates which CTU the top-left CTU of the i-th sub-image is from the top of the image. Here, the topmost CTU is the 0th CTU from the top.

[0770] In addition, the dimensions of a sub-image include its width and height. Specifically, by Figure 98 In this context, `subpic_width_minus1[i]` represents the width of the i-th sub-image. Additionally, from... Figure 98 In this context, subpic_height_minus1[i] represents the height of the i-th subpic.

[0771] More specifically, `subpic_width_minus1[i]` represents the width of the i-th subpicture in CTB units. More specifically, `subpic_width_minus1[i]` represents the number of CTUs in the horizontal (width) direction of the i-th subpicture. For example, `subpic_width_minus1[i]` is the value obtained by subtracting 1 from the number of CTUs in the width direction of the i-th subpicture.

[0772] Additionally, `subpic_height_minus1[i]` represents the height of the subpicture in CTB units. More specifically, `subpic_height_minus1[i]` represents the number of CTUs in the vertical (height) direction of the i-th subpicture. For example, `subpic_height_minus1[i]` is the value obtained by subtracting 1 from the number of CTUs in the height direction of the i-th subpicture.

[0773] The overall size of the image is determined by... Figure 98 In the `pic_width_max_in_luma_samples` and `pic_height_max_in_luma_samples`, pic_width_max_in_luma_samples is encoded in units of luminance samples. `pic_width_max_in_luma_samples` represents the width of the image. `pic_height_max_in_luma_samples` represents the height of the image. When extracting sub-bitstreams associated with sub-images from the bitstream associated with the image, the SPS of the sub-bitstream is generated by replacing the width (pic_width_max_in_luma_samples) and height (pic_height_max_in_luma_samples) of the image within the bitstream's SPS with the width and height of the sub-image, respectively.

[0774] In the SPS of a bitstream, the SPS of a sub-bitstream can be generated by replacing the values ​​of `pic_width_max_in_luma_samples` and `pic_height_max_in_luma_samples` of the image with the values ​​of `subpic_width_minus1` and `sub_pic_height_minus1` of the corresponding sub-images. Alternatively, the SPS of the bitstream can be copied by rewriting `pic_width_max_in_luma_samples` and `pic_height_max_in_luma_samples` with the corresponding sub-image's `subpic_width_minus1` and `sub_pic_height_minus1`, or the SPS of the bitstream associated with the image can be copied without updating, and the SPS of the sub-bitstream can be calculated.

[0775] Figure 99 This is a diagram illustrating an example of sub-image boundaries and consistent clipping windows within an image.

[0776] The first aspect of this disclosure relates to an import restriction that sub-pictures should not be defined outside the conformance clipping window contained in the bitstream. A sub-picture not being contained within the conformance clipping window means that the sub-bitstream of that sub-picture is a stream that does not contain any display object area. For a bitstream containing a sub-picture not contained within the conformance clipping window, since the rules defined for the conformance clipping window are not applied, a non-compliant bitstream may be generated. This leads to the problem that even a decoding device that complies with the rules may produce an undecodeable sub-bitstream. Therefore, the restriction is introduced that sub-pictures should not be defined outside the conformance clipping window contained in the bitstream.

[0777] Figure 99 An example is shown in a bitstream suitable for the first form of this disclosure, where all sub-pictures E, F, K, J, and L are completely blocked because all sub-pictures E, F, K, J, and L are completely outside the uniform clipping window.

[0778] on the contrary, Figure 99 Any of the sub-images A, B, C, D, G, H, and I shown are allowed to be defined as sub-images according to the first aspect of this disclosure because at least a portion of them are located inside the uniform clipping window.

[0779] The limitations of the first aspect of this disclosure can be represented as follows: The left, right, top, and bottom offsets of the consistent cropping window are represented by the variables `sps_conf_win_left_offset`, `sps_conf_win_right_offset`, `sps_conf_win_top_offset`, and `sps_conf_win_bottom_offset`, and the overall width and height of the image are represented by the variables `pic_width_max_in_luma_samples` and `pic_height_max_in_luma_samples`. The size of the coding tree unit (CTU is square, so both width and height are consistent) is represented by the variable `CtbSizeY`, and the upsampling coefficients between chromatic difference and luminance are represented by `SubWidthC` in the horizontal direction and by `SubHeightC` in the vertical direction (e.g., in the case of 4:2:0 content, both `SubWidthC` and `SubHeightC` are equal to 2).

[0780] For all subpicks i in the range from 0 to sps_num_subpics_minus1, all of the following conditions (a) to (d) are true requirements for the bitstream to be suitable.

[0781] (a) (subpic_ctu_top_left_x[i] × CtbSizeY) is less than (pic_width_max_in_luma_samples-1-sps_conf_win_right_offset × SubWidthC). That is, the left end of the subpicture is located to the left of the right end of the consistency clipping window.

[0782] (b) ((subpic_ctu_top_left_x[i]+subpic_width_minus1[i]+1)×CtbSizeY-1) is greater than (sps_conf_win_left_offset×SubWidthC). That is, the right end of the subpicture is located to the right of the left end of the consistency clipping window.

[0783] (c) (subpic_ctu_top_left_y[i] × CtbSizeY) is less than (pic_height_max_in_luma_samples-1-sps_conf_win_bottom_offset × SubHeightC). That is, the top of the subpicture is located above the bottom of the consistency clipping window.

[0784] (d) ((subpic_ctu_top_left_y[i]+subpic_height_minus1[i]+1)×CtbSizeY-1) is greater than (sps_conf_win_top_offset×SubHeightC). That is, the bottom of the subpicture is located below the top of the consistent clipping window.

[0785] Additionally, as mentioned above, the upsampling coefficients expressed in SubWidthC and SubHeightC vary depending on the image format. Furthermore, the units of sps_conf_win_xxxx_offset (where xxxx is one of left, right, top, and bottom) also vary depending on the image format.

[0786] Specifically, in the 4:4:4 format, SubWidthC is 1 and SubHeightC is 1. Furthermore, in this case, the units of sps_conf_win_xxxx_offset correspond to a region of 1 luminance sample size in the horizontal direction and a region of 1 luminance sample size in the vertical direction.

[0787] Additionally, in the 4:2:2 format, SubWidthC is 1 and SubHeightC is 2. Furthermore, in this case, the units of sps_conf_win_xxxx_offset correspond to a region with 1 luminance sample size in the horizontal direction and a region with 2 luminance samples size in the vertical direction.

[0788] Additionally, in the 4:2:0 format, SubWidthC is 2 and SubHeightC is 2. Furthermore, in this case, the units of sps_conf_win_xxxx_offset correspond to a region with 2 luminance samples in the horizontal direction and a region with 2 luminance samples in the vertical direction.

[0789] Therefore, by multiplying sps_conf_win_xxxx_offset by SubWidthC or SubHeightC, the offset value corresponding to the number of luminance samples can be obtained.

[0790] [Example of a variation of the first form] As a variation of the first form, in the above four conditions, "less than" can also be replaced with "below", and "greater than" can also be replaced with "above".

[0791] Furthermore, the above four conditions can be replaced with conditions that indicate that at least part of the sub-image is located inside the consistent clipping window, as long as they represent conditions that indicate that the sub-image is also located inside the consis...

Claims

1. An encoding device, wherein, have: Circuits; and The memory is connected to the circuit. The circuit is in operation. Each of the multiple sub-images contained in the design image. The configuration information representing the configurations of multiple sub-images contained within an image is encoded. Each of the plurality of sub-images is encoded. The circuit is in operation. Each of the plurality of sub-images is designed such that at least one pixel in each sub-image is contained within the same consistent cropping window in the image. The conditions for designing the plurality of sub-images include that the left end of each of the plurality of sub-images is located to the left of the right end of the consistency clipping window.

2. A decoding device, wherein, have: Circuits; and The memory is connected to the circuit. The circuit is in operation. The configuration information representing the configurations of multiple sub-images contained in an image is decoded, and the multiple sub-images are determined according to the conditions of the multiple sub-images. Decode each of the plurality of sub-images. The condition is that at least one pixel in each of the plurality of sub-images is contained within the same consistent cropping window of the image, which serves as the display object area. The condition includes that the left end of each of the plurality of sub-images is located to the left of the right end of the consistent cropping window.

3. An encoding method, wherein, Each of the multiple sub-images contained in the design image. The configuration information representing the individual configurations of multiple sub-images within an image is encoded. Each of the plurality of sub-images is encoded. Each of the plurality of sub-images is designed such that at least one pixel in each sub-image is contained within the same consistent cropping window in the image. The conditions for designing the plurality of sub-images include that the left end of each of the plurality of sub-images is located to the left of the right end of the consistency clipping window.

4. A decoding method, wherein, The configuration information representing the configurations of multiple sub-images contained in an image is decoded, and the multiple sub-images are determined according to the conditions of the multiple sub-images. Decode each of the plurality of sub-images. The condition is that at least one pixel in each of the plurality of sub-images is contained within the same consistent cropping window of the image, which serves as the display object area. The condition includes that the left end of each of the plurality of sub-images is located to the left of the right end of the consistent cropping window.

5. A bit stream transmitting device, wherein, have: Memory; and The circuit is connected to the memory. The circuit is in operation. Generate the first parameter representing the configuration of each of the multiple sub-images contained in the image. The circuit is in operation. Send a bit stream containing the first parameter. The first parameter indicates that each of the plurality of sub-images in the image is configured such that at least one pixel contained in each sub-image is included in the same consistent cropping window in the image. The left end of each of the plurality of sub-images is located to the left of the right end of the consistent cropping window.

6. A non-transitory storage medium for storing computer programs and bit streams, wherein, When the computer program is executed by a processor, the encoding method of claim 3 is implemented to generate the bit stream. The bitstream includes a first parameter, which represents the configuration of each of the multiple sub-images contained in the image. The first parameter indicates that each of the plurality of sub-images in the image is configured such that at least one pixel contained in each sub-image is included in the same consistent cropping window in the image. The left end of each of the plurality of sub-images is located to the left of the right end of the consistent cropping window.

Images