Encoding device, decoding device, transmission device, encoding method, decoding method and transmission method

Abstract

The disclosure relates to a transmission device, an encoding device, a decoding device, a transmission method, an encoding method, and a decoding method. This transmission device is provided with: a circuit that, when an image to be processed is a progressive scan image during operation, writes one syntactic element indicating a color difference positioning type into a bit stream; the color difference positioning type is information indicating a position of a color difference sample with respect to a luminance sample for a frame included in the image to be processed, and the color difference positioning type is information indicating a position of a color difference sample with respect to a luminance sample in a case where the image to be processed is not a progressive scan image. Two syntax elements representing the color difference positioning type for each of the two fields included in the image to be processed are written into the bitstream, and the bitstream is transmitted.

Description

[0001] This application is a divisional of the patent application with the application date of December 8, 2020, the application number of 202080083566.9, and the invention name of "Encoding device, decoding device, encoding method, and decoding method". TECHNICAL FIELD

[0002] The present invention relates to video encoding, and particularly relates to systems, components, and methods in encoding and decoding of moving images. BACKGROUND

[0003] Video encoding technology has progressed from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). Along with this progress, in order to handle the increasing amount of digital video data in various uses, there has been a constant need to provide improvements and optimizations of video encoding technology. The present invention relates to further progress, improvements, and optimizations in video encoding.

[0004] Furthermore, Non-Patent Literature 1 relates to an example of an existing standard related to the above-described video encoding technology.

[0005] Prior Art Documents

[0006] Non-Patent Literature

[0007] Non-Patent Literature 1: H.265 (ISO / IEC 23008-2 HEVC) / HEVC (High Efficiency Video Coding) SUMMARY OF THE INVENTION

[0008] PROBLEMS TO BE SOLVED BY THE INVENTION

[0009] With regard to the above-described encoding method, for improvement of encoding efficiency, improvement of picture quality, reduction of processing amount, reduction of circuit size, or appropriate selection of elements or actions such as filters, blocks, sizes, motion vectors, reference pictures, or reference blocks, it is desirable to propose a new method.

[0010] The present invention provides a structure or method that can contribute to one or more of, for example, improvement of encoding efficiency, improvement of picture quality, reduction of processing amount, reduction of circuit size, improvement of processing speed, and appropriate selection of elements or actions. Furthermore, the present invention can include a structure or method that can contribute to benefits other than the above.

[0011] Means for solving problems

[0012] For example, a sending device in one form of the present invention includes a circuit and a memory connected to the circuit. When the circuit is in operation, if the processing object image is a line-by-line scan image, one syntactic element representing a color difference positioning type is written into a bit stream. The color difference positioning type is information representing the position of a color difference sample relative to a luminance sample for a frame included in the processing object image. If the processing object image is not a line-by-line scan image, two syntactic elements representing the color difference positioning type respectively for two fields included in the processing object image are written into the bit stream, and the bit stream is sent.

[0013] In video coding technology, new methods are expected to be proposed in order to improve coding efficiency, improve image quality, reduce circuit size, and the like.

[0014] The structures or methods of the various embodiments or portions thereof of the present invention can achieve, for example, at least one of improved coding efficiency, improved image quality, reduced encoding / decoding processing load, reduced circuit size, or improved encoding / decoding processing speed. Alternatively, the structures or methods of the various embodiments or portions thereof of the present invention can appropriately select components / operations such as filters, blocks, sizes, motion vectors, reference pictures, and reference blocks during encoding and decoding. Furthermore, the present invention also encompasses the disclosure of structures or methods that can provide benefits beyond those described above. For example, structures or methods that improve coding efficiency while suppressing increases in processing load can be used.

[0015] The present invention provides further advantages and effects according to the present invention. These advantages and / or effects are achieved through several embodiments and features described in the present invention and the accompanying drawings, but all advantages and / or effects do not necessarily need to be provided in order to achieve one or more advantages and / or effects.

[0016] In addition, these general or specific forms can be implemented by systems, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0017] Effects of the Invention

[0018] The structure or method of one aspect of the present invention can contribute to one or more of, for example, improved coding efficiency, improved image quality, reduced processing load, reduced circuit size, improved processing speed, and appropriate selection of elements or operations. Furthermore, the structure or method of one aspect of the present invention can also contribute to benefits other than those listed above. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram showing an example of the configuration of a transmission system according to an embodiment.

[0020] Figure 2 This is a diagram showing an example of the hierarchical structure of data in a stream.

[0021] Figure 3 This is a diagram showing an example of the structure of a slice.

[0022] Figure 4 This is a diagram showing an example of a tile structure.

[0023] Figure 5 This is a diagram showing an example of a coding structure in the case of scalable coding.

[0024] Figure 6 This is a diagram showing an example of a coding structure in the case of scalable coding.

[0025] Figure 7 This is a block diagram showing an example of the functional structure of the encoding device according to the embodiment.

[0026] Figure 8 This is a block diagram showing an example of an implementation of an encoding device.

[0027] Figure 9 This is a flowchart showing an example of the overall encoding process performed by the encoding device.

[0028] Figure 10 This is a diagram showing an example of block division.

[0029] Figure 11 This is a diagram showing an example of the functional configuration of a dividing unit.

[0030] Figure 12 A diagram showing an example of a segmentation pattern.

[0031] Figure 13A This is a diagram showing an example of a syntax tree of a segmentation pattern.

[0032] Figure 13B This is a diagram showing another example of a syntax tree of a segmentation pattern.

[0033] Figure 14 This is a table showing the transformation basis functions corresponding to each transformation type.

[0034] Figure 15 This is a diagram showing an example of SVT.

[0035] Figure 16 This is a flowchart showing an example of processing performed by the conversion unit.

[0036] Figure 17This is a flowchart showing another example of the processing performed by the conversion unit.

[0037] Figure 18 This is a block diagram showing an example of the functional configuration of a quantization unit.

[0038] Figure 19 This is a flowchart showing an example of quantization performed by the quantization unit.

[0039] Figure 20 This is a block diagram showing an example of the functional structure of the entropy coding unit.

[0040] Figure 21 This is a diagram showing the CABAC process in the entropy coding unit.

[0041] Figure 22 This is a block diagram showing an example of the functional configuration of a loop filter unit.

[0042] Figure 23A This is a diagram showing an example of the shape of a filter used in an ALF (adaptive loop filter).

[0043] Figure 23B FIG. 1 is a diagram showing another example of the shape of the filter used in ALF.

[0044] Figure 23C FIG. 1 is a diagram showing another example of the shape of the filter used in ALF.

[0045] Figure 23D This is a diagram showing an example in which a Y sample (first component) is used for CCALF of Cb and CCALF of Cr (a plurality of components different from the first component).

[0046] Figure 23E is a diagram showing a diamond-shaped filter.

[0047] Figure 23F It is a diagram showing an example of JC-CCALF.

[0048] Figure 23G This is a diagram showing examples of weight_index candidates of JC-CCALF.

[0049] Figure 24 This is a block diagram showing an example of a detailed configuration of a loop filter unit functioning as a DBF.

[0050] Figure 25 This is a diagram showing an example of deblocking filtering having filter characteristics that are symmetric with respect to block boundaries.

[0051] Figure 26 This is a diagram for explaining an example of a block boundary on which a deblocking filtering process is performed.

[0052] Figure 27 is a graph showing an example of Bs value.

[0053] Figure 28 is a flowchart showing an example of the processing by the prediction section of the encoding apparatus.

[0054] Figure 29 is a flowchart showing another example of the processing by the prediction section of the encoding apparatus.

[0055] Figure 30 is a flowchart showing another example of the processing by the prediction section of the encoding apparatus.

[0056] Figure 31 is a graph showing an example of 67 intra prediction modes in intra prediction.

[0057] Figure 32 is a flowchart showing an example of the processing by the intra prediction section.

[0058] Figure 33 is a graph showing an example of each reference picture.

[0059] Figure 34 is a conceptual diagram showing an example of a reference picture list.

[0060] Figure 35 is a flowchart showing a flow of the basic processing of inter prediction.

[0061] Figure 36 is a flowchart showing an example of MV derivation.

[0062] Figure 37 is a flowchart showing another example of MV derivation.

[0063] Figure 38A is a graph showing an example of classification of each mode of MV derivation.

[0064] Figure 38B is a graph showing an example of classification of each mode of MV derivation.

[0065] Figure 39 is a flowchart showing an example of inter prediction based on normal inter mode.

[0066] Figure 40 is a flowchart showing an example of inter prediction based on normal merge mode.

[0067] Figure 41 is a graph for explaining an example of MV derivation processing based on normal merge mode.

[0068] Figure 42This is a diagram for explaining an example of MV derivation processing based on the HMVP mode.

[0069] Figure 43 This is a flowchart showing an example of FRUC (frame rate up conversion).

[0070] Figure 44 This is a diagram for explaining an example of pattern matching (bidirectional matching) between two blocks along a motion trajectory.

[0071] Figure 45 This is a diagram for explaining an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture.

[0072] Figure 46A This is a diagram for explaining an example of derivation of a sub-block-based MV in an affine mode using two control points.

[0073] Figure 46B This is a diagram for explaining an example of derivation of MVs in sub-block units in the affine mode using three control points.

[0074] Figure 47A This is a conceptual diagram for explaining an example of MV derivation of control points in the affine mode.

[0075] Figure 47B This is a conceptual diagram for explaining an example of MV derivation of control points in the affine mode.

[0076] Figure 47C This is a conceptual diagram for explaining an example of MV derivation of control points in the affine mode.

[0077] Figure 48A A diagram for explaining an affine pattern having two control points.

[0078] Figure 48B A diagram for explaining an affine pattern having three control points.

[0079] Figure 49A This is a conceptual diagram for explaining an example of a method for deriving MVs of control points when the number of control points in an already coded block and a current block is different.

[0080] Figure 49B This is a conceptual diagram for explaining another example of a method for deriving an MV of a control point when the number of control points in an already coded block and a current block is different.

[0081] Figure 50 This is a flowchart showing an example of processing in the affine merge mode.

[0082] Figure 51is a flowchart showing an example of the process of the affine inter mode.

[0083] Figure 52A is a diagram for explaining generation of a prediction image of 2 triangles.

[0084] Figure 52B is a conceptual diagram showing an example of a first part of a first partition and a first sample set and a second sample set.

[0085] Figure 52C is a conceptual diagram showing a first part of a first partition.

[0086] Figure 53 is a flowchart showing an example of the triangle mode.

[0087] Figure 54 is a diagram showing an example of the ATMVP mode in which an MV is derived in a sub-block unit.

[0088] Figure 55 is a diagram showing a relationship between the merge mode and the DMVR (dynamic motion vector refreshing).

[0089] Figure 56 is a conceptual diagram for explaining an example of the DMVR.

[0090] Figure 57 is a conceptual diagram for explaining another example of the DMVR for deciding an MV.

[0091] Figure 58A is a diagram showing an example of a motion search in the DMVR.

[0092] Figure 58B is a flowchart showing an example of a motion search in the DMVR.

[0093] Figure 59 is a flowchart showing an example of generation of a prediction image.

[0094] Figure 60 is a flowchart showing another example of generation of a prediction image.

[0095] Figure 61 is a flowchart for explaining an example of a prediction image correction process based on the OBMC (overlapped block motion compensation).

[0096] Figure 62 is a conceptual diagram for explaining an example of the prediction image correction process based on the OBMC.

[0097] Figure 63This is a diagram for explaining a model assuming uniform linear motion.

[0098] Figure 64 This is a flowchart showing an example of inter-frame prediction according to BIO.

[0099] Figure 65 This is a diagram showing an example of the functional structure of an inter-frame prediction unit that performs inter-frame prediction according to BIO.

[0100] Figure 66A This is a diagram for explaining an example of a method for generating a predicted image using a brightness correction process based on LIC (local illumination compensation).

[0101] Figure 66B This is a flowchart showing an example of a method for generating a predicted image using a brightness correction process based on LIC.

[0102] Figure 67 This is a block diagram showing the functional structure of a decoding device according to an embodiment.

[0103] Figure 68 This is a block diagram showing an implementation example of a decoding device.

[0104] Figure 69 This is a flowchart showing an example of the overall decoding process performed by the decoding device.

[0105] Figure 70 This is a diagram showing the relationship between the division determination unit and other components.

[0106] Figure 71 This is a block diagram showing an example of the functional structure of the entropy decoding unit.

[0107] Figure 72 This is a diagram showing the CABAC process in the entropy decoding unit.

[0108] Figure 73 This is a block diagram showing an example of the functional configuration of an inverse quantization unit.

[0109] Figure 74 This is a flowchart showing an example of inverse quantization performed by the inverse quantization unit.

[0110] Figure 75 This is a flowchart showing an example of processing performed by the inverse transformation unit.

[0111] Figure 76 This is a flowchart showing another example of the processing performed by the inverse transformation unit.

[0112] Figure 77 This is a block diagram showing an example of the functional configuration of a loop filter unit.

[0113] Figure 78 This is a flowchart showing an example of processing performed by the prediction unit of the decoding device.

[0114] Figure 79 This is a flowchart showing another example of processing performed by the prediction unit of the decoding device.

[0115] Figure 80A This is a flowchart showing a part of another example of processing performed by the prediction unit of the decoding device.

[0116] Figure 80B This is a flowchart showing the remaining portion of another example of processing performed by the prediction unit of the decoding device.

[0117] Figure 81 This is a diagram showing an example of processing performed by the intra prediction unit of the decoding device.

[0118] Figure 82 This is a flowchart showing an example of MV derivation in the decoding device.

[0119] Figure 83 This is a flowchart showing another example of MV derivation in the decoding device.

[0120] Figure 84 This is a flowchart showing an example of inter-frame prediction based on the normal inter-frame mode in a decoding device.

[0121] Figure 85 This is a flowchart illustrating an example of inter-frame prediction based on the normal merge mode in a decoding device.

[0122] Figure 86 This is a flowchart showing an example of inter-frame prediction based on the FRUC mode in the decoding device.

[0123] Figure 87 This is a flowchart illustrating an example of inter-frame prediction based on the affine merge mode in a decoding device.

[0124] Figure 88 This is a flowchart illustrating an example of inter prediction based on the affine inter mode in a decoding device.

[0125] Figure 89 This is a flowchart showing an example of inter-frame prediction based on the triangular mode in a decoding device.

[0126] Figure 90 This is a flowchart showing an example of a DMVR-based motion search in a decoding device.

[0127] Figure 91 This is a flowchart showing a detailed example of a motion search based on DMVR in a decoding device.

[0128] Figure 92 is a flowchart showing an example of generation of a prediction image in a decoding device.

[0129] Figure 93 is a flowchart showing another example of generation of a prediction image in a decoding device.

[0130] Figure 94 is a flowchart showing an example of correction of a prediction image based on OBMC in a decoding device.

[0131] Figure 95 is a flowchart showing an example of correction of a prediction image based on BIO in a decoding device.

[0132] Figure 96 is a flowchart showing an example of correction of a prediction image based on LIC in a decoding device.

[0133] Figure 97 is a diagram showing an example of a VUI (Video Usability Information) syntax structure.

[0134] Figure 98 is a flowchart showing an example of actions performed by a decoding device of the first aspect.

[0135] Figure 99 is a flowchart showing actions performed by an encoding device.

[0136] Figure 100 is a flowchart showing actions performed by a decoding device.

[0137] Figure 101 is a diagram showing an overall structure of a content supply system that implements a content distribution service.

[0138] Figure 102 is a diagram showing an example of a display screen of a web page.

[0139] Figure 103 is a diagram showing an example of a display screen of a web page.

[0140] Figure 104 is a diagram showing an example of a smart phone.

[0141] Figure 105 is a block diagram showing an example of a structure of a smart phone. DETAILED DESCRIPTION

[0142] [INTRODUCTION]

[0143] The coding device of one aspect of the present disclosure includes a circuit and a memory connected to the circuit. In operation, the circuit determines whether a processing target image is a progressive image, encodes one syntax element indicating a chroma sample position type into a bitstream when the processing target image is determined to be the progressive image, the chroma sample position type being information indicating a position of a chroma sample with respect to a luma sample for a frame included in the processing target image, and encodes two syntax elements each indicating the chroma sample position type for two fields included in the processing target image into the bitstream when the processing target image is determined not to be the progressive image.

[0144] Thus, the coding device encodes only one syntax element indicating a chroma sample position type when the processing target image is a progressive image, and thus the processing amount is reduced. Therefore, the coding efficiency of the coding device is improved.

[0145] For example, the circuit can determine whether the processing target image is a progressive image or an interlaced image using a first flag and a second flag included in a video usability information syntax structure, the first flag indicating whether the processing target image is the progressive image, and the second flag indicating whether the processing target image is the interlaced image.

[0146] Thus, even when the processing target image includes an image for which a display mode (i.e., a scanning mode in a display) is not specified, for example, the coding device can determine whether the processing target image is a progressive image or an interlaced image. Therefore, the coding device can more appropriately process the processing target image, and thus the coding efficiency is improved.

[0147] For example, the circuit can further encode a third flag into the bitstream, the third flag indicating whether information indicating a position of a chroma sample with respect to a luma sample for the frame is included in the bitstream as the one syntax element or the two syntax elements, encode the one syntax element into the bitstream when the third flag indicates that the information indicating the position of the chroma sample with respect to the luma sample for the frame is included in the bitstream and the processing target image is determined to be the progressive image using the first flag and the second flag, and encode the two syntax elements into the bitstream when the third flag indicates that the information indicating the position of the chroma sample with respect to the luma sample for the frame is included in the bitstream and the processing target image is determined to be the interlaced image using the first flag and the second flag.

[0148] This allows the encoding device to switch the syntax elements to be encoded depending on whether the image being processed is a progressive scan image or an interlaced scan image. Therefore, the encoding device can process the image being processed more appropriately, thereby improving encoding efficiency.

[0149] In addition, a decoding device in one form of the present invention includes a circuit and a memory connected to the circuit, and when the circuit is in operation, it determines whether the processing object image is an interlaced scan image. When it is determined that the processing object image is an interlaced scan image, one syntax element representing a color difference positioning type is decoded from a bit stream. The color difference positioning type is information indicating the position of a color difference sample relative to a luminance sample for a frame included in the processing object image. When it is determined that the processing object image is not an interlaced scan image, two syntax elements respectively representing the color difference positioning type for two fields included in the processing object image are decoded from the bit stream.

[0150] Thus, when the image being processed is a progressive image, the decoding device only needs to decode one syntax element indicating the color difference positioning type, thereby reducing the amount of processing and improving the processing efficiency of the decoding device.

[0151] For example, the circuit may use the first flag and the second flag contained in the video availability information syntax structure to determine whether the image to be processed is a progressive scan image or an interlaced scan image, wherein the first flag indicates whether the image to be processed is a progressive scan image, and the second flag indicates whether the image to be processed is an interlaced scan image.

[0152] Thus, even when the processing target video includes an image whose display format (i.e., the scanning method used in the display) is unspecified, the decoding device can still determine whether the processing target video is a progressive scan video or an interlaced scan video. This allows the decoding device to more appropriately process the processing target video, thereby improving processing efficiency.

[0153] For example, it can also be that the circuit further decodes a third flag from the bitstream, the third flag indicating whether information indicating positions of the chroma samples with respect to the luma samples for a frame is included in the bitstream as the one syntax element or the two syntax elements, in a case where the third flag indicates that the information indicating the positions of the chroma samples with respect to the luma samples for the frame is included in the bitstream, and it is determined using the first flag and the second flag that the processing target image is a progressive image, the one syntax element is decoded from the bitstream, and in a case where the third flag indicates that the information indicating the positions of the chroma samples with respect to the luma samples for the frame is included in the bitstream, and it is determined using the first flag and the second flag that the processing target image is an interlaced image, the two syntax elements are decoded from the bitstream.

[0154] Thus, the decoding device can switch the syntax element to be decoded according to whether the processing target image is a progressive image or an interlaced image. Therefore, the decoding device can perform more appropriate processing on the processing target image, and thus the processing efficiency is improved.

[0155] In addition, the encoding method of an aspect of the present disclosure determines whether a processing target image is a progressive image, in a case where it is determined that the processing target image is a progressive image, encodes one syntax element indicating a chroma positioning type, which is information indicating positions of chroma samples with respect to luma samples for a frame included in the processing target image, to a bitstream, and in a case where it is determined that the processing target image is not a progressive image, encodes two syntax elements respectively indicating the chroma positioning type for two fields included in the processing target image to the bitstream.

[0156] Thus, the device executing the encoding method encodes only the one syntax element indicating the chroma positioning type in a case where the processing target image is a progressive image, and thus the processing amount is reduced. Therefore, the encoding efficiency of the device executing the encoding method is improved.

[0157] In addition, the decoding method of an aspect of the present disclosure determines whether a processing target image is a progressive image, in a case where it is determined that the processing target image is a progressive image, decodes one syntax element indicating a chroma positioning type from a bitstream, the chroma positioning type being information indicating positions of chroma samples with respect to luma samples for a frame included in the processing target image, and in a case where it is determined that the processing target image is not a progressive image, decodes two syntax elements respectively indicating the chroma positioning type for two fields included in the processing target image from the bitstream.

[0158] Thus, the apparatus that executes the decoding method decodes only one syntax element indicating a color difference positioning type in a case where the processing target image is a progressive image, and thus the processing amount is reduced. Therefore, the processing efficiency of the apparatus that executes the decoding method is improved.

[0159] [Definitions of Terms]

[0160] As an example, each term can be defined in the following manner.

[0161] (1) Image

[0162] It is a unit of data composed of a set of pixels, composed of a picture or a block smaller than a picture, and includes still images in addition to moving images.

[0163] (2) Picture

[0164] It is a processing unit of an image composed of a set of pixels, sometimes referred to as a frame or a field.

[0165] (3) Block

[0166] It is a processing unit including a set of a certain number of pixels, and the name is not limited as exemplified in the following examples. In addition, the shape is not limited, and for example, of course, a rectangle composed of MxN pixels, a square composed of MxM pixels, and also a triangle, a circle, and other shapes are included.

[0167] (Examples of Blocks)

[0168] • Slice / Tile / Brick

[0169] • CTU / Superblock / Basic Partition Unit

[0170] • VPDU / Hardware Processing Partition Unit

[0171] • CU / Processing Block Unit / Prediction Block Unit (PU) / Orthogonal Transform Block Unit (TU) / Unit

[0172] • Subblock

[0173] (4) Pixel / Sample

[0174] It is a point that is the smallest unit constituting an image, and includes not only an integer position pixel but also a decimal position pixel generated based on the integer position pixel.

[0175] (5) Pixel Value / Sample Value

[0176] It is an inherent value that a pixel has, and of course includes a luminance value, a color difference value, a gray scale of RGB, and also a depth value, or a 0, 1 binary.

[0177] (6) Logo

[0178] In addition to 1 bit, multiple bits are also possible, for example, parameters or indexes of 2 or more bits may be used. In addition, not only binary values ​​but also multiple values ​​using numbers in other bases may be used.

[0179] (7) Signal

[0180] Symbolization and encoding are performed to transmit information, and include not only discrete digital signals but also analog signals that take continuous values.

[0181] (8) Stream / Bitstream

[0182] A stream or bitstream is a sequence of digital data or a stream of digital data. A stream or bitstream can be a single stream or can be composed of multiple streams divided into multiple layers. Furthermore, this includes transmission over a single transmission path using serial communication as well as transmission over multiple transmission paths using packet communication.

[0183] (9) Difference / Differential

[0184] In the case of scalars, in addition to simple difference (xy), any difference operation can be performed, including the absolute value of the difference (|xy|), square difference (x^2-y^2), square root of the difference (√(xy)), weighted difference (ax-by: a and b are constants), and offset difference (x-y+a: a is the offset).

[0185] (10) and

[0186] In the case of scalars, in addition to the simple sum (x+y), any sum operation can be performed, including the absolute value of the sum (|x+y|), the square sum (x^2+y^2), the square root of the sum (√(x+y)), the weighted sum (ax+by: a and b are constants), and the offset sum (x+y+a: a is the offset).

[0187] (11) based on

[0188] This also includes cases where elements other than the one being based are added. In addition, in addition to cases where a direct result is obtained, this also includes cases where a result is obtained via an intermediate result.

[0189] (12) Use (used, using)

[0190] This also includes the case where elements other than the ones to be used are added. In addition, in addition to the case where a direct result is obtained, the case where a result is obtained via an intermediate result is also included.

[0191] (13) prohibit (prohibit, forbid)

[0192] It can also be called non-permission. In addition, non-prohibition or permission does not necessarily mean obligation.

[0193] (14) Restriction (limit, restriction / restrict / restricted)

[0194] It can also be called "not allowed." Furthermore, not prohibiting or allowing something does not necessarily mean an obligation. Furthermore, a partial prohibition in terms of quantity or quality is sufficient, and this includes a complete prohibition.

[0195] (15) Chromatic aberration (chroma)

[0196] An adjective denoted by the symbols Cb and Cr, specifying that a sample array or a single sample represents one of two color difference signals associated with a primary color. Furthermore, the term "chrominance" can be used instead of the term "chroma."

[0197] (16) Luma

[0198] An adjective that specifies that a sample arrangement or a single sample represents a monochromatic signal associated with a primary color, and is represented by the symbol or subscript Y or L. The term luminance can also be used instead of luma.

[0199] [Explanation related to the record]

[0200] In the drawings, the same reference numerals represent the same or similar components. In addition, the sizes and relative positions of the components in the drawings are not necessarily drawn to scale.

[0201] The following embodiments are described in detail with reference to the accompanying drawings. The embodiments described below are intended to be inclusive or specific examples. The numerical values, shapes, materials, components, configurations and connections of components, steps, and the relationships and order of steps shown in the following embodiments are merely examples and are not intended to limit the scope of the claims.

[0202] The following describes embodiments of encoding and decoding devices. The embodiments are examples of encoding and decoding devices to which the processing and / or structures described in the various aspects of the present invention can be applied. The processing and / or structures can also be implemented in encoding and decoding devices that differ from the embodiments. For example, the processing and / or structures applied to the embodiments may include any of the following.

[0203] (1) Any of the multiple components of the encoding device or decoding device described in the embodiments of the present invention may be replaced by another component described in any of the embodiments of the present invention, or these components may be combined.

[0204] (2) In the encoding device or decoding device of the embodiment, the functions or processes performed by some of the multiple components of the encoding device or decoding device may be modified by adding, replacing, deleting, or other arbitrary changes. For example, any function or process may be replaced by another function or process described in any of the various aspects of the present invention, or these functions or processes may be combined.

[0205] (3) In the method implemented by the encoding device or decoding device of the embodiment, any changes such as addition, replacement, or deletion may be made to a portion of the multiple processes included in the method. For example, any process in the method may be replaced with another process described in one of the various aspects of the present invention, or these processes may be combined.

[0206] (4) Some of the multiple components constituting the encoding device or decoding device of the embodiment may be combined with a component described in any of the aspects of the present invention, may be combined with a component having a portion of the functions described in any of the aspects of the present invention, or may be combined with a component that performs a portion of the processing performed by a component described in any of the aspects of the present invention.

[0207] (5) A component having a portion of the functions of the encoding device or decoding device of the embodiment, or a component implementing a portion of the processing of the encoding device or decoding device of the embodiment, is combined with or replaced with a component described in any of the aspects of the present invention, a component having a portion of the functions described in any of the aspects of the present invention, or a component implementing a portion of the processing described in any of the aspects of the present invention;

[0208] (6) In a method implemented by an encoding device or decoding device according to an embodiment, one of the multiple processes included in the method is replaced by one of the processes described in each aspect of the present invention or a similar process, or a combination of these processes;

[0209] (7) Some of the multiple processes included in the method implemented by the encoding device or decoding device of the embodiment may be combined with the process described in any of the aspects of the present invention.

[0210] (8) The implementation of the processes and / or configurations described in the aspects of the present application is not limited to the encoding apparatus or the decoding apparatus of the embodiments. For example, the processes and / or configurations can also be implemented in an apparatus used for a purpose different from the motion picture encoding or the motion picture decoding disclosed in the embodiments.

[0211] [System configuration]

[0212] Figure 1 is a schematic diagram showing an example of the configuration of the transmission system of the present embodiment.

[0213] The transmission system Trs is a system that transmits a stream generated by encoding an image and decodes the transmitted stream. Such a transmission system Trs includes, for example, as shown in Figure 1 , an encoding apparatus 100, a network Nw, and a decoding apparatus 200.

[0214] An image is input to the encoding apparatus 100. The encoding apparatus 100 generates a stream by encoding the input image and outputs the stream to the network Nw. The stream contains, for example, an encoded image and control information for decoding the encoded image. The image is compressed by the encoding.

[0215] Further, the original image before being encoded, which is input to the encoding apparatus 100, is also referred to as an original image, an original signal, or an original sample. In addition, the image can be a moving image or a still image. In addition, the image is a higher-order concept of a sequence, a picture, a block, and the like, and is not limited by a region in space and time unless otherwise specified. In addition, the image is constituted by an arrangement of pixels or pixel values, and a signal or a pixel value representing the image is also referred to as a sample. In addition, the stream can be referred to as a bit stream, an encoded bit stream, a compressed bit stream, or an encoded signal. Furthermore, the encoding apparatus can also be referred to as an image encoding apparatus or a motion picture encoding apparatus, and the encoding method of the encoding apparatus 100 can also be referred to as an encoding method, an image encoding method, or a motion picture encoding method.

[0216] The network Nw transmits the stream generated by the encoding apparatus 100 to the decoding apparatus 200. The network Nw can be the Internet, a wide area network (WAN), a small-scale network (LAN), or a combination thereof. The network Nw is not necessarily limited to a bidirectional communication network, but can also be a unidirectional communication network that transmits a broadcast wave such as a terrestrial digital broadcast or a satellite broadcast. In addition, the network Nw can also be replaced by a storage medium in which a stream of a DVD (Digital Versatile Disc), a BD (Blu-Ray Disc (registered trademark)), or the like is recorded.

[0217] The decoding device 200 decodes a stream transmitted via the network Nw to generate a decoded image, for example, an uncompressed image. For example, the decoding device decodes the stream using a decoding method corresponding to the encoding method of the encoding device 100 .

[0218] In addition, the decoding device may also be referred to as an image decoding device or a moving image decoding device, and the decoding method of the decoding device 200 may also be referred to as a decoding method, an image decoding method, or a moving image decoding method.

[0219] [Data structure]

[0220] Figure 2 is a diagram showing an example of a hierarchical structure of data in a stream. The stream includes, for example, a video sequence. The video sequence is, for example, Figure 2 As shown in (a), it includes VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), SEI (Supplemental Enhancement Information) and multiple pictures.

[0221] The VPS includes coding parameters common to the multiple layers in a moving picture composed of multiple layers and coding parameters associated with the multiple layers included in the moving picture or with each layer.

[0222] The SPS includes parameters used for the sequence, that is, encoding parameters that the decoding device 200 refers to in order to decode the sequence. For example, the encoding parameters may also represent the width or height of the picture. In addition, there may be multiple SPSs.

[0223] The PPS contains parameters used for pictures, namely, coding parameters that decoding apparatus 200 references to decode each picture in a sequence. For example, these coding parameters may include a reference value for the quantization width used in picture decoding and a flag indicating the use of weighted prediction. Furthermore, multiple PPSs may exist. SPSs and PPSs are sometimes referred to simply as parameter sets.

[0224] like Figure 2 As shown in (b) of FIG. 1 , a picture may include a picture header and one or more slices. The picture header includes coding parameters that the decoding apparatus 200 refers to in order to decode the one or more slices.

[0225] like Figure 2 As shown in (c), a slice includes a slice header and one or more tiles. The slice header includes coding parameters that the decoding device 200 refers to in order to decode the one or more tiles.

[0226] like Figure 2 As shown in (d), a brick includes one or more CTUs (Coding Tree Units).

[0227] Furthermore, a picture may not contain slices but may contain tile groups instead of slices. In this case, a tile group contains one or more tiles. Alternatively, a tile may contain a slice.

[0228] CTU is also called super block or basic partition unit. Figure 2 As shown in (e) of FIG. 1 , such a CTU includes a CTU header and one or more CUs (Coding Units). The CTU header includes coding parameters that the decoding apparatus 200 refers to in order to decode one or more CUs.

[0229] A CU can also be split into multiple small CUs. Figure 2 As shown in (f), the CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information used to predict the CU, and the residual coefficient information is information representing the prediction residual described later. In addition, the CU is basically the same as the PU (Prediction Unit) and the TU (Transform Unit), but for example, in the SBT described later, it may also include multiple TUs smaller than the CU. In addition, the CU may also process each VPDU (Virtual Pipeline Decoding Unit) that constitutes the CU. The VPDU is a fixed unit that can be processed in one stage when pipeline processing is performed in hardware, for example.

[0230] In addition, the stream may not have Figure 2 The hierarchy of any part of the hierarchies shown. In addition, the order of these hierarchies can be swapped, and any hierarchy can be replaced by another hierarchy. In addition, the picture of the object being processed by the apparatus such as the encoding apparatus 100 or the decoding apparatus 200 at the current time point is referred to as the current picture. If the processing is encoding, the current picture is synonymous with the encoding object picture, and if the processing is decoding, the current picture is synonymous with the decoding object picture. In addition, the block of the object such as a CU or a CU etc. being processed by the apparatus such as the encoding apparatus 100 or the decoding apparatus 200 at the current time point is referred to as the current block. If the processing is encoding, the current block is synonymous with the encoding object block, and if the processing is decoding, the current block is synonymous with the decoding object block.

[0231] [Structural slices / tiles of the image]

[0232] In order to decode a picture in parallel, the picture is sometimes constituted by a slice unit or a tile unit.

[0233] A slice is a basic coding unit constituting a picture. A picture is constituted by one or more slices, for example. Further, a slice is constituted by one or more consecutive CTUs.

[0234] Figure 3 is a diagram showing an example of a structure of a slice. For example, a picture contains 11 x 8 CTUs, and is divided into 4 slices (slices 1 to 4). Slice 1 is constituted by 16 CTUs, for example, slice 2 is constituted by 21 CTUs, for example, slice 3 is constituted by 29 CTUs, for example, and slice 4 is constituted by 22 CTUs, for example. Here, each CTU within a picture belongs to a certain slice. The shape of a slice becomes a shape in which a picture is divided in the horizontal direction. The boundary of a slice need not be a picture end, but can be somewhere in the boundary of a CTU within a picture. The processing order (coding order or decoding order) of CTUs in a slice is, for example, a raster scan order. Further, a slice contains a slice header and coding data. In the slice header, the CTU address of the beginning of the slice, the slice type, and the like, which are characteristics of the slice, can be described.

[0235] A tile is a unit of a rectangular region constituting a picture. Each tile can be assigned a number called Tileld in a raster scan order.

[0236] Figure 4 is a diagram showing an example of a structure of a tile. For example, a picture contains 11 x 8 CTUs, and is divided into 4 rectangular region tiles (tiles 1 to 4). In the case of using a tile, the processing order of CTUs is changed compared to the case of not using a tile. In the case of not using a tile, a plurality of CTUs within a picture are processed in a raster scan order, for example. In the case of using a tile, in each of a plurality of tiles, at least one CTU is processed in a raster scan order, for example. For example, as shown in Figure 4 , the processing order of a plurality of CTUs contained in tile 1 is an order from the left end of the 1st column of tile 1 toward the right end of the 1st column of tile 1, and then from the left end of the 2nd column of tile 1 toward the right end of the 2nd column of tile 1.

[0237] Further, one tile sometimes contains one or more slices, and one slice sometimes contains one or more tiles.

[0238] In addition, a picture may also be composed of tile set units. A tile set may contain one or more tile groups, or may contain one or more tiles. A picture may be composed of only one of a tile set, a tile group, and a tile. For example, the order in which a plurality of tiles are scanned in raster order for each tile set is set as the basic encoding order of the tiles. A collection of one or more tiles having a continuous basic encoding order in each tile set is set as a tile group. Such a picture may also be generated by the segmentation unit 102 described later (see Figure 7 )constitute.

[0239] [Scalable Coding]

[0240] Figure 5 and Figure 6 This is a diagram showing an example of the structure of a scalable stream.

[0241] like Figure 5 As shown, encoding device 100 can generate a temporally and spatially scalable stream by encoding multiple pictures by dividing them into one of multiple layers. For example, encoding device 100 achieves scalability by encoding each picture layer, with enhancement layers positioned above a base layer. This type of encoding of each picture is called scalable encoding. This allows decoding device 200 to switch the image quality of the image displayed by decoding the stream. Specifically, decoding device 200 determines which layer to decode based on internal factors such as its own performance and external factors such as the state of the communication band. As a result, decoding device 200 can freely switch between decoding the same content at low and high resolutions. For example, a user of this stream might be viewing a moving image on a smartphone while on the move, then viewing the remainder of the moving image on a device such as an internet TV after returning home. Furthermore, the smartphone and the device may each be equipped with a decoding device 200 with either the same or different performance. In this case, if the device decodes the higher-level layer of the stream, the user can view high-quality moving images upon returning home. This eliminates the need for the encoding device 100 to generate multiple streams having the same content but different image quality, thereby reducing the processing load.

[0242] Furthermore, the enhancement layer may also include metadata such as statistical information based on the image. Alternatively, the decoding device 200 may generate a high-definition moving image by super-resolutioning the base layer picture based on the metadata. Super-resolution may be achieved by either increasing the signal-to-noise ratio at the same resolution or increasing the resolution. Meta-information includes information for determining linear or nonlinear filter coefficients used in the super-resolution process, as well as information for determining parameter values ​​in filtering, machine learning, or least-squares operations used in the super-resolution process.

[0243] Alternatively, the image may be divided into tiles according to the meaning of each object in the image. In this case, the decoding device 200 can decode only a portion of the area in the image by selecting the tile to be decoded. Furthermore, the attributes of the object (person, car, ball, etc.) and the position in the image (coordinate position in the same image, etc.) can be saved as meta information. In this case, the decoding device 200 can determine the position of the desired object based on the meta information and decide the tile containing the object. For example, Figure 6 As shown, a data storage structure different from pixel data such as SEI in HEVC can also be used to store meta-information. This meta-information, for example, indicates the position, size, or color of the main object.

[0244] Alternatively, the meta-information may be stored in units consisting of multiple pictures, such as streams, sequences, or random access units. This allows the decoding device 200 to determine the time at which a specific person appears in a moving image, and, using this time and picture-based information, to identify the picture in which the object appears and the location of the object within that picture.

[0245] [Encoding device]

[0246] Next, the encoding device 100 according to the embodiment will be described. Figure 7 1 is a block diagram showing an example of the functional configuration of the encoding device 100 according to the embodiment. The encoding device 100 encodes an image in units of blocks.

[0247] like Figure 7 As shown, the encoding device 100 is a device that encodes an image in units of blocks, and includes a segmentation unit 102, a subtraction unit 104, a transformation unit 106, a quantization unit 108, an entropy coding unit 110, an inverse quantization unit 112, an inverse transformation unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. The intra-frame prediction unit 124 and the inter-frame prediction unit 126 each constitute a part of a prediction processing unit.

[0248] [Encoding device installation example]

[0249] Figure 8 1 is a block diagram showing an implementation example of the coding device 100. The coding device 100 includes a processor a1 and a memory a2. For example, Figure 7 The multiple components of the encoding device 100 shown are Figure 8 The processor a1 and the memory a2 shown are implemented.

[0250] Processor a1 is a circuit that processes information and can access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit that encodes images. Processor a1 can also be a processor such as a CPU. In addition, processor a1 can also be a collection of multiple electronic circuits. In addition, for example, processor a1 can also play a role in Figure 7 The functions of the multiple components of the encoding device 100 excluding the components for storing information are shown.

[0251] Memory a2 is a dedicated or general-purpose memory that stores information used by processor a1 to encode images. Memory a2 can be an electronic circuit or connected to processor a1. Alternatively, memory a2 can be included in processor a1. Alternatively, memory a2 can be a collection of multiple electronic circuits. Memory a2 can be a magnetic disk or optical disk, or can be a storage device or recording medium. Memory a2 can be either non-volatile or volatile memory.

[0252] For example, the memory a2 may store coded images or a stream corresponding to the coded images. In addition, the memory a2 may store a program for the processor a1 to encode images.

[0253] In addition, for example, memory a2 can also serve as Figure 7 The memory a2 can be used to store information among the multiple components of the encoding device 100 shown in FIG. Figure 7 The functions of the block memory 118 and the frame memory 122 are shown. More specifically, the memory a2 can store reconstructed images (specifically, reconstructed blocks and reconstructed pictures, etc.).

[0254] In addition, in the encoding device 100, it is not necessary to install Figure 7 Not all of the multiple components shown may perform all of the multiple processes described above. Figure 7 Some of the multiple components shown may be included in other devices, and some of the multiple processes described above may be executed by other devices.

[0255] Below, after describing the overall processing flow of the encoding device 100 , each component included in the encoding device 100 will be described.

[0256] [Overall flow of encoding processing]

[0257] Figure 9 3 is a flowchart showing an example of the overall encoding process performed by the encoding device 100 .

[0258] First, the segmentation unit 102 of the encoding device 100 segments the picture contained in the original image into a plurality of fixed-size blocks (128×128 pixels) (step Sa_1). The segmentation unit 102 then selects a segmentation pattern for the fixed-size blocks (step Sa_2). Specifically, the segmentation unit 102 further segments the fixed-size blocks into a plurality of blocks that conform to the selected segmentation pattern. The encoding device 100 then performs steps Sa_3 through Sa_9 on each of the plurality of blocks.

[0259] The prediction processing unit composed of the intra prediction unit 124 and the inter prediction unit 126 and the prediction control unit 128 generates a predicted image of the current block (step Sa_3). The predicted image is also called a prediction signal, a prediction block, or a prediction sample.

[0260] Next, the subtraction unit 104 generates the difference between the current block and the predicted image as a prediction residual (step Sa_4). The prediction residual is also called a prediction error.

[0261] Next, the transform unit 106 and the quantization unit 108 transform and quantize the predicted image to generate a plurality of quantization coefficients (step Sa_5).

[0262] Next, the entropy coding unit 110 generates a stream by encoding (specifically, entropy coding) the plurality of quantized coefficients and prediction parameters related to generation of a predicted image (step Sa_6).

[0263] Next, the inverse quantization unit 112 and the inverse transformation unit 114 restore the prediction residual by performing inverse quantization and inverse transformation on the plurality of quantized coefficients (step Sa_7).

[0264] Next, the adder 116 reconstructs the current block by adding the predicted image to the restored prediction residual (step Sa_8). This generates a reconstructed image. Furthermore, a reconstructed image is also referred to as a reconstructed block. In particular, a reconstructed image generated by the encoding device 100 is also referred to as a locally decoded block or locally decoded image.

[0265] When the reconstructed image is generated, the loop filter unit 120 filters the reconstructed image as necessary (step Sa_9).

[0266] Then, the encoding device 100 determines whether encoding of the entire picture is completed (step Sa_10 ), and if it is determined that encoding is not completed (No in step Sa_10 ), it repeats the process from step Sa_2 .

[0267] Furthermore, in the above example, encoding device 100 selects a single partitioning pattern for fixed-size blocks and encodes each block according to that partitioning pattern. However, encoding device 100 may also encode each block according to each of multiple partitioning patterns. In this case, encoding device 100 may evaluate the cost for each of the multiple partitioning patterns and, for example, select the stream obtained by encoding according to the partitioning pattern with the lowest cost as the final output stream.

[0268] Furthermore, the processing of steps Sa_1 to Sa_10 may be performed sequentially by the encoding device 100 , or a portion or a plurality of these processes may be performed in parallel or in a different order.

[0269] The encoding process performed by encoding device 100 is a hybrid encoding process using predictive coding and transform coding. Predictive coding is performed using a coding loop comprised of a subtraction unit 104, a transform unit 106, a quantization unit 108, an inverse quantization unit 112, an inverse transform unit 114, an addition unit 116, a loop filter unit 120, a block memory 118, a frame memory 122, an intra-frame prediction unit 124, an inter-frame prediction unit 126, and a prediction control unit 128. Specifically, the prediction processing unit comprised of the intra-frame prediction unit 124 and the inter-frame prediction unit 126 constitutes part of the coding loop.

[0270] [Division]

[0271] The segmentation unit 102 segments each picture included in the original image into a plurality of blocks, and outputs each block to the subtraction unit 104. For example, the segmentation unit 102 first segments the picture into blocks of a fixed size (for example, 128×128 pixels). The fixed-size blocks are sometimes called coding tree units (CTUs). Furthermore, the segmentation unit 102 segments each fixed-size block into blocks of a variable size (for example, less than 64×64 pixels), for example, based on recursive quadtree and / or binary tree block segmentation. That is, the segmentation unit 102 selects a segmentation pattern. The variable-size blocks are sometimes called coding units (CUs), prediction units (PUs), or transform units (TUs. In addition, in various installation examples, it is not necessary to distinguish between CUs, PUs, and TUs, and a part or all of the blocks in the picture may be used as processing units of CUs, PUs, or TUs.

[0272] Figure 10 : is a diagram showing an example of block division in an embodiment. Figure 10 In the figure, the solid line represents the block boundary based on quadtree block partitioning, and the dotted line represents the block boundary based on binary tree block partitioning.

[0273] Here, the block 10 is a square block of 128 x 128 pixels. The block 10 is first partitioned into four square blocks of 64 x 64 pixels (quadtree block partitioning).

[0274] The upper left 64×64 pixel square block is further vertically divided into two rectangular blocks of 32×64 pixels each, and the left 32×64 pixel square block is further vertically divided into two rectangular blocks of 16×64 pixels each (binary tree block division). As a result, the upper left 64×64 pixel square block is divided into two 16×64 pixel rectangular blocks 11 and 12, and a 32×64 pixel rectangular block 13.

[0275] The upper right square block of 64×64 pixels is horizontally divided into two rectangular blocks 14 and 15 each consisting of 64×32 pixels (binary tree block division).

[0276] The 64×64 pixel square block in the lower left is divided into four square blocks of 32×32 pixels each (quadtree block division). The upper left block and the lower right block of the four square blocks of 32×32 pixels each are further divided. The 32×32 pixel square block in the upper left is vertically divided into two rectangular blocks of 16×32 pixels each, and the rectangular block of 16×32 pixels on the right is further horizontally divided into two square blocks of 16×16 pixels each (binary tree block division). The 32×32 pixel square block in the lower right is horizontally divided into two rectangular blocks of 32×16 pixels each (binary tree block division). As a result, the 64×64 pixel square block in the lower left corner is divided into a rectangular block 16 of 16×32 pixels, two square blocks 17 and 18 of 16×16 pixels each, two square blocks 19 and 20 of 32×32 pixels each, and two rectangular blocks 21 and 22 of 32×16 pixels each.

[0277] The lower right block 23 consisting of 64×64 pixels is not divided.

[0278] As above, in Figure 10 In FIG, block 10 is divided into 13 variable-size blocks 11 to 23 based on recursive quadtree and binary tree block partitioning. This type of partitioning is sometimes called QTBT (quad-tree plus binary tree) partitioning.

[0279] In addition, Figure 10In the example above, one block is divided into four or two blocks (quadtree or binary tree block division), but the division is not limited to these. For example, one block can also be divided into three blocks (ternary tree division). Division including this ternary tree division is called MBT (multi type tree) division.

[0280] Figure 11 1 is a diagram showing an example of the functional structure of the segmentation unit 102. Figure 11 As shown, the division unit 102 may include a block division determination unit 102a. As an example, the block division determination unit 102a may perform the following processing.

[0281] The block division determination unit 102a collects block information from, for example, the block memory 118 or the frame memory 122 and determines the above-mentioned division pattern based on the block information. The division unit 102 divides the original image according to the division pattern and outputs one or more blocks obtained by the division to the subtraction unit 104.

[0282] Furthermore, the block division determination unit 102a outputs, for example, parameters indicating the above-described division pattern to the transformation unit 106, the inverse transformation unit 114, the intra prediction unit 124, the inter prediction unit 126, and the entropy coding unit 110. The transformation unit 106 can transform the prediction residual based on the parameters, and the intra prediction unit 124 and the inter prediction unit 126 can generate a predicted image based on the parameters. Furthermore, the entropy coding unit 110 can also perform entropy coding on the parameters.

[0283] As an example, parameters related to the segmentation pattern may be written into the stream as follows.

[0284] Figure 12 This figure shows examples of partitioning patterns. Examples include: four-partitioning (QT), which divides a block into two in both the horizontal and vertical directions; three-partitioning (HT or VT), which divides a block in the same direction at a ratio of 1:2:1; two-partitioning (HB or VB), which divides a block in the same direction at a ratio of 1:1; and no partitioning (NS).

[0285] In the case of four-way division and no division, the division pattern does not have the block division direction, while in the case of two-way division and three-way division, the division pattern has division direction information.

[0286] Figure 13A and Figure 13B This is a diagram showing an example of a syntax tree of a segmentation pattern. Figure 13AIn the example of FIG. 8, first, information indicating whether or not to perform division (S: Split flag) is present, and then information indicating whether or not to perform quad-tree division (QT: QT flag) is present. Then, information indicating whether to perform ternary-tree division or binary-tree division (TT: TT flag or BT: BT flag) is present, and finally information indicating the division direction (Ver: Vertical flag or Hor: Horizontal flag) is present. In addition, the same processing can be further repeatedly applied to division to each of one or more blocks obtained by such division based on the division pattern. That is, as an example, the determination of whether or not to perform division, whether or not to perform quad-tree division, whether the division method is horizontal or vertical, and whether to perform ternary-tree division or binary-tree division can be recursively performed, and the results of the determination performed can be encoded in the order disclosed by the syntax tree of FIG. 8. Figure 13A

[0287] In addition, in the syntax tree of FIG. 8, the information is arranged in the order of S, QT, TT, and Ver, but the information can be arranged in the order of S, QT, Ver, and BT. That is, in the example of FIG. 8, first, information indicating whether or not to perform division (S: Split flag) is present, and then information indicating whether or not to perform quad-tree division (QT: QT flag) is present. Then, information indicating the division direction (Ver: Vertical flag or Hor: Horizontal flag) is present, and finally information indicating whether to perform ternary-tree division or binary-tree division (BT: BT flag or TT: TT flag) is present. Figure 13A Figure 13B In addition, the division pattern described herein is an example, and a division pattern other than the division pattern described can be used, or only a part of the division pattern described can be used.

[0288] In addition, the division pattern described herein is an example, and a division pattern other than the division pattern described can be used, or only a part of the division pattern described can be used.

[0289] [Subtracting section]

[0290] The subtracting section 104 subtracts a prediction image (a prediction image input from the prediction control section 128) from an original image in units of blocks input from and divided by the dividing section 102. That is, the subtracting section 104 calculates a prediction residual of a current block. Also, the subtracting section 104 outputs the calculated prediction residual to the transforming section 106.

[0291] The original image is an input signal of the encoding apparatus 100, and is, for example, a signal (e.g., a luma signal and two chroma signals) indicating an image of each picture constituting a moving image.

[0292] [Transforming section]

[0293] ​​The transform unit 106 transforms the spatial domain prediction residual into frequency domain transform coefficients and outputs the transform coefficients to the quantization unit 108. Specifically, the transform unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the spatial domain prediction residual.

[0294] Alternatively, the transform unit 106 may adaptively select a transform type from a plurality of transform types and transform the prediction residual into transform coefficients using a transform basis function corresponding to the selected transform type. Such a transform is sometimes referred to as an explicit multiple core transform (EMT) or an adaptive multiple transform (AMT). Furthermore, the transform basis function is sometimes referred to as simply a basis.

[0295] The plurality of transform types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Furthermore, these transform types may be described as DCT2, DCT5, DCT8, DST1, and DST7, respectively. Figure 14 is a table showing the transformation basis functions corresponding to each transformation type. Figure 14 Where N represents the number of input pixels. The selection of a transform type from among these multiple transform types may depend on, for example, the type of prediction (intra-frame prediction, inter-frame prediction, etc.) or the intra-frame prediction mode.

[0296] Information indicating whether such EMT or AMT is applied (e.g., an EMT flag or an AMT flag) and information indicating the selected transform type are typically signaled at the CU level. However, signaling of this information is not limited to the CU level and may also be performed at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0297] Further, the transform unit 106 can also perform a re-transformation on the transform coefficients (i.e., the transform result). Such a re-transformation is referred to as AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transform unit 106 performs a re-transformation on each sub-block (e.g., a 4x4 pixel sub-block) included in a block of transform coefficients corresponding to the intra prediction residual. Information indicating whether or not to apply NSST and information related to a transform matrix used in NSST are generally signaled at the CU level. In addition, the signaling of these pieces of information does not need to be limited to the CU level, and can be at another level (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0298] The transform unit 106 can apply both a separable transform and a non-separable transform. The separable transform refers to a manner in which a plurality of transforms are performed in each direction separately with respect to the number of dimensions of the input, and the non-separable transform refers to a manner in which, when the input is multi-dimensional, two or more dimensions are regarded as one dimension and a transform is performed on the dimensions together.

[0299] For example, as an example of the non-separable transform, there is a manner in which, when the input is a block of 4x4 pixels, the block is regarded as one arrangement having 16 elements, and a transform process is performed on the arrangement with a 16x16 transform matrix.

[0300] Further, in a further example of the non-separable transform, a transform in which Givens rotation is performed on an arrangement having 16 elements a plurality of times (Hypercube Givens Transform) can be performed after the input block of 4x4 pixels is regarded as the arrangement.

[0301] In the transform in the transform unit 106, the type of transform to be performed on the transform basis function in the frequency domain can be switched according to the region within the CU. As an example, there is SVT (Spatially Varying Transform).

[0302] Figure 15 is a diagram illustrating an example of SVT.

[0303] In the SVT, as Figure 15As shown, the CU is divided into two equal parts in the horizontal direction or the vertical direction, and only the area on either side is transformed into the frequency domain. The transformation type can be set for each area, for example, using DST7 and DCT8. For example, for the area at position 0 in the two areas obtained by dividing the CU into two equal parts in the vertical direction, DST7 and DCT8 can be used. Or, for the area at position 1 in the two areas, DST7 is used. Similarly, for the area at position 0 in the two areas obtained by dividing the CU into two equal parts in the horizontal direction, DST7 and DCT8 are used. Or, for the area at position 1 in the two areas, DST7 is used. In such a Figure 15 In the example shown, only one of the two regions within the CU is transformed, while the other is not. However, it is also possible to transform both regions separately. In addition, the partitioning method is not limited to bisection; it can also be quartered. In addition, it is possible to be more flexible and encode the information indicating the partitioning method and perform signaling in the same way as CU partitioning. In addition, SVT is sometimes also called SBT (Sub-block Transform).

[0304] The aforementioned AMT and EMT can also be referred to as MTS (Multiple Transform Selection). When MTS is applied, a transform type such as DST7 or DCT8 can be selected, and information indicating the selected transform type can be encoded as index information for each CU. On the other hand, as a process of selecting the transform type used in the orthogonal transform based on the shape of the CU without encoding the index information, there is a process called IMTS (Implicit MTS). When IMTS is applied, for example, if the shape of the CU is rectangular, DST7 is used on the short side of the rectangle and DCT2 is used on the long side, and orthogonal transforms are performed separately. In addition, for example, when the shape of the CU is square, if MTS is valid within the sequence, DCT2 is used for orthogonal transform, and if MTS is invalid, DST7 is used for orthogonal transform. DCT2 and DST7 are examples, and other transform types can be used, or the combination of transform types used can be set to different combinations. IMTS can be used only in intra-frame predicted blocks, or it can be used together in intra-frame predicted blocks and inter-frame predicted blocks.

[0305] In the above, the three processes of MTS, SBT, and IMTS have been described as the selection process for selectively switching the transform type used in the orthogonal transform. However, all three selection processes may be valid, or only some of the selection processes may be selectively valid. Whether each selection process is valid can be identified by flag information in the header such as the SPS. For example, if all three selection processes are valid, one of the three selection processes is selected for orthogonal transform in units of CUs. In addition, as long as the selection process for selectively switching the transform type can achieve at least one of the following four functions [1] to [4], a selection process different from the above three selection processes may be used, or each of the above three selection processes may be replaced by another process. Function [1] is a function of performing an orthogonal transform on the entire range within the CU and encoding information indicating the transform type used in the transform. Function [2] is a function of performing an orthogonal transform on the entire range of the CU, not encoding information indicating the transform type, and determining the transform type based on a prescribed rule. Function [3] is a function of performing an orthogonal transform on a part of the CU and encoding information indicating the transform type used in the transform. Function [4] is a function of performing an orthogonal transform on a part of the CU area, and determining the transform type based on a predetermined rule without encoding information indicating the transform type used in the transform.

[0306] Furthermore, the presence or absence of each of MTS, IMTS, and SBT can be determined for each processing unit, for example, in sequence units, picture units, tile units, slice units, CTU units, or CU units.

[0307] In addition, the tool for selectively switching the transformation type in the present invention can also be renamed as a method for adaptively selecting a basis used in the transformation process, a selection process, or a process for selecting a basis. In addition, the tool for selectively switching the transformation type can also be renamed as a mode for adaptively selecting the transformation type.

[0308] Figure 16 This is a flowchart showing an example of processing performed by the conversion unit 106.

[0309] For example, the transform unit 106 determines whether to perform an orthogonal transform (step St_1). If the decision is to perform an orthogonal transform (yes in step St_1), the transform unit 106 selects a transform type from a plurality of transform types (step St_2). Next, the transform unit 106 performs an orthogonal transform by applying the selected transform type to the prediction residual of the current block (step St_3). The transform unit 106 then outputs information indicating the selected transform type to the entropy coding unit 110, which encodes the information (step St_4). On the other hand, if the decision is not to perform an orthogonal transform (no in step St_1), the transform unit 106 outputs information indicating that an orthogonal transform is not performed to the entropy coding unit 110, which encodes the information (step St_5). The decision to perform an orthogonal transform in step St_1 can be based on, for example, the size of the transform block, the prediction mode applied to the CU, and so on. Alternatively, information indicating the transform type used for the orthogonal transform may not be encoded, and an orthogonal transform may be performed using a predetermined transform type.

[0310] Figure 17 : is a flowchart showing another example of the processing performed by the conversion unit 106. Figure 17 The example shown is similar to Figure 16 The example shown is also an example of orthogonal transform in the case of applying a method of selectively switching the transform type used in the orthogonal transform.

[0311] As an example, the first transform type group may include DCT2, DCT7, and DCT8. Furthermore, as an example, the second transform type group may include DCT2. Furthermore, the transform types included in the first and second transform type groups may overlap in part or may be completely different.

[0312] Specifically, the transform unit 106 determines whether the transform size is less than or equal to a predetermined value (step Su_1). If it is determined to be less than or equal to the predetermined value (yes in step Su_1), the transform unit 106 performs an orthogonal transform on the prediction residual of the current block using the transform type included in the first transform type group (step Su_2). Next, the transform unit 106 outputs information indicating which transform type, from the one or more transform types included in the first transform type group, is to be used to the entropy coding unit 110, thereby encoding the information (step Su_3). On the other hand, if it is determined that the transform size is not less than or equal to the predetermined value (no in step Su_1), the transform unit 106 performs an orthogonal transform on the prediction residual of the current block using the second transform type group (step Su_4).

[0313] In step Su_3, the information indicating the transform type used for the orthogonal transform may be information indicating a combination of a transform type applied to the vertical direction and a transform type applied to the horizontal direction of the current block. Furthermore, the first transform type group may include only one transform type, and the information indicating the transform type used for the orthogonal transform may not be encoded. The second transform type group may include multiple transform types, and information indicating the transform type used for the orthogonal transform among one or more transform types included in the second transform type group may be encoded.

[0314] Alternatively, the transform type may be determined based solely on the transform size. Furthermore, if the transform type used for orthogonal transform is determined based on the transform size, the determination is not limited to determining whether the transform size is equal to or smaller than a predetermined value.

[0315] [Quantitative Department]

[0316] The quantization unit 108 quantizes the transform coefficients output from the transform unit 106. Specifically, the quantization unit 108 scans the multiple transform coefficients of the current block in a predetermined scanning order and quantizes the transform coefficients based on the quantization parameter (QP) corresponding to the scanned transform coefficients. The quantization unit 108 then outputs the multiple quantized transform coefficients of the current block (hereinafter referred to as quantized coefficients) to the entropy coding unit 110 and the inverse quantization unit 112.

[0317] The predetermined scanning order is the order used for quantization / inverse quantization of transform coefficients. For example, the predetermined scanning order is defined by ascending order (from low frequency to high frequency) or descending order (from high frequency to low frequency) of frequency.

[0318] The quantization parameter (QP) is a parameter that defines the quantization step size (quantization width). For example, if the value of the quantization parameter increases, the quantization step size also increases. In other words, if the value of the quantization parameter increases, the error in the quantization coefficient (quantization error) increases.

[0319] Quantization also sometimes uses a quantization matrix. For example, multiple quantization matrices may be used to correspond to frequency transform sizes such as 4×4 and 8×8, prediction modes such as intra-frame prediction and inter-frame prediction, and pixel components such as luma and chroma. Furthermore, quantization involves digitizing values ​​sampled at predetermined intervals and assigning them to predetermined levels. In this technical field, rounding, rounding, or scaling are sometimes used.

[0320] Methods for using a quantization matrix include using a quantization matrix directly set on the encoding device 100 side and using a default quantization matrix (default matrix). By directly setting the quantization matrix on the encoding device 100 side, a quantization matrix suitable for the characteristics of the image can be set. However, this method has the disadvantage of increasing the amount of code due to encoding the quantization matrix. Alternatively, rather than directly using the default quantization matrix or the encoded quantization matrix, a quantization matrix used for quantization of the current block can be generated based on the default quantization matrix or the encoded quantization matrix.

[0321] On the other hand, there is also a method of performing quantization so that the coefficients of high-frequency components and low-frequency components are the same without using a quantization matrix. In addition, this method is equivalent to using a quantization matrix (flat matrix) in which all coefficients have the same value.

[0322] The quantization matrix can be encoded at, for example, sequence level, picture level, slice level, brick level, or CTU level.

[0323] When using a quantization matrix, the quantization unit 108, for example, scales the quantization width calculated based on the quantization parameter for each transform coefficient using the value of the quantization matrix. Quantization processing performed without using a quantization matrix may also be processing that quantizes the transform coefficients based on the quantization width calculated based on the quantization parameter. Furthermore, in quantization processing without using a quantization matrix, the quantization width may be multiplied by a predetermined value common to all transform coefficients in a block.

[0324] Figure 18 This is a block diagram showing an example of the functional configuration of the quantization unit 108 .

[0325] The quantization unit 108 includes, for example, a differential quantization parameter generation unit 108 a , a predicted quantization parameter generation unit 108 b , a quantization parameter generation unit 108 c , a quantization parameter storage unit 108 d , and a quantization processing unit 108 e .

[0326] Figure 19 This is a flowchart showing an example of quantization performed by the quantization unit 108 .

[0327] As an example, the quantization unit 108 may be based on Figure 19 The flowchart shown in the figure performs quantization on a per-CU basis. Specifically, the quantization parameter generator 108c determines whether to perform quantization (step Sv_1). If the determination is to perform quantization (yes in step Sv_1), the quantization parameter generator 108c generates a quantization parameter for the current block (step Sv_2) and stores the quantization parameter in the quantization parameter storage unit 108d (step Sv_3).

[0328] Next, the quantization processing unit 108e quantizes the transform coefficients of the current block using the quantization parameter generated in step Sv_2 (step Sv_4). The predicted quantization parameter generation unit 108b then obtains the quantization parameter for a processing unit different from the current block from the quantization parameter storage unit 108d (step Sv_5). Based on the obtained quantization parameter, the predicted quantization parameter generation unit 108b generates a predicted quantization parameter for the current block (step Sv_6). The differential quantization parameter generation unit 108a calculates the difference between the quantization parameter for the current block generated by the quantization parameter generation unit 108c and the predicted quantization parameter for the current block generated by the predicted quantization parameter generation unit 108b (step Sv_7). This difference calculation generates a differential quantization parameter. The differential quantization parameter generation unit 108a outputs the differential quantization parameter to the entropy coding unit 110, which encodes the differential quantization parameter (step Sv_8).

[0329] In addition, the differential quantization parameter can also be encoded at the sequence level, picture level, slice level, brick level, or CTU level. In addition, the initial value of the quantization parameter can be encoded at the sequence level, picture level, slice level, brick level, or CTU level. In this case, the quantization parameter can be generated using the initial value of the quantization parameter and the differential quantization parameter.

[0330] Furthermore, the quantization unit 108 may include a plurality of quantizers, and may apply dependent quantization for quantizing transform coefficients using a quantization method selected from a plurality of quantization methods.

[0331] [Entropy coding unit]

[0332] Figure 20 110 is a block diagram showing an example of the functional configuration of the entropy coding unit 110 .

[0333] The entropy coding unit 110 performs entropy coding on the quantization coefficients input from the quantization unit 108 and the prediction parameters input from the prediction parameter generation unit 130, thereby generating a stream. This entropy coding employs, for example, CABAC (Context-based Adaptive Binary Arithmetic Coding). Specifically, the entropy coding unit 110 includes, for example, a binarization unit 110a, a context control unit 110b, and a binary arithmetic coding unit 110c. The binarization unit 110a performs binarization, converting multi-value signals such as quantization coefficients and prediction parameters into binary signals. Examples of binarization methods include Truncated Rice Binarization, Exponential Golomb codes, and Fixed Length Binarization. The context control unit 110b derives a context value, that is, the probability of occurrence of a binary signal, based on the characteristics of a syntactic element or the surrounding conditions. Examples of methods for deriving this context value include bypass, syntactic element reference, upper / left neighboring block reference, and hierarchy information reference. The binary arithmetic coding unit 110 c performs arithmetic coding on the binarized signal using the derived context value.

[0334] Figure 21 3 is a diagram showing the flow of CABAC in the entropy coding unit 110 .

[0335] First, the CABAC in the entropy coding unit 110 is initialized. During this initialization, the binary arithmetic coding unit 110c is initialized and an initial context value is set. The binarization unit 110a and the binary arithmetic coding unit 110c then sequentially perform binarization and arithmetic coding on, for example, multiple quantized coefficients of a CTU. At this time, the context control unit 110b updates the context value each time arithmetic coding is performed. The context control unit 110b then backs off the context value as post-processing. This backed-off context value is used, for example, as the initial context value for the next CTU.

[0336] [Inverse quantization unit]

[0337] The inverse quantization unit 112 inversely quantizes the quantized coefficients input from the quantization unit 108. Specifically, the inverse quantization unit 112 inversely quantizes the quantized coefficients of the current block in a predetermined scanning order. Furthermore, the inverse quantization unit 112 outputs the inversely quantized transform coefficients of the current block to the inverse transform unit 114.

[0338] [Inverse transformation unit]

[0339] The inverse transform unit 114 restores the prediction residual by performing inverse transform on the transform coefficient input from the inverse quantization unit 112. Specifically, the inverse transform unit 114 restores the prediction residual of the current block by performing inverse transform on the transform coefficient corresponding to the transform performed by the transform unit 106. Further, the inverse transform unit 114 outputs the restored prediction residual to the addition unit 116.

[0340] In addition, the restored prediction residual generally does not coincide with the prediction error calculated by the subtraction unit 104 because information is lost by quantization. That is, the restored prediction residual generally includes quantization error.

[0341] [Addition unit]

[0342] The addition unit 116 reconstructs the current block by adding the prediction residual input from the inverse transform unit 114 to the prediction image input from the prediction control unit 128. As a result, the reconstructed image is generated. Further, the addition unit 116 outputs the reconstructed image to the block memory 118 and the loop filter unit 120.

[0343] [Block memory]

[0344] The block memory 118 is, for example, a storage unit for storing the block referred to in the intra prediction and the block within the current picture. Specifically, the block memory 118 stores the reconstructed image output from the addition unit 116.

[0345] [Frame memory]

[0346] The frame memory 122 is, for example, a storage unit for storing the reference picture used in the inter prediction, and is also referred to as a frame buffer. Specifically, the frame memory 122 stores the reconstructed image filtered by the loop filter unit 120.

[0347] [Loop filter unit]

[0348] The loop filter unit 120 performs loop filtering on the reconstructed image output from the addition unit 116, and outputs the reconstructed image filtered to the frame memory 122. The loop filtering refers to filtering used within the encoding loop (in-loop filtering), and includes, for example, adaptive loop filtering (ALF), deblocking filtering (DF or DBF), and sample adaptive offset (SAO).

[0349] Figure 22 is a block diagram showing an example of the functional structure of the loop filter unit 120.

[0350] For example Figure 22As shown, the loop filter unit 120 includes a deblocking filter processing unit 120a, an SAO processing unit 120b, and an ALF processing unit 120c. The deblocking filter processing unit 120a applies the above-mentioned deblocking filter processing to the reconstructed image. The SAO processing unit 120b applies the above-mentioned SAO processing to the reconstructed image after the deblocking filter processing. In addition, the ALF processing unit 120c applies the above-mentioned ALF processing to the reconstructed image after the SAO processing. Details about ALF and deblocking filtering will be described later. SAO processing is a process that improves image quality by reducing ringing (a phenomenon in which pixel values ​​around edges are deformed in a fluctuating manner) and correcting deviations in pixel values. In the SAO processing, for example, there are edge offset processing and band offset processing. In addition, the loop filter unit 120 may not have Figure 22 The disclosed processing units may also include only a part of the processing units. In addition, the loop filter unit 120 may also be configured in accordance with Figure 22 A structure in which the above-mentioned respective processes are performed in an order different from the processing order disclosed in .

[0351] [Loop Filter Section > Adaptive Loop Filter]

[0352] In ALF, a least squares error filter is used to remove coding distortion. For example, for each 2×2 pixel sub-block in the current block, one filter is selected from multiple filters based on the direction and activity of the local gradient.

[0353] Specifically, a sub-block (e.g., a 2×2 pixel sub-block) is first classified into multiple classes (e.g., 15 or 25 classes). The sub-blocks are classified, for example, based on the direction and activity of the gradient. In a specific example, a classification value C (e.g., C = 5D + A) is calculated using the gradient direction value D (e.g., 0 to 2 or 0 to 4) and the gradient activity value A (e.g., 0 to 4). Based on the classification value C, the sub-blocks are then classified into multiple classes.

[0354] The gradient direction value D is derived, for example, by comparing gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). Furthermore, the gradient activity value A is derived, for example, by summing the gradients in multiple directions and quantizing the sum.

[0355] Based on the result of such classification, a filter to be used for the sub-block is determined from among a plurality of filters.

[0356] As the shape of the filter used in the ALF, for example, a circularly symmetric shape is used. Figures 23A to 23C FIG. 1 is a diagram showing a plurality of examples of filter shapes used in ALF. Figure 23A represents a 5×5 diamond shape filter, Figure 23B represents a 7×7 diamond shape filter, Figure 23C Represents a 9×9 diamond-shaped filter. Information representing the filter shape is typically signaled at the picture level. However, signaling of the filter shape information need not be limited to the picture level and can also be performed at other levels (e.g., sequence level, slice level, tile level, CTU level, or CU level).

[0357] The on / off of ALF can also be determined at the picture level or the CU level. For example, regarding luminance, whether to use ALF can be determined at the CU level, and regarding chrominance, whether to use ALF can be determined at the picture level. The information indicating whether ALF is on / off is usually signaled at the picture level or the CU level. In addition, the signaling of the information indicating whether ALF is on / off does not need to be limited to the picture level or the CU level, and can also be at other levels (for example, the sequence level, the slice level, the brick level, or the CTU level).

[0358] As described above, one filter is selected from a plurality of filters to apply ALF processing to a sub-block. For each of these multiple filters (e.g., up to 15 or 25 filters), a coefficient set consisting of the multiple coefficients used in the filter is typically signaled at the picture level. However, the signaling of the coefficient set is not limited to the picture level and may be performed at other levels (e.g., sequence level, slice level, tile level, CTU level, CU level, or sub-block level).

[0359] [Loop Filter > Cross Component Adaptive Loop Filter]

[0360] Figure 23D This is a diagram showing an example in which a Y sample (first component) is used for CCALF of Cb and CCALF of Cr (a plurality of components different from the first component). Figure 23E is a diagram showing a diamond-shaped filter.

[0361] An example of CC-ALF is to use a linear diamond filter ( Figure 23D 、 Figure 23E) applied to the luma channel of each chroma component. For example, the filter coefficients are sent in APS, scaled by a factor of 2A10, and rounded for fixed-point representation. The application of the filter is controlled to be variable block size and signaled by a context-coded flag received per block of samples. The block size and CC-ALF enable flag are received at slice level per chroma component. The syntax and semantics of CC-ALF are provided in Appendix. In this document, block sizes of 16x16, 32x32, 64x64, 128x128 are supported (in chroma samples).

[0362] [Joint Chroma Cross Component Adaptive Loop Filter (JC-CCALF)]

[0363] Figure 23F is an example of a picture representing JC-CCALF. Figure 23G is an example of a picture representing JC-CCALF weight_index candidates.

[0364] One example of JC-CCALF uses only one CCALF filter, generates one CCALF filter output as the chroma adjustment signal for only one color component, and applies appropriately weighted versions of the same chroma adjustment signal to the other color components. In this way, the complexity of the existing CCALF is roughly halved.

[0365] The weight value is coded as a sign flag and a weight index. The weight index, denoted as weight_index, is coded as 3 bits, specifying the size of the JC-CCALF weight JcCcWeight. It cannot be equal to 0. The size of JcCcWeight is determined as follows.

[0366] • For weight_index less than or equal to 4, JcCcWeight is equal to weight_index » 2.

[0367] • For other cases, JcCcWeight is equal to 4 / (weight_index - 4).

[0368] The block-level on / off control of the ALF filtering for Cb and Cr is separate. This is the same as for CCALF, and two separate sets of block-level on / off control flags are coded. Here, unlike for CCALF, the on / off control block sizes for Cb and Cr are the same, so only one block size variable is coded.

[0369] [Loop Filter Section > Deblocking Filter]

[0370] In the deblocking filtering process, the loop filter unit 120 performs filtering on block boundaries of the reconstructed image to reduce distortion generated at the block boundaries.

[0371] Figure 24 This is a block diagram showing an example of a detailed configuration of the deblocking filter processing unit 120 a .

[0372] The deblocking filter processing unit 120 a includes, for example, a boundary determination unit 1201 , a filter determination unit 1203 , a filter processing unit 1205 , a processing determination unit 1208 , a filter characteristic determination unit 1207 , and switches 1202 , 1204 , and 1206 .

[0373] The boundary determination unit 1201 determines whether there is a pixel (ie, target pixel) to be subjected to deblocking filtering near a block boundary and outputs its determination result to the switch 1202 and the processing determination unit 1208 .

[0374] When the boundary determination unit 1201 determines that the target pixel exists near a block boundary, the switch 1202 outputs the image before filtering to the switch 1204. Conversely, when the boundary determination unit 1201 determines that the target pixel does not exist near a block boundary, the switch 1202 outputs the image before filtering to the switch 1206. The image before filtering is composed of the target pixel and at least one surrounding pixel located around the target pixel.

[0375] The filter determination unit 1203 determines whether to perform deblocking filtering on the target pixel based on the pixel value of at least one surrounding pixel located around the target pixel, and then outputs the determination result to the switch 1204 and the processing determination unit 1208 .

[0376] When the filter determination unit 1203 determines that the deblocking filtering process is to be performed on the target pixel, the switch 1204 outputs the pre-filtering image obtained via the switch 1202 to the filter processing unit 1205. Conversely, when the filter determination unit 1203 determines that the deblocking filtering process is not to be performed on the target pixel, the switch 1204 outputs the pre-filtering image obtained via the switch 1202 to the switch 1206.

[0377] When the pre-filtered image is obtained via switches 1202 and 1204 , the filter processing unit 1205 performs deblocking filtering on the target pixel using the filter characteristics determined by the filter characteristic determination unit 1207 . The filter processing unit 1205 then outputs the filtered pixel to the switch 1206 .

[0378] According to the control of the processing determination unit 1208 , the switch 1206 selectively outputs pixels that have not been processed by the deblocking filter and pixels that have been processed by the deblocking filter by the filter processing unit 1205 .

[0379] The processing determination unit 1208 controls the switch 1206 based on the respective determination results of the boundary determination unit 1201 and the filtering determination unit 1203. That is, when the boundary determination unit 1201 determines that the object pixel exists near the block boundary and the filtering determination unit 1203 determines that the deblocking filtering process is to be performed on the object pixel, the processing determination unit 1208 outputs the pixel processed by the deblocking filtering from the switch 1206. In addition, in cases other than the above-mentioned cases, the processing determination unit 1208 outputs the pixel that has not been processed by the deblocking / filtering process from the switch 1206. By repeatedly outputting such pixels, the image processed by the filtering process is output from the switch 1206. In addition, Figure 24 The illustrated configuration is an example of the configuration of the deblocking filter processing unit 120 a , and the deblocking filter processing unit 120 a may have another configuration.

[0380] Figure 25 This is a diagram showing an example of a deblocking filter having a filter characteristic that is symmetric with respect to a block boundary.

[0381] In the deblocking filter process, for example, using pixel values ​​and quantization parameters, one of two deblocking filters with different characteristics, that is, a strong filter and a weak filter, is selected. Figure 25 As shown, when pixels p0 to p2 and pixels q0 to q2 exist across a block boundary, the pixel values ​​of pixels q0 to q2 are changed to pixel values ​​q'0 to q'2 by performing the calculation shown in the following equation.

[0382] q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8

[0383] q'1=(p0+q0+q1+q2+2) / 4

[0384] q'2=(p0+q0+q1+3×q2+2×q3+4) / 8

[0385] In the above equations, p0 through p2 and q0 through q2 are the pixel values ​​of pixels p0 through p2 and q0 through q2, respectively. Furthermore, q3 is the pixel value of pixel q3, which is adjacent to pixel q2 on the side opposite the block boundary. On the right side of each of the above equations, the coefficients multiplied by the pixel values ​​of each pixel used in the deblocking filter process are the filter coefficients.

[0386] Furthermore, during the deblocking filter process, clipping can be performed so that the calculated pixel value remains unchanged even if it exceeds a threshold. In this clipping process, the calculated pixel value based on the above equation is clipped to "pre-calculated pixel value ± 2 × threshold value" using a threshold determined by the quantization parameter. This prevents excessive smoothing.

[0387] Figure 26 This is a diagram for explaining an example of a block boundary on which a deblocking filtering process is performed. Figure 27 This is a diagram showing an example of BS value.

[0388] The block boundary for deblocking filtering is, for example, Figure 26 The deblocking filter is performed in units of, for example, 4 rows or 4 columns. First, for Figure 26 The block P and block Q shown are as follows: Figure 27 That determines the Bs (Boundary Strength) value.

[0389] according to Figure 27 The Bs value can determine whether to perform deblocking filtering with different strengths even at block boundaries belonging to the same image. When the Bs value is 2, deblocking filtering is performed on the color difference signal. When the Bs value is 1 or more and the specified conditions are met, deblocking filtering is performed on the luminance signal. In addition, the determination conditions of the Bs value are not limited to Figure 27 The conditions shown can also be determined based on other parameters.

[0390] [Prediction Unit (Intra-frame Prediction Unit / Inter-frame Prediction Unit / Prediction Control Unit)]

[0391] Figure 28 This is a flowchart showing an example of processing performed by the prediction unit of the encoding device 100. Furthermore, as an example, the prediction unit is composed of all or part of the components of the intra-frame prediction unit 124, the inter-frame prediction unit 126, and the prediction control unit 128. The prediction processing unit includes, for example, the intra-frame prediction unit 124 and the inter-frame prediction unit 126.

[0392] The prediction unit generates a predicted image for the current block (step Sb_1). Predicted images include, for example, intra-frame predicted images (intra-frame prediction signals) or inter-frame predicted images (inter-frame prediction signals). Specifically, the prediction unit generates a predicted image for the current block using a reconstructed image obtained by generating predicted images for other blocks, generating prediction residuals, generating quantized coefficients, restoring the prediction residuals, and adding the predicted images.

[0393] The reconstructed image can be, for example, an image of a reference picture or an image of a picture that contains the coded block (i.e., the other block described above) within a current picture, i.e., the picture that contains the current block. The coded block within the current picture is, for example, a neighboring block of the current block.

[0394] Figure 29 is a flowchart showing another example of the processing performed by the prediction section of the encoding apparatus 100.

[0395] The prediction section generates a prediction image by the first method (step Sc la), generates a prediction image by the second method (step Sc lb), and generates a prediction image by the third method (step Sc lc). The first method, the second method, and the third method are mutually different methods for generating a prediction image, and can be, for example, an inter prediction method, an intra prediction method, and another prediction method, respectively. In such prediction methods, the reconstructed image described above can also be used.

[0396] Next, the prediction section evaluates the prediction images generated in steps Sc la, Sc lb, and Sc lc, respectively (step Sc 2). For example, the prediction section evaluates the prediction images by calculating the cost C for the prediction images generated in each of steps Sc la, Sc lb, and Sc lc, and comparing the costs C of the prediction images. In addition, the cost C is calculated by the equation of the R-D optimization model, for example, C = D + λ x R. In this equation, D is the coding distortion of the prediction image, and is represented by, for example, the sum of the absolute values of the differences between the pixel values of the current block and the pixel values of the prediction image. Furthermore, R is the bit rate of the stream. In addition, λ is an undetermined multiplier of, for example, Lagrange.

[0397] Next, the prediction section selects one of the prediction images generated in steps Sc la, Sc lb, and Sc lc (step Sc 3). That is, the prediction section selects the method or mode for obtaining the final prediction image. For example, the prediction section selects the prediction image of the smallest cost C on the basis of the costs C calculated for the prediction images. Alternatively, the evaluation of step Sc 2 and the selection of the prediction image in step Sc 3 can also be performed on the basis of the parameters used in the encoding processing. The encoding apparatus 100 can signal information for determining the selected prediction image, method, or mode as a stream. The information can be, for example, a flag or the like. As a result, the decoding apparatus 200 can generate a prediction image in the method or mode selected in the encoding apparatus 100 on the basis of the information. Furthermore, in the example shown in FIG. 8, the prediction section selects one of the prediction images after generating the prediction images by the respective methods. However, the prediction section can select the method or mode on the basis of the parameters for the encoding processing described above before generating the prediction images, and can generate the prediction images in accordance with the method or mode. Figure 29 ​

[0398] For example, the first and second modes are intra prediction and inter prediction, respectively, and the prediction section can select a final prediction image for the current block from among prediction images generated in accordance with these prediction modes.

[0399] Figure 30 is a flowchart of another example of processing performed by the prediction section of the encoding apparatus 100.

[0400] First, the prediction section generates a prediction image by intra prediction (step Sd la), and generates a prediction image by inter prediction (step Sd lb). Further, the prediction image generated by intra prediction is also referred to as an intra prediction image, and the prediction image generated by inter prediction is also referred to as an inter prediction image.

[0401] Next, the prediction section evaluates each of the intra prediction image and the inter prediction image (step Sd 2). The above-described cost C can also be used in this evaluation. Then, the prediction section can select, as the final prediction image of the current block, the prediction image for which the minimum cost C is calculated from among the intra prediction image and the inter prediction image (step Sd 3). That is, the prediction mode or the mode used to generate the prediction image of the current block is selected.

[0402] [Intra Prediction Section]

[0403] The intra prediction section 124 performs intra prediction (also referred to as in-picture prediction) of the current block with reference to the blocks within the current picture saved in the block memory 118, thereby generating a prediction image (i.e., an intra prediction image) of the current block. Specifically, the intra prediction section 124 generates an intra prediction image by performing intra prediction with reference to the pixel values (e.g., luminance values, color difference values) of the blocks adjacent to the current block, and outputs the intra prediction image to the prediction control section 128.

[0404] For example, the intra prediction section 124 performs intra prediction using one of a plurality of intra prediction modes specified in advance. The plurality of intra prediction modes generally includes one or more non-directional prediction modes and a plurality of directional prediction modes.

[0405] The one or more non-directional prediction modes include, for example, a Planar prediction mode and a DC prediction mode specified in the H.265 / HEVC specification.

[0406] The plurality of directional prediction modes include, for example, 33 directional prediction modes specified in the H.265 / HEVC specification. Alternatively, the plurality of directional prediction modes can include 32 directional prediction modes in addition to the 33 directional prediction modes (a total of 65 directional prediction modes). Figure 31This figure shows all 67 intra prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in intra prediction. The solid arrows represent the 33 directions specified by the H.265 / HEVC specification, and the dotted arrows represent the additional 32 directions (2 non-directional prediction modes in Figure 31 (not shown in the figure).

[0407] In various implementations, intra-frame prediction of chrominance blocks can also refer to luma blocks. That is, the chrominance component of the current block can be predicted based on the luma component of the current block. This type of intra-frame prediction is sometimes called CCLM (cross-component linear model) prediction. Such an intra-frame prediction mode for chrominance blocks that refers to luma blocks (e.g., CCLM mode) can also be added as one of the intra-frame prediction modes for chrominance blocks.

[0408] The intra-frame prediction unit 124 may also modify the intra-predicted pixel values ​​based on the gradients of reference pixels in the horizontal and vertical directions. Intra-frame prediction with such modification is sometimes called PDPC (position-dependent intraprediction combination). Information indicating whether PDPC is used (e.g., a PDPC flag) is typically signaled at the CU level. However, this information is not necessarily signaled at the CU level and may be signaled at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0409] Figure 32 This is a flowchart showing an example of processing performed by the intra prediction unit 124 .

[0410] The intra-frame prediction unit 124 selects one intra-frame prediction mode from a plurality of intra-frame prediction modes (step Sw_1). Then, the intra-frame prediction unit 124 generates a predicted image according to the selected intra-frame prediction mode (step Sw_2). Next, the intra-frame prediction unit 124 determines the MPM (Most Probable Modes) (step Sw_3). The MPM is composed of, for example, 6 intra-frame prediction modes. Two of the 6 intra-frame prediction modes can be a Planar prediction mode and a DC prediction mode, and the remaining 4 modes can be directional prediction modes. Then, the intra-frame prediction unit 124 determines whether the intra-frame prediction mode selected in step Sw_1 is included in the MPM (step Sw_4).

[0411] When it is determined that the selected intra prediction mode is included in the MPM (Yes in step Sw_4), the intra prediction section 124 sets the MPM flag to 1 (step Sw_5), generates information indicating the selected intra prediction mode in the MPM (step Sw_6). The MPM flag set to 1 and the information indicating the intra prediction mode are encoded as prediction parameters by the entropy encoding section 110, respectively.

[0412] On the other hand, when it is determined that the selected intra prediction mode is not included in the MPM (No in step Sw_4), the intra prediction section 124 sets the MPM flag to 0 (step Sw_7). Alternatively, the intra prediction section 124 does not set the MPM flag. Then, the intra prediction section 124 generates information indicating the selected intra prediction mode among one or more intra prediction modes not included in the MPM (step Sw_8). The MPM flag set to 0 and the information indicating the intra prediction mode are encoded as prediction parameters by the entropy encoding section 110, respectively. The information indicating the intra prediction mode indicates, for example, any one of 0 to 60.

[0413] [INTER PREDICTION SECTION]

[0414] The inter prediction section 126 performs inter prediction (also referred to as inter picture prediction) of a current block with reference to a reference picture different from the current picture stored in the frame memory 122, thereby generating a predicted image (inter prediction image). Inter prediction is performed in units of the current block or a current sub-block within the current block. A sub-block is included in a block and is a unit smaller than the block. The size of the sub-block can be 4 x 4 pixels, can be 8 x 8 pixels, or can be a size other than these. The size of the sub-block can also be switched in units of a slice, a tile, or a picture.

[0415] For example, the inter prediction section 126 performs motion search within the reference picture for the current block or the current sub-block, and searches for a reference block or sub-block most consistent with the current block or the current sub-block. Then, the inter prediction section 126 acquires motion information (e.g., a motion vector) that compensates for motion or change from the reference block or sub-block to the current block or sub-block. The inter prediction section 126 performs motion compensation (or motion prediction) based on the motion information, thereby generating an inter prediction image of the current block or sub-block. Then, the inter prediction section 126 outputs the generated inter prediction image to the prediction control section 128.

[0416] The motion information used in the motion compensation is signaled as the inter prediction image in various forms. For example, a motion vector can be signaled. As another example, a difference between a motion vector and a prediction motion vector predictor can be signaled.

[0417] [See image list]

[0418] Figure 33 is a diagram showing an example of each reference picture, Figure 34 1 is a conceptual diagram showing an example of a reference picture list. The reference picture list is a list showing one or more reference pictures stored in the frame memory 122. Figure 33 In the figure, rectangles represent pictures, arrows represent reference relationships between pictures, the horizontal axis represents time, I, P and B in the rectangles represent intra-frame prediction pictures, single-prediction pictures and double-prediction pictures respectively, and the numbers in the rectangles represent the decoding order. Figure 33 As shown in , the decoding order of each picture is I0, P1, B2, B3, B4, and the display order of each picture is I0, B3, B2, B4, P1. Figure 34 As shown in the figure, the reference picture list is a list of candidates for reference pictures. For example, one picture (or slice) can have more than one reference picture list. For example, if the current picture is a single-predicted picture, one reference picture list is used. If the current picture is a double-predicted picture, two reference picture lists are used. Figure 33 and Figure 34 In the example, picture B3, which is the current picture currPic, has two reference picture lists, namely the L0 list and the L1 list. When the current picture currPic is picture B3, the candidates for the reference pictures of the current picture currPic are I0, P1 and B2, and each reference picture list (i.e., the L0 list and the L1 list) represents these pictures. The inter-frame prediction unit 126 or the prediction control unit 128 specifies which picture in each reference picture list to actually refer to by referring to the picture index refidxLx. Figure 34 , reference pictures P1 and B2 are specified by reference picture indices refIdxL0 and refIdxL1.

[0419] Such a reference picture list can be generated in sequence units, picture units, slice units, tile units, CTU units, or CU units. Furthermore, a reference picture index indicating a reference picture used for inter prediction among the reference pictures shown in the reference picture list can be encoded at the sequence level, picture level, slice level, tile level, CTU level, or CU level. Furthermore, a common reference picture list can be used in multiple inter prediction modes.

[0420] [Basic process of inter-frame prediction]

[0421] Figure 35 This is a flowchart showing the basic process of inter-frame prediction.

[0422] The inter prediction unit 126 first generates a predicted image (steps Se_1 to Se_3). Next, the subtraction unit 104 generates a difference between the current block and the predicted image as a prediction residual (step Se_4).

[0423] Here, in generating a predicted image, the inter-frame prediction unit 126 generates the predicted image by, for example, determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and performing motion compensation (step Se_3). Furthermore, in determining the MV, the inter-frame prediction unit 126 determines the MV by, for example, selecting a candidate motion vector (candidate MV) (step Se_1) and deriving the MV (step Se_2). The candidate MV is selected by, for example, the inter-frame prediction unit 126 generating a candidate MV list and selecting at least one candidate MV from the candidate MV list. Previously derived MVs may be added to the candidate MV list as candidate MVs. Furthermore, in deriving the MV, the inter-frame prediction unit 126 may further select at least one candidate MV from the at least one candidate MV and determine the selected at least one candidate MV as the MV of the current block. Alternatively, the inter-frame prediction unit 126 may determine the MV of the current block by searching the region of the reference picture indicated by each of the at least one selected candidate MVs. The operation of searching the region of the reference picture may also be referred to as motion estimation.

[0424] Furthermore, in the above example, steps Se_1 to Se_3 are performed by the inter-frame prediction unit 126 . However, for example, the processing of step Se_1 or step Se_2 may be performed by other components included in the encoding device 100 .

[0425] In addition, a candidate MV list may be prepared for each process in each inter-frame prediction mode, or a common candidate MV list may be used in multiple inter-frame prediction modes. Figure 9 The processing of steps Sa_3 and Sa_4 shown in FIG. In addition, the processing of step Se_3 is equivalent to Figure 30 The processing of step Sd_1b.

[0426] [MV export process]

[0427] Figure 36 This is a flowchart showing an example of MV derivation.

[0428] The inter-frame prediction unit 126 can derive the MV of the current block in a mode of encoding motion information (e.g., MV). In this case, for example, the motion information can be encoded as a prediction parameter and signaled. That is, the encoded motion information is included in the stream.

[0429] Alternatively, the inter prediction section 126 can derive the MV in a mode in which the motion information is not coded. In this case, the motion information is not included in the stream.

[0430] Here, the mode of the MV derivation has a normal inter mode, a normal merge mode, an FRUC mode, and an affine mode, and the like, which will be described later. Among these modes, the mode in which the motion information is coded has the normal inter mode, the normal merge mode, and the affine mode (specifically, an affine inter mode and an affine merge mode), and the like. Further, the motion information can include not only the MV but also prediction MV selection information, which will be described later. Further, the mode in which the motion information is not coded has the FRUC mode, and the like. The inter prediction section 126 selects the mode for deriving the MV of the current block from among these multiple modes, and derives the MV of the current block using the selected mode.

[0431] Figure 37 is a flowchart showing another example of the MV derivation.

[0432] The inter prediction section 126 can derive the MV of the current block in a mode in which the differential MV is coded. In this case, for example, the differential MV is coded as a prediction parameter, and is signaled. That is, the coded differential MV is included in the stream. The differential MV is the difference between the MV of the current block and a prediction MV thereof. Further, the prediction MV is a predicted motion vector.

[0433] Alternatively, the inter prediction section 126 can derive the MV in a mode in which the differential MV is not coded. In this case, the coded differential MV is not included in the stream.

[0434] Here, as described above, the mode of the MV derivation has the normal inter mode, the normal merge mode, the FRUC mode, and the affine mode, and the like, which will be described later. Among these modes, the mode in which the differential MV is coded has the normal inter mode, and the affine mode (specifically, the affine inter mode), and the like. Further, the mode in which the differential MV is not coded has the FRUC mode, the normal merge mode, and the affine mode (specifically, the affine merge mode), and the like. The inter prediction section 126 selects the mode for deriving the MV of the current block from among these multiple modes, and derives the MV of the current block using the selected mode.

[0435] [MV Derivation Mode]

[0436] Figure 38A and Figure 38B is a diagram showing an example of the classification of each mode of the MV derivation. For example, as Figure 38AAs shown in FIG, the MV derivation mode is classified into three major modes according to whether the motion information is encoded and whether the differential MV is encoded. The three modes are inter-frame mode, merge mode, and FRUC (frame rate up-conversion) mode. The inter-frame mode is a mode for performing motion search and encoding the motion information and differential MV. For example, Figure 38B As shown, inter-frame mode includes affine inter-frame mode and normal inter-frame mode. Merge mode is a mode that does not perform motion search, but selects MV from the surrounding coded blocks and uses the MV to derive the MV of the current block. This merge mode is basically a mode that encodes motion information without encoding the differential MV. For example, Figure 38B As shown, the merge modes include normal merge mode (sometimes also called normal merge mode or regular merge mode), MMVD (Merge with Motion Vector Difference) mode, CIIP (Combined inter merge / intraprediction) mode, triangle mode, ATMVP mode, and affine merge mode. Among the modes included in the merge mode, MMVD mode exceptionally encodes the differential MV. Furthermore, the aforementioned affine merge mode and affine inter mode are included in the affine mode. The affine mode assumes an affine transformation and derives the MV of each of the multiple sub-blocks constituting the current block as the MV of the current block. The FRUC mode derives the MV of the current block by searching between already coded regions, and does not encode either motion information or the differential MV. Details of these modes will be described later.

[0437] in addition, Figure 38A and Figure 38B The classification of each mode shown is an example and is not limited to this. For example, when encoding the differential MV in the CIIP mode, the CIIP mode is classified as the inter mode.

[0438] [MV Export > Normal Interframe Mode]

[0439] Normal inter mode is an inter prediction mode that derives the MV of the current block by finding a block similar to the image of the current block from an area of ​​a reference picture indicated by a candidate MV. In addition, in this normal inter mode, a differential MV is encoded.

[0440] Figure 39 This is a flowchart showing an example of inter prediction based on the normal inter mode.

[0441] First, the inter prediction section 126 acquires a plurality of candidate MVs for the current block based on information such as MVs of a plurality of coded blocks located around the current block in time or space (step Sg_1). That is, the inter prediction section 126 creates a list of candidate MVs.

[0442] Next, the inter prediction section 126 extracts N (N is an integer of 2 or more) candidate MVs from the plurality of candidate MVs acquired in step Sg_1 as prediction MV candidates in a predetermined priority order (step Sg_2). Note that the priority order is predetermined for each of the N candidate MVs.

[0443] Next, the inter prediction section 126 selects one of the N prediction MV candidates as a prediction MV for the current block (step Sg_3). At this time, the inter prediction section 126 encodes prediction MV selection information for identifying the selected prediction MV into a stream. That is, the inter prediction section 126 outputs the prediction MV selection information as a prediction parameter to the entropy coding section 110 via the prediction parameter generation section 130.

[0444] Next, the inter prediction section 126 refers to a coded reference picture to derive the MV of the current block (step Sg_4). At this time, the inter prediction section 126 also encodes a difference value between the derived MV and the prediction MV as a difference MV into a stream. That is, the inter prediction section 126 outputs the difference MV as a prediction parameter to the entropy coding section 110 via the prediction parameter generation section 130. Note that the coded reference picture is a picture composed of a plurality of blocks reconstructed after coding.

[0445] Finally, the inter prediction section 126 generates a prediction image of the current block by performing motion compensation on the current block using the derived MV and the coded reference picture (step Sg_5). The processes of steps Sg_1 to Sg_5 are performed for each block. For example, when the processes of steps Sg_1 to Sg_5 are performed for all blocks included in a slice, respectively, the inter prediction using the normal inter mode ends for the slice. Also, when the processes of steps Sg_1 to Sg_5 are performed for all blocks included in a picture, respectively, the inter prediction using the normal inter mode ends for the picture. Further, it can also be that the processes of steps Sg_1 to Sg_5, when performed for a part of the blocks included in a slice, end the inter prediction using the normal inter mode for the slice. Similarly, it can also be that the processes of steps Sg_1 to Sg_5, when performed for a part of the blocks included in a picture, end the inter prediction using the normal inter mode for the picture.

[0446] The predicted image is the inter prediction signal described above. Information indicating the inter prediction mode (in the above example, the normal inter mode) used to generate the predicted image, contained in the coded signal, is coded as, for example, prediction parameters.

[0447] The candidate MV list can also be used in conjunction with lists used in other modes. Furthermore, processing related to the candidate MV list can be applied to processing related to lists used in other modes. Processing related to the candidate MV list includes, for example, extracting or selecting candidate MVs from the candidate MV list, rearranging candidate MVs, or deleting candidate MVs.

[0448] [MV Export > Normal Merge Mode]

[0449] Normal merge mode is an inter-frame prediction mode that derives the MV of the current block by selecting a candidate MV from a candidate MV list as the MV of the current block. Normal merge mode is a narrowly defined merge mode, sometimes also referred to simply as merge mode. In this embodiment, a distinction is made between normal merge mode and merge mode, with merge mode used in a broad sense.

[0450] Figure 40 This is a flowchart showing an example of inter-frame prediction based on the normal merge mode.

[0451] First, the inter prediction unit 126 obtains a plurality of candidate MVs for the current block based on information on a plurality of coded block MVs located temporally or spatially around the current block (step Sh_1). That is, the inter prediction unit 126 creates a candidate MV list.

[0452] Next, the inter-frame prediction unit 126 selects one candidate MV from the multiple candidate MVs obtained in step Sh_1 to derive the MV for the current block (step Sh_2). At this point, the inter-frame prediction unit 126 encodes MV selection information identifying the selected candidate MV into the stream. Specifically, the inter-frame prediction unit 126 outputs the MV selection information as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0453] Finally, the inter prediction section 126 generates a prediction image of the current block by performing motion compensation on the current block using the derived MV and the coded reference picture (step Sh_3). The processes of steps Sh_l to Sh_3 are performed, for example, for each block. For example, when the processes of steps Sh_l to Sh_3 are performed for all the blocks included in a slice, respectively, the inter prediction using the normal merge mode for the slice ends. Also, when the processes of steps Sh_l to Sh_3 are performed for all the blocks included in a picture, respectively, the inter prediction using the normal merge mode for the picture ends. Further, the processes of steps Sh_l to Sh_3 can also be such that, when performed for a part of the blocks included in a slice, the inter prediction using the normal merge mode for the slice ends. Likewise, the processes of steps Sh_l to Sh_3 can also be such that, when performed for a part of the blocks included in a picture, the inter prediction using the normal merge mode for the picture ends.

[0454] Further, information indicating the inter prediction mode (in the above example, the normal merge mode) used in the generation of the prediction image is coded as, for example, a prediction parameter, in the stream.

[0455] Figure 41 is a diagram for explaining an example of the MV derivation process for the current picture based on the normal merge mode.

[0456] First, the inter prediction section 126 generates a candidate MV list in which candidate MVs are registered. As the candidate MVs, there are: a spatial neighboring candidate MV, which is an MV possessed by a plurality of coded blocks located in the vicinity of the current block in space; a temporal neighboring candidate MV, which is an MV possessed by a block in the vicinity of the position of the current block in the coded reference picture; a combined candidate MV, which is an MV generated by combining the MV values of the spatial neighboring candidate MV and the temporal neighboring candidate MV; and a zero candidate MV, which is an MV having a value of zero, and the like.

[0457] Next, the inter prediction section 126 determines, as the MV of the current block, one candidate MV selected from among the plurality of candidate MVs registered in the candidate MV list.

[0458] Further, in the entropy coding section 110, a signal indicating which candidate MV is selected, i.e., merge_idx, is coded by being described in the stream.

[0459] Further, in the candidate MV list explained in Figure 41 the number of the candidate MVs registered in the candidate MV list is an example, and can be a different number from that in the diagram, or a structure in which a part of the kinds of the candidate MVs in the diagram is not included, or a structure in which a candidate MV other than the kinds of the candidate MVs in the diagram is added.

[0460] The MV of the current block derived by the normal merge mode can also be used to perform DMVR (dynamic motion vector refreshing) to determine the final MV. In addition, in the normal merge mode, the differential MV is not encoded, but in the MMVD mode, the differential MV is encoded. The MMVD mode selects one candidate MV from the candidate MV list in the same way as the normal merge mode, but encodes the differential MV. Figure 38B As shown, such MMVD can also be classified as a merge mode along with the normal merge mode. In addition, the differential MV in MMVD mode may not be the same as the differential MV used in inter-frame mode. For example, the derivation of the differential MV in MMVD mode may require less processing than the derivation of the differential MV in inter-frame mode.

[0461] Alternatively, a CIIP (Combined inter merge / intra prediction) mode may be performed in which a predicted image generated by inter prediction and a predicted image generated by intra prediction are superimposed to generate a predicted image for the current block.

[0462] In addition, the candidate MV list may also be referred to as a candidate list. In addition, merge_idx is MV selection information.

[0463] [MV Export > HMVP Mode]

[0464] Figure 42 This is a diagram for explaining an example of MV derivation processing of the current picture based on the HMVP mode.

[0465] In normal merge mode, one candidate MV is selected from a candidate MV list generated with reference to an already coded block (e.g., a CU), thereby determining the MV of the current block (e.g., a CU). Other candidate MVs can also be registered in the candidate MV list. This mode of registering other candidate MVs is called HMVP mode.

[0466] In the HMVP mode, candidate MVs are managed using a FIFO (First-In First-Out) buffer for HMVP, separate from the candidate MV list in the normal merge mode.

[0467] In the FIFO buffer, motion information such as the MV of the previously processed blocks is sequentially saved from the new FIFO buffer. In the management of this FIFO buffer, each time a block is processed, the MV of the latest block (i.e., the CU processed immediately before) is saved in the FIFO buffer, and the MV of the oldest CU in the FIFO buffer (i.e., the CU processed first) is deleted from the FIFO buffer. Figure 42In the example shown, HMVP1 is the MV of the newest block, and HMVP5 is the MV of the oldest block.

[0468] Then, for example, the inter-frame prediction unit 126 sequentially checks, starting with HMVP1, for each MV managed in the FIFO buffer to see if the MV is different from all candidate MVs registered in the candidate MV list for normal merge mode. Furthermore, if the inter-frame prediction unit 126 determines that the MV managed in the FIFO buffer is different from all candidate MVs, it may add the MV as a candidate MV to the candidate MV list for normal merge mode. In this case, the number of candidate MVs registered in the FIFO buffer may be one or more.

[0469] In this way, by using the HMVP mode, not only can the MVs of blocks that are spatially or temporally adjacent to the current block be added to the candidate, but also the MVs of blocks processed in the past can be added to the candidate. As a result, by expanding the variation of candidate MVs in the normal merge mode, the possibility of improving coding efficiency becomes higher.

[0470] Furthermore, the MV can also be motion information. That is, the candidate MV list and the information stored in the FIFO buffer can include not only the MV value but also information indicating the reference picture, the reference direction, the number of pictures, etc. Furthermore, the block can be, for example, a CU.

[0471] in addition, Figure 42 The candidate MV list and FIFO buffer are one example, and the candidate MV list and FIFO buffer can also be Figure 42 Lists or buffers of different sizes, or in the same Figure 42 A structure for registering candidate MVs in a different order. The processing described here is common to both the encoding device 100 and the decoding device 200 .

[0472] Furthermore, HMVP mode can be applied to modes other than normal merge mode. For example, motion information such as MVs of blocks previously processed in affine mode can be sequentially stored in a new FIFO buffer and used as candidate MVs. The application of HMVP mode to affine mode is also referred to as historical affine mode.

[0473] [MV Export > FRUC Mode]

[0474] The motion information can also not be signaled from the encoding apparatus 100 side, but derived at the decoding apparatus 200 side. For example, the motion information can also be derived by performing a motion search at the decoding apparatus 200 side. In such a case, the motion search is performed at the decoding apparatus 200 side without using the pixel values of the current block. Such a mode of performing a motion search at the decoding apparatus 200 side is an FRUC (frame rate up-conversion) mode or a PMMVD (pattern matched motion vector derivation) mode, or the like.

[0475] Figure 43 An example of FRUC processing is shown in FIG. 12. First, referring to the MVs of each of the coded blocks that are spatially or temporally adjacent to the current block, a list is generated that represents these MVs as candidate MVs (i.e., it is a candidate MV list, and can also be common to the candidate MV list of the normal merge mode) (step Si_1). Next, a best candidate MV is selected from among the plurality of candidate MVs registered in the candidate MV list (step Si_2). For example, the evaluation value of each of the candidate MVs included in the candidate MV list is calculated, and based on this evaluation value, one candidate is selected as the best candidate MV. Also, based on the selected best candidate MV, the MV for the current block is derived (step Si_4). Specifically, for example, the selected best candidate MV is derived as it is as the MV for the current block. Further, for example, the MV for the current block can also be derived by performing pattern matching in the surrounding area of the position within the reference picture corresponding to the selected best candidate MV. That is, a search using pattern matching in the reference picture and the evaluation value can be performed on the surrounding area of the best candidate MV, and in the case where there is an MV with a better value of the evaluation value, the best candidate MV is updated to this MV, and this is taken as the final MV of the current block. The update to the MV with a better evaluation value can not be performed.

[0476] Finally, the inter-frame prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Si_5). The processing of steps Si_1 to Si_5 is performed on each block, for example. For example, when the processing of steps Si_1 to Si_5 is performed on all blocks included in the slice, the inter-frame prediction using the FRUC mode for the slice is completed. In addition, when the processing of steps Si_1 to Si_5 is performed on all blocks included in the picture, the inter-frame prediction using the FRUC mode for the picture is completed. In addition, the processing of steps Si_1 to Si_5 may be performed on a part of the blocks instead of all blocks included in the slice, and the inter-frame prediction using the FRUC mode for the slice is completed. Similarly, when the processing of steps Si_1 to Si_5 is performed on a part of the blocks included in the picture, the inter-frame prediction using the FRUC mode for the picture is completed.

[0477] Even when processing is performed in sub-block units, the same processing as that in the block unit may be performed.

[0478] The evaluation value can be calculated using various methods. For example, a reconstructed image of a region within a reference picture corresponding to the MV can be compared with a reconstructed image of a predetermined region (e.g., as shown below, this region can be a region of another reference picture or a region of an adjacent block in the current picture). The difference in pixel values ​​between the two reconstructed images can then be calculated and used to evaluate the MV. Furthermore, the evaluation value can be calculated using other information besides the difference.

[0479] Next, we'll explain pattern matching in detail. First, a candidate MV from the candidate MV list (also called a merge list) is selected as the starting point for a pattern matching search. Pattern matching can be done using either primary or secondary pattern matching. Primary and secondary pattern matching are also known as bilateral matching and template matching, respectively.

[0480] [MV Export > FRUC > Bidirectional Matching]

[0481] In the first pattern matching, pattern matching is performed between two blocks in two different reference pictures that are along the motion trajectory of the current block. Therefore, in the first pattern matching, the area in the other reference pictures along the motion trajectory of the current block is used as the predetermined area for calculating the candidate MV evaluation value.

[0482] Figure 44: is a diagram for explaining an example of the first pattern matching (bidirectional matching) between two blocks in two reference pictures along the motion trajectory. Figure 44 As shown, in the first pattern matching, two MVs (MV0, MV1) are derived by searching for the best-matching pair among pairs of blocks in two different reference pictures (Ref0, Ref1) along the motion trajectory of the current block (Cur block). Specifically, for the current block, the difference between the reconstructed image at a specified position in the first coded reference picture (Ref0) specified by the candidate MV and the reconstructed image at a specified position in the second coded reference picture (Ref1) specified by a symmetric MV scaled by the display time interval is derived, and the evaluation value is calculated using the obtained difference value. The candidate MV with the best evaluation value among multiple candidate MVs can be selected as the best candidate MV.

[0483] Under the assumption of a continuous motion trajectory, the MVs (MV0, MV1) indicating the two reference blocks are proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, if the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the two reference pictures are equal, mirror-symmetric MVs in both directions are derived in the first pattern matching.

[0484] [MV Export > FRUC > Template Matching]

[0485] In the second pattern matching (template matching), pattern matching is performed between a template in the current picture (blocks adjacent to the current block in the current picture (e.g., blocks above and / or to the left)) and a block in the reference picture. Therefore, in the second pattern matching, blocks adjacent to the current block in the current picture are used as the specified area for calculating the candidate MV evaluation value.

[0486] Figure 45 is a diagram for explaining an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture. Figure 45 As shown, in the second pattern matching, the MV of the current block is derived by searching the reference picture (Ref0) for a block that best matches the block adjacent to the current block (Cur block) in the current picture (Cur Pic). Specifically, for the current block, the difference between the reconstructed image of the coded area adjacent to the left and above, or one of the two, is derived, and the reconstructed image at the same position in the coded reference picture (Ref0) specified by the candidate MV is used to calculate the evaluation value. The candidate MV with the best evaluation value among multiple candidate MVs is selected as the best candidate MV.

[0487] Information indicating whether or not such a representation adopts the FRUC mode (for example, referred to as a FRUC flag) is signaled at the CU level. Furthermore, in the case where the FRUC mode is adopted (for example, in the case where the FRUC flag is true), information indicating the method of pattern matching (first pattern matching or second pattern matching) that can be adopted is signaled at the CU level. In addition, the signaling of these pieces of information need not be limited to the CU level, and can be another level (for example, a sequence level, a picture level, a slice level, a tile level, a CTU level, or a subblock level).

[0488] [MV derivation > affine mode]

[0489] The affine mode is a mode in which an MV is generated using an affine transformation, for example, an MV can be derived in a subblock unit based on MVs of a plurality of neighboring blocks. This mode is sometimes referred to as an affine motion compensation prediction mode.

[0490] Figure 46A is a diagram for explaining an example of derivation of an MV in a subblock unit based on MVs of a plurality of neighboring blocks. In Figure 46A , a current block includes, for example, 16 subblocks each consisting of 4 x 4 pixels. Here, a motion vector v0 of a top-left control point of the current block is derived based on MVs of neighboring blocks, and similarly, a motion vector v1 of a top-right control point of the current block is derived based on MVs of neighboring subblocks. Then, the two motion vectors v0 and v1 are projected and motion vectors (v x , v y ) of each subblock within the current block are derived according to the following equation (1A).

[0491] [Equation 1]

[0492]

[0493] Here, x and y respectively represent a horizontal position and a vertical position of a subblock, and w represents a predetermined weight coefficient.

[0494] Information indicating such an affine mode (for example, referred to as an affine flag) can be signaled at the CU level. Furthermore, the signaling of the information indicating the affine mode need not be limited to the CU level, and can be another level (for example, a sequence level, a picture level, a slice level, a tile level, a CTU level, or a subblock level).

[0495] In addition, in such an affine mode, there can be several modes in which the derivation method of the MVs of the top-left and top-right control points is different. For example, in the affine mode, there are two modes, an affine inter (also referred to as an affine normal inter) mode and an affine merge mode.

[0496] Figure 46B is a diagram for explaining an example of the derivation of the MV of a sub-block unit in the affine mode using 3 control points. In Figure 46B , the current block includes 16 sub-blocks of 4x4 pixels. Here, the motion vector v0 of the top-left corner control point of the current block is derived based on the MVs of the neighboring blocks. Likewise, the motion vector v1 of the top-right corner control point of the current block is derived based on the MVs of the neighboring blocks, and the motion vector v2 of the bottom-left corner control point of the current block is derived based on the MVs of the neighboring blocks. Then, the 3 motion vectors v0, v1, and v2 are projected to derive the motion vectors (v x , v y ) of the respective sub-blocks within the current block according to the following equation (1B).

[0497] [Equation 2]

[0498]

[0499] Here, x and y represent the horizontal position and the vertical position of the center of the sub-block, respectively, and w and h represent predetermined weight coefficients. Also, w can represent the width of the current block, and h can represent the height of the current block.

[0500] The affine mode using mutually different numbers of control points (for example, 2 and 3) can also be signaled by switching at the CU level. Also, information indicating the number of control points of the affine mode used at the CU level can be signaled at another level (for example, the sequence level, the picture level, the slice level, the tile level, the CTU level, or the sub-block level).

[0501] Also, in such an affine mode having 3 control points, several modes can be included, which differ in the method of deriving the MVs of the top-left, top-right, and bottom-left corner control points. For example, in the affine mode having 3 control points, there are 2 modes, namely, the affine inter mode and the affine merge mode, as in the affine mode having 2 control points.

[0502] Also, in the affine mode, the size of each sub-block included in the current block is not limited to 4x4 pixels, and can be another size. For example, the size of each sub-block can be 8x8 pixels.

[0503] [MV derivation > affine mode > control point]

[0504] Figure 47A , Figure 47B and Figure 47C is a conceptual diagram for explaining an example of the derivation of the MV of a control point in the affine mode.

[0505] In the affine mode, as in Figure 47AAs shown, for example, based on multiple MVs corresponding to blocks coded in affine mode among the already coded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) adjacent to the current block, a predicted MV for each of the control points of the current block is calculated. Specifically, the already coded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) are examined in this order to determine the first valid block coded in affine mode. The MVs of the control points of the current block are calculated based on the multiple MVs corresponding to the determined blocks.

[0506] For example, Figure 47B As shown in FIG2 , when block A adjacent to the left side of the current block is encoded in an affine mode having two control points, motion vectors v3 and v4 are derived that are projected onto the upper left corner and upper right corner of the encoded block including block A. Then, based on the derived motion vectors v3 and v4, motion vector v0 of the upper left control point and motion vector v1 of the upper right control point of the current block are calculated.

[0507] For example, Figure 47C As shown in FIG5 , when block A adjacent to the left side of the current block is encoded in an affine mode with three control points, motion vectors v3, v4, and v5 are derived that are projected onto the positions of the upper left corner, upper right corner, and lower left corner of the encoded block containing block A. Then, based on the derived motion vectors v3, v4, and v5, motion vector v0 of the upper left control point, motion vector v1 of the upper right control point, and motion vector v2 of the lower left control point of the current block are calculated.

[0508] in addition, Figures 47A to 47C The MV derivation method shown can be used in the following Figure 50 The derivation of the MV of each control point of the current block in step Sk_1 shown in FIG. 1 can also be used in the following Figure 51 The derivation of the predicted MV of each control point of the current block in step Sj_1 is shown.

[0509] Figure 48A and Figure 48B This is a conceptual diagram for explaining another example of derivation of control point MV in the affine mode.

[0510] Figure 48A A diagram for explaining an affine pattern having two control points.

[0511] In this affine model, if Figure 48AAs shown in FIG, an MV selected from the MVs of the coded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the upper left corner control point of the current block. Similarly, an MV selected from the MVs of the coded blocks D and E adjacent to the current block is used as the motion vector v1 of the upper right corner control point of the current block.

[0512] Figure 48B A diagram for explaining an affine pattern having three control points.

[0513] In this affine model, if Figure 48B As shown in FIG, an MV selected from the MVs of the coded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the upper left corner control point of the current block. Similarly, an MV selected from the MVs of the coded blocks D and E adjacent to the current block is used as the motion vector v1 of the upper right corner control point of the current block. Furthermore, an MV selected from the MVs of the coded blocks F and G adjacent to the current block is used as the motion vector v2 of the lower left corner control point of the current block.

[0514] in addition, Figure 48A and Figure 48B The MV derivation method shown can be used in the following Figure 50 The derivation of the MV of each control point of the current block in step Sk_1 shown in FIG. 1 can also be used for the following Figure 51 The predicted MV of each control point of the current block is derived in step Sj_1.

[0515] Here, for example, when affine modes with different numbers of control points (eg, 2 and 3) are switched at the CU level and signaled, the number of control points may differ between the coded block and the current block.

[0516] Figure 49A and Figure 49B This is a conceptual diagram for explaining an example of a method for deriving MVs of control points when the number of control points in an already coded block and a current block is different.

[0517] For example, Figure 49A As shown, the current block has three control points: the top-left corner, the top-right corner, and the bottom-left corner. Block A, adjacent to the left of the current block, is coded in an affine mode with two control points. In this case, motion vectors v3 and v4 are derived, projected onto the top-left and top-right corners of the coded block containing block A. Then, based on the derived motion vectors v3 and v4, motion vector v0 for the top-left control point and motion vector v1 for the top-right control point of the current block are calculated. Furthermore, based on the derived motion vectors v0 and v1, motion vector v2 for the bottom-left control point is calculated.

[0518] For example, Figure 49B As shown, the current block has two control points, the upper left and upper right corners, and block A, adjacent to the left of the current block, is coded in affine mode with three control points. In this case, motion vectors v3, v4, and v5 are derived, projected onto the positions of the upper left, upper right, and lower left corners of the coded block containing block A. Then, based on the derived motion vectors v3, v4, and v5, motion vectors v0 for the upper left control point and v1 for the upper right control point of the current block are calculated.

[0519] in addition, Figure 49A and Figure 49B The MV derivation method shown can be used in the following Figure 50 The derivation of the MV of each control point of the current block in step Sk_1 shown in FIG. 1 can also be used for the following Figure 51 The predicted MV of each control point of the current block is derived in step Sj_1.

[0520] [MV Export > Affine Mode > Affine Merge Mode]

[0521] Figure 50 This is a flowchart showing an example of the affine merge mode.

[0522] In the affine merge mode, first, the inter prediction unit 126 derives the MV of each control point of the current block (step Sk_1). Figure 46A As shown, it is the top left and top right corner points of the current block, or as Figure 46B As shown in FIG, these are the points at the upper left corner, upper right corner, and lower left corner of the current block. In this case, the inter prediction unit 126 may also encode MV selection information for identifying the derived two or three MVs into the stream.

[0523] For example, when using Figures 47A to 47C In the case of the MV derivation method shown, as Figure 47A As shown, the inter-frame prediction unit 126 checks the encoded blocks A (left), block B (top), block C (top right), block D (bottom left) and block E (top left) in this order, and determines the initial valid block encoded in the affine mode.

[0524] The inter-frame prediction unit 126 uses the first valid block encoded in the determined affine mode to derive the MV of the control point. For example, when block A is determined and has two control points, Figure 47BAs shown, the inter-frame prediction unit 126 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block based on the motion vectors v3 and v4 of the upper left corner and the upper right corner of the coded block including the block A. For example, the inter-frame prediction unit 126 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block by projecting the motion vectors v3 and v4 of the upper left corner and the upper right corner of the coded block onto the current block.

[0525] Alternatively, in the case where block A is determined and block A has 3 control points, as Figure 47C As shown, the inter-frame prediction unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block based on the motion vectors v3, v4, and v5 of the upper left corner, upper right corner, and lower left corner of the coded block including block A. For example, the inter-frame prediction unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block by projecting the motion vectors v3, v4, and v5 of the upper left corner, upper right corner, and lower left corner of the coded block onto the current block.

[0526] In addition, as mentioned above, Figure 49A As shown, determine block A, and when block A has 2 control points, calculate the MV of 3 control points, or as mentioned above Figure 49B As shown, block A is determined, and when block A has three control points, the MVs of two control points are calculated.

[0527] Next, the inter-frame prediction unit 126 performs motion compensation on each of the multiple sub-blocks included in the current block. That is, for each of the multiple sub-blocks, the inter-frame prediction unit 126 uses two motion vectors v0 and v1 and the above-mentioned equation (1A), or uses three motion vectors v0, v1, and v2 and the above-mentioned equation (1B) to calculate the MV of the sub-block as an affine MV (step Sk_2). Then, the inter-frame prediction unit 126 performs motion compensation on the sub-block using these affine MVs and the encoded reference picture (step Sk_3). When the processes of steps Sk_2 and Sk_3 are performed on all sub-blocks included in the current block, the process of generating a predicted image using the affine merge mode for the current block is completed. In other words, motion compensation is performed on the current block, and a predicted image for the current block is generated.

[0528] In addition, in step Sk_1, the candidate MV list mentioned above may also be generated. The candidate MV list may also be a list containing candidate MVs derived using multiple MV derivation methods for each control point. Multiple MV derivation methods may be Figures 47A to 47C The MV derivation method shown, Figure 48A and Figure 48B The derivation method of the MV shown in Figure 49A and Figure 49B any combination of the derivation method of the MV shown in and the derivation method of other MVs.

[0529] In addition, the candidate MV list can also include a candidate MV of a mode other than the affine mode that predicts in a sub-block unit.

[0530] In addition, as the candidate MV list, for example, a candidate MV list including a candidate MV of the affine merge mode with 2 control points and a candidate MV of the affine merge mode with 3 control points can be generated. Alternatively, a candidate MV list including a candidate MV of the affine merge mode with 2 control points and a candidate MV list including a candidate MV of the affine merge mode with 3 control points can be generated respectively. Alternatively, a candidate MV list including a candidate MV of one of the affine merge mode with 2 control points and the affine merge mode with 3 control points can be generated. The candidate MV can be, for example, an MV of the coded blocks A (left), B (above), C (upper right), D (lower left), and E (upper left), or an MV of a valid block among these blocks.

[0531] Further, as the MV selection information, an index indicating which candidate MV in the candidate MV list can be transmitted.

[0532] [MV derivation > affine mode > affine inter mode]

[0533] Figure 51 is a flowchart indicating an example of the affine inter mode.

[0534] In the affine inter mode, first, the inter prediction section 126 derives a prediction MV (v0, v1) or (v0, v1, v2) of each of the 2 or 3 control points of the current block (step Sj_1). As shown in Figure 46A or Figure 46B The control points are, for example, a point of the upper left corner, the upper right corner, or the lower left corner of the current block.

[0535] For example, in the case of using the derivation method of the MV shown in Figure 48A and Figure 48B The inter prediction section 126 derives the prediction MV (v0, v1) or (v0, v1, v2) of each of the control points of the current block by selecting an MV of a certain block among the coded blocks in the vicinity of each of the control points of the current block shown in Figure 48A or Figure 48B At this time, the inter prediction section 126 encodes prediction MV selection information for identifying the selected 2 or 3 prediction MVs into the stream.

[0536] ​For example, the inter-frame prediction unit 126 may determine which block's MV from among the already coded blocks adjacent to the current block to select as the predicted MV for the control point by using a cost evaluation or the like, and may include a flag indicating which predicted MV was selected in the bitstream. Specifically, the inter-frame prediction unit 126 outputs the predicted MV selection information, such as the flag, as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0537] Next, the inter-frame prediction unit 126 performs motion search (steps Sj_3 and Sj_4) while simultaneously updating the predicted MV selected or derived in step Sj_1 (step Sj_2). Specifically, the inter-frame prediction unit 126 uses the MV of each sub-block corresponding to the predicted MV to be updated as an affine MV and calculates it using equation (1A) or equation (1B) (step Sj_3). The inter-frame prediction unit 126 then performs motion compensation on each sub-block using these affine MVs and the coded reference picture (step Sj_4). Each time the predicted MV is updated in step Sj_2, the processes of steps Sj_3 and Sj_4 are repeated for all blocks within the current block. As a result, during the motion search loop, the inter-frame prediction unit 126 determines, for example, the predicted MV that yields the minimum cost as the MV of the control point (step Sj_5). At this point, the inter-frame prediction unit 126 also encodes the difference between the determined MV and the predicted MV as a differential MV into the stream. That is, the inter prediction unit 126 outputs the difference MV as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130 .

[0538] Finally, the inter-frame prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the determined MV and the encoded reference picture (step Sj_6).

[0539] In addition, in step Sj_1, the candidate MV list mentioned above may also be generated. The candidate MV list may also be a list containing candidate MVs derived using multiple MV derivation methods for each control point. Multiple MV derivation methods may be Figures 47A to 47C The MV derivation method shown, Figure 48A and Figure 48B The MV derivation method shown, Figure 49A and Figure 49B The MV derivation method shown above and any combination of other MV derivation methods can be used.

[0540] Furthermore, the candidate MV list may include candidate MVs of modes other than the affine mode that perform prediction in sub-block units.

[0541] Furthermore, as a candidate MV list, a candidate MV list including candidate MVs of an affine inter mode with two control points and candidate MVs of an affine inter mode with three control points may be generated. Alternatively, a candidate MV list including candidate MVs of an affine inter mode with two control points and a candidate MV list including candidate MVs of an affine inter mode with three control points may be generated separately. Alternatively, a candidate MV list including candidate MVs of either an affine inter mode with two control points or an affine inter mode with three control points may be generated. The candidate MVs may be, for example, the MVs of the coded blocks A (left), B (top), C (top right), D (bottom left), and E (top left), or the MVs of valid blocks among these blocks.

[0542] Alternatively, an index indicating which candidate MV in the candidate MV list is selected may be sent as the predicted MV selection information.

[0543] [MV Export > Triangle Mode]

[0544] In the above example, the inter-frame prediction unit 126 generates a single rectangular predicted image for a rectangular current block. However, the inter-frame prediction unit 126 may generate multiple predicted images of shapes other than a rectangle for the current rectangular block and combine these multiple predicted images to generate a final rectangular predicted image. A shape other than a rectangle may be, for example, a triangle.

[0545] Figure 52A This is a diagram for explaining the generation of predicted images of two triangles.

[0546] The inter-frame prediction unit 126 performs motion compensation on the first triangular partition within the current block using the first MV of the first partition, thereby generating a predicted image for the triangle. Similarly, the inter-frame prediction unit 126 performs motion compensation on the second triangular partition within the current block using the second MV of the second partition, thereby generating a predicted image for the triangle. Furthermore, the inter-frame prediction unit 126 combines these predicted images to generate a predicted image of the same rectangle as the current block.

[0547] Alternatively, as the predicted image for the first partition, a rectangular first predicted image corresponding to the current block may be generated using the first MV. Alternatively, as the predicted image for the second partition, a rectangular second predicted image corresponding to the current block may be generated using the second MV. Alternatively, the predicted image for the current block may be generated by performing a weighted addition of the first and second predicted images. Furthermore, the weighted addition may be performed only on a portion of the region sandwiching the boundary between the first and second partitions.

[0548] Figure 52BThis is a conceptual diagram illustrating an example of a first portion of a first partition overlapping a second partition, and a first sample set and a second sample set that can be weighted as part of a correction process. The first portion can be, for example, one-quarter the width or height of the first partition. In another example, the first portion can have a width corresponding to N samples adjacent to the edge of the first partition. Here, N is an integer greater than zero, such as the integer 2. Figure 52B The rectangular partition represents a rectangular portion having a width that is one-fourth of the width of the first partition. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples inside the first portion. Figure 52B The central example shows a rectangular partition having a height that is one-quarter the height of partition 1. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples inside the first portion. Figure 52B The example on the right shows a triangular partition of a polygonal part with a height corresponding to two samples. Here, the first sample set contains samples outside the first part and samples inside the first part, and the second sample set contains samples inside the first part.

[0549] The first portion may be a portion of the first partition that overlaps with an adjacent partition. Figure 52C This is a conceptual diagram showing the first portion of the first partition, which is a portion of the first partition that overlaps with a portion of an adjacent partition. For simplicity of explanation, a rectangular partition is shown with a portion overlapping a spatially adjacent rectangular partition. Partitions of other shapes, such as triangular partitions, can be used, and the overlapping portion may also overlap with spatially or temporally adjacent partitions.

[0550] Furthermore, an example in which predicted images are generated for each of two partitions using inter-frame prediction has been shown, but a predicted image may be generated for at least one partition using intra-frame prediction.

[0551] Figure 53 This is a flowchart showing an example of the triangle mode.

[0552] In triangular mode, the inter-frame prediction unit 126 first divides the current block into a first partition and a second partition (step Sx_1). At this time, the inter-frame prediction unit 126 may encode partition information, which is information related to the division into each partition, as prediction parameters into the stream. Specifically, the inter-frame prediction unit 126 may output the partition information as prediction parameters to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0553] Next, the inter prediction section 126 first acquires a plurality of candidate MVs for the current block based on information such as MVs of a plurality of already encoded blocks located around the current block in time or in space (step Sx_2). That is, the inter prediction section 126 creates a candidate MV list.

[0554] Then, the inter prediction section 126 selects a candidate MV of the first partition and a candidate MV of the second partition as the first MV and the second MV, respectively, from among the plurality of candidate MVs acquired in step Sx_1 (step Sx_3). At this time, the inter prediction section 126 can also encode MV selection information for identifying the selected candidate MVs as the prediction parameters to the stream. That is, the inter prediction section 126 can output the MV selection information as the prediction parameters to the entropy encoding section 110 via the prediction parameter generation section 130.

[0555] Next, the inter prediction section 126 performs motion compensation using the selected first MV and the already encoded reference picture, thereby generating a first prediction image (step Sx_4). Likewise, the inter prediction section 126 performs motion compensation using the selected second MV and the already encoded reference picture, thereby generating a second prediction image (step Sx_5).

[0556] Finally, the inter prediction section 126 performs weighted addition of the first prediction image and the second prediction image, thereby generating a prediction image of the current block (step Sx_6).

[0557] In addition, in the example shown in FIG. 8, the first partition and the second partition are each a triangle, but can be a trapezoid, and can each be a different shape from the other. Also, in the example shown in FIG. 8, the current block is composed of two partitions, but can be composed of three or more partitions. Figure 52A Figure 52A In addition, the first partition and the second partition can also be repeated. That is, the first partition and the second partition can include the same pixel region. In this case, a prediction image in the first partition and a prediction image in the second partition can also be used to generate a prediction image of the current block.

[0558] In addition, in this example, an example in which prediction images are generated by inter prediction in both the first partition and the second partition is shown, but prediction images can also be generated by intra prediction for at least one of the partitions.

[0559] In addition, the candidate MV list used to select the first MV and the candidate MV list used to select the second MV can be different, or can be the same candidate MV list.

[0560] In addition, the candidate MV list used to select the first MV and the candidate MV list used to select the second MV can be different, or can be the same candidate MV list.

[0561] ​Further, the partition information can include an index indicating a split direction in which the current block is split into a plurality of partitions. The MV selection information can also include an index indicating the selected first MV and an index indicating the selected second MV. One index can indicate a plurality of information. For example, one index that collectively indicates a part or all of the partition information and a part or all of the MV selection information can be encoded.

[0562] [MV derivation > ATMVP mode]

[0563] Figure 54 is a diagram indicating an example of the ATMVP mode in which the MV is derived in a sub-block unit.

[0564] The ATMVP mode is a mode classified as the merge mode. For example, in the ATMVP mode, a candidate MV in a sub-block unit is registered in a candidate MV list used for the normal merge mode.

[0565] Specifically, in the ATMVP mode, first, as shown in Figure 54 , a temporal MV reference block that has correspondence with the current block is determined in an encoded reference picture specified by a MV (MV0) of a block adjacent to the lower left of the current block. Next, for each sub-block within the current block, a MV used at the time of encoding of a region within the temporal MV reference block corresponding to the sub-block is determined. The MV thus determined is included in a candidate MV list as a candidate MV of the sub-block of the current block. In a case where such a candidate MV of each sub-block is selected from the candidate MV list, motion compensation is performed on the sub-block using the candidate MV as a MV of the sub-block. Thereby, a predicted image of each sub-block is generated.

[0566] In addition, in the example shown in Figure 54 , a block adjacent to the lower left of the current block is used as the peripheral MV reference block, but a block other than this can also be used. In addition, the size of the sub-block can be 4x4 pixels, 8x8 pixels, or a size other than this. The size of the sub-block can also be switched in units of a slice, a tile, or a picture, and the like.

[0567] [Motion search > DMVR]

[0568] Figure 55 is a diagram indicating the relationship of the merge mode and the DMVR.

[0569] The inter-frame prediction unit 126 derives the MV of the current block in merge mode (step S1_1). Next, the inter-frame prediction unit 126 determines whether to perform an MV search, i.e., a motion search (step S1_2). If it is determined not to perform a motion search (No in step S1_2), the inter-frame prediction unit 126 determines the MV derived in step S1_1 as the final MV for the current block (step S1_4). In other words, in this case, the MV of the current block is determined in merge mode.

[0570] On the other hand, if it is determined in step S1_1 that a motion search is to be performed (Yes in step S1_2), the inter prediction unit 126 searches the surrounding area of ​​the reference picture indicated by the MV derived in step S1_1 to derive a final MV for the current block (step S1_3). In other words, in this case, the MV of the current block is determined by DMVR.

[0571] Figure 56 This is a conceptual diagram for explaining an example of DMVR for determining MV.

[0572] First, for example, in merge mode, candidate MVs (L0 and L1) are selected for the current block. Then, based on candidate MV (L0), reference pixels are determined from the first reference picture (L0), a coded picture in the L0 list. Similarly, based on candidate MV (L1), reference pixels are determined from the second reference picture (L1), a coded picture in the L1 list. A template is generated by averaging these reference pixels.

[0573] Next, using this template, the surrounding areas of the candidate MVs in the first reference picture (L0) and the second reference picture (L1) are searched, and the MV with the lowest cost is determined as the final MV for the current block. Alternatively, the cost can be calculated using, for example, the difference between the pixel values ​​of the template and the pixel values ​​of the search area, as well as the candidate MV values.

[0574] Even if it is not the process described here, any process can be used as long as it can search the vicinity of the candidate MV and derive the final MV.

[0575] Figure 57 This is a conceptual diagram for explaining another example of DMVR for determining MV. Figure 57 The example shown is the same as Figure 56 The example of DMVR shown is different in that the cost is calculated without generating a template.

[0576] First, the inter prediction unit 126 searches for the reference blocks included in the reference pictures of the L0 list and the L1 list based on the candidate MV, that is, the initial MV, obtained from the candidate MV list. Figure 57As shown, the initial MV corresponding to the reference block in the L0 list is InitMV_L0, and the initial MV corresponding to the reference block in the L1 list is InitMV_L1. During motion search, the inter-frame prediction unit 126 first sets a search position for the reference picture in the L0 list. The difference vector representing this set search position, specifically the difference vector from the position indicated by the initial MV (i.e., InitMV_L0) to this search position, is MVd_L0. The inter-frame prediction unit 126 then determines a search position in the reference picture in the L1 list. This search position is represented by the difference vector from the position indicated by the initial MV (i.e., InitMV_L1) to this search position. Specifically, the inter-frame prediction unit 126 determines MVd_L1 by mirroring the difference vector MVd_L0. That is, the inter-frame prediction unit 126 sets the search position at a position symmetrical to the position indicated by the initial MV in each of the L0 and L1 lists. The inter prediction section 126 calculates the sum of absolute differences (SAD) of pixel values ​​within the block of the search position as a cost for each search position, and finds a search position that minimizes the cost.

[0577] Figure 58A is a diagram showing an example of motion search in DMVR. Figure 58B : is a flowchart showing an example of the motion search.

[0578] First, in Step 1, the inter-frame prediction unit 126 calculates the cost for the search position (also called the starting point) represented by the initial MV and the eight surrounding search positions. The inter-frame prediction unit 126 then determines whether the cost of a search position other than the starting point is the lowest. If the cost of a search position other than the starting point is the lowest, the inter-frame prediction unit 126 moves to the search position with the lowest cost and proceeds to Step 2. On the other hand, if the cost of the starting point is the lowest, the inter-frame prediction unit 126 skips Step 2 and proceeds to Step 3.

[0579] In Step 2, the inter-frame prediction unit 126 uses the search position moved based on the processing results of Step 1 as a new starting point and performs the same search as in Step 1. Furthermore, the inter-frame prediction unit 126 determines whether the cost of a search position other than the starting point is the lowest. If the cost of a search position other than the starting point is the lowest, the inter-frame prediction unit 126 proceeds to Step 4. On the other hand, if the cost of the starting point is the lowest, the inter-frame prediction unit 126 proceeds to Step 3.

[0580] In Step 4 , the inter prediction unit 126 treats the search position of the starting point as the final search position, and determines the difference between the position indicated by the initial MV and the final search position as a difference vector.

[0581] In Step 3, the inter-frame prediction unit 126 determines the fractional-precision pixel position with the lowest cost based on the costs at four points located above, below, and to the left and right of the starting point in Step 1 or Step 2, and sets this pixel position as the final search position. This fractional-precision pixel position is determined by weighted summing the vectors ((0, 1), (0, -1), (-1, 0), and (1, 0)) located above, below, and to the left, using the costs of the search positions of each of the four points as weights. The inter-frame prediction unit 126 then determines the difference between the position represented by the initial MV and this final search position as a difference vector.

[0582] [Motion Compensation > BIO / OBMC / LIC]

[0583] In motion compensation, there are modes for generating a predicted image and then correcting the predicted image. Examples of these modes include BIO, OBMC, and LIC, which will be described later.

[0584] Figure 59 This is a flowchart showing an example of generating a predicted image.

[0585] The inter prediction unit 126 generates a predicted image (step Sm_1 ), and corrects the predicted image using any of the above-mentioned modes (step Sm_2 ).

[0586] Figure 60 This is a flowchart showing another example of generating a predicted image.

[0587] The inter-frame prediction unit 126 derives the MV of the current block (step Sn_1). Next, the inter-frame prediction unit 126 generates a predicted image using this MV (step Sn_2) and determines whether to perform correction processing (step Sn_3). If it is determined that correction processing is to be performed (yes in step Sn_3), the inter-frame prediction unit 126 corrects the predicted image to generate a final predicted image (step Sn_4). Furthermore, in the LIC described later, luminance and chrominance may also be corrected in step Sn_4. On the other hand, if it is determined that correction processing is not to be performed (no in step Sn_3), the inter-frame prediction unit 126 outputs the predicted image as the final predicted image without correction (step Sn_5).

[0588] [Motion Compensation > OBMC]

[0589] Not only the motion information of the current block obtained by the motion search, but also the motion information of the neighboring block can be used to generate the inter prediction image. Specifically, the inter prediction image can also be generated in the sub-block unit within the current block by weighted adding a prediction image based on the motion information obtained by the motion search (within the reference picture) and a prediction image based on the motion information of the neighboring block (within the current picture). Such inter prediction (motion compensation) is sometimes referred to as OBMC (overlapped block motion compensation) or OBMC mode.

[0590] In the OBMC mode, information indicating the size of the sub-block for OBMC (e.g., referred to as OBMC block size) can also be signaled at the sequence level. Also, information indicating whether to apply the OBMC mode (e.g., referred to as OBMC flag) can also be signaled at the CU level. In addition, the level of the signaling of these information does not need to be limited to the sequence level and the CU level, but can be other levels (e.g., the picture level, the slice level, the tile level, the CTU level, or the sub-block level).

[0591] The OBMC mode is described in more detail. Figure 61 and Figure 62 are a flowchart and a conceptual diagram for explaining the outline of the prediction image correction process based on OBMC.

[0592] First, as shown in Figure 62 , a prediction image (Pred) based on the usual motion compensation is obtained using the MV assigned to the current block. In Figure 62 , the arrow "MV" points to the reference picture, and indicates which block of the current picture refers to in order to obtain the prediction image.

[0593] Next, the MV (MV_L) already derived for the encoded left neighboring block is applied (reused) to the current block to obtain a prediction image (Pred_L). The MV (MV_L) is indicated by the arrow "MV_L" pointing from the current block to the reference picture. Then, the 1st correction of the prediction image is performed by overlapping the 2 prediction images Pred and Pred_L. This has the effect of mixing the boundaries between the neighboring blocks.

[0594] Likewise, the MV (MV_U) that has been derived for the coded upper neighboring block is applied (reused) to the current block, resulting in a prediction picture (Pred_U). The MV (MV_U) is indicated by the arrow "MV_U" pointing from the current block to the reference picture. Then, a second correction of the prediction picture is performed by overlaying the prediction picture Pred_U with the prediction picture that has been corrected for the first time (e.g. Pred and Pred_L). This has the effect of blending the boundaries between neighboring blocks. The prediction picture resulting from the second correction is the final prediction picture for the current block with the boundaries to the neighboring blocks being blended (smoothed).

[0595] Further, the above-described example is a two-path correction method using the left and upper neighboring blocks, but the correction method can also be a three-path or more path correction method using the right and / or lower neighboring blocks as well.

[0596] In addition, the region to which the overlay is performed can not be the entire pixel region of the block, but only a partial region near the block boundary.

[0597] In addition, the prediction picture correction processing of the OBMC for obtaining one prediction picture Pred by overlaying one reference picture with the additional prediction pictures Pred_L and Pred_U is described here. However, in the case where the prediction picture is corrected based on multiple reference pictures, the same processing can be applied to the multiple reference pictures respectively. In this case, by performing the OBMC picture correction based on multiple reference pictures, after the corrected prediction pictures are obtained from the respective reference pictures, the final prediction picture is obtained by further overlaying the obtained multiple corrected prediction pictures.

[0598] In addition, in the OBMC, the unit of the current block can be the PU unit, or the sub-block unit obtained by further dividing the PUs.

[0599] As a method of determining whether or not to apply the OBMC, for example, there is a method of using a signal indicating whether or not to apply the OBMC, i.e. obmc_flag. As a specific example, the encoding device 100 can determine whether or not the current block belongs to a region of complex motion. The encoding device 100 encodes by setting the value 1 to the obmc_flag and applying the OBMC in the case where it belongs to the region of complex motion, and encodes without applying the OBMC in the case where it does not belong to the region of complex motion, by setting the value 0 to the obmc_flag. On the other hand, in the decoding device 200, by decoding the obmc_flag described in the stream, whether or not to apply the OBMC is switched according to the value.

[0600] [Motion compensation > BIO]

[0601] Next, a method of deriving the MV will be described. First, a mode of deriving the MV based on a model assuming constant velocity straight line motion will be described. This mode is sometimes referred to as a bi-directional optical flow (BIO) mode. In addition, this bi-directional optical flow can be expressed as BDOF instead of BIO.

[0602] Figure 63 is a diagram for explaining the model assuming constant velocity straight line motion. In Figure 63 , (v x , v y ) represents a velocity vector, τ0, τ1 represent distances in time between a current picture (Cur Pic) and two reference pictures (Ref0, Ref1), respectively. (MVx0, MVy0) represents the MV corresponding to the reference picture Ref0, and (MVx1, MVy1) represents the MV corresponding to the reference picture Ref1.

[0603] At this time, under the assumption of constant velocity straight line motion of the velocity vector (v x , v y ), (MVx0, MVy0) and (MVx1, MVy1) are represented as (vxτ0, vyτ0) and (-vxτ1, -vyτ1), respectively, and the following optical flow equation (2) holds.

[0604]

Equation 3

[0605]

[0606] Here, I(k) represents a luminance value of the reference image k (k = 0, 1) after motion compensation. This optical flow equation represents that the sum of (i) a time differential of the luminance value, (ii) a product of a horizontal component of a velocity in the horizontal direction and a spatial gradient of the reference image, and (iii) a product of a vertical component of a velocity in the vertical direction and a spatial gradient of the reference image is equal to zero. It can also be that, based on a combination of this optical flow equation and Hermite interpolation, a block unit motion vector obtained from a candidate MV list or the like is corrected in pixel units.

[0607] In addition, the MV can also be derived on the decoding apparatus 200 side by a method different from the derivation of the motion vector based on the model assuming constant velocity straight line motion. For example, a motion vector can also be derived in sub-block units based on MVs of a plurality of neighboring blocks.

[0608] Figure 64 is a flowchart representing an example of inter prediction according to BIO. In addition, Figure 65is a diagram showing an example of a functional configuration of the inter prediction section 126 that performs the inter prediction according to BIO.

[0609] As shown in Figure 65 , the inter prediction section 126 includes, for example, a memory 126a, an interpolated image derivation section 126b, a gradient image derivation section 126c, an optical flow derivation section 126d, a correction value derivation section 126e, and a prediction image correction section 126f. Note that the memory 126a can also be the frame memory 122.

[0610] The inter prediction section 126 derives two motion vectors (M0, M1) using two reference pictures (Ref0, Ref1) different from a picture (Cur Pic) that includes the current block. Then, the inter prediction section 126 derives a prediction image of the current block using the two motion vectors (M0, M1) (step Sy_1). Note that the motion vector M0 is a motion vector (MVx0, MVy0) corresponding to the reference picture Ref0, and the motion vector M1 is a motion vector (MVx1, MVy1) corresponding to the reference picture Ref1.

[0611] Next, the interpolated image derivation section 126b refers to the memory 126a and derives an interpolated image I 0 of the current block using the motion vector M0 and the reference picture L0. In addition, the interpolated image derivation section 126b refers to the memory 126a and derives an interpolated image I 1 of the current block using the motion vector M1 and the reference picture L1 (step Sy_2). Here, the interpolated image I 0 is an image included in the reference picture Ref0, which is derived for the current block, and the interpolated image I 1 is an image included in the reference picture Ref1, which is derived for the current block. The interpolated image I 0 and the interpolated image I 1 may each be the same size as the current block. Alternatively, in order to appropriately derive a gradient image described later, the interpolated image I 0 and the interpolated image I 1 may each be an image larger than the current block. Furthermore, the interpolated image I 0 and I 1 may include a prediction image derived by applying the motion vectors (M0, M1) and the reference pictures (L0, L1) and a motion compensation filter.

[0612] In addition, the gradient image derivation section 126c derives gradient images (Ix 0 , Ix 1 , Iy 0 , and Iy 1 of the current block from the interpolated image I 0 and the interpolated image I 1) (step Sy_3). Further, the gradient image in the horizontal direction is (Ix 0 , Ix 1 ), and the gradient image in the vertical direction is (Iy 0 , Iy 1 ). The gradient image derivation section 126c can derive the gradient image by applying a gradient filter to the interpolated image, for example. The gradient image only needs to indicate the amount of spatial variation of pixel values in the horizontal direction or the vertical direction.

[0613] Next, the optical flow derivation section 126d derives the optical flow (vx, vy) as the above-mentioned velocity vector using the interpolated images (I 0 , I 1 ) and the gradient images (Ix 0 , Ix 1 , Iy 0 , Iy 1 ) in units of the plurality of sub-blocks that constitute the current block (step Sy_4). The optical flow is a coefficient that corrects the amount of spatial movement of a pixel and can also be referred to as a local motion estimation value, a correction motion vector, or a correction weight vector. As an example, the sub-block can be a sub-CU of 4x4 pixels. In addition, the derivation of the optical flow can not be performed in units of sub-blocks, but can be performed in units of pixels or other units.

[0614] Next, the inter prediction section 126 corrects the prediction image of the current block using the optical flow (vx, vy). For example, the correction value derivation section 126e derives a correction value of the value of a pixel included in the current block using the optical flow (vx, vy) (step Sy_5). Also, the prediction image correction section 126f can correct the prediction image of the current block using the correction value (step Sy_6). In addition, the correction value can be derived in units of each pixel, or can be derived in units of a plurality of pixels or sub-blocks.

[0615] Further, the processing flow of the BIO is not limited to the processing disclosed in Figure 64 . The processing can implement only a part of the processing disclosed in Figure 64 , can add or replace a different processing, or can be executed in a different processing order.

[0616] [Motion compensation > LIC]

[0617] Next, an example of a mode in which a prediction image (prediction) is generated using LIC (local illumination compensation) will be described.

[0618] Figure 66A is a diagram for explaining an example of a prediction image generation method using LIC-based luminance correction processing. In addition, Figure 66Bis a flowchart showing an example of a method of generating a prediction image using the LIC.

[0619] First, the inter prediction section 126 derives MVs from the coded reference pictures, and acquires a reference image corresponding to the current block (step Sz_1).

[0620] Next, the inter prediction section 126 extracts information indicating how luminance values change in the reference picture and the current picture for the current block (step Sz_2). This extraction is performed based on luminance pixel values of the coded left-adjacent reference region (peripheral reference region) and the coded upper-adjacent reference region (peripheral reference region) in the current picture, and luminance pixel values at equivalent positions in the reference picture specified by the derived MVs. Then, the inter prediction section 126 calculates a luminance correction parameter using the information indicating how luminance values change (step Sz_3).

[0621] The inter prediction section 126 generates a prediction image for the current block by performing luminance correction processing on the reference image in the reference picture specified by the MVs using the luminance correction parameter (step Sz_4). That is, the prediction image as the reference image in the reference picture specified by the MVs is corrected based on the luminance correction parameter. In this correction, luminance can be corrected, or color difference can be corrected. That is, a correction parameter of color difference can be calculated using information indicating how color difference changes, and color difference correction processing can be performed.

[0622] In addition, Figure 66A The shape of the peripheral reference region in the above example can be used, or a shape other than this can be used.

[0623] Furthermore, the processing of generating a prediction image from one reference picture is described here, but the same applies to the case of generating a prediction image from a plurality of reference pictures, and a prediction image can be generated after performing luminance correction processing on the reference images acquired from each of the reference pictures in the same manner as described above.

[0624] As a method of determining whether or not to apply the LIC, for example, there is a method of using lic_flag as a signal indicating whether or not to apply the LIC. As a specific example, in the encoding device 100, it is determined whether or not the current block belongs to a region in which luminance changes have occurred, and in the case where it belongs to a region in which luminance changes have occurred, a value of 1 is set as lic_flag, and the LIC is applied to perform encoding, and in the case where it does not belong to a region in which luminance changes have occurred, a value of 0 is set as lic_flag, and the LIC is not applied to perform encoding. On the other hand, in the decoding device 200, lic_flag described in the stream can be decoded, and whether or not to apply the LIC is switched according to the value to perform decoding.

[0625] Another method for determining whether to apply LIC is to determine whether LIC is applied to neighboring blocks. As a specific example, when the current block is processed in merge mode, the inter-frame prediction unit 126 determines whether the neighboring, already-encoded blocks selected during MV derivation in merge mode were coded using LIC. Based on the result, the inter-frame prediction unit 126 switches whether to apply LIC and performs coding. In this example, the same process also applies to the decoding device 200.

[0626] use Figure 66A and Figure 66B LIC (luminance correction processing) has been described above, and its details will be described below.

[0627] First, the inter prediction unit 126 derives an MV for acquiring a reference image corresponding to the current block from a reference picture that is an already coded picture.

[0628] Next, the inter-frame prediction unit 126 uses the luminance pixel values ​​of the coded neighboring reference areas to the left and above the current block, as well as the luminance pixel values ​​at the same position in the reference picture specified by the MV, to extract information indicating how the luminance values ​​in the reference picture and the current picture change, and calculates a luminance correction parameter. For example, the luminance pixel value of a pixel in the neighboring reference area in the current picture is set to p0, and the luminance pixel value of the pixel in the neighboring reference area at the same position in the reference picture is set to p1. The inter-frame prediction unit 126 calculates coefficients A and B for optimizing the solution A×p1+B=p0 for multiple pixels in the neighboring reference area as luminance correction parameters.

[0629] Next, the inter-frame prediction unit 126 uses the brightness correction parameters to perform brightness correction on the reference image within the reference picture specified by the MV to generate a predicted image for the current block. For example, the brightness pixel value in the reference image is set to p2, and the brightness pixel value of the predicted image after brightness correction is set to p3. The inter-frame prediction unit 126 calculates A×p2+B=p3 for each pixel in the reference image to generate the predicted image after brightness correction.

[0630] Alternatively, you can use Figure 66A For example, a region including a predetermined number of pixels thinned out from the upper adjacent pixels and the left adjacent pixels may be used as the peripheral reference region. In addition, the peripheral reference region is not limited to the region adjacent to the current block, and may also be a region not adjacent to the current block. Figure 66AIn the illustrated example, the peripheral reference region in the reference picture is a region specified by the MV of the current picture from the peripheral reference region in the current picture, but can also be a region specified by another MV. For example, the other MV can also be the MV of the peripheral reference region in the current picture.

[0631] In addition, the operation in the encoding device 100 is described here, but the operation in the decoding device 200 is the same.

[0632] Further, LIC can be applied not only to luminance but also to color difference. At this time, a correction parameter can be derived individually for each of Y, Cb, and Cr, or a common correction parameter can be used for any one of them.

[0633] Further, LIC can also be applied in sub-block units. For example, a peripheral reference region of a current sub-block and a peripheral reference region of a reference sub-block in a reference picture specified by the MV of the current sub-block can also be used to derive a correction parameter.

[0634] [Prediction control section]

[0635] The prediction control section 128 selects one of the intra-predicted image (pixels or signals output from the intra-prediction section 124) and the inter-predicted image (pixels or signals output from the inter-prediction section 126), and outputs the selected predicted image to the subtraction section 104 and the addition section 116.

[0636] [Prediction parameter generation section]

[0637] The prediction parameter generation section 130 can output, as a prediction parameter, information related to intra-prediction, inter-prediction, and selection of a predicted image in the prediction control section 128, to the entropy encoding section 110. The entropy encoding section 110 can generate a stream based on the prediction parameter input from the prediction parameter generation section 130 and the quantization coefficients input from the quantization section 108. The prediction parameter can also be used in the decoding device 200. The decoding device 200 can also receive and decode the stream and perform the same processing as the prediction processing performed in the intra-prediction section 124, the inter-prediction section 126, and the prediction control section 128. The prediction parameter can include a selected prediction signal (e.g., an MV, a prediction type, or a prediction mode used by the intra-prediction section 124 or the inter-prediction section 126), or any index, flag, or value based on or representing the prediction processing performed in the intra-prediction section 124, the inter-prediction section 126, and the prediction control section 128.

[0638] [Decoding device]

[0639] Next, a decoding device 200 capable of decoding a stream output from the above-described encoding device 100 will be described. Figure 67is a block diagram showing an example of a functional configuration of a decoding apparatus 200 according to an embodiment. The decoding apparatus 200 is an apparatus that decodes an encoded image, i.e., a stream, in a block unit.

[0640] As shown in Figure 67 , the decoding apparatus 200 includes an entropy decoding section 202, an inverse quantization section 204, an inverse transform section 206, an addition section 208, a block memory 210, a loop filter section 212, a frame memory 214, an intra prediction section 216, an inter prediction section 218, a prediction control section 220, a prediction parameter generation section 222, and a partition decision section 224. In addition, the intra prediction section 216 and the inter prediction section 218 each constitute a part of a prediction processing section.

[0641] [Example of Installation of Decoding Apparatus]

[0642] Figure 68 is a block diagram showing an example of installation of the decoding apparatus 200. The decoding apparatus 200 includes a processor b1 and a memory b2. For example, as shown in Figure 67 , the plurality of constituent elements of the decoding apparatus 200 are installed by the processor b1 and the memory b2. Figure 68

[0643] The processor b1 is a circuit that performs information processing, and is a circuit that can access the memory b2. For example, the processor b1 is a special-purpose or general-purpose electronic circuit that decodes a stream. The processor b1 can also be a processor such as a CPU. In addition, the processor b1 can also be a collection of a plurality of electronic circuits. In addition, for example, the processor b1 can also function as a plurality of constituent elements of the decoding apparatus 200 other than the constituent elements for storing information. Figure 67

[0644] The memory b2 is a special-purpose or general-purpose memory that stores information used for the processor b1 to decode a stream. The memory b2 can be an electronic circuit, or can be connected to the processor b1. In addition, the memory b2 can be included in the processor b1. In addition, the memory b2 can be a collection of a plurality of electronic circuits. In addition, the memory b2 can be a magnetic disk or an optical disk, or can be represented as a storage or a recording medium. In addition, the memory b2 can be a nonvolatile memory, or can be a volatile memory.

[0645] For example, the memory b2 can store an image, or can store a stream. In addition, a program for the processor b1 to decode a stream can also be stored in the memory b2.

[0646] In addition, for example, the memory b2 can also function as a plurality of constituent elements of the decoding apparatus 200 other than the constituent elements for storing information. Figure 67 ​​the role of the constituent element for storing information among the plurality of constituent elements of the decoding apparatus 200 illustrated in FIG. 2. Specifically, the memory b2 can function as a memory for storing a reconstructed image (specifically, a reconstructed block or a reconstructed picture, or the like) and a memory for storing a decoded image (specifically, a decoded block or a decoded picture, or the like). Figure 67 the roles of the block memory 210 and the frame memory 214 illustrated in FIG. 2. More specifically, in the memory b2, a reconstructed image (specifically, a reconstructed block or a reconstructed picture, or the like) can be stored.

[0647] In addition, in the decoding apparatus 200, all of the plurality of constituent elements illustrated in FIG. 2 can not be installed, and all of the plurality of processes described above can not be performed. Figure 67 In addition, in the decoding apparatus 200, all of the plurality of constituent elements illustrated in FIG. 2 can not be installed, and all of the plurality of processes described above can not be performed. Figure 67 In addition, in the decoding apparatus 200, all of the plurality of constituent elements illustrated in FIG. 2 can not be installed, and all of the plurality of processes described above can not be performed.

[0648] Hereinafter, after the flow of the overall process of the decoding apparatus 200 is described, each constituent element included in the decoding apparatus 200 is described. Furthermore, for each constituent element included in the decoding apparatus 200, a detailed description is omitted for the constituent element that performs the same process as the constituent element included in the encoding apparatus 100. For example, the inverse quantization section 204, the inverse transform section 206, the addition section 208, the block memory 210, the frame memory 214, the intra prediction section 216, the inter prediction section 218, the prediction control section 220, and the loop filter section 212 included in the decoding apparatus 200 respectively perform the same processes as the inverse quantization section 112, the inverse transform section 114, the addition section 116, the block memory 118, the frame memory 122, the intra prediction section 124, the inter prediction section 126, the prediction control section 128, and the loop filter section 120 included in the encoding apparatus 100.

[0649] [Overall flow of decoding process]

[0650] Figure 69 is a flowchart indicating an example of the overall decoding process performed by the decoding apparatus 200.

[0651] First, the split decision section 224 of the decoding apparatus 200 decides the split pattern of each of a plurality of fixed-size blocks (128 x 128 pixels) included in a picture based on the parameters input from the entropy decoding section 202 (step Sp_1). The split pattern is the split pattern selected by the encoding apparatus 100. Then, the decoding apparatus 200 performs the processes of steps Sp_2 to Sp_6 on each of the plurality of blocks constituting the split pattern.

[0652] The entropy decoding section 202 decodes (specifically, entropy decodes) the encoded quantized coefficients and the prediction parameters of the current block (step Sp_2).

[0653] Next, the inverse quantization unit 204 and the inverse transformation unit 206 perform inverse quantization and inverse transformation on the plurality of quantized coefficients to restore the prediction residual of the current block (step Sp_3).

[0654] Next, the prediction processing unit composed of the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220 generates a predicted image of the current block (step Sp_4).

[0655] Next, the adding unit 208 reconstructs the current block into a reconstructed image (also referred to as a decoded image block) by adding the predicted image to the prediction residual (step Sp_5).

[0656] Then, when the reconstructed image is generated, the loop filter unit 212 filters the reconstructed image (step Sp_6).

[0657] Then, the decoding device 200 determines whether decoding of the entire picture is completed (step Sp_7 ). If it is determined that decoding is not completed (No in step Sp_7 ), the processing from step Sp_1 is repeatedly executed.

[0658] Furthermore, the processing of steps Sp_1 to Sp_7 may be performed sequentially by the decoding device 200 , and a plurality of some of these processing may be performed in parallel or in a different order.

[0659] [Division Decision Section]

[0660] Figure 70 : is a diagram showing the relationship between the division determination unit 224 and other components. As an example, the division determination unit 224 may also perform the following processing.

[0661] The partition decision unit 224 collects block information from, for example, the block memory 210 or the frame memory 214 and further obtains parameters from the entropy decoding unit 202. Furthermore, the partition decision unit 224 may determine a partitioning pattern for the fixed-size blocks based on the block information and parameters. Furthermore, the partition decision unit 224 may output information indicating the determined partitioning pattern to the inverse transform unit 206, the intra-frame prediction unit 216, and the inter-frame prediction unit 218. The inverse transform unit 206 may inversely transform the transform coefficients based on the partitioning pattern indicated by the information from the partition decision unit 224. The intra-frame prediction unit 216 and the inter-frame prediction unit 218 may generate a predicted image based on the partitioning pattern indicated by the information from the partition decision unit 224.

[0662] [Entropy decoding unit]

[0663] Figure 71 This is a block diagram showing an example of the functional configuration of the entropy decoding unit 202 .

[0664] The entropy decoding unit 202 entropy decodes the stream to generate quantization coefficients, prediction parameters, and parameters related to the segmentation pattern. CABAC is used, for example, in this entropy decoding. Specifically, the entropy decoding unit 202 includes, for example, a binary arithmetic decoding unit 202a, a context control unit 202b, and a multi-valued unit 202c. The binary arithmetic decoding unit 202a performs arithmetic decoding on the stream as a binary signal using the context values ​​derived by the context control unit 202b. Similar to the context control unit 110b of the encoding device 100, the context control unit 202b derives context values, i.e., the probability of occurrence of the binary signal, based on the characteristics of the syntactic elements or the surrounding conditions. The multi-valued unit 202c performs debinarization, converting the binary signal output from the binary arithmetic decoding unit 202a into a multi-valued signal representing the aforementioned quantization coefficients, etc. This debinarization is performed using the binarization method described above.

[0665] The entropy decoding unit 202 outputs the quantized coefficients to the inverse quantization unit 204 in units of blocks. The entropy decoding unit 202 may also output the stream to the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220 (see Figure 1 The intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220 can perform the same prediction processing as that performed by the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 on the encoding device 100 side.

[0666] [Entropy decoding unit]

[0667] Figure 72 3 is a diagram showing the CABAC flow in the entropy decoding unit 202 .

[0668] Initialization is first performed in the CABAC encoding process within the entropy decoding unit 202. This initialization initializes the binary arithmetic decoding unit 202c and sets the initial context value. The binary arithmetic decoding unit 202c and the multi-valued encoding unit 202c then perform arithmetic decoding and multi-valued encoding on, for example, the coded data of a CTU. At this time, the context control unit 202b updates the context value each time arithmetic decoding is performed. The context control unit 202b then backs off the context value as post-processing. This backed-off context value is used, for example, as the initial context value for the next CTU.

[0669] [Inverse quantization unit]

[0670] The inverse quantization section 204 inversely quantizes the quantization coefficients of the current block that are input from the entropy decoding section 202. Specifically, the inverse quantization section 204 inversely quantizes the quantization coefficients of the current block on a per-quantization coefficient basis based on the quantization parameter corresponding to the quantization coefficient. Then, the inverse quantization section 204 outputs the inversely quantized quantization coefficients (i.e., the transform coefficients) of the current block to the inverse transform section 206.

[0671] Figure 73 is a block diagram that shows an example of the functional structure of the inverse quantization section 204.

[0672] The inverse quantization section 204 includes, for example, a quantization parameter generation section 204a, a predicted quantization parameter generation section 204b, a quantization parameter storage section 204d, and an inverse quantization processing section 204e.

[0673] Figure 74 is a flowchart that shows an example of the inverse quantization performed by the inverse quantization section 204.

[0674] As an example, the inverse quantization section 204 can perform the inverse quantization processing on a per-CU basis according to the flow shown in Figure 74 Specifically, the quantization parameter generation section 204a determines whether or not to perform inverse quantization (step Sv_11). Here, when it is determined to perform inverse quantization (Yes in step Sv_11), the quantization parameter generation section 204a acquires the differential quantization parameter of the current block from the entropy decoding section 202 (step Sv_12).

[0675] Next, the predicted quantization parameter generation section 204b acquires the quantization parameter of a processing unit other than the current block from the quantization parameter storage section 204d (step Sv_13). The predicted quantization parameter generation section 204b generates the predicted quantization parameter of the current block based on the acquired quantization parameter (step Sv_14).

[0676] Then, the quantization parameter generation section 204a adds the differential quantization parameter of the current block acquired from the entropy decoding section 202 and the predicted quantization parameter of the current block generated by the predicted quantization parameter generation section 204b (step Sv_15). By this addition, the quantization parameter of the current block is generated. Furthermore, the quantization parameter generation section 204a saves the quantization parameter of the current block in the quantization parameter storage section 204d (step Sv_16).

[0677] Next, the inverse quantization processing section 204e inversely quantizes the quantization coefficients of the current block into the transform coefficients using the quantization parameter generated in step Sv_15 (step Sv_17).

[0678] In addition, the differential quantization parameter can also be decoded at the bit sequence level, picture level, slice level, brick level, or CTU level. In addition, the initial value of the quantization parameter can also be decoded at the sequence level, picture level, slice level, brick level, or CTU level. In this case, the quantization parameter can be generated using the initial value of the quantization parameter and the differential quantization parameter.

[0679] Furthermore, the inverse quantization unit 204 may include a plurality of inverse quantizers, and may inverse quantize the quantized coefficients using an inverse quantization method selected from a plurality of inverse quantization methods.

[0680] [Inverse transformation unit]

[0681] The inverse transform unit 206 restores the prediction residual by performing inverse transform on the transform coefficients input from the inverse quantization unit 204 .

[0682] For example, when the information read from the stream indicates that EMT or AMT is applied (for example, the AMT flag is true), the inverse transform unit 206 performs an inverse transform on the transform coefficients of the current block based on the read information indicating the transform type.

[0683] Furthermore, for example, when the information read from the stream indicates that NSST is applied, the inverse transform unit 206 applies inverse re-transformation to the transform coefficients.

[0684] Figure 75 This is a flowchart showing an example of processing performed by the inverse transformation unit 206.

[0685] For example, the inverse transform unit 206 determines whether information indicating that an orthogonal transform is not to be performed exists in the stream (step St_11). If this information is not present (No in step St_11), the inverse transform unit 206 obtains information indicating the transform type decoded by the entropy decoding unit 202 (step St_12). Next, based on this information, the inverse transform unit 206 determines the transform type to be used for the orthogonal transform in the encoding device 100 (step St_13). The inverse transform unit 206 then performs an inverse orthogonal transform using the determined transform type (step St_14).

[0686] Figure 76 This is a flowchart showing another example of the processing performed by the inverse transformation unit 206.

[0687] For example, the inverse transform unit 206 determines whether the transform size is equal to or smaller than a predetermined value (step Su 11). When it is determined that the transform size is equal to or smaller than the predetermined value (Yes in step Su 11), the inverse transform unit 206 acquires information indicating which one of one or more transform types included in the first transform type group is used by the encoding apparatus 100 from the entropy decoding unit 202 (step Su 12). In addition, such information is decoded by the entropy decoding unit 202 and output to the inverse transform unit 206.

[0688] The inverse transform unit 206 determines a transform type used in orthogonal transform in the encoding apparatus 100 on the basis of the information (step Su 13). Then, the inverse transform unit 206 inversely orthogonal-transforms the transform coefficients of the current block using the determined transform type (step Su 14). On the other hand, when it is determined that the transform size is not equal to or smaller than the predetermined value (No in step Su 11), the inverse transform unit 206 inversely orthogonal-transforms the transform coefficients of the current block using the second transform type group (step Su 15).

[0689] In addition, as an example, the inverse orthogonal transform performed by the inverse transform unit 206 can be implemented in accordance with the flowchart shown in FIG. 6. In addition, the information indicating the transform type used in orthogonal transform can not be decoded, and the inverse orthogonal transform can be performed using a predetermined transform type. In addition, specifically, the transform type is DST7 or DCT8 or the like, and an inverse transform basis function corresponding to the transform type is used in the inverse orthogonal transform. Figure 75 Or Figure 76

[0690] [Addition unit]

[0691] The addition unit 208 reconstructs the current block by adding the prediction residual as input from the inverse transform unit 206 to the prediction image as input from the prediction control unit 220. That is, a reconstructed image of the current block is generated. Furthermore, the addition unit 208 outputs the reconstructed image of the current block to the block memory 210 and the loop filter unit 212.

[0692] [Block memory]

[0693] The block memory 210 is a storage unit that stores a block within the current picture referred to in intra prediction. Specifically, the block memory 210 stores the reconstructed image output from the addition unit 208.

[0694] [Loop filter unit]

[0695] The loop filter unit 212 applies loop filtering to the reconstructed image generated by the addition unit 208, and outputs the reconstructed image on which the filtering has been performed to the frame memory 214 and a display apparatus or the like.

[0696] ​In a case where the information indicating the on / off of the ALF read out from the stream indicates the on of the ALF, one filter is selected from among the plurality of filters based on the direction and the activity of the gradient of the locality, and the selected filter is applied to the reconstructed image.

[0697] Figure 77 is a block diagram indicating an example of a functional structure of the loop filter 212. Further, the loop filter 212 has the same structure as the loop filter 120 of the encoding apparatus 100.

[0698] The loop filter 212, for example, as shown in Figure 77 , has a deblocking filter processing section 212a, an SAO processing section 212b, and an ALF processing section 212c. The deblocking filter processing section 212a applies the deblocking filter processing described above to the reconstructed image. The SAO processing section 212b applies the SAO processing described above to the reconstructed image after the deblocking filter processing. In addition, the ALF processing section 212c applies the ALF processing described above to the reconstructed image after the SAO processing. Further, the loop filter 212 can not have all the processing sections disclosed in Figure 77 , but can have only a part of the processing sections. In addition, the loop filter 212 can be a structure that performs each of the above-described processes in an order different from the order of the processes disclosed in Figure 77

[0699] [Frame Memory]

[0700] The frame memory 214 is a storage section for storing a reference picture used in inter prediction, and is also called a frame buffer. Specifically, the frame memory 214 stores the reconstructed image after the filtering performed by the loop filter 212.

[0701] [Prediction Section (Intra Prediction Section / Inter Prediction Section / Prediction Control Section)]

[0702] Figure 78 is a flowchart indicating an example of the processing performed by the prediction section of the decoding apparatus 200. Further, as an example, the prediction section is constituted by all or a part of the elements of the intra prediction section 216, the inter prediction section 218, and the prediction control section 220. The prediction processing section, for example, includes the intra prediction section 216 and the inter prediction section 218.

[0703] ​The prediction unit generates a prediction image of the current block (step Sq_1). The prediction image is also referred to as a prediction signal or a prediction block. In addition, in the prediction signal, for example, there is an intra prediction signal or an inter prediction signal. Specifically, the prediction unit generates the prediction image of the current block using a reconstructed image that has been obtained by performing generation of a prediction image for another block, restoration of a prediction residual, and addition of the prediction images. The prediction unit of the decoding apparatus 200 generates the same prediction image as the prediction image generated by the prediction unit of the encoding apparatus 100. That is, the generation method of the prediction image used in these prediction units is common or corresponds to each other.

[0704] The reconstructed image can be, for example, an image of a reference picture or an image of a picture that contains the decoded block (i.e., the above-described another block) in the current picture, i.e., the current picture. The decoded block in the current picture is, for example, a neighboring block of the current block.

[0705] Figure 79 is a flowchart showing another example of the processing performed by the prediction unit of the decoding apparatus 200.

[0706] The prediction unit determines a method or mode for generating a prediction image (step Sr_1). For example, the method or mode can be determined based on, for example, a prediction parameter or the like.

[0707] In a case where it is determined that the first method is the mode for generating a prediction image, the prediction unit generates a prediction image in accordance with the first method (step Sr_2a). In addition, in a case where it is determined that the second method is the mode for generating a prediction image, the prediction unit generates a prediction image in accordance with the second method (step Sr_2b). In addition, in a case where it is determined that the third method is the mode for generating a prediction image, the prediction unit generates a prediction image in accordance with the third method (step Sr_2c).

[0708] The first method, the second method, and the third method are mutually different methods for generating a prediction image, and can be, for example, an inter prediction method, an intra prediction method, and another prediction method. In such a prediction method, the above-described reconstructed image can also be used.

[0709] Figure 80A and Figure 80B is a flowchart showing another example of the processing performed in the prediction unit of the decoding apparatus 200.

[0710] As an example, the prediction unit can also perform the prediction processing in accordance with the flow shown in Figure 80A and Figure 80B In addition, Figure 80A and Figure 80BThe intra block copy shown is a mode belonging to inter prediction, in which a block included in the current picture is referenced as a reference image or reference block. That is, in the intra block copy, a picture different from the current picture is not referenced. Figure 80A The PCM mode shown is a mode belonging to intra-frame prediction, and is a mode in which neither transformation nor quantization is performed.

[0711] [Intra-frame prediction unit]

[0712] The intra prediction unit 216 performs intra prediction based on the intra prediction mode read from the stream, referring to blocks in the current picture stored in the block memory 210, thereby generating a predicted image (i.e., an intra predicted image) for the current block. Specifically, the intra prediction unit 216 performs intra prediction by referring to pixel values ​​(e.g., luminance values ​​and chrominance values) of blocks adjacent to the current block, thereby generating an intra predicted image and outputting the intra predicted image to the prediction control unit 220.

[0713] Furthermore, when an intra prediction mode that refers to a luminance block is selected in intra prediction of a chrominance block, the intra prediction unit 216 may predict the chrominance component of the current block based on the luminance component of the current block.

[0714] Furthermore, when the information read from the stream indicates application of PDPC, the intra prediction unit 216 corrects the pixel value after intra prediction based on the gradient of the reference pixel in the horizontal and vertical directions.

[0715] Figure 81 This is a diagram showing an example of processing performed by the intra prediction unit 216 of the decoding device 200 .

[0716] The intra-frame prediction unit 216 first determines whether an MPM flag indicating 1 exists in the stream (step Sw_11). Here, when it is determined that the MPM flag indicating 1 exists (yes in step Sw_11), the intra-frame prediction unit 216 obtains information indicating the intra-frame prediction mode selected in the encoding device 100 in the MPM from the entropy decoding unit 202 (step Sw_12). In addition, this information is decoded by the entropy decoding unit 202 and output to the intra-frame prediction unit 216. Next, the intra-frame prediction unit 216 determines the MPM (step Sw_13). The MPM is composed of, for example, 6 intra-frame prediction modes. Then, the intra-frame prediction unit 216 determines the intra-frame prediction mode indicated by the information obtained in step Sw_12 from the multiple intra-frame prediction modes included in the MPM (step Sw_14).

[0717] On the other hand, when it is determined in step Sw_11 that the MPM flag indicating 1 is not present in the stream (NO in step Sw_11), the intra prediction section 216 acquires information indicating the intra prediction mode selected in the encoding apparatus 100 (step Sw_15). That is, the intra prediction section 216 acquires information indicating the intra prediction mode selected in the encoding apparatus 100 from among one or more intra prediction modes not included in the MPM from the entropy decoding section 202. Further, this information is decoded by the entropy decoding section 202 and output to the intra prediction section 216. Then, the intra prediction section 216 decides the intra prediction mode indicated by the information acquired in step Sw_15 from among one or more intra prediction modes not included in the MPM (step Sw_17).

[0718] The intra prediction section 216 generates a prediction image in accordance with the intra prediction mode decided in step Sw_14 or step Sw_17 (step Sw_18).

[0719] [Inter prediction section]

[0720] The inter prediction section 218 predicts a current block with reference to a reference picture stored in the frame memory 214. Prediction is performed in units of a current block or a sub-block within the current block. In addition, a sub-block is included in a block and is a unit smaller than a block. The size of a sub-block can be 4x4 pixels, 8x8 pixels, or a size other than these. The size of a sub-block can also be switched in units of a slice, a tile, or a picture.

[0721] For example, the inter prediction section 218 performs motion compensation using motion information (e.g., an MV) read from a stream (e.g., a prediction parameter output from the entropy decoding section 202), thereby generating an inter prediction image of a current block or a sub-block, and outputs the inter prediction image to the prediction control section 220.

[0722] In a case where information read from a stream indicates that the OBMC mode is applied, the inter prediction section 218 generates an inter prediction image using not only motion information of a current block obtained by motion search but also motion information of a neighboring block.

[0723] Further, in a case where information read from a stream indicates that the FRUC mode is applied, the inter prediction section 218 performs motion search in accordance with a method (bi-directional matching or template matching) matching a pattern read from a stream, thereby deriving motion information. Also, the inter prediction section 218 performs motion compensation (prediction) using the derived motion information.

[0724] Further, the inter prediction section 218 derives the MV based on a model assuming constant velocity straight line motion in a case where the BIO mode is applied. Further, in a case where information read out from the stream indicates that the affine mode is applied, the inter prediction section 218 derives the MV in a sub-block unit based on MVs of a plurality of neighboring blocks.

[0725] [Flow of MV derivation]

[0726] Figure 82 is a flowchart showing an example of MV derivation in the decoding apparatus 200.

[0727] The inter prediction section 218 determines, for example, whether or not to decode motion information (e.g., MV). For example, the inter prediction section 218 can determine based on a prediction mode included in the stream, or can determine based on other information included in the stream. Here, when it is determined to decode the motion information, the inter prediction section 218 derives the MV of the current block in a mode in which the motion information is decoded. On the other hand, when it is determined not to decode the motion information, the inter prediction section 218 derives the MV in a mode in which the motion information is not decoded.

[0728] Here, the modes of MV derivation are a normal inter mode, a normal merge mode, an FRUC mode, an affine mode, and the like, which will be described later. Among these modes, the modes in which the motion information is decoded are the normal inter mode, the normal merge mode, and the affine mode (specifically, the affine inter mode and the affine merge mode), and the like. Further, the motion information can include not only the MV but also prediction MV selection information, which will be described later. Further, the modes in which the motion information is not decoded are the FRUC mode, and the like. The inter prediction section 218 selects the mode for deriving the MV of the current block from among these plurality of modes, and derives the MV of the current block using the selected mode.

[0729] Figure 83 is a flowchart showing another example of MV derivation in the decoding apparatus 200.

[0730] The inter prediction section 218 determines, for example, whether or not to decode the difference MV, for example, the inter prediction section 218 can determine based on a prediction mode included in the stream, or can determine based on other information included in the stream. Here, when it is determined to decode the difference MV, the inter prediction section 218 can derive the MV of the current block in a mode in which the difference MV is decoded. In this case, for example, the difference MV included in the stream is decoded as a prediction parameter.

[0731] On the other hand, when it is determined not to decode the difference MV, the inter prediction section 218 derives the MV in a mode in which the difference MV is not decoded. In this case, the encoded difference MV is not included in the stream.

[0732] As described above, the MV derivation modes include the normal inter mode, normal merge mode, FRUC mode, and affine mode, which will be described later. Among these modes, the normal inter mode and affine mode (specifically, affine inter mode) encode the differential MV. Furthermore, the FRUC mode, normal merge mode, and affine mode (specifically, affine merge mode) do not encode the differential MV. The inter prediction unit 218 selects a mode from these multiple modes for deriving the MV of the current block and uses the selected mode to derive the MV of the current block.

[0733] [MV Export > Normal Interframe Mode]

[0734] For example, when the information decoded from the stream indicates that the normal inter mode is applied, the inter prediction unit 218 derives an MV in the normal merge mode based on the information decoded from the stream, and performs motion compensation (prediction) using the MV.

[0735] Figure 84 This is a flowchart showing an example of inter prediction performed in the normal inter mode in the decoding apparatus 200 .

[0736] The inter-frame prediction unit 218 of the decoding device 200 performs motion compensation on each block. To do this, the inter-frame prediction unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks temporally or spatially surrounding the current block (step Sg_11). In other words, the inter-frame prediction unit 218 creates a candidate MV list.

[0737] Next, the inter-frame prediction unit 218 extracts N (N is an integer greater than or equal to 2) candidate MVs from the plurality of candidate MVs obtained in step Sg_11 as motion vector predictor candidates (also referred to as predicted MV candidates) in a predetermined order of priority (step Sg_12). Alternatively, the order of priority may be predetermined for each of the N predicted MV candidates.

[0738] Next, the inter prediction unit 218 decodes the predicted MV selection information from the input stream, and uses the decoded predicted MV selection information to select one predicted MV candidate from the N predicted MV candidates as the predicted MV of the current block (step Sg_13).

[0739] Next, the inter prediction unit 218 decodes the difference MV from the input stream, and derives the MV of the current block by adding the difference value of the decoded difference MV to the selected prediction MV (step Sg_14).

[0740] Finally, the inter-frame prediction unit 218 generates a predicted image for the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Sg_15). The processes of steps Sg_11 to Sg_15 are performed for each block. For example, when the processes of steps Sg_11 to Sg_15 are performed for all blocks included in a slice, inter-frame prediction using the normal inter-frame mode for that slice is completed. Alternatively, when the processes of steps Sg_11 to Sg_15 are performed for all blocks included in a picture, inter-frame prediction using the normal inter-frame mode for that picture is completed. Furthermore, the processes of steps Sg_11 to Sg_15 may be performed not for all blocks included in a slice, but for a portion of the blocks, in which case inter-frame prediction using the normal inter-frame mode for that slice is completed. Similarly, when the processes of steps Sg_11 to Sg_15 are performed for a portion of the blocks included in a picture, inter-frame prediction using the normal inter-frame mode for that picture is completed.

[0741] [MV Export > Normal Merge Mode]

[0742] For example, when the information read from the stream indicates application of the normal merge mode, the inter prediction unit 218 derives an MV in the normal merge mode and performs motion compensation (prediction) using the MV.

[0743] Figure 85 This is a flowchart illustrating an example of inter-frame prediction based on the normal merge mode in the decoding apparatus 200 .

[0744] The inter-frame prediction unit 218 first obtains multiple candidate MVs for the current block based on information such as MVs of multiple decoded blocks temporally or spatially surrounding the current block (step Sh_11). In other words, the inter-frame prediction unit 218 creates a candidate MV list.

[0745] Next, the inter-frame prediction unit 218 selects one candidate MV from the multiple candidate MVs obtained in step Sh_11 to derive the MV of the current block (step Sh_12). Specifically, the inter-frame prediction unit 218 obtains MV selection information included as a prediction parameter in the stream, for example, and selects the candidate MV identified by the MV selection information as the MV of the current block.

[0746] Finally, the inter prediction section 218 generates a prediction image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Sh_13). The processes of steps Sh_11 to Sh_13 are performed, for example, for each block. For example, when the processes of steps Sh_11 to Sh_13 are performed for all blocks included in a slice respectively, the inter prediction using the normal merge mode for the slice ends. Also, when the processes of steps Sh_11 to Sh_13 are performed for all blocks included in a picture respectively, the inter prediction using the normal merge mode for the picture ends. Further, the processes of steps Sh_11 to Sh_13 can also be that, when the processes are performed for a part of blocks included in a slice, the inter prediction using the normal merge mode for the slice ends. The processes of steps Sh_11 to Sh_13 can also be that, likewise, when the processes are performed for a part of blocks included in a picture, the inter prediction using the normal merge mode for the picture ends.

[0747] [MV derivation > FRUC mode]

[0748] For example, in a case where the information read out from the stream indicates the application of the FRUC mode, the inter prediction section 218 derives an MV in the FRUC mode, and performs motion compensation (prediction) using the MV. In this case, the motion information is not signaled from the encoding apparatus 100 side, but is derived at the decoding apparatus 200 side. For example, the decoding apparatus 200 can also derive the motion information by performing a motion search. In this case, the decoding apparatus 200 does not use the pixel values of the current block to perform the motion search.

[0749] Figure 86 is a flowchart indicating an example of the inter prediction based on the FRUC mode in the decoding apparatus 200.

[0750] First, the inter-frame prediction unit 218 refers to the MVs of each decoded block that is spatially or temporally adjacent to the current block and generates a list representing these MVs as candidate MVs (that is, a candidate MV list, which can also be shared with the candidate MV list of the normal merge mode) (step Si_11). Next, the inter-frame prediction unit 218 selects the best candidate MV from the multiple candidate MVs registered in the candidate MV list (step Si_12). For example, the inter-frame prediction unit 218 calculates an evaluation value for each candidate MV included in the candidate MV list and selects one candidate MV as the best candidate MV based on the evaluation value. Then, the inter-frame prediction unit 218 derives the MV for the current block based on the selected best candidate MV (step Si_14). Specifically, for example, the selected best candidate MV is directly derived as the MV for the current block. Alternatively, for example, the MV for the current block can be derived by performing pattern matching in the surrounding area of ​​the position in the reference picture corresponding to the selected best candidate MV. Specifically, a search is performed using pattern matching and evaluation values ​​in the reference image for the area surrounding the best candidate MV. If an MV with a good evaluation value exists, the best candidate MV may be updated to that MV and set as the final MV for the current block. Updating to an MV with a better evaluation value may not be performed.

[0751] Finally, the inter-frame prediction unit 218 performs motion compensation on the current block using the derived MV and the decoded reference picture to generate a predicted image for the current block (step Si_15). The processing of steps Si_11 to Si_15 is performed on each block, for example. For example, when the processing of steps Si_11 to Si_15 is performed on all blocks included in a slice, the inter-frame prediction using the FRUC mode for the slice is completed. In addition, when the processing of steps Si_11 to Si_15 is performed on all blocks included in a picture, the inter-frame prediction using the FRUC mode for the picture is completed. The processing can also be performed in sub-block units in the same way as the above-mentioned block unit.

[0752] [MV Export > Affine Merge Mode]

[0753] For example, when the information read from the stream indicates application of the affine merge mode, the inter prediction unit 218 derives an MV in the affine merge mode and performs motion compensation (prediction) using the MV.

[0754] Figure 87 This is a flowchart illustrating an example of inter-frame prediction based on the affine merge mode in the decoding device 200 .

[0755] In the affine merge mode, the inter prediction unit 218 first derives the MV of each control point of the current block (step Sk_11). Figure 46AAs shown, the control points are the top left and top right corners of the current block, or as Figure 46B As shown, these are the points at the upper left corner, upper right corner, and lower left corner of the current block.

[0756] For example, when using Figures 47A to 47C In the case of the MV derivation method shown, as Figure 47A As shown, the inter-frame prediction unit 218 checks the decoded blocks A (left), B (top), C (top right), D (bottom left) and E (top left) in this order to determine the first valid block decoded in affine mode.

[0757] The inter-frame prediction unit 218 uses the first valid block decoded in the determined affine mode to derive the MV of the control point. For example, if block A is determined and block A has two control points, Figure 47B As shown, the inter-frame prediction unit 218 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block by projecting the motion vectors v3 and v4 of the upper left corner and upper right corner of the decoded block including block A onto the current block. Thus, the MV of each control point is derived.

[0758] In addition, if Figure 49A As shown, if block A is determined and block A has two control points, the MV of three control points can also be calculated, or as shown in Figure 49B As shown, block A is determined, and when block A has three control points, MVs of two control points are calculated.

[0759] Furthermore, when the stream includes MV selection information as a prediction parameter, the inter prediction unit 218 may use the MV selection information to derive the MV of each control point of the current block.

[0760] Next, the inter-frame prediction unit 218 performs motion compensation on each of the multiple sub-blocks included in the current block. That is, for each of the multiple sub-blocks, the inter-frame prediction unit 218 uses two motion vectors v0 and v1 and the above-mentioned equation (1A), or uses three motion vectors v0, v1, and v2 and the above-mentioned equation (1B) to calculate the MV of the sub-block as an affine MV (step Sk_12). Then, the inter-frame prediction unit 218 uses these affine MVs and the decoded reference picture to perform motion compensation on the sub-block (step Sk_13). When the processing of steps Sk_12 and Sk_13 is performed on all sub-blocks included in the current block, the inter-frame prediction using the affine merge mode for the current block is completed. That is, motion compensation is performed on the current block, and a predicted image for the current block is generated.

[0761] In addition, in step Sk_11, the candidate MV list mentioned above may also be generated. The candidate MV list may also be a list containing candidate MVs derived using multiple MV derivation methods for each control point. Multiple MV derivation methods may be Figures 47A to 47C The MV derivation method shown, Figure 48A and Figure 48B The MV derivation method shown, Figure 49A and Figure 49B The MV derivation method shown above and any combination of other MV derivation methods can be used.

[0762] Furthermore, the candidate MV list may include candidate MVs of modes other than the affine mode that perform prediction in sub-block units.

[0763] Furthermore, as a candidate MV list, for example, a candidate MV list including candidate MVs of an affine merge pattern with two control points and a candidate MV list including candidate MVs of an affine merge pattern with three control points may be generated. Alternatively, a candidate MV list including candidate MVs of an affine merge pattern with two control points and a candidate MV list including candidate MVs of an affine merge pattern with three control points may be generated separately. Alternatively, a candidate MV list including candidate MVs of either an affine merge pattern with two control points or an affine merge pattern with three control points may be generated.

[0764] [MV Export > Affine Inter Mode]

[0765] For example, when the information read from the stream indicates application of the affine inter mode, the inter prediction unit 218 derives an MV in the affine inter mode and performs motion compensation (prediction) using the MV.

[0766] Figure 88 This is a flowchart illustrating an example of inter prediction based on the affine inter mode in the decoding apparatus 200 .

[0767] In the affine inter mode, first, the inter prediction unit 218 derives the predicted MV (v0, v1) or (v0, v1, v2) of each of the two or three control points of the current block (step Sj_11). Figure 46A or Figure 46B As shown, it is the point at the upper left corner, upper right corner or lower left corner of the current block.

[0768] The inter-frame prediction unit 218 obtains the prediction MV selection information included in the stream as a prediction parameter, and uses the MV identified by the prediction MV selection information to derive the prediction MV of each control point of the current block. Figure 48A and Figure 48B In the case of the MV derivation method shown in FIG. 1 , the inter-frame prediction unit 218 selects Figure 48Aor Figure 48B The predicted MV (v0, v1) or (v0, v1, v2) of the control point of the current block is derived from the MV of the block identified by the predicted MV selection information in the decoded blocks near each control point of the current block.

[0769] Next, the inter-frame prediction unit 218 obtains, for example, each differential MV included as a prediction parameter in the stream, and adds the predicted MV of each control point of the current block to the differential MV corresponding to the predicted MV (step Sj_12). Thus, the MV of each control point of the current block is derived.

[0770] Next, the inter-frame prediction unit 218 performs motion compensation on each of the multiple sub-blocks included in the current block. That is, for each of the multiple sub-blocks, the inter-frame prediction unit 218 uses two motion vectors v0 and v1 and the above-mentioned equation (1A), or uses three motion vectors v0, v1, and v2 and the above-mentioned equation (1B) to calculate the MV of the sub-block as an affine MV (step Sj_13). Then, the inter-frame prediction unit 218 uses these affine MVs and the decoded reference picture to perform motion compensation on the sub-block (step Sj_14). When the processing of steps Sj_13 and Sj_14 is performed on all sub-blocks included in the current block, the inter-frame prediction using the affine merge mode for the current block is completed. That is, motion compensation is performed on the current block, and a predicted image for the current block is generated.

[0771] Furthermore, in step Sj_11, the candidate MV list described above may be generated in the same manner as in step Sk_11.

[0772] [MV Export > Triangle Mode]

[0773] For example, when the information read from the stream indicates application of the triangular mode, the inter prediction unit 218 derives an MV in the triangular mode and performs motion compensation (prediction) using the MV.

[0774] Figure 89 This is a flowchart illustrating an example of inter-frame prediction based on the triangular mode in the decoding apparatus 200 .

[0775] In triangular mode, the inter-frame prediction unit 218 first divides the current block into a first partition and a second partition (step Sx_11). At this time, the inter-frame prediction unit 218 can obtain partition information related to the division into each partition from the stream as a prediction parameter. Furthermore, the inter-frame prediction unit 218 can divide the current block into the first partition and the second partition based on the partition information.

[0776] Next, the inter-frame prediction unit 218 first obtains multiple candidate MVs for the current block based on information such as MVs of multiple decoded blocks located temporally or spatially around the current block (step Sx_12). In other words, the inter-frame prediction unit 218 creates a candidate MV list.

[0777] The inter-frame prediction unit 218 then selects the candidate MV for the first partition and the candidate MV for the second partition as the first and second MVs, respectively, from the multiple candidate MVs obtained in step Sx_11 (step Sx_13). At this time, the inter-frame prediction unit 218 may also obtain MV selection information from the stream as a prediction parameter to identify the selected candidate MVs. The inter-frame prediction unit 218 then selects the first and second MVs based on the MV selection information.

[0778] Next, the inter-frame prediction unit 218 performs motion compensation using the selected first MV and the decoded reference picture to generate a first predicted image (step Sx_14). Similarly, the inter-frame prediction unit 218 performs motion compensation using the selected second MV and the decoded reference picture to generate a second predicted image (step Sx_15).

[0779] Finally, the inter prediction unit 218 generates a prediction image of the current block by performing weighted addition on the first prediction image and the second prediction image (step Sx_16).

[0780] [Sports Search>DMVR]

[0781] For example, when the information read from the stream indicates application of DMVR, the inter prediction unit 218 performs motion search using DMVR.

[0782] Figure 90 3 is a flowchart showing an example of a motion search based on DMVR in the decoding apparatus 200 .

[0783] The inter-frame prediction unit 218 first derives the MV of the current block in merge mode (step S1_11). Next, the inter-frame prediction unit 218 searches the surrounding area of ​​the reference picture represented by the MV derived in step S1_11 to derive the final MV for the current block (step S1_12). In other words, the MV of the current block is determined using DMVR.

[0784] Figure 91 This is a flowchart showing a detailed example of the motion search based on DMVR in the decoding device 200.

[0785] First, the inter-frame prediction unit 218 Figure 58AIn Step 1, the inter prediction section 218 calculates the cost of the search position indicated by the initial MV (also referred to as a starting point) and eight search positions located around it. Then, the inter prediction section 218 determines whether the cost of the search position other than the starting point is the minimum. In this case, when it is determined that the cost of the search position other than the starting point is the minimum, the inter prediction section 218 moves to the search position of which the cost is the minimum, and performs the processing of Step 2. Figure 58A On the other hand, if the cost of the starting point is the minimum, the inter prediction section 218 skips the processing of Step 2 and performs the processing of Step 3. Figure 58A

[0786] In Step 2, the inter prediction section 218 performs the same search as the processing of Step 1, using the search position moved according to the result of the processing of Step 1 as a new starting point. Then, the inter prediction section 218 determines whether the cost of the search position other than the starting point is the minimum. In this case, if the cost of the search position other than the starting point is the minimum, the inter prediction section 218 performs the processing of Step 4. On the other hand, if the cost of the starting point is the minimum, the inter prediction section 218 performs the processing of Step 3. Figure 58A

[0787] In Step 4, the inter prediction section 218 processes the search position of the starting point as a final search position, and determines the difference between the position indicated by the initial MV and the final search position as a difference vector.

[0788] In Step 3, the inter prediction section 218 determines a decimal-precision pixel position of which the cost is the minimum, based on the costs of four points above, below, left, and right of the starting point in Step 1 or Step 2, and processes the pixel position as a final search position. The decimal-precision pixel position is determined by performing weighted addition of vectors ((0, 1), (0, -1), (-1, 0), (1, 0)) located above, below, left, and right of the four points, using the costs of the search positions of the four points as weights. Then, the inter prediction section 218 determines the difference between the position indicated by the initial MV and the final search position as a difference vector. Figure 58A [Motion compensation > BIO / OBMC / LIC]

[0789] For example, in a case where information read out from the stream indicates an application of a modification of a prediction image, the inter prediction section 218 modifies the prediction image in accordance with the modification mode when generating the prediction image. The mode is, for example, BIO, OBMC, and LIC described above.

[0790]

[0791] Figure 92 is a flowchart indicating an example of generation of a prediction image in the decoding apparatus 200. is a flowchart indicating an example of generation of a prediction image in the decoding apparatus 200.​​

[0792] The inter prediction section 218 generates a prediction image (step Sm_11), and corrects the prediction image by any of the above-described modes (step Sm_12).

[0793] Figure 93 is a flowchart showing another example of generation of a prediction image in the decoding apparatus 200.

[0794] The inter prediction section 218 derives the MV of the current block (step Sn_11). Next, the inter prediction section 218 generates a prediction image using the MV (step Sn_12), and determines whether or not to perform the correction process (step Sn_13). For example, the inter prediction section 218 acquires a prediction parameter included in the stream, and determines whether or not to perform the correction process based on the prediction parameter. The prediction parameter is, for example, a flag indicating whether or not to apply each of the above-described modes. Here, when it is determined to perform the correction process (Yes in step Sn_13), the inter prediction section 218 generates a final prediction image by correcting the prediction image (step Sn_14). Further, in LIC, the luminance and color difference of the prediction image can be corrected in step Sn_14. On the other hand, when it is determined not to perform the correction process (No in step Sn_13), the inter prediction section 218 outputs the prediction image as it is as a final prediction image without correcting it (step Sn_15).

[0795] [OBMC]

[0796] For example, in a case where the information read out from the stream indicates the application of OBMC, the inter prediction section 218 corrects the prediction image in accordance with OBMC when generating the prediction image.

[0797] Figure 94 is a flowchart showing an example of correction of a prediction image based on OBMC in the decoding apparatus 200. In addition, Figure 94 The flowchart of Figure 62 shows a flow of correction of a prediction image using the current picture and the reference picture as shown in

[0798] First, as shown in Figure 62 , the inter prediction section 218 acquires a prediction image (Pred) based on normal motion compensation using the MV allocated to the current block.

[0799] Next, the inter prediction section 218 applies (reuses) the MV (MV_L) already derived with respect to the decoded left neighboring block to the current block, and acquires a prediction image (Pred_L). Then, the inter prediction section 218 performs the 1st correction of the prediction image by superimposing the two prediction images Pred and Pred_L. This has an effect of mixing the boundaries between the neighboring blocks.

[0800] Likewise, the inter prediction section 218 applies (reuses) the MV (MV_U) that has been derived for the decoded upper neighboring block to the current block, to obtain a prediction image (Pred_U). Then, the inter prediction section 218 performs a second correction of the prediction image by superimposing the prediction image Pred_U and the prediction image that has been subjected to the first correction (e.g., Pred and Pred_L). This has the effect of blending the boundaries between the neighboring blocks. The prediction image obtained by the second correction is the final prediction image of the current block that is blended (smoothed) with the boundaries of the neighboring blocks.

[0801] [Motion compensation > BIO]

[0802] For example, in a case where the information read out from the stream indicates the application of BIO, the inter prediction section 218 corrects the prediction image in accordance with BIO when generating the prediction image.

[0803] Figure 95 is a flowchart that shows an example of the correction of the prediction image based on BIO in the decoding apparatus 200.

[0804] As shown in Figure 63 , the inter prediction section 218 derives two motion vectors (M0, M1) using two reference pictures (Ref0, Ref1) that are different from the picture (CurPic) that contains the current block. Then, the inter prediction section 218 derives the prediction image of the current block using the two motion vectors (M0, M1) (step Sy_11). Note that the motion vector M0 is a motion vector (MVx0, MVy0) corresponding to the reference picture Ref0, and the motion vector M1 is a motion vector (MVx1, MVy1) corresponding to the reference picture Ref1.

[0805] Next, the inter prediction section 218 derives an interpolated image I 0 for the current block using the motion vector M0 and the reference picture L0 (step Sy_12). Here, the interpolated image I 1 is an image contained in the reference picture Ref0 that is derived for the current block. The interpolated image I 0 is an image contained in the reference picture Ref1 that is derived for the current block. The interpolated image I 1 and the interpolated image I 0 may each be an image of the same size as the current block. Alternatively, in order to appropriately derive a gradient image to be described later, the interpolated image I 1 and the interpolated image I 0 may each be an image larger than the current block. Furthermore, the interpolated image I 1 and I 0 may each be an image larger than the current block. Furthermore, the interpolated image I 1It may include a predicted image derived by applying a motion vector (M0, M1), a reference picture (L0, L1), and a motion compensation filter.

[0806] In addition, the inter-frame prediction unit 218 uses the interpolation image I 0 and interpolated image I 1 Export the gradient image of the current block (Ix 0 , 1x 1 , Iy 0 , Iy 1 )(Step Sy_13). In addition, the gradient image in the horizontal direction is (Ix 0 , 1x 1 ), the vertical gradient image is (Ix 0 , 1x 1 The inter-frame prediction unit 218 may also derive the gradient image by, for example, applying a gradient filter to the interpolated image. The gradient image may be any image that represents the spatial variation of pixel values ​​in the horizontal or vertical direction.

[0807] Next, the inter-frame prediction unit 218 uses the interpolation image (I 0 , I 1 ) and gradient image (Ix 0 , 1x 1 , Iy 0 , Iy 1 ) derives the optical flow (vx, vy) as the velocity vector (step Sy_14). As an example, the sub-block may be a sub-CU of 4x4 pixels.

[0808] Next, the inter-frame prediction unit 218 uses the optical flow (vx, vy) to correct the predicted image of the current block. For example, the inter-frame prediction unit 218 uses the optical flow (vx, vy) to derive correction values ​​for the values ​​of the pixels included in the current block (step Sy_15). The inter-frame prediction unit 218 then uses the correction values ​​to correct the predicted image of the current block (step Sy_16). The correction values ​​may be derived for each pixel, for multiple pixels, or for sub-blocks.

[0809] In addition, the BIO process is not limited to Figure 95 The disclosed processing can be implemented only Figure 95 A part of the disclosed processing may be added to or replaced with a different processing, or may be executed in a different processing order.

[0810] [Motion Compensation > LIC]

[0811] For example, when the information read from the stream indicates application of LIC, the inter prediction unit 218 modifies the predicted image in accordance with LIC when generating the predicted image.

[0812] Figure 96 This is a flowchart showing an example of modification of a predicted image based on LIC in the decoding apparatus 200 .

[0813] First, the inter prediction unit 218 obtains a reference image corresponding to the current block from a decoded reference picture using MV (step Sz_11 ).

[0814] Next, the inter-frame prediction unit 218 extracts information indicating how the luminance value of the current block changes between the reference picture and the current picture (step Sz_12). Figure 66A As shown, this extraction is performed based on the luminance pixel values ​​of the decoded left-adjacent reference region (peripheral reference region) and the decoded upper-adjacent reference region (peripheral reference region) in the current picture, and the luminance pixel values ​​at the same position in the reference picture specified by the derived MV. The inter-frame prediction unit 218 then calculates the luminance correction parameter using information indicating how the luminance value changes (step Sz_13).

[0815] The inter-frame prediction unit 218 generates a predicted image for the current block by applying the brightness correction parameters to the reference image within the reference picture specified by the MV (step Sz_14). Specifically, the predicted image, which is the reference image within the reference picture specified by the MV, is corrected based on the brightness correction parameters. This correction can be performed to correct brightness or color difference.

[0816] [Prediction Control Department]

[0817] The prediction control unit 220 selects either the intra-frame prediction image or the inter-frame prediction image, and outputs the selected prediction image to the addition unit 208. Generally speaking, the structure, function, and processing of the prediction control unit 220, the intra-frame prediction unit 216, and the inter-frame prediction unit 218 on the decoding device 200 side may correspond to the structure, function, and processing of the prediction control unit 128, the intra-frame prediction unit 124, and the inter-frame prediction unit 126 on the encoding device 100 side.

[0818] [First Form]

[0819] Hereinafter, an encoding device 100 and a decoding device 200 according to a first embodiment of the present invention will be described.

[0820] The encoding apparatus 100 determines whether the processing target image is a progressive image (hereinafter, also referred to as a progressive content), and in a case where it is determined that the processing target image is a progressive image, encodes one syntax element indicating a color difference positioning type, which is information indicating positions of color difference samples with respect to luminance samples for a frame included in the processing target image, into a bitstream. In a case where it is determined that the processing target image is not a progressive image, encodes two syntax elements respectively indicating color difference positioning types for two fields included in the processing target image into the bitstream.

[0821] Further, the decoding apparatus 200 determines whether the processing target image is a progressive image, and in a case where it is determined that the processing target image is a progressive image, decodes one syntax element indicating a color difference positioning type, which is information indicating positions of color difference samples with respect to luminance samples for a frame included in the processing target image, from a bitstream. In a case where it is determined that the processing target image is not a progressive image, decodes two syntax elements respectively indicating color difference positioning types for two fields included in the processing target image from the bitstream.

[0822] The encoding apparatus 100 and the decoding apparatus 200 determine whether the processing target image is a progressive image or an interlaced image using, for example, a first flag and a second flag included in a video usability information (hereinafter, also referred to as VUI) syntax structure, the first flag indicating whether the processing target image is a progressive image, and the second flag indicating whether the processing target image is an interlaced image (hereinafter, also referred to as an interlaced content).

[0823] Hereinafter, the internal structure of the VUI in the encoding apparatus 100 will be described with reference to the drawings.

[0824] [Internal structure of VUI syntax in encoding apparatus]

[0825] Figure 97is a diagram showing an example of a VUI syntax structure. Information included in the VUI syntax structure is not necessary for decoding luminance samples and chroma samples of a video signal. However, such information is indispensable for properly displaying the video signal on a display. Moreover, in today's world, video is either HDR (High Dynamic Range) or SDR (Standard Dynamic Range), can use a color gamut of SD (Standard Definition), HD (High Definition), or UHD (Ultra High Definition), and can use chroma sample positions (hereinafter also referred to as chroma positioning) of ITU-R Recommendation (hereinafter referred to as Recommendation) BT.709 (Broadcasting service (television) 709) or BT.2020 (Broadcasting service (television) 2020) with respect to positions of luminance samples, so such information is even more indispensable. In particular, in Recommendations BT.2020 and BT.2100, chroma sample positions of Type 2 have been used instead of chroma sample positions of Type 0 used in previous forms, so it is indispensable to transmit information about chroma sample positions for progressive content. If this information is transmitted even for progressive content using two syntax elements chroma_sample_loc_type_top_field and chroma_sample_loc_type_bottom_field each representing 2 fields, it is unclear how a display reacts to different values for the 2 fields (e.g., top field and bottom field) for progressive content.

[0826] Therefore, the encoding apparatus 100 of the first aspect of the present disclosure determines whether the processing target image is progressive content, and in a case where it is determined that the processing target image is progressive content, encodes one syntax element chroma_sample_loc_type_frame indicating a chroma positioning type that is information indicating positions of chroma samples with respect to luminance samples for frames included in the processing target image into a bitstream.

[0827] For example, in the first aspect of the present disclosure, as Figure 97As shown in the syntax of , two flags, general_progressive_source_flag and general_interlaced_source_flag, are used to determine whether the image being processed is progressive content. In addition, general_progressive_source_flag is an example of a first flag indicating whether the image being processed is progressive content, and general_interlaced_source_flag is an example of a second flag indicating whether the image being processed is interlaced content.

[0828] In the first aspect of the present disclosure, when it is determined that the processing target image is not an interlaced scan image, two syntax elements chroma_sample_loc_type_top_field and chroma_sample_loc_type_bottom_field respectively indicating the color difference positioning type for two types of fields included in the processing target image are encoded into the bitstream.

[0829] In addition, if Figure 97 As shown, the encoding device 100 may also encode a third flag (e.g., chroma_loc_info_present_flag) into the bitstream, the third flag indicating whether the bitstream includes information indicating the positions of chroma samples relative to luma samples for frames included in the current image as one or two of the above-mentioned syntax elements. In this case, for example, when the third flag indicates that the bitstream includes information indicating the positions of chroma samples relative to luma samples for frames included in the current image, and when the first and second flags are used to determine that the current image is a progressive scan image, the encoding device 100 may encode one syntax element into the bitstream. Furthermore, for example, when the third flag indicates that the bitstream includes information indicating the positions of chroma samples relative to luma samples for frames, and when the first and second flags are used to determine that the current image is not progressive scan content, the encoding device 100 may encode two syntax elements into the bitstream.

[0830] Alternatively, the first and second flags may be represented within the general_constraint_info syntax structure. Furthermore, since these two flags convey information about the coded video signal rather than the bitstream (unlike other flags within the general_constraint_info syntax structure), they may be moved to the VUI syntax structure instead of the general_constraint_info syntax structure and repeated (i.e., duplicated) within the VUI syntax.

[0831] In addition, the internal structure of the VUI syntax described above is common to the encoding apparatus 100 and the decoding apparatus 200.

[0832] [Action example of decoding apparatus]

[0833] Next, the action of the decoding apparatus 200 when receiving a processing target image will be described with the case where a video signal having a 4:2:0 (YUV) color difference format is received as an example.

[0834] Figure 98 is a flowchart showing an example of the action of the decoding apparatus 200 in the first mode. Hereinafter, the information shown in the VUI syntax structure is referred to as VUI information.

[0835] First, when the decoding apparatus 200 receives a video signal including 4:2:0 (YUV) color difference content (step S101), the VUI information is read from the bitstream (step S102).

[0836] Next, the decoding apparatus 200 checks the values of two flags, general_progressive_source_flag (so-called first flag) and general_interlaced_source_flag (so-called second flag), included in the VUI information read from the bitstream in step S102. Then, the decoding apparatus 200 determines that the content (hereinafter referred to as processing target image) included in the video signal received in step S101 is progressive content (not shown) in the case where the value of general_progressive_source_flag is equal to 1 and the value of general_interlaced_source_flag is equal to 0 (Yes in step S103).

[0837] In the case where it is determined that the processing target image is progressive content, the decoding apparatus 200 upsamples the color difference samples of the entire image (i.e., frame) using one syntax element chroma_sample_loc_type_frame indicating a color difference positioning type, which is information indicating the position of the color difference samples with respect to the luminance samples for the frame included in the processing target image (step S104). Thereby, the color difference samples corresponding to the position of the luminance samples are generated.

[0838] More specifically, in step S104, the decoding apparatus 200 decodes 1 syntax element from the bitstream. Next, the decoding apparatus 200 invokes a process for upsampling the chroma samples of the frames (also referred to as frame images) included in the processing target video using the chroma location type indicated by the decoded 1 syntax element (not shown). Next, the decoding apparatus 200 upsamples the chroma samples of the entire image for each image (so-called frame image) included in the processing target video in accordance with the invoked process (step S104). Such a process generally depends on the application or specific practice.

[0839] On the other hand, the decoding apparatus 200 checks the values of the 2 flags included in the VUI information read from the bitstream in step S102, and in the case where the general_progressive_source_flag (i.e., 1st flag) is not equal to 1 and / or the general_interlaced_source_flag (i.e., 2nd flag) is not equal to 0 (NO in step S103), determines that the content (i.e., processing target video) included in the video signal received in step S101 is interlaced content or unknown content (not shown). Further, the unknown content means content whose processing target video is unclear whether it is interlaced or progressive, content whose processing target video cannot be determined whether it is interlaced or progressive, or content whose information indicating frame / field is not encoded, and the like.

[0840] In other words, the decoding apparatus 200 (1) determines that the processing target video is progressive content in the case where the general_progressive_source_flag = 1 and the general_interlaced_source_flag = 0, (2) determines that the processing target video is interlaced content in the case where the general_progressive_source_flag = 0 and the general_interlaced_source_flag = 1, (3) determines that the processing target video is unknown content in the case where the general_progressive_source_flag = 0 and the general_interlaced_source_flag = 0, and (4) determines that the processing target video is content whose information indicating frame / field is transmitted by a method other than VUI (e.g., SEI message, etc.) in the case where the general_progressive_source_flag = 1 and the general_interlaced_source_flag = 1.

[0841] If the current image is determined to be interlaced content or unknown content, the decoding apparatus 200 uses the two syntax elements chroma_sample_loc_type_top_field and chroma_sample_loc_type_bottom_field indicating the chroma positioning type, which is information indicating the position of the chroma samples relative to the luma samples for the two fields included in the current image, to upsample the chroma samples of the two fields (steps S105 and S106). This generates chroma samples corresponding to the luma sample positions.

[0842] More specifically, when the decoding device 200 determines that the current image is interlaced or unknown content, it decodes two syntax elements (not shown) from the bitstream that indicate the chroma location type for each of the two fields. Next, the decoding device 200 invokes two processes for upsampling the chroma samples of the two fields (e.g., the top and bottom fields) contained in the current image using the chroma location types indicated by the two decoded syntax elements (not shown). The decoding device 200 then upsamples the chroma samples of the two fields in each image (so-called frame image) contained in the current image according to the two invoked processes. For example, the decoding device 200 uses the syntax element chroma_sample_loc_type_top_field, which indicates the chroma location type relative to luma samples for the top field, of the two decoded syntax elements, to upsample the chroma samples of the top field (step S105). Next, the decoding apparatus 200 upsamples the chroma samples of the bottom field using the syntax element chroma_sample_loc_type_bottom_field indicating the chroma location type relative to the luma samples for the bottom field, of the two decoded syntax elements (step S106 ).

[0843] In addition, the processing of step S105 and step S106 can also be carried out in the same manner as Figure 98 The order shown is reversed. For example, the decoding device 200 may first upsample the color difference samples of the bottom field and then upsample the color difference samples of the top field for the two fields included in the processing target image. Furthermore, if the bitstream is encoded using frames rather than fields, the decoding device 200 needs to convert the frames into fields so that the two processes described above can be performed on each of the two fields.

[0844] After the decoding device 200 repeatedly performs the processing of step S104 or steps S105 to S106 on all pictures included in the processing target video, the processing ends.

[0845] Note that this processing flow is merely an example, and the order of the described processing may be changed, part of the described processing may be removed, or undescribed processing may be added.

[0846] For example, between step S102 and step S103, the following processing may be added: a process of decoding the third flag chroma_loc_info_present_flag from the bit stream, the third flag indicating whether the bit stream contains information indicating the positions of chroma samples relative to luminance samples for frames included in the processing object image as the above-mentioned one syntactic element or the above-mentioned two syntactic elements; and a process of determining whether the decoded third flag indicates that information indicating the positions of chroma samples relative to luminance samples for frames included in the processing object image is included in the bit stream.

[0847] Note that, although the decoding device 200 is described here as an example, if “decoding” is replaced with “encoding”, the above-described processing is common to the encoding device 100 and the decoding device 200 .

[0848] [Effect of the first form]

[0849] As described above, according to the encoding device 100 of the first aspect of the present disclosure, when the processing target image is a progressive scan image, only one syntax element indicating the color difference positioning type is encoded for each frame included in the processing target image. When the processing target image is not a progressive scan image, two syntax elements indicating the color difference positioning type are encoded for each of the two types of fields included in the processing target image. This reduces the amount of code and improves encoding efficiency.

[0850] Furthermore, according to the decoding device 200 of the first aspect of the present disclosure, when the processing target video is a progressive scan video, only one syntax element indicating the color difference positioning type for each frame included in the processing target video is decoded. When the processing target video is not a progressive scan video, two syntax elements indicating the color difference positioning type for each of the two fields included in the processing target video are decoded. Consequently, the processing efficiency of the decoding device 200 is improved due to a reduced processing load.

[0851] Moreover, according to the first aspect of the present disclosure, even if the processing target image is a progressive scan content with a 4:2:0 color difference format, it is possible to achieve improved coding efficiency, improved image quality, reduced processing volume, and improved processing efficiency.

[0852] Furthermore, not all components described in the first embodiment are necessarily required, and the encoding device 100 and the decoding device 200 may include only a portion of the components of the first embodiment. Furthermore, not all of the above-described processing is necessarily required, and the encoding device 100 and the decoding device 200 may include a portion of the above-described processing.

[0853] [Representative example of treatment]

[0854] Representative examples of the processing performed by the encoding device 100 and the decoding device 200 described above are described below.

[0855] Figure 99 1 is a flowchart showing the operation of the coding device 100. The coding device 100 includes a circuit and a memory connected to the circuit. The circuit and memory included in the coding device 100 may also correspond to Figure 8 The circuitry of the coding device 100 operates as follows: a processor a1 and a memory a2.

[0856] For example, the circuit of the encoding device 100 determines whether the processing object image is an interlaced scan image. If it is determined that the processing object image is a interlaced scan image, one syntactic element representing the chroma positioning type is encoded into the bit stream. The chroma positioning type is information representing the position of the chroma sample relative to the luminance sample for the frame included in the processing object image. If it is determined that the processing object image is not an interlaced scan image, two syntactic elements representing the chroma positioning type for the two fields included in the processing object image are encoded into the bit stream.

[0857] Thus, when the target video is a progressive video, the encoding device 100 encodes only one syntax element indicating the color difference positioning type, thereby reducing the amount of processing and improving the encoding efficiency of the encoding device 100.

[0858] For example, the circuit of the encoding device 100 may use the first flag and the second flag contained in the video availability information syntax structure to determine whether the image to be processed is a progressive scan image or an interlaced scan image, wherein the first flag indicates whether the image to be processed is a progressive scan image, and the second flag indicates whether the image to be processed is an interlaced scan image.

[0859] Thus, even when the processing target video includes an image whose display method (i.e., the scanning method of the display) is not specified, the encoding device 100 can still determine whether the processing target video is a progressive scan video or an interlaced scan video. Therefore, the encoding device 100 can perform more appropriate processing on the processing target video, thereby improving encoding efficiency.

[0860] For example, it can also be that the circuit of the encoding device 100 further encodes a 3rd flag into the bitstream, the 3rd flag indicating whether information indicating positions of chroma samples with respect to luma samples for frames included in the processing target image is included in the bitstream as one syntax element or two syntax elements, in a case where the 3rd flag indicates that the information indicating the positions of the chroma samples with respect to the luma samples for the frames included in the processing target image is included in the bitstream, and it is determined using the 1st flag and the 2nd flag that the processing target image is a progressive image, one syntax element is encoded into the bitstream, in a case where the 3rd flag indicates that the information indicating the positions of the chroma samples with respect to the luma samples for the frames included in the processing target image is included in the bitstream, and it is determined using the 1st flag and the 2nd flag that the processing target image is an interlaced image, two syntax elements are encoded into the bitstream.

[0861] Thus, the encoding device 100 can switch syntax elements to be encoded depending on whether the processing target image is a progressive image or an interlaced image. Therefore, the encoding device 100 can perform more appropriate processing on the processing target image.

[0862] Figure 100 is a flowchart indicating actions performed by the decoding device 200. For example, the decoding device 200 includes a circuit and a memory connected to the circuit. The circuit and the memory included in the decoding device 200 can correspond to the processor b1 and the memory b2 illustrated in Figure 68 . The circuit of the decoding device 200 performs the following in the actions.

[0863] For example, the circuit of the decoding device 200 determines whether the processing target image is a progressive image, decodes one syntax element indicating a chroma positioning type for frames included in the processing target image in a case where it is determined that the processing target image is a progressive image, the chroma positioning type indicating information indicating positions of chroma samples with respect to luma samples, and decodes two syntax elements respectively indicating chroma positioning types for two fields included in the processing target image in a case where it is determined that the processing target image is not a progressive image.

[0864] Thus, the decoding device 200 decodes only one syntax element indicating a chroma positioning type in a case where the processing target image is a progressive image, and thus the processing amount is reduced. Therefore, the processing efficiency of the decoding device 200 is improved.

[0865] For example, the circuit of the decoding device 200 may use the first flag and the second flag contained in the video availability information syntax structure to determine whether the image to be processed is a progressive scan image or an interlaced scan image, wherein the first flag indicates whether the image to be processed is a progressive scan image, and the second flag indicates whether the image to be processed is an interlaced scan image.

[0866] Thus, even when the processing target video includes an video whose display format (i.e., the scanning format of the display) is not specified, the decoding device 200 can still determine whether the processing target video is a progressive scan video or an interlaced scan video. Therefore, the decoding device 200 can more appropriately process the processing target video, thereby improving processing efficiency.

[0867] For example, the circuit of the decoding device 200 may also decode a third flag from the bitstream, wherein the third flag indicates whether the bitstream includes information indicating the positions of chrominance samples relative to luminance samples for frames included in the processing target image as one syntactic element or two syntactic elements. When the third flag indicates that the bitstream includes information indicating the positions of chrominance samples relative to luminance samples for frames included in the processing target image, and when the first flag and the second flag are used to determine that the processing target image is a progressive scan image, one syntactic element is decoded from the bitstream; and when the third flag indicates that the bitstream includes information indicating the positions of chrominance samples relative to luminance samples for frames included in the processing target image, and when the first flag and the second flag are used to determine that the processing target image is an interlaced scan image, two syntactic elements are decoded from the bitstream.

[0868] Thus, the decoding device 200 can switch the syntax elements to be decoded depending on whether the processing target video is a progressive scan video or an interlaced scan video. Therefore, the decoding device 200 can process the processing target video more appropriately, thereby improving processing efficiency.

[0869] In addition, as described above, each component may also be a circuit. These circuits may constitute a single circuit as a whole, or they may be different circuits. In addition, each component may be implemented by a general-purpose processor or a dedicated processor.

[0870] Furthermore, the processing performed by a specific component may be performed by another component. Furthermore, the order in which the processing is performed may be changed, and multiple processing may be performed simultaneously. Furthermore, the encoding and decoding apparatus may include the encoding apparatus 100 and the decoding apparatus 200.

[0871] While the embodiments of the encoding device 100 and the decoding device 200 have been described above based on a plurality of examples, the embodiments of the encoding device 100 and the decoding device 200 are not limited to these examples. The embodiments of the encoding device 100 and the decodin...

Claims

1. A sending device, wherein: have: circuits; and a memory connected to the circuit, The circuit is in action, When the image to be processed is a progressive scan image, a syntax element indicating a color difference positioning type is written into the bitstream, wherein the color difference positioning type is information indicating the position of color difference samples relative to luma samples for a frame included in the image to be processed. When the processing target image is not a progressive scan image, two syntax elements representing the color difference positioning type for two fields included in the processing target image are written into the bitstream. The bit stream is transmitted.

2. A coding device, wherein: have: circuits; and a memory connected to the circuit, The circuit is in action, A first flag indicating whether the processing target image is a progressive scan image and a second flag indicating whether the processing target image is an interlaced scan image are encoded into a bit stream, encoding, into the bitstream, a third flag indicating whether information indicating positions of chrominance samples relative to luma samples for a frame is included in the bitstream as one syntactic element or as two syntactic elements, When the third flag indicates that information indicating the position of the chrominance sample relative to the luma sample for the frame is included in the bitstream, When the processing target image is a progressive scan image, encoding the one syntax element indicating a color difference positioning type into the bitstream, the color difference positioning type being information indicating the position of the color difference samples relative to the luma samples for the frame included in the processing target image, When the processing target image is not a progressive scan image, the two syntax elements respectively indicating the color difference positioning type for the two fields included in the processing target image are encoded into the bit stream.

3. A decoding device, wherein: have: circuits; and a memory connected to the circuit, The circuit is in action, Decoding from the bit stream a first flag indicating whether the image to be processed is a progressive scan image, and a second flag indicating whether the image to be processed is an interlaced scan image, decoding, from the bitstream, a third flag indicating whether information indicating positions of chrominance samples relative to luma samples for a frame is included in the bitstream as one syntax element or two syntax elements; When the third flag indicates that information indicating the position of the chrominance sample relative to the luma sample for the frame is included in the bitstream, Using the first flag and the second flag, it is determined whether the processing target image is a progressive scan image. When it is determined that the processing target image is a progressive scan image, the one syntactic element indicating a chroma positioning type is decoded from the bit stream, the chroma positioning type being information indicating the position of the chroma samples relative to the luminance samples for the frame included in the processing target image, When it is determined that the processing target image is not an interlaced scan image, the two syntax elements respectively indicating the color difference positioning type for the two fields included in the processing target image are decoded from the bit stream.

4. A sending device, wherein: have: circuits; and a memory connected to the circuit, The circuit is in action, A first flag indicating whether the image to be processed is a progressive scan image and a second flag indicating whether the image to be processed is an interlaced scan image are written into the bit stream, writing a third flag into the bitstream indicating whether information indicating positions of chrominance samples relative to luma samples for a frame is included in the bitstream as one syntactic element or two syntactic elements, When the third flag indicates that information indicating the position of the chrominance sample relative to the luma sample for the frame is included in the bitstream, When the processing target image is a progressive scan image, the one syntax element indicating a color difference positioning type is written into the bitstream, the color difference positioning type being information indicating the position of the color difference samples relative to the luminance samples for the frame included in the processing target image, When the processing target image is not a progressive scan image, the two syntax elements respectively indicating the color difference positioning type for the two fields included in the processing target image are written into the bitstream, The bit stream is transmitted.

5. A sending method, wherein: When the image to be processed is a progressive scan image, a syntax element indicating a color difference positioning type is written into the bitstream, wherein the color difference positioning type is information indicating the position of color difference samples relative to luma samples for a frame included in the image to be processed. When the processing target image is not a progressive scan image, two syntax elements representing the color difference positioning type for two fields included in the processing target image are written into the bitstream. The bit stream is transmitted.

6. A coding method, wherein: A first flag indicating whether the processing target image is a progressive scan image and a second flag indicating whether the processing target image is an interlaced scan image are encoded into a bit stream, encoding, into the bitstream, a third flag indicating whether information indicating positions of chrominance samples relative to luma samples for a frame is included in the bitstream as one syntactic element or as two syntactic elements, When the third flag indicates that information indicating the position of the chrominance sample relative to the luma sample for the frame is included in the bitstream, When the processing target image is a progressive scan image, encoding the one syntax element indicating a color difference positioning type into the bitstream, the color difference positioning type being information indicating the position of the color difference samples relative to the luma samples for the frame included in the processing target image, When the processing target image is not a progressive scan image, the two syntax elements respectively indicating the color difference positioning type for the two fields included in the processing target image are encoded into the bit stream.

7. A decoding method, wherein: Decoding from the bit stream a first flag indicating whether the image to be processed is a progressive scan image, and a second flag indicating whether the image to be processed is an interlaced scan image, decoding, from the bitstream, a third flag indicating whether information indicating positions of chrominance samples relative to luma samples for a frame is included in the bitstream as one syntax element or two syntax elements; When the third flag indicates that information indicating the position of the chrominance sample relative to the luma sample for the frame is included in the bitstream, Using the first flag and the second flag, it is determined whether the processing target image is a progressive scan image. When it is determined that the processing target image is a progressive scan image, the one syntactic element indicating a chroma positioning type is decoded from the bit stream, the chroma positioning type being information indicating the position of the chroma samples relative to the luminance samples for the frame included in the processing target image, When it is determined that the processing target image is not an interlaced scan image, the two syntax elements respectively indicating the color difference positioning type for the two fields included in the processing target image are decoded from the bit stream.

8. A method of sending, wherein: A first flag indicating whether the image to be processed is a progressive scan image and a second flag indicating whether the image to be processed is an interlaced scan image are written into the bit stream, writing a third flag indicating whether information indicating positions of chrominance samples relative to luma samples for a frame is included in the bitstream as one syntactic element or two syntactic elements, When the third flag indicates that information indicating the position of the chrominance sample relative to the luma sample for the frame is included in the bitstream, When the processing target image is a progressive scan image, the one syntax element indicating a color difference positioning type is written into the bitstream, the color difference positioning type being information indicating the position of the color difference samples relative to the luminance samples for the frame included in the processing target image, When the processing target image is not a progressive scan image, the two syntax elements respectively indicating the color difference positioning type for the two fields included in the processing target image are written into the bitstream, The bit stream is transmitted.

Images