Encoding device, decoding device, and transmitting device

Abstract

To allow further advancement, improvement, and optimization to be achieved in encoding of blocks using predicted chrominance samples.SOLUTION: An encoding device determines whether a first VPDU is to be split into smaller blocks and whether a second VPDU is to be split into smaller blocks based on split information. In response to a determination that the first VPDU is not to be split into smaller blocks and the second VPDU is to be split into smaller blocks, a block of chrominance samples is predicted without using luma samples. In response to a determination that the first VPDU is to be split into smaller blocks and the second VPDU is to be split into smaller blocks, the block of chrominance samples is predicted using luma samples. In response to a determination that the first VPDU is not to be split into smaller blocks and the second VPDU is not to be split into smaller block, the block of chrominance samples is predicted using luma samples.SELECTED DRAWING: Figure 98

Description

Technical Field

[0001] The present disclosure relates to video coding, and more particularly to video encoding and decoding systems and components, such as encoding blocks using predicted chrominance samples, and video encoding and decoding methods.

Background Art

[0002] Video coding technologies have evolved from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). Along with this evolution, there is a constant need to provide improvements and optimizations to video coding technologies to handle the ever-increasing amounts of digital video data in various applications. The present disclosure relates to further advancements, improvements, and optimizations in video coding, particularly in the encoding of blocks using predicted chrominance samples.

Summary of the Invention

Means for Solving the Problems

[0003] In one embodiment, the encoding device comprises a circuit and a memory connected to the circuit. The circuit determines, based on division information, whether to divide the first VPDU (virtual pipeline decoding unit) into smaller blocks and whether to divide the second VPDU into smaller blocks. If the determination is not to divide the first VPDU into smaller blocks and to divide the second VPDU into smaller blocks, the device predicts blocks of color difference samples without using luminance samples. If the determination is to divide the first VPDU into smaller blocks and to divide the second VPDU into smaller blocks, the device predicts blocks of color difference samples using luminance samples. If the determination is not to divide the first VPDU into smaller blocks and to not divide the second VPDU into smaller blocks, the device predicts blocks of color difference samples using luminance samples. The device encodes the blocks using the predicted color difference samples.

[0004] In video coding technology, there is a demand for new methods to improve coding efficiency, improve image quality, and reduce circuit size. Implementations of each embodiment of the disclosure, including the components of the embodiments of the disclosure considered individually or in various combinations, enable at least one of the following: improved coding efficiency, improved image quality, reduced coding / decoding processing resource usage, reduced circuit size, or improved coding / decoding processing speed.

[0005] Furthermore, implementations of each embodiment in this disclosure, including the components of the embodiments of this disclosure considered individually or in various combinations, enable appropriate selection of components / operations such as filters, blocks, sizes, motion vectors, reference pictures, and reference blocks in encoding and decoding. This disclosure also includes disclosures of configurations or methods that may provide benefits other than those mentioned above, such as configurations or methods that improve encoding efficiency while suppressing an increase in processing resource usage.

[0006] Further advantages and effects of the disclosed embodiments will be evident from the specification and drawings. Such advantages and / or effects are obtained by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided to obtain one or more advantages and / or effects.

[0007] These general or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium, or as any combination of a system, method, integrated circuit, computer program, and recording medium. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing an example of the functional configuration of a transmission system according to an embodiment. [Figure 2] Figure 2 is a conceptual diagram showing an example of the hierarchical structure of data in a stream. [Figure 3] Figure 3 is a conceptual diagram showing an example of a slice configuration. [Figure 4] Figure 4 is a conceptual diagram showing an example of tile configuration. [Figure 5] Figure 5 is a conceptual diagram showing an example of an encoding structure during scalable encoding. [Figure 6] Figure 6 is a conceptual diagram showing an example of an encoding structure during scalable encoding. [Figure 7] Figure 7 is a block diagram showing an example of the functional configuration of an encoding device according to an embodiment. [Figure 8] Figure 8 is a functional block diagram showing an example implementation of an encoding device. [Figure 9] Figure 9 is a flowchart showing an example of the overall encoding process by the encoding device. [Figure 10] Figure 10 is a conceptual diagram showing an example of block division. [Figure 11] Figure 11 is a block diagram showing an example of the functional configuration of a divided section according to the embodiment. [Figure 12] Figure 12 is a conceptual diagram showing an example of a division pattern. [Figure 13A] Figure 13A is a conceptual diagram showing an example of a syntax tree for a partitioning pattern. [Figure 13B] Figure 13B is a conceptual diagram showing another example of a syntax tree for a partitioning pattern. [Figure 14] Figure 14 is a table showing examples of transformation basis functions corresponding to each transformation type. [Figure 15] Figure 15 is a conceptual diagram showing an example of SVT (spatially varying transform). [Figure 16] Figure 16 is a flowchart showing an example of processing by the conversion unit. [Figure 17] Figure 17 is a flowchart showing another example of processing by the conversion unit. [Figure 18] Figure 18 is a block diagram showing an example of the functional configuration of the quantization unit according to the embodiment. [Figure 19] Figure 19 is a flowchart showing an example of quantization processing by the quantization unit. [Figure 20] Figure 20 is a block diagram showing an example of the functional configuration of the entropy coding unit according to the embodiment. [Figure 21] Figure 21 is a conceptual diagram showing an example of the CABAC (context-based adaptive binary arithmetic coding) processing flow in the entropy coding unit. [Figure 22] Figure 22 is a block diagram showing an example of the functional configuration of the loop filter unit according to the embodiment. [Figure 23A] Figure 23A is a conceptual diagram showing an example of the filter shape used in an ALF (adaptive loop filter). [Figure 23B] Figure 23B is a conceptual diagram showing another example of the filter shape used in ALF. [Figure 23C] Figure 23C is a conceptual diagram showing another example of the filter shape used in ALF. [Figure 23D] Figure 23D is a conceptual diagram showing an example of the CCALF (cross component ALF) flow. [Figure 23E] Figure 23E is a conceptual diagram showing the filter shape used in CCALF. [Figure 23F] Figure 23F is a conceptual diagram showing an example of the JC-CCALF (Joint Chroma CCALF) flow. [Figure 23G] Figure 23G is a table showing examples of weight_index candidates for JC-CCALF. [Figure 24] Figure 24 is a block diagram showing an example of a detailed configuration of a loop filter section that functions as a DBF (deblocking filter). [Figure 25] Figure 25 is a conceptual diagram showing an example of a deblocking filter with symmetrical filter characteristics with respect to block boundaries. [Figure 26] Figure 26 is a conceptual diagram illustrating an example of a block boundary where deblocking filtering is performed. [Figure 27] Figure 27 is a conceptual diagram showing an example of a Bs (Boundary Strength) value. [Figure 28] Figure 28 is a flowchart showing an example of the processing performed in the prediction unit of the encoding device. [Figure 29] Figure 29 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device. [Figure 30] Figure 30 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device. [Figure 31] Figure 31 is a conceptual diagram showing 67 intra-prediction modes in an embodiment of intra-prediction. [Figure 32] Figure 32 is a flowchart showing an example of processing by the intra-prediction unit. [Figure 33] Figure 33 is a conceptual diagram showing an example of each reference picture. [Figure 34]Figure 34 is a conceptual diagram showing an example of a reference picture list. [Figure 35] Figure 35 is a flowchart showing an example of the basic processing flow for interpretation. [Figure 36] Figure 36 is a flowchart showing an example of the MV (motion vector) derivation process. [Figure 37] Figure 37 is a flowchart showing another example of the MV derivation process. [Figure 38A] Figure 38A is a conceptual diagram showing an example of the characteristics of each mode in the MV derivation. [Figure 38B] Figure 38B is a conceptual diagram showing an example of the characteristics of each mode in the MV derivation. [Figure 39] Figure 39 is a flowchart showing an example of inter-prediction processing using normal inter-mode. [Figure 40] Figure 40 is a flowchart showing an example of interpretation processing using normal merge mode. [Figure 41] Figure 41 is a conceptual diagram illustrating an example of MV derivation processing using normal merge mode. [Figure 42] Figure 42 is a conceptual diagram illustrating an example of the MV derivation process of the current picture using HMVP merge mode. [Figure 43] Figure 43 is a flowchart showing an example of FRUC (frame rate up conversion) processing. [Figure 44] Figure 44 is a conceptual diagram illustrating an example of pattern matching (bilateral matching) between two blocks along a motion trajectory. [Figure 45] Figure 45 is a conceptual diagram illustrating an example of pattern matching (template matching) between a template in the current picture and a block in a referenced picture. [Figure 46A] Figure 46A is a conceptual diagram illustrating an example of deriving the MV at the subblock level based on the MVs of multiple adjacent blocks. [Figure 46B]Figure 46B is a conceptual diagram illustrating an example of deriving the subblock unit MV in affine mode using three control points. [Figure 47A] Figure 47A is a conceptual diagram illustrating an example of the MV derivation of the control point in affine mode. [Figure 47B] Figure 47B is a conceptual diagram illustrating an example of the MV derivation of the control point in affine mode. [Figure 47C] Figure 47C is a conceptual diagram illustrating an example of the MV derivation of the control point in affine mode. [Figure 48A] Figure 48A is a conceptual diagram illustrating an affine mode with two control points. [Figure 48B] Figure 48B is a conceptual diagram illustrating an affine mode with three control points. [Figure 49A] Figure 49A is a conceptual diagram illustrating an example of a method for deriving the control point MV when the number of control points differs between the encoded block and the current block. [Figure 49B] Figure 49B is a conceptual diagram illustrating another example of how to derive the control point MV when the number of control points differs between the encoded block and the current block. [Figure 50] Figure 50 is a flowchart showing an example of processing in affine merge mode. [Figure 51] Figure 51 is a flowchart showing an example of affine intermode processing. [Figure 52A] Figure 52A is a conceptual diagram illustrating the generation of two triangular prediction images. [Figure 52B] Figure 52B is a conceptual diagram showing the first portion of the first partition overlapping with the second partition, as well as examples of the first and second sample sets which may be weighted as part of the correction process. [Figure 52C] Figure 52C is a conceptual diagram showing the first part of the first partition, which is a portion of the first partition that overlaps with a portion of an adjacent partition. [Figure 53] Figure 53 is a flowchart showing an example of triangle mode processing. [Figure 54] Figure 54 is a conceptual diagram showing an example of an ATMVP (Advanced Temporal Motion Vector Prediction) mode in which MV is derived at the subblock level. [Figure 55] Figure 55 is a flowchart showing the relationship between merge mode and DMVR (dynamic motion vector refreshing). [Figure 56] Figure 56 is a conceptual diagram illustrating an example of a DMVR. [Figure 57] Figure 57 is a conceptual diagram illustrating another example of DMVR for determining MV. [Figure 58A] Figure 58A is a conceptual diagram showing an example of motion search in a DMVR. [Figure 58B] Figure 58B is a flowchart showing an example of motion detection processing in a DMVR. [Figure 59] Figure 59 is a flowchart showing an example of the process for generating predicted images. [Figure 60] Figure 60 is a flowchart showing another example of the predictive image generation process. [Figure 61] Figure 61 is a flowchart illustrating an example of predictive image correction processing using OBMC (overlapped block motion compensation). [Figure 62] Figure 62 is a conceptual diagram illustrating an example of predictive image correction processing using OBMC. [Figure 63] Figure 63 is a conceptual diagram illustrating a model that assumes uniform linear motion. [Figure 64] Figure 64 is a flowchart showing an example of an interpretation prediction process according to BIO. [Figure 65] Figure 65 is a functional block diagram showing an example of the functional configuration of the inter-prediction unit that performs inter-prediction according to BIO. [Figure 66A] Figure 66A is a conceptual diagram illustrating an example of a predictive image generation method using luminance correction processing by LIC (local illumination compensation). [Figure 66B] Figure 66B is a flowchart showing an example of a predictive image generation method using luminance correction processing by LIC. [Figure 67] Figure 67 is a block diagram showing the functional configuration of a decoding device according to an embodiment. [Figure 68] Figure 68 is a functional block diagram showing an example implementation of a decoding device. [Figure 69] Figure 69 is a flowchart showing an example of the overall decoding process by the decoding device. [Figure 70] Figure 70 is a conceptual diagram showing the relationship between the division decision unit and other components. [Figure 71] Figure 71 is a block diagram showing an example of the functional configuration of the entropy decoding unit. [Figure 72] Figure 72 is a conceptual diagram showing an example of the CABAC processing flow in the entropy decoding unit. [Figure 73] Figure 73 is a block diagram showing an example of the functional configuration of the inverse quantization unit. [Figure 74] Figure 74 is a flowchart showing an example of inverse quantization processing by the inverse quantization unit. [Figure 75] Figure 75 is a flowchart showing an example of processing by the inverse transform unit. [Figure 76] Figure 76 is a flowchart showing another example of processing by the inverse transform unit. [Figure 77] Figure 77 is a block diagram showing an example of the functional configuration of the loop filter section. [Figure 78] Figure 78 is a flowchart showing an example of the processing performed in the prediction unit of the decoding device. [Figure 79] Figure 79 is a flowchart showing another example of the processing performed in the prediction unit of the decoding device. [Figure 80]Figure 80 is a flowchart showing another example of the processing performed in the prediction unit of the decoding device. [Figure 81] Figure 81 shows an example of processing performed by the intra-prediction unit of the decoding device. [Figure 82] Figure 82 is a flowchart showing an example of the MV derivation process in a decoding device. [Figure 83] Figure 83 is a flowchart showing another example of the MV derivation process in a decoding device. [Figure 84] Figure 84 is a flowchart showing an example of inter-prediction processing using normal inter-mode in a decoding device. [Figure 85] Figure 85 is a flowchart showing an example of interpretation processing using normal merge mode in a decoding device. [Figure 86] Figure 86 is a flowchart showing an example of interpretation processing using FRUC mode in a decoding device. [Figure 87] Figure 87 is a flowchart showing an example of interprediction processing using affine merge mode in a decoding device. [Figure 88] Figure 88 is a flowchart showing an example of inter-prediction processing using affine intermode in a decoding device. [Figure 89] Figure 89 is a flowchart showing an example of interpretation processing using triangle mode in a decoding device. [Figure 90] Figure 90 is a flowchart showing an example of motion detection processing by a DMVR in a decoding device. [Figure 91] Figure 91 is a flowchart showing an example of motion detection processing by a DMVR in a decoding device. [Figure 92] Figure 92 is a flowchart showing an example of the predictive image generation process in a decoding device. [Figure 93] Figure 93 is a flowchart showing another example of the predictive image generation process in a decoding device. [Figure 94]Figure 94 is a flowchart showing an example of the correction process of predicted images by OBMC in a decoding device. [Figure 95] Figure 95 is a flowchart showing an example of the BIO-based correction process for predicted images in a decoding device. [Figure 96] Figure 96 is a flowchart showing an example of the correction process of the predicted image by the LIC in the decoding device. [Figure 97] Figure 97 is a flowchart showing an example of the process of decoding blocks using predicted color difference samples. [Figure 98] Figure 98 is a flowchart showing an example of a process for decoding blocks using predicted color difference samples. [Figure 99] Figure 99 is a conceptual diagram illustrating an example of determining whether or not a color difference block to be processed is located within an M×N non-overlapping area that matches an M×N grid of color difference samples. [Figure 100] Figure 100 is a conceptual diagram illustrating an example of determining whether or not a color difference block to be processed is located within an M×N non-overlapping area that matches an M×N grid of color difference samples. [Figure 101] Figure 101 is a conceptual diagram illustrating the VPDU (virtual pipeline decoder unit). [Figure 102] Figure 102 is a conceptual diagram illustrating an example of determining whether the VPDU being processed can predict the blocks of the color difference samples using luminance samples. [Figure 103] Figure 103 is a conceptual diagram illustrating an example of a method for determining whether or not to divide the luminance VPDU into smaller blocks. [Figure 104] Figure 104 is a conceptual diagram illustrating additional decisions that may be considered when determining whether or not to use luminance samples to predict the color difference samples of a block. [Figure 105] Figure 105 is a conceptual diagram illustrating an example of considering combinations of conditions in determining whether or not to predict the color difference sample of a block using a luminance sample. [Figure 106]Figure 106 is a conceptual diagram illustrating an example of considering combinations of conditions in determining whether or not to predict the color difference sample of a block using a luminance sample. [Figure 107] Figure 107 is a conceptual diagram illustrating an example of considering combinations of conditions in determining whether or not to predict the color difference sample of a block using a luminance sample. [Figure 108] Figure 108 is a conceptual diagram illustrating an example of considering combinations of conditions in determining whether or not to predict the color difference sample of a block using a luminance sample. [Figure 109] Figure 109 is a conceptual diagram illustrating an example of considering combinations of conditions in determining whether or not to predict the color difference sample of a block using a luminance sample. [Figure 110] Figure 110 is a conceptual diagram illustrating an example of considering combinations of conditions in determining whether or not to predict the color difference sample of a block using a luminance sample. [Figure 111] Figure 111 is a conceptual diagram showing an example of a non-rectangular partition. [Figure 112] Figure 112 is a diagram showing the overall configuration of a content supply system that realizes a content distribution service. [Figure 113] Figure 113 is a conceptual diagram showing an example of a web page display screen. [Figure 114] Figure 114 is a conceptual diagram showing an example of a web page display screen. [Figure 115] Figure 115 is a block diagram showing an example of a smartphone. [Figure 116] Figure 116 is a block diagram showing an example of a smartphone's functional configuration. [Modes for carrying out the invention]

[0009] In drawings, unless otherwise indicated, the same reference number refers to the same or similar component. Furthermore, the size and relative position of components in drawings are not necessarily depicted to a consistent scale.

[0010] The embodiments will be described in detail below with reference to the drawings. Note that the embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement and connection forms of components, steps, relationships and sequences of steps shown in the following embodiments are examples only and are not intended to limit the scope of the claims.

[0011] Embodiments of encoding and decoding devices are described below. These embodiments are examples of encoding and decoding devices to which the processes and / or configurations described in each aspect of this disclosure can be applied. The processes and / or configurations can also be implemented in encoding and decoding devices different from those in the embodiments. For example, with respect to the processes and / or configurations applicable to the embodiments, one of the following may be implemented:

[0012] (1) Any of the multiple components of the encoding or decoding device of the embodiments described in each aspect of the Disclosure may be replaced or combined with other components described in any of the aspects of the Disclosure.

[0013] (2) In the encoding or decoding device of the embodiment, any modifications such as addition, replacement, or deletion of functions or processes performed by some of the multiple components of the encoding or decoding device may be made. For example, any of the functions or processes may be replaced or combined with other functions or processes described in any of the embodiments of this disclosure.

[0014] (3) In the methods performed by the encoding or decoding apparatus of the embodiment, any modifications, such as additions, replacements, and deletions, may be made to some of the processes included in the method. For example, any of the processes in the method may be replaced with or combined with other processes described in any of the embodiments of this disclosure.

[0015] (4) Some of the multiple components constituting the encoding or decoding device of the embodiment may be combined with components described in any of the embodiments of this disclosure, or with components that have some of the functions described in any of the embodiments of this disclosure, or with components that perform some of the processing performed by the components described in any of the embodiments of this disclosure.

[0016] (5) Components that provide some of the functions of the encoding or decoding device of the embodiment, or components that perform some of the processing of the encoding or decoding device of the embodiment, may be combined with or replaced with components described in any of the aspects of the disclosure and components that provide some of the functions described in any of the aspects of the disclosure, or components that perform some of the processing described in any of the aspects of the disclosure.

[0017] (6) In a method performed by an encoding or decoding device of an embodiment, any of the processes included in the method may be replaced or combined with any of the processes described in any of the embodiments of the present disclosure.

[0018] (7) Some of the processes included in the methods performed by the encoding or decoding device of the embodiment may be combined with the processes described in any of the embodiments of this disclosure.

[0019] (8) The methods of carrying out the processes and / or configurations described in each aspect of the present disclosure are not limited to the encoding or decoding devices of the embodiments. For example, the processes and / or configurations may be carried out in devices used for purposes other than the video encoding or video decoding disclosed in the embodiments.

[0020] [Definition of Terms] The following definitions may be used as examples for each term.

[0021] An image is a unit of data composed of a collection of pixels, consisting of pictures and smaller blocks. Images include both still images and videos.

[0022] A picture is an image processing unit composed of a collection of pixels, and is sometimes called a frame or field. A picture may take the form of, for example, a luminance sample array in a monochrome format, or a luminance sample array and its two corresponding color difference sample arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0023] A block is a processing unit of a set containing a specific number of pixels. Blocks can also have any shape. For example, they can be rectangles (M×N pixels), squares (M×M pixels), triangles, circles, and other shapes. Examples of blocks include slices, tiles, bricks, CTUs, superblocks, basic division units, VPDUs, hardware processing division units, CUs, processing block units, prediction block units (PUs), orthogonal transformation block units (TUs), units, and subblocks. A block may take the form of an M×N array of samples or an M×N array of transformation coefficients. For example, a block may be a square or rectangular region of pixels containing one luminance matrix and two chrominance matrices.

[0024] A pixel or sample is the smallest unit of a point that makes up an image. A pixel or sample includes not only pixels at integer positions but also pixels at subpixel positions generated based on pixels at integer positions.

[0025] Pixel values ​​or sample values ​​are unique values ​​that a pixel possesses. Pixel values ​​or sample values ​​include one or more luminance values, chrominance values, RGB tones, depth values, or binary values ​​such as 0 or 1.

[0026] Chroma (or chrominance) is the intensity of a color and is usually represented by the symbols Cb and Cr, indicating that a sample array value or a single sample value represents one of two color difference signals related to a primary color.

[0027] Luminance (or luma) is the brightness of an image and is usually represented by a symbol, or a subscript Y or L, indicating that the sample array value or single sample value represents the monochrome signal value related to the primary color.

[0028] A flag can be a single bit or a multi-bit value, and may be, for example, a parameter or index value. Furthermore, a flag may be a binary flag indicating a binary value, or it may indicate a non-binary parameter value.

[0029] A signal transmits information that is symbolized or encoded by the signal. Signals include discretized digital signals and analog signals that take continuous values.

[0030] A stream or bitstream is a digital data string in a digital data flow. A stream or bitstream may consist of a single stream or multiple streams divided into multiple layers. It may also be transmitted by serial communication over a single transmission path or by packet communication over multiple transmission paths.

[0031] The term "difference" refers to various mathematical differences, including simple differences (xy), absolute differences (|xy|), squared differences (x^2-y^2), square roots of differences (√(xy)), weighted differences (ax-by: a, b is a constant), and offset differences (x-y+a: a is the offset). For scalar quantities, a simple difference is sufficient, although the difference operation may be included.

[0032] The term "sum" refers to various mathematical sums, including simple sums (x+y), absolute values ​​of sums (|x+y|), sums of squares (x^2+y^2), square roots of sums (√(x+y)), weighted sums (ax+by: a, b is a constant), and offset sums (x+y+a: a is the offset). For scalar quantities, a simple sum is sufficient, although summation operations may be included.

[0033] A frame is a combination of the top and bottom fields. Sample rows 0, 2, 4, ... originate from the top field, while sample rows 1, 3, 5, ... originate from the bottom field.

[0034] A slice is an integer number of coded tree units, which consist of one independent slice segment and all subsequent dependent slice segments (if any) that precede the next independent slice segment (if any) within the same access unit.

[0035] A tile is a rectangular region of a coding tree block within a specific tile column or row of a picture. A tile may also be a rectangular region of a frame. Tiles are intended to be able to be decoded and coded independently, although loop filters spanning tile edges may still be applied.

[0036] A coding tree unit (CTU) may be a coding tree block of luminance samples of a picture having three sample arrays, or two corresponding coding tree blocks of chrominance samples. Alternatively, a CTU may be a coding tree block of one sample from a monochrome picture and a picture encoded using a syntax structure and three separate color planes used to encode the samples. A superblock may be a 64x64 pixel square block consisting of one or two mode information blocks, or recursively divided into four 32x32 blocks, each of which can be further divided.

[0037] [System Configuration] First, the transmission system according to this embodiment will be described. Figure 1 is a schematic diagram showing an example of the configuration of the transmission system 400 according to this embodiment.

[0038] The transmission system 400 is a system that transmits a stream generated by encoding an image and decodes the transmitted stream. As shown in the figure, the transmission system 400 includes an encoding device 100, a network 300, and a decoding device 200, for example, as shown in Figure 1.

[0039] An image is input to the encoding device 100. The encoding device 100 generates a stream by encoding the input image and outputs the stream to the network 300. The stream includes, for example, the encoded image and control information for decoding the encoded image. The image is compressed by this encoding process.

[0040] The image before encoding by the encoding device 100 is also called the original image, original signal, or original sample. The image may be a moving image or a still image. The image is a higher-level concept than sequences, pictures, and blocks, and is not limited by spatial and temporal domains unless otherwise specified. The image consists of a sequence of pixels or pixel values, and the signal or pixel values ​​representing the image are also called samples. The stream may be called a bitstream, encoded bitstream, compressed bitstream, or encoded signal. Furthermore, the encoding device 100 may be called an image encoding device or a moving image encoding device, and the encoding method by the encoding device 100 may be called an encoding method, an image encoding method, or a moving image encoding method.

[0041] Network 300 transmits the stream generated by the encoding device 100 to the decoding device 200. Network 300 is connected to the Internet, Wide Area Network (WAN). Network 300 may be a local area network (LAN), a small-scale network (LAN), or a combination thereof. Network 300 is not necessarily limited to a bidirectional communication network, but may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Network 300 may also be replaced by a storage medium that records streams such as DVD (Digital Versatile Disc) or BD (Blu-Ray Disc®).

[0042] The decoding device 200 generates a decoded image, which is, for example, an uncompressed image, by decoding the stream transmitted by the network 300. For example, the decoding device decodes the stream according to a decoding method that corresponds to the encoding method by the encoding device 100.

[0043] The decoding device 200 may also be called an image decoding device or a video decoding device, and the decoding method performed by the decoding device 200 may be called a decoding method, an image decoding method, or a video decoding method.

[0044] [Data structure] Figure 2 is a conceptual diagram showing an example of a data hierarchical structure in a stream. For convenience, Figure 2 will be explained with reference to the transmission system 400 in Figure 1. The stream includes, for example, a video sequence. This video sequence includes, for example, one or more VPS (Video Parameter Sets), one or more SPS (Sequence Parameter Sets), one or more PPS (Picture Parameter Sets), SEI (Supplemental Enhancement Information), and multiple pictures, as shown in Figure 2(a).

[0045] In a video composed of multiple layers, the VPS may include encoding parameters common to multiple layers, as well as encoding parameters related to the multiple layers included in the video, or to individual layers.

[0046] The SPS includes parameters used for the sequence, i.e., encoding parameters that the decoding device 200 references to decode the sequence. For example, these encoding parameters may represent the width or height of the picture. Multiple SPSs may exist.

[0047] The PPS includes parameters used for the picture, i.e., encoding parameters that the decoding device 200 references to decode each picture in the sequence. For example, these encoding parameters may include a reference value for the quantization width used to decode the picture and a flag indicating the application of weighted prediction. Multiple PPSs may exist. The SPS and PPS are sometimes simply referred to as parameter sets.

[0048] The picture may include a picture header and one or more slices, as shown in Figure 2(b). The picture header includes encoding parameters that the decoding device 200 references to decode the one or more slices.

[0049] A slice includes a slice header and one or more bricks, as shown in Figure 2(c). The slice header includes encoding parameters that the decoding device 200 references to decode the one or more bricks.

[0050] A brick is one or more CTUs (Coding Trees) as shown in Figure 2(d). Includes Unit)

[0051] Note that a picture may not contain slices, but instead contain tile groups. In this case, a tile group contains one or more tiles. Also, a brick may contain slices.

[0052] A CTU is also called a superblock or basic division unit. A CTU includes a CTU header and one or more CUs (Coding Units), as shown in Figure 2(e). As illustrated, a CTU includes four CUs (10), CU (11), CU (12), and CU (13). The CTU header contains coding parameters that the decoding device 200 refers to in order to decode one or more CUs.

[0053] A CU may be divided into multiple smaller CUs. As shown in the figure, CU(10) is not divided into smaller CUs, CU(11) is divided into four smaller CUs (110), CU(111), CU(112), and CU(113), CU(12) is not divided into smaller CUs, and CU(13) is divided into seven smaller CUs (1310), CU(1311), CU(1312), CU(1313), CU(132), CU(133), and CU(134). Furthermore, as shown in Figure 2(f), a CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information for predicting that CU, and the residual coefficient information is information that shows the predicted residual, which will be described later. Note that a CU is basically the same as a PU (Prediction Unit) and a TU (Transform Unit), but in the case of an SBT (sub-block transform) described later, it may contain multiple TUs smaller than the CU. Also, a CU may be processed for each VPDU (Virtual Pipeline Decoding Unit) that constitutes it. A VPDU is a fixed unit that can be processed in one stage when performing pipeline processing in hardware, for example.

[0054] Note that a stream does not necessarily have all of the layers shown in Figure 2. Also, the order of these layers may be changed, and any layer may be replaced by another layer. The picture that is currently being processed by a device such as the encoding device 100 or the decoding device 200 is called the current picture. If the processing is encoding, the current picture is synonymous with the picture to be encoded, and if the processing is decoding, the current picture is synonymous with the picture to be decoded. Also, a block, such as CU or CU, that is currently being processed by a device such as the encoding device 100 or the decoding device 200 is called the current block. If the processing is encoding, the current block is synonymous with the block to be encoded, and if the processing is decoding, the current block is synonymous with the block to be decoded.

[0055] [Picture composition: slice / tile] To allow picture encoding / decoding to proceed in parallel, pictures may be composed of slices or tiles.

[0056] A slice is the basic encoding unit that makes up a picture. A picture is composed of, for example, one or more slices. A slice, in turn, consists of one or more CTUs.

[0057] Figure 3 is a conceptual diagram showing an example of slice configuration. For example, in Figure 3, the picture contains 11 × 8 CTUs and is divided into four slices (slices 1-4). Slice 1 consists of, for example, 16 CTUs, slice 2 consists of, for example, 21 CTUs, slice 3 consists of, for example, 29 CTUs, and slice 4 consists of, for example, 22 CTUs. Here, each CTU in the picture belongs to one of the slices. The shape of the slice is the horizontal division of the picture. The boundaries of the slice do not have to be at the edges of the screen, but can be anywhere among the boundaries of the CTUs within the screen. The processing order (encoding order or decoding order) of the CTUs in a slice is, for example, the raster scan order. A slice also includes a slice header and encoded data. The slice header may describe the characteristics of the slice, such as the CTU address of the beginning of the slice and the slice type.

[0058] A tile is a rectangular area that makes up a picture. Picture tiles may be assigned a number called a TileId in the order of their raster scan.

[0059] Figure 4 is a conceptual diagram showing an example of tile configuration. For example, in Figure 4, the picture contains 11 × 8 CTUs and is divided into four rectangular tiles (tiles 1-4). When tiles are used, the processing order of CTUs may differ from when tiles are not used. When tiles are not used, multiple CTUs in a picture are usually processed, for example, in raster scan order. When tiles are used, in each of the multiple tiles, at least one CTU is processed, for example, in raster scan order. For example, as shown in Figure 4, the processing order of multiple CTUs contained in tile 1 is from the left end of the first column of tile 1 to the right end of the first column of tile 1, and then from the left end of the second column of tile 1 to the right end of the second column of tile 1.

[0060] Note that one tile may contain one or more slices, and one slice may contain one or more tiles.

[0061] A picture may be composed of tilesets. A tileset may contain one or more tile groups, or one or more tiles. A picture may be composed of any one of a tileset, a tile group, or a tile. For example, the order in which multiple tiles are scanned in raster order for each tileset is defined as the basic coding order of the tiles. Within each tileset, a collection of one or more tiles that follow the same basic coding order is defined as a tile group. Such a picture may be composed of the division unit 102 (see Figure 7) described later.

[0062] [Scalable encoding] Figures 5 and 6 are conceptual diagrams illustrating an example of a scalable stream configuration, and for convenience, they will be explained with reference to Figure 1.

[0063] As shown in Figure 5, the encoding device 100 may generate a temporally / spatially scalable stream by encoding each of the multiple pictures into one of the multiple layers. For example, the encoding device 100 achieves scalability by encoding each picture layer by layer, with an enhancement layer existing above the base layer. This type of encoding of each picture is called scalable encoding. This allows the decoding device 200 to switch the image quality displayed by decoding the stream. In other words, the decoding device 200 may decide which layers to decode depending on internal factors such as its own performance and external factors such as the state of the communication bandwidth. As a result, the decoding device 200 can freely switch between decoding the same content into low-resolution content and high-resolution content. For example, a user of the stream might watch part of the video on their smartphone while on the go, and then watch the rest of the video on a device such as an internet TV after returning home. Note that each of the aforementioned smartphones and devices incorporates a decoding device 200 with the same or different performance. In this case, if the device decodes up to the upper layers of the stream, the user can view high-definition video after returning home. This eliminates the need for the encoding device 100 to generate multiple streams with the same content but different image quality, thereby reducing the processing load.

[0064] Furthermore, the enhancement layer may include metadata based on image statistics. The decoding device 200 may generate a high-resolution video by super-resolution the picture of the base layer based on the metadata. Super-resolution may include, for example, improving the signal-to-noise ratio at the same resolution or increasing the resolution. The metadata may include information for identifying linear or nonlinear filter coefficients used in the super-resolution process, or information for identifying parameter values ​​in the filtering process, machine learning, or least-squares operation used in the super-resolution process.

[0065] In this embodiment, the picture may be divided into tiles or the like, depending on the meaning of each object within the picture. In this case, the decoding device 200 may decode only a portion of the picture by selecting the tiles to be decoded. Furthermore, the attributes of the objects (e.g., person, car, ball) and their positions within the picture (e.g., coordinate positions within the same picture) may be stored as metadata. In this case, the decoding device 200 can identify the position of a desired object based on the metadata and determine the tile containing that object. For example, as shown in Figure 6, the metadata is stored using a data storage structure different from the image data, such as the SEI (supplemental enhancement information) message in HEVC. This metadata may indicate, for example, the position, size, or color of the main object.

[0066] Furthermore, metadata may be stored in units consisting of multiple pictures, such as streams, sequences, or random access units. This allows the decoding device 200 to obtain information such as the time when a specific person appears in the video, and by using that time and the information in the picture units, it can identify the picture in which the object (person) exists and the position of the object within that picture.

[0067] [Encoding device] Next, an encoding device according to an embodiment will be described. Figure 7 is a block diagram showing the functional configuration of the encoding device 100 according to the embodiment. The encoding device 100 is a video encoding device that encodes moving images in block units.

[0068] As shown in Figure 7, the encoding device 100 is a device that encodes an image in block units and comprises a division unit 102, a subtraction unit 104, a transformation unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transformation unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130. As shown in the figure, the intra prediction unit 124 and the inter prediction unit 126 are each part of the prediction control unit.

[0069] The encoding device 100 can be implemented, for example, by a general-purpose processor and memory. In this case, when a software program stored in memory is executed by the processor, the processor functions as a splitting unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a loop filter unit 120, an intra prediction unit 124, an inter prediction unit 126, and a prediction control unit 128. Alternatively, the encoding device 100 may be implemented as one or more dedicated electronic circuits corresponding to the splitting unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a loop filter unit 120, an intra prediction unit 124, an inter prediction unit 126, and a prediction control unit 128.

[0070] [Example of an encoding device implementation] Figure 8 is a functional block diagram showing an example implementation of the encoding device 100. The encoding device 100 includes a processor a1 and memory a2. For example, the multiple components of the encoding device 100 shown in Figure 7 are implemented by the processor a1 and memory a2 shown in Figure 8.

[0071] Processor a1 is a circuit that performs information processing and is connected to memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit for encoding images. Processor a1 may be a processor such as a CPU. Alternatively, processor a1 may be a collection of multiple electronic circuits. Furthermore, for example, processor a1 may play the role of multiple components of the encoding device 100 shown in Figure 7.

[0072] Memory a2 is a dedicated or general-purpose memory in which information for the processor a1 to encode an image is stored. Memory a2 may be an electronic circuit and may be connected to the processor a1. Memory a2 may also be included in the processor a1. Memory a2 may also be a collection of multiple electronic circuits. Memory a2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory a2 may also be a non-volatile memory or a volatile memory.

[0073] For example, memory a2 may store the image to be encoded, or the bitstream corresponding to the encoded image. Alternatively, memory a2 may store a program for processor a1 to encode the image.

[0074] Furthermore, memory a2 may also function as one of the components of the encoding device 100 shown in Figure 7, etc., that stores information. For example, memory a2 may function as the block memory 118 and frame memory 122 shown in Figure 7. More specifically, memory a2 may store reconstructed blocks or reconstructed pictures, etc.

[0075] Furthermore, it is not necessary for the encoding device 100 to implement all of the components shown in Figure 7, nor is it necessary for all of the processes described herein to be performed. Some of the components shown in Figure 7 may be included in other devices, and some of the processes described herein may be performed by other devices.

[0076] The following describes the overall processing flow of the encoding device 100, followed by a description of each component included in the encoding device 100.

[0077] [Overall flow of the encoding process] Figure 9 is a flowchart showing an example of the overall encoding process by the encoding device 100, and for convenience, it will be explained with reference to Figure 7.

[0078] First, the division unit 102 of the encoding device 100 divides the picture contained in the input image into multiple fixed-size blocks (for example, 128 x 128 pixels) (step Sa_1). Then, the division unit 102 selects a division pattern for these fixed-size blocks (also called block shapes) (step Sa_2). In other words, the division unit 102 further divides the fixed-size blocks into multiple blocks that constitute the selected division pattern. Then, the encoding device 100 performs steps Sa_3 to Sa_9 for each of these multiple blocks (i.e., the blocks to be encoded).

[0079] The prediction processing unit, consisting of the intra-prediction unit 124 and the inter-prediction unit 126, and the prediction control unit 128 generate a prediction image of the current block (step Sa_3). The prediction image may also be called a prediction signal, prediction block, or prediction sample.

[0080] Next, the subtraction unit 104 generates the difference between the current block and the predicted image as the predicted residual (step Sa_4). The predicted residual may also be called the prediction error.

[0081] Next, the transformation unit 106 and the quantization unit 108 generate multiple quantization coefficients by performing transformation and quantization on the predicted image (step Sa_5). These multiple quantization coefficients may also be called coefficient blocks.

[0082] Next, the entropy coding unit 110 generates a stream by coding (specifically, entropy coding) its multiple quantization coefficients and prediction parameters related to the generation of the predicted image (step Sa_6). The stream may also be called an encoded bitstream or a compressed bitstream.

[0083] Next, the inverse quantization unit 112 and the inverse transformation unit 114 restore the predicted residuals by performing inverse quantization and inverse transformation on a plurality of quantization coefficients (step Sa_7).

[0084] Next, the summing unit 116 reconstructs the current block by adding the predicted image to the recovered predicted residual (step Sa_8). This generates a reconstructed image. The reconstructed image may also be called a reconstructed block or a decoded image block.

[0085] Once this reconstructed image is generated, the loop filter unit 120 performs filtering on the reconstructed image as needed (step Sa_9).

[0086] Then, the encoding device 100 determines whether or not the encoding of the entire picture is complete (step Sa_10). If it determines that it is not complete (No. in step Sa_10), it repeats the process from step Sa_2 for the next block of the picture.

[0087] In the example described above, the encoding device 100 selects one partitioning pattern for a fixed-size block and encodes each block according to that partitioning pattern. However, it may also encode each block according to multiple partitioning patterns. In this case, the encoding device 100 may evaluate the cost of each of the multiple partitioning patterns and select, for example, the stream obtained by encoding according to the partitioning pattern with the smallest cost as the output stream.

[0088] As illustrated, these steps Sa_1 to Sa_10 are performed sequentially by the encoding device 100. Alternatively, some of these processes may be performed in parallel, or their order may be changed.

[0089] The encoding process performed by such an encoding device 100 is a hybrid encoding using predictive encoding and transformative encoding. Furthermore, predictive encoding is performed by an encoding loop consisting of a subtraction unit 104, a transformer unit 106, a quantization unit 108, an inverse quantization unit 112, an inverse transformer unit 114, an addition unit 116, a loop filter unit 120, a block memory 118, a frame memory 122, an intra-prediction unit 124, an inter-prediction unit 126, and a prediction control unit 128. In other words, the prediction processing unit consisting of the intra-prediction unit 124 and the inter-prediction unit 126 constitutes a part of the encoding loop.

[0090] [Divided part] The splitting unit 102 divides each picture contained in the original image into multiple blocks and outputs each block to the subtraction unit 104. For example, the splitting unit 102 first divides the picture into blocks of a fixed size (e.g., 128x128 pixels). Other fixed block sizes may also be applied. These fixed-size blocks are sometimes called coding tree units (CTUs). Then, the splitting unit 102 divides each of the fixed-size blocks into blocks of a variable size (e.g., 64x64 pixels or less) based on, for example, a recursive quadtree and / or binary tree block splitting. In other words, the splitting unit 102 selects a splitting pattern. These variable-size blocks are sometimes called coding units (CUs), prediction units (PUs), or transformation units (TUs). Note that in various processing examples, CUs, PUs, and TUs do not need to be distinguished, and some or all of the blocks in the picture may become processing units of CUs, PUs, or TUs.

[0091] Figure 10 is a conceptual diagram showing an example of block partitioning in the embodiment. In Figure 10, solid lines represent block boundaries due to quadtree block partitioning, and dashed lines represent block boundaries due to binary tree block partitioning.

[0092] Here, block 10 is a 128x128 pixel square block (128x128 block). This 128x128 block 10 is first divided into four 64x64 pixel square blocks (quadtree block partitioning).

[0093] The top-left 64x64 pixel square block is further divided vertically into two rectangular blocks, each consisting of 32x64 pixels. The left 32x64 pixel rectangular block is further divided vertically into two rectangular blocks, each consisting of 16x64 pixels (binary tree block partitioning). As a result, the top-left 64x64 pixel block is divided into two 16x64 pixel rectangular blocks 11 and 12 and a 32x64 pixel rectangular block 13.

[0094] The 64x64 pixel block in the upper right is horizontally divided into two rectangular blocks 14 and 15, each consisting of 64x32 pixels (binary tree block partitioning).

[0095] The 64x64 pixel square block in the bottom left is divided into four 32x32 pixel square blocks (quadrutree block partitioning). Of these four 32x32 pixel square blocks, the top left and bottom right blocks are further divided. The top left 32x32 pixel square block is vertically divided into two 16x32 pixel rectangular blocks, and the right 16x32 pixel rectangular block is further horizontally divided into two 16x16 pixel square blocks (binary tree block partitioning). The bottom right 32x32 pixel square block is horizontally divided into two 32x16 pixel rectangular blocks (binary tree block partitioning). As a result, the 64x64 pixel square block in the lower left is divided into a 16x32 pixel rectangular block 16, two 16x16 pixel square blocks 17 and 18, two 32x32 pixel square blocks 19 and 20, and two 32x16 pixel rectangular blocks 21 and 22.

[0096] Block 23, consisting of 64x64 pixels in the lower right corner, will not be divided.

[0097] As described above, in Figure 10, block 10 is divided into 13 variable-sized blocks 11-23 based on a recursive quad-tree and binary tree block partition. This type of partition is sometimes called a QTBT (quad-tree plus binary tree) partition.

[0098] In Figure 10, one block was divided into four or two blocks (quadrutree or binary tree block partitioning), but partitioning is not limited to these. For example, one block may be divided into three blocks (ternary tree block partitioning). Partitioning that includes such ternary tree block partitioning is sometimes called MBT (multi-type tree) partitioning.

[0099] Figure 11 is a block diagram showing an example of the functional configuration of the division unit 102 according to the embodiment. As shown in Figure 11, the division unit 102 may include a block division determination unit 102a. The block division determination unit 102a may perform the following processing as an example.

[0100] The block division determination unit 102a may, for example, acquire or read block information from the block memory 118 and / or the frame memory 122, and determine a division pattern (for example, the division pattern described above) based on that block information. The division unit 102 divides the original image according to the division pattern and outputs one or more blocks obtained by the division to the subtraction unit 104.

[0101] Furthermore, the block division determination unit 102a outputs one or more parameters indicating the determined division pattern (for example, the division pattern described above) to the transformation unit 106, the inverse transformation unit 114, the intra prediction unit 124, the inter prediction unit 126, and the entropy coding unit 110. The transformation unit 106 may transform the prediction residual based on the one or more parameters, and the intra prediction unit 124 and the inter prediction unit 126 may generate a prediction image based on the one or more parameters. The entropy coding unit 110 may also perform entropy coding on the one or more parameters.

[0102] The parameters related to the splitting pattern may be written to the stream as follows, for example.

[0103] Figure 12 is a conceptual diagram showing examples of division patterns. Division patterns include, for example, quadripartition (QT), which divides a block into two horizontally and two vertically; tripartition (HT or VT), which divides a block in the same direction in a 1:2:1 ratio; duplicate division (HB or VB), which divides a block in the same direction in a 1:1 ratio; and no division (NS).

[0104] Note that in the case of 4 divisions and no division, the division pattern does not have a block division direction, while in the case of 2 divisions and 3 divisions, the division pattern has division direction information.

[0105] Figure 13A is a conceptual diagram showing an example of a syntax tree for a partitioning pattern.

[0106] Figure 13B is a conceptual diagram showing another example of a syntax tree for a partitioning pattern.

[0107] Figures 13A and 13B show examples of syntax trees for splitting patterns. In the example in Figure 13A, first there is information indicating whether or not to split (S: Split flag), then there is information indicating whether or not to split into four (QT: QT flag). Next there is information indicating whether to split into three or two (TT: TT flag or BT: BT flag), and there is information indicating the direction of splitting (Ver: Vertical flag or Hor: Horizontal flag). Note that the same splitting process may be repeatedly applied to each of the one or more blocks obtained by splitting using such a splitting pattern. That is, as an example, the determination of whether or not to split, whether or not to split into four, whether the splitting method is horizontal or vertical, and whether to split into three or two may be performed recursively, and the results of the determinations performed may be encoded into a stream according to the encoding order disclosed in the syntax tree shown in Figure 13A.

[0108] Furthermore, in the syntax tree shown in Figure 13A, the information is arranged in the order of S, QT, TT, and Ver, but it may also be arranged in the order of S, QT, Ver, and BT. In other words, in the example in Figure 13B, first there is information indicating whether or not to perform a split (S: Split flag), then there is information indicating whether or not to perform a four-way split (QT: QT flag). Next there is information indicating the direction of the split (Ver: Vertical flag or Hor: Horizontal flag), and then there is information indicating whether to perform a two-way split or a three-way split (BT: BT flag or TT: TT flag).

[0109] Note that the division patterns described here are just examples; you may use other division patterns, or only a part of the division patterns described.

[0110] [Subtraction Unit] The subtraction unit 104 subtracts the predicted image (predicted samples input from the prediction control unit 128, described later) from the original image in block units that are input from the division unit 102 and divided by the division unit 102. In other words, the subtraction unit 104 calculates the predicted residual (also called the error) of the current block. The subtraction unit 104 then outputs the calculated predicted residual to the conversion unit 106.

[0111] The source image may also be an image input to the encoding device 100 as signals representing the images of each picture that make up the moving image (for example, a luminance (luma) signal and two chroma (chroma) signals). The signals representing the images may also be called samples.

[0112] [Conversion section] The conversion unit 106 converts the predicted residual in the spatial domain into conversion coefficients in the frequency domain and outputs the conversion coefficients to the quantization unit 108. Specifically, the conversion unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the predicted residual in the spatial domain, for example. The predetermined DCT or DST may be predetermined.

[0113] The transformation unit 106 may adaptively select a transformation type from among several transformation types and use a transformation basis function corresponding to the selected transformation type to convert the predicted residuals into transformation coefficients. Such a transformation is sometimes called an EMT (explicit multiple core transform) or an AMT (adaptive multiple transform). The transformation basis function is also sometimes called a basis.

[0114] Multiple transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. These transformation types may also be denoted as DCT2, DCT5, DCT8, DST1, and DST7, respectively. Figure 14 is a table showing examples of transformation basis functions corresponding to each transformation type. In Figure 14, N represents the number of input pixels. The selection of a transformation type from among these multiple transformation types may depend, for example, on the type of prediction (e.g., intra-prediction and inter-prediction) or on the intra-prediction mode.

[0115] Information indicating whether or not to apply such EMT or AMT (e.g., called an EMT flag or AMT flag) and information indicating the selected conversion type are typically signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0116] Furthermore, the transformation unit 106 may retransform the transformation coefficients (i.e., the transformation result). Such retransformation is sometimes called AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transformation unit 106 performs retransformation for each subblock (e.g., a 4x4 pixel subblock) contained in the block of transformation coefficients corresponding to the intra-predicted residual. Information indicating whether or not to apply NSST and information regarding the transformation matrix used for NSST are usually signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0117] The transformation unit 106 may be subjected to either a separable transformation or a non-separable transformation. A separable transformation is a method in which the input is separated into directions equal to the number of dimensions and transformed multiple times, while a non-separable transformation is a method in which, when the input is multidimensional, two or more dimensions are treated as one dimension and transformed together.

[0118] For example, one example of a non-separable transformation is to treat a 4x4 pixel block as a single array with 16 elements and then perform a transformation on that array using a 16x16 transformation matrix.

[0119] Another example of a non-separable transformation is a Hypercube Givens Transform, which treats a 4x4 pixel input block as a single array with 16 elements and then performs multiple Givens rotations on that array.

[0120] In the conversion unit 106, the conversion type of the conversion basis function used to convert to the frequency domain can be switched depending on the region within the CU. One example is SVT (Spatially Varying Transform).

[0121] Figure 15 is a conceptual diagram showing an example of SVT.

[0122] In SVT, as shown in Figure 15, the CU is divided into two equal parts horizontally or vertically, and only one of the regions is converted to the frequency domain. The conversion type may be set for each region, for example, DST7 and DCT8 can be used. For example, of the two regions obtained by dividing the CU vertically, DST7 and DCT8 may be used for the region at position 0. Or, of those two regions, DST7 is used for the region at position 1. Similarly, of the two regions obtained by dividing the CU horizontally, DST7 and DCT8 are used for the region at position 0. Or, of those two regions, DST7 is used for the region at position 1. In this example shown in Figure 15, only one of the two regions in the CU is converted, and the other is not, but it is also possible to convert both regions. Furthermore, the division method may be not only bisecting but also quartet-dividing. It is also possible to make it more flexible by encoding information indicating the division method and signaling it in the same way as CU division. Note that SVT is sometimes also called SBT (Sub-block Transform).

[0123] The aforementioned AMT and EMT may also be called MTS (Multiple Transform Selection). When applying MTS, a transformation type such as DST7 or DCT8 can be selected, and information indicating the selected transformation type may be encoded as index information for each CU. On the other hand, there is a process called IMTS (Implicit MTS) that selects the transformation type to be used for orthogonal transformation without encoding the index information. When applying IMTS, for example, if the shape of the CU is rectangular, the shorter side of the rectangle may be orthogonally transformed using DST7 and the longer side using DCT2. Also, for example, if the shape of the CU is square, if MTS is enabled in the sequence, DCT2 may be used, and if MTS is disabled, DST7 may be used to perform the orthogonal transformation. DCT2 and DST7 are just examples, and other transformation types may be used, and different combinations of transformation types can be used. IMTS may be usable only in intra-prediction blocks, or it may be usable in both intra-prediction blocks and inter-prediction blocks.

[0124] The above describes three selection processes, MTS, SBT, and IMTS, which selectively switch the transformation type used for orthogonal transformation. However, all three selection processes may be applied, or only some of them may be applied selectively. Whether to apply one or more selection processes can be identified, for example, by flag information in the header such as SPS. For example, if all three selection processes are available, one of the three selection processes is selected for each CU to perform the orthogonal transformation. Note that the selection process for selectively switching the transformation type may be a different selection process from the three selection processes described above, or each of the three selection processes may be replaced with another process. Generally, at least one of the following four functions [1] to [4] is performed. Function [1] is a function that orthogonally transforms the entire range within the CU and encodes information indicating the transformation type used for the transformation. Function [2] is a function that orthogonally transforms the entire range within the CU and determines the transformation type based on predetermined rules without encoding information indicating the transformation type. Function [3] is a function that orthogonally transforms a portion of the CU and encodes information indicating the transformation type used for the transformation. Function [4] is a function that orthogonally transforms a portion of the CU and determines the transformation type based on predetermined rules without encoding information indicating the transformation type used. The predetermined rules may be defined in advance.

[0125] The application of MTS, IMTS, and / or SBT may be determined on a per-processing-unit basis. For example, the application may be determined on a per-sequence, per-picture, per-brick, per-slice, per-CTU, or per-CU basis.

[0126] Furthermore, the tool for selectively switching the conversion type in this disclosure may be rephrased as a method for adaptively selecting a base to be used in the conversion process, a selection process, or a process for selecting a base. Alternatively, the tool for selectively switching the conversion type may be rephrased as a mode for adaptively selecting the conversion type.

[0127] Figure 16 is a flowchart showing an example of processing by the conversion unit 106, and for convenience, it will be explained with reference to Figure 7.

[0128] For example, the transformation unit 106 determines whether or not to perform an orthogonal transformation (step St_1). If the transformation unit 106 determines to perform an orthogonal transformation (Yes in step St_1), it selects a transformation type to be used for the orthogonal transformation from among multiple transformation types (step St_2). Next, the transformation unit 106 performs the orthogonal transformation by applying the selected transformation type to the predicted residual of the current block (step St_3). Then, the transformation unit 106 outputs information indicating the selected transformation type to the entropy encoding unit 110, thereby encoding that information (step St_4). On the other hand, if the transformation unit 106 determines not to perform an orthogonal transformation (No in step St_1), it outputs information indicating that an orthogonal transformation will not be performed to the entropy encoding unit 110, thereby encoding that information (step St_5). Note that the determination of whether or not to perform an orthogonal transformation in step St_1 may be based on, for example, the size of the transformation block, the prediction mode applied to the CU, etc. Also, the information indicating the transformation type to be used for the orthogonal transformation may not be encoded, and the orthogonal transformation may be performed using a defined transformation type. The specified conversion type may be predetermined.

[0129] Figure 17 is a flowchart showing an example of processing by the conversion unit 106, and for convenience, it will be explained with reference to Figure 7. The example shown in Figure 17 is an example of an orthogonal transformation in which a method of selectively switching the transformation type used for the orthogonal transformation is applied, similar to the example shown in Figure 16.

[0130] For example, the first group of conversion types may include DCT2, DST7, and DCT8. Also, for example, the second group of conversion types may include DCT2. Furthermore, the conversion types included in the first group of conversion types and the second group of conversion types may overlap in some respects, or they may all be different conversion types.

[0131] The transformation unit 106 determines whether the transformation size is less than or equal to a predetermined value (step Su_1). If it determines that it is less than or equal to the predetermined value (Yes in step Su_1), the transformation unit 106 orthogonally transforms the predicted residual of the current block using the transformation types included in the first transformation type group (step Su_2). Next, the transformation unit 106 outputs information to the entropy encoding unit 110 indicating which of the one or more transformation types included in the first transformation type group will be used, thereby encoding that information (step Su_3). On the other hand, if the transformation unit 106 determines that the transformation size is not less than or equal to the predetermined value (No in step Su_1), it orthogonally transforms the predicted residual of the current block using the second transformation type group (step Su_4). The predetermined value may be a threshold value or a predetermined value.

[0132] In step Su_3, the information indicating the transformation type used for the orthogonal transformation may be information indicating a combination of transformation types applied vertically and horizontally to the current block. Also, the first group of transformation types may contain only one transformation type, and the information indicating the transformation type used for the orthogonal transformation does not need to be encoded. The second group of transformation types may contain multiple transformation types, and the information indicating the transformation type used for the orthogonal transformation from one or more transformation types included in the second group of transformation types may be encoded.

[0133] Furthermore, the transformation type may be indicated based on the transformation size without encoding information indicating the transformation type. Note that the process of determining the transformation type to be used for orthogonal transformations based on the transformation size is not limited to determining whether the transformation size is less than or equal to a predetermined value.

[0134] [Quantization section] The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization unit 108 scans a plurality of conversion coefficients in the current block in a predetermined scan order and quantizes the conversion coefficients based on the quantization parameter (QP) corresponding to the scanned conversion coefficients. The quantization unit 108 then outputs the plurality of quantized conversion coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy coding unit 110 and the inverse quantization unit 112. The predetermined scan order may be set in advance.

[0135] A predetermined scan order is the order for quantization / inverse quantization of the conversion coefficients. For example, a predetermined scan order may be defined as ascending frequency (from low to high frequency) or descending frequency (from high to low frequency).

[0136] The quantization parameter (QP) is a parameter that defines the quantization step (quantization width). For example, if the value of the quantization parameter increases, the quantization step also increases. In other words, if the value of the quantization parameter increases, the error in the quantization coefficient (quantization error) increases.

[0137] Furthermore, quantization matrices may be used for quantization. For example, several types of quantization matrices may be used corresponding to frequency conversion sizes such as 4x4 and 8x8, prediction modes such as intra-prediction and inter-prediction, and pixel components such as luminance and chrominance. Note that quantization refers to the process of digitizing values ​​sampled at predetermined intervals and associating them with predetermined levels. In this technical field, it may also be referred to using other expressions such as rounding, scaling, and rounding, or rounding, rounding, and scaling may be employed. The predetermined intervals and levels may be predetermined.

[0138] There are two methods for using the quantization matrix: using a quantization matrix directly set on the encoding device 100, and using a default quantization matrix (default matrix). On the encoding device 100, by directly setting the quantization matrix, a quantization matrix corresponding to the image features can be set. However, in this case, there may be a disadvantage in that the amount of code increases due to the encoding of the quantization matrix. Furthermore, instead of using the default quantization matrix or the encoded quantization matrix as is, a quantization matrix to be used for quantizing the current block may be generated based on the default quantization matrix or the encoded quantization matrix.

[0139] On the other hand, there is also a method that quantizes the coefficients of the high-frequency and low-frequency components without using a quantization matrix. This method can be considered equivalent to the method that uses a quantization matrix (a flat matrix) where the coefficients have the same value.

[0140] The quantization matrix may be encoded at, for example, the sequence level, picture level, slice level, brick level, or CTU level. The quantization matrix may also be specified, for example, by an SPS (Sequence Parameter Set) or a PPS (Picture Parameter Set). An SPS contains the parameters used for the sequence, and a PPS contains the parameters used for the picture. SPS and PPS are sometimes simply referred to as parameter sets.

[0141] When a quantization matrix is ​​used, the quantization unit 108 scales the quantization width, which is obtained from quantization parameters, for each conversion coefficient using the values ​​of the quantization matrix. A quantization process performed without a quantization matrix may be a process that quantizes the conversion coefficients based on the quantization width obtained from quantization parameters. In a quantization process performed without a quantization matrix, the quantization width may be multiplied by a predetermined value common to all conversion coefficients in a block. This predetermined value may be set in advance.

[0142] Figure 18 is a block diagram showing an example of the functional configuration of the quantization unit according to the embodiment. The quantization unit 108 includes, for example, a differential quantization parameter generation unit 108a, a prediction quantization parameter generation unit 108b, a quantization parameter generation unit 108c, a quantization parameter storage unit 108d, and a quantization processing unit 108e.

[0143] Figure 19 is a flowchart showing an example of the quantization process performed by the quantization unit 108, and for convenience, it will be explained with reference to Figures 7 and 18.

[0144] For example, the quantization unit 108 may perform quantization for each CU based on the flowchart shown in Figure 19. Specifically, the quantization parameter generation unit 108c determines whether or not to perform quantization (step Sv_1). If it is determined that quantization should be performed (Yes in step Sv_1), the quantization parameter generation unit 108c generates quantization parameters for the current block (step Sv_2) and stores these quantization parameters in the quantization parameter storage unit 108d (step Sv_3).

[0145] Next, the quantization processing unit 108e quantizes the conversion coefficients of the current block using the quantization parameters generated in step Sv_2 (step Sv_4). Then, the prediction quantization parameter generation unit 108b obtains quantization parameters for a processing unit different from the current block from the quantization parameter storage unit 108d (step Sv_5). Based on the obtained quantization parameters, the prediction quantization parameter generation unit 108b generates prediction quantization parameters for the current block (step Sv_6). The difference quantization parameter generation unit 108a calculates the difference between the quantization parameters of the current block generated by the quantization parameter generation unit 108c and the prediction quantization parameters of the current block generated by the prediction quantization parameter generation unit 108b (step Sv_7). Difference quantization parameters may be generated by calculating this difference. The differential quantization parameter generation unit 108a outputs the differential quantization parameter to the entropy encoding unit 110, thereby encoding the differential quantization parameter (step Sv_8).

[0146] The differential quantization parameters may be encoded at the sequence level, picture level, slice level, brick level, or CTU level. The initial values ​​of the quantization parameters may also be encoded at the sequence level, picture level, slice level, brick level, or CTU level. In this case, the quantization parameters may be generated using the initial values ​​of the quantization parameters and the differential quantization parameters.

[0147] The quantization unit 108 may be equipped with multiple quantizers, and may also apply dependent quantization, which quantizes the conversion coefficients using a quantization method selected from multiple quantization methods.

[0148] [Entropy coding unit] Figure 20 is a block diagram showing an example of the functional configuration of the entropy coding unit 110, and for convenience, it will be explained with reference to Figure 7. The entropy coding unit 110 generates a stream by performing entropy coding on the quantization coefficients input from the quantization unit 108 and the prediction parameters input from the prediction parameter generation unit 130. For example, CABAC (Context-based Adaptive Binary Arithmetic Coding) is used for this entropy coding. Specifically, the entropy coding unit 110 comprises, for example, a binarization unit 110a, a context control unit 110b, and a binary arithmetic coding unit 110c. The binarization unit 110a performs binarization, converting multi-level signals such as quantization coefficients and prediction parameters into binary signals. For example, the binarization method is Truncated Rice Binarization, Exponential Examples include Golomb codes and Fixed Length Binarization. The context control unit 110b derives context values, i.e., the probability of a binary signal occurring, according to the characteristics of the syntax elements or the surrounding circumstances. Methods for deriving these context values ​​include, for example, bypass, syntax element referencing, upper and left adjacent block referencing, hierarchical information referencing, and others. The binary arithmetic coding unit 110c performs arithmetic coding on the binarized signal using the derived context values.

[0149] Figure 21 is a conceptual diagram showing an example of the CABAC processing flow in the entropy coding unit 110. First, initialization is performed in the CABAC in the entropy coding unit 110. This initialization involves initialization in the binary arithmetic coding unit 110c and setting of an initial context value. The binarization unit 110a and the binary arithmetic coding unit 110c may then sequentially perform binarization and arithmetic coding for each of the multiple quantization coefficients of the CTU, for example. The context control unit 110b may update the context value each time arithmetic coding is performed. The context control unit 110b may then save the context value as a post-processing step. This saved context value may be used, for example, as the initial value of the context value for the next CTU.

[0150] [Dequantization section] The inverse quantization unit 112 inversely quantizes the quantization coefficients input from the quantization unit 108. Specifically, the inverse quantization unit 112 inversely quantizes the quantization coefficients of the current block in a predetermined scanning order. Then, the inverse quantization unit 112 outputs the inversely quantized conversion coefficients of the current block to the inverse conversion unit 114. The predetermined scanning order may be set in advance.

[0151] [Inverse Transformation Section] The inverse transform unit 114 restores the predicted residual by performing an inverse transform on the transformation coefficients input from the inverse quantization unit 112. Specifically, the inverse transform unit 114 restores the predicted residual of the current block by performing an inverse transform on the transformation coefficients corresponding to the transformation by the transformation unit 106. The inverse transform unit 114 then outputs the restored predicted residual to the summation unit 116.

[0152] Furthermore, the recovered prediction residuals usually do not match the prediction residuals calculated by the subtraction unit 104 because information is typically lost due to quantization. In other words, the recovered prediction residuals usually contain quantization errors.

[0153] [Addition section] The adder 116 reconstructs the current block by adding the predicted residual input from the inverse transformer 114 and the predicted image input from the prediction control unit 128. As a result, a reconstructed image is generated. The adder 116 then outputs the reconstructed image to the block memory 118 and the loop filter unit 120. The reconstructed block is sometimes called a local decoded block.

[0154] [Block memory] The block memory 118 is a storage unit for storing blocks within the current picture used, for example, in intra prediction. Specifically, the block memory 118 stores the reconstructed image output from the adder 116.

[0155] [Frame memory] The frame memory 122 is a storage unit for storing reference pictures used, for example, in interpretation, and is sometimes called a frame buffer. Specifically, the frame memory 122 stores the reconstructed image filtered by the loop filter unit 120.

[0156] [Loop Filter Section] The loop filter unit 120 applies loop filtering to the reconstructed image output from the adder unit 116 and outputs the filtered reconstructed image to the frame memory 122. A loop filter is a filter used within the encoding loop (in-loop filter), and includes, for example, adaptive loop filters (ALF), deblocking filters (DF or DBF), and sample adaptive offset (SAO) filters.

[0157] Figure 22 is a block diagram showing an example of the functional configuration of the loop filter unit 120 according to the embodiment. The loop filter unit 120 comprises, for example, a deblocking filter processing unit 120a, an SAO processing unit 120b, and an ALF processing unit 120c, as shown in Figure 22. The deblocking filter processing unit 120a applies the above-described deblocking filter processing to the reconstructed image. The SAO processing unit 120b applies the above-described SAO processing to the reconstructed image after the deblocking filter processing. The ALF processing unit 120c applies the above-described ALF processing to the reconstructed image after the SAO processing. Details of ALF and deblocking filters will be described later. SAO processing is a process that improves image quality by reducing ringing (a phenomenon in which pixel values ​​are distorted in a wave-like manner around edges) and correcting the shift in pixel values. Examples of SAO processing include edge offset processing and band offset processing. Furthermore, the loop filter unit 120 does not necessarily have to include all the processing units disclosed in Figure 22; it may include some of the processing units, or it may include additional processing units. Also, the loop filter unit 120 may be configured to perform the above-mentioned processing in an order different from the processing order disclosed in Figure 22, and it does not have to perform all of the processing.

[0158] [Loop Filter Section > Adaptive Loop Filter] In ALF, a least-squares error filter is applied to remove coding distortion. For example, for each 2x2 pixel subblock within the current block, one filter selected from several filters is applied based on the direction and activity of the local gradient.

[0159] Specifically, first, subblocks (e.g., 2x2 pixel subblocks) are classified into multiple classes (e.g., 15 or 25 classes). The classification of subblocks may be based, for example, on the direction and activity of the gradient. In a specific example, a classification value C (e.g., C = 5D + A) is calculated using the gradient direction value D (e.g., 0 to 2 or 0 to 4) and the gradient activity value A (e.g., 0 to 4). Then, based on the classification value C, the subblocks are classified into multiple classes.

[0160] The gradient direction value D is derived, for example, by comparing gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). The gradient activation value A is derived, for example, by adding the gradients in multiple directions and quantizing the sum.

[0161] Based on the results of this classification, a filter for a subblock may be determined from among multiple filters.

[0162] For example, circularly symmetrical shapes are used as filter shapes in ALF. Figures 23A to 23C are conceptual diagrams showing several examples of filter shapes used in ALF. Figure 23A shows a 5x5 diamond-shaped filter, Figure 23B shows a 7x7 diamond-shaped filter, and Figure 23C shows a 9x9 diamond-shaped filter. Information indicating the filter shape is usually signaled at the picture level. However, the signaling of information indicating the filter shape is not limited to the picture level and may be at other levels (e.g., sequence level, slice level, tile level, CTU level, or CU level).

[0163] The ALF (Automatic Laser Level) on / off status may be determined, for example, at the picture level or CU (Camera Unit) level. For example, the decision to apply ALF to luminance may be made at the CU level, and the decision to apply ALF to color difference may be made at the picture level. Information indicating whether ALF is on or off is usually signaled at the picture level or CU level. However, the signaling of information indicating whether ALF is on or off is not limited to the picture level or CU level, but may be at other levels (e.g., sequence level, slice level, tile level, or CTU level).

[0164] Furthermore, as described above, one filter is selected from among several filters and the ALF process is applied to the subblock. For each of these filters (for example, up to 15 or 25 filters), the set of coefficients used in that filter is usually signaled at the picture level. However, the signaling of the coefficient set is not limited to the picture level; it may be at other levels (for example, sequence level, slice level, tile level, CTU level, CU level, or subblock level).

[0165] [Loop Filters > Cross Component Adaptive Loop Filter] Figure 23D is a conceptual diagram showing an example of a CC-ALF (cross component ALF) flow. Figure 23E is a conceptual diagram showing an example of a filter shape used in CC-ALF, such as the CC-ALF in Figure 23D. One example of CC-ALF in Figures 23D and 23E operates by applying a linear diamond-shaped filter to the luminance channel of each chromatic difference component. For example, the filter coefficients are transmitted in APS, scaled by a factor of 2^10, and rounded for fixed-point representation. For example, in Figure 23D, a Y sample (first component) is used for the CCALF of Cb and the CCALF of Cr (components different from the first component).

[0166] The application of the filter may be controlled by a variable block size and notified by a context-encoded flag received for each sample block. The block size and CC-ALF enable flag may be received at the slice level for each chrominance component. CC-ALF supports various block sizes, e.g., 16x16, 32x32, 64x64, and 128x128 (for chrominance samples).

[0167] [Loop Filters > Joint Chroma Cross Component Adaptive Loop Filter] An example of a combined chrominance CCALF is shown in Figures 23F and 23G. Figure 23F is a diagram illustrating an example of the combined chrominance CCALF flow. Figure 23G is a table showing an example of a weight_index candidate. As illustrated, one CCALF filter is used to generate one CCALF filter output as a chrominance adjustment signal for one color component, and a weighted version of the same chrominance adjustment signal is applied to the other color component. In this way, the complexity of the existing CCALF is roughly halved. The weight values ​​may be encoded into a sign flag and a weight index. The weight index (indicated as weight_index) may be encoded in 3 bits and specify the magnitude of a non-zero JC-CCALF weight JcCcWeight. The magnitude of JcCcWeight may be determined, for example, as follows:

[0168] If weight_index is 4 or less, JcCcWeight is equal to weight_index>>2.

[0169] Otherwise, JcCcWeight is equal to 4 / (weight_index-4).

[0170] The block-level on / off control of the ALF filtering for Cb and Cr may be separate. This is the same as CCALF, and two separate sets of block-level on / off control flags may be encoded. Here, unlike CCALF, since the on / off control block sizes of Cb and Cr are the same, only one block size variable may be encoded.

[0171] [Loop Filter Section > Deblocking Filter] In the deblocking filter process, the loop filter section 120 reduces the distortion generated at the block boundary of the reconstructed image by performing filter processing on the block boundary.

[0172] FIG. 24 is a block diagram showing an example of the detailed configuration of the deblocking filter processing section 120a of the loop filter 120 (see FIGS. 7 and 22) that functions as a deblocking filter.

[0173] The deblocking filter processing section 120a includes, for example, a boundary determination section 1201, a filter determination section 1203, a filter processing section 1205, a processing determination section 1208, a filter characteristic determination section 1207, and switches 1202, 1204, and 1206.

[0174] The boundary determination section 1201 determines whether the pixel to be deblocking-filtered (i.e., the target pixel) exists near the block boundary. Then, the boundary determination section 1201 outputs the determination result to the switches 1202 and the processing determination section 1208.

[0175] If the boundary determination unit 1201 determines that the target pixel is located near a block boundary, switch 1202 outputs the image before filtering to switch 1204. Conversely, if the boundary determination unit 1201 determines that the target pixel is not located near a block boundary, switch 1202 outputs the image before filtering to switch 1206. The image before filtering consists of the target pixel and at least one surrounding pixel located around that target pixel.

[0176] The filter determination unit 1203 determines whether or not to perform a deblocking filter on the target pixel based on the pixel values ​​of at least one surrounding pixel located around the target pixel. The filter determination unit 1203 then outputs the determination result to the switch 1204 and the processing determination unit 1208.

[0177] If the filter determination unit 1203 determines that deblocking filtering should be performed on the target pixel, switch 1204 outputs the pre-filtered image acquired via switch 1202 to the filter processing unit 1205. Conversely, if the filter determination unit 1203 determines that deblocking filtering should not be performed on the target pixel, switch 1204 outputs the pre-filtered image acquired via switch 1202 to switch 1206.

[0178] When the filter processing unit 1205 acquires an image before filtering via switches 1202 and 1204, it performs a deblocking filter process on the target pixel, using the filter characteristics determined by the filter characteristic determination unit 1207. The filter processing unit 1205 then outputs the filtered pixel to switch 1206.

[0179] Switch 1206 selectively outputs pixels that have not undergone deblocking and filtering, and pixels that have undergone deblocking and filtering by the filtering processing unit 1205, in accordance with the control by the processing determination unit 1208.

[0180] The processing determination unit 1208 controls the switch 1206 based on the determination results of the boundary determination unit 1201 and the filter determination unit 1203. Specifically, if the boundary determination unit 1201 determines that a target pixel is near a block boundary, and the filter determination unit 1203 determines that the target pixel should undergo deblocking and filtering, the processing determination unit 1208 outputs the deblocked and filtered pixel from the switch 1206. In other cases, the processing determination unit 1208 outputs the unfiltered pixel from the switch 1206. This output of pixels is repeated, and the filtered image is output from the switch 1206. Note that the configuration shown in Figure 24 is just one example of the configuration of the deblocking and filtering processing unit 120a, and the deblocking and filtering processing unit 120a may have various configurations.

[0181] Figure 25 is a conceptual diagram showing an example of a deblocking filter with symmetrical filter characteristics with respect to block boundaries.

[0182] In deblocking filtering, for example, one of two deblocking filters with different characteristics, namely a strong filter and a weak filter, may be selected using pixel values ​​and quantization parameters. In a strong filter, as shown in Figure 25, if pixels p0~p2 and pixels q0~q2 exist on either side of a block boundary, the respective pixel values ​​of pixels q0~q2 are changed to pixel values ​​q'0~q'2 by performing the operation shown in the following equation, for example.

[0183] q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8 q'1=(p0+q0+q1+q2+2) / 4 q'2=(p0+q0+q1+3×q2+2×q3+4) / 8

[0184] In the above equations, p0~p2 and q0~q2 are the pixel values ​​of pixels p0~p2 and pixels q0~q2, respectively. Also, q3 is the pixel value of pixel q3, which is adjacent to pixel q2 on the opposite side of the block boundary. Furthermore, the coefficient multiplied by the pixel value of each pixel used in the deblocking filter process on the right-hand side of each of the above equations is the filter coefficient.

[0185] Furthermore, in the deblocking filter process, clipping may be performed to ensure that the calculated pixel values ​​do not exceed a threshold. For example, in this clipping process, the calculated pixel values ​​according to the above formula may be clipped to "pre-calculation pixel value ± 2 × threshold" using a threshold determined from the quantization parameters. This prevents excessive smoothing.

[0186] Figure 26 is a conceptual diagram illustrating an example of a block boundary where deblocking filtering is performed. Figure 27 is a conceptual diagram showing an example of a Boundary Strength (BS) value.

[0187] The block boundaries on which deblocking filtering is performed are, for example, the CU, PU, ​​or TU boundaries of an 8x8 pixel block as shown in Figure 26. Deblocking filtering may be performed in units of, for example, 4 rows or 4 columns. First, for blocks P and Q shown in Figure 26, the Bs (Boundary Strength) value is determined as shown in Figure 27.

[0188] According to the Bs value in Figure 27, it may be determined whether or not to perform deblocking filtering of different strengths, even for block boundaries belonging to the same image. Deblocking filtering is performed on the chrominance signal when the Bs value is 2. Deblocking filtering is performed on the luminance signal when the Bs value is 1 or greater and predetermined conditions are met. These predetermined conditions may be set in advance. Note that the criteria for determining the Bs value are not limited to those shown in Figure 27 and may be determined based on other parameters.

[0189] [Prediction Unit (Intra Prediction Unit, Inter Prediction Unit, Prediction Control Unit)] FIG. 28 is a flowchart showing an example of the processing performed by the prediction unit of the encoding apparatus 100. Note that the prediction unit includes all or some of the components of the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128. The prediction processing unit includes, for example, the intra prediction unit 124 and the inter prediction unit 126.

[0190] The prediction unit generates a predicted image of the current block (step Sb_1). The predicted image may be referred to as a prediction signal or a prediction block. Note that the prediction signal includes, for example, an intra prediction image (intra prediction signal) or an inter prediction image (inter prediction signal). The prediction unit generates a predicted image of the current block using a reconstructed image that has already been obtained by performing generation of a predicted image for other blocks, generation of a prediction residual, generation of quantization coefficients, restoration of the prediction residual, and addition of the predicted image.

[0191] The reconstructed image may be, for example, an image of a reference picture, or an image of an encoded block (i.e., the above-described other block) in the current picture that is the picture including the current block. The encoded block in the current picture is, for example, an adjacent block of the current block.

[0192] FIG. 29 is a flowchart showing another example of the processing performed by the prediction unit of the encoding apparatus 100.

[0193] The prediction unit generates a predicted image in a first method (step Sc_1a), generates a predicted image in a second method (step Sc_1b), and generates a predicted image in a third method (step Sc_1c). The first method, the second method, and the third method are different methods for generating a predicted image, and may be, for example, an inter prediction method, an intra prediction method, and other prediction methods, respectively. In these prediction methods, the above-described reconstructed image may be used.

[0194] Next, the prediction unit evaluates the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_2). For example, the prediction unit calculates a cost C for each of the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c, and evaluates the predicted images by comparing their costs C. Note that the cost C may be calculated using the formula of the RD optimization model, for example, C = D + λ × R. In this formula, D is the coding distortion of the predicted image, and can be expressed, for example, as the sum of the absolute differences between the pixel values ​​of the current block and the pixel values ​​of the predicted image. R is the bitrate of the stream. λ is, for example, the Lagrange multiplier.

[0195] Next, the prediction unit selects one of the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_3). In other words, the prediction unit selects a method or mode to obtain the final predicted image. For example, the prediction unit selects the predicted image with the smallest cost C based on the cost C calculated for those predicted images. Alternatively, the evaluation in step Sc_2 and the selection of the predicted image in step Sc_3 may be based on parameters used in the coding process. The coding device 100 may signal information to identify the selected predicted image, method, or mode into a stream. This information may be, for example, a flag. This allows the decoding device 200 to generate a predicted image according to the method or mode selected by the coding device 100 based on this information. In the example shown in Figure 29, the prediction unit selects one of the predicted images after generating a predicted image for each method. However, the prediction unit may select a method or mode based on the parameters used in the coding process described above before generating those predicted images, and then generate the predicted image according to that method or mode.

[0196] For example, the first method and the second method are intra-prediction and inter-prediction, respectively, and the prediction unit may select the final predicted image for the current block from the predicted images generated according to these prediction methods.

[0197] Figure 30 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device 100.

[0198] First, the prediction unit generates a predicted image by intra-prediction (step Sd_1a) and then generates a predicted image by inter-prediction (step Sd_1b). The predicted image generated by intra-prediction is also called the intra-prediction image, and the predicted image generated by inter-prediction is also called the inter-prediction image.

[0199] Next, the prediction unit evaluates both the intra-predicted image and the inter-predicted image (step Sd_2). The cost C mentioned above may be used for this evaluation. The prediction unit may then select the prediction image with the smallest cost C from the intra-predicted image and the inter-predicted image as the final prediction image for the current block (step Sd_3). In other words, a prediction method or mode for generating the prediction image for the current block is selected.

[0200] The prediction unit then selects the prediction image with the smallest cost C from the intra-predicted image and inter-predicted image as the final prediction image for the current block (step Sd_3). In other words, a prediction method or mode for generating the prediction image for the current block is selected.

[0201] [Intra Prediction Unit] The intra-prediction unit 124 generates a prediction signal (i.e., an intra-predicted image) for the current block by performing intra-prediction (also called in-screen prediction) of the current block by referring to the block in the current picture stored in the block memory 118. Specifically, the intra-prediction unit 124 generates an intra-predicted image by performing intra-prediction by referring to the pixel values ​​(e.g., luminance value, chrominance value) of the block adjacent to the current block, and outputs the intra-predicted image to the prediction control unit 128.

[0202] For example, the intra-prediction unit 124 performs intra-prediction using one of a defined intra-prediction mode. The defined intra-prediction modes typically include one or more non-directional prediction modes and multiple directional prediction modes. The defined modes may be predetermined.

[0203] One or more non-directional prediction modes include, for example, the Planar prediction mode and DC prediction mode as defined in the H.265 / HEVC (high-efficiency video coding) standard.

[0204] Multiple directional prediction modes include, for example, the 33 prediction modes defined in the H.265 / HEVC standard. Note that multiple directional prediction modes may also include 32 additional prediction modes (a total of 65 directional prediction modes). Figure 31 is a conceptual diagram showing all 67 intra-prediction modes (2 non-directional prediction modes and 65 directional prediction modes) in intra-prediction. Solid arrows represent the 33 directions defined in the H.265 / HEVC standard, and dashed arrows represent the additional 32 directions (the 2 non-directional prediction modes are not shown in Figure 31).

[0205] In various processing examples, luminance blocks may be referenced in the intra-prediction of chrominance blocks. That is, the chrominance components of the current block may be predicted based on the luminance components of the current block. Such intra-prediction is sometimes called CCLM (cross-component linear model) prediction. Such an intra-prediction mode for chrominance blocks that references luminance blocks (e.g., called CCLM mode) may be added as one of the intra-prediction modes for chrominance blocks.

[0206] The intra-prediction unit 124 may correct the pixel values ​​after intra-prediction based on the gradient of the horizontal / vertical reference pixels. Intra-prediction with such correction is sometimes called PDPC (position dependent intra-prediction combination). Information indicating whether or not PDPC is applied (e.g., called a PDPC flag) is usually signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0207] Figure 32 is a flowchart showing an example of processing by the intra prediction unit 124.

[0208] The intra-prediction unit 124 selects one intra-prediction mode from a plurality of intra-prediction modes (step Sw_1). Then, the intra-prediction unit 124 generates a predicted image according to the selected intra-prediction mode (step Sw_2). Next, the intra-prediction unit 124 determines the MPM (Most Probable Modes) (step Sw_3). The MPM consists of, for example, six intra-prediction modes. For example, two of the six intra-prediction modes may be a Planar prediction mode and a DC prediction mode, and the remaining four modes may be directional prediction modes. Then, the intra-prediction unit 124 determines whether the intra-prediction mode selected in step Sw_1 is included in the MPM (step Sw_4).

[0209] If the selected intra-prediction mode is determined to be included in the MPM (Yes in step Sw_4), the intra-prediction unit 124 sets the MPM flag to 1 (step Sw_5) and generates information from the MPM indicating the selected intra-prediction mode (step Sw_6). Note that the MPM flag set to 1 and the information indicating the intra-prediction mode may each be encoded as prediction parameters by the entropy encoding unit 110.

[0210] On the other hand, if the intra-prediction unit 124 determines that the selected intra-prediction mode is not included in the MPM (No in step Sw_4), it sets the MPM flag to 0 (step Sw_7). Alternatively, the intra-prediction unit 124 does not set the MPM flag. Then, the intra-prediction unit 124 generates information indicating the selected intra-prediction mode from among one or more intra-prediction modes not included in the MPM (step Sw_8). Note that the MPM flag set to 0 and the information indicating the intra-prediction mode may each be encoded as prediction parameters by the entropy coding unit 110. The information indicating the intra-prediction mode may, for example, represent a value between 0 and 60.

[0211] [International Prediction Department] The inter-prediction unit 126 generates a predicted image (inter-predicted image) by performing inter-prediction (also called inter-screen prediction) of the current block by referring to a reference picture stored in the frame memory 122 that is different from the current picture. Inter-prediction is performed in units of the current block or the current subblock within the current block (for example, a 4x4 block). A subblock is contained within a block and is a smaller unit than a block. The size of a subblock may be in the form of a slice, brick, or picture.

[0212] For example, the interpretation unit 126 performs motion estimation within a reference picture for the current block or current subblock and finds the reference block or subblock that best matches the current block or current subblock. The interpretation unit 126 then obtains motion information (e.g., motion vectors) that compensates for the movement or change from the reference block or subblock to the current block or subblock. Based on this motion information, the interpretation unit 126 performs motion compensation (or motion prediction) and generates an interpretation prediction image of the current block or subblock. The interpretation unit 126 outputs the generated interpretation prediction image to the prediction control unit 128.

[0213] The motion information used for motion compensation may be signaled as an interpretation signal in various forms. For example, the motion vector may be signaled. Another example is that the difference between the motion vector and the predicted motion vector (motion vector predictor) may be signaled.

[0214] [Reference Picture List] Figure 33 is a conceptual diagram showing an example of each reference picture, and Figure 34 is a conceptual diagram showing an example of a reference picture list. The reference picture list is a list showing one or more reference pictures stored in the frame memory 122. In Figure 33, rectangles represent pictures, arrows indicate the reference relationships between pictures, the horizontal axis represents time, I, P, and B in the rectangles represent intra-prediction pictures, single-prediction pictures, and double-prediction pictures, respectively, and the numbers in the rectangles indicate the decoding order. As shown in Figure 33, the decoding order of each picture is I0, P1, B2, B3, B4, and the display order of each picture is I0, B3, B2, B4, P1. As shown in Figure 34, the reference picture list is a list representing candidate reference pictures, and for example, one picture (or slice) may have one or more reference picture lists. For example, if the current picture is a single-prediction picture, one reference picture list is used, and if the current picture is a double-prediction picture, two reference picture lists are used. In the examples in Figures 33 and 34, picture B3, which is the current picture currPic, has two reference picture lists, the L0 list and the L1 list. When the current picture currPic is picture B3, the candidate reference pictures for that current picture currPic are I0, P1, and B2, and each reference picture list (i.e., the L0 list and the L1 list) indicates these pictures. The interpretation unit 126 or the prediction control unit 128 specifies which picture in each reference picture list to actually reference using the reference picture index refIdxLx. In Figure 34, reference pictures P1 and B2 are specified by the reference picture indices refIdxL0 and refIdxL1.

[0215] Such reference picture lists may be generated on a sequence, picture, slice, brick, CTU, or CU basis. Furthermore, the reference picture index indicating the reference picture referenced in interpretation among the reference pictures shown in the reference picture list may be encoded at the sequence, picture, slice, brick, CTU, or CU level. Additionally, a common reference picture list may be used across multiple interpretation modes.

[0216] [Basic flow of interpretation] Figure 35 is a flowchart showing an example of the basic flow of the interpretation prediction process.

[0217] The interpretation unit 126 first generates a predicted image (steps Se_1 to Se_3). Next, the subtraction unit 104 generates the difference between the current block and the predicted image as the predicted residual (step Se_4).

[0218] Here, the interpretation unit 126 generates a predicted image by determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and performing motion compensation (step Se_3). In determining the MV, the interpretation unit 126 determines the MV by selecting a candidate motion vector (candidate MV) (step Se_1) and deriving the MV (step Se_2). The selection of a candidate MV is performed, for example, by the interpretation unit 126 generating a candidate MV list and selecting at least one candidate MV from the candidate MV list. Note that previously derived MVs may be added as candidate MVs to the candidate MV list. In the MV derivation, the interpretation unit 126 may determine the selected at least one candidate MV as the MV of the current block by further selecting at least one candidate MV from at least one candidate MV. Alternatively, the interpretation unit 126 may determine the MV of the current block by searching the region of the reference picture indicated by each of the selected candidate MVs. This search of the reference picture region may be called motion estimation.

[0219] Furthermore, in the example described above, steps Se_1 to Se_3 are performed by the interpretation unit 126, but processing such as step Se_1 or step Se_2 may be performed by other components included in the encoding device 100.

[0220] Furthermore, a candidate MV list may be created for each process in each interpretation mode, or a common candidate MV list may be used across multiple interpretation modes. Also, the processes in steps Se_3 and Se_4 correspond to the processes in steps Sa_3 and Sa_4 shown in Figure 9, respectively. In addition, the process in step Se_3 corresponds to the process in step Sd_1b in Figure 30.

[0221] [MV Derivation Flow] Figure 36 is a flowchart showing an example of the MV derivation process.

[0222] The interpretation unit 126 may derive the MV of the current block in a mode that encodes motion information (e.g., MV). In this case, for example, the motion information may be encoded as prediction parameters and signaled. That is, the encoded motion information is included in the stream.

[0223] Alternatively, the interpretation unit 126 may derive MV in a mode that does not encode motion information. In this case, motion information is not included in the stream.

[0224] Here, the modes for MV derivation include the normal inter mode, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that encode motion information include the normal inter mode, normal merge mode, and affine mode (specifically, the affine inter mode and affine merge mode). Note that the motion information may include not only the MV but also the predicted MV selection information, which will be described later. Modes that do not encode motion information include the FRUC mode, etc. The inter prediction unit 126 selects a mode from these multiple modes for deriving the MV of the current block and derives the MV of the current block using the selected mode.

[0225] Figure 37 is a flowchart showing another example of MV derivation.

[0226] The interpretation unit 126 may derive the MV of the current block in a mode that encodes the differential MV. In this case, for example, the differential MV may be encoded as a prediction parameter and signaled. That is, the encoded differential MV is included in the stream. This differential MV is the difference between the MV of the current block and its predicted MV. The predicted MV is the predicted motion vector.

[0227] Alternatively, the interpretation unit 126 may derive the MV in a mode that does not encode the differential MV. In this case, the encoded differential MV is not included in the stream.

[0228] As mentioned above, the modes for MV derivation include the normal inter, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that encode differential MV include the normal inter mode and affine mode (specifically, affine inter mode). Modes that do not encode differential MV include the FRUC mode, normal merge mode, and affine mode (specifically, affine merge mode). The inter prediction unit 126 selects a mode from these multiple modes to derive the MV of the current block, and uses the selected mode to derive the MV of the current block.

[0229] [Modes for MV derivation] Figures 38A and 38B are conceptual diagrams illustrating an example of the classification of each mode of MV derivation. For example, as shown in Figure 38A, the modes of MV derivation can be broadly classified into three modes depending on whether or not motion information is encoded and whether or not differential MV is encoded. The three modes are intermode, merge mode, and FRUC (frame rate up-conversion) mode. Intermode is a mode that performs motion search and encodes motion information and differential MV. For example, as shown in Figure 38B, intermode includes affine intermode and normal intermode. Merge mode is a mode that does not perform motion search and selects an MV from a surrounding encoded block and uses that MV to derive the MV of the current block. This merge mode basically encodes motion information and does not encode differential MV. For example, as shown in Figure 38B, merge mode includes normal merge mode (sometimes called regular merge mode or normal merge mode), MMVD (Merge with Motion Vector Difference) mode, and CIIP (Combined This includes inter-merge / intra-prediction mode, triangle mode, ATMVP mode, and affine merge mode. In the MMVD mode, one of the modes included in merge mode, the differential MV is exceptionally encoded. The aforementioned affine merge mode and affine inter-mode are modes included in affine mode. Affine mode is a mode that assumes an affine transformation and derives the MV of each of the multiple sub-blocks constituting the current block as the MV of the current block. FRUC mode is a mode that derives the MV of the current block by performing a search between encoded regions, and does not encode either motion information or differential MV. Details of each of these modes will be described later.

[0230] Note that the classification of each mode shown in Figures 38A and 38B is just an example and is not limited to this classification. For example, if a differential MV is encoded in CIIP mode, that CIIP mode is classified as an intermode.

[0231] [MV Derivation > Normal Intermode] The normal intermode is an interpretation mode that derives the MV of the current block based on blocks similar to the image of the current block, using the region of the reference picture indicated by the candidate MV. In this normal intermode, the differential MV is also encoded.

[0232] Figure 39 is a flowchart showing an example of inter-prediction processing in normal inter-mode.

[0233] The interpretation unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks surrounding the current block in time or space (step Sg_1). In other words, the interpretation unit 126 creates a candidate MV list.

[0234] Next, the interpretation unit 126 extracts N candidate MVs (where N is an integer greater than or equal to 2) from among the multiple candidate MVs obtained in step Sg_1, each of them as a predicted motion vector candidate (also called a predicted MV candidate), according to a predetermined priority order (step Sg_2). Note that this priority order may be predetermined for each of the N candidate MVs.

[0235] Next, the interpretation unit 126 selects one predicted MV candidate from among the N predicted MV candidates as the predicted motion vector (also called the predicted MV) for the current block (step Sg_3). At this time, the interpretation unit 126 encodes predicted MV selection information into a stream to identify the selected predicted MV. In other words, the interpretation unit 126 outputs the predicted MV selection information as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0236] Next, the interpretation unit 126 refers to the encoded reference picture and derives the MV of the current block (step Sg_4). At this time, the interpretation unit 126 further encodes the difference between the derived MV and the predicted MV as the difference MV into the stream. In other words, the interpretation unit 126 outputs the difference MV as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130. The encoded reference picture is a picture consisting of multiple blocks that have been reconstructed after encoding.

[0237] Finally, the inter-prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Sg_5). The processes in steps Sg_1 to Sg_5 are performed for each block. For example, when the processes in steps Sg_1 to Sg_5 are performed for all blocks in a slice, the inter-prediction using the normal inter-mode for that slice is completed. Similarly, when the processes in steps Sg_1 to Sg_5 are performed for all blocks in a picture, the inter-prediction using the normal inter-mode for that picture is completed. Note that the processes in steps Sg_1 to Sg_5 are not performed for all blocks in a slice, but only for some blocks, in which case the inter-prediction using the normal inter-mode for that slice may be completed. The same applies to the processes in steps Sg_1 to Sg_5. When the processes in steps Sg_1 to Sg_5 are performed for some blocks in a picture, the inter-prediction using the normal inter-mode for that picture may be completed.

[0238] The predicted image is the interprediction signal described above. Furthermore, information indicating the interprediction mode used to generate the predicted image (normal intermode in the example above), which is included in the encoded signal, is encoded, for example, as a prediction parameter.

[0239] The candidate MV list may be the same as the list used in other modes. Furthermore, processing related to the candidate MV list may be applied to processing related to lists used in other modes. This processing related to the candidate MV list may include, for example, extracting or selecting candidate MVs from the candidate MV list, rearranging candidate MVs, or deleting candidate MVs.

[0240] [MV Derivation > Normal Merge Mode] Normal merge mode is an interpretation mode that derives an MV by selecting a candidate MV from a candidate MV list as the MV of the current block. Note that normal merge mode is a type of merge mode and is sometimes simply called merge mode. In this embodiment, normal merge mode and merge mode are distinguished, and merge mode is used in a broad sense.

[0241] Figure 40 is a flowchart showing an example of inter prediction using normal merge mode.

[0242] The interpretation unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks surrounding the current block in time or space (step Sh_1). In other words, the interpretation unit 126 creates a candidate MV list.

[0243] Next, the interpretation unit 126 derives the MV of the current block by selecting one candidate MV from among the multiple candidate MVs obtained in step Sh_1 (step Sh_2). At this time, the interpretation unit 126 encodes MV selection information to identify the selected candidate MV into a stream. In other words, the interpretation unit 126 outputs the MV selection information as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0244] Finally, the inter prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Sh_3). The processes in steps Sh_1 to Sh_3 are performed for each block, for example. For example, when the processes in steps Sh_1 to Sh_3 are performed for all blocks included in a slice, the inter prediction using normal merge mode for that slice is completed. Similarly, when the processes in steps Sh_1 to Sh_3 are performed for all blocks included in a picture, the inter prediction using normal merge mode for that picture is completed. Note that the processes in steps Sh_1 to Sh_3 are not performed for all blocks included in a slice, but only for some blocks, in which case the inter prediction using normal merge mode for that slice may be completed. The same applies to the processes in steps Sh_1 to Sh_3. When the processes in steps Sh_1 to Sh_3 are performed for some blocks included in a picture, the inter prediction using normal merge mode for that picture may be completed.

[0245] Furthermore, information indicating the inter-prediction mode used to generate the predicted image (normal merge mode in the example above), which is included in the encoded signal, is encoded in the stream, for example, as prediction parameters.

[0246] Figure 41 is a conceptual diagram illustrating an example of the MV derivation process for the current picture using normal merge mode.

[0247] First, the interpretation unit 126 generates a candidate MV list in which candidate MVs are registered. Candidate MVs include spatially adjacent candidate MVs, which are MVs of multiple encoded blocks located spatially around the current block; temporally adjacent candidate MVs, which are MVs of nearby blocks projected onto the current block's position in the encoded reference picture; combined candidate MVs, which are MVs generated by combining the MV values ​​of spatially adjacent candidate MVs and temporally adjacent candidate MVs; and zero candidate MVs, which are MVs with a value of zero.

[0248] Next, the interpretation unit 126 selects one candidate MV from among the multiple candidate MVs registered in the candidate MV list, thereby determining that one candidate MV as the MV of the current block.

[0249] Furthermore, the entropy coding unit 110 writes and encodes merge_idx, a signal indicating which candidate MV was selected, into a stream.

[0250] Note that the candidate MVs registered in the candidate MV list explained in Figure 41 are just examples, and the number of candidates may differ from the number shown in the figure, the configuration may not include some of the types of candidate MVs shown in the figure, or it may include candidate MVs other than those shown in the figure.

[0251] The final MV may be determined by performing DMVR (dynamic motion vector refreshing), described later, using the MV of the current block derived by the normal merge mode. In normal merge mode, motion information is encoded, but the differential MV is not. MMVD mode selects one candidate MV from a list of candidate MVs, similar to normal merge mode, but encodes the differential MV. Such MMVD may be classified as a merge mode along with normal merge mode, as shown in Figure 38B. The differential MV in MMVD mode does not have to be the same as the differential MV used in inter-mode; for example, the derivation of the differential MV in MMVD mode may be a less computationally intensive process than the derivation of the differential MV in inter-mode.

[0252] Alternatively, a CIIP (Combined inter merge / intra prediction) mode may be used to generate a prediction image for the current block by overlaying the prediction image generated by inter prediction with the prediction image generated by intra prediction.

[0253] The candidate MV list may also be referred to simply as the candidate list. Furthermore, merge_idx is the MV selection information.

[0254] [MV Derivation > HMVP Mode] Figure 42 is a conceptual diagram illustrating an example of the MV derivation process for the current picture using HMVP mode.

[0255] In normal merge mode, the MV of the current block (e.g., CU) is determined by selecting one candidate MV from the MV list generated by referencing the encoded block (e.g., CU). Other candidate MVs may be added to this candidate MV list. This mode, where other candidate MVs are added, is called HMVP mode.

[0256] In HMVP mode, candidate MVs are managed using a separate FIFO (First-In First-Out) server for HMVP, in addition to the candidate MV list used in normal merge mode.

[0257] The FIFO buffer stores motion information such as MVs of blocks that have been processed in the past, in reverse chronological order. In the management of this FIFO buffer, each time a block is processed, the MV of the most recent block (i.e., the most recently processed CU) is stored in the FIFO buffer, and in its place, the MV of the oldest CU (i.e., the most recently processed CU) in the FIFO buffer is removed. In the example shown in Figure 42, HMVP1 is the MV of the most recent block, and HMVP5 is the MV of the oldest block.

[0258] For example, the interpretation unit 126 checks each MV managed in the FIFO buffer, starting with HMVP1, whether that MV is different from all the candidate MVs already registered in the candidate MV list for normal merge mode. If the interpretation unit 126 determines that it is different from all the candidate MVs, it may add the MV managed in the FIFO buffer as a candidate MV to the candidate MV list for normal merge mode. At this time, one or more candidate MVs in the FIFO buffer may be registered (added to the candidate MV list).

[0259] By using HMVP mode in this way, it becomes possible to include not only MVs of spatially or temporally adjacent blocks to the current block, but also MVs of previously processed blocks as candidates. As a result, the variety of candidate MVs in normal merge mode is broadened, which increases the likelihood of improving encoding efficiency.

[0260] Furthermore, the aforementioned MV may also be motion information. In other words, the information stored in the candidate MV list and FIFO buffer may include not only the MV value, but also information indicating the referenced picture, the direction and number of references, etc. Also, the aforementioned block may be, for example, a CU.

[0261] Note that the candidate MV list and FIFO buffer in Figure 42 are just examples, and the candidate MV list and FIFO buffer may be lists or buffers of different sizes than those in Figure 42, or the candidate MVs may be registered in a different order than those in Figure 42. Also, the processing described here may be common to both the encoding device 100 and the decoding device 200.

[0262] Furthermore, HMVP mode can be applied to modes other than normal merge mode. For example, motion information such as MV of blocks previously processed in affine mode can be stored in the FIFO buffer in chronological order from newest to oldest and used as candidate MV, potentially improving efficiency. A mode in which HMVP mode is applied to affine mode may be called history affine mode.

[0263] [MV Derivation > FRUC Mode] Motion information may be derived on the decoding side without being signaled from the encoding side. For example, motion information may be derived by performing a motion search on the decoding side 200. In this embodiment, the decoding side performs a motion search without using the pixel values ​​of the current block. Modes in which the decoding side 200 performs a motion search without using the pixel values ​​of the current block include FRUC (frame rate up-conversion) mode or PMMVD (pattern matched motion vector derivation) mode.

[0264] Figure 43 shows an example of FRUC processing in flowchart format. First, a list is generated that refers to the MVs of each encoded block spatially or temporally adjacent to the current block and indicates those MVs as candidate MVs (i.e., a candidate MV list, which may be the same as the candidate MV list for normal merge mode) (step Si_1).

[0265] Next, the best candidate MV is selected from among the multiple candidate MVs registered in the candidate MV list (step Si_2). For example, an evaluation value is calculated for each candidate MV included in the candidate MV list, and one candidate MV is selected based on that evaluation value. Then, based on the selected candidate MV, the MV for the current block is derived (step Si_4). Specifically, for example, the selected candidate MV (best candidate MV) is derived as the MV for the current block. Alternatively, for example, the MV for the current block may be derived by performing pattern matching in the area surrounding the location in the reference picture that corresponds to the location in the reference picture of the selected candidate MV. That is, a search using pattern matching and evaluation values ​​is performed in the area surrounding the best candidate MV, and if there is an MV with a better evaluation value, the best candidate MV may be updated to that MV and made the final MV for the current block. In one embodiment, it is not necessary to update to an MV with a better evaluation value.

[0266] Finally, the inter-prediction unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the encoded reference picture (step Si_5). The processes in steps Si_1 to Si_5 are performed for each block, for example. For example, when the processes in steps Si_1 to Si_5 are performed for all blocks included in a slice, the inter-prediction using FRUC mode for that slice is completed. Similarly, when the processes in steps Si_1 to Si_5 are performed for all blocks included in a picture, the inter-prediction using FRUC mode for that picture is completed. Note that the processes in steps Si_1 to Si_5 are not performed for all blocks included in a slice, but only for some blocks, in which case the inter-prediction using FRUC mode for that slice is completed. Likewise, when the processes in steps Si_1 to Si_5 are performed for some blocks included in a picture, the inter-prediction using FRUC mode for that picture is completed.

[0267] Similar processing may be performed at the subblock level.

[0268] The evaluation value may be calculated by various methods. For example, the reconstructed image of a region in the reference picture corresponding to the MV is compared with the reconstructed image of a predetermined region (which may be, for example, a region in another reference picture or a region in an adjacent block of the current picture, as shown below). The predetermined region may be set in advance.

[0269] Furthermore, the difference in pixel values ​​between the two reconstructed images may be calculated and used as the evaluation value for MV. Alternatively, the evaluation value may be calculated using other information in addition to the difference value.

[0270] Next, we will explain in detail an example of pattern matching. First, one candidate MV included in the candidate MV list (e.g., merge list) is selected as the starting point for the search using pattern matching. For example, either first-order pattern matching or second-order pattern matching may be used. First-order pattern matching and second-order pattern matching are sometimes called bilateral matching and template matching, respectively.

[0271] [MV Derivation > FRUC > Bilateral Matching] In the first pattern matching, pattern matching is performed between two blocks in two different reference pictures that are aligned with the motion trajectory of the current block. Therefore, in the first pattern matching, a region in another reference picture aligned with the motion trajectory of the current block is used as a predetermined region for calculating the evaluation value of the candidate MV described above. This predetermined region may be set in advance.

[0272] Figure 44 is a conceptual diagram illustrating an example of first pattern matching (bilateral matching) between two blocks in two reference pictures along a motion trajectory. As shown in Figure 44, in first pattern matching, two MVs (MV0, MV1) are derived by searching for the best-matching pair of blocks in two different reference pictures (Ref0, Ref1) that are along the motion trajectory of the current block. Specifically, for the current block, the difference is derived between the reconstructed image at a specified position in the first encoded reference picture (Ref0) specified by the candidate MV and the reconstructed image at a specified position in the second encoded reference picture (Ref1) specified by the symmetric MV obtained by scaling the candidate MV by the display time interval. An evaluation value is then calculated using the obtained difference value. Among the multiple candidate MVs, the candidate MV with the best evaluation value can be selected as the final MV, potentially yielding good results.

[0273] Under the assumption of a continuous motion trajectory, the MV(MV0, MV1) pointing to two reference blocks is proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, if the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the two reference pictures are equal, then the first pattern matching derives a mirror-symmetric bidirectional MV.

[0274] [MV Derivation > FRUC > Template Matching] In the second pattern matching (template matching), pattern matching is performed between the template in the current picture (blocks adjacent to the current block in the current picture (e.g., blocks above and / or to the left)) and the blocks in the reference picture. Therefore, in the second pattern matching, the blocks adjacent to the current block in the current picture are used as a predetermined area for calculating the evaluation value of the candidate MV described above.

[0275] Figure 45 is a conceptual diagram illustrating an example of pattern matching (template matching) between a template in the current picture and a block in the reference picture. As shown in Figure 45, in the second pattern matching, the MV of the current block is derived by searching in the reference picture (Ref0) for the block that best matches the block adjacent to the current block in the current picture (Cur Pic). Specifically, for the current block, the difference between the reconstructed image of both or either of the left-adjacent and upper-adjacent encoded regions and the reconstructed image at the equivalent position in the encoded reference picture (Ref0) specified by the candidate MV is derived, and an evaluation value is calculated using the obtained difference value. The candidate MV with the best evaluation value among multiple candidate MVs may be selected as the best candidate MV.

[0276] Information indicating whether or not to apply such a FRUC mode (e.g., called a FRUC flag) may be signaled at the CU level. Furthermore, if FRUC mode is applied (e.g., if the FRUC flag is true), information indicating the applicable pattern matching method (first pattern matching or second pattern matching) may be signaled at the CU level. Note that the signaling of this information is not limited to the CU level, but may be at other levels (e.g., sequence level, picture level, slice level, tile level, CTU level, or subblock level).

[0277] [MV Derivation > Affine Mode] The affine mode is a mode in which the MV is generated using an affine transform. For example, the MV may be derived on a subblock basis based on the MVs of multiple adjacent blocks. This mode is sometimes called the affine motion compensation prediction mode.

[0278] Figure 46A is a conceptual diagram illustrating an example of deriving the MV of a subblock based on the MV of multiple adjacent blocks. In Figure 46A, the current block contains, for example, 16 subblocks consisting of 4x4 pixels. Here, the motion vector v0 of the upper left corner control point of the current block is derived based on the MV of the adjacent blocks, and similarly, the motion vector v1 of the upper right corner control point of the current block is derived based on the MV of the adjacent subblock. Then, by projecting the two motion vectors v0 and v1 using the following equation (1A), the motion vector (v) of each subblock within the current block is obtained. x ,v y ) may be derived.

[0279]

number

[0280] Here, x and y represent the horizontal and vertical positions of the subblock, respectively, and w represents a predetermined weighting coefficient. The predetermined weighting coefficient may be set in advance.

[0281] Information indicating such affine modes (e.g., called an affine flag) may be signaled at the CU level. However, the signaling of this information indicating affine modes is not limited to the CU level, but may be at other levels (e.g., sequence level, picture level, slice level, tile level, CTU level, or subblock level).

[0282] Furthermore, such affine modes may include several modes with different methods for deriving the MV of the upper-left and upper-right corner control points. For example, there are two affine modes: the affine inter (also called the affine normal inter) mode and the affine merge mode.

[0283] Figure 46B is a conceptual diagram illustrating an example of deriving the motion vector (MV) for a subblock unit in affine mode using three control points. In Figure 46B, the current block includes, for example, a subblock consisting of 16 4x4 pixels. Here, the motion vector v0 of the upper left corner control point of the current block is derived based on the MV of the adjacent block. Similarly, the motion vector v1 of the upper right corner control point of the current block is derived based on the MV of the adjacent block, and the motion vector v2 of the lower left corner control point of the current block is derived based on the MV of the adjacent block. Then, by projecting the three motion vectors v0, v1, and v2 using the following equation (1B), the motion vector (v) of each subblock within the current block is obtained. x ,v y ) may be derived.

[0284]

number

[0285] Here, x and y represent the horizontal and vertical positions of the subblock center, respectively, and w and h represent weighting coefficients, which may be predetermined weighting coefficients. In this embodiment, w may represent the width of the current block, and h may represent the height of the current block.

[0286] Affine modes using different numbers of control points (e.g., two and three) may be switched and signaled at the CU level. Information indicating the number of control points for the affine modes used at the CU level may also be signaled at other levels (e.g., sequence level, picture level, slice level, tile level, CTU level, or subblock level).

[0287] Furthermore, an affine mode having three control points may include several modes in which the MV of the upper left, upper right, and lower left corner control points is derived in different ways. For example, an affine mode having three control points has two modes, similar to the affine mode having two control points described above: an affine intermode and an affine merge mode.

[0288] In affine mode, the size of each subblock included in the current block is not limited to 4x4 pixels; it can be any other size. For example, each subblock may be 8x8 pixels.

[0289] [MV Derivation > Affine Mode > Control Point] Figures 47A, 47B, and 47C are conceptual diagrams illustrating an example of MV derivation of control points in affine mode.

[0290] In affine mode, as shown in Figure 47A, the predicted MV for each control point of the current block is calculated based on multiple MVs corresponding to blocks encoded in affine mode among the encoded blocks adjacent to the current block: Block A (left), Block B (top), Block C (upper right), Block D (lower left), and Block E (upper left). Specifically, these blocks are examined in the order of encoded Block A (left), Block B (top), Block C (upper right), Block D (lower left), and Block E (upper left), and the first valid block encoded in affine mode is identified. Based on the multiple MVs corresponding to this identified block, the predicted MV for the control point of the current block is calculated.

[0291] For example, as shown in Figure 47B, if block A, which is adjacent to the left of the current block, is encoded in affine mode with two control points, then motion vectors v3 and v4 are derived, projected onto the upper-left and upper-right corners of the encoded block containing block A. Then, from the derived motion vectors v3 and v4, the motion vector v0 of the upper-left corner control point and the motion vector v1 of the upper-right corner control point of the current block are calculated.

[0292] For example, as shown in Figure 47C, if block A adjacent to the left of the current block is encoded in affine mode with three control points, motion vectors v3, v4, and v5 are derived projected onto the upper-left, upper-right, and lower-left corners of the encoded block containing block A. Then, from the derived motion vectors v3, v4, and v5, the motion vector v0 of the upper-left corner control point, the motion vector v1 of the upper-right corner control point, and the motion vector v2 of the lower-left corner control point of the current block are calculated.

[0293] The MV derivation method shown in Figures 47A to 47C may be used to derive the MV of each control point in the current block in step Sk_1 shown in Figure 50, or it may be used to derive the predicted MV of each control point in the current block in step Sj_1 shown in Figure 51, which will be described later.

[0294] Figures 48A and 48B are conceptual diagrams illustrating another example of the derivation of the control point MV in affine mode.

[0295] Figure 48A is a conceptual diagram illustrating an example of an affine mode with two control points.

[0296] In this affine mode, as shown in Figure 48A, the MV selected from the respective MVs of the encoded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the upper left corner control point of the current block. Similarly, the MV selected from the respective MVs of the encoded blocks D and E adjacent to the current block is used as the motion vector v1 of the upper right corner control point of the current block.

[0297] Figure 48B is a conceptual diagram illustrating an example of an affine mode with three control points.

[0298] In this affine mode, as shown in Figure 48B, the MV selected from each of the MVs of the encoded blocks A, B, and C adjacent to the current block is used as the motion vector v0 of the upper left corner control point of the current block. Similarly, the MV selected from each of the MVs of the encoded blocks D and E adjacent to the current block is used as the motion vector v1 of the upper right corner control point of the current block. Furthermore, the MV selected from each of the MVs of the encoded blocks F and G adjacent to the current block is used as the motion vector v2 of the lower left corner control point of the current block.

[0299] The MV derivation methods shown in Figures 48A and 48B may be used to derive the MV of each control point in the current block in step Sk_1 shown in Figure 50, which will be described later, or they may be used to derive the predicted MV of each control point in the current block in step Sj_1 shown in Figure 51, which will be described later.

[0300] Here, for example, when switching between affine modes with different numbers of control points (e.g., two and three) at the CU level to generate signals, the number of control points may differ between the encoded block and the current block.

[0301] Figures 49A and 49B are conceptual diagrams illustrating an example of a method for deriving the control point MV when the number of control points differs between the encoded block and the current block.

[0302] For example, as shown in Figure 49A, the current block has three control points at the upper left corner, upper right corner, and lower left corner, and is encoded in affine mode with block A adjacent to the left of the current block having two control points. In this case, motion vectors v3 and v4 are derived by projecting them onto the upper left and upper right corner positions of the encoded block containing block A. Then, from the derived motion vectors v3 and v4, the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block are calculated. Furthermore, from the derived motion vectors v0 and v1, the motion vector v2 of the lower left corner control point is calculated.

[0303] For example, as shown in Figure 49B, the current block is encoded in affine mode, having two control points at the upper left and upper right corners, and block A, adjacent to the left of the current block, has three control points. In this case, motion vectors v3, v4, and v5 are derived by projecting them onto the upper left, upper right, and lower left corners of the encoded block containing block A. Then, from the derived motion vectors v3, v4, and v5, the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block are calculated.

[0304] The MV derivation methods shown in Figures 49A and 49B may be used to derive the MV of each control point in the current block in step Sk_1 shown in Figure 50, which will be described later, or they may be used to derive the predicted MV of each control point in the current block in step Sj_1 shown in Figure 51, which will be described later.

[0305] [MV Derivation > Affine Mode > Affine Merge Mode] Figure 50 is a flowchart showing an example of processing in affine merge mode.

[0306] In affine merge mode, the interpretation unit 126 first derives the MVs of each control point in the current block (step Sk_1). The control points are the upper left and upper right corners of the current block, as shown in Figure 46A, or the upper left, upper right, and lower left corners of the current block, as shown in Figure 46B. The interpretation unit 126 may encode MV selection information into a stream to identify two or three of the derived MVs.

[0307] For example, when using the MV derivation method shown in Figures 47A to 47C, the interpretation unit 126 examines the encoded blocks in the order of A (left), B (top), C (upper right), D (lower left), and E (upper left), as shown in Figure 47A, and identifies the first valid block encoded in affine mode.

[0308] The interpretation unit 126 derives the motion vectors (MV) of the control points using the first valid block encoded in the identified affine mode. For example, if block A is identified and block A has two control points, as shown in Figure 47B, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block from the motion vectors v3 and v4 of the upper left and upper right corners of the encoded block containing block A. For example, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block by projecting the motion vectors v3 and v4 of the upper left and upper right corners of the encoded block onto the current block.

[0309] Alternatively, if block A is identified and block A has three control points, as shown in Figure 47C, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block from the motion vectors v3, v4, and v5 of the upper left, upper right, and lower left corners of the encoded block containing block A. For example, the interpretation unit 126 calculates the motion vector v0 of the upper left corner control point, the motion vector v1 of the upper right corner control point, and the motion vector v2 of the lower left corner control point of the current block by projecting the motion vectors v3, v4, and v5 of the upper left, upper right, and lower left corners of the encoded block onto the current block.

[0310] Furthermore, as shown in Figure 49A above, if block A is identified and block A has two control points, the MV of three control points may be calculated. Alternatively, as shown in Figure 49B above, if block A is identified and block A has three control points, the MV of two control points may be calculated.

[0311] Next, the interpretation unit 126 performs motion compensation for each of the multiple subblocks contained in the current block. That is, for each of the multiple subblocks, the interpretation unit 126 calculates the MV of that subblock as an affine MV, for example, using two motion vectors v0 and v1 and equation (1A) described above, or using three motion vectors v0, v1 and v2 and equation (1B) described above (step Sk_2). Then, the interpretation unit 126 performs motion compensation for that subblock using those affine MVs and the encoded reference picture (step Sk_3). Once steps Sk_2 and Sk_3 have been executed for each of the subblocks contained in the current block, the process of generating a predicted image using the affine merge mode for that current block is completed. In other words, motion compensation is performed on the current block, and a predicted image for that current block is generated.

[0312] In step Sk_1, the above-mentioned candidate MV list may be generated. The candidate MV list may be a list containing candidate MVs derived for each control point using multiple MV derivation methods. The multiple MV derivation methods may be any combination of, for example, the MV derivation methods shown in Figures 47A to 47C, the MV derivation methods shown in Figures 48A and 48B, the MV derivation methods shown in Figures 49A and 49B, and other MV derivation methods.

[0313] The candidate MV list may also include candidate MVs from modes other than affine mode, where prediction is performed on a sub-block basis.

[0314] Furthermore, the candidate MV list may include, for example, candidate MVs for affine merge modes with two control points and candidate MVs for affine merge modes with three control points. Alternatively, a candidate MV list containing candidate MVs for affine merge modes with two control points and a candidate MV list containing candidate MVs for affine merge modes with three control points may be generated separately. Alternatively, a candidate MV list may be generated containing candidate MVs for one of the modes: affine merge mode with two control points and affine merge mode with three control points. The candidate MVs may be, for example, the MVs of encoded blocks A (left), B (top), C (upper right), D (lower left), and E (upper left), or they may be the MVs of valid blocks among those blocks.

[0315] Alternatively, you may send an index indicating which candidate MV from the candidate MV list is being selected as MV selection information.

[0316] [MV Derivation > Affine Mode > Affine Intermode] Figure 51 is a flowchart showing an example of affine intermode processing.

[0317] In affine intermode, the interpretation unit 126 first derives the predicted MV(v0,v1) or (v0,v1,v2) for each of two or three control points of the current block (step Sj_1). The control points are, for example, the upper left corner, upper right corner, or lower left corner of the current block, as shown in Figure 46A or Figure 46B.

[0318] For example, when using the MV derivation method shown in Figures 48A and 48B, the interpretation unit 126 derives the predicted MV (v0,v1) or (v0,v1,v2) of the control point of the current block by selecting the MV of one of the encoded blocks near each control point of the current block shown in Figure 48A or Figure 48B. At this time, the interpretation unit 126 encodes predicted MV selection information into a stream to identify the two or three selected predicted MVs.

[0319] For example, the interpretation unit 126 may determine which block's MV from the encoded blocks adjacent to the current block to select as the predicted MV for the control point using cost evaluation or the like, and write a flag indicating which predicted MV was selected to the bitstream. In other words, the interpretation unit 126 outputs the predicted MV selection information, such as a flag, as a prediction parameter to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0320] Next, the interpretation unit 126 performs motion search (steps Sj_3 and Sj_4) while updating the predicted MVs selected or derived in step Sj_1 (step Sj_2). That is, the interpretation unit 126 calculates the MV of each subblock corresponding to the updated predicted MV as an affine MV using the above-mentioned equation (1A) or equation (1B) (step Sj_3). Then, the interpretation unit 126 performs motion compensation for each subblock using these affine MVs and encoded reference pictures (step Sj_4). The processing in steps Sj_3 and Sj_4 is performed for all blocks in the current block each time the predicted MV is updated in step Sj_2. As a result, the interpretation unit 126 determines, for example, the predicted MV that yields the smallest cost in the motion search loop as the MV of the control point (step Sj_5). At this time, the interpretation unit 126 further encodes the difference between the determined MV and the predicted MV as the difference MV into a stream. In other words, the interpretation unit 126 outputs the difference MV as a prediction parameter to the entropy coding unit 110 via the prediction parameter generation unit 130.

[0321] Finally, the interpretation unit 126 generates a predicted image of the current block by performing motion compensation on the current block using the determined MV and the encoded reference picture (step Sj_6).

[0322] In step Sj_1, the above-mentioned candidate MV list may be generated. The candidate MV list may be a list containing candidate MVs derived for each control point using multiple MV derivation methods. The multiple MV derivation methods may be any combination of, for example, the MV derivation methods shown in Figures 47A to 47C, the MV derivation methods shown in Figures 48A and 48B, the MV derivation methods shown in Figures 49A and 49B, and other MV derivation methods.

[0323] The candidate MV list may also include candidate MVs from modes other than affine mode, where prediction is performed on a sub-block basis.

[0324] Furthermore, a candidate MV list may be generated that includes candidate MVs for affine intermodes with two control points and candidate MVs for affine intermodes with three control points. Alternatively, a candidate MV list containing candidate MVs for affine intermodes with two control points and a candidate MV list containing candidate MVs for affine intermodes with three control points may be generated separately. Alternatively, a candidate MV list may be generated that includes candidate MVs for one of the modes: affine intermodes with two control points and affine intermodes with three control points. The candidate MVs may be, for example, the MVs of encoded blocks A (left), B (top), C (upper right), D (lower left), and E (upper left), or they may be the MVs of valid blocks among those blocks.

[0325] Additionally, as predicted MV selection information, you may send an index indicating which candidate MV from the candidate MV list it is.

[0326] [MV Derivation > Triangle Mode] In the example described above, the interpretation unit 126 generates one rectangular prediction image for the current rectangular block. However, the interpretation unit 126 may generate multiple prediction images of shapes different from rectangles for the current rectangular block, and then combine these multiple prediction images to generate the final rectangular prediction image. The shapes different from rectangles may be, for example, triangles.

[0327] Figure 52A is a conceptual diagram illustrating the generation of two triangular prediction images.

[0328] The interpretation unit 126 generates a predicted triangular image by performing motion compensation on the first triangular partition within the current block using the first MV of that first partition. Similarly, the interpretation unit 126 generates a predicted triangular image by performing motion compensation on the second triangular partition within the current block using the second MV of that second partition. Then, the interpretation unit 126 combines these predicted images to generate a predicted rectangular image identical to that of the current block.

[0329] Furthermore, a first rectangular prediction image corresponding to the current block may be generated using the first MV as the prediction image for the first partition. Similarly, a second rectangular prediction image corresponding to the current block may be generated using the second MV as the prediction image for the second partition. The prediction image for the current block may also be generated by weighting and adding the first and second prediction images. Note that the weighting and addition may be applied only to a portion of the region straddling the boundary between the first and second partitions.

[0330] Figure 52B is a conceptual diagram showing an example of a first portion of a first partition overlapping with a second partition, as well as a first and second sample set that may be weighted as part of a correction process. The first portion may be, for example, one-quarter of the width or height of the first partition. In another example, the first portion may have a width corresponding to N samples adjacent to the edge of the first partition, where N is an integer greater than zero, for example, N may be the integer 2. Figure 52B shows a rectangular partition having a rectangular portion with a width of one-quarter of the width of the first partition. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples within the first portion. The central example in Figure 52B shows a rectangular partition having a rectangular portion with a height of one-quarter of the height of the first partition. Here, the first sample set includes samples outside the first portion and samples inside the first portion, and the second sample set includes samples within the first portion. The example on the right in Figure 52B shows a triangular partition having polygonal sections of height corresponding to two samples. Here, the first sample set includes samples outside the first section and samples inside the first section, and the second sample set includes samples within the first section.

[0331] The first portion may be the portion of the first partition that overlaps with an adjacent partition. Figure 52C is a conceptual diagram showing the first portion of the first partition, which is the portion of the first partition that overlaps with a portion of an adjacent partition. For simplicity of explanation, a rectangular partition is shown that has a portion that overlaps with a spatially adjacent rectangular partition. Partitions of other shapes, such as triangular partitions, may be used, and the overlapping portion may overlap with a spatially or temporally adjacent partition.

[0332] Furthermore, while an example is shown in which prediction images are generated for each of the two partitions using interpretation, prediction images may also be generated for at least one partition using intrapretation.

[0333] Figure 53 is a flowchart showing an example of triangle mode processing.

[0334] In triangle mode, the interpretation unit 126 first divides the current block into a first partition and a second partition (step Sx_1). At this time, the interpretation unit 126 may encode partition information, which is information about the division into each partition, into the stream as prediction parameters. In other words, the interpretation unit 126 may output partition information as prediction parameters to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0335] Next, the interpretation unit 126 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple encoded blocks surrounding the current block in time or space (step Sx_2). In other words, the interpretation unit 126 creates a candidate MV list.

[0336] Then, the interpretation unit 126 selects the candidate MVs for the first partition and the candidate MVs for the second partition from among the multiple candidate MVs obtained in step Sx_2 as the first MV and the second MV, respectively (step Sx_3). At this time, the interpretation unit 126 may encode MV selection information for identifying the selected candidate MVs into the stream as prediction parameters. In other words, the interpretation unit 126 may output the MV selection information as prediction parameters to the entropy encoding unit 110 via the prediction parameter generation unit 130.

[0337] Next, the interpretation unit 126 generates a first predicted image by performing motion compensation using the selected first MV and the encoded reference picture (step Sx_4). Similarly, the interpretation unit 126 generates a second predicted image by performing motion compensation using the selected second MV and the encoded reference picture (step Sx_5).

[0338] Finally, the interpretation unit 126 generates a predicted image of the current block by weighting and adding the first predicted image and the second predicted image (step Sx_6).

[0339] In the example shown in Figure 52A, the first and second partitions are triangular, but they may also be trapezoidal, or they may have different shapes from each other. Furthermore, in the example shown in Figure 52A, the current block is composed of two partitions, but it may be composed of three or more partitions.

[0340] Furthermore, the first and second partitions may overlap. That is, the first and second partitions may contain the same pixel region. In this case, the predicted image of the current block may be generated using the predicted image in the first partition and the predicted image in the second partition.

[0341] Furthermore, while this example shows that prediction images are generated by interpretation for both partitions, prediction images may also be generated by intrapretation for at least one partition.

[0342] Note that the candidate MV list for selecting the first MV and the candidate MV list for selecting the second MV may be different, or they may be the same.

[0343] The partition information may include an index indicating the partitioning direction for dividing the current block into multiple partitions. The MV selection information may include an index indicating the first selected MV and an index indicating the second selected MV. A single index may represent multiple pieces of information. For example, a single index may be encoded that represents part or all of the partition information and part or all of the MV selection information together.

[0344] [MV Derivation > ATMVP Mode] Figure 54 is a conceptual diagram showing an example of an ATMVP (Advanced Temporal Motion Vector Prediction) mode in which MV is derived at the subblock level.

[0345] ATMVP mode is a mode classified as a merge mode. For example, in ATMVP mode, candidate MVs at the subblock level are registered in the candidate MV list used in normal merge mode.

[0346] Specifically, in ATMVP mode, first, as shown in Figure 54, the time MV reference block associated with the current block is identified in the encoded reference picture specified by the MV (MV0) of the block adjacent to the lower left of the current block. Next, for each subblock within the current block, the MV used when encoding the region corresponding to that subblock within the time MV reference block is identified. The MVs identified in this way are included in the candidate MV list as candidate MVs for the subblocks of the current block. When a candidate MV for each subblock is selected from the candidate MV list, motion compensation using that candidate MV as the subblock's MV is performed for that subblock. This generates a predicted image for each subblock.

[0347] In the example shown in Figure 54, the block adjacent to the lower left of the current block was used as the peripheral MV reference block, but other blocks may be used. Also, the size of the subblock may be 4x4 pixels, 8x8 pixels, or other sizes. The size of the subblock may be switched in units such as slices, bricks, or pictures.

[0348] [Motion Detection > DMVR] Figure 55 is a flowchart showing the relationship between merge mode and DMVR (Decoder Motion Vector Refinement).

[0349] The interpretation unit 126 derives the MV of the current block in merge mode (step Sl_1). Next, the interpretation unit 126 determines whether or not to perform an MV search, i.e., a motion search (step Sl_2). If the interpretation unit 126 determines not to perform a motion search (No in step Sl_2), it determines the MV derived in step Sl_1 as the final MV for the current block (step Sl_4). In other words, in this case, the MV of the current block is determined in merge mode.

[0350] On the other hand, if it is determined in step Sl_1 to perform a motion search (Yes in step Sl_2), the interpretation unit 126 derives the final MV for the current block by searching the surrounding region of the reference picture indicated by the MV derived in step Sl_1 (step Sl_3). In other words, in this case, the MV for the current block is determined by the DMVR.

[0351] Figure 56 is a conceptual diagram illustrating an example of DMVR processing for determining MV.

[0352] First, for example in merge mode, candidate MVs (L0 and L1) are selected for the current block. Then, according to candidate MV (L0), reference pixels are identified from the first reference picture (L0), which is an encoded picture in the L0 list. Similarly, according to candidate MV (L1), reference pixels are identified from the second reference picture (L1), which is an encoded picture in the L1 list. A template is generated by taking the average of these reference pixels.

[0353] Next, using the template, the surrounding regions of candidate MVs for the first reference picture (L0) and the second reference picture (L1) are searched, and the MV with the minimum cost is determined as the final MV for the current block. The cost may be calculated, for example, using the difference between each pixel value in the template and each pixel value in the search region, as well as the candidate MV value.

[0354] Any other process that can explore the vicinity of a candidate MV and derive the final MV is acceptable, even if it is not the exact process described here.

[0355] Figure 57 is a conceptual diagram illustrating another example of a DMVR for determining MV. Unlike the DMVR example shown in Figure 56, this example in Figure 57 calculates the cost without generating a template.

[0356] First, the interpretation unit 126 searches around the reference blocks contained in the reference pictures of the L0 list and L1 list, respectively, based on the initial MV, which is a candidate MV obtained from the candidate MV list. For example, as shown in Figure 57, the initial MV corresponding to the reference block in the L0 list is InitMV_L0, and the initial MV corresponding to the reference block in the L1 list is InitMV_L1. In motion search, the interpretation unit 126 first sets the search position for the reference picture in the L0 list. The difference vector indicating the set search position, specifically the difference vector from the position indicated by the initial MV (i.e., InitMV_L0) to that search position, is MVd_L0. Then, the interpretation unit 126 determines the search position in the reference picture of the L1 list. This search position is indicated by the difference vector from the position indicated by the initial MV (i.e., InitMV_L1) to that search position. Specifically, the interpretation unit 126 determines this difference vector as MVd_L1 by mirroring MVd_L0. In other words, the interpretation unit 126 searches for a position that is symmetrical to the position indicated by the initial MV in the reference pictures of the L0 list and the L1 list. For each search position, the interpretation unit 126 calculates a cost such as the sum of the absolute differences in pixel values ​​within the block at that search position (SAD), and finds the search position that minimizes this cost.

[0357] Figure 58A is a conceptual diagram showing an example of motion search in a DMVR, and Figure 58B is a flowchart showing an example of that motion search process.

[0358] First, in Step 1, the interpretation unit 126 calculates the cost of the search position indicated by the initial MV (also called the starting point) and the eight surrounding search positions. Then, the interpretation unit 126 determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 126 determines that the cost of the search positions other than the starting point is the minimum, it moves to the search position with the minimum cost and proceeds to Step 2. On the other hand, if the cost of the starting point is the minimum, the interpretation unit 126 skips Step 2 and proceeds to Step 3.

[0359] In Step 2, the interpretation unit 126 uses the search position moved according to the processing result of Step 1 as a new starting point and performs the same search as in Step 1. The interpretation unit 126 then determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 126 determines that the cost of the search positions other than the starting point is the minimum, it proceeds to Step 4. On the other hand, if the interpretation unit 126 determines that the cost of the starting point is the minimum, it proceeds to Step 3.

[0360] In Step 4, the interpretation unit 126 treats the starting position as the final search position and determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0361] In Step 3, the interpretation unit 126 determines the pixel position with the minimum cost based on the costs at the four points above, below, to the left and right of the starting point in Step 1 or Step 2, and sets that pixel position as the final search position. This decimal-precision pixel position is determined by weighting the vector of the four points above, below, to the left and right ((0,1),(0,-1),(-1,0),(1,0)) with the cost at each of the four search positions as the weight. The interpretation unit 126 then determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0362] [Motion compensation > BIO / OBMC / LIC] Motion compensation includes modes that generate a predictive image and then correct that predictive image. These modes include, for example, BIO (bi-directional optical flow), OBMC (overlapped block motion compensation), and LIC (local illumination compensation), which will be described later.

[0363] Figure 59 is a flowchart showing an example of the process for generating predicted images.

[0364] The interpretation unit 126 generates a predicted image (step Sm_1) and corrects the predicted image using one of the modes described above (step Sm_2).

[0365] Figure 60 is a flowchart showing another example of the predictive image generation process.

[0366] The interpretation unit 126 determines the MV of the current block (step Sn_1). Next, the interpretation unit 126 generates a predicted image using the MV (step Sn_2) and determines whether or not to perform correction processing (step Sn_3). If the interpretation unit 126 determines that correction processing should be performed (Yes in step Sn_3), it generates the final predicted image by correcting the predicted image (step Sn_4). In the LIC described later, brightness and color difference may also be corrected in step Sn_4. On the other hand, if the interpretation unit 126 determines that no correction processing should be performed (No in step Sn_3), it outputs the predicted image as the final predicted image without correction (step Sn_5).

[0367] [Motion Compensation > OBMC] Interpretation images may be generated using not only the motion information of the current block obtained through motion search, but also the motion information of adjacent blocks. Specifically, interpretation images may be generated at the sub-block level within the current block by weighted addition of a prediction image based on motion information obtained through motion search (in the reference picture) and a prediction image based on the motion information of adjacent blocks (in the current picture). Such interpretation (motion compensation) is sometimes called OBMC (oval-wrapped block motion compensation) or OBMC mode.

[0368] In OBMC mode, information indicating the size of the subblock for OBMC (e.g., called the OBMC block size) may be signaled at the sequence level. Furthermore, information indicating whether or not to apply OBMC mode (e.g., called the OBMC flag) may be signaled at the CU level. Note that the signaling levels of this information are not limited to the sequence level and CU level, but may be other levels (e.g., picture level, slice level, brick level, CTU level, or subblock level).

[0369] The OBMC mode will be explained in more detail. Figures 61 and 62 are flowcharts and conceptual diagrams illustrating the overview of the predictive image correction process using OBMC.

[0370] First, as shown in Figure 62, a predicted image (Pred) is obtained using normal motion compensation with the MV assigned to the current block. In Figure 62, the arrow "MV" points to the reference picture, indicating what the current block of the current picture is referencing in order to obtain the predicted image.

[0371] Next, the previously derived MV (MV_L) for the encoded left-adjacent block is applied (reused) to the current block to obtain the predicted image (Pred_L). The MV (MV_L) is indicated by an arrow “MV_L” pointing from the current block to the reference picture. Then, the first correction of the predicted image is performed by superimposing the two predicted images, Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.

[0372] Similarly, the previously derived MV (MV_U) for the encoded upper adjacent block is applied (reused) to the current block to obtain the predicted image (Pred_U). The MV (MV_U) is indicated by an arrow “MV_U” pointing from the current block to the reference picture. Then, the predicted image Pred_U is superimposed on the predicted image that has undergone the first correction (e.g., Pred and Pred_L) to perform a second correction of the predicted image. This has the effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is the final predicted image of the current block, with the boundaries with adjacent blocks blended (smoothed).

[0373] The above example is a two-pass correction method using left-adjacent and top-adjacent blocks, but the correction method may also be a three-pass or more-pass correction method using right-adjacent and / or bottom-adjacent blocks.

[0374] Furthermore, the area to be superimposed does not have to be the entire pixel area of ​​the block, but rather only a portion of the area near the block boundary.

[0375] Here, we have described the OBMC predictive image correction process for obtaining a single predictive image Pred by superimposing additional predictive images Pred_L and Pred_U onto a single reference picture. However, if the predictive image is corrected based on multiple reference images, the same process may be applied to each of the multiple reference pictures. In such cases, by performing OBMC image correction based on multiple reference pictures, a corrected predictive image is obtained from each reference picture, and then the final predictive image is obtained by further superimposing these multiple corrected predictive images.

[0376] In OBMC, the unit of a current block may be a PU unit, or it may be a subblock unit obtained by further dividing a PU.

[0377] One method for determining whether to apply OBMC is to use an obmc_flag signal, which indicates whether or not to apply OBMC. For example, the encoding device 100 may determine whether the current block belongs to a region with complex motion. If the encoding device 100 belongs to a region with complex motion, it sets the obmc_flag to a value of 1 and applies OBMC to perform encoding. If the block does not belong to a region with complex motion, it sets the obmc_flag to a value of 0 and encodes the block without applying OBMC. On the other hand, the decoding device 200 decodes the obmc_flag written in the stream and switches whether or not to apply OBMC depending on its value to perform decoding.

[0378] [Motion Compensation > BIO] Next, we will explain how to derive MV. First, we will describe the mode of deriving MV based on a model that assumes uniform linear motion. This mode is sometimes called the BIO (bi-directional optical flow) mode. Also, this bi-directional optical flow may be written as BDOF instead of BIO.

[0379] FIG. 63 is a diagram for explaining a model assuming a constant velocity linear motion. In FIG. 63, (v x , v y ) indicates a velocity vector, and τ0 and τ1 respectively indicate the temporal distances between the current picture (Cur Pic) and two reference pictures (Ref0, Ref1). (MV x0 , MV y0 ) indicates the MV corresponding to the reference picture Ref0, and (MV x1 , MV y1 ) indicates the MV corresponding to the reference picture Ref1.

[0380] At this time, under the assumption of a constant velocity linear motion of the velocity vector (v x , v y ), (MV x0 , MV y0 ) and (MV x1 , MV y1 ) are respectively represented as (v xτ0 , v yτ0 ) and (-v xτ1 , -v yτ1 ), and the following optical flow equation (2) holds.

[0381]

Equation

[0382] Here, I(k) indicates the motion-compensated luminance value of the reference picture k (k = 0, 1) after motion compensation. This optical flow equation indicates that the sum of (i) the temporal derivative of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is equal to zero. Based on the combination of this optical flow equation and Hermite interpolation, the motion vector in block units obtained from a candidate MV list or the like may be corrected in pixel units. <0(001575> Furthermore, the motion vector (MV) may be derived on the decoding device 200 side using a method different from the derivation of the motion vector based on a model that assumes uniform linear motion. For example, the motion vector may be derived on a sub-block basis based on the MVs of multiple adjacent blocks.

[0384] Figure 64 is a flowchart showing an example of inter-prediction processing according to BIO. Figure 65 is a functional block diagram showing an example of the functional configuration of the inter-prediction unit 126 that performs inter-prediction according to BIO.

[0385] As shown in Figure 65, the interpretation unit 126 includes, for example, a memory 126a, an interpolation image derivation unit 126b, a gradient image derivation unit 126c, an optical flow derivation unit 126d, a correction value derivation unit 126e, and a prediction image correction unit 126f. Note that the memory 126a may be a frame memory 122.

[0386] The interpretation unit 126 derives two motion vectors (M0, M1) using two reference pictures (Ref0, Ref1) that are different from the picture (Cur Pic) containing the current block. Then, the interpretation unit 126 derives a predicted image of the current block using these two motion vectors (M0, M1) (step Sy_1). Note that motion vector M0 corresponds to the motion vector (MV) of reference picture Ref0. x0 MV y0 ) and motion vector M1 is the motion vector (MV) corresponding to reference picture Ref1. x1 MV y1 )

[0387] Next, the interpolated image derivation unit 126b refers to the memory 126a and uses the motion vector M0 and the reference picture L0 to create an interpolated image I of the current block. 0 The interpolated image derivation unit 126b also references memory 126a and uses motion vector M1 and reference picture L1 to derive the interpolated image I of the current block. 1 Derive the following (step Sy_2). Here, interpolated image I 0This is the image contained in the reference picture Ref0, which is derived for the current block, and is the interpolated image I 1 This is the image contained in reference picture Ref1, which is derived for the current block. Interpolated image I 0 and interpolated image I 1 Each of these may be the same size as the current block. Alternatively, interpolated image I 0 and interpolated image I 1 Each of these may be a larger image than the current block in order to properly derive the gradient image described later. Furthermore, interpolated image I 0 and I 1 This may include a motion vector (M0, M1) and a reference picture (L0, L1), and a predicted image derived by applying a motion compensation filter.

[0388] Furthermore, the gradient image derivation unit 126c generates an interpolated image I 0 and interpolated image I 1 From, the gradient image of the current block (Ix 0 ,Ix 1 Iy 0 Iy 1 ) is derived (Step Sy_3). Note that the horizontal gradient image is (Ix 0 ,Ix 1 ) and the vertical gradient image is (Iy 0 Iy 1 The gradient image derivation unit 126c may derive the gradient image, for example, by applying a gradient filter to the interpolated image. The gradient image may show the amount of spatial change in pixel values ​​along the horizontal direction, the vertical direction, or both.

[0389] Next, the optical flow derivation unit 126d interpolates the image (I) in units of multiple subblocks that constitute the current block. 0 ,I 1 ) and gradient image (Ix 0 ,Ix 1 Iy 0 Iy 1 ) using the above-mentioned velocity vector, optical flow (v x ,vy ) is derived (step Sy_4). Optical flow is a coefficient that corrects the amount of spatial movement of pixels, and may also be called local motion estimate, corrected motion vector, or corrected weight vector. For example, a subblock may be a 4x4 pixel subCU. Note that the derivation of optical flow may be performed in units other than subblocks, such as pixels.

[0390] Next, the interpretation unit 126 calculates the optical flow (v x ,v y The predicted image of the current block is corrected using ). For example, the correction value derivation unit 126e uses optical flow (v x ,v y The correction value for the pixel values ​​included in the current block is derived using (step Sy_5). The prediction image correction unit 126f may then correct the prediction image of the current block using the correction value (step Sy_6). The correction value may be derived for each pixel, or for multiple pixels or subblocks.

[0391] Note that the BIO processing flow is not limited to the processes disclosed in Figure 64. For example, only a portion of the processes disclosed in Figure 64 may be performed, different processes may be added or replaced, or processes may be executed in a different order.

[0392] [Motion Compensation > LIC] Next, we will describe an example of a mode that generates a predicted image (prediction) using LIC (local illumination compensation) processing.

[0393] Figure 66A is a conceptual diagram illustrating an example of a predictive image generation method using luminance correction processing by LIC. Figure 66B is a flowchart illustrating an example of the predictive image generation method using the LIC.

[0394] First, the interpretation unit 126 derives the MV from the encoded reference picture and obtains the reference image corresponding to the current block (step Sz_1).

[0395] Next, the interpretation unit 126 extracts information for the current block indicating how the luminance values ​​have changed between the reference picture and the current picture (step Sz_2). This extraction is performed based on the luminance pixel values ​​of the encoded left adjacent reference region (surrounding reference region) and encoded upper adjacent reference region (surrounding reference region) in the current picture, and the luminance pixel values ​​at the equivalent positions in the reference picture specified by the derived MV. Then, the interpretation unit 126 calculates luminance correction parameters using the information indicating how the luminance values ​​have changed (step Sz_3).

[0396] The interpretation unit 126 generates a predicted image for the current block by performing a brightness correction process on the reference image in the reference picture specified in MV, applying its brightness correction parameter (step Sz_4). In other words, the predicted image, which is the reference image in the reference picture specified in MV, is corrected based on the brightness correction parameter. This correction may involve brightness correction, color difference correction, or both. That is, color difference correction parameters may be calculated using information indicating how the color difference has changed, and color difference correction processing may be performed.

[0397] Note that the shape of the peripheral reference region in Figure 66A is just one example, and other shapes may be used.

[0398] Furthermore, although this explanation describes the process of generating a predicted image from a single reference picture, the process is similar when generating predicted images from multiple reference pictures. Alternatively, the brightness correction process may be applied to each reference picture obtained from the reference picture in the same manner as described above before generating the predicted image.

[0399] One method for determining whether to apply LIC is to use a signal called lic_flag, which indicates whether or not to apply LIC. For example, in the encoding device 100, it is determined whether the current block belongs to a region where a brightness change is occurring. If it belongs to a region where a brightness change is occurring, the value of lic_flag is set to 1 and LIC is applied and encoding is performed. If it does not belong to a region where a brightness change is occurring, the value of lic_flag is set to 0 and encoding is performed without applying LIC. On the other hand, in the decoding device 200, the lic_flag written in the stream may be decoded, and the applicability of LIC may be switched according to its value before decoding.

[0400] Another method for determining whether to apply LIC is, for example, to determine whether LIC has been applied to the surrounding blocks. Specifically, if the current block is being processed in merge mode, the interpretation unit 126 determines whether the surrounding encoded blocks selected during MV derivation in merge mode were encoded with LIC. The interpretation unit 126 then switches whether to apply LIC and performs encoding based on the result. In this example, the same process is applied to the decoding device 200.

[0401] The LIC (Luminance Correction Processing) was explained using Figures 66A and 66B, but the details will be explained below.

[0402] First, the interpretation unit 126 derives an MV for obtaining the reference image corresponding to the current block from the reference picture, which is an encoded picture.

[0403] Next, the interpretation unit 126 extracts information indicating how the luminance values ​​have changed between the reference picture and the current picture, using the luminance pixel values ​​of the left-adjacent and upper-adjacent encoded peripheral reference regions and the luminance pixel values ​​at equivalent positions in the reference picture specified by MV, and calculates luminance correction parameters. For example, let p0 be the luminance pixel value of a pixel in the peripheral reference region of the current picture, and p1 be the luminance pixel value of a pixel in the peripheral reference region of the reference picture at an equivalent position to that pixel. The interpretation unit 126 calculates coefficients A and B as luminance correction parameters to optimize A×p1+B=p0 for multiple pixels in the peripheral reference region.

[0404] Next, the interpretation unit 126 generates a predicted image for the current block by performing a brightness correction process on the reference image in the reference picture specified by MV using a brightness correction parameter. For example, let p2 be the brightness pixel value in the reference image, and p3 be the brightness pixel value of the predicted image after brightness correction processing. The interpretation unit 126 generates the predicted image after brightness correction processing by calculating A × p2 + B = p3 for each pixel in the reference image.

[0405] Furthermore, a portion of the peripheral reference area shown in Figure 66A may be used. For example, a region containing a predetermined number of pixels obtained by thinning out the upper adjacent pixels and the left adjacent pixels may be used as the peripheral reference area. Also, the peripheral reference area is not limited to the region adjacent to the current block, but may be a region not adjacent to the current block. In the example shown in Figure 66A, the peripheral reference area in the reference picture is the region specified by the MV of the current picture, but it may be a region specified by another MV. For example, the other MV may be the MV of the peripheral reference area in the current picture.

[0406] Although the operation of the encoding device 100 has been described here, the operation of the decoding device 200 is similar.

[0407] Furthermore, LIC may be applied not only to luminance but also to chrominance. In this case, correction parameters may be derived individually for each of Y, Cb, and Cr, or a common correction parameter may be used for any of them.

[0408] Furthermore, LIC processing may be applied on a subblock basis. For example, correction parameters may be derived using the surrounding reference region of the current subblock and the surrounding reference region of the reference subblock within the reference picture specified by the MV of the current subblock.

[0409] [Prediction Control Unit] The prediction control unit 128 selects either an intra-prediction image (an image or signal output from the intra-prediction unit 124) or an inter-prediction image (an image or signal output from the inter-prediction unit 126), and outputs the selected prediction image as a prediction signal to the subtraction unit 104 and the addition unit 116.

[0410] [Prediction parameter generation unit] The prediction parameter generation unit 130 may output information regarding intra-prediction, inter-prediction, and the selection of predicted images in the prediction control unit 128 as prediction parameters to the entropy coding unit 110. The entropy coding unit 110 may generate a stream based on the prediction parameters input from the prediction parameter generation unit 130 and the quantization coefficients input from the quantization unit 108. The prediction parameters may be used by the decoding device 200. The decoding device 200 may receive and decode the stream and perform the same processing as the prediction processing performed in the intra-prediction unit 124, inter-prediction unit 126, and prediction control unit 128. The prediction parameters may include, for example, a selected prediction signal (e.g., MV, prediction type, or prediction mode used in the intra-prediction unit 124 or inter-prediction unit 126), or arbitrary indices, flags, or values ​​based on or indicating the prediction processing performed in the intra-prediction unit 124, inter-prediction unit 126, and prediction control unit 128.

[0411] [Decoding device] Next, a decoding device 200 capable of decoding the stream output from the above-described encoding device 100 will be explained. Figure 67 is a block diagram showing an example of the functional configuration of the decoding device 200 according to the embodiment. The decoding device 200 is a device that decodes the encoded image stream in block units.

[0412] As shown in Figure 67, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, a prediction control unit 220, a prediction parameter generation unit 222, and a division decision unit 224. The intra prediction unit 216 and the inter prediction unit 218 are each configured as part of the prediction processing unit.

[0413] [Example of a decryption device implementation] Figure 68 is a functional block diagram showing an example implementation of the decoding device 200. The decoding device 200 comprises a processor b1 and memory b2. For example, several components of the decoding device 200 shown in Figure 67 are implemented by the processor b1 and memory b2 shown in Figure 68.

[0414] Processor b1 is a circuit that performs information processing and is a circuit that can access memory b2. For example, processor b1 is a dedicated or general-purpose electronic circuit for decoding streams. Processor b1 may be a processor such as a CPU. Alternatively, processor b1 may be a collection of multiple electronic circuits. Furthermore, for example, processor b1 may play the role of multiple components of the decoding device 200 shown in Figure 67, etc., excluding the component for storing information.

[0415] Memory b2 is a dedicated or general-purpose memory that stores information for processor b1 to decode the stream. Memory b2 may be an electronic circuit and may be connected to processor b1. Memory b2 may also be included in processor b1. Memory b2 may also be a collection of multiple electronic circuits. Memory b2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory b2 may also be non-volatile memory or volatile memory.

[0416] For example, memory b2 may store an image or a stream. Alternatively, memory b2 may store a program for processor b1 to decode the stream.

[0417] Furthermore, for example, memory b2 may play the role of an information storage component among the multiple components of the decoding device 200 shown in Figure 67, etc. Specifically, memory b2 may play the role of the block memory 210 and frame memory 214 shown in Figure 67. More specifically, reconstructed images (specifically, reconstructed blocks or reconstructed pictures, etc.) may be stored in memory b2.

[0418] Furthermore, the decoding device 200 does not necessarily have to implement all of the components shown in Figure 67, etc., nor does it have to perform all of the processes described herein. Some of the components shown in Figure 67, etc., may be included in other devices, and some of the processes described herein may be performed by other devices.

[0419] The following describes the overall processing flow of the decoding device 200, followed by a description of each component included in the decoding device 200. Note that detailed explanations of components included in the decoding device 200 that perform the same processing as those included in the encoding device 100 will be omitted. For example, the inverse quantization unit 204, inverse transform unit 206, adder unit 208, block memory 210, frame memory 214, intra prediction unit 216, inter prediction unit 218, prediction control unit 220, and loop filter unit 212 included in the decoding device 200 perform the same processing as the inverse quantization unit 112, inverse transform unit 114, adder unit 116, block memory 118, frame memory 122, intra prediction unit 124, inter prediction unit 126, prediction control unit 128, and loop filter unit 120 included in the encoding device 100.

[0420] [Overall flow of the decryption process] Figure 69 is a flowchart showing an example of the overall decoding process by the decoding device 200.

[0421] First, the partitioning determination unit 224 of the decoding device 200 determines the partitioning pattern for each of the multiple fixed-size blocks (128 × 128 pixels) contained in the picture based on the parameters input from the entropy decoding unit 202 (step Sp_1). This partitioning pattern is the partitioning pattern selected by the encoding device 100. Then, the decoding device 200 performs steps Sp_2 to Sp_6 for each of the multiple blocks that constitute that partitioning pattern.

[0422] The entropy decoding unit 202 decodes (specifically, performs entropy decoding) the encoded quantization coefficients and prediction parameters of the current block (step Sp_2).

[0423] Next, the inverse quantization unit 204 and the inverse transformation unit 206 reconstruct the predicted residual of the current block by performing inverse quantization and inverse transformation on multiple quantization coefficients (step Sp_3).

[0424] Next, the prediction processing unit, consisting of the intra-prediction unit 216, the inter-prediction unit 218, and the prediction control unit 220, generates a prediction image of the current block (step Sp_4).

[0425] Next, the summing unit 208 reconstructs the current block into a reconstructed image (also called a decoded image block) by adding the predicted image to the predicted residual (step Sp_5).

[0426] Then, once this reconstructed image is generated, the loop filter unit 212 performs filtering on the reconstructed image (step Sp_6).

[0427] The decryption device 200 then determines whether or not the entire picture has been decrypted (step Sp_7). If it determines that it has not been completed (No. in step Sp_7), it repeats the process from step Sp_1.

[0428] The processes in steps Sp_1 to Sp_7 may be performed sequentially by the decoding device 200, some of these processes may be performed in parallel, and the order may be changed.

[0429] [Partitioning decision section] Figure 70 is a conceptual diagram showing the relationship between the division decision unit 224 and other components in the embodiment. The division decision unit 224 may perform the following processing as an example.

[0430] The division decision unit 224 collects block information from, for example, the block memory 210 or the frame memory 214, and further obtains parameters from the entropy decoding unit 202. The division decision unit 224 may then determine a division pattern for fixed-size blocks based on the block information and parameters. The division decision unit 224 may then output information indicating the determined division pattern to the inverse transform unit 206, the intra prediction unit 216, and the inter prediction unit 218. The inverse transform unit 206 may perform an inverse transform on the transformation coefficients based on the division pattern indicated by the information from the division decision unit 224. The intra prediction unit 216 and the inter prediction unit 218 may generate a predicted image based on the division pattern indicated by the information from the division decision unit 224.

[0431] [Entropy Decoder] Figure 71 is a block diagram showing an example of the functional configuration of the entropy decoding unit 202.

[0432] The entropy decoding unit 202 generates quantization coefficients, prediction parameters, and parameters related to the partition pattern by entropy decoding the stream. For example, CABAC is used for this entropy decoding. Specifically, the entropy decoding unit 202 comprises, for example, a binary arithmetic decoding unit 202a, a context control unit 202b, and a multileveling unit 202c. The binary arithmetic decoding unit 202a arithmetically decodes the stream into a binary signal using the context value derived by the context control unit 202b. Similar to the context control unit 110b of the encoding device 100, the context control unit 202b derives context values, i.e., the probability of a binary signal occurring, according to the characteristics of the syntax elements or the surrounding circumstances. The multileveling unit 202c performs debinarization, converting the binary signal output from the binary arithmetic decoding unit 202a into a multilevel signal showing the quantization coefficients and the like. This multileveling is performed according to the binaryization method described above.

[0433] The entropy decoding unit 202 outputs the quantization coefficients to the inverse quantization unit 204 in block units. The entropy decoding unit 202 may also output the prediction parameters included in the stream (see Figure 1) to the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220. The intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220 can perform the same prediction processing as the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128 on the encoding device 100 side.

[0434] Figure 72 is a conceptual diagram showing an example of the CABAC processing flow in the entropy decoding unit 202.

[0435] First, the CABAC in the entropy decoding unit 202 is initialized. This initialization includes the initialization of the binary arithmetic decoding unit 202a and the setting of an initial context value. Then, the binary arithmetic decoding unit 202a and the multi-leveling unit 202c perform arithmetic decoding and multi-leveling on the encoded data of, for example, the CTU. At this time, the context control unit 202b updates the context value each time arithmetic decoding is performed. Then, as a post-processing step, the context control unit 202b saves the context value. This saved context value is used, for example, as the initial value of the context value for the next CTU.

[0436] [Dequantization section] The inverse quantization unit 204 inversely quantizes the quantization coefficients of the current block, which are input from the entropy decoding unit 202. Specifically, for each quantization coefficient of the current block, the inverse quantization unit 204 inversely quantizes the quantization coefficient based on the quantization parameter corresponding to that quantization coefficient. The inverse quantization unit 204 then outputs the inversely quantized quantization coefficients (i.e., transformation coefficients) of the current block to the inverse transformation unit 206.

[0437] Figure 73 is a block diagram showing an example of the functional configuration of the inverse quantization unit 204.

[0438] The inverse quantization unit 204 includes, for example, a quantization parameter generation unit 204a, a prediction quantization parameter generation unit 204b, a quantization parameter storage unit 204d, and an inverse quantization processing unit 204e.

[0439] Figure 74 is a flowchart showing an example of the inverse quantization process performed by the inverse quantization unit 204.

[0440] The inverse quantization unit 204 may, for example, perform inverse quantization processing for each CU based on the flow shown in Figure 74. Specifically, the quantization parameter generation unit 204a determines whether or not to perform inverse quantization (step Sv_11). If it is determined that inverse quantization should be performed (Yes in step Sv_11), the quantization parameter generation unit 204a obtains the difference quantization parameters of the current block from the entropy decoding unit 202 (step Sv_12).

[0441] Next, the predictive quantization parameter generation unit 204b obtains quantization parameters for a processing unit different from the current block from the quantization parameter storage unit 204d (step Sv_13). Based on the obtained quantization parameters, the predictive quantization parameter generation unit 204b generates predictive quantization parameters for the current block (step Sv_14).

[0442] Then, the quantization parameter generation unit 204a generates the quantization parameters of the current block based on the difference quantization parameters of the current block obtained from the entropy decoding unit 202 and the predicted quantization parameters of the current block generated by the predicted quantization parameter generation unit 204b (step Sv_15). For example, the difference quantization parameters of the current block may be generated by adding the difference quantization parameters of the current block obtained from the entropy decoding unit 202 and the predicted quantization parameters of the current block generated by the predicted quantization parameter generation unit 204b. The quantization parameter generation unit 204a then stores the quantization parameters of the current block in the quantization parameter storage unit 204d (step Sv_16).

[0443] Next, the inverse quantization processing unit 204e inversely quantizes the quantization coefficients of the current block into transformation coefficients using the quantization parameters generated in step Sv_15 (step Sv_17).

[0444] The differential quantization parameters may be decoded at the bit sequence level, picture level, slice level, brick level, or CTU level. The initial values ​​of the quantization parameters may also be decoded at the sequence level, picture level, slice level, brick level, or CTU level. In this case, the quantization parameters may be generated using the initial values ​​of the quantization parameters and the differential quantization parameters.

[0445] The inverse quantization unit 204 may be equipped with multiple inverse quantizers, and the quantization coefficients may be inversely quantized using an inverse quantization method selected from multiple inverse quantization methods.

[0446] [Inverse Transformation Section] The inverse transform unit 206 recovers the predicted residual by inversely transforming the transformation coefficients, which are input from the inverse quantization unit 204.

[0447] For example, if the information read from the stream indicates that EMT or AMT should be applied (e.g., the AMT flag is true), the inverse transformer 206 inversely transforms the transformation coefficients of the current block based on the information indicating the type of transformation read.

[0448] For example, if the information read from the stream indicates that NSST should be applied, the inverse conversion unit 206 applies inverse re-conversion to the conversion coefficients.

[0449] Figure 75 is a flowchart showing an example of processing by the inverse transform unit 206.

[0450] For example, the inverse transform unit 206 determines whether or not information indicating that an orthogonal transform is not performed exists in the stream (step St_11). If it is determined that such information does not exist (No in step St_11) (for example, nothing is indicated about whether orthogonal transforms are performed, or it is indicated that orthogonal transforms are performed), the inverse transform unit 206 obtains information indicating the type of transformation, which has been decoded by the entropy decoding unit 202 (step St_12). Next, the inverse transform unit 206 determines the type of transformation used for the orthogonal transform of the encoding device 100 based on that information (step St_13). Then, the inverse transform unit 206 performs an inverse orthogonal transform using the determined type of transformation (step St_14). As shown in Figure 75, if it is determined that information indicating that an orthogonal transform is not performed exists (Yes in step St_11) (for example, it is clearly indicated that an orthogonal transform is not performed: it is not indicated that an orthogonal transform is performed), the orthogonal transform is not performed.

[0451] Figure 76 is a flowchart showing another example of processing by the inverse transform unit 206.

[0452] For example, the inverse transform unit 206 determines whether the transformation size is less than or equal to a predetermined value (step Su_11). The predetermined value may be set in advance. If it is determined that the transformation size is less than or equal to the predetermined value (Yes in step Su_11), the inverse transform unit 206 obtains information from the entropy decoding unit 202 indicating which of the one or more transformation types included in the first transformation type group was used by the encoding device 100 (step Su_12). This information is decoded by the entropy decoding unit 202 and output to the inverse transform unit 206.

[0453] Based on this information, the inverse transform unit 206 determines the type of transformation used in the orthogonal transformation in the encoding device 100 (step Su_13). Then, the inverse transform unit 206 performs an inverse orthogonal transformation on the transformation coefficients of the current block using the determined transformation type (step Su_14). On the other hand, if the inverse transform unit 206 determines in step Su_11 that the transformation size is not less than or equal to a predetermined value (No. in step Su_11), it performs an inverse orthogonal transformation on the transformation coefficients of the current block using a second group of transformation types (step Su_15).

[0454] The inverse orthogonal transformation by the inverse transformation unit 206 may, for example, be performed for each TU according to the flow shown in Figure 75 or Figure 76. Alternatively, the inverse orthogonal transformation may be performed using a predetermined transformation type without decoding the information indicating the transformation type used in the orthogonal transformation. The predetermined transformation type may be a predefined transformation type or a default transformation type. Specifically, the transformation type may be DST7 or DCT8, and the inverse transformation basis function corresponding to that transformation type is used in the inverse orthogonal transformation.

[0455] [Addition section] The summing unit 208 reconstructs the current block by adding the predicted residual, which is the input from the inverse transform unit 206, and the predicted image, which is the input from the prediction control unit 220. In other words, a reconstructed image of the current block is generated. The summing unit 208 then outputs the reconstructed image of the current block to the block memory 210 and the loop filter unit 212.

[0456] [Block memory] The block memory 210 is a block referenced in intra prediction and is a storage unit for storing blocks within the current picture. Specifically, the block memory 210 stores the reconstructed image output from the adder 208.

[0457] [Loop Filter Section] The loop filter unit 212 applies a loop filter to the reconstructed image generated by the adder unit 208, and outputs the filtered reconstructed image to the frame memory 214 and a display device, etc.

[0458] If the information interpreted from the stream indicating ALF on / off indicates ALF is on, one filter is selected from several filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed image.

[0459] Figure 77 is a block diagram showing an example of the functional configuration of the loop filter unit 212. Note that the loop filter unit 212 has the same configuration as the loop filter unit 120 of the encoding device 100.

[0460] The loop filter unit 212 includes, for example, a deblocking filter processing unit 212a, an SAO processing unit 212b, and an ALF processing unit 212c, as shown in Figure 77. The deblocking filter processing unit 212a applies the above-described deblocking filter processing to the reconstructed image. The SAO processing unit 212b applies the above-described SAO processing to the reconstructed image after the deblocking filter processing. The ALF processing unit 212c applies the above-described ALF processing to the reconstructed image after the SAO processing. Note that the loop filter unit 212 does not have to include all the processing units disclosed in Figure 77, and may include only some of the processing units. Furthermore, the loop filter unit 212 may be configured to perform each of the above-described processes in an order different from the processing order disclosed in Figure 77, and does not have to perform all the processes shown in Figure 77.

[0461] [Frame memory] The frame memory 214 is a memory unit for storing reference pictures used in interpretation, and is sometimes called a frame buffer. Specifically, the frame memory 214 stores the reconstructed image that has been filtered by the loop filter unit 212.

[0462] [Prediction Unit (Intra Prediction Unit, Inter Prediction Unit, Prediction Control Unit)] Figure 78 is a flowchart showing an example of processing performed in the prediction unit of the decoding device 200. For example, the prediction unit consists of all or some of the components of the intra-prediction unit 216, the inter-prediction unit 218, and the prediction control unit 220. The prediction processing unit includes, for example, the intra-prediction unit 216 and the inter-prediction unit 218.

[0463] The prediction unit generates a predicted image of the current block (step Sq_1). This predicted image is also called a predicted signal or predicted block. The predicted signal may include, for example, an intra-prediction signal or an inter-prediction signal. Specifically, the prediction unit generates the predicted image of the current block using a reconstructed image already obtained by generating predicted images for other blocks, restoring prediction residuals, and adding predicted images. The prediction unit of the decoder 200 generates the same predicted image as the prediction unit of the encoder 100. In other words, the methods for generating predicted images used by these prediction units are common to or correspond to each other.

[0464] The reconstructed image may be, for example, the image of the reference picture, or it may be the image of the decoded block (i.e., the other block mentioned above) within the current picture, which is the picture containing the current block. The decoded block within the current picture is, for example, the adjacent block to the current block.

[0465] Figure 79 is a flowchart showing another example of the processing performed in the prediction unit of the decoding device 200.

[0466] The prediction unit determines a method or mode for generating the predicted image (step Sr_1). For example, this method or mode may be determined based on prediction parameters, for example.

[0467] If the prediction unit determines a first method as the mode for generating the predicted image, it generates the predicted image according to that first method (step Sr_2a). If the prediction unit determines a second method as the mode for generating the predicted image, it generates the predicted image according to that second method (step Sr_2b). If the prediction unit determines a third method as the mode for generating the predicted image, it generates the predicted image according to that third method (step Sr_2c).

[0468] The first, second, and third methods are different methods for generating predictive images, and may be, for example, an interpretation method, an intrapretation method, and other prediction methods, respectively. These prediction methods may use the reconstructed images described above.

[0469] Figure 80 is a flowchart showing another example of the processing performed in the prediction unit of the decoding device 200.

[0470] The prediction unit may perform prediction processing according to the flow shown in Figure 80 as an example. Note that the intrablock copy shown in Figure 80 is one mode belonging to interprediction, in which blocks included in the current picture are referenced as reference images or reference blocks. In other words, in intrablock copy, pictures different from the current picture are not referenced. Also, the PCM mode shown in Figure 80 is one mode belonging to intraprediction, in which no transformation or quantization is performed.

[0471] [Intra Prediction Unit] The intra-prediction unit 216 generates a predicted image of the current block (i.e., an intra-predicted image) by performing intra-prediction based on the intra-prediction mode decoded from the stream, by referring to the blocks in the current picture stored in the block memory 210. Specifically, the intra-prediction unit 216 generates an intra-predicted image by performing intra-prediction by referring to the pixel values ​​(e.g., luminance values, chrominance values) of blocks adjacent to the current block, and outputs the intra-predicted image to the prediction control unit 220.

[0472] Furthermore, if an intra-prediction mode that references a luminance block is selected in the intra-prediction of a color difference block, the intra-prediction unit 216 may predict the color difference component of the current block based on the luminance component of the current block.

[0473] Furthermore, if the information interpreted from the stream indicates the application of PDPC, the intra-prediction unit 216 corrects the pixel value after intra-prediction based on the gradient of the reference pixels in the horizontal / vertical directions.

[0474] Figure 81 shows an example of processing by the intra-prediction unit 216 of the decoding device 200.

[0475] First, the intra-prediction unit 216 determines whether or not an MPM flag indicating 1 exists in the stream (step Sw_11). If it determines that an MPM flag indicating 1 exists (Yes in step Sw_11), the intra-prediction unit 216 obtains information from the entropy decoding unit 202 indicating the intra-prediction mode selected by the encoding device 100 from the MPM (step Sw_12). This information is decoded by the entropy decoding unit 202 and output to the intra-prediction unit 216. Next, the intra-prediction unit 216 determines the MPM (step Sw_13). The MPM consists of, for example, six intra-prediction modes. Then, the intra-prediction unit 216 determines the intra-prediction mode indicated by the information obtained in step Sw_12 from among the multiple intra-prediction modes included in the MPM (step Sw_14).

[0476] Meanwhile, if the intra-prediction unit 216 determines in step Sw_11 that there is no MPM flag indicating 1 in the stream (No. in step Sw_11), it obtains information indicating the intra-prediction mode selected by the encoding device 100 (step Sw_15). In other words, the intra-prediction unit 216 obtains information from the entropy decoding unit 202 indicating the intra-prediction mode selected by the encoding device 100 from among one or more intra-prediction modes not included in the MPM. This information is decoded by the entropy decoding unit 202 and output to the intra-prediction unit 216. The intra-prediction unit 216 then determines the intra-prediction mode indicated by the information obtained in step Sw_15 from among the one or more intra-prediction modes not included in the MPM (step Sw_17).

[0477] The intra-prediction unit 216 generates a predicted image according to the intra-prediction mode determined in step Sw_14 or step Sw_17 (step Sw_18).

[0478] [International Prediction Department] The interpretation unit 218 predicts the current block by referring to a reference picture stored in the frame memory 214. Prediction is performed in units of the current block or subblocks within the current block. A subblock is a unit contained within a block and is smaller than a block. The size of a subblock may be 4x4 pixels, 8x8 pixels, or other sizes. The size of a subblock may be switched to units such as slices, bricks, or pictures.

[0479] For example, the interpretation unit 218 generates an interpretation image of the current block or subblock by performing motion compensation using motion information (e.g., MV) decoded from the stream (e.g., prediction parameters output from the entropy decoding unit 202), and outputs the interpretation image to the prediction control unit 220.

[0480] If the information read from the stream indicates that OBMC mode should be applied, the interpretation unit 218 generates an interpretation prediction image using not only the motion information of the current block obtained by motion search, but also the motion information of adjacent blocks.

[0481] Furthermore, if the information deciphered from the stream indicates that FRUC mode should be applied, the interpretation unit 218 derives motion information by performing a motion search according to the pattern matching method (bilateral matching or template matching) deciphered from the stream. Then, the interpretation unit 218 performs motion compensation (prediction) using the derived motion information.

[0482] Furthermore, when the BIO mode is applied, the interpretation unit 218 derives the MV based on a model that assumes uniform linear motion. Also, if the information read from the stream indicates that the affine mode is applied, the interpretation unit 218 derives the MV on a sub-block basis based on the MVs of multiple adjacent blocks.

[0483] [MV Derivation Flow] Figure 82 is a flowchart showing an example of the MV derivation process in the decoding device 200.

[0484] The interpretation unit 218 determines, for example, whether or not to decode motion information (e.g., MV). For example, the interpretation unit 218 may make this determination according to the prediction mode included in the stream, or it may make this determination based on other information included in the stream. Here, if the interpretation unit 218 determines to decode the motion information, it derives the MV of the current block in the mode for decoding the motion information. On the other hand, if the interpretation unit 218 determines not to decode the motion information, it derives the MV in the mode for not decoding the motion information.

[0485] Here, the modes for MV derivation include the normal inter mode, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that decode motion information include the normal inter mode, normal merge mode, and affine mode (specifically, affine inter mode and affine merge mode). Note that the motion information may include not only MV but also predicted MV selection information, which will be described later. Modes that do not decode motion information include the FRUC mode, etc. The inter prediction unit 218 selects a mode from these multiple modes for deriving the MV of the current block and derives the MV of the current block using the selected mode.

[0486] Figure 83 is a flowchart showing another example of the MV derivation process in the decoding device 200.

[0487] The interpretation unit 218 determines, for example, whether or not to decode the differential MV. For example, the interpretation unit 218 may make this determination according to the prediction mode included in the stream, or it may make this determination based on other information included in the stream. Here, if the interpretation unit 218 determines to decode the differential MV, it may derive the MV of the current block in the mode for decoding the differential MV. In this case, for example, the differential MV included in the stream is decoded as a prediction parameter.

[0488] On the other hand, if the interpretation unit 218 determines that it does not decode the differential MV, it derives the MV in a mode that does not decode the differential MV. In this case, the encoded differential MV is not included in the stream.

[0489] As mentioned above, the modes for MV derivation include the normal inter, normal merge mode, FRUC mode, and affine mode, which will be described later. Of these modes, the modes that encode differential MV include the normal inter mode and affine mode (specifically, affine inter mode). Modes that do not encode differential MV include the FRUC mode, normal merge mode, and affine mode (specifically, affine merge mode). The inter prediction unit 218 selects a mode from these multiple modes to derive the MV of the current block, and uses the selected mode to derive the MV of the current block.

[0490] [MV Derivation > Normal Intermode] For example, if the information interpreted from the stream indicates that the normal intermode should be applied, the interpretation unit 218 derives the MV in normal merge mode based on the information interpreted from the stream and uses that MV to perform motion compensation (prediction).

[0491] Figure 84 is a flowchart showing an example of inter-prediction processing using normal inter-mode in the decoding device 200.

[0492] The interpretation unit 218 of the decoding device 200 performs motion compensation for each block. First, the interpretation unit 218 obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks surrounding the current block in time or space (step Sg_11). In other words, the interpretation unit 218 creates a list of candidate MVs.

[0493] Next, the interpretation unit 218 extracts N candidate MVs (where N is an integer greater than or equal to 2) from among the multiple candidate MVs obtained in step Sg_11 as predicted motion vector candidates (also called predicted MV candidates) according to a predetermined priority order (step Sg_12). Note that this priority order may be predetermined for each of the N predicted MV candidates.

[0494] Next, the interpretation unit 218 decodes the predicted MV selection information from the input stream and uses the decoded predicted MV selection information to select one predicted MV candidate from the N predicted MV candidates as the predicted MV for the current block (step Sg_13).

[0495] Next, the interpretation unit 218 decodes the differential MV from the input stream and derives the MV of the current block by adding the decoded differential MV (the difference value) to the selected predicted MV (step Sg_14).

[0496] Finally, the inter-prediction unit 218 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Sg_15). The processes in steps Sg_11 to Sg_15 are performed for each block. For example, when the processes in steps Sg_11 to Sg_15 are performed for each block contained in a slice, the inter-prediction using the normal inter-mode for that slice is completed. Similarly, when the processes in steps Sg_11 to Sg_15 are performed for each block contained in a picture, the inter-prediction using the normal inter-mode for that picture is completed. Note that the processes in steps Sg_11 to Sg_15 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using the normal inter-mode for that slice may be completed. The same applies to pictures in steps Sg_11 to Sg_15. When the processes in steps Sg_11 to Sg_15 are performed for some blocks contained in a picture, the inter-prediction using the normal inter-mode for that picture may be completed.

[0497] [MV Derivation > Normal Merge Mode] For example, if the information read from the stream indicates the application of normal merge mode, the interpretation unit 218 derives MV in normal merge mode and uses that MV to perform motion compensation (prediction).

[0498] Figure 85 is a flowchart showing an example of interpretation processing using normal merge mode in the decoding device 200.

[0499] The interpretation unit 218 first obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks surrounding the current block in terms of time or space (step Sh_11). In other words, the interpretation unit 218 creates a candidate MV list.

[0500] Next, the interpretation unit 218 derives the MV of the current block by selecting one candidate MV from among the multiple candidate MVs obtained in step Sh_11 (step Sh_12). Specifically, the interpretation unit 218 obtains, for example, MV selection information included in the stream as prediction parameters, and selects the candidate MV identified by that MV selection information as the MV of the current block.

[0501] Finally, the inter-prediction unit 218 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Sh_13). The processes in steps Sh_11 to Sh_13 are performed for each block, for example. For example, when the processes in steps Sh_11 to Sh_13 are performed for each of the blocks contained in a slice, the inter-prediction using normal merge mode for that slice is completed. Similarly, when the processes in steps Sh_11 to Sh_13 are performed for each of the blocks contained in a picture, the inter-prediction using normal merge mode for that picture is completed. Note that the processes in steps Sh_11 to Sh_13 are not performed for all blocks contained in a slice, but only for some blocks, in which case the inter-prediction using normal merge mode for that slice may be completed. The same applies to pictures in steps Sh_11 to Sh_13. When the processes in steps Sh_11 to Sh_13 are performed for some of the blocks contained in a picture, the inter-prediction using normal merge mode for that picture may be completed.

[0502] [MV Derivation > FRUC Mode] For example, if the information decoded from the stream indicates the application of FRUC mode, the interpretation unit 218 derives the MV in FRUC mode and uses that MV to perform motion compensation (prediction). In this case, the motion information is not signaled from the encoding device 100 but is derived from the decoding device 200. For example, the decoding device 200 may derive the motion information by performing a motion search. In this case, the decoding device 200 performs a motion search without using the pixel values ​​of the current block.

[0503] Figure 86 is a flowchart showing an example of interpretation processing using FRUC mode in the decoding device 200.

[0504] First, the interpretation unit 218 refers to the MVs of each decoded block that is spatially or temporally adjacent to the current block and generates a list of candidate MVs (i.e., a candidate MV list, which may be the same as the candidate MV list for normal merge mode) (step Si_11). Next, the interpretation unit 218 selects the best candidate MV from among the multiple candidate MVs registered in the candidate MV list (step Si_12). For example, the interpretation unit 218 calculates an evaluation value for each candidate MV included in the candidate MV list and selects one candidate MV as the best candidate MV based on that evaluation value. Then, the interpretation unit 218 derives the MV for the current block based on the selected best candidate MV (step Si_14). Specifically, for example, the selected best candidate MV is directly derived as the MV for the current block. Alternatively, for example, the MV for the current block may be derived by performing pattern matching in the area surrounding the position in the reference picture corresponding to the selected best candidate MV. In other words, a search is performed in the area surrounding the best candidate MV using pattern matching and evaluation values ​​in the reference picture. If an MV with a better evaluation value is found, the best candidate MV may be updated to that MV and made the final MV of the current block. In this embodiment, it is not necessary to update to an MV with a better evaluation value.

[0505] Finally, the inter-prediction unit 218 generates a predicted image of the current block by performing motion compensation on the current block using the derived MV and the decoded reference picture (step Si_15). The processes in steps Si_11 to Si_15 are performed for each block, for example. For example, when the processes in steps Si_11 to Si_15 are performed for each of the blocks contained in a slice, the inter-prediction using FRUC mode for that slice is completed. Similarly, when the processes in steps Si_11 to Si_15 are performed for each of the blocks contained in a picture, the inter-prediction using FRUC mode for that picture is completed. Sub-blocks may also be processed in the same way as the block units described above.

[0506] [MV Derivation > Affine Merge Mode] For example, if the information deciphered from the stream indicates the application of affine merge mode, the interpretation unit 218 derives the motion video (MV) in affine merge mode and uses that MV to perform motion compensation (prediction).

[0507] Figure 87 is a flowchart showing an example of interpretation processing using affine merge mode in the decoding device 200.

[0508] In affine merge mode, the interpretation unit 218 first derives the MV of each control point of the current block (step Sk_11). The control points are the upper left and upper right corners of the current block, as shown in Figure 46A, or the upper left, upper right, and lower left corners of the current block, as shown in Figure 46B.

[0509] For example, when using the MV derivation method shown in Figures 47A to 47C, the interpretation unit 218 examines the decoded blocks in the order of Block A (left), Block B (top), Block C (upper right), Block D (lower left), and Block E (upper left), as shown in Figure 47A, and identifies the first valid block decoded in affine mode. The interpretation unit 218 uses the identified first valid block decoded in affine mode to derive the MV of the control points. For example, if Block A is identified and Block A has two control points, as shown in Figure 47B, the interpretation unit 218 calculates the motion vector v0 of the upper left corner control point and the motion vector v1 of the upper right corner control point of the current block from the motion vectors v3 and v4 of the upper left and upper right corners of the decoded block containing Block A. This derives the MV of each control point.

[0510] Furthermore, as shown in Figure 49A, if block A is identified and block A has two control points, the MV of three control points may be calculated. Alternatively, as shown in Figure 49B, if block A is identified and block A has three control points, the MV of two control points may be calculated.

[0511] Furthermore, if the stream includes MV selection information as a prediction parameter, the interpretation unit 218 may use that MV selection information to derive the MV of each control point in the current block.

[0512] Next, the interpretation unit 218 performs motion compensation for each of the multiple subblocks contained in the current block. That is, for each of the multiple subblocks, the interpretation unit 218 calculates the MV of that subblock as an affine MV using two motion vectors v0 and v1 and equation (1A) described above, or using three motion vectors v0, v1 and v2 and equation (1B) described above (step Sk_12). Then, the interpretation unit 218 performs motion compensation for that subblock using those affine MVs and the decoded reference picture (step Sk_13). Once steps Sk_12 and Sk_13 have been executed for each of the subblocks contained in the current block, the interpretation using the affine merge mode for that current block is completed. In other words, motion compensation is performed for the current block, and a predicted image of that current block is generated.

[0513] In step Sk_11, the above-mentioned candidate MV list may be generated. The candidate MV list may be a list containing candidate MVs derived for each control point using multiple MV derivation methods. The multiple MV derivation methods may be any combination of the MV derivation methods shown in Figures 47A to 47C, the MV derivation methods shown in Figures 48A and 48B, the MV derivation methods shown in Figures 49A and 49B, and other MV derivation methods.

[0514] The candidate MV list may also include candidate MVs from modes other than affine mode, where prediction is performed on a sub-block basis.

[0515] Furthermore, the candidate MV list may include, for example, candidate MVs for an affine merge mode with two control points and candidate MVs for an affine merge mode with three control points. Alternatively, a candidate MV list containing candidate MVs for an affine merge mode with two control points and a candidate MV list containing candidate MVs for an affine merge mode with three control points may be generated separately. Alternatively, a candidate MV list may be generated containing candidate MVs for one of the modes: an affine merge mode with two control points or an affine merge mode with three control points.

[0516] [MV Derivation > Affine Intermode] For example, if the information interpreted from the stream indicates the application of affine intermode, the interpretation unit 218 derives the motion video (MV) using affine intermode and performs motion compensation (prediction) using that MV.

[0517] Figure 88 is a flowchart showing an example of inter-prediction processing using affine intermode in the decoding device 200.

[0518] In affine intermode, the interpretation unit 218 first derives the predicted MV(v0,v1) or (v0,v1,v2) for each of two or three control points of the current block (step Sj_11). The control points are, for example, the upper left corner, upper right corner, or lower left corner of the current block, as shown in Figure 46A or Figure 46B.

[0519] The interpretation unit 218 acquires predicted MV selection information included in the stream as prediction parameters, and uses the MVs identified by the predicted MV selection information to derive the predicted MV for each control point in the current block. For example, when using the MV derivation method shown in Figures 48A and 48B, the interpretation unit 218 derives the predicted MV (v0,v1) or (v0,v1,v2) for the control point of the current block by selecting the MV of the block identified by the predicted MV selection information from among the decoded blocks near each control point in the current block shown in Figure 48A or Figure 48B.

[0520] Next, the interpretation unit 218 obtains, for example, each differential MV included in the stream as a prediction parameter, and adds the predicted MV of each control point in the current block to the differential MV corresponding to that predicted MV (step Sj_12). This derives the MV of each control point in the current block.

[0521] Next, the interpretation unit 218 performs motion compensation for each of the multiple subblocks contained in the current block. That is, for each of the multiple subblocks, the interpretation unit 218 calculates the MV of that subblock as an affine MV using two motion vectors v0 and v1 and equation (1A) described above, or using three motion vectors v0, v1 and v2 and equation (1B) described above (step Sj_13). Then, the interpretation unit 218 performs motion compensation for that subblock using those affine MVs and the decoded reference picture (step Sj_14). Once steps Sj_13 and Sj_14 have been executed for each of the subblocks contained in the current block, the interpretation using the affine merge mode for that current block is completed. In other words, motion compensation is performed for the current block, and a predicted image of that current block is generated.

[0522] In step Sj_11, the candidate MV list described above may be generated, similar to step Sk_11.

[0523] [MV Derivation > Triangle Mode] For example, if the information interpreted from the stream indicates the application of triangle mode, the interpretation unit 218 derives MV in triangle mode and uses that MV to perform motion compensation (prediction).

[0524] Figure 89 is a flowchart showing an example of interpretation processing using triangle mode in the decoding device 200.

[0525] In triangle mode, the interpretation unit 218 first divides the current block into a first partition and a second partition (step Sx_11). For example, the interpretation unit 218 may obtain partition information, which is information about the division into each partition, from the stream as prediction parameters. Then, the interpretation unit 218 may divide the current block into a first partition and a second partition according to that partition information.

[0526] Next, the interpretation unit 218 obtains multiple candidate MVs for the current block based on information such as the MVs of multiple decoded blocks surrounding the current block in time or space (step Sx_12). In other words, the interpretation unit 218 creates a candidate MV list.

[0527] Then, the interpretation unit 218 selects the candidate MV for the first partition and the candidate MV for the second partition from among the multiple candidate MVs obtained in step Sx_11 as the first MV and the second MV, respectively (step Sx_13). At this time, the interpretation unit 218 may obtain MV selection information to identify the selected candidate MVs from the stream as prediction parameters. Then, the interpretation unit 218 may select the first MV and the second MV according to that MV selection information.

[0528] Next, the interpretation unit 218 generates a first predicted image by performing motion compensation using the selected first MV and the decoded reference picture (step Sx_14). Similarly, the interpretation unit 218 generates a second predicted image by performing motion compensation using the selected second MV and the decoded reference picture (step Sx_15).

[0529] Finally, the interpretation unit 218 generates a predicted image of the current block by weighting and adding the first predicted image and the second predicted image (step Sx_16).

[0530] [Motion Detection > DMVR] For example, if the information deciphered from the stream indicates the application of DMVR, the interpretation unit 218 performs motion search using DMVR.

[0531] Figure 90 is a flowchart showing an example of motion detection processing by the DMVR in the decoding device 200.

[0532] The interpretation unit 218 first derives the MV of the current block in merge mode (step Sl_11). Next, the interpretation unit 218 derives the final MV for the current block by searching the surrounding region of the reference picture indicated by the MV derived in step Sl_11 (step Sl_12). In other words, the MV of the current block is determined by the DMVR.

[0533] Figure 91 is a flowchart showing an example of motion detection processing by the DMVR in the decoding device 200.

[0534] First, in Step 1 shown in Figure 58A, the interpretation unit 218 calculates the cost of the search position indicated by the initial MV (also called the starting point) and the eight surrounding search positions. Then, the interpretation unit 218 determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 218 determines that the cost of the search positions other than the starting point is the minimum, it moves to the search position with the minimum cost and performs the process shown in Figure 58A in Step 2. On the other hand, if the cost of the starting point is the minimum, the interpretation unit 218 skips the process shown in Figure 58A in Step 2 and proceeds to the process shown in Step 3.

[0535] In Step 2, shown in Figure 58A, the interpretation unit 218 uses the search position moved according to the processing result of Step 1 as a new starting point and performs a search similar to the process in Step 1. The interpretation unit 218 then determines whether the cost of the search positions other than the starting point is the minimum. If the interpretation unit 218 determines that the cost of the search positions other than the starting point is the minimum, it proceeds to Step 4. On the other hand, if the interpretation unit 218 determines that the cost of the starting point is the minimum, it proceeds to Step 3.

[0536] In Step 4, the interpretation unit 218 treats the starting position as the final search position and determines the difference between the position indicated by the initial MV and the final search position as the difference vector.

[0537] In Step 3, shown in Figure 58A, the interpretation unit 218 determines the pixel position with the minimum cost based on the costs of the four points above, below, to the left and right of the starting point in Step 1 or Step 2, and sets that pixel position as the final search position.

[0538] The decimal-precision pixel position is determined by weighting and adding the vector of four points ((0,1), (0,-1), (-1,0), (1,0)) located above, below, left, and right, with the cost of each of these four points at their respective search positions being used as weights. The interpretation unit 218 then determines the difference between the position indicated by the initial MV and the final search position as a difference vector.

[0539] [Motion compensation > BIO / OBMC / LIC] For example, if the information read from the stream indicates the application of correction to the predicted image, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to the correction mode. These modes include, for example, the BIO, OBMC, and LIC mentioned above.

[0540] Figure 92 is a flowchart showing an example of the predictive image generation process in the decoding device 200.

[0541] The interpretation unit 218 generates a predicted image (step Sm_11) and corrects the predicted image using one of the modes described above (step Sm_12).

[0542] Figure 93 is a flowchart showing another example of the predictive image generation process in the decoding device 200.

[0543] The interpretation unit 218 derives the MV of the current block (step Sn_11). Next, the interpretation unit 218 generates a predicted image using the MV (step Sn_12) and determines whether or not to perform correction processing (step Sn_13). For example, the interpretation unit 218 obtains prediction parameters included in the stream and determines whether or not to perform correction processing based on these prediction parameters. These prediction parameters are, for example, flags indicating whether or not to apply each of the above modes. If the interpretation unit 218 determines that correction processing should be performed (Yes in step Sn_13), it generates the final predicted image by correcting the predicted image (step Sn_14). In LIC, the brightness and color difference of the predicted image may be corrected in step Sn_14. On the other hand, if the interpretation unit 218 determines that no correction processing should be performed (No in step Sn_13), it outputs the predicted image as the final predicted image without correction (step Sn_15).

[0544] [Motion Compensation > OBMC] For example, if the information interpreted from the stream indicates the application of OBMC, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to OBMC.

[0545] Figure 94 is a flowchart showing an example of the correction process of the predicted image by OBMC in the decoding device 200. The flowchart in Figure 94 shows the flow of correction of the predicted image using the current picture and reference picture shown in Figure 62.

[0546] First, as shown in Figure 62, the interpretation unit 218 acquires a predicted image (Pred) using normal motion compensation with the MV assigned to the current block.

[0547] Next, the interpretation unit 218 applies (reuses) the already derived MV (MV_L) for the decoded left adjacent block to the current block to obtain a predicted image (Pred_L). Then, the interpretation unit 218 performs the first correction of the predicted image by superimposing the two predicted images, Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.

[0548] Similarly, the interpretation unit 218 applies (reuses) the already derived MV (MV_U) for the decoded upper adjacent block to the current block to obtain a predicted image (Pred_U). Then, the interpretation unit 218 performs a second correction of the predicted image by superimposing the predicted image Pred_U onto the first corrected predicted image (e.g., Pred and Pred_L). This has the effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is the final predicted image of the current block, with the boundaries with adjacent blocks blended (smoothed).

[0549] [Motion Compensation > BIO] For example, if the information interpreted from the stream indicates the application of BIO, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to BIO.

[0550] Figure 95 is a flowchart showing an example of the correction process of the predicted image by BIO in the decoding device 200.

[0551] As shown in Figure 63, the interpretation unit 218 derives two motion vectors (M0, M1) using two reference pictures (Ref0, Ref1) that are different from the picture (Cur Pic) containing the current block. Then, the interpretation unit 218 derives a predicted image of the current block using these two motion vectors (M0, M1) (step Sy_11). Note that motion vector M0 corresponds to the motion vector (MV) of reference picture Ref0. x0 MV y0 ) and motion vector M1 is the motion vector (MV) corresponding to reference picture Ref1. x1 MV y1 )

[0552] Next, the interpretation unit 218 interpolates the current block image I using the motion vector M0 and the reference picture L0. 0 The following is derived. Furthermore, the interpretation unit 218 uses the motion vector M1 and the reference picture L1 to interpolate the current block image I 1 Derive the following (step Sy_12). Here, interpolated image I 0 This is the image contained in the reference picture Ref0, which is derived for the current block, and is the interpolated image I 1 This is the image contained in reference picture Ref1, which is derived for the current block. Interpolated image I 0 and interpolated image I 1 Each of these may be the same size as the current block. Alternatively, interpolated image I 0 and interpolated image I 1 Each of these may be a larger image than the current block in order to properly derive the gradient image described later. Furthermore, interpolated image I 0 and I 1 This may include a motion vector (M0, M1) and a reference picture (L0, L1), and a predicted image derived by applying a motion compensation filter.

[0553] Furthermore, the interpolation prediction unit 218 generates interpolated image I 0 and interpolated image I 1 From, the gradient image of the current block (Ix 0 ,Ix 1 Iy 0 Iy 1 ) is derived (step Sy_13). Note that the horizontal gradient image is (Ix 0 ,Ix 1 ) and the vertical gradient image is (Iy 0 Iy 1 The interpretation unit 218 may, for example, derive a gradient image by applying a gradient filter to the interpolated image. The gradient image only needs to show the amount of spatial change in pixel values ​​along the horizontal or vertical direction.

[0554] Next, the interpretation unit 218 interpolates the image (I) in units of multiple subblocks that constitute the current block. 0 ,I 1 ) and gradient image (Ix 0 ,Ix 1 Iy 0 Iy 1 The optical flow (vx, vy), which is the velocity vector described above, is derived using (step Sy_14). As an example, the subblock may be a 4x4 pixel subCU.

[0555] Next, the interpretation unit 218 corrects the predicted image of the current block using optical flow (vx, vy). For example, the interpretation unit 218 derives correction values ​​for the pixel values ​​included in the current block using optical flow (vx, vy) (step Sy_15). The interpretation unit 218 may then correct the predicted image of the current block using the correction values ​​(step Sy_16). The correction values ​​may be derived for each pixel, or for multiple pixels or subblocks.

[0556] Note that the BIO processing flow is not limited to the processing disclosed in Figure 95. Only a portion of the processing disclosed in Figure 95 may be performed, different processing may be added or replaced, or the processing may be executed in a different order.

[0557] [Motion Compensation > LIC] For example, if the information interpreted from the stream indicates the application of LIC, the interpretation unit 218 generates a predicted image and then corrects the predicted image according to LIC.

[0558] Figure 96 is a flowchart showing an example of the correction process of the predicted image by the LIC in the decoding device 200.

[0559] First, the interpretation unit 218 uses MV to obtain the reference image corresponding to the current block from the decoded reference picture (step Sz_11).

[0560] Next, the interpretation unit 218 extracts information for the current block indicating how the luminance values ​​have changed between the reference picture and the current picture (step Sz_12). This extraction may be performed based on the luminance pixel values ​​of the decoded left adjacent reference region (peripheral reference region) and the decoded upper adjacent reference region (peripheral reference region) in the current picture, as shown in Figure 66A, and the luminance pixel values ​​at the equivalent positions in the reference picture specified by the derived MV. Then, the interpretation unit 218 calculates luminance correction parameters using the information indicating how the luminance values ​​have changed (step Sz_13).

[0561] The interpretation unit 218 generates a predicted image for the current block by performing a brightness correction process on the reference image in the reference picture specified in MV, applying its brightness correction parameter (step Sz_14). In other words, the predicted image, which is the reference image in the reference picture specified in MV, is corrected based on the brightness correction parameter. This correction may involve brightness correction or color difference correction.

[0562] [Prediction Control Unit] The prediction control unit 220 selects either an intra-prediction image or an inter-prediction image and outputs the selected prediction image to the summing unit 208. Overall, the configuration, functions, and processing of the prediction control unit 220, intra-prediction unit 216, and inter-prediction unit 218 on the decoding device 200 side correspond to the configuration, functions, and processing of the prediction control unit 128, intra-prediction unit 124, and inter-prediction unit 126 on the encoding device 100 side.

[0563] [Decoding using predicted color difference samples] In the first embodiment, a determination is made as to whether the color difference sample block of the current block can be predicted using the luminance sample. The predicted color difference sample is used for decoding the block. For example, the embodiment may use a process to determine whether or not to enable a tool such as CCLM that predicts a color difference signal using the decoding result of the luminance signal in the decoding method or encoding method.

[0564] Figure 97 is a flowchart showing an example of a process 1000 for decoding blocks using predicted color difference samples, which may be performed, for example, by the encoding device 100 in Figure 7 or the decoding device 200 in Figure 67. For convenience, Figure 97 will be described with reference to the decoding device 200 in Figure 67.

[0565] In S1001, the decoding device 200 determines whether the color difference block to be processed is located within an M×N non-overlapping area that matches the M×N grid of the color difference sample. Figures 99 and 100 are conceptual diagrams illustrating an example of determining whether the color difference block to be processed is located within an M×N non-overlapping area that matches the M×N grid of the color difference sample. In some formats, such as the YUV420 format, a 16×16 pixel area of ​​color difference corresponds to a 32×32 pixel area of ​​luminance. As shown in Figures 99 and 100, a color difference block contained within a 32×32 luminance area that matches the 16×16 color difference grid is determined to be located within an M×N non-overlapping area that matches the M×N grid of the color difference sample. A color difference block not contained within a 32×32 luminance area is not determined to be located within an M×N non-overlapping area that matches the M×N grid of the color difference sample. Even if the color difference block to be processed straddles the boundary of the corresponding luminance block, if it is contained within the same VPDU, the color difference sample may be acquired using the luminance sample. For example, in Figure 99, chrominance block A predicts the sample in chrominance block A-1 using the corresponding sample in luminance block B, and predicts the sample in chrominance block A-2 using the corresponding sample in luminance block C.

[0566] As shown in Figure 100, the color difference blocks are contained within a grid (a 16x16 grid as shown), and the luminance blocks at the same location are also located within the same 32x32 area. Therefore, the color difference sample of the illustrated color difference block may be predicted using the luminance sample for the color difference block.

[0567] In some embodiments, color difference samples of blocks that are not determined to be within an M×N non-overlapping area matching the M×N grid of color difference samples do not need to be predicted using luminance samples. However, color difference samples of blocks that are determined to be within an M×N non-overlapping area matching the M×N grid of color difference samples may be predicted using luminance samples, for example, by default, when other conditions such as those described later with reference to S1002 are met.

[0568] As shown in Figure 97, if in S1001 it is not determined that there is a color difference block to be processed within an M×N non-overlapping area that matches the M×N grid of the color difference sample, process 1000 proceeds from S1001 to S1004, in which the decoding device 200 predicts the color difference sample of the block without using a luminance sample. Process 1000 proceeds from S1004 to S1005, in which the decoding device 200 decodes the block using the predicted color difference sample. If in S1001 it is determined that there is a color difference block to be processed within an M×N non-overlapping area, process 1000 proceeds from S1001 to S1002.

[0569] In S1002, the decoding device 200 determines whether or not to divide the luminance VPDU to be processed into smaller blocks. A VPDU is a unit of parallel processing in the encoding or decoding process, and is, for example, 64 x 64 in size. The size of the VPDU may be determined by a standard or the like, or it may be encoded into a stream.

[0570] The decision of whether or not to divide the luminance VPDU to be processed into smaller blocks can be made in various ways. Several examples will be described in detail later with reference to Figures 102 and 103.

[0571] If, in S1002, it is determined not to divide the luminance VPDU to be processed into smaller blocks, process 1000 proceeds from S1002 to S1004, in which case the decoder 200 predicts the color difference sample of the block without using the luminance sample. Process 1000 proceeds from S1004 to S1005, in which case the decoder 200 decodes the block using the predicted color difference sample. If, in S1002, it is determined to divide the luminance VPDU to be processed into smaller blocks, process 1000 proceeds from S1002 to S1003, in which case the decoder 200 predicts the color difference sample of the block using the luminance sample. Process 1000 proceeds from S1003 to S1005, in which case the decoder 200 decodes the block using the predicted color difference sample. In some embodiments, for example, as will be described later with reference to Figures 104 to 110, an additional decision may be made to determine whether or not to decode the color difference sample of a block using a luminance sample.

[0572] Figure 98 is a flowchart illustrating another example of the process 2000 for decoding blocks using predicted color difference samples, which may be performed, for example, by the encoding device 100 in Figure 7 or the decoding device 200 in Figure 67. For convenience, Figure 98 will be described with reference to the decoding device 200 in Figure 67.

[0573] In S2001, the decoding device 200 determines whether or not to divide the first VPDU and the second VPDU into smaller blocks. The determination of whether or not to divide the luminance VPDU to be processed into smaller blocks can be made in various ways. Several examples will be described in detail later with reference to Figures 102 and 103.

[0574] If, in S2001, it is determined that the first VPDU should not be divided into smaller blocks, but the second VPDU should be divided into smaller blocks, then process 2000 proceeds from S2001 to S2002, in which case the decoder 200 predicts the color difference sample of the block without using the luminance sample. Process 2000 then proceeds from S2002 to S2004, in which case the decoder 200 decodes the block using the predicted color difference sample.

[0575] If, in S2001, it is determined that the first luminance VPDU is not to be divided into smaller blocks, and the second VPDU is not to be divided into smaller blocks, then process 2000 proceeds from S2001 to S2003, in which case the decoder 200 predicts the color difference sample of the block using the luminance sample. Process 2000 proceeds from S2003 to S2004, in which case the decoder 200 decodes the block using the predicted color difference sample. In some embodiments, for example, as will be described later with reference to Figures 104 to 110, additional decisions may be made to determine whether or not to decode the color difference sample of the block using the luminance sample.

[0576] Figure 101 is a conceptual diagram illustrating VPDU. VPDU is a non-overlapping area representing the buffer size for pipeline stages. The left side of Figure 101 (label a) shows an example of a 128×128 CTU with four 64×64 VPDUs. The right side of Figure 101 (label b) shows an example of a 128×128 CTU with sixteen 32×32 VPDUs. When the VPDU is 64×64, both M and N are set to 16. If the VPDU is further subdivided, the resulting CU size will be less than or equal to 2M×2N (32×32). In the YUV420 format, a 16×16 chrominance area corresponds to a 32×32 luminance area, so pixels in a 16×16 chrominance grid can be predicted based on pixels in the corresponding 32×32 luminance grid. Therefore, once the decoding of the 32x32 luminance region is complete, the decoding of the process that predicts color difference from luminance can be started in the 16x16 color difference region. In the case of the YUV444 format, the MxN luminance region corresponds to the MxN color difference region. In step S1002 in Figure 97 or step S2001 in Figure 98, if 2Mx2N is half the size of the VPDU both horizontally and vertically, it may be determined whether or not to further divide the VPDU into one or more layers. However, this is only the case when it is 1 / 4 of the VPDU. The embodiment may also determine whether or not the divided CU is 2Mx2N or less, for example, whether or not to further divide the CU into two or more layers.

[0577] Figure 102 is a conceptual diagram illustrating an example of determining whether a VPDU to be processed can predict blocks of color difference samples using luminance samples, based on whether or not the luminance VPDU is divided into blocks. The left side shows the luminance CTU, and the right side shows the corresponding color difference CTU. As shown in the figure, luminance VPDU0 is divided into blocks, while luminance VPDU1 is not. Therefore, referring to process 1000 in Figure 97, the color difference samples of VPDU0 may be predicted using luminance samples, while the color difference samples of VPDU1 may not be predicted using luminance samples.

[0578] Figure 103 is a conceptual diagram illustrating two examples of methods for determining whether to divide a luminance VPDU into smaller blocks. In the first example shown on the left side of Figure 103 (label a), the decision of whether to divide the luminance VPDU may be based on a division flag associated with the luminance VPDU. As shown, if the value of the division flag is 1, the VPDU is divided (and, referring to process 1000 in Figure 97, the color difference sample of the block may be predicted using the luminance sample). If the value of the division flag is 0, the VPDU is not divided (and, referring to process 1000 in Figure 97, the color difference sample of the block may not be predicted using the luminance sample). Other division flag values ​​may be used to determine whether the luminance VPDU can be divided.

[0579] In the second example shown on the right side of Figure 103 (label b), the decision of whether or not to split the luminance VPDU may be made based on the quadtree splitting depth of the luminance block of the VPDU. As shown in the figure, the quadtree splitting depth of the luminance block of VPDU0 is greater than 1, so referring to process 1000 in Figure 97, when decoding the block of VPDU0, the color difference sample may be predicted using the luminance sample. On the other hand, the quadtree splitting depth of the block of VPDU1 is 1 or less, so referring to process 1000 in Figure 97, when decoding the block of VPDU0, the color difference sample does not need to be predicted using the luminance sample. The decision of whether or not to split the luminance VPDU may also be made using other splitting depth values.

[0580] Figure 104 is a conceptual diagram illustrating additional decisions that may be considered when determining whether or not to predict the color difference sample of a block using a luminance sample. As shown in the figure, one additional decision that may be used when determining whether or not to predict the color difference sample of a block using a luminance sample is whether or not the size of the block to be processed is less than or equal to the threshold block size.

[0581] The threshold block size may be the default block size, the signaled block size, or a predetermined block size, and may also be the luminance or chrominance block size. For example, if the threshold block size is a 16x16 luminance block size, the luminance block size of VPDU0 is larger than 16x16, so it may be determined that the chrominance sample of the block is not determined using the luminance sample. In S1002 of Figure 97 or S2001 of Figure 98, the threshold block size may be used to determine whether or not to divide the luminance VPDU into smaller blocks.

[0582] The embodiments of process 1000 in Figure 97 and process 2000 in Figure 98 may be modified in various ways. For example, process 1000 or 2000 may be modified to perform more steps than those shown, or fewer steps than those shown, or in a different order, or to combine or divide the steps. For example, process 1000 may be modified to determine whether to predict a color difference sample of a block based on other judgments, such as the size of the block to be processed, as described with reference to Figure 103, before S1001 or S1002. In another example, process 1000 may be modified to omit S1001. In another example, the embodiment of process 2000 in Figure 98 may be modified to perform step S1001 before step S2001. In another example, in S2001, it may be determined whether to divide the first VPDU and the second VPDU into smaller blocks.

[0583] Figure 105 is a conceptual diagram illustrating an example of a combination of conditions considered in determining whether or not to predict a block's color difference sample using a luminance sample. As shown in Figure 105, an example of a combination of conditions is whether or not the luminance VPDU and its corresponding color difference VPDU have a quadtree division depth of 2 or more. Since both luminance VPDU0 and color difference VPDU0 have a quadtree division depth of 2 or more, the color difference sample of color difference VPDU0 may be predicted using a luminance sample. However, since luminance VPDU1 has a quadtree division depth of less than 2, one of the conditions is not met, and the color difference sample of color difference VPDU1 is predicted without using a luminance sample.

[0584] Figure 106 is a conceptual diagram illustrating another example considering combinations of conditions in determining whether or not to predict a block's color difference sample using a luminance sample. As shown in Figure 106, an example of a combination of conditions is (i) whether the quadtree division depth of the luminance VPDU is 2 or greater, (ii) whether the quadtree division depth of the corresponding color difference VPDU is equal to 1, and (iii) whether or not the 32×32 color difference division threshold condition is met (for example, if the color difference size is 32×32, the block is not divided). Since the luminance VPDU0 has a quadtree division depth of 2 or greater, condition (i) is met, and since the color difference VPDU0 has a quadtree division depth equal to 1, condition (ii) is met, and since the color difference VPDU is not divided into blocks smaller than 32×32, all three conditions are met, and the color difference sample of color difference VPDU0 may be predicted using a luminance sample. However, since the color difference VPDU1 has blocks smaller than the 32x32 threshold, condition (iii) is not met, and the color difference sample of the color difference VPDU1 is predicted without using the luminance sample.

[0585] Figure 107 is a conceptual diagram illustrating another example of considering combinations of conditions in determining whether or not to predict a block's chrominance sample using a luminance sample. As shown in Figure 107, an example of a combination of conditions is (i) whether the quadtree splitting depth of the luminance VPDU is 2 or greater, (ii) whether the quadtree splitting depth of the corresponding chrominance VPDU is equal to 1, and (iii) whether or not the 32x32 chrominance splitting threshold condition is met (for example, if the chrominance size is 32x32, the block is not split). In Figure 107, qtDepthC represents the quadtree splitting depth of the chrominance, and mtDepthC represents the multi-tree splitting depth of the chrominance. A quadtree split may be followed by another quadtree split or multi-tree split (two- or three-part split). Identification of the chrominance quadtree split ends at depth 1, and the condition chromaSplit32x32 == CU_DONT_SPLIT is added (condition iii discussed with reference to Figure 106). This means that there is no further subdivision at the 32x32 color difference level. Assuming that the quadtree subdivision depth qtDepthl for luminance is 2 or greater, only color difference VPDU0 satisfies all three conditions, and the color difference sample for color difference VPDU0 may be predicted using luminance samples. Color difference VPDU1 has a quadtree subdivision depth of 2, and the block is divided into blocks smaller than 32x32, so the color difference sample for VPDU1 is predicted without using luminance samples. Color difference VPDU2 has a quadtree subdivision depth of 1, but the block is divided into blocks smaller than 32x32, so the color difference sample for VPDU2 is predicted without using luminance samples. Color difference VPDU3 has a quadtree subdivision depth of 1, but the block is divided into blocks smaller than 32x32, so the color difference sample for VPDU3 is predicted without using luminance samples.

[0586] Figure 108 is a conceptual diagram illustrating another example considering combinations of conditions in determining whether or not to predict chromatic difference samples of a block using luminance samples. In the example, the combinations of conditions are: (i) whether the quadtree division depth of the luminance VPDU is 2 or greater, (ii) whether the quadtree division depth of the corresponding chromatic difference VPDU is equal to 1, and (iii) whether or not a vertical or horizontal trisection follows a horizontal chromatic difference division of size 32×32. In VPDU0, the conditions are met. The VPDU is horizontally divided into two 16×32 blocks, and these blocks are not further divided using horizontal or vertical trisections. Therefore, chromatic difference samples in all blocks of VPDU0 may be predicted using luminance samples. In VPDU1, the conditions are met for the lower 16×32 block which is not further trisected, and chromatic difference samples in the lower 16×32 block may be predicted using luminance samples. Due to the further vertical trisection, the conditions are not met for the upper 16x32 block of VPDU1, and the color difference sample for the upper 16x32 block of VPDU1 is predicted without using a luminance sample.

[0587] Figure 109 is a conceptual diagram illustrating another example considering combinations of conditions in determining whether or not to predict a block's color difference sample using a luminance sample. As shown in Figure 109, an example of a combination of conditions is: (i) whether the quadtree division depth of the luminance VPDU is equal to 1, (ii) whether or not the luminance division threshold condition of 64×64 is met (for example, if the luminance size is 64×64, the block is not divided), (iii) whether or not the quadtree division depth of the corresponding color difference VPDU is equal to 1, and (iv) whether or not the color difference division threshold condition of 32×32 is met (for example, if the color difference size is 32×32, the block is not divided). VPDU0 satisfies all four conditions, and the color difference sample of VPDU0 may be predicted using a luminance sample. Color difference VPDU1 has a quadtree division depth of 2, and the block is divided into blocks smaller than 32×32, so the color difference sample of VPDU1 is predicted without using a luminance sample.

[0588] Figure 110 is a conceptual diagram illustrating another example considering combinations of conditions in determining whether or not to predict a block's color difference sample using a luminance sample. As shown in Figure 110, if any of the conditions are true, a block's color difference sample may be predicted using a luminance sample. Examples of condition combinations include: (i) whether the quadtree division depth of the luminance VPDU is 2 or more, and whether the quadtree division depth of the color difference VPDU is 2 or more; (ii) whether the quadtree division depth of the luminance VPDU is equal to 1, satisfies the 64×64 luminance division threshold condition (e.g., if the luminance size is 64×64, the block is not divided), whether the quadtree division depth of the corresponding color difference VPDU is equal to 1, and satisfies the 32×32 color difference division threshold condition (e.g., if the color difference size is 32×32, the block is not divided); (iii) (iv) Whether the quadtree division depth of the luminance VPDU is 2 or more, the quadtree division depth of the corresponding chrominance VPDU is equal to 1, and the 32x32 chrominance division threshold condition is met (for example, if the chrominance size is 32x32, the block is not divided), (iv) whether the quadtree division depth of the luminance VPDU is 2 or more, the quadtree division depth of the corresponding chrominance VPDU is equal to 1, the chrominance division of the 32x32 block is horizontal, and whether smaller chrominance blocks than 32x32 are not divided or are vertically divided. The block of VPDU1 violates all four conditions, so the chrominance sample of VPDU1 is predicted without using the luminance sample. To reduce the latency of chrominance prediction (from the luminance sample) within the 32x32 sample, taking the scan order into consideration, the conditions in the example in Figure 110 may be used. For example, in VPDU1, chrominance block 0 must wait for the reconstruction of luminance block 0 for prediction. Furthermore, color difference block 1 must wait for the reconstruction of luminance block 0 and luminance block 1 for prediction. To avoid such latency, color difference prediction may be performed without using luminance samples.

[0589] The blocks described in each embodiment may be replaced with rectangular or non-rectangular partitions. Figure 111 shows examples of non-rectangular partitions, such as triangular, L-shaped, pentagonal, hexagonal, and polygonal partitions. Other non-rectangular partitions may be used, or various combinations of shapes may be used. The term "partition" in each embodiment may be replaced with the term "prediction unit." Furthermore, the term "partition" in each embodiment may be replaced with the term "sub-prediction unit." Furthermore, the term "partition" in each embodiment may be replaced with the term "encoding unit."

[0590] Other conditions may be used. For example, in one embodiment, when an encoding mode that predicts color difference from luminance, such as CCLM, is enabled, the first division of the VPDU may always be four divisions. In another embodiment, if a predetermined number of four divisions are not applied to at least one VPDU in the CTU, for example, the first VPDU in the CTU in the scan order, then CCLM may be disabled for all VPDUs in the CTU.

[0591] CCLM may be defined as an intra-prediction mode that uses mode information such as intra_chroma_pred_mode. An index number indicating an intra-prediction mode and each mode may be mapped one-to-one in a table, but if CCLM is disabled, the table entries for CCLM are unnecessary. Therefore, the index number is encoded, and the signal can be encoded with a reduced number of bits. In this embodiment, the table indicating the intra-prediction mode may be switched depending on whether CCLM is enabled or disabled. For example, by referring to the division flag information indicating the division of luminance into four parts, if the luminance is not divided into a predetermined size or less in the VPDU, it may be determined that CCLM is disabled, and the table corresponding to when CCLM is disabled is used. Otherwise, the table used when CCLM can be used may be used. In this embodiment, a table including an entry for CCLM may be used without switching tables. However, if CCLM is disabled, it is not necessary to refer to the entry for CCLM.

[0592] Referring to Figure 98, for example, in some embodiments, the block to be processed may be included in a first VPDU. In some embodiments, the block to be processed may be included in a second VPDU. In some embodiments, the first VPDU is a luminance VPDU. In some embodiments, the first VPDU is a chrominance VPDU. In some embodiments, the second VPDU is a luminance VPDU. In some embodiments, the second VPDU is a chrominance VPDU. In some embodiments, the first VPDU is a luminance VPDU and the second VPDU is a chrominance VPDU. In some embodiments, whether or not to divide a VPDU into smaller blocks may be determined based on a division flag associated with the VPDU. In some embodiments, the division flag is a quadtree division flag. In some embodiments, the division flag is a binary tree division flag. In some embodiments, the division flag is a ternary tree division flag. In some embodiments, the determination of whether or not to divide a VPDU into smaller blocks may be based on a plurality of division flags associated with the VPDU. In some embodiments, the determination of whether or not to divide a VPDU into smaller blocks may be based on the block division depth. In some embodiments, the block partitioning depth is the quadtree partitioning depth. In some embodiments, the block partitioning depth is the binary tree partitioning depth. In some embodiments, the block partitioning depth is the terntree partitioning depth. In some embodiments, the decision of whether or not to partition the VPDU is repeated until a threshold block size is reached. In some embodiments, the threshold block size is a default threshold block size and may be predetermined. In some embodiments, the threshold block size is signaled. In some embodiments, the partition shapes of smaller blocks are limited to a predetermined set of shapes, and the block sizes of smaller blocks are limited to a predetermined set of block sizes. These examples have been described with reference to decoding devices and decoding methods, but examples of these embodiments may also be used in encoding devices and encoding methods.

[0593] Among the effects, one particularly significant is the ability to reduce reconstruction latency and improve hardware implementation flexibility by determining whether the color difference sample used to decode the block being processed can be predicted using the luminance sample.

[0594] One or more embodiments disclosed herein may be implemented in combination with at least some of the other embodiments herein. Furthermore, some processes, some configurations of the apparatus, some syntax, etc., described in the flowcharts of one or more embodiments disclosed herein may be implemented in combination with the other embodiments. Embodiments described with reference to the components of an encoding device may similarly be implemented with the corresponding components of a decoding device.

[0595] [Implementation and Application] In each of the above embodiments, each functional or operational block can typically be implemented by an MPU (micro processing unit) and memory, etc. Furthermore, the processing performed by each functional block may be implemented as a program execution unit, such as a processor, which reads and executes software (programs) recorded on a recording medium such as ROM. This software may be distributed. This software may be recorded on various recording media such as semiconductor memory. It is also possible to implement each functional block using hardware (dedicated circuits). Furthermore, various combinations of hardware and software may be employed.

[0596] The processing described in each embodiment may be implemented by centralized processing using a single device (system), or by distributed processing using multiple devices. Furthermore, the processor executing the above program may be one or multiple. In other words, centralized processing may be performed, or distributed processing may be performed.

[0597] The embodiments of this disclosure are not limited to those described above, and various modifications are possible, which are also included within the scope of the embodiments of this disclosure.

[0598] Furthermore, here we will describe application examples of the video encoding method (image encoding method) or video decoding method (image decoding method) shown in each of the above embodiments, and various systems for implementing these application examples. Such systems may be characterized by having an image encoding device using the image encoding method, an image decoding device using the image decoding method, or an image encoding and decoding device that includes both. Other configurations of such systems can be appropriately modified as needed.

[0599] [Usage example] Figure 112 shows the overall configuration of a suitable content supply system ex100 for realizing a content distribution service. The service area for the communication service is divided into cells of a desired size, and within each cell, there are base stations ex106, ex107, ex108, ex109, and ex110, which are fixed radio stations in the illustrated example.

[0600] In this content supply system ex100, various devices such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the internet ex101 via an internet service provider ex102 or a communication network ex104, and base stations ex106~ex110. The content supply system ex100 may also be configured to connect any combination of the above devices. In various implementations, the devices may be directly or indirectly interconnected via a telephone network or short-range wireless, etc., without going through base stations ex106~ex110. Furthermore, the streaming server ex103 may be connected to various devices such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 via the internet ex101, etc. The streaming server ex103 may also be connected to terminals in a hotspot on an airplane ex117 via satellite ex116.

[0601] Note that instead of base stations ex106~ex110, wireless access points or hotspots may be used. Also, streaming server ex103 may be connected directly to the communication network ex104 without going through the internet ex101 or internet service provider ex102, or it may be connected directly to the airplane ex117 without going through satellite ex116.

[0602] Camera ex113 may be any device capable of taking still images and videos, such as a digital camera. Smartphone ex115 may be a smartphone, mobile phone, or PHS (Personal Handyphone System) that supports mobile communication systems such as 2G, 3G, 3.9G, 4G, and the upcoming 5G.

[0603] Home appliance ex114 refers to appliances such as refrigerators or equipment included in household fuel cell cogeneration systems.

[0604] In the content supply system ex100, live streaming becomes possible when a terminal with a shooting function is connected to the streaming server ex103 via a base station ex106 or the like. In live streaming, a terminal (such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and a terminal inside an airplane ex117) may perform the encoding process described in each of the above embodiments on still images or video content captured by a user using the terminal, or it may multiplex the video data obtained by encoding with sound data encoded from the sound corresponding to the video, and then transmit the obtained data to the streaming server ex103. In other words, each terminal functions as an image encoding device according to one aspect of this disclosure.

[0605] Meanwhile, the streaming server ex103 streams the content data sent to the requesting client. The client is a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or a terminal on an airplane ex117, etc., that is capable of decoding the encoded data. Each device that receives the distributed data decodes and plays back the received data. That is, each device may function as an image decoding device according to one aspect of this disclosure.

[0606] [Distributed Processing] Furthermore, the streaming server ex103 may consist of multiple servers or computers that distribute data processing, recording, and distribution. For example, the streaming server ex103 may be implemented by a CDN (Content Delivery Network), where content delivery is achieved through a network connecting numerous edge servers distributed worldwide. In a CDN, the physically closest edge server may be dynamically assigned depending on the client. By caching and delivering content to the edge server, latency can be reduced. In addition, if several types of errors occur or the communication state changes due to increased traffic, processing can be distributed among multiple edge servers, the delivery entity can be switched to another edge server, or delivery can be continued by bypassing the failed part of the network, thus enabling high-speed and stable delivery.

[0607] Furthermore, beyond the distributed processing of the distribution itself, the encoding process of the captured data can be performed on each terminal, on the server side, or shared among them. For example, encoding generally involves two processing loops. In the first loop, the complexity or code amount of the image at the frame or scene level is detected. In the second loop, processing is performed to improve encoding ef...

Claims

1. Circuits and, The circuit comprises a memory connected to the aforementioned circuit, In operation, the aforementioned circuit Based on the division information, it is determined whether or not to divide the first VPDU (virtual pipeline decoding unit) into smaller blocks, and whether or not to divide the second VPDU into smaller blocks. For the determination to not divide the first VPDU into smaller blocks, and to divide the second VPDU into smaller blocks, the blocks of color difference samples are predicted without using luminance samples. For the determination to divide the first VPDU into smaller blocks and the second VPDU into smaller blocks, the blocks of the color difference samples are predicted using luminance samples. For the determination that the first VPDU will not be divided into smaller blocks, and the second VPDU will not be divided into smaller blocks, the blocks of the color difference samples are predicted using luminance samples. The block is encoded using the predicted color difference samples. Encoding device.

2. Circuits and, The circuit comprises a memory connected to the aforementioned circuit, In operation, the aforementioned circuit Based on the division information, it is determined whether or not to divide the first VPDU (virtual pipeline decoding unit) into smaller blocks, and whether or not to divide the second VPDU into smaller blocks. For the determination to not divide the first VPDU into smaller blocks, and to divide the second VPDU into smaller blocks, the blocks of color difference samples are predicted without using luminance samples. For the determination to divide the first VPDU into smaller blocks and the second VPDU into smaller blocks, the blocks of the color difference samples are predicted using luminance samples. For the determination that the first VPDU will not be divided into smaller blocks, and the second VPDU will not be divided into smaller blocks, the blocks of the color difference samples are predicted using luminance samples. The block is decoded using the predicted color difference sample. Decoding device.

3. Circuits and, The circuit comprises a memory connected to the aforementioned circuit, In operation, the aforementioned circuit Based on the division information, it is determined whether or not to divide the first VPDU (virtual pipeline decoding unit) into smaller blocks, and whether or not to divide the second VPDU into smaller blocks. For the determination to not divide the first VPDU into smaller blocks, and to divide the second VPDU into smaller blocks, the blocks of color difference samples are predicted without using luminance samples. For the determination to divide the first VPDU into smaller blocks and the second VPDU into smaller blocks, the blocks of the color difference samples are predicted using luminance samples. For the determination that the first VPDU will not be divided into smaller blocks, and the second VPDU will not be divided into smaller blocks, the blocks of the color difference samples are predicted using luminance samples. The block is encoded using the predicted color difference samples, A bitstream containing the above-encoded block is generated, The generated bitstream is transmitted. Transmitter.

Images