Systems and methods for video coding

Abstract

An image encoder has circuitry coupled to a memory. The circuitry segments a current image block into a plurality of partitions. The circuitry predicts a first motion vector from a single prediction motion vector candidate set for a first partition of the plurality of partitions, and encodes the first partition using the first motion vector.

Description

[0001] This application is a divisional application of application number 201980047145.8 filed on July 11, 2019 and entitled “System and method for video encoding”. Technical Field

[0002] The present disclosure relates to video encoding, and in particular to video encoding and decoding systems, components, and methods for performing inter-frame prediction functions to construct a prediction of a current frame based on a reference frame. Background Art

[0003] With the development of video coding technology, from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Codec), there is still a continuous demand for improvement and optimization of video coding technology to handle the growing amount of digital video data in various applications. The present disclosure relates to further advancements, improvements and optimizations in video coding, in particular, inter-frame prediction functions for building a prediction of a current frame based on a reference frame. Summary of the invention

[0004] In one aspect, an image encoder includes a circuit and a memory coupled to the circuit. In operation, the circuit divides a current image block into a plurality of partitions. A first motion vector is predicted from a set of uni-prediction motion vector candidates for a first partition of the plurality of partitions. The first partition is encoded using the first motion vector.

[0005] In one aspect, an image encoder includes: a partitioner, which, in operation, receives and partitions an original picture into blocks; a first adder, which, in operation, receives blocks from the partitioner and receives predictions from a prediction controller; and subtracts each prediction from its corresponding block to output a residual; a transformer, which, in operation, performs a transform on the residual output from the adder to output a transform coefficient; a quantizer, which, in operation, quantizes the transform coefficient to generate a quantized transform coefficient; an entropy encoder, which, in operation, encodes the quantized transform coefficient to generate a bitstream; an inverse quantizer and transformer, which, in operation, inversely quantizes the quantized transform coefficient to obtain a transform coefficient and inversely transforms the transform coefficient to obtain a residual; a second adder, which, in operation, adds the residual output from the inverse quantizer and inverse transformer to the prediction output from the prediction controller to reconstruct the block; and a prediction controller, which is coupled to an inter-frame predictor, an intra-frame predictor, and a memory. The inter-frame predictor generates a prediction of a current block based on a reference block in an encoded reference picture in operation. The intra-frame predictor generates a prediction of a current block based on an encoded reference block in a current picture in operation. The inter-frame predictor divides a current image block into a plurality of partitions in operation. Predicts a first motion vector from a single prediction motion vector candidate set for a first partition of the plurality of partitions. Encodes the first partition using the first motion vector.

[0006] In one aspect, an image encoding method includes: dividing a current image block into a plurality of partitions, predicting a first motion vector from a single prediction motion vector candidate set of a first partition among the plurality of partitions, and encoding the first partition using the first motion vector.

[0007] In one aspect, an image decoder includes a circuit and a memory coupled to the circuit. The circuit is operable to partition a current image block into a plurality of partitions. A first motion vector is predicted from a single prediction motion vector candidate set for a first partition of the plurality of partitions. The first motion vector is used to decode the first partition.

[0008] In one aspect, an image decoder includes: an entropy decoder, which receives and decodes a coded bitstream in operation to obtain quantized transform coefficients; an inverse quantizer and an inverse transformer, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder, which in operation adds the residual output from the inverse quantizer and the inverse transformer to the prediction output from the prediction controller to reconstruct the block; and a prediction controller, which is coupled to an inter-frame predictor, an intra-frame predictor, and a memory. The inter-frame predictor generates a prediction of a current block based on a reference block in a decoded reference picture in operation. The intra-frame predictor generates a prediction of a current block based on an encoded reference block in a current picture in operation. The inter-frame predictor, in operation, divides a current image block into a plurality of partitions. A first motion vector is predicted from a single prediction motion vector candidate set of a first partition among the plurality of partitions. The first partition is decoded using the first motion vector.

[0009] In one aspect, an image decoding method includes: dividing a current image block into a plurality of partitions, predicting a first motion vector from a single prediction motion vector candidate set of a first partition among the plurality of partitions, and decoding the first partition using the first motion vector.

[0010] Some implementations of the embodiments of the present disclosure can improve coding efficiency, can simplify the encoding / decoding process, can accelerate the speed of the encoding / decoding process, and can efficiently select appropriate components / operations used in encoding and decoding, such as appropriate filters, block sizes, motion vectors, reference pictures, reference blocks, etc.

[0011] Other benefits and advantages of the disclosed embodiments will become apparent from the description and drawings. Benefits and / or advantages may be obtained individually from the various embodiments and features of the description and drawings, and all embodiments and features need not be provided to obtain one or more such benefits and / or advantages.

[0012] It should be noted that the general or specific embodiments may be implemented as a system, a method, an integrated circuit, a computer program, a storage medium, or any selective combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram showing a functional configuration of an encoder according to an embodiment.

[0014] Figure 2 is a flow chart showing one example of the overall encoding process performed by the encoder.

[0015] Figure 3 is a conceptual diagram showing an example of block division.

[0016] Figure 4Ais a conceptual diagram showing an example of a slice configuration.

[0017] Figure 4B is a conceptual diagram showing an example of a tile configuration.

[0018] Figure 5A is a diagram showing transform basis functions for various transform types.

[0019] Figure 5B is a conceptual diagram illustrating an example space-varying transform (SVT).

[0020] Fig. 6A is a conceptual diagram showing an example of a filter shape used in an adaptive loop filter (ALF).

[0021] Figure 6B is a conceptual diagram showing another example of a filter shape used in ALF.

[0022] Figure 6C is a conceptual diagram showing another example of a filter shape used in ALF.

[0023] Figure 7 is a block diagram showing one example of a specific configuration of a loop filter used as a deblocking filter (DBF).

[0024] Figure 8 is a conceptual diagram illustrating an example of a deblocking filter having symmetric filtering characteristics with respect to a block boundary.

[0025] Fig. 9 is a conceptual diagram for illustrating a block boundary on which a deblocking filtering process is performed.

[0026] Fig.10 is a conceptual diagram showing an example of a Bs value.

[0027] Fig.11 is a flow chart showing one example of a process performed by a prediction processor of an encoder.

[0028] Fig.12 is a flow chart illustrating another example of a process performed by a prediction processor of an encoder.

[0029] Fig.13 is a flow chart illustrating another example of a process performed by a prediction processor of an encoder.

[0030] Fig.14 is a conceptual diagram showing sixty-seven intra prediction modes used in intra prediction in the embodiment.

[0031] Fig.15 is a flowchart illustrating an example basic processing flow for inter-frame prediction.

[0032] Fig.16 is a flow chart showing one example of derivation of motion vectors.

[0033] Fig.17 is a flow chart illustrating another example of derivation of motion vectors.

[0034] Fig.18 is a flow chart illustrating another example of derivation of motion vectors.

[0035] Fig.19 is a flowchart illustrating an example of inter prediction in normal inter mode.

[0036] Fig. 20 is a flowchart illustrating an example of inter prediction in merge mode.

[0037] Fig.21 is a conceptual diagram for illustrating an example of a motion vector derivation process in merge mode.

[0038] Fig. 22 is a flow chart illustrating one example of a frame rate up-conversion (FRUC) process.

[0039] Fig.23 is a conceptual diagram for illustrating an example of pattern matching (bilateral matching) between two blocks along a motion trajectory.

[0040] Fig.24 is a conceptual diagram for illustrating an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture.

[0041] Fig.25A is a conceptual diagram for illustrating an example of deriving a motion vector for each subblock based on motion vectors of a plurality of neighboring blocks.

[0042] Fig.25B is a conceptual diagram for illustrating an example of deriving a motion vector for each subblock in an affine mode using three control points.

[0043] Fig.26A is a conceptual diagram for illustrating the affine merge mode.

[0044] Fig.26B is a conceptual diagram for illustrating an affine merge mode using two control points.

[0045] Fig.26C is a conceptual diagram for illustrating an affine merging mode using three control points.

[0046] Fig. 27is a flowchart showing one example of a process in affine merge mode.

[0047] Fig.28A is a conceptual diagram for illustrating an affine inter mode using two control points.

[0048] Fig.28B is a conceptual diagram for illustrating an affine inter mode using three control points.

[0049] Fig.29 is a flowchart for illustrating an example of a process in the affine inter mode.

[0050] Fig. 30A is a conceptual diagram for illustrating an affine inter mode, in which a current block has three control points and a neighboring block has two control points.

[0051] Fig. 30B is a conceptual diagram for illustrating an affine inter mode, in which a current block has two control points and a neighboring block has three control points.

[0052] Fig.31A is a flow chart illustrating a merge mode process including decoder motion vector refinement (DMVR).

[0053] Fig.31B is a conceptual diagram for illustrating an example of a DMVR process.

[0054] Fig.32 is a flowchart showing an example of generating a predicted image.

[0055] Fig.33 is a flowchart showing another example of generating a predicted image.

[0056] Fig.34 is a flowchart showing another example of generating a predicted image.

[0057] Fig.35 is a flowchart illustrating one example of a predicted image correction process performed by an overlapped block motion compensation (OBMC) process.

[0058] Fig.36 is a conceptual diagram for illustrating one example of a predicted image correction process performed by the OBMC process.

[0059] Fig.37 is a conceptual diagram for illustrating the generation of two triangular prediction images.

[0060] Fig.38 This is a conceptual diagram for illustrating a model assuming uniform linear motion.

[0061] Fig.39is a conceptual diagram for illustrating one example of a predicted image generation method using a brightness correction process performed by a local illumination compensation (LIC) process.

[0062] Fig.40 is a block diagram showing an installation example of an encoder.

[0063] Fig.41 is a block diagram showing a functional configuration of a decoder according to an embodiment.

[0064] Fig.42 is a flow chart showing one example of the overall decoding process performed by the decoder.

[0065] Fig.43 is a flow chart showing one example of a process performed by a prediction processor of a decoder.

[0066] Fig.44 is a flow chart showing another example of a process performed by a prediction processor of a decoder.

[0067] Fig.45 is a flow chart showing an example of inter prediction in conventional inter mode in a decoder.

[0068] Fig.46 is a block diagram showing an installation example of a decoder.

[0069] Fig.47A is a flowchart illustrating an example of a process flow of partitioning an image block into a plurality of partitions including at least a first partition and a second partition, predicting a motion vector from a motion vector candidate set of at least the first partition, and performing further processing according to one embodiment.

[0070] Fig.47B is a flowchart illustrating an example of a process flow of partitioning an image block into multiple partitions including at least a first partition and a second partition, predicting a motion vector from a single prediction motion vector candidate set of at least the first partition, and performing further processing according to one embodiment.

[0071] Fig.48 is a conceptual diagram for illustrating an exemplary method of partitioning an image block into a first partition and a second partition.

[0072] Fig.49 is a conceptual diagram for illustrating adjacent and non-adjacent spatially adjacent partitions of a first partition.

[0073] Fig.50 is a conceptual diagram for illustrating uni-prediction and bi-prediction motion vector candidates for an image block.

[0074] Fig.51 is a conceptual diagram for illustrating determination of a single-prediction motion vector from a dual-prediction motion vector.

[0075] Fig.52 is a conceptual diagram for illustrating determination of a single-prediction motion vector from a dual-prediction motion vector.

[0076] Fig.53 is a conceptual diagram for illustrating determination of a single-prediction motion vector from a dual-prediction motion vector.

[0077] Fig.54 is a conceptual diagram for illustrating generation of one or more single-predictive motion vectors from a single-predictive motion vector.

[0078] Fig.55 is a conceptual diagram for illustrating generation of one or more single-predictive motion vectors from a single-predictive motion vector.

[0079] Fig.56 is a conceptual diagram for illustrating generation of one or more single-predictive motion vectors from a single-predictive motion vector.

[0080] Fig.57 is a conceptual diagram for illustrating unidirectional motion vectors of a first partition and a second partition of an image block.

[0081] Fig.58 is a flowchart showing an example of the following process flow according to one embodiment: predicting the motion vector of a partition of an image block from a first single-prediction motion vector candidate set or from a second motion vector candidate set including dual-prediction motion vector candidates and single-prediction motion vector candidates according to the shape of the partition, and performing further processing.

[0082] Fig.59 is a flowchart showing an example of the following process flow according to one embodiment: predicting the motion vector of a partition of an image block from a first single-prediction motion vector candidate set or from a second motion vector candidate set including dual-prediction motion vector candidates and single-prediction motion vector candidates, and performing further processing.

[0083] Fig.60 is a flowchart showing an example of a process flow according to one embodiment: deriving a motion vector of a first partition and a motion vector of a second partition from candidate sets of motion vectors of the first partition and the second partition, and performing further processing.

[0084] Fig.61 is a block diagram showing the overall configuration of a content providing system for realizing a content distribution service.

[0085] Fig.62is a conceptual diagram showing an example of a coding structure in scalable coding.

[0086] Fig.63 is a conceptual diagram showing an example of a coding structure in scalable coding.

[0087] Fig.64 is a conceptual diagram showing an example of a display screen of a web page.

[0088] Fig.65 is a conceptual diagram showing an example of a display screen of a web page.

[0089] Fig.66 is a block diagram illustrating one example of a smart phone.

[0090] Fig.67 is a block diagram showing an example of the configuration of a smartphone. DETAILED DESCRIPTION

[0091] Hereinafter, embodiments will be described with reference to the accompanying drawings. Note that the embodiments described below all illustrate general or specific examples. The numerical values, shapes, materials, components, arrangement and connection of components, steps, relationships and sequences of steps, etc. indicated in the following embodiments are merely examples and are not intended to limit the scope of the claims.

[0092] Embodiments of encoders and decoders are described below. Embodiments are examples of encoders and decoders, and the processes and / or configurations presented in the description of aspects of the present disclosure may be applied to encoders and decoders. The processes and / or configurations may also be implemented in encoders and decoders that are different from the encoders and decoders according to the embodiments. For example, with respect to the processes and / or configurations applied to the embodiments, any of the following may be implemented:

[0093] (1) Any component of an encoder or a decoder according to the embodiments presented in the description of aspects of the present disclosure may be replaced with or combined with another component presented anywhere in the description of aspects of the present disclosure.

[0094] (2) In an encoder or decoder according to an embodiment, any changes may be made to the functions or processes performed by one or more components of the encoder or decoder, such as addition, replacement, removal, etc. of functions or processes. For example, any function or process may be replaced by or combined with another function or process appearing anywhere in the description of various aspects of the present disclosure.

[0095] (3) In the method implemented by the encoder or decoder according to the embodiment, any changes may be made, such as adding, replacing and removing one or more processes included in the method. For example, any process in the method may be replaced or combined with another process appearing anywhere in the description of the aspects of the present disclosure.

[0096] (4) One or more components included in an encoder or decoder according to an embodiment may be combined with components presented anywhere in the description of various aspects of the present disclosure, may be combined with components presented with one or more functions included anywhere in the description of various aspects of the present disclosure, and may be combined with components that implement one or more processes implemented by components presented in the description of various aspects of the present disclosure.

[0097] (5) Components including one or more functions of an encoder or decoder according to an embodiment, or components implementing one or more processes of an encoder or decoder according to an embodiment may be combined with or replaced with components presented anywhere in the description of the various aspects of the present disclosure, with components including one or more functions presented anywhere in the description of the various aspects of the present disclosure, or with components implementing one or more processes presented anywhere in the description of the various aspects of the present disclosure.

[0098] (6) In a method implemented by an encoder or a decoder according to an embodiment, any process included in the method may be replaced or combined with a process presented anywhere in the description of various aspects of the present disclosure or with any corresponding or equivalent process.

[0099] (7) One or more processes included in the method implemented by an encoder or a decoder according to the embodiment may be combined with a process presented anywhere in the description of various aspects of the present disclosure.

[0100] (8) The implementation of the processes and / or configurations presented in the description of aspects of the present disclosure is not limited to the encoder or decoder according to the embodiments. For example, the processes and / or configurations may be implemented in a device for a purpose different from the motion picture encoder or motion picture decoder disclosed in the embodiments.

[0101] (Encoder)

[0102] First, an encoder according to the embodiment will be described. Figure 1 1 is a block diagram showing a functional configuration of an encoder 100 according to an embodiment. The encoder 100 is a video encoder that encodes a video in units of blocks.

[0103] like Figure 1 As shown, the encoder 100 is a device for encoding an image in block units, including a divider 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116, a block memory 118, a loop filter 120, a frame memory 122, an intra-frame predictor 124, an inter-frame predictor 126 and a prediction controller 128.

[0104] The encoder 100 is implemented as, for example, a general purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor functions as a segmenter 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116, a loop filter 120, an intra-frame predictor 124, an inter-frame predictor 126, and a prediction controller 128. Alternatively, the encoder 100 may be implemented as one or more dedicated electronic circuits corresponding to the segmenter 102, the subtractor 104, the transformer 106, the quantizer 108, the entropy encoder 110, the inverse quantizer 112, the inverse transformer 114, the adder 116, the loop filter 120, the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128.

[0105] Hereinafter, the overall flow of a process performed by the encoder 100 is described, and then each constituent element included in the encoder 100 will be described.

[0106] (Overall flow of the encoding process)

[0107] Figure 2 is a flowchart illustrating one example of the overall encoding process performed by the encoder 100 .

[0108] First, the divider 102 of the encoder 100 divides each picture included in the input image as a video into a plurality of blocks having a fixed size (e.g., 128×128 pixels) (step Sa_1). Then, the divider 102 selects a division pattern (also referred to as a block shape) for the fixed-size block (step Sa_2). In other words, the divider 102 further divides the fixed-size block into a plurality of blocks forming the selected division pattern. For each of the plurality of blocks, the encoder 100 performs steps Sa_3 to Sa_9 for the block (i.e., the current block to be encoded).

[0109] In other words, the prediction processor including all or part of the intra predictor 124, the inter predictor 126 and the prediction controller 128 generates a prediction signal (also referred to as a prediction block) of the current block to be encoded (also referred to as the current block) (step Sa_3).

[0110] Next, the subtractor 104 generates the difference between the current block and the predicted block as a prediction residual (also referred to as a difference block) (step Sa_4).

[0111] Next, the transformer 106 transforms the difference block, and the quantizer 108 quantizes the result to generate a plurality of quantized coefficients (step Sa_5). Note that a block having a plurality of quantized coefficients is also referred to as a coefficient block.

[0112] Next, the entropy encoder 110 encodes the coefficient block and the prediction parameters related to the generation of the prediction signal (specifically, entropy encoding) to generate an encoded signal (step Sa_6). Note that the encoded signal is also called an encoded bit stream, a compressed bit stream or a stream.

[0113] Next, the inverse quantizer 112 inversely quantizes the coefficient block, and the inverse transformer 114 inversely transforms the result to restore a plurality of prediction residuals (ie, difference blocks) (step Sa_7).

[0114] Next, the adder 116 adds the prediction block to the restored difference block to reconstruct the current block into a reconstructed image (also referred to as a reconstructed block or a decoded image block) (step Sa_8). In this way, a reconstructed image is generated.

[0115] When the reconstructed image is generated, the loop filter 120 filters the reconstructed image as necessary (step Sa_9).

[0116] The encoder 100 then determines whether encoding of the entire picture has been completed (step Sa_10). When it is determined that encoding has not been completed (No in step Sa_10), the process from step Sa_2 is repeatedly performed.

[0117] Although in the above example, the encoder 100 selects one partitioning pattern for a fixed-size block and encodes each block according to the partitioning pattern, it should be noted that each block may be encoded according to a corresponding one of a plurality of partitioning patterns. In this case, the encoder 100 may evaluate the cost of each of the plurality of partitioning patterns and, for example, may select an encoded signal obtainable by encoding the encoded signal as output according to the partitioning pattern that produces the minimum cost.

[0118] As shown in the figure, the processes in steps Sa_1 to Sa_10 are sequentially performed by the encoder 100. Alternatively, two or more processes may be performed in parallel, the processes may be reordered, and so on.

[0119] (Splitters)

[0120] The splitter 102 splits each picture included in the input video into a plurality of blocks, and outputs each block to the subtractor 104. For example, the splitter 102 first splits the picture into blocks of a fixed size (e.g., 128×128). Other fixed block sizes may be used. Fixed-size blocks are also referred to as coding tree units (CTUs). The splitter 102 then splits each fixed-size block into blocks of variable sizes (e.g., 64×64 or smaller) based on recursive quadtree and / or binary tree block splitting. In other words, the splitter 102 selects a splitting pattern. Variable-size blocks are also referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). It should be noted that in various processing examples, there is no need to distinguish between CUs, PUs, and TUs. All or some blocks in a picture may be processed in units of CUs, PUs, or TUs.

[0121] Figure 3 is a conceptual diagram showing an example of block segmentation according to an embodiment. Figure 3 , solid lines represent block boundaries of blocks divided by quadtree block partitioning, and dotted lines represent block boundaries of blocks divided by binary tree block partitioning.

[0122] Here, the block 10 is a square block having 128×128 pixels (128×128 block). The 128×128 block 10 is first divided into four square 64×64 blocks (quadtree block division).

[0123] The upper left 64×64 block is further vertically divided into two rectangular 32×64 blocks, and the left 32×64 block is further vertically divided into two rectangular 16×64 blocks (binary tree block division). As a result, the upper left 64×64 block is divided into two 16×64 blocks 11 and 12 and one 32×64 block 13.

[0124] The upper right 64×64 block is partitioned into two rectangular 64×32 blocks 14 and 15 in the horizontal direction (binary tree block partitioning).

[0125] First, the 64×64 block on the lower left is divided into four square 32×32 blocks (quadtree block division). The upper left block and the lower right block among the four 32×32 blocks are further divided. The 32×32 block on the upper left is divided into two rectangular 16×32 blocks in the vertical direction, and the 16×32 block on the right is further divided into two 16×16 blocks in the horizontal direction (binary tree block division). The 32×32 block on the lower right is divided horizontally into two 32×16 blocks (binary tree block division). As a result, the 64×64 block on the lower left is divided into a 16×32 block 16, two 16×16 blocks 17 and 18, two 32×32 blocks 19 and 20, and two 32×16 blocks 21 and 22.

[0126] The lower right 64×64 block 23 is not segmented.

[0127] As mentioned above, in Figure 3 In FIG. 1 , based on recursive quadtree and binary tree block partitioning, the block 10 is partitioned into thirteen variable-sized blocks 11 to 23. This type of partitioning is also called quadtree plus binary tree (QTBT) partitioning.

[0128] It should be noted that in Figure 3 In the example above, a block is partitioned into four or two blocks (quadtree or binary tree block partitioning), but the partitioning is not limited to these examples. For example, a block can be partitioned into three blocks (ternary block partitioning). The partitioning including this ternary block partitioning is also called multi-type tree (MBT) partitioning.

[0129] (Image structure: slice / tile)

[0130] The picture may be configured in units of one or more slices or tiles so that the picture may be decoded in parallel. The picture configured in units of one or more slices or tiles may be configured by the partitioner 102.

[0131] A slice is a basic coding unit contained in a picture. A picture may include, for example, one or more slices. In addition, a slice includes one or more consecutive coding tree units (CTUs).

[0132] Figure 4A : is a conceptual diagram showing an example of a slice configuration. For example, a picture includes 11×8 CTUs and is partitioned into four slices (slices 1-4). Slice 1 contains 16 CTUs, slice 2 contains 21 CTUs, slice 3 contains 29 CTUs, and slice 4 contains 22 CTUs. Here, each CTU in the picture belongs to one of the slices. The shape of each slice is a shape that can be obtained by horizontally partitioning the picture. The boundary of each slice does not need to be consistent with one end of the image, and can be consistent with any boundary between CTUs in the image. The processing order (coding order or decoding order) of the CTUs in the slice is, for example, a raster scan order. The slice includes header information and encoded data. The characteristics of the slice can be described in the header information. These characteristics include the CTU address of the top CTU in the slice, the slice type, etc.

[0133] A tile is a unit of a rectangular area included in a picture. Each tile may be assigned a number called a TileId in a raster scan order.

[0134] Figure 4Bis a conceptual diagram indicating an example of a tile configuration. For example, a picture includes 11×8 CTUs and is partitioned into four rectangular-area tiles (tiles 1 to 4). When tiles are used, the processing order of CTUs is changed from the processing order when tiles are not used. When tiles are not used, CTUs in a picture are processed in raster scan order. When tiles are used, at least one CTU in each tile is processed in raster scan order. For example, Figure 4B As shown, the processing order of the CTUs included in tile 1 is from the left end of the first row of tile 1 to the right end of the first row of tile 1 and then from the left end of the second row of tile 1 to the right end of the second row of tile 1.

[0135] Note that a tile may include one or more slices, and a slice may include one or more tiles.

[0136] (Subtractor)

[0137] The subtractor 104 subtracts the prediction signal (prediction sample input from the prediction controller 128 indicated below) from the original signal (original sample) in units of blocks input from the divider 102 and divided by the divider 102. In other words, the subtractor 104 calculates the prediction error (also referred to as residual) of the block to be encoded (hereinafter also referred to as the current block). The subtractor 104 then outputs the calculated prediction error (residual) to the transformer 106.

[0138] The original signal is a signal that has been input into the encoder 100 and represents an image (eg, a luminance signal and two chrominance signals) of each picture included in the video. Hereinafter, the signal representing the image is also referred to as a sample.

[0139] (Converter)

[0140] The transformer 106 transforms the prediction error in the spatial domain into a transform coefficient in the frequency domain and outputs the transform coefficient to the quantizer 108. More specifically, the transformer 106 applies, for example, a defined discrete cosine transform (DCT) or a discrete sine transform (DST) to the prediction error in the spatial domain. The defined DCT or DST may be predefined.

[0141] It should be noted that the transformer 106 can adaptively select a transform type from among a plurality of transform types, and can transform the prediction error into a transform coefficient by using a transform basis function corresponding to the selected transform type. This type of transform is also called explicit multi-core transform (EMT) or adaptive multi-transform (AMT).

[0142] Transform types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 5Ais a diagram indicating transform basis functions for example transform types. Figure 5A In , N indicates the number of input pixels. For example, selecting a transform type from among a plurality of transform types may depend on a prediction type (one of intra prediction and inter prediction), and may depend on an intra prediction mode.

[0143] Information indicating whether such EMT or AMT is applied (e.g., referred to as an EMT flag or an AMT flag) and information indicating the selected transform type are usually signaled at the CU level. It should be noted that the signaling of such information does not necessarily need to be performed at the CU level, but may be performed at another level (e.g., at a bit sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0144] In addition, the transformer 106 can retransform the transform coefficients (transform results). This retransformation is also called an adaptive secondary transform (AST) or a non-separable secondary transform (NSST). For example, the transformer 106 performs retransformation in units of sub-blocks (e.g., 4×4 sub-blocks) included in the transform coefficient block corresponding to the intra-frame prediction error. Information indicating whether NSST is applied and information about the transform matrix to be used in NSST are typically transmitted by signal at the CU level. It should be noted that the signaling of such information does not necessarily need to be performed at the CU level, but can be performed at another level (e.g., at the sequence level, picture level, slice level, tile level, or CTU level).

[0145] The transformer 106 may employ a separable transform and an inseparable transform. A separable transform is a method in which a transform is performed multiple times by performing a transform separately for each of a plurality of directions according to the dimensionality of the input. An inseparable transform is a method of performing a collective transform in which two or more dimensions in a multi-dimensional input are collectively considered as a single dimension.

[0146] In one example of a non-separable transform, when the input is a 4x4 block, the 4x4 block is treated as a single array containing 16 elements, and the transform applies a 16x16 transform matrix to the array.

[0147] In another example of a non-separable transform, a 4×4 input block is treated as a single array containing 16 elements, and then a transform that applies a number of given rotations to the array (a hypercube given transform) can be performed.

[0148] In the transformation in the transformer 106, the type of basis to be transformed into the frequency domain may be switched according to the region in the CU. Examples include a spatially varying transform (SVT). Figure 5BAs shown, in SVT, the CU is divided into two equal regions in the horizontal or vertical direction, and only one region is transformed to the frequency domain. The transform base type can be set for each region. For example, DST7 and DST8 are used. In this example, only one of the two regions in the CU is transformed, and the other is not transformed. However, both regions can be transformed. In addition, the partitioning method is not limited to partitioning into two equal regions, but can be more flexible. For example, the CU can be partitioned into four equal regions, or information indicating the partitioning can be encoded separately and transmitted with a signal in the same way as the CU partitioning. It should be noted that SVT is also called sub-block transform (SBT).

[0149] (Quantizer)

[0150] The quantizer 108 quantizes the transform coefficients output from the transformer 106. More specifically, the quantizer 108 scans the transform coefficients of the current block in a determined scanning order, and quantizes the scanned transform coefficients based on quantization parameters (QP) corresponding to the transform coefficients. The quantizer 108 then outputs the quantized transform coefficients of the current block (hereinafter also referred to as quantized coefficients) to the entropy encoder 110 and the inverse quantizer 112. The determined scanning order may be predetermined.

[0151] The determined scanning order is the order for quantizing / inverse quantizing the transform coefficients. For example, the determined scanning order may be defined as an ascending order of frequency (from low frequency to high frequency) or a descending order of frequency (from high frequency to low frequency).

[0152] The quantization parameter (QP) is a parameter that defines the quantization step size (quantization width). For example, when the value of the quantization parameter increases, the quantization step size also increases. In other words, when the value of the quantization parameter increases, the quantization error increases.

[0153] In addition, a quantization matrix can be used for quantization. For example, several quantization matrices can be used corresponding to the following: frequency transform size (e.g., 4×4 and 8×8), prediction modes such as intra-frame prediction and inter-frame prediction, and pixel components such as brightness and chrominance pixel components. It should be noted that quantization refers to digitizing values ​​sampled at intervals determined corresponding to a determined level. In the art, other expressions such as rounding and scaling can be used to refer to quantization, and quantization can employ rounding and scaling. The determined intervals and levels can be predetermined.

[0154] The method of using the quantization matrix includes a method of using the quantization matrix set directly on the encoder side and a method of using the quantization matrix set to the default (default matrix). On the encoder side, the quantization matrix suitable for the characteristics of the image can be set by directly setting the quantization matrix. However, this case has the disadvantage of increasing the amount of code for encoding the quantization matrix.

[0155] There is a method of quantizing high-frequency coefficients and low-frequency coefficients without using a quantization matrix. Note that this method is equivalent to a method using a quantization matrix (a flat matrix) whose coefficients have the same value.

[0156] The quantization matrix may be specified using, for example, a sequence parameter set (SPS) or a picture parameter set (PPS). The SPS includes parameters for a sequence, and the PPS includes parameters for a picture. Each of the SPS and the PPS may be referred to simply as a parameter set.

[0157] (Entropy Encoder)

[0158] The entropy encoder 110 generates an encoded signal (encoded bit stream) based on the quantization coefficients that have been input from the quantizer 108. More specifically, for example, the entropy encoder 110 binarizes the quantization coefficients and arithmetically encodes the binary signal, and outputs a compressed bit stream or sequence.

[0159] (Inverse Quantizer)

[0160] The inverse quantizer 112 inversely quantizes the quantized coefficients that have been input from the quantizer 108. More specifically, the inverse quantizer 112 inversely quantizes the quantized coefficients of the current block in a determined scanning order. The inverse quantizer 112 then outputs the inversely quantized transform coefficients of the current block to the inverse transformer 114. The determined scanning order may be predetermined.

[0161] (Inverter)

[0162] The inverse transformer 114 restores the prediction error (residual) by inversely transforming the transform coefficients input from the inverse quantizer 112. More specifically, the inverse transformer 114 restores the prediction error of the current block by applying an inverse transform corresponding to the transform applied by the transformer 106 to the transform coefficients. The inverse transformer 114 then outputs the restored prediction error to the adder 116.

[0163] Note that, since information is lost in the quantization, the recovered prediction error does not match the prediction error calculated by the subtractor 104. In other words, the recovered prediction error typically includes the quantization error.

[0164] (Adder)

[0165] The adder 116 reconstructs the current block by adding the prediction error input from the inverse transformer 114 and the prediction sample input from the prediction controller 128. The adder 116 then outputs the reconstructed block to the block memory 118 and the loop filter 120. The reconstructed block is also called a local decoded block.

[0166] (Block Storage)

[0167] The block memory 118 is, for example, a storage device for storing a block in a picture to be encoded (hereinafter referred to as a current picture) to be referenced in intra prediction. More specifically, the block memory 118 stores the reconstructed block output from the adder 116.

[0168] (Frame Memory)

[0169] The frame memory 122 is, for example, a storage device for storing a reference picture used in inter-frame prediction, and is also referred to as a frame buffer. More specifically, the frame memory 122 stores the reconstructed block filtered by the loop filter 120.

[0170] (Loop Filter)

[0171] The loop filter 120 applies a loop filter to the block reconstructed by the adder 116, and outputs the filtered reconstructed block to the frame memory 122. The loop filter is a filter used in an encoding loop (an in-loop filter), and includes, for example, a deblocking filter (DF or DBF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF).

[0172] In ALF, a least square error filter for eliminating compression artifacts is applied. For example, for each of the 2×2 sub-blocks in the current block, one filter selected from among multiple filters based on the direction and activity of the local gradient is applied.

[0173] More specifically, first, each sub-block (e.g., each 2×2 sub-block) is classified into one of a plurality of categories (e.g., fifteen or twenty-five categories). The classification of the sub-block is based on gradient directionality and activity. For example, a classification index C (e.g., C=5D+A) is derived based on gradient directionality D (e.g., 0 to 2 or 0 to 4) and gradient activity A (e.g., 0 to 4). Then, based on the classification index C, each sub-block is classified into one of a plurality of categories.

[0174] For example, the gradient directivity D is calculated by comparing the gradients in a plurality of directions (eg, horizontal direction, vertical direction, and two diagonal directions). Also, the gradient activity A is calculated, for example, by adding the gradients in a plurality of directions and quantizing the addition result.

[0175] Based on the result of this classification, a filter to be used for each sub-block is determined from among a plurality of filters.

[0176] The filter shape used in the ALF is, for example, a circularly symmetric filter shape. FIG. 6A to FIG. 6C An example of filter shape used in ALF is shown. Fig. 6A A 5x5 diamond filter is shown. Figure 6B A 7x7 diamond filter is shown. Figure 6C A 9×9 diamond filter is shown. Information indicating the filter shape is typically signaled at the picture level. Note that signaling of information indicating the filter shape does not necessarily need to be performed at the picture level, but can be performed at another level (e.g., at the sequence level, slice level, tile level, CTU level, or CU level).

[0177] For example, the turning on or off of ALF is determined at the picture level or the CU level. For example, a decision on whether to apply ALF to luminance may be made at the CU level, and a decision on whether to apply ALF to chrominance may be made at the picture level. Information indicating whether ALF is turned on or off is typically sent at the picture level or the CU level. Note that the signaling of information indicating whether ALF is turned on or off does not necessarily need to be performed at the picture level or the CU level, and may be performed at another level (e.g., at the sequence level, the slice level, the tile level, or the CTU level).

[0178] The coefficients set for a plurality of selectable filters (e.g., fifteen or up to twenty-five filters) are typically signaled at the picture level. It should be noted that the signaling of the set coefficients does not necessarily need to be performed at the picture level, but may be performed at another level (e.g., at the sequence level, slice level, tile level, CTU level, CU level, or sub-block level).

[0179] (Loop Filter > Deblocking Filter)

[0180] In the deblocking filter, the loop filter 120 performs a filtering process on a block boundary in a reconstructed image in order to reduce distortion occurring at the block boundary.

[0181] Figure 7 is a block diagram showing one example of a specific configuration of the loop filter 120 serving as a deblocking filter.

[0182] The loop filter 120 includes: a boundary determiner 1201 ; a filter determiner 1203 ; a filter executor 1205 ; a process determiner 1208 ; a filter characteristic determiner 1207 ; and switches 1202 , 1204 , and 1206 .

[0183] The boundary determiner 1201 determines whether there is a pixel to be deblocking filtered (ie, the current pixel) around the block boundary. The boundary determiner 1201 then outputs the determination result to the switch 1202 and the processing determiner 1208.

[0184] In the case where the boundary determiner 1201 has determined that the current pixel exists around the block boundary, the switch 1202 outputs the unfiltered image to the switch 1204. In the opposite case (where the boundary determiner 1201 determines that the current pixel does not exist around the block boundary), the switch 1202 outputs the unfiltered image to the switch 1206.

[0185] The filter determiner 1203 determines whether to perform deblocking filtering on the current pixel based on the pixel value of at least one surrounding pixel located around the current pixel. Then, the filter determiner 1203 outputs the determination result to the switch 1204 and the processing determiner 1208.

[0186] In the case where the filter determiner 1203 has determined that the deblocking filtering is to be performed on the current pixel, the switch 1204 outputs the unfiltered image obtained by the switch 1202 to the filter executor 1205. In the opposite case (where the filter determiner 1203 has determined that the deblocking filtering is not to be performed on the current pixel), the switch 1204 outputs the unfiltered image obtained by the switch 1202 to the switch 1206.

[0187] When the unfiltered image is obtained through the switches 1202 and 1204, the filter executor 1205 performs deblocking filtering of the filter characteristics determined by the filter characteristic determiner 1207 on the current pixel. The filter executor 1205 then outputs the filtered pixel to the switch 1206.

[0188] Under the control of the processing determiner 1208 , the switch 1206 selectively outputs pixels that have not been deblock filtered and pixels that have been deblock filtered by the filter executor 1205 .

[0189] The processing determiner 1208 controls the switch 1206 based on the determination results made by the boundary determiner 1201 and the filter determiner 1203. In other words, when the boundary determiner 1201 has determined that the block boundary period exists in the current pixel and the filter determiner 1203 has determined that the deblocking filtering is performed on the current pixel, the processing determiner 1208 causes the switch 1206 to output the pixel that has been deblocking filtered. In addition, in addition to the above case, the processing determiner 1208 causes the switch 1206 to output the pixel that has not been deblocking filtered. By repeating the output of the pixel in this way, the filtered image is output from the switch 1206.

[0190] Figure 8 is a conceptual diagram illustrating an example of a deblocking filter having symmetric filtering characteristics with respect to a block boundary.

[0191] In the deblocking filter process, the pixel value and the quantization parameter are used to select one of two deblocking filters with different characteristics, namely a strong filter and a weak filter. In the case of a strong filter, such as Figure 8As shown, pixels p0 to p2 and pixels q0 to q2 exist across the block boundary, and the pixel values ​​of the corresponding pixels q0 to q2 are changed to pixel values ​​q'0 to q'2 by, for example, calculating according to the following expression.

[0192] q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8

[0193] q'1=(p0+q0+q1+q2+2) / 4

[0194] q'2=(p0+q0+q1+3×q2+2×q3+4) / 8

[0195] Note that in the above expressions, p0 to p2 and q0 to q2 are pixel values ​​of the corresponding pixels p0 to p2 and pixels q0 to q2. In addition, q3 is the pixel value of the adjacent pixel q3 located on the opposite side of the pixel q2 with respect to the block boundary. In addition, on the right side of each expression, the coefficient multiplied by the corresponding pixel value of the pixel to be used for deblocking filtering is the filter coefficient.

[0196] In addition, in the deblocking filter, clipping can be performed so that the calculated pixel value is not set to be higher than the threshold value. In the clipping process, the pixel value calculated according to the above expression is clipped to a value obtained according to "calculated pixel value ± 2 × threshold value" using a threshold value determined based on a quantization parameter. In this way, excessive smoothing can be prevented.

[0197] Fig. 9 is a conceptual diagram for illustrating a block boundary on which a deblocking filtering process is performed. Fig.10 is a conceptual diagram indicating an example of a Bs value.

[0198] The block boundaries on which the deblocking filtering process is performed are, for example, Fig. 9 The deblocking filtering process can be performed in units of four rows or four columns. First, for Fig. 9 The blocks P and Q shown in Fig.10 The boundary strength (Bs) value is determined as indicated.

[0199] according to Fig.10 The Bs value in the image determines whether to use different strengths to perform the deblocking filtering process on the block boundaries belonging to the same image. When the Bs value is 2, the deblocking filtering process for the chrominance signal is performed. When the Bs value is 1 or more and the determination condition is met, the deblocking filtering process is performed on the luminance signal. The determination condition can be predetermined. It should be noted that the condition for determining the Bs value is not limited to Fig.10 those shown in , and the Bs value can be determined based on another parameter.

[0200] (Prediction processor (intra-frame predictor, inter-frame predictor, prediction controller))

[0201] Fig.11 is a flowchart showing one example of a process performed by the prediction processor of the encoder 100. Note that the prediction processor includes all or part of the following constituent elements: an intra predictor 124; an inter predictor 126; and a prediction controller 128.

[0202] The prediction processor generates a prediction image of the current block (step Sb_1). The prediction image is also called a prediction signal or a prediction block. It should be noted that the prediction signal is, for example, an intra-frame prediction signal or an inter-frame prediction signal. Specifically, the prediction processor uses the prediction image of the current block that has been generated by generating a prediction block, generating a difference block, generating a coefficient block, restoring a difference block, and generating a decoded image block.

[0203] The reconstructed image may be, for example, an image in a reference picture, or an image of a coding block in a current picture, where the current picture is a picture including the current block. The coding block in the current picture is, for example, an adjacent block of the current block.

[0204] Fig.12 is a flow chart illustrating another example of a process performed by the prediction processor of encoder 100 .

[0205] The prediction processor generates a prediction image using a first method (step Sc_1a), generates a prediction image using a second method (step Sc_1b), and generates a prediction image using a third method (step Sc_1c). The first method, the second method, and the third method may be different methods for generating a prediction image. Each of the first to third methods may be an inter-frame prediction method, an intra-frame prediction method, or another prediction method. The above-mentioned reconstructed image may be used in these prediction methods.

[0206] Next, the prediction processor selects any one of the multiple prediction methods generated in steps Sc_1a, Sc_1b and Sc_1c (step Sc_2). The selection of the prediction image (i.e., the selection of the method or mode for obtaining the final prediction image) can be performed by calculating a cost for each generated prediction image and based on the cost. Alternatively, the selection of the prediction image can be based on the parameters used in the encoding process. The encoder 100 can convert information for identifying the selected prediction image, method or mode into a coded signal (also called a coded bitstream). The information can be, for example, a flag, etc. In this way, the decoder is able to generate a prediction image according to the method or mode selected based on the information in the encoder 100. It should be noted that in Fig.12In the example shown, after the prediction image is generated using the corresponding method, the prediction processor selects any prediction image. However, the prediction processor can select a method or mode based on the parameters used in the above encoding process before generating the prediction image, and can generate the prediction image according to the selected method or mode.

[0207] For example, the first method and the second method may be intra prediction and inter prediction, respectively, and the prediction processor may select a final prediction image for the current block from prediction images generated according to the prediction methods.

[0208] Fig.13 is a flow chart illustrating another example of a process performed by the prediction processor of encoder 100 .

[0209] First, the prediction processor generates a prediction image using intra prediction (step Sd_1a), and generates a prediction image using inter prediction (step Sd_1b). Note that the prediction image generated by intra prediction is also referred to as an intra prediction image, and the prediction image generated by inter prediction is also referred to as an inter prediction image.

[0210] Next, the prediction processor evaluates each of the intra-frame prediction image and the inter-frame prediction image (step Sd_2). The cost can be used in the evaluation. In other words, the prediction processor calculates the cost C for each of the intra-frame prediction image and the inter-frame prediction image. The cost C can be calculated according to the expression of the RD optimization model, for example, C=D+λ×R. In this expression, D represents the encoding distortion of the predicted image, and is expressed as, for example, the sum of the absolute differences between the pixel values ​​of the current block and the pixel values ​​of the predicted image. In addition, R represents the predicted encoding amount of the predicted image, specifically, the amount of encoding required to encode the motion information, etc. used to generate the predicted image. In addition, λ represents a multiplier, for example, according to the method of the Lagrange multiplier.

[0211] Then, the prediction processor selects the prediction image for which the minimum cost C has been calculated among the intra prediction image and the inter prediction image as the final prediction image of the current block (step Sd_3). In other words, a prediction method or mode for generating the prediction image of the current block is selected.

[0212] (Intra Predictor)

[0213] The intra predictor 124 generates a prediction signal (intra prediction signal) by performing intra prediction (also referred to as intra prediction) of the current block with reference to one or more blocks in the current picture and stored in the block memory 118. More specifically, the intra predictor 124 generates the intra prediction signal by performing intra prediction with reference to samples (e.g., luminance and / or chrominance values) of one or more blocks adjacent to the current block, and then outputs the intra prediction signal to the prediction controller 128.

[0214] For example, the intra predictor 124 performs intra prediction by using one of a plurality of intra prediction modes that have been defined. The intra prediction modes include one or more non-directional prediction modes and a plurality of directional prediction modes. The defined modes may be predefined.

[0215] The one or more non-directional prediction modes include, for example, planar prediction mode and DC prediction mode defined in the H.265 / High Efficiency Video Coding (HEVC) standard.

[0216] The plurality of directional prediction modes include, for example, 33 directional prediction modes defined in the H.265 / HEVC standard. It should be noted that, in addition to the 33 directional prediction modes, the plurality of directional prediction modes may also include 32 directional prediction modes (a total of 65 directional prediction modes). Fig.14 : is a conceptual diagram showing a total of sixty-seven intra-frame prediction modes (two non-directional prediction modes and sixty-five directional prediction modes) that can be used in intra-frame prediction. The solid arrows represent the 33 directions defined in the H.265 / HEVC standard, while the dotted arrows represent the additional 32 directions ( Fig.14 The two non-directional prediction modes are not shown).

[0217] In various processing examples, a luma block may be referenced in intra prediction of a chroma block. In other words, a chroma component of a current block may be predicted based on a luma component of the current block. Such intra prediction is also referred to as cross-component linear model (CCLM) prediction. An intra prediction mode for a chroma block that references such a luma block (also referred to as, for example, a CCLM mode) may be added as one of the intra prediction modes for a chroma block.

[0218] The intra predictor 124 may correct the pixel values ​​of the intra prediction based on the horizontal / vertical reference pixel gradients. The intra prediction with such correction is also referred to as position-dependent intra prediction combination (PDPC). Information indicating whether PDPC is applied (e.g., referred to as a PDPC flag) is typically signaled at the CU level. It should be noted that the signaling of such information does not necessarily need to be performed at the CU level, but may be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0219] (Inter-frame predictor)

[0220] The inter-frame predictor 126 generates a prediction signal (inter-frame prediction signal) by performing inter-frame prediction (also called inter-frame prediction) of the current block with reference to one or more blocks in a reference picture (which is different from the current picture and is stored in the frame memory 122). Inter-frame prediction is performed in units of the current block or the current sub-block (e.g., 4×4 block) in the current block. For example, the inter-frame predictor 126 performs motion estimation in the reference picture of the current block or the current sub-block, and finds the reference block or sub-block that best matches the current block or the current sub-block. Then, the inter-frame predictor 126 obtains motion information (e.g., motion vector) that compensates for the motion or change from the reference block or sub-block to the current block or sub-block. The inter-frame predictor 126 generates an inter-frame prediction signal of the current block or sub-block by performing motion compensation (or motion prediction) based on the motion information. The inter-frame predictor 126 outputs the generated inter-frame prediction signal to the prediction controller 128.

[0221] The motion information used in motion compensation can be signaled in various forms as an inter-frame prediction signal. For example, a motion vector can be signaled. As another example, the difference between a motion vector and a motion vector predictor can be signaled.

[0222] (Basic process of inter-frame prediction)

[0223] Fig.15 is a flowchart illustrating an example basic processing flow for inter-frame prediction.

[0224] First, the inter-frame predictor 126 generates a prediction signal (steps Se_1 to Se_3). Next, the subtractor 104 generates a difference between the current block and the predicted image as a prediction residual (step Se_4).

[0225] Here, in the generation of the predicted image, the inter-frame predictor 126 generates the predicted image by determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and motion compensation (step Se_3). In addition, when determining the MV, the inter-frame predictor 126 determines the MV by selecting a motion vector candidate (MV candidate) (step Se_1) and deriving the MV (step Se_2). The selection of the MV candidate is performed by, for example, selecting at least one MV candidate from the MV candidate list. Alternatively, in the derivation of the MV, the inter-frame predictor 126 may further select at least one MV candidate from at least one MV candidate, and determine the selected at least one MV candidate as the MV of the current block. Alternatively, the inter-frame predictor 126 may determine the MV of the current block by performing estimation in a reference picture area specified by each of the selected at least one MV candidate. Note that the estimation in the reference picture area may be referred to as motion estimation.

[0226] In addition, although steps Se_1 to Se_3 are performed by the inter-frame predictor 126 in the above-described example, the process (which is, for example, step Se_1, step Se_2, etc.) may be performed by another constituent element included in the encoder 100.

[0227] (Motion vector export process)

[0228] Fig.16 is a flow chart illustrating one example of motion vector derivation.

[0229] The inter-frame predictor 126 derives the MV of the current block in a mode (e.g., MV) for encoding motion information. In this case, for example, the motion information is encoded as a prediction parameter and transmitted with a signal. In other words, the encoded motion information is included in the encoded signal (also referred to as an encoded bitstream).

[0230] Alternatively, the inter-frame predictor 126 derives the MV in a mode in which motion information is not encoded. In this case, motion information is not included in the encoded signal.

[0231] Here, the MV derivation mode may include a conventional inter mode, a merge mode, a FRUC mode, an affine mode, etc., which are described later. Modes in which motion information is encoded include a conventional inter mode, a merge mode, an affine mode (specifically, an affine inter mode and an affine merge mode), etc. Note that the motion information may include not only the MV but also motion vector predictor selection information, which will be described later. Modes in which no motion information is encoded include the FRUC mode, etc. The inter predictor 126 selects a mode for deriving the MV of the current block from these modes, and uses the selected mode to derive the MV of the current block.

[0232] Fig.17 is a flow chart illustrating another example of derivation of motion vectors.

[0233] The inter-frame predictor 126 derives the MV of the current block in a mode of encoding the MV difference. In this case, for example, the MV difference is encoded as a prediction parameter and transmitted with a signal. In other words, the encoded MV difference is included in the encoded signal. The MV difference is the difference between the MV of the current block and the MV predictor.

[0234] Alternatively, the inter-frame predictor 126 derives the MV in a mode in which the MV difference is not encoded. In this case, the encoded MV difference is not included in the encoded signal.

[0235] Here, as described above, the MV derivation mode includes a normal inter mode, a merge mode, a FRUC mode, an affine mode, etc. described later. The mode in which the MV difference is encoded includes a normal inter mode, an affine mode (specifically, an affine inter mode), etc. The mode in which the MV difference is not encoded includes a FRUC mode, a merge mode, an affine mode (specifically, an affine merge mode), etc. The inter predictor 126 selects a mode for deriving the MV of the current block from a plurality of modes, and derives the MV of the current block using the selected mode.

[0236] (Motion vector export process)

[0237] Fig.18 is a flowchart showing another example of the derivation of a motion vector. The MV derivation mode as an inter-frame prediction mode includes a plurality of modes, and is roughly divided into a mode in which the MV difference is encoded and a mode in which the motion vector difference is not encoded. The modes in which the MV difference is not encoded include a merge mode, a FRUC mode, an affine mode (specifically, an affine merge mode), and the like. These modes will be described in detail later. In short, the merge mode is a mode for deriving the MV of the current block by selecting a motion vector from the encoded surrounding blocks, and the FRUC mode is a mode for deriving the MV of the current block by performing estimation between the encoded regions. The affine mode is a mode for deriving a motion vector of each of a plurality of sub-blocks included in the current block as the MV of the current block (assuming an affine transformation).

[0238] More specifically, as shown, when the inter-frame prediction mode information indicates 0 (0 in Sf_1), the inter-frame predictor 126 uses the merge mode (Sf_2) to derive the motion vector. When the inter-frame prediction mode information indicates 1 (1 in Sf_1), the inter-frame predictor 126 uses the FRUC mode (Sf_3) to derive the motion vector. When the inter-frame prediction mode information indicates 2 (2 in Sf_1), the inter-frame predictor 126 uses the affine mode (specifically, the affine merge mode) (Sf_4) to derive the motion vector. When the inter-frame prediction mode information indicates 3 (3 in Sf_1), the inter-frame predictor 126 uses a mode in which the MV difference is encoded (e.g., the conventional inter-frame mode (Sf_5)) to derive the motion vector.

[0239] (MV Export > Regular Interframe Mode)

[0240] The normal inter mode is an inter prediction mode for deriving the MV of the current block from a block similar to the image of the current block in a reference picture area specified by the MV candidate. In this normal inter mode, the MV difference is encoded.

[0241] Fig.19is a flowchart illustrating an example of inter prediction in normal inter mode.

[0242] First, the inter-frame predictor 126 obtains a plurality of MV candidates for the current block based on information such as MVs of a plurality of coding blocks temporally or spatially surrounding the current block (step Sg_1). In other words, the inter-frame predictor 126 generates a MV candidate list.

[0243] Next, the inter-frame predictor 126 extracts N (2 or a greater integer) MV candidates as motion vector predictor candidates (also referred to as MV predictor candidates) from the plurality of MV candidates obtained in step Sg_1 according to the determined priority order (step Sg_2). Note that the priority order may be predetermined for each of the N MV candidates.

[0244] Next, the inter-frame predictor 126 selects one motion vector predictor candidate from the N motion vector predictor candidates as the motion vector predictor (also referred to as MV predictor) of the current block (step Sg_3). At this time, the inter-frame predictor 126 encodes motion vector predictor selection information for identifying the selected motion vector predictor in the stream. It should be noted that the stream is a coded signal or a coded bit stream as described above.

[0245] Next, the inter-frame predictor 126 derives the MV of the current block by referring to the encoded reference picture (step Sg_4). At this time, the inter-frame predictor 126 also encodes the difference between the derived MV and the motion vector predictor as the MV difference in the stream. It should be noted that the encoded reference picture is a picture including multiple blocks that have been reconstructed after encoding.

[0246] Finally, the inter-frame predictor 126 generates a predicted image of the current block by performing motion compensation of the current block using the derived MV and the encoded reference picture (step Sg_5). Note that, as described above, the predicted image is an inter-frame prediction signal.

[0247] In addition, information indicating the inter prediction mode (normal inter mode in the above example) used to generate the predicted image is encoded as a prediction parameter, for example.

[0248] It should be noted that the MV candidate list can also be used as a list used in another mode. In addition, the process related to the MV candidate list can be applied to the process related to the list for use in another mode. The process related to the MV candidate list includes, for example, extracting or selecting MV candidates from the MV candidate list, reordering of MV candidates, or deletion of MV candidates.

[0249] (MV Export > Merge Mode)

[0250] The merge mode is an inter-frame prediction mode, which is used to select an MV candidate from an MV candidate list as the MV of the current block, thereby deriving the MV.

[0251] Fig. 20 is a flowchart illustrating an example of inter prediction in merge mode.

[0252] First, the inter-frame predictor 126 obtains a plurality of MV candidates for the current block based on information such as MVs of a plurality of coding blocks temporally or spatially surrounding the current block (step Sh_1). In other words, the inter-frame predictor 126 generates a MV candidate list.

[0253] Next, the inter-frame predictor 126 selects one MV candidate from the multiple MV candidates obtained in step Sh_1, thereby deriving the MV of the current block (step Sh_2). At this time, the inter-frame predictor 126 encodes MV selection information for identifying the selected MV candidate in the stream.

[0254] Finally, the inter predictor 126 generates a predicted image of the current block by performing motion compensation of the current block using the derived MV and the encoded reference picture (step Sh_3).

[0255] In addition, for example, information indicating the inter prediction mode (merge mode in the above example) used to generate a predicted image and included in the encoded signal is encoded as a prediction parameter.

[0256] Fig.21 is a conceptual diagram for illustrating an example of a motion vector derivation process of a current picture in a merge mode.

[0257] First, an MV candidate list is generated in which MV predictor candidates are registered. Examples of MV predictor candidates include: spatially adjacent MV predictors, which are MVs of multiple coding blocks spatially surrounding the current block; temporally adjacent MV predictors, which are MVs of surrounding blocks on which the position of the current block in the coding reference picture is projected; combined MV predictors, which are MVs generated by combining the MV values ​​of spatially adjacent MV predictors and the MVs of temporally adjacent MV predictors; and zero MV predictors, which are MVs having a zero value.

[0258] Next, one MV predictor is selected from among a plurality of MV predictors registered in the MV predictor list, and the selected MV predictor is determined as the MV of the current block.

[0259] Furthermore, the variable length encoder describes and encodes merge_idx, which is a signal indicating which MV predictor has been selected, in the stream.

[0260] It should be noted that in Fig.21The MV predictors registered in the MV predictor list described in are examples. The number of MV predictors may be different from the number of MV predictors in the figure, and the MV predictor list may be configured in the following manner: some of the types of MV predictors in the figure may not be included, or one or more MV predictors are included in addition to the types of MV predictors in the figure.

[0261] The final MV may be determined by performing a decoder motion vector refinement process (DMVR) to be described later, using the MV of the current block derived in the merge mode.

[0262] Note that the MV predictor candidate is the above-mentioned MV candidate, and the MV predictor list is the above-mentioned MV candidate list. Note that the MV candidate list may be referred to as a candidate list. In addition, merge_idx is MV selection information.

[0263] (MV Export > FRUC Mode)

[0264] The motion information may be derived on the decoder side without being signaled from the encoder side. Note that, as described above, the merge mode defined in the H.265 / HEVC standard may be used. Additionally, for example, the motion information may be derived by performing motion estimation on the decoder side. In an embodiment, on the decoder side, motion estimation is performed without using any pixel values ​​in the current block.

[0265] Here, a mode for performing motion estimation at the decoder side is described. The mode for performing motion estimation at the decoder side may be referred to as a Pattern Matching Motion Vector Derivation (PMMVD) mode or a Frame Rate Up Conversion (FRUC) mode.

[0266] Fig. 22An example of the FRUC process is shown in the form of a flowchart. First, a list of multiple candidates each having a motion vector (MV) predictor (i.e., a MV candidate list that can also be used as a merge list) is generated by referring to the motion vector in the coding block that is adjacent to the current block in space or time (step Si_1). Next, the best MV candidate is selected from the multiple MV candidates registered in the MV candidate list (step Si_2). For example, the evaluation values ​​of the corresponding MV candidates included in the MV candidate list are calculated, and an MV candidate is selected based on the evaluation value. Based on the selected candidate motion vector, the motion vector of the current block is then derived (step Si_4). More specifically, for example, the selected motion vector candidate (best MV candidate) is directly derived as the motion vector of the current block. In addition, for example, the motion vector of the current block can be derived using pattern matching in the surrounding area of ​​the reference picture whose position in the reference picture corresponds to the selected motion vector candidate. In other words, pattern matching and evaluation values ​​can be used in the surrounding area of ​​the best MV candidate to perform estimation, and when there is an MV that produces a better evaluation value, the best MV candidate can be updated to an MV that produces a better evaluation value, and the updated MV can be determined as the final MV of the current block. A configuration is also possible in which a process for updating the best MV candidate to an MV with a better evaluation value is not performed.

[0267] Finally, the inter-frame predictor 126 generates a predicted image of the current block by performing motion compensation of the current block using the derived MV and the encoded reference picture (step Si_5).

[0268] A similar process may be performed in units of sub-blocks.

[0269] The evaluation value may be calculated according to various methods. For example, a comparison may be made between a reconstructed image in a region in a reference picture corresponding to a motion vector and a reconstructed image in a determined region (the determined region may be, for example, a region in another reference picture or a region in an adjacent block of a current picture, as shown below). The determined region may be predetermined.

[0270] The difference between the pixel values ​​of the two reconstructed images can be used for the evaluation value of the motion vector. Note that information other than the difference value can be used to calculate the evaluation value.

[0271] Next, an example of pattern matching is described in detail. First, through pattern matching, an MV candidate included in the MV candidate list (e.g., merged list) is selected as the starting point of estimation. For example, as pattern matching, first pattern matching or second pattern matching can be used. First pattern matching and second pattern matching are also called bilateral matching and template matching, respectively.

[0272] (MV Export > FRUC > Bilateral Matching)

[0273] In the first pattern matching, pattern matching is performed between two blocks along the motion trajectory of the current block, the two blocks being two blocks in two different reference pictures. Therefore, in the first pattern matching, an area in another reference picture along the motion trajectory of the current block is used as a determination area for calculating the evaluation value of the above-mentioned candidate. The determination area may be predetermined.

[0274] Fig.23 is a conceptual diagram for illustrating an example of first pattern matching (bilateral matching) between two blocks in two reference pictures along a motion trajectory. Fig.23 As shown, in the first mode matching, two motion vectors (MV0, MV1) are derived by estimating the best matching pair among the pairs of two blocks in two different reference pictures (Ref0, Ref1) which are two blocks along the motion trajectory of the current block (current block). More specifically, the difference between the reconstructed image at the specified position in the first coded reference picture (Ref0) specified by the MV candidate and the reconstructed image at the specified position in the second coded reference picture (Ref1) specified by the symmetric MV is derived for the current block, the symmetric MV being obtained by scaling the MV candidate at a display time interval, and the obtained difference value is used to calculate the evaluation value. The MV candidate that produces the best evaluation value among multiple MV candidates and is likely to produce good results can be selected as the final MV.

[0275] Under the assumption of a continuous motion trajectory, the motion vectors (MV0, MV1) of the two reference blocks are specified to be proportional to the temporal distance (TD0, TD1) between the current picture (current picture) and the two reference pictures (Ref0, Ref1). For example, when the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the corresponding two reference pictures are equal to each other, a mirror-symmetric bidirectional motion vector is derived in the first pattern matching.

[0276] (MV Export > FRUC > Template Matching)

[0277] In the second pattern matching (template matching), pattern matching is performed between a block in the reference picture and a template in the current picture (a template is a block adjacent to the current block in the current picture (the adjacent block is, for example, an adjacent block on the upper and / or left side)). Therefore, in the second pattern matching, a block adjacent to the current block in the current picture is used as a determination area for calculating the evaluation value of the above-mentioned candidate.

[0278] Fig.24 is a conceptual diagram for illustrating an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture. Fig.24As shown, in the second mode matching, the motion vector of the current block (current block) is derived by estimating a block in the reference picture (Ref0) that best matches a block adjacent to the current block in the current picture (current picture). More specifically, it is possible to derive a difference between a reconstructed image in a coding area adjacent to the left and above or adjacent to the left or above and a reconstructed image in a corresponding area in the coding reference picture (Ref0) and specified by an MV candidate, calculate an evaluation value using the value of the obtained difference, and select an MV candidate that produces the best evaluation value among multiple MV candidates as the best MV candidate.

[0279] Such information indicating whether the FRUC mode is applied (e.g., referred to as a FRUC flag) may be signaled at the CU level. In addition, when the FRUC mode is applied (e.g., when the FRUC flag is true), information indicating an applicable mode matching method (first mode matching or second mode matching) may be signaled at the CU level. It should be noted that the signaling of such information does not necessarily need to be performed at the CU level, but may be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0280] (MV Export > Affine Mode)

[0281] Next, the affine mode for deriving a motion vector in units of subblocks based on motion vectors of a plurality of neighboring blocks is described. This mode is also called the affine motion compensation prediction mode.

[0282] Fig.25A : is a conceptual diagram for illustrating an example of deriving a motion vector for each sub-block based on motion vectors of a plurality of adjacent blocks. Fig.25A In the example, the current block includes sixteen 4×4 sub-blocks. Here, the motion vector V0 at the upper left corner control point in the current block is derived based on the motion vectors of the adjacent blocks, and similarly, the motion vector V1 at the upper right corner control point in the current block is derived based on the motion vectors of the adjacent sub-blocks. The two motion vectors v0 and v1 can be projected according to the expression (1A) indicated below, and the motion vector (v x , v y ).

[0283] [Mathematical formula 1]

[0284]

[0285] Here, x and y represent the horizontal position and the vertical position of the sub-block, respectively, and w represents the determined weighting coefficient. The determined weighting coefficient may be predetermined.

[0286] Such information indicating the affine mode (e.g., called an affine flag) may be signaled at the CU level. It should be noted that the signaling of the information indicating the affine mode does not necessarily need to be performed at the CU level, and may be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0287] In addition, the affine mode may include several modes for different methods of deriving motion vectors at the upper left control point and the upper right control point. For example, the affine mode includes two modes, which are the affine inter mode (also called the affine normal inter mode) and the affine merge mode.

[0288] (MV Export > Affine Mode)

[0289] Fig.25B : is a conceptual diagram for illustrating an example of deriving a motion vector for each sub-block in an affine mode using three control points. Fig.25B , the current block includes sixteen 4×4 blocks. Here, the motion vector V0 at the upper left corner control point of the current block is derived based on the motion vector of the adjacent block, and similarly, the motion vector V1 at the upper right corner control point of the current block is derived based on the motion vector of the adjacent block, and the motion vector V2 at the lower left corner control point of the current block is derived based on the motion vector of the adjacent block. The three motion vectors v0, v1, and v2 can be projected according to the expression (1B) indicated below, and the motion vector (v x , v y ).

[0290] [Mathematical formula 2]

[0291]

[0292] Here, x and y represent the horizontal position and the vertical position of the center of the sub-block, respectively, w represents the width of the current block, and h represents the height of the current block.

[0293] Affine modes using different numbers of control points (e.g., two and three control points) can be switched and signaled at the CU level. It should be noted that the information at the CU level indicating the number of control points in the affine mode used can be signaled at another level (e.g., sequence level, picture level, slice level, tile level, CTU level, or sub-block level).

[0294] In addition, such an affine mode in which three control points are used may include different methods for deriving motion vectors at the upper left, upper right and lower left control points. For example, the affine mode includes two modes, which are an affine inter mode (also called an affine regular inter mode) and an affine merge mode.

[0295] (MV Export > Affine Merge Mode)

[0296] Fig.26A , Fig.26B and Fig.26C is a conceptual diagram for illustrating the affine merge mode.

[0297] like Fig.26A As shown, in the affine merge mode, for example, based on a plurality of motion vectors corresponding to blocks encoded according to the affine mode among the coded blocks A (left), B (upper), C (upper right), D (lower left), and E (upper left) adjacent to the current block, the motion vector predictor at the corresponding control point of the current block is calculated. More specifically, the coded blocks A (left), B (upper), C (upper right), D (lower left), and E (upper left) are checked in the order listed, and the first valid block encoded according to the affine mode is identified. The motion vector predictor at the control point of the current block is calculated based on the plurality of motion vectors corresponding to the identified blocks.

[0298] For example, Fig.26B As shown, when block A adjacent to the left side of the current block has been encoded according to the affine mode using two control points, motion vectors v3 and v4 projected at the upper left corner position and the upper right corner position of the encoded block including block A are derived. Then, a motion vector predictor v0 at the upper left control point of the current block and a motion vector predictor v1 at the upper right control point of the current block are calculated based on the derived motion vectors v3 and v4.

[0299] For example, Fig.26C As shown, when block A adjacent to the left side of the current block has been encoded according to the affine mode using three control points, motion vectors v3, v4, and v5 projected at the upper left corner position, the upper right corner position, and the lower left corner position of the encoded block including block A are derived. Then, a motion vector predictor v0 at the upper left control point of the current block, a motion vector predictor v1 at the upper right control point of the current block, and a motion vector predictor v2 at the lower left control point of the current block are calculated based on the derived motion vectors v3, v4, and v5.

[0300] It should be noted that this method for deriving a motion vector predictor can be used in the later described Fig.29 In step Sj_1, the motion vector predictor of the corresponding control point of the current block is derived.

[0301] Fig. 27 is a flowchart illustrating an example of affine merge mode.

[0302] In the affine merge mode shown, first, the inter-frame predictor 126 derives the MV predictor of the corresponding control point of the current block (step Sk_1). Fig.25AAs shown, the control points are the upper left corner point of the current block and the upper right corner point of the current block, or as Fig.25B As shown, these are the upper left corner point of the current block, the upper right corner point of the current block, and the lower left corner point of the current block.

[0303] In other words, Fig.26A As shown, the inter-frame predictor 126 examines the encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) in the order listed and identifies the first valid block encoded according to the affine mode.

[0304] When block A is identified and block A has two control points, such as Fig.26B As shown, the inter-frame predictor 126 calculates the motion vector v0 of the upper left corner control point of the current block and the motion vector v1 of the upper right corner control point of the current block based on the motion vectors v3 and v4 of the upper left corner and the upper right corner of the encoded block including block A. For example, the inter-frame predictor 126 calculates the motion vector v0 at the upper left corner control point of the current block and the motion vector v1 at the upper right corner control point of the current block by projecting the motion vectors v3 and v4 of the upper left corner and the upper right corner of the encoded block onto the current block.

[0305] Alternatively, when block A is identified and has three control points, as Fig.26C As shown, the inter-frame predictor 126 calculates the motion vector v0 at the upper left control point of the current block, the motion vector v1 at the upper right control point of the current block, and the motion vector v2 at the lower left control point of the current block based on the motion vectors v3, v4, and v5 at the upper left, upper right, and lower left corners of the coded blocks including block A. For example, the inter-frame predictor 126 calculates the motion vector v0 at the upper left control point of the current block, the motion vector v1 at the upper right control point of the current block, and the motion vector v2 at the lower left control point of the current block by projecting the motion vectors v2, v4, and v5 of the upper left, upper right, and lower left corners of the coded blocks onto the current block.

[0306] Next, the inter-frame predictor 126 performs motion compensation on each of the multiple sub-blocks included in the current block. In other words, the inter-frame predictor 126 calculates the motion vector of each sub-block in the multiple sub-blocks as an affine MV (step Sk_2) by using (i) two motion vector predictors v0 and v1 and the above expression (1A) or (ii) three motion vector predictors v0, v1 and v2 and the above expression (1B). Then, the inter-frame predictor 126 uses these affine MVs and the encoded reference picture to perform motion compensation of the sub-blocks (step Sk_3). As a result, motion compensation of the current block is performed to generate a predicted image of the current block.

[0307] (MV Export > Affine Inter-frame Mode)

[0308] Fig.28A is a conceptual diagram for illustrating an affine inter mode using two control points.

[0309] In affine inter mode, such as Fig.28A As shown, a motion vector selected from the motion vectors of the coded blocks A, B, and C adjacent to the current block is used as the motion vector predictor v0 at the upper left corner control point of the current block. Similarly, a motion vector selected from the motion vectors of the coded blocks D and E adjacent to the current block is used as the motion vector predictor v1 at the upper right corner control point of the current block.

[0310] Fig.28B is a conceptual diagram for illustrating an affine inter mode using three control points.

[0311] In affine inter mode, such as Fig.28B As shown, a motion vector selected from the motion vectors of the coded blocks A, B, and C adjacent to the current block is used as the motion vector predictor v0 at the upper left corner control point of the current block. Similarly, a motion vector selected from the motion vectors of the coded blocks D and E adjacent to the current block is used as the motion vector predictor v1 at the upper right corner control point of the current block. Furthermore, a motion vector selected from the motion vectors of the coded blocks F and G adjacent to the current block is used as the motion vector predictor v2 at the lower left corner control point of the current block.

[0312] Fig.29 is a flowchart showing an example of affine inter mode.

[0313] In the affine inter mode shown, first, the inter predictor 126 derives the MV predictors (v0, v1) or (v0, v1, v2) of the corresponding two or three control points of the current block (step Sj_1). Fig.25A As shown, the control points are the upper left corner point of the current block and the upper right corner point of the current block, or as Fig.25B As shown, these are the upper left corner point of the current block, the upper right corner point of the current block, and the lower left corner point of the current block.

[0314] In other words, the inter-frame predictor 126 selects Fig.28A or Fig.28B The inter-frame predictor 126 derives the motion vector predictor (v0, v1) or (v0, v1, v2) of the corresponding two or three control points of the current block by using the motion vector of any block in the coding block near the corresponding control point of the current block shown in . At this time, the inter-frame predictor 126 encodes the motion vector predictor selection information for identifying the selected two motion vectors in the stream.

[0315] For example, the inter-frame predictor 126 may use cost evaluation or the like to determine a block from among encoded blocks adjacent to the current block from which to select a motion vector as a motion vector predictor at a control point, and may describe a flag in a bitstream indicating which motion vector predictor has been selected.

[0316] Next, the inter-frame predictor 126 performs motion estimation (steps Sj_3 and Sj_4) while updating the motion vector predictor selected or derived in step Sj_1 (step Sj_2). In other words, the inter-frame predictor 126 uses the above expression (1A) or expression (1B) to calculate the motion vector of each sub-block corresponding to the updated motion vector predictor as an affine MV (step Sj_3). Then, the inter-frame predictor 126 uses these affine MVs and the encoded reference picture to perform motion compensation of the sub-blocks (step Sj_4). As a result, for example, the inter-frame predictor 126 determines the motion vector predictor that generates the minimum cost at the control point in the motion estimation loop as a motion vector (step Sj_5). At this time, the inter-frame predictor 126 also encodes the difference between the determined MV and the motion vector predictor as an MV difference in the stream.

[0317] Finally, the inter-frame predictor 126 generates a predicted image of the current block by performing motion compensation of the current block using the determined MV and the encoded reference picture (step Sj_6).

[0318] (MV Export > Affine Inter-frame Mode)

[0319] When affine modes in which different numbers of control points (eg, two and three control points) are used can be switched and signaled at the CU level, the number of control points in an encoded block and the number of control points in a current block may be different from each other. Fig. 30A and Fig. 30B is a conceptual diagram for illustrating a method of deriving a motion vector predictor at a control point when the number of control points in a coding block and the number of control points in a current block are different from each other.

[0320] For example, Fig. 30A As shown, when the current block has three control points at the upper left corner, the upper right corner, and the lower left corner, and the block A adjacent to the left side of the current block has been encoded according to the affine mode in which two control points are used, motion vectors v3 and v4 projected at the upper left corner position and the upper right corner position in the encoded block including the block A are derived. Then, the motion vector predictor v0 of the upper left corner control point of the current block and the motion vector predictor v1 of the upper right corner control point of the current block are calculated based on the derived motion vectors v3 and v4. In addition, the motion vector predictor v2 of the lower left corner control point is calculated based on the derived motion vectors v0 and v1.

[0321] For example, Fig. 30B As shown, when the current block has two control points at the upper left and upper right corners, and the block A adjacent to the left side of the current block has been encoded according to the affine mode in which three control points are used, motion vectors v3, v4, and v5 projected at the upper left corner position, the upper right corner position, and the lower left corner position in the encoded block including the block A are derived. Then, a motion vector predictor v0 at the upper left control point of the current block and a motion vector predictor v1 at the upper right control point of the current block are calculated based on the derived motion vectors v3, v4, and v5.

[0322] It should be noted that this method for deriving a motion vector predictor can be used in Fig.29 In step Sj_1, the motion vector predictor of the corresponding control point of the current block is derived.

[0323] (MV Export > DMVR)

[0324] Fig.31A is a flow chart showing the relationship between merge mode and DMVR.

[0325] The inter-frame predictor 126 derives the motion vector of the current block according to the merge mode (step S1_1). Next, the inter-frame predictor 126 determines whether to perform estimation of the motion vector, i.e., motion estimation (step S1_2). Here, when it is determined not to perform motion estimation (No in step S1_2), the inter-frame predictor 126 determines the motion vector derived in step S1_1 as the final motion vector of the current block (step S1_4). In other words, in this case, the motion vector of the current block is determined according to the merge mode.

[0326] When it is determined in step S1_1 that motion estimation is performed (Yes in step S1_2), the inter-frame predictor 126 derives the final motion vector of the current block by estimating the surrounding area of ​​the reference picture specified by the motion vector derived in step S1_1 (step S1_3). In other words, in this case, the motion vector of the current block is determined according to DMVR.

[0327] Fig.31B is a conceptual diagram for illustrating an example of a DMVR process for determining a MV.

[0328] First, the best MVP that has been set as the current block (e.g., in merge mode) is determined as the MV candidate. Reference pixels are identified from a first reference picture (L0), which is a coded picture in the LO direction according to the MV candidate (LO). Similarly, reference pixels are identified from a second reference picture (L1), which is a coded picture in the L1 direction according to the MV candidate (L1). A template is generated by calculating the average value of these reference pixels.

[0329] Next, each surrounding area of ​​the MV candidates of the first reference picture (L0) and the second reference picture (L1) is estimated, and the MV that generates the minimum cost is determined as the final MV. Note that the cost value can be calculated, for example, using the difference between each pixel value in the template and the corresponding pixel value in the pixel value in the estimated area, the value of the MV candidate, etc.

[0330] It is noted that the processes, configurations, and operations described herein are generally substantially common between an encoder and a decoder to be described later.

[0331] It is not always necessary to perform exactly the same example process described here. Any process that enables deriving the final MV by estimating in the surrounding area of ​​the MV candidate can be used.

[0332] (Motion Compensation > BIO / OBMC)

[0333] Motion compensation involves a mode for generating a predicted image and correcting the predicted image. The mode is, for example, BIO and OBMC described later.

[0334] Fig.32 is a flowchart showing an example of generating a predicted image.

[0335] The inter-frame predictor 126 generates a predicted image (step Sm_1 ), and corrects the predicted image, for example, according to any of the modes described above (step Sm_2 ).

[0336] Fig.33 is a flowchart showing another example of generating a predicted image.

[0337] The inter-frame predictor 126 determines the motion vector of the current block (step Sn_1). Next, the inter-frame predictor 126 generates a predicted image (step Sn_2), and determines whether to perform a correction process (step Sn_3). Here, when it is determined to perform the correction process (Yes in step Sn_3), the inter-frame predictor 126 generates a final predicted image by correcting the predicted image (step Sn_4). When it is determined not to perform the correction process (No in step Sn_3), the inter-frame predictor 126 outputs the predicted image as the final predicted image without correcting the predicted image (step Sn_5).

[0338] In addition, motion compensation refers to a mode for correcting the brightness of a predicted image when generating a predicted image. This mode is, for example, LIC which will be described later.

[0339] Fig.34 is a flowchart showing another example of generating a predicted image.

[0340] The inter-frame predictor 126 derives a motion vector of the current block (step So_1). Next, the inter-frame predictor 126 determines whether to perform a brightness correction process (step So_2). Here, when it is determined that the brightness correction process is performed (Yes in step So_2), the inter-frame predictor 126 generates a predicted image when the brightness correction process is performed (step So_3). In other words, the predicted image is generated using LIC. When it is determined that the brightness correction process is not performed (No in step So_2), the inter-frame predictor 126 generates a predicted image by performing conventional motion compensation without performing the brightness correction process (step So_4).

[0341] (Motion Compensation > OBMC)

[0342] It should be noted that in addition to the motion information of the current block obtained from motion estimation, the motion information of the adjacent blocks can also be used to generate the inter-frame prediction signal. More specifically, the inter-frame prediction signal can be generated in units of sub-blocks in the current block by performing weighted addition on the prediction signal based on the motion information obtained from motion estimation (in the reference picture) and the prediction signal based on the motion information of the adjacent blocks (in the current picture). This inter-frame prediction (motion compensation) is also called overlapped block motion compensation (OBMC).

[0343] In the OBMC mode, information indicating a sub-block size for OBMC (e.g., referred to as an OBMC block size) may be signaled at a sequence level. In addition, information indicating whether the OBMC mode is applied (e.g., referred to as an OBMC flag) may be signaled at a CU level. It is to be noted that the signaling of such information does not necessarily need to be performed at the sequence level and the CU level, and may be performed at another level (e.g., a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0344] An example of the OBMC mode will be described in more detail. Fig.35 and Fig.36 is a flowchart and a conceptual diagram for showing an outline of a predicted image correction process performed by the OBMC process.

[0345] First, if Fig.36 As shown in , a predicted image (Pred) is obtained by conventional motion compensation using a motion vector (MV) assigned to a processing target (current) block. Fig.36 In FIG. 8 , the arrow “MV” points to the reference picture and indicates what the current block of the current picture refers to in order to obtain the predicted image.

[0346] Next, a predicted image (Pred_L) is obtained by applying a motion vector (MV_L) that has been derived for a coding block adjacent to the left side of the current block to the current block (reusing the motion vector for the current block). The motion vector (MV_L) is indicated by an arrow "MV_L", which indicates a reference picture from the current block. A first correction of the predicted image is performed by overlapping the two predicted images Pred and Pred_L. This provides the effect of blending the boundaries between adjacent blocks.

[0347] Likewise, a predicted image (Pred_U) is obtained by applying a motion vector (MV_U) that has been derived for a coding block adjacent to the current block above to the current block (reusing the motion vector for the current block). The motion vector (MV_U) is indicated by an arrow "MV_U", which indicates a reference picture from the current block. A second correction of the predicted image is performed by overlapping the predicted image Pred_U with a predicted image (e.g., Pred and Pred_L) on which the first correction has been performed. This provides an effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is a predicted image in which the boundaries between adjacent blocks have been blended (smoothed), and is therefore the final predicted image of the current block.

[0348] Although the above example is a two-path correction method using left and upper neighboring blocks, it is noted that the correction method may be a three-path or more correction method also using right and / or lower neighboring blocks.

[0349] Note that the area in which such overlapping is performed may be only a portion of the area near the block boundary, rather than the pixel area of ​​the entire block.

[0350] Note that the prediction image correction process according to OBMC for obtaining one prediction image Pred from one reference picture by overlapping additional prediction images Pred_L and Pred_U has been described above. However, when the prediction image is corrected based on a plurality of reference images, a similar process may be applied to each of the plurality of reference pictures. In this case, after obtaining a corrected prediction image from the corresponding reference picture by performing OBMC image correction based on the plurality of reference pictures, the obtained corrected prediction images are further overlapped to obtain a final prediction image.

[0351] It is to be noted that, in OBMC, the unit of the current block may be the unit of the prediction block, or the unit of a sub-block obtained by further dividing the prediction block.

[0352] One example of a method for determining whether to apply the OBMC process is a method for using obmc_flag, which is a signal indicating whether the OBMC process is applied. As a specific example, the encoder determines whether the current block belongs to an area with complex motion. When the block belongs to an area with complex motion, the encoder sets the value of obmc_flag to "1", and when encoding, the OBMC process is applied; and when the block does not belong to an area with complex motion, the value of obmc_flag is set to "0", and the block is encoded without applying the OBMC process. The decoder switches between application and non-application of the OBMC process by decoding the obmc_flag written in the stream (e.g., a compressed sequence) and decoding the block by switching between application and non-application of the OBMC process according to the flag value.

[0353] In the above example, the inter-frame predictor 126 generates one rectangular prediction image for the rectangular current block. However, the inter-frame predictor 126 may generate a plurality of prediction images, each of which has a shape different from the rectangular shape of the rectangular current block, and may combine the plurality of prediction images to generate a final rectangular prediction image. The shape different from the rectangle may be, for example, a triangle.

[0354] Fig.37 is a conceptual diagram for illustrating the generation of two triangular prediction images.

[0355] The inter-frame predictor 126 generates a triangular predicted image by performing motion compensation on the first partition having a triangular shape in the current block using the first MV of the first partition to generate a triangular predicted image. Similarly, the inter-frame predictor 126 generates a triangular predicted image by performing motion compensation on the second partition having a triangular shape in the current block using the second MV of the second partition to generate a triangular predicted image. Then, the inter-frame predictor 126 generates a predicted image having the same rectangular shape as that of the current block by combining these predicted images.

[0356] It should be noted that although Fig.37 In the example shown, the first partition and the second partition are triangular, but the first partition and the second partition may be trapezoidal or other shapes different from each other. Fig.37 In the example shown, the current block includes two partitions, and the current block may include three or more partitions.

[0357] In addition, the first partition and the second partition may overlap each other. In other words, the first partition and the second partition may include the same pixel area. In this case, the predicted image in the first partition and the predicted image in the second partition may be used to generate the predicted image of the current block.

[0358] In addition, although an example in which a predicted image is generated for each of two partitions using inter prediction, a predicted image may be generated for at least one partition using intra prediction.

[0359] (Motion Compensation > BIO)

[0360] Next, a method for deriving motion vectors is described. First, a mode for deriving motion vectors based on a model assuming uniform linear motion will be described. This mode is also called bidirectional optical flow (BIO) mode.

[0361] Fig.38 This is a conceptual diagram for illustrating a model assuming uniform linear motion. Fig.38 , (vx, vy) indicates a velocity vector, and τ0 and τ1 indicate a temporal distance between a current picture (current picture) and two reference pictures (Ref0, Ref1). (MVx0, MVy0) indicates a motion vector corresponding to the reference picture Ref0, and (MVx1, MVy1) indicates a motion vector corresponding to the reference picture Ref1.

[0362] Here, assuming that the velocity vector (v x ,v y ) shows uniform linear motion, (MVx0, MVy0) and (MVx1, MVy1) are respectively expressed as (v x τ0,v y τ0) and (-v x τ1,-v y τ1), and the following optical flow equation (2) can be used.

[0363] [Mathematical formula 3]

[0364]

[0365] Here, I(k) represents the motion compensated luminance value of reference picture k (k=0, 1). The optical flow equation shows that the sum of the following is equal to zero: (i) the temporal derivative of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image. The motion vector of each block obtained from, for example, the merge list can be corrected on a pixel basis based on a combination of the optical flow equation and Hermitian interpolation.

[0366] It is to be noted that a method other than deriving a motion vector based on a model assuming uniform linear motion may be used to derive a motion vector on the decoder side. For example, a motion vector may be derived in units of sub-blocks based on motion vectors of neighboring blocks.

[0367] (Motion Compensation > LIC)

[0368] Next, an example of a mode in which a predicted image (prediction) is generated by using a local illumination compensation (LIC) process will be described.

[0369] Fig.39 is a conceptual diagram for illustrating one example of a predicted image generation method using a brightness correction process performed by the LIC process.

[0370] First, MV is derived from the encoded reference picture, and the reference image corresponding to the current block is obtained.

[0371] Next, information indicating how the brightness value between the reference picture and the current picture changes is extracted for the current block. This extraction is performed based on the brightness pixel values ​​for the encoded left adjacent reference area (surrounding reference area) and the encoded upper adjacent reference area (surrounding reference area) and the brightness pixel values ​​at the corresponding positions in the reference picture specified by the derived MV. The brightness correction parameter is calculated by using the information indicating how the brightness value changes.

[0372] A prediction image for the current block is generated by performing a brightness correction process in which a brightness correction parameter is applied to a reference image in a reference picture specified by the MV.

[0373] It should be noted that Fig.39 The shape of the surrounding reference area shown in FIG. 1 is just one example; the surrounding reference area may have a different shape.

[0374] In addition, although the process of generating a predicted image from a single reference picture has been described here, the case where a predicted image is generated based on a plurality of reference pictures can be described in the same manner. The predicted image can be generated after performing a brightness correction process on a reference image obtained from a reference picture in the same manner as described above.

[0375] One example of a method for determining whether to apply the LIC process is a method for using lic_flag, which is a signal indicating whether the LIC process is applied. As a specific example, the encoder determines whether the current block belongs to an area with brightness changes. When the block belongs to an area with brightness changes, the encoder sets the value of lic_flag to "1" and applies the LIC process when encoding, and when the block does not belong to an area with brightness changes, the value of lic_flag is set to "0" and the current block is encoded without applying the LIC process. The decoder can decode the lic_flag written in the stream and decode the current block by switching between application and non-application of the LIC process according to the flag value.

[0376] An example of another method of determining whether to apply the LIC process is a determination method based on whether the LIC process is applied to the surrounding blocks. In a specific example, when the merge mode is used on the current block, it is determined whether the LIC process is applied in the encoding of the surrounding coding blocks selected when deriving the MV in the merge mode process. According to the result, encoding is performed by switching between application and non-application of the LIC process. Note that, also in this example, the same process is applied in the process on the decoder side.

[0377] The following describes in detail the reference Fig.39 An embodiment of a described brightness correction (LIC) process is shown.

[0378] First, the inter predictor 126 derives a motion vector for obtaining a reference image corresponding to a current block to be encoded from a reference picture as an encoded picture.

[0379] Next, the inter-frame predictor 126 uses the luminance pixel value of the encoded surrounding reference area adjacent to the left or above the current block and the luminance value of the corresponding position in the reference picture specified by the motion vector, extracts information indicating how the luminance value of the reference picture is changed to the luminance value of the current picture, and calculates the luminance correction parameter. For example, it is assumed that the luminance pixel value of a given pixel in the surrounding reference area in the current picture is p0, and it is assumed that the luminance pixel value of the pixel corresponding to the given pixel in the surrounding reference area in the reference picture is p1. The inter-frame predictor 126 calculates coefficients A and B for optimizing A×p1+B=p0 as luminance correction parameters for multiple pixels in the surrounding reference area.

[0380] Next, the inter-frame predictor 126 performs a brightness correction process using the brightness correction parameters of the reference image in the reference picture specified by the motion vector to generate a predicted image of the current block. For example, assume that the brightness pixel value in the reference image is p2, and the brightness pixel value of the predicted image after brightness correction is p3. After performing the brightness correction process by calculating A×p2+B=p3 for each pixel in the reference image, the inter-frame predictor 126 generates a predicted image.

[0381] It should be noted that Fig.39 The shape of the surrounding reference area shown in FIG. 1 is an example; a different shape other than the shape of the surrounding reference area may be used. Fig.39 A portion of the surrounding reference area shown in . For example, an area having a determined number of pixels extracted from each of the upper adjacent pixels and the left adjacent pixels can be used as the surrounding reference area. The determined number of pixels can be predetermined.

[0382] In addition, the surrounding reference area is not limited to the area adjacent to the current block, and may be an area not adjacent to the current block. Fig.39 In the example shown, the surrounding reference area in the reference picture is an area specified by a motion vector in the current picture from the surrounding reference area in the current picture. However, an area specified by another motion vector is also possible. For example, the other motion vector may be a motion vector in the surrounding reference area in the current picture.

[0383] Although operations performed by encoder 100 have been described herein, it is noted that decoder 200 generally performs similar operations.

[0384] It should be noted that the LIC process can be applied not only to luminance but also to chrominance. At this time, correction parameters can be derived individually for each of Y, Cb and Cr, or common correction parameters can be used for any one of Y, Cb and Cr.

[0385] In addition, the LIC process may be applied in units of subblocks. For example, correction parameters may be derived using surrounding reference areas in the current subblock and surrounding reference areas in the reference subblock in the reference picture specified by the MV of the current subblock.

[0386] (Predictive Controller)

[0387] The inter predictor 128 selects one of the intra prediction signal (the signal output from the intra predictor 124) and the inter prediction signal (the signal output from the inter predictor 126), and outputs the selected signal to the subtractor 104 and the adder 116 as a prediction signal.

[0388] like Figure 1 As shown, in various encoder examples, the prediction controller 128 can output prediction parameters that are input to the entropy encoder 110. The entropy encoder 110 can generate a coded bitstream (or sequence) based on the prediction parameters input from the prediction controller and the quantization coefficients input from the quantizer 108. The prediction parameters can be used in the decoder. The decoder can receive and decode the coded bitstream and perform the same process as the prediction process performed by the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128. The prediction parameters may include (i) a selection prediction signal (e.g., a motion vector, a prediction type, or a prediction mode used by the intra-frame predictor 124 or the inter-frame predictor 126), or (ii) an optional index, flag, or value based on the prediction process performed in each of the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128 or indicating the prediction process.

[0389] (Encoder installation example)

[0390] Fig.401 is a block diagram showing an example of an installation of the encoder 100. The encoder 100 includes a processor a1 and a memory a2. For example, Figure 1 The various components of the encoder 100 shown may be mounted on Fig.40 On the processor a1 and memory a2 shown in .

[0391] The processor a1 is a circuit that performs information processing and can be accessed by the memory a2. For example, the processor a1 is a dedicated or general-purpose electronic circuit that encodes a video. The processor a1 may be a processor such as a CPU. In addition, the processor a1 may be an aggregation of multiple electronic circuits. In addition, for example, the processor a1 may serve as Figure 1 The roles of two or more constituent elements among the multiple constituent elements of the encoder 100 etc. shown.

[0392] The memory a2 is a dedicated or general-purpose memory for storing information used by the processor a1 to encode the video. The memory a2 may be an electronic circuit and may be connected to the processor a1. In addition, the memory a2 may be included in the processor a1. In addition, the memory a2 may be an aggregation of multiple electronic circuits. In addition, the memory a2 may be a magnetic disk, an optical disk, etc., or may be represented as a storage device, a recording medium, etc. In addition, the memory a2 may be a non-volatile memory or a volatile memory.

[0393] For example, the memory a2 may store a video to be encoded or a bit stream corresponding to the encoded video. In addition, the memory a2 may store a program for causing the processor a1 to encode the video.

[0394] In addition, for example, memory a2 can serve as Figure 1 The memory a2 may serve as two or more components for storing information among the multiple components of the encoder 100, etc. Figure 1 The roles of the block memory 118 and the frame memory 122 are shown. More specifically, the memory a2 can store reconstructed blocks, reconstructed pictures, etc.

[0395] It should be noted that in the encoder 100, it is not necessary to implement Figure 1 All multiple constituent elements indicated in, etc., and not all of the above-mentioned processes may be performed. Figure 1 A portion of the constituent elements indicated in the diagram may be included in another device, or a portion of the above-described process may be performed by another device.

[0396] (Decoder)

[0397] Next, a decoder capable of decoding, for example, an encoded signal (encoded bit stream) output from the above-described encoder 100 will be described. Fig.412 is a block diagram showing a functional configuration of a decoder 200 according to an embodiment. The decoder 200 is a video decoder that decodes a video in units of blocks.

[0398] like Fig.41 As shown, the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an intra-frame predictor 216, an inter-frame predictor 218 and a prediction controller 220.

[0399] The decoder 200 is implemented as, for example, a general purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor functions as an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a loop filter 212, an intra-frame predictor 216, an inter-frame predictor 218, and a prediction controller 220. Alternatively, the decoder 200 may be implemented as one or more dedicated electronic circuits corresponding to the entropy decoder 202, the inverse quantizer 204, the inverse transformer 206, the adder 208, the loop filter 212, the intra-frame predictor 216, the inter-frame predictor 218, and the prediction controller 220.

[0400] Hereinafter, an overall flow of a process performed by the decoder 200 is described, and then each constituent element included in the decoder 200 will be described.

[0401] (Overall flow of the decoding process)

[0402] Fig.42 is a flowchart showing one example of the entire decoding process performed by the decoder 200.

[0403] First, the entropy decoder 202 of the decoder 200 identifies a partition pattern of a block having a fixed size (e.g., 128×128 pixels) (step Sp_1). The partition pattern is a partition pattern selected by the encoder 100. Then, the decoder 200 performs the process of steps Sp_2 to Sp_6 for each of a plurality of blocks of the partition pattern.

[0404] In other words, the entropy decoder 202 decodes (specifically, entropy decodes) the prediction parameters and encoded quantization coefficients of the current block to be decoded (also referred to as the current block) (step Sp_2).

[0405] Next, the inverse quantizer 204 performs inverse quantization on the plurality of quantized coefficients, and the inverse transformer 206 performs inverse transformation on the result to restore a plurality of prediction residuals (ie, difference blocks) (step Sp_3).

[0406] Next, the prediction processor including all or part of the intra predictor 216, the inter predictor 218, and the prediction controller 220 generates a prediction signal (also referred to as a prediction block) for the current block (step Sp_4).

[0407] Next, the adder 208 adds the prediction block to the difference block to generate a reconstructed image of the current block (also referred to as a decoded image block) (step Sp_5).

[0408] When the reconstructed image is generated, the loop filter 212 performs filtering on the reconstructed image (step Sp_6).

[0409] The decoder 200 then determines whether decoding of the entire picture has been completed (step Sp_7). When it is determined that decoding has not been completed (No in step Sp_7), the decoder 200 repeatedly performs the process starting from step Sp_1.

[0410] As shown in the figure, the processes of steps Sp_1 to Sp_7 are sequentially performed by the decoder 200. Alternatively, two or more processes may be performed in parallel, the processing order of two or more processes may be modified, and so on.

[0411] (Entropy Decoder)

[0412] The entropy decoder 202 performs entropy decoding on the decoded bit stream. More specifically, for example, the entropy decoder 202 arithmetically decodes the encoded bit stream into a binary signal. Then, the entropy decoder 202 de-binarizes the binary signal. In this way, the entropy decoder 202 outputs the quantization coefficients of each block to the inverse quantizer 204. The entropy decoder 202 can be included in the encoded bit stream (see Figure 1 ) is output to the intra predictor 216, the inter predictor 218 and the prediction controller 220. The intra predictor 216, the inter predictor 218 and the prediction controller 220 in the embodiment are capable of performing the same prediction process as the prediction process performed by the intra predictor 124, the inter predictor 126 and the prediction controller 128 on the encoder side.

[0413] (Inverse Quantizer)

[0414] The inverse quantizer 204 inversely quantizes the quantized coefficients of the block to be decoded (hereinafter referred to as the current block) input from the entropy decoder 202. More specifically, the inverse quantizer 204 inversely quantizes the quantized coefficients of the current block based on the quantization parameters corresponding to the quantized coefficients. The inverse quantizer 204 then outputs the inversely quantized transform coefficients of the current block to the inverse transformer 206.

[0415] (Inverter)

[0416] The inverse transformer 206 recovers the prediction error by inversely transforming the transform coefficients (which are input from the inverse quantizer 204).

[0417] For example, when information parsed from the encoded bitstream indicates that EMT or AMT is to be applied (eg, when the AMT flag is true), the inverse transformer 206 inversely transforms transform coefficients of the current block based on information indicating the parsed transform type.

[0418] Furthermore, for example, when the information parsed from the encoded bit stream indicates that NSST is to be applied, the inverse transformer 206 applies a secondary inverse transform to the transform coefficients.

[0419] (Adder)

[0420] The adder 208 reconstructs the current block by adding the prediction error (which is the input from the inverse transformer 206) and the prediction sample (which is the input from the prediction controller 220). The adder 208 then outputs the reconstructed block to the block memory 210 and the loop filter 212.

[0421] (Block Storage)

[0422] The block memory 210 is a storage device for storing a block in a picture to be decoded (hereinafter referred to as a current picture) and to be referenced in intra prediction. More specifically, the block memory 210 stores the reconstructed block output from the adder 208.

[0423] (Loop Filter)

[0424] The loop filter 212 applies a loop filter to the block reconstructed by the adder 208 and outputs the filtered reconstructed block to the frame memory 214, a display device, or the like.

[0425] When the information indicating ON or OFF of ALF parsed from the encoded bitstream indicates that ALF is ON, one filter is selected from among a plurality of filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed block.

[0426] (Frame Memory)

[0427] The frame memory 214 is, for example, a storage device for storing a reference picture used in inter-frame prediction, and is also referred to as a frame buffer. More specifically, the frame memory 214 stores the reconstructed block filtered by the loop filter 212.

[0428] (Prediction processor (intra-frame predictor, inter-frame predictor, prediction controller))

[0429] Fig.43is a flowchart showing one example of a process performed by the prediction processor of the decoder 200. Note that the prediction processor includes all or part of the following constituent elements: an intra predictor 216; an inter predictor 218; and a prediction controller 220.

[0430] The prediction processor generates a prediction image of the current block (step Sq_1). The prediction image is also called a prediction signal or a prediction block. Note that the prediction signal is, for example, an intra-frame prediction signal or an inter-frame prediction signal. Specifically, the prediction processor uses a reconstructed image obtained by generating a prediction block, generating a difference block, generating a coefficient block, restoring a difference block, and generating a decoded image block to generate a prediction image of the current block.

[0431] The reconstructed image may be, for example, an image in a reference picture, or an image of a decoded block in a current picture (which is a picture including the current block). The decoded block in the current picture may be, for example, a neighboring block of the current block.

[0432] Fig.44 is a flow chart illustrating another example of a process performed by the prediction processor of the decoder 200 .

[0433] The prediction processor determines a method or mode for generating a prediction image (step Sr_1). For example, the method or mode may be determined based on prediction parameters or the like.

[0434] When the first method is determined as the mode for generating a predicted image, the prediction processor generates a predicted image according to the first method (step Sr_2a). When the second method is determined as the mode for generating a predicted image, the prediction processor generates a predicted image according to the second method (step Sr_2b). When the third method is determined as the mode for generating a predicted image, the prediction processor generates a predicted image according to the third method (step Sr_2c).

[0435] The first method, the second method, and the third method may be different methods for generating a predicted image. Each of the first to third methods may be an inter-frame prediction method, an intra-frame prediction method, or another prediction method. The above-mentioned reconstructed image may be used in these prediction methods.

[0436] (Intra Predictor)

[0437] The intra predictor 216 generates a prediction signal (intra prediction signal) by performing intra prediction based on the intra prediction mode parsed from the encoded bit stream by referring to one or more blocks in the current picture stored in the block memory 210. More specifically, the intra predictor 216 generates the intra prediction signal by performing intra prediction with reference to samples (e.g., luminance and / or chrominance values) of one or more blocks adjacent to the current block, and then outputs the intra prediction signal to the prediction controller 220.

[0438] It should be noted that when an intra prediction mode in which a luma block is referenced in intra prediction of a chroma block is selected, the intra predictor 216 may predict a chroma component of the current block based on a luma component of the current block.

[0439] Furthermore, when information parsed from the encoded bitstream indicates that PDPC is to be applied, the intra predictor 216 corrects the intra-predicted pixel value based on the horizontal / vertical reference pixel gradients.

[0440] (Inter-frame predictor)

[0441] The inter-frame predictor 218 predicts the current block by referring to the reference picture stored in the frame memory 214. Inter-frame prediction is performed in units of the current block or a sub-block (e.g., a 4×4 block) in the current block. For example, the inter-frame predictor 218 generates an inter-frame prediction signal of the current block or sub-block by performing motion compensation using motion information (e.g., a motion vector) parsed from the encoded bit stream (e.g., a prediction parameter output from the entropy decoder 202), and outputs the inter-frame prediction signal to the prediction controller 220.

[0442] Note that when information parsed from the encoded bitstream indicates that the OBMC mode is to be applied, the inter predictor 218 generates an inter prediction signal using motion information of neighboring blocks (in addition to the motion information of the current block obtained from motion estimation).

[0443] In addition, when the information parsed from the coded bit stream indicates that the FRUC mode is to be applied, the inter-frame predictor 218 derives motion information by performing motion estimation according to the pattern matching method (bilateral matching or template matching) parsed from the coded bit stream. Then, the inter-frame predictor 218 performs motion compensation (prediction) using the derived motion information.

[0444] In addition, when the BIO mode is to be applied, the inter-frame predictor 218 derives a motion vector based on a model assuming uniform linear motion. In addition, when the information parsed from the encoded bitstream indicates that the affine motion compensation prediction mode is to be applied, the inter-frame predictor 218 derives a motion vector for each sub-block based on the motion vectors of the neighboring blocks.

[0445] (MV Export > Regular Interframe Mode)

[0446] When the information parsed from the encoded bitstream indicates that the normal inter mode is to be applied, the inter predictor 218 derives an MV based on the information parsed from the encoded bitstream and performs motion compensation (prediction) using the MV.

[0447] Fig.45 is a flowchart illustrating an example of inter prediction in a normal inter mode in the decoder 200 .

[0448] The inter-frame predictor 218 of the decoder 200 performs motion compensation for each block. The inter-frame predictor 218 obtains multiple MV candidates for the current block based on information such as MVs of multiple decoded blocks temporally or spatially surrounding the current block (step Ss_1). In other words, the inter-frame predictor 218 generates a MV candidate list.

[0449] Next, the inter-frame predictor 218 extracts N (2 or a greater integer) MV candidates as motion vector predictor candidates (also referred to as MV predictor candidates) from the plurality of MV candidates obtained in step Ss_1 according to the determined priority order (step Ss_2). It should be noted that the priority order may be predetermined for each of the N MV predictor candidates.

[0450] Next, the inter-frame predictor 218 decodes the motion vector predictor selection information from the input stream (i.e., the encoded bit stream), and uses the decoded motion vector predictor selection information to select an MV predictor candidate from N MV predictor candidates as the motion vector (also called MV predictor) of the current block (step Ss_3).

[0451] Next, the inter predictor 218 decodes the MV difference from the input stream, and derives the MV of the current block by adding the difference value which is the decoded MV difference and the selected motion vector predictor (step Ss_4).

[0452] Finally, the inter predictor 218 generates a predicted image of the current block by performing motion compensation of the current block using the derived MV and the decoded reference picture (step Ss_5).

[0453] (Predictive Controller)

[0454] The prediction controller 220 selects an intra prediction signal or an inter prediction signal and outputs the selected prediction signal to the adder 208. In general, the configuration, function, and process of the prediction controller 220, the intra predictor 216, and the inter predictor 218 on the decoder side may correspond to the configuration, function, and process of the prediction controller 128, the intra predictor 124, and the inter predictor 126 on the encoder side.

[0455] (Decoder installation example)

[0456] Fig.46 2 is a block diagram showing an example of an installation of the decoder 200. The decoder 200 includes a processor b1 and a memory b2. For example, Fig.41 The multiple components of the decoder 200 are installed in Fig.46 As shown, processor b1 and memory b2.

[0457] The processor b1 is a circuit that performs information processing and can be accessed by the memory b2. For example, the processor b1 is a dedicated or general-purpose electronic circuit that decodes the video (i.e., the encoded bit stream). The processor b1 can be a processor such as a CPU. In addition, the processor b1 can be an aggregation of multiple electronic circuits. In addition, for example, the processor b1 can serve as Fig.41 The roles of two or more constituent elements among the multiple constituent elements of the decoder 200 and the like shown.

[0458] The memory b2 is a dedicated or general memory for storing information used by the processor b1 to decode the encoded bit stream. The memory b2 may be an electronic circuit and may be connected to the processor b1. In addition, the memory b2 may be included in the processor b1. In addition, the memory b2 may be an aggregation of multiple electronic circuits. In addition, the memory b2 may be a magnetic disk, an optical disk, etc., or may be represented as a memory, a recording medium, etc. In addition, the memory b2 may be a non-volatile memory or a volatile memory.

[0459] For example, the memory b2 may store a video or a bit stream. In addition, the memory b2 may store a program for causing the processor b1 to decode the encoded bit stream.

[0460] In addition, for example, memory b2 can serve as Fig.41 The memory b2 may serve as two or more components for storing information among the multiple components of the decoder 200, etc. Fig.41 The roles of the block memory 210 and the frame memory 214 are shown. More specifically, the memory b2 can store reconstructed blocks, reconstructed pictures, etc.

[0461] It should be noted that in the decoder 200, it is not necessary to implement Fig.41 All the multiple constituent elements shown in the figure may not be included, and all the processes described above may not be performed. Fig.41 A portion of the constituent elements indicated in the diagram may be included in another device, or a portion of the above-described process may be performed by another device.

[0462] (Definition of terms)

[0463] The corresponding terms can be defined as shown in the following examples.

[0464] A picture is either an array of luma samples in monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color formats. A picture can be either a frame or a field.

[0465] A frame is composed of a top field and a bottom field, where sample lines 0, 2, 4, ... originate from the top field and sample lines 1, 3, 5, ... originate from the bottom field.

[0466] A slice is an integer number of coding tree units contained in an independent slice segment and all subsequent dependent slice segments (if any) before the next independent slice segment (if any) within the same access unit.

[0467] A tile is a rectangular area of ​​a coding treeblock within a specific tile column and a specific tile row in a picture. A tile can be a rectangular area of ​​a frame that is intended to be independently decodable and coded, but loop filtering across tile edges can still be applied.

[0468] A block is an MxN (M columns by N rows) array of samples or an MxN array of transform coefficients. A block can be a square or rectangular area of ​​pixels that includes one luma matrix and two chroma matrices.

[0469] A Coding Tree Unit (CTU) can be a coding tree block of luma samples for a picture with three sample arrays, or two corresponding coding tree blocks of chroma samples. Alternatively, a CTU can be a coding tree block of samples for a monochrome picture and one of the pictures encoded using three separate color planes and syntax structures for encoding the samples. A superblock can be a square block of 64×64 pixels consisting of 1 or 2 mode information blocks, or can be recursively divided into four 32×32 blocks, which themselves can be further divided.

[0470] Fig.47A 1 is a flowchart illustrating an example of a process flow 1000 according to one embodiment: segmenting an image block into a plurality of partitions including at least a first partition and a second partition, predicting a motion vector from a set of motion vector candidates of at least the first partition, and performing further processing. The process flow 1000 may be, for example, Figure 1 Encoder 100, Fig.41 The decoder 200 etc. executes.

[0471] In step S1001 , an image block is divided into a plurality of partitions including at least a first partition, and the first partition may or may not have a non-rectangular shape. Fig.48 is a conceptual diagram for illustrating an exemplary method of dividing an image block into a first partition and a second partition. Fig.48 As shown, the image block can be divided into two or more partitions having various shapes. Fig.48The example illustrations include: an image block that is segmented from the upper left corner of the image block to the lower right corner of the image block to create a first partition and a second partition that both have a non-rectangular shape (e.g., a triangular shape); an image block that is segmented into an L-shaped partition and a rectangular partition; an image block that is segmented into a pentagonal partition and a triangular partition; an image block that is segmented into a hexagonal partition and a pentagonal partition; and an image block that is segmented into two polygonal partitions. The various partition shapes shown can be formed by segmenting the image block in other ways. For example, two triangular partitions can be formed by segmenting the image block from the upper right corner of the image block to the lower left corner of the image block to create a first partition and a second partition that both have a triangular shape. In some embodiments, two or more partitions of the image block may have overlapping portions.

[0472] In step S1002, the process predicts a first motion vector from a motion vector candidate set of at least a first partition. The motion vector candidates of the motion vector candidate list may include motion vector candidates derived from spatial or temporal neighboring partitions of at least the first partition. In step S1003, at least the first partition is encoded or decoded using the first motion vector.

[0473] Fig.49 is a conceptual diagram for illustrating adjacent and non-adjacent spatially adjacent partitions of a first partition of a current picture. An adjacent spatially adjacent partition is a partition adjacent to the first partition in the current picture. A non-adjacent spatially adjacent partition is a partition spaced apart from the first partition in the current picture. In some embodiments, Fig.47A The set of candidate motion vectors in S1002 may be derived from spatially adjacent partitions of at least the first partition in the current picture.

[0474] In some embodiments, the motion vector candidate set can be derived from a motion vector candidate list, such as a motion vector candidate list used in an inter-prediction mode (e.g., merge mode, skip mode, or inter-mode). Such a list may include single-prediction motion vector candidates and dual-prediction motion vector candidates.

[0475] Fig.50 is a conceptual diagram for illustrating uni-predictive and bi-predictive motion vector candidates for image blocks of a current picture. A uni-predictive motion vector candidate is a single motion vector for a current block in a current picture relative to a single reference picture. Fig.50 As shown at the top of , a single prediction motion vector candidate is a motion vector from a block of a current picture to a block of a reference picture, where the reference picture appears before the current picture in display order. In some embodiments, the reference picture may appear after the current picture in display order.

[0476] The bi-prediction motion vector candidate includes two motion vectors: a first motion vector of the current block relative to a first reference picture, and a second motion vector of the current block relative to a second reference picture. Fig.50 As shown, the lower left dual prediction motion vector candidate has a first motion vector from a block of the current picture to a block of the first reference picture, and a second motion vector from a block of the current picture to a block of the second reference picture. As shown, the first reference picture and the second reference picture appear before the current picture in display order. Fig.50 The bi-predictive motion vector candidate in the lower right corner of has a first motion vector from a block of the current picture to a block of a first reference picture, and a second motion vector from a block of the current picture to a block of a second reference picture. The first reference picture appears before the current picture in display order, and the second reference picture appears after the current picture in display order.

[0477] In some embodiments, the motion vector candidate set from which the motion vector of at least one partition is predicted may be a single-prediction motion vector candidate set. Fig.47B is a flowchart showing an example of a process flow 1000' according to one embodiment: partitioning an image block into a plurality of partitions including at least a first partition and a second partition, predicting a motion vector from a single prediction motion vector candidate set of at least the first partition, and performing further processing. The process flow 1000' may be, for example, Figure 1 Encoder 100, Fig.41 The decoder 200 etc. executes. Fig.47B The process flow 1000' and Fig.47A The process flow 1000 differs in that, in Fig.47B In step 1002', the motion vector is predicted from a single set of predictive motion vector candidates. Using a single set of predictive motion vector candidates can help reduce memory bandwidth requirements and the number of operations required to encode the partition.

[0478] The set of single-predicted motion vector candidates may be derived from the list of motion vector candidates, for example, by including only the single-predicted motion vectors of the list in the set of motion vector candidates. However, a single-predicted motion vector may also be derived from a dual-predicted motion vector in various ways, and a single-predicted motion vector may be derived from a single-predicted motion vector candidate in various ways.

[0479] For example, an index may be used to derive a uni-predicted motion vector candidate from a bi-predicted motion vector candidate in a motion vector candidate list for inclusion in a uni-predicted motion vector candidate set. For example, a bi-directional motion vector candidate may have an associated index that identifies a first reference picture in a first list (e.g., reference picture list L0) and identifies a second reference picture in a second list (e.g., reference picture list L1). Table 1 below shows an example mapping of an index of a bi-predicted motion vector candidate to a first reference picture in a first reference picture list and a second reference picture in a second reference picture list.

[0480] [Table 1]

[0481]

[0482] Table 1: Example mapping of bi-predictive motion vector candidate indices to reference pictures in a reference picture list

[0483] Referring to Table 1, the bi-predictive motion vector candidate with index 1 points to reference picture 0 in reference picture list L0, and points to reference picture 8 in reference picture list L1. Two single-predictive motion vector candidates may be derived from: the bi-predictive motion vector candidate with index 1, a single-predictive motion vector candidate for a block relative to the current block and reference picture 0 based on reference picture list L0, and a single-predictive motion vector for a block relative to the current block and reference picture 8 based on reference picture list L1. One or both of the single-predictive motion vector candidates may be included in a single-predictive motion vector candidate set, from which a current partition-predicted motion vector for a block of a current image is predicted (e.g., a single-predictive motion vector based on reference picture list L0, a single-predictive motion vector based on reference picture list L1, or both). Note that, for the current picture, reference picture 8 in list L0 is the same as reference picture 8 in list L1.

[0484] Similarly, the bi-predictive motion vector candidate with index 2 points to reference picture 8 in reference picture list L0 and points to reference picture 16 in reference picture list L1; two uni-predictive motion vector candidates can be derived from the following: the bi-predictive motion vector candidate with index 2, the bi-predictive motion vector candidate for the block relative to the current block and reference picture 8, and the bi-predictive motion vector for the block relative to the current block and reference picture 16. One or both of the uni-predictive motion vector candidates can be included in a uni-predictive motion vector candidate set from which the motion vector of the current partition of the block of the current image can be predicted (e.g., a uni-predictive motion vector candidate based on reference picture list L0, a uni-predictive motion vector candidate based on reference picture list L1, or both).

[0485] In some embodiments, only uni-predictive motion vector candidates based on reference picture list L0 are included in a uni-predictive motion vector candidate set derived from bi-predictive motion vector candidates in the candidate list. In some embodiments, only uni-predictive motion vector candidates based on reference picture list L1 are included in the uni-predictive motion vector candidate set. In some embodiments, the uni-predictive motion vector candidate set is an ordered set, for example, where uni-predictive motion vector candidates based on reference picture list L0 are followed in the uni-predictive motion vector candidate set by uni-predictive motion vector candidates based on reference picture list L1.

[0486] In some embodiments, the motion vectors of the dual-predictive motion vector candidates to be included in the single-predictive motion vector candidate set can be determined based on the display order or coding order of the pictures. A general rule can be to include the motion vectors of the dual-predictive motion vector candidates, which point to the closest reference picture in time or the earliest reference picture. Figure 51 to Figure 53 is a conceptual diagram for illustrating determination of a single-prediction motion vector from dual-prediction motion vector candidates based on a display or encoding order of pictures to be included in a single-prediction motion vector candidate set.

[0487] In some embodiments, the motion vector of the reference picture that is closest in display order to the bi-predictive motion vector of the current picture is included in the single predictive motion vector candidate set. Fig.51 As shown, the bi-predictive motion vector candidates include Mv0 pointing to reference picture 0 and Mv1 pointing to reference picture 1. Mv1 is selected as a uni-predictive motion vector to be included in the uni-predictive motion vector set because reference picture 1 is closer to the current picture than reference picture 0 in display order.

[0488] In some embodiments, the motion vector of the reference picture that is closest in coding order to the bi-predictive motion vector of the current picture is included in the uni-predictive motion vector candidate set. Fig.52 As shown, the bi-predictive motion vector candidates include Mv0 pointing to reference picture 1 and Mv1 pointing to reference picture 2. Mv0 is selected as a uni-predictive motion vector to be included in the uni-predictive motion vector set because reference picture 1 is closer to the current picture than reference picture 2 in coding order.

[0489] In some embodiments, the motion vector of the reference picture of the bi-predictive motion vector (which is the reference picture that precedes the current picture in display order) is included in the uni-predictive motion vector candidate set. Fig.52 As shown, the bi-predictive motion vector candidates include Mv0 pointing to reference picture 1 and Mv1 pointing to reference picture 2. Mv0 is selected as a uni-predictive motion vector to be included in the uni-predictive motion vector set because reference picture 1 is a reference picture that appears before the current picture in display order.

[0490] In some embodiments, the motion vector of the reference picture of the bi-predictive motion vector (which is the reference picture after the current picture in display order) is included in the uni-predictive motion vector candidate set. Fig.52 As shown, the bi-predictive motion vector candidates include Mv0 pointing to reference picture 1 and Mv1 pointing to reference picture 2. Mv1 is selected as a uni-predictive motion vector to be included in the uni-predictive motion vector set because reference picture 2 is a reference picture after the current picture in display order.

[0491] In some embodiments, the above-described example methods of deriving a single-predictive motion vector from a bi-predictive motion vector may be combined. For example, in some embodiments, the motion vector of the reference picture of the bi-predictive motion vector closest to the current picture and preceding the current picture in display order may be included in the single-predictive motion vector candidate set. Fig.53 As shown, the dual-prediction motion vector candidates include Mv0 pointing to reference picture 1 and Mv1 pointing to reference picture 2. Mv0 is selected as the single-prediction motion vector to be included in the single-prediction motion vector set because reference picture 1 precedes the current picture in the display order, even though reference picture 2 may be closer or the same distance as the current picture in the display order.

[0492] In another example, in some embodiments, the motion vector of the reference picture of the bi-predictive motion vector that is closest to the current picture and after the current picture in display order may be included in the uni-predictive motion vector candidate set. Fig.53 As shown, the dual-prediction motion vector candidates include Mv0 pointing to reference picture 1 and Mv1 pointing to reference picture 2. Mv1 is selected as the single-prediction motion vector to be included in the single-prediction motion vector set because even though reference picture 1 may be closer or the same distance as the current picture in display order, reference picture 2 is after the current picture in display order.

[0493] In another example, in some embodiments, the motion vector of the reference picture of the bi-predictive motion vector (closest to the current picture in display order and pointing to reference picture list 0) can be included in the set of uni-predictive motion vector candidates. Other combinations of ways to derive uni-predictive motion vector candidates from the motion vector candidate list can be used (e.g., including uni-predictive motion vector candidates from the list and uni-predictive motion vector candidates derived from the bi-predictive motion vector candidates in the list).

[0494] As mentioned above, a single-predictive motion vector in a single-predictive motion vector set may also be derived from a single-predictive motion vector. Figure 54 to Figure 56 is a conceptual diagram for illustrating generation of one or more single-predictive motion vectors from a single-predictive motion vector.

[0495] For example, in some embodiments, only single-prediction motion vectors in a candidate motion vector list may be included in a single-prediction motion vector candidate set, and additional single-prediction motion vectors may be derived from the single-prediction motion vectors in the candidate motion vector list and included in the single-prediction motion vector set. Fig.54 A single-predicted motion vector Mv0 is shown in a candidate motion vector list included in a single-predicted motion vector candidate set. Mv0 points to a reference picture in list L0 associated with the index of the single-predicted motion vector. In addition to Mv0, Mv0' (a mirror image of Mv0) pointing to a reference picture list L1 associated with the index of the single-predicted motion vector is also included in the single-predicted motion vector set.

[0496] In another example, Fig.55 1 shows a single prediction motion vector Mv0 included in the candidate motion vector list in the single prediction motion vector candidate set. Mv0 points to the first reference picture in the list L0 associated with the index of the single prediction motion vector. In addition to Mv0, a scaled motion vector of Mv0 may also be included in the single prediction motion vector candidate set. Fig.55 As shown, Mv0' (which is a scaled version of Mv0 pointing to the second reference picture in list L0) and MV0" (which is a scaled version of Mv0 pointing to the third reference picture in list L0) are also included in the single-prediction motion vector set.

[0497] In some embodiments, individual motion vectors of the list of uni-predictive motion vector candidates and the list of bi-predictive motion vector candidates may be included in the uni-predictive motion vector candidate set. Figures 51 to 53 , both Mv0 and Mv1 will be included in the single-prediction motion vector candidate set, as well as any single-prediction motion vector candidates included in the list. The single-prediction motion vector candidate set may typically include five or six single-prediction motion vector candidates (e.g., five for a partition and six for a block).

[0498] In some embodiments, the uni-predictive motion vector to be included in the set of uni-predictive motion vectors may be derived from the two motion vectors of the bi-predictive motion vector candidate. Fig.56 An example of deriving a single-prediction motion vector to be included in a set from two motion vectors of a dual-prediction motion vector candidate is shown. Fig.56 As shown, the dual-predictive motion vector candidates include Mv0 pointing to reference picture 0 and Mv1 pointing to reference picture 1. Mv1 (a mirror image of Mv1') is generated, which points to reference picture 0 (the reference picture pointed to by Mv0). Mv0 and Mv1' can be averaged, and the resulting vector can be included in the set of single-predictive motion vector candidates (e.g., together with any single-predictive motion vector candidates in the list).

[0499] In some embodiments, the single predicted motion vector of the list and the average motion vector of two motion vectors of adjacent partitions of at least one partition are included in the set. In some embodiments, the single predicted motion vector of the list and the motion vector derived from the motion vector of the adjacent partition are included in the set. The adjacent partitions can be encoded using a motion vector derivation model, for example in an affine mode. In some embodiments, the single predicted motion vector of the list and a weighted combination of multiple motion vectors of the adjacent partitions are included in the set. The weights applied to the motion vectors in the multiple motion vectors can be based on the position, size, encoding mode, etc. of the adjacent partitions, for example.

[0500] In an embodiment, the first motion vector is predicted (see Fig.47A S1002 and Fig.47B S1002′) may include selecting a motion vector from a first set of motion vector candidates, and comparing the selected motion vector with motion vectors of neighboring partitions. Fig.57 is a conceptual diagram for illustrating unidirectional motion vectors of the first and second partitions of an image block. Fig.57 As shown, Mv0 can be selected from the motion vector candidate set for at least the first partition (shown as a white triangle), and Mv1 can be selected from the motion vector candidate set for the adjacent partition (shown as a shaded triangle). Note that the two triangular partitions of the block do not have the same single prediction motion vector candidate.

[0501] In an embodiment, Mv0 is a uni-prediction motion vector and is used to predict the motion vector of the first partition.

[0502] In an embodiment, Mv0 is a dual-prediction motion vector candidate and Mv1 is a single-prediction motion vector candidate. For example, a motion vector of Mv0 pointing to the same prediction direction as Mv1 pointing (e.g., pointing to list L0 or pointing to list L1) may be selected as a motion vector for predicting a first motion vector for at least a first partition. In another example, for example, a motion vector of Mv0 pointing to the opposite prediction direction to which Mv1 points (e.g., pointing to list L0 or list L1) may be selected as a motion vector for predicting a first motion vector for at least a first partition.

[0503] In an embodiment, both Mv0 and Mv1 are dual-prediction motion vector candidates. The difference between Mv0 and Mv1 may be determined, and for example, if the difference in the L0 direction is greater than or equal to the difference in the L1 direction, the motion vector of Mv0 in the L0 direction may be used to predict the first motion vector, otherwise the motion vector of Mv0 in the L1 direction may be used. In another example, if the difference in the L0 direction is greater than or equal to the difference in the L1 direction, the motion vector of Mv0 in the L1 direction may be used to predict the first motion vector, otherwise the motion vector of Mv0 in the L0 direction may be used.

[0504] In another example, predicting the first motion vector may include selecting a motion vector from a set of uni-predictive motion vectors based on a position or a size of a uni-predictive motion vector candidate.

[0505] The single-prediction motion vector candidate set for the adjacent partition may include, for example, five single-prediction motion vector candidates (e.g., {Mv1, Mv2, Mv3, Mv4, Mv5}). If Mv2 is selected as the single-prediction motion vector from which the motion vector is predicted for the adjacent partition, Mv2 will not be included in (or may be excluded from) the single-prediction motion vector candidate set from which the first motion vector is predicted for at least the first partition. If Mv2 is removed from the set of single-prediction motion vector candidate sets (e.g., Mv6), Mv2 may be replaced by another single-prediction motion vector.

[0506] In an embodiment, whether to use the first motion vector candidate set (which may contain both uni-predictive and bi-predictive motion vector candidates) or the second uni-predictive motion vector candidate set to predict the motion vector of a partition of the image block may be based on various criteria.

[0507] For example, Fig.58 is a flowchart showing an example of the following process flow 2000 according to one embodiment: predicting the motion vector of a partition of an image block from a first single-prediction motion vector candidate set or from a second motion vector candidate set (which includes dual-prediction motion vector candidates and single-prediction motion vector candidates) based on the shape of the partition, and performing further processing.

[0508] In step S2001, an image block is divided into a plurality of partitions including at least a first partition, where the first partition may or may not have a non-rectangular shape. Fig.48 , which is an example of segmenting an image block into multiple partitions.

[0509] In step S2002, the process flow 2000 judges or determines whether the current partition (e.g., as shown in the figure, at least the first partition) is a rectangular partition. When it is determined at step S2002 that the current partition is a rectangular partition, the process flow 2000 proceeds to S2004. When it is determined at S2002 that the current partition is not a rectangular partition, the process flow 2000 proceeds to S2003.

[0510] In step S2004, a first motion vector is predicted from a first single-prediction motion vector candidate set, for example, the first single-prediction motion vector candidate set may be generated as described above. In step S2003, the first motion vector is predicted from a second motion vector candidate set, which may include both single-prediction and dual-prediction motion vector candidates. In step S2005, the current partition is encoded or decoded using the first motion vector.

[0511] In another example, Fig.59 is a flowchart showing an example of the following process flow according to one embodiment: predicting the motion vector of a partition of an image block from a first single-prediction motion vector candidate set or from a second motion vector candidate set (including dual-prediction motion vector candidates and single-prediction motion vector candidates) based on the size of the block or the size of the partition, and performing further processing.

[0512] In step S3001, the image block is divided into a plurality of partitions including at least a first partition, where the first partition may or may not have a non-rectangular shape. Fig.48 , which is an example of segmenting an image block into multiple partitions.

[0513] In step S3002, the process flow 3000 judges or determines whether the size of the current block or current partition is greater than a threshold size. When it is determined at step S3002 that the size of the current block or current partition is not greater than the threshold size, the process flow 3000 proceeds to S3004. When it is determined at S3002 that the size of the current block or partition is greater than the threshold size, the process flow 3000 proceeds to S3003.

[0514] In step S3004, a first motion vector is predicted from a first single-prediction motion vector candidate set, for example, the first single-prediction motion vector candidate set may be generated as described above. In step S3003, the first motion vector is predicted from a second motion vector candidate set, which may include both single-prediction and dual-prediction motion vector candidates. In step S3005, the current partition is encoded or decoded using the first motion vector.

[0515] Various threshold sizes may be used. For example, a 16×16 pixel threshold size may be used, and a single prediction motion vector candidate set may be used to process blocks or partitions smaller than 16×16 pixels, while a dual prediction may be used to process larger sizes. In addition, the size of the image block or partition may be the width of the image block or partition, the height of the image block or partition, the ratio of the width to the height of the image block or partition, the ratio of the height to the width of the image block or partition, the number of luminance samples of the image block or partition, the number of samples of the image block or partition, etc., and various combinations thereof.

[0516] Fig.60 is a flow chart illustrating an example of a process flow 4000 of deriving a motion vector for a first partition and a motion vector for a second partition from a set of candidate motion vectors for the first partition and the second partition, and performing further processing, according to one embodiment.

[0517] In step S4001, the image block is divided into a plurality of partitions including at least a first partition and a second partition, wherein the first partition and the second partition may or may not have a non-rectangular shape. For example, the first partition and the second partition may be triangular-shaped partitions. Fig.48 , which is an example of segmenting an image block into multiple partitions.

[0518] In step S4002, the process flow 4000 creates a first motion vector candidate set for the first and second partitions. The first motion vector candidate set may be a motion vector candidate list of the image block. The motion vector candidates of the motion vector candidate list may include a dual-prediction motion vector candidate and a single-prediction motion vector candidate. In an embodiment, the first motion vector candidate set may be a dual-prediction motion vector candidate set.

[0519] In step S4003, at least two motion vectors are selected from the first motion vector candidate set. The selected motion vector candidates may include a dual-prediction motion vector candidate and a single-prediction motion vector candidate. In an embodiment, the selected motion vector candidate is a dual-prediction motion vector candidate.

[0520] In step S4004, the process flow 4000 derives a first motion vector for the first partition and a second motion vector for the second partition according to the comparison result based on at least two selected motion vector candidates. This can be done in various ways.

[0521] For example, for a total of four motion vectors, at least two selected motion vector candidates may be dual-prediction motion vector candidates each having two motion vectors. The derived first and second derived motion vectors may be two motion vectors of the four motion vectors with the largest magnitude differences among all four motion vectors of the at least two selected motion vector candidates. The derived first and second motion vectors are single-prediction motion vectors.

[0522] In another example, for a total of four motion vectors, at least two selected motion vector candidates may be dual-prediction motion vector candidates, each having two motion vectors. The derived first and second derived motion vectors may be two motion vectors of the four motion vectors with the smallest magnitude difference among all four motion vectors of the at least two selected motion vector candidates. The derived first and second motion vectors are single-prediction motion vectors.

[0523] In another example, at least one of the at least two selected motion vector candidates may be a dual-prediction motion vector candidate having two motion vectors (three or four motion vectors in total). The derived first and second derived motion vectors may be two motion vectors of the motion vectors of the selected motion vector candidates that are the largest or smallest of the motion vectors of the at least two selected motion vector candidates. The derived first and second motion vectors are uni-prediction motion vectors. In step S4005, the first motion vector, the first and second partitions are encoded or decoded using the first and second derived motion vectors.

[0524] One or more aspects disclosed herein may be performed by combining at least a portion of other aspects in the present disclosure. In addition, one or more aspects disclosed herein may be performed by combining a portion of the process indicated in any flow chart according to these aspects, a portion of the configuration of any device, a portion of grammar, etc. with other aspects.

[0525] (Implementation Methods and Applications)

[0526] As described in each of the above embodiments, each function or operation block can generally be implemented as, for example, an MPU (microprocessing unit) and a memory. In addition, the process performed by each function block can be implemented as a program execution unit, such as a processor that reads and executes software (programs) recorded on a recording medium such as a ROM. The software can be distributed. The software can be recorded on various recording media such as semiconductor memories. Note that each function block can also be implemented as hardware (special purpose circuits). Various combinations of hardware and software can be used.

[0527] The processing described in each embodiment can be implemented via integrated processing using a single device (system), and alternatively, can be implemented via distributed processing using multiple devices. In addition, the processor that executes the above program can be a single processor or multiple processors. In other words, integrated processing can be performed, and alternatively, distributed processing can be performed.

[0528] The embodiments of the present disclosure are not limited to the above exemplary embodiments; various modifications may be made to the exemplary embodiments, and the results thereof are also included in the scope of the embodiments of the present disclosure.

[0529] Next, an application example of the motion image encoding method (image encoding method) and the motion image decoding method (image decoding method) described in each of the above embodiments, and various systems that implement the application example will be described. Such a system may be characterized by including: an image encoder that adopts the image encoding method; an image decoder that adopts the image decoding method; or an image encoder-decoder that includes both an image encoder and an image decoder. Other configurations of such a system may be modified as appropriate.

[0530] (Usage example)

[0531] Fig.61 The overall configuration of a content providing system ex100 suitable for implementing content distribution services is shown. An area where communication services are provided is divided into cells of desired sizes, and base stations ex106, ex107, ex108, ex109, and ex110 (fixed wireless stations in the example shown) are located in the respective cells.

[0532] In the content providing system ex100, devices including a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via an Internet service provider ex102 or a communication network ex104 and base stations ex106 to ex110. The content providing system ex100 can combine and connect any combination of the above devices. In various embodiments, the devices can be directly connected together, or indirectly connected together via a telephone network or near field communication rather than via base stations ex106 to ex110. In addition, a streaming server ex103 can be connected to devices including a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 via, for example, the Internet ex101. The streaming server ex103 can also be connected to a terminal in a hot spot such as in an airplane ex117 via a satellite ex116.

[0533] Note that wireless access points or hotspots may be used instead of base stations ex106 to ex110. The streaming server ex103 may be directly connected to the communication network ex104 instead of through the Internet ex101 or the Internet service provider ex102, and may be directly connected to the aircraft ex117 instead of through the satellite ex116.

[0534] The camera ex113 is a device capable of capturing still images and videos, such as a digital camera. The smartphone ex115 is a smartphone device, a cellular phone, or a Personal Handyphone System (PHS) phone that can operate under mobile communication system standards of 2G, 3G, 3.9G, and 4G systems and the next-generation 5G system.

[0535] The household appliance ex114 is, for example, a refrigerator or a device included in a household fuel cell cogeneration system.

[0536] In the content providing system ex100, a terminal including an image and / or video capturing function can perform real-time streaming, for example, by connecting to a streaming server ex103 via, for example, a base station ex106. When performing real-time streaming, a terminal (for example, a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or a terminal in an airplane ex117) can perform the encoding process described in the above embodiments on a still image or video content captured by a user via the terminal, can multiplex video data obtained by encoding and audio data obtained by encoding audio corresponding to the video, and can send the obtained data to the streaming server ex103. In other words, according to an aspect of the present disclosure, the terminal functions as an image encoder.

[0537] The streaming server ex103 streams the transmitted content data to the client that requested the stream. Examples of the client include a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and a terminal in an airplane ex117 that can decode the above-mentioned coded data. The device that receives the streamed data can decode and reproduce the received data. In other words, according to one aspect of the present disclosure, each device can be used as an image decoder.

[0538] (Distributed Processing)

[0539] The streaming server ex103 may be implemented as a plurality of servers or computers between which tasks such as processing, recording, and streaming of data are divided. For example, the streaming server ex103 may be implemented as a content delivery network (CDN) that streams content via a network connecting a plurality of edge servers located around the world. In a CDN, edge servers physically close to a client may be dynamically assigned to a client. Content is cached and streamed to edge servers to reduce loading time. For example, if some type of error or change in connectivity occurs due to a traffic peak, data may be streamed at a stable high speed because the affected portion of the network may be avoided by, for example, dividing processing between a plurality of edge servers, or switching streaming duties to a different edge server and continuing streaming.

[0540] Decentralization is not limited to the division of processing for streaming; the encoding of captured data can be divided between terminals and performed by the terminal, performed on the server side, or both. In one example, in typical encoding, the processing is performed in two loops. The first loop is used to detect the complexity of the image on a frame-by-frame or scene-by-scene basis, or to detect the encoding load. The second loop is used for processing to maintain image quality and improve encoding efficiency. For example, the processing load of the terminal can be reduced and the quality and encoding efficiency of the content can be improved by having the terminal perform the first loop of encoding and the server side receiving the content perform the second loop of encoding. In this case, when a decoding request is received, it is possible to receive and reproduce the encoded data generated by the first loop performed by one terminal on another terminal in approximately real time. This makes it possible to achieve smooth real-time streaming.

[0541] In another example, the camera ex113 or the like extracts a feature quantity (feature quantity or characteristic quantity) from an image, compresses data related to the feature quantity into metadata, and transmits the compressed metadata to a server. For example, the server determines the importance of an object based on the feature quantity, and changes the quantization accuracy accordingly to perform compression suitable for the meaning (or content importance) of the image. During the second pass of compression performed by the server, the feature quantity data is particularly effective in improving the accuracy and efficiency of motion vector prediction. In addition, the terminal can process encoding with a relatively low processing load such as variable length coding (VLC), and encoding with a relatively high processing load such as context adaptive binary arithmetic coding (CABAC) can be processed by the server.

[0542] In yet another example, there is a situation where multiple terminals, such as in a stadium, shopping mall, or factory, capture multiple videos of approximately the same scene. In this case, for example, the encoding can be decentralized by dividing the processing tasks on a per-unit basis between the multiple terminals that capture the video (and, if necessary, other terminals that do not capture the video) and the server. The unit can be, for example, a group of pictures (GOP), a picture, or a tile obtained by dividing a picture. This can reduce loading time and achieve closer to real-time streaming.

[0543] Since the videos have roughly the same scenes, the server can perform management and / or instructions so that the video captured by the terminal can be cross-referenced. In addition, the server can receive the encoded data from the terminal, change the reference relationship between the data items, or correct or replace the picture itself, and then perform encoding. This makes it possible to generate streams for individual data items with improved quality and efficiency.

[0544] In addition, the server can stream the video data after performing transcoding to convert the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (such as VP9), can convert H.264 to H.265, etc.

[0545] Thus, the encoding may be performed by a terminal or one or more servers. Therefore, although the device performing the encoding is referred to as a "server" or a "terminal" in the following description, some or all of the processes performed by the server may be performed by the terminal, and similarly, some or all of the processes performed by the terminal may also be performed by the server. This also applies to the decoding process.

[0546] (3D, multi-angle)

[0547] The use of images or videos combined from images or videos of different scenes captured simultaneously or images or videos of the same scene captured from different angles through multiple terminals (e.g., camera ex113 and / or smartphone ex115) has increased. Videos captured by the terminals can be combined based on, for example, relative positional relationships obtained separately between the terminals or areas with matching feature points in the videos.

[0548] In addition to encoding two-dimensional motion images, the server can also encode still images automatically or at a user-specified time point based on scene analysis of motion pictures, and send the encoded still images to the receiving terminal. In addition, when the server can obtain the relative position relationship between the video capture terminals, in addition to the two-dimensional motion images, the server can also generate the three-dimensional geometry of the scene based on videos of the same scene captured from different angles. The server can separately encode the three-dimensional data generated from, for example, a point cloud, and based on the results of identifying or tracking people or objects using the three-dimensional data, the server can select or reconstruct and generate a video to be sent to the receiving terminal from the videos captured by multiple terminals.

[0549] This allows the user to enjoy the scene by freely selecting the video corresponding to the video capture terminal, and allows the user to enjoy the content obtained by extracting the video at the selected viewpoint from the three-dimensional data reconstructed from multiple images or videos. In addition, like the video, the sound can be recorded from relatively different angles, and the server can multiplex the audio from a specific angle or space with the corresponding video and send the multiplexed video and audio.

[0550] In recent years, content consisting of the real world and the virtual world, such as virtual reality (VR) and augmented reality (AR) content, has also become popular. In the case of VR images, the server can create images from viewpoints of both the left eye and the right eye, and perform encoding that allows reference between the two viewpoint images, such as multi-view coding (MVC), and alternatively, the image can be encoded as a separate stream without reference. When the image is decoded into a separate stream, the stream can be synchronized when reproduced so that a virtual three-dimensional space can be recreated according to the user's point of view.

[0551] In the case of an AR image, the server can overlay virtual object information existing in the virtual space onto the camera information representing the real space based on the three-dimensional position or movement from the user's perspective. The decoder can acquire or store virtual object information and three-dimensional data, generate a two-dimensional image based on the movement from the user's perspective, and then generate superimposed data by seamlessly connecting the images. Alternatively, in addition to the request for virtual object information, the decoder can also send the movement from the user's perspective to the server. The server can generate superimposed data based on the three-dimensional data stored in the server according to the received movement, and encode and stream the generated superimposed data to the decoder. Note that the superimposed data usually includes an alpha value indicating transparency in addition to the RGB value, and the server sets the alpha value of the part other than the object generated from the three-dimensional data to, for example, 0, and can perform encoding while those parts are transparent. Alternatively, the server can set the background to a determined RGB value (e.g., a chroma key) and generate data that sets the area other than the object as the background. The determined RGB value can be predetermined.

[0552] Similarly, the decoding of streaming data can be performed by a client (e.g., a terminal), performed on the server side, or divided between them. In one example, a terminal can send a reception request to a server, the requested content can be received and decoded by another terminal, and the decoded signal can be sent to a device with a display. Regardless of the processing power of the communication terminal itself, high-quality image data can be reproduced by decentralized processing and appropriate selection of content. In another example, for example, when a TV is receiving large-size image data, an area of ​​a picture such as a tile obtained by dividing a picture can be decoded and displayed on one or more personal terminals of one or more viewers of the TV. This allows viewers to share a large picture view, and each viewer can check his or her assigned area, or inspect the area more closely.

[0553] In the case where there may be multiple wireless connections at short, medium and long distances (indoors or outdoors), content can be received seamlessly using streaming system standards such as MPEG-DASH. Users can switch between data in real time while freely selecting decoders or display devices including user terminals, displays arranged indoors or outdoors, etc. In addition, using information such as about the user's location, data decoding can be performed while switching which terminal handles decoding and which terminal handles the display of content. This allows users to map and display information on the walls of nearby buildings embedded with devices capable of displaying content, or on parts of the ground while on the way to their destination. In addition, the bit rate of the received data can also be switched based on the accessibility of the encoded data on the network, such as when the encoded data is cached on a server that can be quickly accessed from the receiving terminal, or when the encoded data is copied to an edge server in a content delivery service.

[0554] (Scalable Coding)

[0555] Reference Fig.62 The scalable stream shown in the embodiment is used to describe the switching of the content, and the scalable stream is compressed and encoded through the implementation of the motion picture encoding method described in the above embodiment. The server can have such a configuration, in which the content is switched while utilizing the temporal and / or spatial scalability of the stream, which is achieved by dividing and encoding the layers, such as Fig.62 As shown. Note that there may be a plurality of individual streams with the same content but different qualities. In other words, by determining which layer to decode based on internal factors such as the processing power on the decoder side and external factors such as the communication bandwidth, the decoder side can freely switch between low-resolution content and high-resolution content while decoding. For example, in the case where a user wants to continue watching a video that the user previously watched on the smartphone ex115 while on the move, for example, at home on a device such as a TV connected to the Internet, the device can simply decode the same stream into different layers, which reduces the load on the server side.

[0556] In addition to the above configuration, in which scalability is achieved by encoding pictures by layer, where the enhancement layer is located above the base layer, the enhancement layer may also include metadata based on, for example, statistical information about the image. The decoder side can generate high image quality content by performing super-resolution imaging on the pictures in the base layer based on the metadata. Super-resolution imaging can improve the signal-to-noise ratio while maintaining and / or increasing the resolution. The metadata includes information for identifying linear or nonlinear filter coefficients used in the super-resolution process, or information for identifying parameter values ​​of a filtering process, machine learning, or least squares method used in the super-resolution process.

[0557] Alternatively, a configuration may be provided in which a picture is divided into tiles, for example, according to the meaning of an object in the image. On the decoder side, only a partial area is decoded by selecting the tile to be decoded. In addition, by storing the properties of the object (person, car, ball, etc.) and the position of the object in the video (coordinates in the same image) as metadata, the decoder side can identify the position of the desired object based on the metadata and determine which tile or tiles include the object. For example, Fig.63 As shown, a data storage structure different from pixel data can be used to store metadata, such as SEI (Supplementary Enhancement Information) messages in HEVC. The metadata indicates, for example, the position, size or color of the main object.

[0558] The metadata can be stored in units of multiple pictures, such as streams, sequences, or random access units. The decoder side can obtain, for example, the time when a specific person appears in the video, and by matching the time information with the picture unit information, the picture in which the object exists can be identified, and the position of the object in the picture can be determined.

[0559] (Web Optimization)

[0560] For example, Fig.64 An example of a display screen of a web page on the computer ex111 is shown. Fig.65 FIG. 4 shows an example of a display screen of a web page on a smartphone ex115. Fig.64 and Fig.65 As shown, a web page may include multiple image links as links to image content, and the appearance of the web page may differ depending on the device used to view the web page. When multiple image links are viewable on the screen, until the user explicitly selects the image link, or until the image link is located in the approximate center of the screen or the entire image link fits in the screen, the display device (decoder) may display a still image included in the content or an I picture as the image link; may display a video such as an animated gif using multiple still images or I pictures; or may receive only the base layer, and decode and display the video.

[0561] When the user selects an image link, the display device performs decoding while giving the highest priority to the base layer, for example. Note that if there is information in the HTML code of the web page indicating that the content is scalable, the display device can decode up to the enhancement layer. In addition, in order to ensure real-time reproduction, before a selection is made or when the bandwidth is severely limited, the display device can reduce the delay between the time point at which the leading picture is decoded and the time point at which the decoded picture is displayed by decoding and displaying only forward reference pictures (I pictures, P pictures, forward reference B pictures) (i.e., the delay between the start of decoding of the content and the display of the content). Furthermore, the display device can intentionally ignore the reference relationship between pictures and roughly decode all B and P pictures as forward reference pictures, and then perform regular decoding as the number of pictures received increases over time.

[0562] (Autonomous Driving)

[0563] When transmitting and receiving still images or video data (e.g., two-dimensional or three-dimensional map information for autonomous or assisted driving cars), in addition to the image data belonging to one or more layers, the receiving terminal may also receive information about, for example, weather or road construction as metadata, and associate the metadata with the image data when decoding. Note that the metadata may be assigned per layer, and may alternatively be simply multiplexed with the image data.

[0564] In this case, since the car, drone, airplane, etc. including the receiving terminal is movable, the receiving terminal can seamlessly receive and perform decoding by transmitting information indicating the receiving terminal while switching between base stations among base stations ex106 to ex110. Furthermore, the receiving terminal can dynamically select the degree of receiving metadata or the degree of updating map information, for example, according to the selection made by the user, the user's situation, and / or the bandwidth of the connection.

[0565] In the content providing system ex100, the client can receive, decode, and reproduce the encoded information sent by the user in real time.

[0566] (Streaming individual content)

[0567] In the content providing system ex100, in addition to high image quality, long content distributed by a video distribution entity, unicast or multicast streaming of low image quality, and short content from individuals are also possible. Such content from individuals may become more popular. The server may first perform editing processing on the content before the encoding process in order to refine the individual content. For example, this can be achieved using the following configuration.

[0568] While capturing video or image content, or after capturing and accumulating content, the server performs recognition processing, such as capture error processing, scene search processing, meaning analysis, and / or object detection processing, based on the original data or encoded data in real time. Then, based on the results of the recognition processing, the server (when prompted or automatically) edits the content, examples of which include: corrections such as focus and / or motion blur correction; removal of low-priority scenes, such as scenes with lower brightness than other pictures, or scenes that are not in focus; object edge adjustment; and hue adjustment. The server encodes the edited data based on the editing results. It is known that videos that are too long tend to receive fewer views. Therefore, in order to keep the content within a specific length proportional to the length of the original video, in addition to the above-mentioned low-priority scenes, the server can also automatically crop out scenes with low motion based on the image processing results. Alternatively, the server can generate and encode a video summary based on the analysis results of the meaning of the scene.

[0569] There are instances where individual content may include content that infringes copyright, moral rights, portrait rights, etc. Such instances may put the creator in a disadvantageous position, such as when content sharing exceeds the scope expected by the creator. Thus, for example, before encoding, the server may, for example, edit the image so as to blur the faces of people on the periphery of the screen or blur the interior of the house. In addition, the server may be configured to recognize the faces of people other than registered people in the image to be encoded, and when such faces appear in the image, a mosaic filter may be applied to, for example, the face of the person. Alternatively, as pre-processing or post-processing for encoding, the user may specify an area of ​​the image including people or an area of ​​the background to be processed for copyright reasons. The server may, for example, process the specified area by replacing the area with a different image or blurring the area. If the area includes a person, the person can be tracked in a moving picture, and when the person moves, the head area of ​​the person can be replaced with another image.

[0570] Since there is a demand for watching personally generated content in real time and the data size of this content tends to be small, the decoder can first receive the base layer as the highest priority and perform decoding and reproduction, but this may vary depending on the bandwidth. When the content is reproduced twice or more, for example, when the decoder receives the enhancement layer during the decoding and reproduction of the base layer and loops the reproduction, the decoder can reproduce a high-image quality video including the enhancement layer. If the stream is encoded using such scalable coding, the video may be of low quality when it is not selected or at the beginning of the video, but it can provide an experience in which the image quality of the stream gradually improves in an intelligent manner. This is not limited to scalable coding; the same experience can be provided by configuring a single stream of a low-quality stream from the first reproduction and a second stream encoded using the first stream as a reference.

[0571] (Other Embodiments and Application Examples)

[0572] The encoding and decoding can be performed by an LSI (Large Scale Integrated Circuit) ex500 (see Fig.61 ) is executed. LSI ex500 may be composed of a single chip or a plurality of chips. Software for encoding and decoding moving images may be integrated into a certain type of recording medium (e.g., a CD-ROM, a floppy disk, or a hard disk) that can be read by, for example, the computer ex111, and may be encoded and decoded using the software. Furthermore, when the smartphone ex115 is equipped with a camera, video data obtained by the camera may be transmitted. In this case, the video data may be encoded by the LSI ex500 included in the smartphone ex115.

[0573] Please note that LSI ex500 can be configured to download and activate applications. In this case, the terminal first determines whether it is compatible with the scheme used to encode the content, or whether it can perform a specific service. When the terminal is not compatible with the encoding scheme of the content, or when the terminal is not able to perform a specific service, the terminal can first download the codec or application software, and then acquire and reproduce the content.

[0574] In addition to the example of the content providing system ex100 using the Internet ex101, at least the moving picture encoder (image encoder) or moving picture decoder (image decoder) described in the above embodiments can be implemented in a digital broadcasting system. The same encoding process and decoding process can be applied to the transmission and reception of broadcast radio waves superimposed with multiplexed audio and video data using, for example, a satellite, although this is equipped for multicast, and unicast is easier for the content providing system ex100.

[0575] (Hardware Configuration)

[0576] Fig.66 Shows Fig.61 More details of the smartphone ex115 shown in. Fig.67An example of a configuration of a smartphone ex115 is shown. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves to and from the base station ex110, a camera ex465 capable of capturing video and still images, and a display ex458 for displaying decoded data, such as video captured by the camera ex465 and video received by the antenna ex450. The smartphone ex115 also includes a user interface ex466 such as a touch panel, an audio output unit ex457 such as a speaker for outputting voice or other audio, an audio input unit ex456 such as a microphone for audio input, a memory ex467 capable of storing decoded data such as captured video or still images, recorded audio, received video or still images, and mails, as well as decoded data, and a slot ex464 which is an interface of the SIM ex468 for authorizing access to a network and various data. Note that an external memory may be used instead of the memory ex467.

[0577] A main controller ex460 that can comprehensively control the display ex458 and the user interface ex466, a power supply circuit ex461, a user interface input controller ex462, a video signal processor ex455, a camera interface ex463, a display controller ex459, a modulator / demodulator ex452, a multiplexer / demultiplexer ex453, an audio signal processor ex454, a slot ex464, and a memory ex467 are connected via a bus ex470.

[0578] When the user turns on the power button of the power circuit ex461, the smartphone ex115 is powered on to enter an operable state, and power is supplied to each component from the battery pack.

[0579] The smart phone ex115 performs processing such as calling and data transmission based on control performed by the main controller ex460 including a CPU, ROM, and RAM. When making a call, the audio signal recorded by the audio input unit ex456 is converted into a digital audio signal by the audio signal processor ex454 (spread spectrum processing is applied to the digital audio signal and digital-to-analog conversion is performed by the modulator / demodulator ex452), and frequency conversion processing is applied by the transmitter / receiver ex451, and the resulting signal is transmitted through the antenna ex450. The received data is amplified, frequency converted, and analog-to-digital converted, inversely spread spectrum processed by the modulator / demodulator ex452, converted into an analog audio signal by the audio signal processor ex454, and then output from the audio output unit ex457. In the data transmission mode, for example, text, still image, or video data can be transmitted under the control of the main controller ex460 via the user interface input controller ex462 based on the operation of the user interface ex466 of the main body. Similar transmission and reception processing is performed. In the data transmission mode, when transmitting video, still images, or video and audio, the video signal processor ex455 compresses and encodes the video signal stored in the memory ex467 or the video signal input from the camera ex465 via the motion image encoding method described in the above embodiments, and transmits the encoded video data to the multiplexer / demultiplexer ex453. The audio signal processor ex454 encodes the audio signal recorded by the audio input unit ex456 while the camera ex465 is capturing video or still images, and transmits the encoded audio data to the multiplexer / demultiplexer ex453. The multiplexer / demultiplexer ex453 multiplexes the encoded video data and the encoded audio data using a determined scheme, modulates and converts the data using a modulator / demodulator (modulator / demodulator circuit) ex452 and a transmitter / receiver ex451, and transmits the result via the antenna ex450. The determined scheme may be predetermined.

[0580] For example, when a video attached to an e-mail or a chat or a video linked from a web page is received, in order to decode the multiplexed data received via the antenna ex450, the multiplexer / demultiplexer ex453 demultiplexes the multiplexed data to divide the multiplexed data into a bit stream of video data and a bit stream of audio data, supplies the encoded video data to the video signal processor ex455 via the synchronous bus ex470, and supplies the encoded audio data to the audio signal processor ex454 via the synchronous bus ex470. The video signal processor ex455 decodes the video signal using a moving image decoding method corresponding to the moving image encoding method described in the above embodiment, and the video or still image included in the linked moving image file is displayed on the display ex458 via the display controller ex459. The audio signal processor ex454 decodes the audio signal and outputs the audio from the audio output unit ex457. Since real-time streaming is becoming more and more popular, there may be instances where the reproduction of audio may be socially inappropriate depending on the user's environment. Therefore, as an initial value, a configuration that reproduces only video data, ie, does not reproduce audio signals, may be preferred; audio may be synchronized and reproduced only when an input is received (eg, when a user clicks on video data).

[0581] Although the smartphone ex115 is used in the above example, other embodiments are also conceivable: a transceiver terminal including both an encoder and a decoder; a transmitter terminal including only an encoder; a receiver terminal including only a decoder. In the description of the digital broadcasting system, an example is given in which multiplexed data obtained as a result of video data and audio data being multiplexed is received or transmitted. However, the multiplexed data may be video data multiplexed with data other than audio data (e.g., text data related to the video). Furthermore, video data itself may be received or transmitted instead of the multiplexed data.

[0582] Although the main controller ex460 including the CPU is described as controlling the encoding or decoding process, various terminals generally include a GPU. Therefore, a configuration is acceptable in which a large area is processed at a time by utilizing the execution capability of the GPU via a memory shared by the CPU and the GPU or a memory including addresses managed to allow the CPU and the GPU to be used in common. This makes it possible to shorten the encoding time, maintain the real-time nature of the stream, and reduce latency. In particular, for example, processing related to motion estimation, deblocking filtering, sample adaptive offset (SAO), and transform / quantization can be efficiently performed all at once by the GPU instead of the CPU in units of pictures.

Claims

1. An image encoder, comprising: Circuit; as well as a memory coupled to the circuit; Wherein, the circuit is in operation: Determine the current image block using a coding tree unit (CTU); Determine a plurality of partitions in the current image block, the plurality of partitions comprising a first partition and a second partition in the current image block, each of the first partition and the second partition having a non-rectangular shape, and the first partition overlaps with the second partition; selecting a first uni-predictive motion vector from among the bi-predictive motion vector candidates for the first partition; and The first partition is encoded using the first single-predictive motion vector, wherein The first uni-predictive motion vector is derived from the bi-predictive motion vector candidate based on an index associated with the bi-predictive motion vector candidate.

2. An image decoder, comprising: Circuit; as well as a memory coupled to the circuit; Wherein, the circuit is in operation: Determine the current image block using a coding tree unit (CTU); Determine a plurality of partitions in the current image block, the plurality of partitions comprising a first partition and a second partition in the current image block, each of the first partition and the second partition having a non-rectangular shape, and the first partition overlaps with the second partition; selecting a first uni-predictive motion vector from among the bi-predictive motion vector candidates for the first partition; and The first partition is decoded using the first uni-predictive motion vector, wherein the first uni-predictive motion vector is derived from a bi-predictive motion vector candidate based on an index associated with the bi-predictive motion vector candidate.

3. A non-transitory computer-readable medium storing a bitstream, the bitstream comprising a current image block, the bitstream causing a decoder to perform partition processing on the current image block, In the partitioning process: The current image block is determined using a coding tree unit (CTU); Determine a first partition and a second partition in the current image block, each of the first partition and the second partition has a non-rectangular shape, and the first partition overlaps with the second partition; For the first partition, selecting a first single-prediction motion vector from dual-prediction motion vector candidates of the first partition; as well as The first partition is decoded using the first single-predictive motion vector, wherein The first uni-predictive motion vector is derived from the bi-predictive motion vector candidate based on an index associated with the bi-predictive motion vector candidate.

Images