Systems and methods for video coding

By using motion vectors to predict and weight images in video encoding, and optimizing inter- and intra-frame prediction, the problem of inefficient video encoding in the prior art is solved, and a more efficient encoding and decoding process is achieved.

CN118972565APending Publication Date: 2024-11-15PANASONIC INTELLECTUAL PROPERTY CORP OF AMERICA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411332100.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-09-07
Filing Date
2019-09-03
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing video encoding technology has problems of inefficiency in inter-frame prediction, especially in the prediction process of building current frames based on reference frames, and it is difficult to effectively handle the growing amount of digital video data in various applications.

Method used

Using an image encoder and decoder system, one or more motion vectors are used to predict and weight the partitions of the image, generate the prediction signal of the current frame through inter-frame and intra-frame predictors, and use entropy encoding and inverse transformation, etc. Steps are encoded and decoded to optimize component selection and operation process during the encoding process.

Benefits of technology

Improves encoding efficiency, accelerates the encoding and decoding process, and enables efficient selection of appropriate components and operations, such as appropriate filters, block sizes, motion vectors and reference pictures, improving the performance of video encoding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972565A_ABST
    Figure CN118972565A_ABST
Patent Text Reader

Abstract

The invention relates to a system and method for video coding. An image encoder or decoder includes a circuit and a memory. The circuitry, in operation, predicts a first set of samples of a first partition using one or more motion vectors including a first motion vector, and predicts a second set of samples of a first portion of the first partition using one or more motion vectors from a second partition. Samples in a first set of samples of a first portion of the first partition and samples in a second set of samples of the first portion of the first partition are weighted. A motion vector for the first portion of the first partition is stored based on one or both of the first motion vector and the second motion vector. The first partition is encoded or decoded using weighted samples of at least a first portion of the first partition.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application. The original application was a patent application submitted to the China Patent Office on February 25, 2021 (the international application date is September 3, 2019), with application number 201980056038.1, and the name of the invention is “System and method for video encoding”. Technical Field

[0002] The present disclosure relates to video encoding, and in particular to video encoding and decoding systems, components, and methods for performing inter-frame prediction functions to construct a prediction of a current frame based on a reference frame. Background Art

[0003] With the development of video coding technology, from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Codec), there is always a need to improve and optimize video coding technology to handle the growing amount of digital video data in various applications. The present disclosure relates to further advancements, improvements and optimizations in video coding, in particular, inter-frame prediction functions for building a prediction of a current frame based on a reference frame. Summary of the invention

[0004] In one aspect, an image encoder includes a circuit and a memory coupled to the circuit. The circuit, in operation, predicts a first set of samples of a first partition of a current picture using one or more motion vectors including a first motion vector, and predicts a second set of samples of a first portion of the first partition using one or more motion vectors from a second partition different from the first partition. Samples in the first set of samples of the first portion of the first partition and samples in the second set of samples of the first portion of the first partition are weighted. A motion vector for the first portion of the first partition is stored based on one or both of the first motion vector and the second motion vector. The first partition is encoded using at least the weighted samples of the first portion of the first partition.

[0005] In one aspect, an image encoder includes: a partitioner, in operation, receiving and partitioning an original picture into blocks; a first adder, in operation, receiving blocks from the partitioner and receiving predictions from a prediction controller, and subtracting each prediction from the block corresponding to each prediction to output a residual; a transformer, in operation, performing a transformation on the residual output from the adder to output a transformation coefficient; a quantizer, in operation, quantizing the transformation coefficient to generate a quantized transformation coefficient; an entropy encoder, in operation, encoding the quantized transformation coefficient to generate a bitstream; an inverse quantizer and an inverse transformer, in operation, inverse quantizing the quantized transformation coefficient to obtain a transformation coefficient, and inverse transforming the transformation coefficient to obtain a residual; a second adder, in operation, adding the residual output from the inverse quantizer and the inverse transformer to the prediction output from the prediction controller to reconstruct the block; and a prediction controller, coupled to an inter-frame predictor, an intra-frame predictor, and a memory. The inter-frame predictor generates a prediction of a current block based on a reference block in an encoded reference picture in operation. The intra predictor is operable to generate a prediction of a current block based on a coded reference block in a current picture. The inter predictor is operable to predict a first set of samples of a first partition of the current picture using one or more motion vectors including a first motion vector, and to predict a second set of samples of a first portion of the first partition using one or more motion vectors from a second partition different from the first partition. Samples in the first set of samples of the first portion of the first partition and samples in the second set of samples of the first portion of the first partition are weighted. A motion vector for the first portion of the first partition is stored based on one or both of the first motion vector and the second motion vector. The first partition is encoded using at least the weighted samples of the first portion of the first partition.

[0006] In one aspect, a method of encoding includes: predicting a first set of samples of a first partition of a current picture using one or more motion vectors including a first motion vector; and predicting a second set of samples of a first portion of the first partition using one or more motion vectors from a second partition different from the first partition. Weighting samples in the first set of samples of the first portion of the first partition and samples in the second set of samples of the first portion of the first partition. Based on one or both of the first motion vector and the second motion vector, storing a motion vector for the first portion of the first partition. Encoding the first partition using at least the weighted samples of the first portion of the first partition.

[0007] In one aspect, an image decoder includes a circuit and a memory coupled to the circuit. The circuit, in operation, predicts a first set of samples of a first partition of a current picture using one or more motion vectors including a first motion vector, and predicts a second set of samples of a first portion of the first partition using one or more motion vectors from a second partition different from the first partition. Samples in the first set of samples of the first portion of the first partition and samples in the second set of samples of the first portion of the first partition are weighted. A motion vector for the first portion of the first partition is stored based on one or both of the first motion vector and the second motion vector. The first partition is decoded using the weighted samples of at least the first portion of the first partition.

[0008] In one aspect, an image decoder includes: an entropy decoder, which receives and decodes a coded bitstream in operation to obtain quantized transform coefficients; an inverse quantizer and inverse transformer, which in operation inversely quantizes the quantized transform coefficients to obtain transform coefficients, and inversely transforms the transform coefficients to obtain residuals; an adder, which in operation adds the residual output from the inverse quantizer and inverse transformer to the prediction output from the prediction controller to reconstruct the block; and a prediction controller, which is coupled to an inter-frame predictor, an intra-frame predictor, and a memory. The inter-frame predictor is in operation to generate a prediction of a current block based on a reference block in a decoded reference picture. The intra-frame predictor is in operation to generate a prediction of a current block based on an encoded reference block in the current picture. The inter-frame predictor is in operation to predict a first group of samples of a first partition of the current picture using one or more motion vectors including a first motion vector, and to predict a second group of samples of a first portion of the first partition using one or more motion vectors from a second partition different from the first partition. Samples in the first group of samples of the first portion of the first partition and samples in the second group of samples of the first portion of the first partition are weighted. Based on one or both of the first motion vector and the second motion vector, a motion vector for a first portion of the first partition is stored. The first partition is decoded using at least the weighted samples of the first portion of the first partition.

[0009] In one aspect, a decoding method includes: predicting a first set of samples of a first partition of a current picture using one or more motion vectors including a first motion vector; and predicting a second set of samples of a first portion of the first partition using one or more motion vectors from a second partition different from the first partition. Weighting samples in the first set of samples of the first portion of the first partition and samples in the second set of samples of the first portion of the first partition. Based on one or both of the first motion vector and the second motion vector, storing a motion vector for the first portion of the first partition. Decoding the first partition using at least the weighted samples of the first portion of the first partition.

[0010] Some implementations of the embodiments of the present disclosure can improve coding efficiency, can simplify the encoding / decoding process, can accelerate the speed of the encoding / decoding process, and can efficiently select appropriate components / operations used in encoding and decoding, such as appropriate filters, block sizes, motion vectors, reference pictures, reference blocks, etc.

[0011] Additional benefits and advantages of the disclosed embodiments will become apparent from the description and drawings. Benefits and / or advantages can be obtained individually from various embodiments and features of the description and drawings, and it is not necessary to provide all embodiments and features to obtain one or more of such benefits and / or advantages.

[0012] It should be noted that the general or specific embodiments may be implemented as a system, a method, an integrated circuit, a computer program, a storage medium, or any selective combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram showing a functional configuration of an encoder according to an embodiment.

[0014] Figure 2 is a flow chart showing one example of the overall encoding process performed by the encoder.

[0015] Figure 3 is a conceptual diagram showing an example of block division.

[0016] Figure 4A is a conceptual diagram showing an example of a slice configuration.

[0017] Figure 4B is a conceptual diagram showing an example of a tile configuration.

[0018] Figure 5A is a graph showing the transform basis functions for various transform types.

[0019] Figure 5B is a conceptual diagram illustrating an example space-varying transform (SVT).

[0020] Fig. 6A is a conceptual diagram showing an example of a filter shape used in an adaptive loop filter (ALF).

[0021] Figure 6B is a conceptual diagram showing another example of a filter shape used in ALF.

[0022] Figure 6C is a conceptual diagram showing another example of a filter shape used in ALF.

[0023] Figure 7is a block diagram showing one example of a specific configuration of a loop filter used as a deblocking filter (DBF).

[0024] Figure 8 is a conceptual diagram illustrating an example of a deblocking filter having symmetric filtering characteristics with respect to a block boundary.

[0025] Fig. 9 is a conceptual diagram for illustrating a block boundary on which a deblocking filtering process is performed.

[0026] Fig.10 is a conceptual diagram showing an example of a Bs value.

[0027] Fig.11 is a flow chart showing one example of a process performed by a prediction processor of an encoder.

[0028] Fig.12 is a flow chart illustrating another example of a process performed by a prediction processor of an encoder.

[0029] Fig.13 is a flow chart illustrating another example of a process performed by a prediction processor of an encoder.

[0030] Fig.14 is a conceptual diagram showing sixty-seven intra prediction modes used in intra prediction in the embodiment.

[0031] Fig.15 is a flowchart illustrating an example basic processing flow for inter-frame prediction.

[0032] Fig.16 is a flow chart showing one example of derivation of motion vectors.

[0033] Fig.17 is a flow chart illustrating another example of derivation of motion vectors.

[0034] Fig.18 is a flow chart illustrating another example of derivation of motion vectors.

[0035] Fig.19 is a flowchart illustrating an example of inter prediction in normal inter mode.

[0036] Fig. 20 is a flowchart illustrating an example of inter prediction in merge mode.

[0037] Fig.21 is a conceptual diagram for illustrating an example of a motion vector derivation process in merge mode.

[0038] Fig. 22 is a flow chart illustrating one example of a frame rate up-conversion (FRUC) process.

[0039] Fig.23 is a conceptual diagram for illustrating an example of pattern matching (bilateral matching) between two blocks along a motion trajectory.

[0040] Fig.24 is a conceptual diagram for illustrating an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture.

[0041] Fig.25A is a conceptual diagram for illustrating an example of deriving a motion vector for each subblock based on motion vectors of a plurality of neighboring blocks.

[0042] Fig.25B is a conceptual diagram for illustrating an example of deriving a motion vector for each subblock in an affine mode using three control points.

[0043] Fig.26A is a conceptual diagram for illustrating the affine merge mode.

[0044] Fig.26B is a conceptual diagram for illustrating an affine merge mode using two control points.

[0045] Fig.26C is a conceptual diagram for illustrating an affine merging mode using three control points.

[0046] Fig. 27 is a flowchart showing one example of a process in affine merge mode.

[0047] Fig.28A is a conceptual diagram for illustrating an affine inter mode using two control points.

[0048] Fig.28B is a conceptual diagram for illustrating an affine inter mode using three control points.

[0049] Fig.29 is a flowchart for illustrating an example of a process in the affine inter mode.

[0050] Fig. 30A is a conceptual diagram for illustrating an affine inter mode, in which a current block has three control points and a neighboring block has two control points.

[0051] Fig. 30B is a conceptual diagram for illustrating an affine inter mode, in which a current block has two control points and a neighboring block has three control points.

[0052] Fig.31A is a flow chart illustrating a merge mode process including decoder motion vector refinement (DMVR).

[0053] Fig.31B is a conceptual diagram for illustrating an example of a DMVR process.

[0054] Fig.32 is a flowchart showing an example of generating a predicted image.

[0055] Fig.33 is a flowchart showing another example of generating a predicted image.

[0056] Fig.34 is a flowchart showing another example of generating a predicted image.

[0057] Fig.35 is a flowchart illustrating one example of a predicted image correction process performed by an overlapped block motion compensation (OBMC) process.

[0058] Fig.36 is a conceptual diagram for illustrating one example of a predicted image correction process performed by the OBMC process.

[0059] Fig.37 is a conceptual diagram for illustrating the generation of two triangular prediction images.

[0060] Fig.38 This is a conceptual diagram for illustrating a model assuming uniform linear motion.

[0061] Fig.39 is a conceptual diagram for illustrating one example of a predicted image generation method using a brightness correction process performed by a local illumination compensation (LIC) process.

[0062] Fig.40 is a block diagram showing an installation example of an encoder.

[0063] Fig.41 is a block diagram showing a functional configuration of a decoder according to an embodiment.

[0064] Fig.42 is a flow chart showing one example of the overall decoding process performed by the decoder.

[0065] Fig.43 is a flow chart showing one example of a process performed by a prediction processor of a decoder.

[0066] Fig.44 is a flow chart showing another example of a process performed by a prediction processor of a decoder.

[0067] Fig.45 is a flow chart showing an example of inter prediction in conventional inter mode in a decoder.

[0068] Fig.46is a block diagram showing an installation example of a decoder.

[0069] Fig.47 is a flowchart illustrating an overall process flow of partitioning an image block into a plurality of partitions including at least a second partition and a first partition having a non-rectangular shape (eg, a triangle) and performing further processing according to one embodiment.

[0070] Fig.48 Two exemplary methods of partitioning an image block into first partitions having non-rectangular shapes (eg, triangles) and second partitions (also having non-rectangular shapes in the illustrated example) are shown.

[0071] Fig.49 An example of a boundary smoothing process is shown, which includes weighting a first value of a boundary pixel based on a first partition prediction and a second value of a boundary pixel based on a second partition prediction.

[0072] Fig.50 Three further samples of a boundary smoothing process are shown, the boundary smoothing process comprising weighting a first value of a boundary pixel based on a first partition prediction and a second value of a boundary pixel based on a second partition prediction.

[0073] Fig.51 is a table of sample parameters ("first index values") and the information sets respectively encoded by the parameters.

[0074] Fig.52 It is a table showing the binarization of parameters (index values).

[0075] Fig.53 is a flow chart illustrating a process of segmenting an image block into a plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0076] Fig.54 An example of partitioning an image block into a plurality of partitions is shown, the plurality of partitions including a second partition and a first partition having a non-rectangular shape (a triangle in the example shown).

[0077] Fig.55 A further example of partitioning the image block into a plurality of partitions is shown, the plurality of partitions comprising a second partition and a first partition having a non-rectangular shape, which in the example shown is a polygon having at least five sides and corners.

[0078] Fig.56 is a flow chart illustrating a boundary smoothing process including weighting a first value of a boundary pixel based on a first partition prediction and a second value of a boundary pixel based on a second partition prediction.

[0079] Fig.57AAn example of a boundary smoothing process is shown, in which boundary pixels are predicted to have a first value to be weighted based on a first partition and a second value to be weighted based on a second partition.

[0080] Fig.57B An example of a boundary smoothing process is shown, in which boundary pixels are predicted to have a first value to be weighted based on a first partition and a second value to be weighted based on a second partition.

[0081] Fig.57C An example of a boundary smoothing process is shown, in which boundary pixels are predicted to have a first value to be weighted based on a first partition and a second value to be weighted based on a second partition.

[0082] Fig.57D An example of a boundary smoothing process is shown, in which boundary pixels are predicted to have a first value to be weighted based on a first partition and a second value to be weighted based on a second partition.

[0083] Fig.58 The present invention is a flowchart showing a method performed on an encoder side, which partitions an image block into a plurality of partitions based on partition parameters representing the partitions and writes one or more parameters including the partition parameters into a bitstream in entropy coding, wherein the plurality of partitions include a second partition and a first partition having a non-rectangular shape.

[0084] Fig.59 The present invention is a flowchart showing a method performed on a decoder side, parsing one or more parameters from a bitstream, partitioning an image block into multiple partitions based on the partition parameters, and decoding a first partition and a second partition, wherein the bitstream includes partition parameters indicating that the image block is partitioned into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape.

[0085] Fig.60 is a sample partition parameter ("first index value") and a table which may be a set of information jointly encoded by the partition parameters, respectively indicating that the image block is divided into a plurality of partitions, the plurality of partitions including the second partition and the first partition having a non-rectangular shape.

[0086] Fig.61 is a table of sample combinations of a first parameter and a second parameter, one of the first parameter and the second parameter being a partition parameter indicating partitioning of the image block into a plurality of partitions, the plurality of partitions including the second partition and the first partition having a non-rectangular shape.

[0087] Fig.621. The present invention is a flowchart showing an example of a process flow of predicting a first group of samples of a first partition of a current picture using a first motion vector, predicting a second group of samples of a first part included in the first partition using a second motion vector, weighting the first and second groups of samples, storing at least one of the first and second motion vectors for the first partition, encoding or decoding the first partition using the weighted samples, and performing further processing according to one embodiment.

[0088] Fig.63 is a conceptual diagram for illustrating an exemplary method of partitioning an image block into a first partition and a second partition.

[0089] Fig.64 is a conceptual diagram for illustrating adjacent and non-adjacent spatially neighboring partitions of a first partition.

[0090] Fig.65 is a conceptual diagram for illustrating uni-predictive and bi-predictive motion vector candidates for an image block.

[0091] Fig.66 is a conceptual diagram for illustrating an example of a first portion of a first partition and a first group of samples and a second group of samples.

[0092] Fig.67 is a conceptual diagram for illustrating a first portion of a first partition, which is a portion of the first partition overlapping a portion of an adjacent partition.

[0093] Fig.68 is a conceptual diagram for illustrating an example case where a first motion vector and a second motion vector are uni-prediction motion vectors pointing to different pictures in different reference picture lists.

[0094] Fig.69 is a conceptual diagram for illustrating an example case where a first motion vector and a second motion vector are uni-prediction motion vectors pointing to a picture in a single reference picture list.

[0095] Fig.70 is a conceptual diagram for illustrating an example case where a first motion vector and a second motion vector are uni-prediction motion vectors pointing to the same reference picture in the same reference picture list.

[0096] Fig.71 is a conceptual diagram for illustrating an example case where a first motion vector and a second motion vector are uni-prediction motion vectors pointing to the same reference picture in the same reference picture list.

[0097] Fig.72 is a conceptual diagram for illustrating an example case where a first motion vector and a second motion vector are uni-prediction motion vectors pointing to the same reference picture in the same reference picture list.

[0098] Fig.73 is a block diagram showing the overall configuration of a content providing system for realizing a content distribution service.

[0099] Fig.74 is a conceptual diagram showing an example of a coding structure in scalable coding.

[0100] Fig.75 is a conceptual diagram showing an example of a coding structure in scalable coding.

[0101] Fig.76 is a conceptual diagram showing an example of a display screen of a web page.

[0102] Fig.77 is a conceptual diagram showing an example of a display screen of a web page.

[0103] Fig.78 is a block diagram illustrating one example of a smartphone.

[0104] Fig.79 is a block diagram showing an example of the configuration of a smartphone. DETAILED DESCRIPTION

[0105] Hereinafter, embodiments will be described with reference to the accompanying drawings. Note that the embodiments described below all illustrate general or specific examples. The numerical values, shapes, materials, components, arrangement and connection of components, steps, relationships and sequences of steps, etc. shown in the following embodiments are only examples and are not intended to limit the scope of the claims.

[0106] Embodiments of encoders and decoders are described below. Embodiments are examples of encoders and decoders to which the processes and / or configurations presented in the description of aspects of the present disclosure may be applied. The processes and / or configurations may also be implemented in encoders and decoders that are different from the encoders and decoders according to the embodiments. For example, with respect to the processes and / or configurations applied to the embodiments, any of the following may be implemented:

[0107] (1) Any component of an encoder or a decoder according to the embodiments presented in the description of aspects of the present disclosure may be replaced with or combined with another component presented anywhere in the description of aspects of the present disclosure.

[0108] (2) In an encoder or decoder according to an embodiment, any changes may be made to the functions or processes performed by one or more components of the encoder or decoder, such as addition, replacement, removal, etc. of functions or processes. For example, any function or process may be replaced by or combined with another function or process appearing anywhere in the description of various aspects of the present disclosure.

[0109] (3) In the method implemented by the encoder or decoder according to the embodiment, any changes may be made, such as adding, replacing and removing one or more processes included in the method. For example, any process in the method may be replaced or combined with another process appearing anywhere in the description of the aspects of the present disclosure.

[0110] (4) One or more components included in an encoder or decoder according to an embodiment may be combined with components presented anywhere in the description of various aspects of the present disclosure, may be combined with components presented with one or more functions included anywhere in the description of various aspects of the present disclosure, and may be combined with components that implement one or more processes implemented by components presented in the description of various aspects of the present disclosure.

[0111] (5) A component including one or more functions of an encoder or decoder according to an embodiment, or a component implementing one or more processes of an encoder or decoder according to an embodiment may be combined with or replaced with a component presented anywhere in the description of the various aspects of the present disclosure, with a component including one or more functions presented anywhere in the description of the various aspects of the present disclosure, or with a component implementing one or more processes presented anywhere in the description of the various aspects of the present disclosure.

[0112] (6) In a method implemented by an encoder or a decoder according to an embodiment, any process included in the method may be replaced or combined with a process presented anywhere in the description of various aspects of the present disclosure or with any corresponding or equivalent process.

[0113] (7) One or more processes included in the method implemented by an encoder or a decoder according to the embodiment may be combined with the processes presented anywhere in the description of various aspects of the present disclosure.

[0114] (8) The implementation of the process and / or configuration presented in the description of aspects of the present disclosure is not limited to the encoder or decoder according to the embodiment. For example, the process and / or configuration may be implemented in a device for a purpose different from the motion picture encoder or motion picture decoder disclosed in the embodiment.

[0115] (Encoder)

[0116] First, an encoder according to the embodiment will be described. Figure 1 1 is a block diagram showing a functional configuration of an encoder 100 according to an embodiment. The encoder 100 is a video encoder that encodes a video in units of blocks.

[0117] like Figure 1As shown, the encoder 100 is a device for encoding an image in units of blocks, and includes a divider 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116, a block memory 118, a loop filter 120, a frame memory 122, an intra-frame predictor 124, an inter-frame predictor 126 and a prediction controller 128.

[0118] The encoder 100 is implemented as, for example, a general purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor functions as a segmenter 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116, a loop filter 120, an intra-frame predictor 124, an inter-frame predictor 126, and a prediction controller 128. Alternatively, the encoder 100 may be implemented as one or more dedicated electronic circuits corresponding to the segmenter 102, the subtractor 104, the transformer 106, the quantizer 108, the entropy encoder 110, the inverse quantizer 112, the inverse transformer 114, the adder 116, the loop filter 120, the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128.

[0119] Hereinafter, the overall flow of a process performed by the encoder 100 is described, and then each constituent element included in the encoder 100 will be described.

[0120] (Overall flow of the encoding process)

[0121] Figure 2 is a flowchart illustrating one example of the overall encoding process performed by the encoder 100 .

[0122] First, the divider 102 of the encoder 100 divides each picture included in the input image as a video into a plurality of blocks having a fixed size (e.g., 128×128 pixels) (step Sa_1). Then, the divider 102 selects a partitioning pattern (also referred to as a block shape) for the fixed-size block (step Sa_2). In other words, the divider 102 further divides the fixed-size block into a plurality of blocks forming the selected partitioning pattern. The encoder 100 performs steps Sa_3 to Sa_9 for a block (i.e., a current block to be encoded) for each of the plurality of blocks.

[0123] In other words, the prediction processor including all or part of the intra predictor 124, the inter predictor 126 and the prediction controller 128 generates a prediction signal (also referred to as a prediction block) of the current block to be encoded (also referred to as the current block) (step Sa_3).

[0124] Next, the subtractor 104 generates the difference between the current block and the predicted block as a prediction residual (also referred to as a difference block) (step Sa_4).

[0125] Next, the transformer 106 transforms the difference block, and the quantizer 108 quantizes the result to generate a plurality of quantized coefficients (step Sa_5). Note that a block having a plurality of quantized coefficients is also referred to as a coefficient block.

[0126] Next, the entropy encoder 110 encodes the coefficient block and the prediction parameters related to the generation of the prediction signal (specifically, entropy encoding) to generate an encoded signal (step Sa_6). Note that the encoded signal is also called an encoded bit stream, a compressed bit stream or a stream.

[0127] Next, the inverse quantizer 112 performs inverse quantization on the coefficient block, and the inverse transformer 114 performs inverse transformation on the result to restore a plurality of prediction residuals (ie, difference blocks) (step Sa_7).

[0128] Next, the adder 116 adds the prediction block to the restored difference block to reconstruct the current block into a reconstructed image (also referred to as a reconstructed block or a decoded image block) (step Sa_8). In this way, a reconstructed image is generated.

[0129] When the reconstructed image is generated, the loop filter 120 performs filtering on the reconstructed image as necessary (step Sa_9).

[0130] The encoder 100 then determines whether encoding of the entire picture has been completed (step Sa_10). When it is determined that encoding has not been completed ("No" in step Sa_10), the process from step Sa_2 is repeatedly executed.

[0131] Although in the above example, the encoder 100 selects one partition pattern for a fixed-size block and encodes each block according to the partition pattern, it should be noted that each block may be encoded according to a corresponding one of a plurality of partition patterns. In this case, the encoder 100 may estimate the cost of each of the plurality of partition patterns, and may select, for example, an encoded signal that can be obtained by encoding according to the partition pattern that produces the minimum cost as the output encoded signal.

[0132] As shown in the figure, the processes in steps Sa_1 to Sa_10 are sequentially performed by the encoder 100. Alternatively, two or more processes may be performed in parallel, the processes may be reordered, and so on.

[0133] (Splitters)

[0134] The splitter 102 splits each picture included in the input video into a plurality of blocks, and outputs each block to the subtractor 104. For example, the splitter 102 first splits the picture into blocks of a fixed size (e.g., 128×128). Other fixed block sizes may be used. Fixed-size blocks are also referred to as coding tree units (CTUs). The splitter 102 then splits each fixed-size block into blocks of variable sizes (e.g., 64×64 or smaller) based on recursive quadtree and / or binary tree block splitting. In other words, the splitter 102 selects a splitting pattern. Variable-size blocks are also referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). It should be noted that in various processing examples, there is no need to distinguish between CUs, PUs, and TUs; all or some blocks in a picture may be processed in units of CUs, PUs, or TUs.

[0135] Figure 3 is a conceptual diagram showing an example of block segmentation according to an embodiment. Figure 3 , solid lines represent block boundaries of blocks partitioned by quadtree block partitioning, and dotted lines represent block boundaries of blocks partitioned by binary tree block partitioning.

[0136] Here, the block 10 is a square block having 128×128 pixels (128×128 block). The 128×128 block 10 is first divided into four square 64×64 blocks (quadtree block division).

[0137] The upper left 64×64 block is further vertically divided into two rectangular 32×64 blocks, and the left 32×64 block is further vertically divided into two rectangular 16×64 blocks (binary tree block division). As a result, the upper left 64×64 block is divided into two 16×64 blocks 11 and 12 and one 32×64 block 13.

[0138] The upper right 64×64 block is partitioned into two rectangular 64×32 blocks 14 and 15 in the horizontal direction (binary tree block partitioning).

[0139] First, the 64×64 block on the lower left is divided into four square 32×32 blocks (quadtree block division). The upper left block and the lower right block in the four 32×32 blocks are further divided. The 32×32 block on the upper left is divided into two rectangular 16×32 blocks in the vertical direction, and the 16×32 block on the right is further divided into two 16×16 blocks in the horizontal direction (binary tree block division). The 32×32 block on the lower right is divided horizontally into two 32×16 blocks (binary tree block division). As a result, the 64×64 block on the lower left is divided into a 16×32 block 16, two 16×16 blocks 17 and 18, two 32×32 blocks 19 and 20, and two 32×16 blocks 21 and 22.

[0140] The lower right 64×64 block 23 is not segmented.

[0141] As mentioned above, in Figure 3 In FIG. 1 , based on recursive quadtree and binary tree block partitioning, the block 10 is partitioned into thirteen variable-sized blocks 11 to 23. This type of partitioning is also called quadtree plus binary tree (QTBT) partitioning.

[0142] It should be noted that in Figure 3 In the example above, a block is partitioned into four or two blocks (quadtree or binary tree block partitioning), but the partitioning is not limited to these examples. For example, a block can be partitioned into three blocks (ternary block partitioning). The partitioning including this ternary block partitioning is also called multi-type tree (MBT) partitioning.

[0143] (Image structure: slice / tile)

[0144] The picture may be configured in units of one or more slices or tiles so that the picture may be decoded in parallel. The picture configured in units of one or more slices or tiles may be configured by the partitioner 102.

[0145] A slice is a basic coding unit included in a picture. A picture may include, for example, one or more slices. In addition, a slice includes one or more consecutive coding tree units (CTUs).

[0146] Figure 4A : is a conceptual diagram showing an example of slice construction. For example, a picture includes 11×8 CTUs and is divided into four slices (slices 1-4). Slice 1 includes 16 CTUs, slice 2 includes 21 CTUs, slice 3 includes 29 CTUs, and slice 4 includes 22 CTUs. Here, each CTU in the picture belongs to one of the slices. The shape of each slice is a shape that can be obtained by horizontally partitioning the picture. The boundary of each slice does not have to coincide with the end of the image and can coincide with any boundary between CTUs in the image. The processing order (coding order or decoding order) of the CTUs in the slice is, for example, a raster scan order. The slice includes header information and encoded data. The characteristics of the slice can be described in the header information. These characteristics include the CTU address of the top CTU in the slice, the slice type, and the like.

[0147] A tile is a unit of a rectangular area included in a picture. Each tile may be assigned a number called a TileId in a raster scan order.

[0148] Figure 4Bis a conceptual diagram indicating an example of a tile configuration. For example, a picture includes 11×8 CTUs and is partitioned into four rectangular-area tiles (tiles 1-4). When tiles are used, the processing order of CTUs is changed from the processing order when tiles are not used. When tiles are not used, CTUs in a picture are processed in raster scan order. When tiles are used, at least one CTU in each tile is processed in raster scan order. For example, Figure 4B As shown, the processing order of the CTUs included in tile 1 is from the left end of the first row of tile 1 to the right end of the first row of tile 1, and then from the left end of the second row of tile 1 to the right end of the second row of tile 1.

[0149] Note that a tile may include one or more slices, and a slice may include one or more tiles.

[0150] (Subtractor)

[0151] The subtractor 104 subtracts the prediction signal (prediction sample input from the prediction controller 128 indicated below) from the original signal (original sample) in units of blocks input from the divider 102 and divided by the divider 102. In other words, the subtractor 104 calculates the prediction error (also referred to as the residual) of the block to be encoded (hereinafter also referred to as the current block). The subtractor 104 then outputs the calculated prediction error (residual) to the transformer 106.

[0152] The original signal is a signal that has been input to the encoder 100 and represents an image (eg, a luminance signal and two chrominance signals) of each picture included in the video. Hereinafter, a signal representing an image is also referred to as a sample.

[0153] (Converter)

[0154] The transformer 106 transforms the prediction error in the spatial domain into a transform coefficient in the frequency domain and outputs the transform coefficient to the quantizer 108. More specifically, the transformer 106 applies, for example, a defined discrete cosine transform (DCT) or a discrete sine transform (DST) to the prediction error in the spatial domain. The defined DCT or DST may be predefined.

[0155] It should be noted that the transformer 106 can adaptively select a transform type from among a plurality of transform types, and can transform the prediction error into a transform coefficient by using a transform basis function corresponding to the selected transform type. This transform is also called explicit multi-core transform (EMT) or adaptive multi-transform (AMT).

[0156] Transform types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 5Ais a diagram indicating transform basis functions for example transform types. Figure 5A In , N represents the number of input pixels. For example, selecting a transform type from among a plurality of transform types may depend on a prediction type (one of intra prediction and inter prediction) and may depend on an intra prediction mode.

[0157] Information indicating whether such EMT or AMT is applied (e.g., referred to as an EMT flag or an AMT flag) and information indicating the selected transform type are generally signaled at the CU level. It should be noted that the signaling of such information does not necessarily need to be performed at the CU level, but may be performed at another level (e.g., at a bit sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0158] In addition, the transformer 106 can retransform the transform coefficients (transform results). This retransformation is also called an adaptive secondary transform (AST) or a non-separable secondary transform (NSST). For example, the transformer 106 performs retransformation in units of sub-blocks (e.g., 4×4 sub-blocks) included in the transform coefficient block corresponding to the intra-frame prediction error. Information indicating whether NSST is applied and information about the transform matrix to be used in NSST are typically signaled at the CU level. It should be noted that the signaling of such information does not necessarily need to be performed at the CU level, but can be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0159] The transformer 106 may employ a separable transform and an inseparable transform. A separable transform is a method in which a transform is performed multiple times by performing a transform individually on each of multiple directions according to the dimensionality of the input. An inseparable transform is a method of performing a collective transform in which two or more dimensions in a multi-dimensional input are collectively regarded as a single dimension.

[0160] In one example of a non-separable transform, when the input is a 4×4 block, the 4×4 block is treated as a single array comprising 16 elements, and the transform applies a 16×16 transform matrix to the array.

[0161] In another example of a non-separable transform, a 4x4 input block is treated as a single array comprising 16 elements, and then a transform that applies a number of given rotations to the array (a hypercube given transform) may be performed.

[0162] In the transform in the transformer 106, the type of basis to be transformed into the frequency domain may be switched according to the region in the CU. Examples include a spatially varying transform (SVT). Figure 5BAs shown, in SVT, the CU is divided into two equal regions in the horizontal or vertical direction, and only one region is transformed into the frequency domain. The transform base type can be set for each region. For example, DST7 and DST8 are used. In this example, only one of the two regions in the CU is transformed, and the other is not transformed. However, both regions can be transformed. In addition, the partitioning method is not limited to partitioning into two equal regions, but can be more flexible. For example, the CU can be partitioned into four equal regions, or the information indicating the partitioning can be separately encoded and signaled in the same way as the CU partitioning. It should be noted that SVT is also called sub-block transform (SBT).

[0163] (Quantizer)

[0164] The quantizer 108 quantizes the transform coefficients output from the transformer 106. More specifically, the quantizer 108 scans the transform coefficients of the current block in a determined scanning order, and quantizes the scanned transform coefficients based on quantization parameters (QP) corresponding to the transform coefficients. The quantizer 108 then outputs the quantized transform coefficients of the current block (hereinafter also referred to as quantized coefficients) to the entropy encoder 110 and the inverse quantizer 112. The determined scanning order may be predetermined.

[0165] The determined scanning order is the order for quantizing / inverse quantizing the transform coefficients. For example, the determined scanning order may be defined as an ascending order of frequency (from low frequency to high frequency) or a descending order of frequency (from high frequency to low frequency).

[0166] The quantization parameter (QP) is a parameter that defines the quantization step size (quantization width). For example, when the value of the quantization parameter increases, the quantization step size also increases. In other words, when the value of the quantization parameter increases, the quantization error increases.

[0167] In addition, a quantization matrix can be used for quantization. For example, several quantization matrices can be used corresponding to frequency transform sizes such as 4×4 and 8×8, prediction modes such as intra-frame prediction and inter-frame prediction, and pixel components such as luminance and chrominance pixel components. It should be noted that quantization refers to digitizing values ​​sampled at intervals determined corresponding to determined levels. In the art, other expressions such as rounding and scaling can be used to refer to quantization, and rounding and scaling can be adopted. The determined intervals and levels can be predetermined.

[0168] The method of using the quantization matrix includes a method of using the quantization matrix that has been directly set on the encoder side and a method of using the quantization matrix that has been set to a default value (default matrix). On the encoder side, a quantization matrix suitable for the characteristics of the image can be set by directly setting the quantization matrix. However, this case has the disadvantage of increasing the amount of code used to encode the quantization matrix.

[0169] There is a method of quantizing high-frequency coefficients and low-frequency coefficients without using a quantization matrix. Note that this method is equivalent to a method using a quantization matrix (a flat matrix) whose coefficients have the same value.

[0170] The quantization matrix may be specified using, for example, a sequence parameter set (SPS) or a picture parameter set (PPS). The SPS includes parameters for a sequence, and the PPS includes parameters for a picture. Each of the SPS and the PPS may be referred to simply as a parameter set.

[0171] (Entropy Encoder)

[0172] The entropy encoder 110 generates an encoded signal (encoded bit stream) based on the quantization coefficients that have been input from the quantizer 108. More specifically, for example, the entropy encoder 110 binarizes the quantization coefficients and arithmetically encodes the binary signal, and outputs a compressed bit stream or sequence.

[0173] (Inverse Quantizer)

[0174] The inverse quantizer 112 inversely quantizes the quantized coefficients that have been input from the quantizer 108. More specifically, the inverse quantizer 112 inversely quantizes the quantized coefficients of the current block in a determined scanning order. The inverse quantizer 112 then outputs the inversely quantized transform coefficients of the current block to the inverse transformer 114. The determined scanning order may be predetermined.

[0175] (Inverter)

[0176] The inverse transformer 114 restores the prediction error (residual) by inversely transforming the transform coefficients that have been input from the inverse quantizer 112. More specifically, the inverse transformer 114 restores the prediction error of the current block by applying an inverse transform corresponding to the transform applied by the transformer 106 to the transform coefficients. The inverse transformer 114 then outputs the restored prediction error to the adder 116.

[0177] Note that since information is lost in the quantization, the recovered prediction error does not match the prediction error calculated by the subtractor 104. In other words, the recovered prediction error typically includes the quantization error.

[0178] (Adder)

[0179] The adder 116 reconstructs the current block by adding the prediction error input from the inverse transformer 114 and the prediction sample input from the prediction controller 128. The adder 116 then outputs the reconstructed block to the block memory 118 and the loop filter 120. The reconstructed block is also called a local decoded block.

[0180] (Block Storage)

[0181] The block memory 118 is, for example, a storage device for storing a block in a picture to be encoded (hereinafter referred to as a current picture) to be referenced in intra prediction. More specifically, the block memory 118 stores the reconstructed block output from the adder 116.

[0182] (Frame Memory)

[0183] The frame memory 122 is, for example, a storage device for storing a reference picture used in inter-frame prediction, and is also referred to as a frame buffer. More specifically, the frame memory 122 stores the reconstructed block filtered by the loop filter 120.

[0184] (Loop Filter)

[0185] The loop filter 120 applies a loop filter to the block reconstructed by the adder 116, and outputs the filtered reconstructed block to the frame memory 122. The loop filter is a filter used in an encoding loop (an in-loop filter), and includes, for example, a deblocking filter (DF or DBF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF).

[0186] In ALF, a least square error filter for eliminating compression artifacts is applied. For example, for each of the 2×2 sub-blocks in the current block, one filter selected from among multiple filters based on the direction and activity of the local gradient is applied.

[0187] More specifically, first, each sub-block (e.g., each 2×2 sub-block) is classified into one of a plurality of categories (e.g., fifteen or twenty-five categories). The classification of the sub-block is based on gradient directionality and activity. For example, a classification index C (e.g., C=5D+A) is derived based on gradient directionality D (e.g., 0 to 2 or 0 to 4) and gradient activity A (e.g., 0 to 4). Then, based on the classification index C, each sub-block is classified into one of a plurality of categories.

[0188] For example, the gradient directivity D is calculated by comparing the gradients in a plurality of directions (eg, horizontal direction, vertical direction, and two diagonal directions). Also, the gradient activity A is calculated, for example, by adding the gradients in a plurality of directions and quantizing the addition result.

[0189] Based on the result of this classification, a filter to be used for each sub-block is determined from among a plurality of filters.

[0190] The filter shape to be used in the ALF is, for example, a circularly symmetric filter shape. FIG. 6A to FIG. 6C An example of filter shape used in ALF is shown. Fig. 6A shows a 5×5 diamond filter, Figure 6B shows a 7×7 diamond filter, and Figure 6C A 9×9 diamond filter is shown. Information indicating the filter shape is typically signaled at the picture level. Note that such signaling of information indicating the filter shape does not necessarily need to be performed at the picture level, but can be performed at another level (e.g., at the sequence level, slice level, tile level, CTU level, or CU level).

[0191] For example, turning on or off ALF is determined at the picture level or the CU level. For example, a decision on whether to apply ALF to luma may be made at the CU level, and a decision on whether to apply ALF to chroma may be made at the picture level. Information indicating turning on or off ALF is typically signaled at the picture level or the CU level. Note that signaling of information indicating turning on or off ALF does not necessarily need to be performed at the picture level or the CU level, and may be performed at another level (e.g., at a sequence level, a slice level, a tile level, or a CTU level).

[0192] The coefficients set for a plurality of selectable filters (e.g., fifteen or up to twenty-five filters) are typically signaled at the picture level. It should be noted that the signaling of the set coefficients does not necessarily need to be performed at the picture level, but may be performed at another level (e.g., at the sequence level, slice level, tile level, CTU level, CU level, or sub-block level).

[0193] (Loop Filter > Deblocking Filter)

[0194] In the deblocking filter, the loop filter 120 performs a filtering process on a block boundary in a reconstructed image in order to reduce distortion occurring at the block boundary.

[0195] Figure 7 is a block diagram showing one example of a specific configuration of the loop filter 120 serving as a deblocking filter.

[0196] The loop filter 120 includes: a boundary determiner 1201 ; a filter determiner 1203 ; a filter executor 1205 ; a process determiner 1208 ; a filter characteristic determiner 1207 ; and switches 1202 , 1204 , and 1206 .

[0197] The boundary determiner 1201 determines whether there is a pixel to be deblocking filtered (ie, the current pixel) around the block boundary. The boundary determiner 1201 then outputs the determination result to the switch 1202 and the processing determiner 1208.

[0198] In the case where the boundary determiner 1201 has determined that the current pixel exists around the block boundary, the switch 1202 outputs the unfiltered image to the switch 1204. In the opposite case, where the boundary determiner 1201 has determined that the current pixel does not exist around the block boundary, the switch 1202 outputs the unfiltered image to the switch 1206.

[0199] The filter determiner 1203 determines whether to perform deblocking filtering on the current pixel based on the pixel value of at least one surrounding pixel located around the current pixel. Then, the filter determiner 1203 outputs the determination result to the switch 1204 and the processing determiner 1208.

[0200] In the case where the filter determiner 1203 has determined that the deblocking filtering is to be performed on the current pixel, the switch 1204 outputs the unfiltered image obtained by the switch 1202 to the filter executor 1205. In the opposite case, where the filter determiner 1203 has determined that the deblocking filtering is not to be performed on the current pixel, the switch 1204 outputs the unfiltered image obtained by the switch 1202 to the switch 1206.

[0201] When the unfiltered image is obtained through the switches 1202 and 1204, the filter executor 1205 performs deblocking filtering with the filter characteristics determined by the filter characteristic determiner 1207 on the current pixel. The filter executor 1205 then outputs the filtered pixel to the switch 1206.

[0202] Under the control of the processing determiner 1208 , the switch 1206 selectively outputs pixels that have not been deblock filtered and pixels that have been deblock filtered by the filter executor 1205 .

[0203] The processing determiner 1208 controls the switch 1206 based on the determination results made by the boundary determiner 1201 and the filter determiner 1203. In other words, when the boundary determiner 1201 has determined that the block boundary period exists in the current pixel and the filter determiner 1203 has determined that the deblocking filtering is performed on the current pixel, the processing determiner 1208 causes the switch 1206 to output the pixel that has been deblocking filtered. In addition, in addition to the above case, the processing determiner 1208 causes the switch 1206 to output the pixel that has not been deblocking filtered. By repeating the output of the pixel in this way, the filtered image is output from the switch 1206.

[0204] Figure 8 is a conceptual diagram illustrating an example of a deblocking filter having symmetric filtering characteristics with respect to a block boundary.

[0205] In the deblocking filter process, the pixel value and the quantization parameter are used to select one of two deblocking filters with different characteristics, namely a strong filter and a weak filter. In the case of a strong filter, such as Figure 8As shown, pixels p0 to p2 and pixels q0 to q2 exist across the block boundary, and the pixel values ​​of the corresponding pixels q0 to q2 are changed to pixel values ​​q'0 to q'2 by performing calculation according to the following expression, for example.

[0206] q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8

[0207] q'1=(p0+q0+q1+q2+2) / 4

[0208] q'2=(p0+q0+q1+3×q2+2×q3+4) / 8

[0209] Note that in the above expressions, p0 to p2 and q0 to q2 are pixel values ​​of the corresponding pixels p0 to p2 and pixels q0 to q2. In addition, q3 is the pixel value of the adjacent pixel q3 located on the opposite side of the pixel q2 with respect to the block boundary. In addition, on the right side of each expression, the coefficient multiplied by the corresponding pixel value of the pixel to be used for deblocking filtering is the filter coefficient.

[0210] In addition, in the deblocking filter, clipping can be performed so that the calculated pixel value is not set above the threshold. In the clipping process, the pixel value calculated according to the above expression is clipped to a value obtained according to "calculated pixel value ± 2 × threshold" using a threshold determined based on a quantization parameter. In this way, excessive smoothing can be prevented.

[0211] Fig. 9 is a conceptual diagram for illustrating a block boundary on which a deblocking filtering process is performed. Fig.10 is a conceptual diagram showing an example of a Bs value.

[0212] The block boundaries on which the deblocking filtering process is performed are, for example, Fig. 9 The deblocking filtering process can be performed in units of four rows or four columns. First, for Fig. 9 The blocks P and Q shown in Fig.10 The boundary strength (Bs) value is determined as shown.

[0213] according to Fig.10 The Bs value in the image determines whether to use different strengths to perform the deblocking filtering process on the block boundary belonging to the same image. When the Bs value is 2, the deblocking filtering process for the chrominance signal is performed. When the Bs value is 1 or more and the determination condition is met, the deblocking filtering process is performed on the luminance signal. The determination condition can be predetermined. It should be noted that the condition for determining the Bs value is not limited to Fig.10 Those shown in , the Bs value can be determined based on another parameter.

[0214] (Prediction processor (intra-frame predictor, inter-frame predictor, prediction controller))

[0215] Fig.11 is a flowchart showing one example of a process performed by the prediction processor of the encoder 100. Note that the prediction processor includes all or part of the following constituent elements: an intra predictor 124; an inter predictor 126; and a prediction controller 128.

[0216] The prediction processor generates a prediction image of the current block (step Sb_1). The prediction image is also called a prediction signal or a prediction block. It should be noted that the prediction signal is, for example, an intra-frame prediction signal or an inter-frame prediction signal. Specifically, the prediction processor uses a reconstructed image obtained by generating a prediction block, generating a difference block, generating a coefficient block, restoring a difference block, and generating a decoded image block to generate a prediction image of the current block.

[0217] The reconstructed image may be, for example, an image in a reference picture, or an image of a coding block in a current picture, where the current picture is a picture including the current block. The coding block in the current picture is, for example, an adjacent block of the current block.

[0218] Fig.12 is a flow chart illustrating another example of a process performed by the prediction processor of encoder 100 .

[0219] The prediction processor generates a prediction image using a first method (step Sc_1a), generates a prediction image using a second method (step Sc_1b), and generates a prediction image using a third method (step Sc_1c). The first method, the second method, and the third method may be different methods for generating a prediction image. Each of the first to third methods may be an inter-frame prediction method, an intra-frame prediction method, or another prediction method. The above-mentioned reconstructed image may be used in these prediction methods.

[0220] Next, the prediction processor selects any one of the multiple prediction methods generated in steps Sc_1a, Sc_1b and Sc_1c (step Sc_2). The selection of the prediction image (i.e., the selection of the method or mode for obtaining the final prediction image) can be performed by calculating a cost for each generated prediction image and based on the cost. Alternatively, the selection of the prediction image can be based on the parameters used in the encoding process. The encoder 100 can convert information for identifying the selected prediction image, method or mode into a coded signal (also called a coded bitstream). The information can be, for example, a flag, etc. In this way, the decoder is able to generate a prediction image according to the method or mode selected based on the information in the encoder 100. It should be noted that in Fig.12In the example shown, after the prediction image is generated using the corresponding method, the prediction processor selects any prediction image. However, the prediction processor can select a method or mode based on the parameters used in the above encoding process before generating the prediction image, and can generate the prediction image according to the selected method or mode.

[0221] For example, the first method and the second method may be intra prediction and inter prediction, respectively, and the prediction processor may select a final prediction image for the current block from prediction images generated according to the prediction methods.

[0222] Fig.13 is a flow chart illustrating another example of a process performed by the prediction processor of encoder 100 .

[0223] First, the prediction processor generates a prediction image using intra prediction (step Sd_1a), and generates a prediction image using inter prediction (step Sd_1b). Note that the prediction image generated by intra prediction is also referred to as an intra prediction image, and the prediction image generated by inter prediction is also referred to as an inter prediction image.

[0224] Next, the prediction processor estimates each of the intra-frame prediction image and the inter-frame prediction image (step Sd_2). The cost can be used in the estimation. In other words, the prediction processor calculates the cost C for each of the intra-frame prediction image and the inter-frame prediction image. The cost C can be calculated according to the expression of the RD optimization model, for example, C=D+λ×R. In this expression, D represents the encoding distortion of the predicted image, and is expressed as, for example, the sum of the absolute differences between the pixel values ​​of the current block and the pixel values ​​of the predicted image. In addition, R represents the predicted encoding amount of the predicted image, specifically, the amount of encoding required to encode the motion information, etc. used to generate the predicted image. In addition, λ represents a multiplier, for example, according to the method of the Lagrange multiplier.

[0225] Then, the prediction processor selects the prediction image for which the minimum cost C has been calculated among the intra prediction image and the inter prediction image as the final prediction image of the current block (step Sd_3). In other words, a prediction method or mode for generating the prediction image of the current block is selected.

[0226] (Intra Predictor)

[0227] The intra predictor 124 generates a prediction signal (intra prediction signal) by performing intra prediction (also referred to as intra prediction) of the current block with reference to one or more blocks in the current picture and stored in the block memory 118. More specifically, the intra predictor 124 generates the intra prediction signal by performing intra prediction with reference to samples (e.g., luminance and / or chrominance values) of one or more blocks adjacent to the current block, and then outputs the intra prediction signal to the prediction controller 128.

[0228] For example, the intra predictor 124 performs intra prediction by using one of a plurality of intra prediction modes that have been defined. The intra prediction modes include one or more non-directional prediction modes and a plurality of directional prediction modes. The defined modes may be predefined.

[0229] The one or more non-directional prediction modes include, for example, planar prediction mode and DC prediction mode defined in the H.265 / High Efficiency Video Coding (HEVC) standard.

[0230] The plurality of directional prediction modes include, for example, 33 directional prediction modes defined in the H.265 / HEVC standard. It should be noted that, in addition to the 33 directional prediction modes, the plurality of directional prediction modes may also include 32 directional prediction modes (a total of 65 directional prediction modes). Fig.14 : is a conceptual diagram showing a total of sixty-seven intra-frame prediction modes (two non-directional prediction modes and sixty-five directional prediction modes) that can be used in intra-frame prediction. The solid arrows represent the 33 directions defined in the H.265 / HEVC standard, while the dotted arrows represent the additional 32 directions ( Fig.14 The two non-directional prediction modes are not shown).

[0231] In various processing examples, a luma block may be referenced in intra prediction of a chroma block. In other words, a chroma component of a current block may be predicted based on a luma component of the current block. Such intra prediction is also referred to as cross-component linear model (CCLM) prediction. An intra prediction mode for a chroma block (where such a luma block is referenced) (also referred to as, for example, a CCLM mode) may be added as one of the intra prediction modes for a chroma block.

[0232] The intra predictor 124 may correct the pixel values ​​of the intra prediction based on the horizontal / vertical reference pixel gradients. The intra prediction with such correction is also referred to as position-dependent intra prediction combination (PDPC). Information indicating whether PDPC is applied (e.g., referred to as a PDPC flag) is typically signaled at the CU level. It should be noted that the signaling of such information does not necessarily need to be performed at the CU level, but may be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0233] (Inter-frame predictor)

[0234] The inter-frame predictor 126 generates a prediction signal (inter-frame prediction signal) by performing inter-frame prediction (also called inter-frame prediction) of the current block with reference to one or more blocks in the reference picture that are different from the current picture and stored in the frame memory 122. Inter-frame prediction is performed in units of the current block or the current sub-block (e.g., 4×4 block) in the current block. For example, the inter-frame predictor 126 performs motion estimation in the reference picture of the current block or the current sub-block, and finds the reference block or sub-block that best matches the current block or the current sub-block. Then, the inter-frame predictor 126 obtains motion information (e.g., motion vector) that compensates for the motion or change from the reference block or sub-block to the current block or sub-block. The inter-frame predictor 126 generates an inter-frame prediction signal of the current block or sub-block by performing motion compensation (or motion prediction) based on the motion information. The inter-frame predictor 126 outputs the generated inter-frame prediction signal to the prediction controller 128.

[0235] The motion information used in motion compensation can be signaled in various forms as an inter-frame prediction signal. For example, a motion vector can be signaled. As another example, the difference between a motion vector and a motion vector predictor can be signaled.

[0236] (Basic process of inter-frame prediction)

[0237] Fig.15 is a flowchart illustrating an example basic processing flow for inter-frame prediction.

[0238] First, the inter-frame predictor 126 generates a prediction signal (steps Se_1 to Se_3). Next, the subtractor 104 generates a difference between the current block and the predicted image as a prediction residual (step Se_4).

[0239] Here, in the generation of the predicted image, the inter-frame predictor 126 generates the predicted image by determining the motion vector (MV) of the current block (steps Se_1 and Se_2) and motion compensation (step Se_3). In addition, when determining the MV, the inter-frame predictor 126 determines the MV by selecting a motion vector candidate (MV candidate) (step Se_1) and deriving the MV (step Se_2). The selection of the MV candidate is performed by, for example, selecting at least one MV candidate from the MV candidate list. Alternatively, in the derivation of the MV, the inter-frame predictor 126 may further select at least one MV candidate from at least one MV candidate, and determine the selected at least one MV candidate as the MV of the current block. Alternatively, the inter-frame predictor 126 may determine the MV of the current block by performing estimation in a reference picture area specified by each MV candidate of the selected at least one MV candidate. Note that the estimation in the reference picture area may be referred to as motion estimation.

[0240] In addition, although steps Se_1 to Se_3 are performed by the inter-frame predictor 126 in the above-described example, processes such as step Se_1 , step Se_2 , and the like may be performed by another constituent element included in the encoder 100 .

[0241] (Motion vector export process)

[0242] Fig.16 is a flow chart showing one example of derivation of motion vectors.

[0243] The inter-frame predictor 126 derives the MV of the current block in a mode for encoding motion information (e.g., MV). In this case, for example, the motion information is encoded as a prediction parameter and sent with a signal. In other words, the encoded motion information is included in the encoded signal (also referred to as an encoded bitstream).

[0244] Alternatively, the inter-frame predictor 126 derives the MV in a mode in which motion information is not encoded. In this case, motion information is not included in the encoded signal.

[0245] Here, the MV derivation mode may include a conventional inter mode, a merge mode, a FRUC mode, an affine mode, etc., which are described later. Among the modes, the modes for encoding motion information include a conventional inter mode, a merge mode, an affine mode (specifically, an affine inter mode and an affine merge mode), etc. It should be noted that the motion information may include not only the MV but also the motion vector predictor selection information, which will be described later. Modes in which no motion information is encoded include the FRUC mode, etc. The inter predictor 126 selects a mode for deriving the MV of the current block from these modes, and uses the selected mode to derive the MV of the current block.

[0246] Fig.17 is a flow chart illustrating another example of derivation of motion vectors.

[0247] The inter-frame predictor 126 derives the MV of the current block in a mode of encoding the MV difference. In this case, for example, the MV difference is encoded as a prediction parameter and sent with a signal. In other words, the encoded MV difference is included in the encoded signal. The MV difference is the difference between the MV of the current block and the MV predictor.

[0248] Alternatively, the inter-frame predictor 126 derives the MV in a mode in which the MV difference is not encoded. In this case, the encoded MV difference is not included in the encoded signal.

[0249] Here, as described above, the MV derivation mode includes a normal inter mode, a merge mode, a FRUC mode, an affine mode, etc. described later. Among the modes, the mode in which the MV difference is encoded includes a normal inter mode, an affine mode (specifically, an affine inter mode), etc. The mode in which the MV difference is not encoded includes a FRUC mode, a merge mode, an affine mode (specifically, an affine merge mode), etc. The inter predictor 126 selects a mode for deriving the MV of the current block from a plurality of modes, and derives the MV of the current block using the selected mode.

[0250] (Motion vector export process)

[0251] Fig.18 : is a flowchart showing another example of the derivation of a motion vector. The MV derivation mode as an inter-frame prediction mode includes a plurality of modes, and is roughly divided into a mode in which the MV difference is encoded and a mode in which the motion vector difference is not encoded. The modes in which the MV difference is not encoded include a merge mode, a FRUC mode, an affine mode (specifically, an affine merge mode), and the like. These modes will be described in detail later. In short, the merge mode is a mode for deriving the MV of the current block by selecting a motion vector from the encoded surrounding blocks, and the FRUC mode is a mode for deriving the MV of the current block by performing estimation between the encoded regions. The affine mode is a mode for deriving the motion vector of each of the plurality of sub-blocks included in the current block as the MV of the current block assuming an affine transformation.

[0252] More specifically, when the inter-frame prediction mode information indicates 0 (0 in Sf_1), the inter-frame predictor 126 uses the merge mode (Sf_2) to derive the motion vector. When the inter-frame prediction mode information indicates 1 (1 in Sf_1), the inter-frame predictor 126 uses the FRUC mode (Sf_3) to derive the motion vector. When the inter-frame prediction mode information indicates 2 (2 in Sf_1), the inter-frame predictor 126 uses the affine mode (specifically, the affine merge mode) (Sf_4) to derive the motion vector. When the inter-frame prediction mode information indicates 3 (3 in Sf_1), the inter-frame predictor 126 uses a mode for encoding MV differences (e.g., the normal inter-frame mode (Sf_5)) to derive the motion vector.

[0253] (MV Export > Regular Interframe Mode)

[0254] The normal inter mode is an inter prediction mode for deriving the MV of the current block from a reference picture area specified by the MV candidate based on a block similar to the image of the current block. In this normal inter mode, the MV difference is encoded.

[0255] Fig.19 is a flowchart illustrating an example of inter prediction in normal inter mode.

[0256] First, the inter-frame predictor 126 obtains a plurality of MV candidates for the current block based on information such as MVs of a plurality of coding blocks temporally or spatially surrounding the current block (step Sg_1). In other words, the inter-frame predictor 126 generates a MV candidate list.

[0257] Next, the inter-frame predictor 126 extracts N (an integer of 2 or more) MV candidates as motion vector predictor candidates (also referred to as MV predictor candidates) from the multiple MV candidates obtained in step Sg_1 according to the determined priority order (step Sg_2). Note that the priority order can be predetermined for each of the N MV candidates.

[0258] Next, the inter-frame predictor 126 selects one motion vector predictor candidate from the N motion vector predictor candidates as the motion vector predictor (also referred to as MV predictor) of the current block (step Sg_3). At this time, the inter-frame predictor 126 encodes motion vector predictor selection information for identifying the selected motion vector predictor in the stream. It should be noted that the stream is a coded signal or a coded bit stream as described above.

[0259] Next, the inter-frame predictor 126 derives the MV of the current block by referring to the encoded reference picture (step Sg_4). At this time, the inter-frame predictor 126 also encodes the difference between the derived MV and the motion vector predictor as an MV difference in the stream. It should be noted that the encoded reference picture is a picture including multiple blocks that have been reconstructed after encoding.

[0260] Finally, the inter-frame predictor 126 generates a predicted image of the current block by performing motion compensation of the current block using the derived MV and the encoded reference picture (step Sg_5). Note that, as described above, the predicted image is an inter-frame prediction signal.

[0261] In addition, information indicating the inter prediction mode (normal inter mode in the above example) used to generate the predicted image is encoded as a prediction parameter, for example.

[0262] It should be noted that the MV candidate list can also be used as a list used in another mode. In addition, the process related to the MV candidate list can be applied to the process related to the list used for another mode. The process related to the MV candidate list includes, for example, extracting or selecting MV candidates from the MV candidate list, reordering of MV candidates, or deletion of MV candidates.

[0263] (MV Export > Merge Mode)

[0264] The merge mode is an inter-frame prediction mode used to select an MV candidate from an MV candidate list as the MV of the current block, thereby deriving the MV.

[0265] Fig. 20 is a flowchart illustrating an example of inter prediction in merge mode.

[0266] First, the inter-frame predictor 126 obtains a plurality of MV candidates for the current block based on information such as MVs of a plurality of coding blocks temporally or spatially surrounding the current block (step Sh_1). In other words, the inter-frame predictor 126 generates a MV candidate list.

[0267] Next, the inter-frame predictor 126 selects one MV candidate from the multiple MV candidates obtained in step Sh_1, thereby deriving the MV of the current block (step Sh_2). At this time, the inter-frame predictor 126 encodes MV selection information for identifying the selected MV candidate in the stream.

[0268] Finally, the inter predictor 126 generates a predicted image of the current block by performing motion compensation of the current block using the derived MV and the encoded reference picture (step Sh_3).

[0269] In addition, for example, information indicating the inter prediction mode (merge mode in the above example) used to generate a predicted image and included in the encoded signal is encoded as a prediction parameter.

[0270] Fig.21 is a conceptual diagram for illustrating an example of a motion vector derivation process of a current picture in a merge mode.

[0271] First, an MV candidate list in which MV predictor candidates are registered is generated. Examples of MV predictor candidates include: spatially adjacent MV predictors, which are MVs of multiple coding blocks spatially positioned around the current block; temporally adjacent MV predictors, which are MVs of surrounding blocks on which the position of the current block in the coding reference picture is projected; combined MV predictors, which are MVs generated by combining the MV values ​​of spatially adjacent MV predictors and the MVs of temporally adjacent MV predictors; and zero MV predictors, which are MVs having a zero value.

[0272] Next, one MV predictor is selected from among a plurality of MV predictors registered in the MV predictor list, and the selected MV predictor is determined as the MV of the current block.

[0273] Furthermore, the variable length encoder describes and encodes merge_idx, which is a signal indicating which MV predictor has been selected, in the stream.

[0274] It should be noted that in Fig.21The MV predictors registered in the MV predictor list described in are examples. The number of MV predictors may be different from the number of MV predictors in the figure, and the MV predictor list may be configured in the following manner: some types of MV predictors in the figure may not be included, or one or more MV predictors other than the types of MV predictors in the figure may be included.

[0275] The final MV may be determined by performing a decoder motion vector refinement process (DMVR) described later using the MV of the current block derived in the merge mode.

[0276] Note that the MV predictor candidate is the above-mentioned MV candidate, and the MV predictor list is the above-mentioned MV candidate list. Note that the MV candidate list may be referred to as a candidate list. In addition, merge_idx is MV selection information.

[0277] (MV Export > FRUC Mode)

[0278] The motion information may be derived on the decoder side without being signaled from the encoder side. Note that, as described above, the merge mode defined in the H.265 / HEVC standard may be used. Additionally, for example, the motion information may be derived by performing motion estimation on the decoder side. In an embodiment, on the decoder side, motion estimation is performed without using any pixel values ​​in the current block.

[0279] Here, a mode for performing motion estimation at the decoder side is described.The mode for performing motion estimation at the decoder side may be referred to as a pattern matching motion vector derivation (PMMVD) mode or a frame rate up conversion (FRUC) mode.

[0280] Fig. 22An example of a FRUC process is shown in the form of a flowchart. First, a list of multiple candidates each having a motion vector (MV) predictor is generated by referring to the motion vectors in the coding blocks that are adjacent to the current block in space or time (i.e., a MV candidate list that can also be used as a merge list) (step Si_1). Next, the best MV candidate is selected from the multiple MV candidates registered in the MV candidate list (step Si_2). For example, an estimated value of the corresponding MV candidate included in the MV candidate list is calculated, and an MV candidate is selected based on the estimated value. Based on the selected candidate motion vector, the motion vector of the current block is then derived (step Si_4). More specifically, for example, the selected motion vector candidate (best MV candidate) is directly derived as the motion vector of the current block. In addition, for example, the motion vector of the current block can be derived using pattern matching in the surrounding area of ​​a position in a reference picture, where the position in the reference picture corresponds to the selected motion vector candidate. In other words, estimation using pattern matching and estimation values ​​may be performed in the surrounding area of ​​the best MV candidate, and when there is an MV that produces a better estimated value, the best MV candidate may be updated to the MV that produces the better estimated value, and the updated MV may be determined as the final MV of the current block. It is also possible that a configuration is performed in which there is no process for updating the best MV candidate to the MV with a better evaluation value.

[0281] Finally, the inter-frame predictor 126 generates a predicted image of the current block by performing motion compensation of the current block using the derived MV and the encoded reference picture (step Si_5).

[0282] A similar process may be performed in units of sub-blocks.

[0283] The estimated value may be calculated according to various methods. For example, a comparison is made between a reconstructed image in a region corresponding to the motion vector in a reference picture and a reconstructed image in a determined region (the region may be, for example, a region in another reference picture or a region in an adjacent block of the current picture, as shown below). The determined region may be predetermined.

[0284] The difference between the pixel values ​​of the two reconstructed images can be used for the estimate of the motion vector. Note that information other than the difference can be used to calculate the estimate.

[0285] Next, an example of pattern matching is described in detail. First, through pattern matching, one MV candidate included in the MV candidate list (e.g., merge list) is selected as the starting point of estimation. For example, as pattern matching, first pattern matching or second pattern matching can be used. First pattern matching and second pattern matching are also called bilateral matching and template matching, respectively.

[0286] (MV Export > FRUC > Bilateral Matching)

[0287] In the first pattern matching, pattern matching is performed between two blocks along the motion trajectory of the current block, the two blocks being two blocks in two different reference pictures. Therefore, in the first pattern matching, an area in another reference picture along the motion trajectory of the current block is used as a determined area for calculating the above-mentioned candidate estimation value. The determined area may be predetermined.

[0288] Fig.23 is a conceptual diagram for illustrating an example of first pattern matching (bilateral matching) between two blocks in two reference pictures along a motion trajectory. Fig.23 As shown, in the first pattern matching, two motion vectors (MV0, MV1) are derived by estimating the best matching pair among the pairs of two blocks in two different reference pictures (Ref0, Ref1) (which are two blocks along the motion trajectory of the current block (Cur block)). More specifically, the difference between the reconstructed image at the specified position in the first coded reference picture (Ref0) specified by the MV candidate and the reconstructed image at the specified position in the second coded reference picture (Ref1) specified by the symmetric MV is derived for the current block, and the symmetric MV is obtained by scaling the MV candidate at a display time interval, and the estimated value is calculated using the value of the obtained difference. The MV candidate that produces the best estimated value among multiple MV candidates and is likely to produce good results can be selected as the final MV.

[0289] Under the assumption of a continuous motion trajectory, the motion vectors (MV0, MV1) of the two reference blocks are specified to be proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, when the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the corresponding two reference pictures are equal to each other, a mirror-symmetric bidirectional motion vector is derived in the first pattern matching.

[0290] (MV Export > FRUC > Template Matching)

[0291] In the second pattern matching (template matching), pattern matching is performed between a block in the reference picture and a template in the current picture (a template is a block adjacent to the current block in the current picture (an adjacent block is, for example, an adjacent block above and / or on the left)). Therefore, in the second pattern matching, a block adjacent to the current block in the current picture is used as a determined region for calculating the above-mentioned candidate estimated value.

[0292] Fig.24 is a conceptual diagram for illustrating an example of pattern matching (template matching) between a template in a current picture and a block in a reference picture. Fig.24As shown, in the second pattern matching, the motion vector of the current block (Cur block) is derived by estimating in the reference picture (Ref0) a block that best matches a block adjacent to the current block in the current picture (Cur Pic). More specifically, it is possible to derive a difference between a reconstructed image in a coding area adjacent to the upper left or adjacent to the left or upper side and a reconstructed image in a corresponding area in the coding reference picture (Ref0) and specified by an MV candidate, calculate an estimated value using the value of the obtained difference, and select an MV candidate that produces the best estimated value among multiple MV candidates as the best MV candidate.

[0293] Such information indicating whether the FRUC mode is applied (e.g., referred to as a FRUC flag) may be signaled at the CU level. In addition, when the FRUC mode is applied (e.g., when the FRUC flag is true), information indicating an applicable pattern matching method (first pattern matching or second pattern matching) may be signaled at the CU level. It should be noted that the signaling of such information does not necessarily need to be performed at the CU level, but may be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0294] (MV Export > Affine Mode)

[0295] Next, the affine mode for deriving a motion vector in units of subblocks based on motion vectors of a plurality of neighboring blocks is described. This mode is also called the affine motion compensation prediction mode.

[0296] Fig.25A : is a conceptual diagram for illustrating an example of deriving a motion vector for each sub-block based on motion vectors of a plurality of adjacent blocks. Fig.25A In the example, the current block includes sixteen 4×4 sub-blocks. Here, the motion vector V0 at the upper left corner control point of the current block is derived based on the motion vectors of the adjacent blocks, and similarly, the motion vector V1 at the upper right corner control point of the current block is derived based on the motion vectors of the adjacent sub-blocks. The two motion vectors v0 and v1 can be projected according to the expression (1A) shown below, and the motion vector (v x , v y ).

[0297] [Mathematical formula 1]

[0298]

[0299] Here, x and y represent the horizontal position and the vertical position of the sub-block, respectively, and w represents the determined weighting coefficient. The determined weighting coefficient may be predetermined.

[0300] Such information indicating the affine mode (e.g., called an affine flag) may be signaled at the CU level. It should be noted that the signaling of the information indicating the affine mode does not necessarily need to be performed at the CU level, and may be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0301] In addition, the affine mode may include several modes for different methods of deriving motion vectors at the upper left control point and the upper right control point. For example, the affine mode includes two modes, namely, the affine inter mode (also called the affine regular inter mode) and the affine merge mode.

[0302] (MV Export > Affine Mode)

[0303] Fig.25B : is a conceptual diagram for illustrating an example of deriving a motion vector for each sub-block in an affine mode using three control points. Fig.25B , the current block includes sixteen 4×4 blocks. Here, the motion vector V0 at the upper left corner control point of the current block is derived based on the motion vector of the adjacent block, and similarly, the motion vector V1 at the upper right corner control point of the current block is derived based on the motion vector of the adjacent block, and the motion vector V2 of the lower left corner control point of the current block is derived based on the motion vector of the adjacent block. The three motion vectors v0, v1, and v2 can be projected according to the expression (1B) shown below, and the motion vector (v x , v y ).

[0304] [Mathematical formula 2]

[0305]

[0306] Here, x and y represent the horizontal position and the vertical position of the center of the sub-block, respectively, w represents the width of the current block, and h represents the height of the current block.

[0307] Affine modes using different numbers of control points (e.g., two and three control points) can be switched and signaled at the CU level. It should be noted that the information indicating the number of control points in the affine mode used at the CU level can be signaled at another level (e.g., sequence level, picture level, slice level, tile level, CTU level, or sub-block level).

[0308] In addition, such an affine mode in which three control points are used may include different methods for deriving motion vectors at the upper left, upper right and lower left control points. For example, the affine mode includes two modes, which are an affine inter mode (also called an affine regular inter mode) and an affine merge mode.

[0309] (MV Export > Affine Merge Mode)

[0310] Fig.26A , Fig.26B and Fig.26C is a conceptual diagram for illustrating the affine merge mode.

[0311] like Fig.26A As shown, in the affine merge mode, for example, based on a plurality of motion vectors corresponding to blocks encoded according to the affine mode among the coded blocks A (left), B (upper), C (upper right), D (lower left), and E (upper left) adjacent to the current block, a motion vector predictor at a corresponding control point of the current block is calculated. More specifically, the coded blocks A (left), B (upper), C (upper right), D (lower left), and E (upper left) are checked in the order listed, and the first valid block encoded according to the affine mode is identified. The motion vector predictor at the control point of the current block is calculated based on the plurality of motion vectors corresponding to the identified blocks.

[0312] For example, Fig.26B As shown, when block A adjacent to the left side of the current block has been encoded according to the affine mode using two control points, motion vectors v3 and v4 projected at the upper left corner position and the upper right corner position of the encoded block including block A are derived. Then, a motion vector predictor v0 at the upper left control point of the current block and a motion vector predictor v1 at the upper right control point of the current block are calculated based on the derived motion vectors v3 and v4.

[0313] For example, Fig.26C As shown, when block A adjacent to the left side of the current block has been encoded according to the affine mode using three control points, motion vectors v3, v4, and v5 projected at the upper left corner position, the upper right corner position, and the lower left corner position of the encoded block including block A are derived. Then, a motion vector predictor v0 at the upper left control point of the current block, a motion vector predictor v1 at the upper right control point of the current block, and a motion vector predictor v2 at the lower left control point of the current block are calculated based on the derived motion vectors v3, v4, and v5.

[0314] It should be noted that this method for deriving a motion vector predictor can be used in the later described Fig.29 In step Sj_1, the motion vector predictor of the corresponding control point of the current block is derived.

[0315] Fig. 27 is a flowchart illustrating an example of affine merge mode.

[0316] In the affine merge mode shown, first, the inter-frame predictor 126 derives the MV predictor of the corresponding control point of the current block (step Sk_1). Fig.25AAs shown, the control points are the upper left corner point of the current block and the upper right corner point of the current block, or as Fig.25B As shown, these are the upper left corner point of the current block, the upper right corner point of the current block, and the lower left corner point of the current block.

[0317] In other words, Fig.26A As shown, the inter-frame predictor 126 examines the encoded blocks A (left), B (top), C (top right), D (bottom left), and E (top left) in the order listed and identifies the first valid block encoded according to the affine mode.

[0318] When block A is identified and block A has two control points, such as Fig.26B As shown, the inter-frame predictor 126 calculates the motion vector v0 at the upper left corner control point of the current block and the motion vector v1 at the upper right corner control point of the current block based on the motion vectors v3 and v4 at the upper left corners and the upper right corners of the encoded blocks including block A. For example, the inter-frame predictor 126 calculates the motion vector v0 at the upper left corner control point of the current block and the motion vector v1 at the upper right corner control point of the current block by projecting the motion vectors v3 and v4 at the upper left corners and the upper right corners of the encoded blocks onto the current block.

[0319] Alternatively, when block A is identified and has three control points, as Fig.26C As shown, the inter-frame predictor 126 calculates the motion vector v0 at the upper left control point of the current block, the motion vector v1 at the upper right control point of the current block, and the motion vector v2 at the lower left control point of the current block based on the motion vectors v3, v4, and v5 at the upper left, upper right, and lower left corners of the coding block including block A. For example, the inter-frame predictor 126 calculates the motion vector v0 at the upper left control point of the current block, the motion vector v1 at the upper right control point of the current block, and the motion vector v2 at the lower left control point of the current block by projecting the motion vectors v3, v4, and v5 at the upper left, upper right, and lower left corners of the coding block onto the current block.

[0320] Next, the inter-frame predictor 126 performs motion compensation on each of the multiple sub-blocks included in the current block. In other words, the inter-frame predictor 126 calculates the motion vector of each sub-block in the multiple sub-blocks as an affine MV (step Sk_2) by using (i) two motion vector predictors v0 and v1 and the above expression (1A) or (ii) three motion vector predictors v0, v1 and v2 and the above expression (1B). Then, the inter-frame predictor 126 uses these affine MVs and the encoded reference picture to perform motion compensation of the sub-blocks (step Sk_3). As a result, motion compensation of the current block is performed to generate a predicted image of the current block.

[0321] (MV Export > Affine Inter-frame Mode)

[0322] Fig.28A is a conceptual diagram for illustrating an affine inter mode using two control points.

[0323] In affine inter mode, such as Fig.28A As shown, a motion vector selected from the motion vectors of the coded blocks A, B, and C adjacent to the current block is used as the motion vector predictor v0 at the upper left corner control point of the current block. Similarly, a motion vector selected from the motion vectors of the coded blocks D and E adjacent to the current block is used as the motion vector predictor v1 at the upper right corner control point of the current block.

[0324] Fig.28B is a conceptual diagram for illustrating an affine inter mode using three control points.

[0325] In affine inter mode, such as Fig.28B As shown, a motion vector selected from the motion vectors of the coded blocks A, B, and C adjacent to the current block is used as the motion vector predictor v0 at the upper left corner control point of the current block. Similarly, a motion vector selected from the motion vectors of the coded blocks D and E adjacent to the current block is used as the motion vector predictor v1 at the upper right corner control point of the current block. Furthermore, a motion vector selected from the motion vectors of the coded blocks F and G adjacent to the current block is used as the motion vector predictor v2 at the lower left corner control point of the current block.

[0326] Fig.29 is a flowchart showing an example of affine inter mode.

[0327] In the affine inter mode shown, first, the inter predictor 126 derives the MV predictors (v0, v1) or (v0, v1, v2) of the corresponding two or three control points of the current block (step Sj_1). Fig.25A As shown, the control points are the upper left corner point of the current block and the upper right corner point of the current block, or as Fig.25B As shown, these are the upper right corner point of the current block and the lower left corner point of the current block.

[0328] In other words, the inter-frame predictor 126 generates Fig.28A or Fig.28B The inter-frame predictor 126 selects the motion vector of any block from the coding blocks near the corresponding control point of the current block shown in , to derive the motion vector predictor (v0, v1) or (v0, v1, v2) of the corresponding two or three control points of the current block. At this time, the inter-frame predictor 126 encodes the motion vector predictor selection information for identifying the selected two motion vectors in the stream.

[0329] For example, the inter-frame predictor 126 may use cost estimation or the like to determine a block from which a motion vector as a motion vector predictor at a control point is selected among coding blocks adjacent to the current block, and the inter-frame predictor 126 may describe a flag in a bitstream indicating which motion vector predictor has been selected.

[0330] Next, the inter-frame predictor 126 performs motion estimation (steps Sj_3 and Sj_4) while updating the motion vector predictor selected or derived in step Sj_1 (step Sj_2). In other words, the inter-frame predictor 126 uses the above expression (1A) or expression (1B) to calculate the motion vector of each sub-block corresponding to the updated motion vector predictor as an affine MV (step Sj_3). Then, the inter-frame predictor 126 uses these affine MVs and the encoded reference picture to perform motion compensation of the sub-blocks (step Sj_4). As a result, for example, the inter-frame predictor 126 determines the motion vector predictor that produces the minimum cost as the motion vector at the control point in the motion estimation loop (step Sj_5). At this time, the inter-frame predictor 126 also encodes the difference between the determined MV and the motion vector predictor as an MV difference in the stream.

[0331] Finally, the inter-frame predictor 126 generates a predicted image of the current block by performing motion compensation of the current block using the determined MV and the encoded reference picture (step Sj_6).

[0332] (MV Export > Affine Inter-frame Mode)

[0333] When affine modes in which different numbers of control points (eg, two and three control points) are used can be switched and signaled at the CU level, the number of control points in the coding block and the number of control points in the current block may be different from each other. Fig. 30A and Fig. 30B is a conceptual diagram for illustrating a method of deriving a motion vector predictor at a control point when the number of control points in a coding block and the number of control points in a current block are different from each other.

[0334] For example, Fig. 30A As shown, when the current block has three control points at the upper left corner, the upper right corner, and the lower left corner, and the block A adjacent to the left side of the current block has been encoded according to the affine mode in which two control points are used, motion vectors v3 and v4 projected at the upper left corner position and the upper right corner position in the encoded block including the block A are derived. Then, the motion vector predictor v0 at the upper left corner control point of the current block and the motion vector predictor v1 at the upper right corner control point of the current block are calculated based on the derived motion vectors v3 and v4. In addition, the motion vector predictor v2 at the lower left corner control point is calculated based on the derived motion vectors v0 and v1.

[0335] For example, Fig. 30B As shown, when the current block has two control points at the upper left and upper right corners, and the block A adjacent to the left side of the current block has been encoded according to the affine mode using three control points, motion vectors v3, v4, and v5 projected at the upper left corner position, the upper right corner position, and the lower left corner position in the encoded block including the block A are derived. Then, the motion vector predictor v0 at the upper left control point of the current block and the motion vector predictor v1 at the upper right control point of the current block are calculated based on the derived motion vectors v3, v4, and v5.

[0336] It should be noted that this method for deriving a motion vector predictor can be used in Fig.29 In step Sj_1, the motion vector predictor of the corresponding control point of the current block is derived.

[0337] (MV Export > DMVR)

[0338] Fig.31A is a flow chart showing the relationship between merge mode and DMVR.

[0339] The inter-frame predictor 126 derives the motion vector of the current block according to the merge mode (step S1_1). Next, the inter-frame predictor 126 determines whether to perform estimation of the motion vector, i.e., motion estimation (step S1_2). Here, when it is determined not to perform motion estimation (No in step S1_2), the inter-frame predictor 126 determines the motion vector derived in step S1_1 as the final motion vector of the current block (step S1_4). In other words, in this case, the motion vector of the current block is determined according to the merge mode.

[0340] When it is determined in step S1_1 that motion estimation is performed (yes in step S1_2), the inter-frame predictor 126 derives the final motion vector of the current block by estimating the surrounding area of ​​the reference picture specified by the motion vector derived in step S1_1 (step S1_3). In other words, in this case, the motion vector of the current block is determined according to DMVR.

[0341] Fig.31B is a conceptual diagram for illustrating an example of a DMVR process for determining a MV.

[0342] First, the best MVP that has been set as the current block (e.g., in merge mode) is determined as the MV candidate. Reference pixels are identified from a first reference picture (L0), which is a coded picture in the LO direction according to the MV candidate (LO). Similarly, reference pixels are identified from a second reference picture (L1), which is a coded picture in the L1 direction according to the MV candidate (L1). A template is generated by calculating the average value of these reference pixels.

[0343] Next, each surrounding area of ​​the MV candidates of the first reference picture (L0) and the second reference picture (L1) is estimated, and the MV that generates the minimum cost is determined as the final MV. Note that the cost value can be calculated, for example, using the difference between each pixel value in the template and a corresponding one of the pixel values ​​in the estimated area, the value of the MV candidate, etc.

[0344] It is noted that the processes, configurations, and operations described herein are generally substantially common between an encoder and a decoder to be described later.

[0345] It is not always necessary to perform exactly the same example process described here. Any process for enabling the derivation of a final MV by estimating in the surrounding area of ​​the MV candidate may be used.

[0346] (Motion Compensation > BIO / OBMC)

[0347] Motion compensation involves a mode for generating a predicted image and correcting the predicted image. The mode is, for example, BIO and OBMC which will be described later.

[0348] Fig.32 is a flowchart showing an example of generating a predicted image.

[0349] The inter-frame predictor 126 generates a predicted image (step Sm_1 ), and corrects the predicted image, for example, according to any of the modes described above (step Sm_2 ).

[0350] Fig.33 is a flowchart showing another example of generating a predicted image.

[0351] The inter-frame predictor 126 determines the motion vector of the current block (step Sn_1). Next, the inter-frame predictor 126 generates a predicted image (step Sn_2), and determines whether to perform a correction process (step Sn_3). Here, when it is determined to perform the correction process (Yes in step Sn_3), the inter-frame predictor 126 generates a final predicted image by correcting the predicted image (step Sn_4). When it is determined not to perform the correction process (No in step Sn_3), the inter-frame predictor 126 outputs the predicted image as the final predicted image without correcting the predicted image (step Sn_5).

[0352] In addition, motion compensation refers to a mode for correcting the brightness of a predicted image when generating a predicted image. This mode is, for example, LIC which will be described later.

[0353] Fig.34 is a flowchart showing another example of generating a predicted image.

[0354] The inter-frame predictor 126 derives a motion vector of the current block (step So_1). Next, the inter-frame predictor 126 determines whether to perform a brightness correction process (step So_2). Here, when it is determined that the brightness correction process is performed (Yes in step So_2), the inter-frame predictor 126 generates a predicted image while performing the brightness correction process (step So_3). In other words, the predicted image is generated using LIC. When it is determined that the brightness correction process is not performed (No in step So_2), the inter-frame predictor 126 generates a predicted image by performing conventional motion compensation without performing the brightness correction process (step So_4).

[0355] (Motion Compensation > OBMC)

[0356] It should be noted that in addition to the motion information of the current block obtained from motion estimation, the motion information of the adjacent blocks can also be used to generate the inter-frame prediction signal. More specifically, the inter-frame prediction signal can be generated in units of sub-blocks in the current block by performing weighted addition on the prediction signal based on the motion information obtained from motion estimation (in the reference picture) and the prediction signal based on the motion information of the adjacent blocks (in the current picture). This inter-frame prediction (motion compensation) is also called overlapped block motion compensation (OBMC).

[0357] In the OBMC mode, information indicating a sub-block size for OBMC (e.g., referred to as an OBMC block size) may be signaled at the sequence level. In addition, information indicating whether the OBMC mode is applied (e.g., referred to as an OBMC flag) may be signaled at the CU level. It should be noted that the signaling of such information does not necessarily need to be performed at the sequence level and the CU level, but may be performed at another level (e.g., a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0358] An example of the OBMC mode will be described in more detail. Fig.35 and Fig.36 is a flowchart and a conceptual diagram for showing an outline of a predicted image correction process performed by the OBMC process.

[0359] First, if Fig.36 As shown in , a predicted image (Pred) is obtained by conventional motion compensation using a motion vector (MV) assigned to a processing target (current) block. Fig.36 , the arrow “MV” points to the reference picture and indicates what the current block of the current picture refers to in order to obtain the predicted image.

[0360] Next, a predicted image (Pred_L) is obtained by applying a motion vector (MV_L) that has been derived for a coding block adjacent to the left side of the current block to the current block (reusing the motion vector for the current block). The motion vector (MV_L) is indicated by an arrow "MV_L", which indicates a reference picture from the current block. A first correction of the predicted image is performed by overlapping the two predicted images Pred and Pred_L. This provides the effect of blending the boundaries between adjacent blocks.

[0361] Likewise, a predicted image (Pred_U) is obtained by applying a motion vector (MV_U) that has been derived for a coding block adjacent to the current block above to the current block (reusing the motion vector for the current block). The motion vector (MV_U) is indicated by an arrow "MV_U", which indicates a reference picture from the current block. A second correction of the predicted image is performed by overlapping the predicted image Pred_U with a predicted image (e.g., Pred and Pred_L) on which the first correction has been performed. This provides an effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is a predicted image in which the boundaries between adjacent blocks have been blended (smoothed), and is therefore the final predicted image of the current block.

[0362] Although the above example is a two-path correction method using left and upper neighboring blocks, it is noted that the correction method may be a three-path or more correction method also using right and / or lower neighboring blocks.

[0363] Note that the area where such overlapping is performed may be only a portion of the area near the block boundary, rather than the pixel area of ​​the entire block.

[0364] Note that the prediction image correction process according to OBMC for obtaining one prediction image Pred from one reference picture by overlapping additional prediction images Pred_L and Pred_U has been described above. However, when the prediction image is corrected based on a plurality of reference images, a similar process may be applied to each of the plurality of reference pictures. In this case, after obtaining a corrected prediction image from the corresponding reference picture by performing OBMC image correction based on the plurality of reference pictures, the obtained corrected prediction images are further overlapped to obtain a final prediction image.

[0365] It should be noted that in OBMC, the unit of the current block may be the unit of the prediction block, or the unit of a sub-block obtained by further dividing the prediction block.

[0366] An example of a method for determining whether to apply the OBMC process is a method for using obmc_flag, which is a signal indicating whether the OBMC process is applied. As a specific example, the encoder determines whether the current block belongs to an area with complex motion. When the block belongs to an area with complex motion, the encoder sets the value of obmc_flag to "1", and when encoding, the OBMC process is applied; and when the block does not belong to an area with complex motion, the value of obmc_flag is set to "0", and the block is encoded without applying the OBMC process. The decoder switches between application and non-application of the OBMC process by decoding the obmc_flag written in the stream (e.g., a compressed sequence) and decoding the block by switching between application and non-application of the OBMC process according to the flag value.

[0367] In the above example, the inter-frame predictor 126 generates one rectangular prediction image for the rectangular current block. However, the inter-frame predictor 126 may generate a plurality of prediction images, each of which has a shape different from the rectangular shape of the rectangular current block, and may combine the plurality of prediction images to generate a final rectangular prediction image. The shape different from the rectangle may be, for example, a triangle.

[0368] Fig.37 is a conceptual diagram for illustrating the generation of two triangular prediction images.

[0369] The inter-frame predictor 126 generates a triangular predicted image by performing motion compensation on the first partition having a triangular shape in the current block using the first MV of the first partition to generate a triangular predicted image. Similarly, the inter-frame predictor 126 generates a triangular predicted image by performing motion compensation on the second partition having a triangular shape in the current block using the second MV of the second partition to generate a triangular predicted image. Then, the inter-frame predictor 126 generates a predicted image having the same rectangular shape as that of the current block by combining these predicted images.

[0370] It should be noted that although Fig.37 In the example shown, the first partition and the second partition are triangular, but the first partition and the second partition may be trapezoidal or other shapes different from each other. Fig.37 In the example shown, the current block includes two partitions, but the current block may include three or more partitions.

[0371] In addition, the first partition and the second partition may overlap each other. In other words, the first partition and the second partition may include the same pixel area. In this case, the predicted image in the first partition and the predicted image in the second partition may be used to generate the predicted image of the current block.

[0372] In addition, although an example in which a predicted image is generated for each of two partitions using inter prediction, a predicted image may be generated for at least one partition using intra prediction.

[0373] (Motion Compensation > BIO)

[0374] Next, a method for deriving motion vectors is described. First, a mode for deriving motion vectors based on a model assuming uniform linear motion will be described. This mode is also called the Bidirectional Optical Flow (BIO) mode.

[0375] Fig.38 This is a conceptual diagram for illustrating a model assuming uniform linear motion. Fig.38 In , (vx, vy) indicates a velocity vector, and τ0 and τ1 indicate the temporal distance between the current picture (Cur Pic) and two reference pictures (Ref0, Ref1). (MVx0, MVy0) indicates a motion vector corresponding to the reference picture Ref0, and (MVx1, MVy1) indicates a motion vector corresponding to the reference picture Ref1.

[0376] Here, assuming that the velocity vector (v x ,v y ) exhibits uniform linear motion, (MVx0, MVy0) and (MVx1, MVy1) are expressed as (v x τ0,v y τ0) and (-v x τ1,-v y τ1), and the following optical flow equation (2) can be used.

[0377] [Mathematical formula 3]

[0378]

[0379] Here, I(k) represents the motion compensated luminance value of reference picture k (k=0, 1). The optical flow equation shows that the sum of the following is equal to zero: (i) the temporal derivative of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image. The motion vector of each block obtained from, for example, the merge list can be corrected in units of pixels based on a combination of the optical flow equation and Hermitian interpolation.

[0380] It should be noted that methods other than deriving motion vectors based on a model assuming uniform linear motion may be used to derive motion vectors at the decoder side. For example, motion vectors may be derived in units of sub-blocks based on motion vectors of neighboring blocks.

[0381] (Motion Compensation > LIC)

[0382] Next, an example of a mode in which a predicted image (prediction) is generated by using a local illumination compensation (LIC) process will be described.

[0383] Fig.39 is a conceptual diagram for illustrating one example of a predicted image generation method using a brightness correction process performed through the LIC process.

[0384] First, MV is derived from the encoded reference picture, and the reference image corresponding to the current block is obtained.

[0385] Next, information indicating how the brightness value between the reference picture and the current picture changes is extracted for the current block. This extraction is performed based on the brightness pixel values ​​for the encoded left adjacent reference area (surrounding reference area) and the encoded upper adjacent reference area (surrounding reference area) and the brightness pixel values ​​at the corresponding positions in the reference picture specified by the derived MV. The brightness correction parameter is calculated by using the information indicating how the brightness value changes.

[0386] A prediction image for the current block is generated by performing a brightness correction process in which a brightness correction parameter is applied to a reference image in a reference picture specified by the MV.

[0387] It should be noted that Fig.39 The shape of the surrounding reference area shown in FIG. 1 is only one example; the surrounding reference area may have a different shape.

[0388] In addition, although the process of generating a predicted image from a single reference picture has been described here, the case of generating a predicted image from a plurality of reference pictures can be described in the same manner. The predicted image can be generated after performing a brightness correction process on the reference image obtained from the reference picture in the same manner as described above.

[0389] One example of a method for determining whether to apply the LIC process is a method for using lic_flag, which is a signal indicating whether the LIC process is applied. As a specific example, the encoder determines whether the current block belongs to an area with brightness changes. When the block belongs to an area with brightness changes, the encoder sets the value of lic_flag to "1" and applies the LIC process when encoding, and when the block does not belong to an area with brightness changes, the value of lic_flag is set to "0" and the current block is encoded without applying the LIC process. The decoder can decode the lic_flag written in the stream by switching between application and non-application of the LIC process according to the flag value, and decode the current block.

[0390] One example of a different method of determining whether to apply the LIC process is a determination method based on whether the LIC process is performed on the surrounding blocks. In a specific example, when the merge mode is used on the current block, it is determined whether the LIC process is applied in the encoding of the surrounding coding blocks selected when deriving the MV in the merge mode process. According to the result, encoding is performed by switching between application and non-application of the LIC process. Note that also in this example, the same process is applied in the process on the decoder side.

[0391] The following describes in detail the reference Fig.39 An embodiment of a brightness correction (LIC) process is described.

[0392] First, the inter predictor 126 derives a motion vector for obtaining a reference image corresponding to a current block to be encoded from a reference picture as an encoded picture.

[0393] Next, the inter-frame predictor 126 uses the luminance pixel value of the encoded surrounding reference area adjacent to the left or above the current block and the luminance value of the corresponding position in the reference picture specified by the motion vector, extracts information indicating how the luminance value of the reference picture becomes the luminance value of the current picture, and calculates the luminance correction parameter. For example, it is assumed that the luminance pixel value of a given pixel in the surrounding reference area in the current picture is p0, and it is assumed that the luminance pixel value of the pixel corresponding to the given pixel in the surrounding reference area in the reference picture is p1. The inter-frame predictor 126 calculates coefficients A and B for optimizing A×p1+B=p0 as luminance correction parameters for multiple pixels in the surrounding reference area.

[0394] Next, the inter-frame predictor 126 performs a brightness correction process using the brightness correction parameters of the reference image in the reference picture specified by the motion vector to generate a predicted image of the current block. For example, assume that the brightness pixel value in the reference image is p2, and the brightness pixel value of the predicted image after brightness correction is p3. After performing the brightness correction process by calculating A×p2+B=p3 for each pixel in the reference image, the inter-frame predictor 126 generates a predicted image.

[0395] It should be noted that Fig.39 The shape of the surrounding reference area shown in FIG. 1 is an example; other shapes than the shape of the surrounding reference area may be used. Fig.39 A portion of the surrounding reference area shown in . For example, an area having a determined number of pixels extracted from each of the upper adjacent pixels and the left adjacent pixels can be used as the surrounding reference area. The determined number of pixels can be predetermined.

[0396] In addition, the surrounding reference area is not limited to the area adjacent to the current block, and may be an area not adjacent to the current block. Fig.39 In the example shown, the surrounding reference area in the reference picture is an area in the current picture specified by a motion vector from the surrounding reference area in the current picture. However, an area specified by another motion vector is also possible. For example, another motion vector may be a motion vector in the surrounding reference area in the current picture.

[0397] Although operations performed by encoder 100 have been described herein, it is noted that decoder 200 generally performs similar operations.

[0398] It should be noted that the LIC process can be applied not only to luminance but also to chrominance. At this time, correction parameters can be derived individually for each of Y, Cb and Cr, or common correction parameters can be used for any one of Y, Cb and Cr.

[0399] In addition, the LIC process may be applied in units of subblocks. For example, correction parameters may be derived using surrounding reference areas in a reference subblock specified by the MV of the current subblock and surrounding reference areas in the current subblock in the reference picture.

[0400] (Predictive Controller)

[0401] The inter predictor 128 selects one of the intra prediction signal (the signal output from the intra predictor 124) and the inter prediction signal (the signal output from the inter predictor 126), and outputs the selected signal to the subtractor 104 and the adder 116 as a prediction signal.

[0402] like Figure 1 As shown, in various encoder examples, the prediction controller 128 can output prediction parameters that are input to the entropy encoder 110. The entropy encoder 110 can generate a coded bit stream (or sequence) based on the prediction parameters input from the prediction controller and the quantization coefficients input from the quantizer 108. The prediction parameters can be used in the decoder. The decoder can receive and decode the coded bit stream and perform the same process as the prediction process performed by the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128. The prediction parameters may include (i) a selection prediction signal (e.g., a motion vector, a prediction type, or a prediction mode used by the intra-frame predictor 124 or the inter-frame predictor 126), or (ii) an optional index, flag, or value based on the prediction process performed in each of the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128 or indicating the prediction process.

[0403] (Encoder installation example)

[0404] Fig.40 1 is a block diagram showing an example of an installation of the encoder 100. The encoder 100 includes a processor a1 and a memory a2. For example, Figure 1 The various components of the encoder 100 shown may be mounted on Fig.40 On the processor a1 and memory a2 shown in .

[0405] Processor a1 is a circuit that performs information processing and can be accessed by memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit that encodes video. Processor a1 may be a processor such as a CPU. In addition, processor a1 may be an aggregation of multiple electronic circuits. In addition, for example, processor a1 may act as Figure 1 etc. The roles of two or more constituent elements among the multiple constituent elements of the encoder 100 shown in the figure.

[0406] The memory a2 is a dedicated or general-purpose memory for storing information used by the processor a1 to encode the video. The memory a2 may be an electronic circuit and may be connected to the processor a1. In addition, the memory a2 may be included in the processor a1. In addition, the memory a2 may be an aggregation of multiple electronic circuits. In addition, the memory a2 may be a magnetic disk, an optical disk, etc., or may be represented as a storage device, a recording medium, etc. In addition, the memory a2 may be a non-volatile memory or a volatile memory.

[0407] For example, the memory a2 may store a video to be encoded or a bit stream corresponding to the encoded video. In addition, the memory a2 may store a program for causing the processor a1 to encode the video.

[0408] In addition, for example, memory a2 can act as Figure 1 The memory a2 may serve as two or more components for storing information among the multiple components of the encoder 100 shown in FIG. Figure 1 The roles of the block memory 118 and the frame memory 122 are shown. More specifically, the memory a2 can store reconstructed blocks, reconstructed pictures, etc.

[0409] It should be noted that in the encoder 100, it is not necessary to implement Figure 1 All multiple constituent elements indicated in the above may not be included, and all the above processes may not be performed. Figure 1 A part of the constituent elements indicated in the above-mentioned figure may be included in another device, or a part of the above-mentioned process may be performed by another device.

[0410] (Decoder)

[0411] Next, a decoder capable of decoding, for example, an encoded signal (encoded bit stream) output from the above-described encoder 100 will be described. Fig.41 2 is a block diagram showing a functional configuration of a decoder 200 according to an embodiment. The decoder 200 is a video decoder that decodes a video in units of blocks.

[0412] like Fig.41 As shown, the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an intra-frame predictor 216, an inter-frame predictor 218 and a prediction controller 220.

[0413] The decoder 200 is implemented as, for example, a general purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor functions as an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a loop filter 212, an intra-frame predictor 216, an inter-frame predictor 218, and a prediction controller 220. Alternatively, the decoder 200 may be implemented as one or more dedicated electronic circuits corresponding to the entropy decoder 202, the inverse quantizer 204, the inverse transformer 206, the adder 208, the loop filter 212, the intra-frame predictor 216, the inter-frame predictor 218, and the prediction controller 220.

[0414] Hereinafter, the overall flow of a process performed by the decoder 200 is described, and then each constituent element included in the decoder 200 will be described.

[0415] (Overall flow of the decoding process)

[0416] Fig.42 is a flowchart showing one example of the entire decoding process performed by the decoder 200.

[0417] First, the entropy decoder 202 of the decoder 200 identifies a partition pattern of a block having a fixed size (e.g., 128×128 pixels) (step Sp_1). The partition pattern is a partition pattern selected by the encoder 100. Then, the decoder 200 performs the process of steps Sp_2 to Sp_6 for each of a plurality of blocks of the partition pattern.

[0418] In other words, the entropy decoder 202 decodes (specifically, entropy decodes) the encoded quantization coefficients and prediction parameters of the current block to be decoded (also referred to as the current block) (step Sp_2).

[0419] Next, the inverse quantizer 204 performs inverse quantization on the plurality of quantized coefficients, and the inverse transformer 206 performs inverse transformation on the result to restore a plurality of prediction residuals (ie, difference blocks) (step Sp_3).

[0420] Next, the prediction processor including all or part of the intra predictor 216, the inter predictor 218, and the prediction controller 220 generates a prediction signal (also referred to as a prediction block) for the current block (step Sp_4).

[0421] Next, the adder 208 adds the prediction block to the difference block to generate a reconstructed image of the current block (also referred to as a decoded image block) (step Sp_5).

[0422] When the reconstructed image is generated, the loop filter 212 performs filtering on the reconstructed image (step Sp_6).

[0423] The decoder 200 then determines whether decoding of the entire picture has been completed (step Sp_7). When it is determined that decoding has not been completed (No in step Sp_7), the decoder 200 repeatedly performs the process starting from step Sp_1.

[0424] As shown in the figure, the processes of steps Sp_1 to Sp_7 are sequentially performed by the decoder 200. Alternatively, two or more processes may be performed in parallel, the processing order of two or more processes may be modified, and so on.

[0425] (Entropy Decoder)

[0426] The entropy decoder 202 performs entropy decoding on the decoded bit stream. More specifically, for example, the entropy decoder 202 arithmetically decodes the coded bit stream into a binary signal. Then, the entropy decoder 202 de-binarizes the binary signal. In this way, the entropy decoder 202 outputs the quantization coefficients of each block to the inverse quantizer 204. The entropy decoder 202 can be included in the coded bit stream (see Figure 1 ) is output to the intra predictor 216, the inter predictor 218 and the prediction controller 220. The intra predictor 216, the inter predictor 218 and the prediction controller 220 in the embodiment are capable of performing the same prediction process as the prediction process performed by the intra predictor 124, the inter predictor 126 and the prediction controller 128 on the encoder side.

[0427] (Inverse Quantizer)

[0428] The inverse quantizer 204 inversely quantizes the quantized coefficients of the block to be decoded (hereinafter referred to as the current block), which are input from the entropy decoder 202. More specifically, the inverse quantizer 204 inversely quantizes the quantized coefficients of the current block based on the quantization parameters corresponding to the quantized coefficients. The inverse quantizer 204 then outputs the inversely quantized transform coefficients of the current block to the inverse transformer 206.

[0429] (Inverter)

[0430] The inverse transformer 206 restores the prediction error by inversely transforming the transform coefficients (which are input from the inverse quantizer 204 ).

[0431] For example, when information parsed from the encoded bitstream indicates that EMT or AMT is to be applied (eg, when the AMT flag is true), the inverse transformer 206 inversely transforms transform coefficients of the current block based on information indicating the parsed transform type.

[0432] Furthermore, for example, when the information parsed from the encoded bit stream indicates that NSST is to be applied, the inverse transformer 206 applies a secondary inverse transform to the transform coefficients.

[0433] (Adder)

[0434] The adder 208 reconstructs the current block by adding the prediction error (which is the input from the inverse transformer 206) and the prediction sample (which is the input from the prediction controller 220). The adder 208 then outputs the reconstructed block to the block memory 210 and the loop filter 212.

[0435] (Block Storage)

[0436] The block memory 210 is a storage device for storing a block in a picture to be decoded (hereinafter referred to as a current picture) and to be referenced in intra prediction. More specifically, the block memory 210 stores the reconstructed block output from the adder 208.

[0437] (Loop Filter)

[0438] The loop filter 212 applies a loop filter to the block reconstructed by the adder 208 and outputs the filtered reconstructed block to the frame memory 214, a display device, or the like.

[0439] When the information indicating on or off of the ALF parsed from the encoded bitstream indicates that the ALF is on, a filter is selected from among a plurality of filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed block.

[0440] (Frame Memory)

[0441] The frame memory 214 is, for example, a storage device for storing a reference picture used in inter-frame prediction, and is also referred to as a frame buffer. More specifically, the frame memory 214 stores the reconstructed block filtered by the loop filter 212.

[0442] (Prediction processor (intra-frame predictor, inter-frame predictor, prediction controller))

[0443] Fig.43 is a flowchart showing one example of a process performed by the prediction processor of the decoder 200. Note that the prediction processor includes all or part of the following constituent elements: an intra predictor 216; an inter predictor 218; and a prediction controller 220.

[0444] The prediction processor generates a prediction image of the current block (step Sq_1). The prediction image is also called a prediction signal or a prediction block. Note that the prediction signal is, for example, an intra-frame prediction signal or an inter-frame prediction signal. Specifically, the prediction processor uses a reconstructed image obtained by generating a prediction block, generating a difference block, generating a coefficient block, restoring a difference block, and generating a decoded image block to generate a prediction image of the current block.

[0445] The reconstructed image may be, for example, an image in a reference picture, or an image of a decoded block in a current picture (which is a picture including the current block). The decoded block in the current picture may be, for example, a neighboring block of the current block.

[0446] Fig.44 is a flow chart illustrating another example of a process performed by the prediction processor of the decoder 200 .

[0447] The prediction processor determines a method or mode for generating a prediction image (step Sr_1). For example, the method or mode may be determined based on prediction parameters or the like.

[0448] When the first method is determined as the mode for generating a predicted image, the prediction processor generates a predicted image according to the first method (step Sr_2a). When the second method is determined as the mode for generating a predicted image, the prediction processor generates a predicted image according to the second method (step Sr_2b). When the third method is determined as the mode for generating a predicted image, the prediction processor generates a predicted image according to the third method (step Sr_2c).

[0449] The first method, the second method, and the third method may be different methods for generating a predicted image. Each of the first to third methods may be an inter-frame prediction method, an intra-frame prediction method, or another prediction method. The above-mentioned reconstructed image may be used in these prediction methods.

[0450] (Intra Predictor)

[0451] The intra predictor 216 generates a prediction signal (intra prediction signal) by performing intra prediction with reference to one or more blocks in the current picture stored in the block memory 210 based on the intra prediction mode parsed from the encoded bit stream. More specifically, the intra predictor 216 generates the intra prediction signal by performing intra prediction with reference to samples (e.g., luminance and / or chrominance values) of one or more blocks adjacent to the current block, and then outputs the intra prediction signal to the prediction controller 220.

[0452] It should be noted that when the intra prediction mode in which the luma block is referenced in the intra prediction of the chroma block is selected, the intra predictor 216 may predict the chroma component of the current block based on the luma component of the current block.

[0453] Furthermore, when information parsed from the encoded bitstream indicates that PDPC is to be applied, the intra predictor 216 corrects the intra-predicted pixel value based on the horizontal / vertical reference pixel gradients.

[0454] (Inter-frame predictor)

[0455] The inter-frame predictor 218 predicts the current block by referring to the reference picture stored in the frame memory 214. Inter-frame prediction is performed in units of the current block or a sub-block (e.g., a 4×4 block) in the current block. For example, the inter-frame predictor 218 generates an inter-frame prediction signal of the current block or sub-block by performing motion compensation using motion information (e.g., a motion vector) parsed from the encoded bit stream (e.g., a prediction parameter output from the entropy decoder 202), and outputs the inter-frame prediction signal to the prediction controller 220.

[0456] Note that when information parsed from the encoded bitstream indicates that the OBMC mode is to be applied, the inter predictor 218 generates an inter prediction signal using motion information of neighboring blocks in addition to motion information of the current block acquired from motion estimation.

[0457] In addition, when the information parsed from the encoded bit stream indicates that the FRUC mode is to be applied, the inter-frame predictor 218 derives motion information by performing motion estimation according to the pattern matching method (bilateral matching or template matching) parsed from the encoded bit stream. Then, the inter-frame predictor 218 performs motion compensation (prediction) using the derived motion information.

[0458] In addition, when the BIO mode is to be applied, the inter-frame predictor 218 derives a motion vector based on a model assuming uniform linear motion. In addition, when the information parsed from the encoded bitstream indicates that the affine motion compensation prediction mode is to be applied, the inter-frame predictor 218 derives a motion vector for each sub-block based on the motion vectors of the neighboring blocks.

[0459] (MV Export > Regular Interframe Mode)

[0460] When the information parsed from the encoded bit stream indicates that the normal inter mode is to be applied, the inter predictor 218 derives an MV based on the information parsed from the encoded bit stream, and performs motion compensation (prediction) using the MV.

[0461] Fig.45 is a flowchart illustrating an example of inter prediction in a normal inter mode in the decoder 200 .

[0462] The inter-frame predictor 218 of the decoder 200 performs motion compensation for each block. The inter-frame predictor 218 obtains multiple MV candidates for the current block based on information such as MVs of multiple decoded blocks temporally or spatially surrounding the current block (step Ss_1). In other words, the inter-frame predictor 218 generates a MV candidate list.

[0463] Next, the inter-frame predictor 218 extracts N (2 or more integers) MV candidates as motion vector predictor candidates (also referred to as MV predictor candidates) from the multiple MV candidates obtained in step Ss_1 according to the determined priority order (step Ss_2). It should be noted that the priority order can be predetermined for each of the N MV predictor candidates.

[0464] Next, the inter-frame predictor 218 decodes the motion vector predictor selection information from the input stream (i.e., the encoded bit stream), and uses the decoded motion vector predictor selection information to select an MV predictor candidate from N MV predictor candidates as the motion vector (also called MV predictor) of the current block (step Ss_3).

[0465] Next, the inter predictor 218 decodes the MV difference from the input stream, and derives the MV of the current block by adding the difference value which is the decoded MV difference to the selected motion vector predictor (step Ss_4).

[0466] Finally, the inter predictor 218 generates a predicted image of the current block by performing motion compensation of the current block using the derived MV and the decoded reference picture (step Ss_5).

[0467] (Predictive Controller)

[0468] The prediction controller 220 selects an intra prediction signal or an inter prediction signal and outputs the selected prediction signal to the adder 208. In general, the configuration, function, and process of the prediction controller 220, the intra predictor 216, and the inter predictor 218 on the decoder side may correspond to the configuration, function, and process of the prediction controller 128, the intra predictor 124, and the inter predictor 126 on the encoder side.

[0469] (Decoder installation example)

[0470] Fig.46 2 is a block diagram showing an example of an installation of the decoder 200. The decoder 200 includes a processor b1 and a memory b2. For example, Fig.41 The multiple components of the decoder 200 shown are installed in Fig.46 As shown, processor b1 and memory b2.

[0471] Processor b1 is a circuit that performs information processing and can be accessed by memory b2. For example, processor b1 is a dedicated or general-purpose electronic circuit that decodes video (i.e., encoded bit stream). Processor b1 can be a processor such as a CPU. In addition, processor b1 can be an aggregation of multiple electronic circuits. In addition, for example, processor b1 can act as Fig.41 The roles of two or more constituent elements or the like among the multiple constituent elements of the decoder 200 shown.

[0472] The memory b2 is a dedicated or general memory for storing information used by the processor b1 to decode the encoded bit stream. The memory b2 may be an electronic circuit and may be connected to the processor b1. In addition, the memory b2 may be included in the processor b1. In addition, the memory b2 may be an aggregation of multiple electronic circuits. In addition, the memory b2 may be a disk, an optical disk, etc., or may be represented as a memory, a recording medium, etc. In addition, the memory b2 may be a non-volatile memory or a volatile memory.

[0473] For example, the memory b2 may store a video or a bit stream. In addition, the memory b2 may store a program for causing the processor b1 to decode the encoded bit stream.

[0474] In addition, for example, memory b2 can act as Fig.41 The memory b2 may serve as a memory for storing information. Fig.41 The roles of the block memory 210 and the frame memory 214 are shown. More specifically, the memory b2 can store reconstructed blocks, reconstructed pictures, etc.

[0475] It should be noted that in the decoder 200, it is not necessary to implement Fig.41 All the multiple constituent elements shown in the figure may not be included, and all the processes described above may not be performed. Fig.41 A portion of the constituent elements indicated in the diagram may be included in another device, or a portion of the above-described process may be performed by another device.

[0476] (Definition of terms)

[0477] The corresponding terms can be defined as shown in the following examples.

[0478] A picture is an array of luma samples in monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color formats. A picture can be a frame or a field.

[0479] A frame is composed of a top field and a bottom field, where sample lines 0, 2, 4, ... originate from the top field and sample lines 1, 3, 5, ... originate from the bottom field.

[0480] A slice is an integer number of coding tree units contained in an independent slice segment and all subsequent dependent slice segments (if any) before the next independent slice segment in the same access unit.

[0481] A tile is a rectangular area of ​​a coding treeblock within a specific tile column and a specific tile row in a picture. A tile can be a rectangular area of ​​a frame that is intended to be independently decodable and coded, but loop filtering across tile edges can still be applied.

[0482] A block is an MxN (M columns by N rows) array of samples or an MxN array of transform coefficients. A block can be a square or rectangular area of ​​pixels that includes one luma matrix and two chroma matrices.

[0483] A Coding Tree Unit (CTU) can be a coding tree block of luma samples for a picture with three sample arrays, or two corresponding coding tree blocks of chroma samples. Alternatively, a CTU can be a coding tree block of samples for a monochrome picture and one of such pictures that is coded using three separate color planes and syntax structures for coding the samples. A superblock can be a square block of 64×64 pixels consisting of 1 or 2 mode information blocks, or recursively partitioned into four 32×32 blocks, which themselves can be further partitioned.

[0484] (Non-rectangular partitions)

[0485] In the prediction controller 128 coupled to the intra-frame predictor 124 and the inter-frame predictor 126 on the encoder side (see Figure 1 ), and in a prediction controller 220 coupled to the intra-frame predictor 216 and the inter-frame predictor 218 on the decoder side (see Fig.41 ), so far, the partitions (or blocks or sub-blocks of variable size) for which motion information (eg motion vectors) is obtained by segmenting each block have always been rectangular, such as Figure 3 As shown. The inventors have found that, in various embodiments, generating partitions having non-rectangular shapes such as triangles according to the content of the image in the picture leads to improvements in image quality and coding efficiency. In the following, various embodiments will be described in which at least one partition partitioned from the image block for prediction purposes has a non-rectangular shape. Note that these embodiments are equally applicable to the encoder side (prediction controller 128 coupled to intra predictor 124 and inter predictor 126) and the decoder side (prediction controller 220 coupled to intra predictor 216 and inter predictor 218), and can be used in Figure 1 The encoder or Fig.41It is implemented in a decoder, etc.

[0486] Fig.47 is a flowchart showing an example of a process of partitioning an image block into partitions including at least a first partition having a non-rectangular shape (e.g., a triangle) and a second partition and performing further processing including encoding (or decoding) the image block into a reconstructed combination of the first and second partitions.

[0487] In step S1001, the image block is divided into partitions including a first partition having a non-rectangular shape and a second partition which may or may not have a non-rectangular shape. Fig.48 As shown, the image block can be partitioned from the upper left corner of the image block to the lower right corner of the image block to create a first partition and a second partition both having a non-rectangular shape (e.g., a triangle), or the image block can be partitioned from the upper right corner of the image block to the lower left corner of the image block to create a first partition and a second partition both having a non-rectangular shape (e.g., a triangle). Fig.48 and 53 -55 describes various examples of non-rectangular segmentation.

[0488] In step S1002, the process predicts a first motion vector for a first partition and predicts a second motion vector for a second partition. For example, the prediction of the first and second motion vectors may include selecting the first motion vector from a first set of motion vector candidates and selecting the second motion vector from a second set of motion vector candidates.

[0489] In step S1003 , a motion compensation process is performed to obtain a first partition using the first motion vector derived in step S1002 above, and to obtain a second partition using the second motion vector derived in step S1002 above.

[0490] In step S1004, a prediction process is performed for the image block as a (reconstructed) combination of the first partition and the second partition. The prediction process may include a boundary smoothing process to smooth the boundary between the first partition and the second partition. For example, the boundary smoothing process may involve weighting a first value of a boundary pixel predicted based on the first partition and a second value of a boundary pixel predicted based on the second partition. This will be referred to below. Fig.49 , 50 , 56 and 57A-57D describe various implementations of the boundary smoothing process.

[0491] In step S1005, the process encodes or decodes the image block using one or more parameters, the one or more parameters including a partition parameter indicating partitioning the image block into a second partition and a first partition having a non-rectangular shape. Fig.51As summarized in the table of FIG. 1 , the partition parameter (“first index value”) may have an influence on, for example, the partition direction to be applied in the partition (e.g., from top left to bottom right or from top right to bottom left, as Fig.48 As shown in FIG. 1002 , the first and second motion vectors derived in step S1002 are jointly encoded. Fig.51 , 52 and 58-61 describe in detail the details of such partitioning syntax operations involving one or more parameters including partitioning parameters.

[0492] Fig.53 2000 is a flow chart showing a process 2000 for segmenting an image block. In step S2001, the process segments the image into a plurality of partitions, including a first partition having a non-rectangular shape and a second partition that may or may not have a non-rectangular shape. Fig.48 As shown, the image block can be divided into a first partition having a triangular shape and a second partition also having a triangular shape. There are many other examples in which the image block is divided into multiple partitions, including a first partition and a second partition, wherein at least the first partition has a non-rectangular shape. The non-rectangular shape can be a triangle, a trapezoid, and a polygon with at least five sides and angles.

[0493] For example, Fig.54 As shown, the image block can be partitioned into two triangular partitions; the image block can be partitioned into more than two triangular partitions (for example, three triangular partitions); the image block can be partitioned into a combination of triangular partitions and rectangular partitions; or the image block can be partitioned into a combination of triangular partitions and polygonal partitions.

[0494] like Fig.55 As further shown, the image block can be divided into L-shaped (polygonal) partitions and rectangular partitions; the image block can be divided into pentagonal (polygonal) partitions and triangular partitions; the image block can be divided into hexagonal (polygonal) partitions and pentagonal (polygonal) partitions; or the image block can be divided into multiple polygonal partitions.

[0495] Back to Fig.53, in step S2002, the process predicts a first motion vector for a first partition, for example, by selecting the first partition from a first set of motion vector candidates, and predicts a second motion vector for a second partition, for example, by selecting the second partition from a second set of motion vector candidates. For example, the first set of motion vector candidates may include motion vectors of partitions adjacent to the first partition, and the second set of motion vector candidates may include motion vectors of partitions adjacent to the second partition. The adjacent partitions may be one or both of spatially adjacent partitions and temporally adjacent partitions. Some examples of spatially adjacent partitions include partitions located to the left, lower left, lower, lower right, right, upper right, upper, or upper left of the partition being processed. An example of a temporally adjacent partition is a co-located partition in a reference picture of an image block.

[0496] In various embodiments, a partition adjacent to the first partition and a partition adjacent to the second partition may be outside the image block from which the first partition and the second partition are segmented. The first group of motion vector candidates may be the same as or different from the second group of motion vector candidates. In addition, at least one of the first group of motion vector candidates and the second group of motion vector candidates may be the same as another, third group of motion vector candidates prepared for the image block.

[0497] In some embodiments, in step S2002, in response to determining that the second partition (similar to the first partition) also has a non-rectangular shape (e.g., a triangle), the process 2000 creates a second set of motion vector candidates (for the non-rectangular second partition) that includes motion vectors of partitions adjacent to the second partition that do not include the first partition (excluding the motion vector of the first partition). On the other hand, in response to determining that the second partition (different from the first partition) has a rectangular shape, the process 2000 creates a second set of motion vector candidates (for the rectangular second partition) that includes motion vectors of partitions adjacent to the second partition that include the first partition.

[0498] In step S2003, the process encodes or decodes the first partition using the first motion vector derived in step S2002 above, and encodes or decodes the second partition using the second motion vector derived in step S2002 above.

[0499] picture Fig.53 As in process 2000, the image block segmentation process may be performed by an image encoder, for example, Figure 1As shown, the image encoder includes a circuit and a memory coupled to the circuit. The circuit performs in operation: dividing the image block into a plurality of partitions, including a second partition and a first partition having a non-rectangular shape (step S2001); predicting a first motion vector for the first partition and a second motion vector for the second partition (step S2002); and encoding the first partition using the first motion vector and encoding the second partition using the second motion vector (step S2003).

[0500] According to another embodiment, Figure 1 As shown, an image encoder is provided, including: a partitioner 102, which receives an original image and partitions it into blocks during operation; an adder 104, which receives blocks from the partitioner and predictions from a prediction controller 128 during operation, and subtracts each prediction from its corresponding block to output a residual; a transformer 106, which performs a transform on the residual output from the adder 104 during operation to output a transform coefficient; a quantizer 108, which quantizes the transform coefficient to generate a quantized transform coefficient during operation; an entropy encoder 110, which encodes the quantized transform coefficient to generate a bitstream during operation; and a prediction controller 128, coupled to an inter-frame predictor 126, an intra-frame predictor 124 and memories 118, 122, wherein the inter-frame predictor 126 generates a prediction of a current block based on a reference block in an encoded reference picture during operation, and the intra-frame predictor 124 generates a prediction of the current block based on an encoded reference block in a current picture during operation. The prediction controller 128 divides the block into a plurality of partitions in operation, including a second partition and a first partition having a non-rectangular shape ( Fig.53 , step S2001); predicting a first motion vector for a first partition and a second motion vector for a second partition (step S2002); and encoding the first partition using the first motion vector and encoding the second partition using the second motion vector (step S2003).

[0501] According to another embodiment, for example Fig.41 As shown, an image decoder is provided, which includes a circuit and a memory coupled to the circuit. The circuit performs in operation: dividing the image block into a plurality of partitions, including a second partition and a first partition having a non-rectangular shape ( Fig.53 , step S2001); predicting a first motion vector for a first partition and a second motion vector for a second partition (step S2002); and decoding the first partition using the first motion vector and decoding the second partition using the second motion vector (step S2003).

[0502] According to another embodiment, there is provided Fig.41The image decoder shown includes: an entropy decoder 202, which receives a coded bit stream in operation and decodes the coded bit stream to obtain quantized transform coefficients; an inverse quantizer 204 and an inverse transformer 206, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder 208, which in operation adds the residual output from the inverse quantizer 204 and the transformer 206 to the prediction output from the prediction controller 220 to reconstruct the block; and a prediction controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216 and memories 210, 214, wherein the inter-frame predictor 218 generates a prediction of the current block based on a reference block in a decoded reference picture in operation, and the intra-frame predictor 216 generates a prediction of the current block based on a decoded reference block in the current picture in operation. The prediction controller 220 divides the image block into a plurality of partitions in operation, including a second partition and a first partition having a non-rectangular shape ( Fig.53 , step S2001); predicting a first motion vector for a first partition and a second motion vector for a second partition (step S2002); and decoding the first partition using the first motion vector and decoding the second partition using the second motion vector (step S2003).

[0503] (Border smoothing)

[0504] As above Fig.47 As described in, according to various embodiments, step S1004 of performing a prediction process for the image block as a (reconstructed) combination of the second partition and the first partition having a non-rectangular shape may involve applying a boundary smoothing process along a boundary between the first partition and the second partition.

[0505] For example, Fig.57B An example of a boundary smoothing process is shown, which involves weighting a first value of a boundary pixel and a second value of the boundary pixel, wherein the first value of the boundary pixel is first predicted based on a first partition and the second value of the boundary pixel is second predicted based on a second partition.

[0506] Fig.56 3000 is a flowchart illustrating an overall boundary smoothing process according to one embodiment, which involves weighting a first value of a boundary pixel for a first prediction based on a first partition and a second value of a boundary pixel for a second prediction based on a second partition. Fig.57A or Fig.48 , Fig.54 and Fig.55 As shown, the image block is divided into a first partition and a second partition along a boundary in which at least the first partition has a non-rectangular shape.

[0507] In step S3002, a group of pixels ( Fig.57A In step S3003, a second prediction is made for a second value of the (same) pixel group of the first partition along the boundary (for example, color, brightness, transparency, etc.), wherein the first value is first predicted using information of the first partition. In step S3003, a second prediction is made for a second value of the (same) pixel group of the first partition along the boundary, wherein the second value is second predicted using information of the second partition. In some embodiments, at least one of the first prediction and the second prediction is an inter-frame prediction process, which predicts the first value and the second value based on a reference partition in an encoded reference picture. Reference Fig.57D In some embodiments, the prediction process predicts a first value of all pixels of the first partition including a pixel group on which the first partition and the second partition overlap ("a first sample group"), and predicts only a second value of a pixel group on which the first and second partitions overlap ("a second group of samples"). In another embodiment, at least one of the first prediction and the second prediction is an intra-prediction process that predicts the first value and the second value based on an encoded reference partition in the current picture. In some embodiments, the prediction method used in the first prediction is different from the prediction method used in the second prediction. For example, the first prediction may include an inter-prediction process, and the second prediction may include an intra-prediction process. The information used for making a first prediction of the first value or a second prediction of the second value may be a motion vector of the first or second partition, an intra-prediction direction, etc.

[0508] In step S3004, a first value predicted using the first partition and a second value predicted using the second partition are weighted. In step S3005, the first partition is encoded or decoded using the weighted first value and second value.

[0509] Fig.57B An example of a boundary smoothing operation is shown, where the first partition and the second partition overlap on (at most) five pixels per row or column. That is, the number of pixel groups per row or column for which a first value is predicted based on the first partition and a second value is predicted based on the second partition is at most five. Fig.57C Another example of a boundary smoothing operation is shown, in which the first partition and the second partition overlap on (at most) three pixels per row or column. That is, the number of pixel groups per row or column for which a first value is predicted based on the first partition and a second value is predicted based on the second partition is at most three.

[0510] Fig.49Another example of a boundary smoothing operation is shown, where the first partition and the second partition overlap on (at most) four pixels per row or column. That is, the number of pixel groups per row or column for which a first value is predicted based on the first partition and a second value is predicted based on the second partition is at most four. In the example shown, weights 1 / 8, 1 / 4, 3 / 4, and 7 / 8 may be applied to the first value of the four pixels in the group, respectively, and weights 7 / 8, 3 / 4, 1 / 4, and 1 / 8 may be applied to the second value of the four pixels in the group, respectively.

[0511] Fig.50 Other examples of boundary smoothing operations are shown, where the first partition and the second partition overlap on zero pixels per row or column (they do not overlap), overlap on (at most) one pixel per row or column, and overlap on (at most) two pixels per row or column. In examples where the first and second partitions do not overlap, a zero weight is applied. In examples where the first partition and the second partition overlap on one pixel per row or column, a weight of 1 / 2 may be applied to the first value of the pixel in the group predicted based on the first partition, and a weight of 1 / 2 may be applied to the second value of the pixel in the group predicted based on the second partition. In examples where the first partition and the second partition overlap on two pixels per row or column, weights of 1 / 3 and 2 / 3 may be applied to the first value of the two pixels in the group predicted based on the first partition, respectively, and weights of 2 / 3 and 1 / 3 may be applied to the second value of the two pixels in the group predicted based on the second partition, respectively.

[0512] According to the above embodiment, the number of pixels in the group on which the first partition and the second partition overlap is an integer. In other embodiments, for example, the number of overlapping pixels in the group can be a non-integer and can be a fraction. Moreover, depending on each application, the weights of the first value and the second value applied to the pixel group can be fractions or integers.

[0513] Similar to Fig.56 The boundary smoothing process of process 3000 may be performed by, for example Figure 1 The image encoder shown in the figure is executed, and the image encoder includes a circuit and a memory coupled to the circuit. In operation, the circuit performs a boundary smoothing operation ( Fig.56 , step S3001). The boundary smoothing operation includes: using information of the first partition, making a first prediction of a first value of a group of pixels of the first partition along the boundary (step S3002); using information of the second partition, making a second prediction of a second value of a group of pixels of the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); and encoding the first partition using the weighted first value and the weighted second value (step S3005).

[0514] According to another embodiment, Figure 1 As shown, an image encoder is provided, including: a partitioner 102, which receives an original image and partitions it into blocks during operation; and an adder 104, which receives blocks from the partitioner and predictions from a prediction controller 128 during operation, and subtracts each prediction from its corresponding block to output a residual; a transformer 106, which transforms the residual output from the adder 104 to output a transform coefficient during operation; a quantizer 108, which quantizes the transform coefficient to generate a quantized transform coefficient during operation; an entropy encoder 110, which encodes the quantized transform coefficient to generate a bitstream during operation; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124 and memories 118, 122, wherein the inter-frame predictor 126 generates a prediction of a current block based on a reference block in an encoded reference picture during operation, and the intra-frame predictor 124 generates a prediction of the current block based on an encoded reference block in a current picture during operation. The prediction controller 128 performs a boundary smoothing operation along a boundary between the second partition segmented from the image block and the first partition having a non-rectangular shape. Fig.56 , step S3001). The boundary smoothing operation includes: using information of the first partition, making a first prediction on a first value of a group of pixels of the first partition along the boundary (step S3002); using information of the second partition, making a second prediction on a second value of a group of pixels of the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); encoding the first partition using the weighted first value and the weighted second value (step S3005).

[0515] According to another embodiment, there is provided an image decoder, such as in Fig.41 As shown in , it includes a circuit and a memory coupled to the circuit. In operation, the circuit performs a boundary smoothing operation along a boundary between a second partition segmented from an image block and a first partition having a non-rectangular shape. Fig.56 , step S3001). The boundary smoothing operation includes: using information of the first partition, making a first prediction on a first value of a group of pixels of the first partition along the boundary (step S3002); using information of the second partition, making a second prediction on a second value of a group of pixels of the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); decoding the first partition using the weighted first value and the weighted second value (step S3005).

[0516] According to another embodiment, there is provided a Fig.41The image decoder shown includes: an entropy decoder 202, which receives and decodes the encoded bit stream in operation to obtain quantized transform coefficients; an inverse quantizer 204 and a transformer 206, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder 208, which in operation adds the residual output from the inverse quantizer 204 and the transformer 206 to the prediction output from the prediction controller 220 to reconstruct the block; and a prediction controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216 and memories 210, 214, wherein the inter-frame predictor 218 generates a prediction for the current block based on a reference block in a decoded reference picture in operation, and the intra-frame predictor 216 generates a prediction for the current block based on a decoded reference block in the current picture in operation. The prediction controller 220 performs a boundary smoothing operation along the boundary between the second partition partitioned from the image block and the first partition having a non-rectangular shape in operation ( Fig.56 , step S3001). The boundary smoothing operation includes: using information of the first partition, making a first prediction on a first value of a group of pixels of the first partition along the boundary (step S3002); using information of the second partition, making a second prediction on a second value of a group of pixels of the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); decoding the first partition using the weighted first value and the weighted second value (step S3005).

[0517] (Entropy encoding and decoding using partition parameter syntax)

[0518] As in Fig.47 As shown in step S1005, according to various embodiments, one or more parameters may be used to encode or decode the image block partitioned into the second partition and the first partition having a non-rectangular shape, wherein the one or more parameters include a partition parameter indicating the non-rectangular partition of the image block. In various embodiments, such partition parameters may be applied to the partition direction (e.g., from the upper left to the lower right or from the upper right to the lower left, see Fig.48 ), and the first motion vector and the second motion vector predicted in step S1002 are jointly encoded, as will be described in more detail below.

[0519] Fig.51 is a table of sample partition parameters ("first index values") and sets of information jointly encoded by the partition parameters. The partition parameters ("first index values") range from 0 to 6 and jointly encode the following items: the partitioning of the image block into the first partition and the second partition (they are both triangles, see Fig.48 ) direction, the first motion vector predicted for the first partition ( Fig.47, step S1002), and predicting a second motion vector for the second partition ( Fig.47 Specifically, the partition direction encoded by partition parameter 0 is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second group of motion vector candidates for the second partition.

[0520] Partition parameter 1 is encoded as follows: the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "first" motion vector listed in the first set of motion vector candidates of the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates of the second partition. Partition parameter 2 is encoded as follows: the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates of the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates of the second partition. Partition parameter 3 is encoded as follows: the partition direction is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates of the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates of the second partition. Partition parameter 4 is encoded as follows: the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates of the first partition, and the second motion vector is the "third" motion vector listed in the second set of motion vector candidates of the second partition. Partition parameter 5 is encoded as follows: the partition direction is from top left to bottom right, the first motion vector is the "third" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 6 is encoded as follows: the partition direction is from top left to bottom right, the first motion vector is the "fourth" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition.

[0521] Fig.58 4 is a flow chart showing a method 4000 performed at the encoder side. In step S4001, the process divides the image block into a plurality of partitions based on a partition parameter indicating the partition, the plurality of partitions including a second partition and a first partition having a non-rectangular shape. For example, as described above Fig.51As shown, the partition parameter may indicate the direction of partitioning the image block (e.g., from the upper right corner to the lower left corner or from the upper left corner to the lower right corner). In step S4002, the process encodes the first partition and the second partition. In step S4003, the process writes one or more parameters including the partition parameter into the bitstream, and the decoder side may receive and decode the bitstream to obtain the one or more parameters so as to perform the same prediction process (as performed on the encoder side) on the first partition and the second partition on the decoder side. One or more parameters including the partition parameter may jointly or separately encode various information fragments, such as the non-rectangular shape of the first partition, the shape of the second partition, the partition direction for partitioning the image block to obtain the first partition and the second partition, the first motion vector of the first partition, the second motion vector of the second partition, etc.

[0522] Fig.59 is a flowchart showing a method 5000 performed on a decoder side. In step S5001, the process parses one or more parameters from a bitstream, wherein the one or more parameters include a partition parameter indicating that an image block is partitioned into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape. The one or more parameters including the partition parameter parsed from the bitstream can be jointly or separately encoded with various information fragments required for the decoder side to perform the same prediction process as performed on the encoder side, such as the non-rectangular shape of the first partition, the shape of the second partition, the partition direction for partitioning the image block to obtain the first partition and the second partition, the first motion vector of the first partition, the second motion vector of the second partition, etc. In step S5002, the process 5000 partitions the image block into a plurality of partitions based on the partition parameter parsed from the bitstream. In step S5003, the process decodes the first partition and the second partition partitioned from the image block.

[0523] Fig.60 is a table of sample partition parameters ("first index values") and information sets jointly encoded by the partition parameters, which is essentially similar to the above Fig.51 In the sample table. Fig.60 In the partition parameter ("first index value") ranges from 0 to 6, and the following items are jointly encoded: the shape of the first and second partitions partitioned from the image block, the direction of partitioning the image block into the first and second partitions, the first motion vector predicted for the first partition ( Fig.47 , step S1002), and the second motion vector predicted for the second partition ( Fig.47 , step S1002). Specifically, the partition parameter 0 is encoded as follows: neither the first partition nor the second partition has a triangular shape, and therefore, the partition direction information is "N / A", the first motion vector information is "N / A", and the second motion vector information is "N / A".

[0524] Partition parameter 1 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 2 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "first" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 3 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 4 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 5 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "third" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 6 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper left corner to the lower right corner, the first motion vector is the "third" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition.

[0525] According to some implementations, the partition parameters (index values) may be binarized according to a binarization scheme that is selected according to the values ​​of at least one or more parameters. Fig.52 An example binarization scheme for binarizing index values ​​(partition parameter values) is shown.

[0526] Figure 25 is a table of sample combinations of first parameters and second parameters, wherein one parameter is a partition parameter indicating partitioning of an image block into a plurality of partitions, the plurality of partitions comprising a second partition and a first partition having a non-rectangular shape. In this example, the partition parameter can be used to indicate the partitioning of the image block without jointly encoding other information encoded by one or more other parameters.

[0527] exist Fig.61 In a first example, a first parameter is used to indicate an image block size, and a second parameter is used as a partition parameter (flag) to indicate that at least one of a plurality of partitions segmented from the image block has a triangular shape. Such a combination of the first parameter and the second parameter can be used to indicate, for example, 1) when the image block size is greater than 64×64, there is no triangular partition, or 2) when the aspect ratio of the image block is greater than 4 (e.g., 64×4), there is no triangular partition.

[0528] exist Fig.61 In the second example, the first parameter is used to indicate the prediction mode, and the second parameter is used as a partition parameter (flag) for indicating that at least one of the plurality of partitions partitioned from the image block has a triangular shape. Such a combination of the first parameter and the second parameter can be used to indicate, for example: 1) when encoding the image block in intra mode, there is no triangular partition.

[0529] exist Fig.61 In the third example of , the first parameter is used as a partition parameter (flag) for indicating that at least one of the plurality of partitions partitioned from the image block has a triangular shape, and the second parameter is used to indicate a prediction mode. Such a combination of the first parameter and the second parameter can be used to indicate, for example: 1) when at least one of the plurality of partitions partitioned from the image block has a triangular shape, the image block must be inter-coded.

[0530] exist Fig.61 In the fourth example of , the first parameter indicates the motion vector of the adjacent block, and the second parameter is used as a segmentation parameter indicating the direction of segmenting the image block into two triangles. Such a combination of the first parameter and the second parameter can be used to indicate, for example: 1) when the motion vector of the adjacent block is a diagonal direction, the direction of segmenting the image block into two triangles is from the upper left corner to the lower right corner.

[0531] exist Fig.61 In a fifth example, the first parameter indicates the intra prediction direction of the adjacent block, and the second parameter is used as a partition parameter indicating the direction of partitioning the image block into two triangles. Such a combination of the first parameter and the second parameter can be used to indicate, for example: 1) when the intra prediction direction of the adjacent block is the anti-diagonal direction, the direction of partitioning the image block into two triangles is from the upper right corner to the lower left corner.

[0532] It should be understood that if Fig.51 , 60As shown in and 61, the table of one or more parameters including partition parameters and which information is jointly or separately encoded is presented only as an example, and various other ways of jointly or separately encoding various information as part of the above-mentioned partition syntax operation are also within the scope of the present disclosure. For example, the partition parameter may indicate that the first partition is a triangle, a trapezoid, or a polygon with at least five sides and angles. The partition parameter may indicate that the second partition has a non-rectangular shape, such as a triangle, a trapezoid, and a polygon with at least five sides and angles. The partition parameter may indicate one or more information about the partition, such as the non-rectangular shape of the first partition, the shape of the second partition (which may be non-rectangular or rectangular), and the partition direction applied to partition the image block into multiple partitions (for example, from the upper left corner of the image block to its lower right corner, and from the upper right corner of the image block to its lower left corner). The partition parameter may jointly encode other information, such as the first motion vector of the first partition, the second motion vector of the second partition, the image block size, the prediction mode, the motion vector of the adjacent block, the intra-frame prediction direction of the adjacent block, etc. Alternatively, any of the further information may be encoded separately by one or more parameters other than the partition parameter.

[0533] Similar to Fig.58 In process 4000, partition syntax operations may be performed by, for example Figure 1 The image encoder shown in the figure is executed, and the image encoder includes a circuit and a memory coupled to the circuit. The circuit performs a partition syntax operation in operation, and the partition syntax operation includes: based on a partition parameter indicating the partition, the image block is partitioned into a plurality of partitions, and the plurality of partitions include a second partition and a first partition having a non-rectangular shape ( Fig.58 , step S4001); encoding a first partition and a second partition (S4002); writing one or more parameters including partition parameters into a bitstream (S4003).

[0534] According to another embodiment, Figure 1As shown, an image encoder is provided, including: a partitioner 102, which receives an original picture and partitions it into blocks during operation; an adder 104, which receives blocks from the partitioner and predictions from a prediction controller 128 during operation, and subtracts each prediction from its corresponding block to output a residual; a transformer 106, which performs a transformation on the residual output from the adder 104 to output a transformation coefficient during operation; a quantizer 108, which quantizes the transformation coefficient to generate a quantized transformation coefficient during operation; an entropy encoder 110, which encodes the quantized transformation coefficient to generate a bitstream during operation; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124 and memories 118, 122, wherein the inter-frame predictor 126 generates a prediction for a current block based on a reference block in an encoded reference picture during operation, and the intra-frame predictor 124 generates a prediction for a current block based on an encoded reference block in a current picture during operation. The prediction controller 128 in operation partitions the image block into a plurality of partitions based on the partition parameters indicating the partition, the plurality of partitions including the second partition and the first partition having a non-rectangular shape ( Fig.58 , step S4001), and encode the first partition and the second partition (step S4002). The entropy encoder 110 writes one or more parameters including partition parameters into the bitstream in operation (step S4003).

[0535] According to another embodiment, there is provided an image decoder, such as Fig.41 As shown, the image decoder includes a circuit and a memory coupled to the circuit. The circuit performs a partition syntax operation in operation, and the partition syntax operation includes: parsing one or more parameters from a bit stream, wherein the one or more parameters include a partition parameter indicating that the image block is divided into a plurality of partitions, and the plurality of partitions include a second partition and a first partition having a non-rectangular shape ( Fig.59 , step S5001); dividing the image block into multiple partitions based on the partition parameter (S5002); decoding the first partition and the second partition (S5003).

[0536] According to another embodiment, there is provided Fig.41The image decoder shown in the figure includes: an entropy decoder 202, which receives and decodes the encoded bit stream to obtain quantized transform coefficients in operation; an inverse quantizer 204 and a transformer 206, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder 208, which in operation adds the residual output from the inverse quantizer 204 and the transformer 206 to the prediction output from the prediction controller 220 to reconstruct the block; and a prediction controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216 and memories 210, 214, wherein the inter-frame predictor 218 generates a prediction for a current block based on a reference block in a decoded reference picture in operation, and the intra-frame predictor 216 generates a prediction for the current block based on a decoded reference block in the current picture in operation. The entropy decoder 202 is operated by parsing one or more parameters from a bitstream, wherein the one or more parameters include a partition parameter indicating partitioning of the image block into a plurality of partitions, wherein the plurality of partitions include a second partition and a first partition having a non-rectangular shape ( Fig.59 , step S5001); partitioning the image block into a plurality of partitions based on a partition parameter (S5002); in some implementations, decoding the first partition and the second partition in cooperation with the prediction controller 220 (S5003).

[0537] According to other examples, the intra predictor may perform the following process.

[0538] All motion vector candidates included in the first group of motion vector candidates may be uni-predictive motion vectors. That is, the intra predictor may determine only uni-predictive motion vectors as motion vector candidates in the first group of motion vector candidates.

[0539] The intra predictor may select only a single predictive motion vector candidate from the first set of motion vector candidates.

[0540] In an embodiment, only a single prediction motion vector may be used to predict a small block. A dual prediction motion vector may be used to predict a large block. For example, the prediction process may include determining the size of an image block. When it is determined that the image block size is greater than a threshold, the prediction may include selecting a first motion vector from a first set of motion vector candidates, and the first set of motion vector candidates may include a single prediction and / or a dual prediction motion vector. When it is determined that the image block size is not greater than a threshold, the prediction may select a first motion vector from a first set of motion vector candidates, and the first set of motion vector candidates may include only a single prediction motion vector.

[0541] (Method for storing motion vectors of weighted regions)

[0542] Fig.62is an example of a flowchart illustrating an example of a process flow 6000 according to one embodiment, the process flow 6000 predicting a first set of samples of a first partition of a current picture having a first motion vector, obtaining a second motion vector from a second partition different from the first partition, predicting a second set of samples of a first portion included in the first partition having the second motion vector, weighting the first set of samples and the second set of samples, storing at least one of the first and second motion vectors for the first partition, and encoding or decoding the first partition using the weighted samples, and performing further processing. The process flow 6000 may be performed, for example, by Figure 1 Encoder 100, Fig.41 The decoder 200 etc. executes.

[0543] In step S6001, a first set of samples of a first partition of a current picture is predicted using one or more motion vectors including a first motion vector. The first partition may or may not have a non-rectangular shape. The first set of samples may be, for example, a block of pixels having an associated motion vector and luminance and chrominance pixel components. For example, each sample may be a 4×4 block of pixels associated with a first motion vector and having associated luminance and chrominance pixel components.

[0544] The first partition may be, for example, a partition of an image block that has been divided into a plurality of partitions. Fig.63 is a conceptual diagram for illustrating an exemplary method of dividing an image block into a first partition and a second partition. Fig.48 , 54 and 55. For example, Fig.63 As shown, the image block can be divided into two or more partitions having various shapes. Fig.63 The example illustrations include: an image block that is segmented from the upper left corner of the image block to the lower right corner of the image block to create a first partition and a second partition that both have a non-rectangular shape (e.g., a triangular shape); an image block that is segmented into an L-shaped partition and a rectangular partition; an image block that is segmented into a pentagonal partition and a triangular partition; an image block that is segmented into a hexagonal partition and a pentagonal partition; and an image block that is segmented into two polygonal partitions. The various partition shapes shown can be formed by segmenting the image block in other ways. For example, two triangular partitions can be formed by segmenting the image block from the upper right corner of the image block to the lower left corner of the image block to create a first partition and a second partition that both have a triangular shape. In some embodiments, two or more partitions of the image block may have overlapping portions.

[0545] In some embodiments, a motion estimation process may be used to obtain the first motion vector. In some embodiments, the first motion vector may be parsed from a bitstream. In some embodiments, the first motion vector may be a motion vector predicted from a set of motion vector candidates for at least the first partition. The motion vector candidates of the motion vector candidate list may include motion vector candidates derived from spatial or temporal neighboring partitions of at least the first partition, or may be derived from a motion vector candidate list for an image block for a prediction mode such as a merge mode, a skip mode, or an inter-frame mode.

[0546] Fig.64 is a conceptual diagram for illustrating adjacent and non-adjacent spatially adjacent partitions of a first partition of a current picture. An adjacent spatially adjacent partition is a partition adjacent to the first partition in the current picture. A non-adjacent spatially adjacent partition is a partition spaced apart from the first partition in the current picture. In some embodiments, a set of motion vector candidates may be derived from spatially adjacent partitions of at least the first partition in the current picture.

[0547] The first motion vector may be, for example, a uni-predictive motion vector or a bi-predictive motion vector. Fig.65 is a conceptual diagram for illustrating uni-predictive and bi-predictive motion vector candidates for image blocks of a current picture. A uni-predictive motion vector candidate is a single motion vector for a current block in a current picture relative to a single reference picture. Fig.65 As shown at the top of , a single prediction motion vector candidate is a motion vector from a block of a current picture to a block of a reference picture, where the reference picture appears before the current picture in display order. In some embodiments, the reference picture may appear after the current picture in display order.

[0548] The dual-prediction motion vector candidate includes two motion vectors: a first motion vector of the current block relative to the first reference picture and a second motion vector of the current block relative to the second reference picture. Fig.65 As shown, the lower left dual prediction motion vector candidate has a first motion vector from a block of the current picture to a block of the first reference picture, and a second motion vector from a block of the current picture to a block of the second reference picture. As shown, the first reference picture and the second reference picture appear before the current picture in display order. Fig.65 The bi-predictive motion vector candidate in the lower right corner has a first motion vector from a block of the current picture to a block of a first reference picture, and a second motion vector from the block of the current picture to a block of a second reference picture. The first reference picture appears before the current picture in display order, and the second reference picture appears after the current picture in display order.

[0549] In some embodiments, some partition shapes may be associated with one type of predicted motion vector. For example, in some embodiments, a triangular partition may be associated with a single predicted motion vector candidate.

[0550] Prediction of the first set of samples may include a motion compensation process with one or more motion vectors including the first motion vector, and the motion compensation process may predict samples from a current picture (intra-block), or predict samples from a different picture (inter-block), or a combination thereof.

[0551] In step S6002, one or more motion vectors including a second motion vector are obtained from a second partition different from the first partition. The second partition may be, for example, a partition co-located in space or time with the first partition. A portion of the second partition may overlap spatially with a portion of the first partition. A motion estimation process may be used to obtain the second motion vector. In some embodiments, the second motion vector may be parsed from a bitstream. In some embodiments, the second motion vector may be a motion vector predicted from a set of motion vector candidates of at least the second partition. The motion vector candidates in the motion vector candidate list may include motion vector candidates derived from spatially or temporally adjacent partitions of the second partition, or may be derived from a motion vector candidate list for an image block used for a prediction mode (e.g., a merge mode, a skip mode, or other inter-frame modes). The second motion vector may be a single-prediction motion vector or a dual-prediction motion vector.

[0552] In step S6003, a second set of samples of a portion of the first partition is predicted using one or more motion vectors including the second motion vector. The first portion of the first partition is smaller than the first partition and is placed adjacent to an edge of the first partition and overlaps with the first set of samples. The prediction of the second set of samples may include a motion compensation process with one or more motion vectors including the second motion vector, and the motion compensation process may predict samples from the current picture (intra-block), or predict samples from different pictures (inter-block), or a combination thereof. The second set of samples may be, for example, a pixel block having an associated motion vector and luminance and chrominance pixel components. For example, each sample in the second set of samples may be a 4×4 pixel block associated with the second motion vector and having associated luminance and chrominance pixel components.

[0553] Fig.66 is a conceptual diagram for illustrating an example of a first portion of a first partition and a first set of samples and a second set of samples. The first portion may be, for example, one quarter of the width or height of the first partition. In another example, the first portion may have a width corresponding to N samples adjacent to an edge of the first partition, where N is an integer greater than zero, for example, N may be an integer 2. As shown in the figure, Fig.66The right example of shows a rectangular partition having a rectangular portion whose width is one quarter of the width of the first partition, wherein the first group of samples includes samples outside the first portion and samples inside the first portion, and the second group of samples includes samples inside the first portion. Fig.66 The center example of shows a rectangular partition having a rectangular portion whose height is one quarter of the height of the first partition, wherein the first set of samples includes samples outside the first portion and samples within the first portion, and the second set of samples includes samples within the first portion. Fig.66 The left example of shows a triangle partitioning of a polygonal portion whose height corresponds to two samples, where a first set of samples includes samples outside the first portion and samples within the first portion, and a second set of samples includes samples within the first portion.

[0554] The first portion may be a portion of the first partition that overlaps with an adjacent partition. Fig.67 is a conceptual diagram for illustrating a first portion of a first partition, which is a portion of the first partition that overlaps with a portion of an adjacent partition. For ease of explanation, a rectangular partition having a portion that overlaps with a spatially adjacent rectangular partition is shown. Partitions having other shapes, such as triangular partitions, may be used, and the overlapping portion may overlap with a partition that is adjacent in space or time. The adjacent partition may be a second partition.

[0555] In step S6004, a weighting process is performed on the first portion included in the first partition using a first set of samples included in the first portion and a subset of a second set of samples included in the first portion. The weighting may be performed as part of a boundary smoothing process, such as an overlapped block motion compensation process (OBMC, see Fig.35 and 36 and its description), reference Figure 47-61 The boundary smoothing process described in the foregoing description and the like. For example, various weights may be assigned to pixels in a first set of samples in the first portion based on the distance of the samples from an edge of a first partition aligned with the first portion. Samples closer to the edge may be assigned lower weights than samples further away from the edge. Similarly, various weights may be assigned to pixels in a second set of samples in the first portion of the first partition based on, for example, the distance of the samples from an edge of the first partition aligned with the first portion. Samples closer to the edge may be assigned higher weights than samples further away from the edge.

[0556] In step S6005, one or more motion vectors are stored for the first portion of the first partition, the stored motion vectors being based on one or both of the first motion vector and the second motion vector. In some embodiments, the stored motion vectors may take into account the reference picture lists pointed to by the first motion vector and the second motion vector.

[0557] For example, Fig.68 is a conceptual diagram for illustrating an example case where the first motion vector and the second motion vector are uni-predictive motion vectors pointing to different pictures in different reference picture lists. In some embodiments, when the first motion vector and the second motion vector are uni-predictive motion vectors pointing to different pictures in different reference picture lists, the first motion vector and the second motion vector may be combined and stored as a bi-predictive motion vector for the first portion of the first partition. Fig.68 As shown, the current picture is POC 4. The first motion vector Mv1 is a uni-predictive motion vector pointing to reference picture POC 0 of reference picture list L0, which includes reference pictures POC 0 to POC 8. The second motion vector Mv2 is a uni-predictive motion vector pointing to reference picture POC 16 of reference picture list L1 including reference pictures POC 8 to POC 16. The first motion vector Mv1 and the second motion vector Mv2 are combined and stored as a bi-predictive motion vector for the first part of the first partition.

[0558] In another example, Fig.69 is a conceptual diagram for illustrating an example case where the first motion vector and the second motion vector are uni-predicted motion vectors pointing to a picture in a single reference picture list. The bi-predicted motion vector points to two different reference picture lists. Therefore, simply combining the first motion vector and the second motion vector will not produce a bi-predicted motion vector. In some embodiments, when the first motion vector and the second motion vector are uni-predicted motion vectors pointing to different reference pictures in a single reference picture list, if one of the reference pictures pointed to is, for example, included in another reference picture list, the motion vector associated with the picture (e.g., the second motion vector if the reference picture it points to is included in two reference picture lists) can be represented by a motion vector pointing to the same reference picture in the other reference picture list. The first motion vector and the second motion vector can be combined and stored as a bi-predicted motion vector for the first part of the first partition by replacing the motion vector associated with the reference picture with a replacement motion vector pointing to a reference picture in another reference picture list. As shown in FIG. Fig.69As shown, the current picture is POC 4. The first motion vector Mv1 is a uni-predictive motion vector pointing to reference picture POC 0 of reference picture list L0, which includes reference pictures POC 0 to POC 8. The second motion vector Mv2 is a uni-predictive motion vector pointing to reference picture POC 8 of reference picture list L0. Reference picture POC 8 is also included in reference picture list L1. The second motion vector Mv2 is replaced by a replacement motion vector Mv2', which points to reference picture POC 8 in reference picture list L1. The first motion vector Mv1 and the replacement second motion vector Mv2' are combined and stored as a bi-predictive motion vector for the first part of the first partition. In other words, different lists pointing to the same picture can be used.

[0559] In another example, Fig.70 , Fig.71 and Fig.72 is a conceptual diagram for illustrating an example case where the first motion vector and the second motion vector are uni-predictive motion vectors pointing to the same reference picture in the same reference picture list. In some embodiments, when the first motion vector and the second motion vector are uni-predictive motion vectors pointing to the same reference picture in the same reference picture list, the first motion vector is stored as a uni-predictive motion vector for the first portion of the first partition. Fig.70 As shown, the current picture is POC 4. The first motion vector Mv1 is a uni-predictive motion vector pointing to reference picture POC 0 of reference picture list L0, which includes reference pictures POC 0 to POC 8. The second motion vector Mv2 is a uni-predictive motion vector pointing to reference picture POC 0 of reference picture list L0. The first motion vector Mv1 is stored as a uni-predictive motion vector for the first part of the first partition.

[0560] In some embodiments, when the first motion vector and the second motion vector are uni-predictive motion vectors pointing to the same reference picture in the same reference picture list, the second motion vector is stored as a uni-predictive motion vector for the first portion of the first partition. Fig.71 As shown, the current picture is POC 4. The first motion vector Mv1 is a uni-predictive motion vector pointing to reference picture POC 0 of reference picture list L0, which includes reference pictures POC 0 to POC 8. The second motion vector Mv2 is a uni-predictive motion vector pointing to reference picture POC 0 of reference picture list L0. The second motion vector Mv2 is stored as a uni-predictive motion vector for the first part of the first partition.

[0561] In some embodiments, when the first motion vector and the second motion vector are uni-predictive motion vectors pointing to the same reference picture in the same reference picture list, the second motion vector is replaced with a flipped motion vector pointing to a reference picture in another reference picture list, and the first motion vector and the flipped motion vector are combined and stored as a bi-predictive motion vector for the first portion of the first partition. Fig.72 As shown, the current picture is POC4. The first motion vector Mv1 is a uni-predictive motion vector pointing to reference picture POC 0 of reference picture list L0, and reference picture list L0 includes reference pictures POC 0 to POC 8. The second motion vector Mv2 is a uni-predictive motion vector pointing to reference picture POC 0 of reference picture list L0. The second motion vector Mv2 is replaced by a flipped motion vector Mv2' pointing to a reference picture in reference picture list L1, as shown in reference picture POC 8 in reference picture list L1. The first motion vector Mv1 and the flipped motion vector Mv2' are combined and stored as a bi-predictive motion vector for the first part of the first partition. The reference picture of another reference picture list may be selected in various ways, for example, a reference picture in reference picture list L1 having the same position as the reference picture pointed to by the second motion vector in reference picture list L0 may be selected, a reference picture in reference picture list L1 having the same distance in display order from the current picture as the reference picture pointed to by the second motion vector in reference picture list L0 may be selected, and so on. The flipped motion vector may be scaled as needed depending on the reference picture in another reference picture list to which the flipped motion vector points.

[0562] In some embodiments, a bi-prediction vector based on the first motion vector and the second motion vector is stored for the first portion of the first partition, where possible, without flipping or scaling one of the first motion vector and the second motion vector (see, e.g., Fig.68 and Fig.69 ), and when it is not possible to generate a bi-predictive motion vector without flipping or scaling one of the first motion vector and the second motion vector, storing a uni-predictive motion vector for the first part of the first partition (see, e.g. Fig.70 and Fig.71 , wherein one of the first motion vector or the second motion vector is stored as a uni-prediction motion vector for a first portion of a first partition.

[0563] In another example, when the first motion vector and the second motion vector are uni-predictive motion vectors pointing to the same reference picture, the first motion vector and the second motion vector may be averaged to generate a uni-predictive motion vector, which is stored as the uni-predictive motion vector for the first portion of the first partition.

[0564] In another example, one of the first motion vector and the second motion vector may be a bi-predictive motion vector and the other may be a uni-predictive motion vector.The bi-predictive motion vector may be stored as a motion vector for the first portion of the first partition.

[0565] In another example, both the first motion vector and the second motion vector may be bi-predictive motion vectors, and a reference picture of the first motion vector may be the same as a reference picture of the second motion vector. The first motion vector and the second motion vector may be averaged to generate a bi-predictive motion vector, which is stored as the bi-predictive motion vector for the first portion of the first partition.

[0566] In another example, both the first motion vector and the second motion vector may be bi-predictive motion vectors, and the reference picture of the first motion vector may be different from the reference picture of the second motion vector. The second motion vector may be scaled to the reference picture of the first motion vector and averaged with the first motion vector to generate an average bi-predictive motion vector, which may be stored as the bi-predictive motion vector for the first portion of the first partition.

[0567] In another example, the first motion vector may be stored as a motion vector of a first portion of a first partition, for example, always or when certain conditions exist. In another example, the second motion vector may be stored as a motion vector of a first portion of a first partition, for example, always or when certain conditions exist. In step S6006, the first partition is encoded or decoded using at least the weighted samples of the first portion of the first partition. For example, the weighted pixel component values ​​(e.g., luminance values, chrominance values) of the samples in the first group of samples and the weighted pixel component values ​​of the corresponding samples in the second group of samples may be combined to determine the pixel component values ​​used in the encoding or decoding process. See, e.g. Figure 47-61 and their corresponding descriptions.

[0568] The disclosed embodiments introduce methods of storing motion vectors for weighted regions (e.g., the first portion of the first partition in the disclosed examples). This can improve the accuracy of motion vector prediction for the next coded partition, which can improve coding efficiency. Note that motion vectors for weighted regions can typically be stored for blocks of four pixels. For example, each sample can be a four-by-four pixel block for which a motion vector is stored. Single-prediction motion vectors or dual-prediction motion vectors for weighted regions can be stored without significantly increasing or not increasing the amount of memory used to store motion vectors for weighted regions.

[0569] Note that the terms "encoding" and "encoding" used in the description of the encoding method and encoding process performed by the image encoding device may be replaced by the term "decoding" for the decoding method and decoding process performed by the image decoding device. Not all of the described processes and elements are always required, and one or more of the described processes may be omitted in some embodiments.

[0570] One or more aspects disclosed herein may be performed by combining at least a portion of other aspects in the present disclosure. In addition, one or more aspects disclosed herein may be performed by combining the partial process indicated in any flow chart according to these aspects, a portion of the configuration of any device, a portion of grammar, etc. with other aspects.

[0571] (Implementation and Application)

[0572] As described in each of the above embodiments, each function or operation block can generally be implemented as, for example, an MPU (microprocessing unit) and a memory. In addition, the process performed by each of the function blocks can be implemented as a program execution unit, such as a processor that reads and executes software (programs) recorded on a recording medium such as a ROM. The software can be distributed. The software can be recorded on various recording media such as semiconductor memories. Note that each function block can also be implemented as hardware (special purpose circuits). Various combinations of hardware and software can be used.

[0573] The processing described in each embodiment can be implemented by integrated processing using a single device (system), or can be implemented by decentralized processing using multiple devices. In addition, the processor that executes the above program can be a single processor or multiple processors. In other words, integrated processing can be performed, or decentralized processing can be performed.

[0574] The embodiments of the present disclosure are not limited to the above exemplary embodiments; various modifications may be made to the exemplary embodiments, and the results thereof are also included in the scope of the embodiments of the present disclosure.

[0575] Next, application examples of the motion picture encoding method (image encoding method) and the motion picture decoding method (image decoding method) described in each of the above embodiments, as well as various systems for implementing the application examples, will be described. Such a system may be characterized by including an image encoder using an image encoding method, an image decoder using an image decoding method, or an image encoder-decoder including both an image encoder and an image decoder. Other configurations of such a system may be modified as appropriate.

[0576] (Usage example)

[0577] Fig.73The overall configuration of a content providing system ex100 suitable for implementing content distribution services is shown. An area where communication services are provided is divided into cells of desired sizes, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations in the shown example, are located in each cell.

[0578] In the content providing system ex100, devices including a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via an Internet service provider ex102 or a communication network ex104 and base stations ex106 to ex110. The content providing system ex100 can combine and connect any combination of the above devices. In various implementations, the devices can be directly or indirectly connected together via a telephone network or near field communication rather than via base stations ex106 to ex110. In addition, a streaming media server ex103 can be connected to devices including a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 via, for example, the Internet ex101. The streaming media server ex103 can also be connected to a terminal in a hotspot such as in an airplane ex117 via a satellite ex116.

[0579] Note that wireless access points or hotspots may be used instead of base stations ex106 to ex110. The streaming server ex103 may be directly connected to the communication network ex104 instead of through the Internet ex101 or the Internet service provider ex102, and may be directly connected to the aircraft ex117 instead of through the satellite ex116.

[0580] The camera ex113 is a device capable of capturing still images and videos, such as a digital camera. The smartphone ex115 is a smartphone device, a cellular phone, or a Personal Handyphone System (PHS) phone that can operate under mobile communication system standards of 2G, 3G, 3.9G, and 4G systems and the next-generation 5G system.

[0581] The household appliance ex114 is, for example, a refrigerator or a device included in a household fuel cell cogeneration system.

[0582] In the content providing system ex100, a terminal including an image and / or video capturing function can perform real-time streaming, for example, by connecting to a streaming server ex103 via, for example, a base station ex106. When performing real-time streaming, a terminal (e.g., a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or a terminal in an airplane ex117) can perform the encoding process described in the above embodiments on still images or video content captured by a user via the terminal, can multiplex video data obtained via encoding and audio data obtained by encoding audio corresponding to the video, and can send the obtained data to the streaming server ex103. In other words, the terminal functions as an image encoder according to one aspect of the present disclosure.

[0583] The streaming media server ex103 streams the sent content data to the client that requested the stream. Examples of the client include a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and a terminal in an airplane ex117, which are capable of decoding the above-mentioned encoded data. The device that receives the streamed data can decode and reproduce the received data. In other words, each of the devices can act as an image decoder according to one aspect of the present disclosure.

[0584] (Decentralized processing)

[0585] Streaming media server ex103 can be implemented as multiple servers or computers that divide tasks such as processing, recording and streaming of data between them. For example, streaming media server ex103 can be implemented as a content delivery network (CDN), which streams content via a network connecting multiple edge servers located around the world. In CDN, edge servers physically close to clients can be dynamically assigned to clients. Content is cached and streamed to edge servers to reduce loading time. For example, in an event where some type of error or connectivity changes due to a traffic peak, data can be stably streamed at high speed, because the affected part of the network can be avoided in the following ways: for example, processing is divided between multiple edge servers, or streaming responsibilities are switched to different edge servers and streaming continues.

[0586] Decentralization is not limited to dividing processing for streaming; the encoding of the captured data can be distributed between terminals on the server side, or performed by the terminal, or both. In one example, in a typical encoding, the processing is performed in two loops. The first loop is used to detect the complexity of the image on a frame-by-frame or scene-by-scene basis, or to detect the encoding load. The second loop is used for processing to maintain image quality and improve encoding efficiency. For example, it is possible to reduce the processing responsibility of the terminal and improve the quality and encoding efficiency of the content by causing the terminal to perform a first loop of encoding, and causing the server side receiving the content to perform a second loop of encoding. In such a case, when a decoding request is received, it is possible that the encoded data generated by the first loop performed on one terminal is received and reproduced on another terminal in approximately real time. This makes it possible to achieve smooth, real-time streaming.

[0587] In another example, the camera ex113 or the like extracts a feature quantity (a certain amount of feature or characteristic) from an image, compresses data related to the feature quantity into metadata, and transmits the compressed metadata to a server. For example, the server determines the importance of an object based on the feature quantity, and changes the quantization accuracy accordingly to perform compression suitable for the meaning (or content importance) of the image. The feature quantity data is particularly effective in improving the accuracy and efficiency of motion vector prediction during the second compression performed by the server. In addition, the terminal can process encoding with a relatively low processing load such as variable length coding (VLC), and encoding with a relatively high processing load such as context adaptive binary arithmetic coding (CABAC) can be processed by the server.

[0588] In another example, there is a situation where multiple terminals, such as in a stadium, a shopping mall, or a factory, capture multiple videos of approximately the same scene. In such a case, for example, decentralized encoding can be achieved by dividing the processing tasks between multiple terminals that capture the video and other terminals and servers that do not capture the video on a per-unit basis. The unit can be, for example, a group of pictures (GOP), a picture, or a tile obtained after dividing a picture. This can reduce loading time and achieve closer to real-time streaming.

[0589] Since the videos have substantially the same scenes, the server can perform management and / or instructions so that the videos captured by the terminal can be cross-referenced. In addition, the server can receive encoded data from the terminal, change the reference relationship between the data items, or correct or replace the pictures itself, and then perform encoding. This makes it possible to generate streams with improved quality and efficiency for individual data items.

[0590] In addition, the server can stream the video data after performing code conversion to convert the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (e.g., VP9), can convert H.264 to H.265, and so on.

[0591] In this way, encoding can be performed by a terminal or one or more servers. Therefore, although the device performing encoding is referred to as a "server" or a "terminal" in the following description, some or all of the processes performed by the server can be performed by the terminal, and similarly, the processing performed by the terminal can be performed by the server. This also applies to the decoding process.

[0592] (3D, multi-angle)

[0593] The use of images or videos combined from images or videos of different scenes captured simultaneously or images or videos of the same scene captured from different angles through multiple terminals (e.g., camera ex113 and / or smartphone ex115) has increased. Videos captured by the terminals can be combined based on, for example, relative positional relationships obtained separately between the terminals or areas with matching feature points in the videos.

[0594] In addition to encoding two-dimensional motion pictures, the server can also encode still images automatically or at a user-specified time point based on scene analysis of the motion pictures, and send the encoded still images to the receiving terminal. In addition, when the server is able to obtain the relative positional relationship between the video capture terminals, in addition to the two-dimensional motion pictures, the server can also generate the three-dimensional geometry of the scene based on videos of the same scene captured from different angles. The server can encode the three-dimensional data generated from, for example, a point cloud, respectively, and based on the results of identifying or tracking a person or object using the three-dimensional data, the server can select or reconstruct and generate a video to be sent to the receiving terminal from the videos captured by multiple terminals.

[0595] This allows the user to enjoy the scene by freely selecting the video corresponding to the video capture terminal, and allows the user to enjoy the content obtained by extracting the video at a selected viewpoint from the three-dimensional data reconstructed from multiple images or videos. In addition, like the video, the sound can be recorded from relatively different angles, and the server can multiplex the audio of a specific angle or space with the corresponding video and send the multiplexed video and audio.

[0596] In recent years, content consisting of the real world and the virtual world, such as virtual reality (VR) and augmented reality (AR) content, has also become popular. In the case of VR images, the server can create images from both the left eye and the right eye viewpoints, and perform encoding that allows reference between the two viewpoint images, such as multi-view coding (MVC), or the image can be encoded as a separate stream without reference. When the image is decoded into a separate stream, the stream can be synchronized when reproduced so as to recreate a virtual three-dimensional space according to the user's viewpoint.

[0597] In the case of an AR image, the server can superimpose virtual object information existing in the virtual space on the camera information representing the real space based on the three-dimensional position or movement from the user's perspective. The decoder can acquire or store virtual object information and three-dimensional data, generate a two-dimensional image based on the movement from the user's perspective, and then generate superimposed data by seamlessly connecting the images. Alternatively, in addition to the request for virtual object information, the decoder can also send the movement from the user's perspective to the server. The server can generate superimposed data based on the three-dimensional data stored in the server according to the received movement, and encode the generated superimposed data and stream it to the decoder. Note that the superimposed data generally includes an alpha value indicating transparency in addition to the RGB value, and the server sets the alpha value of the part other than the object generated from the three-dimensional data to, for example, 0, and can be encoded while these parts are transparent. Alternatively, the server can set the background to a determined RGB value (e.g., a chroma key) and generate data that sets the area outside the object as the background. The determined RGB value can be predetermined.

[0598] Decoding of similarly streamed data can be performed by a client (e.g., a terminal) on the server side, or divided between them. In one example, a terminal can send a reception request to a server, the requested content can be received and decoded by another terminal, and the decoded signal can be sent to a device with a display. Regardless of the processing power of the communication terminal itself, high image quality data can be reproduced by decentralized processing and appropriate selection of content. In another example, for example, when a TV is receiving large-size image data, areas of a picture such as tiles obtained by dividing a picture can be decoded and displayed on a personal terminal or on the terminals of one or more viewers of the TV. This allows viewers to share a large picture view, and each viewer can check their assigned area or view the area more closely.

[0599] In the case where multiple wireless connections can be made indoors, outdoors, at close range, medium range, and long range, content can be received seamlessly using streaming system standards such as MPEG-DASH. Users can switch between data in real time while freely selecting decoders or display devices including user terminals, displays arranged indoors or outdoors, etc. In addition, using information such as the user's location, decoding can be performed while switching which terminal handles decoding and which terminal handles the display of content. When the user is on the way to the destination, this makes it possible to draw and display information on the wall of a nearby building or on a part of the ground in which a device capable of displaying content is embedded. In addition, the bit rate of the received data can also be switched based on the accessibility of the encoded data on the network, such as when the encoded data is cached on a server that can be quickly accessed from the receiving terminal, or when the encoded data is copied to the edge server in the content delivery service.

[0600] (Scalable Coding)

[0601] Reference will be made to Fig.74 The scalable stream shown in is used to describe the switching of the content, which is compressed and encoded by implementing the motion picture encoding method described in the above embodiment. The server may have a configuration in which the content is switched while utilizing the temporal and / or spatial scalability of the stream, which is achieved by dividing and encoding the layers, such as Fig.74 As shown. Note that there can be multiple individual streams with the same content but different qualities. In other words, by determining the layer to be decoded based on internal factors such as the processing power on the decoder side and external factors such as the communication bandwidth, the decoder side can freely switch between low-resolution content and high-resolution content while decoding. For example, in the case where a user wants to continue watching a video that the user previously watched on the smartphone ex115 while on the move, for example, using a device such as a TV connected to the Internet at home, the device can simply decode the same stream up to a different layer, which reduces the load on the server side.

[0602] In addition, in addition to the above configuration (where scalability is achieved by encoding pictures by layer), and the enhancement layer is located above the base layer, the enhancement layer may also include metadata based on, for example, statistical information about the image. The decoder side can generate content with high image quality by performing super-resolution imaging on the pictures in the base layer based on the metadata. Super-resolution imaging can improve the signal-to-noise ratio while maintaining and / or increasing the resolution. The metadata includes information for identifying linear or nonlinear filter coefficients used in super-resolution processing, or information for identifying parameter values ​​of filtering processing, machine learning, or least squares method used in super-resolution processing.

[0603] Alternatively, a configuration may be provided in which a picture is segmented into, for example, tiles according to the meaning of, for example, an object in the image. On the decoder side, only a partial area is decoded by selecting the tile to be decoded. In addition, by storing the properties of the object (person, car, ball, etc.) and the position of the object in the video (coordinates in the same image) as metadata, the decoder side can identify the position of the desired object based on the metadata and determine which tile or tiles contain the object. For example, Fig.75 As shown, a data storage structure different from pixel data can be used to store metadata, such as SEI (Supplemental Enhancement Information) messages in HEVC. The metadata indicates, for example, the position, size, or color of the main object.

[0604] The metadata can be stored in units of multiple pictures, such as streams, sequences, or random access units. The decoder side can obtain, for example, the time when a specific person appears in the video, and by fitting the time information with the picture unit information, the picture in which the object exists can be identified, and the position of the object in the picture can be determined.

[0605] (Web Optimization)

[0606] Fig.76 An example of a display screen of a web page on the computer ex111 is shown, for example. Fig.77 FIG. 4 shows an example of a display screen of a web page on a smartphone ex115. Fig.76 and Fig.77 As shown in , a web page may include multiple image links as links to image content, and the appearance of the web page may differ depending on the device used to view the web page. When multiple image links are visible on the screen, until the user explicitly selects the image link, or until the image link is located in the approximate center of the screen, or until the entire image link fits the screen, the display device (decoder) may display a still image included in the content or an I picture as the image link; may use multiple still images or I pictures to display a video such as an animated gif; or may receive only the base layer, and decode and display the video.

[0607] When the user selects an image link, the display device performs decoding while, for example, giving the highest priority to the base layer. Note that if there is information in the HTML code of the web page indicating that the content is scalable, the display device can decode up to the enhancement layer. In addition, in order to ensure real-time reproduction, before making a selection or when the bandwidth is strictly limited, the display device can reduce the delay between the time point when the leading picture is decoded and the time point when the decoded picture is displayed (i.e., the delay between the start of content decoding and the display of the content) by decoding and displaying only forward reference pictures (I pictures, P pictures, forward reference B pictures). In addition, the display device can intentionally ignore the reference relationship between pictures, and roughly decode all B and P pictures as forward reference pictures, and then perform regular decoding as the number of pictures received over a period of time increases.

[0608] (Autonomous Driving)

[0609] When sending and receiving still images or video data (e.g., two-dimensional or three-dimensional map information for autonomous or assisted driving cars), the receiving terminal may receive information about, for example, weather or road construction as metadata in addition to image data belonging to one or more layers, and associate the metadata with the image data when decoding. Note that the metadata may be allocated by layer, and may alternatively be multiplexed with the image data alone.

[0610] In this case, since the car, drone, airplane, etc. including the receiving terminal is movable, the receiving terminal can seamlessly receive and perform decoding by transmitting information indicating the location of the receiving terminal while switching between base stations among base stations ex106 to ex110. In addition, the receiving terminal can dynamically select the degree of receiving metadata or the degree of updating, for example, map information, according to the user's selection, the user's condition, and / or the bandwidth of the connection.

[0611] In the content providing system ex100, the client can receive, decode, and reproduce the encoded information sent by the user in real time.

[0612] (Streaming individual content)

[0613] In the content providing system ex100, in addition to high image quality, long content distributed by the video distribution entity, unicast or multicast streams with low image quality, and short content from individuals are also possible. The popularity of such content from individuals is likely to increase further. The server can first perform editing processing on the content before the encoding process in order to refine the individual content. For example, this can be achieved using the following configuration.

[0614] When capturing video or image content in real time, or after capturing and accumulating the content, the server performs recognition processing based on the original data or the encoded data, such as capture error processing, scene search processing, meaning analysis, and / or object detection processing. Then, based on the results of the recognition processing, the server edits the content when prompted or automatically, examples of which include: corrections such as focus and / or motion blur correction; removal of low-priority scenes, such as scenes with lower brightness compared to other pictures, or scenes that are out of focus; object edge adjustment; and hue adjustment. The server encodes the edited data based on the results of the editing. It is well known that videos that are too long will result in a decrease in the number of views. Therefore, in order to keep the content within a specific length equivalent to the length of the original video, in addition to the low-priority scenes mentioned above, the server can also automatically crop out scenes with low motion based on the image processing results. Alternatively, the server can generate and encode a video summary based on the analysis results of the meaning of the scene.

[0615] In some cases, individual content may include content that infringes copyright, moral rights, portrait rights, etc. Such instances may bring disadvantageous situations to the creator, such as when the content is shared beyond the scope that the creator intends. Thus, before encoding, the server can, for example, edit the image so that the faces of people on the periphery of the screen are blurred or the interior of the house is blurred. In addition, the server can be configured to identify the faces of people other than registered people in the image to be encoded, and when such faces appear in the image, a mosaic filter can be applied to, for example, the face of the person. Alternatively, as pre-processing or post-processing for encoding, the user can specify an image area including a person or an area of ​​the background to be processed for copyright reasons. The server can process the specified area by, for example, replacing the area with a different image or blurring the area. If the area includes a person, the person can be tracked in a moving picture, and the head area of ​​the person can be replaced with another image as the person moves.

[0616] Due to the need to watch individually generated content that tends to have a smaller data size in real time, the decoder can first receive the base layer as the highest priority and perform decoding and reproduction, but this may vary depending on the bandwidth. When the content is reproduced twice or more times, for example when the decoder receives the enhancement layer during the decoding and reproduction of the base layer and loops the reproduction, the decoder can reproduce a video with high image quality including the enhancement layer. If the stream is encoded using such scalable coding, the video may be of lower quality when it is not selected or at the beginning of the video, but it can provide an experience in which the image quality of the stream gradually improves in an intelligent manner. This is not limited to scalable coding; a single stream can be configured by reproducing a low-quality stream from the first time and encoding a second stream using the first stream as a reference, thereby providing the same experience.

[0617] (Other implementation and application examples)

[0618] The encoding and decoding can be performed by an LSI (Large Scale Integrated Circuit) ex500 (see Fig.73 ). LSI ex500 may be configured by a single chip or a plurality of chips. Software for encoding and decoding motion pictures may be integrated into a certain type of recording medium (e.g., a CD-ROM, a floppy disk, or a hard disk), and for example, the computer ex111 may read the recording medium, and encoding and decoding may be performed using the software. In addition, when the smartphone ex115 is equipped with a camera, video data acquired by the camera may be transmitted. In this case, the video data may be encoded by LSI ex500 included in the smartphone ex115.

[0619] Note that LSI ex500 can be configured to download and activate applications. In this case, the terminal first determines whether it is compatible with the scheme used to encode the content, or whether it can perform a specific service. When the terminal is incompatible with the encoding scheme of the content, or when the terminal cannot perform a specific service, the terminal can first download a codec or application software, and then obtain and reproduce the content.

[0620] In addition to the example of the content providing system ex100 using the Internet ex101, at least the moving picture encoder (image encoder) or the moving picture decoder (image decoder) described in the above embodiments can be implemented in a digital broadcasting system. The same encoding process and decoding process can be applied to, for example, transmitting and receiving broadcast radio waves superimposed with multiplexed audio and video data using a satellite, even though this is for multicast, unicast is easier for the content providing system ex100.

[0621] (Hardware Configuration)

[0622] Fig.78 Shows Fig.73 More details of the smartphone ex115 shown in. Fig.79An example of a configuration of a smartphone ex115 is shown. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves to and from the base station ex110, a camera ex465 capable of capturing video and still images, and a display ex458 for displaying decoded data such as video captured by the camera ex465 and video received by the antenna ex450. The smartphone ex115 also includes a user interface ex466 such as a touch panel, an audio output unit ex457 such as a speaker for outputting voice or other audio, an audio input unit ex456 such as a microphone for audio input, a memory ex467 capable of storing decoded data such as captured video or still images, recorded audio, received video or still images, mails, and decoded data, and a slot ex464 which is an interface of a SIM ex468 for authorizing access to a network and various data. Note that an external memory may be used in place of the ex467.

[0623] A main controller ex460 that can comprehensively control the display ex458 and the user interface ex466, a power supply circuit ex461, a user interface input controller ex462, a video signal processor ex455, a camera interface ex463, a display controller ex459, a modulator / demodulator ex452, a multiplexer / demultiplexer ex453, an audio signal processor ex454, a slot ex464, and a memory ex467 are connected via a bus ex470.

[0624] When the user turns on the power button of the power circuit ex461, the smartphone ex115 is powered on to enter an operable state, and power is supplied to each component from the battery pack.

[0625] The smart phone ex115 performs processing such as calling and data transmission based on control performed by the main controller ex460 including a CPU, ROM, and RAM. When making a call, the audio signal recorded by the audio input unit ex456 is converted into a digital audio signal by the audio signal processor ex454, and the digital audio signal is spread spectrum processed and digital-to-analog converted by the modulator / demodulator ex452, and frequency converted by the transmitter / receiver ex451, and the resulting signal is transmitted via the antenna ex450. The received data is amplified, frequency converted and analog-to-digital converted, and inverse spread spectrum processed by the modulator / demodulator ex452, converted into an analog audio signal by the audio signal processor ex454, and then output from the audio output unit ex457. In the data transmission mode, for example, based on the operation of the user interface ex466 of the main body, text, still images, or video data can be transmitted via the user interface input controller ex462 under the control of the main controller ex460. Similar transmission and reception processing is performed. In the data transmission mode, when transmitting video, still image, or video and audio, the video signal processor ex455 compresses and encodes the video signal stored in the memory ex467 or the video signal input from the camera ex467 via the motion picture encoding method described in the above embodiments, and transmits the encoded video data to the multiplexer / demultiplexer ex453. The audio signal processor ex454 encodes the audio signal recorded by the audio input unit ex456 while the camera ex465 is capturing video or still image, and transmits the encoded audio data to the multiplexer / demultiplexer ex453. The multiplexer / demultiplexer ex453 multiplexes the encoded video data and the encoded audio data using a determined scheme, modulates and converts the data using a modulator / demodulator (modulator / demodulator circuit) ex452 and a transmitter / receiver ex451, and transmits the result via the antenna ex450. The determined scheme may be predetermined.

[0626] For example, when a video attached to an email or a chat or a video linked from a web page is received, in order to decode the multiplexed data received via the antenna ex450, the multiplexer / demultiplexer ex453 demultiplexes the multiplexed data to divide the multiplexed data into a bit stream of video data and a bit stream of audio data, supplies the encoded video data to the video signal processor ex455 via the synchronous bus ex470, and supplies the encoded audio data to the audio signal processor ex454 via the synchronous bus ex470. The video signal processor ex455 decodes the video signal using a moving picture decoding method corresponding to the moving picture encoding method described in the above embodiment, and the video or still image included in the linked moving picture file is displayed on the display ex458 via the display controller ex459. The audio signal processor ex454 decodes the audio signal and outputs the audio from the audio output unit ex457. Since real-time streaming is becoming more and more popular, the reproduction of audio may be socially inappropriate in some cases depending on the user's environment. Thus, as an initial value, a configuration in which only video data is reproduced, ie, audio signals are not reproduced, may be preferred; audio may be synchronized and reproduced only when an input is received (eg, when a user clicks on video data).

[0627] Although the smartphone ex115 is used in the above example, other implementations are conceivable: a transceiver terminal including both an encoder and a decoder; a transmitter terminal including only an encoder; a receiver terminal including only a decoder. In the description of the digital broadcasting system, an example is given in which multiplexed data obtained by multiplexing video data with audio data is received or transmitted. However, the multiplexed data may be video data multiplexed with data other than audio data (e.g., text data related to the video). Furthermore, video data itself may be received or transmitted instead of the multiplexed data.

[0628] Although the main controller ex460 including the CPU is described as controlling the encoding or decoding process, various terminals generally include a GPU. Thus, a configuration is acceptable in which a large area is processed at once by utilizing the performance capabilities of the GPU via a memory shared by the CPU and the GPU or a memory including addresses managed to allow common use of the CPU and the GPU. This can shorten the encoding time, maintain the real-time nature of the stream, and reduce latency. In particular, processing related to motion estimation, deblocking filtering, sample adaptive offset (SAO), and transform / quantization can be effectively performed all at once by the GPU instead of the CPU in units of pictures.

Claims

1. An image encoder, comprising: Circuit; as well as a memory coupled to the circuit; The circuit performs the following operations during operation: Encode the first partition of the current picture.

2. An image decoder, comprising: Circuit; as well as a memory coupled to the circuit; The circuit performs the following operations during operation: Decode the first partition of the current picture.

3. A non-transitory computer-readable medium storing a bitstream, the bitstream comprising information and instructions, the instructions causing a processor to perform a partition decoding process.