Image encoders and image decoders

Abstract

Provided are an image encoder and an image decoder. The image encoder comprises: a circuit; and a memory coupled to the circuit; wherein the circuit performs an operation on an image block during operation.

Description

[0001] This application was submitted on August 10, 2018, with Chinese patent application number 201880054253.3 (international application number PCT / JP2018 / 030064), and is a divisional application of a patent application with the invention name “Image encoder, image decoder, image encoding method and image decoding method”. Technical Field

[0002] The present disclosure relates to video coding, and in particular to video coding and decoding systems, components, and methods for performing an inter-frame prediction function of constructing a current block based on a reference frame or an intra-frame prediction function of constructing a current block based on an encoded / decoded reference block in a current frame. Background Art

[0003] With the development of video coding technology from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Codec), there is always a need to improve and optimize video coding technology to handle the growing amount of digital video data in various applications. The present disclosure relates to further advancements, improvements, and optimizations in video coding, in particular, in conjunction with an inter-frame prediction function or an intra-frame prediction function, which divides an image block into a plurality of partitions, the plurality of partitions including at least a second partition and a first partition having a non-rectangular shape (e.g., a triangle). Summary of the invention

[0004] An image encoder is provided, comprising: a circuit; and a memory coupled to the circuit; wherein the circuit performs an operation on an image block during operation.

[0005] An image decoder is provided, comprising: a circuit; and a memory coupled to the circuit; wherein the circuit performs an operation on an image block during operation.

[0006] According to one aspect, an image encoder is provided, comprising a circuit and a memory coupled to the circuit. The circuit performs a boundary smoothing operation along a boundary between a second partition segmented from an image block and a first partition having a non-rectangular shape in operation. The boundary smoothing operation includes: using information of the first partition, making a first prediction of a first value of a group of pixels of the first partition along the boundary; and using information of the second partition, making a second prediction of a second value of a group of pixels of the first partition along the boundary; weighting the first value and the second value; and encoding the first partition using the weighted first value and the weighted second value.

[0007] Some implementations of the embodiments of the present disclosure can improve coding efficiency, can simply be the encoding / decoding process, can accelerate the speed of the encoding / decoding process, and can efficiently select appropriate components / operations used in encoding and decoding, such as appropriate filters, block sizes, motion vectors, reference pictures, reference blocks, etc.

[0008] Other benefits and advantages of the disclosed embodiments will become apparent through the specification and drawings. Benefits and / or advantages may be obtained individually through various embodiments and features of the specification and drawings, which need not all be provided to obtain one or more of such benefits and / or advantages.

[0009] It should be noted that the general or specific embodiments may be implemented as a system, a method, an integrated circuit, a computer program, a storage medium, or any selective combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram showing a functional configuration of an encoder according to an embodiment.

[0011] Figure 2 An example of block segmentation is shown.

[0012] Figure 3 is a table indicating transform basis functions for various transform types.

[0013] Figure 4A An example of a filter shape used in an ALF (Adaptive Loop Filter) is shown.

[0014] Figure 4B Another example of a filter shape used in an ALF is shown.

[0015] Figure 4C Another example of a filter shape used in an ALF is shown.

[0016] Figure 5A 67 intra prediction modes used in the example of intra prediction are shown.

[0017] Figure 5B is a flowchart showing one example of a predicted image correction procedure performed in an OBMC (Overlapped Block Motion Compensation) process.

[0018] Figure 5C is a conceptual diagram showing one example of a predicted image correction process performed in the OBMC process.

[0019] Figure 5D is a flowchart showing one example of FRUC (Frame Rate Up Conversion) processing.

[0020] Figure 6 An example of pattern matching between two blocks along a motion trajectory (bidirectional matching) is shown.

[0021] Figure 7 An example of pattern matching (template matching) between a template in a current picture and a block in a reference picture is shown.

[0022] Figure 8 A model assuming uniform linear motion is shown.

[0023] Fig. 9A An example of deriving the motion vector of each sub-block based on the motion vectors of neighboring blocks is shown.

[0024] Fig. 9B One example of a process for deriving motion vectors in merge mode is shown.

[0025] Fig. 9C is a conceptual diagram showing an example of DMVR (Dynamic Motion Vector Refresh) processing.

[0026] Fig.9D One example of a predicted image generation method using a brightness correction process performed by LIC (Local Illumination Compensation) processing is shown.

[0027] Fig.10 is a block diagram showing a functional configuration of a decoder according to an embodiment.

[0028] Fig.11 Detailed description is a flowchart showing an overall processing flow of dividing an image block into a plurality of partitions including at least a second partition and a first partition having a non-rectangular shape (eg, a triangle) and performing further processing according to one embodiment.

[0029] Fig.12 Two exemplary methods of partitioning an image block into second partitions (also having a non-rectangular shape in the illustrated example) and first partitions having a non-rectangular shape (eg, a triangle) are shown.

[0030] Fig.13 An example of a boundary smoothing process is shown, which includes weighting a first value of a boundary pixel based on a first partition prediction and a second value of a boundary pixel based on a second partition prediction.

[0031] Fig.14 Three further examples of a boundary smoothing process are shown that involve weighting first values ​​of boundary pixels based on a first partition prediction and weighting second values ​​of boundary pixels based on a second partition prediction.

[0032] Fig.15is a table of sampling parameters ("first index values") and the information sets respectively encoded by the parameters.

[0033] Fig.16 It is a table showing the binarization of parameters (index values).

[0034] Fig.17 is a flow chart illustrating a process of segmenting an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0035] Fig.18 An example of partitioning an image block into a plurality of partitions is shown, the plurality of partitions comprising a second partition and a first partition having a non-rectangular shape (a triangle in the example shown).

[0036] Fig.19 A further example of partitioning the image block into a plurality of partitions is shown, the plurality of partitions comprising a second partition and a first partition having a non-rectangular shape, which in the example shown is a polygon having at least five sides and corners.

[0037] Fig. 20 is a flow chart illustrating a boundary smoothing process including weighting a first value of boundary pixels based on a first partition prediction and weighting a second value of boundary pixels based on a second partition prediction.

[0038] Fig.21A An example of a boundary smoothing process is shown, in which boundary pixels whose first values ​​are to be weighted are predicted based on a first partition and whose second values ​​are to be weighted are predicted based on a second partition.

[0039] Fig.21B An example of a boundary smoothing process is shown, in which boundary pixels whose first values ​​are to be weighted are predicted based on a first partition and whose second values ​​are to be weighted are predicted based on a second partition.

[0040] Fig. 21C An example of a boundary smoothing process is shown, in which boundary pixels whose first values ​​are to be weighted are predicted based on a first partition and whose second values ​​are to be weighted are predicted based on a second partition.

[0041] Fig.21D An example of a boundary smoothing process is shown, in which boundary pixels whose first values ​​are to be weighted are predicted based on a first partition and whose second values ​​are to be weighted are predicted based on a second partition.

[0042] Fig. 22The present invention is a flowchart showing a method performed on an encoder side to partition an image block into a plurality of partitions based on partition parameters indicating the partition and write one or more parameters including the partition parameters into a bitstream in entropy coding, wherein the plurality of partitions include a second partition and a first partition having a non-rectangular shape.

[0043] Fig.23 A flowchart showing a method for parsing one or more parameters from a bitstream performed on a decoder side, wherein the parameters include a method for indicating partitioning an image block into multiple partitions based on a partition parameter and decoding a first partition and a second partition, wherein the one or more parameters include a partition parameter indicating partitioning the image block into multiple partitions, wherein the multiple partitions include a second partition and a first partition having a non-rectangular shape.

[0044] Fig.24 It is a table of sample partition parameters ("first index values") and a set of information that can be jointly encoded by the partition parameters, respectively, wherein the sample partition parameters respectively indicate that the image block is divided into multiple partitions, and the multiple partitions include a second partition and a first partition having a non-rectangular shape.

[0045] Fig.25 is a table of sample combinations of a first parameter and a second parameter, wherein one of the first parameter and the second parameter is a partition parameter indicating partitioning of the image block into a plurality of partitions, the plurality of partitions including the second partition and the first partition having a non-rectangular shape.

[0046] Fig.26 The overall configuration of a content providing system for realizing a content distribution service is shown.

[0047] Fig. 27 An example of a coding structure in scalable coding is shown.

[0048] Fig.28 An example of a coding structure in scalable coding is shown.

[0049] Fig.29 An example of a display screen of a web page is shown.

[0050] Fig.30 An example of a display screen of a web page is shown.

[0051] Fig.31 An example of a smartphone is shown.

[0052] Fig.32 is a block diagram showing a configuration example of a smartphone. DETAILED DESCRIPTION

[0053] According to one aspect, an image encoder is provided, comprising a circuit and a memory coupled to the circuit. The circuit performs the following operations in operation: partitioning an image block into a plurality of partitions, the plurality of partitions comprising a second partition and a first partition having a non-rectangular shape; predicting a first motion vector of the first partition and a second motion vector of the second partition; and encoding the first partition using the first motion vector and encoding the second partition using the second motion vector.

[0054] According to another aspect, the second partition has a non-rectangular shape. According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from the group consisting of a triangle, a trapezoid, and a polygon having at least five sides and angles.

[0055] According to another aspect, the prediction includes selecting a first motion vector from a first set of motion vector candidates and selecting a second motion vector from a second set of motion vector candidates. For example, the first set of motion vector candidates may include motion vectors of partitions adjacent to the first partition, and the second set of motion vector candidates may include motion vectors of partitions adjacent to the second partition. The partitions adjacent to the first partition and the partitions adjacent to the second partition may be outside the image block from which the first partition and the second partition are segmented. The adjacent partitions may be one or both of spatially adjacent partitions and temporally adjacent partitions. The first set of motion vector candidates may be the same as or different from the second set of motion vector candidates.

[0056] According to another aspect, the prediction includes: selecting a first motion vector candidate from a first group of motion vector candidates and deriving the first motion vector by adding a first motion vector difference to the first motion vector candidate, and selecting a second motion vector candidate from a second group of motion vector candidates and deriving the second motion vector by adding the second motion vector difference to the second motion vector candidate.

[0057] According to another aspect, an image encoder is provided, comprising: a partitioner, which receives and partitions an original picture into blocks during operation; and an adder, which receives blocks from the partitioner and predictions from a prediction controller during operation and subtracts each prediction from its corresponding block to output a residual; a transformer, which performs a transformation on the residual output from the adder during operation to output a transformation coefficient; a quantizer, which quantizes the transformation coefficient to generate a quantized transformation coefficient during operation; an entropy encoder, which encodes the quantized transformation coefficient to generate a bitstream during operation; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor and a memory, wherein the inter-frame predictor generates a prediction for a current block based on a reference block in an encoded reference picture during operation, and the intra-frame predictor generates a prediction for the current block based on an encoded reference block in a current picture during operation. The prediction controller operates to divide a block into a plurality of partitions, the plurality of partitions including a first partition having a non-rectangular shape and a second partition; predict a first motion vector for the first partition and a second motion vector for the second partition; and encode the first partition using the first motion vector and encode the second partition using the second motion vector.

[0058] According to another aspect, an image encoding method is provided, which generally includes three steps: dividing an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; and predicting a first motion vector of the first partition and a second motion vector of the second partition; encoding the first partition using the first motion vector and encoding the second partition using the second motion vector.

[0059] According to another aspect, an image decoder is provided, comprising a circuit and a memory coupled to the circuit. The circuit performs in operation: partitioning an image block into a plurality of partitions, the plurality of partitions comprising a second partition and a first partition having a non-rectangular shape; predicting a first motion vector of the first partition and a second motion vector of the second partition; decoding the first partition using the first motion vector, and decoding the second partition using the second motion vector.

[0060] According to another aspect, the second partition has a non-rectangular shape. According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from the group consisting of a triangle, a trapezoid, and a polygon having at least five sides and angles.

[0061] According to another aspect, an image decoder is provided, comprising: an entropy decoder, which receives and decodes an encoded bitstream in operation to obtain quantized transform coefficients; and an inverse quantizer and a transformer, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder, which in operation adds the residual output from the inverse quantizer and the transformer to the prediction output from the prediction controller to reconstruct the block; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor and a memory, wherein the inter-frame predictor generates a prediction of a current block based on a reference block in a decoded reference picture in operation, and the intra-frame predictor generates a prediction of the current block based on a decoded reference block in the current picture in operation. The prediction controller divides the image block into a plurality of partitions in operation, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; predicts a first motion vector of the first partition and a second motion vector of the second partition; decodes the first partition using the first motion vector, and decodes the second partition using the second motion vector.

[0062] According to another aspect, an image decoding method is provided, which generally includes three steps: dividing an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; and predicting a first motion vector of the first partition and a second motion vector of the second partition; using the first motion vector to decode the first partition, and using the second motion vector to decode the second partition.

[0063] According to one aspect, an image encoder is provided, comprising a circuit and a memory coupled to the circuit. The circuit performs a boundary smoothing operation along a boundary between a second partition segmented from an image block and a first partition having a non-rectangular shape in operation. The boundary smoothing operation includes: using information of the first partition, making a first prediction of a first value of a group of pixels of the first partition along the boundary; and using information of the second partition, making a second prediction of a second value of a group of pixels of the first partition along the boundary; weighting the first value and the second value; and encoding the first partition using the weighted first value and the weighted second value.

[0064] According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from the group consisting of a triangle, a trapezoid, and a polygon having at least five sides and angles. According to yet another aspect, the second partition has a non-rectangular shape.

[0065] According to another aspect, at least one of the first prediction and the second prediction is an inter-prediction process that predicts the first value and the second value based on a reference partition in an encoded reference picture. The inter-prediction process may predict a first value of pixels of a first partition including a group of pixels, and may predict a second value of a group of pixels of only the first partition.

[0066] According to another aspect, at least one of the first prediction and the second prediction is an intra prediction process that predicts the first value and the second value based on an encoded reference partition in the current picture.

[0067] According to another aspect, the prediction method used in the first prediction is different from the prediction method used in the second prediction.

[0068] According to another aspect, the number of the set of pixels for which the first value and the second value are predicted in each row or column is an integer. For example, when the number of the set of pixels in each row or column is four, weights 1 / 8, 1 / 4, 3 / 4, and 7 / 8 may be applied to the first values ​​of the four pixels in the set, respectively, and weights 7 / 8, 3 / 4, 1 / 4, and 1 / 8 may be applied to the second values ​​of the four pixels in the set, respectively. As another example, when the number of the set of pixels in each row or column is two, weights 1 / 3 and 2 / 3 may be applied to the first values ​​of the two pixels in the set, respectively, and weights 2 / 3 and 1 / 3 may be applied to the second values ​​of the two pixels in the set, respectively.

[0069] According to another aspect, the weights may be integer values ​​or may be fractional values.

[0070] According to another aspect, an image encoder is provided, the image encoder comprising: a partitioner, which receives and partitions an original picture into blocks in operation; and an adder, which receives blocks from the partitioner and predictions from a prediction controller in operation, and subtracts each prediction from its corresponding block to output a residual; a transformer, which performs a transformation on the residual output from the adder in operation to output a transformation coefficient; a quantizer, which quantizes the transformation coefficient in operation to generate a quantized transformation coefficient; an entropy encoder, which encodes the quantized transformation coefficient in operation to generate a bitstream; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor generates a prediction of a current block based on a reference block in an encoded reference picture in operation, and the intra-frame predictor generates a prediction of a current block based on an encoded reference block in a current picture in operation. The prediction controller performs a boundary smoothing operation along a boundary between a second partition partitioned from the image block and a first partition having a non-rectangular shape in operation. The boundary smoothing operation includes: using information of the first partition, making a first prediction of a first value of a group of pixels of the first partition along the boundary; and using information of the second partition, making a second prediction of a second value of a group of pixels of the first partition along the boundary; weighting the first value and the second value; encoding the first partition using the weighted first value and the weighted second value.

[0071] According to another aspect, an image encoding method is provided, which performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The method generally includes four steps: using information of the first partition, making a first prediction of a first value of a group of pixels of the first partition along the boundary; and using information of the second partition, making a second prediction of a second value of a group of pixels of the first partition along the boundary; weighting the first value and the second value; encoding the first partition using the weighted first value and the weighted second value.

[0072] According to another aspect, an image decoder is provided, comprising a circuit and a memory coupled to the circuit. The circuit performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape in operation. The boundary smoothing operation includes: using information of the first partition, making a first prediction of a first value of a group of pixels of the first partition along the boundary; and using information of the second partition, making a second prediction of a second value of a group of pixels of the first partition along the boundary; weighting the first value and the second value; and decoding the first partition using the weighted first value and the weighted second value.

[0073] According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from the group consisting of a triangle, a trapezoid, and a polygon having at least five sides and angles. According to another aspect, the second partition has a non-rectangular shape.

[0074] According to another aspect, at least one of the first prediction and the second prediction is an inter-prediction process that predicts the first value and the second value based on a reference partition in an encoded reference picture. The inter-prediction process may predict a first value of pixels of a first partition including a group of pixels, and may predict a second value of a group of pixels of only the first partition.

[0075] According to another aspect, at least one of the first prediction and the second prediction is an intra prediction process that predicts the first value and the second value based on an encoded reference partition in the current picture.

[0076] According to another aspect, an image decoder is provided, comprising: an entropy decoder, which receives and decodes an encoded bitstream in operation to obtain quantized transform coefficients; and an inverse quantizer and a transformer, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder, which in operation adds the residual output from the inverse quantizer and the transformer to the prediction output from the prediction controller to reconstruct the block; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor and a memory, wherein the inter-frame predictor generates a prediction of a current block based on a reference block in a decoded reference picture in operation, and the intra-frame predictor generates a prediction of the current block based on a decoded reference block in a current picture in operation. The prediction controller performs a boundary smoothing operation along a boundary between a second partition partitioned from the image block and a first partition having a non-rectangular shape in operation. The boundary smoothing operation includes: using information of the first partition, making a first prediction of a first value of a group of pixels of the first partition along the boundary; and using information of the second partition, making a second prediction of a second value of a group of pixels of the first partition along the boundary; weighting the first value and the second value; and decoding the first partition using the weighted first value and the weighted second value.

[0077] According to another aspect, an image decoding method is provided, which performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The method generally includes four steps: using information of the first partition, making a first prediction of a first value of a group of pixels of the first partition along the boundary; and using information of the second partition, making a second prediction of a second value of a group of pixels of the first partition along the boundary; weighting the first value and the second value; and decoding the first partition using the weighted first value and the weighted second value.

[0078] According to one aspect, an image encoder is provided, comprising a circuit and a memory coupled to the circuit. The circuit performs a partition syntax operation in operation, wherein the partition syntax operation includes: partitioning an image block into a plurality of partitions based on a partition parameter indicating the partition, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; encoding the first partition and the second partition; and writing one or more parameters including the partition parameter into a bitstream.

[0079] According to another aspect, the partition parameters indicate that the first partition has a triangular shape.

[0080] According to another aspect, the partition parameters indicate that the second partition has a non-rectangular shape.

[0081] According to another aspect, the partitioning parameters indicate that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0082] According to another aspect, the partition parameters jointly encode a partition direction for partitioning the image block into a plurality of partitions. For example, the partition direction may include: from the upper left corner of the image block to its lower right corner, and from the upper right corner of the image block to its lower left corner. The partition parameters may jointly encode at least a first motion vector of a first partition.

[0083] According to another aspect, one or more parameters other than the partitioning parameter encode a partitioning direction applied to partition the image block into a plurality of partitions. The parameter encoding the partitioning direction may jointly encode at least a first motion vector of the first partition.

[0084] According to another aspect, the partition parameters may jointly encode at least a first motion vector of the first partition. The partition parameters may jointly encode a second motion vector of the second partition.

[0085] According to another aspect, one or more parameters other than the partition parameters may encode at least the first motion vector of the first partition.

[0086] According to another aspect, the one or more parameters are binarized according to a binarization scheme selected according to a value of at least one of the one or more parameters.

[0087] According to another aspect, an image encoder is provided, the image encoder comprising: a partitioner, which receives and partitions an original picture into blocks in operation; and an adder, which receives blocks from the partitioner and predictions from a prediction controller in operation, and subtracts each prediction from its corresponding block to output a residual; a transformer, which performs a transformation on the residual output from the adder in operation to output a transformation coefficient; a quantizer, which quantizes the transformation coefficient in operation to generate a quantized transformation coefficient; an entropy encoder, which encodes the quantized transformation coefficient in operation to generate a bitstream; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor generates a prediction of a current block based on a reference block in an encoded reference picture in operation, and the intra-frame predictor generates a prediction of a current block based on an encoded reference block in a current picture in operation. In operation, the prediction controller partitions the image block into a plurality of partitions based on a partition parameter indicating the partition, the plurality of partitions including a second partition and a first partition having a non-rectangular shape, and encodes the first partition and the second partition. In operation, the entropy encoder writes one or more parameters including partition parameters to the bitstream.

[0088] According to another aspect, a method for encoding an image including a partition syntax operation is provided. The method generally includes three steps: partitioning an image block into a plurality of partitions based on a partition parameter indicating the partition, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; encoding the first partition and the second partition; and writing one or more parameters including the partition parameter into a bitstream.

[0089] According to another aspect, an image decoder is provided, comprising a circuit and a memory coupled to the circuit. The circuit performs a partition syntax operation in operation, wherein the partition syntax operation comprises: parsing one or more parameters from a bitstream, wherein the one or more parameters comprise a partition parameter indicating partitioning of an image block into a plurality of partitions, wherein the plurality of partitions comprise a second partition and a first partition having a non-rectangular shape; partitioning the image block into the plurality of partitions based on the partition parameter; and decoding the first partition and the second partition.

[0090] According to another aspect, the partition parameters indicate that the first partition has a triangular shape.

[0091] According to another aspect, the partition parameters indicate that the second partition has a non-rectangular shape.

[0092] According to another aspect, the partitioning parameters indicate that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon having at least five sides and angles.

[0093] According to another aspect, the partition parameters jointly encode the partition direction for partitioning the image block into multiple partitions. For example, the partition direction includes: from the upper left corner of the image block to its lower right corner, and from the upper right corner of the image block to its lower left corner. The partition parameters can jointly encode at least the first motion vector of the first partition.

[0094] According to another aspect, one or more parameters other than the partitioning parameter encode a partitioning direction applied to partition the image block into a plurality of partitions. The parameter encoding the partitioning direction may jointly encode at least a first motion vector of the first partition.

[0095] According to another aspect, the partition parameters may jointly encode at least a first motion vector of the first partition. The partition parameters may jointly encode a second motion vector of the second partition.

[0096] According to another aspect, one or more parameters other than the partition parameters may encode at least the first motion vector of the first partition.

[0097] According to another aspect, the one or more parameters are binarized according to a binarization scheme selected according to a value of at least one of the one or more parameters.

[0098] According to another aspect, an image decoder is provided, the image decoder comprising: an entropy decoder, which receives and decodes an encoded bitstream in operation to obtain quantized transform coefficients; and an inverse quantizer and a transformer, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder, which in operation adds the residual output from the inverse quantizer and the transformer to the prediction output from the prediction controller to reconstruct the block; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor and a memory, wherein the inter-frame predictor generates a prediction for a current block based on a reference block in a decoded reference picture in operation, and the intra-frame predictor generates a prediction for the current block based on a decoded reference block in a current picture in operation. The entropy decoder in operation: parses one or more parameters from the bitstream, wherein the one or more parameters include a partition parameter indicating that the image block is partitioned into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; partitions the image block into a plurality of partitions based on the partition parameter; and decodes the first partition and the second partition.

[0099] According to another aspect, a method for decoding an image including a partition syntax operation is provided. The method generally includes three steps: parsing one or more parameters from a bitstream, wherein the one or more parameters include a partition parameter indicating partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; partitioning the image block into a plurality of partitions based on the partition parameter; and decoding the first partition and the second partition.

[0100] In the drawings, like reference numerals denote similar elements. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale.

[0101] In the following, embodiments will be described with reference to the accompanying drawings. Note that each of the embodiments described below shows a general or specific example. The numerical values, shapes, materials, components, arrangements and connections of components, steps, relationships and sequences of steps, etc. indicated in the following embodiments are only examples and are not intended to limit the scope of the claims. Therefore, those components disclosed in the following embodiments but not listed in any independent claim defining the broadest inventive concept may be understood as optional components.

[0102] Embodiments of encoders and decoders are described below. The embodiments are examples of encoders and decoders to which the processes and / or configurations presented in the description of aspects of the present disclosure may be applied. The processes and / or configurations may also be implemented in encoders and decoders other than those according to the embodiments. For example, with respect to the processes and / or configurations applied to the embodiments, any of the following may be implemented:

[0103] (1) Any component of an encoder or a decoder according to the embodiments presented in the description of the aspects of the present disclosure may be replaced with or combined with another component presented anywhere in the description of the aspects of the present disclosure.

[0104] (2) In an encoder or decoder according to an embodiment, any changes may be made to the functions or processes performed by one or more components of the encoder or decoder, such as addition, replacement, removal, etc. of functions or processes. For example, any function or process may be replaced or combined with another function or process appearing anywhere in the description of various aspects of the present disclosure.

[0105] (3) In the method implemented by the encoder or decoder according to the embodiment, any changes may be made, such as adding, replacing and removing one or more processes included in the method. For example, any process in the method may be replaced or combined with another process appearing anywhere in the description of various aspects of the present disclosure.

[0106] (4) One or more components included in an encoder or decoder according to an embodiment may be combined with components presented anywhere in the description of various aspects of the present disclosure, may be combined with components including one or more functions presented anywhere in the description of various aspects of the present disclosure, and may be combined with components that implement one or more processes implemented by components presented in the description of aspects of the present disclosure.

[0107] (5) Components including one or more functions of an encoder or decoder according to the embodiment, or components implementing one or more processes of an encoder or decoder according to the embodiment may be combined with or replaced with components presented anywhere in the description of the various aspects of the present disclosure, with components including one or more functions presented anywhere in the description of the various aspects of the present disclosure, or components implementing one or more processes presented anywhere in the description of the various aspects of the present disclosure.

[0108] (6) In a method implemented by an encoder or a decoder according to an embodiment, any process included in the method may be replaced or combined with a process presented anywhere in the description of aspects of the present disclosure or with any corresponding or equivalent process.

[0109] (7) One or more processes included in a method implemented by an encoder or a decoder according to an embodiment may be combined with a process presented anywhere in the description of various aspects of the present disclosure.

[0110] (8) The implementation of the processes and / or configurations presented in the description of various aspects of the present disclosure is not limited to the encoder or decoder according to the embodiments. For example, the processes and / or configurations may be implemented in a device for a purpose different from the motion picture encoder or motion picture decoder disclosed in the embodiments.

[0111] (Encoder)

[0112] First, an encoder according to the embodiment will be described. Figure 1 1 is a block diagram showing a functional configuration of an encoder 100 according to an embodiment. The encoder 100 is a moving picture encoder that encodes a moving picture block by block.

[0113] like Figure 1 As shown, the encoder 100 is a device for encoding a picture block by block, and includes a partitioner 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116, a block memory 118, a loop filter 120, a frame memory 122, an intra-frame predictor 124, an inter-frame predictor 126, and a prediction controller 128.

[0114] The encoder 100 is implemented as, for example, a general purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor functions as a segmenter 102, a subtractor 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116, a loop filter 120, an intra-frame predictor 124, an inter-frame predictor 126, and a prediction controller 128. Alternatively, the encoder 100 may be implemented as one or more dedicated electronic circuits corresponding to the segmenter 102, the subtractor 104, the transformer 106, the quantizer 108, the entropy encoder 110, the inverse quantizer 112, the inverse transformer 114, the adder 116, the loop filter 120, the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128.

[0115] Hereinafter, each component included in the encoder 100 will be described.

[0116] (Splitters)

[0117] The partitioner 102 partitions each image included in the input motion picture into blocks, and outputs each block to the subtractor 104. For example, the partitioner 102 first partitions the image into blocks of a fixed size (e.g., 128×128). Blocks of a fixed size may also be referred to as coding tree units (CTUs). The partitioner 102 then partitions each fixed-size block into blocks of a variable size (e.g., 64×64 or smaller) based on, for example, recursive quadtree and / or binary tree block partitioning. Blocks of variable size may also be referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). In various implementations, it may not be necessary to distinguish between CUs, PUs, and TUs; all or some blocks in a picture may be processed as CUs, PUs, or TUs.

[0118] Figure 2 An example of block segmentation according to an embodiment is shown. Figure 2 , solid lines represent block boundaries of blocks partitioned by quadtree block partitioning, and dotted lines represent block boundaries of blocks partitioned by binary tree block partitioning.

[0119] Here, the block 10 is a square 128×128 pixel block (128×128 block). The 128×128 block 10 is first partitioned into four square 64×64 blocks (quadtree block partitioning).

[0120] The upper left 64×64 block is further vertically divided into two rectangular 32×64 blocks, and the left 32×64 block is further vertically divided into two rectangular 16×64 blocks (binary tree block division). As a result, the upper left 64×64 block is divided into two 16×64 blocks 11 and 12 and one 32×64 block 13.

[0121] The upper right 64×64 block is split horizontally into two rectangular 64×32 blocks 14 and 15 (binary tree block partitioning).

[0122] First, the 64×64 block on the lower left is divided into four square 32×32 blocks (quadtree block partitioning). Among the four 32×32 blocks, the upper left block and the lower right block are further partitioned. The 32×32 block on the upper left is vertically divided into two rectangular 16×32 blocks, and the 16×32 block on the right is further horizontally divided into two 16×16 blocks (binary tree block partitioning). The 32×32 block on the lower right is horizontally partitioned into two 32×16 blocks (binary tree block partitioning). As a result, the 64×64 block on the lower left is partitioned into a 16×32 block 16, two 16×16 blocks 17 and 18, two 32×32 blocks 19 and 20, and two 32×16 blocks 21 and 22.

[0123] The lower right 64×64 block 23 is not segmented.

[0124] As mentioned above, in Figure 2 In the embodiment, based on recursive quadtree and binary tree block partitioning, the block 10 is partitioned into 13 variable-sized blocks 11 to 23. This type of partitioning is also called quadtree plus binary tree (QTBT) partitioning.

[0125] Despite Figure 2 In the example, a block is partitioned into four or two blocks (quadtree or binary tree block partitioning), but the partitioning is not limited to these examples. For example, a block can be partitioned into three blocks (ternary block partitioning). The partitioning including such ternary block partitioning is also called multi-type tree (MBT) partitioning.

[0126] (Subtractor)

[0127] The subtractor 104 subtracts the prediction signal (prediction sample, input from the prediction controller 128, which will be described below) from the original signal (original sample) of each block divided and input by the divider 102. In other words, the subtractor 104 calculates the prediction error (also referred to as "residual") of the block to be encoded (hereinafter also referred to as "current block"). The subtractor 104 then outputs the calculated prediction error (residual) to the transformer 106.

[0128] The original signal is a signal input to the encoder 100, and is a signal representing an image of each picture included in a moving picture (for example, a luminance signal and two chrominance signals). Hereinafter, the signal representing the image is also referred to as a sample.

[0129] (Converter)

[0130] The transformer 106 transforms the spatial domain prediction error into a frequency domain transform coefficient and outputs the transform coefficient to the quantizer 108. More specifically, the transformer 106 applies, for example, a predefined discrete cosine transform (DCT) or discrete sine transform (DST) to the spatial domain prediction error.

[0131] Note that the transformer 106 can adaptively select a transform type from a plurality of transform types and transform the prediction error into a transform coefficient by using a transform basis function corresponding to the selected transform type. This transform is also called explicit multi-core transform (EMT) or adaptive multi-transform (AMT).

[0132] Transform types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 is a graph showing the transformation basis functions for each transformation type. Figure 3 In , N represents the number of input pixels. For example, selecting a transform type from a plurality of transform types may depend on a prediction type (intra prediction and inter prediction) and an intra prediction mode.

[0133] Information indicating whether EMT or AMT is applied (e.g., referred to as an EMT flag or an AMT flag) and information indicating the selected transform type are typically signaled at the CU level. Note that the signaling of such information need not be performed at the CU level, but may be performed at another level (e.g., at a bit sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0134] In addition, the transformer 106 may apply a secondary transform to the transform coefficients (transform results). Such a secondary transform is also referred to as an adaptive secondary transform (AST) or a non-separable secondary transform (NSST). For example, the transformer 106 applies a secondary transform to each sub-block (e.g., each 4×4 sub-block) included in the block of transform coefficients corresponding to the intra-frame prediction error. Information indicating whether NSST is applied and information about the transform matrix used in NSST are typically signaled at the CU level. Note that the signaling of such information does not need to be performed at the CU level, but can be performed at another level (e.g., at a bit sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0135] Either a separate transform or a non-separable transform may be applied in the transformer 106. Separate transform is a method in which a transform is performed multiple times by performing a transform separately for each direction according to the number of dimensions of the input. Non-separable transform is a method of performing a collective transform in which two or more dimensions in a multi-dimensional input are collectively considered as a single dimension.

[0136] In one example of a non-separable transform, when the input is a 4x4 block, the 4x4 block is treated as a single array containing 16 components, and the transform applies a 16x16 transform matrix to the array.

[0137] In another example of a non-separable transform, after treating an input 4×4 block as a single array including 16 components, a transform (eg, a Hypercube-Givens transform) that performs multiple Givens rotations may be applied to the array.

[0138] (Quantizer)

[0139] The quantizer 108 quantizes the transform coefficients output from the transformer 106. More specifically, the quantizer 108 scans the transform coefficients of the current block in a predetermined scanning order, and quantizes the scanned transform coefficients based on a quantization parameter (QP) corresponding to the transform coefficients. Next, the quantizer 108 outputs the quantized transform coefficients of the current block (hereinafter referred to as quantized coefficients) to the entropy encoder 110 and the inverse quantizer 112.

[0140] The predetermined scanning order is an order for quantizing / inverse quantizing transform coefficients. For example, the predetermined scanning order is defined as an ascending order of frequency (from low frequency to high frequency) or a descending order of frequency (from high frequency to low frequency).

[0141] The quantization parameter (QP) is a parameter that defines the quantization step size (quantization width). For example, if the value of the quantization parameter increases, the quantization step size also increases. In other words, if the value of the quantization parameter increases, the quantization error increases.

[0142] (Entropy Encoder)

[0143] The entropy encoder 110 generates an encoded signal (encoded bit stream) based on the quantized coefficients input from the quantizer 108. More specifically, for example, the entropy encoder 110 binarizes the quantized coefficients and arithmetically encodes the binary signal to output a compressed bit stream or sequence.

[0144] (Inverse Quantizer)

[0145] The inverse quantizer 112 inversely quantizes the quantized coefficients input from the quantizer 108. More specifically, the inverse quantizer 112 inversely quantizes the quantized coefficients of the current block in a predetermined scanning order. Then, the inverse quantizer 112 outputs the inversely quantized transform coefficients of the current block to the inverse transformer 114.

[0146] (Inverter)

[0147] The inverse transformer 114 restores the prediction error (residual) by inversely transforming the transform coefficients input from the inverse quantizer 112. More specifically, the inverse transformer 114 restores the prediction error of the current block by applying an inverse transform corresponding to the transform applied to the transform coefficients by the transformer 106. The inverse transformer 114 then outputs the restored prediction error to the adder 116.

[0148] Note that since typically information is lost in quantization, the recovered prediction error does not match the prediction error calculated by the subtractor 104. In other words, the recovered prediction error typically includes the quantization error.

[0149] (Adder)

[0150] The adder 116 reconstructs the current block by summing the prediction error input from the inverse transformer 114 and the prediction sample input from the prediction controller 128. The adder 116 then outputs the reconstructed block to the block memory 118 and the loop filter 120. The reconstructed block is also referred to as a locally decoded block.

[0151] (Block Storage)

[0152] The block memory 118 is a storage device for storing blocks in a picture to be encoded (referred to as a “current picture”) referenced in, for example, intra prediction. More specifically, the block memory 118 stores the reconstructed blocks output from the adder 116 .

[0153] (Loop Filter)

[0154] The loop filter 120 applies a loop filter to the block reconstructed by the adder 116, and outputs the filtered reconstructed block to the frame memory 122. The loop filter is a filter (in-loop filter) used in an encoding loop, and includes, for example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF).

[0155] In ALF, a least square error filter for removing compression artifacts is applied. For example, based on the direction and activity of the local gradient, a filter is selected from multiple filters for each 2×2 sub-block in the current block and applied.

[0156] More specifically, first, each sub-block (e.g., each 2×2 sub-block) is classified into one of a plurality of categories (e.g., 15 or 25 categories). The classification of the sub-block is based on gradient directionality and activity. For example, a classification index C (e.g., C=5D+A) is derived based on gradient directionality D (e.g., 0 to 2 or 0 to 4) and gradient activity A (e.g., 0 to 4). Next, based on the classification index C, each sub-block is classified into one of a plurality of categories.

[0157] For example, the gradient directivity D is calculated by comparing the gradients in multiple directions (eg, horizontal direction, vertical direction, and two diagonal directions). Also, for example, the gradient activity A is calculated by summing the gradients in multiple directions and quantizing the sum.

[0158] Based on the result of such classification, a filter to be used for each sub-block is determined from among a plurality of filters.

[0159] The filter shape used in the ALF is, for example, a circularly symmetric filter shape. Figure 4A , 4B , and 4C show examples of filter shapes used in ALF. Figure 4A shows a 5×5 diamond filter, Figure 4B shows a 7×7 diamond filter, while Figure 4C A 9×9 diamond filter is shown. Information indicating the filter shape is typically signaled at the picture level. Note that the signaling of information indicating the filter shape need not be performed at the picture level, but can be performed at another level (e.g., at the sequence level, slice level, tile level, CTU level, or CU level).

[0160] The enabling or disabling of ALF may be determined at the picture level or at the CU level. For example, for luma, the decision whether to apply ALF may be done at the CU level, and for chroma, the decision whether to apply ALF may be done at the picture level. Information indicating whether ALF is enabled or disabled is typically signaled at the picture level or at the CU level. Note that the signaling of information indicating whether ALF is enabled need not be performed at the picture level or at the CU level, but may be performed at another level (e.g., at the sequence level, the slice level, the tile level, or the CTU level).

[0161] The coefficient sets for multiple selectable filters (e.g., 15 or 25 filters) are typically signaled at the picture level. Note that the signaling of coefficient sets need not be performed at the picture level, but can be performed at another level (e.g., at the sequence level, slice level, tile level, CTU level, CU level, or sub-block level).

[0162] (Frame Memory)

[0163] The frame memory 122 is, for example, a storage device for storing reference pictures used in inter-frame prediction, and is also referred to as a frame buffer. More specifically, the frame memory 122 stores the reconstructed blocks filtered by the loop filter 120.

[0164] (Intra Predictor)

[0165] The intra predictor 124 generates a prediction signal (intra prediction signal) (also referred to as intra prediction) by performing intra prediction on a current block with reference to one or more blocks in the current picture stored in the block memory 118. More specifically, the intra predictor 124 generates the intra prediction signal by performing intra prediction with reference to samples (e.g., luminance and / or chrominance values) of one or more blocks adjacent to the current block, and then outputs the intra prediction signal to the prediction controller 128.

[0166] For example, the intra predictor 124 performs intra prediction by using one of a plurality of predetermined intra prediction modes, which generally include one or more non-directional prediction modes and a plurality of directional prediction modes.

[0167] The one or more non-directional prediction modes include, for example, planar prediction mode and DC prediction mode defined in the H.265 / HEVC standard.

[0168] The plurality of directional prediction modes include, for example, 33 directional prediction modes defined in the H.265 / HEVC standard. Note that in addition to the 33 directional prediction modes, the plurality of directional prediction modes may also include 32 directional prediction modes (for a total of 65 directional prediction modes).

[0169] Figure 5A A total of 67 intra prediction modes (two non-directional prediction modes and 65 directional prediction modes) used in intra prediction are shown. Solid arrows represent 33 directions defined in the H.265 / HEVC standard, while dashed arrows represent another 32 directions (in Figure 5A The two "non-directional" prediction modes are not shown. )

[0170] In various implementations, a luma block may be referenced in chroma block intra prediction. That is, the chroma component of the current block may be predicted based on the luma component of the current block. Such intra prediction is also referred to as cross-component linear model (CCLM) prediction. A chroma block intra prediction mode (e.g., referred to as a CCLM mode) that references a luma block may be added as one of the chroma block intra prediction modes.

[0171] The intra predictor 124 may correct the intra predicted pixel value based on the horizontal / vertical reference pixel gradient. Intra prediction with such correction is also referred to as position dependent intra prediction combination (PDPC). Information indicating whether PDPC is applied (e.g., referred to as a PDPC flag) is typically signaled at the CU level. Note that the signaling of this information need not be performed at the CU level, but may be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, or a CTU level).

[0172] (Inter-frame predictor)

[0173] The inter-frame predictor 126 generates a prediction signal (inter-frame prediction) by inter-frame predicting the current block with reference to one or more blocks in a reference picture that is different from the current picture and stored in the frame memory 122. Inter-frame prediction is performed on each current block or each current sub-block in the current block (for example, each 4×4 block). For example, the inter-frame predictor 126 performs motion estimation for the current block or current sub-block in the reference picture to find the reference block or sub-block that best matches the current block or sub-block in the reference picture, and obtains motion information (for example, motion vector) that compensates (or predicts) motion or changes from the reference block or sub-block to the current block or sub-block. Then, the inter-frame predictor 126 performs motion compensation (or motion prediction) based on the motion information, and generates an inter-frame prediction signal for the current block or sub-block based on the motion information. The inter-frame predictor 126 then outputs the generated inter-frame prediction signal to the prediction controller 128.

[0174] The motion information used in motion compensation can be signaled in various forms as an inter-frame prediction signal. For example, a motion vector can be signaled. As another example, the difference between a motion vector and a motion vector predictor can be signaled.

[0175] Note that in addition to the motion information of the current block obtained from motion estimation, the motion information of the neighboring blocks can also be used to generate the inter-frame prediction signal. More specifically, the inter-frame prediction signal can be generated for each sub-block in the current block by calculating the weighted sum of the prediction signal based on the motion information obtained from motion estimation (in the reference picture) and the prediction signal based on the motion information of the neighboring blocks (in the current picture). Such inter-frame prediction (motion compensation) is also called overlapped block motion compensation (OBMC).

[0176] In the OBMC mode, information indicating the sub-block size used for OBMC (e.g., referred to as OBMC block size) may be signaled at the sequence level. In addition, information indicating whether the OBMC mode is applied (e.g., referred to as OBMC flag) may be signaled at the CU level. Note that the signaling of such information need not be performed at the sequence level and the CU level, but may be performed at another level (e.g., at the picture level, slice level, tile level, CTU level, or sub-block level).

[0177] Hereinafter, the OBMC mode will be described in more detail. Figure 5B It is a flow chart. Figure 5C is a conceptual diagram illustrating a predicted image correction process performed by the OBMC process.

[0178] refer to Figure 5C First, a predicted image (Pred) is obtained by typical motion compensation using the motion vector (MV) assigned to the target (current) block. Figure 5C In FIG. 8 , the arrow “MV” points to the reference picture to indicate what the current block in the current picture is referring to in order to obtain the predicted image.

[0179] Next, a predicted image (Pred_L) is obtained by applying (reusing) the motion vector (MV_L) that has been derived for the encoded neighboring left block to the target (current) block, as indicated by the arrow "MV_L" starting from the current block and pointing to the reference picture to obtain the predicted image Pred_L. Next, the two predicted images Pred and Pred_L are superimposed to perform a first pass of correction of the predicted images, which in one aspect has the effect of blending the boundaries between adjacent blocks.

[0180] Similarly, a predicted image (Pred_U) is obtained by applying (reusing) a motion vector (MV_U) that has been derived for the coded adjacent upper block to the target (current) block, as indicated by the arrow "MV_U" starting from the current block and pointing to the reference picture to obtain the predicted image Pred_U. Next, the predicted image Pred_U is superimposed with the predicted images obtained from the first pass (i.e., Pred and Pred_L) to perform a second pass of correction of the predicted image, which in one aspect has the effect of blending the boundaries between adjacent blocks. The result of the second pass is the final predicted image of the current block, which has a fused (smoothed) boundary with its adjacent blocks.

[0181] Note that the above example is a two-pass correction method using adjacent left and upper blocks, but the method may be a three-pass or higher-pass correction method also using adjacent right and / or lower blocks.

[0182] Note that the area subject to superposition may be the entire pixel area of ​​the block, and may alternatively be a partial block boundary area.

[0183] Note here that the prediction image correction process of OBMC is described as being based on a single reference picture to derive a single prediction image Pred, on which additional prediction images Pred_L and Pred_U are superimposed, but when the prediction image is corrected based on the multiple reference pictures, the same process can be applied to each of the multiple reference pictures. In such a case, after obtaining multiple corrected prediction images by performing image correction of OBMC based on the multiple reference pictures, respectively, the obtained multiple corrected prediction images are also superimposed to obtain a final prediction image.

[0184] Note that in OBMC, the unit of the target block may be a prediction block, and alternatively, may be a subblock obtained by further dividing the prediction block.

[0185] One example of a method for determining whether to implement OBMC processing is to use obmc_flag, which is a signal indicating whether to implement OBMC processing. As a specific example, the encoder can determine whether the target block belongs to an area including complex motion. When the block belongs to an area including complex motion, the encoder sets the value of obmc_flag to "1" and implements OBMC processing during the encoding process; if the block does not belong to an area including complex motion, the value of obmc_flag is set to "0" and the block is encoded without implementing OBMC processing. The decoder switches between whether to implement OBMC processing by decoding obmc_flag (i.e., compression sequence) written in the stream and performing decoding according to the flag value.

[0186] Note that the motion information can be derived on the decoder side without being signaled from the encoder side. For example, the merge mode defined in the H.265 / HEVC standard can be used. In addition, for example, the motion information can be derived by performing motion estimation on the decoder side. In this case, the decoder side can perform motion estimation without using the pixel values ​​of the current block.

[0187] Here, a mode for performing motion estimation at the decoder side will be described. The mode for performing motion estimation at the decoder side is also called a pattern matching motion vector derivation (PMMVD) mode or a frame rate up conversion (FRUC) mode.

[0188] exist Figure 5D An example of FRUC processing is shown in FIG. First, a candidate list of candidates (the candidate list may be a merge list) is generated with reference to motion vectors of coding blocks adjacent to the current block in space or time, each candidate list including a predicted motion vector (MV). Next, the best candidate MV is selected from a plurality of candidate MVs registered in the candidate list. For example, an evaluation value of the candidate MVs included in the candidate list is calculated, and a candidate MV is selected based on the calculated evaluation value.

[0189] Next, the motion vector of the current block is derived from the motion vectors of the selected candidates. More specifically, for example, the motion vector of the current block is calculated as the motion vector of the selected candidate (best candidate MV). Alternatively, the motion vector of the current block can be derived by pattern matching performed near the position corresponding to the motion vector of the selected candidate in the reference picture. In other words, when the vicinity of the best candidate MV is searched using pattern matching in the reference picture and the evaluation value, and an MV with a better evaluation value is found, the best candidate MV can be updated to the MV with the better evaluation value, and the MV with the better evaluation value can be used as the final MV of the current block. A configuration in which a process for updating the MV with a better evaluation value is not implemented is also acceptable.

[0190] In the case where processing is performed in units of sub-blocks, the same processing can be performed.

[0191] The evaluation value may be calculated in various ways. For example, a reconstructed image of a region in a reference picture corresponding to a motion vector is compared with a reconstructed image of a predetermined region (e.g., in another reference picture or in an adjacent block in the current picture as described below), and the difference in pixel values ​​between the two reconstructed images may be calculated and used as the evaluation value of the motion vector. Note that the evaluation value may be calculated by using some other information in addition to the difference.

[0192] Next, pattern matching is described in detail. First, a candidate MV included in a candidate list (e.g., a merged list) is selected as a starting point for searching by pattern matching. The pattern matching used is a first pattern matching or a second pattern matching. The first pattern matching and the second pattern matching are also referred to as bilateral matching and template matching, respectively.

[0193] In the first pattern matching, pattern matching is performed between two blocks in two different reference pictures that are both along the motion trajectory of the current block. Therefore, in the first pattern matching, for a region in a reference picture, a region in another reference picture that matches the motion trajectory of the current block is used as a predetermined region for calculation of the above-mentioned candidate evaluation value.

[0194] Figure 6 An example of a first pattern match (bidirectional match) between two blocks in two reference pictures along a motion trajectory is shown. Figure 6 As shown in, in the first pattern matching, two motion vectors (MV0, MV1) are obtained by finding the best match between two blocks in two different reference pictures (Ref0, Ref1) along the motion trajectory of the current block (Cur block). More specifically, the difference can be obtained between (i) the reconstructed image at the position specified by the candidate MV in the first encoded reference picture (Ref0) and (ii) the reconstructed image at the position specified by the candidate MV in the second encoded reference picture (Ref1) (symmetrically scaled for each display time interval). Then, the difference can be used to derive an evaluation value of the current block. The candidate MV with the best evaluation value can be selected as the final MV among multiple candidate MVs.

[0195] Under the assumption of continuous motion trajectory, the motion vectors (MV0, MV1) pointing to the two reference blocks are proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, when the current picture is temporally located between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, the first pattern matching results in two mirrored bidirectional motion vectors.

[0196] In the second pattern matching (template matching), pattern matching is performed between a template in the current picture (a block adjacent to the current block in the current picture; for example, a top and / or left adjacent block) and a block in the reference picture. Therefore, in the second pattern matching, a block adjacent to the current block in the current picture is used as a predetermined region for the above-mentioned calculation of the candidate evaluation value.

[0197] Figure 7 An example of pattern matching (template matching) between a template in the current picture and a block in the reference picture is shown. Figure 7As shown in, in the second pattern matching, the motion vector of the current block is obtained by searching in the reference picture (Ref0) to find the block that best matches the neighboring block of the current block (Cur block) in the current picture (Cur Pic). More specifically, a difference can be obtained between (i) a reconstructed image of one or both of the encoded adjacent upper and left regions relative to the current block and (ii) a reconstructed image of the same region relative to the block position specified by the candidate MV in the encoded reference picture (Ref0). Then, the difference can be used to derive an evaluation value of the current block. A candidate MV with the best evaluation value can be selected as the best candidate MV among multiple candidate MVs.

[0198] Information indicating whether the FRUC mode is applied (e.g., referred to as a FRUC flag) may be signaled at the CU level. In addition, when the FRUC mode is applied (e.g., when the FRUC flag is set to true), information indicating a matching method (e.g., first mode matching or second mode matching) to which the mode is applicable may be signaled at the CU level. Note that the signaling of such information need not be performed at the CU level, but may be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0199] Next, a method of deriving motion vectors is described. First, a description is given of a mode for deriving motion vectors based on a model assuming uniform linear motion. This mode is also known as the Bidirectional Optical Flow (BIO) mode.

[0200] Figure 8 A model assuming uniform linear motion is shown. Figure 8 In (v x ,v y ) represents a velocity vector, and I0 and I1 represent the temporal distances between the current picture (Cur Pic) and two reference pictures (Ref0, Ref1), respectively. (MVx0, MVy0) represents a motion vector corresponding to the reference picture Ref0, and (MVx1, MVy1) represents a motion vector corresponding to the reference picture Ref1.

[0201] Here, assuming that the velocity vector (vx, vy) exhibits uniform linear motion, (MVx0, MVy0) and (MVx1, MVy1) are represented as (v x I0,v y I0) and (-v x I1,-v y I1), and gives the following optical flow equation (Equation 1).

[0202] [Mathematical formula 1]

[0203]

[0204] Here, I (k) represents the luminance value from the motion compensated reference picture k (k=0, 1). The optical flow equation shows that the sum of the following terms is equal to zero: (i) the temporal derivative of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference picture, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference picture. The motion vector of each block obtained from, for example, a merge list can be corrected pixel by pixel based on a combination of the optical flow equation and Hermite interpolation.

[0205] Note that motion vectors may be derived at the decoder side using methods other than deriving motion vectors based on a model assuming uniform linear motion. For example, the motion vector for each sub-block may be derived based on the motion vectors of neighboring blocks.

[0206] Next, a description is given of a mode in which a motion vector is derived for each subblock based on motion vectors of neighboring blocks. This mode is also called an affine motion compensation prediction mode.

[0207] Fig. 9A An example of deriving the motion vector of each sub-block based on the motion vectors of neighboring blocks is shown. Fig. 9A In the example, the current block includes 16 4×4 sub-blocks. Here, the motion vector v0 of the upper left corner control point in the current block is derived based on the motion vectors of the neighboring sub-blocks. Similarly, the motion vector v1 of the upper right corner control point in the current block is derived based on the motion vectors of the neighboring blocks. Next, using the two motion vectors v0 and v1, the motion vector (v x , v y ).

[0208] [Mathematical formula 2]

[0209]

[0210] Here, x and y are the horizontal and vertical positions of the sub-block, respectively, and w is a predetermined weighting coefficient.

[0211] The affine motion compensation prediction mode may include multiple modes of different methods of deriving motion vectors for the upper left and upper right control points. Information indicating the affine motion compensation prediction mode (e.g., referred to as an affine flag) may be signaled at the CU level. Note that the signaling of information indicating the affine motion compensation prediction mode need not be performed at the CU level, but may be performed at another level (e.g., at a sequence level, a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0212] (Predictive Controller)

[0213] The prediction controller 128 selects the intra prediction signal (output from the intra predictor 124 ) or the inter prediction signal (output from the inter predictor 126 ), and outputs the selected prediction signal to the subtractor 104 and the adder 116 .

[0214] like Figure 1 As shown, in various implementations, the prediction controller 128 may output prediction parameters that are input to the entropy encoder 110. The entropy encoder 110 may generate an encoded bitstream (or sequence) based on the prediction parameters input from the prediction controller 128 and the quantized coefficients input from the quantizer 108. The prediction parameters may be used by a decoder that receives and decodes the encoded bitstream to implement the same prediction process performed in the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128. The prediction parameters may include a selected prediction signal (e.g., a motion vector, a prediction type, or a prediction mode employed in the intra-frame predictor 124 or the inter-frame predictor 126), or any index, flag, or value based on or indicating the prediction process performed in the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128.

[0215] Fig. 9B An example of a process for deriving a motion vector in the current picture in merge mode is shown.

[0216] First, a prediction MV list is generated in which prediction MV candidates are registered. Examples of prediction MV candidates include: spatial neighbor prediction MV, which is the MV of a coding block located in the spatial vicinity of the target block; temporal neighbor prediction MV, which is the MV of a block in the encoded reference picture adjacent to a block located at the same position as the target block; coupled prediction MV, which is an MV generated by combining the MV values ​​of the spatial neighbor prediction MV and the temporal neighbor prediction MV; zero prediction MV, which is an MV with a value of zero.

[0217] Next, the MV of the target block is determined by selecting one predicted MV from among a plurality of predicted MVs registered in the predicted MV list.

[0218] In addition, in the variable length encoder, a signal merge_idx indicating which predicted MV is selected is written and encoded in the stream.

[0219] Note that Fig. 9B The predicted MVs registered in the predicted MV list shown in constitute an example. The number of predicted MVs registered in the predicted MV list can be Fig. 9B If the number shown is different, the predicted MV registered in the predicted MV list can be omitted. Fig. 9B One or more types of predicted MVs are given in the examples, except for Fig. 9B outside the types given in the examples and with Fig. 9B Different from the types given in the example of , the predicted MVs registered in the predicted MV list may include one or more types of predicted MVs.

[0220] The final MV can be determined by performing a DMVR (Dynamic Motion Vector Refresh) process (described later) using the MV of the target block derived in the merge mode.

[0221] Fig. 9C is a conceptual diagram illustrating an example of a DMVR process for determining a MV.

[0222] First, the most suitable MV set for the current block (e.g., in merge mode) is considered as a candidate MV. Next, according to the candidate MV (L0), reference pixels are identified in the first reference picture (L0) which is the encoded image in the L0 direction. Similarly, according to the candidate MV (L1), reference pixels are identified in the second reference picture (L1) which is the encoded picture in the L1 direction. Next, the reference pixels are averaged to form a template.

[0223] Next, using the template, search the surrounding areas of the candidate MVs of the first reference picture (L0) and the second reference picture (L1), and determine the MV with the lowest cost as the final MV. For example, the cost value can be calculated using the difference between each pixel value in the template and each pixel value in the searched area, using the candidate MV, etc.

[0224] Note that the configuration and operation of the processing described here are basically the same on both the encoder side and the decoder side, which will be described below.

[0225] Any processing other than the above-described processing may be used as long as the processing can derive a final MV by searching the surroundings of the candidate MVs.

[0226] Next, a description is given of an example of a mode of generating a predicted image (prediction) using LIC (Local Illumination Compensation) processing.

[0227] Fig.9D One example of a predicted image generation method using a brightness correction process performed by the LIC process is shown.

[0228] First, an MV is derived from the encoded reference picture to obtain a reference image corresponding to the current block.

[0229] Next, for the current block, based on the luminance pixel values ​​of the encoded adjacent left reference area and the encoded adjacent upper reference area in the current picture, and based on the luminance pixel values ​​of the same position in the reference picture specified by the MV, information indicating how the luminance value changes between the reference picture and the current picture is obtained. The information indicating how the luminance value changes is used to calculate the luminance correction parameter.

[0230] The predicted image for the current block is generated by performing a brightness correction process that applies brightness correction parameters to a reference image in a reference picture specified by the MV.

[0231] Notice, Fig.9D The shape of the surrounding reference area shown in FIG. 1 is only one example; the surrounding reference area may have a different shape.

[0232] Furthermore, although a predicted image is generated from a single reference picture in this example, in the case where a predicted image is generated from a plurality of reference pictures, the predicted image may be generated after a brightness correction process is performed on a reference image obtained from the reference picture as described above.

[0233] One example of a method for determining whether to implement LIC processing is to use lic_flag, which is a signal indicating whether to implement LIC processing. As a specific example, the encoder determines whether the current block belongs to an area where brightness changes. When the block belongs to an area where brightness changes, the encoder sets lic_flag to "1" and implements LIC processing when encoding. When the block does not belong to an area where brightness changes, the encoder sets lic_flag to "0" and performs encoding that implements LIC processing. The decoder can switch between whether to perform LIC processing by decoding lic_flag written in the stream and performing decoding according to the flag value.

[0234] An example of different methods of determining whether to implement LIC processing includes discerning whether LIC processing is determined to be implemented on surrounding blocks. In a specific example, when merge mode is used on the current block, it is determined whether LIC processing is applied in the encoding of surrounding coded blocks, which are selected when deriving MVs in merge mode. This determination is then used to further determine whether LIC processing is implemented on the current block. Note that in this example, the same applies to the processing performed on the decoder side.

[0235] (Decoder)

[0236] Next, a decoder capable of decoding an encoded signal (encoded bit stream) output from the encoder 100 will be described. Fig.102 is a block diagram showing a functional configuration of a decoder 200 according to an embodiment. The decoder 200 is a moving picture decoder that decodes a moving picture block by block.

[0237] As in Fig.10 As shown in , the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an intra predictor 216, an inter predictor 218, and a prediction controller 220.

[0238] The decoder 200 is implemented as, for example, a general purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor acts as an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a loop filter 212, an intra-frame predictor 216, an inter-frame predictor 218, and a prediction controller 220. Alternatively, the decoder 200 may be implemented as one or more dedicated electronic circuits corresponding to the entropy decoder 202, the inverse quantizer 204, the inverse transformer 206, the adder 208, the loop filter 212, the intra-frame predictor 216, the inter-frame predictor 218, and the prediction controller 220.

[0239] Hereinafter, each component included in the decoder 200 will be described.

[0240] (Entropy Decoder)

[0241] The entropy decoder 202 performs entropy decoding on the decoded bit stream. More specifically, for example, the entropy decoder 202 arithmetically decodes the encoded bit stream into a binary signal. Then, the entropy decoder 202 de-binarizes the binary signal. The entropy decoder 202 outputs the quantized coefficients of each block to the inverse quantizer 204. The entropy decoder 202 may also de-quantize the prediction parameters (see Figure 1 ) is output to the intra predictor 216, the inter predictor 218, and the prediction controller 220 so that they can perform the same prediction processing as the prediction processing performed on the encoder side in the intra predictor 124, the inter predictor 126, and the prediction controller 128.

[0242] (Inverse Quantizer)

[0243] The inverse quantizer 204 inversely quantizes the quantized coefficients of the block to be decoded (hereinafter referred to as the current block) input from the entropy decoder 202. More specifically, the inverse quantizer 204 inversely quantizes the quantized coefficients of the current block based on the quantization parameters corresponding to the quantized coefficients. Next, the inverse quantizer 204 outputs the inversely quantized coefficients of the current block (i.e., transform coefficients) to the inverse transformer 206.

[0244] (Inverter)

[0245] The inverse transformer 206 restores the prediction error (residual) by inversely transforming the transform coefficient input from the inverse quantizer 204 .

[0246] For example, when information parsed from the encoded bitstream indicates application of EMT or AMT (eg, when the AMT flag is set to true), the inverse transformer 206 inversely transforms transform coefficients of the current block based on information indicating the parsed transform type.

[0247] Furthermore, for example, when the information parsed from the encoded bitstream indicates application of NSST, the inverse transformer 206 applies a secondary inverse transform to the transform coefficients.

[0248] (Adder)

[0249] The adder 208 reconstructs the current block by adding the prediction error input from the inverse transformer 206 and the prediction sample as an input from the prediction controller 220. Then, the adder 208 outputs the reconstructed block to the block memory 210 and the loop filter 212.

[0250] (Block Storage)

[0251] The block memory 210 is a memory for storing a block in a picture to be decoded for reference in intra prediction (hereinafter referred to as a current picture). More specifically, the block memory 210 stores the reconstructed block output from the adder 208.

[0252] (Loop Filter)

[0253] The loop filter 212 applies a loop filter to the block reconstructed by the adder 208 and outputs the filtered reconstructed block to the frame memory 214 and, for example, to a display device.

[0254] When information indicating enabling or disabling of the ALF parsed from the encoded bitstream indicates enabling, one filter is selected from a plurality of filters based on a direction and an activity of a local gradient, and the selected filter is applied to the reconstructed block.

[0255] (Frame Memory)

[0256] The frame memory 214 is a storage device for storing reference pictures used in inter-frame prediction, and is also referred to as a frame buffer. More specifically, the frame memory 214 stores the reconstructed blocks filtered by the loop filter 212.

[0257] (Intra Predictor)

[0258] The intra predictor 216 generates a prediction signal (intra prediction signal) through intra prediction with reference to one or more blocks in the current picture as stored in the block memory 210. More specifically, the intra predictor 216 generates the intra prediction signal through intra prediction with reference to samples (e.g., luminance and / or chrominance values) of one or more blocks adjacent to the current block, and then outputs the intra prediction signal to the prediction controller 220.

[0259] Note that when an intra prediction mode in which a chroma block is intra predicted from a luma block is selected, the intra predictor 216 may predict the chroma component of the current block based on the luma component of the current block.

[0260] Furthermore, when information indicating application of PDPC is parsed from the encoded bitstream (eg, in the prediction parameters output from the entropy decoder 202), the intra predictor 216 corrects the intra predicted pixel value based on the horizontal / vertical reference pixel gradient.

[0261] (Inter-frame predictor)

[0262] The inter-frame predictor 218 predicts the current block with reference to the reference picture stored in the frame memory 214. Inter-frame prediction is performed on each current block or each sub-block in the current block (e.g., each 4×4 block). For example, the inter-frame predictor 218 uses motion information (e.g., motion vector) parsed from the encoded bit stream (e.g., in the prediction parameters output from the entropy decoder 202), generates an inter-frame prediction signal of the current block or sub-block based on motion compensation, and outputs the inter-frame prediction signal to the prediction controller 220.

[0263] When information parsed from the encoded bitstream indicates application of the OBMC mode, the inter predictor 218 generates an inter prediction signal using motion information of a neighboring block in addition to motion information of a current block obtained from motion estimation.

[0264] In addition, when the information parsed from the encoded bitstream indicates the application of the FRUC mode, the inter-frame predictor 218 derives motion information by performing motion estimation according to the pattern matching method (bilateral matching or template matching) parsed from the encoded bitstream. Then, the inter-frame predictor 218 performs motion compensation (prediction) using the derived motion information.

[0265] In addition, when the BIO mode is to be applied, the inter-frame predictor 218 derives a motion vector based on a model assuming uniform linear motion. In addition, when the information parsed from the encoded bitstream indicates that the affine motion compensation prediction mode is to be applied, the inter-frame predictor 218 derives a motion vector for each sub-block based on the motion vectors of the neighboring blocks.

[0266] (Predictive Controller)

[0267] The prediction controller 220 selects an intra prediction signal or an inter prediction signal, and outputs the selected prediction signal to the adder 208. Generally speaking, the configuration, function, and operation of the prediction controller 220, the inter predictor 218, and the intra predictor 216 on the decoder side may correspond to the configuration, function, and operation of the prediction controller 128, the inter predictor 126, and the intra predictor 124 on the encoder side.

[0268] (Non-rectangular partitions)

[0269] In the prediction controller 128 coupled to the intra-frame predictor 124 and the inter-frame predictor 126 on the encoder side (see Figure 1 ), and in a prediction controller 220 coupled to the intra-frame predictor 216 and the inter-frame predictor 218 on the decoder side (see Fig.10 ), so far, the partitions (or blocks or sub-blocks of variable size) for which motion information (eg, motion vectors) is obtained by segmenting each block are always rectangular, such as Figure 2 As shown. The inventors have found that in various implementations, generating partitions having non-rectangular shapes such as triangles can lead to improvements in image quality and coding efficiency based on the content of the image in the picture. In the following, various embodiments will be described in which at least one partition partitioned from the image block for prediction purposes has a non-rectangular shape. Note that these embodiments are equally applicable to the encoder side (prediction controller 128 coupled to intra predictor 124 and inter predictor 126) and the decoder side (prediction controller 220 coupled to intra predictor 216 and inter predictor 218), and can be used in Figure 1 The encoder or Fig.10 is implemented in a decoder, etc.

[0270] Fig.11 is a flowchart showing a process of partitioning an image block into partitions including at least a second partition and a first partition having a non-rectangular shape (e.g., a triangle), and performing an additional process including encoding (or decoding) the image block into a reconstructed combination of the first partition and the second partition.

[0271] In step S1001, the image block is divided into a plurality of partitions, wherein the partitions may include a first partition having a non-rectangular shape and a second partition which may or may not have a non-rectangular shape. Fig.12As shown in , the image block can be partitioned from the upper left corner of the image block to the lower right corner of the image block to create a first partition and a second partition each having a non-rectangular shape (e.g., a triangle), or the image block can be partitioned from the upper right corner of the image block to the lower left corner of the image block to create a first partition and a second partition each having a non-rectangular shape (e.g., a triangle). Fig.12 and 17 -19 to describe various examples of non-rectangular segmentation.

[0272] In step S1002, the process predicts a first motion vector for a first partition and predicts a second motion vector for a second partition. For example, the prediction of the first and second motion vectors may include selecting the first motion vector from a first set of motion vector candidates and selecting the second motion vector from a second set of motion vector candidates.

[0273] In step S1003 , a motion compensation process is performed to obtain a first partition using the first motion vector derived in step S1002 above, and to obtain a second partition using the second motion vector derived in step S1002 above.

[0274] In step S1004, a prediction process is performed on the image block that is a (reconstructed) combination of the first partition and the second partition. The prediction process may include a boundary smoothing process for smoothing the boundary between the first partition and the second partition. For example, the boundary smoothing process may involve weighting a first value of a boundary pixel predicted based on the first partition and a second value of a boundary pixel predicted based on the second partition. Various implementations of the boundary smoothing process will be referred to below. Fig.13 , 14 , 20 and 21A-21D are described.

[0275] In step S1005, the process encodes or decodes the image block using one or more parameters, wherein the one or more parameters include a partition parameter indicating partitioning the image block into a second partition and a first partition having a non-rectangular shape. Fig.15 As summarized in the table of FIG. 1 , for example, the partition parameter (“first index value”) may have an influence on, for example, the partition direction to be applied in the partition (e.g., from top left to bottom right or from top right to bottom left, as Fig.12 The first and second motion vectors obtained in step S1002 are jointly encoded. Fig.15 , 16 and 22-25 describe in detail the details of such partitioning syntax operations involving one or more parameters including partitioning parameters.

[0276] Fig.172000 is a flow chart showing a process 2000 for segmenting an image block. In step S2001, the process segments the image into a plurality of partitions, including a first partition having a non-rectangular shape and a second partition that may or may not have a non-rectangular shape. Fig.12 As shown in , the image block can be divided into a first partition having a triangular shape and a second partition also having a triangular shape. There are many other examples in which the image block is divided into a plurality of partitions including a first partition and a second partition, wherein at least the first partition has a non-rectangular shape. The non-rectangular shape can be a triangle, a trapezoid, and a polygon with at least five sides and angles.

[0277] For example, as in Fig.18 As shown in , the image block can be partitioned into two triangular partitions; the image block can be partitioned into more than two triangular partitions (for example, three triangular partitions); the image block can be partitioned into a combination of triangular partitions and rectangular partitions; or the image block can be partitioned into a combination of triangular partitions and polygonal partitions.

[0278] As in Fig.19 As further shown in , the image block can be divided into L-shaped (polygonal) partitions and rectangular shape partitions; the image block can be divided into pentagonal (polygonal) shape partitions and triangular shape partitions; the image block can be divided into hexagonal (polygonal) shape partitions and pentagonal (polygonal) shape partitions; or the image block can be divided into multiple polygonal partitions.

[0279] Reference Fig.17 , in step S2002, the process predicts a first motion vector of a first partition, for example, by selecting the first partition from a first set of motion vector candidates, and predicts a second motion vector of a second partition, for example, by selecting the second motion vector from a second set of motion vector candidates. For example, the first set of motion vector candidates may include motion vectors of partitions adjacent to the first partition, and the second set of motion vector candidates may include motion vectors of partitions adjacent to the second partition. Adjacent partitions may be one or both of spatially adjacent partitions and temporally adjacent partitions. Some examples of spatially adjacent partitions include partitions located to the left, lower left, lower, lower right, right, upper right, upper, or upper left of the partition being processed. An example of a temporally adjacent partition is a partition located at the same position in a reference picture of an image block.

[0280] In various implementations, a partition adjacent to the first partition and a partition adjacent to the second partition may be outside the image block from which the first partition and the second partition are partitioned. The first group of motion vector candidates may be the same as or different from the second group of motion vector candidates. In addition, at least one of the first group of motion vector candidates and the second group of motion vector candidates may be the same as another group of motion vector candidates prepared for the image block, namely, a third group of motion vector candidates.

[0281] In some implementations, in step S2002, in response to determining that the second partition also has a non-rectangular shape (e.g., a triangle) similar to the first partition, the process 2000 creates a second set of motion vector candidates (for the non-rectangular second partition), which includes motion vectors of partitions adjacent to the second partition (excluding the first partition) (i.e., excluding the motion vector of the first partition). On the other hand, in response to determining that the second partition (different from the first partition) has a rectangular shape, the process 2000 creates a second set of motion vector candidates (for the rectangular second partition), which includes motion vectors of partitions adjacent to the second partition (including the first partition).

[0282] In step S2003, the process encodes or decodes the first partition using the first motion vector derived in step S2002 above, and encodes or decodes the second partition using the second motion vector derived in step S2002 above.

[0283] picture Fig.17 As in process 2000, the image block segmentation process may be performed by an image encoder, for example, as in Figure 1 As shown in , the image encoder includes a circuit and a memory coupled to the circuit. The circuit performs in operation: dividing an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape (step S2001); predicting a first motion vector of the first partition and a second motion vector of the second partition (step S2002); encoding the first partition using the first motion vector, and encoding the second partition using the second motion vector (step S2003).

[0284] According to another embodiment, Figure 1As shown, an image encoder is provided, including: a partitioner 102, which receives an original picture and partitions it into a plurality of blocks during operation; an adder 104, which receives a block from the partitioner and a prediction from a prediction controller 128 during operation, and subtracts each prediction from its corresponding block to output a residual; a transformer 106, which performs a transformation on the residual output from the adder 104 to output a transformation coefficient during operation; a quantizer 108, which quantizes the transformation coefficient to generate a quantized transformation coefficient during operation; an entropy encoder 110, which encodes the quantized transformation coefficient to generate a bitstream during operation; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124, and memories 118 and 122, wherein the inter-frame predictor 126 generates a prediction for a current block based on a reference block in an encoded reference picture during operation, and the intra-frame predictor 124 generates a prediction for a current block based on an encoded reference block in a current picture during operation. The prediction controller 128 divides the block into a plurality of partitions in operation, the plurality of partitions including the second partition and the first partition having a non-rectangular shape ( Fig.17 , step S2001); predicting a first motion vector of a first partition and a second motion vector of a second partition (step S2002); encoding the first partition using the first motion vector, and encoding the second partition using the second motion vector (step S2003).

[0285] According to another embodiment, as in Fig.10 As shown in , an image decoder is provided, which includes a circuit and a memory coupled to the circuit. The circuit performs in operation: dividing the image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape ( Fig.17 , step S2001); predicting a first motion vector for a first partition and a second motion vector for a second partition (step S2002); decoding the first partition using the first motion vector, and decoding the second partition using the second motion vector (step S2003).

[0286] According to another embodiment, there is provided Fig.10The image decoder shown in , which includes: an entropy decoder 202, which receives and decodes the encoded bit stream in operation to obtain quantized transform coefficients; an inverse quantizer 204 and a transformer 206, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder 208, which in operation adds the residual output from the inverse quantizer 204 and the transformer 206 to the prediction output from the prediction controller 220 to reconstruct the block; and a prediction controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216 and memories 210, 214, wherein the inter-frame predictor 218 generates a prediction of the current block based on a reference block in a decoded reference picture in operation, and the intra-frame predictor 216 generates a prediction of the current block based on a decoded reference block in the current picture in operation. The prediction controller 220 divides the image block into a plurality of partitions in operation, the plurality of partitions including a second partition and a first partition having a non-rectangular shape ( Fig.17 , step S2001); predict a first motion vector of a first partition and a second motion vector of a second partition (step S2002); use the first motion vector to decode the first partition, and use the second motion vector to decode the second partition (step S2003).

[0287] (Border smoothing)

[0288] As mentioned above Fig.11 As described in, according to various embodiments, step S1004 of performing a prediction process on an image block which is a (reconstructed) combination of a second partition and a first partition having a non-rectangular shape may involve applying a boundary smoothing process along a boundary between the first partition and the second partition.

[0289] For example, Fig.21B An example of a boundary smoothing process is shown, which involves weighting a first value of a boundary pixel and a second value of the boundary pixel, wherein the first value of the boundary pixel is first predicted based on a first partition and the second value of the boundary pixel is second predicted based on a second partition.

[0290] Fig. 20 3000 is a flow chart showing an overall boundary smoothing process 3000 according to one embodiment, the process 3000 involves weighting a first value of a boundary pixel for a first prediction based on a first partition and a second value of a boundary pixel for a second prediction based on a second partition. In step S3001, as in Fig.21A or above Fig.12 , 18 As shown in and 19, the image block is divided into a first partition and a second partition along a boundary, wherein at least the first partition has a non-rectangular shape.

[0291] In step S3002, a group of pixels ( Fig.21A In step S3003, a second prediction is made for a second value of a (same) group of pixels of the first partition along the boundary (for example, color, brightness, transparency, etc.), wherein the first value is first predicted using information of the first partition. In step S3003, a second prediction is made for a second value of a (same) group of pixels of the first partition along the boundary, wherein the second value is second predicted using information of the second partition. In some implementations, at least one of the first prediction and the second prediction is an inter-frame prediction process that predicts the first value and the second value based on a reference partition in an encoded reference picture. Reference Fig.21D In some implementations, the prediction process predicts a first value of all pixels of the first partition including a set of pixels on which the first partition and the second partition overlap ("a first set of samples"), and predicts a second value of only a set of pixels on which the first partition and the second partition overlap ("a second set of samples"). In another implementation, at least one of the first prediction and the second prediction is an intra-prediction process that predicts the first value and the second value based on an encoded reference partition in the current picture. In some implementations, the prediction method used in the first prediction is different from the prediction method used in the second prediction. For example, the first prediction may include an inter-prediction process, and the second prediction may include an intra-prediction process. The information used to make a first prediction of the first value or a second prediction of the second value may be a motion vector of the first partition or the second partition, an intra-prediction direction, etc.

[0292] In step S3004, a first value predicted using the first partition and a second value predicted using the second partition are weighted. In step S3005, the first partition is encoded or decoded using the weighted first value and second value.

[0293] Fig.21B An example of a boundary smoothing operation is shown, where the first partition and the second partition overlap at five pixels (at most) per row or column. That is, the number of pixels in each row or column for which a first value is predicted based on the first partition and a second value is predicted based on the second partition is at most five. Fig. 21C Another example of a boundary smoothing operation is shown, where the first partition and the second partition overlap at three pixels (at most) per row or column. That is, the number of a group of pixels in each row or column whose first value is predicted based on the first partition and whose second value is predicted based on the second partition is at most three.

[0294] Fig.13Another example of a boundary smoothing operation is shown, where the first partition and the second partition overlap on four pixels (at most) per row or column. That is, the number of pixels in a group in each row or column for which a first value is predicted based on the first partition and a second value is predicted based on the second partition is at most four. In the example shown, weights 1 / 8, 1 / 4, 3 / 4, and 7 / 8 may be applied to the first value of the four pixels in the group, respectively, and weights 7 / 8, 3 / 4, 1 / 4, and 1 / 8 may be applied to the second value of the four pixels in the group, respectively.

[0295] Fig.14 Another example of a boundary smoothing operation is shown, in which the first partition and the second partition overlap on zero pixels per row or column (i.e., they do not overlap), overlap on one pixel (at most) per row or column, and overlap on two pixels (at most) per row or column, respectively. In examples where the first and second partitions do not overlap, a zero weight is applied. In examples where the first partition and the second partition overlap on one pixel per row or column, a weight of 1 / 2 may be applied to the first value of the pixel in the group predicted based on the first partition, and a weight of 1 / 2 may be applied to the second value of the pixel in the group predicted based on the second partition. In examples where the first partition and the second partition overlap on two pixels per row or column, weights of 1 / 3 and 2 / 3 may be applied to the first value of the two pixels in the group predicted based on the first partition, respectively, and weights of 2 / 3 and 1 / 3 may be applied to the second value of the two pixels in the group predicted based on the second partition, respectively.

[0296] According to the above embodiment, the number of pixels in the group where the first partition and the second partition overlap is an integer. In other implementations, for example, the number of overlapping pixels in the group may be a non-integer and may be a fraction. Likewise, the weights applied to the first value and the second value of a group of pixels may be fractions or integers, depending on each application.

[0297] Similar to Fig. 20 In process 3000, the boundary smoothing process may be performed by, for example, Figure 1 The image encoder shown in the figure is executed, and the image encoder includes a circuit and a memory coupled to the circuit. In operation, the circuit performs a boundary smoothing operation ( Fig. 20 , step S3001). The boundary smoothing operation includes: using information of the first partition, making a first prediction on a first value of a group of pixels of the first partition along the boundary (step S3002); using information of the second partition, making a second prediction on a second value of a group of pixels of the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); encoding the first partition using the weighted first value and the weighted second value (step S3005).

[0298] According to another embodiment, as in Figure 1 As shown in , an image encoder is provided, including: a partitioner 102, which receives an original picture and partitions it into blocks during operation; an adder 104, which receives blocks from the partitioner and predictions from a prediction controller 128 during operation, and subtracts each prediction from its corresponding block to output a residual; a transformer 106, which performs a transform on the residual output from the adder 104 during operation to output a transform coefficient; a quantizer 108, which quantizes the transform coefficient to generate a quantized transform coefficient during operation; an entropy encoder 110, which encodes the quantized transform coefficient to generate a bitstream during operation; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124 and memories 118, 122, wherein the inter-frame predictor 126 generates a prediction for a current block based on a reference block in an encoded reference picture during operation, and the intra-frame predictor 124 generates a prediction for a current block based on an encoded reference block in a current picture during operation. The prediction controller 128 performs a boundary smoothing operation along a boundary between a second partition segmented from the image block and a first partition having a non-rectangular shape. Fig. 20 , step S3001). The boundary smoothing operation includes: using information of the first partition, making a first prediction on a first value of a group of pixels of the first partition along the boundary (step S3002); using information of the second partition, making a second prediction on a second value of a group of pixels of the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); encoding the first partition using the weighted first value and the weighted second value (step S3005).

[0299] According to another embodiment, there is provided an image decoder, such as in Fig.10 As shown in , it includes a circuit and a memory coupled to the circuit. In operation, the circuit performs a boundary smoothing operation along the boundary between the second partition segmented from the image block and the first partition having a non-rectangular shape ( Fig. 20 , step S3001). The boundary smoothing operation includes: using information of the first partition, making a first prediction on a first value of a group of pixels of the first partition along the boundary (step S3002); using information of the second partition, making a second prediction on a second value of a group of pixels of the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); decoding the first partition using the weighted first value and the weighted second value (step S3005).

[0300] According to another embodiment, there is provided a Fig.10The image decoder shown includes: an entropy decoder 202, which receives and decodes the encoded bit stream in operation to obtain quantized transform coefficients; an inverse quantizer 204 and a transformer 206, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder 208, which in operation adds the residual output from the inverse quantizer 204 and the transformer 206 to the prediction output from the prediction controller 220 to reconstruct the block; and a prediction controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216 and memories 210, 214, wherein the inter-frame predictor 218 generates a prediction for the current block based on a reference block in a decoded reference picture in operation, and the intra-frame predictor 216 generates a prediction for the current block based on a decoded reference block in the current picture in operation. The prediction controller 220 performs a boundary smoothing operation along the boundary between the second partition partitioned from the image block and the first partition having a non-rectangular shape in operation ( Fig. 20 , step S3001). The boundary smoothing operation includes: using information of the first partition, making a first prediction on a first value of a group of pixels of the first partition along the boundary (step S3002); using information of the second partition, making a second prediction on a second value of a group of pixels of the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); decoding the first partition using the weighted first value and the weighted second value (step S3005).

[0301] (Entropy encoding and decoding using partition parameter syntax)

[0302] As in Fig.11 As shown in step S1005, according to various embodiments, one or more parameters may be used to encode or decode the image block partitioned into the second partition and the first partition having a non-rectangular shape, wherein the one or more parameters include a partition parameter indicating the non-rectangular partition of the image block. In various embodiments, such partition parameters may be applied to the partition direction (e.g., from the upper left to the lower right or from the upper right to the lower left, see Fig.12 ), and the first motion vector and the second motion vector predicted in step S1002 are jointly encoded, as will be described in more detail below.

[0303] Fig.15 is a table of sample partition parameters ("first index values") and sets of information jointly encoded by the partition parameters. The partition parameters ("first index values") range from 0 to 6 and jointly encode the following items: the partitioning of the image block into the first partition and the second partition (they are both triangles, see Fig.12 ) direction, the first motion vector predicted for the first partition ( Fig.11, step S1002), and predicting a second motion vector for the second partition ( Fig.11 Specifically, the partition direction encoded by partition parameter 0 is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second group of motion vector candidates for the second partition.

[0304] Partition parameter 1 is encoded as follows: the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "first" motion vector listed in the first set of motion vector candidates of the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates of the second partition. Partition parameter 2 is encoded as follows: the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates of the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates of the second partition. Partition parameter 3 is encoded as follows: the partition direction is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates of the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates of the second partition. Partition parameter 4 is encoded as follows: the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates of the first partition, and the second motion vector is the "third" motion vector listed in the second set of motion vector candidates of the second partition. Partition parameter 5 is encoded as follows: the partition direction is from top left to bottom right, the first motion vector is the "third" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 6 is encoded as follows: the partition direction is from top left to bottom right, the first motion vector is the "fourth" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition.

[0305] Fig. 22 4 is a flow chart showing a method 4000 performed at the encoder side. In step S4001, the process divides the image block into a plurality of partitions based on a partition parameter indicating the partition, the plurality of partitions including a second partition and a first partition having a non-rectangular shape. For example, as described above Fig.15As shown, the partition parameter may indicate the direction of partitioning the image block (e.g., from the upper right corner to the lower left corner or from the upper left corner to the lower right corner). In step S4002, the process encodes the first partition and the second partition. In step S4003, the process writes one or more parameters including the partition parameter into the bitstream, and the decoder side may receive and decode the bitstream to obtain the one or more parameters so as to perform the same prediction process (as performed on the encoder side) on the first partition and the second partition on the decoder side. One or more parameters including the partition parameter may jointly or separately encode various information fragments, such as the non-rectangular shape of the first partition, the shape of the second partition, the partition direction for partitioning the image block to obtain the first partition and the second partition, the first motion vector of the first partition, the second motion vector of the second partition, etc.

[0306] Fig.23 is a flowchart showing a method 5000 performed on a decoder side. In step S5001, the process parses one or more parameters from a bitstream, wherein the one or more parameters include a partition parameter indicating that an image block is partitioned into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape. The one or more parameters including the partition parameter parsed from the bitstream can be jointly or separately encoded with various information fragments required for the decoder side to perform the same prediction process as performed on the encoder side, such as the non-rectangular shape of the first partition, the shape of the second partition, the partition direction for partitioning the image block to obtain the first partition and the second partition, the first motion vector of the first partition, the second motion vector of the second partition, etc. In step S5002, the process 5000 partitions the image block into a plurality of partitions based on the partition parameter parsed from the bitstream. In step S5003, the process decodes the first partition and the second partition partitioned from the image block.

[0307] Fig.24 is a table of sample partition parameters ("first index values") and information sets jointly encoded by the partition parameters, which is essentially similar to the above Fig.15 In the sample table. Fig.24 In the partition parameter ("first index value") ranges from 0 to 6, and the following items are jointly encoded: the shape of the first and second partitions partitioned from the image block, the direction of partitioning the image block into the first and second partitions, the first motion vector predicted for the first partition ( Fig.11 , step S1002), and the second motion vector predicted for the second partition ( Fig.11 , step S1002). Specifically, the partition parameter 0 is encoded as follows: neither the first partition nor the second partition has a triangular shape, and therefore, the partition direction information is "N / A", the first motion vector information is "N / A", and the second motion vector information is "N / A".

[0308] Partition parameter 1 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 2 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "first" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 3 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 4 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 5 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "third" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 6 encodes the following: the first partition and the second partition are a triangle, the partition direction is from the upper left corner to the lower right corner, the first motion vector is the "third" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition.

[0309] According to some implementations, the partition parameters (index values) may be binarized according to a binarization scheme that is selected according to the values ​​of at least one or more parameters. Fig.16 An example binarization scheme for binarizing index values ​​(partition parameter values) is shown.

[0310] Fig.25 is a table of sample combinations of a first parameter and a second parameter, wherein one parameter is a partition parameter indicating partitioning of the image block into a plurality of partitions, the plurality of partitions including the second partition and the first partition having a non-rectangular shape. In this example, the partition parameter can be used to indicate the partitioning of the image block without jointly encoding other information encoded by one or more other parameters.

[0311] exist Fig.25 In a first example, a first parameter is used to indicate an image block size, and a second parameter is used as a partition parameter (flag) to indicate that at least one of a plurality of partitions segmented from the image block has a triangular shape. Such a combination of the first parameter and the second parameter can be used to indicate, for example, 1) when the image block size is greater than 64×64, there is no triangular partition, or 2) when the aspect ratio of the image block is greater than 4 (e.g., 64×4), there is no triangular partition.

[0312] exist Fig.25 In the second example, the first parameter is used to indicate the prediction mode, and the second parameter is used as a partition parameter (flag) for indicating that at least one of the plurality of partitions partitioned from the image block has a triangular shape. Such a combination of the first parameter and the second parameter can be used to indicate, for example: 1) when encoding the image block in intra mode, there is no triangular partition.

[0313] exist Fig.25 In the third example of , the first parameter is used as a partition parameter (flag) for indicating that at least one of the plurality of partitions partitioned from the image block has a triangular shape, and the second parameter is used to indicate a prediction mode. Such a combination of the first parameter and the second parameter can be used to indicate, for example: 1) when at least one of the plurality of partitions partitioned from the image block has a triangular shape, the image block must be inter-coded.

[0314] exist Fig.25 In a fourth example of , the first parameter indicates a motion vector of an adjacent block, and the second parameter is used as a partition parameter indicating a direction in which the image block is partitioned into two triangles. Such a combination of the first parameter and the second parameter can be used to indicate, for example: 1) when the motion vector of the adjacent block is a diagonal direction, the direction in which the image block is partitioned into two triangles is from the upper left corner to the lower right corner.

[0315] exist Fig.25 In a fifth example, the first parameter indicates the intra prediction direction of the adjacent block, and the second parameter is used as a partition parameter indicating the direction of partitioning the image block into two triangles. Such a combination of the first parameter and the second parameter can be used to indicate, for example: 1) when the intra prediction direction of the adjacent block is the anti-diagonal direction, the direction of partitioning the image block into two triangles is from the upper right corner to the lower left corner.

[0316] It should be understood that if Fig.15 , 24As shown in and 25, the table of one or more parameters including partition parameters and which information is jointly or separately encoded is presented only as an example, and various other ways of jointly or separately encoding various information as part of the above-mentioned partition syntax operation are also within the scope of the present disclosure. For example, the partition parameter may indicate that the first partition is a triangle, a trapezoid, or a polygon with at least five sides and angles. The partition parameter may indicate that the second partition has a non-rectangular shape, such as a triangle, a trapezoid, and a polygon with at least five sides and angles. The partition parameter may indicate one or more information about the partition, such as the non-rectangular shape of the first partition, the shape of the second partition (which may be non-rectangular or rectangular), and the partition direction applied to partition the image block into multiple partitions (for example, from the upper left corner of the image block to its lower right corner, and from the upper right corner of the image block to its lower left corner). The partition parameter may jointly encode other information, such as the first motion vector of the first partition, the second motion vector of the second partition, the image block size, the prediction mode, the motion vector of the adjacent block, the intra-frame prediction direction of the adjacent block, etc. Alternatively, any of the further information may be encoded separately by one or more parameters other than the partition parameter.

[0317] Similar to Fig. 22 In process 4000, partition syntax operations may be performed by, for example Figure 1 The image encoder shown in the figure is executed, and the image encoder includes a circuit and a memory coupled to the circuit. The circuit performs a partition syntax operation in operation, and the partition syntax operation includes: based on a partition parameter indicating the partition, the image block is partitioned into a plurality of partitions, and the plurality of partitions include a second partition and a first partition having a non-rectangular shape ( Fig. 22 , step S4001); encoding a first partition and a second partition (S4002); writing one or more parameters including partition parameters into a bitstream (S4003).

[0318] According to another embodiment, Figure 1As shown, an image encoder is provided, including: a partitioner 102, which receives an original picture and partitions it into blocks during operation; an adder 104, which receives blocks from the partitioner and predictions from a prediction controller 128 during operation, and subtracts each prediction from its corresponding block to output a residual; a transformer 106, which performs a transformation on the residual output from the adder 104 to output a transformation coefficient during operation; a quantizer 108, which quantizes the transformation coefficient to generate a quantized transformation coefficient during operation; an entropy encoder 110, which encodes the quantized transformation coefficient to generate a bitstream during operation; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124 and memories 118, 122, wherein the inter-frame predictor 126 generates a prediction for a current block based on a reference block in an encoded reference picture during operation, and the intra-frame predictor 124 generates a prediction for a current block based on an encoded reference block in a current picture during operation. The prediction controller 128 in operation partitions the image block into a plurality of partitions based on the partition parameters indicating the partition, the plurality of partitions including the second partition and the first partition having a non-rectangular shape ( Fig. 22 , step S4001), and encode the first partition and the second partition (step S4002). The entropy encoder 110 writes one or more parameters including partition parameters into the bitstream in operation (step S4003).

[0319] According to another embodiment, there is provided an image decoder, such as Fig.10 As shown, the image decoder includes a circuit and a memory coupled to the circuit. The circuit performs a partition syntax operation in operation, and the partition syntax operation includes: parsing one or more parameters from a bit stream, wherein the one or more parameters include a partition parameter indicating that the image block is divided into a plurality of partitions, and the plurality of partitions include a second partition and a first partition having a non-rectangular shape ( Fig.23 , step S5001); dividing the image block into multiple partitions based on the partition parameter (S5002); decoding the first partition and the second partition (S5003).

[0320] According to another embodiment, there is provided Fig.10The image decoder shown in the figure includes: an entropy decoder 202, which receives and decodes the encoded bit stream to obtain quantized transform coefficients in operation; an inverse quantizer 204 and a transformer 206, which in operation inversely quantize the quantized transform coefficients to obtain transform coefficients, and inversely transform the transform coefficients to obtain residuals; an adder 208, which in operation adds the residual output from the inverse quantizer 204 and the transformer 206 to the prediction output from the prediction controller 220 to reconstruct the block; and a prediction controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216 and memories 210, 214, wherein the inter-frame predictor 218 generates a prediction for a current block based on a reference block in a decoded reference picture in operation, and the intra-frame predictor 216 generates a prediction for the current block based on a decoded reference block in the current picture in operation. The entropy decoder 202 is operated by parsing one or more parameters from a bitstream, wherein the one or more parameters include a partition parameter indicating partitioning of the image block into a plurality of partitions, wherein the plurality of partitions include a second partition and a first partition having a non-rectangular shape ( Fig.23 , step S5001); partitioning the image block into a plurality of partitions based on a partition parameter (S5002); in some implementations, decoding the first partition and the second partition in cooperation with the prediction controller 220 (S5003).

[0321] (Implementation and Application)

[0322] As described in each of the above embodiments, each function or operation block can generally be implemented as, for example, an MPU (microprocessing unit) and a memory. In addition, the process performed by each of the function blocks can be implemented as a program execution unit, such as a processor that reads and executes software (programs) recorded on a recording medium such as a ROM. The software can be distributed. The software can be recorded on various recording media such as semiconductor memories. Note that each function block can also be implemented as hardware (dedicated circuit).

[0323] The processing described in each embodiment can be implemented by integrated processing using a single device (system), or can be implemented by decentralized processing using multiple devices. In addition, the processor that executes the above program can be a single processor or multiple processors. In other words, integrated processing can be performed, or decentralized processing can be performed.

[0324] The embodiments of the present disclosure are not limited to the above exemplary embodiments; various modifications may be made to the exemplary embodiments, and the results thereof are also included in the scope of the embodiments of the present disclosure.

[0325] Next, application examples of the motion picture encoding method (image encoding method) and the motion picture decoding method (image decoding method) described in each of the above embodiments, as well as various systems for implementing the application examples, will be described. Such a system may be characterized by including an image encoder using an image encoding method, an image decoder using an image decoding method, or an image encoder-decoder including both an image encoder and an image decoder. Other configurations of such a system may be modified as appropriate.

[0326] (Usage example)

[0327] Fig.26 The overall configuration of a content providing system ex100 suitable for implementing content distribution services is shown. An area where communication services are provided is divided into cells of desired sizes, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations in the example shown, are located in each cell.

[0328] In the content providing system ex100, devices including a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via an Internet service provider ex102 or a communication network ex104 and base stations ex106 to ex110. The content providing system ex100 can combine and connect any combination of the above devices. In various implementations, the devices can be directly or indirectly connected together via a telephone network or near field communication rather than via base stations ex106 to ex110. In addition, a streaming media server ex103 can be connected to devices including a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 via, for example, the Internet ex101. The streaming media server ex103 can also be connected to a terminal in a hotspot such as in an airplane ex117 via a satellite ex116.

[0329] Note that wireless access points or hotspots may be used instead of base stations ex106 to ex110. The streaming server ex103 may be directly connected to the communication network ex104 instead of through the Internet ex101 or the Internet service provider ex102, and may be directly connected to the aircraft ex117 instead of through the satellite ex116.

[0330] The camera ex113 is a device capable of capturing still images and videos, such as a digital camera. The smartphone ex115 is a smartphone device, a cellular phone, or a Personal Handyphone System (PHS) phone that can operate under mobile communication system standards of 2G, 3G, 3.9G, and 4G systems and the next-generation 5G system.

[0331] The household appliance ex114 is, for example, a refrigerator or a device included in a household fuel cell cogeneration system.

[0332] In the content providing system ex100, a terminal including an image and / or video capturing function can perform real-time streaming, for example, by connecting to a streaming server ex103 via, for example, a base station ex106. When performing real-time streaming, a terminal (e.g., a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or an airplane ex117) can perform the encoding process described in the above embodiments on a still image or video content captured by a user via the terminal, can multiplex video data obtained via encoding and audio data obtained by encoding audio corresponding to the video, and can send the obtained data to the streaming server ex103. In other words, the terminal functions as an image encoder according to one aspect of the present disclosure.

[0333] The streaming media server ex103 streams the sent content data to the client that requested the stream. Examples of clients include computers ex111, game devices ex112, cameras ex113, home appliances ex114, smartphones ex115, and terminals in airplanes ex117, which are capable of decoding the above-mentioned encoded data. The device that receives the streamed data will decode and reproduce the received data. In other words, each of the devices can act as an image decoder according to one aspect of the present disclosure.

[0334] (Decentralized processing)

[0335] Streaming media server ex103 can be implemented as multiple servers or computers that divide tasks such as processing, recording and streaming of data between them. For example, streaming media server ex103 can be implemented as a content delivery network (CDN), which streams content via a network connecting multiple edge servers located all over the world. In CDN, edge servers physically close to clients are dynamically allocated to clients. Content is cached and streamed to edge servers to reduce loading time. For example, in the event that some type of error or connectivity changes due to traffic peaks, data can be stably streamed at high speed, because the affected part of the network can be avoided in the following ways: for example, processing is divided between multiple edge servers, or streaming responsibilities are switched to different edge servers and streaming continues.

[0336] Decentralization is not limited to dividing processing for streaming; the encoding of the captured data can be distributed between terminals on the server side, or performed by the terminal, or both. In one example, in a typical encoding, the processing is performed in two loops. The first loop is used to detect the complexity of the image on a frame-by-frame or scene-by-scene basis, or to detect the encoding load. The second loop is used for processing to maintain image quality and improve encoding efficiency. For example, it is possible to reduce the processing responsibility of the terminal and improve the quality and encoding efficiency of the content by causing the terminal to perform a first loop of encoding, and causing the server side receiving the content to perform a second loop of encoding. In such a case, when a decoding request is received, it is possible that the encoded data generated by the first loop performed on one terminal is received and reproduced on another terminal in approximately real time. This makes it possible to achieve smooth, real-time streaming.

[0337] In another example, the camera ex113 or the like extracts a feature quantity from an image, compresses data related to the feature quantity into metadata, and transmits the compressed metadata to a server. For example, the server determines the importance of an object based on the feature quantity, and changes the quantization accuracy accordingly to perform compression suitable for the meaning (or content importance) of the image. The feature quantity data is particularly effective in improving the accuracy and efficiency of motion vector prediction during the second compression performed by the server. In addition, the terminal can process encoding with a relatively low processing load such as variable length coding (VLC), and encoding with a relatively high processing load such as context adaptive binary arithmetic coding (CABAC) can be processed by the server.

[0338] In another example, there is a situation where multiple terminals, such as in a stadium, a shopping mall, or a factory, capture multiple videos of approximately the same scene. In such a case, for example, decentralized encoding can be achieved by dividing the processing tasks between multiple terminals that capture the video and other terminals and servers that do not capture the video on a per-unit basis. The unit can be, for example, a group of pictures (GOP), a picture, or a tile obtained after dividing a picture. This can reduce loading time and achieve closer to real-time streaming.

[0339] Since the videos have substantially the same scenes, the server can perform management and / or instructions so that the videos captured by the terminal can be cross-referenced. In addition, the server can receive encoded data from the terminal, change the reference relationship between the data items, or correct or replace the pictures itself, and then perform encoding. This makes it possible to generate streams with improved quality and efficiency for individual data items.

[0340] In addition, the server can stream the video data after performing code conversion to convert the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (e.g., VP9), and can convert H.264 to H.265.

[0341] In this way, encoding can be performed by a terminal or one or more servers. Therefore, although the device performing encoding is referred to as a "server" or a "terminal" in the following description, some or all of the processes performed by the server can be performed by the terminal, and similarly, the processing performed by the terminal can be performed by the server. This also applies to the decoding process.

[0342] (3D, multi-angle)

[0343] The use of images or videos combined from images or videos of different scenes captured simultaneously or images or videos of the same scene captured from different angles through multiple terminals (e.g., camera ex113 and / or smartphone ex115) has increased. Videos captured by the terminals are combined based on, for example, relative positional relationships obtained separately between the terminals or areas with matching feature points in the videos.

[0344] In addition to encoding two-dimensional motion pictures, the server can also encode still images automatically or at a user-specified time point based on scene analysis of the motion pictures, and send the encoded still images to the receiving terminal. In addition, when the server is able to obtain the relative positional relationship between the video capture terminals, in addition to the two-dimensional motion pictures, the server can also generate the three-dimensional geometry of the scene based on videos of the same scene captured from different angles. The server can encode the three-dimensional data generated from, for example, a point cloud, respectively, and based on the results of identifying or tracking a person or object using the three-dimensional data, the server can select or reconstruct and generate a video to be sent to the receiving terminal from the videos captured by multiple terminals.

[0345] This allows the user to enjoy the scene by freely selecting the video corresponding to the video capture terminal, and allows the user to enjoy the content obtained by extracting the video at a selected viewpoint from the three-dimensional data reconstructed from multiple images or videos. In addition, like the video, the sound can be recorded from relatively different angles, and the server can multiplex the audio of a specific angle or space with the corresponding video and send the multiplexed video and audio.

[0346] In recent years, content consisting of the real world and the virtual world, such as virtual reality (VR) and augmented reality (AR) content, has also become popular. In the case of VR images, the server can create images from both the left eye and the right eye viewpoints, and perform encoding that allows reference between the two viewpoint images, such as multi-view coding (MVC), or the image can be encoded as a separate stream without reference. When the image is decoded into a separate stream, the stream can be synchronized when reproduced so as to recreate a virtual three-dimensional space according to the user's viewpoint.

[0347] In the case of an AR image, the server superimposes virtual object information existing in the virtual space on the camera information representing the real space based on the three-dimensional position or movement from the user's perspective. The decoder can acquire or store virtual object information and three-dimensional data, generate a two-dimensional image based on the movement from the user's perspective, and then generate superimposed data by seamlessly connecting the images. Alternatively, in addition to the request for virtual object information, the decoder can also send the movement from the user's perspective to the server. The server can generate superimposed data based on the three-dimensional data stored in the server according to the received movement, and encode the generated superimposed data and stream it to the decoder. Note that the superimposed data includes an alpha value indicating transparency in addition to the RGB value, and the server sets the alpha value of the part other than the object generated from the three-dimensional data to, for example, 0, and can be encoded while these parts are transparent. Alternatively, the server can set the background to a predetermined RGB value (e.g., a chroma key), and generate data that sets the area outside the object as the background.

[0348] The decoding of similarly streamed data can be performed by the client (i.e., terminal) on the server side, or divided between them. In one example, a terminal can send a reception request to a server, the requested content can be received and decoded by another terminal, and the decoded signal can be sent to a device with a display. Regardless of the processing power of the communication terminal itself, high image quality data can be reproduced by decentralized processing and appropriate selection of content. In another example, for example, when a TV is receiving large-size image data, areas of a picture such as tiles obtained by dividing a picture can be decoded and displayed on a personal terminal or on the terminals of one or more viewers of the TV. This allows viewers to share a large picture view, and each viewer can check their assigned area or view the area more closely.

[0349] In the case where multiple wireless connections can be made indoors, outdoors, at close range, medium range, and long range, content can be received seamlessly using streaming system standards such as MPEG-DASH. Users can switch between data in real time while freely selecting decoders or display devices including user terminals, displays arranged indoors or outdoors, etc. In addition, using information such as the user's location, decoding can be performed while switching which terminal handles decoding and which terminal handles the display of content. When the user is on the way to the destination, this makes it possible to draw and display information on the wall of a nearby building or on a part of the ground in which a device capable of displaying content is embedded. In addition, the bit rate of the received data can also be switched based on the accessibility of the encoded data on the network, such as when the encoded data is cached on a server that can be quickly accessed from the receiving terminal, or when the encoded data is copied to the edge server in the content delivery service.

[0350] (Scalable Coding)

[0351] Reference will be made to Fig. 27 The scalable stream shown in is used to describe the switching of the content, which is compressed and encoded by implementing the motion picture encoding method described in the above embodiment. The server may have a configuration in which the content is switched while utilizing the temporal and / or spatial scalability of the stream, which is achieved by dividing and encoding the layers, such as Fig. 27 As shown. Note that there can be multiple individual streams with the same content but different qualities. In other words, by determining the layer to be decoded based on internal factors such as the processing power on the decoder side and external factors such as the communication bandwidth, the decoder side can freely switch between low-resolution content and high-resolution content while decoding. For example, in the case where a user wants to continue watching a video that the user previously watched on the smartphone ex115 while on the move, for example, using a device such as a TV connected to the Internet at home, the device can simply decode the same stream up to a different layer, which reduces the load on the server side.

[0352] In addition, in addition to the above configuration (where scalability is achieved by encoding pictures by layer), and the enhancement layer is located above the base layer, the enhancement layer may also include metadata based on, for example, statistical information about the image. The decoder side can generate content with high image quality by performing super-resolution imaging on the pictures in the base layer based on the metadata. Super-resolution imaging can improve the signal-to-noise ratio while maintaining and / or increasing the resolution. The metadata includes information for identifying linear or nonlinear filter coefficients used in super-resolution processing, or information for identifying parameter values ​​of filtering processing, machine learning, or least squares method used in super-resolution processing.

[0353] Alternatively, a configuration may be provided in which a picture is divided into, for example, tiles according to the meaning of, for example, an object in the image. On the decoder side, only a partial area is decoded by selecting a tile to be decoded. In addition, by storing the properties of an object (a person, a car, a ball, etc.) and the position of the object in the video (coordinates in the same image) as metadata, the decoder side can identify the position of the desired object based on the metadata and determine which tile or tiles contain the object. For example, Fig.28 As shown, a data storage structure different from pixel data can be used to store metadata, such as SEI (Supplementary Enhancement Information) messages in HEVC. The metadata indicates, for example, the position, size, or color of the main object.

[0354] The metadata can be stored in units of multiple pictures, such as streams, sequences, or random access units. The decoder side can obtain, for example, the time when a specific person appears in the video, and by fitting the time information with the picture unit information, the picture in which the object exists can be identified, and the position of the object in the picture can be determined.

[0355] (Web Optimization)

[0356] Fig.29 An example of a display screen of a web page on the computer ex111 is shown, for example. Fig.30 FIG. 4 shows an example of a display screen of a web page on a smartphone ex115. Fig.29 and Fig.30 As shown in , a web page may include multiple image links as links to image content, and the appearance of the web page differs depending on the device used to view the web page. When multiple image links are visible on the screen, until the user explicitly selects the image link, or until the image link is located in the approximate center of the screen, or until the entire image link fits the screen, the display device (decoder) may display a still image included in the content or an I picture as the image link; may use multiple still images or I pictures to display a video such as an animated gif; or may receive only the base layer, and decode and display the video.

[0357] When the user selects an image link, the display device performs decoding while giving the highest priority to the base layer. Note that if there is information in the HTML code of the web page indicating that the content is scalable, the display device can decode up to the enhancement layer. In addition, in order to ensure real-time reproduction, before making a selection or when the bandwidth is strictly limited, the display device can reduce the delay between the time point when the leading picture is decoded and the time point when the decoded picture is displayed (i.e., the delay between the start of content decoding and the display of the content) by decoding and displaying only forward reference pictures (I pictures, P pictures, forward reference B pictures). In addition, the display device can intentionally ignore the reference relationship between pictures, and roughly decode all B and P pictures as forward reference pictures, and then perform regular decoding as the number of pictures received over a period of time increases.

[0358] (Autonomous Driving)

[0359] When sending and receiving still images or video data (e.g., two-dimensional or three-dimensional map information for autonomous or assisted driving cars), the receiving terminal may receive information about, for example, weather or road construction as metadata in addition to image data belonging to one or more layers, and associate the metadata with the image data when decoding. Note that the metadata may be allocated by layer, and may alternatively be multiplexed with the image data alone.

[0360] In this case, since the car, drone, airplane, etc. including the receiving terminal is movable, the receiving terminal can seamlessly receive and perform decoding by transmitting information indicating the location of the receiving terminal while switching between base stations among base stations ex106 to ex110. In addition, the receiving terminal can dynamically select the degree of receiving metadata or the degree of updating, for example, map information, according to the user's selection, the user's condition, and / or the bandwidth of the connection.

[0361] In the content providing system ex100, the client can receive, decode, and reproduce the encoded information sent by the user in real time.

[0362] (Streaming individual content)

[0363] In the content providing system ex100, in addition to high image quality, long content distributed by the video distribution entity, unicast or multicast streams with low image quality, and short content from individuals are also possible. The popularity of such content from individuals is likely to increase further. The server can first perform editing processing on the content before the encoding process in order to refine the individual content. For example, this can be achieved using the following configuration.

[0364] When capturing video or image content in real time, or after capturing and accumulating the content, the server performs recognition processing based on the original data or the encoded data, such as capture error processing, scene search processing, meaning analysis, and / or object detection processing. Then, based on the results of the recognition processing, the server edits the content when prompted or automatically, examples of which include: corrections such as focus and / or motion blur correction; removal of low-priority scenes, such as scenes with lower brightness compared to other pictures, or scenes that are out of focus; object edge adjustment; and hue adjustment. The server encodes the edited data based on the results of the editing. It is well known that videos that are too long will result in a decrease in the number of views. Therefore, in order to keep the content within a specific length equivalent to the length of the original video, in addition to the low-priority scenes mentioned above, the server can also automatically crop out scenes with low motion based on the image processing results. Alternatively, the server can generate and encode a video summary based on the analysis results of the meaning of the scene.

[0365] In some cases, individual content may include content that infringes copyright, moral rights, portrait rights, etc. Such instances may bring disadvantageous situations to the creator, such as when the content is shared beyond the scope that the creator intends. Thus, before encoding, the server can, for example, edit the image so that the faces of people on the periphery of the screen are blurred or the interior of the house is blurred. In addition, the server can be configured to identify the faces of people other than registered people in the image to be encoded, and when such faces appear in the image, a mosaic filter can be applied to, for example, the face of the person. Alternatively, as pre-processing or post-processing for encoding, the user can specify an image area including a person or an area of ​​the background to be processed for copyright reasons. The server can process the specified area by, for example, replacing the area with a different image or blurring the area. If the area includes a person, the person can be tracked in a moving picture, and the head area of ​​the person can be replaced with another image as the person moves.

[0366] Due to the need to watch individually generated content that tends to have a smaller data size in real time, the decoder first receives the base layer as the highest priority and performs decoding and reproduction, but this may vary depending on the bandwidth. When the content is reproduced two or more times, for example when the decoder receives an enhancement layer during decoding and reproduction of the base layer and loops the reproduction, the decoder can reproduce a video with high image quality including the enhancement layer. If the stream is encoded using such scalable coding, the video may be of lower quality when it is not selected or at the beginning of the video, but it can provide an experience in which the image quality of the stream gradually improves in an intelligent manner. This is not limited to scalable coding; a single stream can be configured from a low-quality stream reproduced for the first time and a second stream encoded using the first stream as a reference, thereby providing the same experience.

[0367] (Other implementation and application examples)

[0368] The encoding and decoding can be performed by an LSI (Large Scale Integrated Circuit) ex500 (see Fig.26 ) is executed. LSI ex500 may be configured by a single chip or a plurality of chips. Software for encoding and decoding motion pictures may be integrated into a certain type of recording medium (e.g., a CD-ROM, a floppy disk, or a hard disk), and for example, the computer ex111 may read the recording medium, and encoding and decoding may be performed using the software. In addition, when the smartphone ex115 is equipped with a camera, video data acquired by the camera may be transmitted. In this case, the video data is encoded by the LSI ex500 included in the smartphone ex115.

[0369] Note that LSI ex500 can be configured to download and activate applications. In such a case, the terminal first determines whether it is compatible with the scheme used to encode the content, or whether it can perform a specific service. When the terminal is incompatible with the encoding scheme of the content, or when the terminal cannot perform a specific service, the terminal first downloads the codec or application software, and then obtains and reproduces the content.

[0370] In addition to the example of the content providing system ex100 using the Internet ex101, at least the moving picture encoder (image encoder) or the moving picture decoder (image decoder) described in the above embodiments can be implemented in a digital broadcasting system. The same encoding process and decoding process can be applied to, for example, transmitting and receiving broadcast radio waves superimposed with multiplexed audio and video data using a satellite, even though this is for multicast, unicast is easier for the content providing system ex100.

[0371] (Hardware Configuration)

[0372] Fig.31 Shows Fig.26 More details of the smartphone ex115 shown in. Fig.32An example of a configuration of a smartphone ex115 is shown. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves to and from the base station ex110, a camera ex465 capable of capturing video and still images, and a display ex458 for displaying decoded data such as video captured by the camera ex465 and video received by the antenna ex450. The smartphone ex115 also includes a user interface ex466 such as a touch panel, an audio output unit ex457 such as a speaker for outputting voice or other audio, an audio input unit ex456 such as a microphone for audio input, a memory ex467 capable of storing decoded data such as captured video or still images, recorded audio, received video or still images, mails, and decoded data, and a slot ex464 which is an interface of a SIM ex468 for authorizing access to a network and various data. Note that an external memory may be used in place of the ex467.

[0373] A main controller ex460 that can fully control the display ex458 and the user interface ex466, a power supply circuit ex461, a user interface input controller ex462, a video signal processor ex455, a camera interface ex463, a display controller ex459, a modulator / demodulator ex452, a multiplexer / demultiplexer ex453, an audio signal processor ex454, a slot ex464, and a memory ex467 are connected via a bus ex470.

[0374] When the user turns on the power button of the power circuit ex461, the smartphone ex115 is powered on to enter an operable state, and power is supplied to each component from the battery pack.

[0375] The smart phone ex115 performs processing such as calling and data transmission based on control performed by the main controller ex460 including a CPU, ROM, and RAM. When making a call, the audio signal recorded by the audio input unit ex456 is converted into a digital audio signal by the audio signal processor ex454, and the digital audio signal is spread spectrum processed and digital-to-analog converted by the modulator / demodulator ex452, and frequency converted by the transmitter / receiver ex451, and the resulting signal is transmitted via the antenna ex450. The received data is amplified, frequency converted and analog-to-digital converted, and inversely spread spectrum processed by the modulator / demodulator ex452, converted into an analog audio signal by the audio signal processor ex454, and then output from the audio output unit ex457. In the data transmission mode, for example, based on the operation of the user interface ex466 of the main body, the main controller ex460 transmits text, still images, or video data via the user interface input controller ex462. Similar transmission and reception processing is performed. In the data transmission mode, when transmitting video, still images, or video and audio, the video signal processor ex455 compression-encodes the video signal stored in the memory ex467 or the video signal input from the camera ex467 via the motion picture encoding method described in the above embodiments, and transmits the encoded video data to the multiplexer / demultiplexer ex453. The audio signal processor ex454 encodes the audio signal recorded by the audio input unit ex456 while the camera ex465 is capturing video or still images, and transmits the encoded audio data to the multiplexer / demultiplexer ex453. The multiplexer / demultiplexer ex453 multiplexes the encoded video data and the encoded audio data using a predetermined scheme, modulates and converts the data using the modulator / demodulator (modulator / demodulator circuit) ex452 and the transmitter / receiver ex451, and transmits the result via the antenna ex450.

[0376] For example, when a video attached to an email or a chat or a video linked from a web page is received, in order to decode the multiplexed data received via the antenna ex450, the multiplexer / demultiplexer ex453 demultiplexes the multiplexed data to divide the multiplexed data into a bit stream of video data and a bit stream of audio data, supplies the encoded video data to the video signal processor ex455 via the synchronous bus ex470, and supplies the encoded audio data to the audio signal processor ex454 via the synchronous bus ex470. The video signal processor ex455 decodes the video signal using a moving picture decoding method corresponding to the moving picture encoding method described in the above embodiment, and the video or still image included in the linked moving picture file is displayed on the display ex458 via the display controller ex459. The audio signal processor ex454 decodes the audio signal and outputs the audio from the audio output unit ex457. Since real-time streaming is becoming more and more popular, the reproduction of audio may be socially inappropriate in some cases depending on the user's environment. Thus, as an initial value, a configuration is preferred in which only video data is reproduced, ie, audio signals are not reproduced; audio can be synchronized and reproduced only when an input is received (eg, when a user clicks on video data).

[0377] Although the smartphone ex115 is used in the above example, three other implementations are conceivable: a transceiver terminal including both an encoder and a decoder; a transmitter terminal including only an encoder; a receiver terminal including only a decoder. In the description of the digital broadcasting system, an example is given in which multiplexed data obtained by multiplexing video data with audio data is received or transmitted. However, the multiplexed data may be video data multiplexed with data other than audio data (e.g., text data related to the video). Furthermore, video data itself may be received or transmitted instead of the multiplexed data.

[0378] Although the main controller ex460 including the CPU is described as controlling the encoding or decoding process, various terminals generally include a GPU. Thus, a configuration is acceptable in which a large area is processed at once by utilizing the performance capabilities of the GPU via a memory shared by the CPU and the GPU or a memory including addresses managed to allow common use of the CPU and the GPU. This can shorten the encoding time, maintain the real-time nature of the stream, and reduce latency. In particular, processing related to motion estimation, deblocking filtering, sample adaptive offset (SAO), and transform / quantization can be effectively performed all at once by the GPU instead of the CPU in units of pictures.

Claims

1. An image encoder, comprising: Circuit; as well as a memory coupled to the circuit; Wherein, the circuit is in operation: Using a first motion vector of a first partition, calculating a first value of a pixel of the first partition in a current block; Calculating a second value of a pixel of the second partition in the current block using a second motion vector of the second partition; Using the first motion vector, calculating a third value of a group of pixels between the first partition and the second partition; calculating a fourth value of the set of pixels using the second motion vector; weighting the third value and the fourth value of the group of pixels; encoding the current block using the weighted third value and the weighted fourth value, Wherein the number of pixels in a row in the group of pixels is greater than two.

2. The image encoder according to claim 1, wherein: The weights greater than two applied to the third values ​​of the group of pixels increase in a first direction along the row.

3. The image encoder according to claim 2, wherein: The weights greater than two applied to the fourth values ​​of the group of pixels decrease in the first direction along the row.

4. An image decoder, comprising: Circuit; as well as a memory coupled to the circuit; Wherein, the circuit is in operation: Using a first motion vector of a first partition, calculating a first value of a pixel of the first partition in a current block; Calculating a second value of a pixel of the second partition in the current block using a second motion vector of the second partition; Using the first motion vector, calculating a third value of a group of pixels between the first partition and the second partition; calculating a fourth value of the set of pixels using the second motion vector; weighting the third value and the fourth value of the group of pixels; decoding the current block using the weighted third value and the weighted fourth value, Wherein the number of pixels in a row in the group of pixels is greater than two.

5. The image decoder according to claim 4, wherein: The weights greater than two applied to the third values ​​of the group of pixels increase in a first direction along the row.

6. The image decoder according to claim 5, wherein: The weights greater than two applied to the fourth values ​​of the group of pixels decrease in the first direction along the row.

Images