Image encoder, image decoder, and bitstream generation apparatus

Abstract

An image encoder, an image decoder, and a non-transitory computer readable medium are provided. The image encoder includes circuitry and a memory coupled to the circuitry; wherein the circuitry, in operation, performs the following operations: selects, for a first partition having a non-rectangular shape in a picture block, a first motion vector from a set of motion vector candidates; selects, for a second partition in the picture block that overlaps the first partition, a second motion vector from the set of motion vector candidates; encodes the first partition using the first motion vector; and encodes the second partition using the second motion vector, the circuitry selecting only a single-prediction motion vector from the set of motion vector candidates.

Description

[0001] This application is a divisional application of the patent application No. 201880054452.4 (International Application No. PCT / JP2018 / 030059) filed on August 10, 2018, with the title of “Image encoder, image decoder, image encoding method, and image decoding method”. TECHNICAL FIELD

[0002] The present disclosure relates to video coding, and in particular, to video coding and decoding systems, components, and methods for performing inter-prediction function of constructing a current block based on a reference frame or intra-prediction function of constructing the current block based on an encoded / decoded reference block in the current frame. BACKGROUND

[0003] As video coding technology has evolved from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec), there has always been a need for improvement and optimization of video coding technology to handle the ever-increasing amount of digital video data in various applications. The present disclosure relates to further advancement, improvement, and optimization in video coding, in particular, in connection with inter-prediction function or intra-prediction function that partitions an image block into a plurality of partitions including at least a first partition having a non-rectangular shape (e.g., a triangle) and a second partition. SUMMARY

[0004] According to one aspect, there is provided an image encoder comprising: circuitry; and a memory coupled to the circuitry; wherein the circuitry, in operation, performs the following operations: selecting, for a first partition having a non-rectangular shape in an image block, a first motion vector from a set of motion vector candidates; selecting, for a second partition in the image block that overlaps the first partition, a second motion vector from the set of motion vector candidates; encoding the first partition using the first motion vector; and encoding the second partition using the second motion vector, the circuitry selecting only a single-prediction motion vector from the set of motion vector candidates.

[0005] Further, an image decoder is provided, comprising: circuitry; a memory coupled to the circuitry; wherein the circuitry in operation performs the following: selecting, for a first partition having a non-rectangular shape in a picture block, a first motion vector from a set of motion vector candidates; selecting, for a second partition overlapping the first partition in the picture block, a second motion vector from the set of motion vector candidates; decoding the first partition using the first motion vector; and decoding the second partition using the second motion vector; the circuitry selecting only uni-prediction motion vectors from the set of motion vector candidates.

[0006] Further, a non-transitory computer readable medium storing a bitstream is provided, the bitstream comprising information causing a decoder to perform a partitioning process in accordance therewith, in which: a first motion vector for a first partition having a non-rectangular shape in a picture block is selected from a set of motion vector candidates; a second motion vector for a second partition overlapping the first partition in the picture block is selected from the set of motion vector candidates; the first partition is decoded using the first motion vector; and the second partition is decoded using the second motion vector; only uni-prediction motion vectors are selected from the set of motion vector candidates.

[0007] Further, an image encoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry in operation performs the following: partitioning a picture block into a second partition and a first partition having a non-rectangular shape; predicting a first motion vector for the first partition and a second motion vector for the second partition; and encoding the first partition using the first motion vector and the second partition using the second motion vector.

[0008] Some implementations of embodiments of the present disclosure can improve coding efficiency, can simply be encoding / decoding processes, can speed up the speed of encoding / decoding processes, can efficiently select suitable components / operations used in encoding and decoding, such as suitable filters, block sizes, motion vectors, reference pictures, reference blocks, etc.

[0009] Other benefits and advantages of the disclosed embodiments will become apparent from the description and drawings. Benefits and / or advantages can be separately provided by various embodiments and features of the description and drawings, which need not all be provided in order to obtain one or more of such benefits and / or advantages.

[0010] It should be noted that general or specific embodiments can be implemented as a system, a method, an integrated circuit, a computer program, a storage medium, or any selective combination thereof. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1is a block diagram showing a functional configuration of an encoder according to an embodiment.

[0012] Figure 2 One example of block partitioning is shown.

[0013] Figure 3 is a table indicating transform basis functions of various transform types.

[0014] Figure 4A One example of filter shapes used in ALF (Adaptive Loop Filter) is shown.

[0015] Figure 4B Another example of filter shapes used in ALF is shown.

[0016] Figure 4C Another example of filter shapes used in ALF is shown.

[0017] Figure 5A 67 intra prediction modes used in an example of intra prediction are shown.

[0018] Figure 5B is a flowchart showing one example of a prediction picture correction process performed in OBMC (Overlapped Block Motion Compensation) processing.

[0019] Figure 5C is a conceptual diagram showing one example of a prediction picture correction process performed in OBMC processing.

[0020] Figure 5D is a flowchart showing one example of FRUC (Frame Rate Up Conversion) processing.

[0021] Figure 6 One example of pattern matching between two blocks along a motion trajectory (bi-directional matching) is shown.

[0022] Figure 7 One example of pattern matching between a template in a current picture and a block in a reference picture (template matching) is shown.

[0023] Figure 8 A model assuming uniform linear motion is shown.

[0024] Figure 9A One example of deriving a motion vector for each sub-block based on motion vectors of neighboring blocks is shown.

[0025] Figure 9B One example of a process for deriving a motion vector in merge mode is shown.

[0026] Figure 9Cis a conceptual diagram showing an example of DMVR (Dynamic Motion Vector Refinement) processing.

[0027] Figure 9D An example of a prediction image generation method using a brightness correction process performed by LIC (Local Illumination Compensation) processing is shown.

[0028] Figure 10 is a block diagram showing a functional configuration of a decoder according to an embodiment.

[0029] Figure 11 is a flowchart showing an overall processing flow of dividing an image block into a plurality of partitions including at least a first partition having a non-rectangular shape (e.g., a triangle) and a second partition and performing further processing according to one embodiment.

[0030] Figure 12 Two exemplary methods of dividing an image block into a second partition (also having a non-rectangular shape in the shown example) and a first partition having a non-rectangular shape (e.g., a triangle) are shown.

[0031] Figure 13 An example of a boundary smoothing process is shown, which includes weighting a first value of a boundary pixel based on a first partition prediction and a second value of the boundary pixel based on a second partition prediction.

[0032] Figure 14 Three further examples of a boundary smoothing process are shown, which involve weighting a first value of a boundary pixel based on a first partition prediction and weighting a second value of the boundary pixel based on a second partition prediction.

[0033] Figure 15 is a table of sampling parameters (“first index values”) and information sets encoded by the parameters, respectively.

[0034] Figure 16 is a table showing binarization of a parameter (index value).

[0035] Figure 17 is a flowchart showing a process of dividing an image block into a plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0036] Figure 18 An example of dividing an image block into a plurality of partitions including a second partition and a first partition having a non-rectangular shape (a triangle in the shown example) is shown.

[0037] Figure 19A further example is shown of a segmentation of an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape, which in the shown example is a polygon having at least five edges and corners.

[0038] Figure 20 is a flowchart showing a boundary smoothing process including weighting a first value of a boundary pixel based on a first partition prediction and weighting a second value of the boundary pixel based on a second partition prediction.

[0039] Figure 21A An example of a boundary smoothing process is shown, in which a boundary pixel whose first value is to be weighted based on a first partition prediction and whose second value is to be weighted based on a second partition prediction.

[0040] Figure 21B An example of a boundary smoothing process is shown, in which a boundary pixel whose first value is to be weighted based on a first partition prediction and whose second value is to be weighted based on a second partition prediction.

[0041] Figure 21C An example of a boundary smoothing process is shown, in which a boundary pixel whose first value is to be weighted based on a first partition prediction and whose second value is to be weighted based on a second partition prediction.

[0042] Figure 21D An example of a boundary smoothing process is shown, in which a boundary pixel whose first value is to be weighted based on a first partition prediction and whose second value is to be weighted based on a second partition prediction.

[0043] Figure 22 is a flowchart showing a method performed at an encoder side of segmenting an image block into a plurality of partitions based on partition parameters indicative of the segmentation and writing one or more parameters including the partition parameters into a bitstream in entropy coding, the plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0044] Figure 23 is a flowchart showing a method performed at a decoder side of parsing one or more parameters from a bitstream, the parameters including a method of segmenting an image block into a plurality of partitions based on partition parameters and decoding a first partition and a second partition, the one or more parameters including partition parameters indicative of the segmentation of the image block into the plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0045] Figure 24is a table of sampling partition parameters ("first index values") and a set of information that can be jointly coded by the partition parameters, respectively, the sampling partition parameters respectively indicating a partitioning of an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0046] Figure 25 is a table of a sampling combination of a first parameter and a second parameter, wherein one of the first parameter and the second parameter is a partition parameter indicating a partitioning of an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0047] Figure 26 An overall configuration of a content providing system for implementing a content distribution service is shown.

[0048] Figure 27 An example of an encoding structure in scalable coding is shown.

[0049] Figure 28 An example of an encoding structure in scalable coding is shown.

[0050] Figure 29 An example of a display screen of a web page is shown.

[0051] Figure 30 An example of a display screen of a web page is shown.

[0052] Figure 31 An example of a smartphone is shown.

[0053] Figure 32 A block diagram showing an example of a configuration of a smartphone is shown. DETAILED DESCRIPTION

[0054] According to one aspect, there is provided an image encoder comprising circuitry and a memory coupled to the circuitry. The circuitry in operation performs the following: partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; predicting a first motion vector for the first partition and a second motion vector for the second partition; and encoding the first partition using the first motion vector and the second partition using the second motion vector.

[0055] According to another aspect, the second partition has a non-rectangular shape. According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0056] According to another aspect, the predicting includes selecting a first motion vector from a first set of motion vector candidates and selecting a second motion vector from a second set of motion vector candidates. For example, the first set of motion vector candidates can include motion vectors of partitions neighboring the first partition and the second set of motion vector candidates can include motion vectors of partitions neighboring the second partition. The partitions neighboring the first partition and the partitions neighboring the second partition can be outside of the image block from which the first partition and the second partition are partitioned. The neighboring partitions can be one or both of spatially neighboring partitions and temporally neighboring partitions. The first set of motion vector candidates can be the same as or different from the second set of motion vector candidates.

[0057] According to another aspect, the predicting includes selecting a first motion vector candidate from a first set of motion vector candidates and deriving the first motion vector by adding a first motion vector difference to the first motion vector candidate, and selecting a second motion vector candidate from a second set of motion vector candidates and deriving the second motion vector by adding a second motion vector difference to the second motion vector candidate.

[0058] According to another aspect, an image encoder is provided that includes a partitioner that in operation receives an original picture and partitions the original picture into blocks, and an adder that in operation receives the blocks from the partitioner and predictions from a prediction controller and subtracts each prediction from its corresponding block to output a residual, a transformer that in operation performs a transform on the residuals output from the adder to output transform coefficients, a quantizer that in operation quantizes the transform coefficients to generate quantized transform coefficients, an entropy encoder that in operation encodes the quantized transform coefficients to generate a bitstream, and a prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, where the inter-predictor in operation generates a prediction of a current block based on a reference block in an encoded reference picture and the intra-predictor in operation generates a prediction of the current block based on an encoded reference block in a current picture. The prediction controller in operation partitions the block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape, predicts a first motion vector for the first partition and a second motion vector for the second partition, and encodes the first partition using the first motion vector and the second partition using the second motion vector.

[0059] According to another aspect, an image encoding method is provided that generally includes three steps of partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape, and predicting a first motion vector for the first partition and a second motion vector for the second partition, and encoding the first partition using the first motion vector and the second partition using the second motion vector.

[0060] According to another aspect, an image decoder is provided that includes circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs: partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; and predicting a first motion vector for the first partition and a second motion vector for the second partition; decoding the first partition using the first motion vector and decoding the second partition using the second motion vector.

[0061] According to another aspect, the second partition has a non-rectangular shape. According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0062] According to another aspect, an image decoder is provided that includes: an entropy decoder that, in operation, receives and decodes an encoded bitstream to obtain quantized transform coefficients; and an inverse quantizer and transformer that, in operation, inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual; an adder that, in operation, adds the residual output from the inverse quantizer and transformer to a prediction output from a prediction controller to reconstruct a block; and the prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, wherein the inter-predictor, in operation, generates a prediction for a current block based on a reference block in a decoded reference picture and the intra-predictor, in operation, generates a prediction for the current block based on a decoded reference block in a current picture. The prediction controller, in operation, partitions an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; predicts a first motion vector for the first partition and a second motion vector for the second partition; decodes the first partition using the first motion vector and decodes the second partition using the second motion vector.

[0063] According to another aspect, an image decoding method is provided that generally includes three steps: partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; and predicting a first motion vector for the first partition and a second motion vector for the second partition; decoding the first partition using the first motion vector and decoding the second partition using the second motion vector.

[0064] According to one aspect, an image encoder is provided that includes circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The boundary smoothing operation includes: using information of the first partition, performing a first prediction of first values of a set of pixels of the first partition along the boundary; and using information of the second partition, performing a second prediction of second values of the set of pixels of the first partition along the boundary; weighting the first values and the second values; and encoding the first partition using the weighted first values and the weighted second values.

[0065] According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon having at least five sides and corners. According to yet another aspect, the second partition has a non-rectangular shape.

[0066] According to another aspect, at least one of the first prediction and the second prediction is an inter prediction process that predicts the first values and the second values based on a reference partition in an encoded reference picture. The inter prediction process can predict the first values of pixels of the first partition including the set of pixels, and can predict the second values of only the set of pixels of the first partition.

[0067] According to another aspect, at least one of the first prediction and the second prediction is an intra prediction process that predicts the first values and the second values based on an encoded reference partition in a current picture.

[0068] According to another aspect, a prediction method used in the first prediction is different from a prediction method used in the second prediction.

[0069] According to another aspect, a number of the set of pixels for which the first values and the second values are predicted in each row or each column is an integer. For example, when the number of the set of pixels in each row or each column is four, weights 1 / 8, 1 / 4, 3 / 4, and 7 / 8 can be respectively applied to the first values of the four pixels in the set, and weights 7 / 8, 3 / 4, 1 / 4, and 1 / 8 can be respectively applied to the second values of the four pixels in the set. As another example, when the number of the set of pixels in each row or each column is two, weights 1 / 3 and 2 / 3 can be respectively applied to the first values of the two pixels in the set, and weights 2 / 3 and 1 / 3 can be respectively applied to the second values of the two pixels in the set.

[0070] According to another aspect, the weights can be integer values or can be fractional values.

[0071] According to another aspect, there is provided an image encoder comprising: a partitioner that in operation receives and partitions an original picture into blocks; and an adder that in operation receives the blocks from the partitioner and predictions from a prediction controller and subtracts each prediction from its corresponding block to output a residual; a transformer that in operation performs a transform on the residuals output from the adder to output transform coefficients; a quantizer that in operation quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder that in operation encodes the quantized transform coefficients to generate a bitstream; and a prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, wherein the inter-predictor in operation generates a prediction for a current block based on a reference block in an encoded reference picture, and the intra-predictor in operation generates a prediction for a current block based on an encoded reference block in a current picture. The prediction controller in operation performs a boundary smoothing operation along a boundary between a second partition partitioned out of an image block and a first partition having a non-rectangular shape. The boundary smoothing operation comprises: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary; and using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary; weighting the first values and the second values; and encoding the first partition using the weighted first values and the weighted second values.

[0072] According to another aspect, there is provided an image encoding method that performs a boundary smoothing operation along a boundary between a second partition partitioned out of an image block and a first partition having a non-rectangular shape. The method generally comprises four steps: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary; and using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary; weighting the first values and the second values; and encoding the first partition using the weighted first values and the weighted second values.

[0073] According to another aspect, there is provided an image decoder comprising a circuit and a memory coupled to the circuit. The circuit in operation performs a boundary smoothing operation along a boundary between a second partition partitioned out of an image block and a first partition having a non-rectangular shape. The boundary smoothing operation comprises: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary; and using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary; weighting the first values and the second values; and decoding the first partition using the weighted first values and the weighted second values.

[0074] According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from the group consisting of a triangle, a trapezoid, and a polygon having at least five sides and corners. According to another aspect, the second partition has a non-rectangular shape.

[0075] According to another aspect, at least one of the first prediction and the second prediction is an inter prediction process that predicts the first value and the second value based on a reference partition in an encoded reference picture. The inter prediction process can predict the first value of a pixel of the first partition comprising a set of pixels and can predict the second value of only the set of pixels of the first partition.

[0076] According to another aspect, at least one of the first prediction and the second prediction is an intra prediction process that predicts the first value and the second value based on an encoded reference partition in a current picture.

[0077] According to another aspect, an image decoder is provided, comprising: an entropy decoder that in operation receives and decodes an encoded bitstream to obtain quantized transform coefficients; and an inverse quantizer and transformer that in operation inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual; an adder that in operation adds the residual output from the inverse quantizer and transformer and a prediction output from a prediction controller to reconstruct a block; and the prediction controller coupled to an inter predictor, an intra predictor, and a memory, wherein the inter predictor in operation generates a prediction for a current block based on a reference block in a decoded reference picture and the intra predictor in operation generates a prediction for the current block based on a decoded reference block in a current picture. The prediction controller in operation performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The boundary smoothing operation comprises: using information of the first partition, first predicting a first value of a set of pixels of the first partition along the boundary; and using information of the second partition, second predicting a second value of the set of pixels of the first partition along the boundary; weighting the first value and the second value; and decoding the first partition using the weighted first value and the weighted second value.

[0078] According to another aspect, an image decoding method is provided that performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The method generally comprises four steps: using information of the first partition, first predicting a first value of a set of pixels of the first partition along the boundary; and using information of the second partition, second predicting a second value of the set of pixels of the first partition along the boundary; weighting the first value and the second value; and decoding the first partition using the weighted first value and the weighted second value.

[0079] According to one aspect, an image encoder is provided that includes circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a partitioning syntax operation that includes partitioning an image block into a plurality of partitions based on a partition parameter indicative of the partitioning, the plurality of partitions including a first partition having a non-rectangular shape and a second partition, encoding the first partition and the second partition, and writing one or more parameters including the partition parameter to a bitstream.

[0080] According to another aspect, the partition parameter indicates that the first partition has a triangular shape.

[0081] According to another aspect, the partition parameter indicates that the second partition has a non-rectangular shape.

[0082] According to another aspect, the partition parameter indicates that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0083] According to another aspect, the partition parameter jointly encodes a partitioning direction used to partition the image block into the plurality of partitions. For example, the partitioning direction can include from a top-left corner of the image block to a bottom-right corner thereof, and from a top-right corner of the image block to a bottom-left corner thereof. The partition parameter can jointly encode at least a first motion vector of the first partition.

[0084] According to another aspect, one or more parameters other than the partition parameter encode a partitioning direction applied to partition the image block into the plurality of partitions. The parameters encoding the partitioning direction can jointly encode at least a first motion vector of the first partition.

[0085] According to another aspect, the partition parameter can jointly encode at least a first motion vector of the first partition. The partition parameter can jointly encode a second motion vector of the second partition.

[0086] According to another aspect, one or more parameters other than the partition parameter can encode at least a first motion vector of the first partition.

[0087] According to another aspect, one or more parameters are binarized according to a binarization scheme selected according to a value of at least one of the one or more parameters.

[0088] According to another aspect, there is provided an image encoder comprising: a partitioner that, in operation, receives and partitions an original picture into blocks; and an adder that, in operation, receives the blocks from the partitioner and predictions from a prediction controller and subtracts each prediction from its corresponding block to output a residual; a transformer that, in operation, performs a transform on the residuals output from the adder to output transform coefficients; a quantizer that, in operation, quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder that, in operation, encodes the quantized transform coefficients to generate a bitstream; and a prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, wherein the inter-predictor, in operation, generates a prediction for a current block based on a reference block in an encoded reference picture, and the intra-predictor, in operation, generates a prediction for the current block based on an encoded reference block in a current picture. The prediction controller, in operation, partitions an image block into a plurality of partitions based on a partition parameter indicative of the partitioning, the plurality of partitions including a second partition and a first partition having a non-rectangular shape, and encodes the first partition and the second partition. The entropy encoder, in operation, writes one or more parameters including the partition parameter into the bitstream.

[0089] According to another aspect, there is provided an image encoding method comprising a partition syntax operation. The method generally comprises three steps: partitioning an image block into a plurality of partitions based on a partition parameter indicative of the partitioning, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; and encoding the first partition and the second partition; and writing one or more parameters including the partition parameter into a bitstream.

[0090] According to another aspect, there is provided an image decoder comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a partition syntax operation comprising: parsing one or more parameters from a bitstream, wherein the one or more parameters include a partition parameter indicative of partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; partitioning the image block into the plurality of partitions based on the partition parameter; and decoding the first partition and the second partition.

[0091] According to another aspect, the partition parameter indicates that the first partition has a triangular shape.

[0092] According to another aspect, the partition parameter indicates that the second partition has a non-rectangular shape.

[0093] According to another aspect, the partition parameter indicates that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0094] According to another aspect, partition parameters jointly encode a split direction for splitting an image block into a plurality of partitions. For example, the split direction includes: from a top-left corner to a bottom-right corner of the image block, and from a top-right corner to a bottom-left corner of the image block. The partition parameters can jointly encode at least a first motion vector of a first partition.

[0095] According to another aspect, one or more parameters other than partition parameters encode a split direction applied to split an image block into a plurality of partitions. The parameters encoding the split direction can jointly encode at least a first motion vector of a first partition.

[0096] According to another aspect, partition parameters can jointly encode at least a first motion vector of a first partition. The partition parameters can jointly encode a second motion vector of a second partition.

[0097] According to another aspect, one or more parameters other than partition parameters can encode at least a first motion vector of a first partition.

[0098] According to another aspect, one or more parameters are binarized according to a binarization scheme selected according to a value of at least one of the one or more parameters.

[0099] According to another aspect, an image decoder is provided, the image decoder comprising: an entropy decoder that in operation receives and decodes an encoded bitstream to obtain quantized transform coefficients; and an inverse quantizer and transformer that in operation inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual; an adder that in operation adds the residual output from the inverse quantizer and transformer to a prediction output from a prediction controller to reconstruct a block; and the prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, wherein the inter-predictor in operation generates a prediction of a current block based on a reference block in a decoded reference picture and the intra-predictor in operation generates a prediction of the current block based on a decoded reference block in a current picture. The entropy decoder in operation: parses one or more parameters from the bitstream, wherein the one or more parameters include partition parameters indicating a split of an image block into a plurality of partitions, the plurality of partitions including a first partition having a non-rectangular shape and a second partition; splits the image block into the plurality of partitions based on the partition parameters; and decodes the first partition and the second partition.

[0100] According to another aspect, there is provided an image decoding method including a partition syntax operation. The method generally includes three steps: parsing one or more parameters from a bitstream, wherein the one or more parameters include partition parameters indicating that an image block is partitioned into a plurality of partitions, the plurality of partitions including a first partition having a non-rectangular shape and a second partition; partitioning the image block into the plurality of partitions based on the partition parameters; and decoding the first partition and the second partition.

[0101] In the drawings, the same reference numbers indicate similar elements. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale.

[0102] Hereinafter, embodiments will be described with reference to the accompanying drawings. Note that each of the following described embodiments shows a general or specific example. The numerical values, shapes, materials, components, arrangement and connection of components, steps, relationships and order of steps, and the like indicated in the following embodiments are merely examples and are not intended to limit the scope of the claims. Therefore, components not listed among any independent claims that define the broadest aspects of the application can be understood as optional components in the following embodiments.

[0103] Embodiments of an encoder and a decoder will be described below. The embodiments are examples of an encoder and a decoder to which the processes and / or configurations presented in the description of aspects of the present disclosure are applicable. The processes and / or configurations can also be implemented in an encoder and a decoder different from those according to the embodiments. For example, with respect to the processes and / or configurations applicable to the embodiments, any one of the following can be implemented:

[0104] (1) Any component of an encoder or a decoder according to the embodiments presented in the description of aspects of the present disclosure can be replaced with or combined with another component presented anywhere in the description of aspects of the present disclosure.

[0105] (2) In an encoder or a decoder according to the embodiments, any change can be made to the functions or processes performed by one or more components of the encoder or the decoder, such as addition, replacement, removal, and the like. For example, any function or process can be replaced with or combined with another function or process appearing anywhere in the description of aspects of the present disclosure.

[0106] (3) In a method implemented by an encoder or a decoder according to the embodiments, any change can be made to one or more processes included in the method, such as addition, replacement, and removal. For example, any process in the method can be replaced with or combined with another process appearing anywhere in the description of aspects of the present disclosure.

[0107] (4) One or more components included in an encoder or decoder according to embodiments can be combined with components presented anywhere in the description of aspects of the disclosure, can be combined with components that include one or more functions presented anywhere in the description of aspects of the disclosure, and can be combined with components that implement one or more processes implemented by components presented in the description of aspects of the disclosure.

[0108] (5) Components that include one or more functions of an encoder or decoder according to embodiments, or components that implement one or more processes of an encoder or decoder according to embodiments, can be combined with components presented anywhere in the description of aspects of the disclosure, with components that include one or more functions presented anywhere in the description of aspects of the disclosure, or with components that implement one or more processes presented anywhere in the description of aspects of the disclosure, or can replace components presented anywhere in the description of aspects of the disclosure.

[0109] (6) In a method implemented by an encoder or decoder according to embodiments, any processes included in the method can be replaced or combined with processes presented anywhere in the description of aspects of the disclosure, or with any corresponding or equivalent processes.

[0110] (7) One or more processes included in a method implemented by an encoder or decoder according to embodiments can be combined with processes presented anywhere in the description of aspects of the disclosure.

[0111] (8) Implementations of processes and / or configurations presented in the description of aspects of the disclosure are not limited to encoders or decoders according to embodiments. For example, the processes and / or configurations can be implemented in devices for different purposes than motion picture encoders or motion picture decoders disclosed in embodiments.

[0112] (Encoder)

[0113] First, an encoder according to embodiments will be described. Figure 1 is a block diagram illustrating a functional configuration of an encoder 100 according to embodiments. The encoder 100 is a motion picture encoder that encodes a motion picture on a block-by-block basis.

[0114] As shown in Figure 1 , the encoder 100 is a device that encodes a picture on a block-by-block basis, and includes a partitioner 102, a subtracter 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116, a block memory 118, a loop filter 120, a frame memory 122, an intra predictor 124, an inter predictor 126, and a prediction controller 128.

[0115] The encoder 100 is implemented as, for example, a general-purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor functions as the partitioner 102, the subtracter 104, the transformer 106, the quantizer 108, the entropy encoder 110, the inverse quantizer 112, the inverse transformer 114, the adder 116, the in-loop filter 120, the intra predictor 124, the inter predictor 126, and the prediction controller 128. Alternatively, the encoder 100 can be implemented as one or more dedicated electronic circuits corresponding to the partitioner 102, the subtracter 104, the transformer 106, the quantizer 108, the entropy encoder 110, the inverse quantizer 112, the inverse transformer 114, the adder 116, the in-loop filter 120, the intra predictor 124, the inter predictor 126, and the prediction controller 128.

[0116] Hereinafter, each component included in the encoder 100 will be described.

[0117] (partitioner)

[0118] The partitioner 102 partitions each picture included in the inputted moving picture into blocks, and outputs each block to the subtracter 104. For example, the partitioner 102 first partitions a picture into blocks of a fixed size (e.g., 128x128). The fixed-size blocks can also be referred to as coding tree units (CTUs). Then, the partitioner 102 partitions each fixed-size block into blocks of a variable size (e.g., 64x64 or smaller) based on, for example, recursive quad-tree and / or binary-tree block partitioning. The variable-size blocks can also be referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). In various implementations, there can be no need to distinguish between CUs, PUs, and TUs; all or some of the blocks in a picture can be processed as CUs, PUs, or TUs.

[0119] Figure 2 One example of block partitioning according to an embodiment is shown. In Figure 2 In the example, solid lines indicate block boundaries of blocks partitioned by quad-tree block partitioning, and dashed lines indicate block boundaries of blocks partitioned by binary-tree block partitioning.

[0120] Here, the block 10 is a square 128x128-pixel block (128x128 block). The 128x128 block 10 is first partitioned into four square 64x64 blocks (quad-tree block partitioning).

[0121] The upper-left 64x64 block is further vertically partitioned into two rectangular 32x64 blocks, and the left 32x64 block is further vertically partitioned into two rectangular 16x64 blocks (binary-tree block partitioning). As a result, the upper-left 64x64 block is partitioned into two 16x64 blocks 11 and 12 and one 32x64 block 13.

[0122] The upper right 64x64 block is horizontally split into two rectangular 64x32 blocks 14 and 15 (binary tree block split).

[0123] First, the lower left 64x64 block is split into four square 32x32 blocks (quad tree block split). Among the four 32x32 blocks, the upper left block and the lower right block are further split. The upper left 32x32 block is vertically split into two rectangular 16x32 blocks, and the right 16x32 block is further horizontally split into two 16x16 blocks (binary tree block split). The lower right 32x32 block is horizontally split into two 32x16 blocks (binary tree block split). As a result, the lower left 64x64 block is split into the 16x32 block 16, two 16x16 blocks 17 and 18, two 32x32 blocks 19 and 20, and two 32x16 blocks 21 and 22.

[0124] The lower right 64x64 block 23 is not split.

[0125] As described above, in the Figure 2 , the block 10 is split into 13 variable size blocks 11 to 23 based on recursive quad tree and binary tree block split. This type of split is also referred to as quad tree plus binary tree (QTBT) split.

[0126] Although a block is split into four or two blocks (quad tree or binary tree block split) in the Figure 2 , the split is not limited to these examples. For example, a block can be split into three blocks (ternary block split). The split including such ternary block split is also referred to as multi-type tree (MBT) split.

[0127] (subtracter)

[0128] The subtracter 104 subtracts a prediction signal (prediction sample, input from a prediction controller 128, which will be described hereinafter) from an original signal (original sample) of each block split and input by the splitter 102. In other words, the subtracter 104 calculates a prediction error (also referred to as "residual") of a block to be encoded (also referred to as "current block" hereinafter). The subtracter 104 then outputs the calculated prediction error (residual) to the transformer 106.

[0129] The original signal is a signal input to the encoder 100, and is a signal (e.g., a luminance signal and two chrominance signals) representing an image of each picture included in a moving picture. Hereinafter, the signal representing an image is also referred to as a sample.

[0130] (transformer)

[0131] The transformer 106 transforms the spatial domain prediction error into a frequency domain transform coefficient, and outputs the transform coefficient to the quantizer 108. More specifically, the transformer 106 applies, for example, a predefined discrete cosine transform (DCT) or a discrete sine transform (DST) to the spatial domain prediction error.

[0132] Note that the transformer 106 can adaptively select a transform type from among a plurality of transform types, and transform the prediction error into a transform coefficient by using a transform basis function corresponding to the selected transform type. Such a transform is also referred to as explicit multi-core transform (EMT) or adaptive multi-transform (AMT).

[0133] The transform types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 is a graph indicating a transform basis function for each transform type. In Figure 3 In, N indicates the number of input pixels. For example, the selection of the transform type from among the plurality of transform types can depend on the prediction type (intra prediction and inter prediction) and the intra prediction mode.

[0134] Information indicating whether to apply EMT or AMT (for example, referred to as an EMT flag or an AMT flag) and information indicating the selected transform type are generally signaled at the CU level. Note that the signaling of such information need not be performed at the CU level, but can be performed at another level (for example, at the bit sequence level, the picture level, the slice level, the tile level, or the CTU level).

[0135] Further, the transformer 106 can apply a secondary transform to the transform coefficient (transform result). Such a secondary transform is also referred to as adaptive secondary transform (AST) or non-separable secondary transform (NSST). For example, the transformer 106 applies a secondary transform to each sub-block (for example, each 4x4 sub-block) included in a block of transform coefficients corresponding to the intra prediction error. Information indicating whether to apply NSST and information about a transform matrix used in the NSST are generally signaled at the CU level. Note that the signaling of such information need not be performed at the CU level, but can be performed at another level (for example, at the bit sequence level, the picture level, the slice level, the tile level, or the CTU level).

[0136] A separate transform or a non-separable transform can be applied in the transformer 106. The separate transform is a method in which a transform is performed multiple times by separately performing a transform for each direction according to the number of dimensions of the input. The non-separable transform is a method of performing a collective transform in which two or more dimensions in a multi-dimensional input are collectively regarded as a single dimension.

[0137] In one example of a non-separable transform, when the input is a 4x4 block, the 4x4 block is treated as a single array containing 16 components, and the transform applies a 16x16 transform matrix to the array.

[0138] In another example of a non-separable transform, after treating the input 4x4 block as a single array containing 16 components, a transform that performs multiple Givens rotations (e.g., a Hypercube-Givens transform) can be applied to the array.

[0139] (Quantizer)

[0140] The quantizer 108 quantizes the transform coefficients output from the transformer 106. More specifically, the quantizer 108 scans the transform coefficients of the current block in a predetermined scan order, and quantizes the scanned transform coefficients based on a quantization parameter (QP) corresponding to the transform coefficients. Then, the quantizer 108 outputs the quantized transform coefficients (hereinafter, referred to as quantized coefficients) of the current block to the entropy encoder 110 and the inverse quantizer 112.

[0141] The predetermined scan order is an order for quantizing / dequantizing the transform coefficients. For example, the predetermined scan order is defined as an ascending order of frequency (from low frequency to high frequency) or a descending order of frequency (from high frequency to low frequency).

[0142] The quantization parameter (QP) is a parameter that defines a quantization step (quantization width). For example, if the value of the quantization parameter increases, the quantization step also increases. In other words, if the value of the quantization parameter increases, the quantization error increases.

[0143] (Entropy encoder)

[0144] The entropy encoder 110 generates an encoded signal (encoded bitstream) based on the quantized coefficients input from the quantizer 108. More specifically, for example, the entropy encoder 110 binarizes the quantized coefficients and arithmetically encodes the binarized signal to output a compressed bitstream or sequence.

[0145] (Inverse quantizer)

[0146] The inverse quantizer 112 dequantizes the quantized coefficients input from the quantizer 108. More specifically, the inverse quantizer 112 dequantizes the quantized coefficients of the current block in the predetermined scan order. Then, the inverse quantizer 112 outputs the dequantized transform coefficients of the current block to the inverse transformer 114.

[0147] (Inverse transformer)

[0148] Inverse transformer 114 recovers the prediction error (residual) by performing an inverse transform on the transform coefficients input from inverse quantizer 112. More specifically, inverse transformer 114 recovers the prediction error of the current block by applying an inverse transform corresponding to the transform applied to the transform coefficients by transformer 106. Inverse transformer 114 then outputs the recovered prediction error to adder 116.

[0149] Note that because information is typically lost during quantization, the recovered prediction error does not match the prediction error calculated by subtractor 104. In other words, the recovered prediction error typically includes the quantization error.

[0150] (Adder)

[0151] Adder 116 reconstructs the current block by summing the prediction error input from inverse transformer 114 and the prediction sample input from prediction controller 128. Adder 116 then outputs the reconstructed block to block memory 118 and loop filter 120. The reconstructed block is also referred to as the locally decoded block.

[0152] (Block memory)

[0153] Block memory 118 is a storage device for storing blocks of a picture to be encoded (referred to as the "current picture") referenced in, for example, intra-frame prediction. More specifically, block memory 118 stores reconstructed blocks output from adder 116.

[0154] (Loop filter)

[0155] Loop filter 120 applies a loop filter to the block reconstructed by adder 116 and outputs the filtered, reconstructed block to frame memory 122. Loop filters are filters used in the coding loop (in-loop filters) and include, for example, deblocking filters (DF), sample adaptive offset (SAO), and adaptive loop filters (ALF).

[0156] In ALF, a least-squares error filter is applied to eliminate compression artifacts. For example, based on the direction and activity of the local gradient, a filter is selected from multiple filters for each 2×2 sub-block in the current block and applied.

[0157] More specifically, first, each sub-block (e.g., each 2×2 sub-block) is classified into one of several categories (e.g., 15 or 25 categories). The classification of sub-blocks is based on gradient directionality and activity. For example, a classification index C (e.g., C = 5D + A) is derived based on gradient directionality D (e.g., 0 to 2 or 0 to 4) and gradient activity A (e.g., 0 to 4). Then, based on the classification index C, each sub-block is classified into one of several categories.

[0158] For example, the gradient directionality D is calculated by comparing the gradients of multiple directions (e.g., a horizontal direction, a vertical direction, and two diagonal directions). Further, for example, the gradient activity A is calculated by summing the gradients of multiple directions and quantizing the sum.

[0159] Based on the result of such classification, a filter to be used for each sub-block is determined from among the multiple filters.

[0160] The filter shape used in the ALF is, for example, a circularly symmetric filter shape. Figure 4A 、 4B 、 and 4C show examples of filter shapes used in the ALF. Figure 4A A 5x5 diamond filter is shown in Figure 4B A 7x7 diamond filter is shown in Figure 4C A 9x9 diamond filter is shown in. The information indicating the filter shape is typically signaled at the picture level. Note that the signaling of the information indicating the filter shape need not be performed at the picture level, but can be performed at another level (e.g., at the sequence level, the slice level, the tile level, the CTU level, or the CU level).

[0161] The enablement or disablement of the ALF can be determined at the picture level or the CU level. For example, for luma, the decision whether to apply the ALF can be made on the CU level, and for chroma, the decision whether to apply the ALF can be made on the picture level. The information indicating whether the ALF is enabled or disabled is typically signaled at the picture level or the CU level. Note that the signaling of the information indicating whether the ALF is enabled need not be performed at the picture level or the CU level, but can be performed at another level (e.g., at the sequence level, the slice level, the tile level, or the CTU level).

[0162] The coefficient sets of multiple selectable filters (e.g., 15 or 25 filters) are typically signaled at the picture level. Note that the signaling of the coefficient sets need not be performed at the picture level, but can be performed at another level (e.g., at the sequence level, the slice level, the tile level, the CTU level, the CU level, or the sub-block level).

[0163] (frame memory)

[0164] The frame memory 122 is, for example, a storage device used to store reference pictures used in inter prediction, and is also referred to as a frame buffer. More specifically, the frame memory 122 stores the reconstructed blocks filtered by the loop filter 120.

[0165] (intra predictor)

[0166] The intra predictor 124 generates a prediction signal (intra prediction signal) by intra predicting the current block by referring to one or more blocks in the current picture stored in the block memory 118. More specifically, the intra predictor 124 generates the intra prediction signal by intra predicting the current block by referring to samples (e.g., luma and / or chroma values) of one or more blocks neighboring the current block, and then outputs the intra prediction signal to the prediction controller 128.

[0167] For example, the intra predictor 124 performs intra prediction by using one of a plurality of predetermined intra prediction modes. The intra prediction modes generally include one or more non-directional prediction modes and a plurality of directional prediction modes.

[0168] The one or more non-directional prediction modes include, for example, a planar prediction mode and a DC prediction mode defined in the H.265 / HEVC standard.

[0169] The plurality of directional prediction modes include, for example, 33 directional prediction modes defined in the H.265 / HEVC standard. Note that the plurality of directional prediction modes can include 32 directional prediction modes in addition to the 33 directional prediction modes (for a total of 65 directional prediction modes).

[0170] Figure 5A A total of 67 intra prediction modes (two non-directional prediction modes and 65 directional prediction modes) used in intra prediction are shown. The solid arrows indicate 33 directions defined in the H.265 / HEVC standard, while the dashed arrows indicate the other 32 directions (not shown in the H.265 / HEVC standard). Figure 5A The two "non-directional" prediction modes are not shown in the H.265 / HEVC standard.

[0171] In various implementations, a luma block can be referred to in chroma block intra prediction. That is, a chroma component of a current block can be predicted based on a luma component of the current block. Such intra prediction is also referred to as cross-component linear model (CCLM) prediction. A chroma block intra prediction mode that refers to a luma block (e.g., referred to as a CCLM mode) can be added as one of the chroma block intra prediction modes.

[0172] The intra predictor 124 can correct intra-predicted pixel values based on horizontal / vertical reference pixel gradients. Intra prediction with such correction is also referred to as position-dependent intra prediction combination (PDPC). Information indicating whether to apply PDPC (e.g., referred to as a PDPC flag) is generally signaled at the CU level. Note that signaling of the information need not be performed at the CU level, but can be performed at another level (e.g., at the sequence level, picture level, slice level, tile level, or CTU level).

[0173] (inter-predictor)

[0174] The inter-predictor 126 generates a prediction signal (inter-prediction) by inter-predicting the current block with reference to one or more blocks in a reference picture different from the current picture and stored in the frame memory 122. The inter-prediction is performed for each current block or each current sub-block (e.g., each 4x4 block) in the current block. For example, the inter-predictor 126 performs motion estimation in the reference picture for the current block or the current sub-block to find a reference block or sub-block in the reference picture most matching the current block or sub-block, and obtains motion information (e.g., a motion vector) compensating (or predicting) the motion or changing from the reference block or sub-block to the current block or sub-block. Then, the inter-predictor 126 performs motion compensation (or motion prediction) based on the motion information, and generates an inter-prediction signal for the current block or sub-block based on the motion information. The inter-predictor 126 then outputs the generated inter-prediction signal to the prediction controller 128.

[0175] The motion information used in the motion compensation can be signaled as the inter-prediction signal in various forms. For example, a motion vector can be signaled. As another example, a difference between a motion vector and a motion vector predictor can be signaled.

[0176] Note that, in addition to the motion information of the current block obtained from the motion estimation, the motion information of a neighboring block can be used to generate the inter-prediction signal. More specifically, the inter-prediction signal can be generated for each sub-block in the current block by calculating a weighted sum of a prediction signal based on the motion information obtained from the motion estimation (in the reference picture) and a prediction signal based on the motion information of the neighboring block (in the current picture). Such inter-prediction (motion compensation) is also referred to as overlapped block motion compensation (OBMC).

[0177] In the OBMC mode, information indicating a sub-block size for OBMC (e.g., referred to as OBMC block size) can be signaled at a sequence level. Further, information indicating whether to apply the OBMC mode (e.g., referred to as OBMC flag) can be signaled at a CU level. Note that the signaling of such information need not be performed at the sequence level and the CU level, but can be performed at another level (e.g., at a picture level, a slice level, a tile level, a CTU level, or a sub-block level).

[0178] Hereinafter, the OBMC mode will be described in more detail. Figure 5B is a flowchart, Figure 5C is a conceptual diagram illustrating a prediction image correction process performed by the OBMC process.

[0179] Reference Figure 5CFirst, a prediction picture (Pred) is obtained by typical motion compensation using the motion vector (MV) assigned to the target (current) block. In Figure 5C the arrow "MV" points to the reference picture to indicate what the current block in the current picture is referencing in order to obtain the prediction picture.

[0180] Next, a prediction picture (Pred_L) is obtained by applying (reusing) the motion vector (MV_L) that has been derived for the encoded neighboring left block to the target (current) block, as indicated by the arrow "MV_L" starting from the current block and pointing to the reference picture to obtain the prediction picture Pred_L. Then, the two prediction pictures Pred and Pred_L are superimposed to perform a first pass of correction of the prediction picture, which in one aspect has the effect of blending the boundaries between the neighboring blocks.

[0181] Similarly, a prediction picture (Pred_U) is obtained by applying (reusing) the motion vector (MV_U) that has been derived for the encoded neighboring upper block to the target (current) block, as shown by the arrow "MV_U" starting from the current block and pointing to the reference picture to obtain the prediction picture Pred_U. Then, the prediction picture Pred_U is superimposed with the prediction picture resulting from the first pass (i.e., Pred and Pred_L) to perform a second pass of correction of the prediction picture, which in one aspect has the effect of blending the boundaries between the neighboring blocks. The result of the second pass is the final prediction picture for the current block, which has blended (smoothed) boundaries with its neighboring blocks.

[0182] Note that the above example is a two-pass correction method using the neighboring left and upper blocks, but the method can be a three-pass or higher pass correction method that also uses the neighboring right and / or lower blocks.

[0183] Note that the area subject to superimposition can be the entire pixel area of the block, and alternatively can be a partial block boundary area.

[0184] Note here that the prediction picture correction process of OBMC is described as being based on a single reference picture to derive a single prediction picture Pred, with the additional prediction pictures Pred_L and Pred_U being superimposed on it, but the same process can be applied to each of a plurality of reference pictures when the prediction picture is corrected based on the plurality of reference pictures. In such a case, after a plurality of corrected prediction pictures are obtained by performing the image correction of OBMC based on the plurality of reference pictures respectively, the obtained plurality of corrected prediction pictures are also superimposed to obtain the final prediction picture.

[0185] Note that in OBMC, the unit of the target block can be a prediction block, and alternatively, can be a sub-block obtained by further partitioning a prediction block.

[0186] One example of a method of determining whether to implement the OBMC process is to use obmc_flag, which is a signal indicating whether to implement the OBMC process. As one specific example, an encoder can determine whether a target block belongs to a region including complex motion. When the block belongs to a region including complex motion, the encoder sets the value of obmc_flag to "1" and implements the OBMC process during the encoding process; if the block does not belong to a region including complex motion, the value of obmc_flag is set to "0" and the block is encoded without implementing the OBMC process. A decoder switches between whether to implement the OBMC process by decoding obmc_flag written in a stream (i.e., a compressed sequence) and performing decoding according to the flag value.

[0187] Note that the motion information can be derived at the decoder side without being signaled from the encoder side. For example, a merge mode defined in the H.265 / HEVC standard can be used. Also, for example, the motion information can be derived by performing motion estimation at the decoder side. In this case, the decoder side can perform motion estimation without using pixel values of the current block.

[0188] Here, a mode for performing motion estimation at the decoder side will be described. The mode for performing motion estimation at the decoder side is also referred to as a pattern matching motion vector derivation (PMMVD) mode or a frame rate up conversion (FRUC) mode.

[0189] One example of FRUC processing is shown in Figure 5D A candidate list (which can be a merge list) of candidates is first generated by referring to motion vectors of coded blocks adjacent to the current block in space or time, each of the candidate list including a predicted motion vector (MV). Next, a best candidate MV is selected from among the plurality of candidate MVs registered in the candidate list. For example, evaluation values of the candidate MVs included in the candidate list are calculated, and one candidate MV is selected based on the calculated evaluation values.

[0190] Next, a motion vector of the current block is derived from the motion vector of the selected candidate. More specifically, for example, the motion vector of the current block is calculated as the motion vector of the selected candidate (best candidate MV). Alternatively, the motion vector of the current block can be derived by pattern matching performed in the vicinity of a position in a reference picture corresponding to the motion vector of the selected candidate. In other words, when the vicinity of the best candidate MV is searched using pattern matching in the reference picture and an MV with a better evaluation value is found, the best candidate MV can be updated to the MV with the better evaluation value, and the MV with the better evaluation value can be used as the final MV of the current block. A configuration in which the process for updating the MV with the better evaluation value is not implemented is also acceptable.

[0191] In a case where processing is performed in units of sub-blocks, the same processing can be performed.

[0192] The evaluation value can be calculated in various ways. For example, a reconstructed image of a region in a reference picture corresponding to a motion vector is compared with a reconstructed image of a predetermined region (for example, in another reference picture or in a neighboring block in the current picture, as described below), and a difference in pixel values between the two reconstructed images can be calculated and used as the evaluation value of the motion vector. Note that the evaluation value can be calculated by using some other information in addition to the difference.

[0193] Next, the mode matching is described in detail. First, one of the candidates included in a candidate list (for example, a merge list) is selected as a starting point of the search by mode matching. The mode matching used is either first mode matching or second mode matching. The first mode matching and the second mode matching are also referred to as bilateral matching and template matching, respectively.

[0194] In the first mode matching, the mode matching is performed between two blocks in two different reference pictures that both follow the motion trajectory of the current block. Thus, in the first mode matching, for a region in a reference picture, a region in another reference picture that coincides with the motion trajectory of the current block is used as a predetermined region for the calculation of the above-mentioned candidate evaluation value.

[0195] Figure 6 One example of the first mode matching (bilateral matching) between two blocks in two reference pictures along the motion trajectory is shown. As shown in Figure 6 In the first mode matching, two motion vectors (MV0, MV1) are derived by finding the best match between two blocks in two different reference pictures (Ref0, Ref1) that follow the motion trajectory of the current block (Cur block). More specifically, a difference can be obtained between (i) a reconstructed image at a position specified by a candidate MV in a first encoded reference picture (Ref0) and (ii) a reconstructed image at a position specified by the candidate MV in a second encoded reference picture (Ref1) (each scaled symmetrically in display time interval). This difference can then be used to derive an evaluation value for the current block. A candidate MV with the best evaluation value can be selected from among a plurality of candidate MVs as the final MV.

[0196] Under the assumption of continuous motion trajectory, the motion vectors (MV0, MV1) pointing to the two reference blocks are proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, when the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the two reference pictures are the same, the first mode matching results in two mirror bilateral motion vectors.

[0197] In the second mode matching (template matching), the mode matching is performed between a template (a block neighboring the current block in the current picture; e.g., top and / or left neighboring blocks) in the current picture and a block in the reference picture. Thus, in the second mode matching, the block neighboring the current block in the current picture is used as the predetermined region for the above-mentioned calculation of the candidate evaluation value.

[0198] Figure 7 One example of the mode matching (template matching) between a template in the current picture and a block in the reference picture is shown. As shown in Figure 7 In the second mode matching, the motion vector of the current block is derived by searching in the reference picture (Ref0) to find a block that best matches the neighboring blocks of the current block (Cur block) in the current picture (Cur Pic). More specifically, a difference can be obtained between (i) a reconstructed image with respect to one or both of the encoded neighboring top and left regions of the current block and (ii) a reconstructed image with respect to the same region with respect to a block position specified by a candidate MV in the encoded reference picture (Ref0). The difference can then be used to derive an evaluation value for the current block. The candidate MV with the best evaluation value can be selected as the best candidate MV among a plurality of candidate MVs.

[0199] Information indicating whether to apply the FRUC mode (e.g., referred to as a FRUC flag) can be signaled at the CU level. Further, when the FRUC mode is applied (e.g., when the FRUC flag is set to true), information indicating the matching method (e.g., the first mode matching or the second mode matching) to which the mode applies can be signaled at the CU level. Note that the signaling of such information need not be performed at the CU level, but can be performed at another level (e.g., at the sequence level, the picture level, the slice level, the tile level, the CTU level, or the sub-block level).

[0200] Next, the method of deriving the motion vector is described. First, a description of a mode for deriving the motion vector based on a model assuming uniform linear motion is given. This mode is also referred to as the bilateral optical flow (BIO) mode.

[0201] Figure 8 A model assuming uniform linear motion is shown. In Figure 8In this case, (v x ,v y ) represents a velocity vector, and τ0 and τ1 represent a temporal distance between a current picture (Cur Pic) and two reference pictures (Ref0, Ref1), respectively. (MVx0, MVy0) represents a motion vector corresponding to the reference picture Ref0, and (MVx1, MVy1) represents a motion vector corresponding to the reference picture Ref1.

[0202] Here, in a case where it is assumed that the velocity vector (vx, vy) exhibits uniform linear motion, (MVx0, MVy0) and (MVx1, MVy1) are represented as (v x τ0,v y τ0) and (-v x τ1,-v y τ1), respectively, and the following optical flow equation (Equation 1) is given.

[0203] [Equation 1]

[0204]

[0205] Here, I (k) denotes a luminance value from a reference picture k (k = 0, 1) after motion compensation. The optical flow equation shows that a sum of (i) a time derivative of the luminance value, (ii) a product of a horizontal velocity and a horizontal component of a spatial gradient of the reference picture, and (iii) a product of a vertical velocity and a vertical component of the spatial gradient of the reference picture is equal to zero. A motion vector of each block obtained from, for example, a merge list can be corrected pixel by pixel based on a combination of the optical flow equation and Hermite interpolation.

[0206] Note that a motion vector can be derived at a decoder side using a method other than a model based on an assumption of uniform linear motion. For example, a motion vector of each sub-block can be derived based on motion vectors of neighboring blocks.

[0207] Next, a description is given of a mode in which a motion vector is derived for each sub-block based on motion vectors of neighboring blocks. This mode is also referred to as an affine motion compensation prediction mode.

[0208] Figure 9A An example of deriving a motion vector of each sub-block based on motion vectors of neighboring blocks is shown. In this example, a motion vector of each sub-block is derived based on a motion vector of a neighboring block located in a left upper corner of the sub-block. Figure 9AIn this context, the current block comprises 16 4×4 sub-blocks. Here, the motion vector v0 of the top-left control point in the current block is derived based on the motion vectors of adjacent sub-blocks. Similarly, the motion vector v1 of the top-right control point in the current block is derived based on the motion vectors of adjacent blocks. Then, using the two motion vectors v0 and v1, the motion vector (v1) of each sub-block in the current block is obtained using the following Equation 2. x v y ).

[0209] [Mathematical Expression 2]

[0210]

[0211] Here, x and y are the horizontal and vertical positions of the sub-block, respectively, and w is a pre-determined weighting coefficient.

[0212] Affine motion compensation prediction modes can include multiple modes that derive motion vectors for the top-left and top-right control points using different methods. Information indicating the affine motion compensation prediction mode (e.g., referred to as an affine flag) can be signaled at the CU level. Note that signaling information indicating the affine motion compensation prediction mode does not need to be performed at the CU level, but can be performed at another level (e.g., at the sequence level, picture level, slice level, tile level, CTU level, or sub-block level).

[0213] (Predictive Controller)

[0214] Prediction controller 128 selects an intra-frame prediction signal (output from intra-frame predictor 124) or an inter-frame prediction signal (output from inter-frame predictor 126), and outputs the selected prediction signal to subtractor 104 and adder 116.

[0215] like Figure 1 As shown, in various implementations, prediction controller 128 can output prediction parameters that are input to entropy encoder 110. Entropy encoder 110 can generate an encoded bitstream (or sequence) based on the prediction parameters input from prediction controller 128 and quantized coefficients input from quantizer 108. The prediction parameters can be used by a decoder that receives and decodes the encoded bitstream to perform the same prediction processing as performed in intra-frame predictor 124, inter-frame predictor 126, and prediction controller 128. Prediction parameters can include selected prediction signals (e.g., motion vectors, prediction types, or prediction modes employed in intra-frame predictor 124 or inter-frame predictor 126), or any index, flag, or value based on or indicating the prediction processing performed in intra-frame predictor 124, inter-frame predictor 126, and prediction controller 128.

[0216] Figure 9BAn example of the process for deriving motion vectors in the current image in merge mode is shown.

[0217] First, a list of predicted MVs is generated, in which predicted MV candidates are registered. Examples of predicted MV candidates include: spatially adjacent predicted MVs, which are MVs of coded blocks located spatially near the target block; temporally adjacent predicted MVs, which are MVs of blocks in the encoded reference image that are adjacent to the block located in the same position as the target block; coupled predicted MVs, which are MVs generated by combining the MV values ​​of spatially adjacent predicted MVs and temporally adjacent predicted MVs; and zero predicted MVs, which are MVs with a value of zero.

[0218] Next, the MV of the target block is determined by selecting one of the multiple predicted MVs registered in the predicted MV list.

[0219] Additionally, in a variable-length encoder, the signal merge_idx, which indicates which predicted MV was selected, is written and encoded into the stream.

[0220] Note that in Figure 9B The predicted MVs registered in the predicted MV list shown constitute an example. The number of predicted MVs registered in the predicted MV list can be related to... Figure 9B The quantities shown are different; the predicted MVs registered in the predicted MV list can be omitted. Figure 9B The examples given include one or more types of predicted MVs, except in Figure 9B The types given in the examples are different and are related to Figure 9B The types given in the examples are different; the predicted MVs registered in the predicted MV list can include one or more types of predicted MVs.

[0221] The final MV can be determined by performing DMVR (Dynamic Motion Vector Refresh) processing (described later) using the MV of the target block obtained in merge mode.

[0222] Figure 9C This is a conceptual diagram illustrating an example of DMVR processing used to determine MV.

[0223] First, the most suitable MV set for the current block (e.g., in merge mode) is considered as a candidate MV. Next, based on the candidate MV (L0), reference pixels are identified in a first reference image (L0), which serves as the encoded image in the L0 direction. Similarly, based on the candidate MV (L1), reference pixels are identified in a second reference image (L1), which serves as the encoded image in the L1 direction. Then, the reference pixels are averaged to form a template.

[0224] Next, using the template, a surrounding area of the candidate MVs of the first reference picture (L0) and the second reference picture (L1) is searched, and the MV having the lowest cost is determined as the final MV. For example, the cost value can be calculated using the difference between each pixel value in the template and each pixel value in the searched area, using the candidate MV, and the like.

[0225] Note that the configuration and operation of the processing described here are substantially the same on the encoder side and the decoder side, which will be described below.

[0226] Any processing other than the above-described processing can be used as long as the processing is capable of deriving the final MV by searching the surrounding environment of the candidate MV.

[0227] Next, a description is given of an example of a mode in which a prediction image (prediction) is generated using LIC (Local Illumination Compensation) processing.

[0228] Figure 9D An example of a prediction image generation method using a brightness correction process performed by LIC processing is shown.

[0229] First, the MV is derived from the encoded reference picture to obtain a reference image corresponding to the current block.

[0230] Next, for the current block, based on the brightness pixel values of the encoded neighboring left reference region and the encoded neighboring upper reference region in the current picture, and based on the brightness pixel values of the same positions in the reference picture specified by the MV, information indicating how the brightness values change between the reference picture and the current picture is obtained. The information indicating how the brightness values change is used to calculate the brightness correction parameter.

[0231] The prediction image for the current block is generated by performing a brightness correction process that applies the brightness correction parameter to the reference image in the reference picture specified by the MV.

[0232] Note that, Figure 9D The shape of the surrounding reference region shown in the above example is merely one example; the surrounding reference region can have a different shape.

[0233] Further, although the prediction image is generated from a single reference picture in this example, in the case where the prediction image is generated from a plurality of reference pictures, the prediction image can be generated after the brightness correction process is performed on the reference images obtained from the reference pictures as described above.

[0234] One example of a method for determining whether to implement LIC processing is to use lic_flag, which is a signal indicating whether to implement LIC processing. As one specific example, the encoder determines whether the current block belongs to a region of luminance change. When the block belongs to a region of luminance change, the encoder sets lic_flag to "1" and implements LIC processing at the time of encoding. When the block does not belong to a region of luminance change, the encoder sets lic_flag to "0" and performs encoding that implements LIC processing. The decoder can switch between whether to perform LIC processing by decoding lic_flag written in the stream and performing decoding according to the flag value.

[0235] One example of a different method of determining whether to implement LIC processing includes discriminating whether LIC processing is determined to be implemented on a surrounding block. In one specific example, when a merge mode is used on a current block, it is determined whether LIC processing was applied in encoding of a surrounding encoded block that was selected when the MV was derived in the merge mode. This determination is then used to further determine whether to implement LIC processing on the current block. Note that in this example, the same applies to processing performed on the decoder side.

[0236] (Decoder)

[0237] Next, a decoder that can decode an encoded signal (encoded bitstream) output from the encoder 100 will be described. Figure 10 is a block diagram showing a functional configuration of a decoder 200 according to an embodiment. The decoder 200 is a motion picture decoder that decodes a motion picture on a block-by-block basis.

[0238] As shown in Figure 10 , the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, a summer 208, a block memory 210, a loop filter 212, a frame memory 214, an intra predictor 216, an inter predictor 218, and a prediction controller 220.

[0239] The decoder 200 is implemented as, for example, a general-purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor functions as the entropy decoder 202, the inverse quantizer 204, the inverse transformer 206, the summer 208, the loop filter 212, the intra predictor 216, the inter predictor 218, and the prediction controller 220. Alternatively, the decoder 200 can be implemented as one or a plurality of dedicated electronic circuits corresponding to the entropy decoder 202, the inverse quantizer 204, the inverse transformer 206, the summer 208, the loop filter 212, the intra predictor 216, the inter predictor 218, and the prediction controller 220.

[0240] Hereinafter, each component included in the decoder 200 will be described.

[0241] (Entropy decoder)

[0242] The entropy decoder 202 entropy-decodes the decoded bitstream. More specifically, for example, the entropy decoder 202 arithmetically decodes the encoded bitstream into a binary signal. Then, the entropy decoder 202 debinarizes the binary signal. The entropy decoder 202 outputs the quantized coefficients of each block to the inverse quantizer 204. The entropy decoder 202 can also output the prediction parameters (see Figure 1 ) included in the encoded bitstream to the intra predictor 216, the inter predictor 218, and the prediction controller 220 so that they can perform the same prediction process as that performed on the encoder side in the intra predictor 124, the inter predictor 126, and the prediction controller 128.

[0243] (Inverse quantizer)

[0244] The inverse quantizer 204 inverse-quantizes the quantized coefficients of a block to be decoded (hereinafter referred to as a current block) input from the entropy decoder 202. More specifically, the inverse quantizer 204 inverse-quantizes the quantized coefficients of the current block based on the quantization parameter corresponding to the quantized coefficients. Then, the inverse quantizer 204 outputs the inverse-quantized coefficients (i.e., transform coefficients) of the current block to the inverse transformer 206.

[0245] (Inverse transformer)

[0246] The inverse transformer 206 recovers a prediction error (residual) by inverse-transforming the transform coefficients input from the inverse quantizer 204.

[0247] For example, when the information parsed from the encoded bitstream indicates the application of EMT or AMT (e.g., when the AMT flag is set to true), the inverse transformer 206 inverse-transforms the transform coefficients of the current block based on the information indicating the parsed transform type.

[0248] Further, for example, when the information parsed from the encoded bitstream indicates the application of NSST, the inverse transformer 206 applies a secondary inverse transform to the transform coefficients.

[0249] (Adder)

[0250] The adder 208 reconstructs the current block by adding the prediction error input from the inverse transformer 206 to the prediction sample as input from the prediction controller 220. Then, the adder 208 outputs the reconstructed block to the block memory 210 and the loop filter 212.

[0251] (Block memory)

[0252] The block memory 210 is a memory for storing a block in a picture (hereinafter referred to as a current picture) to be decoded to be referred to in intra prediction. More specifically, the block memory 210 stores the reconstructed block output from the adder 208.

[0253] (loop filter)

[0254] The loop filter 212 applies a loop filter to the block reconstructed by the adder 208, and outputs the filtered reconstructed block to the frame memory 214 and, for example, to a display device.

[0255] When the information indicating the enabling or disabling of the ALF parsed from the encoded bitstream indicates the enabling, one filter is selected from a plurality of filters based on a local gradient direction and activity, and the selected filter is applied to the reconstructed block.

[0256] (frame memory)

[0257] The frame memory 214 is a storage device for storing a reference picture used in inter prediction, and is also referred to as a frame buffer. More specifically, the frame memory 214 stores the reconstructed block filtered by the loop filter 212.

[0258] (intra predictor)

[0259] The intra predictor 216 refers to one or more blocks in the current picture as stored in the block memory 210, generates a prediction signal (intra prediction signal) by intra prediction. More specifically, the intra predictor 216 refers to samples (e.g., luma and / or chroma values) of one or more blocks adjacent to the current block, generates an intra prediction signal by intra prediction, and then outputs the intra prediction signal to the prediction controller 220.

[0260] Note that when an intra prediction mode in which a chroma block is intra predicted from a luma block is selected, the intra predictor 216 can predict a chroma component of the current block based on a luma component of the current block.

[0261] Further, when information indicating the application of PDPC is parsed from the encoded bitstream (e.g., in the prediction parameters output from the entropy decoder 202), the intra predictor 216 corrects intra-predicted pixel values based on horizontal / vertical reference pixel gradients.

[0262] (inter predictor)

[0263] The inter predictor 218 predicts a current block with reference to a reference picture stored in the frame memory 214. Inter prediction is performed on each current block or each sub-block (e.g., each 4x4 block) in the current block. For example, the inter predictor 218 generates an inter prediction signal of the current block or sub-block based on motion compensation using motion information (e.g., a motion vector) parsed from the coded bitstream (e.g., in the prediction parameters output from the entropy decoder 202), and outputs the inter prediction signal to the prediction controller 220.

[0264] When the information parsed from the coded bitstream indicates the application of the OBMC mode, the inter predictor 218 generates an inter prediction signal using motion information of a neighboring block in addition to motion information of the current block obtained from motion estimation.

[0265] Further, when the information parsed from the coded bitstream indicates the application of the FRUC mode, the inter predictor 218 derives motion information by performing motion estimation according to a pattern matching method (bilateral matching or template matching) parsed from the coded bitstream. Then, the inter predictor 218 performs motion compensation (prediction) using the derived motion information.

[0266] Further, when the BIO mode is to be applied, the inter predictor 218 derives a motion vector based on a model assuming uniform linear motion. Further, when the information parsed from the coded bitstream indicates that the affine motion compensation prediction mode is to be applied, the inter predictor 218 derives a motion vector for each sub-block based on motion vectors of neighboring blocks.

[0267] (Prediction controller)

[0268] The prediction controller 220 selects either the intra prediction signal or the inter prediction signal, and outputs the selected prediction signal to the adder 208. Generally, the configuration, functions, and operations of the prediction controller 220, the inter predictor 218, and the intra predictor 216 on the decoder side can correspond to those of the prediction controller 128, the inter predictor 126, and the intra predictor 124 on the encoder side.

[0269] (Non-rectangular partitions)

[0270] In the prediction controller 128 coupled to the intra predictor 124 and the inter predictor 126 on the encoder side (see Figure 1 ), and in the prediction controller 220 coupled to the intra predictor 216 and the inter predictor 218 on the decoder side (see Figure 10 So far, the partitions (or variable size blocks or sub-blocks) for which motion information (e.g., motion vectors) is obtained by partitioning each block are always rectangular as in Figure 2The inventors have discovered that, in various implementations, generating partitions having non-rectangular shapes such as triangles can lead to improvements in image quality and coding efficiency based on the content of the image in the picture. Below, various embodiments will be described in which at least one partition of a partitioning of an image block has a non-rectangular shape for the purpose of prediction. Note that these embodiments are equally applicable to the encoder side (prediction controller 128 coupled to intra-predictor 124 and inter-predictor 126) and the decoder side (prediction controller 220 coupled to intra-predictor 216 and inter-predictor 218), and can be implemented in an encoder of Figure 1 or in a decoder of Figure 10 and the like.

[0271] Figure 11 is a flowchart showing a process that partitions an image block into partitions including at least a first partition having a non-rectangular shape (e.g., a triangle) and a second partition, and performs a further process including encoding (or decoding) the image block into a reconstructed combination of the first partition and the second partition.

[0272] In step S1001, the image block is partitioned into a plurality of partitions, which can include a first partition having a non-rectangular shape and a second partition which can or can not have a non-rectangular shape. For example, as shown in Figure 12 , the image block can be partitioned from the top left corner of the image block to the bottom right corner of the image block to create a first partition and a second partition both having a non-rectangular shape (e.g., a triangle), or the image block can be partitioned from the top right corner of the image block to the bottom left corner of the image block to create a first partition and a second partition both having a non-rectangular shape (e.g., a triangle). Various examples of non-rectangular partitioning will be described below with reference to Figure 12 and 17 -19.

[0273] In step S1002, the process predicts a first motion vector for the first partition, and predicts a second motion vector for the second partition. For example, the prediction of the first and second motion vectors can include selecting the first motion vector from a first set of motion vector candidates, and selecting the second motion vector from a second set of motion vector candidates.

[0274] In step S1003, motion compensation processing is performed to obtain the first partition using the first motion vector derived in step S1002 above, and to obtain the second partition using the second motion vector derived in step S1002 above.

[0275] In step S1004, a prediction process is performed on the image block that is a combination of the first partition and the second partition (reconstructed). The prediction process can include a boundary smoothing process that is used to smooth the boundary between the first partition and the second partition. For example, the boundary smoothing process can involve weighting a first value of a boundary pixel that is predicted based on the first partition and a second value of the boundary pixel that is predicted based on the second partition. Various implementations of the boundary smoothing process will be described below with reference to Figure 13 , 14 , 20 and 21A-21D.

[0276] In step S1005, the process encodes or decodes the image block using one or more parameters that include a partition parameter that indicates that the image block is partitioned into a second partition and a first partition that has a non-rectangular shape. As summarized in the table in Figure 15 , for example, the partition parameter (“first index value”) can jointly encode the partition direction (e.g., from top left to bottom right or from top right to bottom left, as shown in Figure 12 , that is applied in the partitioning and the first and second motion vectors that are derived in step S1002 above. Details regarding such partition syntax operations involving one or more parameters including the partition parameter will be described in detail below with reference to Figure 15 , 16 and 22-25.

[0277] Figure 17 is a flowchart illustrating a process 2000 of partitioning an image block. In step S2001, the process partitions an image into a plurality of partitions that includes a first partition that has a non-rectangular shape and a second partition that can or can not have a non-rectangular shape. As shown in Figure 12 , an image block can be partitioned into a first partition that has a triangular shape and a second partition that also has a triangular shape. There are many other examples in which an image block is partitioned into a plurality of partitions that includes a first partition and a second partition, where at least the first partition has a non-rectangular shape. The non-rectangular shape can be a triangle, a trapezoid, and a polygon with at least five sides and corners.

[0278] For example, as shown in Figure 18 , an image block can be partitioned into two triangular partitions; an image block can be partitioned into more than two triangular partitions (e.g., three triangular partitions); an image block can be partitioned into a combination of a triangular partition and a rectangular partition; or an image block can be partitioned into a combination of a triangular partition and a polygonal partition.

[0279] As shown in Figure 19As further illustrated in FIG. 2, the image block can be partitioned into L-shaped (polygonal) partitions and rectangular-shaped partitions; the image block can be partitioned into pentagon (polygonal)-shaped partitions and triangle-shaped partitions; the image block can be partitioned into hexagon (polygonal)-shaped partitions and pentagon (polygonal)-shaped partitions; or the image block can be partitioned into multiple polygonal partitions.

[0280] Referring back to FIG. 2 Figure 17 In step S2002, the process predicts a first motion vector for a first partition, e.g., by selecting the first partition from a first set of motion vector candidates, and predicts a second motion vector for a second partition, e.g., by selecting the second motion vector from a second set of motion vector candidates. For example, the first set of motion vector candidates can include motion vectors of partitions neighboring the first partition, and the second set of motion vector candidates can include motion vectors of partitions neighboring the second partition. The neighboring partitions can be one or both of spatially neighboring partitions and temporally neighboring partitions. Some examples of spatially neighboring partitions include partitions located to the left, lower-left, lower, lower-right, right, upper-right, upper, or upper-left of the partition being processed. An example of a temporally neighboring partition is a partition in a reference picture of the image block that is located in the same position.

[0281] In various implementations, the partitions neighboring the first partition and the partitions neighboring the second partition can be outside of the image block from which the first and second partitions are partitioned. The first set of motion vector candidates can be the same as or different from the second set of motion vector candidates. Further, at least one of the first set of motion vector candidates and the second set of motion vector candidates can be the same as another set, a third set of motion vector candidates, prepared for the image block.

[0282] In some implementations, in step S2002, in response to determining that the second partition, like the first partition, also has a non-rectangular shape (e.g., a triangle), the process 2000 creates the second set of motion vector candidates (for the non-rectangular second partition) to include motion vectors of the partitions neighboring the second partition (excluding the first partition) (i.e., excluding the motion vector of the first partition). On the other hand, in response to determining that the second partition (different from the first partition) has a rectangular shape, the process 2000 creates the second set of motion vector candidates (for the rectangular second partition) to include motion vectors of the partitions neighboring the second partition (including the first partition).

[0283] In step S2003, the process encodes or decodes the first partition using the first motion vector derived in the above step S2002, and encodes or decodes the second partition using the second motion vector derived in the above step S2002.

[0284] Like the process 2000 of FIG. 2 Figure 17 The image block partitioning process can be performed by an image encoder, e.g., as in FIG. 3, which illustrates an example of a process 3000 for encoding an image block.Figure 1 As shown, the image encoder includes circuitry and a memory coupled to the circuitry. The circuitry performs the following operations: dividing an image block into multiple partitions, including a second partition and a first partition having a non-rectangular shape (step S2001); predicting a first motion vector for the first partition and a second motion vector for the second partition (step S2002); encoding the first partition using the first motion vector and encoding the second partition using the second motion vector (step S2003).

[0285] According to another embodiment, such as Figure 1 As shown, an image encoder is provided, comprising: a segmenter 102 that receives an original image and segments it into multiple blocks; a subtractor 104 that receives blocks from the segmenter and predictions from a prediction controller 128, and subtracts each prediction from its corresponding block to output a residual; a transformer 106 that performs a transform on the residuals output from the subtractor 104 to output transform coefficients; a quantizer 108 that quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder 110 that encodes the quantized transform coefficients to generate a bitstream; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124, and memories 118 and 122, wherein the inter-frame predictor 126 generates a prediction for the current block based on a reference block in an encoded reference image, and the intra-frame predictor 124 generates a prediction for the current block based on an encoded reference block in the current image. In operation, the predictive controller 128 divides the block into multiple partitions, including a second partition and a first partition with a non-rectangular shape. Figure 17 (Step S2001); Predict the first motion vector of the first partition and the second motion vector of the second partition (Step S2002); Encode the first partition using the first motion vector and encode the second partition using the second motion vector (Step S2003).

[0286] According to another embodiment, such as in Figure 10 As shown, an image decoder is provided, which includes circuitry and a memory coupled to the circuitry. The circuitry performs the following in operation: dividing an image patch into multiple partitions, said partitions including a second partition and a first partition having a non-rectangular shape. Figure 17 (Step S2001); Predict the first motion vector of the first partition and the second motion vector of the second partition (Step S2002); Use the first motion vector to decode the first partition and use the second motion vector to decode the second partition (Step S2003).

[0287] According to another embodiment, as follows is provided Figure 10The image decoder shown in FIG. 1 includes an entropy decoder 202 that, in operation, receives and decodes an encoded bitstream to obtain quantized transform coefficients, an inverse quantizer 204 and an inverse transformer 206 that, in operation, inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual, an adder 208 that, in operation, adds the residual output from the inverse quantizer 204 and the inverse transformer 206 to a prediction output from a prediction controller 220 to reconstruct a block, and the prediction controller 220 coupled to an inter-predictor 218, an intra-predictor 216, and memories 210, 214, where the inter-predictor 218, in operation, generates a prediction for a current block based on a reference block in a decoded reference picture and the intra-predictor 216, in operation, generates a prediction for the current block based on a decoded reference block in a current picture. The prediction controller 220, in operation, partitions the image block into a plurality of partitions including a first partition having a non-rectangular shape and a second partition. Figure 17 , step S2001); predicting a first motion vector for the first partition and a second motion vector for the second partition (step S2002); decoding the first partition using the first motion vector and the second partition using the second motion vector (step S2003).

[0288] (boundary smoothing)

[0289] As described above in Figure 11 , according to various embodiments, the step S1004 of performing a prediction process on the image block that is a combination of the (reconstructed) first partition having a non-rectangular shape and the second partition can involve applying a boundary smoothing process along the boundary between the first partition and the second partition.

[0290] For example, Figure 21B One example of a boundary smoothing process is shown that involves weighting a first value of a boundary pixel that is first predicted based on the first partition and a second value of the boundary pixel that is second predicted based on the second partition.

[0291] Figure 20 is a flowchart showing an overall boundary smoothing process 3000 according to one embodiment that involves weighting a first value of a boundary pixel that is first predicted based on the first partition and a second value of the boundary pixel that is second predicted based on the second partition. In step S3001, the image block is partitioned into the first partition and the second partition along the boundary, where at least the first partition has a non-rectangular shape, as in Figure 21A or above Figure 12 , 18 and 19.

[0292] In step S3002, a set of pixels in the first partition along the boundary ( Figure 21A A first prediction is made for a first value (e.g., color, brightness, transparency, etc.) of the "boundary pixels" in the image, wherein the first value is predicted first using information from the first partition. In step S3003, a second prediction is made for a second value of a (same) group of pixels along the boundary of the first partition, wherein the second prediction is made using information from the second partition. In some implementations, at least one of the first and second predictions is an inter-frame prediction process that predicts the first and second values ​​based on a reference partition in an encoded reference image. Figure 21D In some implementations, the prediction process predicts a first value (“first set of samples”) of all pixels in a first partition that includes a set of pixels on which the first and second partitions overlap, and predicts a second value (“second set of samples”) of only a set of pixels on which the first and second partitions overlap. In another implementation, at least one of the first and second predictions is an intra-frame prediction process that predicts the first and second values ​​based on an encoded reference partition in the current image. In some implementations, the prediction method used in the first prediction differs from the prediction method used in the second prediction. For example, the first prediction may include an inter-frame prediction process, and the second prediction may include an intra-frame prediction process. The information used for the first prediction of the first value or the second prediction of the second value may be the motion vector of the first or second partition, the intra-frame prediction direction, etc.

[0293] In step S3004, the first value predicted using the first partition and the second value predicted using the second partition are weighted. In step S3005, the weighted first and second values ​​are used to encode or decode the first partition.

[0294] Figure 21B An example of a boundary smoothing operation is shown, where the first and second partitions overlap by a maximum of five pixels per row or column. That is, the number of pixels in each row or column that predicts the first value based on the first partition and the second value based on the second partition is at most five. Figure 21C Another example of a boundary smoothing operation is shown, where the first and second partitions overlap by a maximum of three pixels per row or column. That is, the number of pixels in each row or column whose first value is predicted based on the first partition and whose second value is predicted based on the second partition is at most three.

[0295] Figure 13Another example of a boundary smoothing operation is shown, in which the first partition and the second partition overlap on (at most) four pixels per row or column. That is, the number of pixels in each row or column for which a first value is predicted based on the first partition and a second value is predicted based on the second partition is at most four. In the example shown, weights of 1 / 8, 1 / 4, 3 / 4, and 7 / 8 can be applied to the first values of the four pixels in the group, respectively, and weights of 7 / 8, 3 / 4, 1 / 4, and 1 / 8 can be applied to the second values of the four pixels in the group, respectively.

[0296] Figure 14 Further examples of a boundary smoothing operation are shown, in which the first partition and the second partition overlap on zero pixels per row or column (i.e., they do not overlap), one pixel per row or column (at most), and two pixels per row or column (at most). In the example in which the first and second partitions do not overlap, zero weight is applied. In the example in which the first partition and the second partition overlap on one pixel per row or column, a weight of 1 / 2 can be applied to the first values of the pixels in the group predicted based on the first partition, and a weight of 1 / 2 can be applied to the second values of the pixels in the group predicted based on the second partition. In the example in which the first partition and the second partition overlap on two pixels per row or column, weights of 1 / 3 and 2 / 3 can be applied to the first values of the two pixels in the group predicted based on the first partition, respectively, and weights of 2 / 3 and 1 / 3 can be applied to the second values of the two pixels in the group predicted based on the second partition, respectively.

[0297] According to the above embodiments, the number of pixels in the group on which the first partition and the second partition overlap is an integer. In other implementations, for example, the number of overlapping pixels in the group can be non-integer and can be fractional. Likewise, the weights applied to the first values and the second values of a group of pixels can be fractional or integer, depending on each application.

[0298] Similar to the process 3000 of FIG. 3A, a boundary smoothing process can be performed by, for example, an image encoder as shown in FIG. 3B, which includes circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation includes first predicting first values of a group of pixels of the first partition along the boundary using information of the first partition (step S3002), second predicting second values of the group of pixels of the first partition along the boundary using information of the second partition (step S3003), weighting the first values and the second values (step S3004), and encoding the first partition using the weighted first values and the weighted second values (step S3005). Figure 20 Figure 1 The circuitry, in operation, performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation includes first predicting first values of a group of pixels of the first partition along the boundary using information of the first partition (step S3002), second predicting second values of the group of pixels of the first partition along the boundary using information of the second partition (step S3003), weighting the first values and the second values (step S3004), and encoding the first partition using the weighted first values and the weighted second values (step S3005). Figure 20 The circuitry, in operation, performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation includes first predicting first values of a group of pixels of the first partition along the boundary using information of the first partition (step S3002), second predicting second values of the group of pixels of the first partition along the boundary using information of the second partition (step S3003), weighting the first values and the second values (step S3004), and encoding the first partition using the weighted first values and the weighted second values (step S3005).​

[0299] According to another embodiment, as shown in Figure 1 , there is provided an image encoder comprising: a partitioner 102 that in operation receives an original picture and partitions it into blocks; a subtractor 104 that in operation receives blocks from the partitioner and predictions from a prediction controller 128 and subtracts each prediction from its corresponding block to output a residual; a transformer 106 that in operation performs a transform on the residuals output from the subtractor 104 to output transform coefficients; a quantizer 108 that in operation quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder 110 that in operation encodes the quantized transform coefficients to generate a bitstream; and a prediction controller 128 coupled to an inter-predictor 126, an intra-predictor 124, and memories 118, 122, wherein the inter-predictor 126 in operation generates a prediction of a current block based on a reference block in an encoded reference picture, the intra-predictor 124 in operation generates a prediction of a current block based on an encoded reference block in a current picture. The prediction controller 128 in operation performs a boundary smoothing operation along a boundary between a second partition partitioned out of an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation comprises: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary (step S3002); using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary (step S3003); weighting the first values and the second values (step S3004); and encoding the first partition using the weighted first values and the weighted second values (step S3005). Figure 20

[0300] According to another embodiment, there is provided an image decoder as shown in Figure 10 , comprising circuitry and a memory coupled with the circuitry. The circuitry in operation performs a boundary smoothing operation along a boundary between a second partition partitioned out of an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation comprises: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary (step S3002); using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary (step S3003); weighting the first values and the second values (step S3004); and decoding the first partition using the weighted first values and the weighted second values (step S3005). Figure 20

[0301] According to another embodiment, there is provided an image decoder as shown in Figure 10 ​​The illustrated image decoder includes an entropy decoder 202 that, in operation, receives and decodes an encoded bitstream to obtain quantized transform coefficients, an inverse quantizer 204 and an inverse transformer 206 that, in operation, inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual, an adder 208 that, in operation, adds the residual output from the inverse quantizer 204 and inverse transformer 206 to a prediction output from a prediction controller 220 to reconstruct a block, and the prediction controller 220 coupled to an inter-predictor 218, an intra-predictor 216, and memories 210, 214, where the inter-predictor 218, in operation, generates a prediction of a current block based on a reference block in a decoded reference picture and the intra-predictor 216, in operation, generates a prediction of the current block based on a decoded reference block in a current picture. The prediction controller 220, in operation, performs a boundary smoothing operation along a boundary between a first partition having a non-rectangular shape and a second partition partitioned from an image block Figure 20 , step S3001). The boundary smoothing operation includes first predicting (step S3002) first values of a set of pixels of the first partition along the boundary using information of the first partition, second predicting (step S3003) second values of the set of pixels of the first partition along the boundary using information of the second partition, weighting (step S3004) the first values and the second values, and decoding (step S3005) the first partition using the weighted first values and the weighted second values.

[0302] (Entropy encoding and decoding using partition parameter syntax)

[0303] As illustrated in Figure 11 , step S1005, according to various embodiments, an image block partitioned into a second partition and a first partition having a non-rectangular shape can be encoded or decoded using one or more parameters including a partition parameter indicating a non-rectangular partitioning of the image block. In various embodiments, such a partition parameter can jointly encode, for example, a partition direction applied to the partitioning (e.g., from top left to bottom right or from top right to bottom left, see Figure 12 ), and first and second motion vectors predicted in step S1002, as will be more fully described below.

[0304] Figure 15 is a table of sample partition parameters ("first index value") and sets of information jointly encoded by the partition parameters, respectively. The partition parameter ("first index value") ranges from 0 to 6 and jointly encodes: a direction of partitioning an image block into a first partition and a second partition (both of which are triangles, see Figure 12 ) and a first motion vector predicted for the first partition Figure 11, step S1002), and predicting a second motion vector (MV2) for the second partition (P2) (step S1003). In particular, partition parameter 0 encodes a split direction from top-left to bottom-right, a first motion vector (MV1) is a "second" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "first" motion vector listed in a second set of motion vector candidates for the second partition (P2). Figure 11

[0305] Partition parameter 1 encodes a split direction from top-right to bottom-left, a first motion vector (MV1) is a "first" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "second" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 2 encodes a split direction from top-right to bottom-left, a first motion vector (MV1) is a "second" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "first" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 3 encodes a split direction from top-left to bottom-right, a first motion vector (MV1) is a "second" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "second" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 4 encodes a split direction from top-right to bottom-left, a first motion vector (MV1) is a "second" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "third" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 5 encodes a split direction from top-left to bottom-right, a first motion vector (MV1) is a "third" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "first" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 6 encodes a split direction from top-left to bottom-right, a first motion vector (MV1) is a "fourth" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "first" motion vector listed in a second set of motion vector candidates for the second partition (P2).

[0306] Figure 22 is a flowchart illustrating a method 4000 performed at an encoder side. In step S4001, the process partitions an image block into a plurality of partitions based on a partition parameter indicating a split, the plurality of partitions including a second partition (P2) and a first partition (P1) having a non-rectangular shape. For example, as described above with respect to FIG. 10, the first partition (P1) has a non-rectangular shape. Figure 15 ​As shown, the partitioning parameters can indicate the direction of segmenting the image patch (e.g., from the upper right corner to the lower left corner or from the upper left corner to the lower right corner). In step S4002, the process encodes the first and second partitions. In step S4003, the process writes one or more parameters, including the partitioning parameters, into a bitstream. The decoder side can receive and decode the bitstream to obtain one or more parameters, thereby performing the same prediction process (as performed on the encoder side) on the first and second partitions. The one or more parameters including the partitioning parameters can jointly or separately encode various information fragments, such as the non-rectangular shape of the first partition, the shape of the second partition, the segmentation direction for segmenting the image patch to obtain the first and second partitions, the first motion vector of the first partition, the second motion vector of the second partition, etc.

[0307] Figure 23 This is a flowchart illustrating method 5000 performed on the decoder side. In step S5001, the process parses one or more parameters from the bitstream, wherein the one or more parameters include partitioning parameters indicating that an image patch is divided into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape. The one or more parameters, including the partitioning parameters parsed from the bitstream, can jointly or separately encode various information fragments required for performing the same prediction processing on the decoder side as on the encoder side, such as the non-rectangular shape of the first partition, the shape of the second partition, the segmentation direction for segmenting the image patch to obtain the first and second partitions, a first motion vector of the first partition, a second motion vector of the second partition, etc. In step S5002, process 5000 segments the image patch into multiple partitions based on the partitioning parameters parsed from the bitstream. In step S5003, the process decodes the first and second partitions segmented from the image patch.

[0308] Figure 24 It is a table of sampling partition parameters (“first index value”) and information sets jointly encoded by the partition parameters, and is essentially similar to the above. Figure 15 The sampling table in [the document / reference table]. Figure 24 In this context, the partitioning parameter (“first index value”) ranges from 0 to 6, and the following items are jointly encoded: the shape of the first and second partitions segmented from the image patch, the orientation in which the image patch is segmented into the first and second partitions, and the first motion vector predicted for the first partition. Figure 11 Step S1002), and the second motion vector predicted for the second partition ( Figure 11 (Step S1002). Specifically, the partition parameter 0 is encoded as follows: neither the first partition nor the second partition has a triangular shape, and therefore, the partitioning direction information is "N / A", the first motion vector information is "N / A", and the second motion vector information is "N / A".

[0309] Partition parameter 1 encodes the following: the first and second partitions are triangles, the split direction is from top-left to bottom-right, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 2 encodes the following: the first and second partitions are triangles, the split direction is from top-right to bottom-left, the first motion vector is the "first" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 3 encodes the following: the first and second partitions are triangles, the split direction is from top-right to bottom-left, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 4 encodes the following: the first and second partitions are triangles, the split direction is from top-left to bottom-right, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 5 encodes the following: the first and second partitions are triangles, the split direction is from top-right to bottom-left, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "third" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 6 encodes the following: the first and second partitions are triangles, the split direction is from top-left to bottom-right, the first motion vector is the "third" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition.

[0310] According to some implementations, the partition parameters (index values) can be binarized according to a binarization scheme that is selected according to the value of at least one or one or more parameters. Figure 16 An example binarization scheme for binarizing index values (partition parameter values) is shown.

[0311] Figure 25 is a table of sample combinations of a first parameter and a second parameter, where one parameter is a partition parameter indicating a partitioning of an image block into a plurality of partitions including a second partition and a first partition having a non-rectangular shape. In this example, the partition parameter can be used to indicate the partitioning of the image block without jointly encoding other information encoded by one or more other parameters.

[0312] In a first example of Figure 25 , the first parameter is used to indicate the image block size, and the second parameter is used as a partition parameter (flag) to indicate that at least one of the plurality of partitions split from the image block has a triangular shape. Such a combination of the first and second parameters can be used to indicate, for example, 1) no triangular partition when the image block size is greater than 64x64, or 2) no triangular partition when the aspect ratio of the image block is greater than 4 (e.g., 64x4).

[0313] In a second example of Figure 25 , the first parameter is used to indicate the prediction mode, and the second parameter is used as a partition parameter (flag) to indicate that at least one of the plurality of partitions split from the image block has a triangular shape. Such a combination of the first and second parameters can be used to indicate, for example, 1) no triangular partition when the image block is coded in intra mode.

[0314] In a third example of Figure 25 , the first parameter is used as a partition parameter (flag) to indicate that at least one of the plurality of partitions split from the image block has a triangular shape, and the second parameter is used to indicate the prediction mode. Such a combination of the first and second parameters can be used to indicate, for example, 1) the image block must be inter coded when at least one of the plurality of partitions split from the image block has a triangular shape.

[0315] In a fourth example of Figure 25 , the first parameter indicates the motion vector of a neighboring block, and the second parameter is used as a split parameter to indicate the direction to split the image block into two triangles. Such a combination of the first and second parameters can be used to indicate, for example, 1) the direction to split the image block into two triangles is from the top-left corner to the bottom-right corner when the motion vector of the neighboring block is a diagonal direction.

[0316] In a fifth example of Figure 25 , the first parameter indicates the intra prediction direction of a neighboring block, and the second parameter is used as a partition parameter to indicate the direction to split the image block into two triangles. Such a combination of the first and second parameters can be used to indicate, for example, 1) the direction to split the image block into two triangles is from the top-right corner to the bottom-left corner when the intra prediction direction of the neighboring block is an anti-diagonal direction.

[0317] It should be appreciated that, as Figure 15 , 24As shown in Figure 25, the table including partitioning parameters and which information is jointly or individually encoded is presented merely as an example, and many other ways of jointly or individually encoding various information as part of the partitioning syntax operations described above are also within the scope of this disclosure. For example, partitioning parameters may indicate that the first partition is a triangle, trapezoid, or a polygon with at least five sides and angles. Partitioning parameters may indicate that the second partition has a non-rectangular shape, such as a triangle, trapezoid, or a polygon with at least five sides and angles. Partitioning parameters may indicate one or more pieces of information about the segmentation, such as the non-rectangular shape of the first partition, the shape of the second partition (which may be non-rectangular or rectangular), and the segmentation direction applied to segment the image patch into multiple partitions (e.g., from the upper left corner of the image patch to its lower right corner, and from the upper right corner of the image patch to its lower left corner). Partitioning parameters may jointly encode other information, such as a first motion vector of the first partition, a second motion vector of the second partition, the image patch size, the prediction mode, the motion vectors of neighboring patches, the intra-frame prediction directions of neighboring patches, etc. Alternatively, any information in the further information may be encoded separately by one or more parameters other than the partitioning parameters.

[0318] Similar to Figure 22 The process 4000, partitioning syntax operations can be performed by, for example Figure 1 The image encoder shown includes circuitry and a memory coupled to the circuitry. The circuitry performs a partitioning syntax operation during operation, which includes: dividing an image patch into multiple partitions based on partitioning parameters indicating the segmentation, the multiple partitions including a second partition and a first partition having a non-rectangular shape. Figure 22 Step S4001); Encode the first partition and the second partition (S4002); Write one or more parameters, including partition parameters, into the bit stream (S4003).

[0319] According to another embodiment, such as Figure 1As shown, there is provided an image encoder comprising: a partitioner 102 that in operation receives an original picture and partitions it into blocks; a subtractor 104 that in operation receives blocks from the partitioner and predictions from a prediction controller 128 and subtracts each prediction from its corresponding block to output a residual; a transformer 106 that in operation performs a transform on the residuals output from the subtractor 104 to output transform coefficients; a quantizer 108 that in operation quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder 110 that in operation encodes the quantized transform coefficients to generate a bitstream; and a prediction controller 128 coupled to an inter-predictor 126, an intra-predictor 124 and memories 118, 122, wherein the inter-predictor 126 in operation generates a prediction of a current block based on a reference block in an encoded reference picture, the intra-predictor 124 in operation generates a prediction of a current block based on an encoded reference block in a current picture. The prediction controller 128 in operation partitions an image block into a plurality of partitions based on partition parameters indicative of the partitioning, the plurality of partitions including a second partition and a first partition having a non-rectangular shape Figure 22 , step S4001), and encoding the first and second partitions (step S4002). The entropy encoder 110 in operation writes one or more parameters including the partition parameters into the bitstream (step S4003).

[0320] According to another embodiment, there is provided an image decoder, e.g. as Figure 10 As shown, the image decoder comprises circuitry and a memory coupled to the circuitry. The circuitry in operation performs a partition syntax operation comprising: parsing one or more parameters from a bitstream, wherein the one or more parameters include partition parameters indicative of partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape Figure 23 , step S5001); partitioning the image block into the plurality of partitions based on the partition parameters (S5002); decoding the first and second partitions (S5003).

[0321] According to a further embodiment, there is provided an image decoder as Figure 10The image decoder shown in FIG. 2 includes an entropy decoder 202 that in operation receives and decodes an encoded bitstream to obtain quantized transform coefficients, an inverse quantizer 204 and an inverse transformer 206 that in operation inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain residuals, an adder 208 that in operation adds the residuals output from the inverse quantizer 204 and inverse transformer 206 to predictions output from a prediction controller 220 to reconstruct a block, and the prediction controller 220 coupled to an inter-predictor 218, an intra-predictor 216, and memories 210, 214, where the inter-predictor 218 in operation generates a prediction for a current block based on a reference block in a decoded reference picture and the intra-predictor 216 in operation generates a prediction for the current block based on a decoded reference block in a current picture. The entropy decoder 202 in operation parses one or more parameters from the bitstream, where the one or more parameters include partition parameters indicating partitioning of an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape (S5001); partitions the image block into the plurality of partitions based on the partition parameters (S5002); and decodes the first partition and the second partition in cooperation with the prediction controller 220 (S5003) in some implementations. Figure 23

[0322] According to other examples, the intra-predictor can perform the following process.

[0323] All of the motion vector candidates included in the first set of motion vector candidates can be single-prediction motion vectors. That is, the intra-predictor can only determine single-prediction motion vectors as motion vector candidates in the first set of motion vector candidates.

[0324] The intra-predictor can only select single-prediction motion vector candidates from the first set of motion vector candidates.

[0325] Only single-prediction motion vectors can be used to predict small blocks. Double-prediction motion vectors can be used to predict large blocks. For example, the prediction process can include determining a size of the image block. When the size of the image block is determined to be greater than a threshold, the prediction can include selecting a first motion vector from the first set of motion vector candidates, and the first set of motion vector candidates can contain single-prediction and / or double-prediction motion vectors. When the size of the image block is determined to be not greater than the threshold, the prediction can select a first motion vector from the first set of motion vector candidates, and the first set of motion vector candidates can contain only single-prediction motion vectors.

[0326] (Implementation and Application)

[0327] ​As described in each of the above embodiments, each functional or operational block can generally be implemented as, for example, an MPU (micro processing unit) and a memory. Furthermore, the process performed by each of the functional blocks can be implemented as a program execution unit, such as a processor that reads and executes software (a program) recorded on a recording medium such as a ROM. The software can be distributed. The software can be recorded on various recording media such as a semiconductor memory. Note that each functional block can also be implemented as hardware (a dedicated circuit).

[0328] The processes described in each of the embodiments can be implemented by using integrated processing of a single apparatus (system), or can be implemented by using decentralized processing of a plurality of apparatuses. Furthermore, the processor that executes the above-described program can be a single processor or a plurality of processors. In other words, integrated processing can be performed, or decentralized processing can be performed.

[0329] Embodiments of the present disclosure are not limited to the above example embodiments; various modifications can be made to the example embodiments, and the results thereof are also included in the scope of the embodiments of the present disclosure.

[0330] Next, application examples of the moving picture encoding method (image encoding method) and the moving picture decoding method (image decoding method) described in each of the above embodiments, and various systems that implement the application examples will be described. Such a system can be characterized by including an image encoder that employs the image encoding method, an image decoder that employs the image decoding method, or both an image encoder and an image decoder. Other configurations of such a system can be modified as appropriate.

[0331] (Usage Examples)

[0332] Figure 26 The overall configuration of a content providing system ex100 suitable for implementing a content distribution service is shown. The area in which a communication service is provided is divided into cells of a desired size, and in the example shown, base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations, are located in the respective cells.

[0333] In the content providing system ex100, devices including a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via the Internet service provider ex102 or the communication network ex104 and the base stations ex106 to ex110. The content providing system ex100 can combine and connect any combination of the above-described devices. In various implementations, the devices can be connected together directly or indirectly via a telephone network or near field communication instead of via the base stations ex106 to ex110. Furthermore, the streaming server ex103 can be connected to the devices including the computer ex111, the game device ex112, the camera ex113, the home appliance ex114, and the smartphone ex115 via, for example, the Internet ex101. The streaming server ex103 can also be connected to terminals in a hotspot in, for example, an airplane ex117 via the satellite ex116.

[0334] Note that wireless access points or hotspots can be used instead of the base stations ex106 to ex110. The streaming server ex103 can be directly connected to the communication network ex104 instead of through the Internet ex101 or the Internet service provider ex102, and can be directly connected to the airplane ex117 instead of through the satellite ex116.

[0335] The camera ex113 is a device capable of capturing still images and videos, such as a digital camera. The smartphone ex115 is a smartphone device, a cellular phone, or a Personal Handyphone System (PHS) phone that can operate under the standards of mobile communication systems of 2G, 3G, 3.9G, and 4G systems and the next generation 5G system.

[0336] The home appliance ex114 is, for example, a refrigerator or a device included in a home fuel cell cogeneration system.

[0337] In the content providing system ex100, a terminal including an image and / or video capturing function is capable of real-time streaming, for example, by connecting to the streaming server ex103 via, for example, the base station ex106. When real-time streaming is performed, the terminal (for example, the computer ex111, the game device ex112, the camera ex113, the home appliance ex114, the smartphone ex115, or the airplane ex117) can perform the encoding process described in the above-described embodiments on still image or video content captured by a user via the terminal, can multiplex video data obtained via encoding and audio data obtained by encoding audio corresponding to the video, and can transmit the obtained data to the streaming server ex103. In other words, the terminal functions as an image encoder according to an aspect of the present disclosure.

[0338] The streaming server ex103 streams the transmitted content data to a client that requests the stream. Examples of the client include a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and a terminal in an airplane ex117, which are capable of decoding the above-mentioned encoded data. The device that receives the streamed data decodes and reproduces the received data. In other words, each of the devices can function as an image decoder according to an aspect of the present disclosure.

[0339] (Decentralized processing)

[0340] The streaming server ex103 can be implemented as a plurality of servers or computers that divide tasks such as processing, recording, and streaming of data among them. For example, the streaming server ex103 can be implemented as a content delivery network (CDN) that streams content via a network connecting a plurality of edge servers located around the world. In the CDN, an edge server that is physically close to a client is dynamically assigned to the client. Content is cached and streamed to the edge server to reduce loading time. For example, in the event of some type of error or change in connectivity due to a traffic peak, it is possible to stably stream data at high speed because it is possible to avoid the affected part of the network by, for example, dividing the processing among a plurality of edge servers, or switching the streaming responsibility to a different edge server and continuing the streaming.

[0341] Decentralization is not limited to dividing processing for streaming; encoding of captured data can be allocated among terminals on the server side, or performed by the terminals, or both. In one example, in typical encoding, the processing is performed in two loops. The first loop is used to detect the complexity of the image on a frame-by-frame or scene-by-scene basis, or to detect the encoding load. The second loop is used for processing that maintains image quality and improves encoding efficiency. For example, it is possible to reduce the processing responsibility of the terminals and improve the quality and encoding efficiency of the content by having the terminals perform the first loop of encoding, and having the server side that receives the content perform the second loop of encoding. In such a case, upon receiving a decoding request, it is possible to receive and reproduce the encoded data produced as a result of the first loop performed on one terminal on another terminal in near real time. It is thus possible to achieve smooth, real-time streaming.

[0342] In another example, the camera ex113 or the like extracts a feature amount from an image, compresses data related to the feature amount as metadata, and transmits the compressed metadata to the server. For example, the server determines the importance of an object based on the feature amount, and changes quantization precision accordingly to perform compression suitable for the meaning (or content importance) of the image. The feature amount data is particularly effective in improving the accuracy and efficiency of motion vector prediction during a second compression process performed by the server. Furthermore, the terminal can process encoding with relatively low processing load such as variable length coding (VLC), and encoding with relatively high processing load such as context adaptive binary arithmetic coding (CABAC) can be processed by the server.

[0343] In yet another example, there is a case in which a plurality of terminals capture a plurality of videos of approximately the same scene, for example, in a stadium, a shopping center, or a factory. In such a case, for example, encoding can be decentralized by dividing processing tasks on a per-unit basis among the plurality of terminals that capture videos and, if necessary, other terminals that do not capture videos, and the server. The unit can be, for example, a group of pictures (GOP), a picture, or a tile obtained by dividing a picture. This can reduce the loading time and enable streaming closer to real time.

[0344] Since the videos have approximately the same scene, the server can perform management and / or instruction so that the videos captured by the terminals can be cross-referenced. Furthermore, the server can receive encoded data from the terminals, change the reference relationship between data items, or correct or replace pictures itself, and then perform encoding. This makes it possible to generate a stream with improved quality and efficiency for individual data items.

[0345] Furthermore, the server can stream video data after performing transcoding to convert the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (e.g., VP9), and can convert H.264 to H.265.

[0346] In this way, encoding can be performed by the terminal or the server or servers. Therefore, although the device that performs encoding is referred to as a "server" or a "terminal" in the description below, some or all of the processes performed by the server can be performed by the terminal, and likewise, the processing performed by the terminal can be performed by the server. The same applies to the decoding process.

[0347] (3D, multi-angle)

[0348] The use of images or videos combined from images or videos of different scenes captured at the same time or from the same scene captured from different angles has increased through a plurality of terminals (for example, a camera ex113 and / or a smartphone ex115). Videos captured by terminals are combined based on, for example, a relative positional relationship individually obtained between terminals or a region having matching feature points in a video.

[0349] In addition to encoding two-dimensional moving pictures, the server can automatically or at a user-specified point in time encode still images based on scene analysis of the moving pictures and transmit the encoded still images to the receiving terminal. Furthermore, when the server is able to obtain a relative positional relationship between video-capturing terminals, in addition to two-dimensional moving pictures, the server can generate a three-dimensional geometry of a scene based on videos of the same scene captured from different angles. The server can encode three-dimensional data generated from, for example, a point cloud, respectively, and based on a result of recognizing or tracking a person or an object using the three-dimensional data, the server can select or reconstruct and generate a video to be transmitted to the receiving terminal from videos captured by a plurality of terminals.

[0350] This allows a user to enjoy a scene by freely selecting a video corresponding to a video-capturing terminal, and allows a user to enjoy content obtained by extracting a video at a selected viewpoint from three-dimensional data reconstructed from a plurality of images or videos. Furthermore, as with a video, sound can be recorded from relatively different angles, and the server can multiplex audio of a specific angle or space with a corresponding video, and transmit the multiplexed video and audio.

[0351] In recent years, content composed of a real world and a virtual world, such as virtual reality (VR) and augmented reality (AR) content, has also become popular. In the case of a VR image, the server can create an image from the viewpoint of both the left eye and the right eye, and perform encoding that allows reference between the two viewpoint images, such as multi-view encoding (MVC), or can encode the images as separate streams without reference. When the images are decoded as separate streams, the streams can be synchronized at the time of reproduction in order to recreate a virtual three-dimensional space according to the viewpoint of a user.

[0352] In the case of an AR image, the server superimposes virtual object information existing in a virtual space onto camera information representing a real space based on a three-dimensional position or motion from the user's perspective. The decoder can acquire or store virtual object information and three-dimensional data, generate a two-dimensional image based on motion from the user's perspective, and then generate superimposed data by seamlessly connecting the images. Alternatively, the decoder can transmit motion from the user's perspective to the server in addition to a request for virtual object information. The server can generate superimposed data based on three-dimensional data stored in the server according to the received motion, and encode and stream the generated superimposed data to the decoder. Note that the superimposed data includes an alpha value representing transparency in addition to an RGB value, and the server sets the alpha value of a portion other than an object generated from three-dimensional data to, for example, 0, and can be encoded while the portion is transparent. Alternatively, the server can set a background to a predetermined RGB value (e.g., chroma key), and generate data setting a region other than the object to the background.

[0353] Decoding of similarly streamed data can be performed at the server side by the client (i.e., terminal), or divided between them. In one example, one terminal can send a reception request to the server, the requested content can be received and decoded by another terminal, and the decoded signal can be transmitted to a device having a display. High image quality data can be reproduced through decentralized processing and appropriate selection of content regardless of the processing capability of the communication terminal itself. In yet another example, when a TV is receiving large size image data, for example, regions of a picture such as tiles obtained by dividing the picture can be decoded and displayed on a personal terminal or a terminal of one or more viewers of the TV. This enables viewers to share a large picture view, and each viewer can check its assigned region, or view the region more closely.

[0354] In a case where a plurality of wireless connections can be performed indoors, outdoors, at close range, at medium range, and at long range, a streaming system standard such as MPEG-DASH can be used to seamlessly receive content. A user can switch between data in real time while freely selecting a decoder or a display device including a user terminal, a display arranged indoors or outdoors, or the like. Furthermore, using, for example, information on a user's location, it is possible to perform decoding while switching which terminal processes decoding and which terminal processes display of content. This makes it possible to draw and display information on a wall of a nearby building or on a portion of the ground that is embedded with a device capable of displaying content when a user is on the way to a destination. Furthermore, it is also possible to switch the bit rate of received data based on the accessibility of encoded data on a network, such as when encoded data is cached on a server that is quickly accessible from a receiving terminal or when encoded data is copied to an edge server in a content delivery service.

[0355] (scalable encoding)

[0356] Switching of content will be described with reference to a scalable stream shown in Figure 27 , which is compressed and encoded by implementation of the moving picture encoding method described in the above-described embodiments. A server can have a configuration in which content is switched while utilizing temporal and / or spatial scalability of a stream, which is achieved by dividing and encoding layers, as shown in Figure 27 . Note that there can be a plurality of individual streams that are the same in content but different in quality. In other words, by determining a layer to be decoded based on internal factors such as processing capability on the decoder side and external factors such as communication bandwidth, the decoder side can freely switch between low-resolution content and high-resolution content while decoding. For example, in a case where a user wants to continue watching a video that the user has previously watched on a smartphone ex115 while moving, using a device such as a television connected to the Internet, for example, in the user's home, the device can simply decode the same stream up to a different layer, which reduces the load on the server side.

[0357] Furthermore, in addition to the above-described configuration in which scalability is achieved by encoding pictures by layer, and in addition to the fact that an enhancement layer is located above a base layer, the enhancement layer can also include metadata based on, for example, statistical information on an image. The decoder side can generate content of high image quality by performing super-resolution imaging on a picture in the base layer based on the metadata. Super-resolution imaging can improve a signal-to-noise ratio while maintaining resolution and / or improving resolution. The metadata includes information for identifying linear or nonlinear filter coefficients used in super-resolution processing, or information for identifying parameter values of filter processing, machine learning, or least squares used in super-resolution processing.

[0358] Alternatively, a configuration can be provided in which a picture is divided into, for example, tiles according to the meaning of the objects in the image. At the decoder side, only part of the area is decoded by selecting the tile to be decoded. Furthermore, by storing the attributes of the objects (person, car, ball, etc.) and the position of the objects in the video (coordinates in the same image) as metadata, the decoder side can identify the position of the desired object based on the metadata and determine which tile or tiles contain the object. For example, as shown in Figure 28 the metadata can be stored using a data storage structure different from the pixel data, such as the SEI (Supplemental Enhancement Information) message in HEVC. This metadata indicates, for example, the position, size, or color of the main object.

[0359] The metadata can be stored in units of multiple pictures, such as a stream, sequence, or random access unit. The decoder side can obtain, for example, the time at which a certain person appears in the video, and by fitting the time information with the picture unit information, the picture in which the object is present can be identified, and the position of the object in the picture can be determined.

[0360] (Web page optimization)

[0361] Figure 29 An example of a display screen of a web page on, for example, a computer ex111 is shown. Figure 30 An example of a display screen of a web page on, for example, a smartphone ex115 is shown. As shown in Figure 29 and Figure 30 The web page can include a plurality of image links that are links to image content, and the appearance of the web page differs depending on the device used to view the web page, as shown in FIGS. 11A to 11C. When a plurality of image links can be seen on the screen, the display device (decoder) can display a still image included in the content or I picture as the image link until the user explicitly selects the image link, or until the image link is located at the approximate center of the screen, or until the entire image link fits on the screen; a plurality of still images or I pictures can be used to display a video such as an animated gif; or only the base layer can be received, and the video can be decoded and displayed.

[0362] When the user selects the image link, the display device performs decoding while giving the highest priority to the base layer. Note that if there is information indicating that the content is scalable in the HTML code of the web page, the display device can decode up to the enhancement layer. Furthermore, in order to guarantee real-time reproduction, the display device can reduce the delay between the point in time at which the leading pictures are decoded and the point in time at which the decoded pictures are displayed (i.e., the delay between the start of decoding of the content and the display of the content) by decoding and displaying only forward reference pictures (I pictures, P pictures, forward reference B pictures) before the selection is made or when the bandwidth is strictly limited. In addition, the display device can intentionally ignore the reference relationship between pictures and roughly decode all B and P pictures as forward reference pictures, and then perform regular decoding as the number of pictures received over time increases.

[0363] (Autonomous driving)

[0364] When transmitting and receiving still image or video data (e.g., two-dimensional or three-dimensional map information for autonomous or assisted driving of a car), the receiving terminal can receive, in addition to the image data belonging to one or more layers, information about, for example, the weather or road construction as metadata, and associate the metadata with the image data at the time of decoding. Note that the metadata can be distributed by layer, and can alternatively be multiplexed with the image data only.

[0365] In such a case, since the car, drone, airplane, or the like that contains the receiving terminal is movable, the receiving terminal can seamlessly receive and perform decoding while switching between the base stations in the base stations ex106 to ex110 by transmitting information indicating the position of the receiving terminal. Furthermore, the receiving terminal can dynamically select the degree of reception of metadata or the degree of update of, for example, map information according to the user's selection, the user's situation, and / or the bandwidth of the connection.

[0366] In the content providing system ex100, the client can receive, decode, and reproduce the encoded information transmitted by the user in real time.

[0367] (Streaming individual content)

[0368] In the content providing system ex100, in addition to high image quality, long content distributed by a video distribution entity, unicast or multicast streaming of low image quality, and short content from individuals are also possible. The popularity of such content from individuals is likely to further increase. The server can first perform an editing process on the content before the encoding process in order to refine the individual content. For example, this can be achieved using the following configuration.

[0369] When real-time capturing video or image contents, or after capturing and accumulating the contents, the server performs recognition processing based on the original data or the encoded data, such as capturing error processing, scene search processing, meaning analysis, and / or object detection processing. Then, based on the results of the recognition processing, the server edits the contents when prompted or automatically, examples of which include correction such as focus and / or motion blur correction, removal of low-priority scenes such as scenes with low brightness compared to other pictures, or scenes with focus out, object edge adjustment, and tone adjustment. The server encodes the edited data based on the results of the editing. As is well known, videos that are too long result in reduced viewing times. Thus, in order to keep the contents within a specific length comparable to the length of the original video, in addition to the low-priority scenes described above, the server can automatically crop scenes with low motion based on the results of the image processing. Alternatively, the server can generate and encode a video summary based on the results of the analysis of the meaning of the scenes.

[0370] In some cases, individual contents can include contents that infringe copyrights, moral rights, portrait rights, and the like. Such instances can put the creator in an unfavorable position, for example, when the creator intends to share the contents beyond the scope. Thus, before encoding, the server can, for example, edit the images so that the faces of people outside the periphery of the screen are blurred or so that the interior of a house is blurred. In addition, the server can be configured to recognize the faces of people other than registered people in the images to be encoded, and when such a face appears in the image, a mosaic filter can be applied to, for example, the face of the person. Alternatively, as pre-processing or post-processing for encoding, the user can designate an image region including a person or a background to be processed for copyright reasons. The server can process the designated region by, for example, replacing the region with a different image or blurring the region. If the region includes a person, the person can be tracked in the moving picture, and the head region of the person can be replaced with another image as the person moves.

[0371] Since there is a need to watch the contents produced by individuals in real time, which tend to be small in data size, the decoder first receives the base layer as the highest priority and performs decoding and reproduction, but this can vary depending on the bandwidth. When the contents are reproduced twice or more, for example, when the decoder receives the enhancement layer during decoding and reproduction of the base layer and loops the reproduction, the decoder can reproduce a video with high image quality including the enhancement layer. If such a scalable encoding is used to encode the stream, the video can be low in quality when in an unselected state or at the beginning of the video, but it can provide an experience in which the image quality of the stream is gradually improved in an intelligent manner. This is not limited to scalable encoding; a single stream can be configured by a low-quality stream from the first reproduction and a second stream encoded using the first stream as a reference, thereby providing the same experience.

[0372] (Other implementation and application examples)

[0373] Encoding and decoding can be performed by an LSI (Large Scale Integrated Circuit) ex500 (see below), which is typically included in each terminal. Figure 26 The LSI ex500 can be configured as a single chip or multiple chips. Software for encoding and decoding moving images can be integrated into a type of recording medium (e.g., CD-ROM, floppy disk, or hard disk), which a computer ex111 can read and use to perform encoding and decoding. Furthermore, when the smartphone ex115 is equipped with a camera, video data acquired by that camera can be transmitted. In this case, the video data is encoded by the LSI ex500 included in the smartphone ex115.

[0374] Note that the LSIex500 can be configured to download and activate applications. In this case, the terminal first determines whether it is compatible with the encoding scheme used to encode the content, or whether it can perform a specific service. When the terminal is incompatible with the content's encoding scheme, or when the terminal cannot perform a specific service, the terminal first downloads the codec or application software, and then acquires and reproduces the content.

[0375] Besides the example of the content delivery system ex100 using the Internet ex101, the motion picture encoder (image encoder) or motion picture decoder (image decoder) described in at least the above embodiments can be implemented in a digital broadcasting system. The same encoding and decoding processes can be applied, for example, to transmitting and receiving broadcast radio waves superimposed with multiplexed audio and video data using satellites, even if this is for multicast, whereas unicast is easier for the content delivery system ex100.

[0376] (Hardware Configuration)

[0377] Figure 31 It shows Figure 26 More details about the smartphone ex115 shown. Figure 32A configuration example of a smartphone ex115 is shown. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves to and from the base station ex110, a camera ex465 capable of capturing video and still images, and a display ex458 for displaying decoded data, such as video captured by the camera ex465 and video received by the antenna ex450. The smartphone ex115 also includes a user interface ex466 such as a touch panel, an audio output unit ex457 such as a speaker for outputting voice or other audio, an audio input unit ex456 such as a microphone for audio input, a memory ex467 capable of storing decoded data such as captured video or still images, recorded audio, received video or still images, mail, and decoded data, and a slot ex464, which is an interface for a SIM ex468, for authorizing access to a network and various data. Note that an external memory can be used instead of ex467.

[0378] A main controller ex460, which can comprehensively control the display ex458 and the user interface ex466, a power supply circuit ex461, a user interface input controller ex462, a video signal processor ex455, a camera interface ex463, a display controller ex459, a modulator / demodulator ex452, a multiplexer / demultiplexer ex453, an audio signal processor ex454, the slot ex464, and the memory ex467 are connected by a bus ex470.

[0379] When the user turns on the power button of the power supply circuit ex461, the smartphone ex115 is powered on to an operable state, and each component is powered from a battery pack.

[0380] The smartphone ex115 performs processing such as calling and data transmission based on control performed by a main controller ex460 including a CPU, a ROM, and a RAM. When a call is made, an audio signal recorded by an audio input unit ex456 is converted into a digital audio signal by an audio signal processor ex454, and spread spectrum processing is performed on the digital audio signal by a modulator / demodulator ex452 and digital-to-analog conversion is performed, and frequency conversion processing is performed by a transmitter / receiver ex451, and the resulting signal is transmitted via an antenna ex450. Received data is amplified, frequency-converted, and analog-to-digital-converted, inverse spread spectrum-processed by the modulator / demodulator ex452, converted into an analog audio signal by the audio signal processor ex454, and then output from an audio output unit ex457. In a data transmission mode, for example, based on operation of a user interface ex466 of the main body, the main controller ex460 transmits text, still images, or video data via a user interface input controller ex462. Similar transmission and reception processing is performed. In the data transmission mode, when video, still images, or video and audio are transmitted, a video signal processor ex455 compressively encodes a video signal stored in a storage ex467 or a video signal input from a camera ex465 via the moving picture encoding method described in the above-described embodiment, and transmits the encoded video data to a multiplexer / demultiplexer ex453. The audio signal processor ex454 encodes an audio signal recorded by the audio input unit ex456 while the camera ex465 is capturing video or still images, and transmits the encoded audio data to the multiplexer / demultiplexer ex453. The multiplexer / demultiplexer ex453 multiplexes the encoded video data and the encoded audio data using a predetermined scheme, modulates and converts the data using the modulator / demodulator (modulator / demodulator circuit) ex452 and the transmitter / receiver ex451, and transmits the result via the antenna ex450.

[0381] For example, when a video attached in an e-mail or a chat or a video linked from a web page is received, in order to decode multiplexed data received via the antenna ex450, the multiplexer / demultiplexer ex453 demultiplexes the multiplexed data to divide the multiplexed data into a bit stream of video data and a bit stream of audio data, supplies the encoded video data to the video signal processor ex455 via the synchronous bus ex470, and supplies the encoded audio data to the audio signal processor ex454 via the synchronous bus ex470. The video signal processor ex455 decodes the video signal using a motion picture decoding method corresponding to the motion picture encoding method described in the above-described embodiments, and a video or still image included in a linked motion picture file is displayed on the display ex458 via the display controller ex459. The audio signal processor ex454 decodes the audio signal, and outputs the audio from the audio output unit ex457. Since real-time streaming is becoming more and more popular, in some cases the reproduction of audio can be socially inappropriate according to the environment of the user. Thereby, as an initial value, a configuration in which only the video data is reproduced and the audio signal is not reproduced is preferable; the audio can be synchronized and reproduced only when an input is received (for example, when the user clicks the video data).

[0382] Although a smartphone ex115 is used in the above example, other three implementations are conceivable: a transceiver terminal including both an encoder and a decoder; a transmitter terminal including only an encoder; a receiver terminal including only a decoder. In the description of the digital broadcasting system, an example is given in which multiplexed data obtained because video data and audio data are multiplexed is received or transmitted. However, the multiplexed data can be video data multiplexed with data other than audio data (for example, text data related to video). Furthermore, video data itself can be received or transmitted instead of multiplexed data.

[0383] Although the main controller ex460 including a CPU is described as controlling the encoding or decoding process, various terminals generally include a GPU. Thereby, a configuration in which a memory shared via a CPU and a GPU or a memory including addresses managed to allow general use of the CPU and the GPU is acceptable, by immediately processing a large area by utilizing the performance capability of the GPU. This can shorten the encoding time, maintain the real-time property of the stream, and reduce the delay. In particular, processing related to motion estimation, deblocking filtering, sample adaptive offset (SAO), and transform / quantization can be efficiently carried out all at once in units of pictures, for example, by the GPU instead of the CPU.

Claims

1. An image encoder, comprising: circuitry; and a memory coupled to the circuitry; wherein the circuitry in operation performs the following: selecting, for a first partition having a non-rectangular shape in a picture block, a first motion vector from a set of motion vector candidates; selecting, for a second partition in the picture block that overlaps the first partition, a second motion vector from the set of motion vector candidates; encoding the first partition using the first motion vector; and encoding the second partition using the second motion vector, the circuitry selecting only a uni-prediction motion vector from the set of motion vector candidates.

2. An image decoder, comprising: circuitry; and a memory coupled to the circuitry; wherein the circuitry in operation performs the following: selecting, for a first partition having a non-rectangular shape in a picture block, a first motion vector from a set of motion vector candidates; selecting, for a second partition in the picture block that overlaps the first partition, a second motion vector from the set of motion vector candidates; decoding the first partition using the first motion vector; and decoding the second partition using the second motion vector; the circuitry selecting only a uni-prediction motion vector from the set of motion vector candidates.

3. A bitstream generation apparatus, comprising: circuitry; and a memory coupled to the circuitry; wherein the circuitry in operation performs the following: generating information that causes a decoder to perform a partitioning process in accordance therewith, and including the information in a bitstream, in the partitioning process: a first motion vector for a first partition having a non-rectangular shape in a picture block is selected from a set of motion vector candidates; a second motion vector for a second partition in the picture block that overlaps the first partition is selected from the set of motion vector candidates; the first partition is decoded using the first motion vector; and the second partition is decoded using the second motion vector; only a uni-prediction motion vector is selected from the set of motion vector candidates. ​

Images