Image encoding method, image decoding method, and bitstream generation apparatus

Abstract

A picture coding method, a picture decoding method, and a non-transitory computer readable medium are provided. The picture coding method includes obtaining a picture block from a coding tree unit (CTU), selecting, for a first partition having a non-rectangular shape in the picture block, a first motion vector from a set of motion vector candidates, selecting, for a second partition in the picture block, a second motion vector from the set of motion vector candidates, coding the first partition using the first motion vector, and coding the second partition using the second motion vector, only a single prediction motion vector being selected from the set of motion vector candidates.

Description

[0001] This application is a divisional application of the patent application No. 201880054452.4 (International Application No. PCT / JP2018 / 030059) filed on August 10, 2018, with the title of “Image encoder, image decoder, image encoding method, and image decoding method”. TECHNICAL FIELD

[0002] The present disclosure relates to video coding, and in particular, to video coding and decoding systems, components, and methods for performing inter prediction function based on reference frames to construct a current block or intra prediction function based on a coded / decoded reference block in a current frame to construct the current block. BACKGROUND

[0003] As video coding technology has evolved from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec), there has always been a need for improvement and optimization of video coding technology to handle the ever-increasing amount of digital video data in various applications. The present disclosure relates to further advancement, improvement, and optimization in video coding, in particular, in connection with inter prediction function or intra prediction function that partitions an image block into a plurality of partitions including at least a first partition having a non-rectangular shape (e.g., a triangle) and a second partition. SUMMARY

[0004] According to one aspect, an image encoding method is provided, comprising: obtaining an image block from a coding tree unit (CTU); selecting, for a first partition having a non-rectangular shape in the image block, a first motion vector from a set of motion vector candidates; selecting, for a second partition in the image block, a second motion vector from the set of motion vector candidates; encoding the first partition using the first motion vector; and encoding the second partition using the second motion vector, only uni-prediction motion vectors being selected from the set of motion vector candidates.

[0005] Further, an image decoding method is provided, comprising: obtaining an image block from a coding tree unit (CTU); selecting, for a first partition having a non-rectangular shape in the image block, a first motion vector from a set of motion vector candidates; selecting, for a second partition in the image block, a second motion vector from the set of motion vector candidates; decoding the first partition using the first motion vector; and decoding the second partition using the second motion vector; only uni-prediction motion vectors being selected from the set of motion vector candidates.

[0006] Further, a non-transitory computer readable medium storing a bitstream is provided, the bitstream including information causing a decoder to perform a partitioning process in which: a first motion vector for a first partition having a non-rectangular shape in a picture block is selected from a set of motion vector candidates; a second motion vector for a second partition in the picture block is selected from the set of motion vector candidates; the first partition is decoded using the first motion vector; and the second partition is decoded using the second motion vector; only uni-prediction motion vectors are selected from the set of motion vector candidates.

[0007] Further, an image encoder is provided, including circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs the following operations: partitions an image block into a second partition and a first partition having a non-rectangular shape; predicts a first motion vector for the first partition and a second motion vector for the second partition; and encodes the first partition using the first motion vector and encodes the second partition using the second motion vector.

[0008] Some implementations of embodiments of the present disclosure can improve coding efficiency, can simply be a coding / decoding process, can speed up the speed of the coding / decoding process, can efficiently select suitable components / operations used in coding and decoding, such as suitable filters, block sizes, motion vectors, reference pictures, reference blocks, etc.

[0009] Other benefits and advantages of the disclosed embodiments will become apparent from the description and drawings. Benefits and / or advantages can be separately provided by various embodiments and features of the description and drawings, which need not all be provided in order to obtain one or more of such benefits and / or advantages.

[0010] It should be noted that general or specific embodiments can be implemented as a system, a method, an integrated circuit, a computer program, a storage medium, or any selective combination thereof. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a block diagram showing a functional configuration of an encoder according to an embodiment.

[0012] Figure 2 One example of block partitioning is shown.

[0013] Figure 3 is a table indicating transform basis functions of various transform types.

[0014] Figure 4A One example of a filter shape used in ALF (Adaptive Loop Filter) is shown.

[0015] Figure 4BAnother example of filter shapes used in ALF is shown.

[0016] Figure 4C Another example of filter shapes used in ALF is shown.

[0017] Figure 5A 67 intra prediction modes used in an example of intra prediction are shown.

[0018] Figure 5B is a flowchart showing an example of a prediction image correction process performed in OBMC (Overlapped Block Motion Compensation) processing.

[0019] Figure 5C is a conceptual diagram showing an example of a prediction image correction process performed in OBMC processing.

[0020] Figure 5D is a flowchart showing an example of FRUC (Frame Rate Up Conversion) processing.

[0021] Figure 6 An example of pattern matching between two blocks along a motion trajectory (bi-directional matching) is shown.

[0022] Figure 7 An example of pattern matching between a template in a current picture and a block in a reference picture (template matching) is shown.

[0023] Figure 8 A model assuming uniform linear motion is shown.

[0024] Figure 9A An example of deriving a motion vector for each sub-block based on motion vectors of neighboring blocks is shown.

[0025] Figure 9B An example of a process for deriving a motion vector in merge mode is shown.

[0026] Figure 9C is a conceptual diagram showing an example of DMVR (Dynamic Motion Vector Refresh) processing.

[0027] Figure 9D An example of a prediction image generation method using a brightness correction process performed by LIC (Local Illumination Compensation) processing is shown.

[0028] Figure 10 is a block diagram showing a functional configuration of a decoder according to an embodiment.

[0029] Figure 11is a flowchart illustrating the overall processing flow of segmenting an image block into a plurality of partitions including at least a first partition having a non-rectangular shape (e.g., a triangle) and a second partition and performing further processing according to one embodiment.

[0030] Figure 12 Two exemplary methods of segmenting an image block into a second partition (also having a non-rectangular shape in the illustrated example) and a first partition having a non-rectangular shape (e.g., a triangle) are shown.

[0031] Figure 13 One example of a boundary smoothing process is shown that includes weighting a first value of a boundary pixel based on a prediction of the first partition and a second value of the boundary pixel based on a prediction of the second partition.

[0032] Figure 14 Three further examples of a boundary smoothing process are shown that involve weighting a first value of a boundary pixel based on a prediction of the first partition and weighting a second value of the boundary pixel based on a prediction of the second partition.

[0033] Figure 15 is a table of sampling parameters ("first index values") and information sets encoded by the parameters, respectively.

[0034] Figure 16 is a table illustrating binarization of a parameter (index value).

[0035] Figure 17 is a flowchart illustrating a process of segmenting an image block into a plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0036] Figure 18 An example of segmenting an image block into a plurality of partitions including a second partition and a first partition having a non-rectangular shape (a triangle in the illustrated example) is shown.

[0037] Figure 19 A further example of segmenting an image block into a plurality of partitions including a second partition and a first partition having a non-rectangular shape, which is a polygon having at least five sides and corners in the illustrated example, is shown.

[0038] Figure 20 is a flowchart illustrating a boundary smoothing process that includes weighting a first value of a boundary pixel based on a prediction of the first partition and weighting a second value of the boundary pixel based on a prediction of the second partition.

[0039] Figure 21AAn example of a boundary smoothing process is shown, in which a boundary pixel whose first value is to be weighted is predicted based on a first partition and whose second value is to be weighted is predicted based on a second partition.

[0040] Figure 21B An example of a boundary smoothing process is shown, in which a boundary pixel whose first value is to be weighted is predicted based on a first partition and whose second value is to be weighted is predicted based on a second partition.

[0041] Figure 21C An example of a boundary smoothing process is shown, in which a boundary pixel whose first value is to be weighted is predicted based on a first partition and whose second value is to be weighted is predicted based on a second partition.

[0042] Figure 21D An example of a boundary smoothing process is shown, in which a boundary pixel whose first value is to be weighted is predicted based on a first partition and whose second value is to be weighted is predicted based on a second partition.

[0043] Figure 22 is a flowchart showing a method performed on an encoder side of partitioning an image block into a plurality of partitions based on a partition parameter indicating the partitioning and writing one or more parameters including the partition parameter in an entropy coding into a bitstream, the plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0044] Figure 23 is a flowchart showing a method performed on a decoder side of parsing one or more parameters from a bitstream, the parameters including a method of partitioning an image block into a plurality of partitions based on a partition parameter and decoding a first partition and a second partition, the one or more parameters including a partition parameter indicating the partitioning of the image block into the plurality of partitions, the plurality of partitions including the second partition and the first partition having a non-rectangular shape.

[0045] Figure 24 is a table of sampling partition parameters ("first index values") and sets of information that can be jointly encoded by the partition parameters, respectively, the sampling partition parameters indicating partitioning of an image block into a plurality of partitions, respectively, the plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0046] Figure 25 is a table of sampling combinations of a first parameter and a second parameter, wherein one of the first parameter and the second parameter is a partition parameter indicating partitioning of an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape.

[0047] Figure 26 An overall configuration of a content providing system for implementing a content distribution service is shown.

[0048] Figure 27 An example of an encoding structure in scalable coding is shown.

[0049] Figure 28 An example of an encoding structure in scalable coding is shown.

[0050] Figure 29 An example of a display screen of a web page is shown.

[0051] Figure 30 An example of a display screen of a web page is shown.

[0052] Figure 31 An example of a smartphone is shown.

[0053] Figure 32 is a block diagram showing an example of a configuration of a smartphone. DETAILED DESCRIPTION

[0054] According to an aspect, there is provided an image encoder comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs the following operations: partitioning an image block into a plurality of partitions, the plurality of partitions including a first partition having a non-rectangular shape and a second partition; predicting a first motion vector for the first partition and a second motion vector for the second partition; and encoding the first partition using the first motion vector and the second partition using the second motion vector.

[0055] According to another aspect, the second partition has a non-rectangular shape. According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0056] According to another aspect, the predicting includes selecting the first motion vector from a first set of motion vector candidates and selecting the second motion vector from a second set of motion vector candidates. For example, the first set of motion vector candidates can include motion vectors of partitions adjacent to the first partition, and the second set of motion vector candidates can include motion vectors of partitions adjacent to the second partition. The partitions adjacent to the first partition and the partitions adjacent to the second partition can be outside of the image block from which the first partition and the second partition are partitioned. The adjacent partitions can be one or both of spatially adjacent partitions and temporally adjacent partitions. The first set of motion vector candidates can be the same as or different from the second set of motion vector candidates.

[0057] According to another aspect, the predicting includes: selecting a first motion vector candidate from a first set of motion vector candidates and deriving the first motion vector by adding a first motion vector difference to the first motion vector candidate, and selecting a second motion vector candidate from a second set of motion vector candidates and deriving the second motion vector by adding a second motion vector difference to the second motion vector candidate.

[0058] According to another aspect, there is provided an image encoder comprising: a partitioner that, in operation, receives an original picture and partitions it into blocks; and an adder that, in operation, receives the blocks from the partitioner and predictions from a prediction controller and subtracts each prediction from its corresponding block to output a residual; a transformer that, in operation, performs a transform on the residuals output from the adder to output transform coefficients; a quantizer that, in operation, quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder that, in operation, encodes the quantized transform coefficients to generate a bitstream; and a prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, wherein the inter-predictor, in operation, generates a prediction of a current block based on a reference block in an encoded reference picture, and the intra-predictor, in operation, generates a prediction of the current block based on an encoded reference block in a current picture. The prediction controller, in operation, partitions the block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; predicts a first motion vector for the first partition and a second motion vector for the second partition; and encodes the first partition using the first motion vector and the second partition using the second motion vector.

[0059] According to another aspect, there is provided an image encoding method generally comprising three steps: partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; and predicting a first motion vector for the first partition and a second motion vector for the second partition; encoding the first partition using the first motion vector and the second partition using the second motion vector.

[0060] According to another aspect, there is provided an image decoder comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs: partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; and predicting a first motion vector for the first partition and a second motion vector for the second partition; decoding the first partition using the first motion vector and the second partition using the second motion vector.

[0061] According to another aspect, the second partition has a non-rectangular shape. According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0062] According to another aspect, an image decoder is provided, comprising: an entropy decoder that, in operation, receives and decodes an encoded bitstream to obtain quantized transform coefficients; and an inverse quantizer and transformer that, in operation, inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual; an adder that, in operation, adds the residual output from the inverse quantizer and transformer to a prediction output from a prediction controller to reconstruct a block; and the prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, wherein the inter-predictor, in operation, generates a prediction for a current block based on a reference block in a decoded reference picture and the intra-predictor, in operation, generates a prediction for the current block based on a decoded reference block in a current picture. The prediction controller, in operation, partitions an image block into a plurality of partitions including a first partition having a non-rectangular shape and a second partition; predicts a first motion vector for the first partition and a second motion vector for the second partition; decodes the first partition using the first motion vector and the second partition using the second motion vector.

[0063] According to another aspect, an image decoding method is provided, generally comprising three steps: partitioning an image block into a plurality of partitions including a first partition having a non-rectangular shape and a second partition; and predicting a first motion vector for the first partition and a second motion vector for the second partition; decoding the first partition using the first motion vector and the second partition using the second motion vector.

[0064] According to one aspect, an image encoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The boundary smoothing operation comprises: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary; and using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary; weighting the first values and the second values; and encoding the first partition using the weighted first values and the weighted second values.

[0065] According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon having at least five sides and corners. According to yet another aspect, the second partition has a non-rectangular shape.

[0066] According to another aspect, at least one of the first predicting and the second predicting is an inter-prediction process that predicts the first values and the second values based on a reference partition in an encoded reference picture. The inter-prediction process can predict the first values of pixels of the first partition including the set of pixels and can predict the second values of only the set of pixels of the first partition.

[0067] According to another aspect, at least one of the first prediction and the second prediction is an intra prediction process that predicts the first value and the second value based on an encoded reference partition in the current picture.

[0068] According to another aspect, a prediction method used in the first prediction is different from a prediction method used in the second prediction.

[0069] According to another aspect, the number of the set of pixels in each row or each column for which the first value and the second value are predicted is an integer. For example, when the number of the set of pixels in each row or each column is four, weights 1 / 8, 1 / 4, 3 / 4, and 7 / 8 can be applied to the first values of the four pixels in the set, respectively, and weights 7 / 8, 3 / 4, 1 / 4, and 1 / 8 can be applied to the second values of the four pixels in the set, respectively. As another example, when the number of the set of pixels in each row or each column is two, weights 1 / 3 and 2 / 3 can be applied to the first values of the two pixels in the set, respectively, and weights 2 / 3 and 1 / 3 can be applied to the second values of the two pixels in the set, respectively.

[0070] According to another aspect, the weights can be integer values or can be fractional values.

[0071] According to another aspect, there is provided an image encoder comprising: a partitioner that, in operation, receives an original picture and partitions it into blocks; and an adder that, in operation, receives the blocks from the partitioner and predictions from a prediction controller and subtracts each prediction from its corresponding block to output a residual; a transformer that, in operation, performs a transform on the residuals output from the adder to output transform coefficients; a quantizer that, in operation, quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder that, in operation, encodes the quantized transform coefficients to generate a bitstream; and a prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, wherein the inter-predictor, in operation, generates a prediction for a current block based on a reference block in an encoded reference picture, and the intra-predictor, in operation, generates a prediction for the current block based on an encoded reference block in the current picture. The prediction controller, in operation, performs a boundary smoothing operation along a boundary between a first partition having a non-rectangular shape and a second partition partitioned out of the image block. The boundary smoothing operation comprises: using information of the first partition, making a first prediction of first values of a set of pixels of the first partition along the boundary; and using information of the second partition, making a second prediction of second values of the set of pixels of the first partition along the boundary; weighting the first values and the second values; and encoding the first partition using the weighted first values and the weighted second values.

[0072] According to another aspect, there is provided an image encoding method that performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The method generally includes four steps: first predicting first values of a set of pixels of the first partition along the boundary using information of the first partition; and second predicting second values of the set of pixels of the first partition along the boundary using information of the second partition; weighting the first values and the second values; and encoding the first partition using the weighted first values and the weighted second values.

[0073] According to another aspect, there is provided an image decoder that includes circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The boundary smoothing operation includes: first predicting first values of a set of pixels of the first partition along the boundary using information of the first partition; and second predicting second values of the set of pixels of the first partition along the boundary using information of the second partition; weighting the first values and the second values; and decoding the first partition using the weighted first values and the weighted second values.

[0074] According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon having at least five sides and corners. According to another aspect, the second partition has a non-rectangular shape.

[0075] According to another aspect, at least one of the first predicting and the second predicting is an inter prediction process that predicts the first values and the second values based on a reference partition in an encoded reference picture. The inter prediction process can predict the first values of pixels of the first partition including the set of pixels, and can predict the second values of only the set of pixels of the first partition.

[0076] According to another aspect, at least one of the first predicting and the second predicting is an intra prediction process that predicts the first values and the second values based on an encoded reference partition in a current picture.

[0077] According to another aspect, an image decoder is provided, comprising: an entropy decoder that in operation receives and decodes an encoded bitstream to obtain quantized transform coefficients; and an inverse quantizer and transformer that in operation inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual; an adder that in operation adds the residual output from the inverse quantizer and transformer to a prediction output from a prediction controller to reconstruct a block; and the prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, wherein the inter-predictor in operation generates a prediction for a current block based on a reference block in a decoded reference picture and the intra-predictor in operation generates a prediction for the current block based on a decoded reference block in a current picture. The prediction controller in operation performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The boundary smoothing operation comprises: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary; and using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary; weighting the first values and the second values; and decoding the first partition using the weighted first values and the weighted second values.

[0078] According to another aspect, an image decoding method is provided that performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape. The method generally comprises four steps: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary; and using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary; weighting the first values and the second values; and decoding the first partition using the weighted first values and the weighted second values.

[0079] According to an aspect, an image encoder is provided that comprises circuitry and a memory coupled to the circuitry. The circuitry in operation performs a partition syntax operation that comprises: partitioning an image block into a plurality of partitions based on a partition parameter indicative of the partitioning, the plurality of partitions comprising a second partition and a first partition having a non-rectangular shape; encoding the first partition and the second partition; and writing one or more parameters including the partition parameter to a bitstream.

[0080] According to another aspect, the partition parameter indicates that the first partition has a triangular shape.

[0081] According to another aspect, the partition parameter indicates that the second partition has a non-rectangular shape.

[0082] According to another aspect, the partition parameter indicates that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0083] According to another aspect, partition parameters jointly encode a split direction for splitting an image block into a plurality of partitions. For example, the split direction can include: from a top-left corner to a bottom-right corner of the image block, and from a top-right corner to a bottom-left corner of the image block. The partition parameters can jointly encode at least a first motion vector of a first partition.

[0084] According to another aspect, one or more parameters other than the partition parameters encode a split direction applied to split an image block into a plurality of partitions. The parameters encoding the split direction can jointly encode at least a first motion vector of a first partition.

[0085] According to another aspect, the partition parameters can jointly encode at least a first motion vector of a first partition. The partition parameters can jointly encode a second motion vector of a second partition.

[0086] According to another aspect, one or more parameters other than the partition parameters can encode at least a first motion vector of a first partition.

[0087] According to another aspect, one or more parameters are binarized according to a binarization scheme selected according to a value of at least one of the one or more parameters.

[0088] According to another aspect, an image encoder is provided that includes a splitter that in operation receives and splits an original picture into blocks, and an adder that in operation receives the blocks from the splitter and predictions from a prediction controller, and subtracts each prediction from its corresponding block to output a residual, a transformer that in operation performs a transform on the residuals output from the adder to output transform coefficients, a quantizer that in operation quantizes the transform coefficients to generate quantized transform coefficients, an entropy encoder that in operation encodes the quantized transform coefficients to generate a bitstream, and a prediction controller coupled to an inter-predictor, an intra-predictor, and a memory, where the inter-predictor in operation generates a prediction of a current block based on a reference block in an encoded reference picture, and the intra-predictor in operation generates a prediction of the current block based on an encoded reference block in a current picture. The prediction controller in operation splits an image block into a plurality of partitions based on partition parameters indicative of the split, the plurality of partitions including a second partition and a first partition having a non-rectangular shape, and encodes the first and second partitions. The entropy encoder in operation writes one or more parameters including the partition parameters to the bitstream.

[0089] According to another aspect, there is provided an image encoding method comprising a partition syntax operation. The method generally comprises three steps: partitioning an image block into a plurality of partitions based on a partition parameter indicative of the partitioning, the plurality of partitions comprising a first partition having a non-rectangular shape and a second partition; and encoding the first and second partitions; and writing one or more parameters including the partition parameter into a bitstream.

[0090] According to another aspect, there is provided an image decoder comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a partition syntax operation comprising: parsing one or more parameters from a bitstream, wherein the one or more parameters include a partition parameter indicative of partitioning an image block into a plurality of partitions, the plurality of partitions comprising a first partition having a non-rectangular shape and a second partition; partitioning the image block into the plurality of partitions based on the partition parameter; and decoding the first and second partitions.

[0091] According to another aspect, the partition parameter indicates that the first partition has a triangular shape.

[0092] According to another aspect, the partition parameter indicates that the second partition has a non-rectangular shape.

[0093] According to another aspect, the partition parameter indicates that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0094] According to another aspect, the partition parameter jointly encodes a partitioning direction used to partition the image block into the plurality of partitions. For example, the partitioning direction comprises: from a top-left corner of the image block to a bottom-right corner thereof, and from a top-right corner of the image block to a bottom-left corner thereof. The partition parameter can jointly encode at least a first motion vector of the first partition.

[0095] According to another aspect, one or more parameters other than the partition parameter encode a partitioning direction applied to partition the image block into the plurality of partitions. The parameter encoding the partitioning direction can jointly encode at least a first motion vector of the first partition.

[0096] According to another aspect, the partition parameter can jointly encode at least a first motion vector of the first partition. The partition parameter can jointly encode a second motion vector of the second partition.

[0097] According to another aspect, one or more parameters other than the partition parameter can encode at least a first motion vector of the first partition.

[0098] According to another aspect, the one or more parameters are binarized according to a binarization scheme selected according to a value of at least one of the one or more parameters.

[0099] According to another aspect, there is provided an image decoder comprising: an entropy decoder that in operation receives and decodes an encoded bitstream to obtain quantized transform coefficients; and an inverse quantizer and transformer that in operation inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual; an adder that in operation adds the residual output from the inverse quantizer and transformer to a prediction output from a prediction controller to reconstruct a block; and the prediction controller coupled to an inter-predictor, an intra-predictor and a memory, wherein the inter-predictor in operation generates a prediction for a current block based on a reference block in a decoded reference picture and the intra-predictor in operation generates a prediction for the current block based on a decoded reference block in a current picture. The entropy decoder in operation: parses one or more parameters from the bitstream, wherein the one or more parameters comprise partition parameters indicative of partitioning an image block into a plurality of partitions, the plurality of partitions comprising a second partition and a first partition having a non-rectangular shape; partitions the image block into the plurality of partitions based on the partition parameters; and decodes the first partition and the second partition.

[0100] According to another aspect, there is provided an image decoding method comprising a partition syntax operation. The method generally comprises three steps: parsing one or more parameters from a bitstream, wherein the one or more parameters comprise partition parameters indicative of partitioning an image block into a plurality of partitions, the plurality of partitions comprising a second partition and a first partition having a non-rectangular shape; partitioning the image block into the plurality of partitions based on the partition parameters; and decoding the first partition and the second partition.

[0101] In the drawings, like reference numerals refer to like elements throughout. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale.

[0102] Hereinafter, embodiments will be described with reference to the accompanying drawings. Note that each of the following described embodiments shows a general or specific example. The numerical values, shapes, materials, components, arrangement and connection of components, steps, relationships and order of steps, and the like indicated in the following embodiments are merely examples and are not intended to limit the scope of the claims. Therefore, components disclosed in the following embodiments that are not enumerated in any independent claim defining the broadest aspects of the application can be understood as optional components.

[0103] Embodiments of an encoder and a decoder will be described below. The embodiments are examples of an encoder and a decoder to which the processes and / or configurations presented in the description of aspects of the present disclosure are applicable. The processes and / or configurations can also be implemented in an encoder and a decoder different from the encoder and the decoder according to the embodiments. For example, with respect to the processes and / or configurations applicable to the embodiments, any one of the following can be implemented:

[0104] (1) Any component of an encoder or decoder according to the embodiments presented in the description of aspects of the disclosure can be replaced or combined with another component presented anywhere in the description of aspects of the disclosure.

[0105] (2) In an encoder or decoder according to the embodiments, any change can be made to the functions or processes performed by one or more components of the encoder or decoder, such as addition, replacement, removal, etc. of functions or processes. For example, any function or process can be replaced or combined with another function or process appearing anywhere in the description of aspects of the disclosure.

[0106] (3) In a method implemented by an encoder or decoder according to the embodiments, any change can be made, such as addition, replacement, and removal of one or more processes included in the method. For example, any process in the method can be replaced or combined with another process appearing anywhere in the description of aspects of the disclosure.

[0107] (4) One or more components included in an encoder or decoder according to the embodiments can be combined with components presented anywhere in the description of aspects of the disclosure, can be combined with components including one or more functions presented anywhere in the description of aspects of the disclosure, and can be combined with components implementing one or more processes implemented by components presented in the description of aspects of the disclosure.

[0108] (5) Components including one or more functions of an encoder or decoder according to the embodiments, or components implementing one or more processes of an encoder or decoder according to the embodiments, can be combined or replaced with components presented anywhere in the description of aspects of the disclosure, with components including one or more functions presented anywhere in the description of aspects of the disclosure, or with components implementing one or more processes presented anywhere in the description of aspects of the disclosure.

[0109] (6) In a method implemented by an encoder or decoder according to the embodiments, any process included in the method can be replaced or combined with a process presented anywhere in the description of aspects of the disclosure or with any corresponding or equivalent process.

[0110] (7) One or more processes included in a method implemented by an encoder or decoder according to the embodiments can be combined with processes presented anywhere in the description of aspects of the disclosure.

[0111] (8) The implementation of the processes and / or configurations presented in the description of aspects of the disclosure is not limited to an encoder or a decoder according to the embodiments. For example, the processes and / or configurations can be implemented in a device for a different purpose from the motion picture encoder or the motion picture decoder disclosed in the embodiments.

[0112] (Encoder)

[0113] First, an encoder according to the embodiments will be described. Figure 1 is a block diagram showing a functional configuration of an encoder 100 according to the embodiments. The encoder 100 is a motion picture encoder that encodes a motion picture on a block-by-block basis.

[0114] As shown in Figure 1 , the encoder 100 is a device that encodes a picture on a block-by-block basis, and includes a partitioner 102, a subtracter 104, a transformer 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse transformer 114, an adder 116, a block memory 118, a loop filter 120, a frame memory 122, an intra predictor 124, an inter predictor 126, and a prediction controller 128.

[0115] The encoder 100 is implemented as, for example, a general-purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor functions as the partitioner 102, the subtracter 104, the transformer 106, the quantizer 108, the entropy encoder 110, the inverse quantizer 112, the inverse transformer 114, the adder 116, the loop filter 120, the intra predictor 124, the inter predictor 126, and the prediction controller 128. Alternatively, the encoder 100 can be implemented as one or a plurality of dedicated electronic circuits corresponding to the partitioner 102, the subtracter 104, the transformer 106, the quantizer 108, the entropy encoder 110, the inverse quantizer 112, the inverse transformer 114, the adder 116, the loop filter 120, the intra predictor 124, the inter predictor 126, and the prediction controller 128.

[0116] Hereinafter, each component included in the encoder 100 will be described.

[0117] (Partitioner)

[0118] The partitioner 102 partitions each picture included in the inputted moving picture into blocks, and outputs each block to the subtracter 104. For example, the partitioner 102 first partitions a picture into blocks of a fixed size (e.g., 128x128). The fixed-size blocks can also be referred to as coding tree units (CTUs). Then, the partitioner 102 partitions each fixed-size block into blocks of variable sizes (e.g., 64x64 or smaller) based on, for example, recursive quad-tree and / or binary-tree block partitioning. The variable-size blocks can also be referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). In various implementations, there can be no need to distinguish between CUs, PUs, and TUs; all or some of the blocks in a picture can be processed as CUs, PUs, or TUs.

[0119] Figure 2 One example of block partitioning according to an embodiment is shown. In Figure 2 In the example, solid lines represent block boundaries of blocks partitioned by quad-tree block partitioning, and dashed lines represent block boundaries of blocks partitioned by binary-tree block partitioning.

[0120] Here, the block 10 is a square 128x128 pixel block (128x128 block). The 128x128 block 10 is first partitioned into four square 64x64 blocks (quad-tree block partitioning).

[0121] The upper-left 64x64 block is further vertically partitioned into two rectangular 32x64 blocks, and the left 32x64 block is further vertically partitioned into two rectangular 16x64 blocks (binary-tree block partitioning). As a result, the upper-left 64x64 block is partitioned into two 16x64 blocks 11 and 12 and one 32x64 block 13.

[0122] The upper-right 64x64 block is horizontally partitioned into two rectangular 64x32 blocks 14 and 15 (binary-tree block partitioning).

[0123] The lower-left 64x64 block is first partitioned into four square 32x32 blocks (quad-tree block partitioning). Among the four 32x32 blocks, the upper-left block and the lower-right block are further partitioned. The upper-left 32x32 block is vertically partitioned into two rectangular 16x32 blocks, and the right 16x32 block is further horizontally partitioned into two 16x16 blocks (binary-tree block partitioning). The lower-right 32x32 block is horizontally partitioned into two 32x16 blocks (binary-tree block partitioning). As a result, the lower-left 64x64 block is partitioned into a 16x32 block 16, two 16x16 blocks 17 and 18, two 32x32 blocks 19 and 20, and two 32x16 blocks 21 and 22.

[0124] The lower-right 64x64 block 23 is not partitioned.

[0125] As described above, inFigure 2 In this case, the block 10 is divided into 13 blocks 11 to 23 of variable sizes based on recursive quadtree and binary tree block splitting. This type of splitting is also referred to as quadtree plus binary tree (QTBT) splitting.

[0126] Although in Figure 2 In this case, the block 10 is divided into 13 blocks 11 to 23 of variable sizes based on recursive quadtree and binary tree block splitting. This type of splitting is also referred to as quadtree plus binary tree (QTBT) splitting.

[0127] (subtracter)

[0128] The subtracter 104 subtracts a prediction signal (prediction sample, input from a prediction controller 128, which will be described later) from an original signal (original sample) of each block divided and input by the divider 102. In other words, the subtracter 104 calculates a prediction error (also referred to as a "residual") of a block to be encoded (also referred to as a "current block" hereinafter). The subtracter 104 then outputs the calculated prediction error (residual) to the transformer 106.

[0129] The original signal is a signal input to the encoder 100, and is a signal (e.g., a luminance signal and two chrominance signals) representing an image of each picture included in a moving picture. Hereinafter, the signal representing the image is also referred to as a sample.

[0130] (transformer)

[0131] The transformer 106 transforms the spatial domain prediction error into a frequency domain transform coefficient, and outputs the transform coefficient to the quantizer 108. More specifically, the transformer 106 applies, for example, a predefined discrete cosine transform (DCT) or a discrete sine transform (DST) to the spatial domain prediction error.

[0132] Note that the transformer 106 can adaptively select a transform type from a plurality of transform types, and transform the prediction error into a transform coefficient by using a transform basis function corresponding to the selected transform type. This transform is also referred to as explicit multi-kernel transform (EMT) or adaptive multi transform (AMT).

[0133] The transform types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 is a graph representing a transform basis function of each transform type. In Figure 3 In this case, N represents the number of input pixels. For example, the selection of the transform type from a plurality of transform types can depend on a prediction type (intra prediction and inter prediction) and an intra prediction mode.

[0134] The information indicating whether to apply EMT or AMT (e.g., referred to as an EMT flag or an AMT flag) and the information indicating the selected transform type are typically signaled at the CU level. Note that the signaling of such information need not be performed at the CU level, but can be performed at another level (e.g., at the bit sequence level, the picture level, the slice level, the tile level, or the CTU level).

[0135] Further, the transformer 106 can apply a secondary transform to the transform coefficients (transform result). Such a secondary transform is also referred to as an adaptive secondary transform (AST) or a non-separable secondary transform (NSST). For example, the transformer 106 applies a secondary transform to each sub-block (e.g., each 4x4 sub-block) included in a block of transform coefficients corresponding to intra prediction error. The information indicating whether to apply NSST and the information related to the transform matrix used in NSST are typically signaled at the CU level. Note that the signaling of such information need not be performed at the CU level, but can be performed at another level (e.g., at the bit sequence level, the picture level, the slice level, the tile level, or the CTU level).

[0136] A separate transform or a non-separable transform can be applied in the transformer 106. The separate transform is a method in which a transform is performed multiple times by separately performing a transform for each direction according to the number of dimensions of the input. The non-separable transform is a method of performing a collective transform in which two or more dimensions in a multi-dimensional input are collectively regarded as a single dimension.

[0137] In one example of the non-separable transform, when the input is a 4x4 block, the 4x4 block is regarded as a single array including 16 components, and the transform applies a 16x16 transform matrix to the array.

[0138] In another example of the non-separable transform, after a 4x4 block of the input is regarded as a single array including 16 components, a transform performing a plurality of Givens rotations (e.g., a Hypercube-Givens transform) can be applied to the array.

[0139] (Quantizer)

[0140] The quantizer 108 quantizes the transform coefficients output from the transformer 106. More specifically, the quantizer 108 scans the transform coefficients of the current block in a predetermined scan order, and quantizes the scanned transform coefficients based on a quantization parameter (QP) corresponding to the transform coefficients. Then, the quantizer 108 outputs the quantized transform coefficients (hereinafter referred to as quantized coefficients) of the current block to the entropy encoder 110 and the inverse quantizer 112.

[0141] The predetermined scan order is an order for quantizing / dequantizing the transform coefficients. For example, the predetermined scan order is defined as an ascending order of frequency (from low frequency to high frequency) or a descending order of frequency (from high frequency to low frequency).

[0142] A quantization parameter (QP) is a parameter that defines a quantization step (quantization width). For example, if the value of the quantization parameter increases, the quantization step also increases. In other words, if the value of the quantization parameter increases, the quantization error increases.

[0143] (Entropy encoder)

[0144] The entropy encoder 110 generates an encoded signal (encoded bitstream) based on the quantized coefficients input from the quantizer 108. More specifically, for example, the entropy encoder 110 binarizes the quantized coefficients and arithmetically encodes the binarized signal to output a compressed bitstream or sequence.

[0145] (Dequantizer)

[0146] The dequantizer 112 dequantizes the quantized coefficients input from the quantizer 108. More specifically, the dequantizer 112 dequantizes the quantized coefficients of the current block in the predetermined scan order. Then, the dequantizer 112 outputs the dequantized transform coefficients of the current block to the inverse transformer 114.

[0147] (Dequantizer)

[0148] The inverse transformer 114 recovers the prediction error (residual) by inverse-transforming the transform coefficients input from the dequantizer 112. More specifically, the inverse transformer 114 recovers the prediction error of the current block by applying an inverse transform corresponding to the transform applied to the transform coefficients by the transformer 106. The inverse transformer 114 then outputs the recovered prediction error to the adder 116.

[0149] Note that the recovered prediction error does not match the prediction error calculated by the subtractor 104 because information is typically lost in quantization. In other words, the recovered prediction error typically includes a quantization error.

[0150] (Adder)

[0151] The adder 116 reconstructs the current block by summing the prediction error input from the inverse transformer 114 and the prediction sample input from the prediction controller 128. The adder 116 then outputs the reconstructed block to the block memory 118 and the loop filter 120. The reconstructed block is also referred to as a locally decoded block.

[0152] (Block memory)

[0153] The block memory 118 is a storage device for storing blocks in a picture to be encoded (referred to as "current picture") that is referred to in, for example, intra prediction. More specifically, the block memory 118 stores the reconstructed blocks output from the adder 116.

[0154] (loop filter)

[0155] The loop filter 120 applies a loop filter to the blocks reconstructed by the adder 116, and outputs the filtered reconstructed blocks to the frame memory 122. The loop filter is a filter used in the encoding loop (in-loop filter), and includes, for example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF).

[0156] In the ALF, a least square error filter for removing compression artifacts is applied. For example, based on a direction and activity of local gradients, one filter is selected from a plurality of filters for each 2x2 sub-block in a current block, and is applied thereto.

[0157] More specifically, first, each sub-block (e.g., each 2x2 sub-block) is classified into one of a plurality of classes (e.g., 15 or 25 classes). The classification of the sub-block is based on gradient directionality and activity. For example, a classification index C (e.g., C = 5D + A) is derived based on a gradient directionality D (e.g., 0 to 2 or 0 to 4) and a gradient activity A (e.g., 0 to 4). Next, each sub-block is classified into one of the plurality of classes based on the classification index C.

[0158] The gradient directionality D is calculated, for example, by comparing gradients of a plurality of directions (e.g., a horizontal direction, a vertical direction, and two diagonal directions). Further, the gradient activity A is calculated, for example, by summing gradients of a plurality of directions and quantizing the sum.

[0159] Based on the result of such classification, a filter to be used for each sub-block is determined from a plurality of filters.

[0160] The filter shape used in the ALF is, for example, a circularly symmetric filter shape. Figure 4A 、 4B , and 4C show examples of filter shapes used in the ALF. Figure 4A shows a 5x5 diamond filter, Figure 4B shows a 7x7 diamond filter, and Figure 4C shows a 9x9 diamond filter. Information indicating the filter shape is generally signaled at a picture level. Note that the signaling of the information indicating the filter shape need not be performed at the picture level, but can be performed at another level (e.g., at a sequence level, a slice level, a tile level, a CTU level, or a CU level).

[0161] The enabling or disabling of ALF can be determined at a picture level or a CU level. For example, for luma, the decision whether to apply ALF can be made on a CU level, and for chroma, the decision whether to apply ALF can be made on a picture level. The information indicating whether ALF is enabled or disabled is typically signaled at a picture level or a CU level. Note that the signaling of the information indicating whether ALF is enabled does not need to be performed at a picture level or a CU level, but can be performed at another level (e.g., at a sequence level, a slice level, a tile level, or a CTU level).

[0162] The coefficient sets of the multiple optional filters (e.g., 15 or 25 filters) are typically signaled at a picture level. Note that the signaling of the coefficient sets does not need to be performed at a picture level, but can be performed at another level (e.g., at a sequence level, a slice level, a tile level, a CTU level, a CU level, or a sub-block level).

[0163] (frame memory)

[0164] The frame memory 122 is, for example, a storage device used to store reference pictures used in inter prediction, and is also referred to as a frame buffer. More specifically, the frame memory 122 stores the reconstructed blocks filtered by the loop filter 120.

[0165] (intra predictor)

[0166] The intra predictor 124 generates a prediction signal (intra prediction signal) (also referred to as an intra predicted signal) by performing intra prediction on the current block with reference to one or more blocks in the current picture stored in the block memory 118. More specifically, the intra predictor 124 generates the intra prediction signal by performing intra prediction with reference to samples (e.g., luma and / or chroma values) of one or more blocks neighboring the current block, and then outputs the intra prediction signal to the prediction controller 128.

[0167] For example, the intra predictor 124 performs intra prediction by using one of a plurality of predetermined intra prediction modes. The intra prediction modes typically include one or more non-directional prediction modes and a plurality of directional prediction modes.

[0168] The one or more non-directional prediction modes include, for example, a planar prediction mode and a DC prediction mode defined in the H.265 / HEVC standard.

[0169] The plurality of directional prediction modes includes, for example, 33 directional prediction modes defined in the H.265 / HEVC standard. Note that the plurality of directional prediction modes can include 32 directional prediction modes in addition to the 33 directional prediction modes (for a total of 65 directional prediction modes).

[0170] Figure 5A A total of 67 intra prediction modes (two non-directional prediction modes and 65 directional prediction modes) are shown to be used in intra prediction. The solid arrows represent the 33 directions defined in the H.265 / HEVC standard, while the dashed arrows represent the additional 32 directions (not shown in the Figure 5A two "non-directional" prediction modes.

[0171] In various implementations, a luma block can be referenced in chroma block intra prediction. That is, a chroma component of a current block can be predicted based on a luma component of the current block. Such intra prediction is also referred to as cross-component linear model (CCLM) prediction. A chroma block intra prediction mode that references a luma block (e.g., referred to as a CCLM mode) can be added as one of the chroma block intra prediction modes.

[0172] The intra predictor 124 can correct the intra-predicted pixel values based on horizontal / vertical reference pixel gradients. The intra prediction with such correction is also referred to as position-dependent intra prediction combination (PDPC). Information indicating whether to apply PDPC (e.g., referred to as a PDPC flag) is typically signaled at the CU level. Note that the signaling of this information need not be performed at the CU level, but can be performed at another level (e.g., at the sequence level, picture level, slice level, tile level, or CTU level).

[0173] (inter-predictor)

[0174] The inter-predictor 126 generates a prediction signal (inter-prediction) by inter-predicting a current block by referring to one or more blocks in a reference picture different from the current picture and stored in the frame memory 122. Inter-prediction is performed for each current block or each current sub-block (e.g., each 4x4 block) in the current block. For example, the inter-predictor 126 performs motion estimation in the reference picture for the current block or current sub-block to find a reference block or sub-block in the reference picture that best matches the current block or sub-block, and obtains motion information (e.g., a motion vector) that compensates (or predicts) the motion or changes from the reference block or sub-block to the current block or sub-block. The inter-predictor 126 then performs motion compensation (or motion prediction) based on the motion information and generates an inter-prediction signal for the current block or sub-block based on the motion information. The inter-predictor 126 then outputs the generated inter-prediction signal to the prediction controller 128.

[0175] Motion information used in motion compensation can be signaled in various forms as an inter prediction signal. For example, a motion vector can be signaled. As another example, a difference between a motion vector and a motion vector predictor can be signaled.

[0176] Note that, in addition to the motion information of the current block obtained from motion estimation, motion information of neighboring blocks can also be used to generate the inter prediction signal. More specifically, the inter prediction signal can be generated in each sub-block in the current block by computing a weighted sum of a prediction signal based on the motion information obtained from motion estimation (in the reference picture) and a prediction signal based on the motion information of the neighboring blocks (in the current picture). Such inter prediction (motion compensation) is also referred to as overlapped block motion compensation (OBMC).

[0177] In the OBMC mode, information indicating the sub-block size for OBMC (e.g., referred to as OBMC block size) can be signaled at the sequence level. In addition, information indicating whether to apply the OBMC mode (e.g., referred to as OBMC flag) can be signaled at the CU level. Note that the signaling of such information need not be performed at the sequence level and the CU level, but can be performed at another level (e.g., at the picture level, the slice level, the tile level, the CTU level, or the sub-block level).

[0178] In the following, the OBMC mode will be described in more detail. Figure 5B is a flowchart, Figure 5C is a conceptual diagram illustrating a prediction picture correction process performed by the OBMC process.

[0179] Referring to Figure 5C , first, a prediction picture (Pred) is obtained by typical motion compensation using the motion vector (MV) assigned to the target (current) block. In Figure 5C , the arrow "MV" points to the reference picture to indicate what the current block in the current picture is referencing in order to obtain the prediction picture.

[0180] Next, a prediction picture (Pred_L) is obtained by applying (reusing) the motion vector (MV_L) that has been derived for the encoded neighboring left block to the target (current) block, as indicated by the arrow "MV_L" starting from the current block and pointing to the reference picture to obtain the prediction picture Pred_L. Then, the two prediction pictures Pred and Pred_L are superimposed to perform a first pass of correction of the prediction picture, which has the effect, among others, of blending the boundaries between the neighboring blocks.

[0181] Similarly, the prediction picture (Pred_U) is obtained by applying (reusing) the motion vector (MV_U) that has been derived for the coded neighboring upper block to the target (current) block, as indicated by the arrow "MV_U" starting from the current block and pointing to the reference picture to obtain the prediction picture Pred_U. Then, the prediction picture Pred_U is superimposed with the prediction pictures (i.e., Pred and Pred_L) obtained from the first pass to perform a second pass of correction of the prediction picture, which in one aspect has the effect of blending the boundaries between neighboring blocks. The result of the second pass is the final prediction picture of the current block, which has blended (smoothed) boundaries with its neighboring blocks.

[0182] Note that the above example is a two-pass correction method using neighboring left and upper blocks, but the method can be a three-pass or higher pass correction method that also uses neighboring right and / or lower blocks.

[0183] Note that the area subject to superimposition can be the entire pixel area of a block, and can alternatively be a partial block boundary area.

[0184] Note here that the prediction picture correction process of OBMC is described as being based on a single reference picture to derive a single prediction picture Pred, with the additional prediction pictures Pred_L and Pred_U being superimposed thereon, but the same process can be applied to each of a plurality of reference pictures when the prediction picture is corrected based on the plurality of reference pictures. In such a case, after a plurality of corrected prediction pictures are obtained by performing the image correction of OBMC based on the plurality of reference pictures, respectively, the obtained plurality of corrected prediction pictures are also superimposed to obtain the final prediction picture.

[0185] Note that in OBMC, the unit of a target block can be a prediction block, and can alternatively be a sub-block obtained by further partitioning a prediction block.

[0186] One example of a method of determining whether to implement the OBMC process is to use obmc_flag, which is a signal indicating whether to implement the OBMC process. As one specific example, the encoder can determine whether a target block belongs to a region including complex motion. When the block belongs to a region including complex motion, the encoder sets the value of obmc_flag to "1" and implements the OBMC process during the encoding process; if the block does not belong to a region including complex motion, the value of obmc_flag is set to "0" and the block is encoded without implementing the OBMC process. The decoder switches between whether to implement the OBMC process by decoding obmc_flag written in the stream (i.e., compressed sequence) and performing decoding according to the flag value.

[0187] Note that the motion information can be derived at the decoder side without being signaled from the encoder side. For example, the merge mode defined in the H.265 / HEVC standard can be used. Further, for example, the motion information can be derived by performing motion estimation at the decoder side. In this case, the motion estimation can be performed at the decoder side without using the pixel values of the current block.

[0188] Here, a mode for performing motion estimation at the decoder side will be described. The mode for performing motion estimation at the decoder side is also referred to as a pattern matching motion vector derivation (PMMVD) mode or a frame rate up conversion (FRUC) mode.

[0189] In Figure 5D One example of the FRUC process is shown in FIG. 1. First, a candidate list of candidates (the candidate list can be a merge list) each including a predicted motion vector (MV) is generated with reference to motion vectors of coded blocks neighboring the current block in space or time. Next, a best candidate MV is selected from among the plurality of candidate MVs registered in the candidate list. For example, evaluation values of the candidate MVs included in the candidate list are calculated, and one candidate MV is selected based on the calculated evaluation values.

[0190] Next, the motion vector of the current block is derived from the motion vector of the selected candidate. More specifically, for example, the motion vector of the current block is calculated as the motion vector of the selected candidate (the best candidate MV). Alternatively, the motion vector of the current block can be derived by pattern matching performed in the vicinity of a position in a reference picture corresponding to the motion vector of the selected candidate. In other words, when the vicinity of the best candidate MV is searched using pattern matching in the reference picture and an evaluation value, and an MV with a better evaluation value is found, the best candidate MV can be updated to the MV with the better evaluation value, and the MV with the better evaluation value can be used as the final MV of the current block. A configuration in which the process for updating the MV with the better evaluation value is not implemented is also acceptable.

[0191] In a case where the process is performed in units of sub-blocks, the same process can be performed.

[0192] The evaluation value can be calculated in various ways. For example, a reconstructed image of a region in a reference picture corresponding to the motion vector is compared with a reconstructed image of a predetermined region (for example, as described below, in another reference picture or in a neighboring block in the current picture), and a difference in pixel values between the two reconstructed images can be calculated and used as the evaluation value of the motion vector. Note that the evaluation value can be calculated by using some other information in addition to the difference.

[0193] Next, the pattern matching is described in detail. First, one of the candidates included in the candidate list (e.g., the merge list) is selected as a starting point for the search by pattern matching. The pattern matching used is either the first pattern matching or the second pattern matching. The first pattern matching and the second pattern matching are also referred to as bilateral matching and template matching, respectively.

[0194] In the first pattern matching, the pattern matching is performed between two blocks in two different reference pictures that both follow the motion trajectory of the current block. Thus, in the first pattern matching, for a region in a reference picture, a region in another reference picture that coincides with the motion trajectory of the current block is used as a predetermined region for the calculation of the above-mentioned candidate evaluation value.

[0195] Figure 6 An example of the first pattern matching (bilateral matching) between two blocks in two reference pictures along the motion trajectory is shown. As shown in Figure 6 In the first pattern matching, two motion vectors (MV0, MV1) are derived by finding the best match between two blocks in two different reference pictures (Ref0, Ref1) that follow the motion trajectory of the current block (Cur block). More specifically, a difference can be obtained between (i) the reconstructed image at the position specified by the candidate MV in the first encoded reference picture (Ref0) and (ii) the reconstructed image at the position specified by the candidate MV in the second encoded reference picture (Ref1), each scaled symmetrically in the display time interval. This difference can then be used to derive an evaluation value for the current block. The candidate MV with the best evaluation value can be selected as the final MV among a plurality of candidate MVs.

[0196] Under the assumption of a continuous motion trajectory, the motion vectors (MV0, MV1) pointing to the two reference blocks are proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, when the current picture is located in time between the two reference pictures and the temporal distance from the current picture to the two reference pictures is the same, the first pattern matching derives two mirror bilateral motion vectors.

[0197] In the second pattern matching (template matching), the pattern matching is performed between a template (a block adjacent to the current block in the current picture; e.g., the top and / or left adjacent block) in the current picture and a block in a reference picture. Thus, in the second pattern matching, the block in the current picture adjacent to the current block is used as a predetermined region for the above-mentioned calculation of the candidate evaluation value.

[0198] Figure 7 An example of the pattern matching between a template in the current picture and a block in a reference picture (template matching) is shown. As shown in Figure 7In the second mode matching, the motion vector of the current block is derived by searching in the reference picture (Ref0) to find a block that best matches a neighboring block of the current block (Cur Block) in the current picture (Cur Pic). More specifically, a difference can be obtained between (i) a reconstructed image with respect to one or both of the coded neighboring top and left regions of the current block and (ii) a reconstructed image with respect to the same regions specified by a candidate MV in the coded reference picture (Ref0). The difference can then be used to derive an evaluation value for the current block. The candidate MV with the best evaluation value can be selected as the best candidate MV among a plurality of candidate MVs.

[0199] The information indicating whether to apply the FRUC mode (e.g., referred to as a FRUC flag) can be signaled at the CU level. In addition, when the FRUC mode is applied (e.g., when the FRUC flag is set to true), the information indicating the matching method (e.g., the first mode matching or the second mode matching) to which the mode applies can be signaled at the CU level. Note that the signaling of such information need not be performed at the CU level, but can be performed at another level (e.g., at the sequence level, the picture level, the slice level, the tile level, the CTU level, or the sub-block level).

[0200] Next, a method of deriving a motion vector is described. First, a description of a mode for deriving a motion vector based on a model assuming uniform linear motion is given. This mode is also referred to as a bi-directional optical flow (BIO) mode.

[0201] Figure 8 A model assuming uniform linear motion is shown in Figure 8 (v x ,v y ) represents a velocity vector, and τ0 and τ1 represent time distances between the current picture (Cur Pic) and two reference pictures (Ref0, Ref1), respectively. (MVx0, MVy0) represents a motion vector corresponding to the reference picture Ref0, and (MVx1, MVy1) represents a motion vector corresponding to the reference picture Ref1.

[0202] Here, (MVx0, MVy0) and (MVx1, MVy1) are represented as (v x τ0,v y τ0) and (-v x τ1,-v y τ1), respectively, under the assumption that the velocity vector (vx, vy) exhibits uniform linear motion, and the following optical flow equation (Equation 1) is given.

[0203] [Math. 1]

[0204]

[0205] Here, I (k) denotes the luminance value from the motion-compensated reference picture k (k = 0, 1). The optical flow equation shows that the sum of (i) the time derivative of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference picture, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference picture equals zero. The motion vector of each block obtained from, for example, the merge list can be corrected pixel by pixel based on a combination of the optical flow equation and Hermite interpolation.

[0206] Note that the motion vector can be derived at the decoder side using a method other than a model based on the assumption of uniform linear motion. For example, the motion vector of each sub-block can be derived based on the motion vectors of neighboring blocks.

[0207] Next, a description is given of a mode in which the motion vector of each sub-block is derived based on the motion vectors of neighboring blocks. This mode is also referred to as an affine motion-compensated prediction mode.

[0208] Figure 9A An example of deriving the motion vector of each sub-block based on the motion vectors of neighboring blocks is shown. In Figure 9A , the current block includes 16 4x4 sub-blocks. Here, the motion vector of the top-left corner control point in the current block is derived based on the motion vectors of neighboring sub-blocks. Similarly, the motion vector of the top-right corner control point in the current block is derived based on the motion vectors of neighboring blocks. Next, using the two motion vectors v0 and v1, the motion vector of each sub-block in the current block is derived using the following equation 2. x , v y ).

[0209] [Equation 2]

[0210]

[0211] Here, x and y are the horizontal and vertical positions of the sub-block, respectively, and w is a predetermined weighting coefficient.

[0212] The affine motion-compensated prediction mode can include a plurality of modes of different methods of deriving the motion vectors of the top-left and top-right corner control points. Information indicating the affine motion-compensated prediction mode (e.g., referred to as an affine flag) can be signaled at the CU level. Note that the signaling of the information indicating the affine motion-compensated prediction mode need not be performed at the CU level, but can be performed at another level (e.g., at the sequence level, the picture level, the slice level, the tile level, the CTU level, or the sub-block level).

[0213] (Prediction controller)

[0214] Prediction controller 128 selects an intra-frame prediction signal (output from intra-frame predictor 124) or an inter-frame prediction signal (output from inter-frame predictor 126), and outputs the selected prediction signal to subtractor 104 and adder 116.

[0215] like Figure 1 As shown, in various implementations, prediction controller 128 can output prediction parameters that are input to entropy encoder 110. Entropy encoder 110 can generate an encoded bitstream (or sequence) based on the prediction parameters input from prediction controller 128 and quantized coefficients input from quantizer 108. The prediction parameters can be used by a decoder that receives and decodes the encoded bitstream to perform the same prediction processing as performed in intra-frame predictor 124, inter-frame predictor 126, and prediction controller 128. Prediction parameters can include selected prediction signals (e.g., motion vectors, prediction types, or prediction modes employed in intra-frame predictor 124 or inter-frame predictor 126), or any index, flag, or value based on or indicating the prediction processing performed in intra-frame predictor 124, inter-frame predictor 126, and prediction controller 128.

[0216] Figure 9B An example of the process for deriving motion vectors in the current image in merge mode is shown.

[0217] First, a list of predicted MVs is generated, in which predicted MV candidates are registered. Examples of predicted MV candidates include: spatially adjacent predicted MVs, which are MVs of coded blocks located spatially near the target block; temporally adjacent predicted MVs, which are MVs of blocks in the encoded reference image that are adjacent to the block located in the same position as the target block; coupled predicted MVs, which are MVs generated by combining the MV values ​​of spatially adjacent predicted MVs and temporally adjacent predicted MVs; and zero predicted MVs, which are MVs with a value of zero.

[0218] Next, the MV of the target block is determined by selecting one of the multiple predicted MVs registered in the predicted MV list.

[0219] Additionally, in a variable-length encoder, the signal merge_idx, which indicates which predicted MV was selected, is written and encoded into the stream.

[0220] Note that in Figure 9B The predicted MVs registered in the predicted MV list shown constitute an example. The number of predicted MVs registered in the predicted MV list can be related to... Figure 9B The quantities shown are different; the predicted MVs registered in the predicted MV list can be omitted. Figure 9B The examples given include one or more types of predicted MVs, except inFigure 9B Examples given in the type of Figure 9B Examples given in the type of

[0221] The final MV can be determined by performing DMVR (Dynamic Motion Vector Refinement) processing (described later) using the MV of the target block derived in the merge mode.

[0222] Figure 9C is a conceptual diagram illustrating an example of DMVR processing for determining an MV.

[0223] First, the most suitable MV set for the current block (e.g., in the merge mode) is considered as a candidate MV. Next, reference pixels are identified in a first reference picture (L0) that is a coded picture in the L0 direction, according to the candidate MV (L0). Similarly, reference pixels are identified in a second reference picture (L1) that is a coded picture in the L1 direction, according to the candidate MV (L1). Next, the reference pixels are averaged to form a template.

[0224] Next, using the template, a surrounding area of the candidate MV of the first reference picture (L0) and the second reference picture (L1) is searched, and the MV having the lowest cost is determined as the final MV. For example, a cost value can be calculated using the difference between each pixel value in the template and each pixel value in the searched area, using the candidate MV, etc.

[0225] Note that the configuration and operation of the processing described here are substantially the same on both the encoder side and the decoder side, which will be described later.

[0226] Any processing other than the above processing can be used as long as the processing is capable of deriving a final MV by searching the surrounding environment of a candidate MV.

[0227] Next, a description is given of an example of a mode in which a prediction image (prediction) is generated using LIC (Local Illumination Compensation) processing.

[0228] Figure 9D An example of a prediction image generation method using a brightness correction process performed by LIC processing is shown.

[0229] First, an MV is derived from a coded reference picture to obtain a reference image corresponding to the current block.

[0230] Next, for the current block, information indicating how the luminance values change between the reference picture and the current picture is obtained based on the luminance pixel values of the encoded neighboring left reference region and the encoded neighboring upper reference region in the current picture, and based on the luminance pixel values of the same positions in the reference picture specified by the MV. The information indicating how the luminance values change is used to calculate the luminance correction parameter.

[0231] The prediction picture for the current block is generated by performing a luminance correction process that applies the luminance correction parameter to the reference picture in the reference picture specified by the MV.

[0232] Note that, Figure 9D The shape of the surrounding reference region shown in FIG. 6 is merely one example; the surrounding reference region can have a different shape.

[0233] Further, although the prediction picture is generated from a single reference picture in this example, in the case where the prediction picture is generated from multiple reference pictures, the prediction picture can be generated after the luminance correction process is performed on the reference picture obtained from the reference picture as described above.

[0234] One example of a method for determining whether to implement the LIC process is to use lic_flag, which is a signal indicating whether to implement the LIC process. As one specific example, the encoder determines whether the current block belongs to a region of luminance change. When the block belongs to a region of luminance change, the encoder sets lic_flag to "1", and implements the LIC process at the time of encoding. When the block does not belong to a region of luminance change, the encoder sets lic_flag to "0", and performs encoding that implements the LIC process. The decoder can switch between whether to perform the LIC process by decoding lic_flag written in the stream and performing decoding according to the flag value.

[0235] One example of a different method for determining whether to implement the LIC process includes discriminating whether the LIC process is determined to be implemented on a surrounding block. In one specific example, when the merge mode is used on the current block, it is determined whether the LIC process is applied in the encoding of a surrounding encoded block that is selected when the MV is derived in the merge mode. This determination is then used to further determine whether to implement the LIC process on the current block. Note that in this example, the same applies to the processing performed on the decoder side.

[0236] (Decoder)

[0237] Next, a decoder that can decode the encoded signal (encoded bitstream) output from the encoder 100 will be described. Figure 10is a block diagram showing a functional configuration of a decoder 200 according to an embodiment. The decoder 200 is a motion picture decoder that decodes a motion picture on a block-by-block basis.

[0238] As shown in Figure 10 , the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, a summer 208, a block memory 210, a loop filter 212, a frame memory 214, an intra predictor 216, an inter predictor 218, and a prediction controller 220.

[0239] The decoder 200 is implemented as, for example, a general-purpose processor and a memory. In this case, when the processor executes a software program stored in the memory, the processor functions as the entropy decoder 202, the inverse quantizer 204, the inverse transformer 206, the summer 208, the loop filter 212, the intra predictor 216, the inter predictor 218, and the prediction controller 220. Alternatively, the decoder 200 can be implemented as one or more dedicated electronic circuits corresponding to the entropy decoder 202, the inverse quantizer 204, the inverse transformer 206, the summer 208, the loop filter 212, the intra predictor 216, the inter predictor 218, and the prediction controller 220.

[0240] Hereinafter, each component included in the decoder 200 will be described.

[0241] (Entropy Decoder)

[0242] The entropy decoder 202 entropy-decodes a decoded bitstream. More specifically, for example, the entropy decoder 202 arithmetically decodes an encoded bitstream into a binary signal. Then, the entropy decoder 202 debinarizes the binary signal. The entropy decoder 202 outputs the quantized coefficients of each block to the inverse quantizer 204. The entropy decoder 202 can also output prediction parameters (see Figure 1 ) included in the encoded bitstream to the intra predictor 216, the inter predictor 218, and the prediction controller 220 so that they can perform the same prediction processing as that performed on the encoder side in the intra predictor 124, the inter predictor 126, and the prediction controller 128.

[0243] (Inverse Quantizer)

[0244] The inverse quantizer 204 inverse-quantizes the quantized coefficients of a block to be decoded (hereinafter referred to as a current block) input from the entropy decoder 202. More specifically, the inverse quantizer 204 inverse-quantizes the quantized coefficients of the current block on the basis of a quantization parameter corresponding to the quantized coefficients. Then, the inverse quantizer 204 outputs the inverse-quantized coefficients (i.e., transform coefficients) of the current block to the inverse transformer 206.

[0245] (inverse transformer)

[0246] The inverse transformer 206 recovers a prediction error (residual) by inversely transforming the transform coefficients input from the inverse quantizer 204.

[0247] For example, when information parsed from the coded bitstream indicates the application of EMT or AMT (e.g., when an AMT flag is set to true), the inverse transformer 206 inversely transforms the transform coefficients of the current block based on the information indicating the parsed transform type.

[0248] Further, for example, when information parsed from the coded bitstream indicates the application of NSST, the inverse transformer 206 applies a secondary inverse transform to the transform coefficients.

[0249] (adder)

[0250] The adder 208 reconstructs the current block by adding the prediction error input from the inverse transformer 206 to the prediction sample that is input from the prediction controller 220. Then, the adder 208 outputs the reconstructed block to the block memory 210 and the loop filter 212.

[0251] (block memory)

[0252] The block memory 210 is a memory for storing a block in a picture (hereinafter referred to as a current picture) to be decoded to be referred to in intra prediction. More specifically, the block memory 210 stores the reconstructed block output from the adder 208.

[0253] (loop filter)

[0254] The loop filter 212 applies a loop filter to the block reconstructed by the adder 208, and outputs the filtered reconstructed block to the frame memory 214 and, for example, to a display device.

[0255] When information indicating whether to enable or disable the ALF parsed from the coded bitstream indicates enabling, one filter is selected from a plurality of filters based on a direction and activity of a local gradient, and the selected filter is applied to the reconstructed block.

[0256] (frame memory)

[0257] The frame memory 214 is a storage device for storing a reference picture used in inter prediction, and is also referred to as a frame buffer. More specifically, the frame memory 214 stores the reconstructed block filtered by the loop filter 212.

[0258] (intra predictor)

[0259] The intra predictor 216 generates a prediction signal (intra-predicted signal) by intra-prediction with reference to one or more blocks in the current picture as stored in the block memory 210. More specifically, the intra predictor 216 generates an intra-predicted signal by intra-prediction with reference to samples (e.g., luma and / or chroma values) of one or more blocks neighboring the current block, and then outputs the intra-predicted signal to the prediction controller 220.

[0260] Note that when an intra-prediction mode in which a chroma block is intra-predicted from a luma block is selected, the intra predictor 216 can predict the chroma component of the current block based on the luma component of the current block.

[0261] Further, when information indicating the application of PDPC is parsed from the encoded bitstream (e.g., in the prediction parameters output from the entropy decoder 202), the intra predictor 216 corrects the intra-predicted pixel values based on the horizontal / vertical reference pixel gradients.

[0262] (inter-predictor)

[0263] The inter-predictor 218 predicts the current block with reference to the reference pictures stored in the frame memory 214. Inter-prediction is performed for each current block or each sub-block (e.g., each 4x4 block) in the current block. For example, the inter-predictor 218 generates an inter-predicted signal for the current block or sub-block based on motion compensation using motion information (e.g., motion vectors) parsed from the encoded bitstream (e.g., in the prediction parameters output from the entropy decoder 202), and outputs the inter-predicted signal to the prediction controller 220.

[0264] When information parsed from the encoded bitstream indicates the application of the OBMC mode, the inter-predictor 218 generates an inter-predicted signal using motion information of neighboring blocks in addition to motion information of the current block obtained from motion estimation.

[0265] Further, when information parsed from the encoded bitstream indicates the application of the FRUC mode, the inter-predictor 218 derives motion information by performing motion estimation according to a pattern matching method (bilateral matching or template matching) parsed from the encoded bitstream. Then, the inter-predictor 218 performs motion compensation (prediction) using the derived motion information.

[0266] Further, when the BIO mode is to be applied, the inter-predictor 218 derives a motion vector based on a model assuming uniform linear motion. Further, when information parsed from the encoded bitstream indicates that the affine motion compensation prediction mode is to be applied, the inter-predictor 218 derives a motion vector for each sub-block based on motion vectors of neighboring blocks.

[0267] (prediction controller)

[0268] Prediction controller 220 selects either an intra-frame prediction signal or an inter-frame prediction signal and outputs the selected prediction signal to adder 208. Generally, the configuration, function, and operation of the decoder-side prediction controller 220, inter-frame predictor 218, and intra-frame predictor 216 can correspond to the configuration, function, and operation of the encoder-side prediction controller 128, inter-frame predictor 126, and intra-frame predictor 124.

[0269] (Non-rectangular partition)

[0270] In the prediction controller 128 of the intra-frame predictor 124 and inter-frame predictor 126 coupled to the encoder side (see...) Figure 1 ), and in the prediction controller 220 of the intra-predictor 216 and inter-predictor 218 coupled to the decoder side (see Figure 10 To date, the partitions (or variable-sized blocks or sub-blocks) obtained by dividing each block to acquire motion information (e.g., motion vectors) have always been rectangular, such as... Figure 2 As shown. The inventors have discovered that, in various implementations, generating partitions with non-rectangular shapes such as triangles leads to improvements in image quality and coding efficiency based on the content of the image in the picture. Various embodiments will be described below, wherein at least one partition segmented from the image patch for prediction purposes has a non-rectangular shape. Note that these embodiments are equally applicable to the encoder side (prediction controller 128 coupled to intra-frame predictor 124 and inter-frame predictor 126) and the decoder side (prediction controller 220 coupled to intra-frame predictor 216 and inter-frame predictor 218), and can be implemented in... Figure 1 The encoder or Figure 10 It is implemented in decoders, etc.

[0271] Figure 11 The flowchart illustrates the following process, which divides an image patch into partitions that include at least a first partition with a non-rectangular shape (e.g., a triangle) and a second partition, and performs an additional process that includes encoding (or decoding) the image patch into a reconstructed combination of the first and second partitions.

[0272] In step S1001, the image patch is divided into multiple partitions, which may include a first partition with a non-rectangular shape and a second partition that may or may not have a non-rectangular shape. For example, as in Figure 12As illustrated in FIG. 19, an image block can be partitioned from a top-left corner of the image block to a bottom-right corner of the image block to create a first partition and a second partition that each have a non-rectangular shape (e.g., a triangle), or can be partitioned from a top-right corner of the image block to a bottom-left corner of the image block to create a first partition and a second partition that each have a non-rectangular shape (e.g., a triangle). Various examples of non-rectangular partitioning will be described below with reference to Figure 12 and 17 -19.

[0273] In step S1002, the process predicts a first motion vector for the first partition, and predicts a second motion vector for the second partition. For example, the prediction of the first and second motion vectors can include selecting the first motion vector from a first set of motion vector candidates, and selecting the second motion vector from a second set of motion vector candidates.

[0274] In step S1003, motion compensation processing is performed to obtain the first partition using the first motion vector derived in step S1002 above, and to obtain the second partition using the second motion vector derived in step S1002 above.

[0275] In step S1004, a prediction process is performed on the image block that is a combination of the (reconstructed) first and second partitions. The prediction process can include a boundary smoothing process that is used to smooth a boundary between the first and second partitions. For example, the boundary smoothing process can involve weighting a first value of a boundary pixel that is predicted based on the first partition, and a second value of the boundary pixel that is predicted based on the second partition. Various implementations of the boundary smoothing process will be described below with reference to Figure 13 , 14 , 20 and 21A-21D.

[0276] In step S1005, the process encodes or decodes the image block using one or more parameters that include a partition parameter that indicates that the image block is partitioned into a second partition and a first partition that has a non-rectangular shape. As summarized in the table in Figure 15 , for example, the partition parameter (the "first index value") can jointly encode the partition direction (e.g., from top-left to bottom-right or from top-right to bottom-left, as illustrated in FIGS. Figure 12 , and the first and second motion vectors derived in step S1002 above. Details of such partition syntax operations involving one or more parameters including the partition parameter will be described in detail below with reference to Figure 15 , 16 and 22-25.

[0277] Figure 17is a flowchart illustrating a process 2000 of partitioning an image block. In step S2001, the process partitions an image into a plurality of partitions, including a first partition having a non-rectangular shape and a second partition which can or can not have a non-rectangular shape. As shown in Figure 12 , an image block can be partitioned into a first partition having a triangular shape and a second partition also having a triangular shape. There are many other examples in which an image block is partitioned into a plurality of partitions including a first partition and a second partition, where at least the first partition has a non-rectangular shape. The non-rectangular shape can be a triangle, a trapezoid, and a polygon having at least five sides and corners.

[0278] For example, as shown in Figure 18 , an image block can be partitioned into two triangular partitions; an image block can be partitioned into more than two triangular partitions (e.g., three triangular partitions); an image block can be partitioned into a combination of triangular partitions and rectangular partitions; or an image block can be partitioned into a combination of triangular partitions and polygonal partitions.

[0279] As further shown in Figure 19 , an image block can be partitioned into an L-shaped (polygonal) partition and a rectangular-shaped partition; an image block can be partitioned into a pentagonal (polygonal)-shaped partition and a triangular-shaped partition; an image block can be partitioned into a hexagonal (polygonal)-shaped partition and a pentagonal (polygonal)-shaped partition; or an image block can be partitioned into a plurality of polygonal partitions.

[0280] Referring back to Figure 17 , in step S2002, the process predicts a first motion vector for the first partition, e.g., by selecting the first partition from a first set of motion vector candidates, and predicts a second motion vector for the second partition, e.g., by selecting the second motion vector from a second set of motion vector candidates. For example, the first set of motion vector candidates can include motion vectors of partitions neighboring the first partition, while the second set of motion vector candidates can include motion vectors of partitions neighboring the second partition. The neighboring partitions can be one or both of spatially neighboring partitions and temporally neighboring partitions. Some examples of spatially neighboring partitions include partitions located to the left, lower-left, lower, lower-right, right, upper-right, upper, or upper-left of the partition being processed. An example of a temporally neighboring partition is a partition in the reference picture of the image block located in the same position.

[0281] In various implementations, the partitions neighboring the first partition and the partitions neighboring the second partition can be outside of the image block from which the first and second partitions are partitioned. The first set of motion vector candidates can be the same as or different from the second set of motion vector candidates. Further, at least one of the first set of motion vector candidates and the second set of motion vector candidates can be the same as a third set of motion vector candidates prepared for the image block.

[0282] In some implementations, in step S2002, in response to determining that the second partition also has a non-rectangular shape (e.g., triangular) similar to the first partition, the process 2000 creates a second set of motion vector candidates (for the non-rectangular second partition) that includes motion vectors of partitions neighboring the second partition (excluding the first partition). On the other hand, in response to determining that the second partition (different from the first partition) has a rectangular shape, the process 2000 creates a second set of motion vector candidates (for the rectangular second partition) that includes motion vectors of partitions neighboring the second partition (including the first partition).

[0283] In step S2003, the process encodes or decodes the first partition using the first motion vector derived in the above step S2002, and encodes or decodes the second partition using the second motion vector derived in the above step S2002.

[0284] As with the process 2000, Figure 17 An image block partitioning process can be performed by an image encoder, e.g., as shown in Figure 1 The image encoder includes circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs: partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape (step S2001); predicting a first motion vector for the first partition and a second motion vector for the second partition (step S2002); encoding the first partition using the first motion vector and encoding the second partition using the second motion vector (step S2003).

[0285] According to another embodiment, as Figure 1As shown, an image encoder is provided, including a partitioner 102 that in operation receives an original picture and partitions it into a plurality of blocks, a subtractor 104 that in operation receives blocks from the partitioner and predictions from a prediction controller 128 and subtracts each prediction from its corresponding block to output a residual, a transformer 106 that in operation performs a transform on the residuals output from the subtractor 104 to output transform coefficients, a quantizer 108 that in operation quantizes the transform coefficients to generate quantized transform coefficients, an entropy encoder 110 that in operation encodes the quantized transform coefficients to generate a bitstream, and a prediction controller 128 coupled to an inter-predictor 126, an intra-predictor 124, and memories 118, 122, where the inter-predictor 126 in operation generates a prediction of a current block based on a reference block in an encoded reference picture, the intra-predictor 124 in operation generates a prediction of a current block based on an encoded reference block in a current picture. The prediction controller 128 in operation partitions a block into a plurality of partitions, including a second partition and a first partition having a non-rectangular shape Figure 17 , step S2001); predicting a first motion vector for the first partition and a second motion vector for the second partition (step S2002); encoding the first partition using the first motion vector and the second partition using the second motion vector (step S2003).

[0286] According to another embodiment, as shown in Figure 10 , an image decoder is provided, including circuitry and a memory coupled to the circuitry. The circuitry in operation performs: partitioning an image block into a plurality of partitions, including a second partition and a first partition having a non-rectangular shape Figure 17 , step S2001); predicting a first motion vector for the first partition and a second motion vector for the second partition (step S2002); decoding the first partition using the first motion vector and the second partition using the second motion vector (step S2003).

[0287] According to another embodiment, as shown in Figure 10The image decoder shown in FIG. 1 includes an entropy decoder 202 that, in operation, receives and decodes an encoded bitstream to obtain quantized transform coefficients, an inverse quantizer 204 and an inverse transformer 206 that, in operation, inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual, an adder 208 that, in operation, adds the residual output from the inverse quantizer 204 and the inverse transformer 206 to a prediction output from a prediction controller 220 to reconstruct a block, and the prediction controller 220 coupled to an inter-predictor 218, an intra-predictor 216, and memories 210, 214, where the inter-predictor 218, in operation, generates a prediction for a current block based on a reference block in a decoded reference picture and the intra-predictor 216, in operation, generates a prediction for the current block based on a decoded reference block in a current picture. The prediction controller 220, in operation, partitions the image block into a plurality of partitions including a first partition having a non-rectangular shape and a second partition. Figure 17 , step S2001); predicting a first motion vector for the first partition and a second motion vector for the second partition (step S2002); decoding the first partition using the first motion vector and the second partition using the second motion vector (step S2003).

[0288] (boundary smoothing)

[0289] As described above in Figure 11 , according to various embodiments, the step S1004 of performing a prediction process on the image block that is a combination of the (reconstructed) first partition having a non-rectangular shape and the second partition can involve applying a boundary smoothing process along the boundary between the first partition and the second partition.

[0290] For example, Figure 21B One example of a boundary smoothing process is shown that involves weighting a first value of a boundary pixel that is first predicted based on the first partition and a second value of the boundary pixel that is second predicted based on the second partition.

[0291] Figure 20 is a flowchart showing an overall boundary smoothing process 3000 according to one embodiment that involves weighting a first value of a boundary pixel that is first predicted based on the first partition and a second value of the boundary pixel that is second predicted based on the second partition. In step S3001, the image block is partitioned into the first partition and the second partition along the boundary, where at least the first partition has a non-rectangular shape, as in Figure 21A or above Figure 12 , 18 and 19.

[0292] In step S3002, a set of pixels in the first partition along the boundary ( Figure 21A A first prediction is made for a first value (e.g., color, brightness, transparency, etc.) of the "boundary pixels" in the image, wherein the first value is predicted first using information from the first partition. In step S3003, a second prediction is made for a second value of a (same) group of pixels along the boundary of the first partition, wherein the second prediction is made using information from the second partition. In some implementations, at least one of the first and second predictions is an inter-frame prediction process that predicts the first and second values ​​based on a reference partition in an encoded reference image. Figure 21D In some implementations, the prediction process predicts a first value (“first set of samples”) of all pixels in a first partition that includes a set of pixels on which the first and second partitions overlap, and predicts a second value (“second set of samples”) of only a set of pixels on which the first and second partitions overlap. In another implementation, at least one of the first and second predictions is an intra-frame prediction process that predicts the first and second values ​​based on an encoded reference partition in the current image. In some implementations, the prediction method used in the first prediction differs from the prediction method used in the second prediction. For example, the first prediction may include an inter-frame prediction process, and the second prediction may include an intra-frame prediction process. The information used for the first prediction of the first value or the second prediction of the second value may be the motion vector of the first or second partition, the intra-frame prediction direction, etc.

[0293] In step S3004, the first value predicted using the first partition and the second value predicted using the second partition are weighted. In step S3005, the weighted first and second values ​​are used to encode or decode the first partition.

[0294] Figure 21B An example of a boundary smoothing operation is shown, where the first and second partitions overlap by a maximum of five pixels per row or column. That is, the number of pixels in each row or column that predicts the first value based on the first partition and the second value based on the second partition is at most five. Figure 21C Another example of a boundary smoothing operation is shown, where the first and second partitions overlap by a maximum of three pixels per row or column. That is, the number of pixels in each row or column whose first value is predicted based on the first partition and whose second value is predicted based on the second partition is at most three.

[0295] Figure 13Another example of a boundary smoothing operation is shown, in which the first partition and the second partition overlap on (at most) four pixels per row or column. That is, the number of pixels in each row or column for which a first value is predicted based on the first partition and a second value is predicted based on the second partition is at most four. In the example shown, weights of 1 / 8, 1 / 4, 3 / 4, and 7 / 8 can be applied to the first values of the four pixels in the group, respectively, and weights of 7 / 8, 3 / 4, 1 / 4, and 1 / 8 can be applied to the second values of the four pixels in the group, respectively.

[0296] Figure 14 Further examples of a boundary smoothing operation are shown, in which the first partition and the second partition overlap on zero pixels per row or column (i.e., they do not overlap), one pixel per row or column (at most), and two pixels per row or column (at most). In the example in which the first and second partitions do not overlap, zero weight is applied. In the example in which the first partition and the second partition overlap on one pixel per row or column, a weight of 1 / 2 can be applied to the first values of the pixels in the group predicted based on the first partition, and a weight of 1 / 2 can be applied to the second values of the pixels in the group predicted based on the second partition. In the example in which the first partition and the second partition overlap on two pixels per row or column, weights of 1 / 3 and 2 / 3 can be applied to the first values of the two pixels in the group predicted based on the first partition, respectively, and weights of 2 / 3 and 1 / 3 can be applied to the second values of the two pixels in the group predicted based on the second partition, respectively.

[0297] According to the above embodiments, the number of pixels in the group on which the first partition and the second partition overlap is an integer. In other implementations, for example, the number of overlapping pixels in the group can be a non-integer and can be a fraction. Likewise, the weights applied to the first values and the second values of a group of pixels can be fractions or integers, depending on each application.

[0298] Similar to the process 3000 of FIG. 3A, a boundary smoothing process can be performed by, for example, an image encoder as shown in FIG. 3B, which includes circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation includes first predicting first values of a group of pixels of the first partition along the boundary using information of the first partition (step S3002), second predicting second values of the group of pixels of the first partition along the boundary using information of the second partition (step S3003), weighting the first values and the second values (step S3004), and encoding the first partition using the weighted first values and the weighted second values (step S3005). Figure 20 Figure 1 The circuitry, in operation, performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation includes first predicting first values of a group of pixels of the first partition along the boundary using information of the first partition (step S3002), second predicting second values of the group of pixels of the first partition along the boundary using information of the second partition (step S3003), weighting the first values and the second values (step S3004), and encoding the first partition using the weighted first values and the weighted second values (step S3005). Figure 20 The circuitry, in operation, performs a boundary smoothing operation along a boundary between a second partition partitioned from an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation includes first predicting first values of a group of pixels of the first partition along the boundary using information of the first partition (step S3002), second predicting second values of the group of pixels of the first partition along the boundary using information of the second partition (step S3003), weighting the first values and the second values (step S3004), and encoding the first partition using the weighted first values and the weighted second values (step S3005).​

[0299] According to another embodiment, as shown in Figure 1 , there is provided an image encoder comprising: a partitioner 102 that in operation receives an original picture and partitions it into blocks; a subtractor 104 that in operation receives blocks from the partitioner and predictions from a prediction controller 128 and subtracts each prediction from its corresponding block to output a residual; a transformer 106 that in operation performs a transform on the residuals output from the subtractor 104 to output transform coefficients; a quantizer 108 that in operation quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder 110 that in operation encodes the quantized transform coefficients to generate a bitstream; and a prediction controller 128 coupled to an inter-predictor 126, an intra-predictor 124, and memories 118, 122, wherein the inter-predictor 126 in operation generates a prediction of a current block based on a reference block in an encoded reference picture, the intra-predictor 124 in operation generates a prediction of a current block based on an encoded reference block in a current picture. The prediction controller 128 in operation performs a boundary smoothing operation along a boundary between a second partition partitioned out of an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation comprises: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary (step S3002); using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary (step S3003); weighting the first values and the second values (step S3004); and encoding the first partition using the weighted first values and the weighted second values (step S3005). Figure 20

[0300] According to another embodiment, there is provided an image decoder as shown in Figure 10 , comprising circuitry and a memory coupled with the circuitry. The circuitry in operation performs a boundary smoothing operation along a boundary between a second partition partitioned out of an image block and a first partition having a non-rectangular shape (step S3001). The boundary smoothing operation comprises: using information of the first partition, first predicting first values of a set of pixels of the first partition along the boundary (step S3002); using information of the second partition, second predicting second values of the set of pixels of the first partition along the boundary (step S3003); weighting the first values and the second values (step S3004); and decoding the first partition using the weighted first values and the weighted second values (step S3005). Figure 20

[0301] According to another embodiment, there is provided an image decoder as shown in Figure 10 ​​The illustrated image decoder includes an entropy decoder 202 that, in operation, receives and decodes an encoded bitstream to obtain quantized transform coefficients, an inverse quantizer 204 and an inverse transformer 206 that, in operation, inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain a residual, an adder 208 that, in operation, adds the residual output from the inverse quantizer 204 and inverse transformer 206 to a prediction output from a prediction controller 220 to reconstruct a block, and the prediction controller 220 coupled to an inter-predictor 218, an intra-predictor 216, and memories 210, 214, where the inter-predictor 218, in operation, generates a prediction of a current block based on a reference block in a decoded reference picture and the intra-predictor 216, in operation, generates a prediction of the current block based on a decoded reference block in a current picture. The prediction controller 220, in operation, performs a boundary smoothing operation along a boundary between a first partition having a non-rectangular shape and a second partition partitioned from an image block Figure 20 , step S3001). The boundary smoothing operation includes first predicting (step S3002) first values of a set of pixels of the first partition along the boundary using information of the first partition, second predicting (step S3003) second values of the set of pixels of the first partition along the boundary using information of the second partition, weighting (step S3004) the first values and the second values, and decoding (step S3005) the first partition using the weighted first values and the weighted second values.

[0302] (Entropy encoding and decoding using partition parameter syntax)

[0303] As illustrated in Figure 11 , step S1005, according to various embodiments, an image block partitioned into a second partition and a first partition having a non-rectangular shape can be encoded or decoded using one or more parameters including a partition parameter indicating a non-rectangular partitioning of the image block. In various embodiments, such a partition parameter can jointly encode, for example, a partition direction applied to the partitioning (e.g., from top left to bottom right or from top right to bottom left, see Figure 12 ), and first and second motion vectors predicted in step S1002, as will be more fully described below.

[0304] Figure 15 is a table of sample partition parameters ("first index value") and sets of information jointly encoded by the partition parameter. The partition parameter ("first index value") ranges from 0 to 6 and jointly encodes: a direction of partitioning an image block into a first partition and a second partition (both of which are triangles, see Figure 12 ) and a first motion vector predicted for the first partition Figure 11, step S1002), and predicting a second motion vector (MV2) for the second partition (P2) (step S1003). In particular, partition parameter 0 encodes a split direction from top-left to bottom-right, a first motion vector (MV1) is a "second" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "first" motion vector listed in a second set of motion vector candidates for the second partition (P2). Figure 11

[0305] Partition parameter 1 encodes a split direction from top-right to bottom-left, a first motion vector (MV1) is a "first" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "second" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 2 encodes a split direction from top-right to bottom-left, a first motion vector (MV1) is a "second" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "first" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 3 encodes a split direction from top-left to bottom-right, a first motion vector (MV1) is a "second" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "second" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 4 encodes a split direction from top-right to bottom-left, a first motion vector (MV1) is a "second" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "third" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 5 encodes a split direction from top-left to bottom-right, a first motion vector (MV1) is a "third" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "first" motion vector listed in a second set of motion vector candidates for the second partition (P2). Partition parameter 6 encodes a split direction from top-left to bottom-right, a first motion vector (MV1) is a "fourth" motion vector listed in a first set of motion vector candidates for the first partition (P1), and a second motion vector (MV2) is a "first" motion vector listed in a second set of motion vector candidates for the second partition (P2).

[0306] Figure 22 is a flowchart illustrating a method 4000 performed at an encoder side. In step S4001, the process partitions an image block into a plurality of partitions based on a partition parameter indicating a split, the plurality of partitions including a second partition (P2) and a first partition (P1) having a non-rectangular shape. For example, as described above with respect to FIG. 10, the first partition (P1) has a non-rectangular shape. Figure 15 ​As shown, the partitioning parameters can indicate the direction of segmenting the image patch (e.g., from the upper right corner to the lower left corner or from the upper left corner to the lower right corner). In step S4002, the process encodes the first and second partitions. In step S4003, the process writes one or more parameters, including the partitioning parameters, into a bitstream. The decoder side can receive and decode the bitstream to obtain one or more parameters, thereby performing the same prediction process (as performed on the encoder side) on the first and second partitions. The one or more parameters including the partitioning parameters can jointly or separately encode various information fragments, such as the non-rectangular shape of the first partition, the shape of the second partition, the segmentation direction for segmenting the image patch to obtain the first and second partitions, the first motion vector of the first partition, the second motion vector of the second partition, etc.

[0307] Figure 23 This is a flowchart illustrating method 5000 performed on the decoder side. In step S5001, the process parses one or more parameters from the bitstream, wherein the one or more parameters include partitioning parameters indicating that an image patch is divided into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape. The one or more parameters, including the partitioning parameters parsed from the bitstream, can jointly or separately encode various information fragments required for performing the same prediction processing on the decoder side as on the encoder side, such as the non-rectangular shape of the first partition, the shape of the second partition, the segmentation direction for segmenting the image patch to obtain the first and second partitions, a first motion vector of the first partition, a second motion vector of the second partition, etc. In step S5002, process 5000 segments the image patch into multiple partitions based on the partitioning parameters parsed from the bitstream. In step S5003, the process decodes the first and second partitions segmented from the image patch.

[0308] Figure 24 It is a table of sampling partition parameters (“first index value”) and information sets jointly encoded by the partition parameters, and is essentially similar to the above. Figure 15 The sampling table in [the document / reference table]. Figure 24 In this context, the partitioning parameter (“first index value”) ranges from 0 to 6, and the following items are jointly encoded: the shape of the first and second partitions segmented from the image patch, the orientation in which the image patch is segmented into the first and second partitions, and the first motion vector predicted for the first partition. Figure 11 Step S1002), and the second motion vector predicted for the second partition ( Figure 11 (Step S1002). Specifically, the partition parameter 0 is encoded as follows: neither the first partition nor the second partition has a triangular shape, and therefore, the partitioning direction information is "N / A", the first motion vector information is "N / A", and the second motion vector information is "N / A".

[0309] Partition parameter 1 encodes the following: the first and second partitions are triangles, the split direction is from top-left to bottom-right, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 2 encodes the following: the first and second partitions are triangles, the split direction is from top-right to bottom-left, the first motion vector is the "first" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 3 encodes the following: the first and second partitions are triangles, the split direction is from top-right to bottom-left, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 4 encodes the following: the first and second partitions are triangles, the split direction is from top-left to bottom-right, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 5 encodes the following: the first and second partitions are triangles, the split direction is from top-right to bottom-left, the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "third" motion vector listed in the second set of motion vector candidates for the second partition. Partition parameter 6 encodes the following: the first and second partitions are triangles, the split direction is from top-left to bottom-right, the first motion vector is the "third" motion vector listed in the first set of motion vector candidates for the first partition, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second partition.

[0310] According to some implementations, the partition parameters (index values) can be binarized according to a binarization scheme that is selected according to the value of at least one or one or more parameters. Figure 16 An example binarization scheme for binarizing index values (partition parameter values) is shown.

[0311] Figure 25 is a table of sample combinations of a first parameter and a second parameter, where one parameter is a partition parameter indicating a partitioning of an image block into a plurality of partitions including a second partition and a first partition having a non-rectangular shape. In this example, the partition parameter can be used to indicate the partitioning of the image block without jointly encoding other information encoded by one or more other parameters.

[0312] In a first example of Figure 25 , the first parameter is used to indicate the image block size, and the second parameter is used as a partition parameter (flag) to indicate that at least one of the plurality of partitions split from the image block has a triangular shape. Such a combination of the first and second parameters can be used to indicate, for example, 1) no triangular partition when the image block size is greater than 64x64, or 2) no triangular partition when the aspect ratio of the image block is greater than 4 (e.g., 64x4).

[0313] In a second example of Figure 25 , the first parameter is used to indicate the prediction mode, and the second parameter is used as a partition parameter (flag) to indicate that at least one of the plurality of partitions split from the image block has a triangular shape. Such a combination of the first and second parameters can be used to indicate, for example, 1) no triangular partition when the image block is coded in intra mode.

[0314] In a third example of Figure 25 , the first parameter is used as a partition parameter (flag) to indicate that at least one of the plurality of partitions split from the image block has a triangular shape, and the second parameter is used to indicate the prediction mode. Such a combination of the first and second parameters can be used to indicate, for example, 1) the image block must be inter coded when at least one of the plurality of partitions split from the image block has a triangular shape.

[0315] In a fourth example of Figure 25 , the first parameter indicates the motion vector of a neighboring block, and the second parameter is used as a split parameter to indicate the direction to split the image block into two triangles. Such a combination of the first and second parameters can be used to indicate, for example, 1) the direction to split the image block into two triangles is from the top-left corner to the bottom-right corner when the motion vector of the neighboring block is a diagonal direction.

[0316] In a fifth example of Figure 25 , the first parameter indicates the intra prediction direction of a neighboring block, and the second parameter is used as a partition parameter to indicate the direction to split the image block into two triangles. Such a combination of the first and second parameters can be used to indicate, for example, 1) the direction to split the image block into two triangles is from the top-right corner to the bottom-left corner when the intra prediction direction of the neighboring block is an anti-diagonal direction.

[0317] It should be appreciated that, as Figure 15 , 24As shown in Figure 25, the table including partitioning parameters and which information is jointly or individually encoded is presented merely as an example, and many other ways of jointly or individually encoding various information as part of the partitioning syntax operations described above are also within the scope of this disclosure. For example, partitioning parameters may indicate that the first partition is a triangle, trapezoid, or a polygon with at least five sides and angles. Partitioning parameters may indicate that the second partition has a non-rectangular shape, such as a triangle, trapezoid, or a polygon with at least five sides and angles. Partitioning parameters may indicate one or more pieces of information about the segmentation, such as the non-rectangular shape of the first partition, the shape of the second partition (which may be non-rectangular or rectangular), and the segmentation direction applied to segment the image patch into multiple partitions (e.g., from the upper left corner of the image patch to its lower right corner, and from the upper right corner of the image patch to its lower left corner). Partitioning parameters may jointly encode other information, such as a first motion vector of the first partition, a second motion vector of the second partition, the image patch size, the prediction mode, the motion vectors of neighboring patches, the intra-frame prediction directions of neighboring patches, etc. Alternatively, any information in the further information may be encoded separately by one or more parameters other than the partitioning parameters.

[0318] Similar to Figure 22 The process 4000, partitioning syntax operations can be performed by, for example Figure 1 The image encoder shown includes circuitry and a memory coupled to the circuitry. The circuitry performs a partitioning syntax operation during operation, which includes: dividing an image patch into multiple partitions based on partitioning parameters indicating the segmentation, the multiple partitions including a second partition and a first partition having a non-rectangular shape. Figure 22 Step S4001); Encode the first partition and the second partition (S4002); Write one or more parameters, including partition parameters, into the bit stream (S4003).

[0319] According to another embodiment, such as Figure 1As shown, there is provided an image encoder comprising: a partitioner 102 that in operation receives an original picture and partitions it into blocks; a subtractor 104 that in operation receives blocks from the partitioner and predictions from a prediction controller 128 and subtracts each prediction from its corresponding block to output a residual; a transformer 106 that in operation performs a transform on the residuals output from the subtractor 104 to output transform coefficients; a quantizer 108 that in operation quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder 110 that in operation encodes the quantized transform coefficients to generate a bitstream; and a prediction controller 128 coupled to an inter-predictor 126, an intra-predictor 124 and memories 118, 122, wherein the inter-predictor 126 in operation generates a prediction of a current block based on a reference block in an encoded reference picture, the intra-predictor 124 in operation generates a prediction of a current block based on an encoded reference block in a current picture. The prediction controller 128 in operation partitions an image block into a plurality of partitions based on partition parameters indicative of the partitioning, the plurality of partitions including a second partition and a first partition having a non-rectangular shape Figure 22 , step S4001), and encoding the first and second partitions (step S4002). The entropy encoder 110 in operation writes one or more parameters including the partition parameters into the bitstream (step S4003).

[0320] According to another embodiment, there is provided an image decoder, e.g. as Figure 10 As shown, the image decoder comprises circuitry and a memory coupled to the circuitry. The circuitry in operation performs a partition syntax operation comprising: parsing one or more parameters from a bitstream, wherein the one or more parameters include partition parameters indicative of partitioning an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape Figure 23 , step S5001); partitioning the image block into the plurality of partitions based on the partition parameters (S5002); decoding the first and second partitions (S5003).

[0321] According to a further embodiment, there is provided an image decoder as Figure 10The image decoder shown in FIG. 2 includes an entropy decoder 202 that in operation receives and decodes an encoded bitstream to obtain quantized transform coefficients, an inverse quantizer 204 and an inverse transformer 206 that in operation inverse quantize the quantized transform coefficients to obtain transform coefficients and inverse transform the transform coefficients to obtain residuals, an adder 208 that in operation adds the residuals output from the inverse quantizer 204 and inverse transformer 206 to predictions output from a prediction controller 220 to reconstruct a block, and the prediction controller 220 coupled to an inter-predictor 218, an intra-predictor 216, and memories 210, 214, where the inter-predictor 218 in operation generates a prediction for a current block based on a reference block in a decoded reference picture and the intra-predictor 216 in operation generates a prediction for the current block based on a decoded reference block in a current picture. The entropy decoder 202 in operation parses one or more parameters from the bitstream, where the one or more parameters include partition parameters indicating partitioning of an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape (S5001); partitions the image block into the plurality of partitions based on the partition parameters (S5002); and decodes the first partition and the second partition in cooperation with the prediction controller 220 (S5003) in some implementations. Figure 23

[0322] According to other examples, the intra-predictor can perform the following process.

[0323] All of the motion vector candidates included in the first set of motion vector candidates can be single-prediction motion vectors. That is, the intra-predictor can only determine single-prediction motion vectors as motion vector candidates in the first set of motion vector candidates.

[0324] The intra-predictor can only select single-prediction motion vector candidates from the first set of motion vector candidates.

[0325] Only single-prediction motion vectors can be used to predict small blocks. Double-prediction motion vectors can be used to predict large blocks. For example, the prediction process can include determining a size of the image block. When the size of the image block is determined to be greater than a threshold, the prediction can include selecting a first motion vector from the first set of motion vector candidates, and the first set of motion vector candidates can contain single-prediction and / or double-prediction motion vectors. When the size of the image block is determined to be not greater than the threshold, the prediction can select a first motion vector from the first set of motion vector candidates, and the first set of motion vector candidates can contain only single-prediction motion vectors.

[0326] (Implementation and Application)

[0327] ​As described in each of the above embodiments, each functional or operational block can generally be implemented as, for example, an MPU (micro processing unit) and a memory. Furthermore, the process performed by each of the functional blocks can be implemented as a program execution unit, such as a processor that reads and executes software (a program) recorded on a recording medium such as a ROM. The software can be distributed. The software can be recorded on various recording media such as a semiconductor memory. Note that each functional block can also be implemented as hardware (a dedicated circuit).

[0328] The processes described in each of the embodiments can be implemented by using integrated processing of a single apparatus (system), or can be implemented by using decentralized processing of a plurality of apparatuses. Furthermore, the processor that executes the above-described program can be a single processor or a plurality of processors. In other words, integrated processing can be performed, or decentralized processing can be performed.

[0329] Embodiments of the present disclosure are not limited to the above example embodiments; various modifications can be made to the example embodiments, and the results thereof are also included in the scope of the embodiments of the present disclosure.

[0330] Next, application examples of the moving picture encoding method (image encoding method) and the moving picture decoding method (image decoding method) described in each of the above embodiments, and various systems that implement the application examples will be described. Such a system can be characterized by including an image encoder that employs the image encoding method, an image decoder that employs the image decoding method, or both an image encoder and an image decoder. Other configurations of such a system can be modified as appropriate.

[0331] (Usage Examples)

[0332] Figure 26 The overall configuration of a content providing system ex100 suitable for implementing a content distribution service is shown. The area in which a communication service is provided is divided into cells of a desired size, and in the example shown, base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations, are located in the respective cells.

[0333] In the content providing system ex100, devices including a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via the Internet service provider ex102 or the communication network ex104 and the base stations ex106 to ex110. The content providing system ex100 can combine and connect any combination of the above-described devices. In various implementations, the devices can be connected together directly or indirectly via a telephone network or near field communication instead of via the base stations ex106 to ex110. Furthermore, the streaming server ex103 can be connected to the devices including the computer ex111, the game device ex112, the camera ex113, the home appliance ex114, and the smartphone ex115 via, for example, the Internet ex101. The streaming server ex103 can also be connected to terminals in a hotspot in, for example, an airplane ex117 via the satellite ex116.

[0334] Note that wireless access points or hotspots can be used instead of the base stations ex106 to ex110. The streaming server ex103 can be directly connected to the communication network ex104 instead of through the Internet ex101 or the Internet service provider ex102, and can be directly connected to the airplane ex117 instead of through the satellite ex116.

[0335] The camera ex113 is a device capable of capturing still images and videos, such as a digital camera. The smartphone ex115 is a smartphone device, a cellular phone, or a Personal Handyphone System (PHS) phone that can operate under the standards of mobile communication systems of 2G, 3G, 3.9G, and 4G systems and the next-generation 5G system.

[0336] The home appliance ex114 is, for example, a refrigerator or a device included in a home fuel cell cogeneration system.

[0337] In the content providing system ex100, a terminal including an image and / or video capturing function is capable of real-time streaming, for example, by connecting to the streaming server ex103 via, for example, the base station ex106. When real-time streaming is performed, the terminal (for example, the computer ex111, the game device ex112, the camera ex113, the home appliance ex114, the smartphone ex115, or the airplane ex117) can perform the encoding process described in the above-described embodiments on still image or video content captured by a user via the terminal, can multiplex video data obtained via encoding and audio data obtained by encoding audio corresponding to the video, and can transmit the obtained data to the streaming server ex103. In other words, the terminal functions as an image encoder according to an aspect of the present disclosure.

[0338] The streaming server ex103 streams the transmitted content data to a client that requests the stream. Examples of the client include a computer ex111, a game device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and a terminal in an airplane ex117, which are capable of decoding the above-mentioned encoded data. The device that receives the streamed data decodes and reproduces the received data. In other words, each of the devices can function as an image decoder according to an aspect of the present disclosure.

[0339] (Decentralized processing)

[0340] The streaming server ex103 can be implemented as a plurality of servers or computers that divide tasks such as processing, recording, and streaming of data among them. For example, the streaming server ex103 can be implemented as a content delivery network (CDN) that streams content via a network connecting a plurality of edge servers located around the world. In the CDN, an edge server that is physically close to a client is dynamically assigned to the client. Content is cached and streamed to the edge server to reduce loading time. For example, in the event of some type of error or change in connectivity due to a traffic peak, it is possible to stably stream data at high speed because it is possible to avoid the affected part of the network by, for example, dividing the processing among a plurality of edge servers, or switching the streaming responsibility to a different edge server and continuing the streaming.

[0341] Decentralization is not limited to dividing processing for streaming; encoding of captured data can be allocated among terminals on the server side, or performed by the terminals, or both. In one example, in typical encoding, the processing is performed in two loops. The first loop is used to detect the complexity of the image on a frame-by-frame or scene-by-scene basis, or to detect the encoding load. The second loop is used for processing that maintains image quality and improves encoding efficiency. For example, it is possible to reduce the processing responsibility of the terminal and improve the quality and encoding efficiency of the content by having the terminal perform the first loop of encoding, and having the server side that receives the content perform the second loop of encoding. In such a case, upon receiving a decoding request, it is possible to receive and reproduce the encoded data produced as a result of the first loop performed on one terminal on another terminal in near real time. It is thus possible to achieve smooth, real-time streaming.

[0342] In another example, the camera ex113 or the like extracts a feature amount from an image, compresses data related to the feature amount as metadata, and transmits the compressed metadata to the server. For example, the server determines the importance of an object based on the feature amount, and changes quantization precision accordingly to perform compression suitable for the meaning (or content importance) of the image. The feature amount data is particularly effective in improving the accuracy and efficiency of motion vector prediction during a second compression process performed by the server. Furthermore, the terminal can process encoding with relatively low processing load such as variable length coding (VLC), and encoding with relatively high processing load such as context adaptive binary arithmetic coding (CABAC) can be processed by the server.

[0343] In yet another example, there is a case in which a plurality of terminals capture a plurality of videos of approximately the same scene, for example, in a stadium, a shopping center, or a factory. In such a case, for example, encoding can be decentralized by dividing processing tasks on a per-unit basis among the plurality of terminals that capture videos and, if necessary, other terminals that do not capture videos, and the server. The unit can be, for example, a group of pictures (GOP), a picture, or a tile obtained by dividing a picture. This can reduce the loading time and enable streaming closer to real time.

[0344] Since the videos have approximately the same scene, the server can perform management and / or instruction so that the videos captured by the terminals can be cross-referenced. Furthermore, the server can receive encoded data from the terminals, change the reference relationship between data items, or correct or replace pictures itself, and then perform encoding. This makes it possible to generate a stream with improved quality and efficiency for individual data items.

[0345] Furthermore, the server can stream video data after performing transcoding to convert the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (e.g., VP9), and can convert H.264 to H.265.

[0346] In this way, encoding can be performed by the terminal or the server or servers. Therefore, although the device that performs encoding is referred to as a "server" or a "terminal" in the description below, some or all of the processes performed by the server can be performed by the terminal, and likewise, the processing performed by the terminal can be performed by the server. The same applies to the decoding process.

[0347] (3D, multi-angle)

[0348] The use of images or videos combined from images or videos of different scenes captured at the same time or from the same scene captured from different angles has increased through a plurality of terminals (for example, a camera ex113 and / or a smartphone ex115). Videos captured by terminals are combined based on, for example, a relative positional relationship individually obtained between terminals or a region having matching feature points in a video.

[0349] In addition to encoding two-dimensional moving pictures, the server can automatically or at a user-specified point in time encode still images based on scene analysis of the moving pictures and transmit the encoded still images to the receiving terminal. Furthermore, when the server is able to obtain a relative positional relationship between video-capturing terminals, in addition to two-dimensional moving pictures, the server can generate a three-dimensional geometry of a scene based on videos of the same scene captured from different angles. The server can encode three-dimensional data generated from, for example, a point cloud, respectively, and based on a result of recognizing or tracking a person or an object using the three-dimensional data, the server can select or reconstruct and generate a video to be transmitted to the receiving terminal from videos captured by a plurality of terminals.

[0350] This allows a user to enjoy a scene by freely selecting a video corresponding to a video-capturing terminal, and allows a user to enjoy content obtained by extracting a video at a selected viewpoint from three-dimensional data reconstructed from a plurality of images or videos. Furthermore, as with a video, sound can be recorded from relatively different angles, and the server can multiplex audio of a specific angle or space with a corresponding video, and transmit the multiplexed video and audio.

[0351] In recent years, content composed of a real world and a virtual world, such as virtual reality (VR) and augmented reality (AR) content, has also become popular. In the case of a VR image, the server can create an image from the viewpoint of both the left eye and the right eye, and perform encoding that allows reference between the two viewpoint images, such as multi-view encoding (MVC), or can encode the images as separate streams without reference. When the images are decoded as separate streams, the streams can be synchronized at the time of reproduction in order to recreate a virtual three-dimensional space according to the viewpoint of a user.

[0352] In the case of an AR image, the server superimposes virtual object information existing in a virtual space onto camera information representing a real space based on a three-dimensional position or motion from the user's perspective. The decoder can acquire or store virtual object information and three-dimensional data, generate a two-dimensional image based on motion from the user's perspective, and then generate superimposed data by seamlessly connecting the images. Alternatively, the decoder can transmit motion from the user's perspective to the server in addition to a request for virtual object information. The server can generate superimposed data based on three-dimensional data stored in the server according to the received motion, and encode and stream the generated superimposed data to the decoder. Note that the superimposed data includes an alpha value representing transparency in addition to an RGB value, and the server sets the alpha value of a portion other than an object generated from three-dimensional data to, for example, 0, and can be encoded while the portion is transparent. Alternatively, the server can set a background to a predetermined RGB value (e.g., chroma key), and generate data setting a region other than the object to the background.

[0353] Decoding of similarly streamed data can be performed at the server side by the client (i.e., terminal), or divided between them. In one example, one terminal can send a reception request to the server, the requested content can be received and decoded by another terminal, and the decoded signal can be transmitted to a device having a display. Regardless of the processing capability of the communication terminal itself, high image quality data can be reproduced by decentralized processing and appropriate selection of content. In yet another example, when a TV is receiving large size image data, for example, regions of a picture such as tiles obtained by dividing the picture can be decoded and displayed on a personal terminal or a terminal of one or more viewers of the TV. This enables viewers to share a large picture view, and each viewer can check its assigned region, or view the region more closely.

[0354] In a case where a plurality of wireless connections can be performed indoors, outdoors, at close range, at medium range, and at long range, a streaming system standard such as MPEG-DASH can be used to seamlessly receive content. A user can switch between data in real time while freely selecting a decoder or a display device including a user terminal, a display arranged indoors or outdoors, or the like. Furthermore, using, for example, information on a user's location, it is possible to perform decoding while switching which terminal processes decoding and which terminal processes display of content. This makes it possible to draw and display information on a wall of a nearby building or on a portion of the ground that is embedded with a device capable of displaying content when a user is on the way to a destination. Furthermore, it is also possible to switch the bit rate of received data based on the accessibility of encoded data on a network, such as when encoded data is cached on a server that is quickly accessible from a receiving terminal or when encoded data is copied to an edge server in a content delivery service.

[0355] (scalable encoding)

[0356] Switching of content will be described with reference to a scalable stream shown in Figure 27 , which is compressed and encoded by implementation of the moving picture encoding method described in the above-described embodiments. A server can have a configuration in which content is switched while utilizing temporal and / or spatial scalability of a stream, which is achieved by dividing and encoding layers, as shown in Figure 27 . Note that there can be a plurality of individual streams that are the same in content but different in quality. In other words, by determining a layer to be decoded based on internal factors such as processing capability on the decoder side and external factors such as communication bandwidth, the decoder side can freely switch between low-resolution content and high-resolution content while decoding. For example, in a case where a user wants to continue watching a video that the user has previously watched on a smartphone ex115 while moving, using a device such as a television connected to the Internet, for example, in the user's home, the device can simply decode the same stream up to a different layer, which reduces the load on the server side.

[0357] Furthermore, in addition to the above-described configuration in which scalability is achieved by encoding pictures by layer, and in addition to the fact that an enhancement layer is located above a base layer, the enhancement layer can also include metadata based on, for example, statistical information on an image. The decoder side can generate content of high image quality by performing super-resolution imaging on a picture in the base layer based on the metadata. Super-resolution imaging can improve a signal-to-noise ratio while maintaining resolution and / or improving resolution. The metadata includes information for identifying linear or nonlinear filter coefficients used in super-resolution processing, or information for identifying parameter values of filter processing, machine learning, or least squares used in super-resolution processing.

[0358] Alternatively, a configuration can be provided in which a picture is divided into, for example, tiles according to the meaning of the objects in the image. At the decoder side, only part of the area is decoded by selecting the tile to be decoded. Furthermore, by storing the attributes of the objects (person, car, ball, etc.) and the position of the objects in the video (coordinates in the same image) as metadata, the decoder side can identify the position of the desired object based on the metadata and determine which tile or tiles contain the object. For example, as shown in Figure 28 the metadata can be stored using a data storage structure different from the pixel data, such as the SEI (Supplemental Enhancement Information) message in HEVC. This metadata indicates, for example, the position, size, or color of the main object.

[0359] The metadata can be stored in units of multiple pictures, such as a stream, sequence, or random access unit. The decoder side can obtain, for example, the time at which a certain person appears in the video, and by fitting the time information with the picture unit information, the picture in which the object is present can be identified, and the position of the object in the picture can be determined.

[0360] (Web page optimization)

[0361] Figure 29 An example of a display screen of a web page on, for example, a computer ex111 is shown. Figure 30 An example of a display screen of a web page on, for example, a smartphone ex115 is shown. As shown in Figure 29 and Figure 30 The web page can include a plurality of image links that are links to image content, and the appearance of the web page differs depending on the device used to view the web page, as shown in FIGS. 11A to 11C. When a plurality of image links can be seen on the screen, the display device (decoder) can display a still image included in the content or I picture as the image link until the user explicitly selects the image link, or until the image link is located at the approximate center of the screen, or until the entire image link fits on the screen; a plurality of still images or I pictures can be used to display a video such as an animated gif; or only the base layer can be received, and the video can be decoded and displayed.

[0362] When the user selects the image link, the display device performs decoding while giving the highest priority to the base layer. Note that if there is information indicating that the content is scalable in the HTML code of the web page, the display device can decode up to the enhancement layer. Furthermore, in order to guarantee real-time reproduction, the display device can reduce the delay between the point in time at which the leading pictures are decoded and the point in time at which the decoded pictures are displayed (i.e., the delay between the start of decoding of the content and the display of the content) by decoding and displaying only forward reference pictures (I pictures, P pictures, forward reference B pictures) before the selection is made or when the bandwidth is strictly limited. In addition, the display device can intentionally ignore the reference relationship between pictures and roughly decode all B and P pictures as forward reference pictures, and then perform regular decoding as the number of pictures received over time increases.

[0363] (Autonomous driving)

[0364] When transmitting and receiving still image or video data (e.g., two-dimensional or three-dimensional map information for autonomous or assisted driving of a car), the receiving terminal can receive, in addition to the image data belonging to one or more layers, information about, for example, the weather or road construction as metadata, and associate the metadata with the image data at the time of decoding. Note that the metadata can be distributed by layer, and can alternatively be multiplexed with the image data only.

[0365] In such a case, since the car, drone, airplane, or the like that contains the receiving terminal is movable, the receiving terminal can seamlessly receive and perform decoding while switching between the base stations in the base stations ex106 to ex110 by transmitting information indicating the position of the receiving terminal. Furthermore, the receiving terminal can dynamically select the degree of reception of metadata or the degree of update of, for example, map information according to the user's selection, the user's situation, and / or the bandwidth of the connection.

[0366] In the content providing system ex100, the client can receive, decode, and reproduce the encoded information transmitted by the user in real time.

[0367] (Streaming individual content)

[0368] In the content providing system ex100, in addition to high image quality, long content distributed by a video distribution entity, unicast or multicast streaming of low image quality, and short content from individuals are also possible. The popularity of such content from individuals is likely to further increase. The server can first perform an editing process on the content before the encoding process in order to refine the individual content. For example, this can be achieved using the following configuration.

[0369] When real-time capturing video or image contents, or after capturing and accumulating the contents, the server performs recognition processing based on the original data or the encoded data, such as capturing error processing, scene search processing, meaning analysis, and / or object detection processing. Then, based on the results of the recognition processing, the server edits the contents when prompted or automatically, examples of which include correction such as focus and / or motion blur correction, removal of low-priority scenes such as scenes with low brightness compared to other pictures, or scenes with focus out, object edge adjustment, and tone adjustment. The server encodes the edited data based on the results of the editing. As is well known, videos that are too long result in reduced viewing times. Thus, in order to keep the contents within a specific length comparable to the length of the original video, in addition to the low-priority scenes described above, the server can automatically crop scenes with low motion based on the results of the image processing. Alternatively, the server can generate and encode a video summary based on the results of the analysis of the meaning of the scenes.

[0370] In some cases, individual contents can include contents that infringe copyrights, moral rights, portrait rights, and the like. Such instances can put the creator in an unfavorable position, for example, when the creator intends to share the contents beyond the scope. Thus, before encoding, the server can, for example, edit the images so that the faces of people outside the periphery of the screen are blurred or so that the interior of a house is blurred. In addition, the server can be configured to recognize the faces of people other than registered people in the images to be encoded, and when such a face appears in the image, a mosaic filter can be applied to, for example, the face of the person. Alternatively, as pre-processing or post-processing for encoding, the user can designate an image region including a person or a background to be processed for copyright reasons. The server can process the designated region by, for example, replacing the region with a different image or blurring the region. If the region includes a person, the person can be tracked in the moving picture, and the head region of the person can be replaced with another image as the person moves.

[0371] Since there is a need to watch the contents produced by individuals in real time, which tend to be small in data size, the decoder first receives the base layer as the highest priority and performs decoding and reproduction, but this can vary depending on the bandwidth. When the contents are reproduced twice or more, for example, when the decoder receives the enhancement layer during decoding and reproduction of the base layer and loops the reproduction, the decoder can reproduce a video with high image quality including the enhancement layer. If such a scalable encoding is used to encode the stream, the video can be low in quality when in an unselected state or at the beginning of the video, but it can provide an experience in which the image quality of the stream is gradually improved in an intelligent manner. This is not limited to scalable encoding; a single stream can be configured by a low-quality stream from the first reproduction and a second stream encoded using the first stream as a reference, thereby providing the same experience.

[0372] (Other implementation and application examples)

[0373] Encoding and decoding can be performed by an LSI (Large Scale Integration) ex500 (see below), which is typically included in each terminal. Figure 26 The LSI ex500 can be configured as a single chip or multiple chips. Software for encoding and decoding moving images can be integrated into a type of recording medium (e.g., CD-ROM, floppy disk, or hard disk), which a computer ex111 can read and use to perform encoding and decoding. Furthermore, when the smartphone ex115 is equipped with a camera, video data acquired by that camera can be transmitted. In this case, the video data is encoded by the LSI ex500 included in the smartphone ex115.

[0374] Note that the LSIex500 can be configured to download and activate applications. In this case, the terminal first determines whether it is compatible with the encoding scheme used to encode the content, or whether it can perform a specific service. When the terminal is incompatible with the content's encoding scheme, or when the terminal cannot perform a specific service, the terminal first downloads the codec or application software, and then acquires and reproduces the content.

[0375] Besides the example of the content delivery system ex100 using the Internet ex101, the motion picture encoder (image encoder) or motion picture decoder (image decoder) described in at least the above embodiments can be implemented in a digital broadcasting system. The same encoding and decoding processes can be applied, for example, to transmitting and receiving broadcast radio waves superimposed with multiplexed audio and video data using satellites, even if this is for multicast, whereas unicast is easier for the content delivery system ex100.

[0376] (Hardware Configuration)

[0377] Figure 31 It shows Figure 26 More details about the smartphone ex115 shown. Figure 32A configuration example of a smartphone ex115 is shown. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves to and from the base station ex110, a camera ex465 capable of capturing video and still images, and a display ex458 for displaying decoded data, such as video captured by the camera ex465 and video received by the antenna ex450. The smartphone ex115 also includes a user interface ex466 such as a touch panel, an audio output unit ex457 such as a speaker for outputting voice or other audio, an audio input unit ex456 such as a microphone for audio input, a memory ex467 capable of storing decoded data such as captured video or still images, recorded audio, received video or still images, mail, and decoded data, and a slot ex464, which is an interface for a SIM ex468, for authorizing access to a network and various data. Note that an external memory can be used instead of ex467.

[0378] A main controller ex460, which can comprehensively control the display ex458 and the user interface ex466, a power supply circuit ex461, a user interface input controller ex462, a video signal processor ex455, a camera interface ex463, a display controller ex459, a modulator / demodulator ex452, a multiplexer / demultiplexer ex453, an audio signal processor ex454, the slot ex464, and the memory ex467 are connected by a bus ex470.

[0379] When the user turns on the power button of the power supply circuit ex461, the smartphone ex115 is powered on to an operable state, and each component is powered from a battery pack.

[0380] The smartphone ex115 performs processing such as calling and data transmission based on control performed by a main controller ex460 including a CPU, a ROM, and a RAM. When a call is made, an audio signal recorded by an audio input unit ex456 is converted into a digital audio signal by an audio signal processor ex454, and spread spectrum processing is performed on the digital audio signal by a modulator / demodulator ex452 and digital-to-analog conversion is performed, and frequency conversion processing is performed by a transmitter / receiver ex451, and the resulting signal is transmitted via an antenna ex450. Received data is amplified, frequency-converted, and analog-to-digital-converted, inverse spread spectrum-processed by the modulator / demodulator ex452, converted into an analog audio signal by the audio signal processor ex454, and then output from an audio output unit ex457. In a data transmission mode, for example, based on operation of a user interface ex466 of the main body, the main controller ex460 transmits text, still images, or video data via a user interface input controller ex462. Similar transmission and reception processing is performed. In the data transmission mode, when video, still images, or video and audio are transmitted, a video signal processor ex455 compressively encodes a video signal stored in a memory ex467 or a video signal input from a camera ex465 via the moving picture encoding method described in the above-described embodiment, and transmits the encoded video data to a multiplexer / demultiplexer ex453. The audio signal processor ex454 encodes an audio signal recorded by the audio input unit ex456 while the camera ex465 is capturing video or still images, and transmits the encoded audio data to the multiplexer / demultiplexer ex453. The multiplexer / demultiplexer ex453 multiplexes the encoded video data and the encoded audio data using a predetermined scheme, modulates and converts the data using the modulator / demodulator (modulator / demodulator circuit) ex452 and the transmitter / receiver ex451, and transmits the result via the antenna ex450.

[0381] For example, when a video attached in an e-mail or a chat or a video linked from a web page is received, in order to decode multiplexed data received via the antenna ex450, the multiplexer / demultiplexer ex453 demultiplexes the multiplexed data to divide the multiplexed data into a bit stream of video data and a bit stream of audio data, supplies the encoded video data to the video signal processor ex455 via the synchronous bus ex470, and supplies the encoded audio data to the audio signal processor ex454 via the synchronous bus ex470. The video signal processor ex455 decodes the video signal using a motion picture decoding method corresponding to the motion picture encoding method described in the above-described embodiments, and a video or still image included in a linked motion picture file is displayed on the display ex458 via the display controller ex459. The audio signal processor ex454 decodes the audio signal, and outputs the audio from the audio output unit ex457. Since real-time streaming is becoming more and more popular, in some cases the reproduction of audio can be socially inappropriate according to the environment of the user. Thereby, as an initial value, a configuration in which only the video data is reproduced and the audio signal is not reproduced is preferable; the audio can be synchronized and reproduced only when an input is received (for example, when the user clicks the video data).

[0382] Although a smartphone ex115 is used in the above example, other three implementations are conceivable: a transceiver terminal including both an encoder and a decoder; a transmitter terminal including only an encoder; a receiver terminal including only a decoder. In the description of the digital broadcasting system, an example is given in which multiplexed data obtained because video data and audio data are multiplexed is received or transmitted. However, the multiplexed data can be video data multiplexed with data other than audio data (for example, text data related to video). Furthermore, video data itself can be received or transmitted instead of multiplexed data.

[0383] Although the main controller ex460 including a CPU is described as controlling the encoding or decoding process, various terminals generally include a GPU. Thereby, a configuration in which a memory shared via a CPU and a GPU or a memory including addresses managed to allow general use of the CPU and the GPU is acceptable, by immediately processing a large area by utilizing the performance capability of the GPU. This can shorten the encoding time, maintain the real-time property of the stream, and reduce the delay. In particular, processing related to motion estimation, deblocking filtering, sample adaptive offset (SAO), and transform / quantization can be efficiently carried out all at once in units of pictures, for example, by the GPU instead of the CPU.

Claims

1. An image encoding method comprising: obtaining a picture block from a coding tree unit (CTU); selecting, for a first partition in the picture block having a non-rectangular shape, a first motion vector from a set of motion vector candidates; selecting, for a second partition in the picture block, a second motion vector from the set of motion vector candidates; encoding the first partition using the first motion vector; and encoding the second partition using the second motion vector, only uni-prediction motion vectors are selected from the set of motion vector candidates.

2. An image decoding method comprising: obtaining a picture block from a coding tree unit (CTU); selecting, for a first partition in the picture block having a non-rectangular shape, a first motion vector from a set of motion vector candidates; selecting, for a second partition in the picture block, a second motion vector from the set of motion vector candidates; decoding the first partition using the first motion vector; and decoding the second partition using the second motion vector; only uni-prediction motion vectors are selected from the set of motion vector candidates.

3. A bitstream generation apparatus comprising: circuitry; and a memory coupled to the circuitry; wherein the circuitry is operative to: generate information that causes a decoder to perform a partitioning process in accordance therewith, and include the information in a bitstream, in the partitioning process: a first motion vector for a first partition in a picture block having a non-rectangular shape is selected from a set of motion vector candidates; a second motion vector for a second partition in the picture block is selected from the set of motion vector candidates; the first partition is decoded using the first motion vector; and the second partition is decoded using the second motion vector; only uni-prediction motion vectors are selected from the set of motion vector candidates. ​ ​ ​

Images