Image encoder, image decoder, and bitstream generation apparatus

Abstract

An image encoder, an image decoder, and a non-transitory computer-readable medium are provided. The image encoder includes: circuitry; and a memory coupled to the circuitry; wherein the circuitry, in operation, performs the following operations: obtaining an image block from a coding tree unit (CTU); selecting a first motion vector from a set of motion vector candidates for a first partition having a non-rectangular shape in the image block; selecting a second motion vector from the set of motion vector candidates for a second partition in the image block that overlaps with the first partition; predicting a first value for the first partition using the first motion vector; encoding the first partition using the first motion vector; and encoding the second partition using the second motion vector, wherein the circuitry selects only a single predicted motion vector from the set of motion vector candidates.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 201880054452.4 (International Application No. PCT / JP2018 / 030059), filed on August 10, 2018, entitled "Image Encoder, Image Decoder, Image Encoding Method and Image Decoding Method". Technical Field

[0002] This disclosure relates to video coding, and more particularly to video coding and decoding systems, components, and methods for performing inter-frame prediction functions that construct the current block based on a reference frame or intra-frame prediction functions that construct the current block based on an encoded / decoded reference block in the current frame. Background Technology

[0003] With the development of video coding technologies from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High-Efficiency Video Coding), and H.266 / VVC (Various Video Codecs), there is a constant need to improve and optimize video coding technologies to handle the ever-increasing volume of digital video data in various applications. This disclosure relates to further advancements, improvements, and optimizations in video coding, particularly in the incorporation of inter-frame prediction or intra-frame prediction functions that segment image blocks into multiple partitions, said partitions including at least a first partition having a non-rectangular shape (e.g., a triangle) and a second partition. Summary of the Invention

[0004] According to one aspect, an image encoder includes: circuitry; and a memory coupled to the circuitry; wherein the circuitry, in operation, performs the following operations: obtaining an image patch from a coding tree unit (CTU); selecting a first motion vector from a set of motion vector candidates for a first partition having a non-rectangular shape in the image patch; selecting a second motion vector from the set of motion vector candidates for a second partition in the image patch that overlaps with the first partition; predicting a first value for the first partition using the first motion vector; encoding the first partition using the first motion vector; and encoding the second partition using the second motion vector, wherein the circuitry selects only a single predicted motion vector from the set of motion vector candidates.

[0005] Furthermore, an image decoder is provided, comprising: circuitry; and a memory coupled to the circuitry; wherein the circuitry performs the following operations in operation: obtaining an image patch from a coding tree unit (CTU); selecting a first motion vector from a set of motion vector candidates for a first partition having a non-rectangular shape in the image patch; selecting a second motion vector from the set of motion vector candidates for a second partition in the image patch that overlaps with the first partition; predicting a first value for the first partition using the first motion vector; decoding the first partition using the first motion vector; and decoding the second partition using the second motion vector; wherein the circuitry selects only a single predicted motion vector from the set of motion vector candidates.

[0006] Furthermore, a non-transitory computer-readable medium is provided that stores a bitstream including information that causes a decoder to perform partitioning processing, in which: an image block is obtained from a coding tree unit (CTU); a first motion vector for a first partition having a non-rectangular shape in the image block is selected from a set of motion vector candidates; a second motion vector for a second partition in the image block that overlaps with the first partition is selected from the set of motion vector candidates; a first value for the first partition is predicted using the first motion vector; the first partition is decoded using the first motion vector; and the second partition is decoded using the second motion vector; and a single predicted motion vector is selected from the set of motion vector candidates.

[0007] Furthermore, an image encoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry performs the following operations in operation: segmenting an image block into a first partition comprising a second partition and a first partition having a non-rectangular shape; predicting a first motion vector for the first partition and a second motion vector for the second partition; and encoding the first partition using the first motion vector and encoding the second partition using the second motion vector.

[0008] Some implementations of the embodiments of this disclosure can improve encoding efficiency, can be simply the encoding / decoding process, can accelerate the encoding / decoding process, and can efficiently select appropriate components / operations used in encoding and decoding, such as appropriate filters, block sizes, motion vectors, reference images, reference blocks, etc.

[0009] Other benefits and advantages of the disclosed embodiments will become apparent from the description and drawings. Benefits and / or advantages may be obtained individually from the various embodiments and features in the description and drawings, and not all of them need to be provided to obtain one or more of such benefits and / or advantages.

[0010] It should be noted that general or specific embodiments can be implemented as systems, methods, integrated circuits, computer programs, storage media, or any alternative combination thereof. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating the functional configuration of an encoder according to an embodiment.

[0012] Figure 2 An example of block splitting is shown.

[0013] Figure 3 It is a table indicating the fundamental transformation functions for various transformation types.

[0014] Figure 4A An example of the filter shape used in ALF (Adaptive Loop Filter) is shown.

[0015] Figure 4B Another example of filter shapes used in ALF is shown.

[0016] Figure 4C Another example of filter shapes used in ALF is shown.

[0017] Figure 5A The example shows 67 intra-prediction modes used in intra-prediction.

[0018] Figure 5B This is a flowchart illustrating an example of the predictive image correction process performed in OBMC (Overlapping Block Motion Compensation) processing.

[0019] Figure 5C This is a conceptual diagram illustrating an example of the predictive image correction process performed in OBMC processing.

[0020] Figure 5D This is a flowchart illustrating an example of FRUC (Frame Rate Upconversion) processing.

[0021] Figure 6 An example of pattern matching (bidirectional matching) between two blocks along a motion trajectory is shown.

[0022] Figure 7 This shows an example of pattern matching (template matching) between a template in the current image and a block in the reference image.

[0023] Figure 8 A model assuming uniform linear motion is shown.

[0024] Figure 9A An example is shown where the motion vector of each sub-block is derived from the motion vectors of adjacent blocks.

[0025] Figure 9B An example of the process for deriving motion vectors in a merged mode is shown.

[0026] Figure 9C This is a conceptual diagram illustrating an example of DMVR (Dynamic Motion Vector Refresh) processing.

[0027] Figure 9D An example of a predictive image generation method using a brightness correction process performed by LIC (Local Illumination Compensation) processing is shown.

[0028] Figure 10 This is a block diagram illustrating the functional configuration of the decoder according to an embodiment.

[0029] Figure 11 This is a flowchart illustrating an overall processing flow according to one embodiment of dividing an image patch into multiple partitions, including at least a first partition and a second partition having a non-rectangular shape (e.g., a triangle), and performing further processing.

[0030] Figure 12 Two exemplary methods are shown to segment an image patch into a second partition (which also has a non-rectangular shape in the example shown) and a first partition with a non-rectangular shape (e.g., a triangle).

[0031] Figure 13 An example of a boundary smoothing process is shown, which includes weighting a first value of a boundary pixel predicted based on a first partition and a second value of a boundary pixel predicted based on a second partition.

[0032] Figure 14 Three more examples of the boundary smoothing process are shown, which involve weighting a first value of the boundary pixels predicted based on a first partition and a second value of the boundary pixels predicted based on a second partition.

[0033] Figure 15 It is a table of sampling parameters (“first index value”) and information sets encoded by the parameters.

[0034] Figure 16 It is a table that shows the binary representation of the parameters (index values).

[0035] Figure 17 This is a flowchart illustrating the process of dividing an image patch into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape.

[0036] Figure 18 An example of segmenting an image patch into multiple partitions is shown, the multiple partitions including a second partition and a first partition having a non-rectangular shape (triangle in the example shown).

[0037] Figure 19 A further example of segmenting an image patch into multiple partitions is shown, the multiple partitions including a second partition and a first partition having a non-rectangular shape, which in the illustrated example is a polygon with at least five sides and corners.

[0038] Figure 20 The flowchart illustrates a boundary smoothing process that includes weighting a first value of the boundary pixels predicted based on a first partition and a second value of the boundary pixels predicted based on a second partition.

[0039] Figure 21A An example of a boundary smoothing process is shown, in which the boundary pixels whose first value is to be weighted are predicted based on a first partition and whose second value is to be weighted are predicted based on a second partition.

[0040] Figure 21B An example of a boundary smoothing process is shown, in which the boundary pixels whose first value is to be weighted are predicted based on a first partition and whose second value is to be weighted are predicted based on a second partition.

[0041] Figure 21C An example of a boundary smoothing process is shown, in which the boundary pixels whose first value is to be weighted are predicted based on a first partition and whose second value is to be weighted are predicted based on a second partition.

[0042] Figure 21D An example of a boundary smoothing process is shown, in which the boundary pixels whose first value is to be weighted are predicted based on a first partition and whose second value is to be weighted are predicted based on a second partition.

[0043] Figure 22 This is a flowchart illustrating a method for dividing an image patch into multiple partitions based on partitioning parameters of an instruction segmentation, performed on the encoder side, and writing one or more parameters, including the partitioning parameters, into a bitstream in entropy coding, wherein the multiple partitions include a second partition and a first partition having a non-rectangular shape.

[0044] Figure 23 This illustrates a method for parsing one or more parameters from a bitstream, performed on the decoder side. The parameters include a flowchart instructing a method for segmenting an image patch into multiple partitions based on partitioning parameters and decoding a first partition and a second partition. The one or more parameters include partitioning parameters instructing the segmentation of the image patch into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape.

[0045] Figure 24It is a table of sampling partition parameters (“first index value”) and a set of information that can be jointly encoded by the partition parameters, the sampling partition parameters indicating that an image patch is divided into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape.

[0046] Figure 25 It is a table of sampled combinations of the first parameter and the second parameter, wherein one of the first parameter and the second parameter is a partitioning parameter that indicates the division of an image patch into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape.

[0047] Figure 26 The overall configuration of the content delivery system used to implement the content distribution service is shown.

[0048] Figure 27 An example of a coding structure in scalable coding is shown.

[0049] Figure 28 An example of a coding structure in scalable coding is shown.

[0050] Figure 29 An example of a webpage display screen is shown.

[0051] Figure 30 An example of a webpage display screen is shown.

[0052] Figure 31 An example of a smartphone is shown.

[0053] Figure 32 This is a block diagram showing an example of a smartphone configuration. Detailed Implementation

[0054] According to one aspect, an image encoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs the following operations: segmenting an image block into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; predicting a first motion vector for the first partition and a second motion vector for the second partition; and encoding the first partition using the first motion vector and encoding the second partition using the second motion vector.

[0055] According to another aspect, the second partition has a non-rectangular shape. According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of triangles, trapezoids, and polygons with at least five sides and angles.

[0056] According to another aspect, the prediction includes selecting a first motion vector from a first set of motion vector candidates and selecting a second motion vector from a second set of motion vector candidates. For example, the first set of motion vector candidates may include motion vectors from partitions adjacent to the first partition, while the second set of motion vector candidates may include motion vectors from partitions adjacent to the second partition. The partitions adjacent to the first partition and the partition adjacent to the second partition may be outside the image patch from which the first and second partitions are segmented. Adjacent partitions may be one or both of spatially adjacent and temporally adjacent partitions. The first set of motion vector candidates may be the same as or different from the second set of motion vector candidates.

[0057] According to another aspect, the prediction includes: selecting a first motion vector candidate from a first group of motion vector candidates, and obtaining a first motion vector by adding a first motion vector difference to the first motion vector candidate; and selecting a second motion vector candidate from a second group of motion vector candidates, and obtaining a second motion vector by adding a second motion vector difference to the second motion vector candidate.

[0058] According to another aspect, an image encoder is provided, comprising: a segmenter that receives and segments an original image into blocks in operation; an adder that receives blocks from the segmenter and predictions from a prediction controller in operation, and subtracts each prediction from its corresponding block to output a residual; a transformer that performs a transform on the residuals output from the adder in operation to output transform coefficients; a quantizer that quantizes the transform coefficients in operation to generate quantized transform coefficients; an entropy encoder that encodes the quantized transform coefficients in operation to generate a bitstream; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor generates a prediction for the current block based on a reference block in an encoded reference image in operation, and the intra-frame predictor generates a prediction for the current block based on an encoded reference block in the current image in operation. The predictive controller divides a block into multiple partitions during operation, including a second partition and a first partition with a non-rectangular shape; predicts a first motion vector for the first partition and a second motion vector for the second partition; and uses the first motion vector to encode the first partition and the second motion vector to encode the second partition.

[0059] According to another aspect, an image encoding method is provided, which typically includes three steps: dividing an image patch into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape; predicting a first motion vector of the first partition and a second motion vector of the second partition; encoding the first partition using the first motion vector and encoding the second partition using the second motion vector.

[0060] According to another aspect, an image decoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs: segmenting an image patch into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape; predicting a first motion vector for the first partition and a second motion vector for the second partition; decoding the first partition using the first motion vector and decoding the second partition using the second motion vector.

[0061] According to another aspect, the second partition has a non-rectangular shape. According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of triangles, trapezoids, and polygons with at least five sides and angles.

[0062] According to another aspect, an image decoder is provided, comprising: an entropy decoder that receives and decodes an encoded bitstream in operation to obtain quantized transform coefficients; an inverse quantizer and a transformer that in operation inversely quantizes the quantized transform coefficients to obtain transform coefficients and performs an inverse transform on the transform coefficients to obtain residuals; an adder that in operation adds the residuals output from the inverse quantizer and the transformer to a prediction output from a prediction controller to reconstruct a block; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor in operation generates a prediction for the current block based on a reference block in a decoded reference image, and the intra-frame predictor in operation generates a prediction for the current block based on a decoded reference block in the current image. The prediction controller in operation divides an image block into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape; predicts a first motion vector of the first partition and a second motion vector of the second partition; decodes the first partition using the first motion vector and decodes the second partition using the second motion vector.

[0063] According to another aspect, an image decoding method is provided, which generally includes three steps: dividing an image patch into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape; predicting a first motion vector of the first partition and a second motion vector of the second partition; using the first motion vector to decode the first partition and using the second motion vector to decode the second partition.

[0064] According to one aspect, an image encoder is provided, comprising circuitry and memory coupled to the circuitry. The circuitry performs a boundary smoothing operation in operation along a boundary between a second partition segmented from an image block and a first partition having a non-rectangular shape. The boundary smoothing operation includes: making a first prediction of a first value for a set of pixels in the first partition along the boundary using information from the first partition; making a second prediction of a second value for a set of pixels in the first partition along the boundary using information from the second partition; weighting the first and second values; and encoding the first partition using the weighted first and weighted second values.

[0065] According to another aspect, the non-rectangular shape is a triangle. According to yet another aspect, the non-rectangular shape is selected from a group consisting of triangles, trapezoids, and polygons with at least five sides and angles. According to yet another aspect, the second partition has a non-rectangular shape.

[0066] According to another aspect, at least one of the first prediction and the second prediction is an inter-frame prediction process that predicts the first and second values ​​based on reference partitions in an encoded reference image. The inter-frame prediction process can predict the first value of pixels in a first partition comprising a set of pixels, and can predict the second value of a set of pixels in only the first partition.

[0067] According to another aspect, at least one of the first prediction and the second prediction is an intra-frame prediction process that predicts the first and second values ​​based on an encoded reference partition in the current image.

[0068] According to another perspective, the forecasting method used in the first forecast is different from the forecasting method used in the second forecast.

[0069] According to another aspect, the number of pixels in each row or column for predicting the first and second values ​​is an integer. For example, when the number of pixels in a group per row or column is four, weights of 1 / 8, 1 / 4, 3 / 4, and 7 / 8 can be applied to the first values ​​of the four pixels in that group, respectively, and weights of 7 / 8, 3 / 4, 1 / 4, and 1 / 8 can be applied to the second values ​​of the four pixels in that group, respectively. As another example, when the number of pixels in a group per row or column is two, weights of 1 / 3 and 2 / 3 can be applied to the first values ​​of the two pixels in that group, respectively, and weights of 2 / 3 and 1 / 3 can be applied to the second values ​​of the two pixels in that group, respectively.

[0070] On the other hand, the weight can be an integer value or a fractional value.

[0071] According to another aspect, an image encoder is provided, comprising: a segmenter that receives and segments an original image into blocks during operation; an adder that receives blocks from the segmenter and predictions from a prediction controller during operation, and subtracts each prediction from its corresponding block to output a residual; a transformer that performs a transform on the residuals output from the adder during operation to output transform coefficients; a quantizer that quantizes the transform coefficients during operation to generate quantized transform coefficients; an entropy encoder that encodes the quantized transform coefficients during operation to generate a bitstream; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor generates a prediction for the current block based on a reference block in an encoded reference image during operation, and the intra-frame predictor generates a prediction for the current block based on an encoded reference block in the current image during operation. The prediction controller performs a boundary smoothing operation during operation along the boundary between a second partition segmented from the image block and a first partition having a non-rectangular shape. The boundary smoothing operation includes: using information from the first partition, making a first prediction of a first value for a set of pixels in the first partition along the boundary; and using information from the second partition, making a second prediction of a second value for a set of pixels in the first partition along the boundary; weighting the first value and the second value; and encoding the first partition using the weighted first value and the weighted second value.

[0072] According to another approach, an image encoding method is provided that performs a boundary smoothing operation along the boundary between a second partition segmented from an image patch and a first partition having a non-rectangular shape. This method typically includes four steps: using information from the first partition, making a first prediction of a first value for a set of pixels in the first partition along the boundary; and using information from the second partition, making a second prediction of a second value for a set of pixels in the first partition along the boundary; weighting the first and second values; and encoding the first partition using the weighted first and weighted second values.

[0073] According to another aspect, an image decoder is provided, comprising circuitry and memory coupled to the circuitry. The circuitry performs a boundary smoothing operation along a boundary between a second partition segmented from an image block and a first partition having a non-rectangular shape. The boundary smoothing operation includes: making a first prediction of a first value for a set of pixels in the first partition along the boundary using information from the first partition; making a second prediction of a second value for a set of pixels in the first partition along the boundary using information from the second partition; weighting the first and second values; and decoding the first partition using the weighted first and weighted second values.

[0074] According to another aspect, the non-rectangular shape is a triangle. According to another aspect, the non-rectangular shape is selected from a group consisting of triangles, trapezoids, and polygons with at least five sides and angles. According to another aspect, the second partition has a non-rectangular shape.

[0075] According to another aspect, at least one of the first prediction and the second prediction is an inter-frame prediction process that predicts the first and second values ​​based on reference partitions in an encoded reference image. The inter-frame prediction process can predict the first value of pixels in a first partition comprising a set of pixels, and can predict the second value of a set of pixels in only the first partition.

[0076] According to another aspect, at least one of the first prediction and the second prediction is an intra-frame prediction process that predicts the first and second values ​​based on an encoded reference partition in the current image.

[0077] According to another aspect, an image decoder is provided, comprising: an entropy decoder that receives and decodes an encoded bitstream in operation to obtain quantized transform coefficients; an inverse quantizer and a transformer that in operation inversely quantizes the quantized transform coefficients to obtain transform coefficients and performs an inverse transform on the transform coefficients to obtain residuals; an adder that in operation adds the residuals output from the inverse quantizer and the transformer to a prediction output from a prediction controller to reconstruct a block; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor in operation generates a prediction for the current block based on a reference block in a decoded reference image, and the intra-frame predictor in operation generates a prediction for the current block based on a decoded reference block in the current image. The prediction controller in operation performs a boundary smoothing operation along the boundary between a second partition segmented from an image block and a first partition having a non-rectangular shape. The boundary smoothing operation includes: using information from the first partition, making a first prediction of a first value for a set of pixels in the first partition along the boundary; and using information from the second partition, making a second prediction of a second value for a set of pixels in the first partition along the boundary; weighting the first value and the second value; and decoding the first partition using the weighted first value and the weighted second value.

[0078] According to another approach, an image decoding method is provided that performs a boundary smoothing operation along the boundary between a second partition segmented from an image patch and a first partition having a non-rectangular shape. This method typically includes four steps: using information from the first partition, making a first prediction of a first value for a set of pixels in the first partition along the boundary; and using information from the second partition, making a second prediction of a second value for a set of pixels in the first partition along the boundary; weighting the first and second values; and decoding the first partition using the weighted first and second values.

[0079] According to one aspect, an image encoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry performs a partitioning syntax operation in operation, the partitioning syntax operation comprising: dividing an image patch into a plurality of partitions, the plurality of partitions including a second partition and a first partition having a non-rectangular shape, based on partitioning parameters indicating the partitioning; encoding the first partition and the second partition; and writing one or more parameters including the partitioning parameters into a bitstream.

[0080] According to another aspect, the partitioning parameters indicate that the first partition has a triangular shape.

[0081] On the other hand, the partitioning parameters indicate that the second partition has a non-rectangular shape.

[0082] According to another aspect, the partitioning parameter indicates that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon with at least five sides and angles.

[0083] According to another approach, the partitioning parameters jointly encode the segmentation direction used to segment an image patch into multiple partitions. For example, the segmentation direction may include: from the upper left corner of the image patch to its lower right corner, and from the upper right corner of the image patch to its lower left corner. The partitioning parameters may jointly encode at least a first motion vector of the first partition.

[0084] According to another approach, one or more parameters other than the partitioning parameters are used to encode the segmentation direction applied to divide the image patch into multiple partitions. The parameters encoding the segmentation direction can jointly encode at least a first motion vector of the first partition.

[0085] According to another approach, the partitioning parameters can jointly encode at least a first motion vector of the first partition. The partitioning parameters can jointly encode a second motion vector of the second partition.

[0086] According to another option, one or more parameters other than the partitioning parameters can encode at least the first motion vector of the first partition.

[0087] According to another method, one or more parameters are binaryized according to a binaryization scheme selected based on the value of at least one of the one or more parameters.

[0088] According to another aspect, an image encoder is provided, comprising: a segmenter that receives and segments an original image into blocks in operation; an adder that receives blocks from the segmenter and predictions from a prediction controller in operation, and subtracts each prediction from its corresponding block to output a residual; a transformer that performs a transform on the residuals output from the adder in operation to output transform coefficients; a quantizer that quantizes the transform coefficients in operation to generate quantized transform coefficients; an entropy encoder that encodes the quantized transform coefficients in operation to generate a bitstream; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor generates a prediction for the current block based on a reference block in an encoded reference image in operation, and the intra-frame predictor generates a prediction for the current block based on an encoded reference block in the current image in operation. The prediction controller, in operation, segments the image block into multiple partitions based on partition parameters indicating segmentation, the multiple partitions including a second partition and a first partition having a non-rectangular shape, and encodes the first partition and the second partition. An entropy encoder writes one or more parameters, including partition parameters, into a bitstream during operation.

[0089] According to another approach, an image encoding method including partitioning syntax operations is provided. This method typically includes three steps: dividing an image patch into multiple partitions based on partitioning parameters indicating the segmentation, the multiple partitions including a second partition and a first partition having a non-rectangular shape; encoding the first and second partitions; and writing one or more parameters, including the partitioning parameters, into a bitstream.

[0090] According to another aspect, an image decoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry performs a partitioning syntax operation in operation, the partitioning syntax operation comprising: parsing one or more parameters from a bitstream, wherein the one or more parameters include partitioning parameters indicating the division of an image patch into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape; dividing the image patch into multiple partitions based on the partitioning parameters; and decoding the first partition and the second partition.

[0091] According to another aspect, the partitioning parameters indicate that the first partition has a triangular shape.

[0092] On the other hand, the partitioning parameters indicate that the second partition has a non-rectangular shape.

[0093] According to another aspect, the partitioning parameter indicates that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon with at least five sides and angles.

[0094] According to another approach, the partitioning parameters jointly encode the segmentation directions used to divide an image patch into multiple partitions. For example, the segmentation directions include: from the top-left corner of the image patch to its bottom-right corner, and from the top-right corner of the image patch to its bottom-left corner. The partitioning parameters may jointly encode at least a first motion vector of the first partition.

[0095] According to another approach, one or more parameters other than the partitioning parameters are used to encode the segmentation direction applied to divide the image patch into multiple partitions. The parameters encoding the segmentation direction can jointly encode at least a first motion vector of the first partition.

[0096] According to another approach, the partitioning parameters can jointly encode at least a first motion vector of the first partition. The partitioning parameters can jointly encode a second motion vector of the second partition.

[0097] According to another option, one or more parameters other than the partitioning parameters can encode at least the first motion vector of the first partition.

[0098] According to another method, one or more parameters are binaryized according to a binaryization scheme selected based on the value of at least one of the one or more parameters.

[0099] According to another aspect, an image decoder is provided, comprising: an entropy decoder that receives and decodes an encoded bitstream in operation to obtain quantized transform coefficients; an inverse quantizer and a transformer that in operation inversely quantizes the quantized transform coefficients to obtain transform coefficients and performs an inverse transform on the transform coefficients to obtain residuals; an adder that in operation adds the residuals output from the inverse quantizer and the transformer to a prediction output from a prediction controller to reconstruct a block; and a prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor in operation generates a prediction for the current block based on a reference block in a decoded reference image, and the intra-frame predictor in operation generates a prediction for the current block based on a decoded reference block in the current image. The entropy decoder in operation: parses one or more parameters from the bitstream, wherein the one or more parameters include partitioning parameters indicating that the image block is divided into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape; divides the image block into multiple partitions based on the partitioning parameters; and decodes the first partition and the second partition.

[0100] According to another approach, an image decoding method including partitioning syntax operations is provided. This method typically includes three steps: parsing one or more parameters from a bitstream, wherein the one or more parameters include partitioning parameters indicating the division of an image patch into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape; dividing the image patch into multiple partitions based on the partitioning parameters; and decoding the first and second partitions.

[0101] In the accompanying drawings, the same reference numerals denote similar elements. The dimensions and relative positions of the elements in the drawings are not necessarily drawn to scale.

[0102] In the following description, embodiments will be illustrated with reference to the accompanying drawings. Note that each of the embodiments described below illustrates a general or specific example. The numerical values, shapes, materials, components, arrangements and connections of components, steps, relationships and sequences of steps, etc., indicated in the following embodiments are merely examples and are not intended to limit the scope of the claims. Therefore, those components disclosed in the following embodiments but not listed in any independent claim defining the broadest inventive concept can be understood as optional components.

[0103] Embodiments of the encoder and decoder will now be described. These embodiments are examples of encoders and decoders, wherein the processes and / or configurations presented in the description of aspects of this disclosure can be applied to the encoder and decoder. The processes and / or configurations can also be implemented in encoders and decoders different from those according to the embodiments. For example, any of the following can be implemented regarding the processes and / or configurations applied to the embodiments:

[0104] (1) Any component of the encoder or decoder of the embodiments presented in the description of aspects of this disclosure may be replaced or combined with another component presented anywhere in the description of aspects of this disclosure.

[0105] (2) In the encoder or decoder according to the embodiments, any changes may be made to the functions or processes performed by one or more components of the encoder or decoder, such as the addition, replacement, or removal of functions or processes. For example, any function or process may be substituted or combined with another function or process that appears anywhere in the description of any aspect of this disclosure.

[0106] (3) In the method implemented by the encoder or decoder according to the embodiments, any changes may be made, such as adding, replacing, and removing one or more processes included in the method. For example, any process in the method may be substituted or combined with another process that appears anywhere in the description of the aspects of this disclosure.

[0107] (4) One or more components included in the encoder or decoder according to the embodiments may be combined with components presented anywhere in the description of the aspects of this disclosure, may be combined with components presenting one or more functions presented anywhere in the description of the aspects of this disclosure, and may be combined with components that implement one or more processes implemented by components presented in the description of the aspects of this disclosure.

[0108] (5) Components including one or more functions of the encoder or decoder according to the embodiment, or components implementing one or more processes of the encoder or decoder according to the embodiment, may be combined or replaced with components presented anywhere in the description of the aspects of this disclosure, components including one or more functions presented anywhere in the description of the aspects of this disclosure, or components implementing one or more processes presented anywhere in the description of the aspects of this disclosure.

[0109] (6) In the method implemented by the encoder or decoder according to the embodiments, any process included in the method may be replaced or combined with any process presented anywhere in the description of aspects of this disclosure or with any corresponding or equivalent process.

[0110] (7) One or more processes included in the method implemented by the encoder or decoder according to the embodiments may be combined with processes presented anywhere in the description of various aspects of this disclosure.

[0111] (8) The implementation of the processes and / or configurations presented in the description of various aspects of this disclosure is not limited to the encoder or decoder according to the embodiments. For example, the processes and / or configurations may be implemented in devices for purposes different from the motion picture encoder or motion picture decoder disclosed in the embodiments.

[0112] (encoder)

[0113] First, the encoder according to an embodiment will be described. Figure 1 This is a block diagram illustrating the functional configuration of an encoder 100 according to an embodiment. The encoder 100 is a motion picture encoder that encodes motion pictures block by block.

[0114] like Figure 1 As shown, encoder 100 is a device for encoding images block by block, and includes segmenter 102, subtractor 104, transformer 106, quantizer 108, entropy encoder 110, inverse quantizer 112, inverse transformer 114, adder 116, block memory 118, loop filter 120, frame memory 122, intra-frame predictor 124, inter-frame predictor 126, and prediction controller 128.

[0115] The encoder 100 is implemented, for example, as a general-purpose processor and memory. In this case, when the processor executes the software program stored in the memory, the processor acts as a divider 102, a subtractor 104, a converter 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse converter 114, an adder 116, a loop filter 120, an intra-frame predictor 124, an inter-frame predictor 126, and a prediction controller 128. Alternatively, the encoder 100 can be implemented as one or more dedicated electronic circuits corresponding to the divider 102, subtractor 104, converter 106, quantizer 108, entropy encoder 110, inverse quantizer 112, inverse converter 114, adder 116, loop filter 120, intra-frame predictor 124, inter-frame predictor 126, and prediction controller 128.

[0116] Each component included in encoder 100 will be described below.

[0117] (Divider)

[0118] Segmenter 102 segments each image included in the input motion picture into blocks and outputs each block to subtractor 104. For example, segmenter 102 first segments the image into fixed-size (e.g., 128×128) blocks. Fixed-size blocks may also be referred to as coding tree units (CTUs). Then, segmenter 102 segments each fixed-size block into variable-size (e.g., 64×64 or smaller) blocks based on, for example, recursive quadtree and / or binary tree block segmentation. Variable-size blocks may also be referred to as coding units (CUs), prediction units (PUs), or transform units (TUs). In various implementations, it may not be necessary to distinguish between CUs, PUs, and TUs; all or some blocks in the image may be processed as CUs, PUs, or TUs.

[0119] Figure 2 An example of block segmentation according to an embodiment is shown. Figure 2 In the diagram, solid lines represent the block boundaries of blocks partitioned by quadtree blocks, while dashed lines represent the block boundaries of blocks partitioned by binary tree blocks.

[0120] Here, block 10 is a square block of 128×128 pixels (128×128 block). First, this 128×128 block 10 is divided into four square blocks of 64×64 (quadtree block partitioning).

[0121] The 64×64 block in the upper left corner is further vertically divided into two rectangular 32×64 blocks, while the 32×64 block on the left is further vertically divided into two rectangular 16×64 blocks (binary tree block partitioning). As a result, the 64×64 block in the upper left corner is divided into two 16×64 blocks 11 and 12 and one 32×64 block 13.

[0122] The 64×64 block in the upper right corner is horizontally divided into two rectangular 64×32 blocks, 14 and 15 (binary tree block division).

[0123] First, divide the bottom left 64×64 block into four 32×32 square blocks (quadtree block division). Then, further divide the top left and bottom right blocks. The top left 32×32 block is vertically divided into two 16×32 rectangular blocks, while the right 16×32 block is further horizontally divided into two 16×16 blocks (binary tree block division). The bottom right 32×32 block is horizontally divided into two 32×16 blocks (binary tree block division). As a result, the bottom left 64×64 block is divided into 16×32 block 16, two 16×16 blocks 17 and 18, two 32×32 blocks 19 and 20, and two 32×16 blocks 21 and 22.

[0124] The 64×64 block 23 in the lower right corner was not divided.

[0125] As mentioned above, in Figure 2 In this method, based on recursive quadtree and binary tree block partitioning, block 10 is divided into 13 variable-size blocks 11 to 23. This type of partitioning is also known as quadtree plus binary tree (QTBT) partitioning.

[0126] Despite Figure 2 In this context, a block can be divided into two or four blocks (quadtree or binary tree block partitioning), but the partitioning is not limited to these examples. For instance, a block can be partitioned into three blocks (ternary block partitioning). Partitioning that includes such ternary block partitioning is also known as multi-type tree (MBT) partitioning.

[0127] (Subtractor)

[0128] Subtractor 104 subtracts the prediction signal (prediction sample, input from prediction controller 128, as described below) from the original signal (original sample) of each block segmented and input by divider 102. In other words, subtractor 104 calculates the prediction error (also called the "residual") of the block to be encoded (hereinafter also referred to as the "current block"). Subtractor 104 then outputs the calculated prediction error (residual) to converter 106.

[0129] The raw signal is the signal input to encoder 100, and is a signal representing the image of each picture included in the moving picture (e.g., luminance signal and two chrominance signals). In the following text, the signal representing the image is also referred to as a sample.

[0130] (Transformer)

[0131] Transformer 106 transforms the spatial domain prediction error into frequency domain transform coefficients and outputs the transform coefficients to quantizer 108. More specifically, transformer 106 applies, for example, a predefined discrete cosine transform (DCT) or discrete sine transform (DST) to the spatial domain prediction error.

[0132] Note that transformer 106 can adaptively select a transformation type from multiple transformation types and transform the prediction error into transformation coefficients by using transformation basis functions corresponding to the selected transformation type. This transformation is also known as explicit multi-kernel transformation (EMT) or adaptive multi-transformation (AMT).

[0133] Transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 This is a graph representing the fundamental transformation functions for each type of transformation. In Figure 3 In this context, N represents the number of input pixels. For example, choosing a transform type from multiple transform types can depend on the prediction type (intra-frame prediction and inter-frame prediction) and the intra-frame prediction mode.

[0134] Information indicating whether EMT or AMT is applied (e.g., referred to as EMT flag or AMT flag) and information indicating the selected transform type are typically signaled at the CU level. Note that signaling of this information does not need to be performed at the CU level, but can be performed at another level (e.g., at the bit sequence level, picture level, slice level, tile level, or CTU level).

[0135] Furthermore, transformer 106 can apply a quadratic transformation to the transform coefficients (the transformation result). Such a quadratic transformation is also known as an adaptive quadratic transformation (AST) or an inseparable quadratic transformation (NSST). For example, transformer 106 applies the quadratic transformation to each sub-block (e.g., each 4×4 sub-block) within a block containing transform coefficients corresponding to the intra-frame prediction error. Information indicating whether NSST is applied and information related to the transform matrix used in NSST are typically signaled at the CU level. Note that signaling of this information does not need to be performed at the CU level, but can be performed at another level (e.g., at the bit sequence level, picture level, slice level, tile level, or CTU level).

[0136] Individual or inseparable transformations can be applied in transformer 106. An individual transformation is a method in which the transformation is performed multiple times by performing the transformation individually for each direction according to the number of dimensions of the input. An inseparable transformation is a method of performing a collective transformation in which two or more dimensions in the multidimensional input are collectively treated as a single dimension.

[0137] In one example of an inseparable transformation, when the input is a 4×4 block, the 4×4 block is treated as a single array containing 16 components, and the transformation applies a 16×16 transformation matrix to that array.

[0138] In another example of an inseparable transformation, after treating the input 4×4 block as a single array comprising 16 components, a transformation that performs multiple Givens rotations (e.g., a Hypercube-Givens transformation) can be applied to that array.

[0139] (Quantizer)

[0140] Quantizer 108 quantizes the transform coefficients output from converter 106. More specifically, quantizer 108 scans the transform coefficients of the current block in a predetermined scan order and quantizes the scanned transform coefficients based on the quantization parameters (QP) corresponding to the transform coefficients. Then, quantizer 108 outputs the quantized transform coefficients of the current block (hereinafter referred to as quantized coefficients) to entropy encoder 110 and inverse quantizer 112.

[0141] A predetermined scan order is the order in which the transform coefficients are quantized / inverse quantized. For example, a predetermined scan order may be defined as ascending frequency (from low to high frequency) or descending frequency (from high to low frequency).

[0142] The quantization parameter (QP) is a parameter that defines the quantization step size (quantization width). For example, if the value of the quantization parameter increases, the quantization step size also increases. In other words, if the value of the quantization parameter increases, the quantization error increases.

[0143] (Entropy encoder)

[0144] The entropy encoder 110 generates an encoded signal (encoded bitstream) based on the quantized coefficients input from the quantizer 108. More specifically, for example, the entropy encoder 110 binaryizes the quantized coefficients and performs arithmetic encoding on the binary signal to output a compressed bitstream or sequence.

[0145] (Inverse quantizer)

[0146] Inverse quantizer 112 inverse quantizes the quantized coefficients input from quantizer 108. More specifically, inverse quantizer 112 inverse quantizes the quantized coefficients of the current block in a predetermined scan order. Then, inverse quantizer 112 outputs the inverse quantized transform coefficients of the current block to inverse transformer 114.

[0147] (Inverse Transformer)

[0148] Inverse transformer 114 recovers the prediction error (residual) by performing an inverse transform on the transform coefficients input from inverse quantizer 112. More specifically, inverse transformer 114 recovers the prediction error of the current block by applying an inverse transform corresponding to the transform applied to the transform coefficients by transformer 106. Inverse transformer 114 then outputs the recovered prediction error to adder 116.

[0149] Note that because information is typically lost during quantization, the recovered prediction error does not match the prediction error calculated by subtractor 104. In other words, the recovered prediction error typically includes the quantization error.

[0150] (Adder)

[0151] Adder 116 reconstructs the current block by summing the prediction error input from inverse transformer 114 and the prediction sample input from prediction controller 128. Adder 116 then outputs the reconstructed block to block memory 118 and loop filter 120. The reconstructed block is also referred to as the locally decoded block.

[0152] (Block memory)

[0153] Block memory 118 is a storage device for storing blocks of a picture to be encoded (referred to as the "current picture") referenced in, for example, intra-frame prediction. More specifically, block memory 118 stores reconstructed blocks output from adder 116.

[0154] (Loop filter)

[0155] Loop filter 120 applies a loop filter to the block reconstructed by adder 116 and outputs the filtered, reconstructed block to frame memory 122. Loop filters are filters used in the coding loop (in-loop filters) and include, for example, deblocking filters (DF), sample adaptive offset (SAO), and adaptive loop filters (ALF).

[0156] In ALF, a least-squares error filter is applied to eliminate compression artifacts. For example, based on the direction and activity of the local gradient, a filter is selected from multiple filters for each 2×2 sub-block in the current block and applied.

[0157] More specifically, first, each sub-block (e.g., each 2×2 sub-block) is classified into one of several categories (e.g., 15 or 25 categories). The classification of sub-blocks is based on gradient directionality and activity. For example, a classification index C (e.g., C = 5D + A) is derived based on gradient directionality D (e.g., 0 to 2 or 0 to 4) and gradient activity A (e.g., 0 to 4). Then, based on the classification index C, each sub-block is classified into one of several categories.

[0158] For example, gradient directionality D is calculated by comparing gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). Furthermore, gradient activity A is calculated, for example, by summing the gradients in multiple directions and quantizing the sum.

[0159] Based on the results of this classification, the filter to be used for each sub-block is determined from multiple filters.

[0160] The filter shape used in ALF is, for example, a circular symmetric filter shape. Figure 4A , 4B Figures 1, 2, and 4C show examples of filter shapes used in ALF. Figure 4A A 5×5 diamond filter is shown. Figure 4B A 7×7 rhombus filter is shown, while Figure 4C A 9×9 diamond-shaped filter is shown. Information indicating the filter shape is typically sent as a signal at the picture level. Note that the signaling of information indicating the filter shape does not need to be performed at the picture level, but can be performed at another level (e.g., at the sequence level, slice level, tile level, CTU level, or CU level).

[0161] ALF (Alternating Current Filter) can be enabled or disabled at the picture level or the Control Unit (CU) level. For example, for luminance, the decision to apply ALF can be made at the CU level, while for chroma, the decision can be made at the picture level. The information indicating whether ALF is enabled or disabled is typically sent as a signal at the picture level or the CU level. Note that the signaling of the information indicating whether ALF is enabled does not need to be performed at the picture level or the CU level, but can be performed at another level (e.g., at the sequence level, slice level, tile level, or CTU level).

[0162] Typically, a set of coefficients for multiple optional filters (e.g., 15 or 25 filters) is signaled at the picture level. Note that signaling of the coefficient set does not need to be performed at the picture level, but can be performed at another level (e.g., at the sequence level, slice level, tile level, CTU level, CU level, or sub-block level).

[0163] (Frame Memory)

[0164] Frame memory 122 is, for example, a storage device used to store reference images used in inter-frame prediction, and is also referred to as a frame buffer. More specifically, frame memory 122 stores reconstructed blocks filtered by loop filter 120.

[0165] (Intra-frame predictor)

[0166] Intra-predictor 124 generates a prediction signal (also referred to as intra-prediction) by performing intra-prediction on the current block with reference to one or more blocks in the current image stored in block memory 118. More specifically, intra-predictor 124 generates the intra-prediction signal by performing intra-prediction on the current block with reference to samples (e.g., luminance and / or chrominance values) of one or more blocks adjacent to the current block, and then outputs the intra-prediction signal to prediction controller 128.

[0167] For example, the intra predictor 124 performs intra prediction by using one of a plurality of predetermined intra prediction modes. The intra prediction modes typically include one or more non-directional prediction modes and a plurality of directional prediction modes.

[0168] One or more non-directional prediction modes include, for example, the planar prediction mode and the DC prediction mode as defined in the H.265 / HEVC standard.

[0169] The multiple directional prediction modes include, for example, the 33 directional prediction modes defined in the H.265 / HEVC standard. Note that in addition to the 33 directional prediction modes, the multiple directional prediction modes may also include 32 directional prediction modes (for a total of 65 directional prediction modes).

[0170] Figure 5A This shows a total of 67 intra-prediction modes used in intra-frame prediction (two non-directional prediction modes and 65 directional prediction modes). Solid arrows represent the 33 directions defined in the H.265 / HEVC standard, while dashed arrows represent the other 32 directions (in...). Figure 5A (The two "non-directional" prediction modes are not shown in the diagram.)

[0171] In various implementations, the luma block can be referenced in chroma block intra-prediction. That is, the chroma components of the current block can be predicted based on the luma components of the current block. Such intra-prediction is also known as cross-component linear model (CCLM) prediction. A chroma block intra-prediction mode that references the luma block (e.g., called a CCLM mode) can be added as one of the chroma block intra-prediction modes.

[0172] Intra-predictor 124 can correct the intra-predicted pixel values ​​based on horizontal / vertical reference pixel gradients. Intra-prediction accompanied by such correction is also known as position-dependent intra-prediction combination (PDPC). Information indicating whether PDPC is applied (e.g., referred to as a PDPC flag) is typically signaled at the CU level. Note that signaling of this information does not need to be performed at the CU level, but can be performed at another level (e.g., at the sequence level, picture level, slice level, tile level, or CTU level).

[0173] (Inter-frame predictor)

[0174] Inter-frame predictor 126 generates a prediction signal (inter-frame prediction) by performing inter-frame prediction on the current block using one or more blocks in a reference image that is different from the current image and stored in frame memory 122. Inter-frame prediction is performed on each current block or each current sub-block within the current block (e.g., each 4×4 block). For example, inter-frame predictor 126 performs motion estimation for the current block or current sub-block in the reference image to find the reference block or sub-block that best matches the current block or sub-block in the reference image, and obtains compensated (or predicted) motion or motion information (e.g., motion vectors) that changes from the reference block or sub-block to the current block or sub-block. Next, inter-frame predictor 126 performs motion compensation (or motion prediction) based on the motion information and generates an inter-frame prediction signal for the current block or sub-block based on the motion information. Inter-frame predictor 126 then outputs the generated inter-frame prediction signal to prediction controller 128.

[0175] Motion information used in motion compensation can be transmitted as inter-frame prediction signals in various forms. For example, motion vectors can be transmitted as signals. As another example, the difference between motion vectors and predicted motion vector values ​​can be transmitted as signals.

[0176] Note that in addition to the motion information of the current block obtained from motion estimation, motion information from neighboring blocks can also be used to generate inter-frame prediction signals. More specifically, inter-frame prediction signals can be generated for each sub-block in the current block by calculating a weighted sum of prediction signals based on motion information obtained from motion estimation (in the reference image) and prediction signals based on motion information from neighboring blocks (in the current image). Such inter-frame prediction (motion compensation) is also known as Overlapping Block Motion Compensation (OBMC).

[0177] In OBMC mode, information indicating the size of the sub-block used for OBMC (e.g., referred to as the OBMC block size) can be signaled at the sequence level. Additionally, information indicating whether OBMC mode is applied (e.g., referred to as the OBMC flag) can be signaled at the CU level. Note that such signaling does not need to be performed at the sequence and CU levels, but can be performed at another level (e.g., at the image level, slice level, tile level, CTU level, or sub-block level).

[0178] The OBMC model will be described in more detail below. Figure 5B It's a flowchart. Figure 5C This is a conceptual diagram illustrating the predictive image correction process performed through OBMC processing.

[0179] refer to Figure 5CFirst, the predicted image (Pred) is obtained using the motion vector (MV) assigned to the target (current) block through typical motion compensation. Figure 5C In the image, the arrow "MV" points to a reference image, indicating what the current block in the current image is referencing in order to obtain the predicted image.

[0180] Next, a predicted image (Pred_L) is obtained by applying (reusing) the motion vector (MV_L) derived for the encoded adjacent left block to the target (current) block, as indicated by the arrow "MV_L" pointing from the current block to the reference image to obtain the predicted image Pred_L. Then, the two predicted images, Pred and Pred_L, are superimposed to perform the first pass of correction to the predicted image, which in one aspect has the effect of blending the boundaries between adjacent blocks.

[0181] Similarly, a predicted image (Pred_U) is obtained by applying (reusing) the motion vectors (MV_U) derived for the encoded neighboring blocks to the target (current) block, as indicated by the arrow "MV_U" pointing from the current block to the reference image to obtain the predicted image Pred_U. Next, the predicted image Pred_U is superimposed with the predicted images obtained from the first pass (i.e., Pred and Pred_L) to perform a second pass of correction to the predicted image, which in one aspect has the effect of blending the boundaries between neighboring blocks. The result of the second pass is the final predicted image of the current block, which has blended (smoothed) boundaries with its neighboring blocks.

[0182] Note that the example above uses a two-pass correction method with adjacent left and top blocks, but this method can also use a three-pass or higher correction method with adjacent right and / or bottom blocks.

[0183] Note that the superimposed area can be the entire pixel region of the block, and alternatively, it can be a portion of the block boundary region.

[0184] Note here that the OBMC predictive image correction process is described as deriving a single predictive image Pred based on a single reference image, on which additional predictive images Pred_L and Pred_U are superimposed. However, when correcting the predictive image based on multiple reference images, the same process can be applied to each of the multiple reference images. In such a case, after obtaining multiple corrected predictive images by performing OBMC image correction separately based on multiple reference images, the obtained multiple corrected predictive images are superimposed to obtain the final predictive image.

[0185] Note that in OBMC, the unit of the target block can be the prediction block, and alternatively, it can be a sub-block obtained by further splitting the prediction block.

[0186] One example of a method for determining whether OBMC processing is implemented is using `obmc_flag`, a signal indicating whether OBMC processing is performed. As a specific example, the encoder can determine whether a target block belongs to a region containing complex motion. When the block belongs to a region containing complex motion, the encoder sets the value of `obmc_flag` to "1" and performs OBMC processing during the encoding process; if the block does not belong to a region containing complex motion, the value of `obmc_flag` is set to "0," and the block is encoded without performing OBMC processing. The decoder toggles between implementing OBMC processing by decoding the `obmc_flag` (i.e., the compressed sequence) written to the stream and performing decoding based on the flag value.

[0187] Note that motion information can be derived at the decoder side without being transmitted via signals from the encoder side. For example, a merging mode defined in the H.265 / HEVC standard can be used. Furthermore, motion information can be derived, for example, by performing motion estimation at the decoder side. In this case, motion estimation can be performed at the decoder side without using the pixel values ​​of the current block.

[0188] Here, the modes used to perform motion estimation on the decoder side will be described. The modes used to perform motion estimation on the decoder side are also known as the Pattern Matching Motion Vector Derivation (PMMVD) mode or the Frame Rate Upconversion (FRUC) mode.

[0189] exist Figure 5D The diagram illustrates an example of FRUC processing. First, a candidate list (which can be a merged list) is generated by referencing the motion vectors of coded blocks spatially or temporally adjacent to the current block, with each candidate list including predicted motion vectors (MVs). Next, the best candidate MV is selected from the multiple candidate MVs registered in the candidate lists. For example, evaluation values ​​are calculated for the candidate MVs included in the candidate lists, and a candidate MV is selected based on the calculated evaluation values.

[0190] Next, the motion vector of the current block is derived from the motion vectors of the selected candidates. More specifically, for example, the motion vector of the current block is calculated as the motion vector of the selected candidate (best candidate MV). Alternatively, the motion vector of the current block can be derived by performing pattern matching in the reference image near the positions corresponding to the motion vectors of the selected candidates. In other words, when searching for the vicinity of the best candidate MV using pattern matching and evaluation values ​​in the reference image, and finding an MV with a better evaluation value, the best candidate MV can be updated to the MV with the better evaluation value, and the MV with the better evaluation value can be used as the final MV of the current block. It is also acceptable not to implement the configuration for updating the MV with the better evaluation value.

[0191] When processing is performed on a sub-block basis, the same processing can be performed.

[0192] The evaluation value can be calculated in various ways. For example, a reconstructed image of a region in a reference image corresponding to the motion vector can be compared with a reconstructed image of a predetermined region (e.g., in another reference image or in an adjacent block in the current image, as described below), and the difference in pixel values ​​between the two reconstructed images can be calculated and used as the evaluation value of the motion vector. Note that, in addition to the difference, the evaluation value can also be calculated using some other information.

[0193] Next, pattern matching is described in detail. First, a candidate MV included in the candidate list (e.g., a merged list) is selected as the starting point for the search by pattern matching. The pattern matching used is either first pattern matching or second pattern matching. First pattern matching and second pattern matching are also referred to as bilateral matching and template matching, respectively.

[0194] In the first pattern matching, pattern matching is performed between two blocks in two different reference images, both of which follow the motion trajectory of the current block. Therefore, in the first pattern matching, for a region in one reference image, a region in another reference image that matches the motion trajectory of the current block is used as a predetermined region for calculating the aforementioned candidate evaluation value.

[0195] Figure 6 This illustrates an example of a first pattern match (bidirectional matching) between two blocks in two reference images along a motion trajectory. (As shown in...) Figure 6 As shown, in the first pattern matching, two motion vectors (MV0, MV1) are derived by finding the best match between two blocks in two different reference images (Ref0, Ref1) along the motion trajectory of the current block (Cur block). More specifically, the difference can be obtained between (i) the reconstructed image at the location specified by the candidate MV in the first encoded reference image (Ref0) and (ii) the reconstructed image at the location specified by the candidate MV in the second encoded reference image (Ref1) (scaled symmetrically for each display time interval). This difference can then be used to derive an evaluation value for the current block. The candidate MV with the best evaluation value can be selected as the final MV from among multiple candidate MVs.

[0196] Under the assumption of continuous motion trajectories, the motion vectors (MV0, MV1) pointing to the two reference blocks are proportional to the temporal distances (TD0, TD1) between the current image (Cur Pic) and the two reference images (Ref0, Ref1). For example, when the current image is located between the two reference images in time, and the temporal distances from the current image to the two reference images are the same, the first pattern matching yields two mirrored bidirectional motion vectors.

[0197] In the second pattern matching (template matching), pattern matching is performed between a template in the current image (a block adjacent to the current block in the current image; for example, the top and / or left adjacent blocks) and a block in the reference image. Therefore, in the second pattern matching, the blocks in the current image adjacent to the current block are used as predetermined regions for the aforementioned calculation of candidate evaluation values.

[0198] Figure 7 This illustrates an example of pattern matching (template matching) between a template in the current image and a block in a reference image. For example, in... Figure 7 As shown, in the second pattern matching, the motion vector of the current block is obtained by searching in the reference image (Ref0) to find the block that best matches the neighboring blocks of the current block (Cur block) in the current image (Cur Pic). More specifically, the difference can be obtained between (i) a reconstructed image of one or both of the encoded adjacent upper and left regions of the current block and (ii) a reconstructed image of the same region relative to the block location specified by the candidate MV in the encoded reference image (Ref0). This difference can then be used to derive an evaluation value for the current block. The candidate MV with the best evaluation value can be selected as the best candidate MV from among the multiple candidate MVs.

[0199] Information indicating whether FRUC mode is applied (e.g., referred to as the FRUC flag) can be signaled at the CU level. Furthermore, when FRUC mode is applied (e.g., when the FRUC flag is set to true), information indicating the matching method applicable to the mode (e.g., first mode match or second mode match) can be signaled at the CU level. Note that signaling such information does not need to be performed at the CU level, but can be performed at another level (e.g., at the sequence level, picture level, slice level, tile level, CTU level, or sub-tile level).

[0200] Next, the method for deriving the motion vector is described. First, a description of the mode used to derive the motion vector based on a model assuming uniform linear motion is given. This mode is also known as the bidirectional optical flow (BIO) mode.

[0201] Figure 8 A model assuming uniform linear motion is shown. In Figure 8In the middle, (v x ,v y Let represent the velocity vector, and τ0 and τ1 represent the time distance between the current image (Cur Pic) and the two reference images (Ref0, Ref1), respectively. (MVx0, MVy0) represents the motion vector corresponding to the reference image Ref0, and (MVx1, MVy1) represents the motion vector corresponding to the reference image Ref1.

[0202] Here, assuming that the velocity vectors (vx, vy) exhibit uniform linear motion, (MVx0, MVy0) and (MVx1, MVy1) are respectively represented as (v x τ0,v y τ0) and (-v x τ1,-v y τ1), and the following optical flow equation (Equation 1) is given.

[0203] [Mathematical Expression 1]

[0204]

[0205] Here, I (k) Let represent the brightness value from the motion-compensated reference image k (k = 0, 1). The optical flow equation states that the sum of the following terms equals zero: (i) the time derivative of the brightness value, (ii) the product of the horizontal velocity and horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and vertical component of the spatial gradient of the reference image. The motion vector of each block obtained, for example, from a merged list can be corrected pixel-by-pixel based on a combination of the optical flow equation and Hermite interpolation.

[0206] Note that motion vectors can be derived on the decoder side using methods other than models based on the assumption of uniform linear motion. For example, the motion vector of each sub-block can be derived based on the motion vectors of adjacent blocks.

[0207] Next, a description of the mode in which motion vectors are derived for each sub-block based on the motion vectors of neighboring blocks is given. This mode is also known as the affine motion compensation prediction mode.

[0208] Figure 9A An example is shown where the motion vector of each sub-block is derived based on the motion vectors of neighboring blocks. Figure 9AIn this context, the current block comprises 16 4×4 sub-blocks. Here, the motion vector v0 of the top-left control point in the current block is derived based on the motion vectors of adjacent sub-blocks. Similarly, the motion vector v1 of the top-right control point in the current block is derived based on the motion vectors of adjacent blocks. Then, using the two motion vectors v0 and v1, the motion vector (v1) of each sub-block in the current block is obtained using the following Equation 2. x v y ).

[0209] [Mathematical Expression 2]

[0210]

[0211] Here, x and y are the horizontal and vertical positions of the sub-block, respectively, and w is a pre-determined weighting coefficient.

[0212] Affine motion compensation prediction modes can include multiple modes that derive motion vectors for the top-left and top-right control points using different methods. Information indicating the affine motion compensation prediction mode (e.g., referred to as an affine flag) can be signaled at the CU level. Note that signaling information indicating the affine motion compensation prediction mode does not need to be performed at the CU level, but can be performed at another level (e.g., at the sequence level, picture level, slice level, tile level, CTU level, or sub-block level).

[0213] (Predictive Controller)

[0214] Prediction controller 128 selects an intra-frame prediction signal (output from intra-frame predictor 124) or an inter-frame prediction signal (output from inter-frame predictor 126), and outputs the selected prediction signal to subtractor 104 and adder 116.

[0215] like Figure 1 As shown, in various implementations, prediction controller 128 can output prediction parameters that are input to entropy encoder 110. Entropy encoder 110 can generate an encoded bitstream (or sequence) based on the prediction parameters input from prediction controller 128 and quantized coefficients input from quantizer 108. The prediction parameters can be used by a decoder that receives and decodes the encoded bitstream to perform the same prediction processing as performed in intra-frame predictor 124, inter-frame predictor 126, and prediction controller 128. Prediction parameters can include selected prediction signals (e.g., motion vectors, prediction types, or prediction modes employed in intra-frame predictor 124 or inter-frame predictor 126), or any index, flag, or value based on or indicating the prediction processing performed in intra-frame predictor 124, inter-frame predictor 126, and prediction controller 128.

[0216] Figure 9BAn example of the process for deriving motion vectors in the current image in merge mode is shown.

[0217] First, a list of predicted MVs is generated, in which predicted MV candidates are registered. Examples of predicted MV candidates include: spatially adjacent predicted MVs, which are MVs of coded blocks located spatially near the target block; temporally adjacent predicted MVs, which are MVs of blocks in the encoded reference image that are adjacent to the block located in the same position as the target block; coupled predicted MVs, which are MVs generated by combining the MV values ​​of spatially adjacent predicted MVs and temporally adjacent predicted MVs; and zero predicted MVs, which are MVs with a value of zero.

[0218] Next, the MV of the target block is determined by selecting one of the multiple predicted MVs registered in the predicted MV list.

[0219] Additionally, in a variable-length encoder, the signal merge_idx, which indicates which predicted MV was selected, is written and encoded into the stream.

[0220] Note that in Figure 9B The predicted MVs registered in the predicted MV list shown constitute an example. The number of predicted MVs registered in the predicted MV list can be related to... Figure 9B The quantities shown are different; the predicted MVs registered in the predicted MV list can be omitted. Figure 9B The examples given include one or more types of predicted MVs, except in Figure 9B The types given in the examples are different and are related to Figure 9B The types given in the examples are different; the predicted MVs registered in the predicted MV list can include one or more types of predicted MVs.

[0221] The final MV can be determined by performing DMVR (Dynamic Motion Vector Refresh) processing (described later) using the MV of the target block obtained in merge mode.

[0222] Figure 9C This is a conceptual diagram illustrating an example of DMVR processing used to determine MV.

[0223] First, the most suitable MV set for the current block (e.g., in merge mode) is considered as a candidate MV. Next, based on the candidate MV (L0), reference pixels are identified in a first reference image (L0), which serves as the encoded image in the L0 direction. Similarly, based on the candidate MV (L1), reference pixels are identified in a second reference image (L1), which serves as the encoded image in the L1 direction. Then, the reference pixels are averaged to form a template.

[0224] Next, using this template, the region surrounding the candidate MVs of the first reference image (L0) and the second reference image (L1) is searched, and the MV with the lowest cost is determined as the final MV. For example, the cost value can be calculated using the difference between each pixel value in the template and each pixel value in the searched region, using candidate MVs, etc.

[0225] Note that the configuration and operation of the processing described here are essentially the same on both the encoder and decoder sides, which will be described below.

[0226] Any processing other than that described above can be used, as long as it can derive the final MV by searching the surrounding environment of the candidate MV.

[0227] Next, a description of an example of using LIC (Local Illumination Compensation) processing to generate a predicted image (predicted) pattern is given.

[0228] Figure 9D An example of a predictive image generation method using a brightness correction process performed via LIC processing is shown.

[0229] First, the MV is derived from the encoded reference image to obtain the reference image corresponding to the current block.

[0230] Next, for the current block, based on the luminance pixel values ​​of the encoded adjacent left reference region and the encoded adjacent upper reference region in the current image, and based on the luminance pixel values ​​at the same location in the reference image specified by MV, information indicating how the luminance value changes between the reference image and the current image is obtained. This information indicating how the luminance value changes is used to calculate the luminance correction parameters.

[0231] The predicted image for the current block is generated by performing a brightness correction process that applies brightness correction parameters to a reference image in a reference picture specified by the MV.

[0232] Notice, Figure 9D The shape of the surrounding reference area shown is merely an example; the surrounding reference area can have different shapes.

[0233] Furthermore, although the predicted image was generated from a single reference image in this example, in the case of generating a predicted image from multiple reference images, the predicted image can be generated after performing a brightness correction process on the reference image obtained from the reference image as described above.

[0234] An example of a method for determining whether to implement LIC processing is to use `lic_flag`, which is a signal indicating whether LIC processing is implemented. As a concrete example, the encoder determines whether the current block belongs to a region of brightness change. When the block belongs to a region of brightness change, the encoder sets `lic_flag` to "1" and implements LIC processing during encoding. When the block does not belong to a region of brightness change, the encoder sets `lic_flag` to "0" and performs encoding that implements LIC processing. The decoder can toggle between implementing LIC processing and not implementing it by decoding the `lic_flag` written to the stream and performing decoding based on the flag value.

[0235] An example of different methods for determining whether LIC processing is implemented includes identifying whether LIC processing is determined to be implemented for surrounding blocks. In a particular example, when merging mode is used on the current block, it is determined whether LIC processing was applied in the encoding of surrounding encoded blocks that were selected when deriving the MV in merging mode. This determination is then used to further determine whether LIC processing is implemented for the current block. Note that in this example, the same applies to processing performed on the decoder side.

[0236] (Decoder)

[0237] Next, a decoder capable of decoding the encoded signal (encoded bitstream) output from encoder 100 will be described. Figure 10 This is a block diagram illustrating the functional configuration of decoder 200 according to an embodiment. Decoder 200 is a motion picture decoder that decodes motion pictures block by block.

[0238] As in Figure 10 As shown, the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an intra-frame predictor 216, an inter-frame predictor 218, and a prediction controller 220.

[0239] The decoder 200 is implemented, for example, as a general-purpose processor and memory. In this case, when the processor executes the software program stored in the memory, the processor acts as the entropy decoder 202, inverse quantizer 204, inverse transformer 206, adder 208, loop filter 212, intra-frame predictor 216, inter-frame predictor 218, and prediction controller 220. Alternatively, the decoder 200 can be implemented as one or more dedicated electronic circuits corresponding to the entropy decoder 202, inverse quantizer 204, inverse transformer 206, adder 208, loop filter 212, intra-frame predictor 216, inter-frame predictor 218, and prediction controller 220.

[0240] Each component included in decoder 200 will be described below.

[0241] (Entropy Decoder)

[0242] Entropy decoder 202 performs entropy decoding on the decoded bitstream. More specifically, for example, entropy decoder 202 arithmetically decodes the encoded bitstream into a binary signal. Then, entropy decoder 202 debinds this binary signal. Entropy decoder 202 outputs the quantized coefficients of each block to inverse quantizer 204. Entropy decoder 202 can also include prediction parameters (see [link to documentation]) in the encoded bitstream. Figure 1 The output is sent to the intra-frame predictor 216, the inter-frame predictor 218, and the prediction controller 220 so that they can perform the same prediction processing as that performed on the encoder side in the intra-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128.

[0243] (Inverse quantizer)

[0244] Inverse quantizer 204 inverse-quantizes the quantized coefficients of the block to be decoded (hereinafter referred to as the current block) input from entropy decoder 202. More specifically, inverse quantizer 204 inverse-quantizes the quantized coefficients of the current block based on the quantization parameters corresponding to the quantized coefficients. Then, inverse quantizer 204 outputs the inverse-quantized coefficients (i.e., transform coefficients) of the current block to inverse transform 206.

[0245] (Inverse Transformer)

[0246] The inverse transformer 206 recovers the prediction error (residual) by performing an inverse transformation on the transform coefficients input from the inverse quantizer 204.

[0247] For example, when the information parsed from the encoded bitstream indicates the application of EMT or AMT (e.g., when the AMT flag is set to true), the inverse transformer 206 performs an inverse transform on the transform coefficients of the current block based on the information indicating the type of transform parsed.

[0248] Furthermore, for example, when the information parsed from the encoded bitstream indicates the application of NSST, the inverse transformer 206 applies a second inverse transform to the transform coefficients.

[0249] (Adder)

[0250] Adder 208 reconstructs the current block by adding the prediction error input from inverter 206 to the prediction sample, which is the input from prediction controller 220. Then, adder 208 outputs the reconstructed block to block memory 210 and loop filter 212.

[0251] (Block memory)

[0252] Block memory 210 is a memory used to store blocks in an image (hereinafter referred to as the current image) to be decoded for reference in intra-frame prediction. More specifically, block memory 210 stores the reconstructed blocks output from adder 208.

[0253] (Loop filter)

[0254] Loop filter 212 applies a loop filter to the block reconstructed by adder 208 and outputs the filtered, reconstructed block to frame memory 214 and, for example, to a display device.

[0255] When the information indicating that ALF parsing from the encoded bitstream is enabled or disabled is enabled, a filter is selected from multiple filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed block.

[0256] (Frame Memory)

[0257] Frame memory 214 is a storage device used to store reference images used in inter-frame prediction, and is also referred to as a frame buffer. More specifically, frame memory 214 stores reconstructed blocks filtered by loop filter 212.

[0258] (Intra-frame predictor)

[0259] Intra-predictor 216 references one or more blocks in the current image, such as those stored in block memory 210, to generate a prediction signal (intra-prediction signal) through intra-frame prediction. More specifically, intra-predictor 216 references samples (e.g., luminance and / or chrominance values) of one or more blocks adjacent to the current block to generate an intra-prediction signal through intra-frame prediction, and then outputs the intra-prediction signal to prediction controller 220.

[0260] Note that when an intra-prediction mode is selected that performs intra-prediction of chroma blocks from luma blocks, the intra-predictor 216 can predict the chroma components of the current block based on the luma components of the current block.

[0261] Furthermore, when information indicating the application of PDPC is parsed from the encoded bitstream (e.g., in the prediction parameters output from the entropy decoder 202), the intra-predictor 216 corrects the intra-predicted pixel values ​​based on the horizontal / vertical reference pixel gradients.

[0262] (Inter-frame predictor)

[0263] Inter-frame predictor 218 refers to a reference image stored in frame memory 214 to predict the current block. Inter-frame prediction is performed for each current block or each sub-block within the current block (e.g., each 4×4 block). For example, inter-frame predictor 218 uses motion information (e.g., motion vectors) parsed from an encoded bitstream (e.g., prediction parameters output from entropy decoder 202) to generate an inter-frame prediction signal for the current block or sub-block based on motion compensation, and outputs the inter-frame prediction signal to prediction controller 220.

[0264] When the information parsed from the encoded bitstream indicates the application of OBMC mode, the inter-frame predictor 218 uses motion information of adjacent blocks in addition to motion information of the current block obtained from motion estimation to generate an inter-frame prediction signal.

[0265] Furthermore, when the information parsed from the encoded bitstream indicates the application of the FRUC mode, the inter-frame predictor 218 derives motion information by performing motion estimation according to the pattern matching method (bilateral matching or template matching) parsed from the encoded bitstream. Then, the inter-frame predictor 218 uses the derived motion information to perform motion compensation (prediction).

[0266] Furthermore, when applying BIO mode, the inter-frame predictor 218 derives the motion vector based on a model assuming uniform linear motion. Additionally, when information parsed from the encoded bitstream indicates that an affine motion compensation prediction mode will be applied, the inter-frame predictor 218 derives the motion vector for each sub-block based on the motion vectors of adjacent blocks.

[0267] (Predictive Controller)

[0268] Prediction controller 220 selects either an intra-frame prediction signal or an inter-frame prediction signal and outputs the selected prediction signal to adder 208. Generally, the configuration, function, and operation of the decoder-side prediction controller 220, inter-frame predictor 218, and intra-frame predictor 216 can correspond to the configuration, function, and operation of the encoder-side prediction controller 128, inter-frame predictor 126, and intra-frame predictor 124.

[0269] (Non-rectangular partition)

[0270] In the prediction controller 128 of the intra-frame predictor 124 and inter-frame predictor 126 coupled to the encoder side (see...) Figure 1 ), and in the prediction controller 220 of the intra-predictor 216 and inter-predictor 218 coupled to the decoder side (see Figure 10 To date, the partitions (or variable-sized blocks or sub-blocks) obtained by dividing each block to acquire motion information (e.g., motion vectors) have always been rectangular, such as... Figure 2As shown. The inventors have discovered that, in various implementations, generating partitions with non-rectangular shapes such as triangles leads to improvements in image quality and coding efficiency based on the content of the image in the picture. Various embodiments will be described below, wherein at least one partition segmented from the image patch for prediction purposes has a non-rectangular shape. Note that these embodiments are equally applicable to the encoder side (prediction controller 128 coupled to intra-frame predictor 124 and inter-frame predictor 126) and the decoder side (prediction controller 220 coupled to intra-frame predictor 216 and inter-frame predictor 218), and can be implemented in... Figure 1 The encoder or Figure 10 It is implemented in decoders, etc.

[0271] Figure 11 The flowchart illustrates the following process, which divides an image patch into partitions that include at least a first partition with a non-rectangular shape (e.g., a triangle) and a second partition, and performs an additional process that includes encoding (or decoding) the image patch into a reconstructed combination of the first and second partitions.

[0272] In step S1001, the image patch is divided into multiple partitions, which may include a first partition with a non-rectangular shape and a second partition that may or may not have a non-rectangular shape. For example, as in Figure 12 As shown, an image patch can be divided from its top-left corner to its bottom-right corner to create a first and second partition, both of which are non-rectangular (e.g., triangular) shapes. Alternatively, an image patch can be divided from its top-right corner to its bottom-left corner to create a first and second partition, both of which are non-rectangular (e.g., triangular) shapes. The following will refer to... Figure 12 and 17 -19 is used to describe various examples of non-rectangular segmentation.

[0273] In step S1002, the process predicts a first motion vector for the first partition and a second motion vector for the second partition. For example, the prediction of the first and second motion vectors may include selecting a first motion vector from a first set of motion vector candidates and selecting a second motion vector from a second set of motion vector candidates.

[0274] In step S1003, motion compensation processing is performed to obtain a first partition using the first motion vector obtained in step S1002 above, and to obtain a second partition using the second motion vector obtained in step S1002 above.

[0275] In step S1004, a prediction process is performed on the image patch that is a combination of the first and second partitions (reconstructed). The prediction process may include a boundary smoothing process for smoothing the boundary between the first and second partitions. For example, the boundary smoothing process may involve weighting a first value of the boundary pixels predicted based on the first partition and a second value of the boundary pixels predicted based on the second partition. Various implementations of the boundary smoothing process will be referenced below. Figure 13 , 14 It is described using 20 and 21A-21D.

[0276] In step S1005, the process encodes or decodes the image patch using one or more parameters, including partitioning parameters indicating that the image patch is divided into a second partition and a first partition having a non-rectangular shape. For example, in Figure 15 The table summarizes, for example, that the partitioning parameter (“first index value”) can relate to, for example, the partitioning direction applied in the partitioning (e.g., from top left to bottom right or from top right to bottom left, as...). Figure 12 (As shown) and the first and second motion vectors obtained in step S1002 above are jointly encoded. The following will refer to Figure 15 , 16 Sections 22-25 describe in detail the partitioning syntax operations involving one or more parameters, including partition parameters.

[0277] Figure 17 This is a flowchart illustrating the process 2000 of segmenting image blocks. In step S2001, the process segments the image into multiple partitions, including a first partition with a non-rectangular shape and a second partition that may or may not have a non-rectangular shape. (As shown in...) Figure 12 As shown, an image patch can be segmented into a first partition with a triangular shape and a second partition that also has a triangular shape. There are many other examples where the image patch is segmented into multiple partitions including the first and second partitions, wherein at least the first partition has a non-rectangular shape. The non-rectangular shape can be a triangle, a trapezoid, or a polygon with at least five sides and angles.

[0278] For example, as in Figure 18 As shown, an image patch can be divided into two triangular partitions; an image patch can be divided into more than two triangular partitions (e.g., three triangular partitions); an image patch can be divided into a combination of triangular and rectangular partitions; or an image patch can be divided into a combination of triangular and polygonal partitions.

[0279] As in Figure 19As further shown, an image patch can be divided into L-shaped (polygonal) partitions and rectangular partitions; an image patch can be divided into pentagonal (polygonal) partitions and triangular partitions; an image patch can be divided into hexagonal (polygonal) partitions and pentagonal (polygonal) partitions; or an image patch can be divided into multiple polygonal partitions.

[0280] Reference Figure 17 In step S2002, the process predicts a first motion vector for the first partition, for example, by selecting a first partition from a first set of motion vector candidates, and predicts a second motion vector for the second partition, for example, by selecting a second motion vector from a second set of motion vector candidates. For example, the first set of motion vector candidates may include motion vectors from partitions adjacent to the first partition, while the second set of motion vector candidates may include motion vectors from partitions adjacent to the second partition. Adjacent partitions may be one or both of spatially adjacent and temporally adjacent partitions. Some examples of spatially adjacent partitions include partitions located to the left, lower left, lower, lower right, right, upper right, upper, or upper left of the partition being processed. Examples of temporally adjacent partitions are partitions located at the same position in a reference image of an image patch.

[0281] In various implementations, the partitions adjacent to the first partition and the partitions adjacent to the second partition can be outside the image patch from which the first and second partitions are divided. The first set of motion vector candidates can be the same as or different from the second set of motion vector candidates. Furthermore, at least one of the first and second sets of motion vector candidates can be the same as another set—a third set of motion vector candidates—prepared for the image patch.

[0282] In some implementations, in step S2002, in response to determining that the second partition, similar to the first partition, also has a non-rectangular shape (e.g., a triangle), process 2000 creates a second set of motion vector candidates (for the non-rectangular second partition), which includes motion vectors of partitions adjacent to the second partition (excluding the first partition) (i.e., excluding the motion vectors of the first partition). On the other hand, in response to determining that the second partition (different from the first partition) has a rectangular shape, process 2000 creates a second set of motion vector candidates (for the rectangular second partition), which includes motion vectors of partitions adjacent to the second partition (including the first partition).

[0283] In step S2003, the process uses the first motion vector obtained in step S2002 above to encode or decode the first partition, and uses the second motion vector obtained in step S2002 above to encode or decode the second partition.

[0284] picture Figure 17 The process is the same as in 2000; the image block segmentation process can be performed by the image encoder, for example, in... Figure 1 As shown, the image encoder includes circuitry and a memory coupled to the circuitry. The circuitry performs the following operations: dividing an image block into multiple partitions, including a second partition and a first partition having a non-rectangular shape (step S2001); predicting a first motion vector for the first partition and a second motion vector for the second partition (step S2002); encoding the first partition using the first motion vector and encoding the second partition using the second motion vector (step S2003).

[0285] According to another embodiment, such as Figure 1 As shown, an image encoder is provided, comprising: a segmenter 102 that receives an original image and segments it into multiple blocks; a subtractor 104 that receives blocks from the segmenter and predictions from a prediction controller 128, and subtracts each prediction from its corresponding block to output a residual; a transformer 106 that performs a transform on the residuals output from the subtractor 104 to output transform coefficients; a quantizer 108 that quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder 110 that encodes the quantized transform coefficients to generate a bitstream; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124, and memories 118 and 122, wherein the inter-frame predictor 126 generates a prediction for the current block based on a reference block in an encoded reference image, and the intra-frame predictor 124 generates a prediction for the current block based on an encoded reference block in the current image. In operation, the predictive controller 128 divides the block into multiple partitions, including a second partition and a first partition with a non-rectangular shape. Figure 17 (Step S2001); Predict the first motion vector of the first partition and the second motion vector of the second partition (Step S2002); Encode the first partition using the first motion vector and encode the second partition using the second motion vector (Step S2003).

[0286] According to another embodiment, such as in Figure 10 As shown, an image decoder is provided, which includes circuitry and a memory coupled to the circuitry. The circuitry performs the following in operation: dividing an image patch into multiple partitions, said multiple partitions including a second partition and a first partition having a non-rectangular shape. Figure 17 (Step S2001); Predict the first motion vector of the first partition and the second motion vector of the second partition (Step S2002); Use the first motion vector to decode the first partition and use the second motion vector to decode the second partition (Step S2003).

[0287] According to another embodiment, as follows is provided Figure 10The image decoder shown includes: an entropy decoder 202 that receives and decodes an encoded bitstream to obtain quantized transform coefficients; an inverse quantizer 204 and an inverse transformer 206 that inverse quantizes the quantized transform coefficients to obtain transform coefficients and performs an inverse transform on the transform coefficients to obtain residuals; an adder 208 that adds the residuals output from the inverse quantizer 204 and the inverse transformer 206 to a prediction output from the prediction controller 220 to reconstruct a block; and a prediction controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216, and memories 210 and 214, wherein the inter-frame predictor 218 generates a prediction for the current block based on a reference block in a decoded reference image, and the intra-frame predictor 216 generates a prediction for the current block based on a decoded reference block in the current image. The prediction controller 220 divides the image block into multiple partitions, including a second partition and a first partition having a non-rectangular shape. Figure 17 (Step S2001); Predict the first motion vector of the first partition and the second motion vector of the second partition (Step S2002); Use the first motion vector to decode the first partition and use the second motion vector to decode the second partition (Step S2003).

[0288] (Boundary smoothing)

[0289] As mentioned above Figure 11 As described in various embodiments, step S1004 of performing a prediction process on an image patch that is a combination of a second partition and a first partition having a non-rectangular shape (reconstructed) can involve applying a boundary smoothing process along the boundary between the first partition and the second partition.

[0290] For example, Figure 21B An example of a boundary smoothing process is shown, which involves weighting a first value and a second value of a boundary pixel, wherein the first value of the boundary pixel is predicted based on a first partition, and the second value of the boundary pixel is predicted based on a second partition.

[0291] Figure 20 This is a flowchart illustrating a global boundary smoothing process 3000 according to one embodiment, which involves weighting a first value of a boundary pixel predicted based on a first partition and a second value of a boundary pixel predicted based on a second partition. In step S3001, as in Figure 21A or the above Figure 12 , 18 As shown in Figure 19, the image patch is divided into a first partition and a second partition along the boundary, wherein at least the first partition has a non-rectangular shape.

[0292] In step S3002, a set of pixels in the first partition along the boundary ( Figure 21A A first prediction is made for a first value (e.g., color, brightness, transparency, etc.) of the "boundary pixels" in the image, wherein the first value is predicted first using information from the first partition. In step S3003, a second prediction is made for a second value of a (same) group of pixels along the boundary of the first partition, wherein the second prediction is made using information from the second partition. In some implementations, at least one of the first and second predictions is an inter-frame prediction process that predicts the first and second values ​​based on a reference partition in an encoded reference image. Figure 21D In some implementations, the prediction process predicts a first value (“first set of samples”) of all pixels in a first partition that includes a set of pixels on which the first and second partitions overlap, and predicts a second value (“second set of samples”) of only a set of pixels on which the first and second partitions overlap. In another implementation, at least one of the first and second predictions is an intra-frame prediction process that predicts the first and second values ​​based on an encoded reference partition in the current image. In some implementations, the prediction method used in the first prediction differs from the prediction method used in the second prediction. For example, the first prediction may include an inter-frame prediction process, and the second prediction may include an intra-frame prediction process. The information used for the first prediction of the first value or the second prediction of the second value may be the motion vector of the first or second partition, the intra-frame prediction direction, etc.

[0293] In step S3004, the first value predicted using the first partition and the second value predicted using the second partition are weighted. In step S3005, the weighted first and second values ​​are used to encode or decode the first partition.

[0294] Figure 21B An example of a boundary smoothing operation is shown, where the first and second partitions overlap by a maximum of five pixels per row or column. That is, the number of pixels in each row or column that predicts the first value based on the first partition and the second value based on the second partition is at most five. Figure 21C Another example of a boundary smoothing operation is shown, where the first and second partitions overlap by a maximum of three pixels per row or column. That is, the number of pixels in each row or column whose first value is predicted based on the first partition and whose second value is predicted based on the second partition is at most three.

[0295] Figure 13Another example of a boundary smoothing operation is shown, where the first and second partitions overlap at most four pixels per row or column. That is, the number of pixels in a group per row or column that predicts a first value based on the first partition and a second value based on the second partition is at most four. In the example shown, weights of 1 / 8, 1 / 4, 3 / 4, and 7 / 8 can be applied to the first value of the four pixels in the group, respectively, and weights of 7 / 8, 3 / 4, 1 / 4, and 1 / 8 can be applied to the second value of the four pixels in the group, respectively.

[0296] Figure 14 Further examples of the boundary smoothing operation are shown, wherein the first and second partitions overlap at zero pixels per row or column (i.e., they do not overlap), overlap at one pixel per row or column (at most), and overlap at two pixels per row or column (at most). In the example where the first and second partitions do not overlap, zero weights are applied. In the example where the first and second partitions overlap at one pixel per row or column, a weight of 1 / 2 can be applied to a first value of the pixels in the group based on the prediction of the first partition, and a weight of 1 / 2 can be applied to a second value of the pixels in the group based on the prediction of the second partition. In the example where the first and second partitions overlap at two pixels per row or column, weights of 1 / 3 and 2 / 3 can be applied to the first values ​​of the two pixels in the group based on the prediction of the first partition, and weights of 2 / 3 and 1 / 3 can be applied to the second values ​​of the two pixels in the group based on the prediction of the second partition, respectively.

[0297] According to the above embodiment, the number of overlapping pixels in the first and second partitions of the group is an integer. In other implementations, for example, the number of overlapping pixels in the group can be a non-integer and can be a fraction. Similarly, depending on each application, the weights applied to the first and second values ​​of a group of pixels can be fractions or integers.

[0298] Similar to Figure 20 The process 3000, the boundary smoothing process can be, for example, as... Figure 1 The image encoder shown includes circuitry and memory coupled to the circuitry. In operation, the circuitry performs a boundary smoothing operation along the boundary between a second partition segmented from an image block and a first partition having a non-rectangular shape. Figure 20 (Step S3001). The boundary smoothing operation includes: using information from the first partition, performing a first prediction on a first value of a set of pixels in the first partition along the boundary (step S3002); using information from the second partition, performing a second prediction on a second value of a set of pixels in the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); and encoding the first partition using the weighted first value and the weighted second value (step S3005).

[0299] According to another embodiment, such as in Figure 1 As shown, an image encoder is provided, comprising: a segmenter 102 that receives an original image and segments it into blocks; a subtractor 104 that receives blocks from the segmenter and predictions from a prediction controller 128, and subtracts each prediction from its corresponding block to output a residual; a transformer 106 that performs a transform on the residuals output from the subtractor 104 to output transform coefficients; a quantizer 108 that quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder 110 that encodes the quantized transform coefficients to generate a bitstream; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124, and memories 118, 122, wherein the inter-frame predictor 126 generates a prediction for the current block based on a reference block in an encoded reference image, and the intra-frame predictor 124 generates a prediction for the current block based on an encoded reference block in the current image. The prediction controller 128 performs a boundary smoothing operation along the boundary between the second partition segmented from the image patch and the first partition having a non-rectangular shape. Figure 20 (Step S3001). The boundary smoothing operation includes: using information from the first partition, performing a first prediction on a first value of a set of pixels in the first partition along the boundary (step S3002); using information from the second partition, performing a second prediction on a second value of a set of pixels in the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); and encoding the first partition using the weighted first value and the weighted second value (step S3005).

[0300] According to another embodiment, an image decoder is provided, such as in Figure 10 As shown, it includes circuitry and a memory coupled to the circuitry. In operation, the circuitry performs a boundary smoothing operation along the boundary between a second partition segmented from an image patch and a first partition having a non-rectangular shape. Figure 20 (Step S3001). The boundary smoothing operation includes: using information from the first partition, performing a first prediction on a first value of a set of pixels in the first partition along the boundary (step S3002); using information from the second partition, performing a second prediction on a second value of a set of pixels in the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); and decoding the first partition using the weighted first value and the weighted second value (step S3005).

[0301] According to another embodiment, a method is provided as follows: Figure 10The image decoder shown includes: an entropy decoder 202 that receives and decodes an encoded bitstream to obtain quantized transform coefficients; an inverse quantizer 204 and an inverse transformer 206 that inverse quantizes the quantized transform coefficients to obtain transform coefficients and performs an inverse transform on the transform coefficients to obtain residuals; an adder 208 that adds the residuals output from the inverse quantizer 204 and the inverse transformer 206 to a prediction output from the prediction controller 220 to reconstruct a block; and a prediction controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216, and memories 210 and 214, wherein the inter-frame predictor 218 generates a prediction for the current block based on a reference block in a decoded reference image, and the intra-frame predictor 216 generates a prediction for the current block based on a decoded reference block in the current image. The prediction controller 220 performs a boundary smoothing operation along the boundary between a second partition segmented from an image block and a first partition having a non-rectangular shape. Figure 20 (Step S3001). The boundary smoothing operation includes: using information from the first partition, performing a first prediction on a first value of a set of pixels in the first partition along the boundary (step S3002); using information from the second partition, performing a second prediction on a second value of a set of pixels in the first partition along the boundary (step S3003); weighting the first value and the second value (step S3004); and decoding the first partition using the weighted first value and the weighted second value (step S3005).

[0302] (Using partition parameter syntax for entropy encoding and decoding)

[0303] As in Figure 11 As shown, in step S1005, according to various embodiments, one or more parameters can be used to encode or decode image patches segmented into a second partition and a first partition having a non-rectangular shape, said one or more parameters including partitioning parameters indicating the non-rectangular segmentation of the image patches. In various embodiments, such partitioning parameters can be applied, for example, to the segmentation direction (e.g., from top left to bottom right or from top right to bottom left, see...). Figure 12 The first motion vector and the second motion vector predicted in step S1002 are jointly encoded, as will be described more fully below.

[0304] Figure 15 This is a table of sampling partition parameters (“first index values”) and a set of information jointly encoded by the partition parameters. The partition parameters (“first index values”) range from 0 to 6 and jointly encode the following: dividing the image patch into a first partition and a second partition (both are triangles, see [link]). Figure 12 The direction of the first motion vector predicted for the first partition ( Figure 11Step S1002), and predicting the second motion vector for the second partition ( Figure 11 (Step S1002). Specifically, the segmentation direction encoded by partition parameter 0 is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "first" motion vector listed in the second group of motion vector candidates in the second partition.

[0305] Partition parameter 1 is encoded as follows: the segmentation direction is from the upper right corner to the lower left corner, the first motion vector is the "first" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "second" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 2 is encoded as follows: the segmentation direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "first" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 3 is encoded as follows: the segmentation direction is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "second" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 4 is encoded as follows: the segmentation direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "third" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 5 is encoded as follows: the segmentation direction is from the top left corner to the bottom right corner, the first motion vector is the "third" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "first" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 6 is encoded as follows: the segmentation direction is from the top left corner to the bottom right corner, the first motion vector is the "fourth" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "first" motion vector listed in the second group of motion vector candidates in the second partition.

[0306] Figure 22 This is a flowchart illustrating method 4000 performed on the encoder side. In step S4001, the process divides an image patch into multiple partitions based on partitioning parameters indicating segmentation, the multiple partitions including a second partition and a first partition having a non-rectangular shape. For example, as described above... Figure 15As shown, the partitioning parameters can indicate the direction of segmenting the image patch (e.g., from the upper right corner to the lower left corner or from the upper left corner to the lower right corner). In step S4002, the process encodes the first and second partitions. In step S4003, the process writes one or more parameters, including the partitioning parameters, into a bitstream. The decoder side can receive and decode the bitstream to obtain one or more parameters, thereby performing the same prediction process (as performed on the encoder side) on the first and second partitions. The one or more parameters including the partitioning parameters can jointly or separately encode various information fragments, such as the non-rectangular shape of the first partition, the shape of the second partition, the segmentation direction for segmenting the image patch to obtain the first and second partitions, the first motion vector of the first partition, the second motion vector of the second partition, etc.

[0307] Figure 23 This is a flowchart illustrating method 5000 performed on the decoder side. In step S5001, the process parses one or more parameters from the bitstream, wherein the one or more parameters include partitioning parameters indicating that an image patch is divided into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape. The one or more parameters, including the partitioning parameters parsed from the bitstream, can jointly or separately encode various information fragments required for performing the same prediction processing on the decoder side as on the encoder side, such as the non-rectangular shape of the first partition, the shape of the second partition, the segmentation direction for segmenting the image patch to obtain the first and second partitions, a first motion vector of the first partition, a second motion vector of the second partition, etc. In step S5002, process 5000 segments the image patch into multiple partitions based on the partitioning parameters parsed from the bitstream. In step S5003, the process decodes the first and second partitions segmented from the image patch.

[0308] Figure 24 It is a table of sampling partition parameters (“first index value”) and information sets jointly encoded by the partition parameters, and is essentially similar to the above. Figure 15 The sampling table in [the document / reference table]. Figure 24 In this context, the partitioning parameter (“first index value”) ranges from 0 to 6, and the following items are jointly encoded: the shape of the first and second partitions segmented from the image patch, the orientation in which the image patch is segmented into the first and second partitions, and the first motion vector predicted for the first partition. Figure 11 Step S1002), and the second motion vector predicted for the second partition ( Figure 11 (Step S1002). Specifically, the partition parameter 0 is encoded as follows: neither the first partition nor the second partition has a triangular shape, and therefore, the partitioning direction information is "N / A", the first motion vector information is "N / A", and the second motion vector information is "N / A".

[0309] Partition parameter 1 is encoded as follows: the first and second partitions are triangles, the segmentation direction is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "first" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 2 is encoded as follows: the first and second partitions are triangles, the segmentation direction is from the upper right corner to the lower left corner, the first motion vector is the "first" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "second" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 3 is encoded as follows: the first and second partitions are triangles, the segmentation direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "first" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 4 is encoded as follows: the first and second partitions are triangles, the segmentation direction is from the upper left corner to the lower right corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "second" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 5 is encoded as follows: the first and second partitions are triangles, the segmentation direction is from the upper right corner to the lower left corner, the first motion vector is the "second" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "third" motion vector listed in the second group of motion vector candidates in the second partition. Partition parameter 6 is encoded as follows: the first and second partitions are triangles, the segmentation direction is from the upper left corner to the lower right corner, the first motion vector is the "third" motion vector listed in the first group of motion vector candidates in the first partition, and the second motion vector is the "first" motion vector listed in the second group of motion vector candidates in the second partition.

[0310] According to some implementations, partition parameters (index values) can be binaryized according to a binaryization scheme selected based on the value of at least one or more parameters. Figure 16 An example binary conversion scheme for binaryizing index values ​​(partition parameter values) is shown.

[0311] Figure 25 This is a table of sampled combinations of the first and second parameters, where one parameter is a partitioning parameter indicating the segmentation of an image patch into multiple partitions, including a second partition and a first partition with a non-rectangular shape. In this example, the partitioning parameter can be used to indicate the segmentation of the image patch without jointly encoding other information encoded by one or more other parameters.

[0312] exist Figure 25 In the first example, the first parameter is used to indicate the image patch size, and the second parameter is used as a partitioning parameter (flag) to indicate that at least one of the multiple partitions divided from the image patch has a triangular shape. Such a combination of the first and second parameters can be used to indicate, for example, 1) that there are no triangular partitions when the image patch size is greater than 64×64, or 2) that there are no triangular partitions when the aspect ratio of the image patch is greater than 4 (e.g., 64×4).

[0313] exist Figure 25 In the second example, the first parameter is used to indicate the prediction mode, and the second parameter is used as a partitioning parameter (flag) to indicate that at least one of the multiple partitions segmented from the image patch has a triangular shape. Such a combination of the first and second parameters can be used to indicate, for example: 1) when the image patch is encoded in intra-frame mode, there are no triangular partitions.

[0314] exist Figure 25 In the third example, the first parameter is used as a partitioning parameter (flag) to indicate that at least one of the multiple partitions segmented from the image patch has a triangular shape, and the second parameter is used to indicate the prediction mode. Such a combination of the first and second parameters can be used to indicate, for example: 1) when at least one of the multiple partitions segmented from the image patch has a triangular shape, the image patch must be inter-frame coded.

[0315] exist Figure 25 In the fourth example, the first parameter indicates the motion vector of the adjacent block, and the second parameter is used as a segmentation parameter to indicate the direction in which the image block is divided into two triangles. Such a combination of the first and second parameters can be used to indicate, for example: 1) when the motion vectors of the adjacent blocks are diagonally aligned, the direction in which the image block is divided into two triangles is from the upper left corner to the lower right corner.

[0316] exist Figure 25 In the fifth example, the first parameter indicates the intra-prediction direction of adjacent blocks, and the second parameter is used as a partitioning parameter to indicate the direction in which the image block is divided into two triangles. Such a combination of the first and second parameters can be used to indicate, for example: 1) when the intra-prediction direction of adjacent blocks is an anti-diagonal direction, the direction in which the image block is divided into two triangles is from the upper right corner to the lower left corner.

[0317] It should be understood that, such as Figure 15 , 24As shown in Figure 25, the table including partitioning parameters and which information is jointly or individually encoded is presented merely as an example, and many other ways of jointly or individually encoding various information as part of the partitioning syntax operations described above are also within the scope of this disclosure. For example, partitioning parameters may indicate that the first partition is a triangle, trapezoid, or a polygon with at least five sides and angles. Partitioning parameters may indicate that the second partition has a non-rectangular shape, such as a triangle, trapezoid, or a polygon with at least five sides and angles. Partitioning parameters may indicate one or more pieces of information about the segmentation, such as the non-rectangular shape of the first partition, the shape of the second partition (which may be non-rectangular or rectangular), and the segmentation direction applied to segment the image patch into multiple partitions (e.g., from the upper left corner of the image patch to its lower right corner, and from the upper right corner of the image patch to its lower left corner). Partitioning parameters may jointly encode other information, such as a first motion vector of the first partition, a second motion vector of the second partition, the image patch size, the prediction mode, the motion vectors of neighboring patches, the intra-frame prediction directions of neighboring patches, etc. Alternatively, any information in the further information may be encoded separately by one or more parameters other than the partitioning parameters.

[0318] Similar to Figure 22 The process 4000, partitioning syntax operations can be performed by, for example Figure 1 The image encoder shown includes circuitry and a memory coupled to the circuitry. The circuitry performs a partitioning syntax operation during operation, which includes: dividing an image patch into multiple partitions based on partitioning parameters indicating the segmentation, the multiple partitions including a second partition and a first partition having a non-rectangular shape. Figure 22 Step S4001); Encode the first partition and the second partition (S4002); Write one or more parameters, including partition parameters, into the bit stream (S4003).

[0319] According to another embodiment, such as Figure 1As shown, an image encoder is provided, comprising: a segmenter 102 that receives an original image and segments it into blocks; a subtractor 104 that receives blocks from the segmenter and predictions from a prediction controller 128, and subtracts each prediction from its corresponding block to output a residual; a transformer 106 that performs a transform on the residuals output from the subtractor 104 to output transform coefficients; a quantizer 108 that quantizes the transform coefficients to generate quantized transform coefficients; an entropy encoder 110 that encodes the quantized transform coefficients to generate a bitstream; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124, and memories 118 and 122, wherein the inter-frame predictor 126 generates a prediction for the current block based on a reference block in an encoded reference image, and the intra-frame predictor 124 generates a prediction for the current block based on an encoded reference block in the current image. In operation, the prediction controller 128 segments an image patch into multiple partitions based on partitioning parameters indicating segmentation, the multiple partitions including a second partition and a first partition having a non-rectangular shape. Figure 22 (Step S4001), and encode the first partition and the second partition (Step S4002). The entropy encoder 110 writes one or more parameters, including partition parameters, into the bit stream during operation (Step S4003).

[0320] According to another embodiment, an image decoder is provided, such as... Figure 10 As shown, the image decoder includes circuitry and a memory coupled to the circuitry. The circuitry performs a partitioning syntax operation in operation, the partitioning syntax operation including: parsing one or more parameters from a bitstream, wherein the one or more parameters include partitioning parameters indicating the division of an image patch into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape. Figure 23 Step S5001); Divide the image block into multiple partitions based on the partitioning parameters (S5002); Decode the first partition and the second partition (S5003).

[0321] According to another embodiment, as follows is provided Figure 10The image decoder shown includes: an entropy decoder 202 that receives and decodes an encoded bitstream to obtain quantized transform coefficients; an inverse quantizer 204 and an inverse transformer 206 that inverse quantizes the quantized transform coefficients to obtain transform coefficients and performs an inverse transform on the transform coefficients to obtain residuals; an adder 208 that adds the residuals output from the inverse quantizer 204 and the inverse transformer 206 to a prediction output from the prediction controller 220 to reconstruct a block; and a prediction controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216, and memories 210 and 214, wherein the inter-frame predictor 218 generates a prediction for the current block based on a reference block in a decoded reference image, and the intra-frame predictor 216 generates a prediction for the current block based on a decoded reference block in the current image. In operation, the entropy decoder 202 parses one or more parameters from the bitstream, wherein the one or more parameters include partitioning parameters indicating the division of an image patch into multiple partitions, the multiple partitions including a second partition and a first partition having a non-rectangular shape. Figure 23 (S5001) The image patch is divided into multiple partitions based on the partitioning parameters (S5002); In some implementations, the first and second partitions are decoded in cooperation with the prediction controller 220 (S5003).

[0322] Based on other examples, an intra-frame predictor can perform the following process.

[0323] All motion vector candidates included in the first group of motion vector candidates can be single predicted motion vectors. That is, the intra-frame predictor can identify only single predicted motion vectors as motion vector candidates in the first group of motion vector candidates.

[0324] The intra-frame predictor can select only a single predicted motion vector candidate from the first set of motion vector candidates.

[0325] A single predicted motion vector can be used to predict small patches. Two predicted motion vectors can be used to predict large patches. For example, the prediction process may include determining the size of an image patch. When the image patch size is determined to be greater than a threshold, the prediction may include selecting a first motion vector from a first set of motion vector candidates, and the first set of motion vector candidates may contain single-predicted and / or double-predicted motion vectors. When the image patch size is determined to be no greater than a threshold, the prediction may select a first motion vector from the first set of motion vector candidates, and the first set of motion vector candidates may contain only a single-predicted motion vector.

[0326] (Implementation and Application)

[0327] As described in each of the embodiments above, each functional or operational block can typically be implemented as, for example, an MPU (microprocessor unit) and memory. Furthermore, the process executed by each functional block can be implemented as a program execution unit, such as a processor that reads and executes software (programs) recorded on a recording medium such as ROM. This software can be distributed. The software can be recorded on various recording media such as semiconductor memory. Note that each functional block can also be implemented as hardware (dedicated circuitry).

[0328] The processes described in each embodiment can be implemented through integrated processing using a single device (system), or through decentralized processing using multiple devices. Furthermore, the processor executing the above programs can be a single processor or multiple processors. In other words, integrated processing or decentralized processing can be performed.

[0329] The embodiments disclosed herein are not limited to the exemplary embodiments described above; various modifications can be made to the exemplary embodiments, and the results are also included within the scope of the embodiments disclosed herein.

[0330] Next, application examples of the motion picture encoding method (image encoding method) and motion picture decoding method (image decoding method) described in each of the above embodiments will be described, along with various systems for implementing these application examples. Such systems may include an image encoder employing the image encoding method, an image decoder employing the image decoding method, or both an image encoder and an image decoder. Other configurations of such systems can be modified as needed.

[0331] (Usage example)

[0332] Figure 26 The overall configuration of a content delivery system ex100 suitable for implementing content distribution services is shown. The area providing communication services is divided into cells of desired size, and in the example shown, base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations, are located in the respective cells.

[0333] In the content providing system ex100, devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via an Internet service provider ex102 or a communication network ex104 and base stations ex106 to ex110. The content providing system ex100 can combine and connect any combination of the aforementioned devices. In various implementations, the devices can be connected directly or indirectly via a telephone network or near-field communication instead of via base stations ex106 to ex110. Furthermore, a streaming media server ex103 can connect to the devices including the computer ex111, gaming device ex112, camera ex113, home appliance ex114, and smartphone ex115 via, for example, the Internet ex101. The streaming media server ex103 can also connect to a terminal in a hotspot, such as an airplane ex117, via satellite ex116.

[0334] Note that wireless access points or hotspots can be used instead of base stations ex106 to ex110. The streaming media server ex103 can connect directly to the communication network ex104 instead of via the internet ex101 or an internet service provider ex102, and can connect directly to the aircraft ex117 instead of via satellite ex116.

[0335] The EX113 camera is a device capable of capturing still images and videos, such as a digital camera. The EX115 smartphone is a smartphone device, cellular phone, or Personal Handheld Phone System (PHS) phone that can operate under mobile communication system standards of 2G, 3G, 3.9G, 4G systems, and the next-generation 5G system.

[0336] Household appliances, such as refrigerators or devices included in a home fuel cell cogeneration system, are examples of such appliances.

[0337] In the content delivery system ex100, a terminal including image and / or video capture capabilities can perform real-time streaming, for example, by connecting to a streaming media server ex103 via a base station ex106. During real-time streaming, the terminal (e.g., a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or an airplane ex117) can perform the encoding processing described in the above embodiments on still images or video content captured by the user via the terminal, can multiplex the video data obtained through encoding and the audio data obtained by encoding the audio corresponding to the video, and can send the obtained data to the streaming media server ex103. In other words, the terminal acts as an image encoder according to one aspect of this disclosure.

[0338] The streaming media server ex103 streams content data to the client requesting the stream. Examples of clients include a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and a terminal within an airplane ex117, all capable of decoding the encoded data. The device receiving the streamed data decodes and reproduces the received data. In other words, each of these devices can act as an image decoder according to one aspect of this disclosure.

[0339] (Decentralized processing)

[0340] The streaming media server ex103 can be implemented as multiple servers or computers that divide tasks such as data processing, recording, and streaming among themselves. For example, the streaming media server ex103 can be implemented as a Content Delivery Network (CDN) that streams content via a network connecting multiple edge servers located around the world. In a CDN, edge servers physically close to the client are dynamically assigned to the client. Content is cached and streamed to the edge servers to reduce loading time. For example, in the event of some type of error or connectivity change due to traffic spikes, data can be streamed stably at high speed because the affected parts of the network can be avoided by, for example, dividing processing among multiple edge servers, or switching streaming responsibilities to different edge servers and continuing streaming.

[0341] Decentralization is not limited to partitioning processing for streaming; the encoding of captured data can be distributed among terminals on the server side, performed by the terminals themselves, or both. In one example, in typical encoding, this processing is performed in two loops. The first loop is used to detect the complexity of the image, or the encoding load, on a frame-by-frame or scene-by-scene basis. The second loop is used to maintain image quality and improve encoding efficiency. For example, it is possible to reduce the processing burden on the terminal and improve the quality and efficiency of the content by having the terminal perform the first loop of encoding, and having the server receiving the content perform the second loop of encoding. In such a case, upon receiving a decoding request, it is possible for the encoded data resulting from the first loop performed on one terminal to be received and reproduced on another terminal in near real-time. This makes smooth, real-time streaming possible.

[0342] In another example, a camera such as ex113 extracts features from an image, compresses data related to those features into metadata, and sends the compressed metadata to a server. For instance, the server determines the importance of objects based on the features and adjusts the quantization precision accordingly to perform compression appropriate to the meaning (or content importance) of the image. Feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction during the second compression pass performed by the server. Furthermore, the terminal can handle codes with relatively low processing overhead, such as variable-length coding (VLC), while codes with relatively high processing overhead, such as context-adaptive binary arithmetic coding (CABAC), can be handled by the server.

[0343] In another example, there exists a situation where multiple terminals in a stadium, shopping mall, or factory capture multiple videos of approximately the same scene. In such cases, for example, processing tasks can be decentralized by dividing the encoding on a unit basis among the multiple terminals capturing the video and, if necessary, other terminals not capturing video, and the server. The unit can be, for example, a group of pictures (GOP), a picture, or tiles obtained by dividing pictures. This reduces loading time and enables streaming that is closer to real-time.

[0344] Because the videos share largely similar scenes, the server can manage and / or instruct the cross-referencing of videos captured by the terminal. Furthermore, the server can receive encoded data from the terminal, alter the reference relationships between data items, or correct or replace images itself, and then perform encoding. This allows for the generation of streams with improved quality and efficiency for individual data items.

[0345] Furthermore, the server can stream video data after performing code conversion to change the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (e.g., VP9) and can convert H.264 to H.265.

[0346] In this way, encoding can be performed by a terminal or one or more servers. Therefore, although the device performing encoding is referred to as a "server" or "terminal" in the following description, some or all of the processes performed by a server can be performed by a terminal, and similarly, processing performed by a terminal can be performed by a server. The same applies to the decoding process.

[0347] (3D, multi-angle)

[0348] The use of images or videos captured simultaneously from different scenes or from the same scene captured from different angles, through multiple terminals (e.g., camera ex113 and / or smartphone ex115), has increased. Videos captured by a terminal are combined based, for example, on relative positional relationships obtained individually between the terminals or on regions in the video that have matching feature points.

[0349] In addition to encoding 2D moving images, the server can also automatically or at user-specified time points encode still images based on scene analysis of the moving images, and send the encoded still images to the receiving terminal. Furthermore, when the server can obtain the relative positional relationships between video capture terminals, it can generate the 3D geometry of a scene based on videos of the same scene captured from different angles, in addition to 2D moving images. The server can encode 3D data generated from, for example, point clouds, and based on the results of using 3D data to identify or track people or objects, the server can select or reconstruct videos captured from multiple terminals and generate videos to be sent to the receiving terminal.

[0350] This allows users to enjoy the scene by freely selecting the video corresponding to the video capture terminal, and also allows users to enjoy content obtained by extracting video from 3D data reconstructed from multiple images or videos at a selected viewpoint. Furthermore, similar to video, sound can be recorded from relatively different angles, and the server can multiplex audio from a specific angle or space with the corresponding video, and send the multiplexed video and audio.

[0351] In recent years, content that combines the real and virtual worlds, such as virtual reality (VR) and augmented reality (AR) content, has become increasingly popular. In the case of VR images, a server can create images from both the left and right eye viewpoints and perform encoding that allows for reference between the two viewpoint images, such as multi-view encoding (MVC). Alternatively, the images can be encoded as separate streams without reference. When the images are decoded into separate streams, these streams can be synchronized during playback to recreate the virtual 3D space according to the user's viewpoint.

[0352] In the case of AR images, the server overlays information about virtual objects existing in virtual space onto camera information representing real space, based on the 3D position or motion from the user's perspective. The decoder can acquire or store the virtual object information and 3D data, generate a 2D image based on the motion from the user's perspective, and then generate the overlay data by seamlessly connecting the images. Alternatively, in addition to requesting virtual object information, the decoder can send the motion from the user's perspective to the server. The server can generate the overlay data based on the received motion and the 3D data stored on the server, and encode and stream the generated overlay data to the decoder. Note that the overlay data includes α values ​​representing transparency in addition to RGB values, and the server sets the α value of parts other than the objects generated from the 3D data to, for example, 0, and can encode these parts as transparent. Alternatively, the server can set the background to predetermined RGB values ​​(e.g., chroma keys) and generate data that sets the area outside the object as the background.

[0353] Decoding of similarly streamed data can be performed by the client (i.e., the terminal) on the server side, or partitioned between them. In one example, a terminal can send a receive request to the server, the requested content can be received and decoded by another terminal, and the decoded signal can be sent to a device with a display. Regardless of the processing power of the communication terminal itself, high-quality image data can be reproduced through decentralized processing and appropriate content selection. In yet another example, for instance, when a TV is receiving large-format image data, regions of the image, such as tiles obtained by dividing the picture, can be decoded and displayed on a personal terminal or on the terminals of one or more viewers of the TV. This allows viewers to share a large image view, and each viewer can examine their assigned region or examine the region more closely.

[0354] In situations where multiple wireless connections are possible indoors, outdoors, at short, medium, and long distances, content can be seamlessly received using streaming system standards such as MPEG-DASH. Users can switch between data in real time, freely choosing decoders or display devices, such as indoor or outdoor displays, including the user terminal. Furthermore, using information such as the user's location, decoding can be performed simultaneously, switching between which terminal handles decoding and which handles content display. This makes it possible to draw and display information on walls or parts of the ground of nearby buildings with embedded devices capable of displaying content, while the user is en route to their destination. Additionally, the bitrate of received data can be switched based on the accessibility of encoded data on the network, such as when to cache encoded data on a server quickly accessible from the receiving terminal, or when to copy encoded data to an edge server within the content delivery service.

[0355] (Scalable encoding)

[0356] Reference Figure 27 The scalable stream shown describes content switching, which is compressed and encoded using the motion picture encoding method described in the above embodiments. The server can be configured to switch content while utilizing the temporal and / or spatial scalability of the stream, achieved by partitioning and encoding layers, such as... Figure 27 As shown. Note that multiple streams with identical content but different qualities can exist. In other words, by determining the layer to be decoded based on internal factors such as the processing power on the decoder side and external factors such as communication bandwidth, the decoder side can freely switch between low-resolution and high-resolution content while decoding. For example, if a user wants to continue watching a video they previously watched on their smartphone ex115 while on the go, for example at home using a device such as an internet-connected TV, the device can simply decode the same stream up to a different layer, which reduces the load on the server side.

[0357] In addition to the above configuration (where scalability is achieved due to layer-by-layer image encoding) and the fact that the enhancement layer sits above the base layer, the enhancement layer can also include metadata based on, for example, statistical information about the image. The decoder can generate high-quality content by performing super-resolution imaging on the images in the base layer based on the metadata. Super-resolution imaging can improve the signal-to-noise ratio while maintaining and / or increasing resolution. The metadata includes information identifying the coefficients of linear or nonlinear filters used in the super-resolution processing, or information identifying the parameter values ​​of filtering, machine learning, or least squares methods used in the super-resolution processing.

[0358] Alternatively, a configuration can be provided in which an image is segmented into tiles, for example, based on the meaning of objects within it. On the decoder side, only a portion of the image is decoded by selecting tiles to be decoded. Furthermore, by storing the attributes of objects (people, cars, balls, etc.) and their positions in the video (coordinates within the same image) as metadata, the decoder side can identify the desired object's location based on this metadata and determine which tiles(s) contain that object. For example, as... Figure 28 As shown, metadata can be stored using a different data storage structure than pixel data, such as the SEI (Supplemental Enhancement Information) message in HEVC. This metadata indicates, for example, the position, size, or color of the main object.

[0359] Metadata can be stored in units of multiple images, such as streams, sequences, or random access units. The decoder can obtain, for example, the time when a specific person appears in the video, and by fitting the time information to the image unit information, it can identify images containing objects and determine the objects' positions within those images.

[0360] (Webpage optimization)

[0361] Figure 29 An example of a webpage display screen, such as on a computer ex111, is shown. Figure 30 This shows an example of a webpage display screen on a smartphone, such as the ex115. Figure 29 and Figure 30 As shown, a webpage may include multiple image links as links to image content, and the appearance of the webpage varies depending on the device used to view it. When multiple image links are visible on the screen, the display device (decoder) may display still images included in the content or I-images as image links until the user explicitly selects an image link, or until the image link is approximately centered on the screen, or until the entire image link fits the screen; multiple still images or I-images may be used to display videos such as animated GIFs; or only the base layer may be received, decoded, and displayed.

[0362] When a user selects an image link, the display device performs decoding while giving it the highest priority to the base layer. Note that if the webpage's HTML code contains information indicating that the content is scalable, the display device can decode up to the enhancement layer. Furthermore, to ensure real-time reproduction, before selection or when bandwidth is severely limited, the display device can reduce the delay between the time of decoding the leading image and the time of displaying the decoded image by only decoding and displaying the forward reference images (I images, P images, and forward reference B images). Alternatively, the display device can intentionally ignore the reference relationships between images and coarsely decode all B and P images into forward reference images, then perform regular decoding as the number of images received over a period of time increases.

[0363] (Autonomous driving)

[0364] When sending and receiving still image or video data (e.g., two-dimensional or three-dimensional map information for autonomous or assisted driving vehicles), the receiving terminal can receive image data belonging to one or more layers, as well as information about, for example, weather or road construction, as metadata, and associate this metadata with the image data during decoding. Note that the metadata can be distributed by layer and, alternatively, can be multiplexed only with the image data.

[0365] In this scenario, since the vehicle, drone, or aircraft containing the receiving terminal is mobile, the receiving terminal can seamlessly receive and perform decoding by sending information indicating its location, while simultaneously switching between base stations ex106 to ex110. Furthermore, based on user selection, user status, and / or connection bandwidth, the receiving terminal can dynamically select the degree to which it receives metadata or updates information such as map data.

[0366] In the content delivery system ex100, clients can receive, decode, and reproduce encoded information sent by users in real time.

[0367] (Streaming individual content)

[0368] In the content delivery system ex100, in addition to high image quality, long content distributed by video distribution entities, low image quality unicast or multicast streams, and short content from individuals are also possible. The popularity of such content from individuals is likely to increase further. The server can perform editing processing on the content before encoding to refine individual content. For example, this can be achieved using the following configuration.

[0369] While capturing video or image content in real time, or after capturing and accumulating content, the server performs recognition processing based on the raw or encoded data, such as capture error handling, scene search processing, meaning analysis, and / or object detection processing. Then, based on the results of the recognition processing, the server edits the content, either prompted or automatically. Examples include: corrections such as focus and / or motion blur correction; removal of low-priority scenes, such as scenes that are less bright than other images or out of focus; object edge adjustment; and tone adjustment. The server encodes the edited data based on the editing results. It is well known that excessively long videos lead to fewer views. Therefore, to keep the content within a specific length comparable to the original video, in addition to the aforementioned low-priority scenes, the server can also automatically crop scenes with low motion based on image processing results. Alternatively, the server can generate and encode video summaries based on the analysis of scene meaning.

[0370] In some cases, individual content may include content that infringes on copyright, moral rights, portrait rights, etc. Such instances can put creators at a disadvantage, for example, when sharing content beyond the scope they intend to cover. Therefore, before encoding, the server can, for example, edit the image to blur faces at the edges of the screen or blur the interior of a house. Furthermore, the server can be configured to identify faces of people other than registered individuals in the image to be encoded, and when such faces appear in the image, a mosaic filter can be applied, for example, to that person's face. Alternatively, as preprocessing or post-processing for encoding, the user can specify an image region that includes people or the background to be processed for copyright reasons. The server can process the specified region by, for example, replacing the region with a different image or blurring it. If the region includes a person, the person can be tracked in a moving image, and the person's head region can be replaced with another image as the person moves.

[0371] Because it requires real-time viewing of individually generated content, which tends to be relatively small in size, the decoder first receives the base layer as the highest priority and performs decoding and playback, although this can vary depending on bandwidth. When content is played back two or more times, for example, when the decoder receives an enhancement layer during the decoding and playback of the base layer and loops through that playback, the decoder can play back high-quality video including the enhancement layer. If the stream is encoded using such scalable coding, the video may be of lower quality when it is unselected or at the beginning of the video, but it can provide an experience where the image quality of the stream gradually improves in an intelligent way. This is not limited to scalable coding; the same experience can be provided by configuring a single stream from a low-quality stream played back the first time and a second stream encoded using that first stream as a reference.

[0372] (Other implementation and application examples)

[0373] Encoding and decoding can be performed by an LSI (Large Scale Integration) ex500 (see below), which is typically included in each terminal. Figure 26 The LSI ex500 can be configured as a single chip or multiple chips. Software for encoding and decoding moving images can be integrated into a type of recording medium (e.g., CD-ROM, floppy disk, or hard disk), which a computer ex111 can read and use to perform encoding and decoding. Furthermore, when the smartphone ex115 is equipped with a camera, video data acquired by that camera can be transmitted. In this case, the video data is encoded by the LSI ex500 included in the smartphone ex115.

[0374] Note that the LSIex500 can be configured to download and activate applications. In this case, the terminal first determines whether it is compatible with the encoding scheme used to encode the content, or whether it can perform a specific service. When the terminal is incompatible with the content's encoding scheme, or when the terminal cannot perform a specific service, the terminal first downloads the codec or application software, and then acquires and reproduces the content.

[0375] Besides the example of the content delivery system ex100 using the Internet ex101, the motion picture encoder (image encoder) or motion picture decoder (image decoder) described in at least the above embodiments can be implemented in a digital broadcasting system. The same encoding and decoding processes can be applied, for example, to transmitting and receiving broadcast radio waves superimposed with multiplexed audio and video data using satellites, even if this is for multicast, whereas unicast is easier for the content delivery system ex100.

[0376] (Hardware Configuration)

[0377] Figure 31 It shows Figure 26 More details about the smartphone ex115 shown. Figure 32An example configuration of the smartphone ex115 is shown. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves from a base station ex110, a camera ex465 capable of capturing video and still images, and a display ex458 for displaying decoded data, such as video captured by the camera ex465 and video received by the antenna ex450. The smartphone ex115 also includes a user interface ex466 such as a touch panel, an audio output unit ex457 such as a speaker for outputting voice or other audio, an audio input unit ex456 such as a microphone for audio input, a memory ex467 capable of storing decoded data such as captured video or still images, recorded audio, received video or still images, emails, and decoded data, and a slot ex464 which is an interface to a SIM ex468 for authorized access to the network and various data. Note that external memory can be used instead of ex467.

[0378] The main controller ex460, power supply circuit ex461, user interface input controller ex462, video signal processor ex455, camera interface ex463, display controller ex459, modulator / demodulator ex452, multiplexer / demultiplexer ex453, audio signal processor ex454, slot ex464, and memory ex467, which can fully control the display ex458 and user interface ex466, are connected via bus ex470.

[0379] When the user turns on the power button of the power circuit ex461, the smartphone ex115 is powered on and put into operation, and power is drawn from the battery pack to each component.

[0380] The smartphone ex115 performs processes such as call and data transmission based on control executed by the main controller ex460, which includes a CPU, ROM, and RAM. When a call is made, the audio signal recorded by the audio input unit ex456 is converted into a digital audio signal by the audio signal processor ex454. This digital audio signal is then spread-spectrum processed and digital-to-analog converted by the modulator / demodulator ex452, and frequency converted by the transmitter / receiver ex451. The resulting signal is transmitted via the antenna ex450. Received data is amplified, frequency converted, and analog-to-digital converted by the modulator / demodulator ex452, and subjected to inverse spread spectrum processing. It is then converted into an analog audio signal by the audio signal processor ex454 and output from the audio output unit ex457. In data transmission mode, for example, based on the operation of the user interface ex466, the main controller ex460 transmits text, still images, or video data via the user interface input controller ex462. Similar send and receive processes are performed. In data transmission mode, when transmitting video, still images, or video and audio, the video signal processor ex455 compresses and encodes the video signal stored in the memory ex467 or the video signal input from the camera ex467 using the motion picture encoding method described in the above embodiments, and sends the encoded video data to the multiplexer / demultiplexer ex453. The audio signal processor ex454 encodes the audio signal recorded by the audio input unit ex456 while the camera ex465 is capturing video or still images, and sends the encoded audio data to the multiplexer / demultiplexer ex453. The multiplexer / demultiplexer ex453 uses a predetermined scheme to multiplex the encoded video data and the encoded audio data, uses a modulator / demodulator (modulator / demodulator circuit) ex452 and a transmitter / receiver ex451 to modulate and convert the data, and transmits the results via the antenna ex450.

[0381] For example, when a video attached to an email or chat, or a video linked from a webpage, is received, in order to decode the multiplexed data received via antenna ex450, multiplexer / demultiplexer ex453 demultiplexes the multiplexed data to divide it into a bitstream of video data and a bitstream of audio data. Encoded video data is provided to video signal processor ex455 via synchronization bus ex470, and encoded audio data is provided to audio signal processor ex454 via synchronization bus ex470. Video signal processor ex455 decodes the video signal using a motion picture decoding method corresponding to the motion picture encoding method described in the above embodiments, and the video or still image included in the linked motion picture file is displayed on display ex458 via display controller ex459. Audio signal processor ex454 decodes the audio signal and outputs audio from audio output unit ex457. As real-time streaming becomes increasingly popular, audio reproduction may be socially inappropriate in some situations, depending on the user's environment. Therefore, as an initial value, it is preferable to configure the system to reproduce only video data and not audio signals; audio can only be synchronized and reproduced when input is received (e.g., when a user clicks on video data).

[0382] Although the example above uses the smartphone ex115, three other implementations are conceivable: a transceiver terminal that includes both an encoder and a decoder; a transmitter terminal that includes only an encoder; and a receiver terminal that includes only a decoder. The description of a digital broadcasting system provides an example where multiplexed data, obtained through the multiplexing of video and audio data, is received or transmitted. However, the multiplexed data can be video data multiplexed with data other than audio data (e.g., text data related to the video). Furthermore, the video data itself, rather than the multiplexed data, can be received or transmitted.

[0383] Although the main controller ex460, which includes the CPU, is described as controlling the encoding or decoding process, various terminals typically include a GPU. Therefore, a configuration is acceptable where large regions are processed immediately by leveraging the performance capabilities of the GPU, via memory shared by the CPU and GPU, or memory including addresses managed to allow general use by both the CPU and GPU. This reduces encoding time, maintains the real-time nature of the stream, and lowers latency. In particular, processing related to motion estimation, deblocking filtering, sample adaptive offset (SAO), and transform / quantization can be efficiently performed all at once, for example, by the GPU instead of the CPU, on a picture-by-picture basis.

Claims

1. An image encoder, comprising: Circuit; as well as A memory coupled to the circuit; The circuit performs the following operations during operation: Image patches are obtained from coding tree units (CTUs). For a first partition in the image block that has a non-rectangular shape, a first motion vector is selected from a set of motion vector candidates; For the second partition in the image block that overlaps with the first partition, a second motion vector is selected from the set of motion vector candidates; Use the first motion vector to predict a first value for the first partition; The first motion vector is used to encode the first partition; and The second motion vector is used to encode the second partition. The circuit selects only a single predicted motion vector from the set of motion vector candidates.

2. An image decoder, comprising: Circuit; as well as A memory coupled to the circuit; The circuit performs the following operations during operation: Image patches are obtained from coding tree units (CTUs). For a first partition in the image block that has a non-rectangular shape, a first motion vector is selected from a set of motion vector candidates; For the second partition in the image block that overlaps with the first partition, a second motion vector is selected from the set of motion vector candidates; Use the first motion vector to predict a first value for the first partition; The first motion vector is used to decode the first partition; and The second motion vector is used to decode the second partition; The circuit selects only a single predicted motion vector from the set of motion vector candidates.

3. A bitstream generation apparatus, comprising: Circuit; as well as A memory coupled to the circuit; The circuit performs the following operations during operation: Generate information that enables the decoder to perform partitioning processing, and Include the information in the bitstream. In the partitioning process: Image patches are obtained from coding tree units, or CTUs. A first motion vector for a first partition with a non-rectangular shape in the image patch is selected from a set of motion vector candidates; The second motion vector for the second partition in the image block that overlaps with the first partition is selected from the set of motion vector candidates; The first value of the first partition is predicted using the first motion vector; The first partition is decoded using the first motion vector; and The second partition is decoded using the second motion vector; Only a single predicted motion vector is selected from the set of motion vector candidates.

Images