Image encoder, image decoder and non-transitory computer readable medium

Abstract

The present invention provides an image encoder including circuitry and a memory coupled to the circuitry. During operation, the circuitry performs a boundary smoothing operation along a boundary between a first segmentation region and a second segmentation region having a non-rectangular shape (e.g., a triangular shape), the first segmentation region and the second segmentation region being split from an image block. The boundary smoothing operation includes: using information from the first segmentation region to first predict a first value for a set of pixels of the first segmentation region along the boundary; using information from the second segmentation region to second predict a second value for the set of pixels of the first segmentation region along the boundary; weighting the first value and the second value; and encoding the first segmentation region using the weighted first value and the weighted second value.

Description

Technical Field

[0001] Invention Field This invention relates to video coding, and more specifically to a video coding and decoding system, components, and methods for performing an inter-frame prediction function based on a reference frame to construct a current block or an intra-frame prediction function based on an encoded / decoded reference block within a current frame. Prior Technology

[0002] Background of the Invention With advancements in video encoding technology from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Encoding), MPEG-LA, H.265 / HEVC (High Efficiency Video Encoding), and H.266 / VVC (Versatile Video Codec), the enduring need remains for improvements and optimizations to video encoding technology to handle the ever-increasing volume of digital video data in various applications. This invention relates to further advancements, improvements, and optimizations in video encoding, specifically, by combining inter-frame prediction or intra-frame prediction functions to divide an image block into a plurality of segmented regions, each of which includes at least a first segmented region and a second segmented region having a non-rectangular shape (e.g., a triangle). Summary of the Invention

[0003] Invention Summary According to one embodiment, an image encoder is provided, including circuitry and a memory coupled to the circuitry. During operation, the circuitry performs a boundary smoothing operation along a boundary between a first segment and a second segment having a non-rectangular shape, the first and second segments being split from an image block. The boundary smoothing operation includes: using information from the first segment to first predict a first value for a set of pixels of the first segment along the boundary; using information from the second segment to second predict a second value for the set of pixels of the first segment along the boundary; weighting the first and second values; and encoding the first segment using the weighted first and second values.

[0004] Some embodiments of the present invention can improve encoding efficiency, simplify the encoding / decoding process, speed up the encoding / decoding process, and efficiently select appropriate components / operations used during encoding and decoding, such as appropriate filters, block sizes, motion vectors, reference images, reference blocks, etc.

[0005] Other benefits and advantages of the disclosed embodiments will become apparent from the description and drawings. Such benefits and / or advantages may be individually obtained by means of the various embodiments and features of the description and drawings, without all of such various embodiments and features needing to be provided in order to obtain one or more of such benefits and / or advantages.

[0006] It should be noted that general or specific embodiments may be implemented as systems, methods, integrated circuits, computer programs, storage media, or any alternative combination thereof. Simple Explanation of the Diagram

[0007] [Figure 1] Figure 1 is a block diagram illustrating the functional configuration of an encoder according to an embodiment.

[0008] [Figure 2] Figure 2 illustrates an example of block splitting.

[0009] [Figure 3] Figure 3 is a table indicating the basis functions of various transformation types.

[0010] [Figure 4A] Figure 4A illustrates an example of a filter shape used in an adaptive loop filter (ALF).

[0011] [Figure 4B] Figure 4B illustrates another example of filter shapes used in ALF.

[0012] [Figure 4C] Figure 4C illustrates another example of filter shapes used in ALF.

[0013] [Figure 5A] Figure 5A illustrates 67 in-frame prediction patterns in one example of in-frame prediction.

[0014] [Figure 5B] Figure 5B is a flowchart illustrating one example of a predictive image correction procedure performed in overlapping block motion compensation (OBMC) processing.

[0015] [Figure 5C] Figure 5C is a conceptual diagram illustrating an example of a predictive image correction procedure performed in OBMC processing.

[0016] [Figure 5D] Figure 5D is a flowchart illustrating an example of frame rate up conversion (FRUC) processing.

[0017] [Figure 6] Figure 6 illustrates an example of pattern matching (two-sided matching) between two blocks along a motion trajectory.

[0018] [Figure 7] Figure 7 illustrates an example of pattern matching (template matching) between a template in the current image and a block in the reference image.

[0019] [Figure 8] Figure 8 illustrates a model using uniform linear motion.

[0020] [Figure 9A] Figure 9A illustrates an example of deriving the motion vectors of each sub-block based on the motion vectors of neighboring blocks.

[0021] [Figure 9B] Figure 9B illustrates an example of a procedure for deriving motion vectors in merge mode.

[0022] [Figure 9C] Figure 9C is a conceptual diagram illustrating one example of dynamic motion vector refreshing (DMVR) processing.

[0023] [Figure 9D] Figure 9D illustrates an example of a predictive image generation method that uses a brightness correction procedure performed by local illumination compensation (LIC) processing.

[0024] [Figure 10] Figure 10 is a block diagram illustrating the functional configuration of a decoder according to one embodiment.

[0025] [Figure 11] Figure 11 is a flowchart illustrating the overall process flow of dividing an image block into a plurality of segments, including a first segment and a second segment having a non-rectangular shape (e.g., a triangle), and performing other processing, according to one embodiment.

[0026] [Figure 12] Figure 12 illustrates two exemplary methods for splitting an image block into a first segmentation region with a non-rectangular shape (e.g., a triangle) and a second segmentation region (also with a non-rectangular shape in the illustrated example).

[0027] [Figure 13] Figure 13 illustrates an example of a boundary smoothing procedure that involves weighting a first value of the boundary pixels predicted based on a first segmentation region and a second value of the boundary pixels predicted based on a second segmentation region.

[0028] [Figure 14] Figure 14 illustrates three additional samples of a boundary smoothing procedure that involves weighting a first value of the boundary pixels predicted based on a first segmentation region and a second value of the boundary pixels predicted based on a second segmentation region.

[0029] [Figure 15] Figure 15 is a table of sample parameters ("first index value") and the set of information encoded by these parameters respectively.

[0030] [Figure 16] Figure 16 is an example of a binary table of parameters (index values).

[0031] [Figure 17] Figure 17 is a flowchart illustrating the procedure of splitting an image block into a plurality of segmented regions, including a first segmented region and a second segmented region with non-rectangular shapes.

[0032] [Figure 18] Figure 18 illustrates an example of splitting an image block into a plurality of segmented regions, including a first segmented region and a second segmented region, which are non-rectangular in shape (triangular in the illustrated example).

[0033] [Figure 19] Figure 19 illustrates another example of splitting an image block into a plurality of segments, including a first segment and a second segment, which are non-rectangular in shape (in the illustrated example, a polygon with at least five sides and five corners).

[0034] [Figure 20] Figure 20 illustrates a flowchart of a boundary smoothing procedure involving weighting a first value of the boundary pixels predicted based on a first segmentation region and a second value of the boundary pixels predicted based on a second segmentation region.

[0035] [Figure 21A] Figure 21A illustrates an example of a boundary smoothing procedure, wherein the first value to be weighted for the boundary pixels is predicted based on a first segmentation region and the second value to be weighted is predicted based on a second segmentation region.

[0036] [Figure 21B] Figure 21B illustrates an example of a boundary smoothing procedure, wherein the first value to be weighted for the boundary pixels is predicted based on a first segmentation region and the second value to be weighted is predicted based on a second segmentation region.

[0037] [Figure 21C] Figure 21C illustrates an example of a boundary smoothing procedure, wherein the first value to be weighted for the boundary pixels is predicted based on a first segmentation region and the second value to be weighted is predicted based on a second segmentation region.

[0038] [Figure 21D] Figure 21D illustrates an example of a boundary smoothing procedure, wherein the first value to be weighted for the boundary pixels is predicted based on a first segmentation region and the second value to be weighted is predicted based on a second segmentation region.

[0039] [Figure 22] Figure 22 is a flowchart illustrating a method performed on the encoder side, which includes splitting an image block into a plurality of segmented regions, including a first segmented region and a second segmented region having a non-rectangular shape, based on segmentation parameters indicating splitting, and writing one or more of the segmentation parameters into a bit stream using entropy encoding.

[0040] [Figure 23] Figure 23 is a flowchart illustrating a method performed on the decoder side, the method including: parsing one or more parameters from a bitstream, the one or more parameters including a segmentation parameter indicating that an image block is split into a plurality of segmented regions including a first segmented region and a second segmented region having a non-rectangular shape; and splitting the image block into a plurality of segmented regions based on the segmentation parameters; and decoding the first segmented region and the second segmented region.

[0041] [Figure 24] Figure 24 is a table of sample segmentation parameters ("first index value"), which respectively indicate: splitting an image block into a plurality of segmentation regions including a first segmentation region and a second segmentation region having a non-rectangular shape; and a set of information that can be jointly encoded by the segmentation parameters.

[0042] [Figure 25] Figure 25 is a table of sample combinations of the first parameter and the second parameter. One of the first parameter and the second parameter is a segmentation parameter that indicates the division of the image block into a plurality of segmentation regions including a first segmentation region and a second segmentation region with non-rectangular shapes.

[0043] [Figure 26] Figure 26 illustrates the overall configuration of a content delivery system used to implement content distribution services.

[0044] [Figure 27] Figure 27 illustrates an example of a coding structure in adjustable coding.

[0045] [Figure 28] Figure 28 illustrates an example of a coding structure in adjustable coding.

[0046] [Figure 29] Figure 29 illustrates an example of a webpage display screen.

[0047] [Figure 30] Figure 30 illustrates an example of a webpage display screen.

[0048] [Figure 31] Figure 31 illustrates an example of a smartphone.

[0049] [Figure 32] Figure 32 is a block diagram illustrating a configuration example of a smart phone. Implementation

[0050] Detailed Description of Preferred Embodiments According to one embodiment, an image encoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs the following: dividing an image block into a plurality of segments including a first segment and a second segment having a non-rectangular shape; predicting a first motion vector for the first segment and a second motion vector for the second segment; and encoding the first segment using the first motion vector and encoding the second segment using the second motion vector.

[0051] According to another embodiment, the second segment has a non-rectangular shape. According to another embodiment, the non-rectangular shape is a triangle. According to another embodiment, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon with at least five sides and five angles.

[0052] According to another approach, the prediction includes selecting the first motion vector from a first set of motion vector candidates and selecting the second motion vector from a second set of motion vector candidates. For example, the first set of motion vector candidates may include motion vectors from segments adjacent to the first segmentation region, and the second set of motion vector candidates may include motion vectors from segments adjacent to the second segmentation region. The segments adjacent to the first segmentation region and the segments adjacent to the second segmentation region may be outside the image blocks that separate the first and second segmentation regions. These adjacent segments may be spatially adjacent segments or temporally adjacent segments, or both. The first set of motion vector candidates may be the same as or different from the second set of motion vector candidates.

[0053] According to another scenario, the prediction includes selecting a first motion vector candidate from a first set of motion vector candidates and deriving the first motion vector by adding a first motion vector difference to the first motion vector candidate, and selecting a second motion vector candidate from a second set of motion vector candidates and deriving the second motion vector by adding a second motion vector difference to the second motion vector candidate.

[0054] According to another embodiment, an image encoder is provided, comprising: a splitter that receives an original image and splits the original image into blocks during operation; an adder that receives the blocks from the splitter and predictions from a prediction controller during operation, and subtracts each prediction from its corresponding block to output a residual portion; a transformer that performs a transform on the residual portions output from the adder during operation to output transform coefficients; a quantizer that quantizes the transform coefficients during operation to generate quantized transform coefficients; an entropy encoder that encodes the quantized transform coefficients during operation to generate a one-bit stream; and the prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor generates a prediction for a current block based on a reference block in an encoded reference image during operation, and the intra-frame predictor generates a prediction for a current block based on an encoded reference block in a current image during operation. During operation, the predictive controller splits the blocks into a plurality of segments, including a first segment and a second segment, each having a non-rectangular shape; predicts a first motion vector for the first segment and a second motion vector for the second segment; and encodes the first segment using the first motion vector and the second segment using the second motion vector.

[0055] According to another approach, an image coding method is provided, which typically includes three steps: dividing an image block into a plurality of segments including a first segment and a second segment having a non-rectangular shape; predicting a first motion vector for the first segment and a second motion vector for the second segment; and encoding the first segment using the first motion vector and encoding the second segment using the second motion vector.

[0056] According to another embodiment, an image decoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs the following: splitting an image block into a plurality of segments including a first segment and a second segment having a non-rectangular shape; predicting a first motion vector for the first segment and a second motion vector for the second segment; and decoding the first segment using the first motion vector and decoding the second segment using the second motion vector.

[0057] According to another embodiment, the second segment has a non-rectangular shape. According to another embodiment, the non-rectangular shape is a triangle. According to another embodiment, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon with at least five sides and five angles.

[0058] According to another embodiment, an image decoder is provided, comprising: an entropy decoder that receives and decodes an encoded bitstream during operation to obtain quantized transform coefficients; an inverse quantizer and an inverse transformer that, during operation, inverse quantizes the quantized transform coefficients to obtain transform coefficients and inverse transforms the transform coefficients to obtain residual portions; an adder that, during operation, adds the residual portions output from the inverse quantizer and the inverse transformer to a prediction output from a prediction controller to reconstruct a block; and the prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor, during operation, generates a prediction of a current block based on a reference block in a decoded reference image, and the intra-frame predictor, during operation, generates a prediction of a current block based on a decoded reference block in a current image. During operation, the predictive controller splits an image block into a plurality of segments, including a first segment and a second segment, each having a non-rectangular shape; predicts a first motion vector for the first segment and a second motion vector for the second segment; and decodes the first segment using the first motion vector and the second segment using the second motion vector.

[0059] According to another approach, an image decoding method is provided, which typically includes three steps: splitting an image block into a plurality of segments including a first segment and a second segment having a non-rectangular shape; predicting a first motion vector for the first segment and a second motion vector for the second segment; and decoding the first segment using the first motion vector and decoding the second segment using the second motion vector.

[0060] According to one embodiment, an image encoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry, during operation, performs a boundary smoothing operation along a boundary between a first segmentation and a second segmentation having a non-rectangular shape, the first and second segmentations being split from an image block. The boundary smoothing operation includes: using information from the first segmentation to first predict a first value for a set of pixels of the first segmentation along the boundary; using information from the second segmentation to second predict a second value for the set of pixels of the first segmentation along the boundary; weighting the first and second values; and encoding the first segmentation using the weighted first and second values.

[0061] According to another example, the non-rectangular shape is a triangle. According to yet another example, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon with at least five sides and five angles. According to yet another example, the second division has a non-rectangular shape.

[0062] According to another embodiment, at least one of the first prediction and the second prediction is a frame-to-frame prediction procedure that predicts the first value and the second value based on a reference segmentation region in an encoded reference image. The frame-to-frame prediction procedure can predict the first value of pixels in the set of pixels of the first segmentation region and can predict the second value of pixels in the set of pixels only of the first segmentation region.

[0063] According to another example, at least one of the first prediction and the second prediction is an in-frame prediction procedure based on an coded reference segmentation region in a current image to predict the first value and the second value.

[0064] According to another scenario, one of the prediction methods used in the first prediction is different from one of the prediction methods used in the second prediction.

[0065] According to another example, the number of pixels in the set for which the first and second values ​​will be predicted in each column or row is an integer. For example, when the number of columns or rows in the set of pixels is four, weights 1 / 8, 1 / 4, 3 / 4, and 7 / 8 can be applied to the first values ​​of the four pixels in the set, and weights 7 / 8, 3 / 4, 1 / 4, and 1 / 8 can be applied to the second values ​​of the four pixels in the set. As another example, when the number of columns or rows in the set of pixels is two, weights 1 / 3 and 2 / 3 can be applied to the first values ​​of the two pixels in the set, and weights 2 / 3 and 1 / 3 can be applied to the second values ​​of the two pixels in the set.

[0066] Depending on the other case, the weights can be integer values ​​or decimal values.

[0067] According to another embodiment, an image encoder is provided, comprising: a splitter that receives an original image and splits the original image into blocks during operation; an adder that receives the blocks from the splitter and predictions from a prediction controller during operation, and subtracts each prediction from its corresponding block to output a residual portion; a transformer that performs a transform on the residual portions output from the adder during operation to output transform coefficients; a quantizer that quantizes the transform coefficients during operation to generate quantized transform coefficients; an entropy encoder that encodes the quantized transform coefficients during operation to generate a one-bit stream; and the prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor generates a prediction for a current block based on a reference block in an encoded reference image during operation, and the intra-frame predictor generates a prediction for a current block based on an encoded reference block in a current image during operation. The predictive controller performs a boundary smoothing operation during operation along a boundary between a first segmentation region and a second segmentation region having a non-rectangular shape, the first and second segmentation regions being split from an image block. The boundary smoothing operation includes: using information from the first segmentation region to first predict a first value for a set of pixels of the first segmentation region along the boundary; using information from the second segmentation region to second predict a second value for the set of pixels of the first segmentation region along the boundary; weighting the first and second values; and encoding the first segmentation region using the weighted first and second values.

[0068] According to another embodiment, an image encoding method is provided to perform a boundary smoothing operation along a boundary between a first segmentation region and a second segmentation region having a non-rectangular shape, the first and second segmentation regions being split from an image block. The method typically includes four steps: using information from the first segmentation region to first predict a first value for a set of pixels of the first segmentation region along the boundary; using information from the second segmentation region to second predict a second value for the set of pixels of the first segmentation region along the boundary; weighting the first and second values; and encoding the first segmentation region using the weighted first and second values.

[0069] According to another embodiment, an image decoder is provided, comprising circuitry and a memory coupled to the circuitry. During operation, the circuitry performs a boundary smoothing operation along a boundary between a first segment and a second segment having a non-rectangular shape, the first and second segments being split from an image block. The boundary smoothing operation includes: using information from the first segment to first predict a first value for a set of pixels of the first segment along the boundary; using information from the second segment to second predict a second value for the set of pixels of the first segment along the boundary; weighting the first and second values; and decoding the first segment using the weighted first and second values.

[0070] According to another pattern, the non-rectangular shape is a triangle. According to yet another pattern, the non-rectangular shape is selected from a group consisting of a triangle, a trapezoid, and a polygon with at least five sides and five angles. According to yet another pattern, the second segment has a non-rectangular shape.

[0071] According to another embodiment, at least one of the first prediction and the second prediction is a frame-to-frame prediction procedure that predicts the first value and the second value based on a reference segmentation region in an encoded reference image. The frame-to-frame prediction procedure can predict the first value of pixels in the set of pixels of the first segmentation region and can predict the second value of pixels in the set of pixels only of the first segmentation region.

[0072] According to another example, at least one of the first prediction and the second prediction is an in-frame prediction procedure based on an coded reference segmentation region in a current image to predict the first value and the second value.

[0073] According to another embodiment, an image decoder is provided, comprising: an entropy decoder that receives and decodes an encoded bitstream during operation to obtain quantized transform coefficients; an inverse quantizer and an inverse transformer that, during operation, inverse quantizes the quantized transform coefficients to obtain transform coefficients and inverse transforms the transform coefficients to obtain residual portions; an adder that, during operation, adds the residual portions output from the inverse quantizer and the inverse transformer to a prediction output from a prediction controller to reconstruct a block; and the prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor, during operation, generates a prediction for a current block based on a reference block in a decoded reference image, and the intra-frame predictor, during operation, generates a prediction for a current block based on a decoded reference block in a current image. The prediction controller, during operation, performs a boundary smoothing operation along a boundary between a first segmentation region and a second segmentation region having a non-rectangular shape, the first and second segmentation regions being split from an image block. The boundary smoothing operation includes: using information from the first segmentation region to first predict a first value of a set of pixels in the first segmentation region along the boundary; using information from the second segmentation region to second predict a second value of the set of pixels in the first segmentation region along the boundary; weighting the first value and the second value; and decoding the first segmentation region using the weighted first value and the weighted second value.

[0074] According to another embodiment, an image decoding method is provided to perform a boundary smoothing operation along a boundary between a first segmentation region and a second segmentation region having a non-rectangular shape, the first and second segmentation regions being split from an image block. The method typically includes four steps: using information from the first segmentation region to first predict a first value for a set of pixels of the first segmentation region along the boundary; using information from the second segmentation region to second predict a second value for the set of pixels of the first segmentation region along the boundary; weighting the first and second values; and decoding the first segmentation region using the weighted first and second values.

[0075] According to one specification, an image encoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a segmentation syntax operation comprising: segmenting an image block into a plurality of segmented regions, including a first segmented region and a second segmented region having a non-rectangular shape, based on a segmentation parameter indicating segmentation; encoding the first segmented region and the second segmented region; and writing one or more of the segmentation parameters into a one-bit stream.

[0076] According to another pattern, the segmentation parameter indicates that the first segmentation region has a triangular shape.

[0077] According to another pattern, the segmentation parameter indicates that the second segmentation region has a non-rectangular shape.

[0078] According to another pattern, the segmentation parameter indicates that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon with at least five sides and five angles.

[0079] According to another approach, the segmentation parameters are jointly encoded to a splitting direction applied to splitting the image block into the plurality of segmented regions. For example, the splitting direction may include: from the upper left corner of one of the image blocks to the lower right corner of one of the image blocks, and from the upper right corner of one of the image blocks to the lower left corner of one of the image blocks. The segmentation parameters may at least jointly encode a first motion vector of one of the first segmented regions.

[0080] According to another configuration, the one or more parameters other than the segmentation parameter are encoded in a splitting direction for splitting the image block into the plurality of segmented regions. The parameter encoding the splitting direction may at least jointly encode a first motion vector of one of the first segmented regions.

[0081] According to another pattern, the segmentation parameters can at least jointly encode a first motion vector of one of the first segmented regions. The segmentation parameters can also jointly encode a second motion vector of one of the second segmented regions.

[0082] According to another pattern, the one or more parameters other than the segmentation parameter may encode at least one of the first motion vectors of the first segmentation region.

[0083] According to another approach, one or more parameters are binarized according to a binary scheme, which is selected based on the value of at least one of the parameters.

[0084] According to another embodiment, an image encoder is provided, comprising: a splitter that receives an original image and splits the original image into blocks during operation; an adder that receives the blocks from the splitter and predictions from a prediction controller during operation, and subtracts each prediction from its corresponding block to output a residual portion; a transformer that performs a transform on the residual portions output from the adder during operation to output transform coefficients; a quantizer that quantizes the transform coefficients during operation to generate quantized transform coefficients; an entropy encoder that encodes the quantized transform coefficients during operation to generate a one-bit stream; and the prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor generates a prediction for a current block based on a reference block in an encoded reference image during operation, and the intra-frame predictor generates a prediction for a current block based on an encoded reference block in a current image during operation. During operation, the predictive controller divides an image block into a plurality of segments, including a first segment and a second segment, each having a non-rectangular shape, based on a segmentation parameter indicating the division, and encodes the first and second segments. During operation, the entropy encoder writes one or more of the segmentation parameters into a one-bit stream.

[0085] According to another approach, an image encoding method including a segmentation syntax operation is provided. This method typically includes three steps: dividing an image block into a plurality of segmentation regions, including a first segmentation region and a second segmentation region having a non-rectangular shape, based on a segmentation parameter indicating the segmentation; encoding the first segmentation region and the second segmentation region; and writing one or more of the segmentation parameters into a one-bit stream.

[0086] According to another embodiment, an image decoder is provided, comprising circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs a segmentation syntax operation comprising: parsing one or more parameters from a one-bit stream, wherein the one or more parameters include a segmentation parameter indicating that an image block is split into a plurality of segmented regions including a first segmented region and a second segmented region having a non-rectangular shape; splitting the image block into the plurality of segmented regions based on the segmentation parameter; and decoding the first segmented region and the second segmented region.

[0087] According to another pattern, the segmentation parameter indicates that the first segmentation region has a triangular shape.

[0088] According to another pattern, the segmentation parameter indicates that the second segmentation region has a non-rectangular shape.

[0089] According to another pattern, the segmentation parameter indicates that the non-rectangular shape is one of a triangle, a trapezoid, and a polygon with at least five sides and five angles.

[0090] According to another approach, the segmentation parameters are jointly encoded to a splitting direction applied to split the image block into the plurality of segmented regions. For example, the splitting direction includes: from the upper left corner of one of the image blocks to the lower right corner of one of the image blocks, and from the upper right corner of one of the image blocks to the lower left corner of one of the image blocks. The segmentation parameters may at least jointly encode a first motion vector of one of the first segmented regions.

[0091] According to another configuration, the one or more parameters other than the segmentation parameter are encoded in a splitting direction for splitting the image block into the plurality of segmented regions. The parameter encoding the splitting direction may at least jointly encode a first motion vector of one of the first segmented regions.

[0092] According to another pattern, the segmentation parameters can at least jointly encode a first motion vector of one of the first segmented regions. The segmentation parameters can also jointly encode a second motion vector of one of the second segmented regions.

[0093] According to another pattern, the one or more parameters other than the segmentation parameter may encode at least one of the first motion vectors of the first segmentation region.

[0094] According to another approach, one or more parameters are binarized according to a binary scheme, which is selected based on the value of at least one of the parameters.

[0095] According to another embodiment, an image decoder is provided, comprising: an entropy decoder that receives and decodes an encoded bitstream during operation to obtain quantized transform coefficients; an inverse quantizer and an inverse transformer that, during operation, inverse quantizes the quantized transform coefficients to obtain transform coefficients and inverse transforms the transform coefficients to obtain residual portions; an adder that, during operation, adds the residual portions output from the inverse quantizer and the inverse transformer to a prediction output from a prediction controller to reconstruct a block; and the prediction controller coupled to an inter-frame predictor, an intra-frame predictor, and a memory, wherein the inter-frame predictor, during operation, generates a prediction of a current block based on a reference block in a decoded reference image, and the intra-frame predictor, during operation, generates a prediction of a current block based on a decoded reference block in a current image. The entropy decoder, in operation, analyzes one or more parameters from a one-bit stream, wherein the one or more parameters include a segmentation parameter indicating that an image block is split into a plurality of segmentation regions including a first segmentation region and a second segmentation region having a non-rectangular shape; splits the image block into the plurality of segmentation regions based on the segmentation parameter; and decodes the first segmentation region and the second segmentation region.

[0096] According to another approach, an image decoding method including a segmentation syntax operation is provided. This method typically includes three steps: parsing one or more parameters from a one-bit stream, wherein the one or more parameters include a segmentation parameter indicating that an image block should be split into a plurality of segmented regions including a first segmented region and a second segmented region having a non-rectangular shape; splitting the image block into the plurality of segmented regions based on the segmentation parameter; and decoding the first segmented region and the second segmented region.

[0097] In the diagram, the same component symbol identifies similar components. The dimensions and relative positions of the components in the diagram are not necessarily drawn to scale.

[0098] In the following description, embodiments will be illustrated with reference to the figures. It should be noted that each of the embodiments described below represents a general or specific example. The numerical values, shapes, materials, components, configurations and connections of components, steps, relationships and sequences of steps indicated in the following embodiments are merely examples and are not intended to limit the scope of the patent application. Therefore, components disclosed in the following embodiments but not listed in any of the independent technical solutions defining the broadest inventive concept may be understood as optional components.

[0099] Embodiments of encoders and decoders will be described below. These embodiments are examples of encoders and decoders, and the programs and / or configurations presented in the description of the embodiments of the present invention are applicable to these embodiments. These programs and / or configurations can also be implemented in encoders and decoders different from those according to these embodiments. For example, with respect to the programs and / or configurations applied to these embodiments, any of the following can be implemented: (1) Any of the components of the encoder or decoder according to the embodiments presented in the description of the present invention may be replaced by or combined with another component presented at any location in the description of the present invention.

[0100] (2) In the encoder or decoder according to the embodiment, any changes can be made to the functions or programs performed by one or more components of the encoder or decoder, such as adding, substituting, or removing such functions or programs. For example, any function or program can be replaced by or combined with another function or program presented at any point in the description of the present invention.

[0101] (3) Any changes may be made to the method implemented by the encoder or decoder according to the embodiment, such as adding, substituting, or removing one or more of the programs included in the method. For example, any program in the method may be replaced by or combined with another program presented at any point in the description of the present invention.

[0102] (4) One or more components included in the encoder or decoder according to the embodiment may be combined with one of the components presented at any location in the description of the present invention, may be combined with one or more functions presented at any location in the description of the present invention, and may be combined with one of the components that implement one or more programs implemented by one of the components presented in the description of the present invention.

[0103] (5) A component including one or more functions of an encoder or decoder according to an embodiment, or a component implementing one or more programs of an encoder or decoder according to an embodiment, may be combined with or replaced by: a component presented at any location in the description of the present invention; a component including one or more functions presented at any location in the description of the present invention; or a component implementing one or more programs presented at any location in the description of the present invention.

[0104] (6) In a method implemented by an encoder or decoder according to an embodiment, any of the programs included in the method may be replaced or combined with: a program presented at any location in the description of the present invention; or any corresponding or equivalent program.

[0105] (7) One or more programs included in the method implemented by the encoder or decoder according to the embodiment may be combined with one of the programs presented at any point in the description of the present invention.

[0106] (8) The implementation of the programs and / or configurations presented in the description of the present invention is not limited to the encoder or decoder according to the embodiments. For example, such programs and / or configurations may be implemented in devices for purposes different from the animation encoder or animation decoder disclosed in the embodiments. (encoder)

[0107] First, an encoder according to one embodiment will be described. Figure 1 is a block diagram illustrating the functional configuration of an encoder 100 according to an embodiment. Encoder 100 is an animation encoder that encodes animation block by block.

[0108] As illustrated in Figure 1, encoder 100 is a device for encoding images block by block, and includes splitter 102, subtractor 104, transformer 106, quantizer 108, entropy encoder 110, dequantizer 112, inverse transformer 114, adder 116, block memory 118, loop filter 120, frame memory 122, in-frame predictor 124, inter-frame predictor 126, and prediction controller 128.

[0109] The encoder 100 is implemented, for example, as a general-purpose processor and memory. In this case, when the software program stored in memory is executed by the processor, the processor functions as a splitter 102, a subtractor 104, a converter 106, a quantizer 108, an entropy encoder 110, an inverse quantizer 112, an inverse converter 114, an adder 116, a loop filter 120, an in-frame predictor 124, an inter-frame predictor 126, and a prediction controller 128. Alternatively, the encoder 100 may be implemented as one or more dedicated electronic circuits corresponding to the following: splitter 102, subtractor 104, converter 106, quantizer 108, entropy encoder 110, inverse quantizer 112, inverse converter 114, adder 116, loop filter 120, in-frame predictor 124, inter-frame predictor 126, and prediction controller 128.

[0110] The components included in encoder 100 will be described below. (Splitter)

[0111] Splitter 102 splits the images included in the input animation into blocks and outputs each block to subtractor 104. For example, splitter 102 first splits the image into blocks of a fixed size (e.g., 128×128). Fixed-size blocks may also be referred to as coding tree units (CTUs). Splitter 102 then splits each fixed-size block into variable-size blocks (e.g., 64×64 or smaller) based on, for example, recursive quadtree and / or binary tree block splitting. Variable-size blocks may also be referred to as write code units (CUs), prediction units (PUs), or transform units (TUs). In various implementations, it may not be necessary to distinguish between CUs, PUs, and TUs; all or some blocks in the image may be processed as CUs, PUs, or TUs.

[0112] Figure 2 illustrates an example of block splitting according to one embodiment. In Figure 2, solid lines represent block boundaries of block splitting according to a quaternion tree, and dashed lines represent block boundaries of block splitting according to a binary tree.

[0113] Here, block 10 is a square block of 128×128 pixels (128×128 block). This 128×128 block 10 is first split into four square blocks of 64×64 (quadtree block split).

[0114] The 64×64 block in the upper left corner is further vertically split into two rectangular 32×64 blocks, and the 32×64 block on the left is further vertically split into two rectangular 16×64 blocks (binary tree block splitting). As a result, the 64×64 block in the upper left corner is split into two 16×64 blocks 11 and 12 and one 32×64 block 13.

[0115] The 64×64 block in the upper right corner is horizontally split into two rectangular 64×32 blocks, 14 and 15 (binary tree block split).

[0116] The lower left 64×64 block is first split into four 32×32 square blocks (quaternion tree block split). The upper left and lower right blocks of these four 32×32 blocks are further split. The upper left 32×32 block is vertically split into two 16×32 rectangular blocks, and the right 16×32 block is further horizontally split into two 16×16 blocks (binary tree block split). The lower right 32×32 block is horizontally split into two 32×16 blocks (binary tree block split). As a result, the lower left 64×64 block is split into 16×32 block 16, two 16×16 blocks 17 and 18, two 32×32 blocks 19 and 20, and two 32×16 blocks 21 and 22.

[0117] Block 23, a 64x64 block in the lower right corner, has not split.

[0118] As described above, in Figure 2, block 10 is split into 13 variable-size blocks 11 to 23 based on recursive quadtree and binary tree block splitting. This type of splitting is also known as quadtree plus binary tree (QTBT) splitting.

[0119] Although in Figure 2, a block is split into four or two blocks (quadtree or binary tree block split), the split is not limited to these instances. For example, a block can be split into three blocks (ternary block split). Splits including this ternary block split are also known as multi-type tree (MBT) splits. (Subtractor)

[0120] Subtractor 104 subtracts a prediction signal (the prediction sample input to the self-predictive controller 128, as described below) from an original signal (original sample) for each block split by splitter 102 and input from splitter 102. In other words, subtractor 104 calculates the prediction error (also referred to as the "residual part") of the block to be encoded (hereinafter referred to as the "current block"). Subtractor 104 then outputs the calculated prediction error (residual part) to converter 106.

[0121] The raw signal is the signal input to encoder 100, and is a signal representing the image of each image included in the animation (e.g., a luminance signal and two chrominance signals). In the following text, the signal representing the image is also referred to as a sample. (Transformer)

[0122] Transformer 106 transforms the spatial domain prediction error into frequency domain transform coefficients and outputs these transform coefficients to quantizer 108. More specifically, transformer 106 applies, for example, a predefined discrete cosine transform (DCT) or discrete sine transform (DST) to the spatial domain prediction error.

[0123] It should be noted that the transformer 106 can adaptively select a transformation type from a plurality of transformation types, and transform the prediction error into transformation coefficients by using a transformation basis function corresponding to the selected transformation type. This type of transformation is also known as explicit multiple core transform (EMT) or adaptive multiple transform (AMT).

[0124] Transform types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 is a graph showing the transform basis functions for each transform type. In Figure 3, N indicates the number of input pixels. For example, selecting a transform type from among several transform types may depend on the prediction type (intra-box prediction and inter-box prediction) and the intra-box prediction mode.

[0125] Information indicating whether EMT or AMT is applied (referred to as, for example, EMT flags or AMT flags) and information indicating the selected transform type are typically communicated at the CU level. It should be noted that this communication does not need to be performed at the CU level and can be performed at another level (e.g., at the bit sequence level, image level, slice level, tile level, or CTU level).

[0126] Furthermore, transformer 106 can apply a quadratic transformation to the transform coefficients (the transformed result). This quadratic transformation is also referred to as an adaptive secondary transform (AST) or a non-separable secondary transform (NSST). For example, transformer 106 applies the quadratic transformation to each sub-block (e.g., each 4×4 sub-block) within a block containing transform coefficients corresponding to the in-box prediction error. Information indicating whether NSST is applied and information related to the transform matrix used in NSST are typically communicated at the CU level. It should be noted that this communication of information does not need to be performed at the CU level and can be performed at another level (e.g., at the sequence level, image level, slice level, block level, or CTU level).

[0127] Individual or inseparable transformations can be applied in transformer 106. An individual transformation is a method of performing the transformation multiple times by performing the transformation individually on each direction according to the number of dimensional inputs. An inseparable transformation is a method of performing a common transformation, in which two or more dimensions in the multidimensional input are treated together as a single dimension.

[0128] In one instance of an inseparable transformation, when the input system is a 4×4 block, the 4×4 block is treated as a single array comprising 16 components, and the transformation applies a 16×16 transformation matrix to this array.

[0129] In another instance of an inseparable transformation, after the input 4×4 block is treated as a single array comprising 16 components, a transformation of multiple Givens rotations (e.g., a hypercube Givens transformation) can be applied to the array. (Quantizer)

[0130] Quantizer 108 quantizes the transform coefficients output from converter 106. More specifically, quantizer 108 scans the transform coefficients of the current block in a predetermined scan order and quantizes the scanned transform coefficients based on the quantization parameters (QP) corresponding to those transform coefficients. Quantizer 108 then outputs the quantized transform coefficients of the current block (hereinafter referred to as quantization coefficients) to entropy encoder 110 and dequantizer 112.

[0131] The predetermined scan order is the order in which the transform coefficients are quantized / dequantized. For example, the predetermined scan order is defined as ascending frequency (from low frequency to high frequency) or descending frequency (from high frequency to low frequency).

[0132] The quantization parameter (QP) defines the quantization step size (quantization width). For example, if the value of the quantization parameter increases, the quantization step size also increases. In other words, if the value of the quantization parameter increases, the quantization error increases. (Entropy encoder)

[0133] The entropy encoder 110 generates an encoded signal (encoded bitstream) based on the quantization coefficients input to the self-quantizer 108. More specifically, for example, the entropy encoder 110 binarizes the quantization coefficients and performs arithmetic encoding on the binary signal to output a compressed bitstream or sequence. (Inverse quantizer)

[0134] Dequantizer 112 dequantizes the quantization coefficients input to dequantizer 108. More specifically, dequantizer 112 dequantizes the quantization coefficients of the current block according to a predetermined scan order. Dequantizer 112 then outputs the dequantized transform coefficients of the current block to inverse transformer 114. (Inverse Transformer)

[0135] Inverse transformer 114 recovers the prediction error (residual portion) by performing an inverse transform on the transform coefficients input to inverse quantizer 112. More specifically, inverse transformer 114 recovers the prediction error of the current block by applying an inverse transform to the transform coefficients corresponding to the transform applied by transformer 106. Inverse transformer 114 then outputs the recovered prediction error to adder 116.

[0136] It should be noted that, since information is often lost during quantization, the recovered prediction error does not match the prediction error calculated by subtractor 104. In other words, the recovered prediction error typically includes the quantization error. (Adder)

[0137] Adder 116 reconstructs the current block by summing the prediction error input to autoreflective transformer 114 and the prediction sample input to autopredictive controller 128. Adder 116 then outputs the reconstructed block to block memory 118 and loop filter 120. The reconstructed block is also referred to as the region decoding block. (Block memory)

[0138] Block memory 118 is a memory used to store blocks in the image to be encoded (referred to as the "current image") for reference, for example, during in-frame prediction. More specifically, block memory 118 stores the reconstructed blocks output by adder 116. (Loop filter)

[0139] The loop filter 120 applies a loop filter to the block reconstructed by the adder 116 and outputs the filtered reconstructed block to the frame memory 122. The loop filter is a filter used in the encoding loop (in-loop filter) and includes, for example, a deblocking filter (DF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF).

[0140] In ALF, a least-squares error filter is applied to remove compression artifacts. For example, one of several filters is selected and applied based on the direction and activity of the regional gradient for each 2×2 sub-block in the current block.

[0141] More specifically, first, each sub-block (e.g., each 2×2 sub-block) is classified into one of several categories (e.g., 15 or 25 categories). The classification of sub-blocks is based on gradient directionality and activity. For example, the classification index C is derived based on gradient directionality D (e.g., 0 to 2 or 0 to 4) and gradient activity A (e.g., 0 to 4) (e.g., C = 5D + A). Then, based on the classification index C, each sub-block is classified into one of the several categories.

[0142] For example, gradient directionality D is calculated by comparing the gradients in a complex number of directions (e.g., horizontal, vertical, and two diagonal directions). Similarly, gradient activity A is calculated by summing the gradients in a complex number of directions and quantizing the sum.

[0143] The filters to be used in each sub-block are determined from a plurality of filters based on the results of this classification.

[0144] The filter shape to be used in ALF is, for example, a circular symmetrical filter shape. Figures 4A, 4B, and 4C illustrate examples of filter shapes used in ALF. Figure 4A illustrates a 5×5 diamond-shaped filter, Figure 4B illustrates a 7×7 diamond-shaped filter, and Figure 4C illustrates a 9×9 diamond-shaped filter. Information indicating the filter shape is typically communicated at the image layer level. It should be noted that the communication of information indicating the filter shape does not need to be performed at the image layer level, but can be performed at another layer (e.g., at the sequence layer, slice layer, block layer, CTU layer, or CU layer).

[0145] Enabling or disabling ALF can be determined at the image layer or the CU layer. For example, the decision to apply ALF for luminance can be made at the CU layer, while the decision to apply ALF for chroma can be made at the image layer. Information indicating whether ALF is enabled or disabled is typically communicated at the image layer or CU layer. It should be noted that the communication of information indicating whether ALF is enabled or disabled does not need to be performed at the image layer or CU layer, and can be performed at another layer (e.g., at the sequence layer, slice layer, block layer, or CTU layer).

[0146] The set of coefficients used for a plurality of selectable filters (e.g., 15 or 25 filters) is typically communicated at the image layer level. It should be noted that the communication of the set of coefficients does not need to be performed at the image layer level, but can be performed at another level (e.g., at the sequence layer, slice layer, block layer, CTU layer, CU layer, or sub-block layer). (Graphic frame memory)

[0147] Frame memory 122 is a storage device used to store reference images, such as those used in inter-frame prediction, and is also referred to as a frame buffer. More specifically, frame memory 122 stores reconstructed blocks filtered by loop filter 120. (In-frame predictor)

[0148] The in-frame predictor 124 generates a prediction signal (in-frame prediction signal) by making an in-frame prediction of the current block (also referred to as in-frame image prediction) by referencing one or more blocks in the current image stored in the block memory 118. More specifically, the in-frame predictor 124 generates an in-frame prediction signal by making an in-frame prediction by referring to samples (e.g., brightness values ​​and / or chroma values) of one or more blocks adjacent to the current block, and then outputs the in-frame prediction signal to the prediction controller 128.

[0149] For example, the in-box predictor 124 performs in-box prediction by using one of a plurality of predefined in-box prediction modes. These in-box prediction modes typically include one or more non-directional prediction modes and a plurality of directional prediction modes.

[0150] The one or more non-directional prediction modes include, for example, the planar prediction mode and the DC prediction mode defined in the H.265 / HEVC standard.

[0151] Multiple direction prediction modes include, for example, the 33 direction prediction modes defined in the H.265 / HEVC standard. It should be noted that, in addition to the 33 direction prediction modes, multiple direction prediction modes may further include 32 direction prediction modes (a total of 65 direction prediction modes).

[0152] Figure 5A illustrates a total of 67 in-box prediction modes (two non-directional prediction modes and 65 directional prediction modes) used for in-box prediction. Solid arrows indicate 33 directions defined in the H.265 / HEVC standard, and dashed arrows indicate an additional 32 directions. (The two "non-directional" prediction modes are not illustrated in Figure 5A.)

[0153] In various implementations, the lightness block can be referenced in the in-frame prediction of the chroma block. That is, the chroma component of the current block can be predicted based on the lightness component of the current block. This in-frame prediction is also known as cross-component linear model (CCLM) prediction. The in-frame prediction mode of the chroma block referencing the lightness block (referred to as, for example, the CCLM mode) can be added as one of the in-frame prediction modes of the chroma block.

[0154] The in-box predictor 124 can correct the predicted pixel values ​​based on horizontal / vertical reference pixel gradients. This correction-related in-box prediction is also known as position-dependent intra-prediction combination (PDPC). Information indicating whether PDPC is applied (such as a PDPC flag) is typically communicated at the CU level. It should be noted that this communication does not need to be performed at the CU level and can be performed at another level (e.g., at the sequence level, image level, slice level, block level, or CTU level). (Inter-frame predictor)

[0155] Inter-frame predictor 126 generates a prediction signal (inter-frame prediction signal) by inter-frame predicting the current block using one or more blocks in a reference image, which is different from the current image and stored in frame memory 122 (also referred to as inter-frame frame prediction). Inter-frame prediction is performed for each current block or for each current sub-block within the current block (e.g., each 4×4 block). For example, inter-frame predictor 126 performs motion estimation on the reference image used for the current block or current sub-block to find the best-matching reference block or sub-block in the reference image, and obtains motion information (e.g., motion vectors) to compensate for (or predict) the movement or change from the reference block or sub-block to the current block or sub-block. Inter-frame predictor 126 then performs motion compensation (or motion prediction) based on this motion information and generates the inter-frame prediction signal for the current block or sub-block based on the motion information. The inter-frame predictor 126 then outputs the generated inter-frame prediction signal to the prediction controller 128.

[0156] Motion information used in motion compensation can be communicated in various forms as inter-frame prediction signals. For example, motion vectors can be communicated. As another example, the difference between the motion vector and the motion vector predictor can be communicated.

[0157] It should be noted that, in addition to the motion information of the current block obtained from self-motion estimation, motion information of neighboring blocks can also be used to generate inter-frame prediction signals. More specifically, the inter-frame prediction signal for each sub-block in the current block can be generated by calculating a weighted sum of the prediction signal based on the motion information obtained from self-motion estimation (in the reference image) and the prediction signal based on the motion information of neighboring blocks (in the current image). This inter-frame prediction (motion compensation) is also known as Overlapping Block Motion Compensation (OBMC).

[0158] In OBMC mode, information indicating the size of sub-blocks used in OBMC (referred to as, for example, OBMC block size) can be communicated at the sequence level. Additionally, information indicating whether OBMC mode is applied (referred to as, for example, OBMC flag) can be communicated at the CU level. It should be noted that this communication does not need to be performed at the sequence and CU levels, but can be performed at another level (e.g., at the image level, slice level, block level, CTU level, or sub-block level).

[0159] The OBMC model will be described in more detail below. Figure 5B is a flowchart and Figure 5C is a conceptual diagram illustrating the predictive image correction procedure performed by OBMC processing.

[0160] Referring to Figure 5C, firstly, the predicted image (Pred) is obtained using the motion vector (MV) assigned to the target (current) block via typical motion compensation. In Figure 5C, the arrow "MV" points to the reference image to indicate which current block in the current image is being referenced in order to obtain the predicted image.

[0161] Next, the predicted image (Pred_L) is obtained by applying (reusing) the motion vector (MV_L) derived for the encoded neighboring left block to the target (current) block indicated by the arrow "MV_L" originating from the current block and pointing to the reference image. Then, the two predicted images, Pred and Pred_L, are superimposed to perform the first stage (pass) of predicted image correction. In one sample, this first stage has the effect of blurring the boundaries between neighboring blocks.

[0162] Similarly, the predicted image (Pred_U) is obtained by applying (reusing) the motion vector (MV_U) derived for the encoded neighboring upper blocks to the target (current) block indicated by the arrow "MV_U" originating from the current block and pointing to the reference image. Next, a second stage of prediction image correction is performed by superimposing the predicted image Pred_U with the predicted images (i.e., Pred and Pred_L) resulting from the first stage. In one sample, this second stage has the effect of blurring the boundaries between neighboring blocks. The result of the second stage is the final predicted image for the current block, with blurred (smooth) boundaries with its neighboring blocks.

[0163] It should be noted that the above example uses a two-stage correction method with the adjacent left block and the upper block, but the method can also use a three-stage or higher-stage correction method with the adjacent right block and / or the lower block.

[0164] It should be noted that the area subject to overlay can be the entire pixel area of ​​the block, or alternatively, a portion of the block boundary area.

[0165] It should be noted that here, the OBMC predictive image correction procedure is described as deriving additional predictive images Pred_L and Pred_U from a single reference image to which a single predictive image Pred will be superimposed. However, when correcting predictive images based on multiple reference images, the same procedure can be applied to each of the multiple reference images. In this case, after obtaining multiple corrected predictive images by performing OBMC image correction based on multiple reference images respectively, the obtained multiple corrected predictive images are further superimposed to obtain the final predictive image.

[0166] It should be noted that in OBMC, the unit of the target block can be the prediction block, and alternatively, it can be the sub-blocks obtained by further dividing the prediction block.

[0167] One example of a method for determining whether to perform OBMC processing is the use of `obmc_flag`, a signal indicating whether OBMC processing is to be performed. As a specific example, the encoder can determine whether a target block belongs to a region containing complex motion. The encoder sets `obmc_flag` to "1" and performs OBMC processing during encoding when the block belongs to a region containing complex motion, and sets `obmc_flag` to "0" and encodes the block without performing OBMC processing when the block does not belong to a region containing complex motion. The decoder switches between performing OBMC processing or not by decoding the `obmc_flag` written to the stream (i.e., the compressed sequence) and performing decoding based on the flag value.

[0168] It should be noted that motion information can be derived on the decoder side without being transmitted from the encoder side. For example, the merging mode defined in the H.265 / HEVC standard can be used. Furthermore, for example, motion information can be derived by performing motion estimation on the decoder side. In this case, the decoder can perform motion estimation without using the pixel values ​​of the current block.

[0169] Here, the mode used to perform motion estimation on the decoder side will be described. The mode used to perform motion estimation on the decoder side is also known as the pattern matching motion vector derivation (PMMVD) mode or the frame rate upconversion (FRUC) mode.

[0170] Figure 5D illustrates an example of FRUC processing. First, candidate lists (which may be merged lists) are generated by referring to the motion vectors of coded blocks that are spatially or temporally adjacent to the current block, each including a predicted motion vector (MV). Next, the best candidate MV is selected from the plurality of candidate MVs registered in the candidate lists. For example, evaluation values ​​are calculated for the candidate MVs included in the candidate lists, and a candidate MV is selected based on the calculated evaluation values.

[0171] Next, the motion vector for the current block is derived from the motion vector of the selected candidate. More specifically, for example, as is currently the case, the motion vector for the current block is calculated as the motion vector of the selected candidate (best candidate MV). Alternatively, the motion vector for the current block can be derived by performing pattern matching corresponding to the motion vector of the selected candidate near a location in the reference image. In other words, when using pattern matching and evaluation values ​​in the reference image to search for the vicinity of the best candidate MV, and finding an MV with a better evaluation value, the best candidate MV can be updated to the MV with the better evaluation value, and the MV with the better evaluation value can be used as the final MV for the current block. Configurations where the processing for updating the MV with the better evaluation value is not implemented are also acceptable.

[0172] The same procedure can be executed when processing is performed on a sub-block basis.

[0173] The evaluation value can be calculated in various ways. For example, a reconstructed image of a region in a reference image corresponding to a motion vector can be compared with a reconstructed image of a predetermined region (which may be in another reference image or in a neighboring block in the current image, for example, as described below), and the difference in pixel values ​​between the two reconstructed images can be calculated and used as the evaluation value of the motion vector. It should be noted that the evaluation value can be calculated using some other information besides this difference.

[0174] Next, pattern matching will be described in detail. First, one candidate MV included in the candidate list (e.g., a merged list) is selected as the starting point for the search based on pattern matching. The pattern matching used is either a first pattern matching or a second pattern matching. The first pattern matching and the second pattern matching are also referred to as two-sided matching and template matching, respectively.

[0175] In the first pattern matching, pattern matching is performed between two blocks in two different reference images, both of which follow the motion trajectory of the current block. Therefore, in the first pattern matching, for one region in the reference image, a region in the other reference image that matches the motion trajectory of the current block is used as a predetermined region for the above calculation of the evaluation value of the candidate.

[0176] Figure 6 illustrates an example of a first pattern matching (two-sided matching) between two blocks in two reference images along a motion trajectory. As illustrated in Figure 6, in the first pattern matching, two motion vectors (MV0, MV1) are derived by finding the best match between two blocks in two different reference images (Ref0, Ref1) along the motion trajectory of the current block (Cur block). More specifically, a difference can be obtained between (i) the reconstructed image at the position specified by the candidate MV in the first coded reference image (Ref0) and (ii) the reconstructed image at the position specified by the candidate MV in the second coded reference image (Ref1) (which is scaled symmetrically according to the display time interval). This difference can then be used to derive an evaluation value for the current block. The candidate MV with the best evaluation value among the plurality of candidate MVs can be selected as the final MV.

[0177] Under the assumption of continuous motion trajectories, the motion vectors (MV0, MV1) pointing to the two reference blocks are proportional to the temporal distances (TD0, TD1) between the current image (Cur Pic) and the two reference images (Ref0, Ref1). For example, when the current image is in time between the two reference images and the temporal distances from the current image to the two reference images are the same, the first pattern matching derives two mirrored bidirectional motion vectors.

[0178] In the second pattern matching (template matching), pattern matching is performed between a template in the current image (a block in the current image that is adjacent to the current block; for example, the top and / or left adjacent block) and a block in the reference image. Therefore, in the second pattern matching, the block in the current image that is adjacent to the current block is used as a predetermined region for the above calculation of the candidate evaluation value.

[0179] Figure 7 illustrates an example of pattern matching (template matching) between a template in the current image and a block in the reference image. As illustrated in Figure 7, in the second pattern matching, the motion vector of the current block is derived by searching in the reference image (Ref0) for a block that best matches a neighboring block of the current block (Cur block) in the current image (Cur Pic). More specifically, a difference can be obtained between (i) the reconstructed image of one or both of the coded adjacent upper and left regions of the current block and (ii) the reconstructed image of the same region relative to the block position defined by the candidate MV in the coded reference image (Ref0). This difference can then be used to derive an evaluation value for the current block. The candidate MV with the best evaluation value among the plurality of candidate MVs can be selected as the best candidate MV.

[0180] Information indicating whether FRUC mode is applied (referred to as, for example, the FRUC flag) can be transmitted at the CU level. Furthermore, when FRUC mode is applied (e.g., when the FRUC flag is set to true), information indicating the applicable matching method for the pattern (e.g., first pattern match or second pattern match) can be transmitted at the CU level. It should be noted that the transmission of this information does not need to be performed at the CU level, but can be performed at another level (e.g., at the sequence level, image level, slice level, block level, CTU level, or sub-block level).

[0181] Next, the method for deriving motion vectors is described. First, a description of a mode for deriving motion vectors based on a model employing uniform linear motion is given. This mode is also known as the bidirectional optical flow (BIO) mode.

[0182] Figure 8 illustrates a model employing uniform linear motion. In Figure 8, (vx, vy) represents the velocity vector, and τ0 and τ1 represent the time distances between the current image (Cur Pic) and the two reference images (Ref 0, Ref 1), respectively. (MVx 0, MVy 0) represents the motion vector corresponding to the reference image Ref 0, and (MVx 1, MVy 1) represents the motion vector corresponding to the reference image Ref 1.

[0183] Here, under the assumption of uniform linear motion represented by the velocity vector (vx, vy), (MVx 0, MVy 0) and (MVx 1, MVy 1) are represented as (v xτ 0, vyτ 0) and (-v xτ 1, -vyτ 1) respectively, and the following optical flow equation (Equation 1) is given. [Mathematical Formula 1] (1)

[0184] Here, I(k) represents the brightness value from the reference image k (k = 0, 1) after motion compensation. The optical flow equation shows that the sum of the following terms equals zero: (i) the temporal derivative of the brightness value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image. Motion vectors of blocks obtained, for example, from a merge list can be corrected pixel-by-pixel based on a combination of this optical flow equation and Hermite interpolation.

[0185] It should be noted that motion vectors can be derived on the decoder side using methods other than those based on a model employing uniform linear motion. For example, motion vectors can be derived for each sub-block based on the motion vectors of neighboring blocks.

[0186] Next, a description of the pattern used to derive motion vectors for each sub-block based on motion vectors from neighboring blocks is given. This pattern is also known as the affine motion compensation prediction pattern.

[0187] Figure 9A illustrates an example of deriving motion vectors for each sub-block based on motion vectors of neighboring blocks. In Figure 9A, the current block comprises 16 (4×4) sub-blocks. Here, the motion vector v0 of the top-left control point in the current block is derived based on the motion vectors of neighboring sub-blocks. Similarly, the motion vector v1 of the top-right control point in the current block is derived based on the motion vectors of neighboring blocks. Then, using the two motion vectors v0 and v1, the motion vectors (vx, vy) of each sub-block in the current block are derived using Equation 2 below. [Mathematical Expression 2] (2)

[0188] Here, x and y are the horizontal and vertical positions of the sub-block, respectively, and w is a predetermined weighting coefficient.

[0189] Affine motion compensation prediction modes can include many modes of different methods for deriving motion vectors for the top-left and top-right control points. Information indicating the affine motion compensation prediction mode (referred to as, for example, affine flags) can be communicated at the CU level. It should be noted that the communication of information indicating the affine motion compensation prediction mode does not need to be performed at the CU level, but can be performed at another level (e.g., at the sequence level, image level, slice level, block level, CTU level, or sub-block level). (Predictive Controller)

[0190] Prediction controller 128 selects either an in-frame prediction signal (output from in-frame predictor 124) or an inter-frame prediction signal (output from inter-frame predictor 126), and outputs the selected prediction signal to subtractor 104 and adder 116.

[0191] As illustrated in Figure 1, in various implementations, the prediction controller 128 may output prediction parameters, which are input to the entropy encoder 110. The entropy encoder 110 may generate an encoded bitstream (or sequence) based on the prediction parameters input to the prediction controller 128 and the quantization coefficients input to the quantizer 108. These prediction parameters may be used by a decoder that receives and decodes the encoded bitstream to perform the same prediction processing as that performed in the in-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128. These prediction parameters may include the selected prediction signal (e.g., motion vector, prediction type, or prediction mode used in the in-frame predictor 124 or the inter-frame predictor 126), or any index, flag, or value based on or indicating the prediction processing performed in the in-frame predictor 124, the inter-frame predictor 126, and the prediction controller 128.

[0192] Figure 9B illustrates an example of a procedure for exporting motion vectors in the current image in merge mode.

[0193] First, a list of predicted MVs is generated, and candidate predicted MVs are registered in this list. Examples of candidate predicted MVs include: spatially proximate predicted MVs, which are MVs of coded blocks located spatially near the target block; temporally proximate predicted MVs, which are MVs of blocks in the coded reference image that are adjacent to blocks in the same location as the target block; coupled predicted MVs, which are MVs generated by combining the MV values ​​of spatially proximate and temporally proximate predicted MVs; and zero predicted MVs, which are MVs with a value of zero.

[0194] Next, the MV of the target block is determined by selecting one of the multiple predicted MVs registered in the predicted MV list.

[0195] In addition, in variable length encoders, the signal merge_idx, which indicates which predicted MV is selected, is written and encoded into the stream.

[0196] It should be noted that the predicted MVs registered in the predicted MV list illustrated in Figure 9B constitute an instance. The number of predicted MVs registered in the predicted MV list may differ from the number illustrated in Figure 9B, and the predicted MVs registered in the predicted MV list may omit one or more of the predicted MV types given in the instance of Figure 9B. Furthermore, the predicted MVs registered in the predicted MV list may include one or more types of predicted MVs other than those given in the instance of Figure 9B or different from those types.

[0197] The final MV can be determined by performing DMVR (Dynamic Motion Vector Regeneration) processing (described later) using the MV of the target block exported in merge mode.

[0198] Figure 9C is a conceptual diagram illustrating one example of DMVR processing used to determine MV.

[0199] First, the most appropriate MV set for the current block (e.g., in merge mode) is considered as a candidate MV. Next, based on the candidate MV (L0), reference pixels are identified in a first reference image (L0), which is an encoded image in the L0 direction. Similarly, based on the candidate MV (L1), reference pixels are identified in a second reference image (L1), which is an encoded image in the L1 direction. These reference pixels are then averaged to form a template.

[0200] Next, using the template, the region surrounding the candidate MVs of the first and second reference images (L0) and (L1) is searched, and the MV with the lowest cost is determined as the final MV. The cost value can be calculated, for example, using the difference between the pixel values ​​in the template and the pixel values ​​in the region searched using the candidate MVs.

[0201] It should be noted that the configuration and operation of the program described here are fundamentally the same in both the encoder and decoder sides described below.

[0202] Any processing other than that described above may be used, as long as that processing can derive the final MV by searching the vicinity of candidate MVs.

[0203] Next, a description of one instance of a pattern for generating a predicted image (prediction) using LIC (Local Illumination Compensation) processing is given.

[0204] Figure 9D illustrates an example of a predictive image generation method that uses a brightness correction procedure performed by LIC processing.

[0205] First, an image image (MV) is derived from the encoded reference image to obtain the reference image corresponding to the current block.

[0206] Next, for the current block, based on the luminance pixel values ​​of the coded neighboring left reference region and the coded neighboring upper reference region in the current image, and based on the luminance pixel values ​​at the same positions in the reference image as defined by MV, information indicating how the luminance value changes between the reference image and the current image is obtained. This information indicating how the luminance value changes is used to calculate a luminance correction parameter.

[0207] A predictive image of the current block is generated by performing a brightness correction procedure, which applies brightness correction parameters to a reference image in a reference image defined by the MV.

[0208] It should be noted that the shape of the surrounding reference area illustrated in Figure 9D is only one example; the surrounding reference area may have different shapes.

[0209] Furthermore, although the predicted image in this example is generated from a single reference image, in the case where the predicted image is generated from multiple reference images, the predicted image can be generated after performing the brightness correction procedure described above on the reference image obtained from the reference image.

[0210] One example of a method for determining whether to implement LIC processing is using `lic_flag`, which is a signal indicating whether LIC processing is to be implemented. As a specific instance, the encoder determines whether the current block belongs to a region with brightness variations. If the block belongs to a region with brightness variations, the encoder sets `lic_flag` to a value of "1" and implements LIC processing during encoding. If the block does not belong to a region with brightness variations, the encoder sets `lic_flag` to a value of "0" and implements LIC processing to perform encoding. The decoder can switch between implementing and not implementing LIC processing by decoding the `lic_flag` written to the stream and performing decoding based on the flag value.

[0211] One example of a different method for determining whether to implement LIC processing includes identifying whether LIC processing is determined to be implemented for surrounding blocks. In a particular instance, when using merge mode for the current block, it is determined whether LIC processing should be applied when encoding surrounding encoded blocks selected when exporting the MV in merge mode. This determination is then used to further determine whether LIC processing should be implemented for the current block. It should be noted that, in this instance, the same applies to processing performed on the decoder side. (Decoder)

[0212] Next, a decoder capable of decoding the encoded signal (encoded bitstream) output by the autoencoder 100 will be described. Figure 10 is a block diagram illustrating the functional configuration of a decoder 200 according to one embodiment. Decoder 200 is an animation decoder that decodes animation block by block.

[0213] As illustrated in Figure 10, the decoder 200 includes an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a block memory 210, a loop filter 212, a frame memory 214, an in-frame predictor 216, an inter-frame predictor 218, and a prediction controller 220.

[0214] The decoder 200 is implemented, for example, as a general-purpose processor and memory. In this case, when the software program stored in memory is executed by the processor, the processor functions as an entropy decoder 202, an inverse quantizer 204, an inverse transformer 206, an adder 208, a loop filter 212, an in-frame predictor 216, an inter-frame predictor 218, and a prediction controller 220. Alternatively, the decoder 200 may be implemented as one or more dedicated electronic circuits corresponding to the entropy decoder 202, the inverse quantizer 204, the inverse transformer 206, the adder 208, the loop filter 212, the in-frame predictor 216, the inter-frame predictor 218, and the prediction controller 220.

[0215] The components included in the decoder 200 will be described below. (Entropy Decoder)

[0216] Entropy decoder 202 performs entropy decoding on the encoded bitstream. More specifically, for example, entropy decoder 202 arithmetically decodes the encoded bitstream into a binary signal. Entropy decoder 202 then debinds the binary signal. Entropy decoder 202 outputs the quantization coefficients of each block to dequantizer 204. Entropy decoder 202 can also output prediction parameters that may be included in the encoded bitstream (see FIG. 1) to in-frame predictor 216, inter-frame predictor 218, and prediction controller 220, such that the in-frame predictor, the inter-frame predictor, and the prediction controller can perform the same prediction processing as that performed on the encoder side in in-frame predictor 124, inter-frame predictor 126, and prediction controller 128. (Inverse quantizer)

[0217] Dequantizer 204 dequantizes the quantization coefficients of the block to be decoded (hereinafter referred to as the current block), which are input from entropy decoder 202. More specifically, dequantizer 204 dequantizes the quantization coefficients of the current block based on the quantization parameters corresponding to the quantization coefficients. Dequantizer 204 then outputs the dequantized coefficients (i.e., transform coefficients) of the current block to inverse transform 206. (Inverse Transformer)

[0218] The inverse transformer 206 recovers the prediction error (residual part) by performing an inverse transformation on the transformation coefficients input to the inverse quantizer 204.

[0219] For example, when the information parsed from the encoded bitstream indicates the application of EMT or AMT (e.g., when the AMT flag is set to true), the inverse transformer 206 performs an inverse transformation on the transform coefficients of the current block based on the indicated parsed transform type.

[0220] Furthermore, for example, when the information parsed from the encoded bitstream indicates the application of NSST, the inverse transformer 206 applies a quadratic inverse transform to the transform coefficients. (Adder)

[0221] Adder 208 reconstructs the current block by summing the prediction error input to reflexive transformer 206 and the prediction sample input from prediction controller 220. Adder 208 then outputs the reconstructed block to block memory 210 and loop filter 212. (Block memory)

[0222] Block memory 210 is a memory used to store blocks in the image to be decoded (hereinafter referred to as the current image) for reference during in-frame prediction. More specifically, block memory 210 stores the reconstructed blocks output from adder 208. (Loop filter)

[0223] The loop filter 212 applies the loop filter to the block reconstructed by the adder 208 and outputs the filtered reconstructed block to the frame memory 214 and, for example, a display device.

[0224] When the information indicating the enable or disable of the ALF is enabled by parsing from the encoded bit stream, one of the multiple filters is selected based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed block. (Graphic frame memory)

[0225] Frame memory 214 is a storage device used to store reference images used in inter-frame prediction and is also referred to as a frame buffer. More specifically, frame memory 214 stores reconstructed blocks filtered by loop filter 212. (In-frame predictor)

[0226] The in-frame predictor 216 generates a prediction signal (in-frame prediction signal) by performing in-frame prediction with reference to one or more blocks in the current image stored in the block memory 210. More specifically, the in-frame predictor 216 generates the in-frame prediction signal by performing in-frame prediction with reference to samples (e.g., brightness values ​​and / or chroma values) of one or more blocks adjacent to the current block, and then outputs the in-frame prediction signal to the prediction controller 220.

[0227] It should be noted that when the in-frame prediction mode of the chroma block predicted from the luminance block is selected, the in-frame predictor 216 can predict the chroma component of the current block based on the luminance component of the current block.

[0228] Furthermore, when the information indicating the application of PDPC is parsed from the encoded bit stream (for example, in the prediction parameters output by the entropy decoder 202), the in-frame predictor 216 corrects the in-frame predicted pixel values ​​based on the horizontal / vertical reference pixel gradients. (Inter-frame predictor)

[0229] Inter-frame predictor 218 refers to a reference image stored in frame memory 214 to predict the current block. Inter-frame prediction is performed for each current block or for each sub-block within the current block (e.g., each 4×4 block). For example, inter-frame predictor 218 generates an inter-frame prediction signal for the current block or sub-block based on motion compensation using motion information (e.g., motion vectors) parsed from an entropy decoder 202 (e.g., in the prediction parameters output by entropy decoder 202), and outputs the inter-frame prediction signal to prediction controller 220.

[0230] When the information parsed from the self-encoded bitstream indicates the application of the OBMC mode, the inter-frame predictor 218 uses motion information of neighboring blocks, in addition to the motion information of the current block obtained from self-motion estimation, to generate an inter-frame prediction signal.

[0231] Furthermore, when the information parsed from the self-encoded bitstream indicates the application of the FRUC pattern, the inter-frame predictor 218 derives motion information by performing motion estimation based on the pattern matching method (two-sided matching or template matching) parsed from the self-encoded bitstream. The inter-frame predictor 218 then uses the derived motion information to perform motion compensation (prediction).

[0232] Furthermore, when the BIO mode is to be applied, the inter-frame predictor 218 derives the motion vector based on a model employing uniform linear motion. Additionally, when the information parsed from the encoded bitstream indicates that the affine motion compensation prediction mode is to be applied, the inter-frame predictor 218 derives the motion vector of each sub-block based on the motion vectors of neighboring blocks. (Predictive Controller)

[0233] Prediction controller 220 selects either an in-frame prediction signal or an inter-frame prediction signal and outputs the selected prediction signal to adder 208. Generally, the configuration, function, and operation of prediction controller 220, inter-frame predictor 218, and in-frame predictor 216 on the decoder side can correspond to the configuration, function, and operation of prediction controller 128, inter-frame predictor 126, and in-frame predictor 124 on the encoder side. (Non-rectangular segmentation)

[0234] In the prediction controller 128 (see FIG. 1) coupled to the in-frame predictor 124 and the inter-frame predictor 126 on the encoder side, and in the prediction controller 220 (see FIG. 10) coupled to the in-frame predictor 216 and the inter-frame predictor 218 on the decoder side, until now, the segmented regions (variable-sized blocks or sub-blocks) obtained from each block from which split motion information (e.g., motion vectors) is obtained have always been rectangular, as shown in FIG. 2. The inventors have discovered that generating segmented regions with non-rectangular shapes, such as triangular shapes, results in improvements in image quality and coding efficiency, depending on the content of the image in the image in various embodiments. In the following, various embodiments will be described in which at least one segmented region for prediction purposes split from an image block has a non-rectangular shape. It should be noted that these embodiments are equally applicable on the encoder side (prediction controller 128 coupled to in-frame predictor 124 and inter-frame predictor 126) and the decoder side (prediction controller 220 coupled to in-frame predictor 216 and inter-frame predictor 218), and can be implemented in the encoder of FIG1 or the like, or in the decoder of FIG10 or the like.

[0235] Figure 11 is a flowchart illustrating an example of a procedure for splitting an image block into a segmentation region that includes at least a first segmentation region and a second segmentation region having a non-rectangular shape (e.g., a triangle), and performing other processes including encoding (or decoding) the image block as a reconstructed combination of the first and second segmentation regions.

[0236] In step S1001, an image block is split into a segmentation region comprising a first segmentation region having a non-rectangular shape and a second segmentation region that may or may not have a non-rectangular shape. For example, as shown in FIG12, an image block may be split from the upper left corner to the lower right corner to form a first segmentation region and a second segmentation region both having a non-rectangular shape (e.g., a triangle), or an image block may be split from the upper right corner to the lower left corner to form a first segmentation region and a second segmentation region both having a non-rectangular shape (e.g., a triangle). Various examples of non-rectangular segmentation will be described below with reference to FIG12 and FIGS17 to 19.

[0237] In step S1002, the program predicts a first motion vector for one of the first segmented regions and a second motion vector for one of the second segmented regions. For example, the prediction of the first and second motion vectors may include selecting the first motion vector from a first set of motion vector candidates and selecting the second motion vector from a second set of motion vector candidates.

[0238] In step S1003, a motion compensation procedure is executed to obtain the first segmentation region using the first motion vector derived in step S1002 above, and to obtain the second segmentation region using the second motion vector derived in step S1002 above.

[0239] In step S1004, a prediction procedure is performed on the image block that is a (reconstructed) combination of the first segmentation region and the second segmentation region. This prediction procedure may include a boundary smoothing procedure to smooth the boundary between the first and second segmentation regions. For example, the boundary smoothing procedure may involve weighting a first value of the boundary pixels predicted based on the first segmentation region and a second value of the boundary pixels predicted based on the second segmentation region. Various implementations of the boundary smoothing procedure will be described below with reference to Figures 13, 14, 20, and 21A to 21D.

[0240] In step S1005, the procedure encodes or decodes the image block using one or more segmentation parameters, which instruct the image block to be split into a first segmentation region and a second segmentation region having a non-rectangular shape. As outlined in the table of Figure 15, for example, the segmentation parameter ("first index value") may jointly encode, for example, the segmentation direction applied in the segmentation (e.g., from top left to bottom right or from top right to bottom left, as shown in Figure 12) and the first and second motion vectors derived above in step S1002. Details of this segmentation syntax operation involving one or more segmentation parameters will be described in detail below with reference to Figures 15, 16, and Figures 22 to 25.

[0241] Figure 17 is a flowchart illustrating the procedure 2000 for splitting an image block. In step S2001, the procedure splits an image into a plurality of segments, including a first segment with a non-rectangular shape and a second segment that may or may not have a non-rectangular shape. As shown in Figure 12, an image block can be split into a first segment with a triangular shape and a second segment that also has a triangular shape. There are numerous other instances where an image block is split into a plurality of segments, including a first segment and a second segment (where at least the first segment has a non-rectangular shape). The non-rectangular shape can be a triangle, a trapezoid, or a polygon with at least five sides and five corners.

[0242] For example, as shown in Figure 18, an image block can be divided into two triangular segments; an image block can be divided into more than two triangular segments (e.g., three triangular segments); an image block can be divided into a combination of triangular and rectangular segments; or an image block can be divided into a combination of triangular and polygonal segments.

[0243] As further shown in Figure 19, an image block can be divided into an L-shaped (polygonal) segmentation area and a rectangular segmentation area; an image block can be divided into a pentagonal (polygonal) segmentation area and a triangular segmentation area; an image block can be divided into a hexagonal (polygonal) segmentation area and a pentagonal (polygonal) segmentation area; or an image block can be divided into multiple polygonal segmentation areas.

[0244] Referring back to Figure 17, in step S2002, the program predicts a first motion vector for the first segmentation region by selecting a first segmentation region from a first set of motion vector candidates, and predicts a second motion vector for the second segmentation region by selecting a second segmentation region from a second set of motion vector candidates. For example, the first set of motion vector candidates may include motion vectors from segmentation regions adjacent to the first segmentation region, and the second set of motion vector candidates may include motion vectors from segmentation regions adjacent to the second segmentation region. These adjacent segmentation regions may be one or both of spatially adjacent and temporally adjacent. Some examples of spatially adjacent segmentation regions include those located to the left, lower left, bottom, lower right, right, upper right, top, or upper left of the segmentation region being processed. Examples of temporarily adjacent segmentation regions are co-located segmentation regions in the reference image of the image block.

[0245] In various implementations, the segmented regions adjacent to the first segmented region and the segmented regions adjacent to the second segmented region may be outside the image block that separates the first segmented region and the second segmented region. The first set of motion vector candidates may be the same as or different from the second set of motion vector candidates. Furthermore, at least one of the first set of motion vector candidates and the second set of motion vector candidates may be the same as another third set of motion vector candidates prepared for the image block.

[0246] In some implementations, in step S2002, in response to determining that the second segmentation region is similar to the first segmentation region and also has a non-rectangular shape (e.g., a triangle), program 2000 generates a second set of motion vector candidates (for non-rectangular second segmentation regions), which includes motion vectors of segmentation regions adjacent to the second segmentation region other than the first segmentation region (i.e., excluding the motion vector of the first segmentation region). On the other hand, in response to determining that the second segmentation region is different from the first segmentation region and has a rectangular shape, program 2000 generates a second set of motion vector candidates (for rectangular second segmentation regions), which includes motion vectors of segmentation regions adjacent to the second segmentation region that include the first segmentation region.

[0247] In step S2003, the program encodes or decodes the first segmented region using the first motion vector derived in step S2002 above, and encodes or decodes the second segmented region using the second motion vector derived in step S2002 above.

[0248] As in procedure 2000 of Figure 17, the image block splitting procedure can be executed by, for example, an image encoder as shown in Figure 1, which includes circuitry and a memory coupled to the circuitry. The circuitry, in operation, performs the following: splitting an image block into a plurality of segments including a first segment and a second segment having a non-rectangular shape (step S2001); predicting a first motion vector for the first segment and a second motion vector for the second segment (step S2002); and encoding the first segment using the first motion vector and encoding the second segment using the second motion vector (step S2003).

[0249] According to another embodiment, as shown in FIG1, an image encoder is provided, comprising: a splitter 102 that receives an original image and splits the original image into blocks during operation; an adder 104 that receives the blocks from the splitter and predictions from a prediction controller 128 during operation, and subtracts each prediction from the corresponding block to output a residual portion; a transformer 106 that performs a transformation on the residual portions output from the adder 104 during operation to output transformation coefficients; and a quantizer 108 that converts the transformed components into a quantizer during operation. The transformation coefficients are quantized to generate quantized transform coefficients; an entropy encoder 110, which, during operation, encodes the quantized transform coefficients to generate a one-bit stream; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124, and memories 118, 122, wherein the inter-frame predictor 126, during operation, generates a prediction of a current block based on a reference block in an encoded reference image, and the intra-frame predictor 124, during operation, generates a prediction of a current block based on an encoded reference block in a current image. The prediction controller 128, during operation, splits the blocks into a plurality of segments including a first segment and a second segment having a non-rectangular shape (Figure 17, step S2001); predicts a first motion vector for the first segment and a second motion vector for the second segment (step S2002); and encodes the first segment using the first motion vector and the second segment using the second motion vector (step S2003).

[0250] According to another embodiment, an image decoder is provided, as shown, for example, in FIG10. The image decoder includes circuitry and a memory coupled to the circuitry. In operation, the circuitry performs the following: splitting an image block into a plurality of segments including a first segment and a second segment having a non-rectangular shape (FIG. 17, step S2001); predicting a first motion vector for the first segment and a second motion vector for the second segment (step S2002); and decoding the first segment using the first motion vector and decoding the second segment using the second motion vector (step S2003).

[0251] According to another embodiment, an image decoder as shown in FIG10 is provided, comprising: an entropy decoder 202, which receives and decodes an encoded bitstream to obtain quantized transform coefficients; an inverse quantizer 204 and an inverse transformer 206, which inoperably inverse quantizes the quantized transform coefficients to obtain transform coefficients and inverse transforms the transform coefficients to obtain residual portions; an adder 208, which inoperably adds the residual portions output from the inverse quantizer 204 and the inverse transformer 206 to a prediction output from the predictive controller 220 to reconstruct a block; and a predictive controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216, and memories 210, 214, wherein the inter-frame predictor 218 inoperably generates a prediction of a current block based on a reference block in a decoded reference image, and the intra-frame predictor 216 inoperably generates a prediction of a current block based on a decoded reference block in a current image. During operation, the prediction controller 220 divides an image block into a plurality of segments, including a first segment and a second segment, each having a non-rectangular shape (Figure 17, step S2001); predicts a first motion vector for the first segment and a second motion vector for the second segment (step S2002); and decodes the first segment using the first motion vector and the second segment using the second motion vector (step S2003). (Boundary smoothing)

[0252] As described above in Figure 11, step S1004 of the prediction procedure for an image block that is a (reconstructed) combination of a first segmentation region and a second segmentation region having a non-rectangular shape, according to various embodiments, may involve the application of a boundary smoothing procedure along the boundary between the first segmentation region and the second segmentation region.

[0253] For example, Figure 21B illustrates an instance of a boundary smoothing procedure that involves weighting a first value of a boundary pixel predicted based on a first segmentation region and a second value of a boundary pixel predicted based on a second segmentation region.

[0254] Figure 20 is a flowchart illustrating a global boundary smoothing procedure 3000 according to one embodiment. The global boundary smoothing procedure involves weighting a first value of a boundary pixel predicted based on a first segmentation region and a second value of a boundary pixel predicted based on a second segmentation region. In step S3001, an image block is divided into a first segmentation region and a second segmentation region along a boundary, wherein at least the first segmentation region has a non-rectangular shape, as shown in Figure 21A or Figures 12, 18 and 19 above.

[0255] In step S3002, a first value (e.g., color, brightness, transparency, etc.) of a set of pixels in a first segmentation region along the boundary (referred to as "boundary pixels" in FIG. 21A) is first predicted, wherein the first value is first predicted using information from the first segmentation region. In step S3003, a second value of a (same) set of pixels in the first segmentation region along the boundary is second predicted, wherein the second value is second predicted using information from the second segmentation region. In one embodiment, at least one of the first prediction and the second prediction is a frame-to-frame prediction procedure based on a reference segmentation region in an encoded reference image to predict the first value and the second value. Referring to FIG. 21D, in some embodiments, the prediction procedure predicts the first value of all pixels in a first segmentation region ("first set of samples") that includes the set of pixels where the first segmentation region and the second segmentation region overlap, and predicts the second value of the set of pixels only where the first segmentation region and the second segmentation region overlap ("second set of samples"). In another embodiment, at least one of the first prediction and the second prediction is an in-box prediction procedure based on an coded reference segmentation region in a current image to predict the first and second values. In some embodiments, one of the prediction methods used in the first prediction differs from one of the prediction methods used in the second prediction. For example, the first prediction may include an inter-box prediction procedure and the second prediction may include an in-box prediction procedure. The information used for the first prediction of the first values ​​or the second prediction of the second values ​​may be the motion vector of the first or second segmentation region, the in-box prediction direction, etc.

[0256] In step S3004, the first values ​​predicted using the first segmentation region and the second values ​​predicted using the second segmentation region are weighted. In step S3005, the weighted first and second values ​​are used to encode or decode the first segmentation region.

[0257] Figure 21B illustrates one example of a boundary smoothing operation, where the first and second segmentation regions overlap by five pixels (to the maximum extent) in each column or row. That is, the set of pixels in each column or row is at most five, for which a first value is predicted based on the first segmentation region and a second value is predicted based on the second segmentation region. Figure 21C illustrates another example of a boundary smoothing operation, where the first and second segmentation regions overlap by three pixels (to the maximum extent) in each column or row. That is, the set of pixels in each column or row is at most three, for which a first value is predicted based on the first segmentation region and a second value is predicted based on the second segmentation region.

[0258] Figure 13 illustrates another example of the boundary smoothing operation, where the first and second segmentation regions overlap by four pixels (to the maximum extent) in each column or row. That is, the set of pixels in each column or row has a maximum of four values, for which first values ​​are predicted based on the first segmentation region and second values ​​are predicted based on the second segmentation region. In the illustrated example, weights 1 / 8, 1 / 4, 3 / 4, and 7 / 8 can be applied to the first values ​​of the four pixels in the set, and weights 7 / 8, 3 / 4, 1 / 4, and 1 / 8 can be applied to the second values ​​of the four pixels in the set.

[0259] Figure 14 illustrates another example of the boundary smoothing operation, where the first and second segmentation regions overlap by zero pixels in each column or row (i.e., these segmentation regions do not overlap), overlap by one pixel in each column or row (to the maximum extent), and overlap by two pixels in each column or row (to the maximum extent). In the example where the first and second segmentation regions do not overlap, zero weight is applied. In the example where the first and second segmentation regions overlap by one pixel in each column or row, weight 1 / 2 can be applied to a first value of pixels in the set predicted based on the first segmentation region, and weight 1 / 2 can be applied to a second value of pixels in the set predicted based on the second segmentation region. In the example where the first and second segmentation regions overlap by two pixels in each column or row, weights 1 / 3 and 2 / 3 can be applied to the first value of two pixels in the set predicted based on the first segmentation region, and weights 2 / 3 and 1 / 3 can be applied to the second value of two pixels in the set predicted based on the second segmentation region.

[0260] According to the embodiments described above, the number of pixels in the set of pixels where the first and second segmentation regions overlap is an integer. In other embodiments, the number of overlapping pixels in the set may be a non-integer and may be, for example, a decimal. Furthermore, the weights applied to the first and second values ​​of the pixel set may be decimals or integers depending on the application.

[0261] The boundary smoothing procedure of process 3000 in Figure 20 can be performed by an image encoder, as shown in Figure 1. For example, the image encoder includes circuitry and a memory coupled to the circuitry. During operation, the circuitry performs a boundary smoothing operation along a boundary between a first segmentation region and a second segmentation region having a non-rectangular shape, the first and second segmentation regions being split from an image block (Figure 20, step S3001). The boundary smoothing operation includes: using information from the first segmentation region to first predict a first value for a set of pixels in the first segmentation region along the boundary (step S3002); using information from the second segmentation region to second predict a second value for the set of pixels in the first segmentation region along the boundary (step S3003); weighting the first and second values ​​(step S3004); and encoding the first segmentation region using the weighted first and second values ​​(step S3005).

[0262] According to another embodiment, as shown in FIG1, an image encoder is provided, comprising: a splitter 102 that receives an original image and splits the original image into blocks during operation; an adder 104 that receives the blocks from the splitter and predictions from a prediction controller 128 during operation, and subtracts each prediction from the corresponding block to output a residual portion; a transformer 106 that performs a transformation on the residual portions output from the adder 104 during operation to output transformation coefficients; and a quantizer 108 that converts the transformed components into a quantizer during operation. The transformation coefficients are quantized to generate quantized transform coefficients; an entropy encoder 110, which encodes these quantized transform coefficients during operation to generate a one-bit stream; and a prediction controller 128 coupled to an inter-frame predictor 126, an intra-frame predictor 124, and memories 118, 122, wherein the inter-frame predictor 126, during operation, generates a prediction of a current block based on a reference block in an encoded reference image, and the intra-frame predictor 124, during operation, generates a prediction of a current block based on an encoded reference block in a current image. The prediction controller 128, during operation, performs a boundary smoothing operation along a boundary between a first segmentation region and a second segmentation region having a non-rectangular shape, the first and second segmentation regions being split from an image block (FIG. 20, step S3001). The boundary smoothing operation includes: using information from the first segmentation region to first predict a first value of a set of pixels in the first segmentation region along the boundary (step S3002); using information from the second segmentation region to second predict a second value of the set of pixels in the first segmentation region along the boundary (step S3003); weighting the first value and the second value (step S3004); and encoding the first segmentation region using the weighted first value and the weighted second value (step S3005).

[0263] According to another embodiment, an image decoder is provided, as shown, for example, in FIG. 10. The image decoder includes circuitry and a memory coupled to the circuitry. During operation, the circuitry performs a boundary smoothing operation along a boundary between a first segmentation region and a second segmentation region having a non-rectangular shape, the first and second segmentation regions being split from an image block (FIG. 20, step S3001). The boundary smoothing operation includes: using information from the first segmentation region to first predict a first value for a set of pixels of the first segmentation region along the boundary (step S3002); using information from the second segmentation region to second predict a second value for the set of pixels of the first segmentation region along the boundary (step S3003); weighting the first and second values ​​(step S3004); and decoding the first segmentation region using the weighted first and second values ​​(step S3005).

[0264] According to another embodiment, an image decoder as shown in FIG10 is provided, comprising: an entropy decoder 202, which receives and decodes an encoded bitstream to obtain quantized transform coefficients; an inverse quantizer 204 and an inverse transformer 206, which inoperably inverse quantizes the quantized transform coefficients to obtain transform coefficients and inverse transforms the transform coefficients to obtain residual portions; an adder 208, which inoperably adds the residual portions output from the inverse quantizer 204 and the inverse transformer 206 to a prediction output from the predictive controller 220 to reconstruct a block; and a predictive controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216, and memories 210, 214, wherein the inter-frame predictor 218 inoperably generates a prediction of a current block based on a reference block in a decoded reference image, and the intra-frame predictor 216 inoperably generates a prediction of a current block based on a decoded reference block in a current image. During operation, the prediction controller 220 performs a boundary smoothing operation along a boundary between a first segmentation region and a second segmentation region having a non-rectangular shape, the first and second segmentation regions being split from an image block. (Figure 20, step S3001) The boundary smoothing operation includes: using information from the first segmentation region to first predict a first value of a set of pixels of the first segmentation region along the boundary (step S3002); using information from the second segmentation region to second predict a second value of the set of pixels of the first segmentation region along the boundary (step S3003); weighting the first value and the second value (step S3004); and decoding the first segmentation region using the weighted first value and the weighted second value (step S3005). (Using entropy encoding and decoding with segmentation parameter syntax)

[0265] As described in Figure 11, in step S1005, according to various embodiments, the image block split into a first segmentation region and a second segmentation region having a non-rectangular shape can be encoded or decoded using one or more parameters including segmentation parameters indicating the non-rectangular split of the image block. In various embodiments, this segmentation parameter can jointly encode, for example, the splitting direction applied to the split (e.g., from top left to bottom right or from top right to bottom left, see Figure 12) and the first and second motion vectors predicted in step S1002, as will be described more fully below.

[0266] Figure 15 is a table of sample segmentation parameters ("first index values") and sets of information jointly encoded by these segmentation parameters. These segmentation parameters ("first index values") range from 0 to 6 and jointly encode: the direction in which the image block is split into a first segmentation region and a second segmentation region, both of which are triangles (see Figure 12); a first motion vector predicted for the first segmentation region (Figure 11, step S1002); and a second motion vector predicted for the second segmentation region (Figure 11, step S1002). Specifically, segmentation parameter 0 encodes the splitting direction from the upper left corner to the lower right corner; the first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first segmentation region; and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second segmentation region.

[0267] Segmentation parameter 1 encodes the splitting direction from the upper right corner to the lower left corner. The first motion vector is the "first" motion vector listed in the first set of motion vector candidates for the first segmentation region, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second segmentation region. Segmentation parameter 2 encodes the splitting direction from the upper right corner to the lower left corner. The first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first segmentation region, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second segmentation region. Segmentation parameter 3 encodes the splitting direction from the upper left corner to the lower right corner. The first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first segmentation region, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second segmentation region. Segmentation parameter 4 encodes the splitting direction from the upper right corner to the lower left corner. The first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first segmentation region, and the second motion vector is the "third" motion vector listed in the second set of motion vector candidates for the second segmentation region. Segmentation parameter 5 encodes the splitting direction from the upper left corner to the lower right corner. The first motion vector is the "third" motion vector listed in the first set of motion vector candidates for the first segmentation region, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second segmentation region. Segmentation parameter 6 encodes the splitting direction from the upper left corner to the lower right corner. The first motion vector is the "fourth" motion vector listed in the first set of motion vector candidates for the first segmentation region, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second segmentation region.

[0268] Figure 22 is a flowchart illustrating method 4000 performed on the encoder side. In step S4001, the procedure divides an image block into a plurality of segmented regions, including a first segmented region and a second segmented region, based on a segmentation parameter indicating the division. For example, as shown in Figure 15 above, the segmentation parameter may indicate the direction of splitting the image block (e.g., from the upper right corner to the lower left corner or from the upper left corner to the lower right corner). In step S4002, the procedure encodes the first segmented region and the second segmented region. In step S4003, the procedure writes one or more parameters, including the segmentation parameter, into a bitstream, which the decoder side can receive and decode to obtain the one or more parameters, thereby performing the same prediction procedure (as performed on the encoder side) for the first and second segmented regions on the decoder side. This includes various segments of information that can be jointly or individually encoded by one or more of the segmentation parameters, such as the non-rectangular shape of the first segmentation region, the shape of the second segmentation region, the splitting direction used to split the image block to obtain the first and second segmentation regions, the first motion vector of the first segmentation region, the second motion vector of the second segmentation region, etc.

[0269] Figure 23 is a flowchart illustrating method 5000 performed on the decoder side. In step S5001, the program analyzes one or more parameters from a bitstream, wherein the one or more parameters include a segmentation parameter indicating that an image block is split into a plurality of segmented regions, including a first segmented region and a second segmented region having a non-rectangular shape. The one or more parameters including the segmentation parameter analyzed from the bitstream can jointly or individually encode various segments of information required for performing the same prediction procedure on the decoder side as on the encoder side, such as the non-rectangular shape of the first segmented region, the shape of the second segmented region, the splitting direction for splitting the image block to obtain the first and second segmented regions, the first motion vector of the first segmented region, the second motion vector of the second segmented region, etc. In step S5002, program 5000 splits the image block into a plurality of segmented regions based on the segmentation parameters analyzed from the bitstream. In step S5003, the program decodes the first segmented region and the second segmented region split from the image block.

[0270] Figure 24 is a table of sample segmentation parameters ("first index values") and the set of information jointly encoded by these segmentation parameters, which is essentially similar to the sample table described above in Figure 15. In Figure 24, the segmentation parameters ("first index values") range from 0 to 6 and jointly encode: the shapes of the first and second segmentation regions from which the image block is split, the direction in which the image block is split into the first and second segmentation regions, the first motion vector predicted for the first segmentation region (Figure 11, step S1002), and the second motion vector predicted for the second segmentation region (Figure 11, step S1002). Specifically, segmentation parameter 0 encodes that neither the first nor the second segmentation region has a triangular shape, and therefore the splitting direction information is "N / A", the first motion vector information is "N / A", and the second motion vector information is "N / A".

[0271] Segmentation parameter 1 encodes the first and second segmentation triangles, with the splitting direction from the upper left to the lower right. The first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first segmentation, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second segmentation. Segmentation parameter 2 encodes the first and second segmentation triangles, with the splitting direction from the upper right to the lower left. The first motion vector is the "first" motion vector listed in the first set of motion vector candidates for the first segmentation, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second segmentation. Segmentation parameter 3 encodes the first and second segmentation triangles, with the splitting direction from the upper right to the lower left. The first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first segmentation, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second segmentation. Segmentation parameter 4 encodes the first and second segmentation triangles, with the splitting direction from the upper left to the lower right. The first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first segmentation, and the second motion vector is the "second" motion vector listed in the second set of motion vector candidates for the second segmentation. Segmentation parameter 5 encodes the first and second segmentation triangles, with the splitting direction from the upper right to the lower left. The first motion vector is the "second" motion vector listed in the first set of motion vector candidates for the first segmentation, and the second motion vector is the "third" motion vector listed in the second set of motion vector candidates for the second segmentation. Segmentation parameter 6 encodes the first and second segmentation triangles, with the splitting direction from the upper left to the lower right. The first motion vector is the "third" motion vector listed in the first set of motion vector candidates for the first segmentation, and the second motion vector is the "first" motion vector listed in the second set of motion vector candidates for the second segmentation.

[0272] According to some implementations, these segmentation parameters (index values) can be binary-coded according to a binary-coded scheme selected based on the value of at least one or more parameters. Figure 16 illustrates a sample binary-coded scheme for binary index values ​​(segmentation parameter values).

[0273] Figure 25 is a table of sample combinations of the first and second parameters, where one of the first and second parameters is a segmentation parameter that indicates the division of an image block into a plurality of segmented regions, including a first segmented region and a second segmented region having a non-rectangular shape. In this example, the segmentation parameter can be used to indicate the division of an image block without jointly encoding other information, which is encoded by one or more of the other parameters.

[0274] In the first example in Figure 25, the first parameter is used to indicate the size of the image block, and the second parameter is used as a segmentation parameter (flag) to indicate that at least one of the plurality of segmented regions split from the image block has a triangular shape. This combination of the first and second parameters can be used to indicate, for example: 1) when the image block size is greater than 64×64, that there is no triangular segmented region; or 2) when the width-to-height ratio of the image block is greater than 4 (e.g., 64x4), that there is no triangular segmented region.

[0275] In the second example in Figure 25, the first parameter is used to indicate the prediction mode, and the second parameter is used as a segmentation parameter (flag) to indicate that at least one of the plurality of segmented regions split from the image block has a triangular shape. This combination of the first and second parameters can be used to indicate, for example, 1) that no triangular segmented region exists when the image block is written in in-frame mode.

[0276] In the third example in Figure 25, the first parameter is used as a segmentation parameter (flag) to indicate that at least one of the plurality of segmented regions split from the image block has a triangular shape, and the second parameter is used to indicate the prediction mode. This combination of the first and second parameters can be used to indicate, for example, 1) that when at least one of the plurality of segmented regions split from the image block has a triangular shape, the image block must be inter-frame coded.

[0277] In the fourth example of Figure 25, the first parameter indicates the motion vector of the neighboring block, and the second parameter is used as a segmentation parameter to indicate the direction in which the image block is split into two triangles. This combination of the first and second parameters can be used to indicate, for example, 1) that when the motion vector of the neighboring block is diagonally oriented, the direction in which the image block is split into two triangles is from the upper left corner to the lower right corner.

[0278] In the fifth example of Figure 25, the first parameter indicates the prediction direction within the frame of the neighboring block, and the second parameter is used as a segmentation parameter to indicate the direction in which the image block is split into two triangles. This combination of the first and second parameters can be used to indicate, for example, 1) that when the prediction direction within the frame of the neighboring block is in the opposite diagonal direction, the direction in which the image block is split into two triangles is from the upper right corner to the lower left corner.

[0279] It should be understood that, as shown in Figures 15, 24, and 25, the tables including one or more segmentation parameters and what information is jointly or individually encoded are presented only as examples, and numerous other ways of jointly or individually encoding various information as part of the segmentation syntax operations described above are within the scope of this invention. For example, the segmentation parameter may indicate that the first segmentation region is a triangle, trapezoid, or a polygon with at least five sides and five corners. The segmentation parameter may indicate that the second segmentation region has a non-rectangular shape, such as a triangle, trapezoid, or a polygon with at least five sides and five corners. The segmentation parameter may indicate one or more segments of information about the split, such as the non-rectangular shape of the first segmentation region, the shape of the second segmentation region (which may be non-rectangular or rectangular), and the splitting direction applied to split the image block into a plurality of segmentation regions (e.g., from the upper left corner of the image block to the lower right corner of the image block, and from the upper right corner of the image block to the lower left corner of the image block). Segmentation parameters can jointly encode other information, such as the first motion vector of the first segmentation region, the second motion vector of the second segmentation region, the image block size, the prediction mode, the motion vectors of neighboring blocks, and the in-frame prediction direction of neighboring blocks. Alternatively, any of the other information can be encoded individually by one or more parameters other than the segmentation parameters.

[0280] As in procedure 4000 of Figure 22, the segmentation syntax operation can be performed by an image encoder, such as that shown in Figure 1, which includes circuitry and a memory coupled to the circuitry. During operation, the circuitry performs a segmentation syntax operation comprising: splitting an image block into a plurality of segments, including a first segment and a second segment, each having a non-rectangular shape, based on a segmentation parameter indicating the splitting (Figure 22, step S4001); encoding the first and second segments (S4002); and writing one or more of the segmentation parameters into a one-bit stream (S4003).

[0281] According to another embodiment, as shown in FIG1, an image encoder is provided, comprising: a splitter 102 that receives an original image and splits the original image into blocks during operation; an adder 104 that receives the blocks from the splitter and predictions from a prediction controller 128 during operation, and subtracts each prediction from the corresponding block to output a residual portion; a transformer 106 that performs a transformation on the residual portions output from the adder 104 during operation to output transformation coefficients; and a quantizer 108 that converts the transformed components into a quantizer during operation. The process involves coefficient quantization to generate quantized transform coefficients; an entropy encoder 110, which, during operation, encodes these quantized transform coefficients to generate a one-bit stream; and a prediction controller 128 coupled to inter-frame predictor 126, intra-frame predictor 124, and memories 118 and 122, wherein the inter-frame predictor 126, during operation, generates a prediction of a current block based on a reference block in an encoded reference image, and the intra-frame predictor 124, during operation, generates a prediction of a current block based on an encoded reference block in a current image. The prediction controller 128, during operation, divides an image block into a plurality of segmentation regions, including a first segmentation region and a second segmentation region having non-rectangular shapes, based on segmentation parameters indicating splitting (Figure 22, step S4001), and encodes the first and second segmentation regions (step S4002). The entropy encoder 110, during operation, writes one or more of these segmentation parameters into a one-bit stream (step S4003).

[0282] According to another embodiment, an image decoder is provided, as shown, for example, in FIG10. The image decoder includes circuitry and a memory coupled to the circuitry. During operation, the circuitry performs a segmentation syntax operation comprising: parsing one or more parameters from a one-bit stream, wherein the one or more parameters include a segmentation parameter indicating that an image block is split into a plurality of segmented regions including a first segmented region and a second segmented region having a non-rectangular shape (FIG. 23, step S5001); splitting the image block into the plurality of segmented regions based on the segmentation parameter (S5002); and decoding the first segmented region and the second segmented region (S5003).

[0283] According to another embodiment, an image decoder as shown in FIG10 is provided, comprising: an entropy decoder 202, which receives and decodes an encoded bitstream to obtain quantized transform coefficients; an inverse quantizer 204 and an inverse transformer 206, which inoperably inverse quantizes the quantized transform coefficients to obtain transform coefficients and inverse transforms the transform coefficients to obtain residual portions; an adder 208, which inoperably adds the residual portions output from the inverse quantizer 204 and the inverse transformer 206 to a prediction output from the predictive controller 220 to reconstruct a block; and a predictive controller 220 coupled to an inter-frame predictor 218, an intra-frame predictor 216, and memories 210, 214, wherein the inter-frame predictor 218 inoperably generates a prediction of a current block based on a reference block in a decoded reference image, and the intra-frame predictor 216 inoperably generates a prediction of a current block based on a decoded reference block in a current image. When the entropy decoder 202 operates, it analyzes one or more parameters from a one-bit stream, wherein the one or more parameters include a segmentation parameter indicating that an image block is split into a plurality of segmentation regions including a first segmentation region and a second segmentation region having a non-rectangular shape (Figure 23, step S5001); splits the image block into the plurality of segmentation regions based on the segmentation parameter (S5002); and in some implementations, it co-decodes the first segmentation region and the second segmentation region with the predictive controller 220 (S5003). (Implementation and Application)

[0284] As described in each of the above embodiments, each functional or operational block can typically be implemented as, for example, a microprocessor unit (MPU) and memory. Furthermore, the program executed by each of the functional blocks can be implemented as a program execution unit, such as a processor that reads and executes software (programs) recorded on a recording medium such as ROM. The software can be distributed. The software can be recorded on various recording media such as semiconductor memory. It should be noted that each functional block can also be implemented as hardware (dedicated circuitry).

[0285] The processes described in each of the embodiments can be implemented using a single device (system) via integrated processing, and alternatively, can be implemented using multiple devices via distributed processing. Furthermore, the processor executing the above programs can be a single processor or multiple processors. In other words, integrated processing can be executed, and alternatively, distributed processing can be executed.

[0286] The embodiments of the present invention are not limited to the above exemplary embodiments; various modifications can be made to the exemplary embodiments, and the results of the modifications are also included within the scope of the embodiments of the present invention.

[0287] Next, application examples of the animation encoding method (image encoding method) and animation decoding method (image decoding method) described in each of the above embodiments will be described, as well as various systems for implementing these application examples. The system may include an image encoder using the image encoding method, an image decoder using the image decoding method, or an image encoder-decoder that includes both the image encoder and the image decoder. Other configurations of this system may be modified as needed. (Usage Examples)

[0288] Figure 26 illustrates the overall configuration of a content delivery system ex100 suitable for implementing content distribution services. The area providing communication services is divided into cells of the desired size, and in the illustrated example, the base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations, are located in their respective cells.

[0289] In the content providing system ex100, devices including a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the Internet ex101 via an Internet service provider ex102 or a communication network ex104 and base stations ex106 to ex110. The content providing system ex100 can combine and connect any combination of the above devices. In various implementations, these devices can be directly or indirectly connected together via a telephone network or near-field communication instead of via base stations ex106 to ex110. Furthermore, a streaming server ex103 can be connected to the devices including the computer ex111, gaming device ex112, camera ex113, home appliance ex114, and smartphone ex115 via, for example, the Internet ex101. The streaming server ex103 can also be connected to a terminal in a hotspot, such as an airplane ex117, via a satellite ex116.

[0290] It should be noted that wireless access points or hotspots can be used to replace base stations ex106 to ex110. Streaming server ex103 can be directly connected to communication network ex104 instead of via Internet ex101 or Internet service provider ex102, and can be directly connected to aircraft ex117 instead of via satellite ex116.

[0291] The ex113 camera series is a device capable of capturing still images and video, such as a digital camera. The ex115 smartphone is a smartphone device, mobile phone, or personal mobile phone system (PHS) capable of operating according to mobile communication system standards of 2G, 3G, 3.9G, 4G systems, and the next-generation 5G system.

[0292] Home appliances, such as refrigerators or devices included in home fuel cell symbiotic systems, are classified as such.

[0293] In the content providing system ex100, a terminal including image and / or video capture functions can perform real-time streaming transmission, for example, by connecting to a streaming server ex103 via a base station ex106. During real-time streaming, a terminal (e.g., a computer ex111, a gaming device ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or an airplane ex117) can perform the encoding processing described in the above embodiments on still images or video content captured by the user via the terminal. It can perform multiplexing on the video data obtained through encoding and the audio data obtained by encoding the audio corresponding to the video, and can transmit the obtained data to the streaming server ex103. In other words, the terminal functions as an image encoder according to one aspect of the present invention.

[0294] The streaming server ex103 transmits the content data stream to the user terminal requesting the stream. Examples of user terminals include computers ex111, gaming devices ex112, cameras ex113, home appliances ex114, smartphones ex115, and terminals inside aircraft ex117, which are capable of decoding the encoded data. The device receiving the streamed data decodes and reproduces the received data. In other words, according to one embodiment of the invention, each of these devices can function as an image decoder. (Decentralized processing)

[0295] The streaming server ex103 can be implemented as a plurality of servers or computers, with tasks such as data processing, recording, and streaming distributed among these servers or computers. For example, the streaming server ex103 can be implemented as a Content Delivery Network (CDN), which streams content via a network connecting multiple edge servers located worldwide. In a CDN, edge servers physically close to the user are dynamically assigned to the user. Content is cached and streamed to edge servers to reduce loading time. In the event of a certain type of error or variation in connectivity, such as due to traffic spikes, it is possible to stream data stably at high speeds because it is possible to avoid affected parts of the network by, for example, distributing processing among a plurality of edge servers or switching streaming operations to different edge servers and continuing streaming.

[0296] Decentralization is not limited to splitting processing only for streaming transmission; the encoding of captured data can be split between terminals and performed by the terminals, on the server side, or both. In one example, in typical encoding, processing is performed in two loops. The first loop is used to detect the complexity of the image on a frame-by-frame or scene-by-scene basis, or to detect the encoding load. The second loop is used to maintain image quality and improve encoding efficiency. For example, it is possible to reduce the processing load on the terminals and improve the quality of the content and encoding efficiency by having the terminals perform the first loop of encoding and the server side of receiving the content perform the second loop of encoding. In this case, upon receiving a decoding request, the encoded data generated by the first loop performed by one terminal can be received and reproduced on another terminal almost instantly. This makes smooth real-time streaming transmission possible.

[0297] In another example, the camera ex113 or similar extracts features from the image, compresses the data associated with those features into post-processing data, and transmits the compressed post-processing data to a server. For instance, the server determines the importance of objects based on the features and adjusts the quantization accuracy accordingly to perform compression appropriate to the meaning (or content importance) of the image. Feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction during the second compression stage performed by the server. Furthermore, encoding methods with relatively low processing load, such as Variable Length Write Code (VLC), can be processed by the terminal, while encoding methods with relatively high processing load, such as Context Adaptive Binary Arithmetic Write Code (CABAC), can be processed by the server.

[0298] In another example, there are multiple videos of similar scenes captured by multiple terminals in, for example, a stadium, shopping mall, or factory. In this case, for instance, encoding can be distributed by dividing the processing task on a per-unit basis among the multiple terminals capturing the video, other terminals not capturing video when necessary, and servers. These units can be, for example, groups of images (GOPs), images, or blocks generated from segmented images. This makes it possible to reduce loading times and achieve near-real-time streaming.

[0299] Since the video feeds depict similar scenes, a server can be used to manage and / or command the video captured by the terminal, allowing for cross-referencing. Furthermore, the server can receive encoded data from the terminal, alter the reference relationships between data items, or correct or replace the image itself, and then perform encoding. This makes it possible to produce streams with the quality and efficiency of individual items of the augmented data.

[0300] Furthermore, the server can stream video data after performing transcoding to change the encoding format of the video data. For example, the server can convert the encoding format from MPEG to VP (e.g., VP9) and can convert H.264 to H.265.

[0301] In this way, encoding can be executed via a terminal or one or more servers. Therefore, although the device performing encoding is referred to as a "server" or "terminal" in the following description, some or all of the programs executed by a server can be executed by a terminal, and similarly, some or all of the programs executed by a terminal can be executed by a server. This also applies to decoding programs. (3D, multi-angle)

[0302] The use of images or videos combined from multiple terminals, such as a camera ex113 and / or a smartphone ex115, simultaneously capturing images or videos of different scenes or the same scene captured from different angles, has increased. The video captured by the terminal is combined based on, for example, the relative positional relationship obtained individually between the terminals or areas in the video with matching feature points.

[0303] In addition to encoding 2D animation, the server can also encode still images (automatically or at user-specified time points) based on scene analysis of the animation, and transmit the encoded still images to the receiving terminal. Furthermore, when the server can obtain the relative positional relationships between video capture terminals, it can generate the 3D geometry of a scene based on video of the same scene captured from different angles, in addition to 2D animation. The server can individually encode 3D data, such as point clouds, and based on the results of using 3D data to identify or track individuals or objects, it can select or reconstruct video from video captured by multiple terminals and generate video to be transmitted to the receiving terminal.

[0304] This allows users to freely select the video corresponding to the video capture terminal to enjoy the scene, and allows users to enjoy the content obtained by extracting the video from three-dimensional data reconstructed from multiple images or videos from a selected perspective. In addition, like video, sound can be recorded from different relative angles, and the server can multiplex audio from a specific angle or space corresponding to the video, and transmit the multiplexed video and audio.

[0305] In recent years, content that combines the real and virtual worlds (such as virtual reality (VR) and augmented reality (AR) content) has become increasingly popular. In the case of VR video, the server can create images from both the left and right eye perspectives and perform encoding of references between the two perspective images, such as multi-view coding (MVC). Alternatively, the images can be encoded as separate streams without references. When the images are decoded into separate streams, these streams can be synchronized during playback to reconstruct a virtual three-dimensional space according to the user's perspective.

[0306] In the case of AR imagery, the server overlays information about virtual objects existing in virtual space onto camera information representing real-world space based on 3D position or movement of the user's viewpoint. The decoder can acquire or store virtual object information and 3D data, generate 2D images based on movement of the user's viewpoint, and then generate overlay data by seamlessly connecting the images. Alternatively, in addition to requests for virtual object information, the decoder can also transmit motion from the user's viewpoint to the server. The server can generate overlay data based on 3D data stored on the server according to the received motion, encode the generated overlay data, and stream it to the decoder. It should be noted that, in addition to RGB values, the overlay data also includes α values ​​indicating transparency, and the server sets the α value to, for example, 0 for areas other than objects generated from 3D data, and can perform encoding when those areas are transparent. Alternatively, the server can set the background to a predetermined RGB value such as chroma key and generate data where areas other than objects are set as the background.

[0307] Similarly, decoding of streamed data can be performed at the user end (i.e., the terminal), on the server side, or segmented between the two. In one example, a terminal can transmit a receive request to the server, the requested content can be received and decoded by another terminal, and the decoded signal can be transmitted to a device with a display. It is possible to reproduce high-quality image data through distributed processing and appropriate content selection, regardless of the processing power of the communication terminal itself. In another example, when a TV receives large image data, such as areas of an image obtained by segmenting the image into squares, areas can be decoded and displayed on one or more terminals of one or more viewers. This allows viewers to share a panoramic view and allows each viewer to check their assigned area or view an area in more detail.

[0308] In situations where multiple wireless connections exist at short, medium, and long distances, indoors, or outdoors, it is possible to seamlessly receive content using streaming transmission system standards such as MPEG-DASH. Users can freely switch between data in real time, choosing from various options, including the decoder or display device of their terminal, and whether the display is located indoors or outdoors. Furthermore, using information such as the user's location, decoding can be performed simultaneously by switching which terminal handles decoding and which terminal handles content display. This makes it possible to map and display information on the walls or ground surfaces of nearby buildings with content display devices while the user is moving towards their destination. Additionally, it is possible to switch the bit rate of received data based on the accessibility of the encoded data on the network, such as when the encoded data is cached on a server that can be quickly accessed from the receiving terminal, or when the encoded data is copied to an edge server in a content delivery service. (Adjustable encoding)

[0309] Content switching will be described with reference to the adjustable stream illustrated in Figure 27, which is compressed and encoded using the animation encoding method described in the above embodiments. The server may have a configuration where content is switched using the temporal and / or spatial scalability of the stream, achieved by dividing it into multiple layers and encoding those layers, as illustrated in Figure 27. It should be noted that multiple individual streams with the same content but different qualities may exist. In other words, by determining which layer to decode based on internal factors such as the processing power on the decoder side and external factors such as communication bandwidth, the decoder side can freely switch between low-resolution and high-resolution content during decoding. For example, if a user wants to continue watching, for instance, a video previously viewed on a smartphone ex115 while on the go, on a device such as a TV connected to the internet at home, the device can simply decode the same stream up to a different layer, thus reducing the server-side load.

[0310] In addition to the configurations described above (where scalability is achieved as a result of images encoded layer by layer), when the enhancement layer is above the base layer, the enhancement layer may include post-processing data based on, for example, statistical information about the image. The decoder can then perform super-resolution imaging on the image in the base layer based on the post-processing data to produce high-quality image content. Super-resolution imaging can improve the signal-to-noise ratio (SN ratio) while maintaining and / or increasing resolution. Post-processing data includes information such as that used in super-resolution processing to identify linear or nonlinear filter coefficients, or information identifying parameter values ​​in filter processing, machine learning, or least-squares methods used in super-resolution processing.

[0311] Alternatively, a configuration can be provided in which the image is divided into, for example, blocks based on the meaning of objects in the image. On the decoder side, only a portion of the area is decoded by selecting the blocks to be decoded. Furthermore, by storing object attributes (people, cars, balls, etc.) and the positions of objects in the video (coordinates within the same image) as post-processing data, the decoder can identify the location of the desired object and determine which blocks(s) contain that object. For example, as illustrated in Figure 28, the post-processing data can be stored using a different data storage structure than pixel data, such as supplemental enhancement information (SEI) messages in HEVC. This post-processing data indicates, for example, the position, size, or color of the primary object.

[0312] The data can be stored in units of multiple images, such as streams, sequences, or random access units. The decoder can obtain, for example, the time when a specific person appears in the video, and by matching the time information with the image unit information, it can identify the image containing that object and determine the position of the object in the image. (Webpage optimization)

[0313] Figure 29 illustrates an example of a webpage displayed on a screen, such as a computer ex111. Figure 30 illustrates an example of a webpage displayed on a screen, such as a smartphone ex115. As illustrated in Figures 29 and 30, a webpage may include multiple image links to image content, and the appearance of the webpage varies depending on the device used to view it. When multiple image links are visible on the screen, the display device (decoder) may display still images included in the content or I-images as image links until the user explicitly selects an image link, or until the image link is approximately centered on the screen, or until the entire image link fits the screen. Multiple still images or I-images may be used to display videos such as animated GIFs; or only the base layer may be received, decoded, and displayed.

[0314] When an image link is selected by the user, the display device performs decoding, giving the base layer the highest priority. It should be noted that if the HTML code of the webpage contains information indicating that the content is adjustable, the display device can decode up to the enhancement layer. Furthermore, to ensure real-time reproduction, before a selection is made or when bandwidth is severely limited, the display device can reduce the delay between the time point of decoding the lead image and the time point of displaying the decoded image (i.e., the delay between the start of content decoding and content display) by only decoding and displaying the forward reference images (I-image, P-image, and forward reference B-image). Further, the display device can intentionally ignore the reference relationships between images and coarsely decode all B and P images as forward reference images before performing normal decoding, because the number of images received increases over time. (Self-driven)

[0315] When transmitting and receiving static images or video data, such as 2D or 3D map information for autonomous or assisted driving of automobiles, the receiving terminal can also receive information such as weather or road structure as post-processing data, in addition to image data belonging to one or more layers, and immediately associate this post-processing data with the image data after decoding. It should be noted that post-processing data can be assigned to each layer, and alternatively, the image data can be simply multiplexed.

[0316] In this scenario, since the vehicle, drone, or aircraft containing the receiving terminal is mobile, the receiving terminal can seamlessly receive and perform decoding when switching between base stations ex106 to ex110 by transmitting information indicating the receiving terminal's location. Furthermore, depending on the user's selection, the user's situation, and / or the connection bandwidth, the receiving terminal can dynamically select the range for receiving background data, or, for example, the range for updating map information.

[0317] In the content delivery system ex100, the client can receive, decode, and reproduce encoded information transmitted by the user in real time. (Streaming of individual content)

[0318] In the content delivery system ex100, in addition to high-quality video, long content distributed via video distribution entities, low-quality unicast or multicast streaming, and short content from individuals are also possible. Such content from individuals is likely to further increase in popularity. The server can perform editing processing on the content before encoding to optimize individual content. For example, this processing can be achieved using the following configuration.

[0319] In real-time, while capturing video or image content, or after content has been captured and accumulated, the server performs identification processing based on the raw or encoded data, such as capture error handling, scene search processing, meaning analysis, and / or object detection processing. Then, based on the results of the identification processing, the server—when prompted or automatically—edits the content. Examples of such editing include: corrections such as focus and / or motion blur correction; removal of low-priority scenes, such as scenes that are less bright or out of focus compared to other images; object edge adjustment; and tone adjustment. The server encodes the edited data based on the editing results. It is well known that excessively long videos tend to receive less view. Therefore, in order to keep the content within a specific length scaled with the length of the original video, in addition to the low-priority scenes mentioned above, the server may also automatically trim scenes with low motion based on the image processing results. Alternatively, the server may generate and encode a video summary based on the analysis of scene meaning.

[0320] Individual instances of content that may infringe on copyright, moral rights, or portrait rights may exist. Such instances can lead to disadvantageous situations for the creator, such as when the content is shared beyond the scope intended by the creator. Therefore, before encoding, the server can, for example, edit the image to blur, for instance, the faces of people around the screen or the interior of a house. Furthermore, the server can be configured to recognize faces of people other than registered individuals in the image to be encoded, and when such faces appear in the image, a mosaic filter can be applied to, for example, the faces of people. Alternatively, as pre-processing or post-processing for encoding, the user can specify areas including images of people or background areas to be processed for copyright reasons. The server can process the specified area by, for example, replacing the specified area with a different image or blurring the specified area. If the area includes a person, that person can be tracked in animation, and the person's head area can be replaced with another image as the person moves.

[0321] Because of the demand for real-time viewing of personally created content (which tends to be smaller in size), the decoder first receives the base layer as the highest priority and performs decoding and playback, although this operation may vary depending on bandwidth. When content is played back two or more times, such as when the decoder receives the enhancement layer during decoding and playback of the base layer and loops playback, the decoder can reproduce high-quality video including the enhancement layer. If the stream is encoded using this adjustable encoding, the video may be of low quality when unselected or at the beginning of the video, but the video provides an experience where the video quality of the stream is gradually enhanced in an intelligent manner. This is not limited to adjustable encoding only; the same experience can be provided by combining a single stream from a low-quality stream used for the first playback with a second stream using the encoding of the first stream as a reference. (Other implementation and application examples)

[0322] Encoding and decoding can be performed by a large-scale integration circuitry (LSI) ex500 (see Figure 26) typically included in each terminal. The LSI ex500 can be a single chip or a combination of multiple chips. Software for encoding and decoding animations can be integrated into a type of recording medium (such as a CD-ROM, flexible disk, or hard disk) that can be read by, for example, a computer ex111, and the encoding and decoding can be performed using software. Furthermore, when the smartphone ex115 is equipped with a camera, video data acquired by the camera can be transmitted. In this case, the video data is encoded using the LSI ex500 included in the smartphone ex115.

[0323] It should be noted that the LSI ex500 can be configured to download and launch an application. In this case, the terminal first determines whether it is compatible with the encoding scheme used to encode the content, or whether it can perform a specific service. When the terminal is incompatible with the content's encoding scheme, or when the terminal cannot perform a specific service, the terminal first downloads the codec or application software and then obtains and reproduces the content.

[0324] Besides the example of the content delivery system ex100 using the Internet ex101, the animation encoder (video encoder) or animation decoder (video decoder) described in at least the above embodiments can be implemented in a digital broadcasting system. The same encoding and decoding processes can be applied to transmit and receive broadcast radio waves superimposed with multiplexed audio and video data using, for example, satellite, even though this is adapted for multicast; however, unicast is easier for the content delivery system ex100. (Hardware configuration)

[0325] Figure 31 illustrates further details of the smartphone ex115 shown in Figure 26. Figure 32 illustrates a configuration example of the smartphone ex115. The smartphone ex115 includes an antenna ex450 for transmitting radio waves to and receiving radio waves from the base station ex110, a camera ex465 capable of capturing video and still images, and a display ex458 for displaying decoded data (such as video captured by the camera ex465 and video received by the antenna ex450). The smartphone ex115 further includes: a user interface ex466, such as a touch panel; an audio output unit ex457, such as a speaker for outputting voice or other audio signals; an audio input unit ex456, such as a microphone for audio input; a memory ex467 capable of storing decoded data, such as captured video or still images, recorded audio, received video or still images and emails, and decoded data; and a slot ex464, which is the interface for a SIM ex468 authorized to access the network and various data. It should be noted that external memory can be used instead of the memory ex467.

[0326] The main controller ex460 (which fully controls the display ex458 and the user interface ex466), power supply circuit ex461, user interface input controller ex462, video signal processor ex455, camera interface ex463, display controller ex459, modulator / demodulator ex452, multiplexer / demultiplexer ex453, audio signal processor ex454, slot ex464, and memory ex467 are connected via bus ex470.

[0327] When the user turns on the power button of the power supply circuit ex461, the smartphone ex115 is powered on and becomes operable, and all components are supplied with power from the battery pack.

[0328] The smartphone ex115 performs processing for tasks such as making calls and transmitting data based on control executed by the main controller ex460, which includes a CPU, ROM, and RAM. When a call is made, the audio signal recorded by the audio input unit ex456 is converted into a digital audio signal by the audio signal processor ex454. Spread spectrum processing is applied to the digital audio signal using a modulator / demodulator ex452 and a digital-to-analog converter. Frequency conversion is applied by the transmitter / receiver ex451, and the received signal is transmitted via the antenna ex450. Received data is amplified, frequency-converted, and analog-to-digital converted by the modulator / demodulator ex452, and inverse spread spectrum processing is performed. It is then converted into an analog audio signal by the audio signal processor ex454 and output from the audio output unit ex457. In data transmission mode, for example, text, still images, or video data are transmitted via the main controller ex460 through the user interface input controller ex462 based on the operation of the user interface ex466. Similar transmission and reception processing is performed. In data transmission mode, when sending video, still images, or video and audio, the video signal processor ex455 compresses and encodes the video signal stored in memory ex467 or the video signal input from the camera ex465 using the animation encoding method described in the above embodiments, and transmits the encoded video data to the multiplexer / demultiplexer ex453. The audio signal processor ex454 encodes the audio signal recorded by the audio input unit ex456 while the camera ex465 is capturing video or still images, and transmits the encoded audio data to the multiplexer / demultiplexer ex453. The multiplexer / demultiplexer ex453 uses a predetermined scheme to multiplex encoded video and encoded audio data, uses a modulator / demodulator (modulator / demodulator circuit) ex452 and a transmitter / receiver ex451 to modulate and convert the data, and transmits the results via an antenna ex450.

[0329] When a video message attached to an email or chat, or a video message linked from a webpage, is received, for example, to decode multiplexed data received via antenna ex450, multiplexer / demultiplexer ex453 demultiplexes the multiplexed data to divide it into a bitstream of video data and a bitstream of audio data. Encoded video data is supplied to video signal processor ex455 via synchronization bus ex470, and encoded audio data is supplied to audio signal processor ex454 via synchronization bus ex470. Video signal processor ex455 decodes the video signal using an animation decoding method corresponding to the animation encoding method described in the above embodiments, and displays the video or still image included in the linked animation file on display ex458 via display controller ex459. Audio signal processor ex454 decodes the audio signal and outputs audio from audio output unit ex457. As live streaming becomes increasingly popular, there may be instances where audio reproduction is inappropriate due to varying user environments. Therefore, as an initial consideration, it is preferable to configure the system to reproduce only video data, i.e., not audio signals; audio may only be synchronized and reproduced when input is received, such as when a user selects video data.

[0330] Although the smartphone ex115 was used in the above example, three other implementations are conceivable: a transceiver terminal including both an encoder and a decoder; a transmitter terminal including only an encoder; and a receiver terminal including only a decoder. In the description of a digital broadcasting system, an example is given where multiplexed data, obtained as the result of multiplexing audio data, is received or transmitted. However, multiplexed data can be video data multiplexed with data other than audio data, such as text data related to the video. Furthermore, video data that is not itself multiplexed data can be received or transmitted.

[0331] Although the main controller, including the CPU, is described as controlling the encoding or decoding program, various terminals often include a GPU. Therefore, it is acceptable to utilize the GPU's execution capabilities to simultaneously process large-area configurations via shared memory between the CPU and GPU, or memory managed to allow shared addresses. This makes it possible to reduce encoding time, maintain the real-time nature of the stream, and reduce latency. Specifically, processing related to motion estimation, deblocking filtering, sample adaptive offset (SAO), and transform / quantization can be efficiently performed by the GPU on an image-by-image basis (e.g., all simultaneously), replacing the CPU.

[0332] From the above discussion, it will be understood that the present invention can be embodied in various forms, including but not limited to the following:

[0333] Example 1, an image encoder comprising: circuitry; and a memory coupled to the circuitry; wherein, during operation, the circuitry determines whether inter-frame prediction is applied to a current block; when inter-frame prediction is applied to the current block, performing a segmentation procedure along a boundary between a first segmentation region and a second segmentation region in the current block, the first segmentation region having a non-rectangular shape, the segmentation procedure comprising: selecting a first motion vector for the first segmentation region from a first set of motion vector candidates; using the first motion vector to calculate a first value of a set of pixels of the first segmentation region along the boundary; using a second motion vector for the second segmentation region to calculate a second value of the set of pixels; weighting the first value and the second value; and encoding the first segmentation region using the weighted first value and the weighted second value; and when inter-frame prediction is not applied to the current block, the circuitry disables the segmentation procedure during operation.

[0334] Example 2, an image decoder, comprising: circuitry; and a memory coupled to the circuitry; wherein, during operation, the circuitry determines whether inter-frame prediction is applied to a current block; when inter-frame prediction is applied to the current block, performing a segmentation procedure along a boundary between a first segmentation region and a second segmentation region in the current block, the first segmentation region having a non-rectangular shape, the segmentation procedure comprising: selecting a first motion vector for the first segmentation region from a first set of motion vector candidates; using the first motion vector to calculate a first value of a set of pixels of the first segmentation region along the boundary; using a second motion vector for the second segmentation region to calculate a second value of the set of pixels; weighting the first value and the second value; and decoding the first segmentation region using the weighted first value and the weighted second value; and when inter-frame prediction is not applied to the current block, the circuitry disables the segmentation procedure during operation.

[0335] Example 3: A non-transitory computer-readable medium storing a bitstream including information in which a decoder performs a segmentation procedure along a boundary between a first segmentation and a second segmentation in a current block, the first segmentation having a non-rectangular shape, the information indicating a prediction pattern for the current block, wherein: a first motion vector for the first segmentation is selected from a first set of motion vector candidates; a first value of a set of pixels in the first segmentation along the boundary is calculated using the first motion vector; a second value of the set of pixels is calculated using a second motion vector for the second segmentation; and a third value of the set of pixels is calculated by weighting the first and second values, wherein the decoder deactivates the segmentation procedure when inter-frame prediction is not applied to the current block.

[0336] 100: Encoder 102: Splitter 104: Subtractor 106: Converter 108: Quantizer 110: Entropy Encoder 112, 204: Dequantizer 114, 206: Inverse converter 116, 208: Adders 118, 210: Block memory 120, 212: Loop filters 122, 214: Frame Memory 124, 216: In-frame predictors 126, 218: Inter-frame predictor 128, 220: Predictive Controller Blocks 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23 Pred, Pred_L, Pred_U: Predicted Imagery MV: Reference Image MV_L, MV_U, V0, V1, MV0, MV1: Motion vectors Ref0, Ref1: Reference images TD0, TD1, τ0, τ1: Time distance 200: Decoder 202: Entropy Decoder S1001, S1002, S1003, S1004, S1005, S2001, S2002, S2003, S3001, S3002, S3003, S3004, S3005, S4001, S4002, S4003, S5001, S5002, S5003: Steps 2000: Procedure for Splitting Image Blocks 3000: Boundary Smoothing Procedure 4000, 5000: Method ex100: Content Provider System ex101: Internet ex102: Internet Service Provider ex103: Streaming Server ex104: Communication Network ex106, ex107, ex108, ex109, ex110: Base stations ex111: Computer ex112: Gaming Device ex113, ex465: Cameras ex114: Home Appliances ex115: Smartphone ex116: Satellite ex117: Airplane ex500: Large-Scale Integrated Circuit (LSI) ex450: Antenna ex451: Transmitter / Receiver ex452: Modulator / Demodulator ex453: Multiplexer / Demultiplexer ex454: Audio signal processor ex455: Video Signal Processor ex456: Audio input unit ex457: Audio Output Unit ex458: Monitor ex459: Display Controller ex460: Main controller ex461: Power supply circuit ex462: User Interface Input Controller ex463: Camera Interface ex464: slot ex466: User Interface ex467: Memory ex468:SIM ex470: Bus

Claims

1. An image encoder comprising: circuitry; and a memory coupled to the circuitry; wherein, during operation, the circuitry: calculates a first pixel value of the first segment in a current block using a first motion vector for a first segment; calculates a second pixel value of the second segment in the current block using a second motion vector for a second segment; calculates a third value of a group of pixels between the first and second segment using the first motion vector; calculates a fourth value of the group of pixels using the second motion vector; weights the third and fourth values ​​of the group of pixels; and encodes the current block using the weighted third and fourth values, wherein, Only the third and fourth values ​​of the pixel group are weighted for the current block, wherein the number of pixels in one column of the pixel group is greater than two, and the weight of the third values ​​of the pixel group being greater than two increases along the column in a first direction, and the weight of the fourth values ​​of the pixel group being greater than two decreases along the column in the first direction.

2. An image encoder comprising: circuitry; and a memory coupled to the circuitry; wherein, during operation, the circuitry: calculates a first pixel value of the first segment in a current block using a first motion vector for a first segment; calculates a second pixel value of the second segment in the current block using a second motion vector for a second segment; calculates a third value of a group of pixels between the first and second segment using the first motion vector; calculates a fourth value of the group of pixels using the second motion vector; weights the third and fourth values ​​of the group of pixels; and encodes the current block using the weighted third and fourth values, wherein, Only the third and fourth values ​​of the group of pixels are weighted for the current block, wherein the number of pixels in one row of the group of pixels is greater than two, and the weight of the third values ​​of the group of pixels being greater than two increases along the row in a first direction, and the weight of the fourth values ​​of the group of pixels being greater than two decreases along the row in the first direction.

3. An image decoder comprising: circuitry; and a memory coupled to the circuitry; wherein, during operation, the circuitry: calculates a first pixel value of the first segment in a current block using a first motion vector for a first segment; calculates a second pixel value of the second segment in the current block using a second motion vector for a second segment; calculates a third value of a group of pixels between the first and second segments using the first motion vector; calculates a fourth value of the group of pixels using the second motion vector; weights the third and fourth values ​​of the group of pixels; and decodes the current block using the weighted third and fourth values, wherein, Only the third and fourth values ​​of the pixel group are weighted for the current block, wherein the number of pixels in one column of the pixel group is greater than two, and the weight of the third values ​​of the pixel group being greater than two increases along the column in a first direction, and the weight of the fourth values ​​of the pixel group being greater than two decreases along the column in the first direction.

4. An image decoder comprising: circuitry; and a memory coupled to the circuitry; wherein, during operation, the circuitry: calculates a first pixel value of the first segment in a current block using a first motion vector for a first segment; calculates a second pixel value of the second segment in the current block using a second motion vector for a second segment; calculates a third value of a group of pixels between the first and second segments using the first motion vector; calculates a fourth value of the group of pixels using the second motion vector; weights the third and fourth values ​​of the group of pixels; and decodes the current block using the weighted third and fourth values, wherein, Only the third and fourth values ​​of the group of pixels are weighted for the current block, wherein the number of pixels in one row of the group of pixels is greater than two, and the weight of the third values ​​of the group of pixels being greater than two increases along the row in a first direction, and the weight of the fourth values ​​of the group of pixels being greater than two decreases along the row in the first direction.

5. A non-transitory computer-readable medium storing a bitstream including information, wherein a decoder performs a segmentation procedure based on the information, in which: a first pixel value of a first segment in a current block is calculated using a first motion vector for the first segment; a second pixel value of a second segment in the current block is calculated using a second motion vector for the second segment; a third value of a group of pixels between the first and second segments is calculated using the first motion vector; a fourth value of the group of pixels is calculated using the second motion vector; the third and fourth values ​​of the group of pixels are weighted; and the current block is decoded using the weighted third and fourth values, wherein... Only the third and fourth values ​​of the pixel group are weighted for the current block, wherein the number of pixels in one column of the pixel group is greater than two, and the weight of the third values ​​of the pixel group being greater than two increases along the column in a first direction, and the weight of the fourth values ​​of the pixel group being greater than two decreases along the column in the first direction.

6. A non-transitory computer-readable medium storing a bitstream including information, wherein a decoder performs a segmentation procedure based on the information, in which: a first pixel value of a first segment in a current block is calculated using a first motion vector for the first segment; a second pixel value of a second segment in the current block is calculated using a second motion vector for the second segment; a third value of a group of pixels between the first and second segments is calculated using the first motion vector; a fourth value of the group of pixels is calculated using the second motion vector; the third and fourth values ​​of the group of pixels are weighted; and the current block is decoded using the weighted third and fourth values, wherein... Only the third and fourth values ​​of the group of pixels are weighted for the current block, wherein the number of pixels in one row of the group of pixels is greater than two, and the weight of the third values ​​of the group of pixels being greater than two increases along the row in a first direction, and the weight of the fourth values ​​of the group of pixels being greater than two decreases along the row in the first direction.

Images