Encoding device, decoding device, and bitstream transmitter

Abstract

To provide a system and a method for video coding.SOLUTION: An encoding device generates a predictive block, uses the predictive block to generate a gradient block of the same size as the predictive block, uses the gradient block to generate a predicted image, and encodes a current block based on the predicted image. The generation of the gradient block includes a process of calculating a gradient value indicating the difference between the value of a right-side sample adjacent to the right of the position of a target sample included in the predictive block and the value of a left-side sample adjacent to the left of the target sample. A first gradient value at the left end of the gradient block is calculated using the value of a first sample at the same position as the first gradient value in the predictive block as the left-side sample, and a second gradient value adjacent to the right of the position of the first gradient value is calculated using the value of the first sample as the left-side sample.SELECTED DRAWING: Figure 24

Description

[Technical Field]

[0001] This disclosure relates to video coding, and more particularly to video coding systems, components, and methods that perform interpredictive functions for predicting the current frame based on a reference frame. [Background technology]

[0002] The video coding standard known as HEVC (High-Efficiency Video Coding) has been standardized by JCT-VC (Joint Collaborative Team on Video Coding).

[0003] Video coding technology has advanced from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), MPEG-LA, H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Codec). With this advancement, there is a constant need to provide improvements and optimizations to video coding technology to handle the ever-increasing volume of digital video data across various applications. This disclosure relates to further advancements, improvements, and optimizations in video coding, particularly in inter-predictive functions that construct predictions of the current frame based on a reference frame. [Overview of the project] [Problems that the invention aims to solve]

[0004] There is a demand for reducing processing load and further improving compression efficiency in encoding and decoding technologies.

[0005] Accordingly, this disclosure provides an encoding device, a decoding device, an encoding method, and a decoding method that can achieve a reduction in processing load and further improvement in compression efficiency. [Means for solving the problem]

[0006] An encoding device according to one aspect of the present disclosure comprises a memory and a circuit connected to the memory, wherein the circuit, in operation, calculates a motion vector indicating a reference range within a reference picture, generates a prediction block by performing interpolation processing using the values ​​of samples contained in the reference picture based on the motion vector, generates a gradient block of the same size as the prediction block using the prediction block, generates a prediction image using the gradient block, encodes the current block based on the prediction image, and the generation of the gradient block involves a process of calculating a horizontal gradient value indicating the difference between the value of a right-side sample adjacent to the right position of the target sample contained in the prediction block and the value of a left-side sample adjacent to the left position of the target sample contained in the prediction block The process includes calculating a vertical gradient value that indicates the difference between the value of a lower sample adjacent to the target sample below and the value of an upper sample adjacent to the target sample above, wherein the first horizontal gradient value at the left end of the gradient block is calculated using the value of a first sample at the same position as the first horizontal gradient value in the prediction block as the left-side sample, the second horizontal gradient value adjacent to the right of the position of the first horizontal gradient value is calculated using the value of the first sample as the left-side sample, the first vertical gradient value at the top end of the gradient block is calculated using the value of a first sample at the same position as the first vertical gradient value in the prediction block as the upper-side sample, and the second vertical gradient value adjacent below the position of the first vertical gradient value is calculated using the value of the first sample as the upper-side sample.

[0007] An encoding device according to one aspect of the present disclosure comprises a memory and a circuit connected to the memory, wherein the circuit generates a prediction block by performing interpolation processing using the values ​​of a sample contained in a reference picture during operation, generates a gradient block of the same size as the prediction block using the prediction block, generates a prediction image using the gradient block, encodes a current block based on the prediction image, the generation of the gradient block includes a process of calculating a gradient value that indicates the difference between the value of a right-side sample adjacent to the right position of a target sample contained in the prediction block and the value of a left-side sample adjacent to the left position of the target sample, the first gradient value at the left end of the gradient block is calculated using the value of a first sample at the same position as the first gradient value in the prediction block as the left-side sample, and the second gradient value adjacent to the right position of the first gradient value is calculated using the value of the first sample as the left-side sample.

[0008] One aspect of the present disclosure is an encoding device for encoding a target block in a picture using interprediction, the encoding device comprising a processor and a memory, the processor using the memory to perform the steps of: obtaining two predicted images from two reference pictures by performing motion compensation using motion vectors corresponding to each of the two reference pictures; obtaining two gradient images from the two reference pictures corresponding to the two predicted images; deriving local motion detection values ​​using the two gradient images and two predicted images in a subblock obtained by dividing the target block; and generating a final predicted image of the target block using the local motion detection values ​​of the subblock, the two gradient images and two predicted images.

[0009] According to another aspect of the present disclosure, an image encoding device is provided comprising a circuit and a memory connected to the circuit. The circuit, in operation, includes at least a prediction process using a motion vector from another picture, the steps of: predicting a first block of prediction samples for a current block of the picture; padding the first block of prediction samples to form a second block of prediction samples that is larger than the first block; calculating at least a gradient using the second block of prediction samples; and encoding the current block using at least the calculated gradient.

[0010] According to another aspect of the present disclosure, an image encoding device is provided comprising a circuit and a memory connected to the circuit. The circuit, in operation, includes at least a prediction process using a motion vector from another picture, the steps of: predicting a first block of prediction samples for a current block of the picture; padding the first block of prediction samples to form a second block of prediction samples that is larger than the first block; performing interpolation using the second block of prediction samples; and encoding the current block using at least the block resulting from the interpolation process.

[0011] According to another aspect of the present disclosure, an image encoding device is provided comprising a circuit and a memory connected to the circuit. The circuit, in operation, includes at least a prediction process using a motion vector from another picture, the steps of: predicting a first block of prediction samples for a current block of the picture; padding a second block of prediction samples adjacent to the current block to form a third block of prediction samples; performing OBMC processing using at least the first and third blocks of prediction samples; and encoding the current block using at least the block resulting from the OBMC processing.

[0012] According to another aspect of the present disclosure, there is provided an image encoding apparatus including a circuit and a memory connected to the circuit. In operation, the circuit performs steps including at least prediction processing using a first motion vector from another picture, the steps comprising: predicting a first block of prediction samples for a current block of a picture; deriving a second motion vector for the current block by dynamic motion vector refreshing (DMVR) processing using at least the first motion vector; performing interpolation processing including padding processing on the current block using the second motion vector; and encoding the current block using at least the block resulting from the interpolation processing.

[0013] According to another aspect of the present disclosure, there is provided an image decoding apparatus including a circuit and a memory connected to the circuit. In operation, the circuit performs steps including at least prediction processing using a motion vector from another picture, the steps comprising: predicting a first block of prediction samples for a current block of a picture; padding the first block of prediction samples to form a second block of prediction samples larger than the first block; calculating at least a gradient using the second block of prediction samples; and decoding the current block using at least the calculated gradient.

[0014] According to another aspect of the present disclosure, there is provided an image decoding apparatus including a circuit and a memory connected to the circuit. In operation, the circuit performs steps including at least prediction processing using a motion vector from another picture, the steps comprising: predicting a first block of prediction samples for a current block of a picture; padding the first block of prediction samples to form a second block of prediction samples larger than the first block; performing interpolation processing using the second block of prediction samples; and decoding the current block using at least the block resulting from the interpolation processing.

[0015] According to another aspect of the present disclosure, there is provided an image decoding apparatus including a circuit and a memory connected to the circuit. In operation, the circuit performs steps including at least prediction processing using a motion vector from another picture, the steps comprising: predicting a first block of prediction samples for a current block of a picture; padding a second block of prediction samples adjacent to the current block to form a third block of prediction samples; performing OBMC processing using at least the first block and the third block of prediction samples; and decoding the current block using at least the block resulting from the OBMC processing.

[0016] According to another aspect of the present disclosure, there is provided an image decoding apparatus including a circuit and a memory connected to the circuit. In operation, the circuit performs steps including at least prediction processing using a first motion vector from another picture, the steps comprising: predicting a first block of prediction samples for a current block of a picture; deriving a second motion vector for the current block by dynamic motion vector refreshing (DMVR) processing using at least the first motion vector; performing interpolation processing including padding processing on the current block using the second motion vector; and decoding the current block using at least the block resulting from the interpolation processing.

[0017] According to another aspect of the present disclosure, there is provided an image encoding method. The image encoding method enables an image encoding apparatus to perform steps according to various aspects of the present disclosure as described herein.

[0018] According to another aspect of the present disclosure, there is provided an image decoding method. The image decoding method enables an image decoding apparatus to perform steps according to various aspects of the present disclosure as described herein.

[0019] These comprehensive and specific embodiments may be implemented using systems, methods, integrated circuits, computer programs, or media such as computer-readable CD-ROMs, or by a combination of systems, methods, integrated circuits, computer programs, and media. [Effects of the Invention]

[0020] This disclosure provides an encoding device, a decoding device, an encoding method, and a decoding method that can achieve a reduction in processing load and further improvement in compression efficiency.

[0021] Some embodiments of the present disclosure enable improved coding efficiency, simplified coding / decoding processes, accelerated coding / decoding speeds, and efficient selection of appropriate components / operations used in coding and decoding, such as appropriate filters, block sizes, motion vectors, reference pictures, and reference blocks.

[0022] Further benefits and advantages of the disclosed embodiments will become apparent from the specification and drawings. Benefits and / or advantages may be obtained individually by various embodiments and features of the specification and drawings, and it is not necessary to provide all of the various embodiments and features of the specification and drawings in order to obtain one or more such benefits and / or advantages.

[0023] Furthermore, comprehensive or specific embodiments may be implemented as systems, methods, integrated circuits, computer programs, storage media, or any combination thereof. [Brief explanation of the drawing]

[0024] In drawings, the same reference numeral indicates the same component. The size and relative position of components in drawings are not necessarily shown to scale. [Figure 1] Figure 1 is a block diagram showing the functional configuration of an encoding device according to an embodiment. [Figure 2]Figure 2 shows an example of block division. [Figure 3] Figure 3 is a table showing the transformation basis functions corresponding to each transformation type. [Figure 4A] Figure 4A shows an example of the filter shape used in an ALF (Adaptive Loop Filter). [Figure 4B] Figure 4B shows another example of the filter shape used in ALF. [Figure 4C] Figure 4C shows another example of the filter shape used in ALF. [Figure 5A] Figure 5A shows the 67 intra-prediction modes in intra-prediction. [Figure 5B] Figure 5B is a flowchart illustrating the overview of the predictive image correction process using OBMC (overlapped block motion compensation). [Figure 5C] Figure 5C is a conceptual diagram illustrating the overview of the predictive image correction process using OBMC processing. [Figure 5D] Figure 5D shows an example of FRUC (frame rate up-conversion). [Figure 6] Figure 6 illustrates pattern matching (bilateral matching) between two blocks along a motion trajectory. [Figure 7] Figure 7 illustrates pattern matching (template matching) between a template in the current picture and a block in the referenced picture. [Figure 8] Figure 8 is a diagram illustrating a model that assumes uniform linear motion. [Figure 9A] Figure 9A is a diagram illustrating the derivation of subblock-level motion vectors based on the motion vectors of multiple adjacent blocks. [Figure 9B] Figure 9B is a diagram illustrating the overview of the motion vector derivation process using merge mode. [Figure 9C]Figure 9C is a conceptual diagram illustrating the overview of DMVR (dynamic motion vector refreshing) processing. [Figure 9D] Figure 9D is a diagram illustrating the outline of a predictive image generation method using brightness correction processing by LIC (local illumination compensation). [Figure 10] Figure 10 is a block diagram showing the functional configuration of a decoding device according to an embodiment. [Figure 11] Figure 11 is a flowchart of the interpretation process according to another embodiment. [Figure 12] Figure 12 is a conceptual diagram used to explain the interpretation prediction according to the embodiment shown in Figure 11. [Figure 13] Figure 13 is a conceptual diagram used to illustrate an example of the reference range of the gradient filter and motion compensation filter according to the embodiment shown in Figure 11. [Figure 14] Figure 14 is a conceptual diagram used to illustrate an example of the reference range of a motion compensation filter according to Modification 1 of the embodiment shown in Figure 11. [Figure 15] Figure 15 is a conceptual diagram used to illustrate an example of the reference range of a gradient filter according to Modification 1 of the embodiment shown in Figure 11. [Figure 16] Figure 16 shows an example of a pixel pattern referenced by deriving local motion detection values ​​according to a modified example 2 of the embodiment shown in Figure 11. [Figure 17] Figure 17 is a flowchart showing an example of an image coding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture. [Figure 18A] Figure 18A is a conceptual diagram illustrating an example of picture order counting for a picture and another picture. In Figure 18A, the picture may be the current picture, and the other picture may be the first picture or the second picture. [Figure 18B]Figure 18B is a conceptual diagram showing another example of picture order counting for a picture and another picture. [Figure 18C] Figure 18C is a conceptual diagram showing yet another example of picture order counts for a picture and another picture. [Figure 19A] Figure 19A shows an example of a padding direction for padding blocks of predicted samples related to an example of an image coding / decoding method as shown in Figure 17. [Figure 19B] Figure 19B shows another example of a padding direction for padding blocks of predicted samples related to an example of an image coding / decoding method as shown in Figure 17. [Figure 20A] Figure 20A shows an example of a process for padding blocks of predicted samples related to an example of an image encoding / decoding method as shown in Figure 17. [Figure 20B] Figure 20B shows another example of the process of padding blocks of predicted samples related to the example of the image coding / decoding method shown in Figure 17. [Figure 20C] Figure 20C shows another example of the process of padding blocks of predicted samples related to the example of the image coding / decoding method shown in Figure 17. [Figure 20D] Figure 20D shows yet another example of the process of padding blocks of predicted samples related to the example of the image coding / decoding method shown in Figure 17. [Figure 21A] Figure 21A shows an example of a gradient filter for a block. [Figure 21B] Figure 21B shows examples of multiple gradient filters for a block. [Figure 21C] Figure 21C shows yet another example of a gradient filter on a block. [Figure 21D] Figure 21D shows yet another example of a gradient filter on a block. [Figure 22] Figure 22 is a flowchart showing an embodiment of an image encoding / decoding method related to an example as shown in Figure 17. [Figure 23]Figure 23 is a conceptual diagram showing an embodiment of the image encoding / decoding method as shown in Figure 22. [Figure 24] Figure 24 is a flowchart showing another embodiment of the image encoding / decoding method related to the example shown in Figure 17. [Figure 25] Figure 25 is a conceptual diagram showing an embodiment of the image encoding / decoding method as shown in Figure 24. [Figure 26] Figure 26 shows an example of a block generated by padding in an embodiment of the image encoding / decoding method shown in Figures 24 and 25. [Figure 27A] Figure 27A is a flowchart showing another alternative example of an image coding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture. [Figure 27B] Figure 27B is a flowchart showing another alternative example of an image coding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture. [Figure 27C] Figure 27C is a flowchart showing yet another alternative example of an image coding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture. [Figure 28] Figure 28 is a conceptual diagram showing an embodiment of the image encoding / decoding method as shown in Figure 27A. [Figure 29] Figure 29 is a conceptual diagram showing an embodiment of the image encoding / decoding method as shown in Figure 27B. [Figure 30] Figure 30 is a conceptual diagram showing an embodiment of the image encoding / decoding method as shown in Figure 27C. [Figure 31] Figure 31 shows an example of adjacent blocks to the current block. [Figure 32A] Figure 32A shows an example of the process of padding blocks of predicted samples related to the image coding / decoding methods shown in Figures 27A, 27B, and 27C. [Figure 32B]Figure 32B shows another example of the process of padding blocks of predicted samples related to the example image coding / decoding method shown in Figures 27A, 27B, and 27C. [Figure 32C] Figure 32C shows another example of the process of padding blocks of predicted samples related to the example image coding / decoding method shown in Figures 27A, 27B, and 27C. [Figure 33A] Figure 33A shows an example of padding direction for padding blocks related to the example image encoding / decoding method shown in Figures 17, 22, 24, 27A, 27B, and 27C. [Figure 33B] Figure 33B shows another example of a padding direction for padding blocks related to the example image encoding / decoding method shown in Figures 17, 22, 24, 27A, 27B, and 27C. [Figure 33C] Figure 33C shows another example of a padding direction for padding blocks related to the example image encoding / decoding method shown in Figures 17, 22, 24, 27A, 27B, and 27C. [Figure 33D] Figure 33D shows yet another example of a padding direction for padding blocks related to the example of an image encoding / decoding method shown in Figures 17, 22, 24, 27A, 27B, and 27C. [Figure 34] Figure 34 shows alternative examples of prediction sample blocks related to the image coding / decoding methods shown in Figures 17, 22, 24, 27A, 27B, and 27C. In these alternative examples, the prediction sample blocks are non-rectangular in shape. [Figure 35] Figure 35 is a flowchart showing another alternative example of an image coding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture. [Figure 36] Figure 36 is a conceptual diagram showing an embodiment of the image encoding / decoding method as shown in Figure 35. [Figure 37A]Figure 37A shows an example of a process for padding blocks of predicted samples according to a second motion vector, relating to an example of an image encoding / decoding method as shown in Figure 35. [Figure 37B] Figure 37B shows another example of the process of padding blocks of predicted samples according to a second motion vector, relating to an example of an image encoding / decoding method as shown in Figure 35. [Figure 38] Figure 38 is a flowchart illustrating yet another alternative example of an image encoding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture. The flowchart in Figure 38 is similar to the flowchart in Figure 35, except that the DMVR (dynamic motion vector refreshing) process in step 3804 further includes a padding process. [Figure 39] Figure 39 is a conceptual diagram showing an embodiment of the image encoding / decoding method as shown in Figure 38. [Figure 40] Figure 40 is a block diagram showing the overall configuration of a content supply system that realizes a content distribution service. [Figure 41] Figure 41 is a conceptual diagram showing an example of an encoding structure during scalable encoding. [Figure 42] Figure 42 is a conceptual diagram showing an example of an encoding structure during scalable encoding. [Figure 43] Figure 43 is a conceptual diagram showing an example of a web page display screen. [Figure 44] Figure 44 is a conceptual diagram showing an example of a web page display screen. [Figure 45] Figure 45 is a block diagram showing an example of a smartphone. [Figure 46] Figure 46 is a block diagram showing an example of a smartphone configuration. [Modes for carrying out the invention]

[0025] The embodiments will be described below in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, relationships and sequences of steps, etc., shown in the following embodiments are examples and are not intended to limit the scope of the claims. Therefore, components disclosed in the following embodiments but not described in the independent claims defining the broadest inventive concept may be understood as any component.

[0026] Embodiments of encoding and decoding devices are described below. These embodiments are examples of encoding and decoding devices to which the processes and / or configurations described in each aspect of this disclosure can be applied. The processes and / or configurations can also be implemented in encoding and decoding devices different from those in the embodiments. For example, with respect to the processes and / or configurations applicable to the embodiments, one of the following may be implemented:

[0027] (1) Any of the multiple components of the encoding or decoding device of the embodiments described in each aspect of the Disclosure may be replaced or combined with other components described in any of the aspects of the Disclosure.

[0028] (2) In the encoding or decoding device of the embodiment, any modifications such as addition, replacement, or deletion of functions or processes performed by some of the multiple components of the encoding or decoding device may be made. For example, any of the functions or processes may be replaced or combined with other functions or processes described in any of the embodiments of this disclosure.

[0029] (3) In the methods performed by the encoding or decoding apparatus of the embodiment, any modifications, such as additions, replacements, and deletions, may be made to some of the processes included in the method. For example, any of the processes in the method may be replaced with or combined with other processes described in any of the embodiments of this disclosure.

[0030] (4) Some of the multiple components constituting the encoding or decoding device of the embodiment may be combined with components described in any of the embodiments of this disclosure, or with components that have some of the functions described in any of the embodiments of this disclosure, or with components that perform some of the processing performed by the components described in any of the embodiments of this disclosure.

[0031] (5) Components that provide some of the functions of the encoding or decoding device of the embodiment, or components that perform some of the processing of the encoding or decoding device of the embodiment, may be combined with or replaced with components described in any of the aspects of the disclosure and components that provide some of the functions described in any of the aspects of the disclosure, or components that perform some of the processing described in any of the aspects of the disclosure.

[0032] (6) In a method performed by an encoding or decoding device of an embodiment, any of the processes included in the method may be replaced or combined with any of the processes described in any of the embodiments of the present disclosure.

[0033] (7) Some of the processes included in the methods performed by the encoding or decoding device of the embodiment may be combined with the processes described in any of the embodiments of this disclosure.

[0034] (8) The methods of carrying out the processes and / or configurations described in each aspect of the present disclosure are not limited to the encoding or decoding devices of the embodiments. For example, the processes and / or configurations may be carried out in devices used for purposes other than the video encoding or video decoding disclosed in the embodiments.

[0035] [Encoding device] First, an overview of the encoding device according to the embodiment will be described. Figure 1 is a block diagram showing the functional configuration of the encoding device 100 according to the embodiment. The encoding device 100 is a video encoding device that encodes video in block units.

[0036] As shown in Figure 1, the encoding device 100 is a device that encodes an image in block units and comprises a division unit 102, a subtraction unit 104, a transformation unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transformation unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, an intra prediction unit 124, an inter prediction unit 126, and a prediction control unit 128.

[0037] The encoding device 100 can be implemented, for example, by a general-purpose processor and memory. In this case, when a software program stored in memory is executed by the processor, the processor functions as a splitting unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a loop filter unit 120, an intra prediction unit 124, an inter prediction unit 126, and a prediction control unit 128. Alternatively, the encoding device 100 may be implemented as one or more dedicated electronic circuits corresponding to the splitting unit 102, a subtraction unit 104, a conversion unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse conversion unit 114, an addition unit 116, a loop filter unit 120, an intra prediction unit 124, an inter prediction unit 126, and a prediction control unit 128.

[0038] The following describes each component included in the encoding device 100.

[0039] [Divided part] The splitting unit 102 divides each picture contained in the input video into multiple blocks and outputs each block to the subtraction unit 104. For example, the splitting unit 102 first divides the picture into blocks of a fixed size (e.g., 128x128). These fixed-size blocks are sometimes called coding tree units (CTUs). Then, based on recursive quadtree and / or binary tree block partitioning, the splitting unit 102 divides each of the fixed-size blocks into blocks of a variable size (e.g., 64x64 or less). These variable-size blocks are sometimes called coding units (CUs), prediction units (PUs), or transformation units (TUs). In various processing examples, CUs, PUs, and TUs do not need to be distinguished, and some or all of the blocks in the picture may be processing units of CUs, PUs, or TUs.

[0040] Figure 2 shows an example of block partitioning in the embodiment. In Figure 2, solid lines represent block boundaries due to quadtree block partitioning, and dashed lines represent block boundaries due to binary tree block partitioning.

[0041] Here, block 10 is a 128x128 pixel square block (128x128 block). This 128x128 block 10 is first divided into four 64x64 square blocks (quadtree block partitioning).

[0042] The top-left 64x64 block is further divided vertically into two rectangular 32x64 blocks, and the left 32x64 block is further divided vertically into two rectangular 16x64 blocks (binary tree block partitioning). As a result, the top-left 64x64 block is divided into two 16x64 blocks 11 and 12 and a 32x64 block 13.

[0043] The 64x64 block in the upper right is horizontally divided into two rectangular 64x32 blocks, 14 and 15 (binary tree block division).

[0044] The bottom-left 64x64 block is divided into four square 32x32 blocks (quadrutree block division). Of the four 32x32 blocks, the top-left and bottom-right blocks are further divided. The top-left 32x32 block is vertically divided into two rectangular 16x32 blocks, and the rightmost 16x32 block is further horizontally divided into two 16x16 blocks (binary tree block division). The bottom-right 32x32 block is horizontally divided into two 32x16 blocks (binary tree block division). As a result, the bottom-left 64x64 block is divided into 16x32 block 16, two 16x16 blocks 17 and 18, two 32x32 blocks 19 and 20, and two 32x16 blocks 21 and 22.

[0045] The 64x64 block 23 in the bottom right will not be divided.

[0046] As described above, in Figure 2, block 10 is divided into 13 variable-sized blocks 11-23 based on recursive quad-tree and binary tree block partitioning. Such partitioning is sometimes called QTBT (quad-tree plus binary tree) partitioning.

[0047] In Figure 2, one block was divided into four or two blocks (quadrutree or binary tree block partitioning), but partitioning is not limited to these. For example, one block may be divided into three blocks (ternary tree block partitioning). Partitioning that includes such ternary tree block partitioning is sometimes called MBT (multi-type tree) partitioning.

[0048] [Subtraction Unit] The subtraction unit 104 subtracts the predicted signal (predicted samples input from the prediction control unit 128, shown below) from the original signal (original sample) in block units that are input from the division unit 102 and divided by the division unit 102. In other words, the subtraction unit 104 calculates the prediction error (also called residual) of the block to be encoded (hereinafter referred to as the current block). The subtraction unit 104 then outputs the calculated prediction error (residual) to the conversion unit 106.

[0049] The source signal is the input signal to the encoding device 100, and is a signal representing the image of each picture that makes up the moving image (for example, a luminance (luma) signal and two chroma (chroma) signals). In the following, the signal representing the image may also be called a sample.

[0050] [Conversion section] The conversion unit 106 converts the prediction error in the spatial domain into conversion coefficients in the frequency domain and outputs the conversion coefficients to the quantization unit 108. Specifically, the conversion unit 106 performs a predetermined discrete cosine transform (DCT) or discrete sine transform (DST) on the prediction error in the spatial domain, for example.

[0051] The transformation unit 106 may also adaptively select a transformation type from among several transformation types and use a transformation basis function corresponding to the selected transformation type to convert the prediction error into transformation coefficients. Such a transformation is sometimes called an EMT (explicit multiple core transform) or an AMT (adaptive multiple transform).

[0052] Multiple transformation types include, for example, DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII. Figure 3 is a table showing the transformation basis functions corresponding to each transformation type. In Figure 3, N represents the number of input pixels. The selection of a transformation type from among these multiple transformation types may depend, for example, on the type of prediction (intra-prediction and inter-prediction) or on the intra-prediction mode.

[0053] Information indicating whether or not to apply EMT or AMT (e.g., called an EMT flag or AMT flag) and information indicating the selected conversion type are typically signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (e.g., bit sequence level, picture level, slice level, tile level, or CTU level).

[0054] Furthermore, the transformation unit 106 may retransform the transformation coefficients (transformation results). Such retransformation is sometimes called AST (adaptive secondary transform) or NSST (non-separable secondary transform). For example, the transformation unit 106 performs retransformation for each subblock (e.g., 4x4 subblock) contained in the block of transformation coefficients corresponding to the intra-prediction error. Information indicating whether or not to apply NSST and information regarding the transformation matrix used for NSST are usually signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (e.g., sequence level, picture level, slice level, tile level, or CTU level).

[0055] The transformation unit 106 may be subjected to either a separable transformation or a non-separable transformation. A separable transformation is a method in which the input is separated into directions equal to the number of dimensions and transformed multiple times, while a non-separable transformation is a method in which, when the input is multidimensional, two or more dimensions are treated as one dimension and transformed together.

[0056] For example, one example of a non-separable transformation is to treat a 4x4 block as a single array with 16 elements and then perform the transformation on that array using a 16x16 transformation matrix.

[0057] Another example of a non-separable transformation is one in which a 4x4 input block is treated as a single array with 16 elements, and then multiple Givens rotations are performed on that array (for example, the Hypercube Givens Transform).

[0058] [Quantization section] The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106. Specifically, the quantization unit 108 scans the conversion coefficients of the current block in a predetermined scanning order and quantizes the conversion coefficients based on the quantization parameter (QP) corresponding to the scanned conversion coefficients. The quantization unit 108 then outputs the quantized conversion coefficients of the current block (hereinafter referred to as quantization coefficients) to the entropy coding unit 110 and the inverse quantization unit 112.

[0059] A predetermined scan order is the order for quantization / inverse quantization of the transformation coefficients. For example, a predetermined scan order is defined as ascending frequency (from low to high frequency) or descending frequency (from high to low frequency).

[0060] The quantization parameter (QP) is a parameter that defines the quantization step (quantization width). For example, if the value of the quantization parameter increases, the quantization step also increases. In other words, if the value of the quantization parameter increases, the quantization error increases.

[0061] [Entropy coding unit] The entropy coding unit 110 generates an encoded signal (encoded bitstream) based on the quantization coefficients input from the quantization unit 108. Specifically, for example, the entropy coding unit 110 binarizes the quantization coefficients, arithmetically encodes the binary signal, and outputs a compressed bitstream or sequence.

[0062] [Dequantization section] The inverse quantization unit 112 inversely quantizes the quantization coefficients input from the quantization unit 108. Specifically, the inverse quantization unit 112 inversely quantizes the quantization coefficients of the current block in a predetermined scanning order. Then, the inverse quantization unit 112 outputs the inversely quantized conversion coefficients of the current block to the inverse conversion unit 114.

[0063] [Inverse Transformation Section] The inverse transform unit 114 restores the prediction error (residual) by performing an inverse transform on the transformation coefficients input from the inverse quantization unit 112. Specifically, the inverse transform unit 114 restores the prediction error of the current block by performing an inverse transform on the transformation coefficients corresponding to the transformation by the transformation unit 106. The inverse transform unit 114 then outputs the restored prediction error to the adder unit 116.

[0064] Furthermore, the recovered prediction error usually does not match the prediction error calculated by the subtraction unit 104 because information is typically lost due to quantization. In other words, the recovered prediction error usually includes quantization errors.

[0065] [Addition section] The adder 116 reconstructs the current block by adding the prediction error input from the inverse transformer 114 and the prediction sample input from the prediction control unit 128. The adder 116 then outputs the reconstructed block to the block memory 118 and the loop filter unit 120. The reconstructed block is sometimes called the local decoded block.

[0066] [Block memory] The block memory 118 is a storage unit for storing blocks within the picture to be encoded ("current picture") that are referenced in intra prediction. Specifically, the block memory 118 stores the reconstructed blocks output from the adder 116.

[0067] [Loop Filter Section] The loop filter unit 120 applies a loop filter to the block reconstructed by the adder unit 116 and outputs the filtered reconstructed block to the frame memory 122. A loop filter is a filter used within the encoding loop (in-loop filter), and includes, for example, a deblocking filter (DF), sample adaptive offset (SAO), and adaptive loop filter (ALF).

[0068] In ALF, a least-squares error filter is applied to remove coding distortion. For example, for each 2x2 subblock within the current block, one filter selected from several filters is applied based on the direction and activity of the local gradient.

[0069] Specifically, first, subblocks (e.g., 2x2 subblocks) are classified into multiple classes (e.g., 15 or 25 classes). The classification of subblocks is based on the direction and activity of the gradient. For example, a classification value C (e.g., C = 5D + A) is calculated using the gradient direction value D (e.g., 0-2 or 0-4) and the gradient activity value A (e.g., 0-4). Then, based on the classification value C, the subblocks are classified into multiple classes.

[0070] The gradient direction value D is derived, for example, by comparing gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). The gradient activation value A is derived, for example, by adding the gradients in multiple directions and quantizing the sum.

[0071] Based on the results of this classification, a filter for the subblock is determined from among multiple filters.

[0072] For example, a circularly symmetric shape is used as the filter shape in ALF. Figures 4A to 4C show several examples of filter shapes used in ALF. Figure 4A shows a 5x5 diamond-shaped filter, Figure 4B shows a 7x7 diamond-shaped filter, and Figure 4C shows a 9x9 diamond-shaped filter. Information indicating the filter shape is usually signaled at the picture level. However, the signaling of information indicating the filter shape is not limited to the picture level and may be at other levels (e.g., sequence level, slice level, tile level, CTU level, or CU level).

[0073] The ALF (Automatic Laser Level) on / off status may be determined at the picture level or the CU (Camera Unit) level. For example, the decision to apply ALF to luminance may be made at the CU level, while the decision to apply ALF to color difference may be made at the picture level. Information indicating whether ALF is on or off is usually signaled at the picture level or the CU level. However, the signaling of information indicating whether ALF is on or off is not limited to the picture level or the CU level, but may be at other levels (e.g., sequence level, slice level, tile level, or CTU level).

[0074] The coefficient sets for multiple selectable filters (e.g., up to 15 or 25 filters) are typically signaled at the picture level. However, signaling of the coefficient sets is not limited to the picture level; it may be at other levels (e.g., sequence level, slice level, tile level, CTU level, CU level, or subblock level).

[0075] [Frame memory] The frame memory 122 is a storage unit for storing reference pictures used for interpretation, and is sometimes called a frame buffer. Specifically, the frame memory 122 stores the reconstructed blocks filtered by the loop filter unit 120.

[0076] [Intra Prediction Unit] The intra-prediction unit 124 generates a prediction signal (intra-prediction signal) by performing intra-prediction (also called in-screen prediction) of the current block by referring to blocks in the current picture, such as those stored in the block memory 118. Specifically, the intra-prediction unit 124 generates an intra-prediction signal by performing intra-prediction by referring to samples (e.g., luminance values, color difference values) of blocks adjacent to the current block, and outputs the intra-prediction signal to the prediction control unit 128.

[0077] For example, the intra-prediction unit 124 performs intra-prediction using one of a predetermined set of intra-prediction modes. The set of intra-prediction modes typically includes one or more non-directional prediction modes and multiple directional prediction modes.

[0078] One or more non-directional prediction modes include, for example, the Planar prediction mode and DC prediction mode as defined in the H.265 / HEVC standard.

[0079] Multiple directional prediction modes include, for example, the 33 directional prediction modes defined in the H.265 / HEVC standard. Alternatively, multiple directional prediction modes may include an additional 32 directional prediction modes (a total of 65 directional prediction modes).

[0080] Figure 5A is a conceptual diagram showing all 67 intra-prediction modes (2 non-directional prediction modes and 65 directional prediction modes) that can be used in intra-prediction. Solid arrows represent the 33 directions specified in the H.265 / HEVC standard, and dashed arrows represent the additional 32 directions (the two "non-directional" prediction modes are not shown in Figure 5A).

[0081] In various processing examples, the luminance block may be referenced in the intra-prediction of the chrominance block. That is, the chrominance component of the current block may be predicted based on the luminance component of the current block. This intra-prediction is sometimes called CCLM (cross-component linear model) prediction. Such an intra-prediction mode for a chrominance block that references a luminance block (e.g., called the CCLM mode) may be added as one of the intra-prediction modes for a chrominance block.

[0082] The intra-prediction unit 124 may correct the pixel values ​​after intra-prediction based on the gradient of the horizontal / vertical reference pixels. Intra-prediction with such correction is sometimes called PDPC (position dependent intra-prediction combination). Information indicating whether or not PDPC is applied (for example, called a PDPC flag) is usually signaled at the CU level. However, the signaling of this information is not limited to the CU level and may be at other levels (for example, sequence level, picture level, slice level, tile level, or CTU level).

[0083] [International Prediction Department] The inter-prediction unit 126 generates a prediction signal (inter-prediction signal) by performing inter-prediction (also called inter-screen prediction) of the current block by referring to a reference picture stored in the frame memory 122 that is different from the current picture. Inter-prediction is performed in units of the current block or the current sub-block within the current block (e.g., a 4x4 block). For example, the inter-prediction unit 126 performs motion estimation within the reference picture for the current block or current sub-block and finds the reference block or sub-block within the reference picture that best matches that current block or sub-block. Then, based on that motion estimation, the inter-prediction unit 126 performs motion compensation (or motion prediction) and obtains motion information (e.g., motion vector) that compensates for (or predicts) the movement or change from the reference block or sub-block to the current block or sub-block, and generates an inter-prediction signal for the current block or sub-block based on that motion information. The inter-prediction unit 126 then outputs the generated inter-prediction signal to the prediction control unit 128.

[0084] The motion information used for motion compensation may be signaled as an interprediction signal in various forms. For example, the motion vector may be signaled. As another example, the difference between the motion vector and the predicted motion vector may be signaled.

[0085] Furthermore, an inter-prediction signal may be generated using not only the motion information of the current block obtained through motion search, but also the motion information of adjacent blocks. Specifically, an inter-prediction signal may be generated for each sub-block within the current block by weighting and adding together a prediction signal based on motion information obtained through motion search (in the reference picture) and a prediction signal based on motion information of adjacent blocks (in the current picture). Such inter-prediction (motion compensation) is sometimes called OBMC (overlapped block motion compensation).

[0086] In OBMC mode, information indicating the size of subblocks for OBMC (e.g., called OBMC block size) may be signaled at the sequence level. Information indicating whether or not OBMC mode is applied (e.g., called OBMC flag) may also be signaled at the CU level. Note that the signaling levels for this information are not limited to sequence and CU levels, but may be other levels (e.g., picture level, slice level, tile level, CTU level, or subblock level).

[0087] Let's explain the OBMC mode in more detail. Figures 5B and 5C are flowcharts and conceptual diagrams illustrating the predictive image correction process using OBMC processing.

[0088] Referring to Figure 5C, first, a predicted image (Pred) is obtained using normal motion compensation with the motion vector (MV) assigned to the current block to be encoded. In Figure 5C, the arrow "MV" points to the reference picture, indicating what the current block within the current picture is referencing in order to obtain the predicted image.

[0089] Next, the motion vector (MV_L) already derived for the encoded left-adjacent block is applied (reused) to the current block to be encoded to obtain the predicted image (Pred_L). The motion vector (MV_L) is indicated by an arrow "MV_L" pointing from the current block to the reference picture. Then, the first correction of the predicted image is performed by superimposing the two predicted images, Pred and Pred_L. This has the effect of blending the boundaries between adjacent blocks.

[0090] Similarly, the motion vector (MV_U) already derived for the encoded upper adjacent block is applied (reused) to the current block to be encoded to obtain the predicted image (Pred_U). The motion vector (MV_U) is indicated by an arrow "MV_U" pointing from the current block to the reference picture. Then, the predicted image Pred_U is superimposed on the predicted image that has undergone the first correction (i.e., Pred and Pred_L) to perform a second correction of the predicted image. In one embodiment, this has the effect of blending the boundaries between adjacent blocks. The predicted image obtained by the second correction is the final predicted image of the current block, with the boundaries with adjacent blocks blended (smoothed).

[0091] While this explanation describes a two-stage correction method using the left adjacent block and the upper adjacent block, it is also possible to use the right adjacent block and the lower adjacent block to perform corrections more than two times.

[0092] Furthermore, the area to be superimposed does not have to be the entire pixel area of ​​the block, but rather only a portion of the area near the block boundary.

[0093] Here, we have described the OBMC predictive image correction process for obtaining a single predictive image Pred by superimposing additional predictive images Pred_L and Pred_U based on a single reference picture. However, if the predictive image is corrected based on multiple reference pictures, the same process may be applied to each of the multiple reference pictures. In such cases, by performing OBMC image correction based on multiple reference pictures, corrected predictive images are obtained from each reference picture, and then these multiple corrected predictive images are further superimposed to obtain the final predictive image.

[0094] In OBMC, the unit of the target block may be the prediction block unit, or it may be a subblock unit obtained by further dividing the prediction block.

[0095] One method for determining whether or not to apply OBMC processing is to use an obmc_flag signal that indicates whether or not to apply OBMC processing. Specifically, in an encoding device, it is determined whether or not the block to be encoded belongs to a region with complex motion. If it belongs to a region with complex motion, the value of obmc_flag is set to "1" and OBMC processing is applied to perform encoding. If it does not belong to a region with complex motion, the value of obmc_flag is set to "0" and encoding is performed without applying OBMC processing. On the other hand, in a decoding device, the obmc_flag written in the stream (i.e., the compressed sequence) is decoded, and the device switches whether or not to apply OBMC processing depending on its value and performs decoding.

[0096] Furthermore, motion information may be derived at the decoding device side without being signaled at the encoding device side. For example, the merge mode specified in the H.265 / HEVC standard may be used. Alternatively, motion information may be derived by performing a motion search at the decoding device side. In this case, the decoding device side may perform the motion search without using the pixel values ​​of the current block.

[0097] Here, we will explain the mode in which motion detection is performed on the decoding device side. This mode in which motion detection is performed on the decoding device side is sometimes called PMMVD (pattern matched motion vector derivation) mode or FRUC (frame rate up-conversion) mode.

[0098] An example of FRUC processing is shown in Figure 5D. First, a list of multiple candidates (which may be the same as the merge list) is generated by referencing the motion vectors of encoded blocks spatially or temporally adjacent to the current block, each having a predicted motion vector (MV). Next, the best candidate MV is selected from the multiple candidate MVs registered in the candidate list. For example, an evaluation value is calculated for each candidate MV included in the candidate list, and one candidate MV is selected based on the evaluation value.

[0099] Then, based on the motion vector of the selected candidate, a motion vector for the current block is derived. Specifically, for example, the motion vector of the selected candidate (best candidate MV) is directly derived as the motion vector for the current block. Alternatively, for example, the motion vector for the current block may be derived by performing pattern matching in the area surrounding the position in the reference picture corresponding to the motion vector of the selected candidate. That is, a search is performed in the area surrounding the best candidate MV using pattern matching in the reference picture and evaluation values, and if an MV with a better evaluation value is found, the best candidate MV may be updated to that MV and made the final MV for the current block. It is also possible to configure the system so that it does not perform the process of updating to an MV with a better evaluation value.

[0100] The same processing method can be used when processing at the sub-block level.

[0101] The evaluation value may be calculated using various methods. For example, the reconstructed image of a region in a reference picture corresponding to the motion vector may be compared with the reconstructed image of a predetermined region (for example, a region in another reference picture, or a region in an adjacent block of the current picture, as described later), and the difference in pixel values ​​between the two reconstructed images may be calculated and used as the evaluation value for the motion vector. In addition to the difference value, other information may also be used to calculate the evaluation value.

[0102] Next, we will explain in detail an example of pattern matching. First, one candidate MV included in the candidate MV list (e.g., merge list) is selected as the starting point for the search using pattern matching. For example, first pattern matching or second pattern matching may be used. First pattern matching and second pattern matching are sometimes called bilateral matching and template matching, respectively.

[0103] In the first pattern matching, pattern matching is performed between two blocks in two different reference pictures that are aligned with the motion trajectory of the current block. Therefore, in the first pattern matching, a region in the other reference picture that is aligned with the motion trajectory of the current block is used as a predetermined region for calculating the evaluation value of the candidate mentioned above, relative to the region in the reference picture.

[0104] Figure 6 illustrates an example of first pattern matching (bilateral matching) between two blocks in two reference pictures along a motion trajectory. As shown in Figure 6, in first pattern matching, two motion vectors (MV0, MV1) are derived by searching for the best-matching pair of blocks in two different reference pictures (Ref0, Ref1) that are along the motion trajectory of the current block. Specifically, for the current block, the difference between the reconstructed image at a specified position in the first encoded reference picture (Ref0) specified by the candidate MV and the reconstructed image at a specified position in the second encoded reference picture (Ref1) specified by a symmetric MV scaled by the display time interval of the candidate MV is derived, and an evaluation value is calculated using the obtained difference value. It is possible to select the candidate MV with the best evaluation value among multiple candidate MVs as the final MV, which can yield good results.

[0105] Under the assumption of a continuous motion trajectory, the motion vectors (MV0, MV1) pointing to two reference blocks are proportional to the temporal distance (TD0, TD1) between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). For example, if the current picture is temporally located between the two reference pictures and the temporal distances from the current picture to the two reference pictures are equal, then the first pattern matching derives two mirror-symmetric bidirectional motion vectors.

[0106] In the second pattern matching (template matching), pattern matching is performed between the template in the current picture (blocks adjacent to the current block in the current picture (e.g., blocks above and / or to the left)) and the blocks in the reference picture. Therefore, in the second pattern matching, the blocks adjacent to the current block in the current picture are used as a predetermined area for calculating the evaluation value of the candidates mentioned above.

[0107] Figure 7 illustrates an example of pattern matching (template matching) between a template in the current picture and a block in the reference picture. As shown in Figure 7, in the second pattern matching, the motion vector of the current block is derived by searching in the reference picture (Ref0) for the block that best matches the block adjacent to the current block (Cur block) in the current picture (Cur Pic). Specifically, for the current block, the difference between the reconstructed image of the encoded region of both or either of the left adjacent and upper adjacent regions is derived, and the reconstructed image of the equivalent position in the encoded reference picture (Ref0) specified by the candidate MV is calculated using the obtained difference value, and the candidate MV with the best evaluation value among multiple candidate MVs is selected as the best candidate MV.

[0108] Information indicating whether or not to apply such a FRUC mode (e.g., called a FRUC flag) may be signaled at the CU level. Furthermore, if FRUC mode is applied (e.g., the FRUC flag is true), information indicating the pattern matching method (e.g., first pattern matching or second pattern matching) (e.g., called a FRUC mode flag) may be signaled at the CU level. Note that the signaling of this information is not limited to the CU level, but may be at other levels (e.g., sequence level, picture level, slice level, tile level, CTU level, or subblock level).

[0109] Next, we will explain how to derive motion vectors. First, we will describe the mode of deriving motion vectors based on a model that assumes uniform linear motion. This mode is sometimes called the BIO (bi-directional optical flow) mode.

[0110] Figure 8 is a diagram illustrating a model that assumes uniform linear motion. In Figure 8, (v x ,v y( ) indicates the velocity vector, and τ0 and τ1 respectively indicate the temporal distances between the current picture (Cur Pic) and the two reference pictures (Ref0, Ref1). (MVx0, MVy0) indicates the motion vector corresponding to the reference picture Ref0, and (MVx1, MVy1) indicates the motion vector corresponding to the reference picture Ref1.

[0111] At this time, under the assumption of a uniform linear motion of the velocity vector (v x , v y ), (MVx0, MVy0) and (MVx1, MVy1) are respectively represented as (v x τ0, v y τ0) and (-v x τ1, -v y τ1), and the following optical flow equation (1) holds.

[0112]

Equation

[0113] Here, I (k) represents the luminance value of the reference image k (k = 0, 1) after motion compensation. This optical flow equation indicates that the sum of (i) the temporal derivative of the luminance value, (ii) the product of the horizontal velocity and the horizontal component of the spatial gradient of the reference image, and (iii) the product of the vertical velocity and the vertical component of the spatial gradient of the reference image is equal to zero. Based on the combination of this optical flow equation and Hermite interpolation, the motion vectors in block units obtained from the merge list or the like may be corrected in pixel units.

[0114] Note that the motion vectors may be derived on the decoder side by a method different from the derivation of the motion vectors based on the model assuming uniform linear motion. For example, the motion vectors may be derived in sub-block units based on the motion vectors of a plurality of adjacent blocks.

[0115] Next, we will describe a mode in which motion vectors are derived at the sub-block level based on the motion vectors of multiple adjacent blocks. This mode is sometimes called the affine motion compensation prediction mode.

[0116] Figure 9A is a diagram illustrating the derivation of subblock-level motion vectors based on the motion vectors of multiple adjacent blocks. In Figure 9A, the current block contains 16 4x4 subblocks. Here, the motion vector v0 of the upper left corner control point of the current block is derived based on the motion vectors of the adjacent blocks. Similarly, the motion vector v1 of the upper right corner control point of the current block is derived based on the motion vectors of the adjacent subblocks. Then, using the two motion vectors v0 and v1, the motion vector (v) of each subblock within the current block is derived by the following equation (2). x ,v y ) is derived.

[0117]

number

[0118] Here, x and y represent the horizontal and vertical positions of the subblock, respectively, and w represents a predetermined weighting coefficient.

[0119] This affine motion compensation prediction mode may include several modes that differ in the method of deriving the motion vectors of the upper-left and upper-right corner control points. Information indicating this affine motion compensation prediction mode (e.g., called an affine flag) may be signaled at the CU level. However, the signaling of information indicating this affine motion compensation prediction mode is not limited to the CU level, but may be at other levels (e.g., sequence level, picture level, slice level, tile level, CTU level, or subblock level).

[0120] [Prediction Control Unit] The prediction control unit 128 selects either the intra-prediction signal (the signal output from the intra-prediction unit 124) or the inter-prediction signal (the signal output from the inter-prediction unit 126), and outputs the selected signal as the prediction signal to the subtraction unit 104 and the addition unit 116.

[0121] As shown in Figure 1, in various processing examples, the prediction control unit 128 may output prediction parameters that are input to the entropy coding unit 110. The entropy coding unit 110 may generate a coded bitstream (or sequence) based on the prediction parameters input from the prediction control unit 128 and the quantization coefficients input from the quantization unit 108. The prediction parameters may be used by a decoder. The decoder may receive and decode the coded bitstream and perform the same processing as the prediction processing performed in the intra-prediction unit 124, inter-prediction unit 126, and prediction control unit 128. The prediction parameters may include a selected prediction signal (e.g., a motion vector, prediction type, or prediction mode used in the intra-prediction unit 124 or inter-prediction unit 126), or any index, flag, or value that is based on or indicates the prediction processing performed in the intra-prediction unit 124, inter-prediction unit 126, and prediction control unit 128.

[0122] In some embodiments, the prediction control unit 128 operates in merge mode and optimizes the motion vector calculated for the current picture using the intra-prediction signal from the intra-prediction unit 124 and the inter-prediction signal from the inter-prediction unit 126. Figure 9B shows an example of the process of deriving the motion vector of the current picture in merge mode.

[0123] First, a list of predicted MVs is generated, containing registered candidates for predicted MVs. Candidates for predicted MVs include spatially adjacent predicted MVs, which are the MVs of multiple encoded blocks located spatially around the target block; temporally adjacent predicted MVs, which are the MVs of nearby blocks projected onto the target block's position in the encoded reference picture; combined predicted MVs, which are generated by combining the MV values ​​of spatially adjacent predicted MVs and temporally adjacent predicted MVs; and zero predicted MVs, which are MVs with a value of zero.

[0124] Next, one predicted MV is selected from the multiple predicted MVs registered in the predicted MV list to determine it as the MV for the target block.

[0125] Furthermore, the variable-length coding unit encodes the merge_idx signal, which indicates which predicted MV was selected, by writing it to a stream.

[0126] Note that the predicted MVs registered in the predicted MV list explained in Figure 9B are just an example, and the number of predicted MVs may differ from the number shown in the figure, the configuration may not include some of the types of predicted MVs shown in the figure, or it may include predicted MVs other than those shown in the figure.

[0127] The final MV may be determined by performing the DMVR (decoder motion vector refinement) process described later, using the MV of the target block derived by merge mode.

[0128] Figure 9C is a conceptual diagram illustrating an example of DMVR processing for determining MV.

[0129] First, the optimal MVP set for the current block (for example, in merge mode) is designated as the candidate MV. Then, according to the candidate MV(L0), reference pixels are identified from the first reference picture (L0), which is an encoded picture in the L0 direction. Similarly, according to the candidate MV(L1), reference pixels are identified from the second reference picture (L1), which is an encoded picture in the L1 direction. A template is generated by taking the average of these reference pixels.

[0130] Next, using the template, the surrounding regions of candidate MVs in the first reference picture (L0) and the second reference picture (L1) are searched, and the MV with the minimum cost is determined as the final MV. The cost value may be calculated, for example, using the difference between each pixel value of the template and each pixel value of the search region, as well as the candidate MV value.

[0131] Typically, the encoding device and the decoding device (described later) share the same basic configuration and operation for the processing described here.

[0132] Any process that can explore the vicinity of a candidate MV and derive the final MV is acceptable, even if it is not the exact process described here.

[0133] Next, we will describe an example of a mode that generates a predicted image (prediction) using LIC (local illumination compensation) processing.

[0134] Figure 9D is a conceptual diagram illustrating an example of a predictive image generation method using brightness correction processing by LIC processing.

[0135] First, the MV is derived from the encoded reference picture to obtain the reference image corresponding to the current block.

[0136] Next, information is extracted showing how the luminance values ​​have changed between the reference picture and the current picture for the current block. This extraction is based on the luminance pixel values ​​of the encoded left adjacent reference region (peripheral reference region) and encoded upper adjacent reference region (peripheral reference region) in the current picture, and the luminance pixel values ​​at the equivalent positions in the reference picture specified by the derived MV. Then, the luminance correction parameter is calculated using the information showing how the luminance values ​​have changed.

[0137] A predicted image for the current block is generated by applying the brightness correction parameters to the reference image within the reference picture specified in MV.

[0138] Note that the shape of the surrounding reference region in Figure 9D is just one example, and other shapes may be used.

[0139] Furthermore, although this explanation describes the process of generating a predicted image from a single reference picture, the process is similar when generating predicted images from multiple reference pictures. Alternatively, the brightness correction process may be applied to each reference picture obtained from the reference picture in the same manner as described above before generating the predicted image.

[0140] One method for determining whether or not to apply LIC processing is to use a signal called lic_flag, which indicates whether or not to apply LIC processing. For example, in an encoding device, it is determined whether the current block belongs to a region where brightness changes are occurring. If it belongs to a region where brightness changes are occurring, the value of lic_flag is set to "1" and LIC processing is applied and encoding is performed. If it does not belong to a region where brightness changes are occurring, the value of lic_flag is set to "0" and encoding is performed without applying LIC processing. On the other hand, in a decoding device, the lic_flag written in the stream may be decoded, and the device may switch whether or not to apply LIC processing depending on its value and perform decoding.

[0141] Another way to determine whether to apply LIC processing is, for example, by checking whether LIC processing was applied to surrounding blocks. A specific example is, if the current block is in merge mode, the system checks whether the surrounding encoded blocks selected during the MV derivation in merge mode were encoded with LIC processing. Based on this result, the system switches whether to apply LIC processing and then performs the encoding. Note that in this example, the same process is applied to the decoding device.

[0142] [Overview of the decryption device] Next, an overview of a decoding device capable of decoding the encoded signal (encoded bitstream) output from the above-mentioned encoding device 100 will be described. Figure 10 is a block diagram showing the functional configuration of the decoding device 200 according to the embodiment. The decoding device 200 is a video decoding device that decodes video in block units.

[0143] As shown in Figure 10, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transform unit 206, an adder unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, an intra prediction unit 216, an inter prediction unit 218, and a prediction control unit 220.

[0144] The decoding device 200 can be implemented, for example, by a general-purpose processor and memory. In this case, when the software program stored in memory is executed by the processor, the processor functions as an entropy decoding unit 202, an inverse quantization unit 204, an inverse transformation unit 206, an addition unit 208, a loop filter unit 212, an intra prediction unit 216, an inter prediction unit 218, and a prediction control unit 220. Alternatively, the decoding device 200 may be implemented as one or more dedicated electronic circuits corresponding to the entropy decoding unit 202, the inverse quantization unit 204, the inverse transformation unit 206, the addition unit 208, the loop filter unit 212, the intra prediction unit 216, the inter prediction unit 218, and the prediction control unit 220.

[0145] The following describes each component included in the decoding device 200.

[0146] [Entropy Decoder] The entropy decoding unit 202 entropically decodes the encoded bitstream. Specifically, the entropy decoding unit 202 arithmetically decodes the encoded bitstream into a binary signal, for example. Then, the entropy decoding unit 202 debinarizes the binary signal. The entropy decoding unit 202 outputs the quantization coefficients in block units to the inverse quantization unit 204. The entropy decoding unit 202 may also output prediction parameters included in the encoded bitstream (see Figure 1) to the intra-prediction unit 216, inter-prediction unit 218, and prediction control unit 220 in the embodiment. The intra-prediction unit 216, inter-prediction unit 218, and prediction control unit 220 can perform the same prediction processing as the intra-prediction unit 124, inter-prediction unit 126, and prediction control unit 128 on the encoding device side.

[0147] [Dequantization section] The inverse quantization unit 204 inversely quantizes the quantization coefficients of the block to be decoded (hereinafter referred to as the current block) input from the entropy decoding unit 202. Specifically, for each quantization coefficient of the current block, the inverse quantization unit 204 inversely quantizes the quantization coefficient based on the quantization parameter corresponding to that quantization coefficient. The inverse quantization unit 204 then outputs the inversely quantized quantization coefficients (i.e., transformation coefficients) of the current block to the inverse transformation unit 206.

[0148] [Inverse Transformation Section] The inverse transform unit 206 recovers the prediction error (residual) by inversely transforming the transformation coefficients input from the inverse quantization unit 204.

[0149] For example, if the information decoded from the encoded bitstream indicates that EMT or AMT should be applied (e.g., the AMT flag is true), the inverse transform unit 206 inversely transforms the transformation coefficients of the current block based on the information indicating the decoded transformation type.

[0150] For example, if the information decoded from the encoded bitstream indicates that NSST should be applied, the inverse transform unit 206 applies inverse retransformation to the transformation coefficients.

[0151] [Addition section] The adder 208 reconstructs the current block by adding the prediction error input from the inverse transformer 206 and the prediction sample input from the prediction control unit 220. The adder 208 then outputs the reconstructed block to the block memory 210 and the loop filter unit 212.

[0152] [Block memory] The block memory 210 is a storage unit for storing blocks that are referenced in intra prediction and are located within the decoded picture (hereinafter referred to as the current picture). Specifically, the block memory 210 stores the reconstructed blocks output from the adder 208.

[0153] [Loop Filter Section] The loop filter unit 212 applies a loop filter to the block reconstructed by the adder unit 208 and outputs the filtered reconstructed block to the frame memory 214 and the display device, etc.

[0154] If the information interpreted from the encoded bitstream indicating ALF on / off indicates ALF is on, one filter is selected from among several filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstruction block.

[0155] [Frame memory] The frame memory 214 is a memory unit for storing reference pictures used for interpretation, and is sometimes called a frame buffer. Specifically, the frame memory 214 stores the reconstructed blocks filtered by the loop filter unit 212.

[0156] [Intra Prediction Unit] The intra-prediction unit 216 generates a prediction signal (intra-prediction signal) by performing intra-prediction based on the intra-prediction mode decoded from the encoded bitstream, and by referring to the blocks in the current picture stored in the block memory 210. Specifically, the intra-prediction unit 216 generates an intra-prediction signal by performing intra-prediction by referring to samples (e.g., luminance values, chrominance values) of blocks adjacent to the current block, and outputs the intra-prediction signal to the prediction control unit 220.

[0157] Furthermore, if an intra-prediction mode that references a luminance block is selected in the intra-prediction of a color difference block, the intra-prediction unit 216 may predict the color difference component of the current block based on the luminance component of the current block.

[0158] Furthermore, if the information decoded from the encoded bitstream (for example, the prediction parameters output from the entropy decoding unit 202) indicates the application of PDPC, the intra-prediction unit 216 corrects the pixel value after intra-prediction based on the gradient of the reference pixels in the horizontal / vertical directions.

[0159] [International Prediction Department] The inter-prediction unit 218 predicts the current block by referring to a reference picture stored in the frame memory 214. Prediction is performed in units of the current block or sub-blocks within the current block (e.g., 4x4 blocks). For example, the inter-prediction unit 218 generates an inter-prediction signal for the current block or sub-block by performing motion compensation using motion information (e.g., motion vectors) decoded from the encoded bitstream (e.g., prediction parameters output from the entropy decoding unit 202), and outputs the inter-prediction signal to the prediction control unit 220.

[0160] If the information decoded from the encoded bitstream indicates that OBMC mode should be applied, the interpretation unit 218 generates an interpretation prediction signal using not only the motion information of the current block obtained by motion search, but also the motion information of the adjacent block.

[0161] Furthermore, if the information decoded from the encoded bitstream indicates that FRUC mode should be applied, the interpretation unit 218 derives motion information by performing a motion search according to the pattern matching method (bilateral matching or template matching) decoded from the encoded stream. Then, the interpretation unit 218 performs motion compensation (prediction) using the derived motion information.

[0162] Furthermore, when the BIO mode is applied, the inter-prediction unit 218 derives motion vectors based on a model that assumes uniform linear motion. Also, if the information decoded from the encoded bitstream indicates that the affine motion compensation prediction mode should be applied, the inter-prediction unit 218 derives motion vectors on a sub-block basis based on the motion vectors of multiple adjacent blocks.

[0163] [Prediction Control Unit] The prediction control unit 220 selects either the intra-prediction signal or the inter-prediction signal and outputs the selected signal as the prediction signal to the summer unit 208. The prediction control unit 220 may also refer to the prediction control unit 128 on the encoding device side and perform various functions and processes such as the merge mode (see Figure 9B), DMVR processing (see Figure 9C), and LIC processing (see Figure 9D) as described above. Overall, the configuration, functions, and processing of the prediction control unit 220, intra-prediction unit 216, and inter-prediction unit 218 on the decoding device side may correspond to the configuration, functions, and processing of the prediction control unit 128, intra-prediction unit 124, and inter-prediction unit 126 on the encoding device side.

[0164] [Details of Interpretation] The following describes examples of other embodiments of interpretation according to the present invention. This embodiment relates to interpretation in so-called BIO mode. In this embodiment, the motion vectors of blocks are corrected not on a pixel-by-pixel basis, but on a sub-block basis, similar to the embodiments shown in Figures 1 to 10. In describing this embodiment, we will focus on the differences from the embodiments shown in Figures 1 to 10.

[0165] Since the configurations of the encoding and decoding devices in this embodiment are substantially the same as those in the embodiments shown in Figures 1 to 10, a description and explanation of these configurations will be omitted.

[0166] [Interface prediction] Figure 11 is a flowchart of interpretation in this embodiment. Figure 12 is a conceptual diagram used to explain interpretation in this embodiment. The following processes are performed by the interpretation prediction unit 126 of the encoding device 100 and the interpretation prediction unit 218 of the decoding device 200.

[0167] As shown in Figure 11, first, block loop processing is performed on multiple blocks within the picture to be encoded / decoded (current picture 1000) (S101~S111). In Figure 12, the block to be encoded / decoded is selected from the multiple blocks as the current block 1001.

[0168] In the block loop processing, the loop processing is performed on the reference pictures, specifically on the first reference picture 1100 (L0) and the second reference picture 1200 (L1), which are already processed pictures (S102-S106).

[0169] In the reference picture loop processing, first, block motion vectors are derived or obtained in order to acquire a predicted image from the reference picture (S103). In Figure 12, the first motion vector 1110 (MV_L0) for the first reference picture 1100 is derived or obtained, and the second motion vector 1210 (MV_L1) for the second reference picture 1200 is derived or obtained. Examples of available motion vector derivation methods include the normal inter-prediction mode, merge mode, and FRUC mode. In the normal inter-prediction mode, the encoding device 100 derives motion vectors by motion search, and the decoding device 200 obtains motion vectors from the bitstream.

[0170] Next, a predicted image is obtained from the reference picture by performing motion compensation using the derived or acquired motion vector (S104). In Figure 12, the first predicted image 1140 is obtained from the first reference picture 1100 by performing motion compensation using the first motion vector 1110. Also, the second predicted image 1240 is obtained from the second reference picture 1200 by performing motion compensation using the second motion vector 1210.

[0171] In motion compensation, a motion compensation filter is applied to the reference picture. The motion compensation filter is an interpolation filter used to obtain a predicted image with sub-pixel accuracy. Using the first reference picture 1100 in Figure 12, pixels in the first prediction block 1120 and pixels in the first interpolation range 1130, including surrounding pixels, are referenced by the motion compensation filter in the first prediction block 1120, indicated by the first motion vector 1110. Similarly, using the second reference picture 1200, pixels in the second prediction block 1220 and pixels in the second interpolation range 1230, including surrounding pixels, are referenced by the motion compensation filter in the second prediction block 1220, indicated by the second motion vector 1210.

[0172] The first interpolation reference range 1130 and the second interpolation range 1230 include the first and second normal reference ranges used to compensate for the motion of the current block 1001 in normal interpretation, which processes using local motion detection values. The first normal reference range is included in the first reference picture 1100, and the second normal reference range is included in the second reference picture 1200. In normal interpretation, motion vectors are derived in the block using motion search, motion compensation is performed on the derived motion vectors in the block, and the motion-compensated image is used as the final predicted image. In other words, local motion detection values ​​are not used in normal interpretation. The first interpolation reference range 1130 and the second interpolation reference range 1230 may coincide with the first and second normal reference ranges.

[0173] Next, a gradient image corresponding to the predicted image is obtained from the reference picture (S105). Each pixel in the gradient image has a gradient value that indicates the spatial slope of the luminance or color difference. The gradient value is obtained by applying a gradient filter to the reference picture. In the first reference picture 1100 in Figure 12, pixels in the first prediction block 1120 and pixels in the first gradient reference range 1135, which includes surrounding pixels, are referenced using a gradient filter for the first prediction block 1120. The first gradient reference range 1135 is included in the first interpolation reference range 1130. In the second reference picture 1200, pixels in the second prediction block 1220 and pixels in the second gradient reference range 1235, which includes surrounding pixels, are referenced using a gradient filter for the second prediction block 1220. The second gradient reference range 1235 is included in the second interpolation reference range 1230.

[0174] The reference picture loop process ends when the predicted image and gradient image are obtained from both the first and second reference pictures (S106). Subsequently, the loop process is performed on multiple subblocks obtained by further dividing the block (S107-S110). The size of each subblock is smaller than the size of the current block (for example, 4x4 pixels).

[0175] In the subblock loop processing, first, the local motion detection value 1300 is derived using the first prediction image 1140, the second prediction image 1240, the first gradient image 1150, and the second gradient image 1250 obtained from the first reference picture 1100 and the second reference picture 1200 (S108). For example, the pixels within the prediction subblock in the first prediction image 1140, the second prediction image 1240, the first gradient image 1150, and the second gradient image 1250 are referenced, and the local motion detection value 1300 is derived for the subblock. The prediction subblock is the region within the first prediction block 1140 and the second prediction block 1240 corresponding to the subblock of the current block 1001. The local motion detection value is sometimes called the corrected motion vector.

[0176] Next, the pixel values ​​of the first prediction image 1140 and the second prediction image 1240, the gradient values ​​of the first gradient image 1150 and the second gradient image 1250, and the local motion detection value 1300 are used to generate the final prediction image 1400 for the subblock. If a final prediction image has been generated for each subblock of the current block, the final prediction image is generated for the current block and the subblock loop processing is completed (S110).

[0177] When the block loop processing is completed (S111), the processing in Figure 11 is terminated.

[0178] Furthermore, the predicted image and gradient image can be obtained at the sub-block level by assigning the motion vector for the current block to each sub-block.

[0179] [Reference range of motion compensation filter and gradient filter in the example of this embodiment] The following describes examples of the reference ranges for the motion compensation filter and gradient filter according to this embodiment.

[0180] Figure 13 is a conceptual diagram used to illustrate an example of the reference range of a gradient filter and a motion compensation filter according to this embodiment.

[0181] In Figure 13, each circle represents a sample. In the example shown in Figure 13, the current block size is 8x8 samples, and the sub-block size is 4x4 samples.

[0182] Reference range 1131 is the reference range of the motion compensation filter applied to the top-left sample 1122 of the first prediction block 1120 (e.g., a square range of 8x8 samples). Reference range 1231 is the reference range of the motion compensation filter applied to the top-left sample 1222 of the second prediction block 1220 (e.g., a square range of 8x8 samples).

[0183] Reference range 1132 is the reference range of the gradient filter applied to the top-left sample 1122 of the first prediction block 1120 (e.g., a square range of 6x6 samples). Reference range 1232 is the reference range of the gradient filter applied to the top-left sample 1222 of the second prediction block 1220 (e.g., a square range of 6x6 samples).

[0184] The motion compensation filter and gradient filter are applied to the remaining samples in the first prediction block 1120 and the second prediction block 1220, referencing samples within the same size reference range at the position corresponding to each sample. As a result, samples within the first interpolation range 1130 and the second interpolation range 1230 are referenced to obtain the first prediction image 1140 and the second prediction image 1240. Similarly, samples within the first gradient range 1135 and the second gradient range 1235 are referenced to obtain the first gradient image 1150 and the second gradient image 1250.

[0185] [Effects, etc.] Thus, the encoding and decoding devices in this embodiment can derive local motion detection values ​​for subblocks. Therefore, while reducing prediction errors using local motion detection values ​​on a subsample basis, the processing load or processing time can be reduced compared to when local motion detection values ​​are derived on a sample basis.

[0186] Furthermore, the encoding and decoding devices in this embodiment can use an interpolation reference range within the normal reference range. Therefore, when generating the final predicted image using local motion detection values ​​in a subblock, there is no need to load new sample data from frame memory during motion compensation, thus suppressing increases in memory capacity and memory bandwidth.

[0187] Furthermore, the encoding and decoding devices in this embodiment can use a gradient reference range within the normal reference range. Therefore, there is no need to load new sample data from frame memory to acquire gradient images, which can suppress increases in memory capacity and memory bandwidth.

[0188] This embodiment may be combined with at least some aspects of other embodiments. Furthermore, some processes, some elements of the apparatus, and some syntax described in the flowchart of this embodiment may be combined with aspects of other embodiments.

[0189] [Modified example 1 of this embodiment] The following describes in detail modified versions of the gradient filter and motion compensation filter according to this embodiment. In Modification 1, the processing for the second predicted image is the same as the processing for the first predicted image, and further explanation is omitted or simplified.

[0190] [Motion compensation filter] First, let's explain the motion compensation filter. Figure 14 is a conceptual diagram used to illustrate an example of the reference range of the motion compensation filter in Modification 1 of this embodiment.

[0191] In the following description, a motion compensation filter with 1 / 4 sample horizontally and 1 / 2 sample vertically is applied to the first prediction block 1120. This motion compensation filter is a so-called 8-tap filter and is represented by the following equation (3).

[0192]

number

[0193] Here, I k [x,y] represents the sample value in the first predicted image with subsample accuracy when k is 0, and the sample value in the second predicted image with subsample accuracy when k is 1. The sample value is a value relative to the sample, for example, the luminance value or chrominance value in the predicted image. Here, w 0.25 and w 0.5 These are the weighting coefficients for 1 / 4 sample precision and 1 / 2 sample precision. k[x,y] represents the sample value in the first predicted image with overall sample accuracy when k is 0, and the sample value in the second predicted image with overall sample accuracy when k is 1.

[0194] For example, when applying the motion compensation filter in equation (3) to the upper left sample 1122 in Figure 14, the values ​​of the horizontally arranged samples within the reference range 1131A are weighted and added row by row, and the sum of those rows is also weighted and added.

[0195] In this modified example, the motion compensation filter for the upper-left sample 1122 refers to samples within the reference range 1131A. The reference range 1131A is a rectangular area extending 3 samples to the left of the upper-left sample 1122, 4 samples to the right, 3 samples above, and 4 samples below.

[0196] This motion compensation filter is applied to all samples within the first prediction block 1120. Therefore, samples within the first interpolation reference range 1130A are referenced by the motion compensation filter for the first prediction block 1120.

[0197] This motion compensation filter is applied to the second prediction block 1220, as well as the first prediction block 1120. That is, samples within reference range 1231A are referenced for the top-left sample 1222, and samples within the second interpolation range 1230A are referenced throughout the entire second prediction block 1220.

[0198] [Gradient filter] The following is a description of the gradient filter. Figure 15 is a conceptual diagram used to illustrate an example of the reference range of the gradient filter in Modification 1 of this embodiment.

[0199] The gradient filter in this modified example is a so-called 5-tap filter, and is represented by the following equations (4) and (5).

[0200]

number

[0201]

number

[0202] Here, I x k [x,y] represents the sample-by-sample horizontal gradient value in the first gradient image when k is 0, and the sample-by-sample horizontal gradient value in the second gradient image when k is 1. y k [x,y] represents the sample-by-sample vertical gradient value in the first gradient image when k is 0, and the sample-by-sample vertical gradient value in the second gradient image when k is 1.

[0203] For example, when applying the gradient filters in equations (4) and (5) to the upper-left sample 1122 in Figure 15, the horizontal gradient sample value is the sample value of five samples arranged horizontally, including the upper-left sample 1122, and is calculated by weighting and adding the sample values ​​in the predicted image with overall sample accuracy. In this case, the weight coefficients have positive or negative values ​​for the samples above or below and to the left or right of the upper-left sample 1122.

[0204] In this modified example, the gradient filter for the upper-left sample 1122 refers to samples within the reference range 1132A. The reference range 1132A is cross-shaped and extends two samples above and below, and to the left and right of the upper-left sample 1122.

[0205] This gradient filter is applied to all samples in the first prediction block 1120. Therefore, samples within the first gradient reference range 1135A are referenced by the motion compensation filter for the first prediction block 1120.

[0206] This gradient filter is applied to the second prediction block 1220, just as it is to the first prediction block 1120. That is, samples within reference range 1232A are referenced for the top-left sample 1222, and samples within the second gradient range 1235A are referenced throughout the entire second prediction block 1220.

[0207] If the motion vector indicating the reference range indicates the subsample position, the sample values ​​in the reference ranges 1132A and 1232A of the gradient filter may be converted to sample values ​​with subsample precision, and the gradient filter may be applied to the converted sample values. Alternatively, a gradient filter having coefficient values ​​obtained by convolving coefficient values ​​for conversion to subsample precision and coefficient values ​​for deriving the gradient value may be applied to the sample values ​​with overall sample precision. In this case, the gradient filter will differ for each subsample position.

[0208] [Deduction of local motion detection values ​​for subblocks] The following is an explanation of how to derive local motion detection values ​​for subblocks. In this example, local motion detection values ​​are derived for the top-left subblock among the subblocks in the current block.

[0209] In this modified example, the horizontal local motion detection value u and the vertical local motion detection value v are derived for the subblock based on the following equation (6).

[0210]

number

[0211] Here is crosssG x sG y sG x 2 ,, sG y 2 sG x dI, and sG y dI is a value calculated in the subblock and is derived based on the following equation (7).

[0212] [Numerical]

[0213] Here, Ω is a set of coordinates for all samples in the predicted sub-block in the area / region of the predicted block corresponding to the sub-block. G x [i, j] represents the sum of the horizontal gradient value of the first gradient image and the horizontal gradient value of the second gradient image. G y [i, j] represents the sum of the vertical gradient value of the first gradient image and the vertical gradient value of the second gradient image. ΔI[i, j] represents the difference value for the first predicted image and the second predicted image. w[i, j] represents a weight coefficient determined by the sample position in the predicted sub-block. However, the same value of the weight coefficient may be used for all samples in the predicted sub-block.

[0214] More specifically, G x [i, j], G y [i, j], and ΔI[i, j] are represented by the following equation (8).

[0215] [Numerical]

[0216] In this way, the local motion detection value is derived in the sub-block.

[0217] [Generation of the final predicted image] The following is an explanation of the generation of the final predicted image. Each sample value p[x, y] in the final predicted image is derived based on the following equation (9) using the sample value I 0 [x, y] in the first predicted image and the sample value I 1 [x, y] in the second predicted image.

[0218] [Numerical]

[0219] Here, b[x,y] represents the correction value for each sample. In equation (9), the sample value I in the first predicted image is 0 [x,y] and the sample value I in the second predicted image 1 The sample value p[x,y] in the final predicted image is calculated by shifting the sum of [x,y] and the correction value b[x,y] one bit to the right. The correction value b[x,y] is expressed by the following equation (10).

[0220]

number

[0221] In equation (10), the difference in horizontal gradient values ​​between the first gradient image and the second gradient image (I x 0 [x,y]-I x 1 Multiply the horizontal local motion detection value (u) by the [x,y] value, and calculate the difference (I) between the vertical gradient values ​​in the first gradient image and the second gradient image. y 0 [x,y]-I y 1 The correction value b[x,y] is calculated by multiplying [x,y] by the vertical local motion detection value (v) and summing the products of these values.

[0222] The calculations explained using equations (6) through (10) are merely examples. Other equations with the same effect may also be used.

[0223] [Effects, etc.] As described above, by using the motion compensation filter and gradient filter in this modified example, local motion detection values ​​can be derived in the subblock. Using the local motion detection values ​​of the subblock derived in this way, the final predicted image of the current block is generated, and the same results as in this embodiment can be obtained.

[0224] This embodiment may be combined with at least some aspects of other embodiments. Furthermore, some processes, some elements of the apparatus, and some syntax described in the flowchart of this embodiment may be combined with aspects of other embodiments.

[0225] [Modified example 2 of this embodiment] In this embodiment and Modification 1 of this embodiment, when deriving local motion detection values, all samples in the prediction subblocks within the prediction block corresponding to the subblocks within the current block are referenced. However, this disclosure is not limited to these examples. For example, only some samples within the prediction subblocks may be referenced. This scenario is described in the following paragraph as Modification 2 of this embodiment.

[0226] In this modified explanation, when deriving local motion detection values ​​in a subblock, only a few samples within the prediction subblock are referenced. For example, in equation (7) in Modified Example 1, instead of a coordinate set Ω for all samples in the prediction subblock, a coordinate set for some samples within the prediction subblock is used. Various patterns can be used as the coordinate set for some samples within the prediction subblock.

[0227] Figure 16 shows an example of a sample pattern referenced by deriving local motion detection values ​​in a modified example 2 of the embodiment. In Figure 16, circles cross-hatched within prediction block 1121 or 1221 indicate referenced samples, and circles not cross-hatched indicate unreferenced samples.

[0228] The seven sample patterns in Figure 16(a)-(g) each represent several samples within prediction subblock 1121 or 1221. All seven of these sample patterns are distinct.

[0229] In (a) to (c) of FIG. 16, only 8 samples out of 16 samples in the prediction sub-block 1121 or 1221 are referred to. In (d) to (g) of FIG. 16, only 4 samples out of 16 samples in the prediction sub-block 1121 or 1221 are referred to. That is, in (a) to (c) of FIG. 16, 8 samples out of 16 samples are decimated, and in (d) to (g) of FIG. 16, 12 samples out of 16 samples are decimated.

[0230] More specifically, in FIG. 16(a), 8 samples, that is, samples taken at every other position in the horizontal and vertical directions are referred to. In FIG. 16(b), two sets of sample pairs arranged on the left and right in the horizontal direction and arranged alternately in the vertical direction are referred to. In FIG. 16(c), in the prediction sub-block 1121 or 1221, the central 4 samples and the corner 4 samples are referred to.

[0231] In FIGS. 16(d) and (e), 2 samples are referred to in the first and third rows in the horizontal direction and the first and third rows in the vertical direction, respectively. In FIG. 16(f), the 4 corner samples are referred to. In FIG. 16(g), the 4 central samples are referred to.

[0232] Based on the two prediction images, a reference pattern may be adaptively selected from a plurality of predetermined patterns. For example, a sample pattern including samples corresponding to representative gradient values in the two prediction images may be selected. More specifically, when the representative gradient value is smaller than the threshold value, a sample pattern including 4 samples (for example, any one of (d) to (g)) may be selected. When the representative gradient value is larger than the threshold value, a sample pattern including 8 samples (for example, any one of (a) to (c)) may be selected.

[0233] When one sample pattern is selected from a plurality of sample patterns, the samples in the prediction sub-block indicating the selected sample pattern are referred to, and a local motion detection value for the sub-block is derived.

[0234] Information indicating the selected sample pattern may also be written to the bitstream. In this case, the decoder may retrieve information from the bitstream and select a sample pattern based on the retrieved information. Information indicating the selected sample pattern may be written to the header on a block, slice, picture, or stream basis.

[0235] Thus, the encoding and decoding devices in this embodiment can derive local motion detection values ​​in a subblock by referring to only a few samples within the prediction subblock. Therefore, compared to referring to all samples, the processing load or processing time can be reduced.

[0236] Furthermore, the encoding and decoding devices in this embodiment can derive local motion detection values ​​in a subblock by referencing only the samples in one sample pattern selected from multiple sample patterns. By switching sample patterns, it is possible to reference samples suitable for deriving local motion detection values ​​in a subblock, thereby reducing prediction errors.

[0237] This embodiment may be combined with at least some aspects of other embodiments. Furthermore, some processes, some elements of the apparatus, and some syntax described in the flowchart of this embodiment may be combined with aspects of other embodiments.

[0238] [Other variations of this embodiment] Encoding and decoding devices in one or more aspects of this disclosure have been described with reference to embodiments and modifications thereof. However, this disclosure is not limited to these embodiments and modifications thereof. Those skilled in the art will readily be able to conceive of various modifications that are applied to these embodiments and modifications thereof and that fall within the scope of one or more aspects of this disclosure, without departing from the spirit and scope of this disclosure.

[0239] For example, the number of taps in the motion compensation filter used in the example of this embodiment and Modification 1 of this embodiment was 8 samples. However, this disclosure is not limited to this example. The number of taps in the motion compensation filter may be any number, as long as the interpolation reference range is included in the normal reference range.

[0240] The number of taps in the gradient filter used in the example of this embodiment and Modification 1 of this embodiment was 5 or 6 samples. However, this disclosure is not limited to this example. The number of taps in the gradient filter may be any number, as long as the gradient reference range is included in the interpolation reference range.

[0241] In the example of this embodiment and in Modification 1 of this embodiment, the first gradient reference range and the second gradient reference range were included in the first interpolation reference range and the second interpolation reference range. However, this disclosure is not limited to this example. For example, the first gradient reference range may coincide with the first interpolation reference range, and the second gradient reference range may coincide with the second interpolation reference range.

[0242] When deriving local motion detection values ​​in a subblock, the sample values ​​may be weighted so that the values ​​for the sample in the center of the prediction subblock are given preferential treatment. Specifically, when deriving local motion detection values, the values ​​for multiple samples within the prediction subblock may be weighted in both the first and second prediction blocks. In this case, a larger weight may be given to the sample in the center of the prediction subblock. That is, the sample in the center of the prediction subblock may be weighted with a larger value than the value used for the sample outside the center of the prediction subblock. More specifically, in Modification 1 of this embodiment, the weight coefficient w[i,j] in equation (7) may be increased to a larger value for coordinate values ​​closer to the center of the prediction subblock.

[0243] When deriving local motion detection values ​​in a subblock, samples belonging to adjacent prediction subblocks may be referenced. Specifically, in both the first and second prediction subblocks, samples from adjacent prediction subblocks may be referenced in addition to the samples within the prediction subblock to derive local motion detection values ​​in the subblock.

[0244] The reference ranges of the motion compensation filter and gradient filter in the examples of this embodiment and Modification 1 of this embodiment are for illustrative purposes only. This disclosure is not limited to these examples.

[0245] In Modification 2 of this embodiment, seven sample patterns were given as examples. However, this disclosure is not limited to these sample patterns. For example, sample patterns obtained by rotating each of the seven sample patterns may be used.

[0246] The weight coefficient values ​​in Modification 1 of this embodiment are merely examples, and this disclosure is not limited to these examples. Similarly, the block size and sub-block size in this embodiment and both of its modifications are also merely examples. This disclosure is not limited to 8×8 and 4×4 sample sizes. Interpretation can be performed using other sizes, similar to the example and both of its modifications.

[0247] This embodiment may be combined with at least some aspects of other embodiments. Furthermore, some processes, some elements of the apparatus, and some syntax described in the flowchart of this embodiment may be combined with aspects of other embodiments.

[0248] As described with reference to Figures 4 and 5, for the reference range of surrounding samples to which a gradient filter is applied in order to obtain gradient values, the sample values ​​within the reference block, indicated by the motion vector assigned to the prediction block used in the prediction process, may be compensated with values ​​other than the sample values ​​described with reference to Figures 4 and 5.

[0249] For example, if the motion vector assigned to a prediction block indicates a fractional sample position, the sample value corresponding to that fractional sample position may be compensated for and used.

[0250] However, if the filtering process references surrounding samples to obtain sample values ​​corresponding to decimal sample positions, it would not be necessary to reference surrounding samples over a wider range than described in Figures 4 and 5.

[0251] To solve the above problems, this disclosure provides, for example, the following methods.

[0252] Figure 17 is a flowchart of an example image coding / decoding method 1700 that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture.

[0253] In the embodiment of the present application, the encoding method is implemented by an encoding device 100 as shown in Figure 1 and described in the corresponding description. In the embodiment, the encoding method is implemented by an interpretation unit 126 of the encoding device 100 in cooperation with other components of the encoding device 100.

[0254] Similarly, in the embodiments of the present application, the decoding method is implemented by a decoding device 200 as shown in Figure 10 and described in the corresponding description. In the embodiments, the decoding method is implemented by an interpretation unit 128 of the decoding device 200 in cooperation with other components of the decoding device 200.

[0255] As shown in Figure 17, the encoding method in this example includes the following steps.

[0256] Step 1702: Predict the first block of prediction samples for the current block of the picture. Here, the step of predicting the first block of prediction samples includes at least a prediction process using motion vectors from another picture.

[0257] Step 1704: Pad the first block of the prediction sample to form the second block of the prediction sample, where the second block is larger than the first block.

[0258] Step 1706: Use the second block of prediction samples to calculate at least the gradient.

[0259] Step 1708: Encode the current block using at least the calculated gradient.

[0260] In this embodiment, another picture may have a different picture order count (POC) than the picture in the time domain of the stream. As shown in Figures 18A, 18B, and 18C, the picture may be the current picture, and the other picture may be the first picture or the second picture. For example, in Figure 18A, if the other picture is the first picture, the POC of the other picture is smaller than the POC of the picture in the left time domain, but larger than the POC of the picture in the right time domain. In Figure 18B, if the other picture is the first picture, the POC of the other picture is smaller than the POC of the picture in both the left and right time domains. In Figure 18C, if the other picture is the first picture, the POC of the other picture is larger than the POC of the picture in both the left and right time domains.

[0261] In other embodiments, the other picture may be an encoded reference picture that is temporally and / or spatially adjacent to the picture. The encoded reference picture may have a POC smaller or larger than the POC of the picture in terms of time and / or space.

[0262] The current block can be arbitrarily selected within the picture. For example, the current block could be a square 4x4 sample block. The size of the current block may be changed to suit the actual prediction accuracy needs.

[0263] Similar to the motion vector shown in Figure 5C, the motion vector in this example of the encoding method points from the current block to another picture, which may be the first or second picture shown in Figures 18A to 18C. In this embodiment, in particular, if the other picture is an encoded reference picture encoded with the same motion vector, the motion vector may be received from the other picture by the interpretation unit 126.

[0264] In step 1702, the interpretation unit 126 of the encoding device 100 predicts the first block of prediction samples for the current block of the picture using motion vectors from another picture and at least the prediction process described above.

[0265] In this embodiment, the first block of the prediction sample may be the same as the prediction block used in the prediction processing described above for a prediction mode such as merge mode or interprediction mode.

[0266] In other embodiments, the first block of prediction samples may be the same as the prediction block used in the motion compensation processing described above, which is performed for prediction modes such as merge mode or interprediction mode.

[0267] In step 1704, the interpretation unit 126 of the encoding device 100 pads the first block of prediction samples to form a second block of prediction samples. The second block of prediction samples is larger than the first block of prediction samples. In this way, more information is held in the second block for subsequent processing, which may favorably provide the interpretation method with improved processing and thereby reduce the memory bandwidth access of the interpretation method. This is hardware-friendly.

[0268] For example, step 1704 may be performed by the interprediction unit 126 of the encoding device 100 by padding the sample with at least two sides of the first block of the prediction sample, where at least two sides of the first block of the prediction sample are not orthogonal.

[0269] In some embodiments, the interpretation unit 126 of the encoding device 100 may pad at least two sides of the first block of the prediction sample in a parallel padding direction. As shown in Figure 19A, the interpretation unit 126 of the encoding device 100 may pad two vertical sides of the 4x4 first block 1902 of the prediction sample in the manner shown by the dotted line in Figure 19A. Alternatively, as shown in Figure 19B, the interpretation unit 126 of the encoding device 100 may pad two horizontal sides of the 4x4 first block 1904 of the prediction sample in the manner shown by the dotted line in Figure 19B.

[0270] In some other embodiments, the interpretation unit 126 of the encoding device 100 may pad the four sides of the first block of prediction samples, as shown in Figures 20A, 20B, 20C, and 20D. Figures 20A, 20B, 20C, and 20D also illustrate four padding processes that can be used to pad the first block of prediction samples. Those skilled in the art will understand that these padding processes may be further used to pad any sample block.

[0271] In the embodiment, as shown in Figure 20A, padding may be performed by mirroring the first block 2002 of the prediction sample in the manner shown by the dotted line in Figure 20A. For example, the interpretation unit 126 of the encoding device 100 may pad the first block of the prediction sample by mirroring the first block 2002 of the prediction sample symmetrically with respect to each edge of the first block 2002. The mirroring may include forming the second block 2000 of the prediction sample by symmetrically copying the sample values ​​of the first block 2002 of the prediction sample (e.g., "A", "B", etc.) to the samples surrounding the first block 2002 of the prediction sample.

[0272] In other embodiments, as shown in Figure 20B, padding may be performed by copying the first block 2004 of the prediction sample in the manner shown by the dotted line in Figure 20B. For example, the interpretation unit 126 of the encoding device 100 may pad the first block 2004 of the prediction sample by copying the sample values ​​of the samples located at each edge of the first block 2004 of the prediction sample (e.g., "A", "B", etc.) to the corresponding samples that surround the first block of the prediction sample and are adjacent to each sample located at each edge of the first block 2004, thereby forming the second block 2001 of the prediction sample.

[0273] Note that the corresponding sample may be a single row of sample adjacent to the first block. The fewer the number of padded sample rows, the higher the accuracy of the second block 2001. Alternatively, the corresponding sample may be a multi-row sample including one row adjacent to the first block. Specifically, the corresponding sample is adjacent to each sample located at each edge of the first block 2004. The more padded sample rows there are, the greater the reduction in memory bandwidth. The number of padded sample rows may be set, for example, based on the size of the first block, or may be predetermined according to standards or other criteria.

[0274] In other embodiments, as shown in Figure 20C, padding may be performed by padding the samples surrounding the first block of prediction samples with a fixed value (e.g., "Q" as shown in Figure 20C) in the manner indicated by the dotted line in Figure 20C. For example, the fixed value may be selected from at least one of 0, 128, 512, a positive integer, the mean of the first block of prediction samples, and the median of the first block of prediction samples. The mean and median may be associated with the sample values ​​of the first block of prediction samples. In the example in Figure 20C, the fixed value is a positive integer Q, and the interpretation unit 126 of the encoding device 100 may pad the first block of prediction samples by padding the samples surrounding the first block of prediction samples with the fixed value Q to form the second block 2050 of prediction samples.

[0275] In other embodiments, padding may be performed by executing a function on the first block of prediction samples. Examples of such functions include filters, polynomial functions, exponential functions, and clipping functions. For simplicity, this example is not illustrated.

[0276] In further embodiments, as shown in Figure 20D, padding may be performed by any combination of mirroring, copying, first value padding, and function execution on the prediction samples as described above. In the example of Figure 20D, the interpretation unit 126 of the encoding device 100 may pad the first block 2008 of the prediction samples by combining copying the first block 2008 of the prediction samples to samples located a first fixed distance away from the first block 2008 and padding samples located a second fixed distance away from the first block 2008 with a fixed value Q, thereby forming the second block 2051 of the prediction samples.

[0277] In the examples shown in Figures 20A to 20D, the interpretation unit 126 of the encoding device 100 pads the 4x4 first blocks 2002, 2004, 2006, and 2008 of the prediction samples to form the 8x8 second blocks 2000, 2001, 2050, and 2051 of the prediction samples.

[0278] Referring again to the examples in Figures 19A and 19B, it can be seen that the interpretation unit 126 of the encoding device 100 pads the 4x4 first blocks 1902 and 1904 of the prediction samples by mirroring the first blocks 1902 and 1904 of the prediction samples to form the second blocks 1900 and 1901 of the prediction samples. In Figure 19A, the second block 1900 of the prediction samples is an 8x4 sample block. In Figure 19B, the second block 1901 of the prediction samples is a 4x8 sample block.

[0279] Therefore, it will be understood by those skilled in the art that the size of the second block of desired prediction samples from the padding process can be changed according to the actual prediction accuracy needs. Advantageously, the padding process reduces the data for subsequent interprediction processing.

[0280] Therefore, the padding step 1704 of this coding method can be designed to pad a first block of M × N size prediction samples to form a second block of prediction samples of any desired size (M + d1) × (N + d2), based on the actual prediction accuracy needs based on the processing / techniques described above with respect to Figures 19A, 19B, and 20A to 20D, and / or different padding directions. It will be understood by those skilled in the art that M may be the same as or different from N, while d1 and d2 are greater than or equal to 0.

[0281] Thanks to padding, small data can be padded to a larger size to generate more accurate predictions for the current block, including more information. Thus, this encoding method requires only small data as input, which is advantageous because it reduces memory bandwidth access for interpretation and provides a more hardware-friendly processing.

[0282] In step 1706, the interprediction unit 126 of the encoding device 100 calculates at least a gradient using the second block of prediction samples. In this embodiment, when calculating at least a gradient, the interprediction unit 126 of the encoding device 100 may apply a gradient filter to the second block of prediction samples to generate at least a derivative that functions as at least one gradient. An example of applying a gradient filter to generate at least one gradient is described above in the section on gradient filters. Thus, the data in the current block of the current picture is referenced by the data from the reference block of the reference picture, which is encoded using each gradient (i.e., derivative) between the reference block and the current block. Such a step favorably facilitates the reduction of memory bandwidth access for interprediction and provides a more hardware-friendly process.

[0283] Two examples of gradient filters are shown in Figures 21A and 21B, respectively. In Figure 21A, a single gradient filter {2, -9, 0, 9, -2} is applied to all sample values ​​in the second block of prediction samples, regardless of the fractional part of the motion vector. In Figure 21A, the process of applying the gradient filter {2, -9, 0, 9, -2} is the same as described earlier in the section on gradient filters. In the example in Figure 21A, equations (4) and (5) are used for all motion vectors.

[0284] In Figure 21B, there are nine gradient filters applied to the sample values ​​in the second block of the prediction sample according to the fractional part of the motion vector: {8,-39,-3,46,-17,5}, {8,-32,-13,50,-18,5}, {7,-27,-20,54,-19,5}, {6,-21,-29,57,-18,5}, {4,-17,-36,60,-15,4}, {3,-9,-44,61,-15,4}, {1,-4,-48,61,-13,3}, {0,1,-54,60,-9,2}, and {-1,4,-57,57,-4,1}. In the example in Figure 21B, the fractional part of the motion vector is a subsample portion of the motion vector, e.g., 1 / 4 sample. Therefore, different gradient filters are selected based on the subsample portion of the motion vector. For example, if the horizontal motion vector has 1 + (1 / 4) samples, the fractional part of the motion vector is 1 / 4. Then, the gradient filter {4, -17, -36, 60, -15, 4} is selected to calculate the horizontal gradient. w[i] in equation (4) is replaced with {4, -17, -36, 60, -15, 4}. In another example, if the vertical motion vector has 4 + (7 / 16) samples, the fractional part of the motion vector is 7 / 16. In this example, the gradient filter {0, 1, -54, 60, -9, 2} is selected to calculate the vertical gradient. w[i] in equation (5) is replaced with {0, 1, -54, 60, -9, 2}.

[0285] Figures 21C and 21D show two other examples of gradient filters. In Figure 21C, a single gradient filter {-1,0,1} is applied to all sample values ​​in the second block of prediction samples, regardless of the fractional part of the motion vector.

[0286] In Figure 21D, a single gradient filter {-1,1} is applied to all sample values ​​in the second block of predicted samples, regardless of the fractional part of the motion vector. In Figures 21C and 21D, the process of applying each gradient filter {-1,0,1} or {-1,1} is the same as described above in the section on gradient filters.

[0287] In step 1708, the interpretation unit 126 of the encoding device 100 encodes the current block of the picture using at least the calculated gradient.

[0288] As shown in Example 1700 in Figure 17, the steps in the decoding method are the same as those in the encoding method, except for the last step 1708. In this last step, the encoding device 100 encodes the current block using at least a gradient, while the decoding device 200 decodes the current block using at least a gradient.

[0289] The above examples and embodiments predict the current block using prediction blocks (which are alternately referenced as blocks of prediction samples) as input. Further, other embodiments of this encoding method, in which the current block is predicted using two prediction blocks as input, will be described below.

[0290] When predicting the current block using two prediction blocks (i.e., two blocks of prediction samples) as input, step 1702 as described above further includes the step of predicting another block of the prediction sample for the current block using another motion vector from yet another picture. The process of predicting another block of prediction samples is the same as the process of predicting the first block of prediction samples. Furthermore, the padding step 1704 as described above includes the step of padding another block of prediction samples to form yet another block of prediction samples.

[0291] Figure 22 shows an embodiment 2200 of an image coding / decoding method that predicts the current block using two reference pictures. In this embodiment, in addition to using an additional prediction block as input, interpolation processing is also performed for subsample accuracy. The embodiment includes the following steps.

[0292] Step 2202: Predict the first and second blocks of prediction samples for the current block of the picture, using at least a prediction process with first and second motion vectors from different pictures.

[0293] Step 2204: Pad the first and second blocks of the prediction sample to form the third and fourth blocks of the prediction sample.

[0294] Step 2206: Perform interpolation on the third and fourth blocks of the prediction samples.

[0295] Step 2208: Calculate the gradient using the result of the interpolation process.

[0296] Step 2210: Encode / decode the current block using at least the result of the interpolation process and the calculated gradient.

[0297] Steps 2202 to 2210 are shown in the conceptual diagram 2300 in Figure 23. Embodiment 2200 of the encoding method, as shown in Figures 22 and 23, can be performed by the image encoding device 100. The decoding method performed by the image decoding device 200 is the same as the encoding method performed by the image encoding device 100 as shown in Figures 22 and 23.

[0298] In step 2202, the first block of predicted samples for the current block of the picture is predicted using at least a prediction process, which predicts samples using a first motion vector from another picture. In this embodiment, the first motion vector may point to a first picture that is different from the current picture.

[0299] Similarly, a second block of predicted samples for the current block of a picture is predicted using at least a prediction process, which predicts samples using a second motion vector from yet another picture. The second motion vector may point to a second picture, which is different from the current picture.

[0300] In one example, the first picture may be different from the second picture. In another example, the first picture may be the same as the second picture.

[0301] In one example, as shown in Figure 18A, at least one of the picture order counts (POCs) of the first and second pictures is smaller than the POC of the current picture, and the other picture order count (POC) of the first and second pictures is larger than the POC of the current picture.

[0302] In other examples, as shown in Figure 18B, the POCs of the first and second pictures may be smaller than the POC of the current picture.

[0303] In another example, as shown in Figure 18C, the POCs of the first and second pictures may be greater than the POC of the current picture.

[0304] The POCs of the first picture, second picture, and current picture shown in Figures 18A to 18C are in the time domain and indicate the encoding order of these pictures in the data stream. The encoding order may be the same as or different from the playback order of these pictures in the data stream (e.g., video clips). Those skilled in the art will understand that the POC may also refer to the order of these pictures in the spatial domain.

[0305] In one example, the first and second blocks of the prediction samples predicted in step 2202 may be the same as the reference blocks used in the motion compensation process performed for prediction modes such as merge mode and interprediction mode.

[0306] In one example, the first and second blocks of the prediction sample may be equal to or greater than the current block.

[0307] As shown in Figure 23, an embodiment of prediction step 2202 in Figure 22 is shown in step 2302. In this embodiment, the current block has a size of M × N, where M may be the same as or different from N. The interpretation unit 126 of the encoding device 100 may predict a first block and a second block of prediction samples, each having a size of (M + d1) × (N + d2), where d1 and d2 may be equal to 0 or greater than 0. An example of the first block of prediction samples is shown by a dotted line. The second block of prediction samples has the same size as the first block. It will be understood that the second block of prediction samples may have a different size from the first block of prediction samples.

[0308] Since more information can be included in the prediction sample, predicting the first and second blocks of a prediction sample that is larger than the current block seems more advantageous and contributes to more accurate prediction results.

[0309] In step 2204, the first block of the prediction sample is padded to form the third block of the prediction sample, and similarly, the second block of the prediction sample is padded to form the fourth block of the prediction sample.

[0310] In one example, the third block of the prediction sample is larger than the first block, and the fourth block is larger than the second block. As shown in padding step 2304 in Figure 23, the third and fourth blocks each have a size of (M+d1+d3)×(N+d2+d4), where d3 and d4 are greater than 0. An example of the third block of the prediction sample is shown by the dotted line. The fourth block of the prediction sample has the same size as the third block. As mentioned above, it will be understood that the fourth block of the prediction sample may have a different size from the third block of the prediction sample.

[0311] In one example, as shown in Figure 20A, the padding step may include a mirroring step of the first and second blocks of the prediction sample. In another example, as shown in Figure 20B, the padding step may include a copying step of the first and second blocks of the prediction sample. In yet another example, the padding step may include a step of padding with a fixed value, as shown in Figure 20C. The fixed value may be at least one of 0, 128, 512, a positive integer, the mean of the prediction sample, and the median of the prediction sample. In some examples, the padding step may include a step of applying a function to the prediction sample. Examples of functions may be filters, polynomial functions, exponential functions, and clipping functions. In some examples, as shown in Figure 20D, the padding step may include any combination of mirroring, copying, fixed-value padding, and applying a function to the prediction sample.

[0312] In step 2206, interpolation is performed on the third and fourth blocks of the predicted sample. The interpolation may include applying interpolation filters to the third and fourth blocks of the predicted sample according to the first and second motion vectors, respectively.

[0313] In one example, the interpolation filter may be the same as the filter used in the motion compensation process performed on the prediction mode, such as merge mode or interprediction mode. In another example, the interpolation filter may be different from the filter used in the motion compensation process performed on the prediction mode as described above.

[0314] As shown in step 2306 of Figure 23, the interpolation process performed on the third and fourth blocks of the prediction sample generates blocks resulting from the interpolation process. Each resulting block has a size of (M+d5)×(N+d6), where d5 and d6 are greater than 0. An example of a block resulting from the interpolation process performed on the third block of the prediction sample is shown by the dotted line. The block resulting from the interpolation process performed on the fourth block of the prediction sample is similar. A portion of each resulting block may be used in the processing of the current block, where the size of this portion is M×N. This portion may be the same as the block resulting from the motion compensation process performed on prediction modes such as merge mode and interprediction mode.

[0315] In step 2208, the gradient is calculated using the blocks resulting from the interpolation process. The gradient calculation may include the step of applying a gradient filter to the sample values ​​in the blocks resulting from the interpolation process in order to generate the derivative values. As shown in step 2308 of Figure 23, the calculated gradient may form M × N size blocks.

[0316] In one example, as shown in Figure 21A, there may be only one gradient filter applied to all sample values ​​within the resulting block, regardless of the fractional parts of the first and second motion vectors.

[0317] In one example, as shown in Figure 21B, there may be nine gradient filters applied to the sample values ​​within the resulting block, depending on the fractional parts of the first and second motion vectors.

[0318] In step 2210, the current block is encoded using at least the block resulting from the interpolation process and the calculated gradient. An example of an encoded block is shown in step 2310 in Figure 23.

[0319] Furthermore, the terms "encoding" and "processing" used in the description of the encoding method performed by the image encoding device 100 in step 2210 are interchangeable with the term "decoding" used in the description of the decoding method performed by the image decoding device 200 in step 2210.

[0320] In this embodiment, by introducing padding into interpretation, the encoding and decoding methods favorably reduce memory bandwidth access for interpretation. In addition, by using padding, this embodiment can generate sufficient prediction results with only one interpolation operation. Thus, this application favorably eliminates additional interpolation from interpretation, which is hardware-friendly.

[0321] Furthermore, in the interpolation processing performed on the third and fourth blocks, interpolation filters with different numbers of taps may be used. For example, in the above example, only an 8-tap interpolation filter is used to obtain a block resulting from interpolation processing with a size of (M+d5)×(N+d6). However, the number of taps in the interpolation filter used to obtain the first region of size M×N within the block resulting from the interpolation processing may differ from the number of taps in the interpolation filter used to obtain the second region, which is a region within the block resulting from interpolation processing other than the first region and is used to generate a gradient block. For example, the number of taps in the interpolation filter used to obtain the second region may be less than the number of taps in the interpolation filter used to obtain the first region. Specifically, if the number of taps in the interpolation filter used to obtain the first region is 8, the number of taps in the interpolation filter used to obtain the second region may be less than 8. Therefore, processing load can be reduced while preventing degradation of image quality.

[0322] Another embodiment of an image encoding / decoding method that predicts the current block using two reference pictures is shown in Figure 24. In this embodiment, the interpolation process is performed before the padding process. The embodiment includes the following steps:

[0323] Step 2402: Predict the first and second blocks of prediction samples for the current block of a picture, using at least a prediction process with first and second motion vectors from different pictures.

[0324] Step 2404: Perform interpolation on the first and second blocks of the prediction samples.

[0325] Step 2406: Padding the interpolation results to form the third and fourth blocks of the predicted samples.

[0326] Step 2408: Calculate the gradient using the third and fourth blocks of the prediction sample.

[0327] Step 2410: Encode / decode the current block using at least the result of the interpolation process and the calculated gradient.

[0328] Steps 2402 to 2410 are shown in the conceptual diagram of Figure 25. Embodiment 2400 of the encoding method, as shown in Figures 24 and 25, can be performed by the image encoding device 100. The decoding method performed by the image decoding device 200 is the same as the encoding method performed by the image encoding device 100 as shown in Figures 24 and 25.

[0329] In step 2402, the first block of predicted samples for the current block of the picture is predicted using at least a prediction process, which predicts samples for the current block using a first motion vector from another picture. In this embodiment, the first motion vector may point to a first picture that is different from the current picture.

[0330] Similarly, a second block of predicted samples for the current block of a picture is predicted using at least a prediction process, which predicts samples using a second motion vector from yet another picture. The second motion vector may point to a second picture, which is different from the current picture.

[0331] In one example, the first picture may be different from the second picture. In another example, the first picture may be the same as the second picture.

[0332] In one example, as shown in Figure 18A, at least one of the picture order counts (POCs) of the first and second pictures is smaller than the POC of the current picture, and the other picture order count (POC) of the first and second pictures is larger than the POC of the current picture.

[0333] In other examples, as shown in Figure 18B, the POCs of the first and second pictures may be smaller than the POC of the current picture.

[0334] In another example, as shown in Figure 18C, the POCs of the first and second pictures may be greater than the POC of the current picture.

[0335] The POCs of the first picture, second picture, and current picture shown in Figures 18A to 18C are in the time domain and indicate the encoding order of these pictures in the data stream. The encoding order may be the same as or different from the playback order of these pictures in the data stream (e.g., video clips). Those skilled in the art will understand that the POC may also refer to the order of these pictures in the spatial domain.

[0336] In one example, the first and second blocks of the prediction samples predicted in step 2402 may be the same as the reference blocks used in the motion compensation process performed for prediction modes such as merge mode and interprediction mode.

[0337] In one example, the first and second blocks of the prediction sample may be equal to or greater than the current block.

[0338] As shown in Figure 25, an embodiment of prediction step 2402 in Figure 24 is shown in step 2502. In this embodiment, the current block has a size of M × N, where M may be the same as or different from N. The interpretation unit 126 of the encoding device 100 may predict a first block and a second block of prediction samples, each having a size of (M + d1) × (N + d2), where d1 and d2 may be equal to 0 or greater than 0. An example of the first block of prediction samples is shown by a dotted line. The second block of prediction samples has the same size as the first block. As described above, it will be understood that the second block of prediction samples may have a different size from the first block of prediction samples.

[0339] Since more information can be included in the prediction sample, predicting the first and second blocks of a prediction sample that is larger than the current block seems more advantageous and contributes to more accurate prediction results.

[0340] In step 2404, interpolation is performed on the first and second blocks of the prediction sample. The interpolation may include the step of applying interpolation filters to the first and second blocks of the prediction sample according to the first and second motion vectors, respectively.

[0341] In one example, the interpolation filter may be the same as the filter used in the motion compensation process performed on the prediction mode, such as merge mode or interprediction mode. In another example, the interpolation filter may be different from the filter used in the motion compensation process performed on the prediction mode as described above.

[0342] As shown in step 2504 of Figure 25, the interpolation process performed on the first and second blocks of the prediction sample generates the resulting blocks. The size of each resulting block is M × N. The resulting blocks may be used to process the current block, and may be the same as the blocks resulting from motion compensation performed on prediction modes such as merge mode and interprediction mode.

[0343] In step 2406, the blocks resulting from the interpolation process performed on the first block are padded to form the third block of the prediction sample. The third block of the prediction sample is larger than the current block. Similarly, the blocks resulting from the interpolation process performed on the second block are padded to form the fourth block of the prediction sample. The fourth block of the prediction sample is larger than the current block.

[0344] As shown in step 2506 of Figure 25, an example of the third block of prediction samples may have a size of (M+d3)x(N+d4), where d3 and d4 are greater than 0. The fourth block of prediction samples has the same size. As mentioned above, it will be understood that the fourth block of prediction samples may have a different size from the third block of prediction samples.

[0345] In one example, as shown in Figure 20A, the padding step may include a step of mirroring the block resulting from the interpolation process. In another example, as shown in Figure 20B, the padding step may include a step of copying the block resulting from the interpolation process. In yet another example, as shown in Figure 20C, the padding step may include a step of padding with a fixed value. The fixed value may be at least one of 0, 128, 512, a positive integer, the mean of the predicted samples, and the median of the predicted samples. In some examples, the padding step may include a step of applying a function to the block resulting from the interpolation process. Examples of functions may be filters, polynomial functions, exponential functions, and clipping functions. In yet another example, as shown in Figure 26, the padding step may include a step of using the predicted samples of the first and second blocks.

[0346] In other examples, the padding step may include a step of applying a second interpolation filter to the first and second blocks. Unlike the interpolation filter performed in step 2404, the second interpolation filter has fewer taps than the interpolation filter performed in step 2404. In other examples, as shown in Figure 20D, the padding step may include any combination of mirroring, copying, fixed-value padding, execution of a function on the resulting blocks, padding of samples using the first and second blocks, and application of the second interpolation filter to the first and second blocks.

[0347] In step 2408, the gradient is calculated using the third and fourth blocks of the prediction sample. The gradient calculation may include the step of applying a gradient filter to the sample values ​​in the third and fourth blocks of the prediction sample to generate derivative values. The gradient filter is as described above with respect to Figures 21A and 21B. As shown in step 2508 of Figure 25, the calculated gradient may form an M × N size block.

[0348] In step 2410, the current block is encoded using at least the block resulting from the interpolation process and the calculated gradient. An example of an encoded block is shown in step 2510 of Figure 25.

[0349] Furthermore, the terms "encoding" and "processing" used in the description of the encoding method performed by the image encoding device 100 in step 2410 are interchangeable with the term "decoding" used in the description of the decoding method performed by the image decoding device 200 in step 2410.

[0350] In this embodiment, by introducing padding into interprediction, the encoding and decoding methods favorably reduce memory bandwidth access for interprediction. In addition, by using padding, this embodiment can generate sufficient prediction results with only one interpolation operation. Thus, this application favorably eliminates additional interpolation from interprediction, which is hardware-friendly. This may retain the same number of interpolation filter operations used in motion compensation operations performed for prediction modes such as merge mode and interprediction mode.

[0351] Figure 27A is a flowchart showing another alternative example of an image coding / decoding method that uses interpretation to generate a prediction for the current block of a picture.

[0352] Steps 2702A to 2708A in Figure 27A are shown in the conceptual diagram in Figure 28. Embodiment 2700A of the encoding method shown in Figures 27A and 28 can be performed by the image encoding device 100. The decoding method performed by the image decoding device 200 is the same as the encoding method performed by the image encoding device 100 shown in Figures 27A and 28.

[0353] As shown in Example 2700A, steps 2702A and 2704A are the same as steps 1702 and 1704 as described with respect to Figure 17. An embodiment of the first block of prediction samples predicted in step 2702A of Figure 27A is shown in step 2802 of Figure 28. In the embodiment shown in step 2802, a first block of prediction samples of size (M+d1)×(N+d2) is predicted for a current block of size M×N. It will be understood by those skilled in the art that M may be the same as or different from N, while d1 and d2 are greater than 0, equal to 0, or less than 0.

[0354] An embodiment of a second block of a prediction sample formed by padding a first block of prediction samples as shown in step 2704A of Figure 27A is shown in step 2804 of Figure 28. As shown in step 2804 of Figure 28, the padding step 2704A of Figure 27A can be designed with care to pad a first block of prediction samples of size (M+d1)×(N+d2) to form a second block of prediction samples having any desired size (M+d1+d3)×(N+d2+d4), based on actual prediction accuracy needs based on different padding directions and / or processing / techniques described above for Figures 19A, 19B, and 20A to 20D. It will be understood by those skilled in the art that d3 and d4 are greater than 0.

[0355] In step 2706A, interpolation is performed on the second block of prediction samples. The interpolation may include the step of applying an interpolation filter to each second block of prediction samples according to the first motion vector. In one example, the interpolation filter may be the same as the filter used in motion compensation processing performed on prediction modes such as merge mode and interprediction mode. In another example, the interpolation filter may be different from the filter used in motion compensation processing performed on the prediction modes described above. An embodiment of the result of the interpolation processing performed in step 2706A in Figure 27A is shown in step 2806 in Figure 28.

[0356] In step 2708A, the current block is encoded using at least the block resulting from the interpolation process performed on the second block of the prediction sample. An embodiment of the current block encoded using the block resulting from the interpolation process, as shown in step 2708A of Figure 27A, is shown in step 2808 of Figure 28.

[0357] Furthermore, the terms "encoding" and "processing" used in the explanation of step 2708A for the encoding method performed by the image encoding device 100 are interchangeable with the term "decoding" used in step 2708A for the decoding method performed by the image decoding device 200.

[0358] Figure 27B is a flowchart showing another alternative example of an image coding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on the reference block of another picture and the adjacent blocks of the current block.

[0359] Steps 2702B to 2708B in Figure 27B are shown in the conceptual diagram in Figure 29. Embodiment 2700B of the encoding method, as shown in Figures 27B and 29, can be performed by the image encoding device 100. The decoding method performed by the image decoding device 200 is the same as the encoding method performed by the image encoding device 100 as shown in Figures 27B and 29.

[0360] Step 2701B predicts a first block of prediction samples for the current block of the picture, and the prediction step includes at least a prediction process using a first motion vector from another picture. An embodiment of the first block of prediction samples predicted in step 2702B of Figure 27B is shown in step 2902, as shown in Figure 29.

[0361] In step 2704B, the second block of prediction samples is padded to form a third block of prediction samples. The second block may be adjacent to the current block. An embodiment of the second block is shown in step 2904 of Figure 29. In this embodiment, the second block has a size of M × N, where M may be the same as or different from N.

[0362] As shown in step 2906 of Figure 29, the interpretation unit 126 of the encoding device 100 may be configured to perform step 2704B, padding the second block of prediction samples to form a third block of prediction samples. The third block of prediction samples may have a size of (M+d3)×N, where d3 may be equal to 0 or greater than 0. An example of the third block of prediction samples is shown by the dotted line.

[0363] In the example shown in Figure 29, the second block of the prediction sample has the same size as the first block. It will be understood that the second block of the prediction sample may have a different size from the first block of the prediction sample.

[0364] Furthermore, in the example in Figure 29, the second block is the left block adjacent to the first block. It should be understood that the second block may be at least one of the upper, left, right, lower, upper-left, upper-right, lower-left, and lower-right blocks adjacent to the first block. A description of the possible positions of the second block is shown in Figure 31, where the second block is adjacent to the current block.

[0365] In one example, as shown in Figure 32A, the padding in step 2704B may include mirroring of the samples in the second block. In another example, as shown in Figure 32B, the padding in step 2704B may include copying of the samples in the second block. In yet another example, as shown in Figure 32C, the padding in step 2704B may include fixed-value padding, where the fixed value may be at least one of 0, 128, 512, a positive integer, the mean of the second block, and the median of the second block. In yet another example, the padding in step 2704B may include the step of performing a function on the samples in the second block. Examples of functions may be filters, polynomial functions, exponential functions, and clipping functions. In yet another example, the padding in step 2704B may include any combination of mirroring, copying, first-value padding, and performing a function on the samples in the second block.

[0366] In the example shown in Figure 29, the padding in step 2704B may include sample padding for only one side of the second block, as shown in Figure 33A. In other examples, the padding in step 2704B may include sample padding for two sides of the second block, where the two sides of the second block are parallel, as shown in Figure 33C, or perpendicular, as shown in Figure 33B. In other examples, the padding in step 2704B may include sample padding for three or more sides of the second block, as shown in Figure 33D.

[0367] In step 2706B, OBMC (overlapped block motion compensation) processing is performed using at least the first and third blocks of the prediction sample. The OBMC processing is as described in the preceding paragraphs of this application. As shown in step 2908 of Figure 29, the OBMC processing generates an OBMC block OBMC.

[0368] In step 2708B, the current block is encoded using at least the results of the OBMC processing. An example of an encoded block is shown in step 2910 of Figure 29.

[0369] Furthermore, the terms "encoding" and "processing" used in the description of the encoding method performed by the image encoding device 100 in step 2708B are interchangeable with the term "decoding" used in the description of the decoding method performed by the image decoding device 200 in step 2708B.

[0370] In this embodiment, by introducing padding processing into the OBMC processing, the encoding and decoding methods advantageously reduce memory bandwidth access for interpretation processing.

[0371] Figure 27C is a flowchart showing yet another alternative example of an image coding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture.

[0372] Steps 2702C to 2710C in Figure 27C are shown in the conceptual diagram in Figure 30. Embodiment 2700C of the encoding method shown in Figures 27C and 30 can be performed by the image encoding device 100. The decoding method performed by the image decoding device 200 is the same as the encoding method performed by the image encoding device 100 as shown in Figures 24 and 25.

[0373] As shown in Example 2700C, steps 2702C and 2704C are the same as steps 1702 and 1704 as described with respect to Figure 17.

[0374] As shown in Figure 30, an embodiment of prediction step 2702C in Figure 27C is shown in step 3002. In this embodiment, the current block has a size of M × N, where M may be the same as or different from N. The interpretation unit 126 of the encoding device 100 may predict a first block of prediction samples having a size of (M + d1) × (N + d2), where d1 and d2 may be equal to 0 or greater than 0. As shown in step 3002 of Figure 30, the current block may have one or more adjacent blocks that can be used later in this example of the method.

[0375] As shown in Figure 30, predicting the first block of a prediction sample that is larger than the current block seems more advantageous because it allows for the inclusion of more information in the prediction sample, and contributes to more accurate prediction results. An example of the first block of a prediction sample is shown by the dotted line in step 3002 of Figure 30.

[0376] As shown in step 3004 of Figure 30, the interpretation unit 126 of the encoding device 100 may be configured to perform step 2704C, padding the first block of prediction samples to form a second block of prediction samples. The second block of prediction samples may have a size of (M+d1+d3)×(N+d2+d4), where d3 and d4 may be equal or greater than 0. An example of the second block of prediction samples is shown by a thin dotted line in step 3004 of Figure 30, where an example of the first block of prediction samples is shown by a thick dotted line.

[0377] As shown in Figure 30, since more information can be included in the padding block of the prediction sample, forming a second block of prediction samples that is larger than the first block seems more advantageous and contributes to more accurate prediction results.

[0378] In step 2706C, interpolation is performed on the second block of prediction samples. The interpolation may include the step of applying an interpolation filter to the second block of prediction samples according to the first motion vector. In one example, the interpolation filter may be the same as the filter used in motion compensation performed for prediction modes such as merge mode and interprediction mode. In other examples, the interpolation filter may be different from the filter used in motion compensation performed for the prediction modes described above.

[0379] As shown in step 3006 of Figure 30, the interpolation process performed on the second block of prediction samples may generate a resulting block. The resulting block may have a size of (M+d5)×(N+d6), where d5 and d6 are greater than 0. An example of a resulting block from the interpolation process performed on the third block of prediction samples is shown by the dotted line in step 3006 of Figure 30. A portion of the resulting block may be used for processing the current block. Here, the size of this portion may be M×N, as shown in step 3008 of Figure 30. This portion may be the same as the resulting block from the motion compensation process performed on prediction modes such as merge mode and interprediction mode.

[0380] In step 2708C, the current block is encoded using at least the block resulting from the interpolation process performed on the second block of the prediction samples. An example of an encoded block is shown in step 3008 of Figure 30.

[0381] Simultaneously with step 2708C, following step 2708C, or prior to step 2708C, OBMC processing is performed on one or more adjacent blocks of the current block in step 2710C. The OBMC processing may use at least the blocks resulting from the interpolation process.

[0382] The OBMC processing in step 2708C is as described in the preceding paragraphs of this application. As shown in step 2906 of Figure 29, the OBMC processing generates one or more OBMC blocks between one or more adjacent blocks and the current block. Using one or more OBMC blocks between one or more adjacent blocks and the current block generated at once, the method of the present disclosure advantageously reduces the memory bandwidth access (i.e., data fetched from off-chip memory, DRAM) required for OBMC processing.

[0383] Furthermore, the terms "encoding" and "processing" used in the description of the encoding method performed by the image encoding device 100 in step 2708C are interchangeable with the term "decoding" used in the description of the decoding method performed by the image decoding device 200 in step 2708C.

[0384] In this disclosure, the blocks of prediction samples described in the above examples and embodiments may be replaced with non-rectangular portions of the prediction samples. Examples of non-rectangular portions may be at least one of the following: triangular portions, L-shaped portions, pentagonal portions, hexagonal portions, and polygonal portions, as shown in Figure 34.

[0385] Those skilled in the art will understand that the non-rectangular shapes are not limited to the shapes shown in Figure 34. Furthermore, the shapes shown in Figure 34 may be freely combined.

[0386] Figure 35 is a flowchart showing another alternative example of an image coding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture.

[0387] Steps 3502 to 3508 in Figure 35 are shown in the conceptual diagram in Figure 36. Embodiment 3500 of the encoding method, as shown in Figures 35 and 36, can be performed by the image encoding device 100. It will be understood that the decoding method performed by the image decoding device 200 is the same as the encoding method performed by the image encoding device 100 as shown in Figures 35 and 36.

[0388] In step 3502, a first block of prediction samples is predicted for the current block of the picture, and the prediction step includes at least a prediction process using a first motion vector from another picture. As shown in Figure 36, an embodiment of the first block of prediction samples predicted in step 3502 of Figure 35 is shown in step 3602. In this embodiment, the current block has a size of M × N, where M may be the same as or different from N. The interpretation unit 126 of the encoding device 100 may predict a first block of prediction samples having a size of (M + d1) × (N + d2), where d1 and d2 may be equal to 0 or greater than 0.

[0389] In step 3504, a second motion vector for the current block is derived using at least the first motion vector by DMVR processing as described in the previous paragraph. An embodiment of the second motion vector is shown in step 3604 of Figure 36. In this embodiment, the second motion vector is shown by a dotted line pointing from the current block to the first block.

[0390] In step 3506, interpolation may be performed on the current block using the second motion vector. The interpolation may include padding. In the embodiment of step 3506, the interpretation unit 126 of the encoding device 100 may be configured to pad the first block of prediction samples according to the second motion vector to generate a second block of prediction samples, and to perform interpolation on the second block of prediction samples using at least the second motion vector. As shown in step 3606 of Figure 36, the second block of prediction samples may have a size of (M+d1+d3)×(N+d2+d4). Here, d3 and d4 may be equal to 0 or greater than 0.

[0391] In the embodiment shown in Figure 36, the second block is an L-shaped portion adjacent to the first block, as shown in Figure 34. It should also be understood that the second block may be at least one of the upper block, left block, right block, lower block, upper left block, upper right block, lower left block, and lower right block adjacent to the first block. Alternatively, the second block may be a triangular portion, an L-shaped portion, a pentagonal portion, a hexagonal portion, and a polygonal portion adjacent to the first block.

[0392] Figures 37A and 37B show two examples of the padding process in step 3506.

[0393] In the example shown in Figure 37A, the padding process in step 3506 may include the step of using the second motion vector derived in step 3504 to pad the sample with a sample on only one side of the first block to form a second block of the predicted sample.

[0394] In the example shown in Figure 37B, the padding process in step 3506 may include a step of padding the sample with two sides of the first block using the second motion vector derived in step 3504 to form a second block of the predicted sample. In the example in Figure 37B, the two sides of the first block are orthogonal, as shown in Figure 33B. Alternatively, the two sides of the first block are parallel, as shown in Figure 33C. In other examples, as shown in Figure 33D, the padding process in step 3506 may include a step of padding the sample with three or more sides of the second block.

[0395] In some examples, as shown in Figure 32A, the padding process in step 3506 may include a step of mirroring the samples of the first block. In other examples, as shown in Figure 32B, the padding process in step 3506 may include a step of copying the samples of the second block. In other examples, as shown in Figure 32C, the padding process in step 3506 may include a step of padding with a fixed value, where the fixed value may be at least one of 0, 128, 512, a positive integer, the mean of the second block, and the median of the second block. In other examples, the padding process in step 3506 may include a step of running a function on the samples of the second block. Examples of functions may be a filter, a polynomial function, an exponential function, and a clipping function. In other examples, the padding process in step 3506 may include any combination of mirroring, copying, padding with the first value, and running a function on the samples of the second block.

[0396] Interpolation is performed on the second block of prediction samples as shown in step 3606. The interpolation may include the step of applying an interpolation filter to the second block of prediction samples according to the second motion vector. In one example, the interpolation filter may be the same as the filter used in motion compensation performed for prediction modes such as merge mode and interprediction mode. In other examples, the interpolation filter may be different from the filter used in motion compensation performed for the prediction modes described above.

[0397] As shown in step 3606 of Figure 36, the interpolation process performed on the second block of prediction samples may generate a resulting block. The size of the resulting block is M × N.

[0398] In step 3508, the current block is encoded using at least the block resulting from the interpolation process performed on the second block of the prediction sample in step 3506. An example of an encoded block is shown in step 3608 of Figure 36.

[0399] Furthermore, the terms "encoding" and "processing" used in the description of the encoding method performed by the image encoding device 100 in step 3508 are interchangeable with the term "decoding" used in the description of the decoding method performed by the image decoding device 200 in step 3508.

[0400] In this embodiment, by introducing padding processing into the OBMC processing, the encoding and decoding methods advantageously reduce memory bandwidth access for DMVR processing.

[0401] Figure 38 is a flowchart showing yet another alternative example of an image coding / decoding method that uses interpretation to generate a prediction of the current block of a picture based on a reference block of another picture.

[0402] Steps 3802, 3804, 3806, and 3808 in Figure 38 are shown in the conceptual diagram in Figure 39. Embodiment 3800 of the encoding method as shown in Figures 38 and 39 can be performed by the image encoding device 100. It will be understood that the decoding method performed by the image decoding device 200 is the same as the encoding method performed by the image encoding device 100 as shown in Figures 38 and 39.

[0403] As described above, steps 3802, 3804, 3806, and 3808 in Figure 38 are the same as the steps in Figure 35, except that the DMVR (dynamic motion vector refreshing) process in step 3804 further includes a padding process. In the embodiment shown in Figure 39, the padding process in step 3804 includes a step of padding a first block of prediction samples according to a first motion vector. In this regard, in step 3804, a second motion vector for the current block is derived using at least a first motion vector based on the padded first block by the DMVR process described in the previous paragraph. An embodiment of the second motion vector is shown in step 3904 of Figure 39 by a dotted line pointing from the current block to the first block.

[0404] Furthermore, the terms "encoding" and "processing" used in the description of the encoding method performed by the image encoding device 100 in step 3808 are interchangeable with the term "decoding" used in the description of the decoding method performed by the image decoding device 200 in step 3808.

[0405] In this embodiment, by introducing padding processing into the OBMC processing, the encoding and decoding methods advantageously reduce memory bandwidth access for DMVR processing.

[0406] In this application, the term "block" as described in the above examples and embodiments may be replaced with the term "prediction unit." Furthermore, the term "block" as described in each aspect may be replaced with the term "sub-prediction unit." Furthermore, the term "block" as described in each aspect may be replaced with the term "coding unit."

[0407] [Implementation and Application] In each of the above embodiments, each functional or operational block can typically be implemented by an MPU (micro processing unit) and memory, etc. Furthermore, the processing performed by each functional block may be implemented as a program execution unit, such as a processor, that reads and executes software (programs) recorded on a recording medium such as ROM. This software may be distributed. This software may be recorded on various recording media such as semiconductor memory. It is also possible to implement each functional block using hardware (dedicated circuits). Various combinations of hardware and software can be employed.

[0408] The processing described in each embodiment may be implemented by centralized processing using a single device (system), or by distributed processing using multiple devices. Furthermore, the processor executing the above program may be one or multiple. In other words, centralized processing may be performed, or distributed processing may be performed.

[0409] The embodiments of this disclosure are not limited to those described above, and various modifications are possible, which are also included within the scope of the embodiments of this disclosure.

[0410] Furthermore, here we will describe application examples of the video encoding method (image encoding method) or video decoding method (image decoding method) shown in each of the above embodiments, and various systems for implementing these application examples. Such systems may be characterized by having an image encoding device using the image encoding method, an image decoding device using the image decoding method, or an image encoding and decoding device that includes both. Other configurations of such systems can be appropriately modified as needed.

[0411] [Usage example] Figure 40 shows the overall configuration of a suitable content supply system ex100 for realizing a content distribution service. The service area for the communication service is divided into cells of a desired size, and within each cell, there are base stations ex106, ex107, ex108, ex109, and ex110, which are fixed radio stations in the illustrated example.

[0412] In this content supply system ex100, various devices such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the internet ex101 via an internet service provider ex102 or a communication network ex104, and base stations ex106~ex110. The content supply system ex100 may also be configured to connect any combination of the above devices. In various implementations, the devices may be directly or indirectly interconnected via a telephone network or short-range wireless, etc., without going through base stations ex106~ex110. Furthermore, the streaming server ex103 may be connected to various devices such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 via the internet ex101, etc. The streaming server ex103 may also be connected to terminals in a hotspot on an airplane ex117 via satellite ex116.

[0413] Note that instead of base stations ex106~ex110, wireless access points or hotspots may be used. Also, streaming server ex103 may be connected directly to the communication network ex104 without going through the internet ex101 or internet service provider ex102, or it may be connected directly to the airplane ex117 without going through satellite ex116.

[0414] The camera ex113 is a device capable of taking still images and videos, such as a digital camera. The smartphone ex115 is a smartphone, mobile phone, or PHS (Personal Handy-phone System) that supports mobile communication systems such as 2G, 3G, 3.9G, 4G, and the upcoming 5G.

[0415] Home appliance ex114 refers to appliances such as refrigerators or equipment included in household fuel cell cogeneration systems.

[0416] In the content supply system ex100, live streaming becomes possible when a terminal with a shooting function is connected to the streaming server ex103 via a base station ex106 or the like. In live streaming, a terminal (such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and a terminal inside an airplane ex117) may perform the encoding process described in each of the above embodiments on still images or video content captured by a user using the terminal, or it may multiplex the video data obtained by encoding with sound data encoded from the sound corresponding to the video, and then transmit the obtained data to the streaming server ex103. In other words, each terminal functions as an image encoding device according to one aspect of this disclosure.

[0417] Meanwhile, the streaming server ex103 streams the content data sent to the requesting client. The client is a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or a terminal on an airplane ex117, etc., that is capable of decoding the encoded data. Each device that receives the distributed data may decode and play back the received data. That is, each device may function as an image decoding device according to one aspect of this disclosure.

[0418] [Distributed Processing] Furthermore, the streaming server ex103 may consist of multiple servers or computers that distribute data processing, recording, and distribution. For example, the streaming server ex103 may be implemented by a CDN (Content Delivery Network), where content delivery is achieved through a network connecting numerous edge servers distributed worldwide. In a CDN, the physically closest edge server can be dynamically assigned depending on the client. Latency can be reduced by caching and delivering content to the edge server. In addition, if several types of errors occur or the communication state changes due to increased traffic, processing can be distributed among multiple edge servers, the delivery entity can be switched to another edge server, or delivery can be continued by bypassing the failed part of the network, thus enabling high-speed and stable delivery.

[0419] Furthermore, beyond the distributed processing of the distribution itself, the encoding process of the captured data can be performed on each terminal, on the server side, or shared among them. For example, encoding generally involves two processing loops. In the first loop, the complexity or code amount of the image at the frame or scene level is detected. In the second loop, processing is performed to improve encoding efficiency while maintaining image quality. For example, if the terminal performs the first encoding process and the server that receives the content performs the second encoding process, it is possible to improve the quality and efficiency of the content while reducing the processing load on each terminal. In this case, if there is a request to receive and decode near real time, the first encoded data from the terminal can be received and played back on other terminals, enabling more flexible real-time distribution.

[0420] Another example is the camera ex113, which extracts features (quantities of features or characteristics) from an image, compresses the data related to the features as metadata, and sends it to the server. The server performs compression according to the meaning (or importance of content) of the image, for example, by determining the importance of objects from the features and switching the quantization precision. Feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction during further compression on the server. Alternatively, a simple encoding such as VLC (Variable Length Coding) may be performed on the terminal, and a more computationally intensive encoding such as CABAC (Context-Adaptive Binary Arithmetic Coding) may be performed on the server.

[0421] Another example is a scenario in a stadium, shopping mall, or factory where multiple video data sets of nearly identical scenes may exist, captured by multiple terminals. In such cases, the encoding process is distributed among the multiple terminals that captured the footage, along with other terminals and servers as needed, by assigning encoding tasks to each unit, for example, at the Group of Picture (GOP) level, picture level, or tile level (a division of a picture). This reduces latency and enables more real-time performance.

[0422] Since multiple video data sets depict essentially the same scene, the server may manage and / or instruct the video data captured by each terminal to reference each other. Alternatively, the server may receive the encoded data from each terminal, change the reference relationships between the multiple data sets, or correct or replace the pictures themselves and re-encode them. This allows for the creation of a stream with improved quality and efficiency for each individual data set.

[0423] Furthermore, the server may transcode the video data to change its encoding method before distributing it. For example, the server may convert an MPEG-based encoding to a VP-based encoding (e.g., VP9), or convert H.264 to H.265, etc.

[0424] Thus, the encoding process can be performed by a terminal or one or more servers. Therefore, in the following, the terms "server" or "terminal" will be used to refer to the entity performing the processing, but some or all of the processing performed by the server may be performed by the terminal, and some or all of the processing performed by the terminal may be performed by the server. The same applies to the decoding process.

[0425] [3D, Multi-angle] It is becoming increasingly common to integrate and utilize images or videos of different scenes, or the same scene, captured from different angles, by multiple cameras ex113 and / or smartphones ex115, which are nearly synchronized with each other. Videos captured by each device can be integrated based on the relative positional relationship between the devices, or on areas where feature points contained in the video coincide.

[0426] The server may not only encode two-dimensional video but also encode still images automatically based on scene analysis of the video, or at a time specified by the user, and transmit them to the receiving terminal. Furthermore, if the server can obtain the relative positional relationship between the shooting terminals, it can generate a three-dimensional shape of the scene based not only on two-dimensional video but also on video of the same scene taken from different angles. The server may separately encode three-dimensional data generated by a point cloud or the like, or it may select or reconstruct video from video taken by multiple terminals to transmit to the receiving terminal based on the results of recognizing or tracking a person or object using the three-dimensional data.

[0427] In this way, users can enjoy scenes by arbitrarily selecting each video corresponding to each shooting terminal, or they can enjoy content in which a video from a selected viewpoint is extracted from 3D data reconstructed using multiple images or videos. Furthermore, along with the video, sound is also collected from multiple different angles, and the server may multiplex the sound from a specific angle or space with the corresponding video and transmit the multiplexed video and sound.

[0428] In recent years, content that links the real world with a virtual world, such as Virtual Reality (VR) and Augmented Reality (AR), has also become popular. In the case of VR images, the server may create separate viewpoint images for the right and left eyes and perform encoding that allows referencing between the viewpoint images using Multi-View Coding (MVC), or it may encode them as separate streams without referencing each other. When decoding the separate streams, it is advisable to synchronize playback so that the virtual 3D space is reproduced according to the user's viewpoint.

[0429] In the case of AR images, the server may superimpose virtual object information from the virtual space onto camera information from the real space, based on its three-dimensional position or the user's viewpoint movement. The decoding device may acquire or retain the virtual object information and three-dimensional data, generate a two-dimensional image according to the user's viewpoint movement, and create superimposed data by smoothly stitching them together. Alternatively, the decoding device may send the user's viewpoint movement to the server in addition to the request for virtual object information. The server may create superimposed data according to the viewpoint movement received from the three-dimensional data held by the server, encode the superimposed data, and distribute it to the decoding device. Typically, superimposed data has an α value indicating transparency in addition to RGB, and the server may set the α value of parts other than the object created from the three-dimensional data to 0, and encode the data in a state where those parts are transparent. Alternatively, the server may set predetermined RGB values ​​as the background, like in chroma keying, and generate data where parts other than the object are the background color. The predetermined RGB values ​​may be predetermined.

[0430] Similarly, the decryption process of the distributed data can be performed by the client (e.g., a terminal), the server, or a shared task between them. For example, one terminal may send a reception request to the server, another terminal may receive the content corresponding to that request, decrypt it, and then transmit the decrypted signal to a device with a display. By distributing the processing and selecting appropriate content regardless of the performance of the communication-capable terminals themselves, it is possible to play back data with high image quality. Another example is that while receiving large image data on a TV or similar device, a portion of the picture, such as tiles, may be decrypted and displayed on the viewer's personal terminal. This allows for sharing the overall picture while simultaneously allowing users to check their own area of ​​responsibility or areas they wish to examine in more detail.

[0431] In situations where multiple short-range, medium-range, or long-range wireless communication networks are available both indoors and outdoors, it may be possible to seamlessly receive content using distribution system standards such as MPEG-DASH. Users may freely select and switch in real time between decoding devices or display devices, such as their own terminals or displays located indoors or outdoors. Furthermore, decoding can be performed while switching between the decoding terminal and the display terminal using the user's location information. This makes it possible to map and display information on a part of the wall or ground of an adjacent building with a displayable device embedded, while the user is moving to their destination. It is also possible to switch the bitrate of the received data based on the ease of access to the encoded data on the network, such as when the encoded data is cached on a server that can be accessed quickly from the receiving terminal, or copied to an edge server in the content delivery service.

[0432] [Scalable encoding] Regarding content switching, we will explain using a scalable stream compressed and encoded using the video encoding method described in each of the embodiments above, as shown in Figure 41. The server may have multiple streams with the same content but different qualities as individual streams, but it may also be configured to switch content by taking advantage of the temporal / spatial scalability of the stream realized by encoding it in layers, as shown in the figure. In other words, the decoding side can freely switch between decoding low-resolution and high-resolution content by deciding which layer to decode according to internal factors such as performance and external factors such as the state of the communication bandwidth. For example, if a user was watching a video on their smartphone ex115 while on the go and wants to continue watching it on a device such as an internet TV after returning home, the device only needs to decode the same stream up to a different layer, thus reducing the burden on the server.

[0433] Furthermore, as described above, in addition to a configuration where each layer encodes a picture and scalability is achieved by an enhancement layer above the base layer, the enhancement layer may also include metadata based on image statistics. The decoding side may generate high-quality content by super-resolution the picture in the base layer based on the metadata. Super-resolution may improve the signal-to-noise ratio while maintaining and / or increasing the resolution. The metadata may include information for identifying linear or nonlinear filter coefficients used in the super-resolution process, or information for identifying parameter values ​​in the filtering process, machine learning, or least-squares operation used in the super-resolution process.

[0434] Alternatively, a configuration may be provided in which the picture is divided into tiles or the like according to the meaning of objects within the image. The decoding side decodes only a portion of the area by selecting the tiles to decode. Furthermore, by storing the attributes of the objects (people, cars, balls, etc.) and their positions in the image (coordinate positions within the same image, etc.) as metadata, the decoding side can identify the position of the desired object based on the metadata and determine the tile containing that object. For example, as shown in Figure 42, the metadata may be stored using a data storage structure different from the pixel data, such as the SEI (supplemental enhancement information) message in HEVC. This metadata indicates, for example, the position, size, or color of the main object.

[0435] Metadata may be stored in units consisting of multiple pictures, such as streams, sequences, or random access units. The decryption side can obtain information such as the time when a specific person appears in the video, and by combining the picture-level information with the time information, it can identify the picture in which the object exists and determine the object's position within the picture.

[0436] [Web page optimization] Figure 43 shows an example of a web page display screen on a computer ex111, etc. Figure 44 shows an example of a web page display screen on a smartphone ex115, etc. As shown in Figures 43 and 44, a web page may contain multiple linked images, which are links to image content, and their appearance may differ depending on the viewing device. When multiple linked images are visible on the screen, the display device (decoder) may display still images or I-pictures of each content as linked images until the user explicitly selects a linked image, or until a linked image approaches the center of the screen or the entire linked image is within the screen, or it may display a video such as a GIF animation using multiple still images or I-pictures, or it may receive only the base layer and decode and display the video.

[0437] When a linked image is selected by the user, the display device performs decoding, prioritizing the base layer, for example. If the HTML of the web page contains information indicating that the content is scalable, the display device may decode up to the enhancement layer. Furthermore, to ensure real-time performance, before selection or when the communication bandwidth is very limited, the display device can decode and display only forward-referenced pictures (I pictures, P pictures, and B pictures that only use forward references), thereby reducing the delay between the decoding time and the display time of the first picture (the delay from the start of content decoding to the start of display). In addition, the display device may deliberately ignore the reference relationships of the pictures and roughly decode all B pictures and P pictures using forward references, performing normal decoding as time passes and more pictures are received.

[0438] [Autonomous driving] Furthermore, when transmitting and receiving still image or video data, such as 2D or 3D map information, for autonomous driving or driving assistance of a vehicle, the receiving terminal may receive metadata such as weather or construction information in addition to image data belonging to one or more layers, and decode these in association with each other. The metadata may belong to a layer, or it may simply be multiplexed with the image data.

[0439] In this case, since the vehicle, drone, or airplane containing the receiving terminal is moving, the receiving terminal can transmit its own location information, enabling seamless reception and decoding while switching between base stations ex106 to ex110. Furthermore, the receiving terminal can dynamically switch how much metadata is received or how much map information is updated, depending on the user's selection, the user's situation, and / or the state of the communication bandwidth.

[0440] The content delivery system ex100 allows the client to receive, decode, and play back encoded information transmitted by the user in real time.

[0441] [Distribution of personal content] Furthermore, the ex100 content delivery system allows for unicast or multicast distribution of not only high-definition, long-duration content from video distribution companies, but also low-definition, short-duration content from individuals. It is expected that the amount of such individual content will continue to increase. To improve the quality of individual content, the server may perform editing before encoding. This can be achieved, for example, using a configuration like the following.

[0442] During shooting, or after shooting, the server performs recognition processing such as detecting shooting errors, searching for scenes, analyzing semantics, and detecting objects from the original image data or encoded data in real time. Based on the recognition results, the server manually or automatically edits the images, correcting out-of-focus or shaky images, deleting less important scenes such as those with lower brightness or out of focus compared to other pictures, emphasizing object edges, and altering color tones. The server then encodes the edited data based on the editing results. It is also known that viewership decreases if the shooting time is too long, so the server may automatically clip scenes with little movement, as well as less important scenes, based on the image processing results, to ensure that the content falls within a specific time range according to the shooting time. Alternatively, the server may generate and encode a digest based on the results of the semantic analysis of the scenes.

[0443] Personal content may contain elements that infringe on copyright, moral rights, or portrait rights, and the scope of sharing may exceed the intended scope, which can be inconvenient for the individual. Therefore, for example, the server may intentionally change the image to one that is out of focus, such as the faces of people at the edges of the screen or the interior of a house, before encoding. Furthermore, the server may recognize whether the face of a person other than those previously registered is visible in the image to be encoded, and if so, it may apply a mosaic effect to the face. Alternatively, as a pre-processing or post-processing step before encoding, the user may specify a person or background area that they want to process from a copyright perspective. The server may replace the specified area with another image or blur the focus. In the case of a person, the server can track the person in a video and replace the image of the person's face.

[0444] Viewing personal content with small data volumes requires real-time processing, so depending on the bandwidth, the decoder may prioritize receiving the base layer first and then decode and play it back. During this time, the decoder may receive the enhancement layer, and if playback is looped or if the content is played back more than once, it may play back the high-quality video including the enhancement layer. With a stream that uses this scalable encoding, it is possible to provide an experience where the video is rough when unselected or at the beginning of viewing, but gradually the stream becomes smarter and the image quality improves. In addition to scalable encoding, a similar experience can be provided even if the rough stream played back the first time and the second stream encoded by referencing the first video are configured as a single stream.

[0445] [Other examples of practical applications] Furthermore, these encoding or decoding processes are generally performed on the LSIex500 present in each terminal. The LSIex500 (see Figure 40) may be a single chip or a configuration consisting of multiple chips. Alternatively, video encoding or decoding software may be embedded in some recording medium (such as a CD-ROM, flexible disk, or hard disk) that can be read by a computer ex111, and the encoding or decoding process may be performed using that software. In addition, if the smartphone ex115 has a camera, video data acquired by that camera may be transmitted. This video data may be data encoded by the LSIex500 present in the smartphone ex115.

[0446] The LSIex500 may also be configured to be activated by downloading application software. In this case, the terminal first determines whether it supports the content encoding method or whether it has the capability to perform the specific service. If the terminal does not support the content encoding method or does not have the capability to perform the specific service, the terminal may download a codec or application software and then acquire and play the content.

[0447] Furthermore, not only the content supply system ex100 via the Internet ex101, but also digital broadcasting systems can incorporate at least one of the video encoding device (image encoding device) or video decoding device (image decoding device) of each of the above embodiments. While the content supply system ex100 has a configuration that is more suited to multicast than unicast, as it transmits and receives multiplexed data with video and sound multiplexed onto broadcast radio waves using satellites, etc., the encoding and decoding processes are similar and can be applied in the same way.

[0448] [Hardware configuration] Figure 45 shows further details of the smartphone ex115 shown in Figure 40. Figure 46 shows an example configuration of the smartphone ex115. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves with the base station ex110, a camera unit ex465 capable of taking video and still images, and a display unit ex458 that displays video captured by the camera unit ex465 and data decoded from video received by the antenna ex450. The smartphone ex115 further includes an operation unit ex466, such as a touch panel, an audio output unit ex457, such as a speaker for outputting voice or sound, an audio input unit ex456, such as a microphone for inputting voice, a memory unit ex467 capable of storing captured video or still images, recorded audio, received video or still images, encoded data such as emails, or decoded data, and a slot unit ex464, which is an interface unit with SIM ex468 for identifying the user and authenticating access to various data, including the network. External memory may be used instead of the memory unit ex467.

[0449] The main control unit ex460, which can comprehensively control the display unit ex458 and the operation unit ex466, is connected to the power supply circuit unit ex461, the operation input control unit ex462, the video signal processing unit ex455, the camera interface unit ex463, the display control unit ex459, the modulation / demodulation unit ex452, the multiplexing / decompression unit ex453, the audio signal processing unit ex454, the slot unit ex464, and the memory unit ex467 via the synchronization bus ex470.

[0450] The power supply circuit unit ex461, when the power key is turned on by the user, starts up the smartphone ex115 into an operational state and supplies power to each component from the battery pack.

[0451] The smartphone ex115 performs tasks such as voice calls and data communication based on the control of the main control unit ex460, which has a CPU, ROM, RAM, etc. During a call, the voice signal picked up by the voice input unit ex456 is converted into a digital voice signal by the voice signal processing unit ex454, spread spectrum processing is performed by the modulation / demodulation unit ex452, digital-to-analog conversion and frequency conversion processing are performed by the transmission / reception unit ex451, and the resulting signal is transmitted via the antenna ex450. Received data is amplified, subjected to frequency conversion and analog-to-digital conversion processing, despread spectrum processing is performed by the modulation / demodulation unit ex452, converted into an analog voice signal by the voice signal processing unit ex454, and then output from the voice output unit ex457. In data communication mode, text, still images, or video data can be transmitted via the operation input control unit ex462 under the control of the main control unit ex460 based on operations such as those performed by the operation unit ex466 on the main unit. Similar transmission and reception processing is performed. When transmitting video, still images, or video and audio in data communication mode, the video signal processing unit ex455 compresses and encodes the video signal stored in the memory unit ex467 or the video signal input from the camera unit ex465 using the video encoding method shown in each of the above embodiments, and sends the encoded video data to the multiplexing / decoding unit ex453. The audio signal processing unit ex454 encodes the audio signal picked up by the audio input unit ex456 while the camera unit ex465 is capturing video or still images, and sends the encoded audio data to the multiplexing / decoding unit ex453. The multiplexing / decoding unit ex453 multiplexes the encoded video data and encoded audio data in a predetermined manner, performs modulation and conversion processing in the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmission / reception unit ex451, and transmits via the antenna ex450. The predetermined manner may be set in advance.

[0452] When receiving video attached to an email or chat, or video linked to a webpage, etc., the multiplexing / decomposition unit ex453 separates the multiplexed data received via antenna ex450, dividing it into a video data bitstream and an audio data bitstream. It then supplies the encoded video data to the video signal processing unit ex455 and the encoded audio data to the audio signal processing unit ex454 via the synchronization bus ex470. The video signal processing unit ex455 decodes the video signal using a video decoding method corresponding to the video encoding method shown in each of the above embodiments, and the video or still image contained in the linked video file is displayed from the display unit ex458 via the display control unit ex459. The audio signal processing unit ex454 decodes the audio signal, and the audio is output from the audio output unit ex457. As real-time streaming is becoming increasingly widespread, audio playback may be socially inappropriate depending on the user's situation. Therefore, as an initial setting, it is preferable to play only the video data without playing the audio signal, and to synchronize the audio playback only when the user performs an action such as clicking on the video data.

[0453] While the smartphone ex115 was used as an example here, other implementation forms are possible for terminals, such as a transmitting terminal with only an encoder and a receiving terminal with only a decoder, in addition to a transmitting / receiving terminal that has both an encoder and a decoder. In the explanation for digital broadcasting systems, multiplexed data in which audio data is multiplexed with video data is received or transmitted. However, the multiplexed data may also contain text data related to the video in addition to audio data. Furthermore, the video data itself may be received or transmitted instead of multiplexed data.

[0454] Although it was explained that the main control unit ex460, including the CPU, controls the encoding or decoding process, various terminals often have a GPU. Therefore, a configuration that leverages the GPU's performance to process a wide area at once using memory shared by the CPU and GPU, or memory whose addresses are managed so that it can be used in common, is also possible. This can shorten the encoding time, ensure real-time performance, and achieve low latency. In particular, it is efficient to perform motion detection, deblocking filters, SAO (Sample Adaptive Offset), and transformation / quantization processes at once on the GPU, rather than on the CPU, in units such as pictures.

[0455] Those skilled in the art will understand that various modifications and / or changes to the disclosure, as shown in the specific embodiments, may be made without departing from the spirit or scope of the disclosure as broadly described. Therefore, these embodiments should be considered to be for illustrative purposes only and not limiting.

[0456] According to this disclosure, various features are provided as follows:

[0457] 1. An encoding device that encodes a block to be processed within a picture using interpretation, Processor and Equipped with memory, The processor uses memory, By performing motion compensation using motion vectors corresponding to each of the two reference pictures, two predicted images are obtained from the two reference pictures. Obtain two gradient images corresponding to two predicted images from two reference pictures, Local motion detection values ​​are derived using two gradient images and two prediction images within the sub-block obtained by dividing the processing block. An encoding device that generates the final predicted image of the target block using the local motion detection values ​​of the subblock, two gradient images, and two predicted images.

[0458] 2. In the step of acquiring two predicted images, if two reference images are acquired with sub-pixel precision, the pixels are interpolated with sub-pixel precision by referencing the pixels within the interpolation reference range that encloses the predicted block indicated by the motion vector in each of the two reference pictures. The encoding device described in Statement 1, wherein the interpolation reference range is normally included in the reference range in order to perform motion compensation on the target block in a normal interpretation where processing is performed using local motion detection values.

[0459] 3. The interpolation reference range is the encoding device described in statement 2, which normally matches the reference range.

[0460] 4. The encoding device described in statement 2 or 3, wherein in the step of obtaining two gradient images, pixels are referenced within a gradient reference range that encloses the prediction block in each of the two reference pictures, and the gradient reference range is included in the interpolation reference range.

[0461] 5. The encoding device described in statement 4, where the gradient reference range matches the interpolation reference range.

[0462] 6. An encoding device according to any one of statements 1 to 5, wherein in the step of deriving local motion detection values, the pixel values ​​contained in the predicted subblocks within the regions corresponding to the subblocks in each of the two reference pictures are used in a weighted manner, and among these pixels, the pixel located in the center of the region is weighted with a larger value than the value used for pixels outside the center of the region.

[0463] 7. The encoding device according to statement 6, in the step of deriving local motion detection values, refers to pixels in another prediction subblock adjacent to the prediction subblock, which is included in the prediction subblock indicated by the motion vector, in addition to pixels in the prediction subblock corresponding to the subblock in each of the two reference pictures.

[0464] 8. An encoding device according to any one of statements 6-7, wherein, in the step of deriving local motion detection values, each of the two reference pictures references only some of the pixels in the predicted subblocks of the region corresponding to the subblock.

[0465] 9. In the step of deriving local motion detection values, (i) In each of the two reference pictures, select a pixel pattern from several different pixel patterns that represent several pixels within the prediction subblock. (ii) In order to derive local motion detection values ​​for a subblock, refer to the pixels in the prediction subblock that represent the selected pixel pattern, The processor is an encoding device as described in statement 8, configured to write information indicating a selected pixel pattern to a bitstream.

[0466] 10. In the step of deriving local motion detection values, (i) Based on two predicted images in each of the two reference pictures, a pixel pattern is adaptively selected from several different pixel patterns that represent some pixels within the predicted subblock. (ii) The encoding device described in statement 8, which references pixels in a prediction subblock that represent a selected pixel pattern in order to derive local motion detection values ​​for a subblock.

[0467] 11. An encoding method for encoding a block to be processed within a picture using interpretation, By performing motion compensation using motion vectors corresponding to each of the two reference pictures, two predicted images are obtained from the two reference pictures. Obtain two gradient images corresponding to two predicted images from two reference pictures, Local motion detection values ​​are derived using two gradient images and two prediction images within the sub-block obtained by dividing the processing block. An encoding method that generates a final predicted image of the target block using local motion detection values ​​of the subblock, two gradient images, and two predicted images.

[0468] 12. A decoding device that decodes a target block in a picture using interpretation, Processor and Equipped with memory, The processor uses memory to perform motion compensation using motion vectors corresponding to each of the two reference pictures, thereby obtaining two predicted images from the two reference pictures. Obtain two gradient images corresponding to two predicted images from two reference pictures, Local motion detection values ​​are derived using two gradient images and two prediction images within the sub-block obtained by dividing the processing block. A decoding device that generates a final predicted image of the target block using local motion detection values ​​of the subblock, two gradient images, and two predicted images.

[0469] 13. The decoder described in Statement 12, in the step of acquiring two predicted images, when two reference images are acquired with sub-pixel precision, the pixels are interpolated with sub-pixel precision by referencing pixels within the interpolation reference range that encloses the predicted block indicated by the motion vector in each of the two reference pictures, and the interpolation reference range is included in the normal reference range in order to perform motion compensation on the target block in normal interprediction, which is processed using local motion detection values.

[0470] 14. The interpolation range matches the normal range of the decoder as described in statement 13.

[0471] 15. A decoder according to statement 13 or 14, wherein in the step of obtaining two gradient images, pixels are referenced within a gradient reference range that encloses the predicted blocks in each of the two reference pictures, and the gradient reference range is included in the interpolation reference range.

[0472] 16. The decoder described in statement 15, where the gradient reference range matches the interpolation reference range.

[0473] 17. A decoder according to any one of statements 12 to 16, wherein in the step of deriving local motion detection values, the pixel values ​​contained in the predicted subblocks within the regions corresponding to the subblocks in each of the two reference pictures are used in a weighted manner, and among these pixels, the pixel located in the center of the region is weighted with a larger value than the value used for pixels outside the center of the region.

[0474] 18. The decoder according to statement 17, in the step of deriving local motion detection values, references pixels in another prediction subblock adjacent to the prediction subblock, which is included in the prediction subblock indicated by the motion vector, in addition to pixels in the prediction subblock corresponding to the subblock in each of the two reference pictures.

[0475] 19. A decoder according to any one of statements 17-18, wherein in the step of deriving local motion detection values, in each of the two reference pictures, only some of the pixels in the predicted subblocks of the region corresponding to the subblock.

[0476] 20. The processor obtains information from the bitstream indicating the selected pixel pattern, In the step of deriving local motion detection values, (i) Based on the acquired information, select a pixel pattern from a plurality of different pixel patterns that represent several pixels in the prediction subblock in each of the two reference pictures; and (ii) Refer to the pixels in the prediction subblock that represent the selected pixel pattern in order to derive a local motion detection value for the subblock, as described in Statement 19.

[0477] 21. In the step of deriving local motion detection values, (i) Based on two predicted images in each of the two reference pictures, a pixel pattern is adaptively selected from several different pixel patterns that represent some pixels within the predicted subblock. (ii) The decoding device described in statement 19, which references pixels in a prediction subblock that represent a selected pixel pattern in order to derive local motion detection values ​​for a subblock.

[0478] 22. A decoding method for decoding a target block in a picture using interpretation, By performing motion compensation using motion vectors corresponding to each of the two reference pictures, two predicted images are obtained from the two reference pictures. Obtain two gradient images corresponding to two predicted images from two reference pictures, Local motion detection values ​​are derived using two gradient images and two prediction images within the sub-block obtained by dividing the processing block. A decoding method that generates a final predicted image of the target block using local motion detection values ​​of the subblock, two gradient images, and two predicted images.

[0479] 23. Circuits and, It comprises a memory connected to the circuit, In operation, the circuit The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The first block of the prediction sample is padded to form the second block of the prediction sample, and the second block is larger than the first block. Using the second block of prediction samples, calculate at least the gradient, An image encoding device that encodes the current block using at least the calculated gradient.

[0480] 24. The image coding apparatus according to claim 23, wherein the first block of prediction samples is a prediction block used in a prediction process performed on a prediction mode which is merge mode or interprediction mode.

[0481] 25. The image coding apparatus according to claim 23, wherein the first block of the prediction sample is a reference block used in motion compensation processing performed on a prediction mode which is merge mode or interprediction mode.

[0482] 26. When the circuit pads the first block of predicted samples to form a second block of predicted samples, The image coding apparatus according to claim 23, wherein at least two sides of a first block of prediction samples are padded to form a second block of prediction samples, and at least two sides of the first block are not orthogonal.

[0483] 27. The circuit, at least when calculating the gradient, The image coding apparatus according to claim 23, wherein a gradient filter is applied to a second block of prediction samples to generate at least a derivative value.

[0484] 28. When the circuit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 23, wherein a second block of predicted samples is formed by mirroring the predicted samples of the first block.

[0485] 29. When the circuit pads the first block of predicted samples to form a second block of predicted samples, The image coding apparatus according to claim 23, wherein a second block of prediction samples is formed by copying the first block of prediction samples.

[0486] 30. The circuit pads the first block of predicted samples to form a second block of predicted samples. The image coding apparatus according to claim 23, wherein a fixed value is padded into a first block of prediction samples to form a second block of prediction samples, the fixed value may be 0, 128, a positive integer, the mean of the first block of prediction samples, or the median of the first block of prediction samples.

[0487] 31. When the circuit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 23, wherein a function is executed on a first block of prediction samples to form a second block of prediction samples.

[0488] 32. When the circuit pads the first block of prediction samples to form a second block of prediction samples, The image encoding apparatus according to claim 23, comprising combining at least two of the following to form a second block of prediction samples: mirroring, copying, fixed-value padding, and execution of a function on a first block of prediction samples.

[0489] 33. The circuit, when predicting the first block of predicted samples for the current block of the picture, further, The image encoding apparatus according to claim 23, wherein the step of predicting another block of prediction samples for the current block of a picture, and predicting another block of prediction samples, comprises at least a prediction process using another motion vector from yet another picture.

[0490] 34. The image encoding apparatus according to claim 33, wherein another picture has a picture order count different from that of the other picture and / or the picture order count of the other picture.

[0491] 35. The circuit pads the first block of predicted samples to form a second block of predicted samples. The image encoding apparatus according to claim 34, further comprising padding another block of the predicted sample to form yet another block of the predicted sample.

[0492] 36. The image coding apparatus according to claim 23, wherein the circuit pads a first block of prediction samples to form a second block of prediction samples, and then the circuit performs interpolation on a prediction mode which is a merge mode or an interprediction mode.

[0493] 37. The image coding apparatus according to claim 23, wherein, before the circuit pads a first block of prediction samples to form a second block of prediction samples, the circuit performs interpolation on a prediction mode which is a merge mode or an interprediction mode.

[0494] 38. The circuit encodes the current block using at least the calculated gradient. The image encoding apparatus according to claim 23, which encodes the current block using a block of predicted samples resulting from interpolation and at least a calculated gradient.

[0495] 39. The circuit, at least when calculating the gradient, The image encoding apparatus according to claim 38, which applies one or more gradient filters to a block of predicted samples resulting from interpolation processing to generate one or more derivative values.

[0496] 40. Circuits and, It comprises a memory connected to the circuit, In operation, the circuit The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The first block of the prediction sample is padded to form the second block of the prediction sample, and the second block is larger than the first block. Interpolation is performed using the second block of prediction samples. An image encoding device that encodes the current block using at least the blocks resulting from interpolation.

[0497] 41. The circuit operates as follows: The image encoding apparatus according to claim 40, wherein the OBMC process predicts one or more adjacent blocks of the current block, and the OBMC process uses at least the blocks resulting from the interpolation process.

[0498] 42. In operation, the circuit pads the first block of predicted samples to form a second block of predicted samples. The image coding apparatus according to claim 40, wherein the sample is padded with two sides of a first block of the predicted sample, and the two sides of the first block are parallel to each other.

[0499] 43. In operation, the circuit pads the first block of predicted samples to form a second block of predicted samples. The image coding apparatus according to claim 40, which pads the sample with three or more sides of the first block of the predicted sample.

[0500] 44. Circuits and, It comprises a memory connected to the circuit, In operation, the circuit The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The second block of the prediction sample is padded to form the third block of the prediction sample, and the second block is adjacent to the current block. Perform OBMC processing using at least the first and third blocks of the prediction sample. An image encoding device that encodes the current block using at least the blocks resulting from OBMC processing.

[0501] 45. Circuits and, It comprises a memory connected to the circuit, In operation, the circuit The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using a first motion vector from another picture. Using at least the first motion vector, a second motion vector for the current block is derived by DMVR processing, Interpolation is performed on the current block using the second motion vector, and the interpolation process includes padding. An image encoding device that encodes the current block using at least the blocks resulting from interpolation.

[0502] 46. ​​When the circuit performs interpolation on the current block, The first block of the predicted sample is padded according to the second motion vector to generate the second block of the predicted sample. The image encoding apparatus according to claim 45, which performs interpolation using at least a second block of predicted samples.

[0503] 47. The image coding apparatus according to claim 46, wherein the circuit pads one or more sides of the first block of the predicted sample according to a second motion vector when padding the first block of the predicted sample to generate a second block of the predicted sample.

[0504] 48. The image coding apparatus according to claim 45, wherein the circuit performs padding on the first block of the predicted sample according to the first motion vector when deriving a second motion vector using DMVR processing.

[0505] 49. In operation, the system includes a splitting unit that receives the original picture and divides it into blocks, a first adding unit that receives the blocks from the splitting unit, makes predictions from the prediction control unit, subtracts each predicted value from the corresponding block and outputs the residual, In operation, the conversion unit converts the residual output from the addition unit and outputs a conversion coefficient, In operation, a quantization unit generates quantized conversion coefficients by quantizing the conversion coefficients, In operation, the entropy encoding unit encodes quantization conversion coefficients to generate a bitstream, In operation, the inverse quantization conversion unit obtains conversion coefficients by inverse quantization of the quantization conversion coefficients, and obtains residuals by inverse transformation of the conversion coefficients. In operation, a second adder reconstructs the block by adding the residual output from the inverse quantization conversion unit and the predicted value output from the prediction control unit. In operation, the system comprises an interpretation unit that generates a prediction of the current block based on the reference block in the encoded reference picture, and a prediction control unit connected to memory. In operation, when generating a prediction of the current block based on the reference block in the encoded reference picture, the interpretation unit: The step of predicting the first block of prediction samples for the current block includes at least a prediction process using motion vectors from an encoded reference picture, The first block of the prediction sample is padded to form the second block of the prediction sample, and the second block is larger than the first block. Using the second block of prediction samples, calculate at least the gradient, An image encoding device that encodes the current block using at least the calculated gradient.

[0506] 50. The image coding apparatus according to claim 49, wherein the first block of prediction samples is a prediction block used in a prediction process performed on a prediction mode which is merge mode or interprediction mode.

[0507] 51. The image coding apparatus according to claim 49, wherein the first block of the prediction sample is a reference block used in motion compensation processing performed on a prediction mode which is merge mode or interprediction mode.

[0508] 52. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 49, wherein at least two sides of a first block of prediction samples are padded to form a second block of prediction samples, and at least two sides of the first block are not orthogonal.

[0509] 53. The interpretation unit, at least when calculating the gradient, The image coding apparatus according to claim 49, wherein a gradient filter is applied to a second block of prediction samples to generate at least a differential value.

[0510] 54. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 49, wherein a second block of predicted samples is formed by mirroring the predicted samples of the first block.

[0511] 55. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 49, wherein a second block of prediction samples is formed by copying the first block of prediction samples.

[0512] 56. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 49, wherein a fixed value is padded into a first block of prediction samples to form a second block of prediction samples, the fixed value may be 0, 128, a positive integer, the mean of the first block of prediction samples, or the median of the first block of prediction samples.

[0513] 57. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 49, wherein a function is executed on a first block of prediction samples to form a second block of prediction samples.

[0514] 58. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 49, comprising combining at least two of the following to form a second block of prediction samples: mirroring, copying, padding of the first value, and execution of a function on a first block of prediction samples.

[0515] 59. When the interpretation unit predicts the first block of prediction samples for the current block of the picture, it further: The image encoding apparatus according to claim 49, wherein the step of predicting another block of prediction samples for the current block, and predicting another block of prediction samples, comprises at least a prediction process using another motion vector from another encoded reference picture.

[0516] 60. The image encoding apparatus according to claim 59, wherein another encoded reference picture has a picture order count different from the picture order count of the encoded reference picture and / or the picture order count of the original picture.

[0517] 61. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, it further performs the following: The image encoding apparatus according to claim 60, wherein another block of the predicted sample is padded to form yet another block of the predicted sample.

[0518] 62. The image coding apparatus according to claim 49, wherein the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, and then the interpretation unit performs interpolation processing on the prediction mode, which is either merge mode or interpretation mode.

[0519] 63. The image coding apparatus according to claim 49, wherein, before the interprediction unit pads the first block of prediction samples to form a second block of prediction samples, the interprediction unit performs interpolation on the prediction mode, which is either merge mode or interprediction mode.

[0520] 64. The interpretation unit, when encoding the current block using at least the calculated gradient, The image encoding apparatus according to claim 49, which encodes the current block using a block of predicted samples resulting from interpolation and at least a calculated gradient.

[0521] 65. The interpretation unit, at least when calculating the gradient, The image encoding apparatus according to claim 64, wherein one or more gradient filters are applied to a block of predicted samples resulting from interpolation processing to generate one or more derivative values.

[0522] 66. In operation, the system includes a splitting unit that receives the original picture and divides it into blocks, a first adding unit that receives the blocks from the splitting unit, makes predictions from the prediction control unit, subtracts each prediction from the corresponding block and outputs the residual, In operation, the conversion unit converts the residual output from the addition unit and outputs a conversion coefficient, In operation, a quantization unit generates quantized conversion coefficients by quantizing the conversion coefficients, In operation, the entropy encoding unit encodes quantization conversion coefficients to generate a bitstream, In operation, the system includes an inverse quantization unit and an inverse transformation unit that inversely quantize the quantization conversion coefficients to obtain the conversion coefficients, and then inversely transform the conversion coefficients to obtain the residuals. In operation, a second adder reconstructs the block by adding the residual output from the inverse quantization conversion unit and the predicted value output from the prediction control unit. In operation, the system comprises an interpretation unit that generates a prediction of the current block based on the reference block in the encoded reference picture, and a prediction control unit connected to memory. In operation, when generating a prediction of the current block based on the reference block in the encoded reference picture, the interpretation unit: The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The first block of the prediction sample is padded to form the second block of the prediction sample, and the second block is larger than the first block. Interpolation is performed using the second block of prediction samples. An image encoding device that encodes the current block using at least the blocks resulting from interpolation.

[0523] 67. The interpretation unit operates as follows: The image encoding apparatus according to claim 66, wherein the OBMC process predicts one or more adjacent blocks of the current block, and the OBMC process uses at least the blocks resulting from the interpolation process.

[0524] 68. In operation, when the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 66, wherein the sample is padded with two sides of a first block of the predicted sample, and the two sides of the first block are parallel to each other.

[0525] 69. In operation, when the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image coding apparatus according to claim 66, which pads samples with respect to three or more sides of the first block of the predicted sample.

[0526] 70. In operation, the system includes a splitting unit that receives the original picture and divides it into blocks, a first adding unit that receives the blocks from the splitting unit, makes predictions from the prediction control unit, subtracts each prediction from the corresponding block and outputs the residual, In operation, the conversion unit converts the residual output from the addition unit and outputs a conversion coefficient, In operation, a quantization unit generates quantized conversion coefficients by quantizing the conversion coefficients, In operation, the entropy encoding unit encodes quantization conversion coefficients to generate a bitstream, In operation, the system includes an inverse quantization unit and an inverse transformation unit that inversely quantize the quantization conversion coefficients to obtain the conversion coefficients, and then inversely transform the conversion coefficients to obtain the residuals. In operation, a second adder reconstructs the block by adding the residual output from the inverse quantization conversion unit and the predicted value output from the prediction control unit. In operation, the system comprises an interpretation unit that generates a prediction of the current block based on the reference block in the encoded reference picture, and a prediction control unit connected to memory. In operation, when generating a prediction of the current block based on the reference block in the encoded reference picture, the interpretation unit: The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The second block of the prediction sample is padded to form the third block of the prediction sample, and the second block is adjacent to the current block. Perform OBMC processing using at least the first and third blocks of the prediction sample. An image encoding device that encodes the current block using at least the blocks resulting from OBMC processing.

[0527] 71. In operation, the system includes a splitting unit that receives the original picture and divides it into blocks, a first adding unit that receives the blocks from the splitting unit, makes predictions from the prediction control unit, subtracts each prediction from the corresponding block and outputs the residual, In operation, the conversion unit converts the residual output from the addition unit and outputs a conversion coefficient, In operation, a quantization unit generates quantized conversion coefficients by quantizing the conversion coefficients, In operation, the entropy encoding unit encodes quantization conversion coefficients to generate a bitstream, In operation, the system includes an inverse quantization unit and an inverse transformation unit that inversely quantize the quantization conversion coefficients to obtain the conversion coefficients, and then inversely transform the conversion coefficients to obtain the residuals. In operation, a second adder reconstructs the block by adding the residual output from the inverse quantization conversion unit and the predicted value output from the prediction control unit. In operation, the system comprises an interpretation unit that generates a prediction of the current block based on the reference block in the encoded reference picture, and a prediction control unit connected to memory. In operation, when generating a prediction of the current block based on the reference block in the encoded reference picture, the interpretation unit: The step of predicting the first block of predicted samples for the current block of a picture includes at least a prediction process using a first motion vector from another picture, and deriving a second motion vector for the current block by DMVR processing using at least the first motion vector. Interpolation is performed on the current block using the second motion vector, and the interpolation process includes padding. An image encoding device that encodes the current block using at least the blocks resulting from interpolation.

[0528] 72. The interpretation unit performs interpolation on the current block. The first block of the predicted sample is padded according to the second motion vector to generate the second block of the predicted sample. The image coding apparatus according to claim 71, which performs interpolation using at least a second block of prediction samples.

[0529] 73. The image coding apparatus according to claim 72, wherein when the interpretation unit pads the first block of the prediction sample to generate a second block of the prediction sample, it pads one or more sides of the first block of the prediction sample according to the second motion vector.

[0530] 74. The image coding apparatus according to claim 71, wherein the interpretation unit performs padding on the first block of the prediction sample according to the first motion vector when deriving a second motion vector using DMVR processing.

[0531] 75. Circuits and, It comprises a memory connected to the circuit, In operation, the circuit The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The first block of the prediction sample is padded to form the second block of the prediction sample, and the second block is larger than the first block. Using the second block of prediction samples, calculate at least the gradient, An image decoding device that decodes the current block using at least the calculated gradient.

[0532] 76. The image decoding apparatus according to claim 75, wherein the first block of the prediction sample is a prediction block used in a prediction process performed on a prediction mode which is merge mode or interprediction mode.

[0533] 77. The image decoding apparatus according to claim 75, wherein the first block of the prediction sample is a reference block used in motion compensation processing performed on a prediction mode which is merge mode or interprediction mode.

[0534] 78. The circuit pads the first block of predicted samples to form a second block of predicted samples. The image decoding apparatus according to claim 75, wherein at least two sides of a first block of prediction samples are padded to form a second block of prediction samples, and at least two sides of the first block are not orthogonal.

[0535] 79. The circuit, at least when calculating the gradient, The image decoding apparatus according to claim 75, wherein a gradient filter is applied to a second block of prediction samples to generate at least a differential value.

[0536] 80. The circuit pads the first block of prediction samples to form a second block of prediction samples. The image decoding apparatus according to claim 75, which mirrors the predicted sample of the first block to form a second block of predicted samples.

[0537] 81. When the circuit pads the first block of prediction samples to form a second block of prediction samples, The image decoding apparatus according to claim 75, which copies the first block of predicted samples to form a second block of predicted samples.

[0538] 82. The circuit pads the first block of predicted samples to form a second block of predicted samples. The image decoding apparatus according to claim 75, wherein a fixed value is padded into a first block of prediction samples to form a second block of prediction samples, the fixed value may be 0, 128, a positive integer, the mean of the first block of prediction samples, or the median of the first block of prediction samples.

[0539] 83. The circuit pads the first block of predicted samples to form a second block of predicted samples. The image decoding apparatus according to claim 75, wherein a function is executed on a first block of predicted samples to form a second block of predicted samples.

[0540] 84. When the circuit pads the first block of predicted samples to form a second block of predicted samples, The image decoding apparatus according to claim 75, comprising combining at least two of the following to form a second block of predicted samples: mirroring, copying, padding of first values, and execution of a function on a first block of predicted samples.

[0541] 85. The circuit, when predicting the first block of predicted samples for the current block of the picture, further, The image decoding apparatus according to claim 75, wherein the step of predicting another block of prediction samples for the current block of a picture, and predicting another block of prediction samples, comprises at least a prediction process using another motion vector from yet another picture.

[0542] 86. The image decoding apparatus according to claim 85, wherein another picture has a picture order count different from that of the other picture and / or the picture order count of the other picture.

[0543] 87. When the circuit pads the first block of predicted samples to form a second block of predicted samples, further, The image decoding apparatus according to claim 86, further comprising padding another block of the predicted sample to form yet another block of the predicted sample.

[0544] 88. The image decoding apparatus according to claim 75, wherein the circuit pads a first block of prediction samples to form a second block of prediction samples, and then the circuit performs interpolation on a prediction mode which is a merge mode or an interprediction mode.

[0545] 89. The image decoding apparatus according to claim 75, wherein the circuit performs interpolation on a prediction mode, which is a merge mode or an interprediction mode, before the circuit pads a first block of prediction samples to form a second block of prediction samples.

[0546] 90. The circuit, when decoding the current block using at least the calculated gradient, The image decoding apparatus according to claim 75, which decodes the current block using a block of predicted samples resulting from interpolation and at least a calculated gradient.

[0547] 91. The circuit, at least when calculating the gradient, The image decoding apparatus according to claim 90, which applies one or more gradient filters to a block of predicted samples resulting from interpolation processing to generate one or more derivative values.

[0548] 92. Circuits and, It comprises a memory connected to the circuit, In operation, the circuit The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The first block of the prediction sample is padded to form the second block of the prediction sample, and the second block is larger than the first block. Interpolation is performed using the second block of prediction samples. An image decoding device that decodes the current block using at least the blocks resulting from interpolation.

[0549] 93. In operation, the circuit The image decoding apparatus according to claim 92, wherein the OBMC process predicts one or more adjacent blocks of the current block, and the OBMC process uses at least the blocks resulting from the interpolation process.

[0550] 94. In operation, the circuit pads the first block of predicted samples to form a second block of predicted samples. The image decoding apparatus according to claim 92, wherein the sample is padded with two sides of a first block of the predicted sample, and the two sides of the first block are parallel to each other.

[0551] 95. In operation, the circuit pads the first block of predicted samples to form a second block of predicted samples. The image decoding apparatus according to claim 92, which pads the sample with respect to three or more sides of the first block of the predicted sample.

[0552] 96. Circuits and, It comprises a memory connected to the circuit, In operation, the circuit The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The second block of the prediction sample is padded to form the third block of the prediction sample, and the second block is adjacent to the current block. Perform OBMC processing using at least the first and third blocks of the prediction sample. An image decoding device that decodes the current block using at least the blocks resulting from OBMC processing.

[0553] 97. Circuits and, It comprises a memory connected to the circuit, In operation, the circuit The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using a first motion vector from another picture. Using at least the first motion vector, a second motion vector for the current block is derived by DMVR processing, Interpolation is performed on the current block using the second motion vector, and the interpolation process includes padding. An image decoding device that decodes the current block using at least the blocks resulting from interpolation.

[0554] 98. When a circuit performs interpolation on the current block, The first block of the predicted sample is padded according to the second motion vector to generate the second block of the predicted sample. The image decoding apparatus according to claim 97, which performs interpolation using at least a second block of predicted samples.

[0555] 99. The image decoding apparatus according to claim 98, wherein the circuit pads one or more sides of the first block of the predicted sample according to a second motion vector when padding the first block of the predicted sample to generate a second block of the predicted sample.

[0556] 100. The image decoding apparatus according to claim 97, wherein the circuit performs padding on the first block of the predicted sample according to the first motion vector when deriving a second motion vector using DMVR processing.

[0557] 101. In operation, the system includes an entropy decoding unit that receives and decodes an encoded bitstream to obtain quantization conversion coefficients, In operation, the inverse quantization conversion unit obtains conversion coefficients by inverse quantization of the quantization conversion coefficients, and obtains residuals by inverse transformation of the conversion coefficients. In operation, an adder unit reconstructs a block by adding the residual output from the inverse quantization conversion unit and the predicted value output from the prediction control unit. In operation, the system comprises an interpretation unit that generates a prediction of the current block of a picture based on the reference block in the decoded reference picture, and a prediction control unit connected to memory. In operation, when generating a prediction of the current block based on the reference block in the decoded reference picture, the interpretation unit: The step of predicting the first block of prediction samples for the current block includes at least a prediction process using motion vectors from a decoded reference picture. The first block of the prediction sample is padded to form the second block of the prediction sample, and the second block is larger than the first block. Using the second block of prediction samples, calculate at least the gradient, An image decoding device that decodes the current block using at least the calculated gradient.

[0558] 102. The image decoding apparatus according to claim 101, wherein the first block of the prediction sample is a prediction block used in a prediction process performed on a prediction mode which is merge mode or interprediction mode.

[0559] 103. The image decoding apparatus according to claim 101, wherein the first block of the prediction sample is a reference block used in motion compensation processing performed on a prediction mode which is merge mode or interprediction mode.

[0560] 104. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image decoding apparatus according to claim 101, wherein at least two sides of a first block of prediction samples are padded to form a second block of prediction samples, and at least two sides of the first block are not orthogonal.

[0561] 105. The interpretation unit, at least when calculating the gradient, The image decoding apparatus according to claim 101, wherein a gradient filter is applied to a second block of prediction samples to generate at least a differential value.

[0562] 106. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image decoding apparatus according to claim 101, which mirrors the predicted sample of the first block to form a second block of predicted samples.

[0563] 107. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image decoding apparatus according to claim 101, which copies the first block of predicted samples to form a second block of predicted samples.

[0564] 108. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image decoding apparatus according to claim 101, wherein a fixed value is padded into a first block of prediction samples to form a second block of prediction samples, the fixed value may be 0, 128, a positive integer, the mean value of the first block of prediction samples, or the median value of the first block of prediction samples.

[0565] 109. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image decoding apparatus according to claim 101, wherein a function is executed on a first block of prediction samples to form a second block of prediction samples.

[0566] 110. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image decoding apparatus according to claim 101, comprising combining at least two of the following: mirroring, copying, padding of first values, and execution of a function on a first block of prediction samples to form a second block of prediction samples.

[0567] 111. When the interpretation unit predicts the first block of prediction samples for the current block of the picture, it further: The image decoding apparatus according to claim 101, wherein the step of predicting another block of prediction samples for the current block and predicting another block of prediction samples includes at least a prediction process using another motion vector from another decoded reference picture.

[0568] 112. The image decoding apparatus according to claim 111, wherein another decoded reference picture has a picture order count different from the picture order count of the decoded reference picture and / or the picture order count of the original picture.

[0569] 113. When the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, The image decoding apparatus according to claim 112, further comprising padding another block of the predicted sample to form yet another block of the predicted sample.

[0570] 114. The image decoding apparatus according to claim 101, wherein the interpretation unit pads the first block of prediction samples to form a second block of prediction samples, and then the interpretation unit performs interpolation processing on the prediction mode, which is either merge mode or interpretation mode.

[0571] 115. The image decoding apparatus according to claim 101, wherein, before the interprediction unit pads the first block of prediction samples to form a second block of prediction samples, the interprediction unit performs interpolation on the prediction mode, which is either merge mode or interprediction mode.

[0572] 116. The interpretation unit, when decoding the current block using at least the calculated gradient, The image decoding apparatus according to claim 101, which decodes the current block using a block of predicted samples resulting from interpolation and at least a calculated gradient.

[0573] 117. The interpretation unit, at least when calculating the gradient, The image decoding apparatus according to claim 116, which applies one or more gradient filters to a block of predicted samples resulting from interpolation processing to generate one or more derivative values.

[0574] 118. In operation, the system includes an entropy decoding unit that receives and decodes an encoded bitstream to obtain quantization conversion coefficients, In operation, the inverse quantization conversion unit obtains conversion coefficients by inverse quantization of the quantization conversion coefficients, and obtains residuals by inverse transformation of the conversion coefficients. In operation, an adder unit reconstructs a block by adding the residual output from the inverse quantization conversion unit and the predicted value output from the prediction control unit. In operation, the system comprises an interpretation unit that generates a prediction of the current block of a picture based on the reference block in the decoded reference picture, and a prediction control unit connected to memory. In operation, when generating a prediction of the current block based on the reference block in the decoded reference picture, the interpretation unit: The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The first block of the prediction sample is padded to form the second block of the prediction sample, and the second block is larger than the first block. Interpolation is performed using the second block of prediction samples. An image decoding device that decodes the current block using at least the blocks resulting from interpolation.

[0575] 119. The circuit operates as follows: The image decoding apparatus according to claim 118, wherein the OBMC process predicts one or more adjacent blocks of the current block, and the OBMC process uses at least the blocks resulting from the interpolation process.

[0576] 120. In operation, the circuit pads the first block of predicted samples to form a second block of predicted samples. The image decoding apparatus according to claim 118, wherein the sample is padded with respect to two sides of a first block of the predicted sample, and the two sides of the first block are parallel to each other.

[0577] 121. In operation, the circuit pads the first block of predicted samples to form a second block of predicted samples. The image decoding apparatus according to claim 118, which pads the sample with respect to three or more sides of the first block of the predicted sample.

[0578] 122. In operation, the system includes an entropy decoding unit that receives and decodes an encoded bitstream to obtain quantization conversion coefficients, In operation, the inverse quantization conversion unit obtains conversion coefficients by inverse quantization of the quantization conversion coefficients, and obtains residuals by inverse transformation of the conversion coefficients. In operation, an adder unit reconstructs a block by adding the residual output from the inverse quantization conversion unit and the predicted value output from the prediction control unit. In operation, the system comprises an interpretation unit that generates a prediction of the current block of a picture based on the reference block in the decoded reference picture, and a prediction control unit connected to memory. In operation, when generating a prediction of the current block based on the reference block in the decoded reference picture, the interpretation unit: The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using motion vectors from another picture. The second block of the prediction sample is padded to form the third block of the prediction sample, and the second block is adjacent to the current block. Perform OBMC processing using at least the first and third blocks of the prediction sample. An image decoding device that decodes the current block using at least the blocks resulting from OBMC processing.

[0579] 123. Circuits and, It comprises a memory connected to the circuit, In operation, the circuit The step of predicting the first block of prediction samples for the current block of a picture includes at least a prediction process using a first motion vector from another picture. Using at least the first motion vector, a second motion vector for the current block is derived by DMVR processing, Interpolation is performed on the current block using the second motion vector, and the interpolation process includes padding. An image decoding device that decodes the current block using at least the blocks resulting from interpolation.

[0580] 124. When a circuit performs interpolation on the current block, The first block of the predicted sample is padded according to the second motion vector to generate the second block of the predicted sample. The image decoding apparatus according to claim 123, which performs interpolation using at least a second block of predicted samples.

[0581] 125. The image decoding apparatus according to claim 124, wherein the circuit pads one or more sides of the first block of the predicted sample according to a second motion vector when padding the first block of the predicted sample to generate a second block of the predicted sample.

[0582] 126. The image decoding apparatus according to claim 123, wherein the circuit performs padding on the first block of the predicted sample according to the first motion vector when deriving a second motion vector using DMVR processing.

[0583] 127. An image encoding method that enables an image encoding device to perform the steps according to any one of claims 1 to 48.

[0584] 128. An image decoding method that enables an image decoding device to perform the steps according to any one of claims 75 to 100. [Industrial applicability]

[0585] This disclosure can be used, for example, in television receivers, digital video recorders, car navigation systems, mobile phones, digital cameras, digital video cameras, etc. [Explanation of Symbols]

[0586] 100 Encoding device 102 Division 104 Subtraction Unit 106 Conversion Unit 108 Quantization section 110 Entropy coding unit 112, 204 Inverse quantization section 114, 206 Inverse Transform Section 116, 208 Addition section 118, 210 block memory 120, 212 Loop filter section 122,214 frame memory 124, 216 Intra Prediction Unit 126, 218 Interpretation Unit 128, 220 Prediction Control Unit 200 Decoders 202 Entropy Decoder 1000 Current Pictures 1001 Current Block 1100 First reference picture 1110 First motion vector 1120 First prediction block 1121, 1221 Prediction subblocks 1122, 1222 Top left pixels 1130, 1130A First interpolation reference range 1131, 1131A, 1132, 1132A, 1231, 1231A, 1232, 1232A Reference range 1135, 1135A First Gradient Reference Range 1140 First Prediction Image 1150 First gradient image 1200 Second reference picture 1210 Second motion vector 1220 Second prediction block 1230, 1230A Second interpolation reference range 1235, 1235A Second Gradient Reference Range 1240 Second Prediction Image 1250 Second gradient image 1300 Local motion detection value 1400 First Prediction Image

Claims

1. Memory and The system comprises a circuit connected to the memory, In operation, the aforementioned circuit Calculate the motion vector that indicates the reference range within the reference picture, Based on the aforementioned motion vector, an interpolation process is performed using the sample values ​​contained in the reference picture to generate a predicted block. Using the aforementioned prediction block, a gradient block of the same size as the prediction block is generated. Using the aforementioned gradient block, a predicted image is generated. Based on the aforementioned predicted image, the current block is encoded, The generation of the aforementioned gradient block is A process for calculating a horizontal gradient value that indicates the difference between the value of the right-side sample adjacent to the position of the target sample included in the prediction block and the value of the left-side sample adjacent to the left of the target sample. The process includes calculating a vertical gradient value that indicates the difference between the value of a lower sample adjacent to the position of the target sample included in the prediction block and the value of an upper sample adjacent to the position of the target sample. The first horizontal gradient value at the left end of the gradient block is calculated using the value of the first sample at the same position as the first horizontal gradient value in the prediction block, as the left-side sample. The second horizontal gradient value adjacent to the right of the position of the first horizontal gradient value is calculated using the value of the first sample as the left-side sample. The first vertical gradient value at the upper end of the gradient block is calculated using the value of the first sample at the same position as the first vertical gradient value in the prediction block, as the upper sample. The second vertical slope value adjacent to the position of the first vertical slope value is calculated using the value of the first sample as the upper sample. Encoding device.

2. Memory and The system comprises a circuit connected to the memory, In operation, the aforementioned circuit Calculate the motion vector that indicates the reference range within the reference picture, Based on the aforementioned motion vector, an interpolation process is performed using the sample values ​​contained in the reference picture to generate a predicted block. Using the aforementioned prediction block, a gradient block of the same size as the prediction block is generated. Using the aforementioned gradient block, a predicted image is generated. Based on the aforementioned predicted image, the current block is decoded. The generation of the aforementioned gradient block is A process for calculating a horizontal gradient value that indicates the difference between the value of the right-side sample adjacent to the position of the target sample included in the prediction block and the value of the left-side sample adjacent to the left of the target sample. The process includes calculating a vertical gradient value that indicates the difference between the value of a lower sample adjacent to the position of the target sample included in the prediction block and the value of an upper sample adjacent to the position of the target sample. The first horizontal gradient value at the left end of the gradient block is calculated using the value of the first sample at the same position as the first horizontal gradient value in the prediction block, as the left-side sample. The second horizontal gradient value adjacent to the right of the position of the first horizontal gradient value is calculated using the value of the first sample as the left-side sample. The first vertical gradient value at the upper end of the gradient block is calculated using the value of the first sample at the same position as the first vertical gradient value in the prediction block, as the upper sample. The second vertical slope value adjacent to the position of the first vertical slope value is calculated using the value of the first sample as the upper sample. Decoding device.

3. Memory and The system comprises a circuit connected to the memory, In operation, the aforementioned circuit The bitstream generated by the encoding method is transmitted, The aforementioned encoding method is Calculate the motion vector that indicates the reference range within the reference picture, Based on the aforementioned motion vector, an interpolation process is performed using the sample values ​​contained in the reference picture to generate a predicted block. Using the aforementioned prediction block, a gradient block of the same size as the prediction block is generated. Using the aforementioned gradient block, a predicted image is generated. The process includes encoding the current block based on the predicted image, The generation of the aforementioned gradient block is A process for calculating a horizontal gradient value that indicates the difference between the value of the right-side sample adjacent to the position of the target sample included in the prediction block and the value of the left-side sample adjacent to the left of the target sample. The process includes calculating a vertical gradient value that indicates the difference between the value of a lower sample adjacent to the position of the target sample included in the prediction block and the value of an upper sample adjacent to the position of the target sample. The first horizontal gradient value at the left end of the gradient block is calculated using the value of the first sample at the same position as the first horizontal gradient value in the prediction block, as the left-side sample. The second horizontal gradient value adjacent to the right of the position of the first horizontal gradient value is calculated using the value of the first sample as the left-side sample. The first vertical gradient value at the upper end of the gradient block is calculated using the value of the first sample at the same position as the first vertical gradient value in the prediction block, as the upper sample. The second vertical slope value adjacent to the position of the first vertical slope value is calculated using the value of the first sample as the upper sample. A bitstream transmitter.

Images