Video decoding device, video encoding device, and prediction image generation device

Abstract

The present invention provides a predicted image generation device, which generates a predicted image by performing motion compensation on multiple reference images. The predicted image generation device comprises a predicted image generation unit, which generates a predicted image by referring to a first reference image, a second reference image, and a gradient correction item.

Description

[0001] This application is a divisional application of the PCT national phase application with a filing date of December 22, 2016 and national application number 2016800808190, which claims priority to the Japanese patent application with a filing date of February 3, 2016 and application number JP 2016-019354. Technical Field

[0002] One embodiment of the present invention relates to a moving picture decoding device, a moving picture encoding device, and a predicted image generating device. Background Art

[0003] In order to efficiently transmit or record moving images, a moving image encoding device that generates encoded data by encoding the moving image and a moving image decoding device that generates a decoded image by decoding the encoded data are used.

[0004] As specific moving picture coding methods, for example, methods proposed in H.264 / MPEG-4.AVC and HEVC (High-Efficiency Video Coding) can be cited.

[0005] In this motion image encoding method, images (pictures) constituting the motion image are managed by a hierarchical structure formed by slices obtained by dividing the image, coding units (sometimes also called coding units) obtained by dividing the slices, and prediction units (PUs) and transform units (TUs) which are blocks obtained by dividing the coding units, and are encoded and decoded on a block basis.

[0006] In addition, in such a moving picture coding method, a prediction image is usually generated based on a local decoded image obtained by encoding / decoding an input image, and a prediction residual (sometimes also referred to as a "difference image" or "residual image") obtained by subtracting the prediction image from the input image (original image) is encoded. Examples of a method for generating a prediction image include inter-picture prediction (inter-frame prediction) and intra-picture prediction (intra-frame prediction).

[0007] In addition, Non-Patent Document 1 can be cited as a recent moving picture encoding and decoding technology.

[0008] Prior art literature

[0009] Non-patent literature

[0010] Non-patent document 1: Video / JVET, "Algorithm Description of Joint ExplorationTestModel 1 (JEM 1)", INTERNATIONAL ORGANIZATION FOR STANDARDIZATIONORGANISATION INTERNATIONALE DE NORMALISATION ISO / IEC JTC1 / SC29 / WG11 CODING OFMOVING PICTURES AND AUDIO, ISO / IEC JTC1 / SC29 / WG11 / N15790, October 2015, Geneva, CH. Summary of the invention

[0011] Problem that the invention aims to solve

[0012] In recent years, in the motion image encoding and decoding technology, the following technology is applied: in the motion compensation process when generating the predicted image, the predicted image is derived by motion compensation using dual prediction based on the target image and two reference images. As a technology for improving the accuracy of the predicted image derived by motion compensation, BIO corrected using gradient information is known in Non-Patent Document 1, but there is a problem that there is still room for further improvement in accuracy of this technology.

[0013] One embodiment of the present invention provides an image decoding device capable of improving the accuracy of a predicted image derived through motion compensation.

[0014] Technical Solution

[0015] In order to solve the above-mentioned problems, a prediction image generation device of one scheme of the present invention generates a prediction image by performing motion compensation on multiple reference images. The prediction image generation device includes a prediction image generation unit, which generates a prediction image with reference to a first reference image, a second reference image, and a gradient correction item. The prediction image generation unit derives a first weight coefficient to be multiplied by a pixel value of each pixel of the first reference image and a second weight coefficient to be multiplied by a pixel value of each pixel of the second reference image based on a picture interval between an object image as a prediction object and the first reference image, and a picture interval between the object image and the second reference image.

[0016] In addition, in order to solve the above-mentioned problem, a predicted image generation device according to one embodiment of the present invention generates a predicted image by performing motion compensation on a plurality of reference images, the predicted image generation device comprising a predicted image generation unit, the predicted image generation unit generating a predicted image with reference to a first reference image, a second reference image, and a gradient correction term, the predicted image generation unit deriving the gradient correction term with reference to each pixel value of a first gradient image representing a gradient along a first direction of the first reference image, a second gradient image representing a gradient along a second direction of the first reference image, a third gradient image representing a gradient along the first direction of the second reference image, and a fourth gradient image representing a gradient along the second direction of the second reference image, wherein each weight coefficient multiplied by each pixel value of the first to fourth gradient images is determined based on a first picture interval between a target image as a prediction target and the first reference image, and a second picture interval between the target image and the second reference image.

[0017] In order to solve the above-mentioned problem, a predicted image generation device according to one aspect of the present invention generates a predicted image by performing motion compensation on a plurality of reference images, the predicted image generation device comprising a predicted image generation unit, the predicted image generation unit generating a predicted image with reference to a first reference image, a second reference image, and a gradient correction term, the predicted image generation unit deriving the gradient correction term with reference to each pixel value of a first gradient image indicating a gradient along a first direction of the first reference image, a second gradient image indicating a gradient along a second direction of the first reference image, a third gradient image indicating a gradient along the first direction of the second reference image, and a fourth gradient image indicating a gradient along the second direction of the second reference image, wherein when a predetermined flag is a first value, each weight coefficient multiplied by each pixel value of the first to fourth gradient images is determined based on a first picture interval that is a picture interval between a target image to be predicted and the first reference image, and a second picture interval that is a picture interval between the target image and the second reference image, and when the predetermined flag is a second value, each weight coefficient multiplied by each pixel value of the first to fourth gradient images is a value equal to each other.

[0018] In addition, in order to solve the above-mentioned problems, a predicted image generation device according to one embodiment of the present invention generates a predicted image by performing motion compensation on a plurality of reference images, the predicted image generation device comprising a predicted image generation unit, the predicted image generation unit generating a predicted image with reference to a first reference image, a second reference image, and a gradient correction item, the predicted image generation unit deriving the gradient correction item with reference to each pixel value of a first gradient image representing a gradient along a first direction of the first reference image, a second gradient image representing a gradient along a second direction of the first reference image, a third gradient image representing a gradient along the first direction of the second reference image, and a fourth gradient image representing a gradient along a second direction of the second reference image, and each weight coefficient to be multiplied by each pixel value of the first to fourth gradient images is determined on a block basis.

[0019] In addition, in order to solve the above-mentioned problem, a predicted image generation device according to one aspect of the present invention generates a predicted image by performing motion compensation on a plurality of reference images, the predicted image generation device comprising a predicted image generation unit, the predicted image generation unit generating a predicted image with reference to a first reference image, a second reference image, and a gradient correction item, the predicted image generation unit deriving the gradient correction item with reference to each pixel value of a first gradient image representing a gradient along a first direction of the first reference image, a second gradient image representing a gradient along a second direction of the first reference image, a third gradient image representing a gradient along the first direction of the second reference image, and a fourth gradient image representing a gradient along the second direction of the second reference image, the predicted image generation unit comprising a first switching unit for switching a gradient filter used when deriving each gradient image.

[0020] In addition, in order to solve the above-mentioned problems, a predicted image generation device according to one embodiment of the present invention generates a predicted image by performing motion compensation on a plurality of reference images, the predicted image generation device comprising a predicted image generation unit, the predicted image generation unit generating a predicted image with reference to a first reference image, a second reference image, and a gradient correction term, the predicted image generation unit deriving the gradient correction term with reference to each pixel value of a first gradient image representing a gradient along a first direction of the first reference image, a second gradient image representing a gradient along a second direction of the first reference image, a third gradient image representing a gradient along the first direction of the second reference image, and a fourth gradient image representing a gradient along a second direction of the second reference image, the predicted image generation unit comprising a clipping unit clipping each weight coefficient multiplied by each pixel value of the first to fourth gradient images.

[0021] In order to solve the above-mentioned problem, a predicted image generation device according to one aspect of the present invention generates a predicted image by performing motion compensation on a plurality of reference images, the predicted image generation device comprising a predicted image generation unit, the predicted image generation unit generating a predicted image with reference to a first reference image, a second reference image, and a gradient correction term, the predicted image generation unit deriving the gradient correction term with reference to each pixel value of a first gradient image indicating a gradient along a first direction of the first reference image, a second gradient image indicating a gradient along a second direction of the first reference image, a third gradient image indicating a gradient along the first direction of the second reference image, and a fourth gradient image indicating a gradient along the second direction of the second reference image, the weight coefficients to be multiplied by each pixel value of the first to fourth gradient images are determined using an intermediate parameter determined with reference to at least any one of the first to fourth gradient images and a normalization term added to the intermediate parameter, and the predicted image generation unit comprising a second switching unit for switching the value of the normalization term.

[0022] In addition, in order to solve the above-mentioned problems, a prediction image generating device of one scheme of the present invention generates a prediction image by performing motion compensation on multiple reference images, and the prediction image generating device comprises: a prediction vector calculation unit; and a prediction image generating unit, which generates a prediction image with reference to a first reference image, a second reference image, and a gradient correction item, the prediction vector calculation unit uses a prediction mode selected from one or more prediction modes including a matching prediction mode to derive a motion vector, and the prediction image generating unit switches the accuracy of the motion vector used for deriving the motion compensated image according to whether BIO prediction is applied when correcting the motion compensated image derived from the motion vector obtained in the matching prediction mode.

[0023] Beneficial Effects

[0024] According to the above configuration, it is possible to improve the accuracy of the predicted image derived by motion compensation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a diagram showing the hierarchical structure of data of a coded stream according to this embodiment.

[0026] Figure 2 It is a diagram showing a pattern of PU partitioning modes. Figure 2 (a) (referred to as Figure 2 (a) In the following text, similar figures are referred to by the same abbreviation to avoid redundancy. Figure 2 (h) shows the partition shapes when the PU partition modes are 2N×2N, 2N×N, 2N×nU, 2N×nD, N×2N, nL×2N, nR×2N, and N×N, respectively.

[0027] Figure 3 This is a conceptual diagram showing an example of a reference picture list.

[0028] Figure 4 This is a conceptual diagram showing an example of a reference picture.

[0029] Figure 5 This is a schematic diagram showing the structure of the image decoding device according to this embodiment.

[0030] Figure 6 This is a schematic diagram showing the structure of the inter-frame prediction parameter decoding unit according to this embodiment.

[0031] Figure 7 This is a schematic diagram showing the structure of the combined prediction parameter derivation unit of this embodiment.

[0032] Figure 8 It is a schematic diagram showing the structure of the AMVP prediction parameter derivation unit of this embodiment.

[0033] Fig. 9 is a conceptual diagram showing an example of vector candidates.

[0034] Fig.10 This is a schematic diagram showing the structure of the inter-frame prediction parameter decoding control unit according to this embodiment.

[0035] Fig.11 This is a schematic diagram showing the structure of the inter-frame prediction image generation unit according to this embodiment.

[0036] Fig.12 This is a block diagram showing the structure of the image encoding device according to this embodiment.

[0037] Fig.13 This is a schematic diagram showing the structure of the inter-frame prediction parameter encoding unit according to this embodiment.

[0038] Fig.14 It is a schematic diagram showing the configuration of an image transmission system according to an embodiment of the present invention.

[0039] Fig.15 This is a flowchart showing an example of switching prediction modes in an image decoding device having motion compensation using BIO prediction.

[0040] Fig.16 This is a flowchart for explaining the flow of processing for deriving a predicted image by the motion compensation unit 3091 having a motion compensation function using BIO prediction.

[0041] Fig.17 : is a diagram showing an example of a process of deriving the concept of gradient change.

[0042] Fig.18 This is a diagram for explaining a method of deriving a correction weight vector depending on a target image and a reference image.

[0043] Fig.19 (a) is a specific example of motion compensation filter mcFilter, Fig.19 (b) is a specific example of gradient compensation filtering.

[0044] Fig. 20 This is a diagram for explaining a method of deriving a correction weight vector using the least squares method.

[0045] Fig.21 This is a block diagram showing a configuration example of a motion compensation unit.

[0046] Fig. 22 This is a flowchart showing an example of the flow of the BIO prediction process performed by the motion compensation unit.

[0047] Fig.23 Yes means Fig. 22 A flowchart of an example of the process of S22a.

[0048] Fig.24 This is a block diagram showing another example of the configuration of the motion compensation unit.

[0049] Fig.25 Yes means Fig. 22 A flowchart of another example of the process of S22a.

[0050] Fig.26 Yes means Fig. 22 This is a flowchart of another example of the step S22a.

[0051] Fig. 27 (a) is an example of a high-resolution filter group. Fig. 27 (b) is an example of a low-resolution filter group.

[0052] Fig.28 yes Fig. 27 Graphs showing the yFrac values ​​of the high-resolution filter group and the low-resolution filter group are shown.

[0053] Fig.29 This is a flowchart showing an example of the flow of a process for switching filter groups when the resolution is dependent.

[0054] Fig.30 It is a flowchart showing the flow of motion vector derivation processing.

[0055] Fig.31 (a) is a diagram showing an example of switching the normalization term when the filter group is dependent. Fig.31(b) is a diagram showing an example of switching the range of clipping processing when the filter group is dependent.

[0056] Fig.32 This is a graph showing the comparison of the prediction accuracy of the BIO prediction model and the matching model.

[0057] Fig.33 It is a diagram showing the configuration of a transmitting device equipped with the above-mentioned image encoding device, and a receiving device equipped with the above-mentioned image decoding device. Fig.33 (a) represents a transmitting device equipped with an image encoding device, Fig.33 (b) shows a receiving device equipped with an image decoding device.

[0058] Fig.34 It is a diagram showing the structure of a recording device equipped with the above-mentioned image encoding device and a reproduction device equipped with the above-mentioned image decoding device. Fig.34 (a) represents a recording device equipped with an image encoding device, Fig.34 (b) shows a playback device equipped with an image decoding device. DETAILED DESCRIPTION

[0059] (First Embodiment)

[0060] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0061] Fig.14 It is a schematic diagram showing the configuration of the image transmission system 1 according to the present embodiment.

[0062] The image transmission system 1 is a system for transmitting a code obtained by encoding an encoding target image and displaying an image obtained by decoding the transmitted code. The image transmission system 1 includes an image encoding device 11 (motion image encoding device), a network 21, an image decoding device 31 (motion image decoding device), and an image display device 41.

[0063] The image coding device 11 inputs a signal T indicating a single-layer or multi-layer image. Layer is a concept used to distinguish multiple pictures when there is more than one picture constituting a certain time. For example, when the same picture is encoded in multiple layers with different image quality and resolution, it becomes scalable coding, and when pictures from different viewpoints are encoded in multiple layers, it becomes view scalable coding. When prediction (inter-layer prediction, inter-viewpoint prediction) is performed between pictures in multiple layers, the coding efficiency is greatly improved. In addition, even in the case of no prediction (simulcast), the coded data can be aggregated.

[0064] It should be noted that the image encoding device 11 and the image decoding device 31 may be single-layer images, or may arbitrarily combine scalable coding and view scalable coding.

[0065] The network 21 transmits the coded stream Te generated by the image coding device 11 to the image decoding device 31. The network 21 is the Internet, a wide area network (WAN), a small network (LAN) or a combination of these. The network 21 is not necessarily limited to a two-way communication network, and may be a one-way or two-way communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. In addition, the network 21 may be replaced by a storage medium such as a DVD (Digital Versatile Disc) or a BD (Blue-ray Disc) that records the coded stream Te.

[0066] The image decoding device 31 decodes each coded stream Te transmitted through the network 21 , and generates one or more decoded layer images Td (decoded viewpoint images Td) that have been respectively decoded.

[0067] The image display device 41 displays all or part of one or more decoded layer images Td generated by the image decoding device 31. For example, in viewpoint scalable coding, a three-dimensional image (stereoscopic image) or a free viewpoint image is displayed when all is displayed, and a two-dimensional image is displayed when a part is displayed. The image display device 41 includes display devices such as a liquid crystal display and an organic EL (Electro-luminescence) display. In addition, in spatial scalable coding and signal-to-noise ratio (SNR) scalable coding, when the image decoding device 31 and the image display device 41 have high processing capabilities, an enhanced layer image with high image quality is displayed, and when they have only low processing capabilities, a base layer image (base layer image) that does not require high processing capabilities and display capabilities of the enhancement layer is displayed.

[0068] <Structure of coded stream Te>

[0069] Before describing in detail the image encoding device 11 and the image decoding device 31 of the present embodiment, the data structure of the coded stream Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described.

[0070] Figure 1: is a diagram showing the hierarchical structure of data in a coded stream Te. The coded stream Te exemplarily includes a sequence and a plurality of pictures constituting the sequence. Figure 1 (a)~ Figure 1 (f) is a diagram that respectively represents a sequence layer of a predetermined sequence SEQ, a picture layer of a predetermined picture PICT, a slice layer of a predetermined slice S, a slice data layer of predetermined slice data, a coding tree layer of a coding tree unit included in the slice data, and a coding unit layer of a coding unit (CU: Coding Unit) included in the coding tree.

[0071] (Sequence Layer)

[0072] In the sequence layer, a set of data that the image decoding device 31 refers to for decoding a sequence SEQ to be processed (hereinafter also referred to as a target sequence) is defined. Figure 1 As shown in (a), it includes a video parameter set (VideoParameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (PictureParameter Set), a picture PICT, and supplemental enhancement information SEI (Supplemental EnhancementInformation). Here, the value after # indicates the layer ID. Figure 1 , an example is shown in which #0 and #1, that is, the encoded data of layer 0 and layer 1 exist, but the type of layer and the number of layers are not limited thereto.

[0073] The video parameter set VPS defines a set of coding parameters common to a plurality of moving images, and a set of coding parameters associated with a plurality of layers included in the moving image and each layer, in a moving image.

[0074] In the sequence parameter set SPS, a set of coding parameters that the image decoding device 31 refers to for decoding the object sequence is specified. For example, the width and height of the picture are specified. It should be noted that there may be multiple SPSs. In this case, any one of the multiple SPSs is selected from the PPS.

[0075] The picture parameter set PPS specifies a set of coding parameters that the image decoding device 31 refers to for decoding each picture in the target sequence. For example, it includes a reference value (pic_init_qp_minus26) of the quantization width (quantization step) used for decoding the picture and a flag (weighted_pred_flag) indicating the application of weighted prediction. It should be noted that there may be multiple PPSs. In this case, any one of the multiple PPSs is selected from each picture in the target sequence.

[0076] (Image layer)

[0077] In the picture layer, a set of data that the image decoding device 31 refers to for decoding a picture PICT (hereinafter also referred to as a target picture) to be processed is defined. Figure 1 As shown in (b), it includes slices S0 to SNS-1 (NS is the total number of slices included in the picture PICT).

[0078] It should be noted that, in the following, when it is not necessary to distinguish the slices S0 to SNS-1, the code subscript may be omitted and described. The same applies to other data included in the coded stream Te and marked with a subscript described below.

[0079] (Slice layer)

[0080] In the slice layer, a set of data that the image decoding device 31 refers to for decoding a slice S (also referred to as a target slice) to be processed is defined. Figure 1 As shown in (c), it includes a slice header SH and slice data SDATA.

[0081] The slice header SH includes a coding parameter group that the image decoding device 31 refers to in order to determine a decoding method for a target slice. Slice type designation information (slice_type) that designates a slice type is an example of a coding parameter included in the slice header SH.

[0082] Examples of slice types that can be specified by slice type designation information include: (1) I slices that use only intra-frame prediction during encoding, (2) P slices that use unidirectional prediction (uniprediction) or intra-frame prediction during encoding, and (3) B slices that use unidirectional, bi-prediction, or intra-frame prediction during encoding.

[0083] It should be noted that the slice header SH may also include a reference (pic_parameter_set_id) to the picture parameter set PPS included in the above sequence layer.

[0084] (Slice data layer)

[0085] The slice data layer defines a set of data that the image decoding device 31 refers to for decoding the slice data SDATA to be processed. Figure 1As shown in (d), a coded tree block (CTB) is included. A CTB is a block of a fixed size (for example, 64×64) constituting a slice, and is sometimes also referred to as a largest coding unit (LCU) or a coded tree unit (CTU).

[0086] (Coding tree layer)

[0087] The coding tree layer is as follows Figure 1 As shown in (e), a set of data that the image decoding device 31 refers to for decoding the coding tree block of the processing object is specified. The coding tree block is divided by recursive quadtree partitioning. The nodes of the tree structure obtained by recursive quadtree partitioning are called coding trees. The middle node of the quadtree is the coding quadtree (CQT: Coded Quad Tree), and the coding tree block itself is also specified as the top-level CQT. The CQT includes a split flag (split_flag), and when split_flag is 1, it is divided into 4 CQTs. When split_flag is 0, the CQT is not split, but has a coding unit (CU: Coded Unit) as a node. The coding unit CU is the end node of the coding tree layer, and no further partitioning is performed in this layer. The coding unit CU becomes the basic unit of the coding process.

[0088] In addition, when the size of the coding tree block CTB is 64×64 pixels, the size of the coding unit may be any one of 64×64 pixels, 32×32 pixels, 16×16 pixels, and 8×8 pixels.

[0089] (Coding unit layer)

[0090] The coding unit layer is as follows Figure 1 As shown in (f), a set of data that the image decoding device 31 refers to for decoding the coding unit to be processed is specified. Specifically, the coding unit is composed of a coding tree, a prediction tree, a transform tree, and a CU header CUF. The coding tree specifies a split flag, a split mode, a prediction mode, etc.

[0091] In the prediction tree, prediction information (reference picture index, motion vector, etc.) of each prediction block obtained by dividing the coding unit into one or more is specified. In other terms, a prediction block is one or more non-repeating regions constituting a coding unit. In addition, the prediction tree contains one or more prediction blocks obtained by the above-mentioned division. It should be noted that the prediction unit obtained by further dividing the prediction block is referred to as a "sub-block" below. A sub-block (prediction block) is composed of one or more pixels. When the size of the prediction block is equal to that of the sub-block, there is one sub-block in the prediction block. When the size of the prediction block is larger than the size of the sub-block, the prediction block is divided into sub-blocks. For example, when the prediction block is 8×8 and the sub-block is 4×4, the prediction block is divided into two parts horizontally and two parts vertically, thereby being divided into four sub-blocks.

[0092] The prediction process is performed for each prediction block (sub-block). Hereinafter, a prediction block as a prediction unit is also referred to as a prediction unit (PU).

[0093] There are two types of partitioning in the prediction tree: intra prediction and inter prediction. Intra prediction is prediction within the same picture, while inter prediction is prediction between different pictures (for example, between display times, between layer images).

[0094] In the case of intra prediction, the partitioning methods are 2N×2N (same size as the coding unit) and N×N.

[0095] In addition, in the case of inter-frame prediction, the partitioning method is encoded according to the PU partitioning mode (part_mode) of the encoded data, and the partitioning includes 2N×2N (the same size as the coding unit), 2N×N, 2N×nU, 2N×nD, N×2N, nL×2N, nR×2N, and N×N. It should be noted that 2N×nU means that the 2N×2N coding unit is divided into two regions of 2N×0.5N and 2N×1.5N from top to bottom. 2N×nD means that the 2N×2N coding unit is divided into two regions of 2N×1.5N and 2N×0.5N from top to bottom. nL×2N means that the 2N×2N coding unit is divided into two regions of 0.5N×2N and 1.5N×2N from left to right. nR×2N means that the 2N×2N coding unit is divided into two regions, 1.5N×2N and 0.5N×1.5N, from left to right. The number of divisions is one, two, or four, so the number of PUs included in the CU is one to four. These PUs are sequentially represented as PU0, PU1, PU2, and PU3.

[0096] Figure 2 (a)~ Figure 2In (h), the position of the boundary of the PU partition in the CU is specifically shown for each partition type.

[0097] It should be noted that Figure 2 (a) shows a 2N×2N PU partition mode in which CU partitioning is not performed.

[0098] also, Figure 2 (b) Figure 2 (c), and Figure 2 (d) shows the shapes of partitions when the PU partition modes are 2N×N, 2N×nU, and 2N×nD. Hereinafter, partitions when the PU partition modes are 2N×N, 2N×nU, and 2N×nD are collectively referred to as horizontally long partitions.

[0099] also, Figure 2 (e) Figure 2 (f), and Figure 2 (g) shows the shapes of partitions when the PU partition mode is N×2N, nL×2N, and nR×2N, respectively. Hereinafter, partitions when the PU partition mode is N×2N, nL×2N, and nR×2N are collectively referred to as vertical partitions.

[0100] In addition, the horizontally long partitions and the vertically long partitions are collectively referred to as rectangular partitions.

[0101] also, Figure 2 (h) shows the shape of the partition when the PU partition mode is N×N. Based on the shape of the partition, Figure 2 (a) and Figure 2 The PU partitioning mode of (h) is called square partitioning. Figure 2 (b)~ Figure 2 The PU partitioning mode of (g) is called non-square partitioning.

[0102] In addition, Figure 2 (a)~ Figure 2 In (h), the number assigned to each region indicates the identification number of the region, and the regions are processed in the order of the identification numbers. That is, the identification numbers indicate the scanning order of the partitions.

[0103] In addition, Figure 2 (a)~ Figure 2 In (h), the upper left is assumed to be the reference point (origin) of the CU.

[0104] In addition, in the transform tree, the coding unit is divided into one or more transform blocks, and the position and size of each transform block are specified. If expressed in other terms, the transform block is one or more non-repeating regions constituting the coding unit. In addition, the transform tree contains one or more transform blocks obtained by the above-mentioned division.

[0105] The transform tree partitioning includes partitioning in which a region having the same size as a coding unit is allocated as a transform block, and partitioning by recursive quadtree partitioning similar to the above-mentioned tree block partitioning.

[0106] The transform process is performed on each transform block. Hereinafter, a transform block as a unit of transform is also referred to as a transform unit (TU).

[0107] (Prediction parameters)

[0108] The prediction image of the prediction unit is derived from the prediction parameters attached to the prediction unit. The prediction parameters include prediction parameters for intra-frame prediction or prediction parameters for inter-frame prediction. The prediction parameters for inter-frame prediction (inter-frame prediction parameters) are described below. The inter-frame prediction parameters are composed of prediction list utilization flags predFlagL0, predFlagL1, reference picture indexes refIdxL0, refIdxL1, and vectors mvL0, mvL1. The prediction list utilization flags predFlagL0, predFlagL1 are flags indicating whether to use the reference picture lists called L0 list and L1 list, respectively. When the value is 1, the corresponding reference picture list is used. It should be noted that when it is described as "a flag indicating whether it is XX" in this specification, 1 is set to XX, and 0 is set to not XX. In logical NOT, logical multiplication, etc., 1 is treated as true and 0 is treated as false (the same applies below). However, other values ​​can also be used as true values ​​and false values ​​in actual devices and methods. When two reference picture lists are used, that is, the case where predFlagL0=1 and predFlagL1=1 corresponds to dual prediction, and when one reference picture list is used, that is, the case where (predFlagL0, predFlagL1)=(1,0) or (predFlagL0, predFlagL1)=(0,1) corresponds to uniprediction. It should be noted that the information of the prediction list utilization flag can also be represented by the inter-frame prediction flag inter_pred_idc described later. Usually, the prediction list utilization flag is used in the prediction image generation unit (prediction image generation device) 308 and the prediction parameter memory 307 described later, and the inter-frame prediction flag inter_pred_idc is used when decoding the information on which reference picture list to use based on the encoded data.

[0109] The syntax elements used to derive the inter-frame prediction parameters included in the encoded data include, for example, a partition mode part_mode, a merge flag merge_flag, a merge index merge_idx, an inter-frame prediction flag inter_pred_idc, a reference picture index refIdxLX, a prediction vector index mvp_LX_idx, and a difference vector mvdLX.

[0110] (See an example of a picture list)

[0111] Next, an example of a reference picture list is described. The reference picture list is stored in the reference picture memory 306 ( Figure 5 )'s reference image (reference image). Figure 3 601 is a conceptual diagram showing an example of a reference picture list. In the reference picture list 601, the five rectangles arranged in a row from left to right represent reference pictures. The codes P1, P2, Q0, P3, and P4 shown in sequence from the left end are codes representing each reference picture. P in P1, etc. represents viewpoint P, and Q in Q0 represents viewpoint Q different from viewpoint P. The subscripts of P and Q represent the picture sequence number POC. The downward arrow directly below refIdxLX indicates that the reference picture index refIdxLX is the index of the reference picture Q0 in the reference picture memory 306.

[0112] (See the example in the picture)

[0113] Next, an example of a reference picture used when deriving a vector is described. Figure 4 is a conceptual diagram showing an example of a reference image. Figure 4 In the figure, the horizontal axis represents the display time and the vertical axis represents the viewpoint. Figure 4 The rectangles in two vertical rows and three horizontal columns (a total of six) shown represent pictures respectively. Among the six rectangles, the rectangles in the second column from the left of the bottom row represent pictures of decoding objects (object pictures), and the remaining five rectangles represent reference pictures respectively. The reference picture Q0 represented by an upward arrow from the object picture is a picture with the same display time as the object picture but a different viewpoint. In displacement prediction based on the object picture as a reference, the reference picture Q0 is used. The reference picture P1 represented by an arrow pointing left from the object picture is a past picture with the same viewpoint as the object picture. The reference picture P2 represented by an arrow pointing right from the object picture is a future picture with the same viewpoint as the object picture. In motion prediction based on the object picture as a reference, the reference picture P1 or P2 is used.

[0114] (Inter prediction flag and prediction list use flag)

[0115] The relationship between the inter-frame prediction flag and the prediction list utilization flags predFlagL0 and predFlagL1 is as follows, and they can be interchanged. Therefore, as an inter-frame prediction parameter, the prediction list utilization flag can be used, and the inter-frame prediction flag can also be used. In addition, below, the judgment using the prediction list utilization flag can be replaced by the inter-frame prediction flag. Conversely, the judgment using the inter-frame prediction flag can also be replaced by the prediction list utilization flag.

[0116] Inter prediction flag = (predFlagL1<<1) + predFlagL0

[0117] predFlagL0 = inter prediction flag & 1

[0118] predFlagL1=inter prediction flag >> 1

[0119] Here, >> is a right shift and << is a left shift.

[0120] (Combined forecast and AMVP forecast)

[0121] The decoding (encoding) method of the prediction parameters includes a merge prediction mode and an AMVP (Adaptive Motion Vector Prediction) mode, and the merge flag merge_flag is a flag used to identify these. Regardless of the merge prediction mode or the AMVP mode, the prediction parameters of the block that has been processed are used to derive the prediction parameters of the object PU. The merge prediction mode is a mode that directly uses the derived prediction parameters of the nearby PU without including the prediction list utilization flag predFlagLX (or the inter-frame prediction flag inter_pred_idc), the reference picture index refIdxLX, and the motion vector mvLX in the encoded data. The AMVP mode is a mode that includes the inter-frame prediction flag inter_pred_idc, the reference picture index refIdxLX, and the motion vector mvLX in the encoded data. It should be noted that the motion vector mvLX is encoded as the prediction vector index mvp_LX_idx that identifies the prediction vector mvpLX and the differential vector mvdLX.

[0122] The inter-frame prediction flag inter_pred_idc is data indicating the type and number of reference pictures, and takes any value of PRED_L0, PRED_L1, and PRED_BI. PRED_L0 and PRED_L1 indicate the use of reference pictures stored in reference picture lists called L0 list and L1 list, respectively, and indicate that one reference picture is used for both (single prediction). The predictions using the L0 list and L1 list are called L0 prediction and L1 prediction, respectively. PRED_BI indicates the use of two reference pictures (double prediction), and indicates the use of two reference pictures stored in the L0 list and the L1 list. The prediction vector index mvp_LX_idx is an index indicating the prediction vector, and the reference picture index refIdxLX is an index indicating the reference picture stored in the reference picture list. It should be noted that LX is a description method used when the L0 prediction and the L1 prediction are not distinguished. By replacing LX with L0 and L1, the parameters for the L0 list and the parameters for the L1 list are distinguished. For example, refIdxL0 is a reference picture index used for L0 prediction, refIdxL1 is a reference picture index used for L1 prediction, and refIdx(refIdxLX) is a flag used when refIdxL0 and refIdxL1 are not to be distinguished.

[0123] The merge index merge_idx is an index indicating whether any prediction parameter among prediction parameter candidates (merge candidates) derived from a processed block is used as a prediction parameter of a decoding target block.

[0124] It should be noted that the “object block” may be a prediction block that is one level higher than the multiple prediction blocks, or may be a coding unit that includes the multiple prediction blocks.

[0125] (Motion vector and displacement vector)

[0126] The motion vector mvLX can be divided into a narrow motion vector (motion vector in the narrow sense) representing the offset between blocks on two pictures at different times, and a displacement vector (disparity vector) representing the offset between two blocks at the same time. In the following description, the motion vector and the displacement vector are not distinguished, and are simply referred to as the motion vector mvLX. The prediction vector and the differential vector related to the motion vector mvLX are respectively called the prediction vector mvpLX and the differential vector mvdLX. Whether the motion vector mvLX and the differential vector mvdLX are motion vectors or displacement vectors is identified using the reference picture index refIdxLX attached to the vector.

[0127] (Configuration of Image Decoding Device)

[0128] Next, the configuration of the image decoding device 31 according to this embodiment will be described. Figure 5 : is a schematic diagram showing the structure of the image decoding device 31 of the present embodiment. The image decoding device 31 includes an entropy decoding unit 301, a prediction parameter decoding unit 302, a reference picture memory (reference image storage unit, frame memory) 306, a prediction parameter memory (prediction parameter storage unit, frame memory) 307, a prediction image generation unit 308, an inverse quantization / inverse DCT unit 311, an addition unit 312, and a residual storage unit 313 (residual recording unit).

[0129] The prediction parameter decoding unit 302 includes an inter prediction parameter decoding unit 303 and an intra prediction parameter decoding unit 304. The prediction image generating unit 308 includes an inter prediction image generating unit 309 and an intra prediction image generating unit 310.

[0130] The entropy decoding unit 301 performs entropy decoding on the coded stream Te input from the outside, separates and decodes each code (syntax element). The separated code includes prediction information for generating a prediction image and residual information for generating a difference image.

[0131] The entropy decoding unit 301 outputs a part of the separated code to the prediction parameter decoding unit 302. The part of the separated code is, for example, the prediction mode PredMode, the partition mode part_mode, the merge flag merge_flag, the merge index merge_idx, the inter-frame prediction flag inter_pred_idc, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx, and the difference vector mvdLX. The control of which code to decode is performed based on the instruction of the prediction parameter decoding unit 302. The entropy decoding unit 301 outputs the quantization coefficient to the inverse quantization / inverse DCT unit 311. The quantization coefficient is a coefficient obtained by performing DCT (Discrete Cosine Transform) on the residual signal and quantizing it in the encoding process.

[0132] The inter prediction parameter decoding unit 303 decodes the inter prediction parameter with reference to the prediction parameter stored in the prediction parameter memory 307 based on the code input from the entropy decoding unit 301 .

[0133] The inter prediction parameter decoding unit 303 outputs the decoded inter prediction parameter to the predicted image generation unit 308, and stores it in the prediction parameter memory 307. The inter prediction parameter decoding unit 303 will be described in detail below.

[0134] The intra prediction parameter decoding unit 304 decodes the intra prediction parameter based on the code input from the entropy decoding unit 301, referring to the prediction parameter stored in the prediction parameter memory 307. The intra prediction parameter is a parameter used in the process of predicting a picture block in one picture, for example, the intra prediction mode IntraPredMode. The intra prediction parameter decoding unit 304 outputs the decoded intra prediction parameter to the prediction image generation unit 308, and stores it in the prediction parameter memory 307.

[0135] The intra-frame prediction parameter decoding unit 304 can also derive intra-frame prediction modes that are different in brightness and color difference. In this case, the intra-frame prediction parameter decoding unit 304 decodes the brightness prediction mode IntraPredModeY as the prediction parameter of brightness, and decodes the color difference prediction mode IntraPredModeC as the prediction parameter of color difference. The brightness prediction mode IntraPredModeY is 35 modes, and plane prediction (0), DC prediction (1), and directional prediction (2 to 34) correspond to it. The chroma prediction mode IntraPredModeC uses any one of plane prediction (0), DC prediction (1), directional prediction (2 to 34), and LM mode (35). The intra-frame prediction parameter decoding unit 304 decodes a flag indicating whether IntraPredModeC is the same mode as the brightness mode. If the flag indicates that it is the same mode as the brightness mode, IntraPredModeY is assigned to IntraPredModeC. If the flag indicates that it is a mode different from the brightness mode, the intra-frame prediction parameter decoding unit 304 can also decode planar prediction (0), DC prediction (1), directional prediction (2~34), and LM mode (35) as IntraPredModeC.

[0136] The reference picture memory 306 stores the blocks of the reference picture (reference picture blocks) generated by the adding unit 312 at predetermined positions for each picture and block to be decoded.

[0137] The prediction parameter memory 307 stores the prediction parameters in a predetermined position according to the picture and block of the decoding object. Specifically, the prediction parameter memory 307 stores the inter-frame prediction parameters decoded by the inter-frame prediction parameter decoding unit 303, the intra-frame prediction parameters decoded by the intra-frame prediction parameter decoding unit 304, and the prediction mode predMode separated by the entropy decoding unit 301. The stored inter-frame prediction parameters include, for example, a prediction list utilization flag predFlagLX (inter-frame prediction flag inter_pred_idc), a reference picture index refIdxLX, and a motion vector mvLX.

[0138] The prediction mode predMode input from the entropy decoding unit 301 is input to the prediction image generation unit 308, and the prediction parameters are input from the prediction parameter decoding unit 302. In addition, the prediction image generation unit 308 reads the reference picture from the reference picture memory 306. The prediction image generation unit 308 generates a prediction picture block P (prediction image) using the input prediction parameters and the read reference picture in the prediction mode indicated by the prediction mode predMode.

[0139] Here, when the prediction mode predMode indicates the inter prediction mode, the inter prediction image generator 309 generates a prediction picture block P by inter prediction using the inter prediction parameters input from the inter prediction parameter decoder 303 and the reference picture read out. The prediction picture block P corresponds to the prediction unit PU. The PU corresponds to a part of a picture formed of a plurality of pixels that is a unit for performing a prediction process as described above, that is, corresponds to a decoding target block for performing a prediction process once.

[0140] The inter-frame prediction image generator 309 reads a reference picture block located at a position indicated by a motion vector mvLX based on the decoding target block from the reference picture memory 306, based on the reference picture indicated by the reference picture index refIdxLX, with respect to the reference picture list (L0 list or L1 list) in which the prediction list utilization flag predFlagLX is 1. The inter-frame prediction image generator 309 predicts the read reference picture block and generates a prediction picture block P. The inter-frame prediction image generator 309 outputs the generated prediction picture block P to the adder 312.

[0141] When the prediction mode predMode indicates an intra-frame prediction mode, the intra-frame prediction image generation unit 310 uses the intra-frame prediction parameters input from the intra-frame prediction parameter decoding unit 304 and the read reference picture to perform intra-frame prediction. Specifically, the intra-frame prediction image generation unit 310 reads out a reference picture block that is a picture to be decoded and is located in a predetermined range from the decoded block in the already decoded block from the reference picture memory 306. The predetermined range is when the decoded block is moved sequentially in the order of so-called raster scanning, for example, any one of the adjacent blocks to the left, upper left, upper, and upper right, and it differs depending on the intra-frame prediction mode. The order of raster scanning is the order of moving from the left end to the right end for each row from the top to the bottom in each picture.

[0142] The intra-prediction image generator 310 predicts the read reference picture block in a prediction mode indicated by the intra-prediction mode IntraPredMode to generate a prediction picture block. The intra-prediction image generator 310 outputs the generated prediction picture block P to the adder 312 .

[0143] In the intra-frame prediction parameter decoding unit 304, when deriving intra-frame prediction modes that are different in brightness and color difference, the intra-frame prediction image generation unit 310 generates a prediction picture block of brightness according to the brightness prediction mode IntraPredModeY through any one of plane prediction (0), DC prediction (1), and directional prediction (2 to 34), and generates a prediction picture block of chroma according to the chroma prediction mode IntraPredModeC through any one of plane prediction (0), DC prediction (1), directional prediction (2 to 344), and LM mode (35).

[0144] The inverse quantization / inverse DCT unit 311 inversely quantizes the quantization coefficients input from the entropy decoding unit 301 to obtain DCT coefficients. The inverse quantization / inverse DCT unit 311 performs inverse DCT (Inverse Discrete Cosine Transform) on the obtained DCT coefficients to calculate a decoded residual signal. The inverse quantization / inverse DCT unit 311 outputs the calculated decoded residual signal to the addition unit 312 and the residual storage unit 313.

[0145] The adding unit 312 generates a reference picture block by adding pixel by pixel the predicted picture block P input from the inter-frame predicted picture generating unit 309 and the intra-frame predicted picture generating unit 310 and the signal value of the decoded residual signal input from the inverse quantization / inverse DCT unit 311. The adding unit 312 stores the generated reference picture block in the reference picture memory 306, and outputs a decoded layer image Td obtained by integrating the generated reference picture block on a picture by picture basis to the outside.

[0146] (Configuration of Inter-frame Prediction Parameter Decoding Unit)

[0147] Next, the configuration of the inter-frame prediction parameter decoding unit 303 will be described.

[0148] Figure 6 303 is a schematic diagram showing the structure of the inter prediction parameter decoding unit 303 of this embodiment. The inter prediction parameter decoding unit 303 includes an inter prediction parameter decoding control unit 3031, an AMVP prediction parameter derivation unit 3032, an addition unit 3035, and a merged prediction parameter derivation unit 3036.

[0149] The inter-frame prediction parameter decoding control unit 3031 instructs the entropy decoding unit 301 to decode the code (syntax elements) associated with the inter-frame prediction, and extracts the code (syntax elements) contained in the encoded data, such as the partition mode part_mode, the merge flag merge_flag, the merge index merge_idx, the inter-frame prediction flag inter_pred_idc, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx, and the differential vector mvdLX.

[0150] The inter-frame prediction parameter decoding control unit 3031 first extracts the merge flag. The inter-frame prediction parameter decoding control unit 3031 indicates that the case of extracting a certain syntax element is to instruct the entropy decoding unit 301 to decode a certain syntax element, and read the corresponding syntax element from the encoded data. Here, when the value indicated by the merge flag is 1, that is, indicating the merge prediction mode, the inter-frame prediction parameter decoding control unit 3031 extracts the merge index merge_idx as the prediction parameter of the merge prediction. The inter-frame prediction parameter decoding control unit 3031 outputs the extracted merge index merge_idx to the merge prediction parameter derivation unit 3036.

[0151] When the merge flag merge_flag is 0, that is, indicating the AMVP prediction mode, the inter-frame prediction parameter decoding control unit 3031 uses the entropy decoding unit 301 to extract the AMVP prediction parameters from the encoded data. As AMVP prediction parameters, there are, for example, the inter-frame prediction flag inter_pred_idc, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx, and the difference vector mvdLX. The inter-frame prediction parameter decoding control unit 3031 outputs the prediction list derived from the extracted inter-frame prediction flag inter_pred_idc to the AMVP prediction parameter derivation unit 3032 and the prediction image generation unit 308 ( Figure 5 ), and in addition, stored in the prediction parameter memory 307 ( Figure 5 ). The inter-frame prediction parameter decoding control unit 3031 outputs the extracted prediction vector index mvp_LX_idx to the AMVP prediction parameter derivation unit 3032. The inter-frame prediction parameter decoding control unit 3031 outputs the extracted difference vector mvdLX to the addition unit 3035.

[0152] Figure 7 : This is a schematic diagram showing the structure of the merge prediction parameter derivation unit 3036 of the present embodiment. The merge prediction parameter derivation unit 3036 includes a merge candidate derivation unit 30361 and a merge candidate selection unit 30362. The merge candidate storage unit 303611 stores the merge candidates input from the merge candidate derivation unit 30361. It should be noted that the merge candidate includes a prediction list using a flag predFlagLX, a motion vector mvLX, and a reference picture index refIdxLX. In the merge candidate storage unit 303611, indexes are assigned to the stored merge candidates according to a prescribed rule.

[0153] The merge candidate derivation unit 30361 directly uses the motion vector of the adjacent block that has been decoded and the reference picture index refIdxLX to derive the merge candidate. In addition, affine prediction can also be used to derive the merge candidate. The method is described in detail below. The merge candidate derivation unit 30361 can use affine prediction for the spatial merge candidate derivation process, temporal merge (inter-frame merge) candidate derivation process, combined merge candidate derivation process, and zero merge candidate derivation process described later. It should be noted that affine prediction is performed in sub-block units, and the prediction parameters are stored in the prediction parameter memory 307 according to the sub-block. Alternatively, affine prediction can also be performed in pixel units.

[0154] (Spatial Merge Candidate Derivation Process)

[0155] As a spatial merge candidate derivation process, the merge candidate derivation unit 30361 reads the prediction parameters (prediction list utilization flag predFlagLX, motion vector mvLX, reference picture index refIdxLX) stored in the prediction parameter memory 307 according to a prescribed rule, and derives the read prediction parameters as merge candidates. The read prediction parameters are prediction parameters of each block within a predetermined range from the decoding target block (for example, all or part of the blocks connected to the lower left end, upper left end, and upper right end of the decoding target block). The merge candidates derived by the merge candidate derivation unit 30361 are stored in the merge candidate storage unit 303611.

[0156] (Temporal Merge Candidate Derivation Process)

[0157] As a temporal merge derivation process, the merge candidate derivation unit 30361 reads the prediction parameter of the block in the reference image including the coordinates of the lower right of the decoding target block from the prediction parameter memory 307 as a merge candidate. The reference image may be specified by, for example, using the reference picture index refIdxLX specified in the slice header, or by using the smallest reference picture index among the reference picture indexes refIdxLX of the blocks adjacent to the decoding target block. The merge candidate derived by the merge candidate derivation unit 30361 is stored in the merge candidate storage unit 303611.

[0158] (Combined Merger Candidate Derivation Process)

[0159] As a combined merge derivation process, the merge candidate derivation unit 30361 derives a combined merge candidate by combining the vectors of two different completed merge candidates that have been derived and stored in the merge candidate storage unit 303611 and the reference picture index as vectors of L0 and L1, respectively. The merge candidate derived by the merge candidate derivation unit 30361 is stored in the merge candidate storage unit 303611.

[0160] (Zero Merging Candidate Derivation Process)

[0161] As a zero merge candidate derivation process, the merge candidate derivation unit 30361 derives a merge candidate whose reference picture index refIdxLX is 0 and whose X component and Y component of the motion vector mvLX are both 0. The merge candidate derived by the merge candidate derivation unit 30361 is stored in the merge candidate storage unit 303611 .

[0162] The merge candidate selection unit 30362 selects a merge candidate to which an index corresponding to the merge index merge_idx input from the inter-prediction parameter decoding control unit 3031 is assigned from the merge candidates stored in the merge candidate storage unit 303611 as the inter-prediction parameter of the target PU. The merge candidate selection unit 30362 stores the selected merge candidate in the prediction parameter memory 307 and outputs it to the prediction image generation unit 308 ( Figure 5 ).

[0163] Figure 8 1 is a schematic diagram showing the structure of the AMVP prediction parameter derivation unit 3032 of the present embodiment. The AMVP prediction parameter derivation unit 3032 includes a vector candidate derivation unit 3033 (vector calculation unit) and a vector candidate selection unit 3034. The vector candidate derivation unit 3033 reads out a vector (motion vector or displacement vector) stored in the prediction parameter memory 307 based on the reference picture index refIdx as a prediction vector mvpLX. The read vector is a vector of each block (for example, all or part of the blocks connected to the lower left end, upper left end, and upper right end of the decoding object block) within a predetermined range from the decoding object block.

[0164] The vector candidate selection unit 3034 selects, as the prediction vector mvpLX, the vector candidate indicated by the prediction vector index mvp_LX_idx input from the inter prediction parameter decoding control unit 3031 from among the vector candidates read out by the vector candidate derivation unit 3033 , and outputs the selected prediction vector mvpLX to the addition unit 3035 .

[0165] The AMVP prediction parameter derivation unit 3032 includes a vector candidate derivation unit 3033 and a vector candidate selection unit 3034. The vector candidate storage unit 30331 stores the vector candidates input from the vector candidate derivation unit 3033. It should be noted that the vector candidates include the prediction vector mvpLX. In the vector candidate storage unit 30331, indices are assigned to the stored vector candidates according to a prescribed rule.

[0166] The vector candidate derivation unit 3033 uses affine prediction to derive vector candidates. The vector candidate derivation unit 3033 may also use affine prediction for the spatial vector candidate derivation process, the temporal vector (inter-frame vector) candidate derivation process, the combined vector candidate derivation process, and the zero vector candidate derivation process described later. It should be noted that affine prediction is performed in sub-block units, and prediction parameters are stored in the prediction parameter memory 307 for each sub-block. Alternatively, affine prediction may also be performed in pixel units.

[0167] Fig. 9 is a conceptual diagram showing an example of vector candidates. Fig. 9 The predicted vector list 602 shown is a list formed of a plurality of vector candidates derived in the vector candidate derivation unit 3033. In the predicted vector list 602, five rectangles arranged in a row from left to right represent areas indicating predicted vectors. The downward arrow just below the second mvp_LX_idx from the left end and mvpLX below it indicate that the predicted vector index mvp_LX_idx is the index of the reference vector mvpLX in the prediction parameter memory 307.

[0168] The vector candidate is generated based on the vector of the block referenced by the vector candidate selection unit 3034. The block referenced by the vector candidate selection unit 3034 is a block that has completed the decoding process, and may also be a block in a predetermined range from the decoding target block (for example, an adjacent block). It should be noted that the adjacent block includes not only blocks that are spatially adjacent to the decoding target block, such as the left block and the upper block, but also blocks that are temporally adjacent to the decoding target block, such as blocks obtained from blocks at the same position as the decoding target block but at different display times.

[0169] The adding unit 3035 adds the prediction vector mvpLX input from the AMVP prediction parameter derivation unit 3032 and the difference vector mvdLX input from the inter-frame prediction parameter decoding control unit 3031 to derive the motion vector mvLX. The adding unit 3035 outputs the derived motion vector mvLX to the prediction image generation unit 308 ( Figure 5 ).

[0170] Fig.101 is a schematic diagram showing the structure of the inter prediction parameter decoding control unit 3031 of the present embodiment. The inter prediction parameter decoding control unit 3031 includes an additional prediction flag decoding unit 30311, a merge index decoding unit 30312, a vector candidate index decoding unit 30313, and a partition mode decoding unit, a merge flag decoding unit, an inter prediction flag decoding unit, a reference picture index decoding unit, a vector difference decoding unit, etc., which are not shown in the figure. The partition mode decoding unit, the merge flag decoding unit, the merge index decoding unit, the inter prediction flag decoding unit, the reference picture index decoding unit, the vector candidate index decoding unit 30313, and the vector difference decoding unit respectively decode the partition mode part_mode, the merge flag merge_flag, the merge index merge_idx, the inter prediction flag inter_pred_idc, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx, and the difference vector mvdLX.

[0171] (Inter-frame prediction image generation unit 309)

[0172] Fig.11 2 is a schematic diagram showing the configuration of the inter-prediction image generation unit 309 according to the present embodiment. The inter-prediction image generation unit 309 includes a motion compensation unit 3091 (prediction image generation device) and a weight prediction unit 3094 .

[0173] (Motion Compensation Unit)

[0174] The motion compensation unit 3091 reads a block located at a position offset by the motion vector mvLX from the position of the decoding target block of the reference picture specified by the reference picture index refIdxLX as a starting point from the reference picture memory 306 based on the prediction list utilization flag predFlagLX, the reference picture index refIdxLX, and the motion vector mvLX input from the inter-frame prediction parameter decoding unit 303, thereby generating a motion compensated image. Here, when the motion vector mvLX is not an integer vector, a filter called a motion compensation filter for generating pixels at a decimal position is implemented to generate a motion compensated image. The motion compensation filter includes a vertical motion compensation filter (mcFilterVer) and a horizontal motion compensation filter (mcFilterHor). Hereinafter, the motion compensated image of the L0 prediction is referred to as predSamplesL0, and the motion compensated image of the L1 prediction is referred to as predSamplesL1. When the two are not distinguished, they are referred to as predSamplesLX. It should be noted that an example of the configuration of the motion compensation unit 3091 will be described later.

[0175] (Weight prediction unit)

[0176] The weight prediction unit 3094 generates a prediction picture block P (prediction picture) by multiplying the input motion displacement image predSamplesLX by the weight coefficient. When one of the reference list utilization flags (predFlagL0 or predFlagL1) is 1 (in the case of single prediction), when weight prediction is not used, the following processing is performed to make the input motion displacement image predSamplesLX (LX is L0 or L1) consistent with the number of pixel bits.

[0177] predSamples[x][y]=Clip3(0, (1<<bitDepth)-1,(predSamplesLX[x][y]+offset1)> >shift1)

[0178] Here, shift1=14-bitDepth, offset1=1<<(shift1-1).

[0179] In addition, when both reference list utilization flags (predFlagL0 and predFlagL1) are 1 (double prediction), without using weighted prediction, the input motion displacement images predSamplesL0 and predSamplesL1 are averaged and the following processing is performed to make them consistent with the number of pixel bits.

[0180] predSamples[x][y]=Clip3(0, (1<<bitDepth)-1,(predSamplesL0[x][y]+predSamplesL1[x][y]+offset2)> >shift2)

[0181] Here, shift2=15-bitDepth, offset2=1<<(shift2-1).

[0182] Moreover, in the case of uni-prediction, when weighted prediction is performed, the weighted prediction unit 3094 derives a weighted prediction coefficient w0 and an offset value (offset) o0 from the encoded data, and performs the following processing.

[0183] predSamples[x][y]=Clip3(0, (1<<bitDepth)-1,((predSamplesLX[x][y]*w0+2log2WD-1)> >log2WD)+o0)

[0184] Here, log2WD is a variable representing a predetermined shift amount.

[0185] Moreover, in the case of bi-prediction, when weighted prediction is performed, the weighted prediction unit 3094 derives weighted prediction coefficients w0, w1, o0, o1 from the encoded data, and performs the following processing.

[0186] predSamples[x][y]=Clip3(0, (1< <bitDepth)-1,(predSamplesL0[x][y]*w0+predSamplesL1[x][y]*w1+((o0+o1+1)<<log2WD))> >(log2WD+1))

[0187] (Switching of prediction modes in a motion compensation unit equipped with gradient change prediction)

[0188] Fig.16 This is a flowchart for explaining the process of deriving a predicted image by the motion compensation unit 3091 having a motion compensation function using a gradient change (BIO) prediction. Fig.16 2 shows the processing flow of the inter-frame prediction parameter decoding unit 303 determining whether the motion compensation unit 3091 applies BIO prediction to derive a predicted image in STEP 3 described later.

[0189] When the inter-frame prediction parameter decoding unit 303 determines that it is not a bi-prediction mode (No in S301, i.e., uni-prediction UniPred), the process proceeds to S305, and the motion compensation unit 3091 performs unidirectional motion compensation. On the other hand, when the inter-frame prediction parameter decoding unit 303 determines that it is a bi-prediction mode (Yes in S301, BiPred), the inter-frame prediction parameter decoding unit 303 continues the determination shown in S302.

[0190] If the L0 reference image refImgL0 and the L1 reference image refImgL1 obtained from the picture memory 306 ( Fig.16 If the two prediction blocks (described as "two prediction blocks") are different reference images (Yes in S302), the process proceeds to S303, and the motion compensation unit 3091 performs motion compensation using BIO prediction to be described later. On the other hand, if the L0 reference image refImgL0 and the L1 reference image refImgL1 obtained from the picture memory 306 are the same reference image (No in S302), the process proceeds to S304, and the motion compensation unit 3091 performs bidirectional motion compensation without applying BIO prediction.

[0191] (Basic concept of gradient change)

[0192] Below, use Fig.17 The gradient change (Optical Flow) is described. Fig.17: is a diagram showing an example of a process of deriving the concept of gradient change.

[0193] When it is assumed that the pixel value I(x, y, t) does not change even if it moves by Δx and Δy during the time interval Δt, that is, the pixel value does not change but only its position changes, Fig.17 The equation 1 is established.

[0194] The change of pixel value at a certain point is as follows Fig.17 As shown in Formula 2, it can be decomposed into gradient change and time change. Here, the gradient change can be expressed as the sum of the partial differential of the pixel value I in the x direction and the partial differential in the y direction, and the time change can be expressed as the partial differential of the pixel value I in the t direction.

[0195] Through the above formula 1 and formula 2, we can deduce Fig.17 Then, by dividing both sides of the equation 3 by the time interval Δt, we can deduce Fig.17 Formula 4.

[0196] Here, if the change in position per unit time in the x direction (e.g., horizontal direction) (Δx / Δt) and the change in position per unit time in the y direction (e.g., vertical direction) (Δy / Δt) are denoted as Vx and Vy, respectively, then Fig.17 Equation 4 can be expressed as an equation in the form of Equation 5.

[0197] This formula 5 indicates that, assuming that the pixel value does not change but only the position changes, the value obtained by multiplying the change in the x direction of the pixel value I by Vx, the value obtained by multiplying the change in the y direction of the pixel value I by Vy, and the value obtained by adding the change in the time of the pixel value I are always 0. That is, under the assumption that the pixel value does not change, the change in gradient and the change in time have a mutually canceling (balanced) relationship. The formula for gradient change indicates that the time change of the pixel value I can be predicted based on the product of the change in the x direction of the pixel value I, the change in position Vx (=Δx / Δt), the change in the y direction, and the change in position Vy (=Δx / Δt). In the following processing description, the change in the x direction of the pixel value I is referred to as the horizontal gradient value lx0, lx1, the change in the y direction of the pixel value I is referred to as the vertical gradient value ly0, ly1, and the change in position (Vx, Vy) is referred to as the correction vector (u, v).

[0198] (About BIO forecast)

[0199] Here, use Figures 15 to 20, the motion compensation using BIO prediction will be described. The motion compensation unit 3091 has a mode for performing motion compensation using Bi-directional optical flow (bi-prediction gradient change: BIO) prediction. The Bi-directional optical flow (bi-prediction gradient change: BIO) prediction applies a gradient correction term to bi-prediction (BiPred) for motion correction.

[0200] That is, the motion compensation unit 3091 uses the following prediction formula

[0201] Pred = {(P0 + P1) + shiftOffset} >> shiftPred (Equation A1),

[0202] to derive the pixel value Pred of the predicted image (the object image to be predicted). Note that P0 is the pixel value of the motion compensation image P0, and P1 is the pixel value of the motion compensation image P1. shiftOffset is a constant for performing round control during shifting, and shiftPred is the shift value.

[0203] Moreover, the motion compensation unit 3091 uses the following prediction formula

[0204] Pred = {(P0 + P1) + modBIO + shiftOffset} >> shiftPred (Equation A2) to derive the pixel value Pred of the predicted image. Note that modBIO is obtained by

[0205] modBIO = {(lx0 - lx1) * u + (ly0 - ly1) * v >> 1} << shiftPred (Equation A3)

[0206] to derive the gradient correction term. In Equation A3, lx0 (the first gradient image), ly0 (the second gradient image), lx1 (the third gradient image), and ly1 (the fourth gradient image) are gradient images respectively. The gradient images lx0 and lx1 represent the gradients along the horizontal direction (x-direction, the first direction), and the gradient images ly0 and ly1 represent the gradients along the vertical direction (y-direction, the second direction). In addition, u and v are weight coefficients multiplied by the pixel values of lx0 and lx1, ly0 and ly1 respectively.

[0207] (Summary of the BIO prediction process)

[0208] First, use Fig.15, the process of the motion compensation unit 3091 generating a predicted image (inter-frame predicted image) with reference to the motion compensation image P0 (first reference image), the motion compensation image P1 (second reference image), and the gradient correction item is explained. Fig.15 This is a flowchart showing the flow of processing in motion compensation using BIO prediction.

[0209] like Fig.15 As shown, in motion compensation using BIO prediction, the motion compensation unit 3091 performs three-stage processing of STEP 1 to STEP 3 described below to derive a predicted image.

[0210] STEP 1: Derive interpolated / predicted pixel values

[0211] The motion compensation images P0 and P1, and the gradient images lx0, lx1, ly0, and ly1 are derived. The motion compensation unit 3091 generates the motion compensation image P0 based on the L0 reference image refImgL0 (first reference image), and generates the motion compensation image P1 based on the L1 reference image refImgL1 (second reference image). Moreover, the motion compensation unit 3091 derives the L0 horizontal gradient image lx0 and the L0 vertical gradient image ly0 based on the L0 reference image refImgL0 (first reference image), and derives the L1 vertical gradient image lx1 and the L1 vertical gradient image ly1 based on the L1 reference image refImgL1. It should be noted that refImgL0, refImgL1, P0, P1, lx0, lx1, ly0, ly1 have the cases of representing the reference image refImgL01[x][y], refImgL1[x][y], motion compensated images P0[x][y], P1[x][y], gradient images lx0[x][y], lx1[x][y], ly0[x][y], ly1[x][y] (x, y are integers in a prescribed range) as two-dimensional information and the cases of representing the reference image values ​​refImgL0, refImgL1, motion compensated image values ​​P0, P1, gradient values ​​lx0, lx1, ly0, ly1.

[0212] STEP 2: Derive gradient correction weights

[0213] Next, the motion compensation unit 3091 derives a correction weight vector (Vx, Vy) (hereinafter, sometimes also referred to as (Vx, Vy) indicating Vx=u, Vy=v). The motion compensation unit 3091 can use Fig.17 The correction weight vector (Vx, Vy) is derived from Equation 5, or it can be derived using the least squares method as described below.

[0214] In use Fig.17When the correction weight vector (Vx, Vy) is derived by using equation 5, the motion compensation unit 3091 uses the gradient changes dI / dx (i.e., lx), dI / dy (i.e., ly) at multiple points, and the time change dI / dt to derive the vector that satisfies Fig.17 The square of the left side of equation 5 can be Vx and Vy which meet the minimum conditions.

[0215] like Fig.18 As shown, the pixel value I of the point I on the object image Cur, the pixel value I0 of the point l0 (lx0, ly0) on the L0 reference image refImgL0, and the pixel value I1 of the point l1 (lx1, ly1) on the L1 reference image refImgL1 correspond to each other. It should be noted that Fig.18 Only the x component of point l0 and the x component of point l1 are shown, but the y component of point l0 and the y component of point l1 are the same. When the assumption of gradient change in the object area (the pixel value does not change but only its position changes) holds, I=I0=I1 holds. When focusing on the gradient as the amount of change in the pixel value in space, the pixel value I0 of a certain point on the L0 reference image refImgL0 can be predicted according to I0=P0+gradient×distance using the pixel value P0 of another point and the gradient at P0 and the distance between I0 and P0. Here, the gradient is (lx0, ly0) and the distance between I0 and P0 is (Vx, Vy). Similarly, the pixel value I1 of a certain point on the L0 reference image refImgL0 can also be derived according to I1=P0+gradient (lx1, ly1)*distance (-Vx, -Vy) using the pixel value P1 of another point and the gradient at P1 and the distance between I1 and P1. Furthermore, for the pixel value of the target point I of the object image Cur, its predicted value can be derived with high accuracy from the average value or weighted average value of the derived pixel value I0 and pixel value I1. Based on the assumption that the pixel value I0 is equal to the pixel value I1, the distance (Vx, Vy) = correction weight vector (u, v) can be derived by deriving parameters (Vx, Vy) that minimize the difference between I0 and I1.

[0216] To be more specific, the pixel value I0 of the point l0 (lx0, ly0) on the L0 reference image refImgL0 can be obtained by using the pixel value P0 of a certain motion compensation image P0 on the L0 reference image refImgL0, the pixel values ​​lx0 and ly0 of the gradient image, and the correction weight vector (u, v) ( Fig.18 Vx corresponds to u, Vy (not shown) corresponds to v), through

[0217] l0=P0+{lx0*u+ly0*v}(Formula A4)

[0218] And obtain.

[0219] Similarly, the pixel value I1 of the point l1 (lx1, ly1) on the L1 reference image refImgL1 can be obtained by using the pixel value P1 of a motion compensation image P1 on the L1 reference image refImgL1, the pixel values ​​lx1, ly1 of the gradient image, and the correction weight vector (u, v).

[0220] l1=P1-{lx1*u+ly1*v}(Formula A5)

[0221] And obtain.

[0222] The motion compensation unit 3091 predicts the pixel value I of the point I on the object image Cur as the image to be predicted as the average of the pixel value l0 of the point l0 on the L0 reference image refImgL0 and the pixel value l1 of the point l1 on the L1 reference image refImgL1, that is,

[0223] I=(l0+l1)>>1

[0224] ={P0+P1+(lx0-lx1)*u+(ly0-ly1)*v}>>1(Formula A6)

[0225] (and Fig.17 Here, the x component u and y component v of the correction weight vector, which are unknowns, can be derived as follows based on the assumption that the pixel value does not change.

[0226] In order to derive the correction weight vector (u, v), the motion compensation unit 3091 calculates the equation assuming that the pixel value l0 of the point l0 on the L0 reference image refImgL0 and the pixel value l1 of the point l1 on the L1 reference image refImgL1 do not change.

[0227] Σ│l0-l1│2=Σ│(P0-P1)+(lx0+lx1)*u+(ly0+ly1)*v│2(Formula A7)

[0228] The obtained values ​​of u and v are minimized and derived. Here, Σ is equivalent to calculating P0, P1, lx0, ly0, ly0, ly1 at the object pixel (x, y) and its surrounding points (x+dx, y+dy) (for example, dx=-2..2, dy=-2..2), and adding the values ​​formed by them. It should be noted that in formula A7, when (P0-P1)=t1, (lx0+lx1)=t2, and (ly0+ly1)=t3, the values ​​of u and v that are minimized by formula A7 are assigned by

[0229] Σ│l0-l1│2=Σ│t1+t2*u+t3*v│2(Formula A8)

[0230] The minimum of the obtained values.

[0231] STEP3: Derive the gradient-corrected dual-prediction image.

[0232] Next, the motion compensation unit 3091 substitutes the correction weight vector (u, v) derived in STEP2 into the formula A3 to derive the gradient correction term modBIO, and uses

[0233] Pred=P0+P1+{(lx0-lx1)*u+(ly0-ly1)*v}>>1(Formula A9)

[0234] To derive the pixel value Pred of the predicted image. It should be noted that the gradient correction term modBIO can also be weakened to 1 / 2, and

[0235] Pred=(P0+P1+{(lx0-lx1)*u+(ly0-ly1)*v}>>1)>>1(Formula A10).

[0236] Next, the processing of each of the above-mentioned STEPs is described in detail. It should be noted that, here, the case where the motion compensation unit 3091 derives (generates) a motion compensated image and a gradient image is cited as an example for description, but the motion compensation unit 3091 may also derive the composition of the pixel values ​​contained in the image instead of deriving the image. That is, the motion compensation unit 3091 may also derive the pixel values ​​of the motion compensated image and the pixel values ​​of the gradient image. That is, each STEP is an independent operation performed on each pixel, so whether each STEP is performed sequentially in a certain block of image units, or each STEP is performed in a certain block of pixel units and repeated only on a certain block of images, the same result can be obtained. Specifically, the processing of each STEP can be to derive the gradient correction item (image) and derive the predicted image Pred[][] after deriving the motion compensated image P0[][], P1[][], gradient images lx0[][], ly0[][], lx1[][], ly1[][] corresponding to a certain prediction block, or to derive the gradient correction item and predictive pixel value Pred for each point of a certain prediction block after deriving the motion compensated pixel values ​​P0, P1 and gradient image values ​​lx0, ly0, lx1, ly1 corresponding to a certain point.

[0237] The following refers to the structure of the motion compensation unit 3091. Fig.21 , and explain the details of each STEP.

[0238] (Details of STEP 1)

[0239] The motion compensation unit 3091 (motion compensation derivation unit 309111) derives motion compensation images P0 and P1 (also referred to as basic motion compensation images) based on the L0 reference image refImgL0 and the L1 reference image refImgL1. Furthermore, the motion compensation unit 3091 (gradient derivation unit 309112) derives horizontal gradient images lx0 and lx1 and vertical gradient images ly0 and ly1 for the derived motion compensation images P0 and P1.

[0240] The motion compensation derivation unit 30911 applies a vertical motion compensation filter (mcFilterVer) to the reference image, and then applies a horizontal motion compensation filter (mcFilterHor), thereby deriving motion compensated images P0 and P1.

[0241] The integer position (xInt, yInt) and phase (xFrac, yFrac) of the reference image of the block intra-coordinate (x, y) of the upper left block coordinate (xPb, yPb) are

[0242] xInt=xPb+(mvLX[0]>>2)+x

[0243] xFrac=mvLX[0]&3

[0244] yInt=yPb+(mvLX[1]>>2)+y

[0245] yFrac=mvLX[1]&3 (Formula A11). Here, although the accuracy of the motion vector is assumed to be 1 / 4 pixel accuracy, the motion vector accuracy is not limited to this, and can also be 1 / 8, 1 / 16, ..., 1 / 64, etc. When the motion vector accuracy is 1 / Mpel accuracy, in (Formula A11), the following formula is used, which uses log2(M) as the shift value for deriving the integer positions xInt and yInt, and uses M-1 as the value used for the logical sum (&) for deriving the phases xFrac and yFrac.

[0246] xInt=xPb+(mvLX[0]>>(log2(M)))+x

[0247] xFrac=mvLX[0]&(M-1)

[0248] yInt=yPb+(mvLX[1]>>(log2(M)))+y

[0249] yFrac=mvLX[1]&(M-1)

[0250] When the bit depth of the filter coefficient is, for example, MC_ACCU, the bit depth of the filtered image becomes the value obtained by adding the bit depth bitDepthY of the reference image to MC_ACCU, that is, bitDepthY+MC_ACCU. In order to return the bit depth of the filtered image to the bit depth of the reference image, a right shift of MC_ACCU is required as a right shift for adjusting the dynamic range. Here, in the case of serial preprocessing of two filters, in the output of the first filtering operation, an intermediate bit depth INTERNAL_BIT_DEPTH higher than bitDepth is used, and in the output of the second filtering operation, it is more appropriate to return from INTERNAL_BIT_DEPTH to bitDepth. At this time, the shift values ​​shift1 and shift2 used to adjust the dynamic range of the first and second filtering can be set to the following values, respectively.

[0251] shift1=bitDepthY-(INTERNAL_BIT_DEPTH-MC_ACCU)

[0252] shift2=MC_ACCU(=6)(Formula A12)

[0253] Here, bitDepthY represents the bit depth of the reference image, INTERNAL_BIT_DEPTH represents the intermediate bit depth, and MC_ACCU represents the accuracy of the motion compensation filter mcFilter. It should be noted that MC_ACCU is not limited to 6, and a value of 3 to 10 can be used.

[0254] Using the vertical motion compensation filter mcFilterVer, when x=0..BLKW-1, y=0..BLKW-1, k=0..TAP-1, offset1=1<<(shift1-1), the product of the motion compensation filter coefficient mcFilter[][] and the reference image refImg[][] is used as the value obtained by adjusting the value range (dynamic range) by the shift value shift1,

[0255] temp[x][y]=(ΣmcFilterVer[yFrac][k]*refImg[xInt][yInt+k-NTAP / 2+1]+offset1)>>shift1

[0256] bitDepth(temp[][])=bitDepthY+MC_ACCU-shift1=INTERNAL_BIT_DEPTH(=14)

[0257] (Formula A13)

[0258] A temporary image temp[][] used as intermediate data is derived. Here, bitDepth(temp[][]) represents the bit depth of the temporary image temp[][]. The bit depth of the temporary image temp[][] is the value obtained by subtracting the right shift value shift1 during the filtering process from the sum of the bit depth bitDepthY of the reference image and the accuracy MC_ACCU of the motion compensation filter. This value is referred to as the intermediate bit depth INTERNAL_BITDEPTH.

[0259] On the other hand, using the horizontal motion compensation filter mcFilterHor, when x=0..BLKW-1, y=0..BLKW-1, k=0..TAP-1, offset2=1<<(shift2-1), the product of the coefficients mcFilter[][] of the motion compensation filter and the temporary image temp[][] derived from the vertical motion compensation filter mcFilterVer is used as the value obtained by adjusting the value range by the shift value shift2,

[0260] PX[x][y]=(ΣmcFilterHor[xFrac][k]*temp[x+k-NTAP / 2+1][y]+offset2)>>shift2

[0261] bitDepth(PX[][])=INTERNAL_BITDEPTH+MC_ACCU-shift2=INTERNAL_BIT_DEPTH

[0262] (Formula A14)

[0263] The motion compensation image PX[][] is derived (PX is P0 or P1). As a specific example of the 8-tap motion compensation filter mcFilterVer and mcFilterHor used in this process, Fig.19 (a) shows a motion compensation filter mcFilter[nFrac][pos] (nFrac=0..3, pos=0..7). NPHASES represents the number of phases, and NTAPS represents the number of taps. Fig.19 The motion compensation filter mcFilter[][] shown in (a) includes {0, 0, 0, 64, 0, 0, 0, 0} and nFrac = 0, {-1, 4, -10, 58, 17, -5, 1, 0} and nFrac = 1, {-1, 4, -11, 40, 40, -11, 4, -1} and yFrac = 2, and {0, 1, -5, 17, 58, -10, 4, -1} and nFrac = 3. Here, nFrac corresponds to xFrac or yFrac.

[0264] Next, the case where the gradient derivation unit 309112 derives the horizontal gradient images lx0 and lx1 is described.

[0265] The integer position (xInt, yInt) and phase (xFrac, yFrac) of the reference image of the block intra-coordinate (x, y) of the upper left block coordinate (xPb, yPb) can be obtained by the above-mentioned formula A11. Then, the motion compensation unit 3091 derives the horizontal gradient images lx0 and lx1 using the above-mentioned formula A12.

[0266] Here, using the vertical motion compensation filter mcFilterVer, when x=0..BLKW-1, y=0..BLKW-1, k=0..TAP-1, offset1=1<<(shift1-1), the temporary image temp[][] is derived by the above formula A13.

[0267] On the other hand, using the horizontal gradient compensation filter gradFilterHor, when x=0..BLKW-1, y=0..BLKW-1, k=0..TAP-1, offset2=1<<(shift2-1), the product of the coefficients gradFilter[][] of the gradient filter and the temporary image temp[][] derived from the vertical motion compensation filter mcFilterVer is used as the value obtained by adjusting the value range by the shift value shift2,

[0268] lxX[x][y]=(ΣgradFilter[xFrac][k]*temp[x+k-NTAP / 2+1][y]+offset2)>>shift2

[0269] bitDepth(lxX[][])=INTERNAL_BITDEPTH+GRAD_ACCU-shift2=INTERNAL_BIT_DEPTH (Formula A15)

[0270] The horizontal gradient image lxX[][] (lxX is lx0 or lx1) is derived. Here, bitDepth(PX[][]) represents the bit depth of the filtered image temp[][]. The bit depth of the filtered image temp[][] is obtained by subtracting the right shift value shift1 during the filtering process from the sum of the bit depth bitDepthY of the reference image and the accuracy MC_ACCU of the motion compensation filter. Here, the bit depth of the temporary filtered image is referred to as INTERNAL_BIT_DEPTH. GRAD_ACCU represents the accuracy of the gradient compensation filter gradFilter. As a specific example of the 8-tap gradient compensation filter gradFilter set in this way, in Fig.19 (b) shows the gradient compensation filter gradFilter[NPHASES][NTAPS].

[0271] Next, the case where the gradient derivation unit 309112 derives the vertical gradient images ly0 and ly1 is described.

[0272] The gradient derivation unit 309112 applies a vertical gradient compensation filter (gradFilterVer) with reference to the L0 reference image refImgL0 and the L1 reference image refImgL1, and then applies a horizontal motion compensation filter (mcFilterHor), thereby derivation of vertical gradient images ly0 and ly1.

[0273] The integer position (xInt, yInt) and phase (xFrac, yFrac) of the reference image of the block intra-coordinate (x, y) of the upper left block coordinate (xPb, yPb) can be obtained by the above-mentioned formula A11. Then, the motion compensation unit 3091 derives the vertical gradient images ly0 and ly1 using the above-mentioned formula A12.

[0274] Here, using the vertical gradient compensation filter gradFilterVer, when x=0..BLKW-1, y=0..BLKW-1, i=0..TAP-1, offset1=1<<(shift1-1), the product of the coefficient mcFilter[][] of the motion compensation filter and the reference image refImg[][] is used as the value obtained by adjusting the value range by the shift value shift1,

[0275] temp[x][y]=(ΣgradFilter[yFrac][i]*refImg[xInt][yInt+i-NTAP / 2+1]+offset1)>>shift1

[0276] bitDepth(temp[][])=bitDepthY+GRAD_ACCU-shift1=INTERNAL_BITDEPTH(=14) (Formula A16)

[0277] Derive the temporary image temp[][].

[0278] On the other hand, using the horizontal motion compensation filter fracFilterHor, when x=0..BLKW-1, y=0..BLKW-1, i=0..TAP-1, offset2=1<<(shift2-1), the product of the coefficients mcFilter[][] of the motion compensation filter and the temporary image temp[][] derived from the vertical motion compensation filter mcFilterVer is used as the value obtained by adjusting the value range by the shift value shift2,

[0279] lxX[x][y]=(ΣfracFilterHor[xFrac][i]*temp[x+i-NTAP / 2+1][y]+offset2)>>shift2

[0280] bitDepth(PX[][])=INTERNAL_BIT_DEPTH+GRAD_ACCU-shift2=INTERNAL_BIT_DEPTH (Formula A17)

[0281] Derive the vertical gradient image lyX[][] (lyX is ly0 or ly1).

[0282] (Details of STEP 2)

[0283] The gradient correction coefficient unit 30912 may also use the least square method to derive the correction weight vector (u, v) that minimizes the value obtained by the above-mentioned formula A7 or formula A8. Fig. 20 Provide explanation. Fig. 20 This is a diagram for explaining a method of deriving a correction weight vector (u, v) using the least squares method.

[0284] The gradient product derivation unit 309121 derives the intermediate parameters s1 to s6 (gradient products). Fig. 20 (a) of equation 6 (above equation A8), it can be expressed as equation 7. Next, if the intermediate parameters s1 to s6 (hereinafter sometimes referred to as "gradient product s1" etc.) are set to

[0285] s1=Σ(t2*t2)

[0286] s2=s4=Σ(t2*t3)

[0287] s3=Σ(-t1*t2)

[0288] s5=Σ(t3*t3)

[0289] s6=Σ(-t1*t3)(Formula A18)

[0290] Then, Formula 7 can be simplified as Formula 8. If Formula 8 is solved for u and v, the correction weight vector (u, v) can be derived as shown in Formula 9. Here, if det = s1s5-s2s4, det1 = s3s5-s2s6, det2 = s1s6-s3s4, it can be expressed as correction weight vector (u, v) = (det1 / det, det2 / det).

[0291] It should be noted that, since t1 = (P0-P1) = t1, t2 = (lx0+lx1) = t2, and t3 = (ly0+ly1), the intermediate parameters s1 to s6 are respectively Fig. 20 (b) is the value expressed by the formula shown.

[0292] The gradient correction coefficient derivation unit 309122 may also adopt a simple solution as described below to derive the correction weight vectors u, v using s1 to s6 to derive the correction weight vectors (u, v). Fig. 20 The determinant of equation 8 shown in (a) and s4=s2 can be obtained

[0293] s1*u+s2*v=s3

[0294] s2*u+s5*v=s6 (Formula A19).

[0295] Here, assuming that s2*v=0, the correction weight vector (u, v) is obtained from the above formula as u=s3 / s1

[0296] v=(s6-s2*u) / s5(Formula A20).

[0297] In addition, the gradient correction coefficient derivation unit 309122 can also perform integer operations, for example, using the integer values uh (i.e., uh = u << shiftBIO) obtained by shifting u to the left by shiftBIO bits and the integer value vh (i.e., vh = v << shiftBIO) obtained by shifting v to the left by shiftBIO bits to replace the correction weight vector (u, v) that actually requires decimal precision operations. When shiftBIO = 5, the precision becomes 1 / 32. It should be noted that the above uh is not a value obtained by shifting the value u with decimal precision to the left by a specified number shiftBIO to derive integer precision, but directly derives uh with a size equivalent to u << shiftBIO through integer operations. Therefore, uh and vh are obtained as

[0298] uh = (s3 << 5) / s1

[0299] vh = ((s6 << 5) - s2 * uh) / s5 (Equation A21).

[0300] It can also be derived through the equation

[0301] v = s6 / s5

[0302] u = (s3 - s2 * v) / s1 (Equation A22)

[0303] to derive.

[0304] It should be noted that when deriving the intermediate parameters s1 to s6 shown in Fig. 20 (b), the gradient correction coefficient derivation unit 309122 can also further insert a regularization term regcost (regularization cost) that has the effect of making the value of the aforementioned gradient correction term modBIO close to 0. That is, the gradient correction coefficient derivation unit 309122 can introduce the regularization term regcost and derive the intermediate parameters s1 and s5 as

[0305] s1 = s1 + regcost

[0306] s5 = s5 + regcost (Equation A23). Here, the regularization term regcost can also be derived, for example, as

[0307] regcost = (1 << 2 * INTERNAL_BIT_DEPTH) >> regshiftBIO (= 5).

[0308] The motion compensation unit 3091 can also derive the values uh and vh obtained by shifting u and v to the left by shiftBIO for integer operations, that is

[0309] uh = (s3 << shiftBIO) / s1

[0310] vh = (s6 << shiftBIO) - s2 * u / s5 (Equation A24)

[0311] Instead of the correction weight vectors (u, v), integer operations are performed.

[0312] At the end of STEP2, the gradient correction coefficient derivation unit 309122 clips uh and vh in a specified value range (-rangeBIO to rangeBIO). That is, the gradient correction coefficient derivation unit 309122 performs the clipping process represented by the equation

[0313] rangeBIO = (1 << shiftBIO) * MVTH

[0314] uh = clip3(-rangeBIO, rangeBIO, uh)

[0315] vh = clip3(-rangeBIO, rangeBIO, vh) (Equation A25)

[0316] The clipping process represented. Note that MVTH is, for example, 2 / 3 pel.

[0317] (Details of STEP3)

[0318] The gradient correction dual - prediction derivation unit 30913 substitutes the correction weight vectors (u, v) derived in STEP2 into Equation A3, derives the gradient correction term modBIO, and uses the above - mentioned Equation A9 to derive the pixel value Pred of the predicted image. Note that the gradient correction term modBIO can also be weakened to 1 / 2 and used as in the above - mentioned Equation A10.

[0319] In the integer operations performed by the gradient correction dual - prediction derivation unit 30913, the pixel value Pred of the predicted image is derived using the prediction formula shown in Equation A2 above. The gradient correction dual - prediction derivation unit 30913 uses the equation

[0320] shiftPred = INTERNAL_BIT_DEPTH - bitDepthY + 1

[0321] modBIO = {(lx0 - lx1) * uh+(ly0 - ly1) * vh} >> shiftBIO2 (Equation A26) to derive the gradient correction term modBIO. Note that

[0322] shiftBIO2=shiftBIO+bitDepth(mcImg)-bitDepth(gradImg)+1 (calculation A27).

[0323] Here, bitDepth(mcImg) is the bit depth of the motion compensation image, which corresponds to the bitDepth(PX[][]) mentioned above. bitDepth(gradImg) is the bit depth of the gradient image, which corresponds to the bitDepth(lxX[][]) and bitDepth(lxX[][]) mentioned above. Therefore, as mentioned above, in each bit depth,

[0324] bitDepth(mcImg)=bitDepth+2*MC_ACCU-(shift1+shift2)

[0325] bitDepth(gradImg)=bitDepth+2*(MC_ACCU+GRAD_ACCU)-(shift1+shift2)

[0326] (Formula A28)

[0327] Established.

[0328] When the accuracy MV_ACCU of the motion compensation filter is equal to the accuracy GRAD_ACCU of the gradient compensation filter, since bitDepth(mcImg)=bitDepth(gradImg), equation (A27) is simplified to

[0329] shiftBIO2=shiftBIO+1 (Formula A29).

[0330] (Configuration Example of Motion Compensation Unit 3091)

[0331] Here, refer to Fig. 22 and Fig.23 and use Fig.21 , the structure of the motion compensation unit 3091 that performs the above-mentioned STEP 1 and STEP 2 is explained. Fig.21 This is a block diagram showing an example of the configuration of the motion compensation unit 3091 . Fig. 22 This is a flowchart showing an example of a processing procedure of BIO prediction performed by the motion compensation unit 3091 . Fig.23 Yes means Fig. 22 A flowchart of an example of the process of S22a.

[0332] Fig.21The motion compensation unit 3091 (prediction image generation device, prediction image generation unit) shown includes a motion compensation gradient unit 30911 and a gradient correction coefficient unit 30912. The motion compensation gradient unit 30911 includes a motion compensation derivation unit 309111 and a gradient derivation unit 309112 (first switching unit), while the gradient correction coefficient unit 30912 includes a gradient product derivation unit 309121 and a gradient correction coefficient derivation unit 309122 (second switching unit, editing unit).

[0333] In STEP 1, the motion compensation derivation unit 309111 reads out the L0 reference image refImgL0[][] and the L1 reference image refImgL1[][] used as reference images from the reference picture memory 306 to generate motion compensation images P0 and P1 ( Fig. 22 S11a) is derived

[0334] Next, the gradient derivation unit 309112 derives horizontal gradient images lx0 (lx0 [][]) and lx1 (lx1 [][]), and vertical gradient images ly0 (ly0 [][]) and ly1 (ly1 [][]) from the motion compensation images P0 (P0 [][]) and P1 (P1 [][]) derived by the motion compensation derivation unit 309111 ( Fig. 22 S12a).

[0335] The filter group (gradient correction filter, gradient filter, filter coefficient) used when the gradient derivation unit 309112 derives the gradient images lx0, lx1, ly0, and ly1 may be a configuration in which a plurality of filter groups with different resolutions are switched. For example, the motion compensation unit 3091 may be a configuration including a gradient derivation unit 309112 (first switching unit) that selects a filter group from a predetermined filter group (e.g., a high resolution filter and a low resolution filter) and switches the filter group.

[0336] Next, in STEP 2, the gradient product derivation unit 309121 derives the above-mentioned intermediate parameters s1 to s6 (gradient products) ( Fig. 22 S21a).

[0337] Next, the gradient correction coefficient derivation unit 309122 uses the intermediate parameters s1(s1[][]) to s6(s6[][]) (gradient product) derived by the gradient product derivation unit 309121 to derive the elements u[][] and v[][] (gradient correction coefficients, uh[], vh[][] in integer operations) of the correction weight vector (u[][], v[][]) ( Fig. 22Here, when the picture structure is random access, the gradient correction coefficient derivation unit 309122 assumes that the distance (T0) (inter-picture distance) between the object image Cur and the L0 reference image refImgL0 is equal to the distance (T1) (inter-picture distance) between the object image Cur and the L1 reference image refImgL1.

[0338] Finally, in STEP3, the gradient-corrected bi-prediction derivation unit 30913 uses the correction weight vector (u, v) derived in STEP2 as a weight coefficient (Formula A3, see Formula A26) multiplied by the motion displacement image predSamplesLX to derive the gradient correction term modBIO[][]( Fig. 22 Then, the gradient-corrected dual-prediction derivation unit 30913 derives the pixel value Pred of the gradient-corrected dual-prediction image (prediction image, prediction image after correction) ( Fig. 22 S32a).

[0339] In addition, if you use Fig.23 To illustrate Fig. 22 An improved example of the processing of S22a is as follows. The gradient correction coefficient derivation unit 309122 adds the normalization term regcost to the intermediate parameters s1~s6 (gradient product) derived by the gradient product derivation unit 309121 (S221a). Next, the gradient correction coefficient derivation unit 309122 derives the values ​​uh and vh (gradient correction coefficients) obtained by left-shifting u and v by shiftBIO for integer operation, respectively, to replace the correction weight vector (u, v) (S222a). Then, the gradient correction coefficient derivation unit 309122 (clipping unit) performs clipping processing to constrain the derived gradient correction coefficients uh and vh (each weight coefficient) to a specified value range (S223a).

[0340] It should be noted that Fig. 22 The process is not limited to this. Fig.25 as well as Fig.26 As shown, S22a may also be a process including step S220a of deriving intermediate parameters s1 to s6 (gradient products) per block, or step S225a of applying filtering to gradient correction coefficients uh and vh (each weight coefficient). Fig.25 as well as Fig.26 An example of deriving the gradient correction coefficients uh and vh (each weight coefficient) according to the processing flow shown will be described later.

[0341] (Other Configuration Examples of the Motion Compensation Unit 3091)

[0342] It should be noted that in Fig.21 In the structure of the motion compensation unit 3091 shown, the intermediate parameters s1 (s1[][]) to s6 (s6[][]) (gradient product) derived by the gradient product derivation unit 309121 are also used to derive the elements u[][] and v[][] (gradient correction coefficients) (STEP 3) of the correction weight vector (u[][], v[][]), but are not limited to this.

[0343] For example, Fig.24 As shown, the gradient-corrected bi-prediction derivation unit 30913 included in the motion compensation unit 3091 may be further provided with an inter-frame distance weight derivation unit 309131. The inter-frame distance weight derivation unit 309131 may be configured as follows: the picture interval T0 (first picture interval) between the target image Cur and the reference image P0 and the picture interval T1 (second picture interval) between the target image Cur and the reference image P1 are obtained from the inter-frame prediction parameter decoding unit 303, and these are used to derive the gradient correction term modBIO[][] (refer to the formula A40, the formula A44, etc. described later), thereby deriving the pixel value Pred of the gradient-corrected bi-prediction image (prediction image). It should be noted that the inter-frame distance weight derivation unit 309131 is set as a part of the gradient bi-prediction derivation unit 30913, but it may also be a part of other units.

[0344] (STEP 1: Generation of gradient image)

[0345] The gradient derivation unit 309112 derives horizontal gradient images lx0 and lx1 and vertical gradient images ly0 and ly1 for the motion compensated images P0 and P1. When deriving the gradient images lx0, lx1, ly0, and ly1, the gradient compensation filter (gradFilter) used by the gradient derivation unit 309112 may also be composed of a plurality of filter groups, such as a high-resolution filter group and a low-resolution filter group. It should be noted that the filter group refers to the filter coefficient gradFilter[nFrac][pos] of the phase nFrac and the position pos (here, nFrac = 0..M-1, pos = 0..NTAPS-1). It should be noted that the set of filter coefficients (filter group) may also be referred to as filter coefficients.

[0346] High-resolution filtering refers to filtering for deriving relatively steep gradients. It is more sensitive to noise because small changes in pixel values ​​have a greater impact. Generally speaking, the filter coefficient [1-1] / 2 for deriving the difference of pixel values ​​of 1 pixel unit is equivalent to it. Low-resolution filtering refers to filtering for deriving relatively gentle gradient changes, and is relatively tolerant to noise. Generally speaking, the filter coefficient [1, 1, -1, -1] / 4 for deriving the difference of 2 pixel units is equivalent to it.

[0347] The gradient change (optical flow) is based on the assumption that the gradient of a certain point is fixed within a specified range. This assumption is relatively valid when the correction vector (u, v) at the pixel position is less than 1 pixel. That is, when the distance of the correction pixel position is less than 1 pixel, the assumption of fixed gradient is valid. However, when the correction vector increases to, for example, 2 to 3 pixels, the assumption of fixed gradient is not valid. High-resolution filtering, for example, can derive steep gradients in pixel units, so the assumption of fixed gradient is valid, and is valid within the range of the correction vector (u, v) at the pixel position is less than 1 pixel. In addition, if we look closely, even if the assumption of fixed gradient is not valid at a distance less than 1 pixel, the assumption of fixed gradient is valid when observing overall gentle changes (not from a curve with jagged changes, but from a curve after applying low-pass filtering). Low-resolution filtering is used to derive such relatively gentle changes, and it can be considered that it is more effective than high-resolution filtering for the correction vector (u, v) at the pixel position, for example, when it exceeds 1 pixel.

[0348] Here, the low-resolution filter group may be a convolution of a high-resolution filter and a low-pass filter, and the high-resolution filter group may also be a high-pass filter. It should be noted that a method of continuously applying a prescribed gradient filter after applying a Gaussian filter to the motion-compensated images P0 and P1 may be introduced to replace the low-resolution filter. It should be noted that the gradient compensation filter applied after applying a Laplace filter to the motion-compensated images P0 and P1 is ideally a high-resolution filter.

[0349] The multiple filter groups switched by the gradient derivation unit 309112 have the following relationship in terms of the difference in resolution. Between a certain filter group A and a filter group B, the fact that the segmentation energy of the filter group A (for example, high resolution filter) is higher than that of the filter group B means that, when referring to the curve graph of the filter coefficients possessed by the filter groups, the curve graph of the filter group A is steeper than that of the filter group B, and the curve graph of the filter group B is gentler than that of the filter group A. Specifically, it means that the filter coefficients of the filter group A and the filter group B satisfy any one of the following relationships.

[0350] For the absolute value of the filter coefficient │gradFilter[nFrac][pos]│, filter group A (high resolution filter group) is greater than filter group B (low resolution filter group). That is, the maximum value of the filter coefficient of filter group A (high resolution filter group) is greater than the maximum value of the filter coefficient of filter group B (low resolution filter) (relationship R1).

[0351] or,

[0352] ·For the maximum value of the absolute value of the difference between adjacent filter coefficients in gradFilter[nFrac][pos] |gradFilter[nFrac][pos]-gradFilter[nFrac][pos+1]|, filter group A (high resolution filter group) is smaller than filter group B (low resolution filter group) (relationship R2).

[0353] ·In the filter coefficient gradFilter[nFracC][pos] corresponding to the center phase nFracC (in the case of 1 / M pixel accuracy filtering, nFracC=M / 2), with reference to the filter coefficient gradFilter[n][posC] of the center position posC (posC=TAP / 2) and the filter coefficient gradFilter[n][posC+1] of posC+1 adjacent to the center position pos, the codes of the two filter coefficients are different from each other in the filter group A (high resolution filter group), and the codes of the two filter coefficients are equal to each other in the filter group B (low resolution filter group) (relationship R3).

[0354] When gradFilter[nFrac][pos] is frequency-converted, the amount of high-frequency components of filter group A is greater than the amount of high-frequency components of filter group B (relationship R4).

[0355] For example, refer to Fig. 27 , due to 1 / 4 pixel accuracy (M = 4) and 8-tap filtering (TAP = 8), the center phase nFracC = 2, and the center position posC = 4. At this time, if the filter coefficients of the example filter1 of the filter group A and the filter coefficients of the example filter2 of the filter group B in the center phase nFracC are extracted, it is as follows.

[0356] {0, -1, 4, -57, 57, -4, 1, 0} >> 4 filter group A (filter1)

[0357] {-1, 2, -50, -53, 53, 50, -2, 1} >> 5 filter group B (filter2)

[0358] The following briefly describes the case where the relationships R1 to R3 hold true in the filtering of the above example.

[0359] (Relationship R1 test) The absolute value of the filter coefficient │gradFilter[nFrac][pos]│ is 57>>4 in filter group A and 53>>5 in filter group B, so relationship R1 holds.

[0360] (Relationship R2 test) The maximum value of the filter coefficient difference is (57-(-57))>>4 in filter group A, and is (53-(-53))>>5 in filter group B, so relationship R2 holds.

[0361] (Relationship R3 Test) When the values ​​of the center position nPosC and the adjacent position nPOC+1 are observed, the codes are different from 57, -4 in filter group A and are equal to 53, 50 in filter group B. Therefore, relationship R3 holds.

[0362] It should be noted that, when the filter coefficient gradFilter[nFrac][pos] is used for the filter operation by integer operation, that is, when the filter coefficient is a value obtained by left-shifting the value of the real filter coefficient by the filter accuracy GRAD_ACCU, the size of the real-valued filter coefficient is a value obtained by right-shifting the integer-valued filter coefficient gradFilter[nFrac][pos] by the filter accuracy GRAD_ACCU. In this case, in the determination of the above-mentioned relationship (relationship R1, relationship R2), the determination is made after the filter coefficient value is divided by 1>>GRAD_ACCU to convert it into a real value.

[0363] For example, considering the accuracy of the filter coefficient, the value obtained by replacing the "absolute value of the filter coefficient |gradFilter[nFrac][pos]|" with "|gradFilter[nFrac][pos]|>>GRAD_ACCU" is used. In addition, it is preferable to use the value obtained by replacing the "absolute value of the filter coefficient difference |gradFilter[nFrac][pos]-gradFilter[nFrac][pos+1]|" with "absolute value of the filter coefficient difference |gradFilter[nFrac][pos]-gradFilter[nFrac][pos+1]|>>GRAD_ACCU". The value obtained by left-shifting the accuracy of filter group A and filter group B as a whole, GRAD_ACCU_A+GRAD_ACCU_B, can also be used for comparison.

[0364] Here, when the filter accuracy of the high-resolution filter group and the low-resolution filter group is equal, there is no need to particularly consider the filter accuracy, and the above-mentioned relationships (relationship R1, relationship R2) can be used directly.

[0365] Fig. 27 (a) is an example of a high-resolution filter group. Fig. 27(b) is an example of a low-resolution filter group. Here, a high-resolution filter group (filter1) with a filter accuracy of 4 (GRAD_ACCU=4) and a low-resolution filter group (filter2) with a filter accuracy of 5 (GRAD_ACCU=5) are exemplified. It should be noted that Fig. 27 The low-resolution filter group in (b) is derived by convolving the high-resolution filter group with a low-pass filter (

[121] filter).

[0366] Fig.28 The position pos is shown in a curve chart relative to Fig. 27 Graph showing the values ​​of the filter coefficient gardFitler[nFrac][pos] at each phase nFrac (nFrac is xFrac or yFrac) of a high-resolution filter group (filter1 in the figure) and a low-resolution filter group (filter2 in the figure). Fig.28 (a) to Fig.28 (d) corresponds to the phases nFrac = 0, nFrac = 1, nFrac = 2, and nFrac = 3, respectively. The horizontal axis of the graph corresponds to the position pos, and the eight pos of pos = 0 ... 7 are represented by values ​​of 1 ... 8. It can be seen that the absolute value of the maximum or minimum value in the low-resolution filter group tends to become smaller than that in the high-resolution filter group.

[0367] It should be noted that, here, the filter groups switched by the gradient derivation unit 309112 are referred to as "high resolution filter group" and "low resolution filter group", but the names are not particularly limited. Fig. 27 As shown, there are at least two filter groups that are different from each other (for example, Fig. 27 In the present invention, the gradient derivation unit 309112 switches filter group 1 (filter1) and filter group 2 (filter2) appropriately. In the following, the filter groups switched by the gradient derivation unit 309112 are referred to as "filter group 1" and "filter group 2" for explanation. In addition, they may be referred to as filter coefficients, such as "filter coefficient 1" and "filter coefficient 2", instead of being referred to as filter groups.

[0368] It should be noted that a configuration in which three or more filter groups, for example, a high-resolution filter group, a medium-resolution filter group, and a low-resolution filter group, are switched may be possible.

[0369] It should be noted that, in the following, when the filter group is switched based on the size of the parameter X, classification is performed by comparing X with a predetermined threshold value, but the index idx used for identification may be directly derived from the parameter X by division, shift, table reference, etc. For example, as described below, a method of derivation by division and shift operation is more appropriate.

[0370] idx=X / TH

[0371] idx=X>>shiftTH

[0372] It should be noted that TH and shiftTH are predetermined constants used to derive the index idx.

[0373] Furthermore, in order to achieve classification at unequal intervals, a table idxTransTbl[] may be used to derive indexes from temporary variables such as shift values, as described below.

[0374] idx=idxTransTbl[X / TH]

[0375] idx=idxTransTbl[X>>shiftTH]

[0376] As idxTransTbl, for example, a table such as the following can be used.

[0377] idxTransTbl=[0, 1, 2, 2, 3, 3, 4, 4, 4, 4]

[0378] Furthermore, a method of finding a filter group based on the derived index idx and referring to a table may also be used.

[0379] For example, X may include resolution (size), block size, frame rate, motion vector, picture interval, quantization step size (quantization parameter), or a combination thereof. Resolution (size) and block size may be derived from the product (width*height) or sum (width+height) of width and height.

[0380] (STEP 1: Switching filter groups when resolution is dependent)

[0381] When the resolution of the motion compensated images P0 and P1 is high, the length of the motion vector tends to become longer overall. Therefore, under this condition, it is preferable to apply a filter group 2 (for example, a low-resolution filter group) that can capture gradients in a relatively wide range.

[0382] The gradient derivation unit 309112 may also be configured to switch the gradient compensation filter gradFilter according to the resolution of the motion compensation images P0 and P1. Specifically, the gradient derivation unit 309112 may be configured to derive the gradient image (lx0, lx1, ly0, ly1) using a filter group whose resolution is lower as the resolution is larger. Fig.29 This is a flowchart showing an example of the flow of processing for switching filter groups (filter coefficients) when the resolution is dependent.

[0383] like Fig.29 As shown, for example, the gradient derivation unit 309112 may apply filter group 2 (filter coefficient 2) when the resolution (size) of the motion compensated images P0 and P1 is above a specified threshold TH (for example, above 1920×1080), and may apply filter group 1 (filter coefficient 1) when the resolution is lower than a specified threshold TH (for example, less than 1920×1080).

[0384] (STEP 1: Filter group switching when dependent on block size)

[0385] When the block size of the prediction block or the sub-block formed by dividing the prediction block containing the object pixel is large, the length of the motion vector tends to become longer overall. Therefore, under this condition, it is preferable to apply a filter group 2 (for example, a low-resolution filter group) that can capture gradients within a relatively wide range.

[0386] Therefore, the gradient derivation unit 309112 may be configured to switch the gradient compensation filter gradFilter according to the size of the block including the pixel to be the object of derivation of the gradient image, that is, the object pixel. Specifically, the gradient derivation unit 309112 may be configured to derive the gradient image (lx0, lx1, ly0, ly1) using a filter group such that the decomposition energy becomes lower as the block size becomes larger.

[0387] For example, the gradient derivation unit 309112 may apply filter group 2 when the block size of the block containing the object pixels included in the motion compensated images P0 and P1 is greater than a specified threshold (for example, greater than 16×16), and may apply filter group 1 when the block size of the block containing the object pixels is less than a specified threshold (for example, less than 16×16).

[0388] (STEP 1: Filter group switching when dependent on frame rate)

[0389] When the resolution / frame rate of the motion compensated images P0 and P1 is large, the length of the motion vector tends to become longer overall. Therefore, under such conditions, it is preferred to apply a filter group 2 (for example, a low-resolution filter group) that can capture gradients within a relatively wide range.

[0390] Therefore, the gradient derivation unit 309112 may also be configured to switch the gradient compensation filter gradFilter according to the frame rate FrameRate of the motion compensation images P0 and P1. Specifically, the gradient derivation unit 309112 may also be configured to derive the gradient image (lx0, lx1, ly0, ly1) using a filter group such that the resolution becomes lower as the frame rate decreases.

[0391] For example, the gradient derivation unit 309112 may also apply filter group 2 when the frame rate is below a specified threshold (for example, FrameRate<=TH, TH=60), and apply filter group 1 when the frame rate is greater than the specified threshold (for example, FrameRate>TH).

[0392] Alternatively, the gradient derivation unit 309112 may apply filter group 2 when the resolution / frame rate of the motion compensated images P0 and P1 is greater than a specified threshold (for example, Width / FrameRate>TH, TH=1920 / 60=32), and may apply filter group 1 when the resolution / frame rate is below the specified threshold (for example, Width / FrameRate<=TH).

[0393] (STEP 1: Filter group switching when dependent on motion vector)

[0394] When the difference between the motion vector MV of the adjacent block (and the motion vector mvLX) is large, the change in motion within the target block tends to be relatively large. Therefore, under such conditions, it is preferable to apply a filter group 2 (for example, a low-resolution filter group) that can capture gradients in a relatively wide range.

[0395] Therefore, the gradient derivation unit 309112 may be configured to switch the gradient compensation filter gradFilter according to the difference between the motion vector MV (motion vector, mvLX) of the object block and the prediction vector PMV (predicted motion vector, mvpLX) of the object block. Specifically, the gradient derivation unit 309112 may be configured to derive the gradient image (lx0, lx1, ly0, ly1) using a filter group such that the larger the difference vector DMV, the motion vector mvLX, and the prediction vector mvpLX, the lower the resolution power.

[0396] For example, the gradient derivation unit 309112 may apply filter group 2 when the difference vector DMV is greater than a predetermined threshold value, and may apply filter group 1 when the difference vector DMV is less than a predetermined threshold value.

[0397] Here, the “difference between the motion vector MV of the target block and the predicted vector PMV (predicted motion vector) of the target block” may be a differential vector DMV (mvdLX) or a differential vector DMV (mvdLX).

[0398] absDMV=│MVx-PMVx│+│MVy-PMVy│

[0399] The absolute value of the difference vector absDMV is obtained by the formula.

[0400] In addition, the prediction vector PMV may be a collocated motion vector (ie, a motion vector at the same position as that of the previous picture), or a motion vector of a left block or a motion vector of an upper block.

[0401] It should be noted that the gradient derivation unit 309112 may switch the configuration of the gradient compensation filter gradFilter based on the "motion vector mvLX of the object block" instead of the "difference with the motion vector MV of the adjacent block".

[0402] (STEP 1: Switching filter groups depending on the distance between images)

[0403] When the distance between pictures (deltaPOC in the above example) is large, the length of the motion vector tends to become longer overall. Therefore, under this condition, it is preferred to apply filter group 2 (for example, a low-resolution filter group) that can capture gradients within a relatively wide range.

[0404] Therefore, the gradient derivation unit 309112 may be configured to switch the filter group according to the inter-picture distance between the target image Cur including the target block and at least one of the motion compensation images P0 and P1 (or the L0 reference image refImgL0 and the L1 reference image refImgL1). Specifically, the gradient derivation unit 309112 may be configured to derive the gradient image (lx0, lx1, ly0, ly1) using a filter group such that the decomposition energy decreases as the inter-picture distance increases.

[0405] For example, the gradient derivation unit 309112 may apply filter group 2 when the distance between pictures is greater than a predetermined threshold, and may apply filter group 1 when the distance between pictures is less than a predetermined threshold.

[0406] Here, the gradient derivation unit 309112 can derive the distance between pictures from the picture sequence number POC (POC_curr) of the object image Cur including the object block, the picture sequence number POC (POC_l0) of the L0 reference image refImgL0, and the picture sequence number POC (POC_11) of the L1 reference image refImgL1. In this case, the distance between pictures is calculated according to the formula

[0407] deltaPOC=│POC_l0-POC_curr│+│POC_l1-POC_curr│ / / average and total deltaPOC=max(│POC_l0-POC_curr│,│POC_l1-POC_curr│) / / maximum value is derived as deltaPOC.

[0408] (STEP 1: Filter group switching when dependent on quantization step size)

[0409] When the quantization step size of the target block as the generation target of the target image Cur is large, the reliability of the change in the fine area can be considered to be low. Under such conditions, it is preferable to apply the filter group 2 (for example, a low resolution filter group) that can capture the gradient in a relatively wide range. It should be noted that, generally, the larger the quantization parameter, the larger the quantization step size becomes. Therefore, the following quantization step size can also be replaced with the quantization parameter and applied.

[0410] Therefore, the gradient derivation unit 309112 switches the filter group acting on the motion compensated image P0 according to the quantization step size of the motion compensated image P0 (or the L0 reference image refImgL0), and switches the filter group acting on the motion compensated image P1 according to the quantization step size of the motion compensated image P1 (or the L1 reference image refImgL1). Specifically, the gradient derivation unit 309112 may be configured to derive the gradient image (lx0, lx1, ly0, ly1) using a filter group such that the decomposition energy decreases as the quantization step size increases.

[0411] Furthermore, the filter group may be switched according to the quantization coefficient of the target block of the target image Cur.

[0412] That is, the gradient derivation unit 309112 may also be configured to switch the filter group according to the quantization coefficient or the quantization step size.

[0413] For example, the gradient derivation unit 309112 may apply filter group 2 when the quantization step size is greater than a predetermined threshold value, and may apply filter group 1 when the quantization step size is less than the predetermined threshold value.

[0414] (STEP 1: Switch filter groups based on flags)

[0415] The gradient derivation unit 309112 may switch the configuration of the filter group by referring to a flag that is a clearly indicated signal.

[0416] For example, the entropy coding unit 104 encodes the filter group index of the filter group to be used in the slice header SH and the picture parameter set PPS. The gradient derivation unit 309112 may switch the configuration of the filter group based on the filter group index decoded from the slice header SH and the picture parameter set PPS.

[0417] (STEP 2: Gradient correction depending on the distance between the target image and the reference image)

[0418] When the picture structure is random access, the motion compensation unit 3091 assumes that the distance (T0) between the target image Cur and the reference image P0 (first reference image) is equal to the distance (T1) between the target image Cur and the reference image P1 (second reference image). However, similarly to the case where the picture structure is LowDelayB, the motion compensation unit 3091 may also depend on Fig.18 The correction weight vector (u, v) is derived based on the distance (T0) between the object image Cur and the reference image P0, and the distance (T1) between the object image Cur and the reference image P1. However, even if only the distance (T0) between the object image Cur and the reference image P0, and the distance (T1) between the object image Cur and the reference image P1 are considered, the performance of the motion compensation processing may not be improved.

[0419] Therefore, the motion compensation unit 3091 may perform motion compensation processing using a weight coefficient that depends on the picture interval (distance between pictures).

[0420] The motion compensation unit 3091 (inter-frame distance weight derivation unit 309131) derives T0 and T1 using the following formula.

[0421] TO=POC_L0-POC_Curr

[0422] T1=POC_curr-POC_L1

[0423] It should be noted that T0 and T1 may also be derived using the following formula.

[0424] T0=POC_curr-POC_L0

[0425] T1=POC_L1-POC_Curr

[0426] Specifically, the gradient product derivation unit 309121 is used to

[0427] Σ│l0-l1│2=Σ│(P0-P1)+(TO*lx0+T1*lx1)*u+(TO*ly0+T1*ly1)*v│2(Formula A30)

[0428] The obtained values ​​of u and v are minimized and derived (corresponding to the above formula A7). In order to derive such u and v, the gradient product derivation unit 309121 satisfies

[0429] s1=Σ(T0*lx0+T1*lx1)2

[0430] s2=s4=Σ(T0*lx0+T1*lx1)(T0*ly0+T1*ly1)

[0431] s3=-Σ(P0+P1)*(T0*lx0+T1*lx1)

[0432] s5=Σ(T0*ly0+T1*ly1)2

[0433] The intermediate parameters s1(s1[][]) to s6(s6[][]) of s6=-Σ(P0-P1)*(T0*ly0+T1*ly1)(Formula A31) are derived (corresponding to the above-mentioned Formula A18). It should be noted that Formula A31 is equivalent to the case where t1=(P0-P1), t2=(T0*lx0+T1*lx1), and t3=(T0*ly0+T1*ly1) derived by taking into account the picture interval are applied to Formula A18.

[0434] Here, the prediction formula of the pixel value I of the point I on the object image Cur as the prediction target image can be expressed as follows based on the pixel value l0 of the point l0 on the L0 reference image refImgL0 and the pixel value l1 of the point l1 on the L1 reference image refImgL1:

[0435] I=(l0+l1)>>1

[0436] ={P0+P1+(T0*lx0+T1*lx1)*u+(T0*ly0+T1*ly1)*v}>>1

[0437] (Formula A32)

[0438] (Corresponding to the above formula A6).

[0439] In addition, the gradient double prediction derivation unit 30913 imports a first weight coefficient corresponding to the picture interval T0 (first picture interval) between the object image Cur and the reference image P0, and a second weight coefficient corresponding to the picture interval T1 (second picture interval) between the object image Cur and the reference image P1 to derive the gradient correction term modBIO[][]. That is, the gradient double prediction derivation unit 30913 uses the prediction formula of the above-mentioned formula A2 to derive the pixel value Pred of the predicted image. It should be noted that in formula A2,

[0440] shiftPred=INTERNAL_BIT_DEPTH-bitDepthY+1,

[0441] modBIO is implemented through

[0442] modBIO={(T0*lx0-T1*lx1)*uh+(T0*ly0-T1*ly1)*vh}>>shiftBIO2(Formula A33)

[0443] The derived gradient correction term (corresponding to the above formula A26).

[0444] That is, the gradient bi-prediction derivation unit 30913 derivates the gradient correction term modBIO by referring to the pixel values ​​of the horizontal gradient images lx0 and lx1 and the vertical gradient images ly0 and ly1, and can also be determined according to the picture interval T0 between the target image Cur and the reference image P0, and the picture interval T1 between the target image Cur and the reference image P1. Here, the picture intervals T0 and T1 can also be referred to as weight coefficients multiplied by the pixel values ​​of the horizontal gradient images lx0 and lx1 and the vertical gradient images ly0 and ly1.

[0445] (STEP 2: Gradient correction depending on the distance between the target image and the reference image)

[0446] The gradient product derivation unit 309121 may also import a first weight coefficient W0 corresponding to the picture interval T0 between the object image Cur and the reference image P0 and a second weight coefficient W1 corresponding to the picture interval T1 between the object image Cur and the reference image P1 to derive the pixel value Pred of the predicted image. That is, the pixel value of each pixel of the reference image P0 may be multiplied by the first weight coefficient W1, and the pixel value of each pixel of the reference image P1 may be multiplied by the second weight coefficient, and the pixel value Pred of the predicted image may be derived in consideration of the temporal weight shiftWeight.

[0447] The prediction formula in this case becomes

[0448] Pred={(W0*P0+W1*P1)+modBIO+shiftOffset}>>shiftWeightPred

[0449] (Equation 34)

[0450] (corresponding to the above formula A2). The gradient product derivation unit 309121 uses the prediction formula to derive the pixel value Pred of the predicted image. It should be noted that modBIO is

[0451] modBIO = {(lx0 - lx1) * uh + (ly0 - ly1) * vh} >> (shiftBIO2 + shiftWeight)

[0452] (Equation A35)

[0453] The derived gradient correction term (corresponding to the above Equation A26). Here, the pixel value shiftWeightPred of the predicted image considering the time weight is obtained by

[0454] shiftWeightPred = shiftPred + shiftWeight

[0455] obtained, and the time weight shiftWeight is, for example, 8.

[0456] The gradient product derivation unit 309121 can also derive the first weight coefficient W0 and the second weight coefficient W1 as

[0457] W0 = (T1 << shiftWeight) / │T0 + T1│

[0458] W1 = (1 << shiftWeight) - W0 (Equation A36)

[0459] or

[0460] W1 = (T0 << shiftWeight) / │T0 + T1│

[0461] W0 = (1 << shiftWeight) - W1 (Equation A37).

[0462] Alternatively, the gradient product derivation unit 309121 can also use td = POC(refImgL1) - POC(refImgL0) based on the information POC representing the output order of the pictures

[0463] tb = POC(currPic) - POC(refImgL0) (Equation A38),

[0464] to derive the first weight coefficient W0 and the second weight coefficient W1 as

[0465] W0 = (1 << shiftWeight) - W1

[0466] W1 = (tb << shiftWeight) / td (Equation A39)

[0467] (corresponding to the above Equation 37).

[0468] Moreover, in addition to importing the first weight coefficient W0 corresponding to the picture interval T0 between the object image Cur and the reference image P0 and the second weight coefficient W1 corresponding to the picture interval T1 between the object image Cur and the reference image P1, the gradient product derivation unit 309121 can also use the weight coefficient that depends on the picture interval to perform motion compensation processing.

[0469] The prediction formula in this case is the same as the above-mentioned formula 34, and the inter-frame distance weight derivation unit 309131 uses this prediction formula to derive the pixel value Pred of the predicted image. It should be noted that modBIO is

[0470] modBIO={(T0*lx0-T1*lx1)*uh+(T0*ly0-T1*ly1)*vh>>(shiftBIO2+shiftWeight)(Formula A40)

[0471] The derived gradient correction term (corresponding to the above formula A35) Here, the pixel value shiftWeightPred of the predicted image taking into account the temporal weight is obtained by shiftPred+shiftWeight, and the temporal weight shiftWeight is 8, for example.

[0472] (STEP 2: Calibrate the distance between the target image and the reference image)

[0473] Alternatively, when the ratio of the picture interval T0 between the object image Cur and the reference image P0 and the picture interval T1 between the object image Cur and the reference image P1 are outside the prescribed range, the gradient product derivation unit 309121 may perform prescribed processing on the picture interval T0 and the picture interval T1 so that the ratio of the picture interval T0 and the picture interval T1 is within the prescribed range, and determine the weight coefficients to be multiplied by the pixel values ​​of the gradient images lx0, lx1, ly0, and ly1 based on the picture interval T0 and the picture interval T1 that have been subjected to the prescribed processing.

[0474] When the gradient product derivation unit 309121 uses a weight coefficient that depends on the picture interval to perform gradient correction processing, for example, when the picture interval T0 between the object image Cur and the reference image P0 and the picture interval T1 between the object image Cur and the reference image P1 are relatively large (for example, T0:T1=1:5, etc.), the weight coefficient is too heavy and motion compensation may not be properly performed.

[0475] Therefore, the gradient product derivation unit 309121 can perform the prescribed processing in a manner that does not deviate too much from T0:T1=1:1 even when the picture interval T0 between the target image Cur and the reference image P0 and the picture interval T1 between the target image Cur and the reference image P1 are relatively large.

[0476] The correction (prescribed processing) performed by the gradient product derivation unit 309121 is described by taking as an example the case where the predetermined range (prescribed range) for T0:T1 is 2:1 to 1:2. In this case, if T0:T1=1:5 (outside the prescribed range), the gradient product derivation unit 309121 clips the picture interval T1 to twice the picture interval T0 so that it becomes a value within the prescribed range of T0:T1 (for example, T0:T1=1:2), and determines each weight coefficient to be multiplied by each pixel value of the gradient images lx0, lx1, ly0, and ly1, rather than directly using the weight coefficient of the ratio of T0:T1.

[0477] Specifically, when the predetermined range (prescribed range) for T0:T1 is 2:1 to 1:2, when the absolute value abs(T0) of the picture interval T0 is greater than twice the absolute value abs(T1) of the picture interval T1, the gradient product derivation unit 309121 clips the picture interval T0 to 2 times. Conversely, when the absolute value abs(T1) of the picture interval T1 is greater than twice the absolute value abs(T0) of the picture interval T0, the gradient product derivation unit 309121 clips the picture interval T1 to 2 times. That is,

[0478] ·if(abs(T0)>2*abs(T1))T0=sgn(T0)*2*abs(T1)

[0479] ·if(abs(T1)>2*abs(T0))T1=sgn(T1)*2*abs(T0)

[0480] (Formula A41).

[0481] Alternatively, when the absolute value abs(T0) of the picture interval T0 is greater than the absolute value abs(T1) of the picture interval T1 by mn / md times, the gradient product derivation unit 309121 clips the picture interval T0 to mn / md times. Conversely, when the absolute value abs(T1) of the picture interval T1 is greater than the absolute value abs(T0) of the picture interval T0 by mn / md times, the gradient product derivation unit 309121 may also clip the picture interval T1 to mn / md times. That is,

[0482] ·if(md*abs(T0)>mn*abs(T1))T0=sgn(T0)*mn*abs(T1) / md

[0483] ·if(md*abs(T1)>mn*abs(T0))T1=sgn(T1)*mn*abs(T0) / md

[0484] (Formula A42) It should be noted that sgn(X) is a function for solving the code of X, and returns 1 when X is greater than 0, and returns -1 when X is less than 0.

[0485] It should be noted that when md is a power of 2,

[0486] ·if(md*abs(T0)>mn*abs(T1))T0=sgn(T0)*mn*abs(T1)>>log2(md)

[0487] ·if(md*abs(T1)>mn*abs(T0))T1=sgn(T1)*mn*abs(T0)>>log2(md)

[0488] (Formula A43) Here, mn represents the numerator of the fraction, and md represents the denominator of the fraction. mn and md are not limited thereto, and examples thereof include mn=2, md=1 (2 times), and mn=3, md=2 (1.5 times).

[0489] (STEP 2: Flag indicating whether gradient correction is performed depending on the distance between the target image and the reference image)

[0490] The gradient product derivation unit 309121 may determine whether to perform gradient correction that depends on the distance between the target image and the reference image, based on a picture interval-dependent gradient flag bio_pic_dist (predetermined flag) included in the coded data.

[0491] That is, the gradient product derivation unit 309121 determines the weight coefficients to be multiplied by the pixel values ​​of the gradient images lx0, lx1, ly0, and ly1 according to the picture interval T0 between the target image Cur and the reference image P0 and the picture interval T1 between the target image Cur and the reference image P1 when the picture interval dependent gradient flag bio_pic_dist is 1 (first value). Then, when the picture interval dependent gradient flag bio_pic_dist is 0 (second value), the weight coefficients to be multiplied by the pixel values ​​of the gradient images lx0, lx1, ly0, and ly1 may be equal to each other. For example, T0=T1=1 may be set.

[0492] Specifically, the gradient product derivation unit 309121 derives intermediate parameters s1(s1[][]) to s6(s6[][]) (corresponding to the above-mentioned formula A18) that satisfy the above-mentioned formula A31 when the picture interval dependent gradient flag bio_pic_dist is 1 (the first value). In this case, modBIO is obtained by

[0493] modBIO={(T0*lx0-T1*lx1)*uh+(T0*ly0-T1*ly1)*vh}>>shiftBIO2(Formula A44)

[0494] The derived gradient correction term (corresponding to the above formula A33).

[0495] On the other hand, when the picture interval dependent gradient flag bio_pic_dist is 0 (second value), it is derived to satisfy Fig. 20 (b) is an intermediate parameter s1 (s1[][]) to s6 (s6[][]). In this case, modBIO is a gradient correction term derived from the above-mentioned equation A26.

[0496] (STEP 2: Determine weight coefficient by block)

[0497] The gradient product derivation unit 309121 may also be configured to determine weight coefficients to be multiplied by pixel values ​​of the gradient images lx0, lx1, ly0, and ly1 on a block basis. For example, the gradient product derivation unit 309121 may also derive intermediate parameters s1 (s1[][]) to s6 (s6[][]) satisfying the above-mentioned formula A31 on a fixed (e.g., 4×4, etc.) block basis.

[0498] The gradient correction coefficient derivation unit 309122 uses the derived intermediate parameters s1 (s1[][]) to s6 (s6[][]) to derive the correction weight vector (u, v).

[0499] In this way, if the intermediate parameters s1 (s1[][]) to s6 (s6[][]) are derived block by block, the unevenness in the intermediate parameters s1 (s1[][]) to s6 (s6[][]) can be reduced and the noise can be reduced. In addition, since the processing of STEP2 can be performed by one operation per block (for example, a block containing 16 pixels), the amount of calculation required for the processing of STEP2 can also be reduced.

[0500] In addition to determining the correction weight vector (u, v) by block (e.g., 4×4 block) as described above, the gradient product derivation unit 309121 can also apply filtering (e.g., interpolation filtering) to the derived correction weight vector (u, v) to derive the correction weight vector (u, v) of a certain pixel.

[0501] For example, the correction weight vector (u, v) of a certain pixel can also be obtained by weighted averaging the correction weight vectors (u, v) of N×N block units.

[0502] u(x,y)=(w00*u(x,y)+w10*u(x+N,y)*w01*u(x,y+N)*w11*u(x+N,y+N))>>n

[0503] v(x, y)=(w00*v(x, y)+w10*v(x+N, y)*w01*v(x, y+N)*w11*v(x+N, y+N))>>n Equation A45)

[0504] The weight can be derived as follows, for example, as the weight coefficient of linear prediction.

[0505] n=log2(N)+1

[0506] w1x=x&(N-1)

[0507] w0x=N-w1

[0508] w1y=x&(N-1)

[0509] w0y=N-w1

[0510] w00=w0x*w0y, w10=w1x*w0y, w01=w0x*w1y, w11=w1x*w1y

[0511] In addition, when the correction weight vector (u, v) of the N×N block unit is compressed by 1 / N and stored, when the correction vector is set to (u0, v0), it can also be obtained by

[0512] u(x,y)=(w00*u0(x,y)+w10*u0(x+1,y)*w01*u0(x,y+1)*w11*u0(x+1,y+N))>>n

[0513] v(x, y)=(w00*v0(x, y)+w10*v0(x+1, y)*w01*v0(x, y+1)*w11*v0(x+N, y+1))>>n (Formula A46)

[0514] To deduce.

[0515] It should be noted that the above-mentioned formula A45 and formula A46 use a single process to perform two-dimensional weighted averaging, but a two-stage process can also be used. That is, a temporary image can be generated by vertical weighted averaging, and the generated temporary image can be further subjected to horizontal weighted averaging to achieve two-dimensional weighted averaging. In addition, the process can also be performed in the order of horizontal and vertical.

[0516] Furthermore, the gradient product derivation unit 309121 may also derive each correction weight vector (u, v) for each pixel by applying a predetermined filter to each correction weight vector (u, v) (weight coefficient) determined for each block (e.g., 4×4 block). As an example of the predetermined filter, a low-pass filter (e.g., a smoothing filter) may be cited.

[0517] For example, the correction weight vector (u, v) for a certain pixel is derived as

[0518] utemp(x,y)=(u(x,y-1)+2*u(x,y)+u(x,y+1))

[0519] u(x,y)=(utem(x-1,y)+2*utem(x,y)+utem(x+1,y))>>4

[0520] vtemp(x,y)=(v(x,y-1)+2*v(x,y)+v(x,y+1))

[0521] v(x,y)=(vtemp(x-1,y)+2*vtemp(x,y)+vtemp(x+1,y))>>4

[0522] (Formula A47).

[0523] In this way, if after reducing the noise of the intermediate parameters s1 (s1[][]) to s6 (s6[][]) derived by block, the prescribed filtering is applied to each correction weight vector (u, v) (weight coefficient) determined by block (for example, 4×4 block), corresponding motion compensation can be performed even in subtle movements.

[0524] (STEP 2: Adaptive gradient correction coefficient clipping)

[0525] As described above, the gradient correction coefficient derivation unit 309122 performs clipping processing on the gradient correction coefficients uh and vh (each weight coefficient) multiplied by each pixel value of the gradient images lx0, lx1, ly0, and ly1. The gradient correction coefficient derivation unit 309122 may switch the value range (range) (-rangeBIO to rangeBIO) of the clipping processing performed on the derived gradient correction coefficients uh and vh from a plurality of value range candidates when the filter group 1 described above is applied in STEP1 and when the filter group 2 described above is applied. In addition, a configuration may be adopted in which the value range is derived based on a parameter.

[0526] Optical flow (gradient change) is a technique based on the assumption that the change in a local area can be approximated using the first-order differential (i.e., gradient). Therefore, it can be said that optical flow (gradient change) is a technique that can only be applied to the case where the correction weight vector (u, v) (and the gradient correction coefficients uh, vh) are small. Therefore, in a relatively small range, it is more appropriate to clip the correction weight vector (u, v).

[0527] However, when filter group 2 (for example, low-resolution filtering) is applied in STEP 1, it is possible to detect gradients in a relatively wide range, that is, a relatively large correction weight vector (u, v). Therefore, when filter group 2 is applied, the value range of the clipping processing of the gradient correction coefficients uh and vh can be within a wider range compared to the case of applying filter group 1 which assumes detection of a relatively small correction weight vector (u, v).

[0528] The motion compensation unit 3091 includes a gradient derivation unit 309112 (first switching unit) for switching the gradient filters used when deriving the gradient images lx0, lx1, ly0, and ly1, and a gradient correction coefficient derivation unit 309122 (editing unit) can switch the range of the editing processing according to the gradient filter selected by the gradient derivation unit 309112.

[0529] That is, Fig.31 As shown in (b), in the gradient correction coefficient derivation unit 309122 which is configured to switch between a plurality of filter groups, the value range of the clipping process performed on the gradient correction coefficients uh and vh may be changed based on the selected filter group.

[0530] It should be noted that the following example shows an example of switching between two clipping processing value ranges, but it can also be a configuration in which the number of categories is set to three or more and more clipping processing value ranges, such as a large value range, a medium value range, and a small value range, are switched.

[0531] It should be noted that, in the following, when the range of the clipping process is switched based on the size of the parameter X, X is classified by comparison with a prescribed threshold value, but the index idx used for identification may be directly derived from the parameter X by division, shifting, table reference, etc. Moreover, it may be a method of finding the range of the clipping process based on the derived index idx and referring to a table. For example, resolution, block size, frame rate, motion vector, picture interval, quantization step (quantization parameter), or a combination thereof may be used in X. It should be noted that the example of the specific process has been described in the configuration of switching the filter group, and therefore, the description is omitted.

[0532] (STEP 2: Switching the range of clipping processing when it depends on resolution)

[0533] When the resolution of the motion compensation images P0 and P1 is high, the length of the motion vector tends to be longer overall, so under this condition, it is preferable to apply the clipping process set to a relatively wide range. On the other hand, when the resolution of the motion compensation images P0 and P1 is low, it is preferable to apply the clipping process set to a relatively narrow range.

[0534] Therefore, the gradient correction coefficient derivation unit 309122 may also be a configuration that derives the range of clipping processing based on the resolution of the motion compensation images P0 and P1. Specifically, the gradient correction coefficient derivation unit 309122 may also be a configuration that uses clipping with a relationship that the larger the resolution, the larger the range of clipping processing becomes. It should be noted that the larger X becomes, the larger Y becomes, which means that in X0 and X1 that are X satisfying X1>X0, Y0 that is Y in the case of X0 and Y1 that is Y in the case of X1 satisfy Y1≥Y0. Therefore, even for two Xs (X0, X1) in the relationship that X becomes larger (X0>X1), the same Y (Y0=Y1) may be used. However, at least two values ​​of Y must be used in the range of X.

[0535] For example, the gradient correction coefficient derivation unit 309122 may also set the value range of the clipping processing to -MVTH0~MVTH0 (for example, MVTH0=2) when the resolution of the motion compensated images P0 and P1 is above the specified threshold TH (for example, above 1920×1080), and set the value range of the clipping processing to -MVTH1~MVTH1 (for example, MVTH1=1 / 3) when the resolution is lower than the specified threshold TH (for example, less than 1920×1080).

[0536] (STEP 2: Switching the range of clipping processing when it depends on block size)

[0537] When the size of the block including the target pixel is large, the length of the motion vector tends to be longer overall, so under this condition, it is preferable to apply the clipping process set to a relatively wide range. On the other hand, when the size of the block including the target pixel is small, it is preferable to apply the clipping process set to a relatively narrow range.

[0538] Therefore, the gradient correction coefficient derivation unit 309122 may switch the range of the clipping process according to the size of the block including the target pixel for deriving the gradient image. Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use clipping such that the larger the block size, the larger the range of the clipping process.

[0539] For example, the gradient correction coefficient derivation unit 309122 may also set the range of the clipping processing to -2 to 2, etc. when the block size of the block containing the object pixels included in the motion compensation images P0 and P1 is greater than a specified threshold (for example, greater than 16×16), and may set the range of the clipping processing to -2 / 3 to 2 / 3, etc. when the block size of the block containing the object pixels is less than a specified threshold (for example, less than 16×16).

[0540] (STEP 2: Switching the range of clipping processing when it depends on the frame rate)

[0541] When the frame rate of the motion compensated images P0 and P1 is low and the resolution / frame rate is high, the length of the motion vector tends to become longer overall. Therefore, under such conditions, it is preferable to apply a clipping process set to a relatively wide range.

[0542] Therefore, the gradient correction coefficient derivation unit 309122 may be configured to switch the range of clipping processing according to the frame rates FrameRate of the motion compensation images P0 and P1. Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use clipping having a relationship that the smaller the frame rate FrameRate (the larger the resolution / frame rate), the larger the range of the clipping processing becomes.

[0543] For example, the gradient correction coefficient derivation unit 309122 may also set the value range of the clipping processing to -2 to 2, etc. when the frame rate is below a specified threshold (for example, FrameRate<=TH, TH=60), and set the value range of the clipping processing to -2 / 3 to 2 / 3, etc. when the frame rate is greater than the specified threshold (for example, FrameRate>TH).

[0544] Alternatively, when the resolution / frame rate of the motion compensated images P0 and P1 is greater than a specified threshold (for example, Width / FrameRate>TH, TH=1920 / 60=32), the value range of the clipping processing is set to -2 to 2; when the resolution / frame rate is below the specified threshold (for example, Width / FrameRate<=TH), the value range of the clipping processing is set to -2 / 3 to 2 / 3.

[0545] (STEP 2: Switching the range of clipping processing when depending on motion vectors)

[0546] When the difference between the motion vector MV of the adjacent block (and the motion vector mvLX) is large, the change of the motion vector in the object block tends to be relatively large. Therefore, under this condition, the range of the clipping process can also be set to be wider to capture the gradient in a relatively wide range.

[0547] Therefore, the gradient correction coefficient derivation unit 309122 may be configured to switch the range of clipping processing according to the difference between the motion vector MV (mvLX) of the target block and the predicted vector PMV (predicted motion vector, mvpLX) of the target block. Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use clipping having a relationship that the larger the difference vector DMV, the larger the range of the clipping processing.

[0548] For example, the gradient correction coefficient derivation unit 309122 may also set the value range of the clipping processing to -2 to 2, etc. when the differential vector DMV is above a specified threshold value, and may set the value range of the clipping processing to -2 / 3 to 2 / 3, etc. when the differential vector DMV is less than a specified threshold value.

[0549] Here, the “difference between the motion vector MV of the target block and the predicted vector PMV (predicted motion vector) of the target block” may be a differential vector DMV (mvdLX) or a differential vector DMV (mvdLX).

[0550] absDMV=│MVx-PMVx│+│MVy-PMVy│

[0551] The absolute value of the difference absDMV is obtained by the formula.

[0552] In addition, the prediction vector PMV may be a collocated motion vector (ie, a motion vector at the same position as that of the previous picture), or a motion vector of a left block or a motion vector of an upper block.

[0553] It should be noted that the gradient correction coefficient derivation unit 309122 may be configured to switch the range of the clipping process based on the "motion vector mvLX of the target block" instead of the "difference with the motion vector MV of the adjacent block". Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use clipping having a relationship that the larger the motion vector mvLX, the larger the range of the clipping process.

[0554] (STEP 2: Switching the range of clipping processing when it depends on the distance between pictures)

[0555] When the distance deltaPOC between pictures is large, the length of the motion vector tends to become longer overall. Therefore, under this condition, the value range of the clipping process can also be set to be wider to capture the gradient in a relatively wide range.

[0556] Therefore, the gradient correction coefficient derivation unit 309122 may be configured to switch the range of clipping processing according to the inter-picture distance between the target image Cur including the target block and at least one of the motion compensation images P0 and P1 (or the L0 reference image refImgL0 and the L1 reference image refImgL1). Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use clipping having a relationship that the larger the inter-picture distance, the larger the range of clipping processing.

[0557] For example, the gradient correction coefficient derivation unit 309122 may also set the value range of the clipping processing to -2 to 2 when the distance between pictures is greater than a specified threshold, and set the value range of the clipping processing to -2 / 3 to 2 / 3 when the distance between pictures is less than a specified threshold.

[0558] (STEP 2: Switching the range of clipping processing when depending on the quantization step size)

[0559] When the quantization step size of the target block to be generated as the target image Cur is large, the reliability of the change in the fine area can be considered to be low. In this case, the range of the clipping process can be set to be wide to capture the gradient in a relatively wide range.

[0560] Therefore, the gradient correction coefficient derivation unit 309122 may also switch the value range of the clipping processing according to the quantization step size of the motion compensated image P0 (or the L0 reference image refImgL0) and the quantization step size of the motion compensated image P1 (or the L1 reference image refImgL1).

[0561] Furthermore, the value range of the clipping process may be switched according to the quantization coefficient of the target block of the target image Cur.

[0562] That is, the gradient correction coefficient derivation unit 309122 may also be configured to switch the value range of the clipping process according to the quantization coefficient or the quantization step size. Specifically, the gradient correction coefficient derivation unit 309122 may also be configured to use clipping having a relationship that the larger the quantization step size, the larger the value range of the clipping process becomes.

[0563] For example, the gradient correction coefficient derivation unit 309122 may also set the value range of the clipping processing to -2 to 2 when the quantization step is greater than a specified threshold, and set the value range of the clipping processing to -2 / 3 to 2 / 3 when the quantization step is less than a specified threshold.

[0564] (STEP 2: Switching the range of clipping processing with reference to the flag)

[0565] The gradient correction coefficient derivation unit 309122 may be configured to switch the value range of the clipping process with reference to a flag that is a clearly indicated signal.

[0566] For example, the entropy coding unit 104 encodes the filter group index of the filter group used in the slice header SH and the picture parameter set PPS. The gradient correction coefficient derivation unit 309122 may also be configured to switch the value range of the clipping process based on the filter group index decoded from the slice header SH and the picture parameter set PPS.

[0567] (STEP 2: Switching of adaptive normalization terms)

[0568] As described above, the gradient correction coefficient derivation unit 309122 adds the normalization term regcost to the intermediate parameters s1 to s6 derived by the gradient product derivation unit 309121. The gradient correction coefficient derivation unit 309122 may switch the value of the normalization term regcost added to the intermediate parameters s1 to s6 in the following manner when the filter group 1 described above is applied and when the filter group 2 described above is applied in STEP1.

[0569] The normalization term regcost is added to the intermediate parameters s1 to s6 in order to suppress the size of the correction weight vector (u, v) (close to 0). When filter group 2 (for example, low resolution filtering) is applied in STEP1, it is possible to detect gradients in a relatively wide range, that is, relatively large correction weight vectors (u, v). Therefore, when filter group 2 is applied, the value of the normalization term regcost added to the intermediate parameters s1 to s6 can also be reduced, so that the derived correction weight vector (u, v) is in a wider range. On the contrary, when filter group 1 is applied, the value of the normalization term regcost added to the intermediate parameters s1 to s6 can also be increased, so that the derived correction weight vector (u, v) is in a small range. It should be noted that the normalization when the value of the normalization term regcost is increased is called "stronger regularization", and the normalization when the value of the normalization term regcost is reduced is called "weaker regularization".

[0570] That is, in the gradient correction coefficient derivation unit 309122 which is configured to switch between a plurality of filter groups, Fig.31 As shown in (a), the value of the normalization term regcost may be changed based on the selected filter group. Specifically, the gradient correction coefficient derivation unit 309122 uses normalization having a relationship that the lower the accuracy of the filter group, the smaller the normalization term becomes.

[0571] The motion compensation unit 3091 includes a gradient derivation unit 309112 (first switching unit) for switching the gradient filter used when deriving the gradient images lx0, lx1, ly0, and ly1, and a gradient correction coefficient derivation unit 309122 (second switching unit) can switch the value of the normalization item regcost according to the gradient filter selected by the gradient derivation unit 309112.

[0572] That is, similar to the configuration in which the gradient correction coefficient deriving unit 309122 switches the range of clipping processing, the gradient correction coefficient deriving unit 309122 may also be a configuration in which the value of the normalization term regcost is switched.

[0573] It should be noted that the following example shows an example of switching two normalization terms, but it is also possible to set the number of categories to three or more and switch more than three normalization terms, such as a large normalization term, a medium normalization term, and a small normalization term.

[0574] It should be noted that, in the following, when the normalization term is switched based on the size of the parameter X, X is classified by comparison with a predetermined threshold value, but the index idx used for identification may be directly derived from the parameter X by division, shifting, table reference, etc. In addition, it is also possible to obtain the normalization term based on the derived index idx and refer to a table. For example, resolution, block size, frame rate, motion vector, picture interval, quantization step (quantization parameter), or a combination thereof may be used in X. It should be noted that the specific processing example has been described in the configuration of switching the filter group, and therefore, the description is omitted.

[0575] (STEP 2: Switching of normalization items when resolution is dependent)

[0576] The gradient correction coefficient derivation unit 309122 may be configured to switch the normalization term according to the resolution of the motion compensation images P0 and P1. Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use normalization such that the normalization term becomes smaller as the resolution increases.

[0577] For example, Fig.31As shown in (a), the gradient correction coefficient derivation unit 309122 may also set the normalization term regcost to be smaller in such a way that a relatively large correction weight vector (u, v) can be derived when the resolution of the motion compensated images P0 and P1 is relatively high, that is, above a prescribed threshold value TH (for example, above 1920×1080), and perform weaker normalization; and may set the normalization term regcost to be larger than that in the case of high resolution, and perform stronger normalization when the resolution is relatively low, that is, below a prescribed threshold value TH (for example, less than 1920×1080).

[0578] (STEP 2: Switching of normalization terms when depending on block size)

[0579] The gradient correction coefficient derivation unit 309122 may switch the normalization term according to the size of the block including the target pixel from which the gradient image is to be derived. Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use normalization such that the larger the block size, the smaller the normalization term.

[0580] For example, the gradient correction coefficient derivation unit 309122 may also set the normalization term regcost to be smaller and perform weaker normalization in a manner that enables the derivation of a relatively large correction weight vector (u, v) when the block size of the block containing the object pixels included in the motion compensated images P0 and P1 is greater than a specified threshold (for example, greater than 16×16), and may set the normalization term regcost to be larger and perform stronger normalization when the block size of the block containing the object pixels is less than a specified threshold (for example, less than 16×16).

[0581] (STEP 2: Switching of normalization items when depending on frame rate)

[0582] The gradient correction coefficient derivation unit 309122 may also be configured to switch the normalization term according to the frame rate FrameRate of the motion compensated images P0 and P1. Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use normalization such that the smaller the frame rate, the smaller the normalization term.

[0583] For example, the gradient correction coefficient derivation unit 309122 may also set the relative normalization term regcost to be smaller and perform weaker normalization in a manner that enables the derivation of a relatively larger correction weight vector (u, v) when the frame rate is below a specified threshold (for example, FrameRate<=TH, TH=60), and may set the normalization term regcost to be larger and perform stronger normalization when the frame rate is greater than the specified threshold (for example, FrameRate>TH).

[0584] (STEP 2: Switching of normalization terms when depending on motion vectors)

[0585] The gradient correction coefficient derivation unit 309122 may be configured to switch the normalization term according to the difference between the motion vector MV (mvLX) of the target block and the prediction vector PMV (predicted motion vector, mvpLX) of the target block. Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use normalization having a relationship that the larger the difference vector, the smaller the normalization term.

[0586] For example, the gradient correction coefficient derivation unit 309122 may also set the normalization term regcost to a smaller value and perform weaker normalization in a manner that enables the derivation of a relatively large correction weight vector (u, v) when the difference vector DMV is above a prescribed threshold value, and may set the normalization term regcost to a larger value and perform stronger normalization when the difference vector DMV is less than a prescribed threshold value.

[0587] It should be noted that the gradient correction coefficient derivation unit 309122 may be configured to switch the normalization term according to the motion vector mvLX of the target block.

[0588] (STEP 2: Switching the normalization term when depending on the distance between images)

[0589] The gradient correction coefficient derivation unit 309122 may be configured to switch the normalization term according to the inter-picture distance between the target image Cur including the target block and at least one of the motion compensation images P0 and P1 (or the L0 reference image refImgL0 and the L1 reference image refImgL1). Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use normalization having a relationship such that the normalization term becomes smaller as the picture interval becomes larger.

[0590] For example, the gradient correction coefficient derivation unit 309122 may also set the normalization term regcost to be smaller and perform weaker regularization in a manner that enables the derivation of a relatively large correction weight vector (u, v) when the distance between pictures is greater than a specified threshold, and may set the normalization term regcost to be larger and perform stronger regularization when the distance between pictures is less than a specified threshold.

[0591] (STEP 2: Switching of normalization terms when depending on quantization step size)

[0592] The gradient correction coefficient derivation unit 309122 may switch the normalization term according to the quantization step size of the motion compensation image P0 (or the L0 reference image refImgL0) and the quantization step size of the motion compensation image P1 (or the L1 reference image refImgL1). Specifically, the gradient correction coefficient derivation unit 309122 may be configured to use normalization having a relationship such that the larger the quantization step size, the smaller the normalization term.

[0593] For example, the gradient correction coefficient derivation unit 309122 may set the normalization term regcost to be smaller in such a way that a relatively large correction weight vector (u, v) can be derived when the quantization step is greater than a prescribed threshold, and perform weaker regularization; and may set the normalization term regcost to be larger in such a way that a relatively large correction weight vector (u, v) can be derived when the quantization step is greater than a prescribed threshold, and perform stronger regularization.

[0594] (STEP 2: Switching of normalization items referring to flags)

[0595] The gradient correction coefficient derivation unit 309122 may be configured to switch the normalization term by referring to a flag that is a signal that explicitly indicates the normalization term.

[0596] For example, the gradient correction coefficient derivation unit 309122 may be configured to switch the normalization item based on the filter group index decoded from the slice header SH and the picture parameter set PPS.

[0597] (Motion vector derivation using matching mode)

[0598] The inter-frame prediction parameter decoding control unit 3031 of the inter-frame prediction parameter decoding unit 303 (prediction vector calculation unit) may also be configured to select any one of the merge mode, AMVP mode, or matching mode (FRUC mode, matching prediction mode) as the prediction mode used when deriving the motion vector mvLX (see Figure 6 ). Alternatively, in addition to the inter-frame prediction parameter decoding control unit 3031, a matching prediction parameter derivation unit (not shown) for selecting a mode used when the inter-frame prediction parameter decoding control unit 3031 derives the motion vector mvLX may be provided.

[0599] Reference Fig.30 , the motion vector derivation process performed by the inter-frame prediction parameter decoding unit 303 is explained. Fig.30 It is a flowchart showing the flow of motion vector derivation processing.

[0600] In the following description, unless otherwise specified, each process is performed by the inter-frame prediction parameter decoding unit 303 .

[0601] First, in step S101, the merge flag merge_flag is decoded, and in step S102, whether merge_flag!=0? is determined.

[0602] When merge_flag!=0 is true (Y in S102), the flag ptn_match_flag indicating whether the matching mode (matching prediction mode) is used is decoded in step S103, and ptn_match_flag!=0 is determined in step S104.

[0603] When ptn_match_flag!=0 is true (Y in S104), in S105, ptn_match_mode, which is a syntax indicating the type of matching mode, is decoded. Next, in step S106, the matching mode specified by ptn_match_mode is executed. When ptn_match_mode==1, bi-prediction matching is selected, and when ptn_match_mode==2, template matching is selected. In the matching mode, for example, it is assumed that the object moves at a constant speed, and the motion vector is derived by matching blocks in a plurality of reference images. For example, it is assumed that the object is an object obtained by the block BLK0 of the L0 reference image and the block BLK1 of the L1 reference image passing through the target block BLKCUR. In this case, based on the assumption that the trajectory connecting the block BLK0 of the L0 reference image and the block BLK1 of the L1 reference image just passes through the target block BLKCUR, the block BLK0 of the L0 reference image and the block BLK1 of the L1 reference image are compared (matched), and a block with a smaller difference between the two is searched and selected. In this case, the motion vector of the L0 reference image for the object block is derived from the positional relationship between the block BLK0 of the L0 reference image and the object block, and the motion vector of the L1 reference image for the object block is derived from the positional relationship between the block BLK1 of the L1 reference image and the object block. The matching mode described above is called "bi-prediction matching". In addition to "bi-prediction matching", "template matching" may be used as a configuration of the matching mode. Template matching assumes that the motion of the object does not change at the block boundary, and takes a difference (matching) between the adjacent area of ​​the object block on the object image (for example, the inverted L-shaped template area formed by the upper row and left column of the object block) and the template area (= inverted L-shaped area) on the reference image, and searches for the motion vector indicated by the block with the minimum difference. The motion vector derived from the assumption that the motion vector in the adjacent area (template area) of the object block obtained by the search is consistent with the motion vector of the object block is used as the motion vector of the object block.

[0604] On the other hand, in step S102, when merge_flag!=0 is false (N in S102), the AMVP mode is selected. More specifically, the inter-frame prediction parameter decoding unit 303 decodes the difference vector mvdLX in S110 and decodes the prediction vector index mvp_LX_idx in S111. Furthermore, the PMV candidate list mvpListLX is derived in S112. Next, in S113,

[0605] mvLX=mvpListLX[mvp_LX_idx]+mvdLX

[0606] To derive the motion vector mvLX.

[0607] In addition, in step S104, when ptn_match_flag!=0 is false, the inter-frame prediction parameter decoding unit 303 decodes the merge index merge_idx in S107. Next, the inter-frame prediction parameter decoding unit 303 derives the merge candidate list mergeCandList in S108, and in S109,

[0608] mvLX = mergeCand[merge_idx]

[0609] To derive the motion vector mvLX.

[0610] (STEP 2: Switching of normalization items referring to flags)

[0611] In matching prediction, if a small unit (e.g., 1 / 4 pixel, 1 / 8 pixel, 1 / 16 pixel) is used as the search accuracy of the motion vector, the accuracy of the predicted image becomes higher, but the amount of processing will increase. If the search accuracy of the motion vector is set to low accuracy (e.g., 1 pixel, 1 / 2 pixel), the amount of processing can be suppressed, but the accuracy of the predicted image will decrease. In addition, it can be considered that in a configuration in which matching prediction and BIO prediction are used together, that is, a configuration in which BIO prediction is performed after matching prediction, the motion compensation image derived from the motion vector obtained by matching can be corrected by BIO prediction. It can be considered that in the case of such a configuration, in matching prediction, even if a motion vector of relatively low accuracy is set, it can be corrected by a high-precision correction motion vector (= correction weight vector (u, v)) derived by the pixel unit according to the gradient derived by the pixel unit by BIO prediction.

[0612] In the image decoding device 31 that derives the motion vector mvLX that adopts (selects) the matching mode, when the motion compensation by BIO prediction is introduced, the motion compensation unit 3091 may set the search accuracy for deriving the motion vector MV (motion vector mvLX) in the matching mode to a lower accuracy (e.g., 1 pixel, 1 / 2 pixel) when the BIO prediction is applied. That is, the accuracy of the motion vector used for deriving the motion compensated image may be switched depending on whether the BIO prediction is applied.

[0613] Fig.32 This is a graph showing the comparison of the prediction accuracy of the BIO prediction model and the matching model. Fig.32 As shown, the range of motion vector MV to which BIO prediction can be applied, i.e., the correction weight vector (u, v), is as small as, for example, 2 / 3 pel at most, but the unit (block size) of spatial correction can be set to 1×1 pixel (i.e., each pixel unit). In addition, in BIO prediction, the accuracy of correction of motion vector MV (accuracy of motion compensation vector (u, v) etc.) is high, for example, it can reach 1 / 64 if shiftBIO=6, and it can reach 1 / 256 if shiftBIO=8.

[0614] On the other hand, in the matching mode, the range of the motion vector MV that can be adapted is as large as, for example, 4 pels at most, but the determination of whether the matching is appropriate requires a relatively large block, so the unit of spatial correction is about 8×8 pixels (i.e., block unit). In addition, in BIO prediction, it is difficult to improve the accuracy of the correction of the motion vector MV due to the processing amount, for example, it is at most about 1 / 8.

[0615] Therefore, when the matching mode and BIO prediction are used together, the accuracy of the vector obtained in the matching mode can also be lower. That is, the search accuracy of the motion vector MV in the matching mode can also be lowered than the accuracy of the motion vector derived in the AMVP mode (for example, 1 / 4 pel). That is, when the motion vector derived for the motion compensation image is derived by the AMVP mode, for example, when it is derived in 1 / 4 pixel units, it can also be derived in 1 / 2 pixel units in the matching prediction used in conjunction with the BIO prediction (BIO prediction is performed in the later stage). By lowering the search accuracy of the motion vector MV in the matching mode than the accuracy of the motion vector derived in the AMVP mode (for example, 1 / 4 pel), the effect of making the search of the motion vector in the matching mode easier is achieved. It should be noted that the part that is not corrected in the matching mode is corrected in the motion compensation performed after the application of the BIO prediction, so it does not become a problem.

[0616] Furthermore, if the motion vector MV is roughly corrected (matched) in advance (every 1 pel or 1 / 2 pel) in the matching mode, even if motion compensation of BIO prediction is applied which can only correct minute motions, sufficient correction can be performed in the matching mode and BIO as a whole.

[0617] It should be noted that BIO prediction is performed based on the assumption that the gradient is fixed in a relatively small range, that is, a range in which the correction weight vector (u, v) is small. Therefore, it is preferred not to perform the operation when the assumption that the correction weight vector (u, v) becomes larger does not hold. However, if the motion vector is derived (corrected) in advance with a certain degree of search accuracy (e.g., 1 pixel, 1 / 2 pixel) by matching in a configuration in which BIO prediction is performed after matching prediction, it can be expected that the correction weight vector (u, v) will reach below the search accuracy.

[0618] (Action switching corresponding to the matching mode of the BIO flag)

[0619] In the image decoding device 31 that derives the motion vector mvLX using the matching mode, when introducing motion compensation performed by BIO prediction, the motion compensation unit 3091 can also switch the search accuracy used to derive the motion vector MV (motion vector mvLX) in the matching mode according to whether the BIO prediction is applied.

[0620] When the BIO flag decoded by the inter-frame prediction parameter decoding unit 303 indicates that the motion compensation unit 3091 performs motion compensation using BIO prediction (hereinafter, marked as "BIO on"), the inter-frame prediction parameter decoding unit 303 sets the search accuracy of the motion vector search in the matching mode to Npel (for example, 1pel, 1 / 2pel). On the other hand, when the BIO flag indicates that motion compensation using BIO prediction is not performed (hereinafter, marked as "BIO off"), the inter-frame prediction parameter decoding unit 303 may also set the search accuracy of the motion vector search in the matching mode to Mpel (for example, 1 / 4 or 1 / 8pel). Here, N>M holds.

[0621] Alternatively, in an image decoding device 31 that derives a motion vector mvLX using a matching mode, when motion compensation performed by BIO prediction is introduced, the motion compensation unit 3091 may switch the size of a block used to derive a motion vector MV (motion vector mvLX) in the matching mode, depending on whether BIO prediction is applied.

[0622] For example, when BIO is turned on, the inter-frame prediction parameter decoding unit 303 may also be configured so that the search accuracy of the motion vector search in the matching mode is performed according to a relatively large block (prediction unit PU) without searching each sub-block (e.g., 4×4) of a smaller block constituting the prediction block. On the other hand, when BIO is turned off, the search for the motion vector in the matching mode may also be performed according to the sub-block of a relatively small block.

[0623] It should be noted that the BIO flag can be a flag for sequence coding, or a flag for sequence parameter set SPS, picture parameter set PPS, or block coding.

[0624] In the image decoding device 31 that derives the motion vector mvLX using the matching mode, when introducing motion compensation by BIO prediction, it is considered that a plurality of modes are provided as the configuration of the above-mentioned matching mode, for example, a configuration having a dual prediction matching mode and a template matching mode. Both the dual prediction matching mode and the BIO prediction mode are based on the assumption that an object moves at a constant speed across a plurality of reference pictures (for BIO prediction, it is assumed that not only the gradient change but also the time change is fixed within a certain range), so it can be considered that BIO prediction is also effective when the dual prediction matching mode is used as the matching mode.

[0625] Therefore, the motion compensation unit 3091 may also perform motion compensation using BIO prediction based on whether the prediction mode decoded by the inter-frame prediction parameter decoding unit 303 is a bi-prediction match. In other words, the motion compensation unit 3091 may also perform motion compensation using BIO prediction based on whether the prediction mode selected for deriving the motion vector of the inter-frame prediction parameter decoding unit 303 is a bi-prediction match mode.

[0626] That is, the following configuration may be adopted: when the matching mode is template matching, the motion compensation unit 3091 does not perform motion compensation with BIO prediction applied (the inter-frame prediction parameter decoding unit 303 does not decode the BIO flag from the encoded data, but sets the BIO flag to a value indicating that BIO is off); on the other hand, when the matching mode is dual prediction matching, the motion compensation unit 3091 performs motion compensation with BIO prediction applied (the inter-frame prediction parameter decoding unit 303 decodes the BIO flag from the encoded data).

[0627] In addition, the following configuration may be adopted: when the matching mode is template matching, the motion compensation unit 3091 does not perform motion compensation with BIO prediction applied (the inter-frame prediction parameter decoding unit 303 sets a value indicating that BIO is off for the BIO flag); on the other hand, when the matching mode is dual-prediction matching, the motion compensation unit 3091 performs motion compensation with BIO prediction applied (the inter-frame prediction parameter decoding unit 303 derives a value indicating that BIO is on as the BIO flag).

[0628] Since the motion is sometimes discontinuous between the target image Cur and the L0 reference image refImgL0 and the target image Cur and the L1 reference image refImgL1 in the case of template matching, the performance of the motion compensation processing may not necessarily be improved even if the motion compensation unit 3091 performs motion compensation using the BIO prediction. Therefore, the motion compensation unit 3091 may be configured not to perform motion compensation using the BIO prediction in the case of template matching. With such a configuration, the amount of processing required for the motion compensation unit 3091 to perform motion compensation using the BIO prediction can be reduced.

[0629] Furthermore, in the image decoding device 31 that derives the motion vector mvLX using the matching mode, when the motion compensation by BIO prediction is introduced, in a configuration having a plurality of modes as the above-mentioned matching mode, for example, a configuration having a dual prediction matching mode and a template matching mode, the configuration may be such that the accuracy of the matching prediction is changed according to the matching mode. That is, in the case where the matching mode is the dual prediction mode, even if the search accuracy of the motion vector is not high, the motion compensation image is corrected by the BIO prediction after the matching mode, thereby generating a prediction image with a higher accuracy. Therefore, in the case where the matching mode is the dual prediction mode, the search unit of the motion vector in the matching mode may be relatively increased (for example, 1 pixel, 1 / 2 pixel), and in the case where the matching mode is the template matching mode, the search unit of the motion vector may be made smaller than that in the case of the dual prediction matching (for example, 1 / 4 pixel, 1 / 8 pixel).

[0630] Furthermore, in the image decoding device 31 that derives the motion vector mvLX using the matching mode, when the motion compensation by BIO prediction is introduced, the structure of the spatial unit (block size) of the matching prediction search may be changed according to the matching mode. Specifically, when the matching mode is the dual prediction mode, even if the motion vector search accuracy is not high, the motion compensation image is corrected by BIO prediction after the matching mode, thereby generating a prediction image with higher accuracy. Therefore, even if the spatial unit of the motion vector is relatively large, it is corrected by BIO prediction in pixel units. Thus, it may be set that: when the matching mode is the dual prediction mode, the search unit (block size) of the motion vector in the matching mode is relatively large (for example, 8×8 pixels, 16×16 pixels), and when the matching mode is the template matching, the spatial unit of the motion vector is small (for example, the length of one side of the inverted L-shaped template is at least 4 to 5).

[0631] (Configuration of Image Coding Device)

[0632] Next, the configuration of the image encoding device 11 according to this embodiment will be described. Fig.12 1 is a block diagram showing the structure of the image coding device 11 of the present embodiment. The image coding device 11 includes a prediction image generation unit 101, a subtraction unit 102, a DCT / quantization unit 103, an entropy coding unit 104, an inverse quantization / inverse DCT unit 105, an addition unit 106, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference picture storage unit, frame memory) 109, a coding parameter determination unit 110, and a prediction parameter coding unit 111. The prediction parameter coding unit 111 includes an inter-frame prediction parameter coding unit 112 and an intra-frame prediction parameter coding unit 113.

[0633] The prediction image generation unit 101 generates a prediction picture block P per block (region formed by dividing the picture) for each picture of each viewpoint of the layer image T input from the outside. Here, the prediction image generation unit 101 reads a reference picture block from the reference picture memory 109 based on the prediction parameter input from the prediction parameter encoding unit 111. The prediction parameter input from the prediction parameter encoding unit 111 is, for example, a motion vector or a displacement vector. The prediction image generation unit 101 reads a reference picture block of a block located at a position indicated by a motion vector or a displacement vector predicted with the encoding target block as a starting point. The prediction image generation unit 101 generates a prediction picture block P using one of a plurality of prediction methods for the read reference picture block. The prediction image generation unit 101 outputs the generated prediction picture block P to the subtraction unit 102. It should be noted that the prediction image generation unit 101 performs the same operation as the prediction image generation unit 308 already described, and therefore the details of the generation of the prediction picture block P are omitted.

[0634] The prediction image generator 101 selects a prediction method, for example, to minimize an error value based on a difference between a signal value of each pixel included in the image block and a signal value of each pixel corresponding to the prediction picture block P. The method of selecting a prediction method is not limited thereto.

[0635] The multiple prediction methods are intra prediction, motion prediction and merge prediction. Motion prediction is the prediction between display times in the above inter prediction. Merge prediction is a prediction using a reference picture block and prediction parameters that have already been coded and are the same as a block located within a predetermined range from the coding target block.

[0636] When intra prediction is selected, the predicted image generation unit 101 outputs a prediction mode IntrapredMode indicating an intra prediction mode used when generating the predicted picture block P to the prediction parameter encoding unit 111 .

[0637] When motion prediction is selected, the prediction image generation unit 101 stores the motion vector mvLX used when generating the prediction picture block P in the prediction parameter memory 108, and outputs it to the inter-frame prediction parameter encoding unit 112. The motion vector mvLX represents a vector from the position of the encoding target block to the position of the reference picture block when generating the prediction picture block P. The information representing the motion vector mvLX includes information representing the reference picture (for example, the reference picture index refIdxLX, the picture sequence number POC), and may also be information representing the prediction parameters. In addition, the prediction image generation unit 101 outputs the prediction mode predMode representing the inter-frame prediction mode to the prediction parameter encoding unit 111.

[0638] When merge prediction is selected, the predicted image generation unit 101 outputs a merge index merge_idx indicating the selected reference picture block to the inter prediction parameter coding unit 112. Furthermore, the predicted image generation unit 101 outputs a prediction mode predMode indicating a merge prediction mode to the prediction parameter coding unit 111.

[0639] The subtraction unit 102 generates a residual signal by subtracting the signal value of the prediction picture block P input from the prediction image generation unit 101 pixel by pixel from the signal value of the block corresponding to the layer image T input from the outside. The subtraction unit 102 outputs the generated residual signal to the DCT / quantization unit 103 and the encoding parameter determination unit 110.

[0640] The DCT / quantization unit 103 performs DCT on the residual signal input from the subtraction unit 102 to calculate a DCT coefficient. The DCT / quantization unit 103 quantizes the calculated DCT coefficient to obtain a quantized coefficient. The DCT / quantization unit 103 outputs the obtained quantized coefficient to the entropy coding unit 104 and the inverse quantization / inverse DCT unit 105.

[0641] The entropy coding unit 104 receives quantization coefficients from the DCT / quantization unit 103 and coding parameters from the coding parameter determination unit 110. The input coding parameters include codes such as reference picture index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX, prediction mode predMode, and merge index merge_idx.

[0642] The entropy coding unit 104 performs entropy coding on the input quantization coefficients and coding parameters to generate a coded stream Te, and outputs the generated coded stream Te to the outside.

[0643] The inverse quantization / inverse DCT unit 105 inversely quantizes the quantization coefficients input from the DCT / quantization unit 103 to obtain DCT coefficients. The inverse quantization / inverse DCT unit 105 performs inverse DCT on the obtained DCT coefficients to calculate a decoded residual signal. The inverse quantization / inverse DCT unit 105 outputs the calculated decoded residual signal to the addition unit 106.

[0644] The adder 106 generates a reference picture block by adding the signal value of the prediction picture block P input from the prediction image generator 101 and the signal value of the decoded residual signal input from the inverse quantization / inverse DCT unit 105 for each pixel. The adder 106 stores the generated reference picture block in the reference picture memory 109 .

[0645] The prediction parameter memory 108 stores the prediction parameter generated by the prediction parameter encoding unit 111 at a predetermined position for each picture and block to be encoded.

[0646] The reference picture memory 109 stores the reference picture blocks generated by the adding unit 106 at predetermined positions for each picture and block to be encoded.

[0647] The coding parameter determination unit 110 selects one of a plurality of sets of coding parameters. The coding parameters are the above-mentioned prediction parameters and parameters to be coded that are generated in association with the prediction parameters. The predicted image generation unit 101 generates a predicted picture block P using each of these sets of coding parameters.

[0648] The coding parameter determination unit 110 calculates the size of the amount of information and the cost value indicating the coding error for each of the multiple sets. The cost value is, for example, the sum of the amount of code and the value obtained by multiplying the square error by the coefficient λ. The amount of code is the amount of information of the coded stream Te obtained by entropy encoding the quantization error and the coding parameter. The square error is the sum of the square values ​​of the residual value of the residual signal calculated in the subtraction unit 102 between pixels. The coefficient λ is a real number greater than a preset zero. The coding parameter determination unit 110 selects a set of coding parameters whose calculated cost value becomes the smallest. As a result, the entropy coding unit 104 outputs the selected set of coding parameters as the coded stream Te to the outside, without outputting the set of coding parameters that are not selected.

[0649] The prediction parameter coding unit 111 derives prediction parameters used when generating a prediction picture based on the parameters input from the prediction image generation unit 101 , and codes the derived prediction parameters to generate a set of coding parameters. The prediction parameter coding unit 111 outputs the generated set of coding parameters to the entropy coding unit 104 .

[0650] The prediction parameter encoding unit 111 stores the prediction parameter corresponding to the parameter selected by the encoding parameter determination unit 110 in the prediction parameter memory 108 among the generated set of encoding parameters.

[0651] When the prediction mode predMode input from the prediction image generator 101 indicates the inter prediction mode, the prediction parameter encoder 111 causes the inter prediction parameter encoder 112 to operate. When the prediction mode predMode indicates the intra prediction mode, the prediction parameter encoder 111 causes the intra prediction parameter encoder 113 to operate.

[0652] The inter-frame prediction parameter encoding unit 112 derives the inter-frame prediction parameter based on the prediction parameter input from the encoding parameter determination unit 110. The inter-frame prediction parameter encoding unit 112 includes, as a configuration for deriving the inter-frame prediction parameter, a decoder 303 (see Figure 5 The structure of the inter-frame prediction parameter encoding unit 112 is the same as that of the inter-frame prediction parameter encoding unit 112. The structure of the inter-frame prediction parameter encoding unit 112 will be described below.

[0653] The intra prediction parameter encoding unit 113 determines the intra prediction mode IntraPredMode indicated by the prediction mode predMode input from the encoding parameter determination unit 110 as a set of inter prediction parameters.

[0654] (Configuration of Inter-frame Prediction Parameter Encoding Unit)

[0655] Next, the configuration of the inter prediction parameter encoding unit 112 will be described. The inter prediction parameter encoding unit 112 is a unit corresponding to the inter prediction parameter decoding unit 303 .

[0656] Fig.13 It is a schematic diagram showing the structure of the inter-frame prediction parameter encoding unit 112 according to this embodiment.

[0657] The inter-frame prediction parameter encoding unit 112 includes a merged prediction parameter derivation unit 1121 , an AMVP prediction parameter derivation unit 1122 , a subtraction unit 1123 , and a prediction parameter integration unit 1126 .

[0658] The combined prediction parameter derivation unit 1121 has the same functions as the combined prediction parameter derivation unit 3036 (see Figure 7 ) has the same structure as the AMVP prediction parameter derivation unit 1122, which has the same structure as the AMVP prediction parameter derivation unit 3032 (see Figure 8 )Same composition.

[0659] The inter-frame prediction parameter encoding control unit 1031 instructs the entropy encoding unit 104 to decode the code (syntax element) associated with the inter-frame prediction, and encodes the code (syntax element) contained in the encoded data, such as the partition mode part_mode, the merge flag merge_flag, the merge index merge_idx, the inter-frame prediction flag inter_pred_idc, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx, and the difference vector mvdLX.

[0660] The inter-frame prediction parameter encoding control unit 1031 is as follows Fig.10 As shown, it includes a merge index coding unit 10312, a vector candidate index coding unit 10313, and a partition mode coding unit, a merge flag coding unit, an inter-frame prediction flag coding unit, a reference picture index coding unit, and a vector difference coding unit, which are not shown. The partition mode coding unit, the merge flag coding unit, the merge index coding unit, the inter-frame prediction flag coding unit, the reference picture index coding unit, the vector candidate index coding unit 10313, and the vector difference coding unit encode the partition mode part_mode, the merge flag merge_flag, the merge index merge_idx, the inter-frame prediction flag inter_pred_idc, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx, and the difference vector mvdLX, respectively.

[0661] In the merge prediction parameter derivation unit 1121, when the prediction mode predMode input from the prediction image generation unit 101 indicates the merge prediction mode, the merge index merge_idx is input from the encoding parameter determination unit 110. The merge index merge_idx is output to the prediction parameter integration unit 1126. The merge prediction parameter derivation unit 1121 reads the reference picture index refIdxLX and the motion vector mvLX of the reference block indicated by the merge index merge_idx in the merge candidate from the prediction parameter memory 108. The merge candidate is a reference block located in a predetermined range from the encoding target block to be encoded (for example, in a reference block adjacent to the lower left end, upper left end, or upper right end of the encoding target block), and is a reference block for which the encoding process has been completed.

[0662] The AMVP prediction parameter derivation unit 1122 has the same functions as the AMVP prediction parameter derivation unit 3032 (see Figure 8 )Same composition.

[0663] In the AMVP prediction parameter derivation unit 1122, when the prediction mode predMode input from the prediction image generation unit 101 indicates the inter-frame prediction mode, the motion vector mvLX is input from the encoding parameter determination unit 110. The AMVP prediction parameter derivation unit 1122 derives the prediction vector mvpLX based on the input motion vector mvLX. The AMVP prediction parameter derivation unit 1122 outputs the derived prediction vector mvpLX to the subtraction unit 1123. It should be noted that the reference picture index refIdx and the prediction vector index mvp_LX_idx are output to the prediction parameter integration unit 1126.

[0664] The subtraction unit 1123 subtracts the prediction vector mvpLX input from the AMVP prediction parameter derivation unit 1122 from the motion vector mvLX input from the encoding parameter determination unit 110 to generate a difference vector mvdLX. The difference vector mvdLX is output to the prediction parameter integration unit 1126.

[0665] When the prediction mode predMode input from the predicted image generation unit 101 indicates the merge prediction mode, the prediction parameter integration unit 1126 outputs the merge index merge_idx input from the encoding parameter determination unit 110 to the entropy encoding unit 104 .

[0666] When the prediction mode predMode input from the predicted image generation unit 101 indicates the inter prediction mode, the prediction parameter integration unit 1126 performs the following processing.

[0667] The prediction parameter integration unit 1126 integrates the reference picture index refIdxLX and the prediction vector index mvp_LX_idx input from the coding parameter determination unit 110 and the difference vector mvdLX input from the subtraction unit 1123. The prediction parameter integration unit 1126 outputs the integrated code to the entropy coding unit 104.

[0668] It should be noted that the inter-frame prediction parameter encoding unit 112 may also include an inter-frame prediction parameter encoding control unit (not shown), which instructs the entropy encoding unit 104 to indicate the code related to the inter-frame prediction (decoding of the syntax elements), and encodes the code (syntax elements) contained in the encoded data, such as the partition mode part_mode, the merge flag merge_flag, the merge index merge_idx, the inter-frame prediction flag inter_pred_idc, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx, and the differential vector mvdLX.

[0669] In this case, the inter-frame prediction parameter encoding control unit includes, in addition to the additional prediction flag encoding unit that encodes the illuminance compensation flag ic_enable_flag for indicating whether to perform additional prediction (see Fig.10 ) also includes a merge index encoding unit (refer to Fig.10 )、Vector candidate index encoding unit (refer to Fig.10 ), and a partition mode encoder, a merge flag encoder, an inter-frame prediction flag encoder, a reference picture index encoder, a vector difference encoder, etc. The partition mode encoder, the merge flag encoder, the merge index encoder, the inter-frame prediction flag encoder, the reference picture index encoder, the vector candidate index encoder, and the vector difference encoder encode the partition mode part_mode, the merge flag merge_flag, the merge index merge_idx, the inter-frame prediction flag inter_pred_idc, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx, and the difference vector mvdLX, respectively.

[0670] It should be noted that a part of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment, such as the entropy decoding unit 301, the prediction parameter decoding unit 302, the prediction image generation unit 101, the DCT / quantization unit 103, the entropy coding unit 104, the inverse quantization / inverse DCT unit 105, the coding parameter determination unit 110, the prediction parameter coding unit 111, the entropy decoding unit 301, the prediction parameter decoding unit 302, the prediction image generation unit 308, and the inverse quantization / inverse DCT unit 311, can be realized by a computer. In this case, the program for realizing the control function can be recorded in a computer-readable recording medium, and the computer system can read the program recorded in the recording medium and execute it. It should be noted that the "computer system" mentioned here refers to a computer system built into any one of the image encoding devices 11-11h and the image decoding devices 31-31h, and a computer system including hardware such as an OS and peripheral devices is used. In addition, the "computer-readable recording medium" refers to a removable medium such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into the computer system. Furthermore, "computer-readable recording media" may include: media that dynamically store programs for a short period of time, such as communication lines when sending programs via networks such as the Internet or communication lines such as telephone lines; media that store programs for a fixed period of time, such as volatile memories inside computer systems that serve as servers or clients in this case. In addition, the above-mentioned program may be a program for realizing a part of the aforementioned functions, or a program that can further realize the aforementioned functions by combining them with programs already recorded in a computer system.

[0671] In addition, part or all of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment may be implemented as an integrated circuit such as LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually processor-based, or part or all of the functional blocks may be integrated to form a processor. In addition, the method of integrated circuitization is not limited to LSI, and it may be implemented by a dedicated circuit or a general-purpose processor. In addition, in the case where an integrated circuit technology that replaces LSI emerges due to the advancement of semiconductor technology, an integrated circuit based on the technology may also be used.

[0672] As mentioned above, one embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to the above, and various design changes and the like can be made within the scope not departing from the gist of the present invention.

[0673] [Application Examples]

[0674] The above-mentioned image encoding device 11 and image decoding device 31 can be mounted on various devices for sending, receiving, recording, and reproducing moving images. It should be noted that the moving images can be natural moving images captured by a camera or the like, or artificial moving images (including CG and GUI) generated by a computer or the like.

[0675] First, refer to Fig.33 , a description will be given of a case where the above-mentioned image encoding device 11 and image decoding device 31 can be used for sending and receiving moving images.

[0676] Fig.33 (a) is a block diagram showing the structure of a transmitting device PROD_A equipped with an image coding device 11. Fig.33 As shown in (a), the transmitting device PROD_A includes: an encoding unit PROD_A1 that obtains encoded data by encoding a moving image, a modulation unit PROD_A2 that obtains a modulated signal by modulating a carrier according to the encoded data obtained by the encoding unit PROD_A1, and a transmitting unit PROD_A3 that transmits the modulated signal obtained by the modulation unit PROD_A2. The above-mentioned image encoding device 11 is used as the encoding unit PROD_A1.

[0677] The transmitting device PROD_A may further include: a camera PROD_A4 for shooting moving images, a recording medium PROD_A5 recording moving images, an input terminal PROD_A6 for inputting moving images from the outside, and an image processing unit A7 for generating or processing images, as a supply source of moving images input to the encoding unit PROD_A1. Fig.33 (a) illustrates that the transmission device PROD_A includes all of these components, but some of them may be omitted.

[0678] It should be noted that the recording medium PROD_A5 may be a medium for recording uncoded moving images, or a medium for recording moving images coded in a recording coding method different from a transmission coding method. In the latter case, it is preferred that a decoding unit (not shown) for decoding coded data read from the recording medium PROD_A5 in accordance with the recording coding method be interposed between the recording medium PROD_A5 and the coding unit PROD_A1.

[0679] Fig.33 (b) is a block diagram showing the structure of a receiving device PROD_B equipped with an image decoding device 31. Fig.33As shown in (b), the receiving device PROD_B includes a receiving unit PROD_B1 that receives a modulated signal, a demodulating unit PROD_B2 that demodulates the modulated signal received by the receiving unit PROD_B1 to obtain coded data, and a decoding unit PROD_B3 that decodes the coded data obtained by the demodulating unit PROD_B2 to obtain a moving image. The above-mentioned image decoding device 31 is used as the decoding unit PROD_B3.

[0680] The receiving device PROD_B may further include a display PROD_B4 for displaying images, a recording medium PROD_B5 for recording moving images, and an output terminal PROD_B6 for outputting the moving images to the outside, as a supply destination for the moving images output by the decoding unit PROD_B3. Fig.33 (b) illustrates that the receiving device PROD_B includes all of these components, but some of them may be omitted.

[0681] It should be noted that the recording medium PROD_B5 may be a medium for recording uncoded moving images, or may be a medium coded in a recording coding method different from a transmission coding method. In the latter case, it is preferred that a coding unit (not shown) that codes the moving images obtained from the decoding unit PROD_B3 in a recording coding method is interposed between the decoding unit PROD_B3 and the recording medium PROD_B5.

[0682] It should be noted that the transmission medium for transmitting the modulated signal can be wireless or wired. In addition, the transmission scheme for transmitting the modulated signal can be broadcast (here, it refers to a transmission scheme in which the transmission destination is not pre-specified) or communication (here, it refers to a transmission scheme in which the transmission destination is pre-specified). That is, the transmission of the modulated signal can be achieved through any one of wireless broadcasting, wired broadcasting, wireless communication, and wired communication.

[0683] For example, a broadcasting station (broadcasting equipment, etc.) / receiving station (television receiver, etc.) of terrestrial digital broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via wireless broadcasting. In addition, a broadcasting station (broadcasting equipment, etc.) / receiving station (television receiver, etc.) of cable television broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via wired broadcasting.

[0684] In addition, a server (workstation, etc.) / client (television receiver, personal computer, smart phone, etc.) of a VOD (Video On Demand) service or a motion picture sharing service using the Internet is an example of a transmitter PROD_A / receiver PROD_B that transmits and receives a modulated signal by communication (usually, a wireless or wired transmission medium is used as a transmission medium in a LAN, and a wired transmission medium is used as a transmission medium in a WAN). Here, a personal computer includes a desktop PC, a laptop PC, and a tablet PC. In addition, a smart phone also includes a multifunctional portable phone terminal.

[0685] It should be noted that the client of the moving image sharing service has the function of decoding the coded data downloaded from the server and displaying it on the display, and also has the function of encoding the moving images captured by the camera and uploading them to the server. That is, the client of the moving image sharing service plays the role of both the sending device PROD_A and the receiving device PROD_B.

[0686] Next, refer to Fig.34 , a description will be given of a case where the above-mentioned image encoding device 11 and image decoding device 31 can be used for recording and reproducing moving images.

[0687] Fig.34 (a) is a block diagram showing the structure of a recording device PROD_C equipped with the above-mentioned image encoding device 11. Fig.34 As shown in (a), the recording device PROD_C includes a coding unit PROD_C1 for coding a moving image to obtain coded data, and a writing unit PROD_C2 for writing the coded data obtained by the coding unit PROD_C1 to the recording medium PROD_M. The above-mentioned image coding device 11 is used as the coding unit PROD_C1.

[0688] It should be noted that the recording medium PROD_M can be (1) a type of recording medium built into the recording device PROD_C such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), or (2) a type of recording medium connected to the recording device PROD_C such as an SD memory card or a USB (Universal Serial Bus) flash memory, or (3) a type of recording medium loaded into a drive device (not shown) built into the recording device PROD_C such as a DVD (Digital Versatile Disc) or a BD (Blu-ray Disc, a registered trademark).

[0689] In addition, the recording device PROD_C may further include: a camera PROD_C3 for shooting moving images, an input terminal PROD_C4 for inputting moving images from the outside, a receiving unit PROD_C5 for receiving moving images, and an image processing unit C6 for generating or processing images, as a supply source of moving images input to the encoding unit PROD_C1. Fig.34 (a) shows an example in which the recording device PROD_C includes all of these components, but some of them may be omitted.

[0690] It should be noted that the receiving unit PROD_C5 can receive uncoded moving images, or can receive coded data coded in a transmission coding method different from the recording coding method. In the latter case, it is preferable to place a transmission decoding unit (not shown) that decodes the coded data coded in the transmission coding method between the receiving unit PROD_C5 and the coding unit PROD_C1.

[0691] Examples of such recording device PROD_C include DVD recorders, BD recorders, and HDD (Hard Disk Drive) recorders (in which case, the input terminal PROD_C4 or the receiving unit PROD_C5 is the main source of motion images). In addition, portable cameras (in which case, the camera PROD_C3 is the main source of motion images), personal computers (in which case, the receiving unit PROD_C5 or the image processing unit C6 is the main source of motion images), and smartphones (in which case, the camera PROD_C3 or the receiving unit PROD_C5 is the main source of motion images) are also examples of such recording device PROD_C.

[0692] Fig.34 (b) is a block diagram showing the structure of a playback device PROD_D equipped with the above-mentioned image decoding device 31. Fig.34 As shown in (b), the playback device PROD_D includes a reading unit PROD_D1 for reading coded data written in the recording medium PROD_M, and a decoding unit PROD_D2 for obtaining a moving image by decoding the coded data read by the reading unit PROD_D1. The above-mentioned image decoding device 31 is used as the decoding unit PROD_D2.

[0693] It should be noted that the recording medium PROD_M can be (1) a type of recording medium that is built into the reproduction device PROD_D, such as an HDD, SSD, etc., or (2) a type of recording medium that is connected to the reproduction device PROD_D, such as an SD memory card, USB flash memory, etc., or (3) a type of recording medium that is loaded into a drive device (not shown) built into the reproduction device PROD_D, such as a DVD, BD, etc.

[0694] In addition, the reproduction device PROD_D may further include a display PROD_D3 for displaying moving images, an output terminal PROD_D4 for outputting moving images to the outside, and a transmission unit PROD_D5 for transmitting moving images, as a supply destination of the moving images output by the decoding unit PROD_D2. Fig.34 (b) illustrates an example in which the playback device PROD_D includes all of these components, but some of them may be omitted.

[0695] It should be noted that the sending unit PROD_D5 can send uncoded moving images, or can send coded data coded in a transmission coding method different from the recording coding method. In the latter case, it is better to place a coding unit (not shown) that encodes moving images in a transmission coding method between the decoding unit PROD_D2 and the sending unit PROD_D5.

[0696] Examples of such a reproduction device PROD_D include a DVD player, a BD player, and an HDD player (in this case, the output terminal PROD_D4 connected to a television receiver or the like becomes the main supply destination of the motion image). In addition, a television receiver (in this case, the display PROD_D3 becomes the main supply destination of the motion image), a digital signage (also called an electronic billboard, an electronic bulletin board, etc., the display PROD_D3 or the transmission unit PROD_D5 becomes the main supply destination of the motion image), a desktop PC (in this case, the output terminal PROD_D4 or the transmission unit PROD_D5 becomes the main supply destination of the motion image), a laptop or tablet PC (in this case, the display PROD_D3 or the transmission unit PROD_D5 becomes the main supply destination of the motion image), and a smartphone (in this case, the display PROD_D3 or the transmission unit PROD_D5 becomes the main supply destination of the motion image) are also examples of such a reproduction device PROD_D.

[0697] (Hardware implementation and software implementation)

[0698] Furthermore, each block of the above-described image decoding device 31 and image encoding device 11 may be implemented in hardware by a logic circuit formed on an integrated circuit (IC chip), or may be implemented in software by a CPU (Central Processing Unit).

[0699] In the latter case, each of the above-mentioned devices has: a CPU that executes commands of a program that realizes each function, a ROM (Read Only Memory) that stores the above-mentioned program, a RAM (Random Access Memory) that expands the above-mentioned program, and a storage device (recording medium) such as a memory that stores the above-mentioned program and various data. Then, the purpose of one embodiment of the present invention can also be achieved in the following manner: the software that realizes the above-mentioned functions, that is, the recording medium that records the program code (executable form program, intermediate code program, source program) of the control program of each of the above-mentioned devices in a computer-readable manner is supplied to each of the above-mentioned devices, and the computer (or CPU, MPU) reads the program code recorded in the recording medium and executes it.

[0700] As the above-mentioned recording medium, for example, tapes such as magnetic tapes and cassette tapes, magnetic disks including floppy disks (registered trademark) / hard disks, disks including CD-ROM (Compact Disc Read-Only Memory) / MO disks (Magneto-Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc) / CD-R (CD Recordable) / Blu-ray Disc (registered trademark) and the like, cards such as IC cards (including memory cards) / optical cards, semiconductor memories such as mask ROM / EPROM (Erasable Programmable Read-Only Memory) / EEPROM (Electrically Erasable and Programmable Read-Only Memory: Electrically Erasable and Programmable Read-Only Memory, registered trademark) / flash ROM, or logic circuits such as PLD (Programmable logic device) and FPGA (Field Programmable Gate Array) can be used.

[0701] In addition, each of the above-mentioned devices can also be configured to be connected to a communication network, and the above-mentioned program code can be supplied via the communication network. The communication network can transmit program codes without any particular limitation. For example, the Internet, intranet, extranet, LAN (Local Area Network), ISDN (Integrated Services Digital Network), VAN (Value-Added Network), CATV (Community Antenna television / Cable Television) communication network, virtual private network (Virtual Private Network), telephone line network, mobile communication network, satellite communication network, etc. can be used. In addition, the transmission medium constituting the communication network can be any medium that can transmit program codes, and is not limited to a specific structure or type. For example, it can be used in wired communication such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line transmission, wired TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, or wireless communication such as IrDA (Infrared Data Association), infrared such as remote controller, Bluetooth (registered trademark), IEEE802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance, registered trademark), mobile phone network, satellite line, terrestrial digital network, etc. It should be noted that one embodiment of the present invention can be realized even in the form of a computer data signal embedded in a carrier wave that embodies the above-mentioned program code through electronic transmission.

[0702] One embodiment of the present invention is not limited to the above-mentioned embodiment, and various changes can be made within the scope of the claims. That is, an embodiment obtained by combining technical solutions appropriately changed within the scope of the claims is also included in the technical scope of the present invention.

[0703] (Cross-reference of related applications)

[0704] This application claims the benefit of priority based on Japanese Patent Application No. 2016-019354, filed on February 3, 2016, the entire contents of which are incorporated herein by reference.

[0705] Industrial Applicability

[0706] An embodiment of the present invention can be preferably applied to an image decoding device that decodes coded data obtained by encoding image data, and an image encoding device that generates coded data obtained by encoding image data. In addition, it can be preferably applied to the data structure of coded data generated by the image encoding device and referred to by the image decoding device.

[0707] Explanation of symbols

[0708] 1: Image transmission system

[0709] 11: Image coding device (moving image coding device)

[0710] 101: Prediction image generation unit

[0711] 102: Subtraction Department

[0712] 103: DCT / Quantization

[0713] 104: Entropy coding unit

[0714] 105: Inverse quantization / inverse DCT section

[0715] 106: Addition Department

[0716] 108: prediction parameter memory (frame memory)

[0717] 109: Reference picture memory (frame memory)

[0718] 110: Coding parameter determination unit

[0719] 111: Prediction parameter encoding unit

[0720] 112: Inter-frame prediction parameter encoding unit

[0721] 1121: Merge prediction parameter derivation section

[0722] 1122: AMVP prediction parameter derivation department

[0723] 1123: Subtraction Department

[0724] 1126: Prediction Parameter Integration Department

[0725] 113: Intra-frame prediction parameter encoding unit

[0726] 21: Network

[0727] 31: Image decoding device (moving image decoding device)

[0728] 301: Entropy decoding unit

[0729] 302: prediction parameter decoding unit

[0730] 303: Inter-frame prediction parameter decoding unit (prediction vector calculation unit)

[0731] 30312: Merge index decoding unit

[0732] 30313: Vector candidate index decoding unit

[0733] 3032: AMVP prediction parameter derivation department

[0734] 3033: Vector candidate derivation unit

[0735] 3035: Addition Department

[0736] 3036: Merge prediction parameter derivation section

[0737] 30361: Merge candidate derivation department

[0738] 303611: Merge candidate storage department

[0739] 30362: Merger Candidate Selection Department

[0740] 304: Intra-frame prediction parameter decoding unit

[0741] 306: Reference picture memory (frame memory)

[0742] 307: prediction parameter memory (frame memory)

[0743] 308: Prediction image generation unit

[0744] 309: Inter-frame prediction image generation unit

[0745] 3091: Motion compensation unit (prediction image generation device, prediction image generation unit)

[0746] 30911: Motion Compensation Gradient Unit

[0747] 309122: Gradient correction coefficient derivation unit (second switching unit, editing unit)

[0748] 309112: Gradient derivation unit (first switching unit)

[0749] 3094: Weight prediction department

[0750] 310: Intra-frame prediction image generation unit

[0751] 311: Inverse quantization / inverse DCT section

[0752] 312: Addition Department

[0753] 41: Image display device

Claims

1. A prediction image generation device, which generates a prediction image by performing motion compensation on a plurality of reference images, characterized in that: A predicted image generation unit is provided, the predicted image generation unit generating a predicted image by referring to a first reference image, a second reference image, and a gradient correction item; The predicted image generation unit generates a first gradient image based on the gradient of the first reference image along a first direction, generates a second gradient image based on the gradient of the first reference image along a second direction, generates a third gradient image based on the gradient of the second reference image along the first direction, and generates a fourth gradient image based on the gradient of the second reference image along the second direction, and derives the gradient correction term according to the product of each pixel value of the first to fourth gradient images and a weight coefficient, The weight coefficient takes a first value or a second value, a method for determining the first value is different from a method for determining the second value, The first value is determined for a block using a sum of the first gradient image and a third gradient image based on a difference between the first reference image and the second reference image, The second value is determined for the block using a difference between the first reference image and the second reference image, a sum of the first gradient image and a third gradient image, a sum of the second gradient image and a fourth gradient image, and the first value.

2. The image generating device according to claim 1, characterized in that: A motion compensation unit is provided, which derives the gradient correction term according to the product of the difference between the first gradient image and the third gradient image and the weight coefficient, and the product of the difference between the second gradient image and the fourth gradient image and the weight coefficient.

3. The image generating device according to claim 2, characterized in that: The motion compensation unit derives the gradient correction term according to the product of the difference between the first gradient image and the third gradient image and the weight coefficient, and the product of the difference between the second gradient image and the fourth gradient image and the weight coefficient, including: The motion compensation unit derives the gradient correction term according to a product of a difference between the first gradient image and the third gradient image and a first value of the weight coefficient, and a product of a difference between the second gradient image and the fourth gradient image and a second value of the weight coefficient.

4. A moving picture decoding device, characterized in that: comprising: the predicted image generation device according to any one of claims 1 to 3, The encoding target image is restored by adding or subtracting the residual image from the predicted image.

5. A motion picture encoding device, characterized in that: comprising: the predicted image generation device according to any one of claims 1 to 3, The residual between the predicted image and the encoding target image is encoded.

6. A motion picture encoding method, characterized in that: The method comprises: generating a predicted image by referring to the first reference image, the second reference image, and the gradient correction item; and Encode the residual between the predicted image and the image to be encoded. The method further includes: generating a first gradient image based on the gradient of the first reference image along a first direction, generating a second gradient image based on the gradient of the first reference image along a second direction, generating a third gradient image based on the gradient of the second reference image along the first direction, and generating a fourth gradient image based on the gradient of the second reference image along the second direction, deriving the gradient correction term according to the product of each pixel value of the first to fourth gradient images and the weight coefficient, The weight coefficient takes a first value or a second value, a method for determining the first value is different from a method for determining the second value, The first value is determined for a block using a sum of the first gradient image and a third gradient image based on a difference between the first reference image and the second reference image, The second value is determined for the block using a difference between the first reference image and the second reference image, a sum of the first gradient image and a third gradient image, a sum of the second gradient image and a fourth gradient image, and the first value.

7. The method according to claim 6, characterized in that The gradient correction term is derived according to a product of a difference between the first gradient image and the third gradient image and the weight coefficient, and a product of a difference between the second gradient image and the fourth gradient image and the weight coefficient.

8. The method according to claim 7, characterized in that The gradient correction term is derived according to the product of the weight coefficient and the difference between the first gradient image and the third gradient image, and the product of the weight coefficient and the difference between the second gradient image and the fourth gradient image, including: The gradient correction term is derived according to a product of a difference between the first gradient image and the third gradient image and a first value of the weight coefficient, and a product of a difference between the second gradient image and the fourth gradient image and a second value of the weight coefficient.

9. A motion picture decoding method, characterized in that: The method comprises: generating a predicted image by referring to the first reference image, the second reference image, and the gradient correction item; and The image to be coded is restored by adding or subtracting the residual image from the predicted image. The method further includes: generating a first gradient image based on the gradient of the first reference image along a first direction, generating a second gradient image based on the gradient of the first reference image along a second direction, generating a third gradient image based on the gradient of the second reference image along the first direction, and generating a fourth gradient image based on the gradient of the second reference image along the second direction, deriving the gradient correction term according to the product of each pixel value of the first to fourth gradient images and the weight coefficient, The weight coefficient takes a first value or a second value, a method for determining the first value is different from a method for determining the second value, The first value is determined for a block using a sum of the first gradient image and a third gradient image based on a difference between the first reference image and the second reference image, The second value is determined for the block using a difference between the first reference image and the second reference image, a sum of the first gradient image and a third gradient image, a sum of the second gradient image and a fourth gradient image, and the first value.

10. The method according to claim 9, characterized in that The gradient correction term is derived according to a product of a difference between the first gradient image and the third gradient image and the weight coefficient, and a product of a difference between the second gradient image and the fourth gradient image and the weight coefficient.

11. The method according to claim 10, characterized in that The gradient correction term is derived according to the product of the weight coefficient and the difference between the first gradient image and the third gradient image, and the product of the weight coefficient and the difference between the second gradient image and the fourth gradient image, including: The gradient correction term is derived according to a product of a difference between the first gradient image and the third gradient image and a first value of the weight coefficient, and a product of a difference between the second gradient image and the fourth gradient image and a second value of the weight coefficient.

Images