Video decoding device and video encoding device

Abstract

To provide an image decoding device and an image encoding device that reduce the complexity of image quality improvement processing.SOLUTION: In an image transmission system including an image encoding device, a network, an image decoding device, and an image display device, the image decoding device includes an inter-prediction parameter decoding unit that has a process of correcting two motion vectors from errors between two predicted images, and performs a process of correcting the two motion vectors when neither of the two predicted images is a weighted prediction.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] An embodiment of the present invention relates to a video decoding device and a video encoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates a decoded image by decoding the encoded data.

[0003] Specific examples of video encoding methods include H.264 / AVC and High-Efficiency Video Coding (HEVC).

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing images, coding tree units (CTUs) obtained by dividing slices, coding units (sometimes called coding units: CUs) obtained by dividing coding tree units, and transform units (TUs) obtained by dividing coding units, and are coded / decoded for each CU.

[0005] In such a video coding method, a predicted image is usually generated based on a locally decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the predicted image from the input image (original image) is coded. Methods for generating a predicted image include inter-prediction and intra-prediction.

[0006] Moreover, Non-Patent Document 1 can be cited as a recent example of a video encoding and decoding technique. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "Versatile Video Coding (Draft 4)", JVET-M1001, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 2019 Summary of the Invention [Problem to be solved by the invention]

[0008] When deriving the bidirectional predictive image in Non-Patent Document 1, there was a problem in that the DMVR process, which uses two predictive images to correct a motion vector to improve the quality of the predicted image, and prediction using BDOF processing, which uses a gradient image to improve the quality of the predicted image (BDOF prediction), were highly complex processes.

[0009] An object of the embodiments of the present invention is to realize an image decoding device and an image encoding device that reduce the complexity of these image quality improvement processes. [Means for solving the problem]

[0010] In order to solve the above problem, an image decoding device according to one aspect of the present invention comprises: A video decoding device that performs a decoder side motion vector refinement (DMVR) process using two reference pictures and motion vectors mvL0 and mvL1, a DMVR unit that executes the DMVR processing when a dmvrFlag indicating whether the DMVR processing is to be performed is TRUE; a weighted prediction unit that performs weighted prediction using a first weighting factor, a first offset, a second weighting factor, and a second offset; the DMVR unit sets the dmvrFlag to TRUE based on a predetermined condition, wherein the predetermined condition for setting the dmvrFlag to TRUE includes that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE; The luma_weight_l0_flag[refIdxL0] indicates whether the first weighting factor and the first offset of luma corresponding to the L0 reference picture indicated by the reference picture index refIdxL0 are present or not; The luma_weight_l1_flag[refIdxL1] indicates whether or not the second weighting factor and the second offset for luma corresponding to the L1 reference picture indicated by the reference picture index refIdxL1 are present.

[0011] Moreover, an image encoding device according to an aspect of the present invention includes: A video encoding device that performs a decoder side motion vector refinement (DMVR) process using two reference pictures and motion vectors mvL0 and mvL1, a DMVR unit that executes the DMVR processing when a dmvrFlag indicating whether the DMVR processing is to be performed is TRUE; a weighted prediction unit that performs weighted prediction using a first weighting factor, a first offset, a second weighting factor, and a second offset; the DMVR unit sets the dmvrFlag to TRUE based on a predetermined condition, wherein the predetermined condition for setting the dmvrFlag to TRUE includes that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE; The luma_weight_l0_flag[refIdxL0] indicates whether the first weighting factor and the first offset of luma corresponding to the L0 reference picture indicated by the reference picture index refIdxL0 are present or not; The luma_weight_l1_flag[refIdxL1] indicates whether or not the second weighting factor and the second offset for luma corresponding to the L1 reference picture indicated by the reference picture index refIdxL1 are present. Effect of the Invention

[0012] According to the above configuration, it is possible to realize an image decoding device and an image encoding device in which the complexity of image quality improvement processing is reduced. [Brief description of the drawings]

[0013] [Figure 1] 1 is a schematic diagram showing a configuration of an image transmission system according to an embodiment of the present invention. [Diagram 2] 1 is a diagram showing the configuration of a transmitting device equipped with a video encoding device according to the present embodiment, and a receiving device equipped with a video decoding device, where (a) shows the transmitting device equipped with the video encoding device, and (b) shows the receiving device equipped with the video decoding device. [Diagram 3] 1 is a diagram showing the configuration of a recording device equipped with a video encoding device according to the present embodiment, and a playback device equipped with a video decoding device, where (a) shows the configuration of a recording device equipped with a video encoding device, and (b) shows the configuration of a playback device equipped with a video decoding device. [Figure 4] FIG. 2 is a diagram showing a hierarchical structure of data in an encoded stream. [Diagram 5] FIG. 13 is a diagram illustrating an example of division of a CTU. [Figure 6] FIG. 2 is a conceptual diagram showing an example of a reference picture and a reference picture list. [Figure 7] FIG. 1 is a schematic diagram showing a configuration of a video decoding device. [Figure 8]FIG. 13 is a schematic diagram showing a configuration of an inter-prediction parameter decoding unit. [Figure 9] 1 is a schematic diagram showing the configuration of a merge prediction parameter derivation unit and an AMVP prediction parameter derivation unit. [Figure 10] FIG. 2 is a schematic diagram showing the configuration of a DMVR unit. [Figure 11] 13 is a flowchart showing a process flow in the DMVR unit. [Figure 12] 11 is a flowchart for explaining a process of determining BDOF by error threshold processing in DMVR. [Figure 13] 11 is a schematic diagram showing a configuration of an inter-prediction image generation unit included in the prediction image generation unit. FIG. [Figure 14] 11 is a flowchart showing an example of the flow of a prediction mode selection process in the video decoding device. [Figure 15] 13 is a flowchart illustrating the flow of a process for deriving a predicted image. [Figure 16] FIG. 2 is a schematic diagram showing the configuration of a BDOF section. [Figure 17] FIG. 13 is a diagram showing an example of an area where a BDOF unit performs padding; [Figure 18] FIG. 1 is a schematic diagram showing a configuration of a video encoding device. [Figure 19] 13 is a schematic diagram showing a configuration of a parameter encoding unit. [Figure 20] 1A and 1B are diagrams illustrating an example of the number of search distance candidates and the number of derivation direction candidates in a video encoding device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] FIG. 1 is a schematic diagram showing the configuration of an image transmission system 1 according to this embodiment.

[0016] The image transmission system 1 is a system that transmits an encoded stream obtained by encoding an image to be encoded, decodes the transmitted encoded stream, and displays an image. The image transmission system 1 includes a video encoding device (image encoding device) 11, a network 21, a video decoding device (image decoding device) 31, and a video display device (image display device) 41.

[0017] An image T is input to the video encoding device 11 .

[0018] The network 21 transmits the encoded stream Te generated by the video encoding device 11 to the video decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be replaced by a storage medium on which the encoded stream Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

[0019] The video decoding device 31 decodes each of the coded streams Te transmitted by the network 21, and generates one or more decoded images Td.

[0020] The video display device 41 displays all or part of one or more decoded images Td generated by the video decoding device 31. The video display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, or a head mounted display (HMD). When the video decoding device 31 has high processing power, the video display device 41 displays high quality images, and when the video decoding device 31 has low processing power, the video display device 41 displays images that do not require high processing power or display power.

[0021] <Operator> The operators used in this specification are described below.

[0022] >> represents a right bit shift, << represents a left bit shift, & represents a bitwise AND, | represents a bitwise OR, |= represents an OR assignment operator, and || represents a logical OR.

[0023] x?y:z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0024] Clip3(a, b, c) is a function that clips c to a value between a and b (inclusive). If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (where a <= b).

[0025] abs(a) is a function that returns the absolute value of a.

[0026] Int(a) is a function that returns the integer value of a.

[0027] floor(a) is a function that returns the largest integer less than or equal to a.

[0028] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0029] a / d represents the division of a by d (truncating the fractional part).

[0030] sign(a) is a function that returns the sign of a.

[0031] a^b represents a to the power of b.

[0032] <Structure of the Encoded Stream Te> Before describing in detail the video encoding device 11 and the video decoding device 31 according to this embodiment, the data structure of the encoded stream Te generated by the video encoding device 11 and decoded by the video decoding device 31 will be described.

[0033] Fig. 4 is a diagram showing a hierarchical structure of data in an encoded stream Te. The encoded stream Te illustratively includes a sequence and a plurality of pictures constituting the sequence. (a) to (f) of Fig. 4 show an encoded video sequence that defines a sequence SEQ, an encoded picture that defines a picture PICT, an encoded slice that defines a slice S, encoded slice data that defines slice data, an encoding tree unit included in the encoded slice data, and an encoding unit included in the encoding tree unit, respectively.

[0034] (Coded Video Sequence) In the coded video sequence, a set of data that the video decoding device 31 refers to in order to decode the sequence SEQ to be processed is specified. As shown in Fig. 4(a), the sequence SEQ includes a video parameter set (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0035] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers and each individual layer included in the video.

[0036] The sequence parameter set SPS specifies a set of coding parameters that the video decoding device 31 refers to in order to decode the target sequence. For example, the width and height of a picture are specified. Note that there may be multiple SPSs. In that case, one of the multiple SPSs is selected from the PPS.

[0037] The picture parameter set PPS specifies a set of coding parameters that the video decoding device 31 refers to in order to decode each picture in the target sequence. For example, the picture parameter set PPS includes a reference value of the quantization width (pic_init_qp_minus26) used in decoding the picture and a flag (weighted_pred_flag) indicating the application of weighted prediction. Note that there may be multiple PPSs. In that case, one of the multiple PPSs is selected for each picture in the target sequence.

[0038] (Encoded Picture) A coded picture defines a set of data to be referenced by the video decoding device 31 in order to decode a picture PICT to be processed. As shown in FIG. 4(b), the picture PICT includes slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0039] In the following description, when there is no need to distinguish between slices 0 to NS-1, the subscripts of the symbols may be omitted. The same applies to other data that are included in the coded stream Te and that are to be described below and that are to be given subscripts.

[0040] (Coded Slice) A coded slice defines a set of data to be referenced by the video decoding device 31 in order to decode a target slice S. As shown in Fig. 4(c), a slice includes a slice header and slice data.

[0041] The slice header includes a group of coding parameters to be referred to by the video decoding device 31 in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates the slice type is an example of a coding parameter included in the slice header.

[0042] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction when encoding, (2) a P slice that uses unidirectional prediction or intra prediction when encoding, and (3) a B slice that uses unidirectional prediction, bidirectional prediction, or intra prediction when encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, when referring to P or B slice, it refers to a slice including a block that can use inter prediction.

[0043] In addition, the slice header may include a reference to a picture parameter set PPS (pic_parameter_set_id).

[0044] (Encoded slice data) The coded slice data defines a set of data to be referenced by the video decoding device 31 in order to decode the slice data to be processed. As shown in Fig. 4(d), the slice data includes a CTU. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0045] (coding tree unit) 4(e) specifies a set of data that the video decoding device 31 refers to in order to decode the CTU to be processed. The CTU is divided into coding units CU, which are basic units of the encoding process, by recursive quad tree division (QT (Quad Tree) division), binary tree division (BT (Binary Tree) division), or ternary tree division (TT (Ternary Tree) division). BT division and TT division are collectively called multi tree division (MT (Multi Tree) division). A node of a tree structure obtained by recursive quad tree division is called a coding node. Intermediate nodes of the quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is specified as the top coding node.

[0046] CT includes, as CT information, a QT split flag (qt_split_cu_flag) indicating whether or not QT split is performed, an MT split flag (mtt_split_cu_flag) indicating the presence or absence of MT split, an MT split direction (mtt_split_cu_vertical_flag) indicating the split direction of MT split, and an MT split type (mtt_split_cu_binary_flag) indicating the split type of MT split. qt_split_cu_flag, mtt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

[0047] 5 is a diagram showing an example of division of a CTU. When qt_split_cu_flag is 1, the coding node is divided into four coding nodes (FIG. 5(b)).

[0048] When qt_split_cu_flag is 0, and mtt_split_cu_flag is 0, the coding node is not split and has one CU as a node (Figure 5(a)). A CU is the terminal node of a coding node and is not split any further. A CU is the basic unit of coding processing.

[0049] When mtt_split_cu_flag is 1, the coding node is MT-split as follows: When mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 1, the coding node is split horizontally into two coding nodes (Fig. 5(d)), and when mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 1, the coding node is split vertically into two coding nodes (Fig. 5(c)). When mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 0, the coding node is split horizontally into three coding nodes (Fig. 5(f)), and when mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 0, the coding node is split vertically into three coding nodes (Fig. 5(e)). These are shown in Fig. 5(g).

[0050] Also, when the size of the CTU is 64x64 pixels, the size of the CU can be any of the following: 64x64 pixels, 64x32 pixels, 32x64 pixels, 32x32 pixels, 64x16 pixels, 16x64 pixels, 32x16 pixels, 16x32 pixels, 16x16 pixels, 64x8 pixels, 8x64 pixels, 32x8 pixels, 8x32 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels, 64x4 pixels, 4x64 pixels, 32x4 pixels, 4x32 pixels, 16x4 pixels, 4x16 pixels, 8x4 pixels, 4x8 pixels, and 4x4 pixels.

[0051] (Encoding Unit) As shown in Fig. 4(f), a set of data to be referenced by the video decoding device 31 in order to decode a coding unit to be processed is defined in the CU. Specifically, the CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines a prediction mode, etc.

[0052] The prediction process may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU. When the size of a CU and a sub-CU are equal, there is one sub-CU in the CU. When the size of a CU is larger than that of a sub-CU, the CU is divided into sub-CUs. For example, when a CU is 8x8 and a sub-CU is 4x4, the CU is divided into 2 parts horizontally and 2 parts vertically, into 4 sub-CUs.

[0053] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times).

[0054] The transform and quantization processes are performed in units of CUs, but the quantized transform coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0055] (Prediction parameters) The predicted image is derived from prediction parameters associated with the block, which include intra-prediction and inter-prediction parameters.

[0056] Hereinafter, prediction parameters of inter prediction will be described. Inter prediction parameters are composed of prediction list use flags predFlagL0, predFlagL1, reference picture indexes refIdxL0, refIdxL1, and motion vectors mvL0, mvL1. Prediction list use flags predFlagL0, predFlagL1 are flags indicating whether or not reference picture lists called L0 list and L1 list are used for inter prediction, respectively, and when the value is 1, the corresponding reference picture list is used for inter prediction. Note that in this specification, when "a flag indicating whether or not XX" is written, a flag other than 0 (for example, 1) is XX, 0 is not XX, and 1 is treated as true and 0 is treated as false in logical negation, logical product, etc. (similarly below). However, in an actual device or method, other values ​​can be used as true and false values.

[0057] Syntax elements for deriving inter prediction parameters include, for example, an affine flag affine_flag, a merge flag merge_flag, a merge index merge_idx, an inter prediction identifier inter_pred_idc, a reference picture index refIdxLX, a prediction vector index mvp_LX_idx, a difference vector mvdLX, and a motion vector precision mode amvr_mode.

[0058] (Reference picture list) The reference picture list is a list of reference pictures stored in the reference picture memory 306. FIG. 6 is a conceptual diagram showing an example of a reference picture and a reference picture list. In FIG. 6(a), a rectangle indicates a picture, an arrow indicates a reference relationship between pictures, the horizontal axis indicates time, I, P, and B in the rectangle indicate an intra picture, a uni-predictive picture, and a bi-predictive picture, respectively, and numbers in the rectangle indicate a decoding order. As shown in the figure, the decoding order of pictures is I0, P1, B2, B3, and B4, and the display order is I0, B3, B2, B4, and P1. FIG. 6(b) shows an example of a reference picture list of a picture B3 (target picture). A reference picture list is a list indicating candidates for a reference picture, and one picture (slice) may have one or more reference picture lists. In the example shown in the figure, the target picture B3 has two reference picture lists, an L0 list RefPicList0 and an L1 list RefPicList1. In each CU, a reference picture index refIdxLX is used to specify which picture in the reference picture list RefPicListX (X=0 or 1) is actually referenced. The diagram shows an example where refIdxL0=2 and refIdxL1=0. Note that LX is a description method used when there is no distinction between L0 prediction and L1 prediction, and hereinafter, parameters for the L0 list and parameters for the L1 list are distinguished by replacing LX with L0 and L1.

[0059] (Merge prediction and AMVP prediction) The prediction parameter decoding (encoding) method includes a merge prediction mode and an AMVP (Advanced Motion Vector Prediction) mode, and the merge flag merge_flag is a flag for identifying these. The merge prediction mode is a mode in which the prediction list usage flag predFlagLX (or inter prediction identifier inter_pred_idc), the reference picture index refIdxLX, and the motion vector mvLX are not included in the encoded data, but are derived from the prediction parameters of the neighboring blocks that have already been processed. The AMVP mode is a mode in which the inter prediction identifier inter_pred_idc, the reference picture index refIdxLX, and the motion vector mvLX are included in the encoded data. The motion vector mvLX is encoded as a prediction vector index mvp_LX_idx that identifies the prediction vector mvpLX, a difference vector mvdLX, and a motion vector precision mode amvr_mode. The merge prediction mode is a mode in which a merge candidate derived from the motion information of the neighboring blocks or the like is selected to obtain a motion vector mvLX (motion vector information). In addition to the merge prediction mode, there may be an affine prediction mode identified by an affine flag affine_flag. As one form of the merge prediction mode, there may be a skip mode identified by a skip flag skip_flag. Note that the skip mode is a mode in which prediction parameters are derived in the same manner as the merge mode, and in which prediction errors (residual images, residual information) are not included in the encoded data. In other words, when the skip flag skip_flag is 1, the target CU includes only the skip flag skip_flag and the syntax related to the merge mode, such as the merge index merge_idx, and the motion vector, residual information, and the like are not included in the encoded data.

[0060] (Motion Vector) The motion vector mvLX indicates the amount of shift between blocks on two different pictures. A prediction vector and a difference vector related to the motion vector mvLX are called a prediction vector mvpLX and a difference vector mvdLX, respectively.

[0061] (Inter prediction identifier inter_pred_idc and prediction list usage flag predFlagLX) The inter prediction identifier inter_pred_idc is a value indicating the type and number of reference pictures, and takes one of the values ​​PRED_L0, PRED_L1, and PRED_BI. PRED_L0 and PRED_L1 indicate uni-prediction using one reference picture managed in the L0 list and L1 list, respectively. PRED_BI indicates bi-prediction BiPred using two reference pictures managed in the L0 list and L1 list.

[0062] The merge index merge_idx is an index indicating which prediction parameter is to be used as the prediction parameter of the current block among prediction parameter candidates (merge candidates) derived from blocks for which processing has been completed.

[0063] The relationship between the inter prediction identifier inter_pred_idc and the prediction list use flags predFlagL0 and predFlagL1 is as follows, and they can be converted into each other.

[0064] inter_pred_idc = (predFlagL1<<1)+predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 (Bi-predictive biPred decision) The flag biPred indicating whether or not the prediction is biPred can be derived based on whether or not two prediction list usage flags are both 1. For example, the flag can be derived by the following formula.

[0065] biPred = (predFlagL0==1 && predFlagL1==1) Alternatively, the flag biPred can be derived based on whether the inter prediction identifier is a value indicating the use of two prediction lists (reference pictures). For example, the flag biPred can be derived by the following formula.

[0066] biPred = (inter_pred_idc==PRED_BI) ? 1 : 0 (Configuration of a video decoding device) The configuration of a video decoding device 31 (FIG. 7) according to this embodiment will be described.

[0067] The video decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generating unit (prediction image generating device) 308, an inverse quantization and inverse transform unit 311, and an adder 312. Note that, in accordance with the video encoding device 11 described below, the video decoding device 31 may also be configured not to include the loop filter 305.

[0068] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), which are not shown, and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively called a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, and PPS, and slice header (slice information) from the encoded data. The CT information decoding unit 3021 decodes the CT from the encoded data. The CU decoding unit 3022 decodes the CU from the encoded data. The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data when a prediction error is included in the TU.

[0069] In cases other than the skip mode (skip_mode==0), the TU decoding unit 3024 decodes the QP update information (quantization correction value) and the quantized prediction error (residual_coding) from the encoded data. More specifically, in cases of skip_mode==0, the TU decoding unit 3024 decodes a flag cu_cbp indicating whether or not a quantized prediction error is included in the current block from the encoded data, and decodes the quantized prediction error when cu_cbp is 1. When cu_cbp does not exist in the encoded data, the TU decoding unit 3024 derives cu_cbp as 0.

[0070] The parameter decoding unit 302 includes an inter prediction parameter decoding unit 303 and an intra prediction parameter decoding unit 304 (not shown). The predicted image generating unit 308 includes an inter prediction image generating unit 309 and an intra prediction image generating unit 310.

[0071] In addition, although an example in which CTU and CU are used as processing units will be described below, the present invention is not limited to this example, and processing may be performed in sub-CU units. Alternatively, CTU and CU may be read as blocks, and sub-CU as sub-blocks, and processing may be performed in block or sub-block units.

[0072] The entropy decoding unit 301 performs entropy decoding on the externally input coded stream Te to decode individual codes (syntax elements). The decoded codes include prediction information for generating a predicted image and prediction errors for generating a difference image.

[0073] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code is, for example, predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0074] (Configuration of inter-prediction parameter decoding unit) The inter prediction parameter decoding unit 303 decodes inter prediction parameters by referring to the prediction parameters stored in the prediction parameter memory 307, based on the code input from the entropy decoding unit 301. In addition, the inter prediction parameter decoding unit 303 outputs the decoded inter prediction parameters to the prediction image generation unit 308, and stores them in the prediction parameter memory 307.

[0075] 8 is a schematic diagram showing the configuration of the inter prediction parameter decoding unit 303 according to this embodiment. The inter prediction parameter decoding unit 303 includes a merge prediction unit 30374, a DMVR unit 30375, a sub-block prediction unit (affine prediction unit) 30372, an MMVD prediction unit 30376, a triangle prediction unit 30377, an AMVP prediction parameter derivation unit 3032, and an adder unit 3038. The merge prediction unit 30374 includes a merge prediction parameter derivation unit 3036. The AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, and the affine prediction unit 30372 are means common to the video encoding device and the video decoding device, and therefore may be collectively referred to as a motion vector derivation unit (motion vector derivation device).

[0076] (Affine prediction part) The affine prediction unit 30372 derives affine prediction parameters for the target block. In this embodiment, the motion vectors (mv0_x, mv0_y) (mv1_x, mv1_y) of two control points (V0, V1) of the target block are derived as the affine prediction parameters. Specifically, the motion vectors of each control point may be derived by predicting from the motion vectors of adjacent blocks of the target block, or the motion vectors of each control point may be derived by adding the predicted vector derived as the motion vector of the control point and a difference vector derived from the encoded data.

[0077] (Merge prediction) 9(a) is a schematic diagram showing a configuration of merge prediction parameter derivation unit 3036 included in merge prediction unit 30374. Merge prediction parameter derivation unit 3036 includes a merge candidate derivation unit 30361 and a merge candidate selection unit 30362. Note that a merge candidate includes a prediction list use flag predFlagLX, a motion vector mvLX, and a reference picture index refIdxLX, and is stored in a merge candidate list. An index is assigned to the merge candidates stored in the merge candidate list according to a predetermined rule.

[0078] The merging candidate derivation unit 30361 derives merging candidates by directly using the motion vectors and reference picture indexes refIdxLX of the decoded adjacent blocks.

[0079] The order in which the merge candidates are stored in the mergeCandList[] is, for example, spatial merge candidates A1, B1, B0, A0, B2, temporal merge candidates Col, pairwise merge candidates avgK, and zero merge candidates ZK. Note that reference blocks that are unavailable (e.g., blocks that are intra-predicted) are not stored in the merge candidate list.

[0080] The merge candidate selection unit 30362 selects a merge candidate N indicated by the merge index merge_idx from among the merge candidates included in the merge candidate list, using the following formula.

[0081] N = mergeCandList[merge_idx] Here, N is a label indicating a merging candidate, and can be A1, B1, B0, A0, B2, Col, avgK, ZK, etc. The motion information of the merging candidate indicated by label N is indicated by (mvLXN[0], mvLXN[1]) and is represented by predFlagLXN, refIdxLXN.

[0082] The merging candidate selection unit 30362 selects the motion information (mvLXN[0], mvLXN[1]), predFlagLXN, and refIdxLXN of the selected merging candidate as inter prediction parameters of the current block. The merging candidate selection unit 30362 stores the inter prediction parameters of the selected merging candidate in the prediction parameter memory 307, and also outputs them to the predicted image generation unit 308.

[0083] (AMVP Prediction) 9B is a schematic diagram showing the configuration of the AMVP prediction parameter derivation unit 3032 according to this embodiment. The AMVP prediction parameter derivation unit 3032 includes a vector candidate derivation unit 3033 and a vector candidate selection unit 3034. The vector candidate derivation unit 3033 derives a prediction vector candidate from the motion vector mvLX of the decoded adjacent block stored in the prediction parameter memory 307 based on the reference picture index refIdxLX, and stores the candidate in a prediction vector candidate list mvpListLX[].

[0084] The vector candidate selection unit 3034 selects the motion vector mvpListLX[mvp_LX_idx] indicated by the prediction vector index mvp_LX_idx from among the prediction vector candidates in the prediction vector candidate list mvpListLX[] as a prediction vector mvpLX. The vector candidate selection unit 3034 outputs the selected prediction vector mvpLX to the addition unit 3038.

[0085] The prediction vector candidates are derived by scaling the motion vectors of the decoded adjacent blocks in a predetermined range from the target block. The adjacent blocks include blocks spatially adjacent to the target block, such as the left block and the upper block, as well as regions temporally adjacent to the target block, such as regions that include the same position as the target block and are obtained from prediction parameters of blocks with different display times.

[0086] The adder 3038 adds the predicted vector mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded differential vector mvdLX to calculate the motion vector mvLX. The adder 3038 outputs the calculated motion vector mvLX to the prediction image generation unit 308 and the prediction parameter memory 307.

[0087] mvLX[0] = mvpLX[0]+mvdLX[0] mvLX[1] = mvpLX[1]+mvdLX[1] The motion vector accuracy mode amvr_mode is a syntax for switching the accuracy of the motion vector derived in the AMVP mode. For example, at amvr_mode = 0, 1, 2, the accuracy is switched between 1 / 4 pixel, 1 pixel, and 4 pixels.

[0088] When the accuracy of the motion vector is 1 / 16 accuracy (MVPREC = 16), in order to change the motion vector differences of 1 / 4, 1, and 4 pixel accuracies to the motion vector differences of 1 / 16 pixel accuracy, the parameter decoding unit 302 may perform inverse quantization using MvShift (= 1 << amvr_mode) derived from amvr_mode as follows.

[0089] mvdLX[0] = mvdLX[0] << (MvShift + 2) mvdLX[1] = mvdLX[1] << (MvShift + 2) Note that the parameter decoding unit 302 may also derive mvdLX[] before shifting by the above MvShift by decoding the following syntax. ·abs_mvd_greater0_flag ·abs_mvd_minus2 ·mvd_sign_flag Then, the parameter decoding unit 302 decodes the differential vector lMvd[] from the syntax by using the following formula.

[0090] lMvd[compIdx] = abs_mvd_greater0_flag[compIdx] * (abs_mvd_minus2[compIdx]+2) * (1-2*mvd_sign_flag[compIdx]) Furthermore, the parameter decoding unit 302 sets the decoded difference vector lMvd[ ] to MvdLX in the case of translational MVD (MotionModelIdc[x][y] == 0), and to MvdCpLX in the case of control point MVD (MotionModelIdc[x][y] != 0).

[0091] if (MotionModelIdc[x][y] == 0) mvdLX[x0][y0][compIdx] = lMvd[compIdx] else mvdCpLX[x0][y0][compIdx] = lMvd[compIdx]<<2 (Motion Vector Scaling) A method for deriving motion vector scaling will be described. Given a motion vector Mv (reference motion vector), a picture PicMv including a block having Mv, a reference picture PicMvRef for Mv, a motion vector after scaling sMv, a picture CurPic including a block having sMv, and a reference picture CurPicRef referenced by sMv, the derivation function MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef) of sMv is expressed by the following formula.

[0092] sMv = MvScale(Mv,PicMv,PicMvRef,CurPic,CurPicRef) = Clip3(-R1,R1-1,sign(distScaleFactor*Mv)*((abs(distScaleFactor*Mv)+round1-1)>>shift1)) distScaleFactor = Clip3(-R2,R2-1,(tb*tx+round2)>>shift2) tx = (16384+abs(td)>>1) / td td = DiffPicOrderCnt(PicMv,PicMvRef) tb = DiffPicOrderCnt(CurPic,CurPicRef) Here, round1, round2, shift1, and shift2 are round and shift values ​​for division using reciprocals, for example, round1=1<<(shift1-1), round2=1<<(shift2-1), shift1=8, shift2=6, etc. DiffPicOrderCnt(Pic1,Pic2) is a function that returns the difference in time information (e.g., POC) between Pic1 and Pic2. R1 and R2 limit the value range to perform processing with limited precision, for example, R1=32768, R2=4096, etc.

[0093] Furthermore, the scaling function MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef) may be expressed by the following formula.

[0094] MvScale(Mv,PicMv,PicMvRef,CurPic,CurPicRef) = Mv*DiffPicOrderCnt(CurPic,CurPicRef) / DiffPicOrderCnt(PicMv,PicMvRef) In other words, Mv may be scaled according to the ratio of the difference in time information between CurPic and CurPicRef to the difference in time information between PicMv and PicMvRef.

[0095] (DMVR Department 30375) Next, a description will be given of a DMVR (Decoder side Motion Vector Refinement) process performed by the DMVR unit 30375. The DMVR process is a process of correcting motion vectors mvL0 and mvL1 using two reference pictures.

[0096] Fig. 10 is a schematic diagram showing the configuration of the DMVR unit 30375. The specific contents of the processing performed by the DMVR unit 30375 will be described with reference to Fig. 10. The DMVR unit 30375 includes a predicted image generation unit 303751 for corrected motion vector search, an initial error generation unit 303752, a motion vector search unit 303753, and a corrected vector derivation unit 303754.

[0097] The DMVR unit 30375 receives the Top left position of the target block (xCb, yCb) -Target block width bW - Height of target block bH - Motion vectors mvL0 and mvL1 with 1 / 16 pixel accuracy Reference pictures refPicL0L and refPicL1L The motion vector change amounts dmvL0 and dmvL1 for correcting mvL0 and mvL1 are derived by referring to the above and output to the inter predicted image generation unit 309.

[0098] First, the corrected motion vector search predicted image generating unit 303751 Top left position of the target subblock (xSb, ySb) -Width of the target sub-block of luminance sbW - Height of the target luminance subblock sbH Motion vector mvLX(X=0, 1) Reference picture refPicLXL(X=0, 1) A predicted image predSamplesLXL having a size of (sbW)*(sbH) is derived by referring to

[0099] The corrected motion vector search predicted image generation unit 303751 derives a motion vector MvLsX (X=0, 1) by the following equation.

[0100] MvLsX[0] = MvLX[0] - 32 MvLsX[1] = MvLX[1] - 32 In addition, the DMVR unit 30375 sets the values ​​of the variables srRange, offsetH[0], offsetV[0], offsetH[1], and offsetV[1] to 2, respectively.

[0101] Let (xIntL, yIntL) be the integer pixel position of the reference block corresponding to pixel position (xL, yL) in the current block. Also, let (xFracL, yFracL) be the offset in 1 / 16 pixel units from (xIntL, yIntL). These coordinates are derived from the integer component (mvLX[0]>>4, mvLX[1]>>4) and fractional component (mvLX[0]&15, mvLX[1]&15) of the motion vector (mvLX[0], mvLX[1]), and indicate the pixel position with fractional precision in the reference picture refPicLXL. For a pixel whose position in predSamplesLXL is (xL, yL) (xL=0, ..., sbW-1, yL=0, ..., sbH-1), the DMVR unit 30375 derives xIntL, yIntL, xFracL, and yFracL according to the following equations.

[0102] xIntL = xSb + (mvLX[0]>>4) + xL yIntL = ySb + (mvLX[1]>>4) + yL xFracL = mvLX[0]&15 yFracL = mvLX[1]&15 Next, DMVR section 30375, ·(xIntL, yIntL) (xFracL, yFracL) ·refPicLXL Derive predSamplesLXL by referring to:

[0103] First, the modified motion vector search predicted image generation unit 303751 derives variables shift1, shift2, shift3, and shift4 according to the following equations.

[0104] shift1 = BitDepthY - 6 offset1 = 1 << (shift1 - 1) shift2 = 4 offset2 = 8 shift3 = 10-BitDepthY offset3 = 1 << (shift3 - 1) shift4 = BitDepthY - 10 In the above formula, BitDepthY is the number of pixel bits.

[0105] Next, the predicted image generation unit 303751 for searching modified motion vector sets picW equal to the value of the picture width pic_width_in_luma_samples. The predicted image generation unit 303751 for searching modified motion vector also sets picH equal to the value of the picture height pic_height_in_luma_samples.

[0106] After that, the corrected motion vector search predicted image generating unit 303751 derives predSamplesLXL as follows. L [p] indicates the filter coefficient for deriving pixel values ​​with 1 / 16 pixel accuracy. fb L The value of [p] depends on the position p (p=1, 2, ..., 15) with 1 / 16 pixel accuracy. Position p is equal to xFracL or yFracL. Increasing the value of p causes fb L [p][0] decreases monotonically, and fb L The value of [p][1] increases monotonically.

[0107] First, the modified motion vector search predicted image generation unit 303751 determines whether or not xFracL and yFracL are each 0. If both xFracL and yFracL are 0, the DMVR unit 30375 derives predSamplesLXL by one of the following equations according to the value of BitDepthY.

[0108] predSamplesLXL = (BitDepthY <= 10)? (refPicLXL[xIntL][yIntL] << shift3) : ((refPicLXL[xIntL][yIntL]+offset3) >> shift4) When xFracL is not 0 and yFracL is 0, the modified motion vector search predicted image generation unit 303751 derives predSamplesLXL by the following formula.

[0109] predSamplesLXL = (fb L [xFracL][0] * refPicLXL[Clip3(0,picW-1,xIntL)][yIntL] + fb L [xFracL][1] * refPicLXL[Clip3(0,picW-1,xIntL+1)][yIntL] + offset1)>>shift1 When xFracL is 0 and yFracL is not 0, the DMVR unit 30375 derives predSamplesLXL by the following formula.

[0110] predSamplesLXL = (fb L [yFracL][0] * refPicLXL[xIntL][Clip3(0,picH-1,yIntL)] + fb L [yFracL][1] * refPicLXL[xIntL][Clip3(0,picH-1,yIntL+1)]+offset1)>>shift1 When neither xFracL nor yFracL is 0, the corrected motion vector search predicted image generation unit 303751 derives predSamplesLXL as follows. First, the DMVR unit 30375 derives temp[n] by the following formula. The derivation process of temp[] is performed n times by changing the reference position. n=0 indicates the first derivation process, and n=1 indicates the second derivation process.

[0111] yPosL = Clip3(0, PicH-1, yIntL+n-3) temp[n] = (fb L [xFracL][0] * refPicLXL[Clip3(0,picW-1,xIntL)][yPosL] + fb L [xFracL][1] * refPicLXL[Clip3(0,picW-1,xIntL+1)][yPosL]+offset1)>>shift1 Then, the DMVR unit 30375 derives predSamplesLXL by the following formula.

[0112] predSamplesLXL = (fb L [yFracL][0] * temp[0] + fb L [yFracL][1] * temp[1])>>shift2 Next, the initial error generator 303752 Target block width nCbW - Height of target block nCbH Two predicted images predSampleL1 and predSampleL2 with size (nCbW+4)x(nCbH+4) Variables offsetH[0], offsetH[1], offsetV[0], and offsetV[1] With reference to the above, a list Sad1 of the sum of absolute differences between pixel values ​​contained in predSampleL1 and predSampleL2 and a variable centerSad are derived.

[0113] The DMVR unit 30375 sets the value of each element of the 2×9 array bC according to the following formula.

[0114] bC[0][0] = -1 bC[1][0] = -1 bC[0][1] = -1 bC[1][1] = 0 bC[0][2] = -1 bC[1][2] = 1 bC[0][3] = 0 bC[1][3] = -1 bC[0][4] = 0 bC[1][4] = 0 bC[0][5] = 0 bC[1][5] = 1 bC[0][6] = 1 bC[1][6] = -1 bC[0][7] = 1 bC[1][7] = 0 bC[0][8] = 1 bC[1][8] = 1 The initial error generator 303752 derives the elements sadList[i] (i=0, . . . , 8) of Sad1 using the following formula.

[0115]

number

[0116] Furthermore, the initial error generator 303752 derives centerSad using the following formula.

[0117]

number

[0118] The initial error generation unit 303752 determines whether centerSad is equal to or greater than (bH>>1)*(bW)*4. dmvrFlag is a flag indicating that DMVR processing is performed when TRUE and that DMVR processing is not performed when FALSE. If centerSad is smaller than (bH>>1)*(bW)*4, the error is small, so the initial error generation unit 303752 determines that DMVR processing does not need to be performed, sets dmvrFlag to FALSE, and proceeds to the inter-prediction image generation unit 309 without correcting the motion vector.

[0119] If centerSad is equal to or greater than (bH>>1)*(bW)*4, the initial error generator 303752 sets dmvrFlag to TRUE, and the motion vector search unit 303753 ·Number of search points n · Element sadList of the list of absolute difference sums of search points Sad1 derives an index bestIdx by referring to n, where n is a positive integer.

[0120] The following describes the case where n = 9. Note that the value of the number of search points n may be other than 9, and other than the method described in this embodiment, for example, the minimum value of the sadList value may be simply selected when n = 25.

[0121] The motion vector search unit 303753 determines whether sadList[1]<sadList[7] and whether sadList[3]<sadList[5].

[0122] If sadList[1] < sadList[7] and sadList[3] < sadList[5], the DMVR unit 30375 sets the value of idx to 0. Then, the motion vector search unit 303753 determines whether or not sadList[1] < sadList[3]. The motion vector search unit 303753 sets the value of bestIdx to 1 if sadList[1] < sadList[3], and sets it to 3 if sadList[1] < sadList[3] is not satisfied.

[0123] Otherwise, if sadList[1] >= sadList[7] and sadList[3] < sadList[5], the motion vector search unit 303753 sets the value of idx to 6. Then, the DMVR unit 30375 determines whether sadList[7] < sadList[3]. The motion vector search unit 303753 sets the value of bestIdx to 7 if sadList[7] < sadList[3], and sets it to 3 if sadList[7] < sadList[3] is not satisfied.

[0124] Otherwise, if sadList[1] < sadList[7] and sadList[3] >= sadList[5], the motion vector search unit 303753 sets the value of idx to 2. Then, the DMVR unit 30375 determines whether or not sadList[1] < sadList[5]. The DMVR unit 30375 sets the value of bestIdx to 1 if sadList[1] < sadList[5], and sets it to 5 if sadList[1] < sadList[5] is not satisfied.

[0125] Otherwise, if sadList[1] >= sadList[7] and sadList[3] >= sadList[5], the motion vector search unit 303753 sets the value of idx to 8. Then, the DMVR unit 30375 determines whether or not sadList[7] < sadList[5]. The DMVR unit 30375 sets the value of bestIdx to 7 if sadList[7] < sadList[5] is true, and sets it to 5 if sadList[7] < sadList[5] is not true.

[0126] Furthermore, the motion vector search unit 303753 determines whether or not sadList[4]<=sadList[bestIdx]. If sadList[4]<=sadList[bestIdx], the DMVR unit 30375 updates the value of bestIdx to 4. On the other hand, if sadList[4]<=sadList[bestIdx] is not true, the DMVR unit 30375 does not update the value of bestIdx.

[0127] Furthermore, the motion vector search unit 303753 determines whether sadList[idx]<sadList[bestIdx]. If sadList[idx]<sadList[bestIdx], the DMVR unit 30375 sets bestIdx to idx. On the other hand, if sadList[idx]<sadList[bestIdx] is not true, the motion vector search unit 303753 does not update the value of bestIdx.

[0128] The motion vector search unit 303753 determines whether the value of bestIdx is 4. If the value of bestIdx is 4, the motion vector search unit 303753 sets halfPelAppliedflag to true.

[0129] If the value of bestIdx is not 4, the motion vector search unit 303753 calculates the values ​​of the variables dmvx and dmvy according to the following formula.

[0130] dmvx = (bestIdx / 3 - 1) dmvy = (bestIdx%3 - 1) Furthermore, the motion vector search unit 303753 updates offsetH and offsetV according to the following formula.

[0131] offsetH[0] = offsetH[0] + dmvx, offsetV[0] = offsetV[0] + dmvy offsetH[1] = offsetH[1] - dmvx, offsetV[1] = offsetV[1] - dmvy The motion vector search unit 303753 derives Sad2 by the same process as the process for deriving Sad1 described above, using the updated offsetH and offsetV. Furthermore, the motion vector search unit 303753 re-derives bestIdx by using Sad2 instead of Sad1.

[0132] The motion vector search unit 303753 determines whether the re-derived value of bestIdx is 4. If the value of bestIdx is 4, the motion vector search unit 303753 sets halfPelAppliedflag to true.

[0133] If the value of bestIdx is not 4, the motion vector search unit 303753 calculates dmvx and dmvy using the following formula.

[0134] dmvx = (bestIdx / 3 - 1), dmvy = (bestIdx%3 - 1) Furthermore, the DMVR unit 30375 calculates dmvL0 and dmvL1 using the following formula.

[0135] dmvL0[0] = 16*dmvx, dmvL0[1] = 16*dmvy dmvL1[0] = -16*dmvx, dmvL1[1] = -16*dmvy When halfPelAppliedflag is true, the motion vector search unit 303753 derives modified dmvL0 and dmvL1 as follows: Note that the following sadList is an element of Sad2 if Sad2 exists, and is an element of Sad1 if Sad2 does not exist.

[0136] First, the motion vector search unit 303753 determines whether sadList[1] + sadList[7] == sadList[4]. If sadList[1] + sadList[7] == sadList[4] and if mrSadT + mrSadB - (mrSadC << 1) == 0, the motion vector search unit 303753 sets dmv[0] = 0. If sadList[1] + sadList[7] == sadList[4] is not true, the motion vector search unit 303753 calculates dmv[0] using the following formula.

[0137] dmv[0] = ((sadList[1] - sadList[7])<<3) / (sadList[1] + sadList[7] - (sadList[4]<<1)) Next, the correction vector derivation unit 303754 determines whether sadList[3] + sadList[5] == sadList[4]. If sadList[3] + sadList[5] == sadList[4] and if mrSadL + mrSadR - (mrSadC << 1) == 0, the correction vector derivation unit 303754 sets dmv[1] = 0. If sadList[3] + sadList[5] == sadList[4] is not true, the correction vector derivation unit 303754 calculates dmv[1] using the following formula.

[0138] dmv[1] = ((sadList[3] - sadList[5])<<3) / (sadList[3] + sadList[5] - (sadList[4]<<1)) Furthermore, the modified vector derivation unit 303754 modifies the motion vectors mvL0 and mvL1 according to the following equation.

[0139] dmvL0[0] = dmvL0[0] + dmv[0] dmvL0[1] = dmvL0[1] + dmv[1] dmvL1[0] = dmvL1[0] - dmv[0] dmvL1[1] = dmvL1[1] - dmv[1] The DMVR unit 30375 calculates a motion vector mvLX by adding the derived difference vector dmvLX to the predicted vector mvpLX input from the merge prediction unit 30374. The DMVR unit 30375 outputs mvLX to the inter predicted image generation unit 309.

[0140] mvLX[0] = mvpLX[0] + dmvLX[0] mvLX[1] = mvpLX[1] + dmvLX[1] Note that the values ​​of dmvLX[0] and dmvLX[1] are limited to the range from -8 to 8, regardless of the number of bits in sadList.

[0141] (DMVR criteria) dmvrFlag is a flag which indicates that DMVR processing is performed when the value is TRUE, and indicates that DMVR processing is not performed when the value is FALSE.

[0142] If a flag in the SPS indicating that DMVR processing is possible is On, the initial error generator 303752 sets dmvrFlag to TRUE. Otherwise, the initial error generator 303752 sets dmvrFlag to FALSE.

[0143] Also, if the merge_flag of the block is TRUE, the initial error generator 303752 may set dmvrFlag to TRUE, otherwise the initial error generator 303752 sets dmvrFlag to FALSE.

[0144] Also, when predFlagL0 and predFlagL1 are both TRUE, that is, when bidirectional prediction is performed, the initial error generator 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generator 303752 sets dmvrFlag to FALSE.

[0145] If the mmvd_flag of the block is FALSE and the mode is not MMVD, the initial error generator 303752 may set dmvrFlag to TRUE. Otherwise, if the mode is MMVD, the initial error generator 303752 sets dmvrFlag to FALSE.

[0146] If DiffPicOrderCnt(currPic, RefPicList

[0000] [refIdxL0]) is equal to DiffPicOrderCnt(RefPicList

[0001] [refIdxL1], currPic), that is, if the current picture currPic is in a positional relationship such that the L0 reference picture RefPicList

[0000] [refIdxL0] and the L1 reference picture RefPicList

[0001] [refIdxL1] are interpolated at an equal distance, the initial error generator 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generator 303752 sets dmvrFlag to FALSE. Here, DiffPicOrderCnt() is a function that derives the difference between the POC (Picture Order Count: picture display order) of two images as follows:

[0147] DiffPicOrderCnt(picA,picB) = PicOrderCnt(picA)-PicOrderCnt(picB) Note that if DiffPicOrderCnt(currPic, RefPicList

[0000] [refIdxL0]) * DiffPicOrderCnt(currPic, RefPicList

[0001] [refIdxL1]) < 0, and the positional relationship is such that interpolation is simply performed, the initial error generation unit 303752 may set dmvrFlag to TRUE, and otherwise set dmvrFlag to FALSE.

[0148] Furthermore, if the size of the processing block is equal to or smaller than a specific value, the initial error generator 303752 may set dmvrFlag to FALSE. For example, if bH is equal to or greater than 8 and bH*bW is 64, the initial error generator 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generator 303752 sets dmvrFlag to FALSE.

[0149] Fig. 11 is a flowchart showing the flow of processing in the DMVR unit 30375. In this embodiment, in addition to the above-mentioned criteria, as shown in Fig. 11, a condition is added such that the DMVR processing is applied only when the GBI processing described later is not applied.

[0150] Specifically, first, the DMVR unit 30375 executes the above-described dmvrFlag determination process (S1101). Next, the DMVR unit 30375 determines whether gbiIdx is 0 or not (S1102). When gbiIdex is a non-zero value, as described below, non-uniform weighted prediction is performed based on the table gbiWLut. When gbiIdx is 0, in addition to the condition that dmvrFlag is TRUE, when gbiIdx is non-zero, dmvrFlag is set to FALSE (S1103).

[0151] Furthermore, the DMVR unit 30375 judges whether or not dmvrFlag is TRUE (S1104), and if it is TRUE, executes DMVR processing (S1105), and if it is FALSE, does not execute it.

[0152] When applying GBI prediction, a weighted prediction is applied, so by taking into account that the error may not be evaluated correctly and limiting the application conditions, the overall amount of processing can be reduced.

[0153] Similarly, in weighted prediction described later, if either L0 prediction or L1 prediction to which DMVR processing is applied performs weighted prediction, dmvrFlag is set to FALSE. Specifically, if luma_weight_l0_flag[refIdxL0] indicating whether or not a luminance weighting factor w0 and an offset o0 exist in the L0 predicted picture and luma_weight_l1_flag[refIdxL1] indicating whether or not a luminance weighting factor w1 and an offset o1 exist in the L1 predicted picture are both FALSE, dmvrFlag is added to the TRUE conditions, and otherwise dmvrFlag is set to FALSE.

[0154] (BDOF determination by error threshold processing in DMVR) In the DMVR, a process for calculating an error between an L0 predicted image and an L1 predicted image is performed, and based on the value of the error at this time, a decision is made in advance as to whether or not to execute BDOF processing, which will be described later and is performed at a subsequent stage.

[0155] FIG. 12 is a flowchart for explaining the process of determining the BDOF by error threshold processing in the DMVR.

[0156] First, the initial error generation unit 303752 sets bdofFlag to TRUE in advance (S1201). Next, the initial error generation unit 303752 derives centerSad (S1202) and judges whether the value of centerSad is equal to or greater than the threshold value (bH>>1)*bW*4 (S1203). If the value of centerSad is smaller than the threshold value, the initial error generation unit 303752 judges that the error is small and sets bdofFlag, which indicates whether or not to perform BDOF processing, to FALSE (S1204), so as not to perform BDOF processing in advance. Since this judgment is the same as that of the initial error generation unit 30752 described above, the motion vector search unit 303753 and correction vector derivation unit 303754 are skipped and DMVR processing is not performed. If the value of centerSad is equal to or greater than the threshold, the motion vector search unit 303753 performs a modified motion vector search (S1205), and as a result, the modified vector derivation unit 303754 determines whether the value of sadList[bestIdx], which is the smallest SAD value of bestIdx, is smaller than the threshold value (bH>>1)*bW*8 (S1206). If the value of sadList[bestIdx] is smaller than the threshold, the modified vector derivation unit 303754 determines that the error is small and sets bdofFlag, which indicates whether or not to perform BDOF processing, to FALSE (S1207), thereby preventing BDOF processing from being performed in advance.

[0157] The threshold value in (S1206) is set to be equal to or greater than the threshold value in (S1203).

[0158] In DMVR processing, in order to search for a corrected motion vector, a process is required to calculate the error between the L0 predicted image and the L1 predicted image. Meanwhile, in BDOF processing, if the error is small, it is ineffective. Therefore, by adding such a process, it is possible to determine whether or not to perform BDOF without adding additional error calculations.

[0159] A BDOF unit described later determines whether bdofFlag is TRUE (S1208), and if Yes, the inter prediction parameter decoding unit 303 performs BDOF processing (S1209). If No, it is determined that BDOF processing is not to be performed on the block.

[0160] (Triangle forecast) Next, the triangle prediction will be described. In the triangle prediction, the target CU is divided into two triangular prediction units with the diagonal or anti-diagonal as the boundary. The predicted image in each triangular prediction unit is derived by applying a weighting mask process according to the pixel position to each pixel of the predicted image of the target CU (rectangular block including the triangular prediction unit). For example, a triangular image can be derived from a rectangular image by multiplying a mask in which the pixels of a triangular area in a rectangular area are 1 and the pixels of areas other than the triangle are 0. The adaptive weighting process of the predicted image is applied to both areas on either side of the diagonal, and one predicted image of the target CU (rectangular block) is derived by the adaptive weighting process using the two predicted images. This process is called a triangle synthesis process. The transform (inverse transform) and quantization (inverse quantization) processes are applied to the entire target CU. Note that the triangle prediction is applied only in the case of the merge prediction mode or the skip mode.

[0161] The triangle prediction unit 30377 derives prediction parameters corresponding to two triangular regions used in triangle prediction, and outputs them to the inter-prediction image generation unit 309. In order to simplify the process, the triangle prediction may be configured not to use bi-prediction. In this case, the inter-prediction parameters of unidirectional prediction are derived in one triangular region. Note that the derivation of the two predicted images and the synthesis using the predicted images are performed by the motion compensation unit 3091 and the triangle synthesis unit 30952.

[0162] (MMVD Prediction Section 30376) The MMVD prediction unit 30376 performs processing in MMVD (Merge with Motion Vector Difference) mode. The MMVD mode is a mode in which a motion vector is obtained by adding a difference vector of a predetermined distance and a predetermined direction to a motion vector derived from a merge candidate (a motion vector derived from a motion vector of an adjacent block, etc.). In the MMVD mode, the MMVD prediction unit 30376 uses the merge candidate and limits the value range of the difference vector to a predetermined distance (e.g., 8 ways) and a predetermined direction (e.g., 4 directions, 8 directions, etc.), thereby efficiently deriving a motion vector.

[0163] The loop filter 305 is a filter provided in the encoding loop, which removes block distortion and ringing distortion to improve image quality. The loop filter 305 applies a filter such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to the decoded image of the CU generated by the adder 312.

[0164] The reference picture memory 306 stores the decoded image of the CU generated by the adder 312 in a location that is determined in advance for each current picture and current CU.

[0165] The prediction parameter memory 307 stores prediction parameters at a predetermined position for each CTU or CU to be decoded. Specifically, the prediction parameter memory 307 stores the parameters decoded by the parameter decoding unit 302 and the prediction mode predMode decoded by the entropy decoding unit 301.

[0166] The prediction image generating unit 308 receives a prediction mode predMode, prediction parameters, and the like. The prediction image generating unit 308 also reads a reference picture from the reference picture memory 306. The prediction image generating unit 308 generates a prediction image of a block or sub-block using the prediction parameters and the read reference picture (reference picture block) in the prediction mode indicated by the prediction mode predMode. Here, the reference picture block is a set of pixels on the reference picture (usually rectangular, so called a block), and is an area to be referenced for generating a prediction image.

[0167] (Inter-prediction image generation unit 309) When the prediction mode predMode indicates an inter prediction mode, the inter prediction image generation unit 309 generates a prediction image of a block or sub-block by inter prediction using the inter prediction parameters input from the inter prediction parameter decoding unit 303 and the read reference picture.

[0168] 13 is a schematic diagram showing the configuration of an inter-prediction image generation unit 309 included in the prediction image generation unit 308 according to this embodiment. The inter-prediction image generation unit 309 includes a motion compensation unit (prediction image generation device) 3091 and a synthesis unit 3095.

[0169] (Motion Compensation) The motion compensation unit 3091 (interpolated image generating unit) generates an interpolated image (motion compensated image) by reading a block at a position shifted by the motion vector mvLX from the position of the target block in the reference picture RefPicLX specified by the reference picture index refIdxLX from the reference picture memory 306 based on the inter prediction parameters (prediction list usage flag predFlagLX, reference picture index refIdxLX, motion vector mvLX) input from the inter prediction parameter decoding unit 303. Here, if the accuracy of the motion vector mvLX is not integer accuracy, a filter for generating pixels at decimal positions called a motion compensation filter is applied to generate the interpolated image.

[0170] The motion compensation unit 3091 first derives an integer position (xInt, yInt) and a phase (xFrac, yFrac) corresponding to coordinates (x, y) in the prediction block using the following formula.

[0171] xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x xFrac = mvLX[0]&(MVPREC-1) yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y yFrac = mvLX[1]&(MVPREC-1) Here, (xPb, yPb) is the upper left coordinate of a block of size bW*bH, where x=0...bW-1, y=0...bH-1, and MVPREC indicates the accuracy of the motion vector mvLX (1 / MVPREC pixel accuracy). For example, MVPREC=16.

[0172] The motion compensation unit 3091 derives a temporary image temp[][] by performing horizontal interpolation processing on the reference picture refImg using an interpolation filter. In the following, Σ is the sum for k=0..NTAP-1, shift1 is a normalization parameter that adjusts the value range, and offset1=1<<(shift1-1).

[0173] temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1 Next, the motion compensation unit 3091 derives the interpolated image Pred[][] by vertically interpolating the temporary image temp[][]. In the following, Σ is the sum for k=0..NTAP-1, shift2 is a normalization parameter that adjusts the value range, and offset2=1<<(shift2-1).

[0174] Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2 (Synthetic section) The synthesis unit 3095 generates a predicted image by referring to the interpolated image input from the motion compensation unit 3091, the inter prediction parameters input from the inter prediction parameter decoding unit 303, and the intra image input from the intra prediction image generation unit 310, and outputs the generated predicted image to the addition unit 312.

[0175] The synthesis unit 3095 includes a combined intra / inter synthesis unit 30951, a triangle synthesis unit 30952, an OBMC unit 30953, and a BDOF unit 30956.

[0176] (Combined intra / inter synthesis processing) The combined intra / inter synthesis unit 30951 generates a predicted image by combining unidirectional prediction, skip mode, merge mode, and intra prediction in AMVP.

[0177] (Triangle synthesis processing) The triangle synthesis unit 30952 generates a predicted image using the above-mentioned triangle prediction.

[0178] (OBMC processing) The OBMC unit 30953 generates a predicted image using OBMC (Overlapped Block Motion Compensation) processing. The OBMC processing includes the following processing. An interpolated image (motion-compensated image) of the target subblock is generated using an interpolated image (PU interpolated image) generated using the inter-prediction parameters added to the target subblock, and an interpolated image (OBMC interpolated image) generated using the motion parameters of the neighboring subblocks of the target subblock. A predicted image is generated by taking a weighted average of the OBMC interpolated image and the PU interpolated image.

[0179] (Weighted Prediction Unit 30954) The weighted prediction unit 309454 generates a predicted image of the block by multiplying the motion compensation images PredL0 and PredL1 by a weighting coefficient. When one of the prediction list usage flags (predFlagL0 or predFlagL1) is 1 (uniprediction) and weighted prediction is not used, the following equation is processed to match the motion compensation image PredLX (LX is L0 or L1) to the pixel bit depth bitDepth.

[0180] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredLX[x][y]+offset1)> >shift1) Here, shift1=Max(2, 14-bitDepth) and offset1=1<<(shift1-1).

[0181] (Bidirectional prediction processing) Furthermore, when both prediction list usage flags (predFlagL0 and predFlagL1) are 1 (bi-prediction BiPred) and weighted prediction is not used, the following equation is processed to average the motion compensation images PredL0 and PredL1 and match them to the pixel bit rate.

[0182] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredL0[x][y]+PredL1[x][y]+offset2)> >shift2) Here, shift2=Max(3,15-bitDepth), offset2=1<<(shift2-1). Hereinafter, this process will be referred to as normal bidirectional prediction.

[0183] Furthermore, in the case of uni-prediction, and a flag indicating whether or not the weighted prediction coefficient w0 and offset o0 are present in the L0 reference picture (luma_weight_l0_flag for luminance, chroma_weight_l0_flag for chrominance) is on, the weighted prediction unit 30954 derives the weighted prediction coefficient w0 and offset o0 from the encoded data in the case of L0 prediction, and performs processing according to the following equation.

[0184] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,((PredL0[x][y]*w0+(1<<(log2WD-1)))> >log2WD)+o0) In the case of L1 prediction, if a flag indicating whether or not the weighted prediction coefficient w1 and offset o1 exist in the L1 reference picture (luma_weight_l1_flag for luminance, chroma_weight_l1_flag for chrominance) is on, the weighted prediction unit 30954 derives the weighted prediction coefficient w1 and offset o1 from the encoded data and performs processing according to the following equation.

[0185] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,((PredL1[x][y]*w1+(1<<(log2WD-1)))> >log2WD)+o1) Here, log2WD is a variable that explicitly sums the values ​​of Log2WeightDenom+shift1 sent in the slice header for luma and chroma discrimination.

[0186] (Weighted Bidirectional Prediction) Furthermore, in the case of bi-predictive BiPred, and flags indicating whether or not weighted prediction coefficients and offsets exist (luma_weight_l0_flag, luma_weight_l1_flag for luminance, chroma_weight_l0_flag, chroma_weight_l1_flag for chrominance) perform weighted prediction, the weighted prediction unit 30954 derives weighted prediction coefficients w0, w1, o0, o1 from the encoded data and performs processing according to the following equations.

[0187] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,(PredL0[x][y]*w0+PredL1[x][y]*w1+((o0+o1+1)<<log2WD))> >(log2WD+1)) (GBI Part 30955) In the above-mentioned weighted prediction, an example in which a predicted image is generated by multiplying an interpolated image by a weighting factor has been described. Here, another example in which a predicted image is generated by multiplying an interpolated image by a weighting factor will be described. In detail, a process of generating a predicted image using generalized bi-prediction (hereinafter, referred to as GBI prediction) will be described. In GBI prediction, a predicted image Pred is generated by multiplying an L0 predicted image PredL0 and an L1 predicted image PredL1 in bi-prediction by weighting factors (w0, w1).

[0188] Furthermore, when generating a predicted image using GBI prediction, the GBI unit 30955 switches the weighting coefficients (w0, w1) for each coding unit. That is, the GBI unit 30954 of the inter-predicted image generation unit 309 sets a weighting coefficient for each coding unit. In GBI prediction, multiple weighting coefficient candidates are predefined, and gbiIdx is an index indicating a weighting coefficient to be used for the current block among multiple weighting coefficient candidates included in the table gbiWLut.

[0189] The GBI unit 30955 checks the flag gbiAppliedFlag indicating whether or not GBI prediction is used. If it is FALSE, the motion compensation unit 3091 generates a predicted image using the following formula.

[0190] Pred[x][y]=Clip3(0,(1< <bitDepth)-1, (PredL0[x][y]+ PredL1[x][y]+offset2)>>shift2 ) Here, the initial state of gbiAppliedFlag is FALSE. The GBI unit 30955 sets gbiAppliedFlag to TRUE when a flag of the SPS indicating that GBI processing is possible is On and bidirectional prediction is performed. Furthermore, as an additional (AND) condition, gbiAppliedFlag may be set to TRUE when gbiIdx, which is an index of the table gbiWLut of the weight coefficient of GBI prediction, is not 0 (the index value when the weights of the L0 predicted image and the L1 predicted image are equal). Furthermore, as an additional (AND) condition, gbiAppliedFlag may be set to TRUE when the block size of the CU is equal to or larger than a certain value.

[0191] When gbiAppliedFlag is true, the GBI unit 30955 derives a predicted image Pred from the weights w0, w1 and PredL0, PredL1 using the following formula.

[0192] Pred[x][y]=Clip3(0,(1< <bitDepth)-1, (w0*PredL0[x][y]+w1*PredL1[x][y]+offset3)>>(shift2+3)) Here, the weighting coefficient w1 is a coefficient derived from the table iWLut[]={4,5,3,10,-2} by gbiIdx explicitly indicated in the syntax. The weighting coefficient w0 is (8-w1). When gbiIdx=0, w0=w1=4, which is equivalent to normal bidirectional prediction.

[0193] shift1, shift2, offset1, and offset2 are derived using the following formula.

[0194] shift1=Max(2,14-bitDepth) shift2=Max(3,15-bitDepth)=shift1+1 offset1=1<<(shift1-1) offset2=1<<(shift2-1) offset3=1<<(shift2+2) Note that there may be a plurality of tables gbiWLut each having a different combination of weighting factors, and the GBI unit 30955 may switch the table used to select a weighting factor depending on whether the picture structure is Low Delay (LB) or not.

[0195] When GBI prediction is used in the AMVP prediction mode, the inter prediction parameter decoding unit 303 decodes gbiIdx and sends it to the GBI unit 30955. When GBI prediction is used in the merge prediction mode, the inter prediction parameter decoding unit 303 decodes the merge index merge_idx, and the merge candidate derivation unit 30361 derives gbiIdx of each merge candidate. Specifically, the merge candidate derivation unit 30361 uses the weighting factor of the adjacent block used to derive the merge candidate as the weighting factor of the merge candidate used for the current block. That is, in the merge mode, the weighting factor used in the past is inherited as the weighting factor of the current block.

[0196] (Selection of prediction mode using GBI prediction) Next, a process of selecting a prediction mode using GBI prediction in the video decoding device 31 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the flow of the process of selecting a prediction mode in the video decoding device 31.

[0197] 14, the inter prediction parameter decoding unit 303 first decodes the skip flag (S1401). If the skip flag indicates the skip mode (YES in S1402), the prediction mode becomes the merge mode (S1403), and the inter prediction parameter decoding unit 303 decodes the merge index (S14031). If the GBI prediction is used, the GBI unit 30955 derives the weighting coefficient derived for the merge candidate as the weighting coefficient for the GBI prediction.

[0198] If the skip flag does not indicate the skip mode (NO in S1402), the inter prediction parameter decoding unit 303 decodes the merge flag (S1407). If the merge flag indicates the merge mode (YES in S1408), the prediction mode becomes the merge mode (S1403), and the inter prediction parameter decoding unit 303 decodes the merge index (S14031). If GBI prediction is used, the GBI unit 30955 derives the weight coefficient derived for the merge candidate as the weight coefficient for GBI prediction.

[0199] If the merge flag does not indicate the merge mode (NO in S1408), the prediction mode is the AMVP mode (S1409).

[0200] In the AMVP mode, the inter prediction parameter decoding unit 303 decodes the inter prediction identifier inter_pred_idc (S14090). Then, the inter prediction parameter decoding unit 303 decodes the difference vector mvdLX (S14091). Then, the inter prediction parameter decoding unit 303 decodes the gbiIdx (S14092), and when the GBI prediction is used, the GBI unit 30955 selects the weighting factor w1 of the GBI prediction from the weighting factor candidates in the gbiWLut table.

[0201] (BDOF prediction) Next, a description will be given of details of prediction using the BDOF process (BDOF prediction) performed by the BDOF unit 30956. In the bi-prediction mode, the BDOF unit 30956 generates a predicted image by referring to two predicted images (a first predicted image and a second predicted image) and a gradient correction term.

[0202] FIG. 15 is a flowchart illustrating the flow of a process for deriving a predicted image.

[0203] When the inter prediction parameter decoding unit 303 determines that the prediction mode is unidirectional prediction of L0 (inter_pred_idc is 0 in S1501), the motion compensation unit 3091 generates an L0 predicted image PredL0[x][y] (S1502). When the inter prediction parameter decoding unit 303 determines that the prediction mode is unidirectional prediction of L1 (inter_pred_idc is 1 in S1501), the motion compensation unit 3091 generates an L1 predicted image PredL1[x][y] (S1503). On the other hand, when the inter prediction parameter decoding unit 303 determines that the prediction mode is bi-prediction mode (inter_pred_idc is 2 in S1501), the process proceeds to the following S1504. In S1504, the synthesis unit 3095 determines whether or not BDOF processing is required by referring to bioAvailableFlag indicating whether or not BDOF processing is performed. When bioAvailableFlag indicates TRUE, the BDOF unit 30956 executes BDOF processing to generate a bidirectional predicted image (S1506). When bioAvailableFlag indicates FALSE, the synthesis unit 3095 generates a predicted image by normal bidirectional predicted image generation (S1505).

[0204] The inter prediction parameter decoding unit 303 may derive TRUE for bioAvailableFlag when the L0 reference image refImgL0 and the L1 reference image refImgL1 are different reference images and the two pictures are in opposite directions with respect to the target picture. Specifically, when the target picture is currPic, bioAvailableFlag indicates TRUE when the condition DiffPicOrderCnt(currPic, refImgL0)*DiffPicOrderCnt(currPic, refImgL1)<0 is satisfied. Here, DiffPicOrderCnt() is a function that derives the difference between POC (Picture Order Count: picture display order) of two images as follows:

[0205] DiffPicOrderCnt(picA,picB) = PicOrderCnt(picA)-PicOrderCnt(picB) A condition that the motion vector of the current block is not a sub-block-based motion vector may be added as a condition for indicating that bioAvailableFlag is TRUE.

[0206] Furthermore, as a condition for bioAvailableFlag to indicate TRUE, a condition that the motion vector of the current picture is not a sub-block-based motion vector may be added.

[0207] Furthermore, as a condition for bioAvailableFlag to indicate TRUE, a condition that the sum of absolute differences between the L0 predicted image and the L1 predicted image of two predicted blocks is equal to or greater than a predetermined value may be added.

[0208] Furthermore, a condition that the predicted image creation mode is a block-based predicted image creation mode may be added as a condition for bioAvailableFlag to indicate TRUE.

[0209] Further, as a condition for bioAvailableFlag to indicate TRUE, a condition may be added that, in weighted prediction, neither L0 prediction nor L1 prediction is performed as weighted prediction. Specifically, the condition for bioAvailableFlag to indicate TRUE is when both luma_weight_l0_flag[refIdxL0] indicating whether or not a luminance weighting factor w0 and an offset o0 are present in the L0 predicted picture and luma_weight_l1_flag[refIdxL1] indicating whether or not a luminance weighting factor w1 and an offset o1 are present in the L1 predicted picture are FALSE.

[0210] FIG. 16 is a schematic diagram showing the configuration of the BDOF unit 30956. The specific contents of the processing performed by the BDOF unit 30956 will be described with reference to FIG. 16. The BDOF processing unit 30956 includes an L0, L1 predicted image generating unit 309561, a gradient image generating unit 309562, a correlation parameter calculation unit 309563, a motion compensation modified value derivation unit 309564, and a BDOF predicted image generating unit 309565. The BDOF unit 30956 generates a predicted image from an interpolated image received from the motion compensation unit 3091 and an inter prediction parameter received from the inter prediction parameter decoding unit 303, and outputs the generated predicted image to the addition unit 312. Note that the process of deriving a motion compensation modified value modBIO (motion compensation modified image) from a gradient image and correcting and deriving the predicted images of PredL0 and PredL1 is called a bidirectional gradient change process.

[0211] FIG. 17 is a diagram showing an example of an area where padding is performed. First, the L0, L1 predicted image generating unit 309561 generates an L0, L1 predicted image used for BDOF processing. The BDOF unit 30956 performs BDOF processing based on the L0, L1 predicted image for each CU unit or sub-CU unit shown in FIG. 17, but in order to obtain a gradient, it requires extra interpolated image information for two pixels around the target CU or sub-CU. The interpolated image information for this part is generated using adjacent integer pixels rather than a normal interpolation filter for gradient image generation described later. In other cases, this part is used as a padding area by copying the surrounding pixels as with the outside of the picture. In addition, the unit of BDOF processing is NxN pixels for each CU unit or sub-CU unit, and the processing itself is performed using (N+2)x(N+2) pixels, which includes one surrounding pixel.

[0212] The gradient image generator 309562 generates a gradient image. In gradient change (Optical Flow), it is assumed that the pixel value of each point does not change, but only its position changes. This can be expressed as follows using the change in pixel value I in the horizontal direction (horizontal gradient value lx) and its position change Vx, the change in pixel value I in the vertical direction (vertical gradient value ly) and its position change Vy, and the time change in pixel value I lt.

[0213] lx * Vx + ly * Vy + lt = 0 Hereafter, the change in position (Vx, Vy) is called the correction weight vector (u, v).

[0214] Specifically, the gradient image generation unit 309562 derives gradient images lx0, ly0, lx1, and ly1 from the following equations: lx0 and lx1 indicate gradients along the horizontal direction, and ly0 and ly1 indicate gradients along the vertical direction.

[0215] lx0[x][y] = (PredL0[x+1][y]-PredL0[x-1][y])>>shift1 ly0[x][y] = (PredL0[x][y+1]-PredL0[x][y-1])>>shift1 lx1[x][y] = (PredL1[x+1][y]-PredL1[x-1][y])>>shift1 ly1[x][y] = (PredL1[x][y+1]-PredL1[x][y-1])>>shift1 Here, shift1=Max(2,14-bitDepth).

[0216] Next, the correlation parameter calculation unit 309563 derives gradient product sums s1, s2, s3, s5, and s6 of (N+2) x (N+2) pixels using one surrounding pixel for each block of N x N pixels in each CU.

[0217] s1 = sum(phiX[x][y]* phiX[x][y]) s2 = sum(phiX[x][y]* phiY[x][y]) s3 = sum(-theta[x][y]* phiX[x][y]) s5 = sum(phiY[x][y]* phiY[x][y]) s6 = sum(-theta[x][y]* phiY[x][y]) Here, sum(a) represents the sum of a for the coordinates (x, y) in the (N+2) x (N+2) pixel block. theta[x][y]= -(PredL1[x][y]>>shift4)+(PredL0[x][y]>>shift4) phiX[x][y] = (lx1[x][y] + lx0[x][y])>>shift5 phiY[x][y] = (ly1[x][y] + ly0[x][y])>>shift5 Where: shift4=Min(8,bitDepth-4) shift5=Min(5,bitDepth-7) Let us assume that.

[0218] Next, the motion compensation modification value derivation unit 309564 derives a correction weight vector (u, v) in units of NxN pixels using the derived gradient product sums s1, s2, s3, s5, and s6.

[0219] u = (s3<<3)>>log2(s1) v = ((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1))>>log2(s5) Here, s2m = s2>>12 and s2s = s2&((1<<12)-1).

[0220] In addition, the range of u and v may be further restricted using clips as follows.

[0221] u = s1>0?Clip3(-th,th,-(s3<<3)>>floor(log2(s1))):0 v = s5>0?Clip3(-th,th,((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1))>>floor(log2(s5))):0 Here, th = Max(2, 1<<(13-bitDepth)). Since the value of th needs to be calculated in conjunction with shift1, the case where the pixel bit length bitDepth is greater than 12 bits is taken into consideration.

[0222] The motion compensation modification value derivation unit 309564 derives motion compensation modification values ​​modBIO[x][y] of NxN pixels using the correction weight vector (u, v) in units of NxN pixels and gradient images lx0, ly0, lx1, and ly1.

[0223] modBIO[x][y] = ((lx1[x][y]-lx0[x][y])*u+(ly1[x][y]-ly0[x][y])*v+1)>>1 (Equation A3). Alternatively, modBIO can be derived using the round function as follows:

[0224] modBIO[x][y] = Round(((lx1[x][y]-lx0[x][y])*u)>>1)+Round(((ly1[x][y]-ly0[x][y])*v)>>1) The BDOF predicted image generation unit 309565 uses the above parameters to derive the pixel value Pred of a predicted image of NxN pixels according to the following equation.

[0225] At this time, the BDOF predicted image generation unit 309565 uses the above parameters to derive the pixel value Pred of the predicted image of NxN pixels according to the following formula.

[0226] Pred[x][y] = Clip3(0, (1<<bitDepth)-1,( PredL0[x][y]+PredL1[x][y]+modBIO[x][y] +offset2)> >shift2) Here, shift2=Max(3,15-bitDepth) and offset2=1<<(shift2-1).

[0227] Then, the BDOF predicted image generation unit 309565 outputs the generated predicted image of the block to the addition unit 312.

[0228] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the entropy decoding unit 301 to obtain transform coefficients. The quantized transform coefficients are coefficients obtained by performing frequency transform such as DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform) on prediction errors and quantizing the transform coefficients in the encoding process. The inverse quantization and inverse transform unit 311 performs inverse frequency transform such as inverse DCT or inverse DST on the obtained transform coefficients to calculate prediction errors. The inverse quantization and inverse transform unit 311 outputs the prediction errors to the adder unit 312. The inverse quantization and inverse transform unit 311 sets all prediction errors to 0 when skip_flag is 1 or cu_cbp is 0.

[0229] The adder 312 generates a decoded image of the block by adding, for each pixel, the predicted image of the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311. The adder 312 stores the decoded image of the block in the reference picture memory 306, and also outputs it to the loop filter 305.

[0230] (Configuration of a video encoding device) Next, the configuration of the video encoding device 11 according to this embodiment will be described. Fig. 18 is a schematic diagram showing the configuration of the video encoding device 11 according to this embodiment. The video encoding device 11 includes a prediction image generating unit 101, a subtraction unit 102, a transformation and quantization unit 103, an inverse quantization and inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determining unit 110, a parameter encoding unit 111, and an entropy encoding unit 104.

[0231] The predicted image generating unit 101 generates a predicted image for each CU, which is an area obtained by dividing each picture of the image T. The predicted image generating unit 101 operates in the same manner as the predicted image generating unit 308 already described, and therefore a description thereof will be omitted.

[0232] The subtraction unit 102 generates a prediction error by subtracting the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

[0233] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing the prediction errors. The transform / quantization unit 103 outputs the quantized transform coefficients to the entropy coding unit 104 and the inverse quantization / inverse transform unit 105.

[0234] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (FIG. 7) in the video decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to the addition unit .

[0235] The entropy coding unit 104 receives the quantized transform coefficients from the transform / quantization unit 103 and the coding parameters from the parameter coding unit 111. The coding parameters include, for example, codes of a reference picture index refIdxLX, a prediction vector index mvp_LX_idx, a difference vector mvdLX, a motion vector precision mode amvr_mode, a prediction mode predMode, and a merge index merge_idx.

[0236] The entropy coding unit 104 entropy codes the division information, the prediction parameters, the quantized transformation coefficients, etc. to generate a coded stream Te, and outputs it.

[0237] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, a CU coding unit 1112 (prediction mode coding unit), and a parameter coding unit 112, all of which are not shown. The CU coding unit 1112 further includes a TU coding unit 1114.

[0238] The operation of each module will be outlined below: The parameter coding unit 111 performs coding processing of parameters such as header information, division information, prediction information, and quantized transformation coefficients.

[0239] The CT information encoding unit 1111 encodes the QT, MT (BT, TT) division information and the like from the encoded data.

[0240] The CU encoding unit 1112 encodes the CU information, prediction information, the TU split flag split_transform_flag, the CU residual flags cbf_cb, cbf_cr, cbf_luma, and the like.

[0241] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information (quantization correction value) and the quantized prediction error (residual_coding).

[0242] The CT information encoding unit 1111 and the CU encoding unit 1112 output syntax elements such as inter prediction parameters (prediction mode predMode, merge flag merge_flag, merge index merge_idx, inter prediction identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX), intra prediction parameters (prev_intra_luma_pred_flag, mpm_idx, rem_selected_mode_flag, rem_selected_mode, rem_non_selected_mode), and quantized transform coefficients to the entropy encoding unit 104.

[0243] (Configuration of the parameter encoding unit) The parameter coding unit 112 derives inter prediction parameters based on the prediction parameters input from the coding parameter determination unit 110. The parameter coding unit 112 includes a part of the same configuration as the configuration in which the inter prediction parameter decoding unit 303 derives inter prediction parameters.

[0244] FIG. 19 is a schematic diagram showing a configuration of the parameter coding unit 112. The configuration of the parameter coding unit 112 will be described. As shown in FIG. 19, the parameter coding unit 112 includes a parameter coding control unit 1121, a merge prediction unit 30374, a sub-block prediction unit (affine prediction unit) 30372, a DMVR unit 30375, an MMVD prediction unit 30376, a Triangle prediction unit 30377, an AMVP prediction parameter derivation unit 3032, and a subtraction unit 1123. The merge prediction unit 30374 includes a merge prediction parameter derivation unit 3036. The parameter coding control unit 1121 includes a merge index derivation unit 11211 and a vector candidate index derivation unit 11212. Furthermore, the parameter coding control unit 1121 derives merge_idx, affine_flag, base_candidate_idx, distance_idx, direction_idx, etc. in the merge index derivation unit 11211, and derives mvpLX, etc. in the vector candidate index derivation unit 11212. The merge prediction parameter derivation unit 3036, the AMVP prediction parameter derivation unit 3032, the affine prediction unit 30372, the MMVD prediction unit 30376, and the triangle prediction unit 30377 may be collectively referred to as a motion vector derivation unit (motion vector derivation device). The parameter coding unit 112 outputs the motion vector mvLX, the reference picture index refIdxLX, the inter prediction identifier inter_pred_idc, or information indicating these to the predicted image generation unit 101. In addition, parameter coding unit 112 outputs merge_flag, skip_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_lX_idx, mvdLX, amvr_mode, and affine_flag to entropy coding unit 104.

[0245] FIG. 20 is a diagram showing an example of the number of candidates for the search distance and the number of candidates for the derivation direction in the video encoding device 11. The parameter encoding control unit 1121 derives parameters (base_candidate_idx, distance_idx, direction_idx, etc.) that represent a difference vector, and outputs them to the MMVD prediction unit 30376. The derivation of a difference vector in the parameter encoding control unit 1121 will be described with reference to FIG. 20. The black circle in the center of the figure is the position indicated by the predicted vector mvpLX, and eight search distances are searched in each of the four directions (up, down, left, and right) centered on this position. mvpLX is the motion vector of the first and second candidates in the merge candidate list, and a search is performed for each of them. There are two predicted vectors in the merge candidate list (the first and second in the list), the search distance is 8, and the search direction is 4, so there are 64 candidates for mvdLX. The mvdLX with the lowest cost among the searched ones is represented by base_candidate_idx, distance_idx, and direction_idx.

[0246] In this way, the MMVD mode is a mode in which a limited number of candidate points are searched for around the predicted vector to derive an appropriate motion vector.

[0247] The merge index derivation unit 11211 derives a merge index merge_idx and outputs it to the merge prediction parameter derivation unit 3036 (merge prediction unit). In the MMVD mode, the merge index derivation unit 11211 sets the value of the merge index merge_idx to the same value as the value of base_candidate_idx. The vector candidate index derivation unit 11212 derives a predicted vector index mvp_lX_idx.

[0248] The merge prediction parameter derivation unit 3036 derives inter prediction parameters based on the merge index merge_idx.

[0249] The AMVP prediction parameter derivation unit 3032 derives a prediction vector mvpLX based on the motion vector mvLX. The AMVP prediction parameter derivation unit 3032 outputs the prediction vector mvpLX to the subtraction unit 1123. Note that the reference picture index refIdxLX and the prediction vector index mvp_lX_idx are output to the entropy encoding unit 104.

[0250] The affine prediction unit 30372 derives inter prediction parameters (affine prediction parameters) for the sub-block.

[0251] The subtraction unit 1123 generates a difference vector mvdLX by subtracting the prediction vector mvpLX, which is the output of the AMVP prediction parameter derivation unit 3032, from the motion vector mvLX input from the coding parameter determination unit 110. The difference vector mvdLX is output to the entropy coding unit 104.

[0252] The adder 106 generates a decoded image by adding, for each pixel, the pixel value of the predicted image of the block input from the predicted image generation unit 101 and the prediction error input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in the reference picture memory 109.

[0253] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters, and may be configured, for example, as only a deblocking filter.

[0254] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0255] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined position for each current picture and CU.

[0256] The coding parameter determination unit 110 selects one set from among a plurality of sets of coding parameters. The coding parameters are the above-mentioned QT, BT or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0257] The coding parameter determination unit 110 calculates an RD cost value indicating the amount of information and the coding error for each of the multiple sets. The coding parameter determination unit 110 selects the set of coding parameters that minimizes the calculated cost value. As a result, the entropy coding unit 104 outputs the selected set of coding parameters as a coded stream Te. The coding parameter determination unit 110 stores the determined coding parameters in the predicted parameter memory 108.

[0258] In addition, a part of the video encoding device 11 and the video decoding device 31 in the above-mentioned embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generating unit 308, the inverse quantization and inverse transform unit 311, the addition unit 312, the predicted image generating unit 101, the subtraction unit 102, the transform and quantization unit 103, the entropy encoding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the encoding parameter determination unit 110, and the parameter encoding unit 111 may be realized by a computer. In this case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into and executed by a computer system. In addition, the "computer system" referred to here is a computer system built into either the video encoding device 11 or the video decoding device 31, and includes hardware such as an OS and peripheral devices. In addition, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include devices that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0259] In addition, a part or the whole of the video encoding device 11 and the video decoding device 31 in the above-mentioned embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 11 and the video decoding device 31 may be individually made into a processor, or a part or the whole may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Furthermore, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used.

[0260] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0261] [Application example] The above-mentioned video encoding device 11 and video decoding device 31 can be mounted on various devices that transmit, receive, record, and play videos. The video may be a natural video captured by a camera or the like, or an artificial video (including CG and GUI) generated by a computer or the like.

[0262] First, it will be described with reference to FIG. 2 that the above-mentioned video encoding device 11 and video decoding device 31 can be used for transmitting and receiving videos.

[0263] Fig. 2(a) is a block diagram showing the configuration of a transmitting device PROD_A equipped with a video encoding device 11. As shown in the figure, the transmitting device PROD_A includes an encoding unit PROD_A1 that obtains encoded data by encoding a video, a modulation unit PROD_A2 that obtains a modulated signal by modulating a carrier wave with the encoded data obtained by the encoding unit PROD_A1, and a transmitting unit PROD_A3 that transmits the modulated signal obtained by the modulation unit PROD_A2. The above-mentioned video encoding device 11 is used as this encoding unit PROD_A1.

[0264] The transmitting device PROD_A may further include, as input sources of moving images to be input to the encoding unit PROD_A1, a camera PROD_A4 for capturing moving images, a recording medium PROD_A5 on which moving images are recorded, an input terminal PROD_A6 for inputting moving images from the outside, and an image processing unit A7 for generating or processing images. In the figure, a configuration in which the transmitting device PROD_A is provided with all of these components is illustrated, but some of them may be omitted.

[0265] The recording medium PROD_A5 may record unencoded video, or may record video encoded by an encoding method for recording that is different from the encoding method for transmission. In the latter case, a decoding unit (not shown) that decodes the encoded data read from the recording medium PROD_A5 according to the encoding method for recording may be interposed between the recording medium PROD_A5 and the encoding unit PROD_A1.

[0266] 2(b) is a block diagram showing the configuration of a receiving device PROD_B equipped with a video decoding device 31. As shown in the figure, the receiving device PROD_B includes a receiving unit PROD_B1 that receives a modulated signal, a demodulation unit PROD_B2 that obtains coded data by demodulating the modulated signal received by the receiving unit PROD_B1, and a decoding unit PROD_B3 that obtains a video by decoding the coded data obtained by the demodulation unit PROD_B2. The above-mentioned video decoding device 31 is used as this decoding unit PROD_B3.

[0267] The receiving device PROD_B may further include, as output destinations of the moving image output by the decoding unit PROD_B3, a display PROD_B4 for displaying the moving image, a recording medium PROD_B5 for recording the moving image, and an output terminal PROD_B6 for outputting the moving image to the outside. In the figure, a configuration in which the receiving device PROD_B includes all of these components is illustrated, but some of them may be omitted.

[0268] The recording medium PROD_B5 may be for recording unencoded video, or may be encoded by an encoding method for recording that is different from the encoding method for transmission. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_B3 and the recording medium PROD_B5, which encodes the video acquired from the decoding unit PROD_B3 according to the encoding method for recording.

[0269] The transmission medium for transmitting the modulated signal may be wireless or wired. The transmission mode for transmitting the modulated signal may be broadcast (here, this refers to a transmission mode in which the destination is not specified in advance) or communication (here, this refers to a transmission mode in which the destination is specified in advance). In other words, the transmission of the modulated signal may be realized by any of wireless broadcast, wired broadcast, wireless communication, and wired communication.

[0270] For example, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for terrestrial digital broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals by wireless broadcasting. Also, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for cable television broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals by cable broadcasting.

[0271] Also, a server (such as a workstation) / client (such as a television receiver, a personal computer, a smartphone, etc.) of a VOD (Video On Demand) service or a video sharing service using the Internet is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals by communication (usually, in a LAN, either wireless or wired is used as a transmission medium, and in a WAN, wired is used as a transmission medium). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Also, smartphones include multi-function mobile phone terminals.

[0272] The client of the video hosting service has a function to decode the encoded data downloaded from the server and display it on a display, as well as a function to encode the video captured by the camera and upload it to the server. That is, the client of the video hosting service functions as both the transmitting device PROD_A and the receiving device PROD_B.

[0273] Next, it will be described with reference to FIG. 3 that the above-mentioned video encoding device 11 and video decoding device 31 can be used for recording and reproducing videos.

[0274] 3(a) is a block diagram showing the configuration of a recording device PROD_C equipped with the above-mentioned video encoding device 11. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that obtains encoded data by encoding a video, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 onto a recording medium PROD_M. The above-mentioned video encoding device 11 is used as this encoding unit PROD_C1.

[0275] The recording medium PROD_M may be (1) a type built into the recording device PROD_C, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), (2) a type connected to the recording device PROD_C, such as an SD memory card or USB (Universal Serial Bus) flash memory, or (3) a type loaded into a drive device (not shown) built into the recording device PROD_C, such as a DVD (Digital Versatile Disc: registered trademark) or BD (Blu-ray Disc: registered trademark).

[0276] The recording device PROD_C may further include, as input sources of moving images to be input to the encoding unit PROD_C1, a camera PROD_C3 for capturing moving images, an input terminal PROD_C4 for inputting moving images from the outside, a receiving unit PROD_C5 for receiving moving images, and an image processing unit PROD_C6 for generating or processing images. In the figure, a configuration in which the recording device PROD_C includes all of these components is illustrated, but some of them may be omitted.

[0277] The receiving unit PROD_C5 may receive unencoded video, or may receive encoded data encoded by an encoding method for transmission that is different from the encoding method for recording. In the latter case, a transmission decoding unit (not shown) that decodes the encoded data encoded by the encoding method for transmission may be interposed between the receiving unit PROD_C5 and the encoding unit PROD_C1.

[0278] Examples of such a recording device PROD_C include a DVD recorder, a BD recorder, and a HDD (Hard Disk Drive) recorder (in this case, the input terminal PROD_C4 or the receiving unit PROD_C5 is the main input source of the moving image). Other examples of such a recording device PROD_C include a camcorder (in this case, the camera PROD_C3 is the main input source of the moving image), a personal computer (in this case, the receiving unit PROD_C5 or the image processing unit C6 is the main input source of the moving image), and a smartphone (in this case, the camera PROD_C3 or the receiving unit PROD_C5 is the main input source of the moving image).

[0279] 3(b) is a block diagram showing the configuration of a playback device PROD_D equipped with the above-mentioned video decoding device 31. As shown in the figure, the playback device PROD_D includes a reading unit PROD_D1 that reads out coded data written to a recording medium PROD_M, and a decoding unit PROD_D2 that obtains video by decoding the coded data read by the reading unit PROD_D1. The above-mentioned video decoding device 31 is used as this decoding unit PROD_D2.

[0280] The recording medium PROD_M may be (1) a type that is built into the playback device PROD_D, such as an HDD or SSD, (2) a type that is connected to the playback device PROD_D, such as an SD memory card or USB flash memory, or (3) a type that is loaded into a drive device (not shown) built into the playback device PROD_D, such as a DVD or BD.

[0281] The playback device PROD_D may further include, as output destinations of the video output by the decoding unit PROD_D2, a display PROD_D3 that displays the video, an output terminal PROD_D4 for outputting the video to the outside, and a transmission unit PROD_D5 that transmits the video. In the figure, the playback device PROD_D is illustrated as having all of these components, but some of them may be omitted.

[0282] The transmission unit PROD_D5 may transmit unencoded video, or may transmit encoded data encoded by an encoding method for transmission different from the encoding method for recording. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_D2 and the transmission unit PROD_D5, which encodes the video by the encoding method for transmission.

[0283] Examples of such a playback device PROD_D include a DVD player, a BD player, and an HDD player (in this case, the output terminal PROD_D4 to which a television receiver or the like is connected is the main output destination of the moving image). In addition, a television receiver (in this case, the display PROD_D3 is the main output destination of the moving image), a digital signage (also called an electronic billboard or electronic bulletin board, and the display PROD_D3 or the transmission unit PROD_D5 is the main output destination of the moving image), a desktop PC (in this case, the output terminal PROD_D4 or the transmission unit PROD_D5 is the main output destination of the moving image), a laptop or tablet PC (in this case, the display PROD_D3 or the transmission unit PROD_D5 is the main output destination of the moving image), and a smartphone (in this case, the display PROD_D3 or the transmission unit PROD_D5 is the main output destination of the moving image) are also examples of such a playback device PROD_D.

[0284] (Hardware and Software Realizations) Furthermore, each block of the above-mentioned video decoding device 31 and video encoding device 11 may be realized in hardware by a logic circuit formed on an integrated circuit (IC chip), or in software by using a CPU (Central Processing Unit).

[0285] In the latter case, each of the above devices includes a CPU that executes the commands of a program that realizes each function, a ROM (Read Only Memory) that stores the above program, a RAM (Random Access Memory) that expands the above program, a storage device (recording medium) such as a memory that stores the above program and various data, etc. The object of the embodiment of the present invention can also be achieved by supplying each of the above devices with a recording medium on which the program code (executable program, intermediate code program, source program) of the control program of each of the above devices, which is software that realizes the above-mentioned functions, is recorded in a computer-readable manner, and the computer (or CPU or MPU) reads and executes the program code recorded on the recording medium.

[0286] Examples of the recording medium that can be used include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy disks (registered trademark) and hard disks, disks including optical disks such as CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical discs), MDs (Mini Discs), DVDs (Digital Versatile Discs: registered trademark), CD-Rs (CD Recordable), and Blu-ray Discs (registered trademark), cards such as IC cards (including memory cards) and optical cards, semiconductor memories such as mask ROMs, EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable and Programmable Read-Only Memory: registered trademark), and flash ROMs, and logic circuits such as PLDs (Programmable logic devices) and FPGAs (Field Programmable Gate Arrays).

[0287] Moreover, each of the above devices may be configured to be connectable to a communication network, and the above program code may be supplied via the communication network. This communication network is not particularly limited as long as it is capable of transmitting the program code. For example, the Internet, an intranet, an extranet, a LAN (Local Area Network), an ISDN (Integrated Services Digital Network), a VAN (Value-Added Network), a CATV (Community Antenna television / Cable Television) communication network, a virtual private network, a telephone line network, a mobile communication network, a satellite communication network, etc. may be used. Moreover, the transmission medium constituting this communication network is not limited to a specific configuration or type as long as it is a medium capable of transmitting the program code. For example, it can be used in wired communication such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carrier, cable TV line, telephone line, ADSL (Asymmetric Digital Subscriber Line) line, etc., or in wireless communication such as infrared such as IrDA (Infrared Data Association) or remote control, BlueTooth (registered trademark), IEEE802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance: registered trademark), mobile phone network, satellite line, terrestrial digital broadcasting network, etc. The embodiment of the present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

[0288] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims.

[0289] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to Japanese Patent Application No. 2019-043097, filed on March 8, 2018, the entire contents of which are incorporated herein by reference.

[0290] (summary) The present invention can also be expressed as follows.

[0291] An image decoding device according to one aspect of the present invention includes: An inter prediction parameter decoding unit having a process of correcting two motion vectors from errors of two predicted images; If neither of the two predicted images is a weighted prediction, a process of correcting the two motion vectors is performed.

[0292] Moreover, an image encoding device according to an aspect of the present invention includes: An inter-prediction parameter coding unit having a process of correcting two motion vectors from errors of two predicted images, If neither of the two predicted images is a weighted prediction, a process of correcting the two motion vectors is performed.

[0293] By adopting such a configuration, when weighted prediction is applied, the error cannot be evaluated correctly and no effect can be obtained, so by limiting the application conditions, the overall amount of processing can be reduced.

[0294] Moreover, an image decoding device according to an aspect of the present invention includes: An inter prediction parameter decoding unit having a process of correcting two motion vectors from error values ​​of two predicted images; a bidirectional gradient change processing unit that generates a predicted image using a gradient image derived from two generated interpolated images by using the parameters decoded by the inter prediction parameter decoding unit, Using the error values ​​of the two predicted images, it is determined whether or not to apply the processing by the bidirectional gradient variation processor.

[0295] Moreover, an image encoding device according to an aspect of the present invention includes: an inter prediction parameter coding unit having a process of correcting two motion vectors from error values ​​of two predicted images; a bidirectional gradient change processing unit that generates a predicted image using a gradient image derived from two generated interpolated images by using the parameters decoded by the inter prediction parameter decoding unit, Using the error values ​​of the two predicted images, it is determined whether or not to apply the processing by the bidirectional gradient variation processor.

[0296] With this configuration, it is necessary to calculate an error value to correct the motion vector, but on the other hand, in the bidirectional gradient change processing unit, when the error is small, it is ineffective. Therefore, by adding such processing, it becomes possible to determine the application of processing by the bidirectional gradient change processing unit without adding an additional error value, and the overall amount of processing can be reduced. [Industrial Applicability]

[0297] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data in which image data is coded, and a video coding device that generates coded data in which image data is coded, and can also be suitably applied to the data structure of coded data that is generated by a video coding device and referenced by the video decoding device. [Explanation of symbols]

[0298] 31 Image Decoding Device 301 Entropy Decoding Unit 302 Parameter Decoding Unit 3020 Header Decoding Unit 303 Inter-prediction parameter decoding unit 304 Intra prediction parameter decoding unit 308 Prediction Image Generation Unit 309 Inter-prediction image generation unit 310 Intra-prediction image generation unit 311 Inverse quantization and inverse transformation unit 312 Addition section 11 Image encoding device 101 Prediction image generation unit 102 Subtraction section 103 Transformation and Quantization Section 104 Entropy coding unit 105 Inverse quantization and inverse transformation unit 107 Loop Filter 110 Encoding parameter determination unit 111 Parameter Encoding Unit 112 Parameter Encoding Unit 1110 Header encoding part 1111 CT information encoder 1112 CU encoding unit (prediction mode encoding unit) 1114 TU encoding section 3091 Motion Compensation Unit 3095 Composite section 30951 Combined Intra / inter synthesis section 30952 Triangle synthesis section 30953 OBMC Department 30954 Weighted Prediction Unit 30955 GBI Division 30956 BDOF section 309561 L0,L1 predicted image generation unit 309562 Gradient image generator 309563 Correlation Parameter Calculation Unit 309564 Motion compensation correction value derivation part 309565 BDOF predicted image generation part

Claims

1. A video decoding device that performs a decoder side motion vector refinement (DMVR) process using two reference pictures and motion vectors mvL0 and mvL1, a DMVR unit that executes the DMVR processing when a dmvrFlag indicating whether the DMVR processing is to be performed is TRUE; a weighted prediction unit that performs weighted prediction using a first weighting factor, a first offset, a second weighting factor, and a second offset; the DMVR unit sets the dmvrFlag to TRUE based on a predetermined condition, wherein the predetermined condition for setting the dmvrFlag to TRUE includes that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE; The luma_weight_l0_flag[refIdxL0] indicates whether the first weighting factor and the first offset of luma corresponding to the L0 reference picture indicated by the reference picture index refIdxL0 are present or not; A video decoding device characterized in that the luma_weight_l1_flag[refIdxL1] indicates whether or not the second weighting coefficient and the second offset for luma corresponding to the L1 reference picture indicated by the reference picture index refIdxL1 exist.

2. A video encoding device that performs a decoder side motion vector refinement (DMVR) process using two reference pictures and motion vectors mvL0 and mvL1, a DMVR unit that executes the DMVR processing when a dmvrFlag indicating whether the DMVR processing is to be performed is TRUE; a weighted prediction unit that performs weighted prediction using a first weighting factor, a first offset, a second weighting factor, and a second offset; the DMVR unit sets the dmvrFlag to TRUE based on a predetermined condition, wherein the predetermined condition for setting the dmvrFlag to TRUE includes that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE; The luma_weight_l0_flag[refIdxL0] indicates whether the first weighting factor and the first offset of luma corresponding to the L0 reference picture indicated by the reference picture index refIdxL0 are present or not; A video encoding device characterized in that the luma_weight_l1_flag[refIdxL1] indicates whether or not the second weighting coefficient and the second offset for luminance corresponding to the L1 reference picture indicated by the reference picture index refIdxL1 exist.

Images