System, encoded stream data structure, encoded stream generation method, moving image decoding device, and moving image encoding device

Abstract

To provide an image decoding device and an image encoding device that reduce the complexity of image quality improvement processing.SOLUTION: A moving image decoding device includes an inter-prediction parameter decoding unit which is included in a parameter decoding unit and which has a process of correcting two motion vectors from errors between two predicted images. The inter-prediction parameter decoding unit performs a process of correcting the two motion vectors when neither of the two predicted images is a weighted prediction.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a system, a data structure of an encoded stream, a method for generating an encoded stream, a video decoding device, and a video encoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates decoded images by decoding the encoded data.

[0003] Specific video encoding methods include, for example, H.264 / AVC and HEVC (High-Efficiency Video Coding) methods.

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing the images, coding tree units (CTUs) obtained by dividing the slices, coding units (sometimes called coding units (CUs)) obtained by dividing the coding tree units, and transform units (TUs) obtained by dividing the coding units, and are coded / decoded for each CU.

[0005] In such video coding methods, a predicted image is usually generated based on a locally decoded image obtained by encoding / decoding an input image, and the predicted image is subtracted from the input image (original image) to obtain a prediction error (sometimes called a "difference image" or "residual image"), which is then coded. Methods for generating predicted images include inter-frame prediction (inter-prediction) and intra-frame prediction (intra-prediction).

[0006] Furthermore, Non-Patent Document 1 is an example of a recent video encoding and decoding technique. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "Versatile Video Coding (Draft 4)", JVET-M1001, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 2019 Summary of the Invention [Problem to be solved by the invention]

[0008] When deriving the bidirectional predicted image in Non-Patent Document 1, there was a problem in that the DMVR process, which uses two predicted images to correct the motion vector to improve the quality of the predicted image, and the prediction using the BDOF process, which uses a gradient image to improve the quality of the predicted image (BDOF prediction), were highly complex processes.

[0009] An object of the embodiments of the present invention is to realize an image decoding device and an image encoding device that reduce the complexity of these image quality improvement processes. [Means for solving the problem]

[0010] In order to solve the above problem, a system according to one aspect of the present invention includes: a video encoding device for generating an encoded stream obtained by encoding a target image to be encoded; a video decoding device that decodes the encoded stream generated by the video encoding device, the video encoding device and the video decoding device are capable of performing a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector; The video encoding device and the video decoding device each include: a DMVR unit that executes the DMVR process based on a value of dmvrFlag; The DMVR unit includes at least (1) The value of the merge flag is TRUE, (2) predFlagL0 and predFlagL1 are both TRUE, (3) mmvd_flag is FALSE, (4) The difference in POC between the current picture and the L0 reference picture is equal to the difference in POC between the L1 reference picture and the current picture; (5) The width of the processing block is 8 or more. (6) bcwIdx is 0, and (7) The value of luma_weight_l0_flag indicating whether or not a weighting factor for L0 prediction exists and the value of luma_weight_l1_flag indicating whether or not a weighting factor for L1 prediction exists are FALSE; setting the value of the dmvrFlag to TRUE based on If the value of the dmvrFlag is TRUE, the DMVR process is executed.

[0011] Furthermore, a data structure of an encoded stream according to one aspect of the present invention is as follows: A data structure of an encoded stream decoded by a video decoding device that performs a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector, The video decoding device a DMVR unit that executes the DMVR process based on a value of dmvrFlag; The data structure of the coded stream is luma_weight_l0_flag indicating whether a weighting factor for L0 prediction exists or not and luma_weight_l1_flag indicating whether a weighting factor for L1 prediction exists or not, and in decoding by the video decoding device, at least (1) the value of the merge flag is TRUE, (2) predFlagL0 and predFlagL1 are both TRUE, (3) mmvd_flag is FALSE, and (4) the difference in POC between the current picture and the L0 reference picture is equal to or greater than the difference in POC between the current picture and the L0 reference picture. and the POC of the current picture are equal, (5) the width of the processing block is 8 or more, (6) bcwIdx is 0, and (7) the value of luma_weight_l0_flag indicating whether a weighting factor for L0 prediction exists and the value of luma_weight_l1_flag indicating whether a weighting factor for L1 prediction exists are FALSE.

[0012] Furthermore, a method for generating an encoded stream according to one aspect of the present invention includes: A method for generating an encoded stream by a video decoding device that performs a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector, comprising: performing the DMVR processing based on a value of dmvrFlag; setting the dmvrFlag to TRUE based on a predetermined condition, wherein the predetermined condition for setting the dmvrFlag to TRUE includes both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] being FALSE; The luma_weight_l0_flag[refIdxL0] indicates whether there is a weighting factor corresponding to the L0 reference picture indicated by the reference picture index refIdxL0; The luma_weight_l1_flag[refIdxL1] indicates whether or not there is a weighting factor corresponding to the L1 reference picture indicated by the reference picture index refIdxL1.

[0013] Furthermore, a video decoding device according to one aspect of the present invention includes: A video decoding device capable of performing a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector, a DMVR unit that executes the DMVR process based on a value of dmvrFlag; The DMVR unit includes at least (1) The value of the merge flag is TRUE, (2) predFlagL0 and predFlagL1 are both TRUE, (3) mmvd_flag is FALSE, (4) The difference in POC between the current picture and the L0 reference picture is equal to the difference in POC between the L1 reference picture and the current picture; (5) The width of the processing block is 8 or more. (6) bcwIdx is 0, and (7) The value of luma_weight_l0_flag indicating whether or not a weighting factor for L0 prediction exists and the value of luma_weight_l1_flag indicating whether or not a weighting factor for L1 prediction exists are FALSE; setting the value of the dmvrFlag to TRUE based on If the value of the dmvrFlag is TRUE, the DMVR process is executed.

[0014] Furthermore, a video encoding device according to an aspect of the present invention includes: A video encoding device capable of performing a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector, a DMVR unit that executes the DMVR process based on a value of dmvrFlag; The DMVR unit includes at least (1) The value of the merge flag is TRUE, (2) predFlagL0 and predFlagL1 are both TRUE, (3) mmvd_flag is FALSE, (4) The difference in POC between the current picture and the L0 reference picture is equal to the difference in POC between the L1 reference picture and the current picture; (5) The width of the processing block is 8 or more. (6) bcwIdx is 0, and (7) The value of luma_weight_l0_flag indicating whether or not a weighting factor for L0 prediction exists and the value of luma_weight_l1_flag indicating whether or not a weighting factor for L1 prediction exists are FALSE; setting the value of the dmvrFlag to TRUE based on If the value of the dmvrFlag is TRUE, the DMVR process is executed. [Effects of the Invention]

[0015] According to the above configuration, it is possible to realize an image decoding device and an image coding device in which the complexity of the image quality improvement process is reduced. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a schematic diagram showing the configuration of an image transmission system according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing the configuration of a transmitting device equipped with a video encoding device according to this embodiment, and a receiving device equipped with a video decoding device, where (a) shows the transmitting device equipped with the video encoding device, and (b) shows the receiving device equipped with the video decoding device. [Figure 3]1A and 1B are diagrams showing the configurations of a recording device equipped with a video encoding device according to an embodiment of the present invention, and a playback device equipped with a video decoding device, where (a) shows the configuration of a recording device equipped with a video encoding device, and (b) shows the configuration of a playback device equipped with a video decoding device. [Figure 4] FIG. 2 is a diagram showing a hierarchical structure of data in an encoded stream. [Figure 5] FIG. 10 is a diagram illustrating an example of division of a CTU. [Figure 6] FIG. 1 is a conceptual diagram illustrating an example of a reference picture and a reference picture list. [Figure 7] FIG. 1 is a schematic diagram illustrating a configuration of a video decoding device. [Figure 8] FIG. 10 is a schematic diagram showing the configuration of an inter-prediction parameter decoding unit. [Figure 9] 1 is a schematic diagram showing the configuration of a merge prediction parameter derivation unit and an AMVP prediction parameter derivation unit. [Figure 10] FIG. 2 is a schematic diagram showing the configuration of a DMVR unit. [Figure 11] 10 is a flowchart showing the flow of processing in the DMVR unit. [Figure 12] 10 is a flowchart illustrating a process for determining BDOF by error threshold processing in DMVR. [Figure 13] FIG. 10 is a schematic diagram showing the configuration of an inter-prediction image generation unit included in the prediction image generation unit. [Figure 14] 10 is a flowchart showing an example of the flow of a prediction mode selection process in the video decoding device. [Figure 15] 10 is a flowchart illustrating the flow of a process for deriving a predicted image. [Figure 16] FIG. 2 is a schematic diagram showing the configuration of a BDOF section. [Figure 17] FIG. 10 is a diagram illustrating an example of an area where a BDOF unit performs padding. [Figure 18] FIG. 1 is a schematic diagram illustrating a configuration of a video encoding device. [Figure 19] FIG. 2 is a schematic diagram illustrating a configuration of a parameter encoding unit. [Figure 20] 10A and 10B are diagrams illustrating an example of the number of search distance candidates and the number of derivation direction candidates in a video encoding device. DETAILED DESCRIPTION OF THE INVENTION

[0017] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] FIG. 1 is a schematic diagram showing the configuration of an image transmission system 1 according to this embodiment.

[0019] The image transmission system 1 is a system that transmits an encoded stream obtained by encoding an image to be encoded, decodes the transmitted encoded stream, and displays an image. The image transmission system 1 includes a video encoding device (image encoding device) 11, a network 21, a video decoding device (image decoding device) 31, and a video display device (image display device) 41.

[0020] An image T is input to the video encoding device 11 .

[0021] The network 21 transmits the coded stream Te generated by the video coding device 11 to the video decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, but may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Furthermore, the network 21 may be replaced by a storage medium on which the coded stream Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

[0022] The video decoding device 31 decodes each of the coded streams Te transmitted over the network 21, and generates one or more decoded images Td.

[0023] The moving image display device 41 displays all or part of one or more decoded images Td generated by the moving image decoding device 31. The moving image display device 41 includes, for example, a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. Examples of the form of the display include stationary, mobile, HMD (Head Mount Display), and the like. Further, when the moving image decoding device 31 has high processing capabilities, the moving image display device 41 displays an image with high image quality, and when it has only lower processing capabilities, it displays an image that does not require high processing capabilities or display capabilities.

[0024] <Operator> The operators used in this specification are described below.

[0025] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || indicates a logical OR.

[0026] x?y:z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0027] Clip3(a, b, c) is a function that clips c to a value between a and b (inclusive). If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (where a <= b).

[0028] abs(a) is a function that returns the absolute value of a.

[0029] Int(a) is a function that returns the integer value of a.

[0030] floor(a) is a function that returns the largest integer less than or equal to a.

[0031] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0032] a / d represents the division of a by d (rounded down to the nearest integer).

[0033] sign(a) is a function that returns the sign of a.

[0034] a^b represents a to the bth power.

[0035] <Structure of the coded stream Te> Before proceeding to a detailed description of the video encoding device 11 and video decoding device 31 according to this embodiment, the data structure of the encoded stream Te generated by the video encoding device 11 and decoded by the video decoding device 31 will be described.

[0036] Fig. 4 is a diagram showing a hierarchical structure of data in a coded stream Te. The coded stream Te illustratively includes a sequence and a plurality of pictures constituting the sequence. (a) to (f) of Fig. 4 show a coded video sequence that defines the sequence SEQ, a coded picture that defines the picture PICT, a coded slice that defines the slice S, coded slice data that defines the slice data, a coding tree unit included in the coded slice data, and a coding unit included in the coding tree unit, respectively.

[0037] (Coded Video Sequence) The coded video sequence defines a set of data that the video decoding device 31 refers to in order to decode the sequence SEQ to be processed. As shown in Fig. 4(a), the sequence SEQ includes a video parameter set (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0038] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers included in the video and each individual layer.

[0039] The sequence parameter set SPS defines a set of coding parameters that the video decoding device 31 references to decode the target sequence. For example, the width and height of a picture are defined. Note that there may be multiple SPSs. In this case, one of the multiple SPSs is selected from the PPS.

[0040] The picture parameter set PPS defines a set of coding parameters that the video decoding device 31 references to decode each picture in the target sequence. For example, the picture parameter set PPS includes a reference value of the quantization width used in decoding the picture (pic_init_qp_minus26) and a flag indicating the application of weighted prediction (weighted_pred_flag). Note that there may be multiple PPSs. In this case, one of the multiple PPSs is selected for each picture in the target sequence.

[0041] (encoded picture) A coded picture defines a set of data that the video decoding device 31 refers to in order to decode a picture PICT to be processed. As shown in FIG. 4(b), the picture PICT includes slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0042] In the following description, when there is no need to distinguish between slices 0 to NS-1, the subscripts of the symbols may be omitted. This also applies to other data that are included in the coded stream Te described below and have subscripts.

[0043] (encoded slice) A coded slice defines a set of data that the video decoding device 31 refers to in order to decode a target slice S. As shown in FIG. 4(c), a slice includes a slice header and slice data.

[0044] The slice header includes a group of coding parameters that the video decoding device 31 refers to in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates the slice type is an example of a coding parameter included in the slice header.

[0045] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction during encoding, (2) a P slice that uses unidirectional prediction or intra prediction during encoding, and (3) a B slice that uses unidirectional prediction, bidirectional prediction, or intra prediction during encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, P and B slices refer to slices that include blocks that can use inter prediction.

[0046] Note that the slice header may include a reference to a picture parameter set PPS (pic_parameter_set_id).

[0047] (encoded slice data) The coded slice data defines a set of data that the video decoding device 31 references in order to decode the slice data to be processed. As shown in Fig. 4(d), the slice data includes a CTU. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0048] (coding tree unit) 4(e) defines a set of data that the video decoding device 31 references to decode the target CTU. The CTU is divided into coding units (CUs), which are basic units of encoding processing, by recursive quad tree (QT) division, binary tree (BT) division, or ternary tree (TT) division. BT division and TT division are collectively called multi-tree (MT) division. A node in the tree structure obtained by recursive quad tree division is called a coding node. The intermediate nodes of the quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is defined as the top-level coding node.

[0049] CT includes, as CT information, a QT split flag (qt_split_cu_flag) indicating whether or not QT splitting is performed, an MT split flag (mtt_split_cu_flag) indicating whether or not MT splitting is performed, an MT split direction (mtt_split_cu_vertical_flag) indicating the split direction of MT splitting, and an MT split type (mtt_split_cu_binary_flag) indicating the split type of MT splitting. qt_split_cu_flag, mtt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

[0050] 5 is a diagram showing an example of division of a CTU. When qt_split_cu_flag is 1, the coding node is divided into four coding nodes (FIG. 5(b)).

[0051] When qt_split_cu_flag is 0 and mtt_split_cu_flag is 0, the coding node is not split and has one CU as a node (Figure 5(a)). A CU is the terminal node of the coding node and is not split any further. A CU is the basic unit of the coding process.

[0052] When mtt_split_cu_flag is 1, the coding node is MT-split as follows: When mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 1, the coding node is split horizontally into two coding nodes (Fig. 5(d)); when mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 1, the coding node is split vertically into two coding nodes (Fig. 5(c)). When mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 0, the coding node is split horizontally into three coding nodes (Fig. 5(f)); when mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 0, the coding node is split vertically into three coding nodes (Fig. 5(e)). These are shown in Fig. 5(g).

[0053] Also, when the size of the CTU is 64x64 pixels, the size of the CU can be any of 64x64 pixels, 64x32 pixels, 32x64 pixels, 32x32 pixels, 64x16 pixels, 16x64 pixels, 32x16 pixels, 16x32 pixels, 16x16 pixels, 64x8 pixels, 8x64 pixels, 32x8 pixels, 8x32 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels, 64x4 pixels, 4x64 pixels, 32x4 pixels, 4x32 pixels, 16x4 pixels, 4x16 pixels, 8x4 pixels, 4x8 pixels, and 4x4 pixels.

[0054] (encoding unit) As shown in Fig. 4(f), a CU defines a set of data that the video decoding device 31 references in order to decode a coding unit to be processed. Specifically, a CU includes a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines a prediction mode, etc.

[0055] The prediction process may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU. When the sizes of a CU and a sub-CU are the same, there is one sub-CU in the CU. When the size of a CU is larger than the size of a sub-CU, the CU is divided into sub-CUs. For example, when a CU is 8x8 and a sub-CU is 4x4, the CU is divided into 2 parts horizontally and 2 parts vertically, into 4 sub-CUs.

[0056] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times).

[0057] The transform and quantization processes are performed in units of CU, but the quantized transform coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0058] (Prediction parameters) The predicted image is derived from prediction parameters associated with the block, which include intra-prediction and inter-prediction parameters.

[0059] Hereinafter, prediction parameters for inter prediction will be described. The inter prediction parameters are composed of prediction list use flags predFlagL0 and predFlagL1, reference picture indices refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. The prediction list use flags predFlagL0 and predFlagL1 are flags indicating whether reference picture lists called L0 lists and L1 lists, respectively, are used for inter prediction, and when the value is 1, the corresponding reference picture list is used for inter prediction. Note that in this specification, when a "flag indicating whether XX is true" is used, a flag other than 0 (for example, 1) is considered to be XX, and 0 is considered to be not XX, and in logical negation, logical product, etc., 1 is treated as true and 0 is treated as false (the same applies below). However, in an actual device or method, other values ​​may be used as true and false values.

[0060] Syntax elements for deriving inter prediction parameters include, for example, an affine flag affine_flag, a merge flag merge_flag, a merge index merge_idx, an inter prediction identifier inter_pred_idc, a reference picture index refIdxLX, a prediction vector index mvp_LX_idx, a difference vector mvdLX, and a motion vector precision mode amvr_mode.

[0061] (Reference Picture List) A reference picture list is a list of reference pictures stored in the reference picture memory 306. FIG. 6 is a conceptual diagram showing an example of reference pictures and reference picture lists. In FIG. 6(a), each rectangle represents a picture, each arrow represents the reference relationship between pictures, the horizontal axis represents time, I, P, and B in the rectangle represent intra-pictures, uni-predictive pictures, and bi-predictive pictures, respectively, and the numbers in the rectangle represent the decoding order. As shown in the figure, the decoding order of pictures is I0, P1, B2, B3, and B4, and the display order is I0, B3, B2, B4, and P1. FIG. 6(b) shows an example of a reference picture list for picture B3 (the target picture). A reference picture list is a list representing candidate reference pictures, and one picture (slice) may have one or more reference picture lists. In the example shown in the figure, the target picture B3 has two reference picture lists: an L0 list RefPicList0 and an L1 list RefPicList1. In each CU, the reference picture index refIdxLX specifies which picture in the reference picture list RefPicListX (X=0 or 1) to actually reference. The diagram shows an example where refIdxL0=2 and refIdxL1=0. Note that LX is a notation method used when there is no distinction between L0 prediction and L1 prediction; hereinafter, parameters for the L0 list and parameters for the L1 list will be distinguished by replacing LX with L0 or L1.

[0062] (Merge prediction and AMVP prediction) Prediction parameter decoding (encoding) methods include merge prediction mode and AMVP (Advanced Motion Vector Prediction, adaptive motion vector prediction) mode, and the merge flag merge_flag is a flag for distinguishing between them. The merge prediction mode is a mode in which the prediction list usage flag predFlagLX (or inter prediction identifier inter_pred_idc), reference picture index refIdxLX, and motion vector mvLX are not included in the encoded data, but are instead derived from prediction parameters of already processed neighboring blocks. The AMVP mode is a mode in which the inter prediction identifier inter_pred_idc, reference picture index refIdxLX, and motion vector mvLX are included in the encoded data. Note that the motion vector mvLX is encoded as a prediction vector index mvp_LX_idx that identifies the prediction vector mvpLX, a difference vector mvdLX, and a motion vector precision mode amvr_mode. The merge prediction mode is a mode in which a merge candidate derived from motion information, etc. of neighboring blocks is selected to obtain a motion vector mvLX (motion vector information). In addition to the merge prediction mode, there may be an affine prediction mode identified by an affine flag affine_flag. As one form of the merge prediction mode, there may be a skip mode identified by a skip flag skip_flag. Note that the skip mode is a mode in which prediction parameters are derived in the same manner as the merge mode, and in which prediction errors (residual images, residual information) are not included in the encoded data. In other words, when the skip flag skip_flag is 1, only syntax related to the merge mode, such as the skip flag skip_flag and the merge index merge_idx, is included for the target CU, and motion vectors, residual information, etc. are not included in the encoded data.

[0063] (motion vector) The motion vector mvLX indicates the amount of shift between blocks on two different pictures. The prediction vector and difference vector related to the motion vector mvLX are called the prediction vector mvpLX and the difference vector mvdLX, respectively.

[0064] (Inter prediction identifier inter_pred_idc and prediction list usage flag predFlagLX) The inter-prediction identifier inter_pred_idc is a value indicating the type and number of reference pictures, and takes one of the values ​​PRED_L0, PRED_L1, or PRED_BI. PRED_L0 and PRED_L1 indicate uni-prediction using one reference picture managed in the L0 list and L1 list, respectively. PRED_BI indicates bi-prediction (BiPred) using two reference pictures managed in the L0 list and L1 list.

[0065] The merge index merge_idx is an index indicating which prediction parameter from among prediction parameter candidates (merge candidates) derived from blocks for which processing has been completed is to be used as the prediction parameter for the current block.

[0066] The relationship between the inter prediction identifier inter_pred_idc and the prediction list use flags predFlagL0 and predFlagL1 is as follows, and they can be converted into each other.

[0067] inter_pred_idc = (predFlagL1<<1)+predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 (Bi-prediction biPred decision) The flag biPred indicating whether or not the prediction is biPred can be derived based on whether or not two prediction list usage flags are both 1. For example, it can be derived using the following formula.

[0068] biPred = (predFlagL0==1 && predFlagL1==1) Alternatively, the flag biPred can be derived based on whether the inter prediction identifier is a value indicating that two prediction lists (reference pictures) are used. For example, the flag biPred can be derived using the following equation:

[0069] biPred = (inter_pred_idc==PRED_BI) ? 1 : 0 (Configuration of video decoding device) The configuration of a video decoding device 31 (FIG. 7) according to this embodiment will be described.

[0070] The video decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generation unit (prediction image generation device) 308, an inverse quantization and inverse transform unit 311, and an adder 312. Note that the video decoding device 31 may also be configured without including the loop filter 305, in accordance with the video coding device 11 described below.

[0071] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), all of which are not shown, and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, and PPS, and slice headers (slice information) from the coded data. The CT information decoding unit 3021 decodes the CT from the coded data. The CU decoding unit 3022 decodes the CU from the coded data. When a prediction error is included in the TU, the TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the coded data.

[0072] In modes other than skip mode (skip_mode==0), the TU decoding unit 3024 decodes QP update information (quantization correction value) and quantized prediction error (residual_coding) from the coded data. More specifically, in the case of skip_mode==0, the TU decoding unit 3024 decodes, from the coded data, a flag cu_cbp indicating whether or not a quantized prediction error is included in the current block, and decodes the quantized prediction error if cu_cbp is 1. If cu_cbp does not exist in the coded data, the TU decoding unit 3024 derives cu_cbp as 0.

[0073] The parameter decoding unit 302 includes an inter prediction parameter decoding unit 303 and an intra prediction parameter decoding unit 304 (not shown). The predicted image generating unit 308 includes an inter prediction image generating unit 309 and an intra prediction image generating unit 310.

[0074] In the following, an example will be described in which CTUs and CUs are used as processing units, but this is not limiting and processing may be performed in sub-CU units. Alternatively, CTUs and CUs may be read as blocks, and sub-CUs as sub-blocks, and processing may be performed in block or sub-block units.

[0075] The entropy decoding unit 301 performs entropy decoding on the externally input coded stream Te to decode individual codes (syntax elements). The decoded codes include prediction information for generating a predicted image and prediction errors for generating a difference image.

[0076] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code includes, for example, predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0077] (Configuration of Inter-Prediction Parameter Decoding Unit) The inter prediction parameter decoding unit 303 decodes inter prediction parameters by referring to the prediction parameters stored in the prediction parameter memory 307, based on the code input from the entropy decoding unit 301. The inter prediction parameter decoding unit 303 also outputs the decoded inter prediction parameters to the prediction image generation unit 308, and stores them in the prediction parameter memory 307.

[0078] 8 is a schematic diagram showing the configuration of the inter prediction parameter decoding unit 303 according to this embodiment. The inter prediction parameter decoding unit 303 includes a merge prediction unit 30374, a DMVR unit 30375, a sub-block prediction unit (affine prediction unit) 30372, an MMVD prediction unit 30376, a triangle prediction unit 30377, an AMVP prediction parameter derivation unit 3032, and an adder 3038. The merge prediction unit 30374 includes a merge prediction parameter derivation unit 3036. The AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, and the affine prediction unit 30372 are means common to the video encoding device and the video decoding device, and therefore may be collectively referred to as a motion vector derivation unit (motion vector derivation device).

[0079] (Affine prediction part) The affine prediction unit 30372 derives affine prediction parameters for the current block. In this embodiment, the affine prediction parameters are motion vectors (mv0_x, mv0_y) (mv1_x, mv1_y) of two control points (V0, V1) of the current block. Specifically, the motion vectors of each control point may be derived by predicting them from the motion vectors of blocks adjacent to the current block, or by adding the predicted vector derived as the motion vector of the control point and a difference vector derived from the encoded data.

[0080] (Merge prediction) 9(a) is a schematic diagram showing the configuration of the merge prediction parameter derivation unit 3036 included in the merge predictor 30374. The merge prediction parameter derivation unit 3036 includes a merge candidate derivation unit 30361 and a merge candidate selection unit 30362. Note that a merge candidate includes a prediction list usage flag predFlagLX, a motion vector mvLX, and a reference picture index refIdxLX, and is stored in a merge candidate list. An index is assigned to the merge candidates stored in the merge candidate list according to a predetermined rule.

[0081] The merging candidate derivation unit 30361 derives merging candidates by directly using the motion vectors and reference picture indices refIdxLX of the decoded neighboring blocks.

[0082] The merge candidate list mergeCandList[] may be stored in the following order: spatial merge candidates A1, B1, B0, A0, B2, temporal merge candidate Col, pairwise merge candidate avgK, and zero merge candidate ZK. Reference blocks that are unavailable (e.g., blocks that are intra-predicted) are not stored in the merge candidate list.

[0083] The merge candidate selection unit 30362 selects a merge candidate N indicated by the merge index merge_idx from among the merge candidates included in the merge candidate list using the following formula.

[0084] N = mergeCandList[merge_idx] Here, N is a label indicating a merge candidate, and can be A1, B1, B0, A0, B2, Col, avgK, ZK, etc. The motion information of the merge candidate indicated by label N is indicated by (mvLXN[0], mvLXN[1]), predFlagLXN, and refIdxLXN.

[0085] The merge candidate selection unit 30362 selects the motion information (mvLXN[0], mvLXN[1]), predFlagLXN, and refIdxLXN of the selected merge candidate as inter prediction parameters for the current block. The merge candidate selection unit 30362 stores the inter prediction parameters of the selected merge candidate in the prediction parameter memory 307 and outputs them to the predicted image generation unit 308.

[0086] (AMVP forecast) 9(b) is a schematic diagram showing the configuration of the AMVP prediction parameter derivation unit 3032 according to this embodiment. The AMVP prediction parameter derivation unit 3032 includes a vector candidate derivation unit 3033 and a vector candidate selection unit 3034. The vector candidate derivation unit 3033 derives predictor vector candidates from the motion vectors mvLX of decoded adjacent blocks stored in the prediction parameter memory 307, based on the reference picture index refIdxLX, and stores the candidates in a predictor vector candidate list mvpListLX[].

[0087] The vector candidate selection unit 3034 selects, from the prediction vector candidates in the prediction vector candidate list mvpListLX[], the motion vector mvpListLX[mvp_LX_idx] indicated by the prediction vector index mvp_LX_idx as the prediction vector mvpLX. The vector candidate selection unit 3034 outputs the selected prediction vector mvpLX to the adder 3038.

[0088] The prediction vector candidates are derived by scaling the motion vectors of decoded neighboring blocks within a predetermined range from the target block. The neighboring blocks include blocks spatially adjacent to the target block, such as a left block and an upper block, as well as regions temporally adjacent to the target block, such as a region obtained from prediction parameters of blocks that include the same position as the target block but have different display times.

[0089] The adder 3038 adds the predicted vector mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded differential vector mvdLX to calculate the motion vector mvLX. The adder 3038 outputs the calculated motion vector mvLX to the prediction image generation unit 308 and the prediction parameter memory 307.

[0090] mvLX[0] = mvpLX[0]+mvdLX[0] mvLX[1] = mvpLX[1]+mvdLX[1] The motion vector accuracy mode amvr_mode is a syntax for switching the accuracy of the motion vector derived in the AMVP mode. For example, at amvr_mode = 0, 1, 2, the accuracy is switched between 1 / 4 pixel, 1 pixel, and 4 pixels.

[0091] When the accuracy of the motion vector is 1 / 16 accuracy (MVPREC = 16), in order to change the motion vector differences of 1 / 4, 1, and 4 pixel accuracies to the motion vector differences of 1 / 16 pixel accuracy, the parameter decoding unit 302 may perform inverse quantization using MvShift (= 1 << amvr_mode) derived from amvr_mode as follows.

[0092] mvdLX[0] = mvdLX[0] << (MvShift + 2) mvdLX[1] = mvdLX[1] << (MvShift + 2) Note that the parameter decoding unit 302 may also decode and derive the mvdLX[] before shifting by the above MvShift using the following syntax. ·abs_mvd_greater0_flag ·abs_mvd_minus2 ·mvd_sign_flag Then, the parameter decoding unit 302 decodes the differential vector lMvd[] from the syntax by using the following formula.

[0093] lMvd[compIdx] = abs_mvd_greater0_flag[compIdx] * (abs_mvd_minus2[compIdx]+2) * (1-2*mvd_sign_flag[compIdx]) Furthermore, the parameter decoding unit 302 sets the decoded difference vector lMvd[ ] to MvdLX in the case of translational MVD (MotionModelIdc[x][y] == 0), and to MvdCpLX in the case of control point MVD (MotionModelIdc[x][y] != 0).

[0094] if (MotionModelIdc[x][y] == 0) mvdLX[x0][y0][compIdx] = lMvd[compIdx] else mvdCpLX[x0][y0][compIdx] = lMvd[compIdx]<<2 (Motion Vector Scaling) A method for deriving motion vector scaling will now be described. Given a motion vector Mv (reference motion vector), a picture PicMv containing a block having Mv, a reference picture PicMvRef of Mv, a motion vector after scaling sMv, a picture CurPic containing a block having sMv, and a reference picture CurPicRef referenced by sMv, the derivation function MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef) of sMv is expressed by the following equation:

[0095] sMv = MvScale(Mv,PicMv,PicMvRef,CurPic,CurPicRef) = Clip3(-R1,R1-1,sign(distScaleFactor*Mv)*((abs(distScaleFactor*Mv)+round1-1)>>shift1)) distScaleFactor = Clip3(-R2,R2-1,(tb*tx+round2)>>shift2) tx = (16384+abs(td)>>1) / td td = DiffPicOrderCnt(PicMv,PicMvRef) tb = DiffPicOrderCnt(CurPic,CurPicRef) Here, round1, round2, shift1, and shift2 are round and shift values ​​for division using reciprocals, for example, round1=1<<(shift1-1), round2=1<<(shift2-1), shift1=8, shift2=6. DiffPicOrderCnt(Pic1,Pic2) is a function that returns the difference in time information (e.g., POC) between Pic1 and Pic2. R1 and R2 limit the value range to perform processing with limited precision, for example, R1=32768, R2=4096.

[0096] Furthermore, the scaling function MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef) may be expressed by the following formula:

[0097] MvScale(Mv,PicMv,PicMvRef,CurPic,CurPicRef) = Mv*DiffPicOrderCnt(CurPic,CurPicRef) / DiffPicOrderCnt(PicMv,PicMvRef) That is, Mv may be scaled according to the ratio between the difference in time information between CurPic and CurPicRef and the difference in time information between PicMv and PicMvRef.

[0098] (DMVR Department 30375) Next, a description will be given of the DMVR (Decoder-side Motion Vector Refinement) processing performed by the DMVR unit 30375. The DMVR processing is processing for correcting motion vectors mvL0 and mvL1 using two reference pictures.

[0099] Fig. 10 is a schematic diagram showing the configuration of the DMVR unit 30375. Specific processing performed by the DMVR unit 30375 will be described using Fig. 10. The DMVR unit 30375 includes a predicted image generation unit 303751 for corrected motion vector search, an initial error generation unit 303752, a motion vector search unit 303753, and a corrected vector derivation unit 303754.

[0100] The DMVR unit 30375 receives the Top left position of the target block (xCb, yCb) Target block width bW Target block height bH 1 / 16 pixel precision motion vectors mvL0 and mvL1 Reference pictures refPicL0L and refPicL1L The motion vector change amounts dmvL0 and dmvL1 for correcting mvL0 and mvL1 are derived by referring to the above and output to the inter-prediction image generation unit 309.

[0101] First, the corrected motion vector search predicted image generation unit 303751 Top left position of the target sub-block (xSb, ySb) -Width of the target luminance sub-block sbW - Height of the target luminance sub-block sbH Motion vector mvLX(X=0, 1) Reference picture refPicLXL(X=0, 1) A predicted image predSamplesLXL having a size of (sbW)*(sbH) is derived by referring to the above.

[0102] The corrected motion vector search predicted image generation unit 303751 derives a motion vector MvLsX (X=0, 1) using the following equation.

[0103] MvLsX[0] = MvLX[0]-32 MvLsX[1] = MvLX[1]-32 The DMVR unit 30375 also sets the values ​​of the variables srRange, offsetH[0], offsetV[0], offsetH[1], and offsetV[1] to 2, respectively.

[0104] Let (xIntL, yIntL) be the integer pixel position of the reference block corresponding to pixel position (xL, yL) in the current block. Let (xFracL, yFracL) be the offset from (xIntL, yIntL) in 1 / 16 pixel units. These coordinates are derived from the integer component (mvLX[0]>>4, mvLX[1]>>4) and fractional component (mvLX[0]&15, mvLX[1]&15) of the motion vector (mvLX[0], mvLX[1]), and indicate the pixel position in fractional precision within the reference picture refPicLXL. For a pixel located at (xL, yL) (xL=0, ..., sbW-1, yL=0, ..., sbH-1) in predSamplesLXL, the DMVR unit 30375 derives xIntL, yIntL, xFracL, and yFracL using the following equations.

[0105] xIntL = xSb + (mvLX[0]>>4) + xL yIntL = ySb + (mvLX[1]>>4) + yL xFracL = mvLX[0]&15 yFracL = mvLX[1]&15 Next, DMVR section 30375, ·(xIntL, yIntL) (xFracL, yFracL) ·refPicLXL predSamplesLXL is derived by referring to

[0106] First, the corrected motion vector search predicted image generation unit 303751 derives variables shift1, shift2, shift3, and shift4 using the following equations.

[0107] shift1 = BitDepthY - 6 offset1 = 1 << (shift1 - 1) shift2 = 4 offset2 = 8 shift3 = 10-BitDepthY offset3 = 1 << (shift3 - 1) shift4 = BitDepthY - 10 In the above formula, BitDepthY is the number of pixel bits.

[0108] Next, the predicted image generation unit for corrected motion vector search 303751 sets picW equal to the value of the picture width pic_width_in_luma_samples. The predicted image generation unit for corrected motion vector search 303751 also sets picH equal to the value of the picture height pic_height_in_luma_samples.

[0109] Thereafter, the predicted image generation unit 303751 for corrected motion vector search derives predSamplesLXL as follows. L [p] indicates the filter coefficient for deriving pixel values ​​with 1 / 16 pixel accuracy. L The value of [p] depends on the position p (p=1, 2, ..., 15) with 1 / 16 pixel accuracy. The position p is equal to xFracL or yFracL. As the value of p increases, the fb L [p][0] decreases monotonically, and fb L The value of [p][1] increases monotonically.

[0110] First, the predicted image generation unit for corrected motion vector search 303751 determines whether xFracL and yFracL are both 0. If both xFracL and yFracL are 0, the DMVR unit 30375 derives predSamplesLXL using one of the following equations according to the value of BitDepthY.

[0111] predSamplesLXL = (BitDepthY <= 10)? (refPicLXL[xIntL][yIntL] << shift3) : ((refPicLXL[xIntL][yIntL]+offset3) >> shift4) When xFracL is not 0 and yFracL is 0, the predicted image generation unit 303751 for corrected motion vector search derives predSamplesLXL by the following formula.

[0112] predSamplesLXL = (fb L [xFracL][0] * refPicLXL[Clip3(0,picW-1,xIntL)][yIntL] + fb L [xFracL][1] * refPicLXL[Clip3(0,picW-1,xIntL+1)][yIntL] + offset1)>>shift1 When xFracL is 0 and yFracL is not 0, the DMVR unit 30375 derives predSamplesLXL by the following formula.

[0113] predSamplesLXL = (fb L [yFracL][0] * refPicLXL[xIntL][Clip3(0,picH-1,yIntL)] + fb L [yFracL][1] * refPicLXL[xIntL][Clip3(0,picH-1,yIntL+1)]+offset1)>>shift1 If neither xFracL nor yFracL is 0, the modified motion vector search predicted image generation unit 303751 derives predSamplesLXL as follows. First, the DMVR unit 30375 derives temp[n] using the following formula. The temp[] derivation process is performed n times by changing the reference position. n=0 represents the first derivation process, and n=1 represents the second derivation process.

[0114] yPosL = Clip3(0, PicH-1, yIntL+n-3) temp[n] = (fb L [xFracL][0] * refPicLXL[Clip3(0,picW-1,xIntL)][yPosL] + fb L [xFracL][1] * refPicLXL[Clip3(0,picW-1,xIntL+1)][yPosL]+offset1)>>shift1 After that, the DMVR unit 30375 derives predSamplesLXL using the following formula.

[0115] predSamplesLXL = (fb L [yFracL][0] * temp[0] + fb L [yFracL][1] * temp[1])>>shift2 Next, the initial error generator 303752 Target block width nCbW Target block height nCbH Two predicted images predSampleL1 and predSampleL2 with size (nCbW+4)x(nCbH+4) Variables offsetH[0], offsetH[1], offsetV[0], and offsetV[1] A list Sad1 of the sum of absolute differences between pixel values ​​included in predSampleL1 and predSampleL2 and a variable centerSad are derived by referring to the above.

[0116] The DMVR unit 30375 sets the value of each element of the 2×9 array bC according to the following formula.

[0117] bC[0][0] = -1 bC[1][0] = -1 bC[0][1] = -1 bC[1][1] = 0 bC[0][2] = -1 bC[1][2] = 1 bC[0][3] = 0 bC[1][3] = -1 bC[0][4] = 0 bC[1][4] = 0 bC[0][5] = 0 bC[1][5] = 1 bC[0][6] = 1 bC[1][6] = -1 bC[0][7] = 1 bC[1][7] = 0 bC[0][8] = 1 bC[1][8] = 1 The initial error generator 303752 derives the elements sadList[i] (i=0, . . . , 8) of Sad1 using the following formula:

[0118]

number

[0119]

number

[0120] If centerSad is equal to or greater than (bH>>1)*(bW)*4, the initial error generator 303752 sets dmvrFlag to TRUE, and the motion vector searcher 303753 ·Number of search points n List of absolute difference sums of search points Sad1 element sadList The index bestIdx is derived by referring to n, where n is a positive integer.

[0121] The following describes the case where n = 9. Note that the value of the number of search points n may be other than 9, and instead of the method described in this embodiment, for example, the minimum value of the sadList value may be simply selected when n = 25.

[0122] The motion vector search unit 303753 determines whether sadList[1]<sadList[7] and whether sadList[3]<sadList[5].

[0123] If sadList[1] < sadList[7] and sadList[3] < sadList[5], the DMVR unit 30375 sets the value of idx to 0. Then, the motion vector search unit 303753 determines whether sadList[1] < sadList[3]. The motion vector search unit 303753 sets the value of bestIdx to 1 if sadList[1] < sadList[3], and to 3 if sadList[1] < sadList[3] is not true.

[0124] Otherwise, if sadList[1] >= sadList[7] and sadList[3] < sadList[5], the motion vector search unit 303753 sets the value of idx to 6. The DMVR unit 30375 then determines whether sadList[7] < sadList[3]. The motion vector search unit 303753 sets the value of bestIdx to 7 if sadList[7] < sadList[3], and sets it to 3 if sadList[7] < sadList[3] is not true.

[0125] Otherwise, if sadList[1] < sadList[7] and sadList[3] >= sadList[5], the motion vector search unit 303753 sets the value of idx to 2. Then, the DMVR unit 30375 determines whether sadList[1] < sadList[5]. The DMVR unit 30375 sets the value of bestIdx to 2. < sadList[5] then set to 1, otherwise set to 5.

[0126] Otherwise, if sadList[1] >= sadList[7] and sadList[3] >= sadList[5], the motion vector search unit 303753 sets the value of idx to 8. Then, the DMVR unit 30375 determines whether sadList[7] < sadList[5]. The DMVR unit 30375 sets the value of bestIdx to 8. < sadList[5] then set it to 7, otherwise set it to 5.

[0127] Furthermore, the motion vector search unit 303753 determines whether sadList[4] <= sadList[bestIdx]. If sadList[4] <= sadList[bestIdx], the DMVR unit 30375 updates the value of bestIdx to 4. On the other hand, if sadList[4] <= sadList[bestIdx] is not true, the DMVR unit 30375 does not update the value of bestIdx.

[0128] Furthermore, the motion vector search unit 303753 determines whether sadList[idx] < sadList[bestIdx]. If sadList[idx] < sadList[bestIdx], the DMVR unit 30375 sets bestIdx to idx. On the other hand, if sadList[idx] < sadList[bestIdx] is not true, the motion vector search unit 303753 does not update the value of bestIdx.

[0129] The motion vector search unit 303753 determines whether the value of bestIdx is 4. If the value of bestIdx is 4, the motion vector search unit 303753 sets halfPelAppliedflag to true.

[0130] If the value of bestIdx is not 4, the motion vector search unit 303753 calculates the values ​​of the variables dmvx and dmvy using the following formulas.

[0131] dmvx = (bestIdx / 3 - 1) dmvy = (bestIdx%3 - 1) Furthermore, the motion vector search unit 303753 updates offsetH and offsetV according to the following formulas.

[0132] offsetH[0] = offsetH[0] + dmvx, offsetV[0] = offsetV[0] + dmvy offsetH[1] = offsetH[1] - dmvx, offsetV[1] = offsetV[1] - dmvy The motion vector search unit 303753 derives Sad2 by the same process as the process for deriving Sad1 described above, using the updated offset H and offset V. Furthermore, the motion vector search unit 303753 derives bestIdx again using Sad2 instead of Sad1.

[0133] The motion vector search unit 303753 determines whether the re-derived value of bestIdx is 4. If the value of bestIdx is 4, the motion vector search unit 303753 sets halfPelAppliedflag to true.

[0134] If the value of bestIdx is not 4, the motion vector search unit 303753 calculates dmvx and dmvy using the following formulas.

[0135] dmvx = (bestIdx / 3 - 1), dmvy = (bestIdx%3 - 1) Furthermore, the DMVR unit 30375 calculates dmvL0 and dmvL1 using the following formulas.

[0136] dmvL0[0] = 16*dmvx, dmvL0[1] = 16*dmvy dmvL1[0] = -16*dmvx, dmvL1[1] = -16*dmvy When halfPelAppliedflag is true, the motion vector search unit 303753 derives the corrected dmvL0 and dmvL1 as follows: Note that the following sadList is an element of Sad2 if Sad2 exists, and is an element of Sad1 if Sad2 does not exist.

[0137] First, the motion vector search unit 303753 determines whether sadList[1] + sadList[7] == sadList[4]. If sadList[1] + sadList[7] == sadList[4] and if mrSadT + mrSadB - (mrSadC<<1) == 0, the motion vector search unit 303753 sets dmv[0] = 0. If sadList[1] + sadList[7] == sadList[4] is not true, the motion vector search unit 303753 calculates dmv[0] using the following formula.

[0138] dmv[0] = ((sadList[1] - sadList[7])<<3) / (sadList[1] + sadList[7] - (sadList[4]<<1)) Next, the correction vector derivation unit 303754 determines whether sadList[3] + sadList[5] == sadList[4]. If sadList[3] + sadList[5] == sadList[4] and if mrSadL + mrSadR - (mrSadC<<1) == 0, the correction vector derivation unit 303754 sets dmv[1] = 0. If sadList[3] + sadList[5] == sadList[4] is not true, the correction vector derivation unit 303754 calculates dmv[1] using the following formula.

[0139] dmv[1] = ((sadList[3] - sadList[5])<<3) / (sadList[3] + sadList[5] - (sadList[4]<<1)) Furthermore, the correction vector derivation unit 303754 corrects the motion vectors mvL0 and mvL1 using the following equations.

[0140] dmvL0[0] = dmvL0[0] + dmv[0] dmvL0[1] = dmvL0[1] + dmv[1] dmvL1[0] = dmvL1[0] - dmv[0] dmvL1[1] = dmvL1[1] - dmv[1] The DMVR unit 30375 calculates a motion vector mvLX by adding the derived difference vector dmvLX to the predicted vector mvpLX input from the merge prediction unit 30374. The DMVR unit 30375 outputs mvLX to the inter predicted image generation unit 309.

[0141] mvLX[0] = mvpLX[0]+dmvLX[0] mvLX[1] = mvpLX[1]+dmvLX[1] The values ​​of dmvLX[0] and dmvLX[1] are limited to the range from -8 to 8 regardless of the number of bits in sadList.

[0142] (DMVR criteria) dmvrFlag is a flag that indicates that DMVR processing is performed when TRUE, and that DMVR processing is not performed when FALSE.

[0143] If the flag of the SPS indicating that DMVR processing is possible is On, the initial error generator 303752 sets dmvrFlag to TRUE. Otherwise, the initial error generator 303752 sets dmvrFlag to FALSE.

[0144] Furthermore, if the merge_flag of the block is TRUE, the initial error generator 303752 may set the dmvrFlag to TRUE, otherwise the initial error generator 303752 sets the dmvrFlag to FALSE.

[0145] Furthermore, if both predFlagL0 and predFlagL1 are TRUE, that is, if bidirectional prediction is performed, the initial error generator 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generator 303752 sets dmvrFlag to FALSE.

[0146] If the mmvd_flag of the block is FALSE and the mode is not MMVD, the initial error generator 303752 may set dmvrFlag to TRUE. Otherwise, if the mode is MMVD, the initial error generator 303752 sets dmvrFlag to FALSE.

[0147] If DiffPicOrderCnt(currPic, RefPicList

[0000] [refIdxL0]) is equal to DiffPicOrderCnt(RefPicList

[0001] [refIdxL1], currPic), that is, if the current picture currPic is in a positional relationship such that the L0 reference picture RefPicList

[0000] [refIdxL0] and the L1 reference picture RefPicList

[0001] [refIdxL1] are interpolated at equal distances, the initial error generator 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generator 303752 sets dmvrFlag to FALSE. Here, DiffPicOrderCnt() calculates the POC (Picture Order Coefficient) of the two images as follows: This is a function that derives the difference between the number of pictures in the display order.

[0148] DiffPicOrderCnt(picA,picB) = PicOrderCnt(picA)-PicOrderCnt(picB) In addition, DiffPicOrderCnt( currPic, RefPicList

[0000] [ refIdxL0 ])*DiffPicOrderCnt( If currPic, RefPicList

[0001] [refIdxL1]<0, and the positional relationship allows simple interpolation, the initial error generator 303752 may set dmvrFlag to TRUE, and otherwise set dmvrFlag to FALSE.

[0149] Furthermore, if the size of the processing block is equal to or smaller than a specific value, the initial error generator 303752 may set dmvrFlag to FALSE. For example, if bH is equal to or greater than 8 and bH*bW is 64, the initial error generator 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generator 303752 sets dmvrFlag to FALSE.

[0150] Fig. 11 is a flowchart showing the flow of processing in the DMVR unit 30375. In this embodiment, in addition to the above-mentioned criteria, as shown in Fig. 11, a condition is added such that DMVR processing is applied only when GBI processing, which will be described later, is not applied.

[0151] Specifically, first, the DMVR unit 30375 executes the above-described dmvrFlag determination process (S1101). Next, the DMVR unit 30375 determines whether gbiIdx is 0 or not (S1102). When gbiIdx is a non-zero value, as will be described later, non-uniform weighted prediction is performed based on the table gbiWLut. When gbiIdx is 0, in addition to the condition that dmvrFlag is set to TRUE, when gbiIdx is non-zero, dmvrFlag is set to FALSE (S1103).

[0152] Furthermore, the DMVR unit 30375 determines whether or not dmvrFlag is TRUE (S1104), and if TRUE, performs DMVR processing. (S1105) is executed, and if it is FALSE, it is not executed.

[0153] When applying GBI prediction, weighted prediction is applied, so taking into account that the error may not be evaluated correctly, the overall processing volume can be reduced by limiting the application conditions.

[0154] Similarly, in weighted prediction (described later), if either L0 prediction or L1 prediction to which DMVR processing is applied performs weighted prediction, dmvrFlag is set to FALSE. Specifically, if luma_weight_l0_flag[refIdxL0], which indicates whether or not the L0 predicted picture has a luminance weighting factor w0 and an offset o0, and luma_weight_l1_flag[refIdxL1], which indicates whether or not the L1 predicted picture has a luminance weighting factor w1 and an offset o1, are both FALSE, dmvrFlag is added to the TRUE conditions; otherwise, dmvrFlag is set to FALSE.

[0155] (BDOF determination by error threshold processing in DMVR) In the DMVR, a process for calculating the error between the L0 predicted image and the L1 predicted image is performed, and based on the value of the error at this time, a decision is made in advance as to whether or not to perform BDOF processing, which will be described later and is performed at a subsequent stage.

[0156] FIG. 12 is a flowchart illustrating the process of determining BDOF by error threshold processing in the DMVR.

[0157] First, the initial error generation unit 303752 sets bdofFlag to TRUE in advance (S1201). Next, the initial error generation unit 303752 derives centerSad (S1202) and determines whether the value of centerSad is equal to or greater than the threshold value (bH>>1)*bW*4 (S1203). If the value of centerSad is smaller than the threshold value, the initial error generation unit 303752 determines that the error is small and sets bdofFlag, which indicates whether or not to perform BDOF processing, to FALSE (S1204), thereby preventing BDOF processing from being performed in advance. Because this determination is the same as that of the initial error generation unit 30752 described above, the motion vector search unit 303753 and correction vector derivation unit 303754 are skipped, and DMVR processing is not performed either. If the value of centerSad is equal to or greater than the threshold, the motion vector search unit 303753 performs a modified motion vector search (S1205), and the modified vector derivation unit 303754 determines (S1206) whether the value of sadList[bestIdx], which is the SAD value of bestIdx that is the smallest as a result, is smaller than the threshold value (bH>>1)*bW*8. If the value of sadList[bestIdx] is smaller than the threshold, the modified vector derivation unit 303754 determines that the error is small and sets bdofFlag, which indicates whether or not to perform BDOF processing, to FALSE (S1207), thereby preventing BDOF processing from being performed in advance.

[0158] The threshold value in (S1206) is set to be equal to or greater than the threshold value in (S1203).

[0159] In DMVR processing, a process is required to calculate the error between the L0 predicted image and the L1 predicted image in order to search for a corrected motion vector. On the other hand, in BDOF processing, if the error is small, it is ineffective. Therefore, by adding such a process, it is possible to determine whether or not to perform BDOF without adding additional error calculations.

[0160] A BDOF unit (described later) determines whether bdofFlag is TRUE (S1208), and if Yes, the inter prediction parameter decoding unit 303 performs BDOF processing (S1209). If No, it determines not to perform BDOF processing on the block.

[0161] (Triangle forecast) Next, we will explain triangle prediction. In triangle prediction, a target CU is divided into two triangular prediction units, with a diagonal or anti-diagonal as the boundary. A predicted image for each triangular prediction unit is derived by applying a weighting mask process to each pixel of the predicted image of the target CU (a rectangular block including the triangular prediction unit) according to the pixel's position. For example, a triangular image can be derived from a rectangular image by multiplying a mask in which pixels in a triangular area within the rectangular area are set to 1 and pixels in areas other than the triangle are set to 0. Adaptive weighting of the predicted image is applied to both areas on either side of the diagonal, and one predicted image of the target CU (rectangular block) is derived by adaptive weighting using the two predicted images. This process is called triangle synthesis. Transformation (inverse transformation) and quantization (inverse quantization) processes are applied to the entire target CU. Note that triangle prediction is only applied in merge prediction mode or skip mode.

[0162] The triangle prediction unit 30377 derives prediction parameters corresponding to two triangular regions used for triangle prediction and outputs them to the inter-prediction image generation unit 309. To simplify the processing, triangle prediction may be configured not to use bi-prediction. In this case, inter-prediction parameters for unidirectional prediction are derived for one triangular region. Note that the derivation of the two predicted images and the synthesis using the predicted images are performed by the motion compensation unit 3091 and triangle synthesis unit 30952.

[0163] (MMVD Prediction Section 30376) The MMVD prediction unit 30376 performs processing in MMVD (Merge with Motion Vector Difference) mode. The MMVD mode is a mode in which a motion vector is obtained by adding a difference vector of a predetermined distance and a predetermined direction to a motion vector derived from a merge candidate (a motion vector derived from the motion vector of an adjacent block, etc.). In the MMVD mode, the MMVD prediction unit 30376 uses merge candidates and limits the value range of the difference vector to a predetermined distance (e.g., eight ways) and a predetermined direction (e.g., four directions, eight directions, etc.), thereby efficiently deriving a motion vector.

[0164] The loop filter 305 is a filter provided in the encoding loop that removes block distortion and ringing distortion to improve image quality. The loop filter 305 applies filters such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to the decoded image of the CU generated by the adder 312.

[0165] The reference picture memory 306 stores the decoded image of the CU generated by the adder 312 in a predetermined location for each current picture and current CU.

[0166] The prediction parameter memory 307 stores prediction parameters at a predetermined location for each CTU or CU to be decoded. Specifically, the prediction parameter memory 307 stores the parameters decoded by the parameter decoding unit 302 and the prediction mode predMode decoded by the entropy decoding unit 301.

[0167] The prediction image generation unit 308 receives input of a prediction mode predMode, prediction parameters, etc. The prediction image generation unit 308 also reads a reference picture from the reference picture memory 306. The prediction image generation unit 308 generates a prediction image of a block or sub-block using the prediction parameters and the read reference picture (reference picture block) in the prediction mode indicated by the prediction mode predMode. Here, the reference picture block is a set of pixels on the reference picture (usually rectangular, and therefore referred to as a block), and is an area referenced to generate a prediction image.

[0168] (Inter-prediction image generation unit 309) When the prediction mode predMode indicates inter prediction mode, the inter prediction image generation unit 309 generates a prediction image of a block or sub-block by inter prediction using the inter prediction parameters input from the inter prediction parameter decoding unit 303 and the read reference picture.

[0169] 13 is a schematic diagram showing the configuration of an inter-prediction image generation unit 309 included in the prediction image generation unit 308 according to this embodiment. The inter-prediction image generation unit 309 includes a motion compensation unit (prediction image generation device) 3091 and a synthesis unit 3095.

[0170] (motion compensation) The motion compensation unit 3091 (interpolated image generation unit) generates an interpolated image (motion-compensated image) by reading, from the reference picture memory 306, a block located at a position shifted by the motion vector mvLX from the position of the current block in the reference picture RefPicLX specified by the reference picture index refIdxLX, based on the inter-prediction parameters (prediction list use flag predFlagLX, reference picture index refIdxLX, motion vector mvLX) input from the inter-prediction parameter decoding unit 303. Here, if the precision of the motion vector mvLX is not integer precision, a filter for generating pixels at decimal positions called a motion compensation filter is applied to generate the interpolated image.

[0171] The motion compensation unit 3091 first derives the integer position (xInt, yInt) and phase (xFrac, yFrac) corresponding to the coordinates (x, y) in the prediction block using the following equations.

[0172] xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x xFrac = mvLX[0]&(MVPREC-1) yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y yFrac = mvLX[1]&(MVPREC-1) Here, (xPb, yPb) are the upper left coordinates of a bW*bH size block, where x = 0...bW-1 and y = 0...bH-1, and MVPREC indicates the precision of the motion vector mvLX (1 / MVPREC pixel precision). For example, MVPREC may be 16.

[0173] The motion compensation unit 3091 derives the temporary image temp[][] by performing horizontal interpolation on the reference picture refImg using an interpolation filter. In the following, Σ is the sum over k, where k=0..NTAP-1, shift1 is a normalization parameter that adjusts the value range, and offset1=1<<(shift1-1).

[0174] temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1 Next, the motion compensation unit 3091 performs vertical interpolation on the temporary image temp[][] to generate an interpolated image Pred Derive [][]. Below, Σ is the sum over k=0..NTAP-1, shift2 is a normalization parameter that adjusts the range of values, and offset2=1<<(shift2-1).

[0175] Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2 (Synthesis section) The synthesis unit 3095 generates a predicted image by referring to the interpolated image input from the motion compensation unit 3091, the inter-prediction parameters input from the inter-prediction parameter decoding unit 303, and the intra image input from the intra-prediction image generation unit 310, and outputs the generated predicted image to the addition unit 312.

[0176] The synthesis unit 3095 includes a combined intra / inter synthesis unit 30951, a triangle synthesis unit 30952, an OBMC unit 30953, and a BDOF unit 30956.

[0177] (Combined intra / inter synthesis processing) The combined intra / inter synthesis unit 30951 generates a predicted image by combining unidirectional prediction, skip mode, merge mode, and intra prediction in AMVP.

[0178] (Triangle synthesis processing) The triangle synthesis unit 30952 generates a predicted image using the above-mentioned triangle prediction.

[0179] (OBMC processing) The OBMC unit 30953 generates a predicted image using OBMC (Overlapped Block Motion Compensation) processing. The OBMC processing includes the following processes. An interpolated image (motion-compensated image) of the target sub-block is generated using an interpolated image (PU interpolated image) generated using the inter-prediction parameters added to the target sub-block and an interpolated image (OBMC interpolated image) generated using the motion parameters of the target sub-block's neighboring sub-blocks. A predicted image is generated by taking a weighted average of the OBMC interpolated image and the PU interpolated image.

[0180] (Weighted Prediction Unit 30954) The weighted prediction unit 309454 generates a predicted image for the block by multiplying the motion compensation images PredL0 and PredL1 by weighting factors. When one of the prediction list usage flags (predFlagL0 or predFlagL1) is 1 (uni-prediction) and weighted prediction is not used, the weighted prediction unit 309454 performs the following equation to adjust the motion compensation image PredLX (LX is L0 or L1) to the pixel bit depth bitDepth.

[0181] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredLX[x][y]+offset1)> >shift1) Here, shift1=Max(2, 14-bitDepth), offset1=1<<(shift1-1).

[0182] (Bidirectional prediction processing) Furthermore, when both prediction list usage flags (predFlagL0 and predFlagL1) are 1 (bi-prediction BiPred) and weighted prediction is not used, the following equation is processed to average the motion compensation images PredL0 and PredL1 and adjust them to the pixel bit rate.

[0183] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredL0[x][y]+PredL1[x][y]+offset2)> >shift2) Here, shift2=Max(3, 15-bitDepth), offset2=1<<(shift2-1). Hereinafter, this process will also be referred to as normal bidirectional prediction.

[0184] Furthermore, if uni-prediction is performed and a flag indicating whether or not the weighted prediction coefficient w0 and offset o0 exist in the L0 reference picture (luma_weight_l0_flag for luminance, chroma_weight_l0_flag for chrominance) is on, the weighted prediction unit 30954 derives the weighted prediction coefficient w0 and offset o0 from the coded data in the case of L0 prediction, and performs processing using the following equations.

[0185] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,((PredL0[x][y]*w0+(1<<(log2WD-1)))> >log2WD)+o0) In the case of L1 prediction, if a flag indicating whether the weighted prediction coefficient w1 and offset o1 exist in the L1 reference picture (luma_weight_l1_flag for luminance, chroma_weight_l1_flag for chrominance) is on, the weighted prediction unit 30954 derives the weighted prediction coefficient w1 and offset o1 from the encoded data and performs processing using the following equations.

[0186] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,((PredL1[x][y]*w1+(1<<(log2WD-1)))> >log2WD)+o1) Here, log2WD is a variable that explicitly sums the values ​​of Log2WeightDenom+shift1 sent in the slice header for luma and chroma.

[0187] (weighted bidirectional prediction processing) Furthermore, when bi-prediction BiPred is used and flags indicating whether or not weighted prediction coefficients and offsets exist (luma_weight_l0_flag, luma_weight_l1_flag for luminance, chroma_weight_l0_flag, chroma_weight_l1_flag for chrominance) perform weighted prediction, the weighted prediction unit 30954 derives weighted prediction coefficients w0, w1, o0, o1 from the coded data and performs processing using the following equations.

[0188] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,(PredL0[x][y]*w0+PredL1[x][y]*w1+((o0+o1+1)<<log2WD))> >(log2WD+1)) (GBI Part 30955) In the above-described weighted prediction, an example has been described in which a predicted image is generated by multiplying an interpolated image by a weighting factor. Here, another example will be described in which a predicted image is generated by multiplying an interpolated image by a weighting factor. In particular, a process of generating a predicted image using generalized bi-prediction (hereinafter referred to as GBI prediction) will be described. In GBI prediction, a predicted image Pred is generated by multiplying an L0 predicted image PredL0 and an L1 predicted image PredL1 in bi-prediction by weighting factors (w0, w1).

[0189] Furthermore, when generating a predicted image using GBI prediction, the GBI unit 30955 switches the weighting factors (w0, w1) for each coding unit. That is, the GBI unit 30954 of the inter-predicted image generation unit 309 sets a weighting factor for each coding unit. In GBI prediction, multiple weighting factor candidates are defined in advance, and gbiIdx is an index indicating the weighting factor to be used for the current block from among the multiple weighting factor candidates included in the table gbiWLut.

[0190] The GBI unit 30955 checks the flag gbiAppliedFlag indicating whether or not GBI prediction is used, and if it is FALSE, the motion compensation unit 3091 generates a predicted image using the following formula.

[0191] Pred[x][y]=Clip3(0,(1< <bitDepth)-1, (PredL0[x][y]+ PredL1[x][y]+offset2)>>shift2 ) Here, the initial state of gbiAppliedFlag is FALSE. The GBI unit 30955 sets gbiAppliedFlag to TRUE when an SPS flag indicating that GBI processing is possible is On and bidirectional prediction is used. Furthermore, as an additional (AND) condition, gbiAppliedFlag may be set to TRUE when gbiIdx, which is an index of the GBI prediction weight coefficient table gbiWLut, is not 0 (the index value when the weights of the L0 predicted image and the L1 predicted image are equal). Furthermore, as an additional (AND) condition, gbiAppliedFlag may be set to TRUE when the block size of the CU is equal to or larger than a certain value.

[0192] When gbiAppliedFlag is true, the GBI unit 30955 derives a predicted image Pred from the weights w0, w1 and PredL0, PredL1 using the following formula.

[0193] Pred[x][y]=Clip3(0,(1< <bitDepth)-1, (w0*PredL0[x][y]+w1*PredL1[x][y]+offset3)>>(shift2+3)) Here, the weighting coefficient w1 is a coefficient derived from the table iWLut[] = {4, 5, 3, 10, -2} by gbiIdx, which is explicitly indicated in the syntax. The weighting coefficient w0 is (8 - w1). When gbiIdx = 0, w0 = w1 = 4, which is equivalent to normal bidirectional prediction.

[0194] shift1, shift2, offset1, and offset2 are derived using the following formulas.

[0195] shift1=Max(2,14-bitDepth) shift2=Max(3,15-bitDepth)=shift1+1 offset1=1<<(shift1-1) offset2=1<<(shift2-1) offset3=1<<(shift2+2) Note that there may be multiple tables gbiWLut having different combinations of weighting factors, and the GBI unit 30955 may switch the table used to select a weighting factor depending on whether the picture structure is Low Delay (LB) or not.

[0196] When GBI prediction is used in AMVP prediction mode, the inter prediction parameter decoding unit 303 decodes gbiIdx and sends it to the GBI unit 30955. When GBI prediction is used in merge prediction mode, the inter prediction parameter decoding unit 303 decodes the merge index merge_idx, and the merge candidate derivation unit 30361 derives gbiIdx for each merge candidate. Specifically, the merge candidate derivation unit 30361 uses the weighting coefficients of the neighboring blocks used to derive the merge candidates as the weighting coefficients of the merge candidates to be used for the current block. In other words, in merge mode, the weighting coefficients used in the past are inherited as the weighting coefficients of the current block.

[0197] (Selection of prediction mode using GBI prediction) Next, a process of selecting a prediction mode using GBI prediction in the video decoding device 31 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the flow of the process of selecting a prediction mode in the video decoding device 31.

[0198] 14, the inter prediction parameter decoding unit 303 first decodes the skip flag (S1401). If the skip flag indicates the skip mode (YES in S1402), the prediction mode becomes the merge mode (S1403), and the inter prediction parameter decoding unit 303 decodes the merge index (S14031). If GBI prediction is used, the GBI unit 30955 derives the weighting coefficients derived for the merge candidates as the weighting coefficients for GBI prediction.

[0199] If the skip flag does not indicate skip mode (NO in S1402), the inter prediction parameter decoding unit 303 decodes the merge flag (S1407). If the merge flag indicates merge mode (YES in S1408), the prediction mode becomes merge mode (S1403), and the inter prediction parameter decoding unit 303 decodes the merge index (S14031). If GBI prediction is used, the GBI unit 30955 derives weighting factors derived for the merge candidates as weighting factors for GBI prediction.

[0200] If the merge flag does not indicate the merge mode (NO in S1408), the prediction mode is the AMVP mode (S1409).

[0201] In AMVP mode, the inter-prediction parameter decoding unit 303 decodes the inter-prediction identifier inter_pred_idc (S14090). Then, the inter-prediction parameter decoding unit 303 decodes the difference vector mvdLX (S14091). Then, the inter-prediction parameter decoding unit 303 decodes gbiIdx (S14092). If GBI prediction is used, the GBI unit 30955 selects a weighting factor w1 for GBI prediction from weighting factor candidates in the gbiWLut table.

[0202] (BDOF prediction) Next, a description will be given of details of prediction using BDOF processing (BDOF prediction) performed by the BDOF unit 30956. In bi-prediction mode, the BDOF unit 30956 generates a predicted image by referring to two predicted images (a first predicted image and a second predicted image) and a gradient correction term.

[0203] FIG. 15 is a flowchart illustrating the flow of processing for deriving a predicted image.

[0204] If the inter prediction parameter decoding unit 303 determines that L0 unidirectional prediction is being performed (inter_pred_idc is 0 in S1501), the motion compensation unit 3091 generates an L0 predicted image PredL0[x][y] (S1502). If the inter prediction parameter decoding unit 303 determines that L1 unidirectional prediction is being performed (inter_pred_idc is 1 in S1501), the motion compensation unit 3091 generates an L1 predicted image PredL1[x][y] (S1503). On the other hand, if the inter prediction parameter decoding unit 303 determines that the mode is bi-prediction (inter_pred_idc is 2 in S1501), the process continues to S1504 below. In S1504, the synthesis unit 3095 references bioAvailableFlag, which indicates whether or not to perform BDOF processing, and determines whether or not BDOF processing is required. If bioAvailableFlag indicates TRUE, the BDOF unit 30956 executes BDOF processing to generate a bidirectional predicted image (S1506). If bioAvailableFlag indicates FALSE, the synthesis unit 3095 generates a predicted image using normal bidirectional predicted image generation (S1505).

[0205] The inter prediction parameter decoding unit 303 may derive TRUE for bioAvailableFlag when the L0 reference image refImgL0 and the L1 reference image refImgL1 are different reference images and the two pictures are in opposite directions with respect to the current picture. Specifically, when the current picture is currPic, bioAvailableFlag indicates TRUE when the condition DiffPicOrderCnt(currPic, refImgL0)*DiffPicOrderCnt(currPic, refImgL1)<0 is satisfied. Here, DiffPicOrderCnt() is a function that derives the difference between the POC (Picture Order Count: picture display order) of two images as follows:

[0206] DiffPicOrderCnt(picA,picB) = PicOrderCnt(picA)-PicOrderCnt(picB) A condition that the motion vector of the current block is not a sub-block-based motion vector may be added as a condition for indicating that bioAvailableFlag is TRUE.

[0207] Furthermore, a condition that the motion vector of the current picture is not a sub-block-based motion vector may be added as a condition for bioAvailableFlag to indicate TRUE.

[0208] Furthermore, a condition that the sum of absolute differences between the L0 predicted image and the L1 predicted image of two predicted blocks is equal to or greater than a predetermined value may be added as a condition for bioAvailableFlag to indicate TRUE.

[0209] Furthermore, a condition that the predicted image creation mode is a block-based predicted image creation mode may be added as a condition for bioAvailableFlag to indicate TRUE.

[0210] Furthermore, a condition that neither L0 prediction nor L1 prediction performs weighted prediction may be added as a condition for bioAvailableFlag to indicate TRUE. Specifically, bioAvailableFlag indicates TRUE when both luma_weight_l0_flag[refIdxL0], which indicates whether the L0 predicted picture has a luma weighting factor w0 and an offset o0, and luma_weight_l1_flag[refIdxL1], which indicates whether the L1 predicted picture has a luma weighting factor w1 and an offset o1, are FALSE.

[0211] FIG. 16 is a schematic diagram showing the configuration of the BDOF unit 30956. Using FIG. 16, the specific contents of the processing performed by the BDOF unit 30956 will be described. The BDOF processing unit 30956 includes an L0,L1 predicted image generation unit 309561, a gradient image generation unit 309562, a correlation parameter calculation unit 309563, a motion compensation modified value derivation unit 309564, and a BDOF predicted image generation unit 309565. The BDOF unit 30956 generates a predicted image from the interpolated image received from the motion compensation unit 3091 and the inter prediction parameters received from the inter prediction parameter decoding unit 303, and outputs the generated predicted image to the adder 312. Note that the process of deriving a motion compensation modified value modBIO (motion compensation modified image) from the gradient image and correcting and deriving the predicted images PredL0 and PredL1 is called bidirectional gradient change processing.

[0212] FIG. 17 is a diagram showing an example of an area where padding is performed. First, the L0,L1 predicted image generation unit 309561 generates an L0,L1 predicted image used in BDOF processing. The BDOF unit 30956 performs BDOF processing based on the L0,L1 predicted image for each CU or sub-CU shown in FIG. 17. However, to calculate the gradient, two extra pixels of interpolated image information surrounding the target CU or sub-CU are required. This interpolated image information is generated using adjacent integer pixels rather than a normal interpolation filter for gradient image generation, as described below. In other cases, this area is used as a padding area by copying the surrounding pixels, as with the outside of the picture. Furthermore, the unit of BDOF processing is NxN pixels, which is equal to or less than the CU or sub-CU unit, and the processing itself is performed using (N+2)x(N+2) pixels, which includes one surrounding pixel.

[0213] The gradient image generation unit 309562 generates a gradient image. Gradient change (Optical Flow) assumes that the pixel value of each point does not change, only its position. This can be expressed as follows using the change in pixel value I in the horizontal direction (horizontal gradient value lx) and its position change Vx, the change in pixel value I in the vertical direction (vertical gradient value ly) and its position change Vy, and the time change in pixel value I lt.

[0214] lx * Vx + ly * Vy + lt = 0 Hereafter, the change in position (Vx, Vy) is called the correction weight vector (u, v).

[0215] Specifically, the gradient image generation unit 309562 derives gradient images lx0, ly0, lx1, and ly1 from the following equations: lx0 and lx1 indicate gradients along the horizontal direction, and ly0 and ly1 indicate gradients along the vertical direction.

[0216] lx0[x][y] = (PredL0[x+1][y]-PredL0[x-1][y])>>shift1 ly0[x][y] = (PredL0[x][y+1]-PredL0[x][y-1])>>shift1 lx1[x][y] = (PredL1[x+1][y]-PredL1[x-1][y])>>shift1 ly1[x][y] = (PredL1[x][y+1]-PredL1[x][y-1])>>shift1 Here, shift1=Max(2, 14-bitDepth).

[0217] Next, the correlation parameter calculation unit 309563 derives gradient product sums s1, s2, s3, s5, and s6 of (N+2) x (N+2) pixels using one surrounding pixel for each N x N pixel block in each CU.

[0218] s1 = sum(phiX[x][y]* phiX[x][y]) s2 = sum(phiX[x][y]* phiY[x][y]) s3 = sum(-theta[x][y]* phiX[x][y]) s5 = sum(phiY[x][y]* phiY[x][y]) s6 = sum(-theta[x][y]* phiY[x][y]) Here, sum(a) represents the sum of a for the coordinates (x, y) within the (N+2) x (N+2) pixel block. theta[x][y]= -(PredL1[x][y]>>shift4)+(PredL0[x][y]>>shift4) phiX[x][y] = (lx1[x][y] + lx0[x][y])>>shift5 phiY[x][y] = (ly1[x][y] + ly0[x][y])>>shift5 where: shift4=Min(8,bitDepth-4) shift5=Min(5,bitDepth-7) Let's say.

[0219] Next, the motion compensation correction value derivation unit 309564 derives a correction weight vector (u, v) in units of NxN pixels using the derived gradient product sums s1, s2, s3, s5, and s6.

[0220] u = (s3<<3)>>log2(s1) v = ((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1))>>log2(s5) Here, s2m=s2>>12 and s2s=s2&((1<<12)-1).

[0221] The range of u and v may be further restricted using clipping as follows:

[0222] u = s1>0?Clip3(-th,th,-(s3<<3)>>floor(log2(s1))):0 v = s5>0?Clip3(-th,th,((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1))>>floor(log2(s5))):0 Here, th = Max(2, 1<<(13-bitDepth)). Because the value of th needs to be calculated in conjunction with shift1, consider the case where the pixel bit length bitDepth is greater than 12 bits.

[0223] The motion compensation modification value derivation unit 309564 derives the motion compensation modification value modBIO[x][y] of NxN pixels using the correction weight vector (u, v) in units of NxN pixels and the gradient images lx0, ly0, lx1, and ly1.

[0224] modBIO[x][y] = ((lx1[x][y]-lx0[x][y])*u+(ly1[x][y]-ly0[x][y])*v+1)>>1 (Equation A3) Alternatively, modBIO can be derived using the round function as follows:

[0225] modBIO[x][y] = Round(((lx1[x][y]-lx0[x][y])*u)>>1)+Round(((ly1[x][y]-ly0[x][y])*v)>>1) The BDOF predicted image generation unit 309565 uses the above parameters to derive the pixel value Pred of the predicted image of NxN pixels according to the following equation.

[0226] At this time, the BDOF predicted image generation unit 309565 uses the above parameters to derive the pixel value Pred of the predicted image of NxN pixels according to the following equation.

[0227] Pred[x][y] = Clip3(0, (1< <bitDepth)-1,( PredL0[x][y]+PredL1[x][y]+modBIO[x][y] +offset2)>>shift2) Here, shift2=Max(3, 15-bitDepth), offset2=1<<(shift2-1).

[0228] Then, the BDOF predicted image generation unit 309565 outputs the generated predicted image of the block to the addition unit 312.

[0229] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the entropy decoding unit 301 to obtain transform coefficients. These quantized transform coefficients are coefficients obtained in the encoding process by performing frequency transform such as DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform) on prediction errors and quantizing them. The inverse quantization and inverse transform unit 311 performs inverse frequency transform such as inverse DCT or inverse DST on the obtained transform coefficients to calculate prediction errors. The inverse quantization and inverse transform unit 311 outputs the prediction errors to the adder 312. The inverse quantization and inverse transform unit 311 sets all prediction errors to 0 when skip_flag is 1 or cu_cbp is 0.

[0230] The adder 312 generates a decoded image for the block by adding, for each pixel, the predicted image for the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311. The adder 312 stores the decoded image for the block in the reference picture memory 306 and also outputs it to the loop filter 305.

[0231] (Configuration of a video encoding device) Next, the configuration of the video encoding device 11 according to this embodiment will be described. Fig. 18 is a schematic diagram showing the configuration of the video encoding device 11 according to this embodiment. The video encoding device 11 includes a prediction image generation unit 101, a subtraction unit 102, a transformation and quantization unit 103, an inverse quantization and inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, a coding parameter determination unit 110, a parameter coding unit 111, and an entropy coding unit 104.

[0232] The predicted image generation unit 101 generates a predicted image for each CU, which is an area obtained by dividing each picture of the image T. The predicted image generation unit 101 operates in the same manner as the predicted image generation unit 308 already described, and therefore a description thereof will be omitted.

[0233] The subtraction unit 102 generates a prediction error by subtracting pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

[0234] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing them. The transform / quantization unit 103 outputs the quantized transform coefficients to the entropy coding unit 104 and the inverse quantization / inverse transform unit 105.

[0235] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (FIG. 7) in the video decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to the adder .

[0236] The entropy coding unit 104 receives the quantized transform coefficients from the transform / quantization unit 103 and the coding parameters from the parameter coding unit 111. The coding parameters include, for example, codes such as a reference picture index refIdxLX, a prediction vector index mvp_LX_idx, a difference vector mvdLX, a motion vector precision mode amvr_mode, a prediction mode predMode, and a merge index merge_idx.

[0237] The entropy coding unit 104 entropy codes the division information, prediction parameters, quantized transform coefficients, etc. to generate and output a coded stream Te.

[0238] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, a CU coding unit 1112 (prediction mode coding unit), and a parameter coding unit 112, all of which are not shown. The CU coding unit 1112 further includes a TU coding unit 1114.

[0239] The operation of each module will be outlined below: The parameter coding unit 111 performs coding processing of parameters such as header information, division information, prediction information, and quantized transform coefficients.

[0240] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information and the like from the encoded data.

[0241] The CU encoding unit 1112 encodes CU information, prediction information, the TU split flag split_transform_flag, the CU residual flags cbf_cb, cbf_cr, and cbf_luma, and the like.

[0242] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information (quantization correction value) and the quantized prediction error (residual_coding).

[0243] The CT information encoding unit 1111 and the CU encoding unit 1112 output syntax elements such as inter-prediction parameters (prediction mode predMode, merge flag merge_flag, merge index merge_idx, inter-prediction identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX), intra-prediction parameters (prev_intra_luma_pred_flag, mpm_idx, rem_selected_mode_flag, rem_selected_mode, rem_non_selected_mode), and quantized transform coefficients to the entropy encoding unit 104.

[0244] (Configuration of the parameter encoding unit) The parameter coding unit 112 derives inter prediction parameters based on the prediction parameters input from the coding parameter determination unit 110. The parameter coding unit 112 includes a configuration that is partially the same as the configuration used by the inter prediction parameter decoding unit 303 to derive inter prediction parameters.

[0245] 19 is a schematic diagram showing the configuration of the parameter coding unit 112. The configuration of the parameter coding unit 112 will be described. As shown in FIG. 19, the parameter coding unit 112 is configured to include a parameter coding control unit 1121, a merge prediction unit 30374, a sub-block prediction unit (affine prediction unit) 30372, a DMVR unit 30375, an MMVD prediction unit 30376, a triangle prediction unit 30377, an AMVP prediction parameter derivation unit 3032, and a subtraction unit 1123. The merge prediction unit 30374 is equipped with a merge prediction parameter derivation unit 3036. The parameter coding control unit 1121 includes a merge index derivation unit 11211 and a vector candidate index derivation unit 11212. Furthermore, the parameter coding control unit 1121 derives merge_idx, affine_flag, base_candidate_idx, distance_idx, direction_idx, etc. in a merge index derivation unit 11211, and derives mvpLX, etc. in a vector candidate index derivation unit 11212. The merge prediction parameter derivation unit 3036, AMVP prediction parameter derivation unit 3032, affine prediction unit 30372, MMVD prediction unit 30376, and triangle prediction unit 30377 may be collectively referred to as a motion vector derivation unit (motion vector derivation device). The parameter coding unit 112 outputs the motion vector mvLX, reference picture index refIdxLX, inter prediction identifier inter_pred_idc, or information indicating these to the predicted image generation unit 101. Furthermore, parameter coding unit 112 outputs merge_flag, skip_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_lX_idx, mvdLX, amvr_mode, and affine_flag to entropy coding unit 104.

[0246] FIG. 20 is a diagram showing an example of the number of search distance candidates and the number of derivation direction candidates in the video encoding device 11. The parameter encoding control unit 1121 derives parameters (base_candidate_idx, distance_idx, direction_idx, etc.) representing a difference vector and outputs them to the MMVD prediction unit 30376. The derivation of a difference vector in the parameter encoding control unit 1121 will be described with reference to FIG. 20. The black circle in the center of the diagram indicates the position indicated by the predicted vector mvpLX, and eight search distances are searched in each of four directions (up, down, left, and right) centered on this position. mvpLX is the motion vector of the first and second candidate in the merge candidate list, and a search is performed for each. There are two predicted vectors in the merge candidate list (the first and second in the list), the search distance is 8, and the search direction is 4, so there are 64 candidates for mvdLX. The mvdLX with the lowest cost among the searched candidates is represented by base_candidate_idx, distance_idx, and direction_idx.

[0247] In this way, the MMVD mode is a mode that searches for limited candidate points centered on a predicted vector and derives an appropriate motion vector.

[0248] The merge index derivation unit 11211 derives a merge index merge_idx and outputs it to the merge prediction parameter derivation unit 3036 (merge prediction unit). In the MMVD mode, the merge index derivation unit 11211 sets the value of the merge index merge_idx to the same value as the value of base_candidate_idx. The vector candidate index derivation unit 11212 derives a predicted vector index mvp_lX_idx.

[0249] The merge prediction parameter derivation unit 3036 derives inter prediction parameters based on the merge index merge_idx.

[0250] The AMVP prediction parameter derivation unit 3032 derives a prediction vector mvpLX based on the motion vector mvLX. The AMVP prediction parameter derivation unit 3032 outputs the prediction vector mvpLX to the subtraction unit 1123. Note that the reference picture index refIdxLX and the prediction vector index mvp_lX_idx are output to the entropy coding unit 104.

[0251] The affine prediction unit 30372 derives inter prediction parameters (affine prediction parameters) for the sub-block.

[0252] The subtraction unit 1123 generates a difference vector mvdLX by subtracting the prediction vector mvpLX output from the AMVP prediction parameter derivation unit 3032 from the motion vector mvLX input from the coding parameter determination unit 110. The difference vector mvdLX is output to the entropy coding unit 104.

[0253] The adder 106 generates a decoded image by adding, for each pixel, the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 and the prediction error input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in the reference picture memory 109.

[0254] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters, and may be configured with only a deblocking filter, for example.

[0255] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0256] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined location for each current picture and CU.

[0257] The coding parameter determination unit 110 selects one set of coding parameters from among a plurality of sets of coding parameters. The coding parameters are the above-mentioned QT, BT, or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0258] The coding parameter determination unit 110 calculates an RD cost value indicating the amount of information and the coding error for each of the multiple sets. The coding parameter determination unit 110 selects the set of coding parameters that minimizes the calculated cost value. As a result, the entropy coding unit 104 outputs the selected set of coding parameters as the coded stream Te. The coding parameter determination unit 110 stores the determined coding parameters in the prediction parameter memory 108.

[0259] Note that parts of the video encoding device 11 and the video decoding device 31 in the above-described embodiments, such as the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generation unit 308, the inverse quantization and inverse transform unit 311, the adder unit 312, the predicted image generation unit 101, the subtractor unit 102, the transform and quantization unit 103, the entropy encoding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the encoding parameter determination unit 110, and the parameter encoding unit 111, may be implemented by a computer. In this case, a program for implementing these control functions may be recorded on a computer-readable recording medium, and the program may be read and executed by a computer system. Note that the term "computer system" used here refers to a computer system built into either the video encoding device 11 or the video decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0260] Furthermore, part or all of the video encoding device 11 and video decoding device 31 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 11 and video decoding device 31 may be individually implemented as a processor, or part or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0261] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like can be made within the scope that does not deviate from the gist of the present invention.

[0262] [Application example] The above-described video encoding device 11 and video decoding device 31 can be mounted on various devices that transmit, receive, record, and play back video. The video may be a natural video captured by a camera or the like, or an artificial video (including CG and GUI) generated by a computer or the like.

[0263] First, it will be explained with reference to FIG. 2 that the above-described video encoding device 11 and video decoding device 31 can be used for transmitting and receiving video.

[0264] 2(a) is a block diagram showing the configuration of a transmitting device PROD_A equipped with a video encoding device 11. As shown in the figure, the transmitting device PROD_A includes an encoding unit PROD_A1 that encodes video to obtain encoded data, a modulation unit PROD_A2 that modulates a carrier wave with the encoded data obtained by the encoding unit PROD_A1 to obtain a modulated signal, and a transmitting unit PROD_A3 that transmits the modulated signal obtained by the modulation unit PROD_A2. The above-described video encoding device 11 is used as this encoding unit PROD_A1.

[0265] The transmitting device PROD_A may further include, as input sources of moving images to be input to the encoding unit PROD_A1, a camera PROD_A4 that captures moving images, a recording medium PROD_A5 on which moving images are recorded, an input terminal PROD_A6 for inputting moving images from the outside, and an image processing unit A7 that generates or processes images. In the figure, the transmitting device PROD_A is shown as having all of these components, but some of them may be omitted.

[0266] The recording medium PROD_A5 may record unencoded video, or may record video encoded using a recording encoding method that is different from the transmission encoding method. In the latter case, a decoding unit (not shown) that decodes the encoded data read from the recording medium PROD_A5 according to the recording encoding method may be interposed between the recording medium PROD_A5 and the encoding unit PROD_A1.

[0267] 2(b) is a block diagram showing the configuration of a receiving device PROD_B equipped with a video decoding device 31. As shown in the figure, the receiving device PROD_B includes a receiving unit PROD_B1 that receives a modulated signal, a demodulating unit PROD_B2 that obtains coded data by demodulating the modulated signal received by the receiving unit PROD_B1, and a decoding unit PROD_B3 that obtains video by decoding the coded data obtained by the demodulating unit PROD_B2. The above-mentioned video decoding device 31 is used as this decoding unit PROD_B3.

[0268] The receiving device PROD_B may further include, as output destinations of the moving image output by the decoding unit PROD_B3, a display PROD_B4 for displaying the moving image, a recording medium PROD_B5 for recording the moving image, and an output terminal PROD_B6 for outputting the moving image to the outside. In the figure, the receiving device PROD_B is shown as having all of these components, but some of them may be omitted.

[0269] The recording medium PROD_B5 may be for recording unencoded video, or may be encoded using an encoding method for recording that is different from the encoding method for transmission. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_B3 and the recording medium PROD_B5, which encodes the video acquired from the decoding unit PROD_B3 according to the encoding method for recording.

[0270] The transmission medium for transmitting the modulated signal may be wireless or wired. The transmission mode for transmitting the modulated signal may be broadcast (here, this refers to a transmission mode in which the destination is not specified in advance) or communication (here, this refers to a transmission mode in which the destination is specified in advance). In other words, the transmission of the modulated signal may be realized by any of wireless broadcasting, wired broadcasting, wireless communication, and wired communication.

[0271] For example, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for terrestrial digital broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via wireless broadcasting. Also, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for cable television broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via cable broadcasting.

[0272] Furthermore, a server (such as a workstation) / client (such as a television receiver, personal computer, or smartphone) of an Internet-based VOD (Video On Demand) service or video sharing service is an example of a transmitter PROD_A / receiver PROD_B that transmits and receives modulated signals via communication (usually, a LAN uses either a wireless or wired transmission medium, while a WAN uses a wired transmission medium). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Smartphones also include multi-function mobile phone terminals.

[0273] The client of the video hosting service has the function of decoding the encoded data downloaded from the server and displaying it on a display, as well as the function of encoding the video images captured by a camera and uploading them to the server. In other words, the client of the video hosting service functions as both the transmitting device PROD_A and the receiving device PROD_B.

[0274] Next, it will be explained with reference to FIG. 3 that the above-described video encoding device 11 and video decoding device 31 can be used for recording and reproducing video.

[0275] 3(a) is a block diagram showing the configuration of a recording device PROD_C equipped with the above-mentioned video encoding device 11. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that obtains encoded data by encoding video, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 onto a recording medium PROD_M. The above-mentioned video encoding device 11 is used as this encoding unit PROD_C1.

[0276] The recording medium PROD_M may be (1) a type that is built into the recording device PROD_C, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), (2) a type that is connected to the recording device PROD_C, such as an SD memory card or USB (Universal Serial Bus) flash memory, or (3) a type that is loaded into a drive device (not shown) built into the recording device PROD_C, such as a DVD (Digital Versatile Disc: registered trademark) or BD (Blu-ray Disc: registered trademark).

[0277] The recording device PROD_C may further include, as input sources of moving images to be input to the encoding unit PROD_C1, a camera PROD_C3 that captures moving images, an input terminal PROD_C4 for inputting moving images from outside, a receiving unit PROD_C5 for receiving moving images, and an image processing unit PROD_C6 that generates or processes images. In the figure, the recording device PROD_C is shown as having all of these components, but some of them may be omitted.

[0278] The receiving unit PROD_C5 may receive unencoded video, or may receive encoded data encoded by a transmission encoding method different from the recording encoding method. In the latter case, a transmission decoding unit (not shown) that decodes the encoded data encoded by the transmission encoding method may be interposed between the receiving unit PROD_C5 and the encoding unit PROD_C1.

[0279] Examples of such a recording device PROD_C include a DVD recorder, a BD recorder, and an HDD (Hard Disk Drive) recorder (in this case, the input terminal PROD_C4 or the receiving unit PROD_C5 is the main input source of the moving image).Other examples of such a recording device PROD_C include a camcorder (in this case, the camera PROD_C3 is the main input source of the moving image), a personal computer (in this case, the receiving unit PROD_C5 or the image processing unit C6 is the main input source of the moving image), and a smartphone (in this case, the camera PROD_C3 or the receiving unit PROD_C5 is the main input source of the moving image).

[0280] 3(b) is a block diagram showing the configuration of a playback device PROD_D equipped with the above-mentioned video decoding device 31. As shown in the figure, the playback device PROD_D includes a reading unit PROD_D1 that reads coded data written on a recording medium PROD_M, and a decoding unit PROD_D2 that obtains video by decoding the coded data read by the reading unit PROD_D1. The above-mentioned video decoding device 31 is used as this decoding unit PROD_D2.

[0281] The recording medium PROD_M may be (1) a type that is built into the playback device PROD_D, such as an HDD or SSD, (2) a type that is connected to the playback device PROD_D, such as an SD memory card or USB flash memory, or (3) a type that is loaded into a drive device (not shown) built into the playback device PROD_D, such as a DVD or BD.

[0282] The playback device PROD_D may further include, as output destinations for the video output by the decoding unit PROD_D2, a display PROD_D3 that displays the video, an output terminal PROD_D4 that outputs the video to the outside, and a transmission unit PROD_D5 that transmits the video. Although the figure shows an example of a configuration in which the playback device PROD_D includes all of these, some of them may be omitted.

[0283] The transmitting unit PROD_D5 may transmit unencoded video, or may transmit encoded data encoded by a transmission encoding method different from the recording encoding method. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_D2 and the transmitting unit PROD_D5, which encodes the video by the transmission encoding method.

[0284] Examples of such a playback device PROD_D include a DVD player, a BD player, and an HDD player (in this case, the output terminal PROD_D4 to which a television receiver or the like is connected is the main output destination of moving images). Other examples of such a playback device PROD_D include a television receiver (in this case, the display PROD_D3 is the main output destination of moving images), a digital signage (also called an electronic billboard or electronic bulletin board, and the display PROD_D3 or the transmitter PROD_D5 is the main output destination of moving images), a desktop PC (in this case, the output terminal PROD_D4 or the transmitter PROD_D5 is the main output destination of moving images), a laptop or tablet PC (in this case, the display PROD_D3 or the transmitter PROD_D5 is the main output destination of moving images), and a smartphone (in this case, the display PROD_D3 or the transmitter PROD_D5 is the main output destination of moving images).

[0285] (hardware and software implementations) Furthermore, each block of the above-mentioned video decoding device 31 and video encoding device 11 may be realized in hardware by a logic circuit formed on an integrated circuit (IC chip), or may be realized in software using a CPU (Central Processing Unit).

[0286] In the latter case, each of the above devices includes a CPU that executes instructions of a program that realizes each function, a ROM (Read Only Memory) that stores the program, and a RAM (Random Access Memory) that expands the program. The device may include a storage device (recording medium) such as an access memory (access memory) for storing the above programs and various data. The object of the embodiment of the present invention can also be achieved by supplying each of the devices with a recording medium on which program code (executable program, intermediate code program, source program) of the control program for each of the devices, which is software for realizing the above functions, is recorded in a computer-readable manner, and having the computer (or CPU or MPU) read and execute the program code recorded on the recording medium.

[0287] Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy disks (registered trademark) and hard disks, and CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical discs), MDs (Mini Discs), DVDs (Digital Versatile Discs: registered trademark), CD-Rs (CD Recordable), and Blu-ray discs. Discs including optical discs such as EEPROM (Erasable Programmable Read-Only Memory: registered trademark), cards such as IC cards (including memory cards) / optical cards, semiconductor memories such as mask ROM / EPROM (Erasable Programmable Read-Only Memory) / EEPROM (Electrically Erasable and Programmable Read-Only Memory: registered trademark) / flash ROM, or logic circuits such as PLD (Programmable logic device) and FPGA (Field Programmable Gate Array) can be used.

[0288] Furthermore, each of the above devices may be configured to be connectable to a communications network, and the program code may be supplied via the communications network. This communications network may be any network capable of transmitting the program code. For example, the Internet, an intranet, an extranet, a local area network (LAN), an integrated services digital network (ISDN), a value-added network (VAN), a community antenna television / cable television (CATV) communications network, a virtual private network, a telephone line network, a mobile communications network, a satellite communications network, etc. Furthermore, the transmission media constituting this communications network may be any medium capable of transmitting the program code, and are not limited to any particular configuration or type. For example, the present invention can be used in wired communication such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carrier, cable TV line, telephone line, or ADSL (Asymmetric Digital Subscriber Line) line, or in wireless communication such as IrDA (Infrared Data Association), infrared such as that used in remote controls, Bluetooth (registered trademark), IEEE 802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance: registered trademark), mobile phone networks, satellite lines, or terrestrial digital broadcasting networks. Note that the present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

[0289] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention.

[0290] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to Japanese Patent Application No. 2019-043097, filed on March 8, 2018, the entire contents of which are incorporated herein by reference.

[0291] (summary) The present invention can also be expressed as follows.

[0292] An image decoding device according to one aspect of the present invention includes: an inter prediction parameter decoding unit having a process of correcting two motion vectors from errors between two predicted images; If neither of the two predicted images is a weighted prediction, the two motion vectors are corrected.

[0293] Furthermore, an image encoding device according to an aspect of the present invention includes: an inter-prediction parameter coding unit having a process of correcting two motion vectors from errors between two predicted images; If neither of the two predicted images is a weighted prediction, the two motion vectors are corrected.

[0294] By adopting such a configuration, if weighted prediction is applied, the error cannot be evaluated correctly and no effect can be obtained. Therefore, by limiting the application conditions, the overall processing volume can be reduced.

[0295] Furthermore, an image decoding device according to an aspect of the present invention includes: an inter prediction parameter decoding unit that corrects two motion vectors from error values ​​of two predicted images; a bidirectional gradient variation processing unit that generates a predicted image using parameters decoded by the inter prediction parameter decoding unit and gradient images derived from two generated interpolated images; The error values ​​of the two predicted images are used to determine whether or not to apply the processing by the bidirectional gradient variation processing unit.

[0296] Furthermore, an image encoding device according to an aspect of the present invention includes: an inter-prediction parameter coding unit having a process of correcting two motion vectors from error values ​​of two predicted images; a bidirectional gradient variation processing unit that generates a predicted image using parameters decoded by the inter prediction parameter decoding unit and gradient images derived from two generated interpolated images; The error values ​​of the two predicted images are used to determine whether or not to apply the processing by the bidirectional gradient variation processing unit.

[0297] With this configuration, it is necessary to calculate an error value to correct the motion vector, while the bidirectional gradient change processing unit is ineffective when the error is small. Therefore, by adding this processing, it becomes possible to determine whether to apply processing by the bidirectional gradient change processing unit without adding an additional error value, thereby reducing the overall processing volume.

[0298] The present invention can also be expressed as follows.

[0299] An image decoding device according to one aspect of the present invention includes: Using two reference pictures and motion vectors mvL0 and mvL1, DMVR (Decoder Side Motion A video decoding device that performs Vector Refinement (Vector Refinement) processing, a DMVR unit that executes the DMVR processing when a dmvrFlag indicating whether the DMVR processing is to be performed is TRUE; a weighted prediction unit that performs weighted prediction using a first weighting factor, a first offset, a second weighting factor, and a second offset; the DMVR unit sets the dmvrFlag to TRUE based on a predetermined condition, wherein the predetermined condition for setting the dmvrFlag to TRUE includes that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE; the luma_weight_l0_flag[refIdxL0] indicates whether the first weighting factor and the first offset of luma corresponding to the L0 reference picture indicated by the reference picture index refIdxL0 are present; The luma_weight_l1_flag[refIdxL1] indicates whether or not the second weighting factor and the second offset for luma corresponding to the L1 reference picture indicated by the reference picture index refIdxL1 exist.

[0300] Furthermore, an image encoding device according to an aspect of the present invention includes: Using two reference pictures and motion vectors mvL0 and mvL1, DMVR (Decoder Side Motion A video encoding device that performs Vector Refinement (Vector Refinement) processing, a DMVR unit that executes the DMVR processing when a dmvrFlag indicating whether the DMVR processing is to be performed is TRUE; a weighted prediction unit that performs weighted prediction using a first weighting factor, a first offset, a second weighting factor, and a second offset; the DMVR unit sets the dmvrFlag to TRUE based on a predetermined condition, wherein the predetermined condition for setting the dmvrFlag to TRUE includes that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE; the luma_weight_l0_flag[refIdxL0] indicates whether the first weighting factor and the first offset of luma corresponding to the L0 reference picture indicated by the reference picture index refIdxL0 are present; The luma_weight_l1_flag[refIdxL1] indicates whether or not the second weighting factor and the second offset for luma corresponding to the L1 reference picture indicated by the reference picture index refIdxL1 exist. [Industrial Applicability]

[0301] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data obtained by coding image data, and a video coding device that generates coded data obtained by coding image data, and can also be suitably applied to the data structure of coded data generated by a video coding device and referenced by the video decoding device. [Explanation of symbols]

[0302] 31 Image decoding device 301 Entropy Decoding Unit 302 Parameter Decoding Unit 3020 Header Decoding Unit 303 Inter-prediction parameter decoding unit 304 Intra prediction parameter decoding unit 308 Prediction Image Generation Unit 309 Inter-prediction image generation unit 310 Intra-prediction image generation unit 311 Inverse quantization and inverse transformation unit 312 Addition section 11 Image encoding device 101 Prediction image generation unit 102 Subtraction section 103 Transformation and Quantization Unit 104 Entropy coding unit 105 Inverse quantization and inverse transformation unit 107 Loop Filter 110 Encoding parameter determination unit 111 Parameter Encoding Unit 112 Parameter Encoding Unit 1110 Header encoding part 1111 CT information encoder 1112 CU encoding unit (prediction mode encoding unit) 1114 TU encoding section 3091 Motion Compensation Unit 3095 Synthesis Section 30951 Combined Intra / inter synthesis section 30952 Triangle synthesis section 30953 OBMC Department 30954 Weighted Prediction Unit 30955 GBI Department 30956 BDOF section 309561 L0,L1 predicted image generation unit 309562 Gradient image generation part 309563 Correlation parameter calculation unit 309564 Motion compensation correction value derivation part 309565 BDOF predicted image generation unit

Claims

1. a video encoding device for generating an encoded stream obtained by encoding a target image to be encoded; a video decoding device that decodes the encoded stream generated by the video encoding device, the video encoding device and the video decoding device are capable of performing a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector; The video encoding device and the video decoding device each include: a DMVR unit that executes the DMVR process based on a value of dmvrFlag; The DMVR unit includes at least (1) The value of the merge flag is TRUE; (2) predFlagL0 and predFlagL1 are both TRUE; (3) mmvd_flag is FALSE, (4) The difference in POC between the current picture and the L0 reference picture is equal to the difference in POC between the L1 reference picture and the current picture; (5) The width of the processing block is 8 or more. (6) bcwIdx is 0, and (7) The value of luma_weight_l0_flag indicating whether or not a weighting factor for L0 prediction exists and the value of luma_weight_l1_flag indicating whether or not a weighting factor for L1 prediction exists are FALSE; setting the value of the dmvrFlag to TRUE based on If the value of the dmvrFlag is TRUE, the system executes the DMVR processing.

2. A data structure of an encoded stream decoded by a video decoding device that performs a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector, The video decoding device a DMVR unit that executes the DMVR process based on a value of dmvrFlag; The data structure of the coded stream is luma_weight_l0_flag indicating whether a weighting factor for L0 prediction exists or not and luma_weight_l1_flag indicating whether a weighting factor for L1 prediction exists or not, and in decoding by the video decoding device, at least (1) a value of a merge flag is TRUE, (2) predFlagL0 and predFlagL1 are both TRUE, (3) mmvd_flag is FALSE, and (4) a difference in POC between the current picture and the L0 reference picture is less than or equal to the difference in POC between the L1 reference picture and the current picture. (4) the width of the processing block is 8 or more; (5) bcwIdx is 0; and (6) the value of luma_weight_l0_flag indicating whether a weighting factor for L0 prediction exists and the value of luma_weight_l1_flag indicating whether a weighting factor for L1 prediction exists are FALSE.

3. 1. A method for generating an encoded stream by a video decoding device that performs a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector, comprising: performing the DMVR processing based on a value of dmvrFlag; setting the dmvrFlag to TRUE based on a predetermined condition, wherein the predetermined condition for setting the dmvrFlag to TRUE includes both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] being FALSE; The luma_weight_l0_flag[refIdxL0] indicates whether there is a weighting factor corresponding to the L0 reference picture indicated by the reference picture index refIdxL0; The method for generating an encoded stream, wherein the luma_weight_l1_flag[refIdxL1] indicates whether or not there is a weighting factor corresponding to the L1 reference picture indicated by the reference picture index refIdxL1.

4. A video decoding device capable of performing a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector, a DMVR unit that executes the DMVR process based on a value of dmvrFlag; The DMVR unit includes at least (1) The value of the merge flag is TRUE; (2) predFlagL0 and predFlagL1 are both TRUE; (3) mmvd_flag is FALSE, (4) The difference in POC between the current picture and the L0 reference picture is equal to the difference in POC between the L1 reference picture and the current picture; (5) The width of the processing block is 8 or more. (6) bcwIdx is 0, and (7) The value of luma_weight_l0_flag indicating whether or not a weighting factor for L0 prediction exists and the value of luma_weight_l1_flag indicating whether or not a weighting factor for L1 prediction exists are FALSE; setting the value of the dmvrFlag to TRUE based on If the value of the dmvrFlag is TRUE, the video decoding device executes the DMVR processing.

5. A video encoding device capable of performing a decoder-side motion vector refinement (DMVR) process using a reference picture and a motion vector, a DMVR unit that executes the DMVR process based on a value of dmvrFlag; The DMVR unit includes at least (1) The value of the merge flag is TRUE; (2) predFlagL0 and predFlagL1 are both TRUE; (3) mmvd_flag is FALSE, (4) The difference in POC between the current picture and the L0 reference picture is equal to the difference in POC between the L1 reference picture and the current picture; (5) The width of the processing block is 8 or more. (6) bcwIdx is 0, and (7) The value of luma_weight_l0_flag indicating whether or not a weighting factor for L0 prediction exists and the value of luma_weight_l1_flag indicating whether or not a weighting factor for L1 prediction exists are FALSE; setting the value of the dmvrFlag to TRUE based on If the value of the dmvrFlag is TRUE, the video encoding device executes the DMVR processing.

Images