Video decoding device, video encoding device, and method for generating encoded streams

Abstract

The present invention provides an image decoding device and an image encoding device that reduce the complexity of image quality enhancement processing. [Solution] The image decoding device has a parameter decoding unit that includes an interprediction parameter decoding unit which processes two motion vectors based on the error between two predicted images, and the interprediction parameter decoding unit processes two motion vectors when neither of the two predicted images is a weighted prediction.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to a video decoding device, a video encoding device, and a method for generating an encoded stream. [Background technology]

[0002] To efficiently transmit or record moving images, a moving image encoding device that generates encoded data by encoding moving images, and a moving image decoding device that generates decoded images by decoding said encoded data are used.

[0003] Specific video coding schemes include, for example, H.264 / AVC and HEVC (High-Efficiency Video Coding).

[0004] In this type of video encoding scheme, the images (pictures) that make up the video are managed by a hierarchical structure consisting of slices obtained by dividing the image, coding tree units (CTUs) obtained by dividing the slices, coding units (sometimes called coding units (CUs)) obtained by dividing the coding tree units, and transform units (TUs) obtained by dividing the coding units. Each CU is then encoded / decoded.

[0005] Furthermore, in such video encoding schemes, a prediction image is typically generated based on a locally decoded image obtained by encoding / decoding the input image. The prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting this prediction image from the input image (original image) is then encoded. Methods for generating the prediction image include inter-frame prediction and intra-frame prediction.

[0006] Furthermore, Non-Patent Document 1 can be cited as an example of recent video encoding and decoding technologies. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "Versatile Video Coding (Draft 4)", JVET-M1001, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 2019 [Overview of the project] [Problems that the invention aims to solve]

[0008] When deriving the bidirectional prediction image described in Non-Patent Document 1, prediction methods such as DMVR processing, which uses two prediction images to correct the motion vector and improve the image quality of the prediction image, and BDOF processing, which uses gradient images to improve the image quality of the prediction image (BDOF prediction), had the problem of being highly complex.

[0009] The embodiments of the present invention aim to realize an image decoding device and an image coding device that reduce the complexity of these image enhancement processes. [Means for solving the problem]

[0010] To solve the above problems, a video decoding device according to one aspect of the present invention is provided. A video decoding device that performs DMVR (Decoder side Motion Vector Refinement) processing using a reference picture and motion vectors, The unit includes a DMVR section that performs the DMVR processing based on the value of dmvrFlag, The DMVR unit sets the dmvrFlag to TRUE based on predetermined conditions. Here, the predetermined condition for dmvrFlag to be TRUE includes the condition that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE. The luma_weight_l0_flag[refIdxL0] indicates whether or not a weight coefficient exists corresponding to the L0 reference picture indicated by the reference picture index refIdxL0, and the luma_weight_l1_flag[refIdxL1] indicates whether or not a weight coefficient exists corresponding to the L1 reference picture indicated by the reference picture index refIdxL1. The DMVR process is executed if the value of dmvrFlag is TRUE.

[0011] Furthermore, a video encoding device according to one aspect of the present invention is A video encoding device that performs DMVR (Decoder-side Motion Vector Refinement) processing using a reference picture and motion vectors, The unit includes a DMVR section that performs the DMVR processing based on the value of dmvrFlag, The DMVR unit sets the dmvrFlag to TRUE based on predetermined conditions. Here, the predetermined condition for dmvrFlag to be TRUE includes the condition that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE. The luma_weight_l0_flag[refIdxL0] indicates whether or not a weight coefficient exists corresponding to the L0 reference picture indicated by the reference picture index refIdxL0, and the luma_weight_l1_flag[refIdxL1] indicates whether or not a weight coefficient exists corresponding to the L1 reference picture indicated by the reference picture index refIdxL1. If the value of said dmvrFlag is TRUE, it is characterized in that said DMVR processing is executed.

[0012] Further, a method for generating an encoded bitstream according to one aspect of the present invention is: A method for generating an encoded bitstream in which DMVR (Decoder side Motion Vector Refinement) processing is performed using a reference picture and a motion vector, the method comprising: if the value of dmvrFlag is TRUE, executing said DMVR processing, setting said dmvrFlag to TRUE based on a predetermined condition, here, the predetermined condition for setting said dmvrFlag to TRUE includes that both luma_weight_l0_flag[ refIdxL0 ] and luma_weight_l1_flag[ refIdxL1 ] are FALSE, wherein said luma_weight_l0_flag[ refIdxL0 ] indicates whether or not a weighting coefficient corresponding to an L0 reference picture indicated by a reference picture index refIdxL0 exists, and said luma_weight_l1_flag[ refIdxL1 ] indicates whether or not a weighting coefficient corresponding to an L1 reference picture indicated by a reference picture index refIdxL1 exists, which is characterized in that:[ Effects of the Invention

[0013] According to the above configuration, an image decoding apparatus and an image encoding apparatus in which the complexity of image quality enhancement processing is reduced can be implemented. Brief Description of the Drawings

[0014] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of an image transmission system according to the present embodiment. [Figure 2]It is a diagram showing the configuration of a transmission device equipped with the moving image encoding apparatus according to the present embodiment and a reception device equipped with the moving image decoding apparatus. (a) shows the transmission device equipped with the moving image encoding apparatus, and (b) shows the reception device equipped with the moving image decoding apparatus. [Figure 3] It is a diagram showing the configuration of a recording device equipped with the moving image encoding apparatus according to the present embodiment and a playback device equipped with the moving image decoding apparatus. (a) shows the recording device equipped with the moving image encoding apparatus, and (b) shows the playback device equipped with the moving image decoding apparatus. [Figure 4] It is a diagram showing the hierarchical structure of encoded stream data. [Figure 5] It is a diagram showing an example of CTU partitioning. [Figure 6] It is a conceptual diagram showing an example of a reference picture and a reference picture list. [Figure 7] It is a schematic diagram showing the configuration of a moving image decoding apparatus. [Figure 8] It is a schematic diagram showing the configuration of an inter prediction parameter decoding unit. [Figure 9] It is a schematic diagram showing the configuration of a merge prediction parameter deriving unit and an AMVP prediction parameter deriving unit. [Figure 10] It is a schematic diagram showing the configuration of a DMVR unit. [Figure 11] It is a flowchart showing the flow of processing in the DMVR unit. [Figure 12] It is a flowchart explaining processing for performing BDOF determination by error threshold processing in DMVR. [Figure 13] It is a schematic diagram showing the configuration of an inter prediction image generation unit included in a prediction image generation unit. [Figure 14] It is a flowchart showing an example of the flow of prediction mode selection processing in a moving image decoding apparatus. [Figure 15] It is a flowchart explaining the flow of processing for deriving a prediction image. [Figure 16] It is a schematic diagram showing the configuration of a BDOF unit. [Figure 17]This figure shows an example of the area where the BDOF (Block Deposition Field) performs padding. [Figure 18] This is a schematic diagram showing the configuration of a video encoding device. [Figure 19] This is a schematic diagram showing the configuration of the parameter coding unit. [Figure 20] This figure shows an example of the number of candidate search distances and derivation directions in a video encoding device. [Modes for carrying out the invention]

[0015] (First Embodiment) Embodiments of the present invention will be described below with reference to the drawings.

[0016] Figure 1 is a schematic diagram showing the configuration of the image transmission system 1 according to this embodiment.

[0017] Image transmission system 1 is a system that transmits an encoded stream containing an image to be encoded, decodes the transmitted encoded stream, and displays the image. Image transmission system 1 consists of a video encoding device (image encoding device) 11, a network 21, a video decoding device (image decoding device) 31, and a video display device (image display device) 41.

[0018] The video encoding device 11 receives image T as input.

[0019] Network 21 transmits the encoded stream Te generated by the video encoding device 11 to the video decoding device 31. Network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. Network 21 is not necessarily limited to a bidirectional communication network; it may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, network 21 may be replaced by a storage medium that records the encoded stream Te, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blu-ray Disc: registered trademark).

[0020] The video decoding device 31 decodes each of the encoded streams Te transmitted by the network 21 and generates one or more decoded images Td.

[0021] The video display device 41 displays all or part of one or more decoded images Td generated by the video decoding device 31. The video display device 41 includes, for example, a liquid crystal display, an organic EL (Electro-luminescence) display, and other display devices. The display can be a stationary, mobile, or HMD (Head Mount Display). Furthermore, the video display device 41 displays high-quality images when the video decoding device 31 has high processing power, and displays images that do not require high processing power or display power when the video decoding device 31 has lower processing power.

[0022] <operators> The operators used in this specification are listed below.

[0023] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is the OR assignment operator, and || represents logical OR.

[0024] x?y:z is a ternary operator that takes the value of y when x is true (non-zero), and takes the value of z when x is false (zero).

[0025] Clip3(a,b,c) is a function that clips c to a value between a and b (inclusive). It returns a when c < a, returns b when c > b, and returns c in other cases (provided that a <= b).

[0026] abs(a) is a function that returns the absolute value of a.

[0027] Int(a) is a function that returns the integer value of a.

[0028] floor(a) is a function that returns the maximum integer not greater than a.

[0029] ceil(a) is a function that returns the minimum integer not less than a.

[0030] a / d represents division of a by d (rounded down to the nearest integer).

[0031] sign(a) is a function that returns the sign of a.

[0032] a^b represents a raised to the power of b.

[0033] <Structure of encoded stream Te> Prior to the detailed description of the moving image encoding apparatus 11 and the moving image decoding apparatus 31 according to the present embodiment, the data structure of the encoded stream Te generated by the moving image encoding apparatus 11 and decoded by the moving image decoding apparatus 31 will be described.

[0034] Figure 4 shows the hierarchical structure of data in the encoded stream Te. The encoded stream Te includes, exemplarily, a sequence and multiple pictures that make up the sequence. Figures 4(a) to (f) show the encoded video sequence that defines the sequence SEQ, the encoded picture that defines the picture PICT, the encoded slice that defines the slice S, the encoded slice data that defines the slice data, the encoded tree unit contained in the encoded slice data, and the encoded unit contained in the encoded tree unit, respectively.

[0035] (Encoded video sequence) In an encoded video sequence, a set of data that the video decoding device 31 references to decode the sequence SEQ to be processed is defined. As shown in Figure 4(a), the sequence SEQ includes a Video Parameter Set, a Sequence Parameter Set SPS, a Picture Parameter Set PPS, a Picture PICT, and Supplemental Enhancement Information SEI.

[0036] The Video Parameter Set (VPS) defines a set of encoding parameters common to multiple video layers in a video composed of multiple layers, as well as a set of encoding parameters associated with the multiple layers included in the video and with each individual layer.

[0037] The sequence parameter set (SPS) defines a set of encoding parameters that the video decoding device 31 references to decode the target sequence. For example, the width and height of the picture are defined. Multiple SPSs may exist. In that case, one of the multiple SPSs is selected from the PPS.

[0038] The Picture Parameter Set (PPS) defines a set of encoding parameters that the video decoding device 31 references to decode each picture in the target sequence. For example, it includes a reference value for the quantization width used for decoding the picture (pic_init_qp_minus26) and a flag indicating the application of weighted prediction (weighted_pred_flag). Multiple PPSs may exist. In that case, one of the multiple PPSs is selected for each picture in the target sequence.

[0039] (Encoded picture) The encoded picture specifies the set of data that the video decoding device 31 references to decode the picture PICT to be processed. As shown in Figure 4(b), the picture PICT includes slices 0 to NS-1 (NS being the total number of slices included in the picture PICT).

[0040] In the following, if it is not necessary to distinguish between slices 0 through NS-1, the code subscripts may be omitted. The same applies to other data included in the coded stream Te described below that have subscripts.

[0041] (Encoded slice) In an encoded slice, a set of data that the video decoding device 31 references to decode the slice S to be processed is defined. As shown in Figure 4(c), the slice includes a slice header and slice data.

[0042] The slice header contains a set of encoding parameters that the video decoding device 31 references to determine the decoding method for the target slice. The slice type specification information (slice_type), which specifies the slice type, is an example of the encoding parameters included in the slice header.

[0043] The slice types that can be specified by the slice type specification information include (1) I slices that use only intra prediction during encoding, (2) P slices that use unidirectional prediction or intra prediction during encoding, and (3) B slices that use unidirectional prediction, bidirectional prediction or intra prediction during encoding. Note that interpretation is not limited to single or bidirectional prediction, and prediction images may be generated using more reference pictures. Hereinafter, when referring to P slices and B slices, we mean slices that contain blocks on which interpretation can be used.

[0044] The slice header may also include a reference to the picture parameter set PPS (pic_parameter_set_id).

[0045] (Encoded slice data) The encoded slice data defines the set of data that the video decoding device 31 references in order to decode the slice data to be processed. As shown in Figure 4(d), the slice data includes CTUs. A CTU is a fixed-size (e.g., 64x64) block that makes up a slice, and is sometimes called a Largest Coding Unit (LCU).

[0046] (Code tree unit) Figure 4(e) defines the set of data that the video decoding device 31 references to decode the CTU to be processed. The CTU is divided into coding units CU, which are the basic units of encoding processing, by recursive quad tree partitioning (QT), binary tree partitioning (BT), or ternary tree partitioning (TT). BT and TT partitioning together are called multi-tree partitioning (MT). The nodes of the tree structure obtained by recursive quad tree partitioning are called coding nodes. The intermediate nodes of quad trees, binary trees, and ternary trees are coding nodes, and the CTU itself is defined as the highest-level coding node.

[0047] The CT (Computation Token) includes the following information: a QT splitting flag (qt_split_cu_flag) indicating whether or not QT splitting is performed, an MT splitting flag (mtt_split_cu_flag) indicating whether or not MT splitting is performed, an MT splitting direction (mtt_split_cu_vertical_flag) indicating the direction of MT splitting, and an MT splitting type (mtt_split_cu_binary_flag) indicating the type of MT splitting. The qt_split_cu_flag, mtt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each coding node.

[0048] Figure 5 shows an example of CTU splitting. When qt_split_cu_flag is 1, the coding node is split into four coding nodes (Figure 5(b)).

[0049] When qt_split_cu_flag is 0 and mtt_split_cu_flag is 0, the coding node is not split and has one CU as a node (Figure 5(a)). The CU is the terminal node of the coding node and cannot be further split. The CU is the basic unit of the coding process.

[0050] When mtt_split_cu_flag is 1, the coding node is split into MT sections as follows: When mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 1, the coding node is horizontally split into two coding nodes (Figure 5(d)); when mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 1, the coding node is vertically split into two coding nodes (Figure 5(c)). Also, when mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 0, the coding node is horizontally split into three coding nodes (Figure 5(f)); and when mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 0, the coding node is vertically split into three coding nodes (Figure 5(e)). These are shown in Figure 5(g).

[0051] Furthermore, if the size of the CTU is 64x64 pixels, the size of the CU can be any of the following: 64x64 pixels, 64x32 pixels, 32x64 pixels, 32x32 pixels, 64x16 pixels, 16x64 pixels, 32x16 pixels, 16x32 pixels, 16x16 pixels, 64x8 pixels, 8x64 pixels, 32x8 pixels, 8x32 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels, 64x4 pixels, 4x64 pixels, 32x4 pixels, 4x32 pixels, 16x4 pixels, 4x16 pixels, 8x4 pixels, 4x8 pixels, and 4x4 pixels.

[0052] (Encoding Unit) As shown in Figure 4(f), the CU defines a set of data that the video decoding device 31 references in order to decode the encoding unit to be processed. Specifically, the CU consists of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines the prediction mode, etc.

[0053] Prediction processing can be performed at the CU (Unit) level or at the subCU level, which is a further division of the CU. If the size of the CU and the subCU are equal, there is one subCU within the CU. If the CU is larger than the size of the subCU, the CU is divided into subCUs. For example, if the CU is 8x8 and the subCU is 4x4, the CU is divided into four subCUs, each consisting of two horizontal and two vertical divisions.

[0054] There are two types of predictions (prediction modes): intra-prediction and inter-prediction. Intra-prediction is prediction within the same picture, while inter-prediction refers to prediction processing performed between different pictures (for example, between display times).

[0055] The transformation and quantization processes are performed in units of CUs, but the quantization transformation coefficients may be entropy-encoded in subblock units such as 4x4.

[0056] (Prediction parameters) The predicted image is derived from the prediction parameters associated with the block. These prediction parameters include intra-prediction and inter-prediction parameters.

[0057] The following describes the prediction parameters for interpretation. The interpretation parameters consist of the prediction list usage flags predFlagL0 and predFlagL1, the reference picture indices refIdxL0 and refIdxL1, and the motion vectors mvL0 and mvL1. The prediction list usage flags predFlagL0 and predFlagL1 indicate whether the reference picture lists, called the L0 list and L1 list respectively, are used for interpretation. If the value is 1, the corresponding reference picture list is used for interpretation. In this specification, when referring to "flags indicating whether or not XX is true," a flag value other than 0 (e.g., 1) is considered true, and 0 is considered false. In logical negation, logical AND, etc., 1 is treated as true and 0 as false (the same applies below). However, in actual devices and methods, other values ​​may be used as true and false values.

[0058] Syntax elements for deriving interpretation parameters include, for example, the affine flag affine_flag, merge flag merge_flag, merge index merge_idx, interpretation identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX, and motion vector accuracy mode amvr_mode.

[0059] (Reference picture list) The reference picture list is a list of reference pictures stored in the reference picture memory 306. Figure 6 is a conceptual diagram showing an example of a reference picture and a reference picture list. In Figure 6(a), rectangles represent pictures, arrows represent the reference relationships between pictures, the horizontal axis represents time, I, P, and B in the rectangles represent intra-picture, single-prediction picture, and double-prediction picture, respectively, and the numbers in the rectangles represent the decoding order. As shown in the figure, the decoding order of the pictures is I0, P1, B2, B3, B4, and the display order is I0, B3, B2, B4, P1. Figure 6(b) shows an example of the reference picture list for picture B3 (target picture). The reference picture list is a list that represents candidates for reference pictures, and a single picture (slice) may have one or more reference picture lists. In the example in the figure, the target picture B3 has two reference picture lists: L0 list RefPicList0 and L1 list RefPicList1. Each CU specifies which picture in the reference picture list RefPicListX (X=0 or 1) to actually reference using the reference picture index refIdxLX. The figure shows an example where refIdxL0=2 and refIdxL1=0. Note that LX is a notation used when there is no distinction between L0 predictions and L1 predictions, and from now on, parameters for the L0 list and parameters for the L1 list will be distinguished by replacing LX with L0 and L1.

[0060] (Merge prediction and AMVP prediction) There are two methods for decoding (encoding) prediction parameters: merge prediction mode and AMVP (Advanced Motion Vector Prediction) mode. The merge flag, merge_flag, is used to distinguish between these modes. In merge prediction mode, the prediction list usage flag predFlagLX (or inter-prediction identifier inter_pred_idc), reference picture index refIdxLX, and motion vector mvLX are not included in the encoded data, but are derived from the prediction parameters of neighboring blocks that have already been processed. In AMVP mode, the inter-prediction identifier inter_pred_idc, reference picture index refIdxLX, and motion vector mvLX are included in the encoded data. The motion vector mvLX is encoded as the prediction vector index mvp_LX_idx, which identifies the prediction vector mvpLX, the difference vector mvdLX, and the motion vector accuracy mode amvr_mode. In merge prediction mode, a merge candidate is selected from the motion information of adjacent blocks, etc., to obtain the motion vector mvLX (motion vector information). In addition to the merge prediction mode, there may also be an affine prediction mode identified by the affine flag affine_flag. As one form of the merge prediction mode, there may also be a skip mode identified by the skip flag skip_flag. The skip mode is a mode that derives prediction parameters in the same way as the merge mode, and does not include prediction errors (residual images, residual information) in the encoded data. In other words, if the skip flag skip_flag is 1, for the target CU, only the skip flag skip_flag and syntax related to the merge mode such as the merge index merge_idx are included, and motion vectors and residual information are not included in the encoded data.

[0061] (Motion vector) The motion vector mvLX represents the amount of shift between blocks in two different pictures. The prediction vector and difference vector for the motion vector mvLX are called the prediction vector mvpLX and the difference vector mvdLX, respectively.

[0062] (Interpretation identifier inter_pred_idc and prediction list usage flag predFlagLX) The interpred_idc identifier is a value that indicates the type and number of reference pictures, and can take one of the following values: PRED_L0, PRED_L1, or PRED_BI. PRED_L0 and PRED_L1 indicate single prediction using one reference picture managed in the L0 list and L1 list, respectively. PRED_BI indicates biprediction (BiPred) using two reference pictures managed in the L0 list and L1 list.

[0063] The merge index merge_idx is an index that indicates which of the candidate predictive parameters (merge candidates) derived from the completed block will be used as the predictive parameters for the target block.

[0064] The relationship between the interpretation identifier inter_pred_idc and the prediction list usage flags predFlagL0 and predFlagL1 is as follows, and they are mutually convertible.

[0065] inter_pred_idc = (predFlagL1<<1)+predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 (Biprediction of biPred) The flag `biPred`, which indicates whether a system uses two prediction lists, can be derived by checking if both of the two prediction list usage flags are set to 1. For example, it can be derived using the following formula.

[0066] biPred = (predFlagL0==1 && predFlagL1==1) Alternatively, the flag biPred can also be derived by determining whether the interpretation identifier is a value that indicates the use of two prediction lists (reference pictures). For example, it can be derived using the following formula:

[0067] biPred = (inter_pred_idc==PRED_BI) ? 1 : 0 (Configuration of the video decoding device) The configuration of the video decoding device 31 (Figure 7) according to this embodiment will be described below.

[0068] The video decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generation unit (prediction image generation device) 308, an inverse quantization / inverse transform unit 311, and an addition unit 312. Note that, in accordance with the video encoding device 11 described later, there is also a configuration in the video decoding device 31 that does not include the loop filter 305.

[0069] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), which are not shown in the figure. The CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, and PPS, and slice headers (slice information) from the encoded data. The CT information decoding unit 3021 decodes CT from the encoded data. The CU decoding unit 3022 decodes CU from the encoded data. The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data if the TU contains a prediction error.

[0070] The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data when the skip mode is not active (skip_mode==0). More specifically, when skip_mode==0, the TU decoding unit 3024 decodes a flag cu_cbp from the encoded data that indicates whether or not the target block contains a quantization prediction error, and decodes the quantization prediction error if cu_cbp is 1. If cu_cbp does not exist in the encoded data, the TU decoding unit 3024 derives cu_cbp as 0.

[0071] Furthermore, the parameter decoding unit 302 is configured to include an inter-prediction parameter decoding unit 303 and an intra-prediction parameter decoding unit 304 (not shown). The prediction image generation unit 308 is configured to include an inter-prediction image generation unit 309 and an intra-prediction image generation unit 310.

[0072] Furthermore, while the following examples use CTU and CU as processing units, processing may be performed in sub-CU units, or the system may be reinterpreted as blocks (CTU and CU) and sub-CU as sub-blocks (sub-CU), and processing may be performed in block or sub-block units.

[0073] The entropy decoding unit 301 performs entropy decoding on the encoded stream Te input from an external source to decode individual codes (syntax elements). The decoded codes include prediction information for generating a predicted image and prediction errors for generating a difference image.

[0074] The entropy decoding unit 301 outputs the decoded codes to the parameter decoding unit 302. The decoded codes include, for example, predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. The control of which codes to decode is performed based on the instructions of the parameter decoding unit 302.

[0075] (Configuration of the interpretation parameter decoding unit) The inter-prediction parameter decoding unit 303 decodes the inter-prediction parameters by referring to the prediction parameters stored in the prediction parameter memory 307 based on the code input from the entropy decoding unit 301. The inter-prediction parameter decoding unit 303 also outputs the decoded inter-prediction parameters to the prediction image generation unit 308 and stores them in the prediction parameter memory 307.

[0076] Figure 8 is a schematic diagram showing the configuration of the inter-prediction parameter decoding unit 303 according to this embodiment. The inter-prediction parameter decoding unit 303 includes a merge prediction unit 30374, a DMVR unit 30375, a sub-block prediction unit (affine prediction unit) 30372, an MMVD prediction unit 30376, a Triangle prediction unit 30377, an AMVP prediction parameter derivation unit 3032, and an addition unit 3038. The merge prediction unit 30374 includes a merge prediction parameter derivation unit 3036. The AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, and the affine prediction unit 30372 are means common to both the video encoding device and the video decoding device, so they may be collectively referred to as the motion vector derivation unit (motion vector derivation device).

[0077] (Affine prediction section) The affine prediction unit 30372 derives affine prediction parameters for the target block. In this embodiment, the motion vectors (mv0_x, mv0_y) and (mv1_x, mv1_y) of two control points (V0, V1) of the target block are derived as affine prediction parameters. Specifically, the motion vectors of each control point may be derived by predicting from the motion vectors of adjacent blocks of the target block, or the motion vectors of each control point may be derived by summing the predicted vector derived as the motion vector of the control point with the difference vector derived from the encoded data.

[0078] (Merge prediction) Figure 9(a) is a schematic diagram showing the configuration of the merge prediction parameter derivation unit 3036 included in the merge prediction unit 30374. The merge prediction parameter derivation unit 3036 comprises a merge candidate derivation unit 30361 and a merge candidate selection unit 30362. A merge candidate consists of a prediction list usage flag predFlagLX, a motion vector mvLX, and a reference picture index refIdxLX, and is stored in the merge candidate list. An index is assigned to the merge candidates stored in the merge candidate list according to a predetermined rule.

[0079] The merge candidate derivation unit 30361 derives merge candidates using the motion vectors and reference picture index refIdxLX of the decoded adjacent blocks as they are.

[0080] The order in which merge candidates are stored in the mergeCandList[] is, for example, spatial merge candidates A1, B1, B0, A0, B2, temporal merge candidate Col, pairwise merge candidate avgK, and zero merge candidate ZK. Note that reference blocks that are unavailable (e.g., blocks that are intra-predicted) are not stored in the merge candidate list.

[0081] The merge candidate selection unit 30362 selects the merge candidate N indicated by the merge index merge_idx from among the merge candidates included in the merge candidate list using the following formula.

[0082] N = mergeCandList[merge_idx] Here, N is a label indicating a merge candidate, and can take the form of A1, B1, B0, A0, B2, Col, avgK, ZK, etc. The movement information of the merge candidate indicated by label N is shown by (mvLXN[0], mvLXN[1]), predFlagLXN, and refIdxLXN.

[0083] The merge candidate selection unit 30362 selects the motion information (mvLXN[0], mvLXN[1]), predFlagLXN, and refIdxLXN of the selected merge candidate as inter-prediction parameters for the target block. The merge candidate selection unit 30362 stores the inter-prediction parameters of the selected merge candidate in the prediction parameter memory 307 and outputs them to the prediction image generation unit 308.

[0084] (AMVP forecast) Figure 9(b) is a schematic diagram showing the configuration of the AMVP prediction parameter derivation unit 3032 according to this embodiment. The AMVP prediction parameter derivation unit 3032 comprises a vector candidate derivation unit 3033 and a vector candidate selection unit 3034. The vector candidate derivation unit 3033 derives prediction vector candidates from the decoded adjacent block motion vector mvLX stored in the prediction parameter memory 307 based on the reference picture index refIdxLX, and stores them in the prediction vector candidate list mvpListLX[].

[0085] The vector candidate selection unit 3034 selects the motion vector mvpListLX[mvp_LX_idx], indicated by the prediction vector index mvp_LX_idx, from among the prediction vector candidates in the prediction vector candidate list mvpListLX[], as the prediction vector mvpLX. The vector candidate selection unit 3034 outputs the selected prediction vector mvpLX to the addition unit 3038.

[0086] The candidate prediction vectors are derived by scaling the motion vectors of decoded adjacent blocks within a predetermined range from the target block. Adjacent blocks include blocks spatially adjacent to the target block, such as the left block and the block above it, as well as regions temporally adjacent to the target block, such as regions obtained from the prediction parameters of blocks that include the same position as the target block but have different display times.

[0087] The addition unit 3038 adds the prediction vector mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded difference vector mvdLX to calculate the motion vector mvLX. The addition unit 3038 outputs the calculated motion vector mvLX to the predicted image generation unit 308 and the prediction parameter memory 307.

[0088] mvLX[0] = mvpLX[0]+mvdLX[0] mvLX[1] = mvpLX[1]+mvdLX[1] The motion vector precision mode amvr_mode is a syntax for switching the precision of a motion vector derived in AMVP mode. For example, when amvr_mode is 0, 1, and 2, 1 / 4 pixel, 1 pixel, and 4 pixel precisions are switched respectively.

[0089] When the precision of a motion vector is set to 1 / 16 precision (MVPREC=16), in order to convert the motion vector difference of 1 / 4, 1, and 4 pixel precisions into a motion vector difference of 1 / 16 pixel precision, the parameter decoding unit 302 may perform inverse quantization using MvShift (=1<<amvr_mode) derived from amvr_mode as described below.

[0090] mvdLX[0] = mvdLX[0] << (MvShift + 2) mvdLX[1] = mvdLX[1] << (MvShift + 2) Furthermore, the parameter decoding unit 302 may derive mvdLX[] before shifting with the MvShift by decoding the following syntaxes. ·abs_mvd_greater0_flag ·abs_mvd_minus2 ·mvd_sign_flag Then, the parameter decoding unit 302 decodes the difference vector lMvd[] from the syntax by using the following formulas.

[0091] lMvd[compIdx] = abs_mvd_greater0_flag[compIdx] * (abs_mvd_minus2[compIdx]+2) * (1-2*mvd_sign_flag[compIdx]) Furthermore, the parameter decoding unit 302 sets the decoded difference vector lMvd[] to MvdLX in the case of translational MVD (MotionModelIdc[x][y] == 0) and to MvdCpLX in the case of control point MVD (MotionModelIdc[x][y] != 0).

[0092] if (MotionModelIdc[x][y] == 0) mvdLX[x0][y0][compIdx] = lMvd[compIdx] else mvdCpLX[x0][y0][compIdx] = lMvd[compIdx]<<2 (Motion vector scaling) This explains how to derive the scaling of motion vectors. If we let Mv be the motion vector (reference motion vector), PicMv be the picture containing the block with Mv, PicMvRef be the reference picture of Mv, sMv be the scaled motion vector, CurPic be the picture containing the block with sMv, and CurPicRef be the reference picture that sMv refers to, then the derivation function for sMv, MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef), is expressed by the following formula.

[0093] sMv = MvScale(Mv,PicMv,PicMvRef,CurPic,CurPicRef) = Clip3(-R1,R1-1,sign(distScaleFactor*Mv)*((abs(distScaleFactor*Mv)+round1-1)>>shift1)) distScaleFactor = Clip3(-R2,R2-1,(tb*tx+round2)>>shift2) tx = (16384+abs(td)>>1) / td td = DiffPicOrderCnt(PicMv,PicMvRef) tb = DiffPicOrderCnt(CurPic,CurPicRef) Here, round1, round2, shift1, and shift2 are the round and shift values ​​used for division using the reciprocal, for example, round1=1<<(shift1-1), round2=1<<(shift2-1), shift1=8, shift2=6, etc. DiffPicOrderCnt(Pic1,Pic2) is a function that returns the difference in time information (e.g., POC) between Pic1 and Pic2. R1 and R2 limit the range of values ​​to perform the processing with limited precision, for example, R1=32768, R2=4096, etc.

[0094] Alternatively, the scaling function MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef) can also be expressed by the following formula.

[0095] MvScale(Mv,PicMv,PicMvRef,CurPic,CurPicRef) = Mv*DiffPicOrderCnt(CurPic,CurPicRef) / DiffPicOrderCnt(PicMv,PicMvRef) In other words, Mv may be scaled according to the ratio of the difference in time information between CurPic and CurPicRef to the difference in time information between PicMv and PicMvRef.

[0096] (DMVR Department 30375) Next, we will explain the DMVR (Decoder side Motion Vector Refinement) processing performed by the DMVR unit 30375. DMVR processing is a process that modifies the motion vectors mvL0 and mvL1 using two reference pictures.

[0097] Figure 10 is a schematic diagram showing the configuration of the DMVR unit 30375. The specific processing performed by the DMVR unit 30375 will be explained using Figure 10. The DMVR unit 30375 comprises a prediction image generation unit 303751 for corrected motion vector search, an initial error generation unit 303752, a motion vector search unit 303753, and a corrected vector derivation unit 303754.

[0098] The DMVR unit 30375 receives information from the merge prediction unit 30374. • Top-left position of the target block (xCb, yCb) • Width of the target block (bW) • Height of the target block bH • 1 / 16 pixel precision motion vectors mvL0 and mvL1 • Reference pictures refPicL0L and refPicL1L Referencing this, the motion vector change amounts dmvL0 and dmvL1 for correcting mvL0 and mvL1 are derived and output to the interpredictive image generation unit 309.

[0099] First, the corrected motion vector search prediction image generation unit 303751 is, • Top-left position (xSb, ySb) of the target subblock • Brightness target subblock width (sbW) • Brightness target subblock height sbH • Motion vector mvLX(X=0, 1) • Reference picture refPicLXL(X=0, 1) Refer to the above to derive the predicted image predSamplesLXL, which has a size of (sbW)*(sbH).

[0100] The corrected motion vector search prediction image generation unit 303751 derives the motion vector MvLsX (X=0, 1) using the following formula.

[0101] MvLsX[0] = MvLX[0]-32 MvLsX[1] = MvLX[1]-32 Furthermore, the DMVR unit 30375 sets the values ​​of the variables srRange, offsetH[0], offsetV[0], offsetH[1], and offsetV[1] to 2.

[0102] Let (xIntL, yIntL) be the position of the pixel in the reference block in integer pixel units corresponding to the pixel position (xL, yL) in the target block. Also, let (xFracL, yFracL) be the offset in units of 1 / 16 pixels from (xIntL, yIntL). These coordinates are derived from the integer components (mvLX[0]>>4, mvLX[1]>>4) and decimal components (mvLX[0]&15, mvLX[1]&15) of the motion vector (mvLX[0], mvLX[1]), and represent the pixel position in decimal precision within the reference picture refPicLXL. For pixels in predSamplesLXL whose position is (xL, yL) (xL=0, ..., sbW-1, yL=0, ..., sbH-1), the DMVR unit 30375 derives xIntL, yIntL, xFracL, and yFracL using the following formulas.

[0103] xIntL = xSb + (mvLX[0]>>4) + xL yIntL = ySb + (mvLX[1]>>4) + yL xFracL = mvLX[0]&15 yFracL = mvLX[1]&15 Next, the DMVR unit 30375 is, ·(xIntL, yIntL) (xFracL, yFracL) ·refPicLXL Refer to this and derive predSamplesLXL.

[0104] First, the corrected motion vector search prediction image generation unit 303751 derives the variables shift1, shift2, shift3, and shift4 using the following formula.

[0105] shift1 = BitDepthY - 6 offset1 = 1 << (shift1 - 1) shift2 = 4 offset2 = 8 shift3 = 10-BitDepthY offset3 = 1 << (shift3 - 1) shift4 = BitDepthY - 10 In the above formula, BitDepthY is the number of pixels in bits.

[0106] Next, the corrected motion vector search prediction image generation unit 303751 sets picW to be equal to the picture width value pic_width_in_luma_samples. The corrected motion vector search prediction image generation unit 303751 also sets picH to be equal to the picture height value pic_height_in_luma_samples.

[0107] Subsequently, the corrected motion vector search prediction image generation unit 303751 derives predSamplesLXL as follows. In the following explanation, fb L [p] represents the filter coefficient for deriving pixel values ​​with 1 / 16 pixel precision. fb L The value of [p] depends on the position p (p=1, 2, ..., 15) with 1 / 16 pixel precision. Position p is equal to xFracL or yFracL. As the value of p increases, fb L [p][0] is monotonically decreasing, and fb L The value of [p][1] increases monotonically.

[0108] First, the corrected motion vector search prediction image generation unit 303751 determines whether xFracL and yFracL are both 0. If both xFracL and yFracL are 0, the DMVR unit 30375 derives predSamplesLXL according to the value of BitDepthY using one of the following formulas.

[0109] predSamplesLXL = (BitDepthY <= 10)? (refPicLXL[xIntL][yIntL] << shift3) : ((refPicLXL[xIntL][yIntL]+offset3) >> shift4) If xFracL is not 0 and yFracL is 0, the corrected motion vector search prediction image generation unit 303751 derives predSamplesLXL using the following formula.

[0110] predSamplesLXL = (fb L [xFracL][0] * refPicLXL[Clip3(0,picW-1,xIntL)][yIntL] + fb L [xFracL][1] * refPicLXL[Clip3(0,picW-1,xIntL+1)][yIntL] + offset1)>>shift1 If xFracL is 0 and yFracL is not 0, the DMVR unit 30375 derives predSamplesLXL using the following formula.

[0111] predSamplesLXL = (fb L [yFracL][0] * refPicLXL[xIntL][Clip3(0,picH-1,yIntL)] + fb L [yFracL][1] * refPicLXL[xIntL][Clip3(0,picH-1,yIntL+1)]+offset1)>>shift1 If neither xFracL nor yFracL is zero, the corrected motion vector search prediction image generation unit 303751 derives predSamplesLXL as follows. First, the DMVR unit 30375 derives temp[n] using the following formula. The derivation process of temp[] is performed n times with the reference position changed. n=0 represents the first derivation process, and n=1 represents the second derivation process.

[0112] yPosL = Clip3(0, PicH-1, yIntL+n-3) temp[n] = (fb L [xFracL][0] * refPicLXL[Clip3(0,picW-1,xIntL)][yPosL] + fb L [xFracL][1] * refPicLXL[Clip3(0,picW-1,xIntL+1)][yPosL]+offset1)>>shift1 Subsequently, the DMVR unit 30375 derives predSamplesLXL according to the following formula.

[0113] predSamplesLXL = (fb L [yFracL][0] * temp[0] + fb L [yFracL][1] * temp[1])>>shift2 Next, the initial error generation unit 303752 · the width of the target block nCbW · the height of the target block nCbH · two predicted images predSampleL1 and predSampleL2 having a size of (nCbW+4)×(nCbH+4) and variables offsetH[0], offsetH[1], offsetV[0], and offsetV[1] with reference to, derives a list Sad1 of sum of absolute differences of pixel values included in predSampleL1 and predSampleL2 and a variable centerSad.

[0114] The DMVR unit 30375 sets the value of each element of the 2×9 array bC according to the following formula.

[0115] bC[0][0] = -1 bC[1][0] = -1 bC[0][1] = -1 bC[1][1] = 0 bC[0][2] = -1 bC[1][2] = 1 bC[0][3] = 0 bC[1][3] = -1 bC[0][4] = 0 bC[1][4] = 0 bC[0][5] = 0 bC[1][5] = 1 bC[0][6] = 1 bC[1][6] = -1 bC[0][7] = 1 bC[1][7] = 0 bC[0][8] = 1 bC[1][8] = 1 The initial error generation unit 303752 derives the elements sadList[i](i=0, ...,8) of Sad1 using the following formula.

number

number

[0116] If centerSad is greater than or equal to (bH>>1)*(bW)*4, the initial error generation unit 303752 sets dmvrFlag to TRUE, and the motion vector search unit 303753 then... ·Number of search points n • The elements of the list Sad1, which is the absolute difference sum of the search points, are in sadList. Refer to the following to derive the index bestIdx, where n is a positive integer.

[0117] The following describes the case where n=9. Note that the value of the search points n may be other than 9. In addition to the method described in this embodiment, for example, n=25 and the minimum value of sadList may be simply selected.

[0118] The motion vector search unit 303753 determines whether sadList[1] < sadList[7] and whether sadList[3] < sadList[5].

[0119] If sadList[1] < sadList[7] and sadList[3] < sadList[5], the DMVR unit 30375 sets the value of idx to 0. Then, the motion vector search unit 303753 determines whether sadList[1] < sadList[3]. The motion vector search unit 303753 sets the value of bestIdx to 1 if sadList[1] < sadList[3], and to 3 otherwise.

[0120] Otherwise, if sadList[1] >= sadList[7] and sadList[3] < sadList[5], the motion vector search unit 303753 sets the value of idx to 6. Then, the DMVR unit 30375 determines whether sadList[7] < sadList[3]. The motion vector search unit 303753 sets the value of bestIdx to 7 if sadList[7] < sadList[3], and to 3 if sadList[7] < sadList[3] is not true.

[0121] Otherwise, if sadList[1] < sadList[7] and sadList[3] >= sadList[5], the motion vector search unit 303753 sets the value of idx to 2. Then, the DMVR unit 30375 determines whether sadList[1] < sadList[5]. The DMVR unit 30375 sets the value of bestIdx to sadList[1] Set to 1 if < sadList[5], and to 5 if sadList[1] < sadList[5] is not true.

[0122] Otherwise, if sadList[1] >= sadList[7] and sadList[3] >= sadList[5], the motion vector search unit 303753 sets the value of idx to 8. Then, the DMVR unit 30375 determines whether sadList[7] < sadList[5]. The DMVR unit 30375 sets the value of bestIdx to sadList[7] If sadList[5] is less than sadList[5], set it to 7; otherwise, set it to 5.

[0123] Furthermore, the motion vector search unit 303753 determines whether sadList[4] <= sadList[bestIdx]. If sadList[4] <= sadList[bestIdx], the DMVR unit 30375 updates the value of bestIdx to 4. On the other hand, if sadList[4] <= sadList[bestIdx] is not true, the DMVR unit 30375 does not update the value of bestIdx.

[0124] Furthermore, the motion vector search unit 303753 determines whether sadList[idx] < sadList[bestIdx]. If sadList[idx] < sadList[bestIdx], the DMVR unit 30375 sets bestIdx to idx. On the other hand, if sadList[idx] < sadList[bestIdx] is not true, the motion vector search unit 303753 does not update the value of bestIdx.

[0125] The motion vector search unit 303753 determines whether the value of bestIdx is 4. If the value of bestIdx is 4, the motion vector search unit 303753 sets halfPelAppliedflag to true.

[0126] If the value of bestIdx is not 4, the motion vector search unit 303753 calculates the values ​​of the variables dmvx and dmvy using the following formula.

[0127] dmvx = (bestIdx / 3 - 1) dmvy = (bestIdx%3 - 1) Furthermore, the motion vector search unit 303753 updates offsetH and offsetV according to the following formula.

[0128] offsetH[0] = offsetH[0] + dmvx, offsetV[0] = offsetV[0] + dmvy offsetH[1] = offsetH[1] - dmvx, offsetV[1] = offsetV[1] - dmvy The motion vector search unit 303753 uses the updated offsetH and offsetV to derive Sad2 in the same way as the process for deriving Sad1 described above. Furthermore, the motion vector search unit 303753 uses Sad2 instead of Sad1 to derive bestIdx again.

[0129] The motion vector search unit 303753 determines whether the value of bestIdx derived again is 4. If the value of bestIdx is 4, the motion vector search unit 303753 sets halfPelAppliedflag to true.

[0130] If the value of bestIdx is not 4, the motion vector search unit 303753 calculates dmvx and dmvy using the following formula.

[0131] dmvx = (bestIdx / 3 - 1), dmvy = (bestIdx%3 - 1) Furthermore, the DMVR unit 30375 calculates dmvL0 and dmvL1 using the following formula.

[0132] dmvL0[0] = 16*dmvx, dmvL0[1] = 16*dmvy dmvL1[0] = -16*dmvx, dmvL1[1] = -16*dmvy If halfPelAppliedflag is true, the motion vector search unit 303753 derives the corrected dmvL0 and dmvL1 as follows. Note that the following sadList consists of elements of Sad2 if Sad2 exists, and elements of Sad1 if Sad2 does not exist.

[0133] First, the motion vector search unit 303753 determines whether sadList[1] + sadList[7] == sadList[4]. If sadList[1] + sadList[7] == sadList[4], and if mrSadT + mrSadB - (mrSadC<<1) == 0, the motion vector search unit 303753 sets dmv[0]=0. If sadList[1] + sadList[7] == sadList[4] is not true, the motion vector search unit 303753 calculates dmv[0] using the following formula.

[0134] dmv[0] = ((sadList[1] - sadList[7])<<3) / (sadList[1] + sadList[7] - (sadList[4]<<1)) Next, the correction vector derivation unit 303754 determines whether sadList[3] + sadList[5] == sadList[4]. If sadList[3] + sadList[5] == sadList[4], and if mrSadL + mrSadR - (mrSadC<<1) == 0, the correction vector derivation unit 303754 sets dmv[1]=0. If sadList[3] + sadList[5] == sadList[4] is not true, the correction vector derivation unit 303754 calculates dmv[1] using the following formula.

[0135] dmv[1] = ((sadList[3] - sadList[5])<<3) / (sadList[3] + sadList[5] - (sadList[4]<<1)) Furthermore, the modified vector derivation unit 303754 modifies the motion vectors mvL0 and mvL1 using the following formula.

[0136] dmvL0[0] = dmvL0[0] + dmv[0] dmvL0[1] = dmvL0[1] + dmv[1] dmvL1[0] = dmvL1[0] - dmv[0] dmvL1[1] = dmvL1[1] - dmv[1] The DMVR unit 30375 calculates the motion vector mvLX by adding the derived difference vector dmvLX to the prediction vector mvpLX input from the merge prediction unit 30374. The DMVR unit 30375 outputs mvLX to the interprediction image generation unit 309.

[0137] mvLX[0] = mvpLX[0]+dmvLX[0] mvLX[1] = mvpLX[1]+dmvLX[1] Note that the values ​​of dmvLX[0] and dmvLX[1] are limited to between -8 and 8, regardless of the number of bits in sadList.

[0138] (DMVR criteria) dmvrFlag is a flag that indicates whether DMVR processing will be performed when TRUE and whether DMVR processing will not be performed when FALSE.

[0139] If the flag indicating that DMVR processing is possible in SPS is On, the initial error generation unit 303752 sets dmvrFlag to TRUE. Otherwise, the initial error generation unit 303752 sets dmvrFlag to FALSE.

[0140] Furthermore, if merge_flag of the block is TRUE, the initial error generation unit 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generation unit 303752 sets dmvrFlag to FALSE.

[0141] Furthermore, if both predFlagL0 and predFlagL1 are TRUE, that is, in the case of bidirectional prediction, the initial error generation unit 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generation unit 303752 sets dmvrFlag to FALSE.

[0142] If mmvd_flag of the block is FALSE, and it is not in MMVD mode, the initial error generation unit 303752 may set dmvrFlag to TRUE. Otherwise, when it is in MMVD mode, the initial error generation unit 303752 sets dmvrFlag to FALSE.

[0143] If DiffPicOrderCnt(currPic, RefPicList

[0000] [refIdxL0]) is equal to DiffPicOrderCnt(RefPicList

[0001] [refIdxL1], currPic), that is, if the current picture currPic is in a positional relationship such that the L0 reference picture RefPicList

[0000] [refIdxL0] and the L1 reference picture RefPicList

[0001] [refIdxL1] are interpolated at equal distances, the initial error generation unit 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generation unit 303752 sets dmvrFlag to FALSE. Here, DiffPicOrderCnt() sets the POC (Picture Order) of the two images as follows. This function derives the difference in the display order of the pictures (Count).

[0144] DiffPicOrderCnt(picA,picB) = PicOrderCnt(picA)-PicOrderCnt(picB) In addition, DiffPicOrderCnt( currPic, RefPicList

[0000] [ refIdxL0 ])*DiffPicOrderCnt( If currPic, RefPicList

[0001] [refIdxL1]) < 0, the initial error generation unit 303752 may set dmvrFlag to TRUE if the positional relationship is such that it is simply interpolated, otherwise it may set dmvrFlag to FALSE.

[0145] Furthermore, if the size of the processing block is less than or equal to a certain value, the initial error generation unit 303752 may set dmvrFlag to FALSE. For example, if bH is 8 or greater and bH*bW is 64, the initial error generation unit 303752 may set dmvrFlag to TRUE. Otherwise, the initial error generation unit 303752 sets dmvrFlag to FALSE.

[0146] Figure 11 is a flowchart showing the processing flow in the DMVR unit 30375. In this embodiment, in addition to the above determination criteria, a condition is added, as shown in Figure 11, that the DMVR processing is applied only when the GBI processing described later is not applied.

[0147] Specifically, first, the DMVR unit 30375 performs the dmvrFlag determination process (S1101) described above. Next, the DMVR unit 30375 determines whether gbiIdx is 0 or not (S1102). As will be described later, when gbiIdx is a non-zero value, a non-equally weighted prediction is made based on the table gbiWLut. In addition to the condition that dmvrFlag is set to TRUE when gbiIdx is 0, dmvrFlag is set to FALSE when gbiIdx is non-zero (S1103).

[0148] Furthermore, the DMVR unit 30375 determines whether dmvrFlag is TRUE or not (S1104), and if it is TRUE, the DMVR process is performed. Execute (S1105), and if it is FALSE, do not execute it.

[0149] When applying GBI forecasting, weighted forecasts are applied, so considering that errors may not be accurately evaluated, the overall processing load can be reduced by limiting the application conditions.

[0150] Similarly, in the weighted prediction described later, if either the L0 prediction or the L1 prediction to which DMVR processing is applied performs weighted prediction, dmvrFlag is set to FALSE. Specifically, if both luma_weight_l0_flag[refIdxL0], which indicates whether the L0 prediction picture has a luminance weight coefficient w0 and offset o0, and luma_weight_l1_flag[refIdxL1], which indicates whether the L1 prediction picture has a luminance weight coefficient w1 and offset o1, are FALSE, then dmvrFlag is set to TRUE; otherwise, dmvrFlag is set to FALSE.

[0151] (BDOF determination using error thresholding in DMVR) DMVR performs a process to calculate the error between the L0 predicted image and the L1 predicted image. Based on the value of this error, it is determined in advance whether or not to perform the BDOF processing described later.

[0152] Figure 12 is a flowchart illustrating the process of determining the BDOF using error thresholding in DMVR.

[0153] First, the initial error generation unit 303752 pre-sets bdofFlag to TRUE (S1201). Next, the initial error generation unit 303752 derives centerSad (S1202) and determines whether the value of centerSad is greater than or equal to the threshold value (bH>>1)*bW*4 (S1203). If the value of centerSad is smaller than the threshold value, the initial error generation unit 303752 determines that the error is small and sets bdofFlag, which indicates whether or not to perform BDOF processing, to FALSE (S1204), thus pre-determining that BDOF processing will not be performed. Since this determination is the same as that of the initial error generation unit 30752 described above, the motion vector search unit 303753 and the correction vector derivation unit 303754 are skipped and DMVR processing is also not performed. If the value of centerSad is greater than or equal to the threshold, the motion vector search unit 303753 performs a corrected motion vector search (S1205), and the corrected vector derivation unit 303754 determines (S1206) whether the value of sadList[bestIdx], which is the SAD value of the minimum bestIdx, is smaller than the threshold value (bH>>1)*bW*8. If the value of sadList[bestIdx] is smaller than the threshold, the corrected vector derivation unit 303754 determines that the error is small and sets bdofFlag, which indicates whether or not to perform BDOF processing, to FALSE (S1207), thereby preventing BDOF processing from being performed in advance.

[0154] Note that the threshold value for (S1206) shall be the same as or greater than the threshold value for (S1203).

[0155] DMVR processing requires calculating the error between the L0 and L1 predicted images in order to search for corrected motion vectors. On the other hand, BDOF processing is ineffective when the error is small. By adding this processing, it becomes possible to determine whether or not to perform BDOF without adding additional error calculations.

[0156] In the BDOF section described later, it is determined whether bdofFlag is TRUE (S1208). If it is Yes, the interpretation parameter decoding unit 303 performs BDOF processing (S1209). If it is No, it is determined that no BDOF processing is performed in that block.

[0157] (Triangle prediction) Next, we will explain Triangle prediction. In Triangle prediction, the target CU is divided into two triangular prediction units, with the diagonal or opposite corner as the boundary. The predicted image for each triangular prediction unit is derived by applying a weighted mask process to each pixel of the prediction image of the target CU (rectangular block containing the triangular prediction unit), according to the pixel's position. For example, a triangular image can be derived from a rectangular image by multiplying it by a mask that sets the pixels in the triangular region within the rectangular area to 1 and the non-triangular region to 0. Adaptive weighting of the prediction image is applied to both regions on either side of the diagonal, and one predicted image of the target CU (rectangular block) is derived through adaptive weighting using the two predicted images. This process is called Triangle synthesis. Transform (inverse transform) and quantization (inverse quantization) processes are applied to the entire target CU. Note that Triangle prediction is only applicable in merge prediction mode or skip mode.

[0158] The Triangle prediction unit 30377 derives prediction parameters corresponding to the two triangular regions used for Triangle prediction and outputs them to the inter-prediction image generation unit 309. For the sake of simplifying processing, Triangle prediction may not use a configuration that does not employ bi-prediction. In this case, inter-prediction parameters for unidirectional prediction are derived for one triangular region. The derivation of the two prediction images and the synthesis using the prediction images are performed by the motion compensation unit 3091 and the Triangle synthesis unit 30952.

[0159] (MMVD prediction unit 30376) The MMVD prediction unit 30376 performs processing in MMVD (Merge with Motion Vector Difference) mode. MMVD mode is a mode in which a motion vector is obtained by adding a difference vector of a predetermined distance and predetermined direction to a motion vector derived from a merge candidate (a motion vector derived from the motion vector of an adjacent block, etc.). In MMVD mode, the MMVD prediction unit 30376 efficiently derives a motion vector by using the merge candidate and restricting the range of values ​​of the difference vector to a predetermined distance (e.g., 8 possibilities) and predetermined direction (e.g., 4 directions, 8 directions, etc.).

[0160] The loop filter 305 is a filter installed within the encoding loop that removes block distortion and ringing distortion, thereby improving image quality. The loop filter 305 applies filters such as a deblocking filter, sample-adaptive offset (SAO), and adaptive loop filter (ALF) to the decoded CU image generated by the summing unit 312.

[0161] The reference picture memory 306 stores the decoded image of the CU generated by the adder 312 at a predetermined location for each target picture and target CU.

[0162] The prediction parameter memory 307 stores prediction parameters at predetermined locations for each CTU or CU to be decoded. Specifically, the prediction parameter memory 307 stores parameters decoded by the parameter decoding unit 302 and prediction modes such as predMode decoded by the entropy decoding unit 301.

[0163] The prediction image generation unit 308 receives the prediction mode (predMode), prediction parameters, etc. The prediction image generation unit 308 also reads a reference picture from the reference picture memory 306. The prediction image generation unit 308 generates a block or subblock prediction image using the prediction mode indicated by the prediction mode (predMode), the prediction parameters, and the read reference picture (reference picture block). Here, a reference picture block is a collection of pixels on the reference picture (usually rectangular, hence called a block), and is the area referenced to generate the prediction image.

[0164] (Interpretation image generation unit 309) When the prediction mode predMode indicates inter-prediction mode, the inter-prediction image generation unit 309 generates a block or sub-block prediction image by inter-prediction using the inter-prediction parameters input from the inter-prediction parameter decoding unit 303 and the read-out reference picture.

[0165] Figure 13 is a schematic diagram showing the configuration of the inter-predictive image generation unit 309 included in the predictive image generation unit 308 according to this embodiment. The inter-predictive image generation unit 309 is composed of a motion compensation unit (predictive image generation device) 3091 and a synthesis unit 3095.

[0166] (Motion compensation) The motion compensation unit 3091 (interpolation image generation unit) generates an interpolated image (motion-compensated image) by reading a block from the reference picture memory 306 that is located at a position shifted by the motion vector mvLX from the position of the target block in the reference picture RefPicLX specified by the reference picture index refIdxLX, based on the inter-prediction parameters (prediction list usage flag predFlagLX, reference picture index refIdxLX, motion vector mvLX) input from the inter-prediction parameter decoding unit 303. Here, if the precision of the motion vector mvLX is not integer precision, a filter called a motion compensation filter is applied to generate pixels at decimal positions, and then the interpolated image is generated.

[0167] The motion compensation unit 3091 first derives the integer position (xInt, yInt) and phase (xFrac, yFrac) corresponding to the coordinates (x, y) within the prediction block using the following formula.

[0168] xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x xFrac = mvLX[0]&(MVPREC-1) yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y yFrac = mvLX[1]&(MVPREC-1) Here, (xPb, yPb) is the top-left coordinate of a block of size bW*bH, x=0...bW-1, y=0...bH-1, and MVPREC represents the precision of the motion vector mvLX (1 / MVPREC pixel precision). For example, MVPREC may be 16.

[0169] The motion compensation unit 3091 derives a temporary image temp[][] by performing horizontal interpolation on the reference picture refImg using an interpolation filter. The following Σ is the sum with respect to k of k=0..NTAP-1, shift1 is a normalization parameter that adjusts the range of values, and offset1=1<<(shift1-1).

[0170] temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1 Next, the motion compensation unit 3091 performs vertical interpolation on the temporary image temp[][] to create the interpolated image Pred Derive [][]. The following Σ is the sum with respect to k from k=0 to NTAP-1, shift2 is a normalization parameter that adjusts the range of values, and offset2=1<<(shift2-1).

[0171] Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2 (Synthesis part) The synthesis unit 3095 generates a predicted image by referring to the interpolated image input from the motion compensation unit 3091, the inter-prediction parameter decoding unit 303, and the intra-prediction image input from the intra-prediction image generation unit 310, and outputs the generated predicted image to the addition unit 312.

[0172] The synthesis unit 3095 comprises a combined intra / inter synthesis unit 30951, a triangle synthesis unit 30952, an OBMC unit 30953, and a BDOF unit 30956.

[0173] (Combined intra / inter synthesis process) The Combined intra / inter synthesis unit 30951 generates a predicted image by combining unidirectional prediction, skip mode, merge mode, and intra prediction in AMVP.

[0174] (Triangle compositing process) The Triangle synthesis unit 30952 generates a predicted image using the Triangle prediction described above.

[0175] (OBMC processing) The OBMC unit 30953 generates a predicted image using OBMC (Overlapped Block Motion Compensation) processing. The OBMC processing includes the following steps: The interpolated image (motion-compensated image) of the target subblock is generated using the interpolated image (PU interpolated image) generated using the interprediction parameters attached to the target subblock, and the interpolated image (OBMC interpolated image) generated using the motion parameters of the adjacent subblocks of the target subblock. • Predictive images are generated by weighting and averaging OBMC interpolated images and PU interpolated images.

[0176] (Weighted prediction unit 30954) The weighted prediction unit 309454 generates a block prediction image by multiplying the motion-compensated images PredL0 and PredL1 by a weight coefficient. If one of the prediction list usage flags (predFlagL0 or predFlagL1) is 1 (single prediction) and weighted prediction is not used, the motion-compensated image PredLX (LX is L0 or L1) is adjusted to the number of pixels bit depth using the following process:

[0177] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredLX[x][y]+offset1)> >shift1) Here, shift1 = Max(2, 14-bit Depth) and offset1 = 1 << (shift1 - 1).

[0178] (Bidirectional prediction processing) Furthermore, if both prediction list usage flags (predFlagL0 and predFlagL1) are 1 (biprediction BiPred) and weight prediction is not used, the following process is performed to average the motion-compensated images PredL0 and PredL1 and adjust them to the number of pixels.

[0179] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredL0[x][y]+PredL1[x][y]+offset2)> >shift2) Here, shift2 = Max(3, 15-bit Depth) and offset2 = 1 << (shift2 - 1). This process will also be referred to as normal bidirectional prediction.

[0180] Furthermore, if a single prediction is performed and a flag indicating whether or not the L0 reference picture has weight prediction coefficients w0 and offset o0 (luma_weight_l0_flag for luminance, chroma_weight_l0_flag for chroma difference) is set to ON, the weighted prediction unit 30954, in the case of L0 prediction, derives the weight prediction coefficients w0 and offset o0 from the encoded data and performs the following processing.

[0181] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,((PredL0[x][y]*w0+(1<<(log2WD-1)))> >log2WD)+o0) In the case of L1 prediction, if a flag indicating whether or not the L1 reference picture contains the weight prediction coefficient w1 and offset o1 (luma_weight_l1_flag for luminance, chroma_weight_l1_flag for chroma difference) is set to ON, the weighted prediction unit 30954 derives the weight prediction coefficient w1 and offset o1 from the encoded data and performs the following calculation.

[0182] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,((PredL1[x][y]*w1+(1<<(log2WD-1)))> >log2WD)+o1) Here, log2WD is a variable that explicitly sums the Log2WeightDenom+shift1 values ​​sent in the slice header for luminance and color discrimination.

[0183] (Weighted bidirectional prediction processing) Furthermore, when BiPred is used for biprediction, and flags indicating the presence or absence of weight prediction coefficients and offsets (luma_weight_l0_flag, luma_weight_l1_flag for luminance, chroma_weight_l0_flag, chroma_weight_l1_flag for chroma difference) are used for weight prediction, the weighted prediction unit 30954 derives the weight prediction coefficients w0, w1, o0, and o1 from the encoded data and performs the following calculation.

[0184] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,(PredL0[x][y]*w0+PredL1[x][y]*w1+((o0+o1+1)<<log2WD))> >(log2WD+1)) (GBI Department 30955) The weighted prediction described above illustrates an example where a predicted image is generated by multiplying an interpolated image by a weight coefficient. Here, we will describe another example of generating a predicted image by multiplying an interpolated image by a weight coefficient. In detail, we will explain the process of generating a predicted image using Generalized bi-prediction (GBI prediction). In GBI prediction, the predicted image Pred is generated by multiplying the L0 predicted image PredL0 and the L1 predicted image PredL1 in bi-prediction by weight coefficients (w0, w1).

[0185] Furthermore, when generating a predicted image using GBI prediction, the GBI unit 30955 switches the weight coefficients (w0, w1) on a coding unit basis. That is, the GBI unit 30954 of the inter-predicted image generation unit 309 sets the weight coefficient for each coding unit. In GBI prediction, multiple weight coefficient candidates are defined in advance, and gbiIdx is an index that indicates the weight coefficient to be used in the target block from among the multiple weight coefficient candidates included in the table gbiWLut.

[0186] The GBI unit 30955 checks the flag gbiAppliedFlag, which indicates whether GBI prediction is used. If it is FALSE, the motion compensation unit 3091 generates a predicted image using the following formula.

[0187] Pred[x][y]=Clip3(0,(1< <bitDepth)-1, (PredL0[x][y]+ PredL1[x][y]+offset2)>>shift2 ) Here, the initial state of gbiAppliedFlag is FALSE. The GBI unit 30955 sets gbiAppliedFlag to TRUE if the flag indicating that GBI processing is possible, which is an SPS flag, is On, and it is a bidirectional prediction. Furthermore, as an additional (AND) condition, gbiAppliedFlag may be set to TRUE if gbiIdx, which is the index of the GBI prediction weight coefficient table gbiWLut, is not 0 (the index value when the weights of the L0 prediction image and the L1 prediction image are equal). Furthermore, as an additional (AND) condition, gbiAppliedFlag may be set to TRUE if the block size of the CU is greater than or equal to a certain value.

[0188] The GBI unit 30955 derives the predicted image Pred from the weights w0, w1 and PredL0, PredL1 using the following formula, when gbiAppliedFlag is true.

[0189] Pred[x][y]=Clip3(0,(1< <bitDepth)-1, (w0*PredL0[x][y]+w1*PredL1[x][y]+offset3)>>(shift2+3)) Here, the weight coefficient w1 is a coefficient derived from the table iWLut[]= {4,5,3,10,-2} using gbiIdx, which is explicitly shown in syntax. The weight coefficient w0 is (8-w1). Note that when gbiIdx=0, w0=w1=4, which is equivalent to normal bidirectional prediction.

[0190] shift1, shift2, offset1, and offset2 are derived using the following formulas.

[0191] shift1=Max(2,14-bitDepth) shift2=Max(3,15-bitDepth)=shift1+1 offset1=1<<(shift1-1) offset2=1<<(shift2-1) offset3=1<<(shift2+2) Furthermore, if there are multiple tables gbiWLut with different combinations of weight coefficients, the GBI unit 30955 may switch the table used to select the weight coefficients depending on whether the picture structure is LowDelay (LB) or not.

[0192] In AMVP prediction mode, when GBI prediction is used, the inter-prediction parameter decoding unit 303 decodes gbiIdx and sends it to the GBI unit 30955. In merge prediction mode, when GBI prediction is used, the inter-prediction parameter decoding unit 303 decodes the merge index merge_idx, and the merge candidate derivation unit 30361 derives the gbiIdx for each merge candidate. Specifically, the merge candidate derivation unit 30361 uses the weight coefficients of the adjacent blocks used to derive the merge candidates as the weight coefficients of the merge candidates used for the target block. In other words, in merge mode, the weight coefficients used in the past are inherited as the weight coefficients of the target block.

[0193] (Selection of forecast mode using GBI forecasting) Next, with reference to Figure 14, the process of selecting a prediction mode using GBI prediction in the video decoding device 31 will be described. Figure 14 is a flowchart showing an example of the flow of the prediction mode selection process in the video decoding device 31.

[0194] As shown in Figure 14, the inter-prediction parameter decoding unit 303 first decodes the skip flag (S1401). If the skip flag indicates that the system is in skip mode (YES in S1402), the prediction mode becomes merge mode (S1403), and the inter-prediction parameter decoding unit 303 decodes the merge index (S14031). If GBI prediction is used, the GBI unit 30955 derives the weight coefficients derived from the merge candidates as the weight coefficients for GBI prediction.

[0195] If the skip flag does not indicate skip mode (NO in S1402), the inter-prediction parameter decoding unit 303 decodes the merge flag (S1407). If the merge flag indicates merge mode (YES in S1408), the prediction mode becomes merge mode (S1403), and the inter-prediction parameter decoding unit 303 decodes the merge index (S14031). If GBI prediction is used, the GBI unit 30955 derives the weight coefficients derived from the merge candidates as the weight coefficients for GBI prediction.

[0196] If the merge flag does not indicate merge mode (NO in S1408), the prediction mode is AMVP mode (S1409).

[0197] In AMVP mode, the interpretation parameter decoding unit 303 decodes the interpretation identifier inter_pred_idc (S14090). Next, the interpretation parameter decoding unit 303 decodes the difference vector mvdLX (S14091). Subsequently, the interpretation parameter decoding unit 303 decodes gbiIdx (S14092), and if GBI prediction is used, the GBI unit 30955 selects the weight coefficient w1 for GBI prediction from the weight coefficient candidates in the gbiWLut table.

[0198] (BDOF prediction) Next, we will explain the details of the prediction using BDOF processing performed by the BDOF unit 30956 (BDOF prediction). In dual prediction mode, the BDOF unit 30956 generates a prediction image by referring to two prediction images (the first prediction image and the second prediction image) and a gradient correction term.

[0199] Figure 15 is a flowchart illustrating the process for deriving the predicted image.

[0200] If the interpretation parameter decoding unit 303 determines that it is a unidirectional prediction for L0 (inter_pred_idc is 0 in S1501), the motion compensation unit 3091 generates the L0 prediction image PredL0[x][y] (S1502). If the interpretation parameter decoding unit 303 determines that it is a unidirectional prediction for L1 (inter_pred_idc is 1 in S1501), the motion compensation unit 3091 generates the L1 prediction image PredL1[x][y] (S1503). On the other hand, if the interpretation parameter decoding unit 303 determines that it is a biprediction mode (inter_pred_idc is 2 in S1501), the process continues to S1504. In S1504, the synthesis unit 3095 refers to bioAvailableFlag, which indicates whether or not to perform BDOF processing, and determines whether or not BDOF processing is necessary. If bioAvailableFlag indicates TRUE, the BDOF unit 30956 performs BDOF processing to generate a bidirectional prediction image (S1506). If bioAvailableFlag indicates FALSE, the synthesis unit 3095 generates a prediction image using the normal bidirectional prediction image generation method (S1505).

[0201] The interpretation parameter decoding unit 303 may derive TRUE for bioAvailableFlag if the L0 reference image refImgL0 and the L1 reference image refImgL1 are different reference images and the two pictures are facing in opposite directions relative to the target picture. Specifically, if the target image is currPic, bioAvailableFlag will show TRUE if the condition DiffPicOrderCnt(currPic,refImgL0)*DiffPicOrderCnt(currPic,refImgL1)<0 is satisfied. Here, DiffPicOrderCnt() is a function that derives the difference in POC (Picture Order Count) between two images, as follows:

[0202] DiffPicOrderCnt(picA,picB) = PicOrderCnt(picA)-PicOrderCnt(picB) As a condition for bioAvailableFlag to be TRUE, we may add the condition that the motion vector of the target block is not a motion vector at the subblock level.

[0203] Additionally, a condition may be added for bioAvailableFlag to indicate TRUE: the motion vector of the target picture is not a motion vector at the subblock level.

[0204] Additionally, a condition may be added for bioAvailableFlag to indicate TRUE: the sum of the absolute differences between the L0 predicted image and the L1 predicted image of two prediction blocks is greater than or equal to a predetermined value.

[0205] Additionally, a condition may be added for bioAvailableFlag to indicate TRUE: that the predictive image creation mode is a block-based predictive image creation mode.

[0206] Additionally, a condition may be added for bioAvailableFlag to indicate TRUE: in weighted prediction, neither L0 prediction nor L1 prediction is weighted. Specifically, if both luma_weight_l0_flag[refIdxL0], which indicates whether the L0 prediction picture has a luminance weight coefficient w0 and offset o0, and luma_weight_l1_flag[refIdxL1], which indicates whether the L1 prediction picture has a luminance weight coefficient w1 and offset o1, are FALSE, then bioAvailableFlag will indicate TRUE.

[0207] Figure 16 is a schematic diagram showing the configuration of the BDOF unit 30956. Using Figure 16, the specific processing performed by the BDOF unit 30956 will be explained. The BDOF processing unit 30956 comprises an L0, L1 prediction image generation unit 309561, a gradient image generation unit 309562, a correlation parameter calculation unit 309563, a motion compensation correction value derivation unit 309564, and a BDOF prediction image generation unit 309565. The BDOF unit 30956 generates a prediction image from the interpolated image received from the motion compensation unit 3091 and the inter-prediction parameter decoding unit 303, and outputs the generated prediction image to the summing unit 312. The process of deriving the motion compensation correction value modBIO (motion compensation correction image) from the gradient image and correcting and deriving the PredL0 and PredL1 prediction images is called the bidirectional gradient change processing.

[0208] Figure 17 shows an example of a region where padding is performed. First, the L0,L1 prediction image generation unit 309561 generates L0 and L1 prediction images used for BDOF processing. The BDOF unit 30956 performs BDOF processing based on the L0 and L1 prediction images for each CU unit or sub-CU unit shown in Figure 17, but in order to calculate the gradient, it requires interpolation image information for two extra pixels surrounding the target CU or sub-CU. This interpolation image information is generated using adjacent integer pixels instead of a normal interpolation filter for gradient image generation, which will be described later. In other cases, this part is used as a padding region, and surrounding pixels are copied and used, similar to the outside of the picture. The unit of BDOF processing is NxN pixels or less than or equal to a CU unit, and the processing itself is performed using (N+2)x(N+2) pixels, which is the sum of the surrounding pixels.

[0209] The gradient image generation unit 309562 generates a gradient image. In gradient change (Optical Flow), it is assumed that the pixel value of each point does not change, but only its position changes. This can be expressed as follows, using the change in the horizontal pixel value I (horizontal gradient value lx) and its position change Vx, the change in the vertical pixel value I (vertical gradient value ly) and its position change Vy, and the time change lt of the pixel value I.

[0210] lx * Vx + ly * Vy + lt = 0 Hereinafter, the change in position (Vx, Vy) is referred to as a corrected weight vector (u, v).

[0211] Specifically, the gradient image generation unit 309562 derives gradient images lx0, ly0, lx1, and ly1 according to the following formula. lx0 and lx1 represent gradients along the horizontal direction, and ly0 and ly1 represent gradients along the vertical direction.

[0212] lx0[x][y] = (PredL0[x+1][y]-PredL0[x-1][y])>>shift1 ly0[x][y] = (PredL0[x][y+1]-PredL0[x][y-1])>>shift1 lx1[x][y] = (PredL1[x+1][y]-PredL1[x-1][y])>>shift1 ly1[x][y] = (PredL1[x][y+1]-PredL1[x][y-1])>>shift1 where shift1=Max(2,14-bitDepth).

[0213] Next, the correlation parameter calculation unit 309563 derives gradient products of sums s1, s2, s3, s5, s6 for (N+2)x(N+2) pixels by using one surrounding pixel for each block of NxN pixels in each CU.

[0214] s1 = sum(phiX[x][y]* phiX[x][y]) s2 = sum(phiX[x][y]* phiY[x][y]) s3 = sum(-theta[x][y]* phiX[x][y]) s5 = sum(phiY[x][y]* phiY[x][y]) s6 = sum(-theta[x][y]* phiY[x][y]) Here, sum(a) represents the sum of a for coordinates (x,y) within a block of (N+2)x(N+2) pixels. Also, theta[x][y]= -(PredL1[x][y]>>shift4)+(PredL0[x][y]>>shift4) phiX[x][y] = (lx1[x][y] + lx0[x][y])>>shift5 phiY[x][y] = (ly1[x][y] + ly0[x][y])>>shift5 Here, shift4 = Min(8, bitDepth-4) shift5 = Min(5, bitDepth-7) Let's assume that.

[0215] Next, the motion compensation correction value derivation unit 309564 uses the derived gradient sum products s1, s2, s3, s5, s6 to derive a correction weight vector (u, v) in NxN pixel units.

[0216] u = (s3<<3)>>log2(s1) v = ((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1))>>log2(s5) Here, s2m = s2 >> 12 and s2s = s2 & ((1 << 12) - 1).

[0217] Furthermore, you can use clipping to further restrict the range of u and v, as shown below.

[0218] u = s1>0?Clip3(-th,th,-(s3<<3)>>floor(log2(s1))):0 v = s5>0?Clip3(-th,th,((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1))>>floor(log2(s5))):0 Here, we set th = Max(2, 1<<(13-bitDepth)). Since the value of th needs to be calculated in conjunction with shift1, we consider the case where the pixel bit length bitDepth is greater than 12 bits.

[0219] The motion compensation correction value derivation unit 309564 derives the modBIO[x][y] motion compensation correction value for NxN pixels using the correction weight vector (u, v) for NxN pixels and the gradient images lx0, ly0, lx1, ly1.

[0220] modBIO[x][y] = ((lx1[x][y]-lx0[x][y])*u+(ly1[x][y]-ly0[x][y])*v+1)>>1 (Equation A3) Alternatively, modBIO can be derived using the round function as follows.

[0221] modBIO[x][y] = Round(((lx1[x][y]-lx0[x][y])*u)>>1)+Round(((ly1[x][y]-ly0[x][y])*v)>>1) The BDOF prediction image generation unit 309565 derives the pixel values ​​Pred of an NxN pixel prediction image using the above parameters and the following formula.

[0222] At this time, the BDOF prediction image generation unit 309565 derives the pixel value Pred of the NxN pixel prediction image using the above parameters and the following formula.

[0223] Pred[x][y] = Clip3(0, (1< <bitDepth)-1,( PredL0[x][y]+PredL1[x][y]+modBIO[x][y] +offset2)>>shift2) Here, shift2 = Max(3, 15-bitDepth) and offset2 = 1 << (shift2 - 1).

[0224] The BDOF prediction image generation unit 309565 then outputs the predicted image of the generated block to the summing unit 312.

[0225] The inverse quantization / inverse transform unit 311 inversely quantizes the quantized transform coefficients input from the entropy decoding unit 301 to obtain transform coefficients. The quantized transform coefficients are coefficients obtained by performing frequency transform such as DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform) on a prediction error and then quantizing the result in an encoding process. The inverse quantization / inverse transform unit 311 performs inverse frequency transform such as inverse DCT or inverse DST on the obtained transform coefficients to calculate a prediction error. The inverse quantization / inverse transform unit 311 outputs the prediction error to the addition unit 312. The inverse quantization / inverse transform unit 311 sets all prediction errors to 0 when skip_flag is 1 or cu_cbp is 0.

[0226] The addition unit 312 adds the prediction image of the block input from the prediction image generation unit 308 and the prediction error input from the inverse quantization / inverse transform unit 311 on a pixel-by-pixel basis, and generates a decoded image of the block. The addition unit 312 stores the decoded image of the block in the reference picture memory 306 and outputs it to the loop filter 305.

[0227] (Configuration of Video Encoding Apparatus) Next, the configuration of the video encoding apparatus 11 according to the present embodiment will be described. FIG. 18 is a schematic diagram showing the configuration of the video encoding apparatus 11 according to the present embodiment. The video encoding apparatus 11 includes a prediction image generation unit 101, a subtraction unit 102, a transform / quantization unit 103, an inverse quantization / inverse transform unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determination unit 110, a parameter encoding unit 111, and an entropy encoding unit 104.

[0228] The prediction image generation unit 101 generates a prediction image for each CU, which is a region obtained by dividing each picture of an image T. The prediction image generation unit 101 operates in the same manner as the prediction image generation unit 308 already described, and thus description thereof is omitted.

[0229] The subtraction unit 102 subtracts the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of image T to generate a prediction error. The subtraction unit 102 outputs the prediction error to the conversion / quantization unit 103.

[0230] The conversion / quantization unit 103 calculates conversion coefficients by frequency conversion for the prediction error input from the subtraction unit 102, and derives quantized conversion coefficients by quantization. The conversion / quantization unit 103 outputs the quantized conversion coefficients to the entropy coding unit 104 and the inverse quantization / inverse conversion unit 105.

[0231] The inverse quantization / inverse transformation unit 105 is the same as the inverse quantization / inverse transformation unit 311 (Figure 7) in the video decoding device 31, and therefore its explanation is omitted. The calculated prediction error is output to the summing unit 106.

[0232] The entropy coding unit 104 receives quantization conversion coefficients from the conversion / quantization unit 103 and coding parameters from the parameter coding unit 111. The coding parameters include codes for, for example, the reference picture index refIdxLX, the prediction vector index mvp_LX_idx, the difference vector mvdLX, the motion vector accuracy mode amvr_mode, the prediction mode predMode, and the merge index merge_idx.

[0233] The entropy coding unit 104 entropy codes the division information, prediction parameters, quantization conversion coefficients, etc., to generate and output an encoded stream Te.

[0234] The parameter coding unit 111 includes a header coding unit 1110 (not shown), a CT information coding unit 1111, a CU coding unit 1112 (predictive mode coding unit), and a parameter coding unit 112. The CU coding unit 1112 further includes a TU coding unit 1114.

[0235] The following describes the general operation of each module. The parameter coding unit 111 performs encoding processing on parameters such as header information, partitioning information, prediction information, and quantization conversion coefficients.

[0236] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information, etc., from the encoded data.

[0237] The CU encoding unit 1112 encodes CU information, prediction information, TU splitting flag split_transform_flag, CU residual flags cbf_cb, cbf_cr, cbf_luma, etc.

[0238] The TU encoding unit 1114 encodes the QP update information (quantization correction value) and the quantization prediction error (residual_coding) when the TU contains a prediction error.

[0239] The CT information coding unit 1111 and the CU coding unit 1112 output syntax elements such as inter-prediction parameters (prediction mode predMode, merge flag merge_flag, merge index merge_idx, inter-prediction identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX), intra-prediction parameters (prev_intra_luma_pred_flag, mpm_idx, rem_selected_mode_flag, rem_selected_mode, rem_non_selected_mode), and quantization conversion coefficients to the entropy coding unit 104.

[0240] (Configuration of the parameter coding section) The parameter coding unit 112 derives inter-prediction parameters based on the prediction parameters input from the coding parameter determination unit 110. The parameter coding unit 112 includes a configuration that is partially identical to the configuration in which the inter-prediction parameter decoding unit 303 derives inter-prediction parameters.

[0241] Figure 19 is a schematic diagram showing the configuration of the parameter coding unit 112. The configuration of the parameter coding unit 112 will now be described. As shown in Figure 19, the parameter coding unit 112 is composed of a parameter coding control unit 1121, a merge prediction unit 30374, a subblock prediction unit (affine prediction unit) 30372, a DMVR unit 30375, an MMVD prediction unit 30376, a Triangle prediction unit 30377, an AMVP prediction parameter derivation unit 3032, and a subtraction unit 1123. The merge prediction unit 30374 includes a merge prediction parameter derivation unit 3036. The parameter coding control unit 1121 includes a merge index derivation unit 11211 and a vector candidate index derivation unit 11212. Furthermore, the parameter coding control unit 1121 derives merge_idx, affine_flag, base_candidate_idx, distance_idx, direction_idx, etc. in the merge index derivation unit 11211, and derives mvpLX, etc. in the vector candidate index derivation unit 11212. The merge prediction parameter derivation unit 3036, AMVP prediction parameter derivation unit 3032, affine prediction unit 30372, MMVD prediction unit 30376, and Triangle prediction unit 30377 may be collectively referred to as the motion vector derivation unit (motion vector derivation device). The parameter coding unit 112 outputs the motion vector mvLX, reference picture index refIdxLX, inter-prediction identifier inter_pred_idc, or information indicating these, to the prediction image generation unit 101. The parameter coding unit 112 also outputs merge_flag, skip_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_lX_idx, mvdLX, amvr_mode, and affine_flag to the entropy coding unit 104.

[0242] Figure 20 shows an example of the number of candidate search distances and derivation directions in the video encoding device 11. The parameter coding control unit 1121 derives parameters representing the difference vector (base_candidate_idx, distance_idx, direction_idx, etc.) and outputs them to the MMVD prediction unit 30376. The derivation of the difference vector in the parameter coding control unit 1121 will be explained with reference to Figure 20. The black circle in the center of the figure is the position pointed to by the prediction vector mvpLX, and eight search distances are searched in each of the four directions (up, down, left, right) around this position. mvpLX is the motion vector of the first and second candidates in the merge candidate list, and a search is performed for each. There are two prediction vectors in the merge candidate list (the first and second in the list), the search distance is 8, and the search direction is 4, so there are 64 candidates for mvdLX. The mvdLX with the lowest cost among those explored is represented by base_candidate_idx, distance_idx, and direction_idx.

[0243] Thus, the MMVD mode is a mode that searches for a limited number of candidate points centered on the prediction vector and derives an appropriate motion vector.

[0244] The merge index derivation unit 11211 derives the merge index merge_idx and outputs it to the merge prediction parameter derivation unit 3036 (merge prediction unit). In MMVD mode, the merge index derivation unit 11211 sets the value of the merge index merge_idx to the same value as the value of base_candidate_idx. The vector candidate index derivation unit 11212 derives the prediction vector index mvp_lX_idx.

[0245] The merge prediction parameter derivation unit 3036 derives the inter prediction parameters based on the merge index merge_idx.

[0246] The AMVP prediction parameter derivation unit 3032 derives the prediction vector mvpLX based on the motion vector mvLX. The AMVP prediction parameter derivation unit 3032 outputs the prediction vector mvpLX to the subtraction unit 1123. The reference picture index refIdxLX and the prediction vector index mvp_lX_idx are output to the entropy coding unit 104.

[0247] The affine prediction unit 30372 derives the interpretation parameters (affine prediction parameters) of the subblock.

[0248] The subtraction unit 1123 subtracts the prediction vector mvpLX, which is the output of the AMVP prediction parameter derivation unit 3032, from the motion vector mvLX input from the coding parameter determination unit 110 to generate the difference vector mvdLX. The difference vector mvdLX is output to the entropy coding unit 104.

[0249] The addition unit 106 generates a decoded image by adding the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 and the prediction error input from the inverse quantization / inverse transform unit 105 for each pixel. The addition unit 106 stores the generated decoded image in the reference picture memory 109.

[0250] The loop filter 107 applies a deblocking filter, SAO, and ALF to the decoded image generated by the summing unit 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters; for example, it may consist of only a deblocking filter.

[0251] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in predetermined locations for each target picture and CU.

[0252] The reference picture memory 109 stores the decoded images generated by the loop filter 107 at predetermined locations for each target picture and CU.

[0253] The coding parameter determination unit 110 selects one set from among several sets of coding parameters. The coding parameters are the QT, BT, or TT segmentation information, prediction parameters, or parameters that are to be coded and generated in relation to these. The prediction image generation unit 101 generates a prediction image using these coding parameters.

[0254] The coding parameter determination unit 110 calculates an RD cost value for each of the multiple sets, which indicates the amount of information and the coding error. The coding parameter determination unit 110 selects the set of coding parameters that minimizes the calculated cost value. As a result, the entropy coding unit 104 outputs the selected set of coding parameters as a coded stream Te. The coding parameter determination unit 110 stores the determined coding parameters in the prediction parameter memory 108.

[0255] Furthermore, some parts of the video encoding device 11 and video decoding device 31 in the above-described embodiment, such as the entropy decoding unit 301, parameter decoding unit 302, loop filter 305, prediction image generation unit 308, inverse quantization / inverse transformation unit 311, addition unit 312, prediction image generation unit 101, subtraction unit 102, transformation / quantization unit 103, entropy encoding unit 104, inverse quantization / inverse transformation unit 105, loop filter 107, encoding parameter determination unit 110, and parameter encoding unit 111, may be implemented using a computer. In that case, a program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Hereinafter, "computer system" refers to a computer system built into either the video encoding device 11 or the video decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. In addition, "computer-readable recording media" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. Moreover, the above-mentioned programs may be for the purpose of realizing some of the functions described above, and may also be programs that can realize the aforementioned functions in combination with programs already recorded in the computer system.

[0256] Furthermore, some or all of the video encoding device 11 and video decoding device 31 in the above-described embodiment may be implemented as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 11 and video decoding device 31 may be individually implemented as a processor, or some or all of them may be integrated into a single processor. In addition, the method of implementing the integrated circuit is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. Furthermore, if an integrated circuit technology that can replace LSIs emerges due to advances in semiconductor technology, an integrated circuit using that technology may be used.

[0257] Although one embodiment of this invention has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes can be made without departing from the spirit of this invention.

[0258] [Application Examples] The video encoding device 11 and video decoding device 31 described above can be installed and used in various devices that transmit, receive, record, and play back video. The video may be natural video captured by a camera or the like, or it may be artificial video (including CG and GUI) generated by a computer or the like.

[0259] First, with reference to Figure 2, we will explain how the aforementioned video encoding device 11 and video decoding device 31 can be used for transmitting and receiving video.

[0260] Figure 2(a) is a block diagram showing the configuration of the transmitter PROD_A equipped with the video encoding device 11. As shown in the figure, the transmitter PROD_A includes an encoding unit PROD_A1 that obtains encoded data by encoding video, a modulation unit PROD_A2 that obtains a modulated signal by modulating a carrier wave with the encoded data obtained by the encoding unit PROD_A1, and a transmitter PROD_A3 that transmits the modulated signal obtained by the modulation unit PROD_A2. The video encoding device 11 described above is used as this encoding unit PROD_A1.

[0261] The transmitting device PROD_A may further include a camera PROD_A4 for capturing moving images, a recording medium PROD_A5 for recording moving images, an input terminal PROD_A6 for receiving moving images from an external source, and an image processing unit A7 for generating or processing images, as input sources for moving images to be input to the encoding unit PROD_A1. In the figure, a configuration in which the transmitting device PROD_A includes all of these is shown as an example, but some may be omitted.

[0262] The recording medium PROD_A5 may contain unencoded video footage, or it may contain video footage encoded using a recording encoding method different from the transmission encoding method. In the latter case, a decoding unit (not shown) may be interposed between the recording medium PROD_A5 and the encoding unit PROD_A1 to decode the encoded data read from the recording medium PROD_A5 according to the recording encoding method.

[0263] Figure 2(b) is a block diagram showing the configuration of the receiver PROD_B equipped with the video decoding device 31. As shown in the figure, the receiver PROD_B includes a receiver PROD_B1 that receives a modulated signal, a demodulation unit PROD_B2 that obtains encoded data by demodulating the modulated signal received by the receiver PROD_B1, and a decoding unit PROD_B3 that obtains a video by decoding the encoded data obtained by the demodulation unit PROD_B2. The video decoding device 31 described above is used as this decoding unit PROD_B3.

[0264] The receiving device PROD_B may further include a display PROD_B4 for displaying the video output from the decoding unit PROD_B3, a recording medium PROD_B5 for recording the video, and an output terminal PROD_B6 for outputting the video externally. The figure illustrates a configuration in which the receiving device PROD_B includes all of these, but some may be omitted.

[0265] The recording medium PROD_B5 may be for recording unencoded video, or it may be encoded using a recording encoding method different from the transmission encoding method. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_B3 and the recording medium PROD_B5, which encodes the video acquired from the decoding unit PROD_B3 according to the recording encoding method.

[0266] The transmission medium for transmitting the modulated signal may be wireless or wired. Furthermore, the transmission method for transmitting the modulated signal may be broadcasting (referring here to a transmission method where the destination is not predetermined) or communication (referring here to a transmission method where the destination is predetermined). In other words, the transmission of the modulated signal may be achieved by wireless broadcasting, wired broadcasting, wireless communication, or wired communication.

[0267] For example, a terrestrial digital broadcasting station (broadcasting equipment, etc.) / receiving station (television receiver, etc.) is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals wirelessly. Similarly, a cable television broadcasting station (broadcasting equipment, etc.) / receiving station (television receiver, etc.) is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via wired broadcasting.

[0268] Furthermore, servers (such as workstations) and clients (such as television sets, personal computers, and smartphones) for internet-based VOD (Video On Demand) services and video sharing services are examples of transmitting devices PROD_A and receiving devices PROD_B that transmit and receive modulated signals via communication (typically, either wireless or wired transmission is used as the transmission medium in a LAN, and wired transmission is used in a WAN). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Smartphones also include multi-function mobile phones.

[0269] Furthermore, the video sharing service client has the function of decrypting encoded data downloaded from the server and displaying it on the screen, as well as the function of encoding video images captured by the camera and uploading them to the server. In other words, the video sharing service client functions as both a transmitting device PROD_A and a receiving device PROD_B.

[0270] Next, with reference to Figure 3, we will explain how the aforementioned video encoding device 11 and video decoding device 31 can be used for recording and playing back video.

[0271] Figure 3(a) is a block diagram showing the configuration of the recording device PROD_C equipped with the video encoding device 11 described above. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that obtains encoded data by encoding video, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 to the recording medium PROD_M. The video encoding device 11 described above is used as this encoding unit PROD_C1.

[0272] The recording medium PROD_M may be (1) a type built into the recording device PROD_C, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), (2) a type connected to the recording device PROD_C, such as an SD memory card or USB (Universal Serial Bus) flash memory, or (3) a type loaded into a drive device (not shown) built into the recording device PROD_C, such as a DVD (Digital Versatile Disc: registered trademark) or BD (Blu-ray Disc: registered trademark).

[0273] Furthermore, the recording device PROD_C may also include a camera PROD_C3 for capturing moving images, an input terminal PROD_C4 for receiving moving images from an external source, a receiving unit PROD_C5 for receiving moving images, and an image processing unit PROD_C6 for generating or processing images, as input sources for moving images to the encoding unit PROD_C1. In the figure, a configuration in which the recording device PROD_C includes all of these is shown as an example, but some may be omitted.

[0274] The receiving unit PROD_C5 may receive unencoded video footage, or it may receive encoded data encoded using a transmission encoding scheme different from the recording encoding scheme. In the latter case, it is preferable to interpose a transmission decoding unit (not shown) between the receiving unit PROD_C5 and the encoding unit PROD_C1 to decode the encoded data encoded using the transmission encoding scheme.

[0275] Examples of such recording devices PROD_C include DVD recorders, BD recorders, and HDD (Hard Disk Drive) recorders (in this case, the input terminal PROD_C4 or the receiver PROD_C5 is the main input source for moving images). Camcorders (in this case, the camera PROD_C3 is the main input source for moving images), personal computers (in this case, the receiver PROD_C5 or the image processing unit C6 is the main input source for moving images), and smartphones (in this case, the camera PROD_C3 or the receiver PROD_C5 is the main input source for moving images) are also examples of such recording devices PROD_C.

[0276] Figure 3(b) is a block diagram showing the configuration of the playback device PROD_D equipped with the video decoding device 31 described above. As shown in the figure, the playback device PROD_D includes a reading unit PROD_D1 that reads encoded data written to the recording medium PROD_M, and a decoding unit PROD_D2 that obtains a video by decoding the encoded data read by the reading unit PROD_D1. The video decoding device 31 described above is used as this decoding unit PROD_D2.

[0277] The recording medium PROD_M may be (1) a type built into the playback device PROD_D, such as an HDD or SSD; (2) a type connected to the playback device PROD_D, such as an SD memory card or USB flash memory; or (3) a type loaded into a drive device (not shown) built into the playback device PROD_D, such as a DVD or BD.

[0278] Furthermore, the playback device PROD_D may also include a display PROD_D3 for displaying the video output from the decoding unit PROD_D2, an output terminal PROD_D4 for outputting the video externally, and a transmission unit PROD_D5 for transmitting the video. The figure illustrates a configuration in which the playback device PROD_D includes all of these components, but some may be omitted.

[0279] The transmitting unit PROD_D5 may transmit unencoded video footage, or it may transmit encoded data encoded using a transmission encoding scheme different from the recording encoding scheme. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_D2 and the transmitting unit PROD_D5 to encode the video footage using the transmission encoding scheme.

[0280] Examples of such playback devices PROD_D include DVD players, BD players, and HDD players (in this case, the output terminal PROD_D4 to which a television receiver is connected becomes the main output destination for the video). Other examples of such playback devices PROD_D include television receivers (in this case, the display PROD_D3 becomes the main output destination for the video), digital signage (also called electronic billboards or electronic display boards, etc., where the display PROD_D3 or the transmitter PROD_D5 becomes the main output destination for the video), desktop PCs (in this case, the output terminal PROD_D4 or the transmitter PROD_D5 becomes the main output destination for the video), laptop or tablet PCs (in this case, the display PROD_D3 or the transmitter PROD_D5 becomes the main output destination for the video), and smartphones (in this case, the display PROD_D3 or the transmitter PROD_D5 becomes the main output destination for the video).

[0281] (Hardware implementation and software implementation) Furthermore, each block of the video decoding device 31 and video encoding device 11 described above may be implemented in hardware by logic circuits formed on an integrated circuit (IC chip), or it may be implemented in software using a CPU (Central Processing Unit).

[0282] In the latter case, each of the above devices consists of a CPU that executes program instructions to realize each function, ROM (Read Only Memory) that stores the above program, and RAM (Random Memory) that loads the above program. The device includes an Access Memory, a storage device (recording medium) such as a memory for storing the above-mentioned program and various data. Furthermore, the objective of the embodiments of the present invention can also be achieved by supplying a recording medium on which the program code (executable program, intermediate code program, source program) of the control program of each of the above-mentioned devices, which is software that realizes the above-mentioned functions, is recorded in a way that can be read by a computer, and the computer (or CPU or MPU) reads and executes the program code recorded on the recording medium.

[0283] Examples of the above recording media include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy disks (registered trademark) and hard disks, and CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical discs), MDs (Mini Discs), DVDs (Digital Versatile Discs: registered trademark), CD-Rs (CD Recordable), and Blu-ray discs. Disks including optical discs such as Disc (registered trademark), cards such as IC cards (including memory cards) and optical cards, semiconductor memories such as mask ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable and Programmable Read-Only Memory: registered trademark), and flash ROM, or logic circuits such as PLD (Programmable logic device) and FPGA (Field Programmable Gate Array) can be used.

[0284] Furthermore, each of the above devices may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. This communication network is not particularly limited, as long as it is capable of transmitting the program code. For example, the Internet, intranet, extranet, LAN (Local Area Network), ISDN (Integrated Services Digital Network), VAN (Value-Added Network), CATV (Community Antenna television / Cable Television) communication network, Virtual Private Network, telephone line network, mobile communication network, satellite communication network, etc., can be used. Also, the transmission medium constituting this communication network is not limited to a specific configuration or type, as long as it is capable of transmitting the program code. For example, it can be used with wired connections such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carriers, cable TV lines, telephone lines, and ADSL (Asymmetric Digital Subscriber Line) lines, as well as wireless connections such as IrDA (Infrared Data Association), infrared (like remote controls), Bluetooth®, IEEE 802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance®), mobile phone networks, satellite lines, and terrestrial digital broadcasting networks. Furthermore, embodiments of the present invention can also be realized in the form of computer data signals embedded in a carrier wave, where the above program code is embodied through electronic transmission.

[0285] The embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the claims. That is, embodiments obtained by combining technical means that have been appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

[0286] (Cross-reference of related applications) This application claims priority to Japanese Patent Application No. 2019-043097, filed on March 8, 2018, and all of its contents are included herein by reference.

[0287] (summary) This invention can also be expressed as follows.

[0288] An image decoding device according to one aspect of the present invention is: It has an interpretation parameter decoding unit that has a process to correct two motion vectors from the error of two predicted images, If neither of the two predicted images is a weighted prediction, the two motion vectors are modified.

[0289] Furthermore, an image coding device according to one aspect of the present invention is It has an interpretation parameter coding unit that has a process to correct two motion vectors based on the error between two predicted images, If neither of the two predicted images is a weighted prediction, the two motion vectors are modified.

[0290] By adopting this configuration, when weighted predictions are applied, the error evaluation cannot be performed correctly, resulting in no effect. Therefore, by restricting the application conditions, the overall processing load can be reduced.

[0291] Furthermore, an image decoding device according to one aspect of the present invention is An interprediction parameter decoding unit that has a process for correcting two motion vectors from the error values ​​of two predicted images, The system includes a bidirectional gradient change processing unit that generates a predicted image using a gradient image derived from two interpolated images using the parameters decoded by the interprediction parameter decoding unit, Using the error values ​​of the two predicted images, a determination is made as to whether or not to apply the processing by the bidirectional gradient change processing unit.

[0292] Furthermore, an image coding device according to one aspect of the present invention is An interprediction parameter coding unit that has a process for correcting two motion vectors from the error values ​​of two predicted images, The system includes a bidirectional gradient change processing unit that generates a predicted image using a gradient image derived from two interpolated images using the parameters decoded by the interprediction parameter decoding unit, Using the error values ​​of the two predicted images, a determination is made as to whether or not to apply the processing by the bidirectional gradient change processing unit.

[0293] With this configuration, it is necessary to calculate the error value in order to correct the motion vector. On the other hand, in the bidirectional gradient change processing unit, if the error is small, it has no effect. By adding this processing, it becomes possible to determine whether to apply the processing by the bidirectional gradient change processing unit without adding additional error values, and the overall amount of processing can be reduced.

[0294] The present invention can also be expressed as follows:

[0295] An image decoding device according to one aspect of the present invention is: Using two reference pictures and motion vectors mvL0 and mvL1, DMVR (Decoder side motion) is created. A video decoding device that performs vector refinement processing, A DMVR unit that executes the DMVR process when the dmvrFlag, which indicates whether or not the DMVR process is performed, is TRUE, The system comprises a weighted prediction unit that performs weighted prediction using a first weight coefficient, a first offset, a second weight coefficient, and a second offset, The DMVR unit sets the dmvrFlag to TRUE based on predetermined conditions, where the predetermined conditions for setting the dmvrFlag to TRUE include both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] being FALSE. The luma_weight_l0_flag[refIdxL0] indicates whether or not the first weight coefficient and the first offset of the luminance corresponding to the L0 reference picture indicated by the reference picture index refIdxL0 exist. The luma_weight_l1_flag[refIdxL1] indicates whether or not the second weight coefficient and second offset of the luminance corresponding to the L1 reference picture indicated by the reference picture index refIdxL1 exist.

[0296] Furthermore, an image coding device according to one aspect of the present invention is Using two reference pictures and motion vectors mvL0 and mvL1, DMVR (Decoder side motion) is created. A video encoding device that performs vector refinement processing, A DMVR unit that executes the DMVR process when the dmvrFlag, which indicates whether or not the DMVR process is performed, is TRUE, The system comprises a weighted prediction unit that performs weighted prediction using a first weight coefficient, a first offset, a second weight coefficient, and a second offset, The DMVR unit sets the dmvrFlag to TRUE based on predetermined conditions, where the predetermined conditions for setting the dmvrFlag to TRUE include both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] being FALSE. The luma_weight_l0_flag[refIdxL0] indicates whether or not the first weight coefficient and the first offset of the luminance corresponding to the L0 reference picture indicated by the reference picture index refIdxL0 exist. The luma_weight_l1_flag[refIdxL1] indicates whether or not the second weight coefficient and second offset of the luminance corresponding to the L1 reference picture indicated by the reference picture index refIdxL1 exist.

[0297] The present invention can also be expressed as follows:

[0298] A system according to one aspect of the present invention is A video encoding device that generates an encoded stream of the image to be encoded, A system comprising: a video decoding device that decodes the encoded stream generated by the video encoding device, The aforementioned video encoding device and video decoding device are capable of performing DMVR (Decoder side Motion Vector Refinement) processing using a reference picture and a motion vector. The aforementioned video encoding device and the aforementioned video decoding device, respectively, The unit includes a DMVR section that performs the DMVR processing based on the value of dmvrFlag, The DMVR unit is at least (1) The value of the merge flag is TRUE. (2) Both predFlagL0 and predFlagL1 are TRUE. (3) mmvd_flag is FALSE, (4) The difference between the POC of the current picture and the L0 reference picture is equal to the difference between the POC of the L1 reference picture and the current picture. (5) The width of the processing block must be 8 or more. (6) bcwIdx is 0, and (7) The value of luma_weight_l0_flag, which indicates whether or not weight coefficients for L0 prediction exist, and the value of luma_weight_l1_flag, which indicates whether or not weight coefficients for L1 prediction exist, are both FALSE. Based on this, the value of dmvrFlag is set to TRUE, The DMVR process is executed if the value of dmvrFlag is TRUE.

[0299] Furthermore, the data structure of the encoded stream according to one aspect of the present invention is: A data structure of an encoded stream decoded by a video decoding device that performs DMVR (Decoder-side Motion Vector Refinement) processing using a reference picture and motion vectors, The aforementioned video decoding device The unit includes a DMVR section that performs the DMVR processing based on the value of dmvrFlag, The data structure of the encoded stream is a luma_weight_l0_flag indicating whether or not weight coefficients for L0 prediction exist and a luma_weight_l1_flag indicating whether or not weight coefficients for L1 prediction exist, wherein in decoding by the video decoding device, at least (1) the value of the merge flag is TRUE, (2) both predFlagL0 and predFlagL1 are TRUE, (3) mmvd_flag is FALSE, and (4) the difference between the POC of the current picture and the L0 reference picture is equal to the L1 reference picture. The luma_weight_l0_flag and luma_weight_l1_flag are characterized in that they are used in the process of setting dmvrFlag to TRUE based on the following conditions: (5) the difference between and the current picture's POC is equal, (6) the width of the processing block is 8 or more, (7) bcwIdx is 0, and (8) the value of luma_weight_l0_flag indicating whether or not a weight coefficient for L0 prediction exists, and the value of luma_weight_l1_flag indicating whether or not a weight coefficient for L1 prediction exists are FALSE.

[0300] Furthermore, a method for generating an encoded stream according to one aspect of the present invention is: A method for generating an encoded stream by a video decoding device that performs DMVR (Decoder-side Motion Vector Refinement) processing using a reference picture and a motion vector, This includes performing the DMVR processing based on the value of dmvrFlag, Based on predetermined conditions, the dmvrFlag is set to TRUE, where the predetermined conditions for setting the dmvrFlag to TRUE include both luma_weight_l0_flag[ refIdxL0 ] and luma_weight_l1_flag[ refIdxL1 ] being FALSE. The aforementioned luma_weight_l0_flag[refIdxL0] indicates whether or not a weight coefficient exists corresponding to the L0 reference picture indicated by the reference picture index refIdxL0. The luma_weight_l1_flag[refIdxL1] is characterized by indicating whether or not a weight coefficient exists corresponding to the L1 reference picture indicated by the reference picture index refIdxL1.

[0301] Furthermore, a video decoding device according to one aspect of the present invention is A video decoding device capable of performing DMVR (Decoder side Motion Vector Refinement) processing using a reference picture and motion vectors, The unit includes a DMVR section that performs the DMVR processing based on the value of dmvrFlag, The DMVR unit is at least (1) The value of the merge flag is TRUE. (2) Both predFlagL0 and predFlagL1 are TRUE. (3) mmvd_flag is FALSE, (4) The difference between the POC of the current picture and the L0 reference picture is equal to the difference between the POC of the L1 reference picture and the current picture. (5) The width of the processing block must be 8 or more. (6) bcwIdx is 0, and (7) The value of luma_weight_l0_flag, which indicates whether or not weight coefficients for L0 prediction exist, and the value of luma_weight_l1_flag, which indicates whether or not weight coefficients for L1 prediction exist, are both FALSE. Based on this, the value of dmvrFlag is set to TRUE, The DMVR process is executed if the value of dmvrFlag is TRUE.

[0302] Furthermore, a video encoding device according to one aspect of the present invention is A video encoding device capable of performing DMVR (Decoder side Motion Vector Refinement) processing using a reference picture and motion vectors, The unit includes a DMVR section that performs the DMVR processing based on the value of dmvrFlag, The DMVR unit is at least (1) The value of the merge flag is TRUE. (2) Both predFlagL0 and predFlagL1 are TRUE. (3) mmvd_flag is FALSE, (4) The difference between the POC of the current picture and the L0 reference picture is equal to the difference between the POC of the L1 reference picture and the current picture. (5) The width of the processing block must be 8 or more. (6) bcwIdx is 0, and (7) The value of luma_weight_l0_flag, which indicates whether or not weight coefficients for L0 prediction exist, and the value of luma_weight_l1_flag, which indicates whether or not weight coefficients for L1 prediction exist, are both FALSE. Based on this, the value of dmvrFlag is set to TRUE, The DMVR process is executed if the value of dmvrFlag is TRUE. [Industrial applicability]

[0303] Embodiments of the present invention can be suitably applied to a video decoding device that decodes encoded data from image data, and a video encoding device that generates encoded data from image data. Furthermore, they can be suitably applied to the data structure of encoded data generated by the video encoding device and referenced by the video decoding device. [Explanation of Symbols]

[0304] 31 Image Decoder 301 Entropy Decoder 302 Parameter Decoding Unit 3020 Header Decoding Section 303 Interpretation parameter decoding unit 304 Intra Prediction Parameter Decoding Unit 308 Predictive Image Generation Unit 309 Interpretation Image Generation Unit 310 Intra Predictive Image Generation Unit 311 Inverse Quantization / Inverse Transformation Section 312 Addition section 11 Image encoding device 101 Predictive Image Generation Unit 102 Subtraction Unit 103 Conversion / Quantization Section 104 Entropy coding unit 105 Inverse Quantization / Inverse Transformation Section 107 Loop Filter 110 Encoding parameter determination unit 111 Parameter coding section 112 Parameter coding section 1110 Header Encoding Section 1111 CT information encoder 1112 CU coding unit (predictive mode coding unit) 1114 TU encoder 3091 Motion compensation unit 3095 Synthesis part 30951 Combined Intra / inter synthesis section 30952 Triangle synthesis section 30953 OBMC Department 30954 Weighted prediction unit 30955 GBI Department 30956 BDOF section 309561 L0, L1 Predictive Image Generation Unit 309562 Gradient Image Generation Unit 309563 Correlation Parameter Calculation Unit 309564 Motion Compensation Correction Value Derivation Unit 309565 BDOF Prediction Image Generation Unit

Claims

1. A video decoding device that performs DMVR (Decoder side Motion Vector Refinement) processing using a reference picture and motion vectors, The unit includes a DMVR section that performs the DMVR processing based on the value of dmvrFlag, The DMVR unit sets the dmvrFlag to TRUE based on predetermined conditions. Here, the predetermined condition for dmvrFlag to be TRUE includes the condition that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE. The luma_weight_l0_flag[refIdxL0] indicates whether or not a weight coefficient exists corresponding to the L0 reference picture indicated by the reference picture index refIdxL0, and the luma_weight_l1_flag[refIdxL1] indicates whether or not a weight coefficient exists corresponding to the L1 reference picture indicated by the reference picture index refIdxL1. A video decoding device characterized in that it performs the DMVR processing if the value of the dmvrFlag is TRUE.

2. A video encoding device that performs DMVR (Decoder-side Motion Vector Refinement) processing using a reference picture and motion vectors, The unit includes a DMVR section that performs the DMVR processing based on the value of dmvrFlag, The DMVR unit sets the dmvrFlag to TRUE based on predetermined conditions. Here, the predetermined condition for dmvrFlag to be TRUE includes the condition that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE. The luma_weight_l0_flag[refIdxL0] indicates whether or not a weight coefficient exists corresponding to the L0 reference picture indicated by the reference picture index refIdxL0, and the luma_weight_l1_flag[refIdxL1] indicates whether or not a weight coefficient exists corresponding to the L1 reference picture indicated by the reference picture index refIdxL1. A video encoding device characterized in that if the value of dmvrFlag is TRUE, the DMVR processing is executed.

3. A method for generating an encoded stream that performs DMVR (Decoder side Motion Vector Refinement) processing using a reference picture and motion vectors, The process includes executing the DMVR process if the value of dmvrFlag is TRUE. Based on predetermined conditions, the dmvrFlag is set to TRUE, Here, the predetermined condition for setting dmvrFlag to TRUE includes the condition that both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are FALSE. The `luma_weight_l0_flag[refIdxL0]` indicates whether or not a weight coefficient exists corresponding to the L0 reference picture indicated by the reference picture index `refIdxL0`, and the `luma_weight_l1_flag[refIdxL1]` indicates whether or not a weight coefficient exists corresponding to the L1 reference picture indicated by the reference picture index `refIdxL1`. A method for generating an encoded stream, characterized by the following:

Images