Moving image decoding device, moving image encoding device, and recording medium

Abstract

Provided are a moving image decoding device, a moving image encoding device, and a recording medium that reduce the complexity of image quality improvement processing. A moving image decoding device is provided with an inter-prediction parameter decoding unit that has a process for correcting two motion vectors on the basis of the error between two predicted images, and that performs a process for correcting the two motion vectors when neither of the two predicted images is weighted prediction.

Description

[0001] This application is a divisional application of the invention patent application filed on March 6, 2020, with application number 202080019033.4 and invention title "Motion Image Decoding Device". Technical Field

[0002] The embodiments of the present invention relate to a motion picture decoding device, a motion picture encoding device, and a recording medium. Background Technology

[0003] In order to efficiently transmit or record moving images, a moving image encoding device is used to generate encoded data by encoding the moving images, and a moving image decoding device is used to generate decoded images by decoding the encoded data.

[0004] Specific motion picture coding methods include H.264 / AVC and HEVC (High-Efficiency Video Coding).

[0005] In this motion picture coding method, the images (pictures) that constitute the motion picture are managed through a hierarchical structure and encoded / decoded by each CU. The hierarchical structure includes slices obtained by segmenting the image, coding tree units (CTUs) obtained by segmenting the slices, coding units (sometimes also called coding units (CUs)) obtained by segmenting the coding tree units, and transformation units (TUs) obtained by segmenting the coding units.

[0006] Furthermore, in such moving image coding methods, a prediction image is typically generated based on a locally decoded image obtained by encoding / decoding the input image, and the prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the prediction image from the input image (original image) is encoded. Methods for generating prediction images include inter-frame prediction and intra-frame prediction.

[0007] In addition, non-patent literature 1 can be cited as a technology for motion image encoding and decoding in recent years.

[0008] Existing technical documents

[0009] Non-patent literature

[0010] Non-patent document 1: "Versatile Video Coding (Draft 4)", JVET-M1001, Joint VideoExploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 2019 Summary of the Invention

[0011] The problem the invention aims to solve

[0012] In deriving bidirectional prediction images, the prediction using DMVR processing and BDOF processing (BDOF prediction) has the problem of high processing complexity. Specifically, DMVR processing uses two prediction images to correct motion vectors to improve the image quality of the prediction image, while BDOF processing uses gradient images to improve the image quality of the prediction image.

[0013] The purpose of embodiments of the present invention is to realize an image decoding device and an image encoding device that reduce the complexity of image quality improvement processing.

[0014] Technical solution

[0015] To address the aforementioned problems, one aspect of the present invention provides an image decoding apparatus having an inter-frame prediction parameter decoding unit that includes a process for correcting two motion vectors based on the error between two predicted images, wherein the correction of the two motion vectors is performed when neither of the two predicted images is a weighted prediction.

[0016] Furthermore, one aspect of the image coding apparatus of the present invention includes an inter-frame prediction parameter coding unit, which has a process for correcting two motion vectors based on the error of two prediction images, and performs the correction of the two motion vectors when neither of the two prediction images is a weighted prediction.

[0017] By adopting this configuration, the error assessment cannot be accurately performed when applying weighted prediction, thus failing to achieve the desired effect. Therefore, the overall processing volume can be reduced by limiting the application conditions.

[0018] Furthermore, one aspect of the image decoding apparatus of the present invention includes: an inter-frame prediction parameter decoding unit having a process for correcting two motion vectors based on error values ​​of two prediction images; and a bidirectional gradient transformation processing unit for generating prediction images using gradient images, wherein the gradient images are derived from two interpolated images generated using parameters decoded by the inter-frame prediction parameter decoding unit, and the error values ​​of the two prediction images are used to determine whether to apply the processing performed by the bidirectional gradient transformation processing unit.

[0019] Furthermore, one aspect of the image coding apparatus of the present invention includes: an inter-frame prediction parameter coding unit having a process for correcting two motion vectors based on error values ​​of two prediction images; and a bidirectional gradient transformation processing unit for generating prediction images using gradient images, wherein the gradient images are derived from two interpolated images generated using parameters decoded by the inter-frame prediction parameter coding unit, and the error values ​​of the two prediction images are used to determine whether to apply the processing performed by the bidirectional gradient transformation processing unit.

[0020] By adopting such a configuration, it is necessary to calculate the error value in order to correct the motion vector. On the other hand, the two-way gradient change processing unit has no effect when the error is small. Therefore, by adding such processing, it is possible to make a judgment on the processing performed by the two-way gradient change processing unit without adding an additional error value, thereby reducing the overall processing load.

[0021] Beneficial effects

[0022] Based on the above configuration, an image decoding device and an image encoding device can be realized that reduce the complexity of image quality improvement processing. Attached Figure Description

[0023] Figure 1 This is a schematic diagram showing the configuration of the image transmission system of this embodiment.

[0024] Figure 2 This diagram illustrates the configuration of a transmitting device equipped with a motion image encoding apparatus according to this embodiment and a receiving device equipped with a motion image decoding apparatus. Figure 2 (a) indicates a transmitting device equipped with a motion picture encoding device. Figure 2 (b) indicates a receiving device equipped with a motion picture decoding device.

[0025] Figure 3 This diagram illustrates the configuration of a recording apparatus equipped with a motion image encoding device according to this embodiment and a playback apparatus equipped with a motion image decoding device. Figure 3 (a) indicates a recording device equipped with a motion picture encoding device. Figure 3 (b) indicates a reproduction device equipped with a motion picture decoding device.

[0026] Figure 4 It is a diagram representing the hierarchical structure of the encoded stream data.

[0027] Figure 5 This is a diagram representing a segmentation example of CTU.

[0028] Figure 6 This is a concept map representing an example of a reference image and a list of reference images.

[0029] Figure 7 This is a schematic diagram showing the configuration of a motion picture decoding device.

[0030] Figure 8 This is a schematic diagram showing the structure of the inter-frame prediction parameter decoding unit.

[0031] Figure 9 This is a schematic diagram showing the structure of the combined prediction parameter derivation unit and the AMVP prediction parameter derivation unit.

[0032] Figure 10 This is a schematic diagram showing the structure of the DMVR unit.

[0033] Figure 11 This is a flowchart illustrating the processing flow within the DMVR department.

[0034] Figure 12 This is a flowchart illustrating the process of determining BDOF based on error threshold processing using DMVR.

[0035] Figure 13 This is a schematic diagram showing the structure of the inter-frame predictive image generation unit included in the predictive image generation unit.

[0036] Figure 14 This is a flowchart illustrating an example of the process for selecting a prediction mode in a motion picture decoding device.

[0037] Figure 15 This is a flowchart illustrating the process of deriving the predicted image processing.

[0038] Figure 16 This is a schematic diagram showing the structure of the BDOF section.

[0039] Figure 17 This is a diagram showing an example of the area where BDOF performs filling.

[0040] Figure 18 This is a schematic diagram showing the configuration of a motion picture encoding device.

[0041] Figure 19 This is a schematic diagram showing the structure of the parameter encoding section.

[0042] Figure 20 This is a diagram representing an example of the number of candidate search distances and candidate derivation directions in a motion picture coding device. Detailed Implementation

[0043] (First Implementation)

[0044] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0045] Figure 1This is a schematic diagram showing the configuration of the image transmission system 1 of this embodiment.

[0046] Image transmission system 1 is a system for transmitting an encoded stream obtained by encoding an image of an encoded object, decoding the transmitted encoded stream, and displaying the image. Image transmission system 1 is configured to include: a moving image encoding device (image encoding device) 11, a network 21, a moving image decoding device (image decoding device) 31, and a moving image display device (image display device) 41.

[0047] The motion picture encoding device 11 is input to the image T.

[0048] Network 21 transmits the encoded stream Te generated by the motion picture encoding device 11 to the motion picture decoding device 31. Network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. Network 21 is not necessarily limited to a two-way communication network; it can also be a one-way communication network transmitting broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, network 21 can also be replaced by a storage medium containing the encoded stream Te, such as a DVD (Digital Versatile Disc) or a Blu-ray Disc (Blu-ray Disc).

[0049] The motion picture decoding device 31 decodes the encoded stream Te transmitted by the network 21 to generate one or more decoded images Td.

[0050] The moving image display device 41 displays all or part of one or more decoded images Td generated by the moving image decoding device 31. The moving image display device 41 may include, for example, a liquid crystal display (LCD), an organic EL (Electroluminescence) display, or other display devices. Examples of display types include fixed, mobile, and HMD (Head-Mounted Display). Furthermore, the moving image display device 41 displays high-quality images when the moving image decoding device 31 has high processing power, and displays images that do not require high processing power or high display power when it has only low processing power.

[0051] <operator>

[0052] The operators used in this specification are described below.

[0053] >> shifts to the right, << shifts to the left, & is bitwise AND, | is bitwise OR, |= is the OR substitution operator, and || represents logical AND.

[0054] x?y:z is a ternary operator that takes y when x is true (other than 0) and takes z when x is false (0).

[0055] Clip3(a, b, c) is a function that clips c to a value between a and b. It returns a when c < a, returns b when c > b, and returns c otherwise (where a <= b).

[0056] abs(a) is a function that returns the absolute value of a.

[0057] Int(a) is a function that returns the integer value of a.

[0058] floor(a) is a function that returns the largest integer less than or equal to a.

[0059] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0060] a / d means a divided by d (truncating the decimal part).

[0061] sign(a) is a function that returns the sign of a.

[0062] a^b means a to the power of b.

[0063] <Structure of the encoded stream Te>

[0064] Before explaining the moving image encoding device 11 and the moving image decoding device 31 of the present embodiment in detail, the data structure of the encoded stream Te generated by the moving image encoding device 11 and decoded by the moving image decoding device 31 will be described.

[0065] Figure 4 It is a diagram showing the hierarchical structure of the data in the encoded stream Te. The encoded stream Te exemplarily includes a sequence and multiple pictures constituting the sequence. Figure 4 Figures (a) to (f) respectively show the encoded video sequence of a given sequence SEQ, the encoded picture of a specified picture PICT, the encoded slice of a specified slice S, the encoded slice data of the specified slice data, the encoded tree unit included in the encoded slice data, and the encoded unit included in the encoded tree unit.

[0066] (Encoded video sequence)

[0067] In the encoded video sequence, a set of data for the moving image decoding device 31 to refer to for decoding the sequence SEQ to be processed is defined. The sequence SEQ is as Figure 4As shown in (a), it includes the Video Parameter Set, the Sequence Parameter Set (SPS), the Picture Parameter Set (PPS), the Picture PCT, and the Supplemental Enhancement Information (SEI).

[0068] The Video Parameter Set (VPS) defines a set of common coding parameters for multiple motion pictures in a motion picture composed of multiple layers, as well as a set of coding parameters for the multiple layers included in the motion picture and associated with each layer.

[0069] The Sequence Parameter Set (SPS) specifies a set of encoding parameters that the moving image decoding device 31 refers to in order to decode the object sequence. For example, it specifies the width and height of the image. It should be noted that multiple SPSs can exist. In this case, any one of the multiple SPSs is selected from the PPS.

[0070] The Picture Parameter Set (PPS) specifies a set of encoding parameters that the moving image decoding device 31 refers to for decoding each picture in the object sequence. These parameters include, for example, a reference value for the quantization width used for picture decoding (pic_init_qp_minus26) and a flag indicating the application of weighted prediction (weighted_pred_flag). It should be noted that multiple PPSs can exist. In this case, any one of the multiple PPSs is selected from the pictures in the object sequence.

[0071] (Encoded image)

[0072] In the encoded image, a set of data is specified for the motion picture decoding device 31 to refer to in order to decode the image PICT of the object being processed. The image PICT is as follows: Figure 4 As shown in (b), it includes slice 0 to slice NS-1 (NS is the total number of slices included in the image PICT).

[0073] It should be noted that, in the following descriptions, where there is no need to distinguish between slices 0 to NS-1, the subscripts of the figure labels may sometimes be omitted. Furthermore, the same applies to other data with subscripts included in the encoded stream Te described below.

[0074] (Encoded slice)

[0075] In the encoded slice, a set of data is specified for the motion picture decoding device 31 to refer to in order to decode the slice S of the object being processed. The slice is as follows: Figure 4 (c) shows the slice header and slice data.

[0076] The slice header includes a set of encoded parameters for the moving image decoding device 31 to refer to in order to determine the decoding method for the object slice. The slice type specification information (slice_type) is an example of the encoded parameters included in the slice header.

[0077] As slice types that can be specified by slice type specification information, the following can be listed: (1) I slices that use only intra-frame prediction during encoding, (2) P slices that use unidirectional prediction or intra-frame prediction during encoding, and (3) B slices that use unidirectional prediction, bidirectional prediction, or intra-frame prediction during encoding. It should be noted that inter-frame prediction is not limited to unidirectional or bidirectional prediction, and more reference images can be used to generate the prediction image. Hereinafter, the cases referred to as P and B slices refer to slices that include blocks that can use inter-frame prediction.

[0078] It should be noted that the slice header may also include a reference to the image parameter set (PPS) (pic_parameter_set_id).

[0079] (Encoded slice data)

[0080] In the encoded slice data, a set of data is specified for the motion picture decoding device 31 to refer to in order to decode the slice data of the object being processed. Slice data such as... Figure 4 (d) shows the CTU. A CTU is a fixed-size (e.g., 64×64) block that makes up a slice, also known as the Largest Coding Unit (LCU).

[0081] (Coding Tree Unit)

[0082] exist Figure 4 In section (e), a set of data is specified for the motion picture decoding device 31 to refer to in order to decode the CTU of the processing object. The CTU is divided into coding units (CUs) as the basic unit of coding processing through recursive quadtree (QT) segmentation, binary tree (BT) segmentation, or ternary tree (TT) segmentation. BT segmentation and TT segmentation are collectively referred to as multitree (MT) segmentation. The nodes of the tree structure obtained by recursive quadtree segmentation are called coding nodes. The intermediate nodes of quadtrees, binary trees, and ternary trees are coding nodes, and the CTU itself is also defined as the top-level coding node.

[0083] CT includes the following information as CT information: a QT splitting flag (qt_split_cu_flag) indicating whether QT splitting is performed, an MT splitting flag (mtt_split_cu_flag) indicating whether MT splitting is performed, an MT splitting direction flag (mtt_split_cu_vertical_flag) indicating the MT splitting direction, and an MT splitting type flag (mtt_split_cu_binary_flag) indicating the MT splitting type. qt_split_cu_flag, mtt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted per coding node.

[0084] Figure 5 This is a diagram representing a CTU split example. With qt_split_cu_flag set to 1, the encoding node is split into 4 encoding nodes ( Figure 5 (b)

[0085] When qt_split_cu_flag is 0, and mtt_split_cu_flag is 0, the encoding nodes are not split, and one CU is retained as a node. Figure 5 (a)). CU is the terminal node of the encoding node and is not further segmented. CU is the basic unit of encoding processing.

[0086] When mtt_split_cu_flag is 1, the encoding node is split into two MT nodes as described below. When mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 1, the encoding node is horizontally split into two encoding nodes. Figure 5 (d) When mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 1, the encoding node is vertically split into 2 encoding nodes. Figure 5 (c)). Furthermore, when mtt_split_cu_vertical_flag is 0 and mtt_split_cu_binary_flag is 0, the encoding node is horizontally split into 3 encoding nodes ( Figure 5 (f) means that when mtt_split_cu_vertical_flag is 1 and mtt_split_cu_binary_flag is 0, the encoding node is vertically split into 3 encoding nodes. Figure 5 (e)). They will be in Figure 5As shown in (g).

[0087] Furthermore, when the size of the CTU is 64×64 pixels, the size of the CU can be any of the following: 64×64 pixels, 64×32 pixels, 32×64 pixels, 32×32 pixels, 64×16 pixels, 16×64 pixels, 32×16 pixels, 16×32 pixels, 16×16 pixels, 64×8 pixels, 8×64 pixels, 32×8 pixels, 8×32 pixels, 16×8 pixels, 8×16 pixels, 8×8 pixels, 64×4 pixels, 4×64 pixels, 32×4 pixels, 4×32 pixels, 16×4 pixels, 4×16 pixels, 8×4 pixels, 4×8 pixels, and 4×4 pixels.

[0088] (Encoding unit)

[0089] like Figure 4 As shown in (f), the CU contains a set of data that the motion picture decoding device 31 refers to in order to decode the encoding unit of the object being processed. Specifically, the CU consists of a CU header CUH, prediction parameters, transform parameters, quantization transform coefficients, etc. The CU header specifies the prediction mode, etc.

[0090] Prediction processing can be performed on a per-unit basis (CU) or on a per-unit basis, based on further subdivisions of the CU into sub-CUs. When the size of the CU and its sub-CUs are equal, there is one sub-CU within the CU. When the size of the CU is larger than the size of its sub-CUs, the CU is divided into sub-CUs. For example, if the CU is 8×8 and the sub-CUs are 4×4, the CU is divided into four sub-CUs, comprising two horizontally divided parts and two vertically divided parts.

[0091] There are two types of prediction (prediction modes): intra-frame prediction and inter-frame prediction. Intra-frame prediction is prediction within the same image, while inter-frame prediction refers to prediction processing performed between different images (e.g., between display times).

[0092] Transform / quantization processing is performed on a unit basis (CU), but quantization transform coefficients can also be entropy encoded on a unit basis (4×4 sub-blocks).

[0093] (Prediction parameters)

[0094] The predicted image is derived from the prediction parameters appended to the block. These prediction parameters include those for intra-frame and inter-frame prediction.

[0095] The prediction parameters for inter-frame prediction are explained below. The inter-frame prediction parameters consist of prediction list utilization flags predFlagL0 and predFlagL1, reference image indices refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. PredFlagL0 and predFlagL1 indicate whether the reference image lists, referred to as L0 and L1 lists, are used for inter-frame prediction. A value of 1 indicates that the corresponding reference image list is used for inter-frame prediction. It should be noted that in this specification, when a flag is referred to as "a flag indicating whether it is ××", a flag other than 0 (e.g., 1) is considered to be ××, and a flag of 0 is considered not to be ××. In logical NOT, logical product, etc., 1 is considered true, and 0 is considered false (the same applies below). However, in actual devices and methods, other values ​​can be used as true and false values.

[0096] The syntax elements used to derive inter-frame prediction parameters include, for example, the affine flag, the merge flag, the merge index, the inter-frame prediction identifier, the reference image index, the prediction vector index, the prediction vector index, the difference vector, and the motion vector precision mode, amvr_mode.

[0097] (Refer to the image list)

[0098] The reference image list is a list of reference images stored in the reference image memory 306. Figure 6 This is a concept diagram representing an example of a reference image and a list of reference images. Figure 6 In (a), rectangles represent images, arrows indicate the reference relationships between images, the horizontal axis represents time, and I, P, and B within the rectangles represent intra-frame images, one-way prediction images, and two-way prediction images, respectively. The numbers within the rectangles indicate the decoding order. For example... Figure 6 As shown in (a), the decoding order of the image is I0, P1, B2, B3, B4, and the display order is I0, B3, B2, B4, P1. Figure 6 Example of a list of reference images for image B3 (the object image) is shown in (b). A list of reference images is a list of candidates for reference images; an image (slice) can have more than one list of reference images. Figure 6 In example (b), object image B3 has two reference image lists: L0 list RefPicList0 and L1 list RefPicList1. In each CU, the reference image index refIdxLX specifies which image in the reference image list RefPicListX (X=0 or 1) is actually referenced. Figure 6 (b) is an example where refIdxL0=2 and refIdxL1=0. It should be noted that LX is a notation used without distinguishing between L0 prediction and L1 prediction. In the following, LX will be replaced with L0 and L1 to distinguish between parameters for the L0 list and parameters for the L1 list.

[0099] (Merged forecast and AMVP forecast)

[0100] There are two methods for decoding (encoding) prediction parameters: merge prediction mode and AMVP (Advanced Motion Vector Prediction) mode. The merge flag is used to identify them. Merge prediction mode is a mode that derives prediction parameters from processed neighboring blocks by not including the prediction list in the encoded data using the predFlagLX (or inter-prediction identifier inter_pred_idc), the reference image index refIdxLX, and the motion vector mvLX. AMVP mode includes the inter-prediction identifier inter_pred_idc, the reference image index refIdxLX, and the motion vector mvLX in the encoded data. It should be noted that the motion vector mvLX is encoded as the prediction vector index mvp_LX_idx, the difference vector mvdLX, and the motion vector precision mode amvr_mode, which identify the prediction vector mvpLX. The merge prediction mode is a mode that selects merge candidates derived from motion information of adjacent blocks to obtain the motion vector mvLX (motion vector information). In addition to the merge prediction mode, there can also be an affine prediction mode identified by the affine flag `affine_flag`. As another type of merge prediction mode, there can be a skip mode identified by the skip flag `skip_flag`. It should be noted that the skip mode refers to the mode used to derive prediction parameters using the same method as the merge mode, but it does not include prediction errors (residual image, residual information) in the encoded data. In other words, when the skip flag `skip_flag` is 1, for the object CU, only the `skip_flag` and the syntax associated with the merge mode, such as the merge index `merge_idx`, are included; motion vectors, residual information, etc., are not included in the encoded data.

[0101] (Motion vector)

[0102] The motion vector mvLX represents the amount of displacement between blocks in two different images. The prediction vector and the difference vector related to the motion vector mvLX are called the prediction vector mvpLX and the difference vector mvdLX, respectively.

[0103] (The inter-frame prediction identifier inter_pred_idc and the prediction list utilize the predFlagLX)

[0104] The inter-frame prediction identifier `inter_pred_idc` is a value representing the type and number of reference images, taking any one of `PRED_L0`, `PRED_L1`, or `PRED_BI`. `PRED_L0` and `PRED_L1` represent unidirectional prediction using a single reference image managed in the L0 and L1 lists, respectively. `PRED_BI` represents bidirectional prediction (BiPred) using two reference images managed in both the L0 and L1 lists.

[0105] The merge index merge_idx is an index indicating whether any of the prediction parameters from the prediction parameter candidates (merge candidates) derived from the processed block are used as prediction parameters for the object block.

[0106] The inter-frame prediction identifier inter_pred_idc and the prediction list can be transformed into each other using the flags predFlagL0 and predFlagL1 as follows.

[0107] inter_pred_idc=(predFlagL1<<1)+predFlagL0

[0108] predFlagL0= inter_pred_idc&1

[0109] predFlagL1= inter_pred_idc>>1

[0110] (Determination of bidirectional prediction biPred)

[0111] The flag indicating whether a BiPred is a bidirectional predictor can be derived from both prediction lists by checking if both flags are 1. For example, it can be derived using the following formula.

[0112] biPred=(predFlagL0==1&&predFlagL1==1)

[0113] Alternatively, the flag biPred can also be derived from whether the inter-frame prediction identifier indicates the use of two prediction lists (see image). For example, it can be derived using the following formula.

[0114] biPred=(inter_pred_idc==PRED_BI)? 1:0

[0115] (Composition of a motion picture decoding device)

[0116] The motion image decoding device 31 of this embodiment ( Figure 7 The composition of ) will be explained.

[0117] The motion picture decoding device 31 is configured to include: an entropy decoding unit 301, a parameter decoding unit 302, a loop filter 305, a reference image memory 306, a prediction parameter memory 307, a prediction image generation unit (prediction image generation device) 308, an inverse quantization / inverse transform unit 311, and an adder unit 312. It should be noted that, according to the motion picture encoding device 11 described later, there is also a configuration where the motion picture decoding device 31 does not include the loop filter 305.

[0118] The parameter decoding unit 302 also includes a header decoding unit 3020 (not shown), a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit). The CU decoding unit 3022 also includes a TU decoding unit 3024. These can be collectively referred to as decoding modules. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, and PPS, and the slice header (slice information) from the encoded data. The CT information decoding unit 3021 decodes the CT from the encoded data. The CU decoding unit 3022 decodes the CU from the encoded data. In the case where the TU includes prediction errors, the TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data.

[0119] The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data when the skip mode is not enabled (skip_mode==0). More specifically, when skip_mode==0, the TU decoding unit 3024 decodes the flag cu_cbp indicating whether the target block contains quantization prediction error from the encoded data; if cu_cbp is 1, the quantization prediction error is decoded. If cu_cbp is not present in the encoded data, the TU decoding unit 3024 derives cu_cbp as 0.

[0120] Furthermore, the parameter decoding unit 302 is configured to include an inter-frame prediction parameter decoding unit 303 (not shown) and an intra-frame prediction parameter decoding unit 304. The prediction image generation unit 308 is configured to include an inter-frame prediction image generation unit 309 and an intra-frame prediction image generation unit 310.

[0121] Furthermore, examples of using CTU and CU as processing units are described below, but this is not the only approach; processing can also be performed on a sub-CU basis. Alternatively, processing can be performed on a block or sub-block basis by replacing CTU and CU with blocks and sub-CUs with sub-blocks.

[0122] The entropy decoding unit 301 performs entropy decoding on the externally input encoded stream Te, decoding each code (syntactic element). The decoded code contains prediction information for generating the predicted image and prediction errors for generating the difference image.

[0123] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code may include, for example, predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. The unit controls which code to decode based on the instructions from the parameter decoding unit 302.

[0124] (The structure of the inter-frame prediction parameter decoding unit)

[0125] The inter-frame prediction parameter decoding unit 303 decodes the inter-frame prediction parameters based on the code input from the entropy decoding unit 301 and with reference to the prediction parameters stored in the prediction parameter memory 307. Furthermore, the inter-frame prediction parameter decoding unit 303 outputs the decoded inter-frame prediction parameters to the prediction image generation unit 308 and stores them in the prediction parameter memory 307.

[0126] Figure 8 This is a schematic diagram showing the configuration of the inter-frame prediction parameter decoding unit 303 according to this embodiment. The inter-frame prediction parameter decoding unit 303 is configured to include: a merging prediction unit 30374, a DMVR unit 30375, a sub-block prediction unit (affine prediction unit) 30372, an MMVD prediction unit 30376, a triangle prediction unit 30377, an AMVP prediction parameter derivation unit 3032, and an addition unit 3038. The merging prediction unit 30374 is configured to include a merging prediction parameter derivation unit 3036. The AMVP prediction parameter derivation unit 3032, the merging prediction parameter derivation unit 3036, and the affine prediction unit 30372 are common units in moving picture coding devices and moving picture decoding devices, and therefore can also be collectively referred to as a motion vector derivation unit (motion vector derivation device).

[0127] (Affine Prediction Department)

[0128] The affine prediction unit 30372 derives the affine prediction parameters of the object block. In this embodiment, the motion vectors (mv0_x, mv0_y) and (mv1_x, mv1_y) of two control points (V0, V1) of the object block are derived as affine prediction parameters. Specifically, the motion vectors of each control point can be derived by predicting the motion vectors of the adjacent blocks of the object block, or the motion vectors of each control point can be derived by summing the predicted vectors of the motion vectors derived as control points with the difference vectors derived from the encoded data.

[0129] (Combined Forecast)

[0130] Figure 9 (a) is a schematic diagram showing the configuration of the merge prediction parameter derivation unit 3036 included in the merge prediction unit 30374. The merge prediction parameter derivation unit 3036 includes a merge candidate derivation unit 30361 and a merge candidate selection unit 30362. It should be noted that the merge candidate configuration includes: a prediction list using a flag predFlagLX, a motion vector mvLX, and a reference image index refIdxLX, which are stored in the merge candidate list. The index is assigned to the merge candidates stored in the merge candidate list according to a prescribed rule.

[0131] The merge candidate derivation unit 30361 directly uses the motion vector of the decoded adjacent block and the reference image index refIdxLX to derive the merge candidate.

[0132] The order in which merge candidate lists are stored is, for example, spatial merge candidate A1, B1, B0, A0, B2, temporal merge candidate Col, paired merge candidate avgK, and zero merge candidate ZK. It should be noted that unusable reference blocks (such as intra-frame prediction blocks) are not stored in the merge candidate list.

[0133] The merge candidate selection unit 30362 selects the merge candidate N, indicated by the merge index merge_idx, from the merge candidates included in the merge candidate list using the following formula.

[0134] N = mergeCandList[merge_idx]

[0135] Here, N represents the label of the merging candidate, which can be A1, B1, B0, A0, B2, Col, avgK, ZK, etc. The motion information of the merging candidate shown by label N is represented by (mvLXN[0], mvLXN[1]), predFlagLXN, refIdxLXN.

[0136] The candidate merging unit 30362 selects the motion information (mvLXN[0], mvLXN[1]), predFlagLXN, and refIdxLXN of the selected merging candidates as the inter-frame prediction parameters of the target block. The candidate merging unit 30362 stores the inter-frame prediction parameters of the selected merging candidates in the prediction parameter memory 307 and outputs them to the prediction image generation unit 308.

[0137] (AMVP Prediction)

[0138] Figure 9Figure (b) is a schematic diagram showing the configuration of the AMVP prediction parameter derivation unit 3032 of the present embodiment. The AMVP prediction parameter derivation unit 3032 includes a vector candidate derivation unit 3033 and a vector candidate selection unit 3034. The vector candidate derivation unit 3033 derives prediction vector candidates based on the reference picture index refIdxLX from the motion vectors mvLX of the decoded adjacent blocks stored in the prediction parameter memory 307, and stores them in the prediction vector candidate list mvpListLX[].

[0139] The vector candidate selection unit 3034 selects the motion vector mvpListLX[mvp_LX_idx] indicated by the prediction vector index mvp_LX_idx among the prediction vector candidates in the prediction vector candidate list mvpListLX[] as the prediction vector mvpLX. The vector candidate selection unit 3034 outputs the selected prediction vector mvpLX to the addition unit 3038.

[0140] Note that the prediction vector candidates are derived by scaling the motion vectors of the decoded adjacent blocks within a range preset for the target block. Note that the adjacent blocks include not only the blocks spatially adjacent to the target block, such as the left block and the upper block, but also the regions temporally adjacent to the target block. For example, the regions obtained according to the prediction parameters of the blocks that have the same position as the target block but different display times.

[0141] The addition unit 3038 adds the prediction vector mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded differential vector mvdLX, and calculates the motion vector mvLX. The addition unit 3038 outputs the calculated motion vector mvLX to the prediction image generation unit 308 and the prediction parameter memory 307.

[0142] mvLX[0]=mvpLX[0]+mvdLX[0]

[0143] mvLX[1]=mvpLX[1]+mvdLX[1]

[0144] The motion vector precision mode amvr_mode is a syntax for switching the precision of the motion vector derived in the AMVP mode. For example, when amvr_mode = 0, 1, 2, it switches the precision to 1 / 4 pixel, 1 pixel, and 4 pixels.

[0145] When the precision of the motion vector is set to 1 / 16 precision (MVPREC = 16), the parameter decoding unit 302 can perform inverse quantization using MvShift (= 1 << amvr_mode) derived from amvr_mode as described below to change the motion vector differential with 1 / 4, 1, 4 pixel precision to a motion vector differential with 1 / 16 pixel precision.

[0146] mvdLX[0]=mvdLX[0]<<(MvShift+2)

[0147] mvdLX[1]=mvdLX[1]<<(MvShift+2)

[0148] It should be noted that the parameter decoding unit 302 can also decode the following syntax to derive mvdLX[] before the above MvShift shift.

[0149] · abs_mvd_greater0_flag

[0150] · abs_mvd_minus2

[0151] · mvd_sign_flag

[0152] Then, the parameter decoding unit 302 decodes the difference vector lMvd[] from the syntax using the following formula.

[0153] lMvd[compIdx]=abs_mvd_greater0_flag[compIdx]*(abs_mvd_minus2[compIdx]+2)*(1-2*mvd_sign_flag[compIdx])

[0154] Furthermore, the parameter decoding unit 302 sets the decoded differential vector lMvd[] to MvdLX in the case of translation MVD (MotionModelIdc[x][y]==0), and sets the decoded differential vector lMvd[] to MvdCpLX in the case of control point MVD (MotionModelIdc[x][y]!=0).

[0155] if (MotionModelIdc[x][y] == 0)

[0156] mvdLX[x0][y0][compIdx]=lMvd[compIdx]

[0157] else

[0158] mvdCpLX[x0][y0][compIdx]=lMvd[compIdx]<<2

[0159] (Motion vector scaling)

[0160] The derivation method for scaling motion vectors is explained. When the motion vector is Mv (reference motion vector), the image PicMv containing the block with Mv, the reference image PicMvRef of Mv, the scaled motion vector sMv, the image CurPic containing the block with sMv, and the reference image CurPicRef referenced by sMv, the derivation function MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef) of sMv is expressed by the following formula.

[0161] sMv=MvScale(Mv,PicMv,PicMvRef,CurPic,CurPicRef)

[0162] =Clip3(-R1, R1-1, sign(distScaleFactor*Mv)*((abs(distScaleFactor*Mv)+round1-1)>>shift1))

[0163] distScaleFactor=Clip3 (-R2, R2-1, (tb*tx+round2)>>shift2)

[0164] tx=(16384+abs(td)>>1) / td

[0165] td=DiffPicOrderCnt(PicMv,PicMvRef)

[0166] tb=DiffPicOrderCnt(CurPic, CurPicRef)

[0167] Here, round1, round2, shift1, and shift2 are rounded and shifted values ​​used for division using reciprocals. For example, round1 = 1 << (shift1 - 1), round2 = 1 << (shift2 - 1), shift1 = 8, shift2 = 6, etc. DiffPicOrderCnt(Pic1, Pic2) is a function that returns the difference between the time information (e.g., POC) of Pic1 and Pic2. R1 and R2 are values ​​that restrict the range to be processed with limited precision, for example, R1 = 32768, R2 = 4096, etc.

[0168] In addition, the scaling function MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef) can be the following formula.

[0169] MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef) = Mv*DiffPicOrderCnt(CurPic, CurPicRef) / DiffPicOrderCnt(PicMv, PicMvRef)

[0170] That is, Mv can be scaled based on the difference between the time information of CurPic and CurPicRef and the difference between the time information of PicMv and PicMvRef.

[0171] (DMVR Department 30375)

[0172] Next, the DMVR (Decoder-side Motion Vector Refinement) processing performed by the DMVR unit 30375 will be explained. DMVR processing refers to the process of correcting motion vectors mvL0 and mvL1 using two reference images.

[0173] Figure 10 This is a schematic diagram showing the configuration of the DMVR unit 30375. (Used...) Figure 10 The specific processing performed by the DMVR unit 30375 will be explained. The DMVR unit 30375 includes: a prediction image generation unit 303751 for corrected motion vector search, an initial error generation unit 303752, a motion vector search unit 303753, and a corrected vector derivation unit 303754.

[0174] DMVR section 30375 refers to the data from the consolidated forecast section 30374.

[0175] • The top-left position of the object block (xCb, yCb)

[0176] • The width of the object block, bW

[0177] • Height of the object block bH

[0178] • Motion vectors mvL0 and mvL1 with 1 / 16 pixel precision

[0179] • Refer to images refPicL0L and refPicL1L

[0180] The changes dmvL0 and dmvL1 used to correct the motion vectors of mvL0 and mvL1 are derived and output to the inter-frame prediction image generation unit 309.

[0181] First, the reference for the motion vector search prediction image generation unit 303751 is corrected.

[0182] • The top-left position of the object's sub-block (xSb, ySb)

[0183] • Width of the object sub-block of brightness (sbW)

[0184] • Height of the object sub-block of brightness sbH

[0185] • Motion vector mvLX (X=0,1)

[0186] • Refer to the image refPicLXL (X=0,1)

[0187] The predicted image predSamplesLXL with size (sbW)*(sbH) is derived.

[0188] In the prediction image generation unit 303751 for correcting motion vector search, the motion vector MvLsX (X=0,1) is derived by the following formula.

[0189] MvLsX[0]=MvLX[0]-32

[0190] MvLsX[1]=MvLX[1]-32

[0191] In addition, the DMVR unit 30375 sets the values ​​of variables srRange, offsetH[0], offsetV[0], offsetH[1], and offsetV[1] to 2 respectively.

[0192] The position of the pixel corresponding to the pixel position (xL, yL) in the object block, in integer pixels of the reference block, is set as (xIntL, yIntL). In addition, the offset from (xIntL, yIntL) in 1 / 16 pixel units is set as (xFracL, yFracL). These coordinates are derived from the integer components (mvLX[0]>>4, mvLX[1]>>4) and fractional components (mvLX[0]&15, mvLX[1]&15) of the motion vector (mvLX[0], mvLX[1]), representing the position of the pixel in the reference image refPicLXL with fractional precision. For the pixel at position (xL, yL) (xL=0, ……, sbW-1, yL=0, ……, sbH-1) in predSamplesLXL, DMVR unit 30375 derives xIntL, yIntL, xFracL and yFracL by the following formula.

[0193] xIntL=xSb+(mvLX[0]>>4)+xL

[0194] yIntL=ySb+(mvLX[1]>>4)+yL

[0195] xFracL=mvLX[0]&15

[0196] yFracL=mvLX[1]&15

[0197] Next, DMVR Department 30375 reference

[0198] ·(xIntL,yIntL)

[0199] ·(xFracL, yFracL)

[0200] · refPicLXL

[0201] Derive predSamplesLXL.

[0202] First, the prediction image generation unit 303751 for correcting motion vector search derives the variables shift1, shift2, shift3 and shift4 through the following formula.

[0203] shift1=BitDepthY-6

[0204] offset1=1 << (shift1-1)

[0205] shift2=4

[0206] offset2=8

[0207] shift3 = 10 - BitDepthY

[0208] offset3 = 1 << (shift3 - 1)

[0209] shift4=BitDepthY-10

[0210] It should be noted that in the above formula, BitDepthY represents the number of pixels in bits.

[0211] Next, the motion vector search prediction image generation unit 303751 is corrected to set picW to a value equal to the image width pic_width_in_luma_samples. Furthermore, the motion vector search prediction image generation unit 303751 is corrected to set the image height pic_height_in_luma_samples to be equal to picH.

[0212] Subsequently, in the predicted image generation unit 303751 for correcting motion vector search, predSamplesLXL is derived as described below. In the following description, fb L[p] represents the filter coefficients used to derive pixel values ​​with 1 / 16 pixel precision. fb L The value of [p] depends on the position p (p = 1, 2, ..., 15) with a precision of 1 / 16 pixels. The position p is equal to xFracL or yFracL. As the value of p increases, fb L [p][0] monotonically decreasing, fb L The value of [p][1] increases monotonically.

[0213] First, in the predicted image generation unit 303751 for corrected motion vector search, it is determined whether xFracL and yFracL are both 0. If either xFracL or yFracL is 0, the DMVR unit 30375 derives predSamplesLXL based on the value of BitDepthY using any of the following formulas.

[0214] predSamplesLXL=(BitDepthY<=10)? (refPicLXL[xIntL][yIntL]<< shift3): ((refPicLXL[xIntL][yIntL]+offset3)>> shift4)

[0215] When xFracL is not 0 and yFracL is 0, the prediction image generation unit 303751 for correcting motion vector search derives predSamplesLXL using the following formula.

[0216] predSamplesLXL=(fb L [xFracL][0]*refPicLXL[Clip3(0,picW-1,xIntL)][yIntL]+fb L [xFracL][1]*refPicLXL[Clip3(0,picW-1,xIntL+1)][yIntL]+offset1)>>shift1

[0217] When xFracL is 0 and yFracL is not 0, DMVR 30375 derives predSamplesLXL using the following formula.

[0218] predSamplesLXL=(fb L [yFracL][0]*refPicLXL[xIntL][Clip3(0,picH-1,yIntL)]+fb L[yFracL][1]*refPicLXL[xIntL][Clip3(0,picH-1,yIntL+1)]+offset1)>>shift1

[0219] When both xFracL and yFracL are not 0, the prediction image generation unit 303751 for correcting motion vector search derives predSamplesLXL as follows. First, the DMVR unit 30375 derives temp[n] using the following formula. The derivation process of temp[] is performed n times by changing the reference position. n=0 indicates the first derivation process, and n=1 indicates the second derivation process.

[0220] yPosL=Clip3(0,PicH-1,yIntL+n-3)

[0221] temp[n] = (fb) L [xFracL][0]*refPicLXL[Clip3(0,picW-1,xIntL)][yPosL]+fb L [xFracL][1]*refPicLXL[Clip3(0,picW-1,xIntL+1)][yPosL]+offset1)>>shift1

[0222] Subsequently, DMVR Department 30375 derived predSamplesLXL using the following formula.

[0223] predSamplesLXL=(fb L [yFracL][0]*temp[0]+fb L [yFracL][1]*temp[1])>>shift2

[0224] Next, the initial error generation unit 303752 refers to

[0225] • The width of the object block is nCbW

[0226] • Height of the object block nCbH

[0227] • Two prediction images, predSampleL1 and predSampleL2, with a size of (nCbW+4) x (nCbH+4).

[0228] • Variables offsetH[0], offsetH[1], offsetV[0], and offsetV[1]

[0229] Derive the list Sad1 and the variable centerSad, which are the sum of absolute differences of pixel values ​​included in predSampleL1 and predSampleL2.

[0230] The DMVR unit 30375 sets the values ​​of each element of the 2×9 array bC as shown in the following formula.

[0231] bC[0][0]=-1 bC[1][0]=-1

[0232] bC[0][1]=-1 bC[1][1]=0

[0233] bC[0][2]=-1 bC[1][2]=1

[0234] bC[0][3]=0 bC[1][3]=-1

[0235] bC[0][4]=0 bC[1][4]=0

[0236] bC[0][5]=0 bC[1][5]=1

[0237] bC[0][6]=1 bC[1][6]=-1

[0238] bC[0][7]=1 bC[1][7]=0

[0239] bC[0][8]=1 bC[1][8]=1

[0240] The initial error generation unit 303752 derives the elements sadList[i] of Sad1 (i=0,……,8) through the following formula.

[0241] [Formula 1]

[0242]

[0243] Furthermore, the initial error generation unit 303752 derives centerSad through the following formula.

[0244] [Formula 2]

[0245]

[0246] The initial error generation unit 303752 determines whether centerSad is equal to or greater than (bH >> 1) * (bW) * 4. The dmvrFlag is a flag indicating that DMVR processing is performed when it is true (TRUE) and not performed when it is false (FALSE). When centerSad is less than (bH >> 1) * (bW) * 4, the error is small. Therefore, the initial error generation unit 303752 determines that DMVR processing is not required, sets dmvrFlag to false, does not correct the motion vector, and proceeds to the inter-frame prediction image generation unit 309.

[0247] When centerSad is equal to or greater than (bH >> 1) * (bW) * 4, the initial error generation unit 303752 sets dmvrFlag to true, and the motion vector search unit 303753 refers to

[0248] · The number of search points n

[0249] · The element sadList of the list Sad1 of the sum of absolute differences of search points

[0250] to derive the index bestIdx. n is a positive integer.

[0251] Hereinafter, the case of n = 9 will be described. It should be noted that the value of the number of search points n can be a value other than 9. For example, in addition to the method described in the present embodiment, the minimum value of the sadList value can be simply selected when n = 25.

[0252] The motion vector search unit 303753 determines whether sadList[1] < sadList[7] and whether sadList[3] < sadList[5].

[0253] When sadList[1] < sadList[7] and sadList[3] < sadList[5], the DMVR unit 30375 sets the value of idx to 0. Thereafter, the motion vector search unit 303753 determines whether sadList[1] < sadList[3]. The motion vector search unit 303753 sets the value of bestIdx to 1 when sadList[1] < sadList[3], and sets the value of bestIdx to 3 when sadList[1] is not less than sadList[3].

[0254] In the case where it is not so, when sadList[1] >= sadList[7] and sadList[3] < sadList[5], the motion vector search unit 303753 sets the value of idx to 6. After that, the DMVR unit 30375 determines whether sadList[7] < sadList[3]. When sadList[7] < sadList[3], the motion vector search unit 303753 sets the value of bestIdx to 7, and when it is not the case that sadList[7] < sadList[3], the motion vector search unit 303753 sets the value of bestIdx to 3.

[0255] In the case where it is not so, when sadList[1] < sadList[7] and sadList[3] >= sadList[5], the motion vector search unit 303753 sets the value of idx to 2. After that, the DMVR unit 30375 determines whether sadList[1] < sadList[5]. When sadList[1] < sadList[5], the DMVR unit 30375 sets the value of bestIdx to 1, and when it is not the case that sadList[1] < sadList[5], the DMVR unit 30375 sets the value of bestIdx to 5.

[0256] In the case where it is not so, when sadList[1] >= sadList[7] and sadList[3] >= sadList[5], the motion vector search unit 303753 sets the value of idx to 8. After that, the DMVR unit 30375 determines whether sadList[7] < sadList[5]. When sadList[7] < sadList[5], the DMVR unit 30375 sets the value of bestIdx to 7, and when it is not the case that sadList[7] < sadList[5], the DMVR unit 30375 sets the value of bestIdx to 5.

[0257] Furthermore, the motion vector search unit 303753 determines whether sadList[4] <= sadList[bestIdx]. When sadList[4] <= sadList[bestIdx], the DMVR unit 30375 updates the value of bestIdx to 4. On the other hand, when it is not the case that sadList[4] <= sadList[bestIdx], the DMVR unit 30375 does not update the value of bestIdx.

[0258] Furthermore, the motion vector search unit 303753 determines whether sadList[idx] < sadList[bestIdx]. If sadList[idx] < sadList[bestIdx], the DMVR unit 30375 sets bestIdx to idx. On the other hand, if it is not the case that sadList[idx] < sadList[bestIdx], the motion vector search unit 303753 does not update the value of bestIdx.

[0259] The motion vector search unit 303753 determines whether the value of bestIdx is 4. If the value of bestIdx is 4, the motion vector search unit 303753 sets halfPelAppliedflag to true.

[0260] If the value of bestIdx is not 4, the motion vector search unit 303753 calculates the values of variables dmvx and dmvy using the following equations.

[0261] dmvx = (bestIdx / 3 - 1)

[0262] dmvy = (bestIdx % 3 - 1)

[0263] Furthermore, the motion vector search unit 303753 updates offsetH and offsetV using the following equations.

[0264] offsetH[0] = offsetH[0] + dmvx, offsetV[0] = offsetV[0] + dmvy

[0265] offsetH[1] = offsetH[1] - dmvx, offsetV[1] = offsetV[1] - dmvy

[0266] The motion vector search unit 303753 derives Sad2 using the updated offsetH and offsetV through the same process as the process of deriving Sad1 above. Furthermore, the motion vector search unit 303753 derives bestIdx again using Sad2 instead of Sad1.

[0267] The motion vector search unit 303753 determines whether the value of bestIdx derived again is 4. If the value of bestIdx is 4, the motion vector search unit 303753 sets halfPelAppliedflag to true.

[0268] When the value of bestIdx is not 4, the motion vector search unit 303753 calculates dmvx and dmvy using the following formula.

[0269] dmvx=(bestIdx / 3-1),dmvy=(bestIdx%3-1)

[0270] Furthermore, the DMVR unit 30375 calculates dmvL0 and dmvL1 using the following formula.

[0271] dmvL0[0]=16*dmvx, dmvL0[1]=16*dmvy

[0272] dmvL1[0]=-16*dmvx, dmvL1[1]=-16*dmvy

[0273] When halfPelAppliedflag is true, the motion vector search unit 303753 derives the corrected dmvL0 and dmvL1 as described below. It should be noted that the following sadList contains elements of Sad2 if Sad2 exists, and elements of Sad1 if Sad2 does not exist.

[0274] First, the motion vector search unit 303753 determines whether sadList[1]+sadList[7]==sadList[4]. In the case that sadList[1]+sadList[7]==sadList[4] and mrSadT+mrSadB-(mrSadC<<1)==0, the motion vector search unit 303753 sets dmv[0]=0. In the case that sadList[1]+sadList[7]==sadList[4] is not true, the motion vector search unit 303753 calculates dmv[0] using the following formula.

[0275] dmv[0]=((sadList[1]-sadList[7])<<3) / (sadList[1]+sadList[7]-(sadList[4]<<1))

[0276] Next, the modified vector derivation unit 303754 determines whether sadList[3]+sadList[5]==sadList[4]. In the case that sadList[3]+sadList[5]==sadList[4] and mrSadL+mrSadR-(mrSadC<<1)==0, the modified vector derivation unit 303754 sets dmv[1]=0. In the case that sadList[3]+sadList[5]==sadList[4] is not true, the modified vector derivation unit 303754 calculates dmv[1] using the following formula.

[0277] dmv[1]=((sadList[3]-sadList[5])<<3) / (sadList[3]+sadList[5]-(sadList[4]<<1))

[0278] Furthermore, the modified vector derivation unit 303754 corrects the motion vectors mvL0 and mvL1 using the following formula.

[0279] dmvL0[0]=dmvL0[0]+dmv[0]

[0280] dmvL0[1]=dmvL0[1]+dmv[1]

[0281] dmvL1[0] = dmvL1[0] - dmv[0]

[0282] dmvL1[1]=dmvL1[1]-dmv[1]

[0283] The DMVR unit 30375 calculates the motion vector mvLX by adding the derived difference vector dmvLX to the prediction vector mvpLX input from the merging prediction unit 30374. The DMVR unit 30375 outputs mvLX to the inter-frame prediction image generation unit 309.

[0284] mvLX[0]=mvpLX[0]+dmvLX[0]

[0285] mvLX[1]=mvpLX[1]+dmvLX[1]

[0286] It should be noted that the values ​​of dmvLX[0] and dmvLX[1] are restricted to between -8 and 8 regardless of the number of bits in sadList.

[0287] (Criteria for determining DMVR)

[0288] dmvrFlag is a flag that indicates whether DMVR processing is performed when it is true and not when it is false.

[0289] When the SPS flag, which indicates that DMVR processing can be performed, is on, the initial error generation unit 303752 sets dmvrFlag to true. Otherwise, the initial error generation unit 303752 sets dmvrFlag to false.

[0290] Furthermore, if merge_flag in this block is true, the initial error generation unit 303752 can set dmvrFlag to true. Otherwise, the initial error generation unit 303752 sets dmvrFlag to false.

[0291] Furthermore, if both predFlagL0 and predFlagL1 are true, that is, in the case of bidirectional prediction, the initial error generation unit 303752 can set dmvrFlag to true. Otherwise, the initial error generation unit 303752 will set dmvrFlag to false.

[0292] If mmvd_flag is false and the block is not in MMVD mode, the initial error generation unit 303752 can set dmvrFlag to true. Otherwise, in MMVD mode, the initial error generation unit 303752 sets dmvrFlag to false.

[0293] When DiffPicOrderCnt(currPic, RefPicList[0][refIdxL0]) equals DiffPicOrderCnt(RefPicList[1][refIdxL1], currPic), that is, when the current image currPic is in a positional relationship with the L0 reference image RefPicList[0][refIdxL0] and the L1 reference image RefPicList[1][refIdxL1] through equidistant interpolation, the initial error generation unit 303752 can set dmvrFlag to true. Otherwise, the initial error generation unit 303752 sets dmvrFlag to false. Here, DiffPicOrderCnt() is a function that derives the difference in POC (Picture Order Count) between the two images as shown below.

[0294] DiffPicOrderCnt(picA,picB)=PicOrderCnt(picA)-PicOrderCnt(picB)

[0295] It should be noted that when DiffPicOrderCnt(currPic, RefPicList[0][refIdxL0]) * DiffPicOrderCnt(currPic, RefPicList[1][refIdxL1]) < 0, the initial error generation unit 303752 can set dmvrFlag to true if the only positional relationship is interpolation, and can set dmvrFlag to false if this is not the case.

[0296] Furthermore, when the size of the processed block is below a certain value, the initial error generation unit 303752 may set dmvrFlag to false. For example, when bH is 8 or higher and bH*bW is 64, the initial error generation unit 303752 may set dmvrFlag to true. Otherwise, the initial error generation unit 303752 will set dmvrFlag to false.

[0297] Figure 11 This is a flowchart illustrating the processing flow in the DMVR unit 30375. In this embodiment, in addition to the determination criteria described above, such as... Figure 11 As shown, an additional condition is added: DMVR processing is applied only when GBI processing described later is not applied.

[0298] Specifically, first, the DMVR unit 30375 performs the dmvrFlag determination process described above (S1101). Next, the DMVR unit 30375 determines whether gbiIdx is 0 (S1102). As described later, when gbiIdx is a non-zero value, an unequal weighted prediction is performed based on the table gbiWLut. When gbiIdx is 0, in addition to setting dmvrFlag to true, when gbiIdx is non-zero, dmvrFlag is set to false (S1103).

[0299] Then, the DMVR unit 30375 determines whether dmvrFlag is true (S1104). If it is true, the DMVR process is executed (S1105). If it is false, the process is not executed.

[0300] When using GBI prediction, weighted prediction is applied, which allows for the consideration of inaccurate assessment errors and reduces the overall processing load by limiting application conditions.

[0301] Similarly, in the weighted prediction described later, when using either the L0 or L1 prediction processed by DMVR for weighted prediction, dmvrFlag is set to false. Specifically, if both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are false, dmvrFlag is added to the true condition; otherwise, dmvrFlag is set to false. Here, luma_weight_l0_flag[refIdxL0] indicates whether the brightness weight coefficient w0 and offset o0 exist in the L0 predicted image, and luma_weight_l1_flag[refIdxL1] indicates whether the brightness weight coefficient w1 and offset o1 exist in the L1 predicted image.

[0302] (BDOF determination based on error threshold processing via DMVR)

[0303] In DMVR, the error between the L0 and L1 predicted images is calculated. However, it is determined in advance whether to perform the BDOF processing, which will be described later, based on the error value at this time.

[0304] Figure 12 This is a flowchart illustrating the process of determining BDOF based on error threshold processing using DMVR.

[0305] First, the initial error generation unit 303752 sets bdofFlag to true beforehand (S1201). Next, the initial error generation unit 303752 derives centerSad (S1202) and determines whether the value of centerSad is greater than or equal to the threshold (bH>>1)*bW*4 (S1203). If the value of centerSad is less than the threshold, the initial error generation unit 303752 determines that the error is small and sets bdofFlag, which indicates whether BDOF processing should be performed, to false (S1204), thus setting it to not perform BDOF processing beforehand. This determination is the same as that of the initial error generation unit 30752 described above; therefore, the motion vector search unit 303753 and the correction vector derivation unit 303754 are skipped, and DMVR processing is not performed. If the value of centerSad is above the threshold, the motion vector search unit 303753 performs a corrected motion vector search (S1205). As a result, the corrected vector derivation unit 303754 determines whether the value of sadList[bestIdx], which is the value of the smallest bestIdx, is less than the threshold (bH>>1)*bW*8 (S1206). If the value of sadList[bestIdx] is less than the threshold, the corrected vector derivation unit 303754 determines that the error is small and sets bdofFlag, which indicates whether BDOF processing is performed, to false (S1207), thus setting it to not perform BDOF processing in advance.

[0306] It should be noted that the threshold of (S1206) is set to the same or greater value as the threshold of (S1203).

[0307] In DMVR processing, in order to perform the search for corrected motion vectors, it is necessary to calculate the error between the L0 and L1 predicted images. On the other hand, BDOF processing has no effect when the error is small. Therefore, by adding such processing, the determination of whether to perform BDOF can be made without additional error calculation.

[0308] In the BDOF section described later, it is determined whether bdofFlag is true (S1208). If it is true, BDOF processing is performed in the inter-frame prediction parameter decoding section 303 (S1209). If it is false, it is determined that BDOF processing will not be performed in this block.

[0309] (Triangle Prediction)

[0310] Next, triangle prediction will be explained. In triangle prediction, the object CU is divided into two triangular prediction units, using the diagonal or opposite diagonal as boundaries. The prediction image in each triangular prediction unit is derived by weighting each pixel of the prediction image of the object CU (including the rectangular block of the triangular prediction unit) according to the pixel position and applying a mask. For example, a triangular image can be derived from a rectangular image by multiplying by a mask that sets pixels in the triangular region within the rectangular area to 1 and pixels outside the triangle to 0. An adaptive weighting of the prediction images is applied to the regions on both sides of the diagonal, and a prediction image of the object CU (rectangular block) is derived by using the adaptive weighting of the two prediction images. This process is called triangle synthesis. Transformation (inverse transform) and quantization (inverse quantization) are applied to the entire object CU. It should be noted that triangle prediction is only applied when there is a merge prediction mode or a skip mode.

[0311] The triangle prediction unit 30377 derives prediction parameters corresponding to the two triangular regions used for triangle prediction and outputs them to the inter-frame prediction image generation unit 309. In triangle prediction, for the sake of processing simplicity, a configuration without bidirectional prediction may be used. In this case, inter-frame prediction parameters for unidirectional prediction are derived within one triangular region. It should be noted that the derivation of the two prediction images and the synthesis of the prediction images are performed in the motion compensation unit 3091 and the triangle synthesis unit 30952.

[0312] (MMVD Prediction Department 30376)

[0313] The MMVD prediction unit 30376 performs processing in MMVD (Merge with Motion Vector Differece) mode. MMVD mode obtains motion vectors by adding a difference vector of a specified distance and direction to a motion vector derived from a merge candidate (such as a motion vector derived from the motion vectors of adjacent blocks). In MMVD mode, the MMVD prediction unit 30376 efficiently derives motion vectors by using merge candidates and limiting the range of the difference vector values ​​to a specified distance (e.g., 8 types) and a specified direction (e.g., 4 directions, 8 directions, etc.).

[0314] The loop filter 305 is a filter located within the encoding loop, used to remove block distortion and ringing distortion to improve image quality. The loop filter 305 performs deblocking filtering, sample adaptive offset (SAO), and adaptive loop filtering (ALF) on the decoded image of the CU generated by the adder 312.

[0315] The image memory 306 stores the CU decoded image generated by the addition unit 312 in a predetermined location according to each object image and object CU.

[0316] The prediction parameter memory 307 stores the prediction parameters in predetermined locations according to the CTU or CU of each decoded object. Specifically, the prediction parameter memory 307 stores parameters decoded by the parameter decoding unit 302 and prediction modes such as predMode decoded by the entropy decoding unit 301.

[0317] The prediction image generation unit 308 is input with a prediction mode (predMode), prediction parameters, etc. Furthermore, the prediction image generation unit 308 reads a reference image from the reference image memory 306. Under the prediction mode indicated by the prediction mode (predMode), the prediction image generation unit 308 uses the prediction parameters and the read-out reference image (reference image block) to generate a prediction image of a block or sub-block. Here, a reference image block refers to a set of pixels (usually rectangular, hence called a block) on the reference image, which is a region referenced for generating the prediction image.

[0318] (Inter-frame prediction image generation unit 309)

[0319] When the prediction mode predMode indicates the inter-frame prediction mode, the inter-frame prediction image generation unit 309 uses the inter-frame prediction parameters input from the inter-frame prediction parameter decoding unit 303 and the read reference image to generate a predicted image of a block or sub-block through inter-frame prediction.

[0320] Figure 13 This is a schematic diagram showing the configuration of the inter-frame prediction image generation unit 309 included in the prediction image generation unit 308 of this embodiment. The inter-frame prediction image generation unit 309 is configured to include a motion compensation unit (prediction image generation device) 3091 and a synthesis unit 3095.

[0321] (Motion compensation)

[0322] The motion compensation unit 3091 (interpolated image generation unit) reads from the reference image memory 306, based on the inter-frame prediction parameters (predFlagLX, reference image index refIdxLX, and motion vector mvLX) input from the inter-frame prediction parameter decoding unit 303, the block located in the reference image RefPicLX specified by the reference image index refIdxLX and shifted by the motion vector mvLX from the position of the object block, thereby generating an interpolated image (motion-compensated image). Here, if the precision of the motion vector mvLX is not integer precision, a filter called motion compensation filtering is implemented to generate pixels at fractional positions, thereby generating the interpolated image.

[0323] The motion compensation unit 3091 first derives the integer position (xInt, yInt) and phase (xFrac, yFrac) corresponding to the coordinates (x, y) in the prediction block using the following formula.

[0324] xInt=xPb+(mvLX[0]>>(log2(MVPREC)))+x

[0325] xFrac=mvLX[0]&(MVPREC-1)

[0326] yInt=yPb+(mvLX[1]>>(log2(MVPREC)))+y

[0327] yFrac=mvLX[1]&(MVPREC-1)

[0328] Here, (xPb, yPb) are the top-left coordinates of a block of size bW*bH, x=0, ..., bW-1, y=0, ..., bH-1, and MVPREC represents the precision of the motion vector mvLX (1 / MVPREC pixel precision). For example, it can be MVPREC=16.

[0329] The motion compensation unit 3091 uses an interpolation filter to perform horizontal interpolation on the reference image refImg, thereby deriving a temporary image temp[][]. The following Σ is the sum of k related to k=0..NTAP-1, shift1 is the normalization parameter for the interval of the adjustment value, offset1=1<<(shift1-1).

[0330] temp[x][y]=(ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1

[0331] Next, the motion compensation unit 3091 derives the interpolated image Pred[][] by performing vertical interpolation on the temporary image temp[][]. The following Σ is the sum related to k from k=0..NTAP-1, shift2 is the normalization parameter for the interval of adjustment values, and offset2=1<<(shift2-1).

[0332] Pred[x][y]=(ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2

[0333] (Synthesis Department)

[0334] The synthesis unit 3095 generates a prediction image by referring to the interpolated image input by the motion compensation unit 3091, the inter-frame prediction parameters input by the inter-frame prediction parameter decoding unit 303, and the intra-frame prediction image generation unit 310, and outputs the generated prediction image to the addition unit 312.

[0335] The compositing unit 3095 includes: a combined intra / inter compositing unit 30951, a triangle compositing unit 30952, an OBMC unit 30953, and a BDOF unit 30956.

[0336] (Combined intra-frame / inter-frame compositing process)

[0337] The combined intra / inter-frame synthesis unit 30951 generates a predicted image by compounding the use of one-way prediction, skip mode, merge mode and intra-frame prediction in AMVP.

[0338] (Triangle composition process)

[0339] The triangle synthesis unit 30952 generates a predicted image using the triangle prediction described above.

[0340] (OBMC processing)

[0341] The OBMC section 30953 uses OBMC (Overlapped block motion compensation) processing to generate a predicted image. The OBMC processing includes the following steps.

[0342] • An interpolated image (motion-compensated image) of the object sub-block is generated using an interpolated image (PU interpolated image) generated with interpolated images (PU interpolated image) generated with interpolated images (OBMC interpolated image) generated with motion parameters of adjacent sub-blocks of the object sub-block.

[0343] • A predicted image is generated by weighted averaging of the OBMC interpolated image and the PU interpolated image.

[0344] (Weighted Forecasting Department 30954)

[0345] The weighted prediction unit 309454 generates a predicted image of the block by multiplying the motion-compensated images PredL0 and PredL1 with weight coefficients. When one of the prediction list flags (predFlagL0 or predFlagL1) is 1 (one-way prediction) and no weighted prediction is used, the following formula is performed to match the motion-compensated image PredLX (LX is L0 or L1) with the number of pixel bits bitDepth.

[0346] Pred[x][y] = Clip3(0, (1 < y) ...<bitDepth)-1,(PredLX[x][y]+offset1)> >shift1)

[0347] Here, shift1 = Max(2, 14-bitDepth), offset1 = 1 << (shift1 - 1).

[0348] (Two-way prediction processing)

[0349] Furthermore, when the prediction list uses flags (predFlagL0 and predFlagL1) set to 1 (bidirectional prediction BiPred) and does not use weighted prediction, the following formula is used to average the motion-compensated images PredL0 and PredL1 and match their average with the number of pixel bits.

[0350] Pred[x][y] = Clip3(0, (1 < y) ...<bitDepth)-1,(PredL0[x][y]+PredL1[x][y]+offset2)> >shift2)

[0351] Here, shift2 = Max(3, 15-bitDepth), offset2 = 1 << (shift2 - 1). This process will also be referred to as the usual bidirectional prediction.

[0352] Furthermore, when a single prediction is performed and a flag indicating whether the weighted prediction coefficients w0 and offset o0 exist in the reference image at L0 (luma_weight_l0_flag in the case of brightness, and chroma_weight_l0_flag in the case of chroma) is enabled, in the case of L0 prediction, the weighted prediction unit 30954 derives the weighted prediction coefficients w0 and offset o0 from the encoded data and performs the following calculation.

[0353] Pred[x][y] = Clip3(0, (1 < y) ... <bitDepth)-1,((PredL0[x][y]*w0+(1<<(log2WD-1)))> >log2WD)+o0)

[0354] In the case of L1 prediction, when the flag indicating whether the weight prediction coefficients w1 and offset o1 exist in the reference image representing L1 (luma_weight_l1_flag in the case of brightness, chroma_weight_l1_flag in the case of color difference) is enabled, the weighted prediction unit 30954 derives the weight prediction coefficients w1 and offset o1 from the encoded data and performs the following calculation.

[0355] Pred[x][y] = Clip3(0, (1 < y) ... <bitDepth)-1,((PredL1[x][y]*w1+(1<<(log2WD-1)))> >log2WD)+o1)

[0356] Here, log2WD is a variable that is explicitly summed by combining the values ​​of Log2WeightDenom+shift1 sent separately in the slice header according to luminance and chromaticity.

[0357] (Weighted bidirectional forecasting)

[0358] Furthermore, when performing weighted prediction using BiPred and flags indicating the presence of weighted prediction coefficients and offsets (luma_weight_l0_flag and luma_weight_l1_flag in the case of luminance, and chroma_weight_l0_flag and chroma_weight_l1_flag in the case of chroma), the weighted prediction unit 30954 derives the weighted prediction coefficients w0, w1, o0, and o1 from the encoded data and performs the following calculation.

[0359] Pred[x][y] = Clip3(0, (1 < y) ... <bitDepth)-1,(PredL0[x][y]*w0+PredL1[x][y]*w1+((o0+o1+1)<<log2WD))> >(log2WD+1))

[0360] (GBI Department 30955)

[0361] In the weighted prediction described above, an example of generating a predicted image by multiplying the interpolated image by weight coefficients was explained. Here, examples other than multiplying the interpolated image by weight coefficients to generate a predicted image will be explained. Specifically, the process of generating a predicted image using Generalized Bi-prediction (hereinafter referred to as GBI prediction) will be explained. In GBI prediction, the predicted image Pred is generated by multiplying the L0 predicted image PredL0 and the L1 predicted image PredL1 from the bi-prediction by weight coefficients (w0, w1).

[0362] Furthermore, when generating a prediction image using GBI prediction, the GBI unit 30955 switches the weight coefficients (w0, w1) on a per-coding-unit basis. That is, the GBI unit 30954 of the inter-frame prediction image generation unit 309 sets the weight coefficients for each coding unit. In GBI prediction, multiple weight coefficient candidates are predefined, and gbiIdx represents the index of the weight coefficient used in the target block among the multiple weight coefficient candidates included in the table gbiWLut.

[0363] The GBI unit 30955 checks the flag gbiAppliedFlag indicating whether GBI prediction is used. If it is false (FALSE), the motion compensation unit 3091 generates a prediction image using the following formula.

[0364] Pred[x][y] = Clip3(0, (1 < y) ... <bitDepth)-1,

[0365] (PredL0[x][y]+PredL1[x][y]+offset2)>>shift2)

[0366] Here, the initial state of gbiAppliedFlag is false. The GBI unit 30955 sets gbiAppliedFlag to true (TRUE) when the SPS flag is on and the flag indicating GBI processing is enabled is on, and bidirectional prediction is enabled. Furthermore, as an AND condition, gbiAppliedFlag can be set to true if the gbiIdx index of the gbiWLut table (which represents the weight coefficients for GBI prediction) is not 0 (the value of the index when the weights of the L0 and L1 prediction images are equal). Furthermore, as an AND condition, gbiAppliedFlag can be set to true if the block size of the CU is a certain value or higher.

[0367] When gbiAppliedFlag is true, GBI30955 derives the predicted image Pred based on the weights w0, w1 and PredL0, PredL1 using the following formula.

[0368] Pred[x][y] = Clip3(0, (1 < y) ... <bitDepth)-1,

[0369] (w0*PredL0[x][y]+w1*PredL1[x][y]+offset3)>>(shift2+3))

[0370] Here, the weight coefficient w1 is derived from the table iWLut[]={4, 5, 3, 10, -2} via gbiIdx, which is explicitly indicated by the syntax. The weight coefficient w0 is set to (8-w1). It should be noted that when gbiIdx=0, w0=w1=4, which is equivalent to the usual bidirectional prediction.

[0371] The formulas shift1, shift2, offset1, and offset2 are derived using the following formulas.

[0372] shift1=Max(2,14-bitDepth)

[0373] shift2=Max(3,15-bitDepth)=shift1+1

[0374] offset1 = 1 << (shift1 - 1)

[0375] offset2 = 1 << (shift2 - 1)

[0376] offset3 = 1 << (shift2 + 2)

[0377] It should be noted that there are multiple tables gbiWLut with different combinations of weight coefficients. GBI 30955 can switch the table used for selecting weight coefficients based on whether the image structure is LowDelay (LB).

[0378] When using GBI prediction in AMVP prediction mode, the inter-frame prediction parameter decoding unit 303 decodes the gbiIdx and sends it to the GBI unit 30955. Furthermore, when using GBI prediction in merge prediction mode, the inter-frame prediction parameter decoding unit 303 decodes the merge index merge_idx, and the merge candidate derivation unit 30361 derives the gbiIdx of each merge candidate. Specifically, the merge candidate derivation unit 30361 uses the weight coefficients of adjacent blocks used for deriving merge candidates as the weight coefficients of merge candidates for the target block. That is, in merge mode, previously used weight coefficients are inherited as the weight coefficients of the target block.

[0379] (Selection of forecasting model using GBI forecasting)

[0380] Next, refer to Figure 14 The selection process for the prediction mode using GBI prediction in the motion picture decoding device 31 will be explained. Figure 14 This is a flowchart illustrating an example of the process for selecting a prediction mode in the motion picture decoding device 31.

[0381] like Figure 14As shown, the inter-frame prediction parameter decoding unit 303 first decodes the skip flag (S1401). When the skip flag indicates that the skip mode is skip mode (YES in S1402), the prediction mode is merge mode (S1403), and the inter-frame prediction parameter decoding unit 303 decodes the merge index (S14031). When using GBI prediction, the GBI unit 30955 derives the weight coefficients derived from the merge candidates into the weight coefficients of the GBI prediction.

[0382] If the skip flag is not indicated as skip mode (NO in S1402), the inter-frame prediction parameter decoding unit 303 decodes the merging flag (S1407). If the merging flag is indicated as merging mode (Yes in S1408), the prediction mode is merging mode (S1403), and the inter-frame prediction parameter decoding unit 303 decodes the merging index (S14031). When using GBI prediction, the GBI unit 30955 derives the weight coefficients derived from the merging candidates into the weight coefficients of the GBI prediction.

[0383] If the merge flag is not indicated as merge mode (no in 1408), the prediction mode is AMVP mode (S1409).

[0384] In AMVP mode, the inter-frame prediction parameter decoding unit 303 decodes the inter-frame prediction identifier inter_pred_idc (S14090). Next, the inter-frame prediction parameter decoding unit 303 decodes the difference vector mvdLX (S14091). Next, the inter-frame prediction parameter decoding unit 303 decodes gbiIdx (S14092). When using GBI prediction, the GBI unit 30955 selects the weight coefficient w1 for GBI prediction from the weight coefficient candidates in the gbiWLut table.

[0385] (BDOF prediction)

[0386] Next, the details of the prediction (BDOF prediction) performed by the BDOF unit 30956 will be explained. The BDOF unit 30956 generates a prediction image in bidirectional prediction mode by referring to two prediction images (a first prediction image and a second prediction image) and a gradient correction term.

[0387] Figure 15 This is a flowchart illustrating the process of deriving the predicted image processing.

[0388] If the inter-frame prediction parameter decoding unit 303 determines that L0 is a unidirectional prediction (inter_pred_idc is 0 in S1501), the motion compensation unit 3091 generates an L0 prediction image PredL0[x][y] (S1502). If the inter-frame prediction parameter decoding unit 303 determines that L1 is a unidirectional prediction (inter_pred_idc is 1 in S1501), the motion compensation unit 3091 generates an L1 prediction image PredL1[x][y] (S1503). On the other hand, if the inter-frame prediction parameter decoding unit 303 determines that bidirectional prediction mode is in place (inter_pred_idc is 2 in S1501), the following processing S1504 is then performed. In S1504, the synthesis unit 3095 determines whether to perform BDOF processing by referring to the bioAvailableFlag, which indicates whether BDOF processing is performed. When bioAvailableFlag indicates true, the BDOF unit 30956 performs BDOF processing to generate a bidirectional prediction image (S1506). When bioAvailableFlag indicates false, the synthesis unit 3095 generates a prediction image through normal bidirectional prediction image generation (S1505).

[0389] The inter-frame prediction parameter decoding unit 303 can deduce that bioAvailableFlag is true when L0 reference image refImgL0 and L1 reference image refImgL1 are different reference images and the two images are in opposite directions relative to the object image. Specifically, when the object image is set to currPic, bioAvailableFlag is true if the condition DiffPicOrderCnt(currPic, refImgL0) * DiffPicOrderCnt(currPic, refImgL1) < 0 is satisfied.

[0390] Here, DiffPicOrderCnt() is a function that derives the difference of the POC (Picture OrderCount: the order in which the pictures are displayed) of two images as shown below.

[0391] DiffPicOrderCnt(picA,picB)=PicOrderCnt(picA)-PicOrderCnt(picB)

[0392] The condition that the motion vector of an appended object block is not a motion vector in units of sub-blocks is used as the condition for bioAvailableFlag to be true.

[0393] In addition, the condition that the motion vector of the object image is not a motion vector in units of sub-blocks can be added as a condition for bioAvailableFlag to be true.

[0394] In addition, the condition that the sum of the absolute differences between the L0 and L1 prediction images of the two prediction blocks is greater than or equal to a specified value can be added as a condition for bioAvailableFlag to indicate that it is true.

[0395] In addition, the condition that the predicted image generation mode is a block-based predicted image generation mode can also be added as a condition for bioAvailableFlag to be true.

[0396] Alternatively, the condition that either L0 or L1 prediction is not weighted can be added to the weighted prediction as a condition for bioAvailableFlag to be true. Specifically, if both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are false, then it is set as a condition for bioAvailableFlag to be true. Here, luma_weight_l0_flag[refIdxL0] indicates whether the brightness weight coefficient w0 and offset o0 exist in the L0 predicted image, and luma_weight_l1_flag[refIdxL1] indicates whether the brightness weight coefficient w1 and offset o1 exist in the L1 predicted image.

[0397] Figure 16 This is a schematic diagram showing the structure of BDOF section 30956. (Using...) Figure 16 The specific processing performed by the BDOF unit 30956 will be explained below. The BDOF processing unit 30956 includes: an L0 / L1 prediction image generation unit 309561, a gradient image generation unit 309562, a correlation parameter calculation unit 309563, a motion compensation correction value derivation unit 309564, and a BDOF prediction image generation unit 309565. The BDOF unit 30956 generates a prediction image based on the interpolated image received from the motion compensation unit 3091 and the inter-frame prediction parameters received from the inter-frame prediction parameter decoding unit 303, and outputs the generated prediction image to the addition unit 312. It should be noted that the process of deriving the motion compensation correction value modBIO (motion compensation correction image) from the gradient image, correcting and deriving the prediction images PredL0 and PredL1, is called bidirectional gradient transformation processing.

[0398] Figure 17This is an example diagram showing the region where filling is performed. First, L0 and L1 predicted images for BDOF processing are generated in the L0 and L1 predicted image generation unit 309561. In the BDOF unit 30956, based on... Figure 17 The L0 and L1 predicted images of each CU unit or sub-CU unit shown are processed using BDOF, but an extra 2 pixels of interpolated image information surrounding the CU or sub-CU being processed are reserved for gradient calculation. This interpolated image information is generated using adjacent integer pixels instead of the usual interpolation filter, and is used for gradient image generation as described later. Otherwise, this portion is copied in the same way as the outer edge of the image and uses the surrounding pixels as a fill area. Furthermore, the unit of BDOF processing is NxN pixels below the CU unit or sub-CU unit, and the processing itself uses (N+2)×(N+2) pixels plus the surrounding 1 pixel.

[0399] A gradient image is generated in the gradient image generation unit 309562. In the gradient change (Optical Flow), it is assumed that the pixel value of each point remains unchanged, and only its position changes. This can be represented by the change of pixel value I in the horizontal direction (horizontal gradient value lx) and its position change Vx, the change of pixel value I in the vertical direction (vertical gradient value ly) and its position change Vy, and the time change lt of pixel value I, as follows.

[0400] lx*Vx+ly*Vy+lt=0

[0401] Hereinafter, the position change (Vx, Vy) will be referred to as the correction weight vector (u, v).

[0402] Specifically, the gradient image generation unit 309562 derives gradient images lx0, ly0, lx1, and ly1 using the following formulas. lx0 and lx1 represent gradients along the horizontal direction, and ly0 and ly1 represent gradients along the vertical direction.

[0403] lx0[x][y]=(PredL0[x+1][y]-PredL0[x-1][y])>>shift1

[0404] ly0[x][y]=(PredL0[x][y+1]-PredL0[x][y-1])>>shift1

[0405] lx1[x][y]=(PredL1[x+1][y]-PredL1[x-1][y])>>shift1

[0406] ly1[x][y]=(PredL1[x][y+1]-PredL1[x][y-1])>>shift1

[0407] Here, shift1 = Max(2, 14-bitDepth).

[0408] Next, the relevant parameter calculation unit 309563 uses the surrounding 1 pixel of each block of NxN pixels in each CU to derive the gradient product of (N+2)x(N+2) pixels and s1, s2, s3, s5, s6.

[0409] s1=sum(phiX[x][y]* phiX[x][y])

[0410] s2=sum(phiX[x][y]* phiY[x][y])

[0411] s3=sum(-theta[x][y]* phiX[x][y])

[0412] s5=sum(phiY[x][y]* phiY[x][y])

[0413] s6=sum(-theta[x][y]* phiY[x][y])

[0414] Here, sum(a) represents the sum of a for coordinates (x, y) within a block of (N+2) x (N+2) pixels. Furthermore,

[0415] theta[x][y]=-(PredL1[x][y]>>shift4)+(PredL0[x][y]>>shift4)

[0416] phiX[x][y]=(lx1[x][y]+lx0[x][y])>>shift5

[0417] phiY[x][y]=(ly1[x][y]+ly0[x][y])>>shift5

[0418] Here, let's set it as

[0419] shift4=Min(8,bitDepth-4)

[0420] shift5=Min(5,bitDepth-7).

[0421] Next, the motion compensation correction value derivation unit 309564 uses the derived gradient product and s1, s2, s3, s5, s6 to derive the correction weight vector (u, v) in NxN pixels.

[0422] u = (s3 << 3) >> log2(s1)

[0423] v=((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1))>>log2(s5)

[0424] Here, s2m = s2 >> 12, s2s = s2 & ((1 << 12) - 1).

[0425] It should be noted that clipping can be further used to limit the range of u and v as shown below.

[0426] u=s1>0? Clip3(-th,th,-(s3<<3)>>floor(log2(s1))): 0

[0427] v=s5>0? Clip3(-th, th, ((s6<<3)-((((u*s2m)<<12)+u*s2s)>>1))>>floor(log2(s5))): 0

[0428] Here, let th = Max(2, 1 << (13-bitDepth)). The value of th needs to be calculated in conjunction with shift1, so consider the case where the pixel bit length bitDepth is greater than 12 bits.

[0429] The motion compensation correction value derivation unit 309564 uses the correction weight vector (u, v) in units of NxN pixels and the gradient images lx0, ly0, lx1, ly1 to derive the motion compensation correction value modBIO[x][y] for NxN pixels.

[0430] The modBIO can be derived using modBIO[x][y]=((lx1[x][y]-lx0[x][y])*u+(ly1[x][y]-ly0[x][y])*v+1)>>1 (formula A3) or the rounding function shown below.

[0431] modBIO[x][y]=Round(((lx1[x][y]-lx0[x][y])*u)>>1)+Round(((ly1[x][y]-ly0[x][y])*v)>>1)

[0432] The BDOF prediction image generation unit 309565 uses the parameters described above to derive the pixel value Pred of the NxN pixel prediction image using the following formula.

[0433] At this time, the BDOF prediction image generation unit 309565 uses the above parameters to derive the pixel value Pred of the NxN pixel prediction image using the following formula.

[0434] Pred[x][y] = Clip3(0, (1 < y) ...<bitDepth)-1,(PredL0[x][y]+PredL1[x][y]+modBIO[x][y]+offset2)> >shift2)

[0435] Here, shift2 = Max(3, 15-bitDepth), offset2 = 1 << (shift2 - 1).

[0436] Then, the BDOF prediction image generation unit 309565 outputs the generated block prediction image to the addition unit 312.

[0437] The inverse quantization / inverse transform unit 311 dequantizes the quantization transform coefficients input from the entropy decoding unit 301 to obtain the transform coefficients. These quantization transform coefficients are obtained by performing frequency transformations such as DCT (Discrete Cosine Transform) and DST (Discrete Sine Transform) on the prediction error during the encoding process and then quantizing it. The inverse quantization / inverse transform unit 311 performs inverse DCT, inverse DST, and other inverse frequency transformations on the obtained transform coefficients to calculate the prediction error. The inverse quantization / inverse transform unit 311 outputs the prediction error to the adder unit 312. The inverse quantization / inverse transform unit 311 sets all prediction errors to 0 when skip_flag is 1 or cu_cbp is 0.

[0438] The adder 312 adds the predicted image of the block input from the predicted image generation unit 308 to the prediction error input from the inverse quantization / inverse transform unit 311 for each pixel to generate the decoded image of the block. The adder 312 stores the decoded image of the block in the reference image memory 306 and outputs it to the loop filter 305.

[0439] (Composition of a motion picture encoding device)

[0440] Next, the configuration of the motion image encoding device 11 in this embodiment will be described. Figure 18 This is a schematic diagram showing the configuration of the motion picture encoding apparatus 11 according to this embodiment. The motion picture encoding apparatus 11 is configured to include: a prediction image generation unit 101, a subtraction unit 102, a transform / quantization unit 103, an inverse quantization / inverse transform unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference image memory (reference image storage unit, frame memory) 109, an encoding parameter determination unit 110, a parameter encoding unit 111, and an entropy encoding unit 104.

[0441] The prediction image generation unit 101 generates a prediction image based on the regions (CU) formed by dividing each image T into its constituent parts. The prediction image generation unit 101 performs the same operation as the prediction image generation unit 308 described previously, and its description is omitted here.

[0442] The subtraction unit 102 subtracts the pixel values ​​of the predicted image of the block input from the prediction image generation unit 101 from the pixel values ​​of the image T to generate a prediction error. The subtraction unit 102 outputs the prediction error to the transform / quantization unit 103.

[0443] The transform / quantization unit 103 calculates the transform coefficients from the prediction error input from the subtraction unit 102 through frequency transformation, and derives the quantized transform coefficients through quantization. The transform / quantization unit 103 outputs the quantized transform coefficients to the entropy encoding unit 104 and the inverse quantization / inverse transform unit 105.

[0444] Inverse quantization / inverse transform unit 105 and inverse quantization / inverse transform unit 311 in motion image decoding device 31 Figure 7 The same applies, so its explanation is omitted here. The calculated prediction error is input into the adder 106.

[0445] In the entropy coding unit 104, quantization transformation coefficients are input from the transform / quantization unit 103, and coding parameters are input from the parameter coding unit 111. The coding parameters include, for example, codes for the reference image index refIdxLX, the prediction vector index mvp_LX_idx, the difference vector mvdLX, the motion vector precision mode amvr_mode, the prediction mode predMode, and the merge index merge_idx.

[0446] The entropy coding unit 104 entropy codes the segmentation information, prediction parameters, quantization transformation coefficients, etc. to generate a coded stream Te and outputs it.

[0447] The parameter encoding unit 111 includes: a header encoding unit 1110 (not shown), a CT information encoding unit 1111, a CU encoding unit 1112 (prediction mode encoding unit), and a parameter encoding unit 112. The CU encoding unit 1112 also includes a TU encoding unit 1114.

[0448] The following is a brief description of the general operation of each module. The parameter encoding unit 111 performs encoding processing of parameters such as header information, segmentation information, prediction information, and quantization transformation coefficients.

[0449] The CT information encoding unit 1111 encodes QT, MT (BT, TT) segmentation information, etc., based on the encoding data.

[0450] The CU encoding unit 1112 encodes CU information, prediction information, TU segmentation flag split_transform_flag, CU residual flags cbf_cb, cbf_cr, cbf_luma, etc.

[0451] When the prediction error is included in the TU, the TU encoding unit 1114 encodes the QP update information (quantization correction value) and the quantization prediction error (residual_coding).

[0452] The CT information coding unit 1111 and the CU coding unit 1112 output syntax elements such as inter-frame prediction parameters (predMode, merge_flag, merge_idx, inter_pred_idc, reference image index refIdxLX, prediction vector index mvp_LX_idx, difference vector mvdLX), intra-frame prediction parameters (prev_intra_luma_pred_flag, mpm_idx, rem_selected_mode_flag, rem_selected_mode, rem_non_selected_mode), and quantization transform coefficients to the entropy coding unit 104.

[0453] (The structure of the parameter encoding section)

[0454] The parameter encoding unit 112 derives inter-frame prediction parameters based on the prediction parameters input from the encoding parameter determination unit 110. The parameter encoding unit 112 includes the same configuration as the inter-frame prediction parameters derived by the inter-frame prediction parameter decoding unit 303.

[0455] Figure 19 This is a schematic diagram showing the configuration of the parameter encoding unit 112. The configuration of the parameter encoding unit 112 will be explained. (See attached diagram.) Figure 19As shown, the parameter encoding unit 112 is configured to include: a parameter encoding control unit 1121, a merging prediction unit 30374, a sub-block prediction unit (affine prediction unit) 30372, a DMVR unit 30375, an MMVD prediction unit 30376, a triangle prediction unit 30377, an AMVP prediction parameter derivation unit 3032, and a subtraction unit 1123. The merging prediction unit 30374 includes a merging prediction parameter derivation unit 3036. The parameter encoding control unit 1121 includes a merging index derivation unit 11211 and a vector candidate index derivation unit 11212. Furthermore, the parameter encoding control unit 1121 derives merge_idx, affine_flag, base_candidate_idx, distance_idx, direction_idx, etc., through the merging index derivation unit 11211, and derives mvpLX, etc., through the vector candidate index derivation unit 11212. The merge prediction parameter derivation unit 3036, AMVP prediction parameter derivation unit 3032, affine prediction unit 30372, MMVD prediction unit 30376, and triangle prediction unit 30377 can also be collectively referred to as the motion vector derivation unit (motion vector derivation device). The parameter encoding unit 112 outputs the motion vector mvLX, the reference image index refIdxLX, the inter-frame prediction identifier inter_pred_idc, or information representing these to the prediction image generation unit 101. In addition, the parameter encoding unit 112 outputs merge_flag, skip_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_lX_idx, mvdLX, amvr_mode, and affine_flag to the entropy encoding unit 104.

[0456] Figure 20 This is a diagram illustrating an example of the number of candidate distances and candidate directions in the motion picture encoding apparatus 11. The parameter encoding control unit 1121 derives parameters representing the difference vector (base_candidate_idx, distance_idx, direction_idx, etc.) and outputs them to the MMVD prediction unit 30376. (See reference...) Figure 20 The derivation of the differential vector in the parameter encoding control unit 1121 will be explained. Figure 20The central black dot represents the position indicated by the prediction vector `mvpLX`. Around this position, eight search distances are searched in four directions (up, down, left, and right). `mvpLX` represents the starting point and the motion vector of the second candidate in the merged candidate list, and searches are performed on them separately. There are two merged candidate lists (the first and second in the list), with a search distance of 8 and four search directions; therefore, there are 64 candidates in `mvdLX`. The `base_candidate_idx`, `distance_idx`, and `direction_idx` represent the `mvdLX` with the lowest cost among the searched values.

[0457] Thus, the MMVD pattern is a pattern that is restricted to searching for candidate points centered on the prediction vector and deriving the appropriate motion vector.

[0458] The merge index derivation unit 11211 derives the merge index merge_idx and outputs it to the merge prediction parameter derivation unit 3036 (merge prediction unit). In MMVD mode, the merge index derivation unit 11211 sets the value of the merge index merge_idx to the same value as the base_candidate_idx. The vector candidate index derivation unit 11212 derives the predicted vector index mvp_lX_idx.

[0459] The merging prediction parameter derivation unit 3036 derives the inter-frame prediction parameters based on the merge index merge_idx.

[0460] The AMVP prediction parameter derivation unit 3032 derives the prediction vector mvpLX based on the motion vector mvLX. The AMVP prediction parameter derivation unit 3032 outputs the prediction vector mvpLX to the subtraction unit 1123. It should be noted that the reference image index refIdxLX and the prediction vector index mvp_lX_idx are output to the entropy coding unit 104.

[0461] The affine prediction unit 30372 derives the inter-frame prediction parameters (affine prediction parameters) of the sub-block.

[0462] The subtraction unit 1123 subtracts the prediction vector mvpLX, which is the output of the AMVP prediction parameter derivation unit 3032, from the motion vector mvLX input by the encoding parameter determination unit 110, to generate the difference vector mvdLX. The difference vector mvdLX is output to the entropy coding unit 104.

[0463] The addition unit 106 adds the pixel values ​​of the block prediction image input from the prediction image generation unit 101 and the prediction error input from the inverse quantization / inverse transform unit 105 to generate a decoded image by adding each pixel. The addition unit 106 stores the generated decoded image in the reference image memory 109.

[0464] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. It should be noted that the loop filter 107 does not necessarily include the above three filters; for example, it may only include a deblocking filter.

[0465] The prediction parameter memory 108 stores the prediction parameters generated by the encoding parameter determination unit 110 in a predetermined location for each object image and CU.

[0466] The image memory 109 stores the decoded images generated by the loop filter 107 in predetermined locations for each object image and each CU.

[0467] The encoding parameter determination unit 110 selects one set from a plurality of sets of encoding parameters. The encoding parameters refer to the QT, BT, or TT segmentation information, prediction parameters, or parameters generated in association with them as encoding objects, as described above. The prediction image generation unit 101 uses these encoding parameters to generate a prediction image.

[0468] The encoding parameter determination unit 110 calculates the RD cost value, representing the amount of information and the encoding error, for each of the multiple sets. The encoding parameter determination unit 110 selects the set of encoding parameters with the smallest calculated cost value. Therefore, the entropy encoding unit 104 outputs the selected set of encoding parameters as the encoded stream Te. The encoding parameter determination unit 110 stores the determined encoding parameters in the prediction parameter memory 108.

[0469] It should be noted that a portion of the motion picture encoding device 11 and motion picture decoding device 31 described above can be implemented using a computer. For example, this includes the entropy decoding unit 301, parameter decoding unit 302, loop filter 305, prediction image generation unit 308, inverse quantization / inverse transform unit 311, addition unit 312, prediction image generation unit 101, subtraction unit 102, transform / quantization unit 103, entropy encoding unit 104, inverse quantization / inverse transform unit 105, loop filter 107, encoding parameter determination unit 110, and parameter encoding unit 111. In this case, the program for implementing this control function can be recorded on a computer-readable recording medium, and the computer system can read and execute the program recorded on the recording medium. It should be noted that the "computer system" mentioned here refers to a computer system built into either the motion picture encoding device 11 or the motion picture decoding device 31, employing hardware including an operating system and peripheral devices. Furthermore, "computer-readable recording medium" refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording medium" can also include: recording media that dynamically stores programs for a short period of time, such as communication lines used to transmit programs via networks like the Internet or telephone lines; and recording media that store programs for a fixed period of time, such as volatile memory within a computer system serving as a server or client in such cases. Furthermore, the aforementioned program can be a program used to implement the above-mentioned functions, or a program that can implement the above-mentioned functions by combining with programs already recorded in the computer system.

[0470] Furthermore, some or all of the motion picture encoding device 11 and motion picture decoding device 31 in the above embodiments can be implemented as integrated circuits such as LSI (Large Scale Integration). Each functional block of the motion picture encoding device 11 and motion picture decoding device 31 can be processorized individually, or some or all can be integrated for processorization. Moreover, the method of integrated circuit implementation is not limited to LSI; it can also be implemented using dedicated circuits or general-purpose processors. Furthermore, if advancements in semiconductor technology lead to integrated circuit technologies that replace LSI, integrated circuits based on such technologies can also be used.

[0471] The above description, with reference to the accompanying drawings, details one embodiment of the invention. However, the specific configuration is not limited to the above embodiment, and various design changes can be made without departing from the spirit of the invention.

[0472] [Application Example]

[0473] The aforementioned moving image encoding device 11 and moving image decoding device 31 can be mounted on various devices for transmitting, receiving, recording, and reproducing moving images. It should be noted that the moving images can be natural moving images captured by a camera or the like, or artificial moving images (including CG and GUI) generated by a computer or the like.

[0474] First, refer to Figure 2 The following describes the situation where the above-described motion picture encoding device 11 and motion picture decoding device 31 can be used for the transmission and reception of motion pictures.

[0475] Figure 2 (a) is a block diagram showing the configuration of the transmitting device PROD_A equipped with the motion picture encoding device 11. For example... Figure 2 As shown in (a), the transmitting device PROD_A includes: an encoding unit PROD_A1 that obtains encoded data by encoding a moving image; a modulation unit PROD_A2 that obtains a modulated signal by modulating a carrier wave using the encoded data obtained by the encoding unit PROD_A1; and a transmitting unit PROD_A3 that transmits the modulated signal obtained by the modulation unit PROD_A2. The moving image encoding device 11 described above is used as the encoding unit PROD_A1.

[0476] As an input source of moving images to the encoding unit PROD_A1, the transmitting device PROD_A may further include: a camera PROD_A4 for capturing moving images, a recording medium PROD_A5 for recording moving images, an input terminal PROD_A6 for inputting moving images from an external source, and an image processing unit A7 for generating or processing images. Figure 2 Example (a) shows that the transmitting device PROD_A has all of these components, but some may be omitted.

[0477] It should be noted that the recording medium PROD_A5 can be a medium that records unencoded motion images, or a medium that records motion images encoded using a recording encoding method different from the encoding method used for transmission. In the latter case, it is preferable that the decoding unit (not shown) that decodes the encoded data read from the recording medium PROD_A5 according to the recording encoding method is located between the recording medium PROD_A5 and the encoding unit PROD_A1.

[0478] Figure 2 (b) is a block diagram showing the configuration of the receiving device PROD_B equipped with the motion picture decoding device 31. Figure 2As shown in (b), the receiving device PROD_B includes: a receiving unit PROD_B1 for receiving a modulated signal, a demodulation unit PROD_B2 for obtaining encoded data by demodulating the modulated signal received by the receiving unit PROD_B1, and a decoding unit PROD_B3 for obtaining a moving image by decoding the encoded data obtained by the demodulation unit PROD_B2. The aforementioned moving image decoding device 31 is used as the decoding unit PROD_B3.

[0479] The receiving device PROD_B, serving as the output destination for the moving images output by the decoding unit PROD_B3, may further include a display PROD_B4 for displaying the moving images, a recording medium PROD_B5 for recording the moving images, and an output terminal PROD_B6 for outputting the moving images to an external device. Figure 2 Example (b) shows the receiving device PROD_B having all of these components, but some may be omitted.

[0480] It should be noted that the recording medium PROD_B5 can be a medium for recording unencoded motion images, or it can be a medium encoded with a recording encoding method different from the encoding method used for transmission. In the latter case, it is preferable that the encoding unit (not shown) that encodes the motion images acquired from the decoding unit PROD_B3 according to the recording encoding method is located between the decoding unit PROD_B3 and the recording medium PROD_B5.

[0481] It should be noted that the transmission medium for modulated signals can be wireless or wired. Furthermore, the transmission scheme for modulated signals can be broadcast (here, a transmission scheme where the destination is not predetermined) or communication (here, a transmission scheme where the destination is predetermined). That is, the transmission of modulated signals can be achieved through any of the following: wireless broadcasting, wired broadcasting, wireless communication, and wired communication.

[0482] For example, a terrestrial digital broadcasting station (broadcasting equipment, etc.) / receiving station (television receiver, etc.) is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via wireless broadcasting. Similarly, a cable television broadcasting station (broadcasting equipment, etc.) / receiving station (television receiver, etc.) is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via cable broadcasting.

[0483] Furthermore, servers (workstations, etc.) and clients (TV receivers, personal computers, smartphones, etc.) using internet-based VOD (Video On Demand) services, motion picture sharing services, etc., are examples of transmitting devices PROD_A and receiving devices PROD_B that transmit and receive modulated signals via communication (typically, either wireless or wired is used as the transmission medium in a LAN, and wired is used in a WAN). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Additionally, smartphones also include multi-functional portable telephone terminals.

[0484] It should be noted that, in addition to decoding the encoded data downloaded from the server and displaying it on the screen, the client of the motion picture sharing service also has the function of encoding motion pictures captured by a camera and uploading them to the server. That is, the client of the motion picture sharing service performs the functions of both the sending device PROD_A and the receiving device PROD_B.

[0485] Next, refer to Figure 3 The following describes the situation where the above-mentioned motion picture encoding device 11 and motion picture decoding device 31 can be used for recording and reproducing motion pictures.

[0486] Figure 3 (a) is a block diagram showing the configuration of the recording device PROD_C equipped with the aforementioned motion picture encoding device 11. Figure 3 As shown in (a), the recording device PROD_C includes an encoding unit PROD_C1 that obtains encoded data by encoding a moving image and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 to the recording medium PROD_M. The moving image encoding device 11 described above is used as the encoding unit PROD_C1.

[0487] It should be noted that the recording medium PROD_M can be (1) a type of recording medium built into the recording device PROD_C, such as HDD (Hard Disk Drive) or SSD (Solid State Drive), or (2) a type of recording medium connected to the recording device PROD_C, such as SD memory card or USB (Universal Serial Bus) flash memory, or (3) a recording medium loaded into a drive (not shown) built into the recording device PROD_C, such as DVD (Digital Versatile Disc) or BD (Blu-ray Disc).

[0488] Furthermore, as an input source of moving images to the encoding unit PROD_C1, the recording device PROD_C may further include: a camera PROD_C3 for capturing moving images, an input terminal PROD_C4 for inputting moving images from the outside, a receiving unit PROD_C5 for receiving moving images, and an image processing unit PROD_C6 for generating or processing images. Figure 3 Example (a) shows that the recording device PROD_C has all of these components, but some may be omitted.

[0489] It should be noted that the receiving unit PROD_C5 can receive unencoded motion images, or it can receive encoded data encoded using a transmission encoding method different from the encoding method used for recording. In the latter case, it is preferable to place the transmission decoding unit (not shown) that decodes the encoded data encoded using the transmission encoding method between the receiving unit PROD_C5 and the encoding unit PROD_C1.

[0490] Examples of such recording devices PROD_C include DVD recorders, BD recorders, and HDD (Hard Disk Drive) recorders (in which case the input terminal PROD_C4 or the receiving unit PROD_C5 is the main input source for moving images). Furthermore, portable camcorders (in which case the camera PROD_C3 is the main input source for moving images), personal computers (in which case the receiving unit PROD_C5 or the image processing unit C6 is the main input source for moving images), and smartphones (in which case the camera PROD_C3 or the receiving unit PROD_C5 is the main input source for moving images) are also examples of such recording devices PROD_C.

[0491] Figure 3 (b) is a block diagram showing the configuration of the playback device PROD_D equipped with the aforementioned motion picture decoding device 31. Figure 3 As shown in (b), the playback device PROD_D includes a readout unit PROD_D1 that reads encoded data written to the recording medium PROD_M and a decoding unit PROD_D2 that obtains a motion picture by decoding the encoded data read out by the readout unit PROD_D1. The motion picture decoding device 31 described above is used as the decoding unit PROD_D2.

[0492] It should be noted that the recording medium PROD_M can be (1) a recording medium built into the playback device PROD_D, such as HDD or SSD, or (2) a recording medium connected to the playback device PROD_D, such as SD memory card or USB flash drive, or (3) a recording medium loaded into a drive device (not shown) built into the playback device PROD_D, such as DVD or BD.

[0493] Furthermore, as the output destination for the moving images output by the decoding unit PROD_D2, the playback device PROD_D may further include: a display PROD_D3 for displaying the moving images, an output terminal PROD_D4 for outputting the moving images to the outside, and a transmitting unit PROD_D5 for transmitting the moving images. Figure 3 Example (b) shows the reproduction device PROD_D having all of these components, but some can be omitted.

[0494] It should be noted that the transmitting unit PROD_D5 can transmit unencoded motion images, or it can transmit encoded data encoded using a transmission encoding method different from the encoding method used for recording. In the latter case, it is preferable to place the encoding unit (not shown) that encodes the motion images using the transmission encoding method between the decoding unit PROD_D2 and the transmitting unit PROD_D5.

[0495] Examples of such playback devices PROD_D include DVD players, BD players, HDD players, etc. (in which case, the output terminal PROD_D4 connected to a television receiver, etc., is the main output destination for moving images). Other examples include television receivers (in which case the display PROD_D3 is the main output destination for moving images), digital signage (also called electronic billboards, electronic bulletin boards, etc., where the display PROD_D3 or the transmitter PROD_D5 is the main output destination for moving images), desktop PCs (in which case, the output terminal PROD_D4 or the transmitter PROD_D5 is the main output destination for moving images), laptop or tablet PCs (in which case, the display PROD_D3 or the transmitter PROD_D5 is the main output destination for moving images), and smartphones (in which case, the display PROD_D3 or the transmitter PROD_D5 is the main output destination for moving images).

[0496] (Hardware implementation and software implementation)

[0497] Furthermore, each of the aforementioned motion picture decoding device 31 and motion picture encoding device 11 can be implemented in hardware using logic circuits formed on an integrated circuit (IC chip), or in software using a CPU (Central Processing Unit).

[0498] In the latter case, the aforementioned devices include: a CPU that executes commands for programs that perform various functions; a ROM (Read Only Memory) that stores the programs; a RAM (Random Access Memory) that expands the programs; and a memory that stores the programs and various data, etc., such as a storage device (recording medium). Furthermore, the objective of embodiments of the present invention is to achieve this by supplying a recording medium containing program code (executable form program, intermediate code program, source program) of the software implementing the aforementioned functions, i.e., the control program of the aforementioned devices, in a computer-readable manner to the aforementioned devices, wherein the computer (or CPU, MPU) reads the program code recorded on the recording medium and executes it.

[0499] As recording media, the following can be used: tapes, cassette tapes, etc.; disks including floppy disks (registered trademark) / hard disks, CD-ROMs (Compact Disc Read-Only Memory), MO discs (Magneto-Optical Disc), MD discs (Mini Disc), DVDs (Digital Versatile Disc), CD-R discs (CD Recordable), Blu-ray discs (Blu-ray Disc); cards (including memory cards) / optical cards; semiconductor memory such as mask ROMs / EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable and Programmable Read-Only Memory), and flash memory ROMs; or logic circuits such as PLDs (Programmable Logic Devices) and FPGAs (Field Programmable Gate Arrays).

[0500] Furthermore, the aforementioned devices can be configured to connect to a communication network and supply the program code via the communication network. This communication network need only be capable of transmitting program code and is not particularly limited. For example, the Internet, intranet, extranet, LAN (Local Area Network), ISDN (Integrated Services Digital Network), VAN (Value-Added Network), CATV (Community Antenna television / Cable Television) communication network, Virtual Private Network, telephone line network, mobile communication network, satellite communication network, etc., can be used. Moreover, the transmission medium constituting this communication network need only be a medium capable of transmitting program code and is not limited to a specific configuration or type. For example, it can be used in wired networks such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line transmission, cable TV lines, telephone lines, and ADSL (Asymmetric Digital Subscriber Line) lines, as well as in wireless networks such as IrDA (Infrared Data Association), infrared (like remote controls), Bluetooth (registered trademark), IEEE 802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance, registered trademark), mobile phone networks, satellite lines, and terrestrial digital broadcasting networks. It should be noted that embodiments of the present invention can also be implemented as computer data signals with embedded carriers that embody the above-mentioned program code via electronic transmission.

[0501] The embodiments of the present invention are not limited to the embodiments described above, and various modifications can be made within the scope of the claims. That is, embodiments obtained by combining technical solutions with appropriate modifications within the scope of the claims are also included within the technical scope of the present invention.

[0502] (Mutual references between related applications)

[0503] This application claims priority to Japanese Patent Application No. 2019-043097, filed on March 8, 2018, and incorporates the entire contents of which hereinafter by reference.

[0504] Industrial availability

[0505] The embodiments of the present invention are preferably applied to a moving image decoding apparatus for decoding encoded data obtained by encoding image data, and to a moving image encoding apparatus for generating encoded data obtained by encoding image data. Furthermore, they are preferably applied to a data structure of encoded data generated by the moving image encoding apparatus and referenced by the moving image decoding apparatus.

[0506] Explanation of reference numerals in the attached figures

[0507] 31 Image Decoding Device

[0508] 301 Entropy Decoding Department

[0509] 302 Parameter Decoding Unit

[0510] 3020 Header Decoding Department

[0511] 303 Inter-frame Prediction Parameter Decoding Unit

[0512] 304-frame intra-prediction parameter decoding unit

[0513] 308 Predictive Image Generation Unit

[0514] 309 Inter-frame Prediction Image Generation Unit

[0515] 310-frame intra-prediction image generation unit

[0516] 311 Inverse Quantization / Inverse Transformation Unit

[0517] 312 Addition Department

[0518] 11 Image Encoding Device

[0519] 101 Predictive Image Generation Unit

[0520] 102 Subtraction Department

[0521] 103 Transformation / Quantization Section

[0522] 104 Entropy Coding Department

[0523] 105 Inverse Quantization / Inverse Transformation Unit

[0524] 107 loop filter

[0525] 110 Coding Parameter Determination Unit

[0526] 111 Parameter Encoding Section

[0527] 112 Parameter Encoding Section

[0528] 1110 Header Code Department

[0529] 1111 CT Information Coding Department

[0530] 1112 CU Coding Unit (Predictive Mode Coding Unit)

[0531] 1114 TU Coding Department

[0532] 3091 Sports Compensation Department

[0533] 3095 Synthesis Department

[0534] Intra / inter-frame synthesis unit of 30951 combination

[0535] 30952 Triangle Composite Part

[0536] 30953 OBMC Department

[0537] 30954 Weighted Forecasting Department

[0538] 30955 GBI Department

[0539] 30956 BDOF Department

[0540] 309561 L0, L1 Predictive Image Generation Unit

[0541] 309562 Gradient Image Generation Unit

[0542] 309563 Related Parameter Calculation Department

[0543] 309564 Derivation of Motion Compensation Correction Value

[0544] 309565 BDOF Predictive Image Generation Unit.

Claims

1. A motion image decoding apparatus, wherein the motion image decoding apparatus uses motion vectors mvL0 and mvL1 and two reference images to perform decoder-side motion vector thinning (DMVR) processing, characterized in that, The motion picture decoding device includes a DMVR unit that calculates a dmvrFlag indicating whether to perform the DMVR process, and performs the DMVR process if the dmvrFlag is true. The dmvrFlag is deduced to be true if the following conditions are met: (1) the flag of the sequence parameter set SPS, which indicates whether the DMVR processing can be performed, is enabled; (2) gbiIdx is 0; and (3) both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are false. If dmvrFlag is not derived to be true, then dmvrFlag is false. The gbiIdx represents the index of the weight coefficient used in the generalized bidirectional prediction of the object block from among the multiple weight coefficient candidates included in the gbiWLut table. The value of gbiIdx=0 indicates that the generalized bidirectional prediction was not applied. The `luma_weight_l0_flag[refIdxL0]` indicates whether the weight coefficients for the L0 reference image represented by the reference image index `refIdxL0` exist. The term luma_weight_l1_flag[refIdxL1] indicates whether the weight coefficients for the L1 reference image represented by the reference image index refIdxL1 exist.

2. A motion picture encoding apparatus, wherein the motion picture encoding apparatus uses motion vectors mvL0 and mvL1 and two reference images to perform decoder-side motion vector thinning (DMVR) processing, characterized in that, The motion picture encoding apparatus includes a DMVR unit that derives a dmvrFlag indicating whether to perform the DMVR process, and performs the DMVR process if the dmvrFlag is true. The dmvrFlag is deduced to be true if the following conditions are met: (1) the flag of the sequence parameter set SPS, which indicates whether the DMVR processing can be performed, is enabled; (2) gbiIdx is 0; and (3) both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are false. If dmvrFlag is not derived to be true, then dmvrFlag is false. The gbiIdx represents the index of the weight coefficient used in the generalized bidirectional prediction of the object block from among the multiple weight coefficient candidates included in the gbiWLut table. The value of gbiIdx=0 indicates that the generalized bidirectional prediction was not applied. The `luma_weight_l0_flag[refIdxL0]` indicates whether the weight coefficients for the L0 reference image represented by the reference image index `refIdxL0` exist. The term luma_weight_l1_flag[refIdxL1] indicates whether the weight coefficients for the L1 reference image represented by the reference image index refIdxL1 exist.

3. A computer-readable recording medium containing a program that causes a computer to perform decoder-side motion vector thinning (DMVR) processing using motion vectors mvL0 and mvL1 and two reference images, characterized in that... The program causes the computer to execute: The dmvrFlag indicating whether the DMVR process can be performed is deduced to be true if the following conditions are met: (1) the flag of the sequence parameter set SPS, i.e. the flag indicating whether the DMVR process can be performed, is enabled; (2) gbiIdx is 0; and (3) both luma_weight_l0_flag[refIdxL0] and luma_weight_l1_flag[refIdxL1] are false. If dmvrFlag is not derived to be true, then dmvrFlag is false. If dmvrFlag is true, the DMVR process is executed. The gbiIdx represents the index of the weight coefficient used in the generalized bidirectional prediction of the object block from among the multiple weight coefficient candidates included in the gbiWLut table. The value of gbiIdx=0 indicates that the generalized bidirectional prediction was not applied. The `luma_weight_l0_flag[refIdxL0]` indicates whether the weight coefficients for the L0 reference image represented by the reference image index `refIdxL0` exist. The term luma_weight_l1_flag[refIdxL1] indicates whether the weight coefficients for the L1 reference image represented by the reference image index refIdxL1 exist.

Images