Image decoders, image encoders, and computer-readable immutable media.

Abstract

The proposed image decoder selectively uses a merging mode in the absence of MMVD mode, thus achieving high encoding performance. The image decoder includes a parameter decoder, and in the case of a normal merging flag indicating normal merging mode, it checks whether the MMVD prediction signaled in the sequence parameter set or similar set is available. If the MMVD prediction is unavailable, it decodes the motion vector information obtained from the merging candidate.

Description

Image decoding device, image encoding device, and image decoding method

[0001] Embodiments of the present invention relate to a prediction image generation device, a moving image decoding device, and a moving image encoding device.

[0002] In order to efficiently transmit or record a moving image, a moving image encoding device that generates encoded data by encoding the moving image, and a moving image decoding device that generates a decoded image by decoding the encoded data are used.

[0003] Specific moving image encoding methods include, for example, the H.264 / AVC and HEVC (High-Efficiency Video Coding) methods.

[0004] In such a moving image encoding method, an image (picture) constituting the moving image is managed by a hierarchical structure consisting of a slice obtained by dividing the image, a coding tree unit (CTU: Coding Tree Unit) obtained by dividing the slice, a coding unit (sometimes called a Coding Unit: CU) obtained by dividing the coding tree unit, and a transform unit (TU: Transform Unit) obtained by dividing the coding unit, and is encoded / decoded for each CU.

[0005] Also, in such a moving image encoding method, usually, a prediction image is generated based on a local decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the prediction image from the input image (original image) is encoded. Methods for generating a prediction image include inter-picture prediction (inter prediction) and intra-picture prediction (intra prediction).

[0006] Also, in Non-Patent Document 1, a regular merge flag is introduced, and a technique for selecting an inter prediction mode from encoded data into two groups: 1) a merge mode and a merge plus distance mode (MMVD mode), and 2) an intra-inter mode (CIIP mode) and a triangle mode is disclosed.

[0007] "Versatile Video Coding (Draft 6)", JVET-O2001-v9, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11"Non-CE4: Merge mode signaling overhead reduction", JVET-O0309-v3, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 2019-07-22

[0008] Non-patent document 1 described a problem where, when MMVD mode was unavailable, there were fewer merge candidate options, resulting in reduced encoding efficiency.

[0009] To solve the above problems, an image decoding device according to one aspect of the present invention is an image decoding device that decodes parameters for generating a prediction image, and comprises a parameter decoding unit that decodes a regular merge flag from merge data indicating whether or not a regular merge mode is used in interprediction, and the parameter decoding unit, if the regular merge flag indicates that the regular merge mode is used in interprediction, checks a flag notified in the sequence parameter set indicating whether or not the motion vector of the merge candidate is valid, and if the value of the flag is 1, decodes an MMVD merge flag indicating whether or not the motion vector of the merge candidate is used to generate the interprediction parameters of the target coding unit, and decodes a merge index, which is an index of the merge candidate list, using the MMVD merge flag.

[0010] An image decoding device according to one aspect of the present invention is characterized in that the merge index indicates that the MMVD merge flag does not use the motion vector of the merge candidate to generate the interpretation parameter, and decoding is performed when the number of merge candidates is greater than 1.

[0011] An image decoding device according to one aspect of the present invention is characterized in that, when the value of the MMVD merge flag is 0, the value of the merge index is estimated to be 0.

[0012] An image encoding device according to one aspect of the present invention is an image encoding device for encoding parameters for generating a prediction image, comprising a parameter encoding unit that encodes a regular merge flag from merge data indicating whether or not a regular merge mode is used in inter-prediction, wherein if the regular merge flag indicates that the regular merge mode is used in inter-prediction, the parameter encoding unit checks a flag notified in the sequence parameter set indicating whether or not a merge candidate motion vector is valid, and if the value of the flag is 1, encodes an MMVD merge flag indicating whether or not the merge candidate motion vector is used to generate the inter-prediction parameters of the target encoding unit, and encodes a merge index, which is an index of the merge candidate list, using the MMVD merge flag.

[0013] An image decoding method according to one aspect of the present invention is an image decoding method for decoding parameters for generating a prediction image, comprising at least the steps of: decoding a regular merge flag from merge data that indicates whether or not a regular merge mode is used in interprediction; if the regular merge flag indicates that the regular merge mode is used in interprediction, checking a flag notified in the sequence parameter set that indicates whether or not a merge candidate motion vector is valid; if the value of the flag is 1, decoding an MMVD merge flag that indicates whether or not the merge candidate motion vector is used to generate the interprediction parameters of the target coding unit; and decoding a merge index, which is an index of a merge candidate list, using the MMVD merge flag.

[0014] According to one aspect of the present invention, the above problems can be solved.

[0015] This is a schematic diagram showing the configuration of the image transmission system according to this embodiment. This diagram shows the configuration of a transmitting device equipped with a video encoding device and a receiving device equipped with a video decoding device according to this embodiment. (a) shows a transmitting device equipped with a video encoding device, and (b) shows a receiving device equipped with a video decoding device. This diagram shows the configuration of a recording device equipped with a video encoding device and a playback device equipped with a video decoding device according to this embodiment. (a) shows a recording device equipped with a video encoding device, and (b) shows a playback device equipped with a video decoding device. This is a diagram showing the hierarchical structure of the data of the encoded stream. This is a diagram showing an example of CTU partitioning. This is a conceptual diagram showing an example of a reference picture and a reference picture list. This is a schematic diagram showing the configuration of the video decoding device. This is a flowchart explaining the general operation of the video decoding device. This is a schematic diagram showing the configuration of the inter-prediction parameter derivation unit. This is a schematic diagram showing the configuration of the merge prediction parameter derivation unit and the AMVP prediction parameter derivation unit. This is a schematic diagram showing the configuration of the inter-prediction image generation unit. This is a block diagram showing the configuration of the video encoding device. This is a schematic diagram showing the configuration of the inter-prediction parameter encoding unit. This is a diagram explaining MMVD. This is a diagram showing the flow of the prediction mode derivation process for inter-prediction. This figure shows the syntax for the prediction mode selection process according to this embodiment. This figure explains the regular merge flag. This flowchart shows the flow of the prediction mode selection process in the video decoding device. This flowchart shows the flow of the prediction mode selection process in the video decoding device. This figure shows the syntax for the prediction mode selection process according to this embodiment.

[0016] (First Embodiment) Embodiments of the present invention will be described below with reference to the drawings.

[0017] Figure 1 is a schematic diagram showing the configuration of the image transmission system 1 according to this embodiment.

[0018] The image transmission system 1 is a system that transmits an encoded stream containing an encoded target image, decodes the transmitted encoded stream, and displays the image. The image transmission system 1 is composed of a video encoding device (image encoding device) 11, a network 21, a video decoding device (image decoding device) 31, and a video display device (image display device) 41.

[0019] The video encoding device 11 receives image T as input.

[0020] Network 21 transmits the encoded stream Te generated by the video encoding device 11 to the video decoding device 31. Network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. Network 21 is not necessarily limited to a bidirectional communication network; it may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, network 21 may be replaced by a storage medium that records the encoded stream Te, such as a DVD (Digital Versatile Disc) or a BD (Blu-ray Disc).

[0021] The video decoding device 31 decodes each of the encoded streams Te transmitted by the network 21 and generates one or more decoded images Td.

[0022] The video display device 41 displays all or part of one or more decoded images Td generated by the video decoding device 31. The video display device 41 includes a display device such as a liquid crystal display or an organic EL (electro-luminescence) display. Examples of display forms include stationary, mobile, and HMD (head-mounted display). Furthermore, if the video decoding device 31 has high processing power, it displays high-quality images, and if it has lower processing power, it displays images that do not require high processing power or display power.

[0023] <Operators> The operators used in this specification are listed below.

[0024] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || represents a logical OR.

[0025] x?y:z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0026] Clip3(a, b, c) is a function that clips c to a value between a and b, returning a if c < a, b if c > b, and c otherwise (where a <= b).

[0027] abs(a) is a function that returns the absolute value of a.

[0028] Int(a) is a function that returns the integer value of a.

[0029] floor(a) is a function that returns the largest integer less than or equal to a.

[0030] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0031] a / d represents the division of a by d (truncating the decimal part).

[0032] <Structure of the Encoded Stream Te> Prior to a detailed description of the moving image encoding device 11 and the moving image decoding device 31 according to the present embodiment, the data structure of the encoded stream Te generated by the moving image encoding device 11 and decoded by the moving image decoding device 31 will be described.

[0033] Figure 4 is a diagram showing the hierarchical structure of the data in the encoded stream Te. The encoded stream Te illustratively includes a sequence and a plurality of pictures constituting the sequence. (a) to (f) of Figure 4 are diagrams showing, respectively, an encoded video sequence that defines the sequence SEQ, an encoded picture that defines the picture PICT, an encoded slice that defines the slice S, encoded slice data that defines the slice data, an encoded tree unit included in the encoded slice data, and an encoded unit included in the encoded tree unit.

[0034] (Encoded Video Sequence) The encoded video sequence defines the set of data that the video decoding device 31 references in order to decode the sequence SEQ to be processed. As shown in Figure 4, the sequence SEQ includes a Video Parameter Set, a Sequence Parameter Set SPS, a Picture Parameter Set PPS, an Adaptation Parameter Set (APS), a Picture PICT, and Supplemental Enhancement Information SEI.

[0035] The Video Parameter Set (VPS) defines a set of encoding parameters common to multiple video layers in a video composed of multiple layers, as well as a set of encoding parameters associated with the multiple layers included in the video and with each individual layer.

[0036] The sequence parameter set (SPS) defines a set of encoding parameters that the video decoding device 31 references to decode the target sequence. For example, the width and height of the picture are defined. Multiple SPSs may exist. In that case, one of the multiple SPSs is selected from the PPS.

[0037] The Picture Parameter Set (PPS) defines a set of encoding parameters that the video decoding device 31 references to decode each picture in the target sequence. For example, it includes a reference value for the quantization width used for decoding the picture (pic_init_qp_minus26) and a flag indicating the application of weighted prediction (weighted_pred_flag). Multiple PPSs may exist. In that case, one of the multiple PPSs is selected for each picture in the target sequence.

[0038] (Encoded Picture) The encoded picture defines the set of data that the video decoding device 31 references in order to decode the picture PICT to be processed. As shown in Figure 4, the picture PICT includes slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0039] In the following, if it is not necessary to distinguish between slices 0 to NS-1, the code subscripts may be omitted. The same applies to other data included in the coded stream Te described below that have subscripts.

[0040] (Encoded Slice) An encoded slice defines the set of data that the video decoding device 31 references in order to decode the slice S to be processed. As shown in Figure 4, the slice includes a slice header and slice data.

[0041] The slice header contains a set of encoding parameters that the video decoding device 31 references to determine the decoding method for the target slice. The slice type specification information (slice_type), which specifies the slice type, is an example of the encoding parameters included in the slice header.

[0042] The slice types that can be specified by the slice type specification information include (1) I-slice, which uses only intra-prediction during encoding; (2) P-slice, which uses unidirectional or intra-prediction during encoding; and (3) B-slice, which uses unidirectional, bidirectional, or intra-prediction during encoding. Note that inter-prediction is not limited to single or bidirectional prediction, and a prediction image may be generated using more reference pictures. Hereinafter, when referring to P-slice and B-slice, it refers to slices that contain blocks on which inter-prediction can be used.

[0043] The slice header may also include a reference to the picture parameter set PPS (pic_parameter_set_id).

[0044] (Symbolized Slice Data) In the symbolized slice data, a set of data that the moving image decoding device 31 refers to for decoding the slice data to be processed is defined. As shown in FIG. 4(d), the slice data includes CTUs. A CTU is a block having a fixed size (for example, 64x64) that constitutes a slice, and may also be referred to as the largest coding unit (LCU).

[0045] (Coding Tree Unit) In FIG. 4, a set of data that the moving image decoding device 31 refers to for decoding the CTU to be processed is defined. The CTU is divided into coding units CU, which are basic units of the encoding process, by recursive quadtree partitioning (QT (Quad Tree) partitioning), binary tree partitioning (BT (Binary Tree) partitioning), or ternary tree partitioning (TT (Ternary Tree) partitioning). The combination of BT partitioning and TT partitioning is called multi-tree partitioning (MT (Multi Tree) partitioning). A node of the tree structure obtained by recursive quadtree partitioning is called a coding node. Intermediate nodes of the quadtree, binary tree, and ternary tree are coding nodes, and the CTU itself is also defined as the topmost coding node.

[0046] The CT includes, as CT information, a CU partitioning flag (split_cu_flag) indicating whether to perform CT partitioning, a QT partitioning flag (qt_split_cu_flag) indicating whether to perform QT partitioning, an MT partitioning direction (mtt_split_cu_vertical_flag) indicating the partitioning direction of MT partitioning, and an MT partitioning type (mtt_split_cu_binary_flag) indicating the partitioning type of MT partitioning. The split_cu_flag, qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each coding node.

[0047] When the split_cu_flag is 1 and the qt_split_cu_flag is 1, the coding node is divided into four coding nodes (FIG. 5(b)).

[0048] When split_cu_flag is 0, the encoding node is not split and has one CU as a node (Fig. 5(a)). The CU is the terminal node of the encoding node and is not split any further. The CU is the basic unit of the encoding process.

[0049] When split_cu_flag is 1 and qt_split_cu_flag is 0, the encoding node is MT split as follows. When mtt_split_cu_binary_flag is 1 and mtt_split_cu_vertical_flag is 0, the encoding node is horizontally split into two encoding nodes (Fig. 5(d)), and when mtt_split_cu_vertical_flag is 1, the encoding node is vertically split into two encoding nodes (Fig. 5(c)). Also, when mtt_split_cu_binary_flag is 0, when mtt_split_cu_vertical_flag is 0, the encoding node is horizontally split into three encoding nodes (Fig. 5(f)), and when mtt_split_cu_vertical_flag is 1, the encoding node is vertically split into three encoding nodes (Fig. 5(e)). These are shown in Fig. 5(g).

[0050] Also, when the size of the CTU is 64x64 pixels, the size of the CU can be any of 64x64 pixels, 64x32 pixels, 32x64 pixels, 32x32 pixels, 64x16 pixels, 16x64 pixels, 32x16 pixels, 16x32 pixels, 16x16 pixels, 64x8 pixels, 8x64 pixels, 32x8 pixels, 8x32 pixels, 16x8 pixels, 8x16 pixels, 8x8 pixels, 64x4 pixels, 4x64 pixels, 32x4 pixels, 4x32 pixels, 16x4 pixels, 4x16 pixels, 8x4 pixels, 4x8 pixels, and 4x4 pixels.

[0051] Different trees may be used for luminance and chrominance. The type of tree is indicated by treeType. For example, if a common tree is used for luminance (Y, cIdx=0) and chrominance (Cb / Cr, cIdx=1,2), the common single tree is indicated by treeType=SINGLE_TREE. If two different trees (DUAL trees) are used for luminance and chrominance, the luminance tree is indicated by treeType=DUAL_TREE_LUMA and the chrominance tree is indicated by treeType=DUAL_TREE_CHROMA.

[0052] (Encoding Unit) Figure 4 defines the set of data that the video decoding device 31 references in order to decode the encoding unit to be processed. Specifically, the CU consists of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines the prediction mode, etc.

[0053] Prediction processing can be performed at the CU (Unit) level or at the subCU level, which is a further division of the CU. If the size of the CU and the subCU are equal, there is one subCU within the CU. If the CU is larger than the size of the subCU, the CU is divided into subCUs. For example, if the CU is 8x8 and the subCU is 4x4, the CU is divided into four subCUs, each consisting of two horizontal and two vertical divisions.

[0054] There are two types of predictions (prediction modes): intra-prediction and inter-prediction. Intra-prediction is prediction within the same picture, while inter-prediction refers to prediction processing performed between different pictures (for example, between display times or between layer images).

[0055] The transformation and quantization processes are performed in units of CUs, but the quantization transformation coefficients may be entropy-encoded in subblock units such as 4x4.

[0056] (Prediction Parameters) Predicted images are derived from prediction parameters associated with blocks. Prediction parameters include intra-prediction and inter-prediction parameters.

[0057] The following describes the prediction parameters for interpretation. The interpretation parameters consist of prediction list usage flags predFlagL0 and predFlagL1, reference picture indices refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. predFlagL0 and predFlagL1 are flags that indicate whether or not a reference picture list (L0 list, L1 list) is used, and if the value is 1, the corresponding reference picture list is used. In this specification, when referring to "flags indicating whether or not XX is true," a flag value other than 0 (e.g., 1) is considered XX, and 0 is considered not XX. In logical negation, logical AND, etc., 1 is treated as true and 0 as false (the same applies below). However, in actual devices and methods, other values ​​may be used as true and false values.

[0058] Syntax elements for deriving interpretation parameters include, for example, the affine flag affine_flag, merge flag merge_flag, merge index merge_idx, MMVD flag mmvd_flag used in merge mode, the interpretation identifier inter_pred_idc and reference picture index refIdxLX used in AMVP mode to select a reference picture, the prediction vector index mvp_LX_idx and difference vector mvdLX used to derive motion vectors, and the motion vector accuracy mode amvr_mode.

[0059] (Reference Picture List) The reference picture list is a list of reference pictures stored in the reference picture memory 306. Figure 6 is a conceptual diagram showing an example of a reference picture and a reference picture list. In Figure 6(a), rectangles represent pictures, arrows represent the reference relationships between pictures, the horizontal axis represents time, I, P, and B in the rectangles represent intra-picture, single-prediction picture, and double-prediction picture, respectively, and the numbers in the rectangles represent the decoding order. As shown in the figure, the decoding order of the pictures is I0, P1, B2, B3, B4, and the display order is I0, B3, B2, B4, P1. Figure 6(b) shows an example of the reference picture list for picture B3 (target picture). The reference picture list is a list that represents candidates for reference pictures, and one picture (slice) may have one or more reference picture lists. In the example in the figure, the target picture B3 has two reference picture lists: L0 list RefPicList0 and L1 list RefPicList1. Each CU specifies which picture in the reference picture list RefPicListX (X=0 or 1) to actually reference using refIdxLX. The figure shows an example where refIdxL0=2 and refIdxL1=0. Note that LX is a notation used when there is no distinction between L0 predictions and L1 predictions, and from now on, parameters for the L0 list and parameters for the L1 list will be distinguished by replacing LX with L0 and L1.

[0060] (Merge Prediction and AMVP Prediction) There are two methods for decoding (encoding) prediction parameters: merge prediction mode and AMVP (Advanced Motion Vector Prediction). merge_flag is a flag used to distinguish between these modes. Merge prediction mode derives the prediction parameters of the target block from the prediction parameters of neighboring blocks that have already been processed, without including the prediction list usage flag predFlagLX, the reference picture index refIdxLX, and the motion vector mvLX in the encoded data. AMVP mode includes inter_pred_idc, refIdxLX, and mvLX in the encoded data. Note that mvLX is encoded as mvp_LX_idx, which identifies the prediction vector mvpLX, and the difference vector mvdLX. In addition to merge prediction mode, there may also be affine prediction mode and MMVD prediction mode.

[0061] inter_pred_idc is a value that indicates the type and number of reference pictures, and can take one of the following values: PRED_L0, PRED_L1, or PRED_BI. PRED_L0 and PRED_L1 indicate single prediction using one reference picture managed in the L0 list and L1 list, respectively. PRED_BI indicates biprediction using two reference pictures managed in the L0 list and L1 list.

[0062] `merge_idx` is an index that indicates which of the candidate predictive parameters (merge candidates) derived from the completed block will be used as the predictive parameters for the target block.

[0063] (Motion vector) mvLX represents the amount of shift between blocks on two different pictures. The prediction vector and difference vector for mvLX are called mvpLX and mvdLX, respectively.

[0064] (Interpretation identifier inter_pred_idc and prediction list usage flag predFlagLX) The relationship between inter_pred_idc, predFlagL0, and predFlagL1 is as follows, and they are mutually convertible.

[0065] inter_pred_idc = (predFlagL1<<1)+predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 Note that the inter-predation parameter may use either the prediction list usage flag or the inter-predation identifier. Also, the determination using the prediction list usage flag may be replaced with the determination using the inter-predation identifier. Conversely, the determination using the inter-predation identifier may be replaced with the determination using the prediction list usage flag.

[0066] (Determination of biprediction biPred) The biPred flag, which indicates whether a prediction is biprediction, can be derived by checking whether both of the two prediction list usage flags are set to 1. For example, it can be derived using the following formula.

[0067] biPred = (predFlagL0==1 && predFlagL1==1) Alternatively, biPred can also be derived by determining whether the interpretation identifier is a value that indicates the use of two prediction lists (reference pictures). For example, it can be derived using the following formula.

[0068] biPred = (inter_pred_idc==PRED_BI) ? 1 : 0 (Intra Prediction Parameters) The following describes the prediction parameters for intra prediction. The intra prediction parameters consist of the luminance prediction mode IntraPredModeY and the color difference prediction mode IntraPredModeC. For example, planar prediction (0), DC prediction (1), and angular prediction (otherwise). In addition, CCLM modes (81-83) may be added for color difference.

[0069] (Configuration of the video decoding device) The configuration of the video decoding device 31 (Figure 7) according to this embodiment will be described below.

[0070] The video decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (predictive image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generation unit (predictive image generation device) 308, an inverse quantization / inverse transform unit 311, and an addition unit 312 and a prediction parameter derivation unit 320. Note that, in accordance with the video encoding device 11 described later, there is also a configuration in the video decoding device 31 that does not include the loop filter 305.

[0071] The parameter decoding unit 302 further comprises a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further comprises a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice headers (slice information) from the encoded data. The CT information decoding unit 3021 decodes CT from the encoded data. The CU decoding unit 3022 decodes CU from the encoded data. The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data if the TU contains a prediction error.

[0072] The TU decoding unit 3024 decodes the QP update information and quantization prediction error from the encoded data when the skip mode is not active (skip_mode==0). More specifically, when skip_mode==0, the TU decoding unit 3024 decodes the flag cu_cbp, which indicates whether or not the target block contains a quantization prediction error, and decodes the quantization prediction error if cu_cbp is 1. If cu_cbp does not exist in the encoded data, it is derived to be 0.

[0073] The TU decoding unit 3024 decodes the index mts_idx, which indicates the transformation basis, from the encoded data. The TU decoding unit 3024 also decodes the index stIdx, which indicates the use of secondary transformations and the transformation basis, from the encoded data. If stIdx is 0, it indicates that the secondary transformation is not applied; if it is 1, it indicates one of the transformations in the set (pair) of secondary transformation basis bases; and if it is 2, it indicates the other transformation in the above pair.

[0074] Furthermore, the TU decoding unit 3024 may decode the subblock conversion flag cu_sbt_flag. If cu_sbt_flag is 1, the CU is divided into multiple subblocks, and the residual is decoded for only one specific subblock. In addition, the TU decoding unit 3024 may decode the flag cu_sbt_quad_flag indicating whether the number of subblocks is 4 or 2, the cu_sbt_horizontal_flag indicating the division direction, and the cu_sbt_pos_flag indicating a subblock containing a non-zero conversion coefficient.

[0075] The predictive image generation unit 308 is configured to include an inter-predictive image generation unit 309 and an intra-predictive image generation unit 310.

[0076] The prediction parameter derivation unit 320 is configured to include an inter-prediction parameter derivation unit 303 and an intra-prediction parameter derivation unit 304.

[0077] Furthermore, while the following examples use CTU and CU as processing units, processing may be performed in sub-CU units, or the system may be reinterpreted as blocks (CTU and CU) and sub-CU as sub-blocks (sub-CU), and processing may be performed in block or sub-block units.

[0078] The entropy decoding unit 301 performs entropy decoding on the encoded stream Te input from an external source to decode individual codes (syntax elements). Entropy coding has two methods: one that uses a context (probability model) adaptively selected according to the type of syntax element and the surrounding circumstances to encode syntax elements in a variable length, and another that uses a predetermined table or calculation formula to encode syntax elements in a variable length. The former, CABAC (Context Adaptive Binary Arithmetic Coding), stores the CABAC state of the context (a probability state index pStateIdx that specifies the type (0 or 1) and probability of the dominant symbol) in memory. The entropy decoding unit 301 initializes all CABAC states at the beginning of a segment (tile, CTU row, slice). The entropy decoding unit 301 converts the syntax elements into a binary string and decodes each bit of the Bin String. When using a context, a context index ctxInc is derived for each bit of the syntax element, the bits are decoded using the context, and the CABAC state of the context used is updated. Bits that are not used with a context are decoded with equal probability (EP, bypass), and the derivation of ctxInc and the CABAC state are omitted. The decoded syntax element contains prediction information for generating a predicted image and prediction errors for generating a difference image.

[0079] The entropy decoding unit 301 outputs the decoded codes to the parameter decoding unit 302. The decoded codes include, for example, the prediction mode predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. The control of which codes to decode is performed based on the instructions of the parameter decoding unit 302.

[0080] (Basic Flowchart) Figure 8 is a flowchart illustrating the general operation of the video decoding device 31.

[0081] (S1100: Parameter set information decoding) The header decoding unit 3020 decodes parameter set information such as VPS, SPS, and PPS from the encoded data.

[0082] (S1200: Slice information decoding) The header decoding unit 3020 decodes the slice header (slice information) from the encoded data.

[0083] The video decoding device 31 then derives a decoded image for each CTU by repeating the processing from S1300 to S5000 for each CTU included in the target picture.

[0084] (S1300: CTU information decoding) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0085] (S1400: CT information decoding) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0086] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

[0087] (S1510: CU information decoding) The CU decoding unit 3022 decodes CU information, prediction information, TU splitting flag split_transform_flag, CU residual flags cbf_cb, cbf_cr, cbf_luma, etc. from the encoded data.

[0088] (S1520: TU information decoding) The TU decoding unit 3024 decodes the QP update information, quantization prediction error, and transformation index mts_idx from the encoded data if the TU contains a prediction error. The QP update information is the difference from the quantization parameter prediction value qPpred, which is the predicted value of the quantization parameter QP.

[0089] (S2000: Predictive image generation) The predictive image generation unit 308 generates a predictive image for each block included in the target CU based on the prediction information.

[0090] (S3000: Inverse Quantization and Inverse Transform) The inverse quantization and inverse transform unit 311 performs inverse quantization and inverse transform processing for each TU included in the target CU.

[0091] (S4000: Decoded image generation) The summing unit 312 generates a decoded image of the target CU by adding the predicted image supplied from the prediction image generation unit 308 and the prediction error supplied from the inverse quantization / inverse transformation unit 311.

[0092] (S5000: Loop filter) Loop filter 305 applies loop filters such as a deblocking filter, SAO, and ALF to the decoded image to generate a decoded image.

[0093] (Configuration of the Interpretation Parameter Derivation Unit) The Interpretation Parameter Derivation Unit 303 derives interpretation parameters by referring to prediction parameters stored in the prediction parameter memory 307 based on syntax elements input from the parameter decoding unit 302. It also outputs the interpretation parameters to the Interpretation Image Generation Unit 309 and the prediction parameter memory 307. As shown in Figure 9, the Interpretation Parameter Derivation Unit 303 and its internal elements, the AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, the affine prediction unit 30372, the MMVD prediction unit 30373, the triangle prediction unit 30377, the DMVR unit 30537, and the MV addition unit 3038, are common means in both the video encoding device and the video decoding device, so they may be collectively referred to as the motion vector derivation unit (motion vector derivation device).

[0094] Figure 15 shows the flow of the prediction mode derivation process for interpretation. The parameter decoding unit 302 decodes the skip flag (cu_skip_flag) (S1600).

[0095] The interpretation parameter derivation unit 303 determines whether the skip flag is 0 or not (S1602).

[0096] If the skip flag is 0, the parameter decoding unit 302 decodes the merge flag (general_merge_flag) (S1604). On the other hand, if the skip flag is not 0, the interpretation parameter derivation unit 303 sets the merge flag to 1 (S1606).

[0097] The parameter decoding unit 302 determines whether the merge flag is 1 or not (S1608).

[0098] If the merge flag is 1, the parameter decoding unit 302 determines that the target block is a merge prediction and derives information related to the merge prediction (S1610). If the merge flag is not 1, the interpretation parameter derivation unit 303 determines that the target block is an AMVP prediction and derives information related to the AMVP prediction (S1612).

[0099] Figure 16 shows the syntax of information related to merge prediction. SYN0001 is the syntax for subblock-level merge prediction, and SYN0002 is the syntax for block-level merge prediction. SYN0002 will be explained using Figure 17.

[0100] (Syntax Decoding of the Regular Merge Flag) Figure 17 is a diagram illustrating the regular merge flag (regular_merge_flag). The regular merge flag is a flag that divides the merge predictions within the interprediction modes into two groups: 1) (narrow sense) merge mode and merge plus distance mode (MMVD mode), and 2) intra-inter mode (CIIP mode) and triangle mode. By balancing multiple (four in this case) prediction modes on the tree, it is possible to achieve high encoding efficiency without increasing bit cost, and low processing delay because the tree does not become deep. The two modes in 1) can be collectively called the regular merge mode.

[0101] (Embodiment 1) The flow of the prediction mode selection process will be described with reference to Figure 18. Figure 18 is a flowchart showing the flow of the prediction mode derivation process in the parameter decoding unit 302 and the interprediction parameter derivation unit 303.

[0102] The parameter decoding unit 302 decodes regular_merge_flag (S1301). If regular_merge_flag==1 (YES in S1302), it checks the value of sps_mmvd_enabled_flag (S1303). sps_mmvd_enabled_flag is a flag notified by the sequence parameter set (SPS), etc., that indicates whether MMVD prediction is available or not. If sps_mmvd_enabled_flag==1, that is, if MMVD prediction is available (YES in S1303), the parameter decoding unit 302 decodes the MMVD flag (mmvd_merge_flag) from the encoded data (S1304).

[0103] If mmvd_merge_flag is 1, it indicates that MMVD mode will be used to generate the interpretation parameters for the target CU. If mmvd_merge_flag is 0, it indicates that MMVD mode will not be used to generate the interpretation parameters for the target CU. If mmvd_merge_flag is not notified, it is set to 0. For example, if sps_mmvd_enabled_flag==0, i.e., MMVD prediction is not available, mmvd_merge_flag is set to 0.

[0104] If mmvd_merge_flag==1, that is, if the MMVD flag indicates that it is in MMVD mode (YES in S1305), the parameter decoding unit 302 decodes the MMVD mode parameters from the encoded data (S1309). Specifically, the parameter decoding unit 302 decodes mmvd_cand_flag, mmvd_distance_idx, and mmvd_direction_idx. mmvd_cand_flag indicates which of the first or second candidate in the merge candidate list will be used for MMVD prediction, as shown in Figure 14(a). mmvd_distance_idx indicates the distance of the difference vector, as shown in Figure 14(c). mmvd_direction_idx indicates the direction of the difference vector, as shown in 14(d).

[0105] If the number of candidates for MMVD prediction, MaxNumMergeCand, is 1 or less, the interpretation parameter derivation unit 303 may set mmvd_cand_flag to 0.

[0106] If mmvd_merge_flag==0, meaning the MMVD flag does not indicate MMVD mode (NO in S1305), and the number of merge candidates MaxNumMergeCand is greater than 1 (YES in S1306), the parameter decoding unit 302 decodes merge_idx (S1307).

[0107] If sps_mmvd_enabled_flag==0 (NO in S1303), or if MaxNumMergeCand is 1 or less (NO in S1306), that is, if merge_idx does not appear, the inter-prediction parameter derivation unit 303 sets merge_idx to 0 (infer).

[0108] The inter-prediction parameter derivation unit 303 activates the MMVD prediction unit 30373 in MMVD mode and the merge prediction parameter derivation unit 3036 in merge mode.

[0109] If regular_merge_flag==0, meaning it is not in regular merge mode (NO in S1302), the parameter decoding unit 302 decodes the CIIP flag (ciip_flag) (S1310). If ciip_flag==1 (YES in S1311), the CIIP parameters are decoded from the encoded data (S1312). merge_idx may also be decoded during CIIP parameter decoding. The interprediction parameter derivation unit 303 outputs these parameters to the interprediction image generation unit 309.

[0110] If ciip_flag==0 (NO in S1311), the inter-prediction parameter derivation unit 303 determines that the target block is in triangle mode, and the parameter decoding unit 302 decodes the triangle parameters (S1313). For example, as triangle parameters, the method of splitting the CU into two, merge_triangle_split_dir, may be used, and merge_triangle_idx0, which is the merge_idx of one of the two split blocks, and merge_triangle_idx1, which is the merge_idx of the other block, may be decoded. If the inter-prediction parameter derivation unit 303 is in triangle mode, it activates the triangle prediction unit 30377.

[0111] In Embodiment 1, by utilizing the regular merge flag, multiple prediction modes can be arranged in a balanced manner on the tree. This has the effect of improving encoding efficiency without increasing the bit cost, and reducing processing delay because the tree does not become too deep.

[0112] (Embodiment 2) Referring to Figures 19 and 20, the flow of the prediction mode derivation process in the parameter decoding unit 302 and the inter-prediction parameter derivation unit 303 of another embodiment of the present invention will be described. Figure 19 is a flowchart showing the flow of the prediction mode derivation process in the inter-prediction parameter derivation unit 303. Figure 20 is a diagram showing the syntax of the prediction mode according to this embodiment. Figure 19 shows the processing corresponding to a part of the syntax in Figure 20.

[0113] In the flowchart in Figure 19 and the syntax in Figure 20, even if MMVD prediction is not enabled by sps_mmvd_enabled_flag (sps_mmvd_enabled_flag=0), merge_idx is decoded, and if merge_idx=1, prediction is performed using merge mode.

[0114] The difference between Figure 19 and 18 lies in the operation in regular merge mode (S1403 to S1409), so the operation in regular merge mode will be described below. The operation when not in regular merge mode is the same as in Embodiment 1.

[0115] The interpretation parameter derivation unit 303 checks the value of sps_mmvd_enabled_flag (S1403). If sps_mmvd_enabled_flag==1, meaning MMVD prediction is available (YES in S1403), the parameter decoding unit 302 decodes the MMVD flag (mmvd_merge_flag) from the encoded data (S1404).

[0116] If mmvd_merge_flag==1, that is, if the MMVD flag indicates that it is in MMVD mode (YES in S1405), the parameter decoding unit 302 decodes the MMVD mode parameters from the encoded data (S1409).

[0117] If mmvd_merge_flag==0 (NO in S1403) or sps_mmvd_enabled_flag==0 (NO in S1405) (MMVD flag is not in MMVD mode), and the number of merge candidates MaxNumMergeCand is greater than 1 (YES in S1406), the parameter decoding unit 302 decodes merge_idx (S1407).

[0118] If MaxNumMergeCand is less than or equal to 1 (NO in S1406), that is, if merge_idx does not appear, the interpretation parameter derivation unit 303 sets merge_idx to 0 (infer).

[0119] The inter-prediction parameter derivation unit 303 activates the MMVD prediction unit 30373 in MMVD mode and the merge prediction parameter derivation unit 3036 in merge mode.

[0120] In Embodiment 2, the regular merge flag is used to divide the system into two groups: 1) merge mode and MMVD mode, and 2) intrainter mode (CIIP mode) and triangle mode. At the branch of 1), the sps_mmvd_enabled_flag, which is decoded from the parameter set, and the mmvd_merge_flag, which is decoded on a per-CU basis, are used to select whether the target CU uses MMVD prediction or merge mode without MMVD. In addition to the case where mmvd_merge_flag is 0, the merge index is decoded even when sps_mmvd_enabled_flag==0 if the number of merge candidates is greater than 1. Therefore, even when MMVD mode is prohibited in the higher-level syntax, the merge mode can be selectively used, resulting in high coding efficiency.

[0121] If affine_flag is 1, i.e., indicates affine prediction mode, the affine prediction unit 30372 derives inter-prediction parameters for each subblock.

[0122] If mmvd_flag is 1, i.e., indicates MMVD prediction mode, the MMVD prediction unit 30373 derives inter-prediction parameters from the merge candidate and difference vector derived by the merge prediction parameter derivation unit 3036.

[0123] When TriangleFlag is 1, that is, when Triangle prediction mode is indicated, the Triangle prediction unit 30377 derives the Triangle prediction parameters.

[0124] If merge_flag is 1, i.e., indicates merge prediction mode, merge_idx is derived and output to the merge prediction parameter derivation unit 3036.

[0125] If merge_flag is 0, i.e., indicates AMVP prediction mode, the AMVP prediction parameter derivation unit 3032 derives mvpLX from inter_pred_idc, refIdxLX or mvp_LX_idx.

[0126] (MV Adder) The MV adder 3038 adds the derived mvpLX and mvdLX to derive mvLX.

[0127] (Affine prediction unit) The affine prediction unit 30372 1) derives motion vectors for two control points CP0, CP1 or three control points CP0, CP1, CP2 of the target block, 2) derives affine prediction parameters for the target block, and 3) derives motion vectors for each subblock from the affine prediction parameters.

[0128] In merge-affine prediction, the motion vectors cpMvLX[] for each control point CP0, CP1, and CP2 are derived from the motion vectors of the adjacent blocks of the target block. In inter-affine prediction, the cpMvLX[] for each control point is derived from the sum of the predicted vectors of each control point CP0, CP1, and CP2 and the difference vector mvdCpLX[] derived from the encoded data.

[0129] The motion vector spMvLX of each subblock that makes up the target block (bW*bH) is derived as the motion vector of the point (xPosCb, yPosCb) located at the center of each subblock.

[0130] The affine prediction unit 30372 derives the affine prediction parameters (mvScaleHor, mvScalerVer, dHorX, dHorY, dHorX, dVerY) for the target block from the motion vector of the control point.

[0131] The affine prediction unit 30372 derives spMvLX[i][j] (i=0,1,2,…,(bW / sbW)-1, j=0,1,2,…,(bH / sbH)-1) in the target block based on the affine prediction parameters of the target block.

[0132] Furthermore, at the coordinates (xSb, ySb) of the top-left block of the subblock, spMvLX[i][j] is assigned to the corresponding mvLX in the screen.

[0133] (Merge Prediction) Figure 10(a) is a schematic diagram showing the configuration of the merge prediction parameter derivation unit 3036 according to this embodiment. The merge prediction parameter derivation unit 3036 includes a merge candidate derivation unit 30361 and a merge candidate selection unit 30362. The merge candidates are composed of prediction parameters (predFlagLX, mvLX, refIdxLX) and are stored in the merge candidate list. An index is assigned to the merge candidates stored in the merge candidate list according to a predetermined rule.

[0134] The merge candidate derivation unit 30361 derives merge candidates using the decoded motion vectors and refIdxLX of adjacent blocks as they are. In addition, the merge candidate derivation unit 30361 may apply the spatial merge candidate derivation process, temporal merge candidate derivation process, pairwise merge candidate derivation process, and zero merge candidate derivation process described later.

[0135] As part of the spatial merge candidate derivation process, the merge candidate derivation unit 30361 reads the prediction parameters stored in the prediction parameter memory 307 according to predetermined rules and sets them as merge candidates. The method for specifying the reference picture is, for example, the prediction parameters relating to each of the adjacent blocks within a predetermined range from the target block (for example, all or part of the blocks adjacent to the left A1, right B1, upper right B0, lower left A0, and upper left B2 of the target block). Each of the merge candidates is called A1, B1, B0, A0, and B2. Here, A1, B1, B0, A0, and B2 are motion information derived from the block containing the following coordinates. Figure 14(b) shows the positions of A1, B1, B0, A0, and B2.

[0136] A1: (xCb - 1, yCb + cbHeight - 1) B1: (xCb + cbWidth - 1, yCb - 1) B0: (xCb + cbWidth, yCb - 1) A0: (xCb - 1, yCb + cbHeight) B2: (xCb - 1, yCb - 1) Let the top-left coordinates of the target block be (xCb, yCb), width cbWidth, and height cbHeight.

[0137] As part of the time merge derivation process, the merge candidate derivation unit 30361 reads the predicted parameters of block C in the reference image, including the lower right CBR or the coordinates of the center of the target block, from the predicted parameter memory 307 to use as merge candidate Col, and stores it in the merge candidate list mergeCandList[].

[0138] The pairwise candidate derivation unit derives the pairwise candidate avgK from the average of two merge candidates (p0Cand, p1Cand) already stored in mergeCandList, and stores it in mergeCandList[].

[0139] mvLXavgK[0] = (mvLXp0Cand[0]+mvLXp1Cand[0]) / 2 mvLXavgK[1] = (mvLXp0Cand[1]+mvLXp1Cand[1]) / 2 The merge candidate derivation unit 30361 derives zero merge candidates Z0, ..., ZM where refIdxLX is 0...M and both the X and Y components of mvLX are 0, and stores them in the merge candidate list.

[0140] The order in which the mergeCandList[] is stored is, for example, spatial merge candidate (A1, B1, B0, A0, B2), temporal merge candidate Col, pairwise candidate avgK, and zero merge candidate ZK. Note that reference blocks that are unavailable (e.g., blocks that are intra-predicted) are not stored in the merge candidate list. i = 0 if( availableFlagA1 ) mergeCandList[ i++ ] = A1 if( availableFlagB1 ) mergeCandList[ i++ ] = B1 if( availableFlagB0 ) mergeCandList[ i++ ] = B0 if( availableFlagA0 ) mergeCandList[ i++ ] = A0 if( availableFlagB2 ) mergeCandList[ i++ ] = B2 if( availableFlagCol ) mergeCandList[ i++ ] = Col if( availableFlagAvgK ) mergeCandList[ i++ ] = avgK if( i < MaxNumMergeCand ) mergeCandList[ i++ ] = ZK The merge candidate selection unit 30362 selects the merge candidate N indicated by merge_idx from among the merge candidates included in the merge candidate list using the following formula.

[0141] N = mergeCandList[merge_idx] Here, N is a label indicating a merge candidate, and can take the form of A1, B1, B0, A0, B2, Col, avgK, ZK, etc. The movement information of the merge candidate indicated by label N is shown by (mvLXN[0], mvLXN[0]), predFlagLXN, and refIdxLXN.

[0142] The selected (mvLXN[0], mvLXN[0]), predFlagLXN, and refIdxLXN are selected as interprediction parameters for the target block. The merge candidate selection unit 30362 stores the interprediction parameters of the selected merge candidates in the prediction parameter memory 307 and outputs them to the interprediction image generation unit 309.

[0143] (MMVD prediction unit 30373) The MMVD prediction unit 30373 obtains mvLX by adding mvdLX of a predetermined distance and predetermined direction to the center vector mvpLX (motion vector mvLXN of merge candidate N) derived by the merge candidate derivation unit 30361. The MMVD prediction unit 30373 derives the center vector mvLX[] using the syntax element mmvd_cand_flag of the encoded data (Figure 14(a)), and derives the difference vector mvpLX[] from mmvd_direction_idx (Figure 14(d)) which indicates the index of the direction table and mmvd_distance_idx (Figure 14(c)) which indicates the index of the distance table.

[0144] The MMVD prediction unit 30373 selects the center vector mvLXN[] using mmvd_cand_flag.

[0145] N = mergeCandList[mmvd_cand_flag] The MMVD prediction unit 30373 derives the direction (MmvdSign[0], MmvdSign[1]) from mmvd_direction_idx and the distance MmvdDistance from mmvd_distance_idx. Note that the table DistanceTable from which MmvdDistance is derived is switched by the flag slice_fpel_mmvd_enabled_flag, which indicates whether to set the precision of the motion vector to integer precision at the slice level. Specifically, if slice_fpel_mmvd_enabled_flag is 0, the DistanceTable[] will be { 1, 2, 4, 8, 16, 32, 64, 128} and if slice_fpel_mmvd_enabled_flag is 1, the DistanceTable[] will be { 4, 8, 16, 32, 64, 128, 256, 512}.

[0146] dir_table_x[] = { 1, -1, 0, 0}; dir_table_y[] = { 0, 0, 1, -1}; MmvdSign[0] = dir_table_x[mmvd_direction_idx]; MmvdSign[1] = dir_table_y[mmvd_direction_idx]; MmvdDistance = DistanceTable[mmvd_distance_idx]; The MMVD prediction unit 30373 derives the differential vector refineMv[] using the product of (MmvdSign[0], MmvdSign[1]) and MmvdDistance.

[0147] firstMv[0] = (MmvdDistance << shiftMMVD) * MmvdSign[0]; firstMv[1] = (MmvdDistance << shiftMMVD) * MmvdSign[1]; Here, shiftMMVD is a value for adjusting the magnitude of the differential vector so as to match the accuracy MVPREC of the motion vector in the motion compensation unit 3091 (interpolation unit). refineMvL0[0] = firstMv[0]; refineMvL0[1] = firstMv[1]; refineMvL1[0] = -firstMv[0]; refineMvL1[1] = -firstMv[1]; Finally, the MMVD prediction unit 30373 derives the motion vector of the MMVD merge candidate from refineMvLX and the center vector mvLXN as follows.

[0148] mvL0[0] = mvL0N[0] + refineMvL0[0] mvL0[1] = mvL0N[1] + refineMvL0[1] mvL1[0] = mvL1N[0] + refineMvL1[0] mvL1[1] = mvL1N[1] + refineMvL1[1] (Triangle prediction) Next, we will explain triangle prediction. In triangle prediction, the target CU is divided into two triangular prediction units with the diagonal or opposite corner as the boundary. The predicted image in each triangular prediction unit is derived by applying a weighted mask process to each pixel of the predicted image of the target CU (rectangular block containing the triangular prediction unit) according to the position of the pixel. Intuitively, a triangular image can be derived from a rectangular image by multiplying by a mask where the top left is 1 and the bottom right is 0. Adaptive weighting of the predicted images is applied to both regions on either side of the diagonal, and a single predicted image of the target CU (rectangular block) is derived through adaptive weighting using the two predicted images. This process is called Triangle synthesis. Transform (inverse transform) and quantization (inverse quantization) processes are applied to the entire target CU. Note that Triangle prediction is only applied in merge prediction mode or skip mode.

[0149] The Triangle prediction unit 30377, in triangle mode, derives prediction parameters corresponding to the two triangular regions used for triangle prediction and supplies them to the inter-prediction image generation unit 309. For the sake of simplifying processing, a configuration that does not use dual prediction is also possible in triangle prediction. In this case, inter-prediction parameters for unidirectional prediction are derived for one triangular region. The derivation of the two prediction images and the synthesis using the prediction images are performed by the motion compensation unit 3091 and the triangle synthesis unit 30952.

[0150] (DMVR) Next, the DMVR (Decoder side Motion Vector Refinement) processing performed by the DMVR unit 30375 will be explained. If merge_flag is 1 or skip_flag is 1, the DMVR unit 30375 modifies the mvLX of the target CU derived by the merge prediction unit 30374 using a reference image. Specifically, when the prediction parameters derived by the merge prediction unit 30374 are dual predictions, the motion vector is modified using the prediction image derived from the motion vectors corresponding to the two reference pictures. The modified mvLX is supplied to the interprediction image generation unit 309.

[0151] (AMVP Prediction) Figure 10(b) is a schematic diagram showing the configuration of the AMVP prediction parameter derivation unit 3032 according to this embodiment. The AMVP prediction parameter derivation unit 3032 comprises a vector candidate derivation unit 3033 and a vector candidate selection unit 3034. The vector candidate derivation unit 3033 derives prediction vector candidates from the decoded motion vectors of adjacent blocks stored in the prediction parameter memory 307 based on refIdxLX and stores them in the prediction vector candidate list mvpListLX[].

[0152] The vector candidate selection unit 3034 selects the motion vector mvpListLX[mvp_LX_idx], indicated by mvp_LX_idx, from among the prediction vector candidates in mvpListLX[] as mvpLX. The vector candidate selection unit 3034 outputs the selected mvpLX to the MV addition unit 3038.

[0153] (MV Addition Unit) The MV addition unit 3038 calculates mvLX by adding the mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded mvdLX. The addition unit 3038 outputs the calculated mvLX to the interprediction image generation unit 309 and the prediction parameter memory 307.

[0154] mvLX[0] = mvpLX[0]+mvdLX[0] mvLX[1] = mvpLX[1]+mvdLX[1] (Motion vector precision) The amvr_mode is a syntax element that switches the precision of the motion vector derived in the AMVP mode. For example, at amvr_mode = 0, 1, 2, the precision switches to 1 / 4 pixel, 1 pixel, and 4 pixels. Instead of amvr_mode, a flag amvr_flag indicating whether it is 1 / 4 and a flag amvr_precision_flag that switches between 1 / 16 and 1 may be used.

[0155] When the precision of the motion vector is 1 / 16 precision, in order to change the motion vector differences of 1 / 4, 1, and 4 pixel precisions to the motion vector differences of 1 / 16 pixel precision, inverse quantization may be performed using MvShift (= 1 << amvr_mode = (amvr_flag + amvr_precision_flag) << 1) derived from amvr_mode as follows.

[0156] MvdLX[0] = MvdLX[0] << (MvShift + 2) MvdLX[1] = MvdLX[1] << (MvShift + 2) Similarly, when the affine_flag is 1, it is derived by the following formula.

[0157] MvShift = amvr_precision_flag? (amvr_precision_flag << 1) : (-(amvr_flag << 1))) MvdCpLX[cpIdx][0] = MvdLX[cpIdx][ 0] << (MvShift + 2) MvdCpLX[cpIdx][1] = MvdLX[cpIdx][1] << (MvShift + 2) Note that the parameter decoding unit 302 may also decode and derive the mvdLX[] before shifting by the above MvShift from the following syntax elements: ・ abs_mvd_greater0_flag ・ abs_mvd_minus2 ・ mvd_sign_flag And the parameter decoding unit 302 decodes the differential vector lMvd[] from the syntax elements by using the following formula.

[0158] lMvd[compIdx] = abs_mvd_greater0_flag[compIdx] * (abs_mvd_minus2[compIdx]+2) * (1-2*mvd_sign_flag[compIdx]) Furthermore, lMvd[] is set to MvdLX in the case of translational MVD (MotionModelIdc==0) and to MvdCpLX in the case of control point MVD (MotionModelIdc!=0).

[0159] If (MotionModelIdc == 0), then MvdLX[compIdx] = lMvd[compIdx]. Otherwise, MvdCpLX[cpIdx][compIdx] = lMvd[cpIdx][compIdx]. Here, compIdx = 0, 1 and cpIdx = 0, 1, 2.

[0160] (Motion Vector Scaling) This section explains how to derive the scaling of motion vectors. If we let Mv be the motion vector (reference motion vector), PicMv be the picture containing the block with Mv, PicMvRef be the reference picture of Mv, sMv be the scaled motion vector, CurPic be the picture containing the block with sMv, and CurPicRef be the reference picture that sMv refers to, then the derivation function for sMv, MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef), is expressed by the following formula.

[0161] sMv = MvScale(Mv,PicMv,PicMvRef,CurPic,CurPicRef) = Clip3(-R1,R1-1,sign(distScaleFactor*Mv)*((abs(distScaleFactor*Mv)+round1-1)>>shift1)) distScaleFactor = Clip3(-R2,R2-1,(tb*tx+round2)>>shift2) tx = (16384+abs(td)>>1) / td td = DiffPicOrderCnt(PicMv,PicMvRef) tb = DiffPicOrderCnt(CurPic,CurPicRef) Here, round1, round2, shift1, and shift2 are the round and shift values ​​used for division using the reciprocal, for example, round1=1<<(shift1-1), round2=1<<(shift2-1), shift1=8, shift2=6, etc. DiffPicOrderCnt(Pic1,Pic2) is a function that returns the difference in time information (e.g., POC) between Pic1 and Pic2. R1 and R2 limit the range of values ​​to perform the processing with limited precision, for example, R1=32768, R2=4096, etc.

[0162] Alternatively, the scaling function MvScale(Mv, PicMv, PicMvRef, CurPic, CurPicRef) can also be expressed by the following formula.

[0163] MvScale(Mv,PicMv,PicMvRef,CurPic,CurPicRef) = Mv*DiffPicOrderCnt(CurPic,CurPicRef) / DiffPicOrderCnt(PicMv,PicMvRef) In other words, Mv may be scaled according to the ratio of the difference in time information between CurPic and CurPicRef to the difference in time information between PicMv and PicMvRef.

[0164] (Configuration of Intra Prediction Parameter Derivation Unit 304) Based on the input from the parameter decoding unit 302, the Intra Prediction Parameter Derivation Unit 304 derives intra prediction parameters, for example, an intra prediction mode, by referring to the prediction parameters stored in the prediction parameter memory 307. The Intra Prediction Parameter Derivation Unit 304 outputs the intra prediction parameters to the prediction image generation unit 308 and also stores them in the prediction parameter memory 307. The Intra Prediction Parameter Derivation Unit 304 may derive different intra prediction modes based on luminance and color difference.

[0165] The loop filter 305 is a filter installed within the encoding loop that removes block distortion and ringing distortion, thereby improving image quality. The loop filter 305 applies filters such as a deblocking filter, sample-adaptive offset (SAO), and adaptive loop filter (ALF) to the decoded CU image generated by the summing unit 312.

[0166] The reference picture memory 306 stores the decoded image of the CU at a predetermined location for each target picture and target CU.

[0167] The prediction parameter memory 307 stores prediction parameters at predetermined locations for each CTU or CU. Specifically, the prediction parameter memory 307 stores parameters decoded by the parameter decoding unit 302 and parameters derived by the prediction parameter derivation unit 320, etc.

[0168] The predictive image generation unit 308 receives the parameters derived by the predictive parameter derivation unit 320. The predictive image generation unit 308 also reads a reference picture from the reference picture memory 306. The predictive image generation unit 308 generates a block or subblock of a predicted image using the parameters and the reference picture (reference picture block) in the prediction mode indicated by predMode. Here, a reference picture block is a collection of pixels on the reference picture (usually rectangular, hence called a block), and is the region referenced to generate the predicted image.

[0169] (Interpretation Image Generation Unit 309) When predMode indicates interpretation mode, the interpretation image generation unit 309 generates a block or subblock prediction image by interpretation using the interpretation parameters input from the interpretation parameter derivation unit 303 and a reference picture.

[0170] Figure 11 is a schematic diagram showing the configuration of the inter-predictive image generation unit 309 included in the predictive image generation unit 308 according to this embodiment. The inter-predictive image generation unit 309 is composed of a motion compensation unit (predictive image generation device) 3091 and a synthesis unit 3095. The synthesis unit 3095 is composed of an IntraInter synthesis unit 30951, a Triangle synthesis unit 30952, a BIO unit 30954, and a weight prediction unit 3094 that generate predictive images for intra-inter-prediction (CIIP mode).

[0171] (Motion Compensation) The motion compensation unit 3091 (interpolation image generation unit 3091) generates an interpolated image (motion-compensated image) by reading a reference block from the reference picture memory 306 based on the inter-prediction parameters (predFlagLX, refIdxLX, mvLX) input from the inter-prediction parameter derivation unit 303. The reference block is the block located at a position shifted by mvLX from the position of the target block on the reference picture RefPicLX specified by refIdxLX. If mvLX is not integer precision, a filter called a motion compensation filter is applied to generate pixels at decimal positions to generate the interpolated image.

[0172] The motion compensation unit 3091 first derives the integer position (xInt, yInt) and phase (xFrac, yFrac) corresponding to the coordinates (x, y) within the prediction block using the following formula.

[0173] xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x xFrac = mvLX[0]&(MVPREC-1) yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y yFrac = mvLX[1]&(MVPREC-1) Here, (xPb,yPb) are the top-left coordinates of a block of size bW*bH, x=0...bW-1, y=0...bH-1, and MVPREC indicates the precision of mvLX (1 / MVPREC pixel precision). For example, MVPREC=16.

[0174] The motion compensation unit 3091 derives a temporary image temp[][] by performing horizontal interpolation on the reference picture refImg using an interpolation filter. The following Σ is the sum with respect to k of k=0..NTAP-1, shift1 is a normalization parameter that adjusts the range of values, and offset1=1<<(shift1-1).

[0175] temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1 Next, the motion compensation unit 3091 derives the interpolated image Pred[][] by vertical interpolation of the temporary image temp[][]. The following Σ is the sum with respect to k from k=0 to NTAP-1, shift2 is a normalization parameter that adjusts the range of values, and offset2=1<<(shift2-1).

[0176] Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2 Note that in the case of biprediction, the above Pred[][] is derived for each L0 list and L1 list (called the interpolated images PredL0[][] and PredL1[][]), and the interpolated image Pred[][] is generated from PredL0[][] and PredL1[][].

[0177] The synthesis unit 3095 includes an IntraInter synthesis unit 30951, a Triangle synthesis unit 30952, a weight prediction unit 3094, and a BIO unit 30954.

[0178] (IntraInter synthesis processing) When ciip_flag is 1, the intra predictive image generation unit 310 sets to planar prediction (IntraPredModeY = INTRA_PLANAR) and generates the predictive image predSamplesIntra[][].

[0179] The interprediction image generation unit 309 generates a predicted image predSamplesInter[][] by performing motion compensation using the motion vector obtained from merge prediction when ciip_flag is 1.

[0180] If ciip_flag is 1, the IntraInter synthesis unit 30951 generates a predicted image predSamplesComb[][] by weighted summing of the inter-predicted image predSamplesInter[][] and the intra-predicted image predSamplesIntra[][] and outputs it to the adder 312.

[0181] predSamplesComb[x][y] = (w * predSamplesIntra[x][y] + (4 - w) * predSamplesInter[x][y] + 2) >> 2 Here, w is set to 3 if both the adjacent blocks above and to the left of the target CU are in intra mode, 1 if both are not in intra mode, and 2 otherwise.

[0182] (Triangle synthesis process) The Triangle synthesis unit 30952 generates a predicted image using the Triangle prediction described above.

[0183] (BIO Prediction) Next, the details of the BIO prediction (Bi-Directional Optical Flow, BDOF processing) performed by the BIO unit 30954 will be explained. In the dual prediction mode, the BIO unit 30954 generates a prediction image by referring to two prediction images (the first prediction image and the second prediction image) and a gradient correction term.

[0184] If the interprediction parameter decoding unit 303 determines that it is a unidirectional prediction of L0, the motion compensation unit 3091 generates PredL0[x][y]. If the interprediction parameter decoding unit 303 determines that it is a unidirectional prediction of L1, the motion compensation unit 3091 generates PredL1[x][y]. On the other hand, if the interprediction parameter decoding unit 303 determines that it is a bidirectional prediction mode, the synthesis unit 3095 refers to bioAvailableFlag, which indicates whether or not to perform BIO processing, and determines whether or not BIO processing is necessary. If bioAvailableFlag indicates TRUE, the BIO unit 30954 performs BIO processing and generates a bidirectional prediction image, and if it indicates FALSE, the synthesis unit 3095 generates a prediction image using normal bidirectional prediction image generation.

[0185] The interprediction parameter decoding unit 303 may derive TRUE for bioAvailableFlag if the L0 reference image refImgL0 and the L1 reference image refImgL1 are different reference images and the two pictures are facing in opposite directions relative to the target picture.

[0186] (Weight Prediction) The weight prediction unit 3094 generates a predicted image of a block by multiplying the interpolated image PredLX by a weight coefficient. If one of the prediction list usage flags (predFlagL0 or predFlagL1) is 1 (single prediction) and weight prediction is not used, the following process is performed to adjust PredLX (LX is L0 or L1) to the number of pixels bit depth.

[0187] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredLX[x][y]+offset1)> >shift1) Here, shift1 = 14-bit Depth and offset1 = 1 << (shift1 - 1). Also, if both prediction list usage flags (predFlagL0 and predFlagL1) are 1 (dual prediction PRED_BI) and weight prediction is not used, the following process is performed by averaging PredL0 and PredL1 to match the number of pixels.

[0188] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredL0[x][y]+PredL1[x][y]+offset2)> >shift2) Here, shift2 = 15-bit Depth and offset2 = 1 << (shift2 - 1).

[0189] Furthermore, when performing single prediction and weight prediction, the weight prediction unit 3094 derives the weight prediction coefficient w0 and offset o0 from the encoded data and performs the following processing.

[0190] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,((PredLX[x][y]*w0+2^(log2WD-1))> >log2WD)+o0) Here, log2WD is a variable that represents a predetermined shift amount.

[0191] Furthermore, when performing dual prediction PRED_BI and weight prediction, the weight prediction unit 3094 derives the weight prediction coefficients w0, w1, o0, and o1 from the encoded data and performs the following processing.

[0192] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,(PredL0[x][y]*w0+PredL1[x][y]*w1+((o0+o1+1)<<log2WD))> >(log2WD+1)) The interpretation image generation unit 309 outputs the predicted image of the generated block to the summing unit 312.

[0193] (Intra-predictive image generation unit 310) When predMode indicates intra-predictive mode, the intra-predictive image generation unit 310 performs intra-predictive prediction using the intra-predictive parameters input from the intra-predictive parameter derivation unit 304 and the reference pixels read from the reference picture memory 306.

[0194] Specifically, the intra-predictive image generation unit 310 reads adjacent blocks on the target picture that are within a predetermined range from the target block from the reference picture memory 306. The predetermined range consists of adjacent blocks to the left, upper left, top, and upper right of the target block, and the area referenced differs depending on the intra-predictive mode.

[0195] The intra-predictive image generation unit 310 generates a predicted image of the target block by referring to the read decoded pixel value and the prediction mode indicated by IntraPredMode. The intra-predictive image generation unit 310 outputs the generated predicted image of the block to the summing unit 312.

[0196] The generation of predicted images based on the intra-prediction mode is described below. In planar prediction, DC prediction, and angular prediction, the decoded surrounding region adjacent to (nearby) the block to be predicted is set as the reference region R. Then, the predicted image is generated by extrapolating the pixels on the reference region R in a specific direction. For example, the reference region R may be set as an L-shaped region including the left and top (or even the top-left, top-right, and bottom-left) of the block to be predicted.

[0197] Planar prediction generates a provisional predicted image by linearly adding reference samples s[x][y] according to the distance between the predicted pixel position and the reference pixel position.

[0198] The inverse quantization / inverse transformation unit 311 inversely quantizes the quantization transformation coefficients input from the parameter decoding unit 302 to obtain the transformation coefficients.

[0199] The summing unit 312 adds the predicted image of the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization / inverse transform unit 311 pixel by pixel to generate a decoded image of the block. The summing unit 312 stores the decoded image of the block in the reference picture memory 306 and also outputs it to the loop filter 305.

[0200] (Configuration of the video encoding device) Next, the configuration of the video encoding device 11 according to this embodiment will be described. Figure 12 is a block diagram showing the configuration of the video encoding device 11 according to this embodiment. The video encoding device 11 is composed of a prediction image generation unit 101, a subtraction unit 102, a transformation / quantization unit 103, an inverse quantization / inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determination unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

[0201] The prediction image generation unit 101 generates a prediction image for each CU. The prediction image generation unit 101 includes the inter-prediction image generation unit 309 and the intra-prediction image generation unit 310, which have already been described, so their description will be omitted.

[0202] The subtraction unit 102 subtracts the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of image T to generate a prediction error. The subtraction unit 102 outputs the prediction error to the conversion and quantization unit 103.

[0203] The conversion / quantization unit 103 calculates conversion coefficients by frequency conversion for the prediction error input from the subtraction unit 102, and derives quantized conversion coefficients by quantization. The conversion / quantization unit 103 outputs the quantized conversion coefficients to the parameter coding unit 111 and the inverse quantization / inverse conversion unit 105.

[0204] The inverse quantization / inverse transformation unit 105 is the same as the inverse quantization / inverse transformation unit 311 (Figure 7) in the video decoding device 31, and therefore its explanation is omitted. The calculated prediction error is output to the summing unit 106.

[0205] The parameter coding unit 111 comprises a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (predictive mode coding unit). The CU coding unit 1112 further comprises a TU coding unit 1114. The general operation of each module is described below.

[0206] The header encoding unit 1110 performs encoding processing on parameters such as header information, partitioning information, prediction information, and quantization conversion coefficients.

[0207] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information, etc.

[0208] The CU encoding unit 1112 encodes CU information, prediction information, division information, etc.

[0209] The TU encoding unit 1114 encodes the QP update information and the quantization prediction error when the TU contains a prediction error.

[0210] The CT information coding unit 1111 and the CU coding unit 1112 supply syntax elements such as inter-prediction parameters (predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX), intra-prediction parameters (intra_luma_mpm_flag, intra_luma_mpm_idx, intra_luma_mpm_reminder, intra_chroma_pred_mode), and quantization conversion coefficients to the parameter coding unit 111.

[0211] The entropy coding unit 104 receives quantization conversion coefficients and coding parameters (partitioning information, prediction parameters) from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output a coded stream Te.

[0212] The prediction parameter derivation unit 120 is a means that includes an inter-prediction parameter coding unit 112 and an intra-prediction parameter coding unit 113, and derives intra-prediction parameters and intra-prediction parameters from parameters input from the coding parameter determination unit 110. The derived intra-prediction parameters and intra-prediction parameters are output to the parameter coding unit 111.

[0213] (Configuration of the Interpretation Parameter Coding Unit) As shown in Figure 13, the Interpretation Parameter Coding Unit 112 is configured to include a parameter coding control unit 1121 and an Interpretation Parameter Derivation Unit 303. The Interpretation Parameter Derivation Unit 303 has the same configuration as the video decoding device. The parameter coding control unit 1121 includes a merge index derivation unit 11211 and a vector candidate index derivation unit 11212.

[0214] The merge index derivation unit 11211 derives merge candidates, etc., and outputs them to the inter-prediction parameter derivation unit 303. The vector candidate index derivation unit 11212 derives prediction vector candidates, etc., and outputs them to the inter-prediction parameter derivation unit 303 and the parameter coding unit 111.

[0215] (Configuration of the Intra Prediction Parameter Coding Unit 113) The Intra Prediction Parameter Coding Unit 113 comprises a parameter coding control unit 1131 and an Intra Prediction Parameter Derivation Unit 304. The Intra Prediction Parameter Derivation Unit 304 has the same configuration as the video decoding device.

[0216] The parameter coding control unit 1131 derives IntraPredModeY and IntraPredModeC. Furthermore, it determines intra_luma_mpm_flag by referring to mpmCandList[]. These prediction parameters are output to the intra prediction parameter derivation unit 304 and the parameter coding unit 111.

[0217] However, unlike the video decoding device, the inputs to the inter-prediction parameter derivation unit 303 and the intra-prediction parameter derivation unit 304 are the coding parameter determination unit 110 and the prediction parameter memory 108, and these are output to the parameter coding unit 111.

[0218] The addition unit 106 generates a decoded image by adding the pixel values ​​of the prediction blocks input from the prediction image generation unit 101 and the prediction error input from the inverse quantization / inverse transform unit 105 for each pixel. The addition unit 106 stores the generated decoded image in the reference picture memory 109.

[0219] The loop filter 107 applies a deblocking filter, SAO, and ALF to the decoded image generated by the summing unit 106. Note that the loop filter 107 does not necessarily have to include the three types of filters mentioned above; for example, it may consist of only a deblocking filter.

[0220] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in predetermined locations for each target picture and CU.

[0221] The reference picture memory 109 stores the decoded images generated by the loop filter 107 at predetermined locations for each target picture and CU.

[0222] The coding parameter determination unit 110 selects one set from among several sets of coding parameters. The coding parameters are the QT, BT, or TT segmentation information, prediction parameters, or parameters that are to be coded and generated in relation to these. The prediction image generation unit 101 generates a prediction image using these coding parameters.

[0223] The coding parameter determination unit 110 calculates an RD cost value for each of the multiple sets, which indicates the magnitude of the information and the coding error. The RD cost value is, for example, the sum of the code amount and the squared error multiplied by a coefficient λ. The code amount is the information amount of the coded stream Te obtained by entropy coding the quantization error and coding parameters. The squared error is the sum of the squares of the prediction errors calculated in the subtraction unit 102. The coefficient λ is a preset real number greater than zero. The coding parameter determination unit 110 selects the set of coding parameters that minimizes the calculated cost value. The coding parameter determination unit 110 outputs the determined coding parameters to the parameter coding unit 111 and the prediction parameter derivation unit 120.

[0224] Furthermore, some parts of the video encoding device 11 and video decoding device 31 in the above-described embodiment, such as the entropy decoding unit 301, parameter decoding unit 302, loop filter 305, prediction image generation unit 308, inverse quantization / inverse transformation unit 311, addition unit 312, prediction parameter derivation unit 320, prediction image generation unit 101, subtraction unit 102, transformation / quantization unit 103, entropy encoding unit 104, inverse quantization / inverse transformation unit 105, loop filter 107, encoding parameter determination unit 110, parameter encoding unit 111, and prediction parameter derivation unit 120, may be implemented using a computer. In that case, a program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Hereinafter, "computer system" refers to a computer system built into either the video encoding device 11 or the video decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. In addition, "computer-readable recording media" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. Moreover, the above-mentioned programs may be for the purpose of realizing some of the functions described above, and may also be programs that can realize the aforementioned functions in combination with programs already recorded in the computer system.

[0225] Furthermore, some or all of the video encoding device 11 and video decoding device 31 in the above-described embodiment may be implemented as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 11 and video decoding device 31 may be individually implemented as a processor, or some or all of them may be integrated into a single processor. In addition, the method of implementing the integrated circuit is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. Furthermore, if an integrated circuit technology that can replace LSIs emerges due to advances in semiconductor technology, an integrated circuit using that technology may be used.

[0226] Although one embodiment of this invention has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes can be made without departing from the spirit of this invention.

[0227] [Application Examples] The video encoding device 11 and video decoding device 31 described above can be installed and used in various devices that transmit, receive, record, and play back video. The video may be natural video captured by a camera or the like, or it may be artificial video (including CG and GUI) generated by a computer or the like.

[0228] First, with reference to Figure 2, we will explain how the aforementioned video encoding device 11 and video decoding device 31 can be used for transmitting and receiving video.

[0229] Figure 2(a) is a block diagram showing the configuration of the transmitter PROD_A equipped with the video encoding device 11. As shown in the figure, the transmitter PROD_A includes an encoding unit PROD_A1 that obtains encoded data by encoding video, a modulation unit PROD_A2 that obtains a modulated signal by modulating a carrier wave with the encoded data obtained by the encoding unit PROD_A1, and a transmitter PROD_A3 that transmits the modulated signal obtained by the modulation unit PROD_A2. The video encoding device 11 described above is used as this encoding unit PROD_A1.

[0230] The transmitting device PROD_A may further include a camera PROD_A4 for capturing moving images, a recording medium PROD_A5 for recording the moving images, an input terminal PROD_A6 for receiving moving images from an external source, and an image processing unit A7 for generating or processing images, as sources for supplying moving images to the encoding unit PROD_A1. In the figure, a configuration in which the transmitting device PROD_A includes all of these is shown as an example, but some may be omitted.

[0231] The recording medium PROD_A5 may contain unencoded video footage, or it may contain video footage encoded using a recording encoding method different from the transmission encoding method. In the latter case, a decoding unit (not shown) may be interposed between the recording medium PROD_A5 and the encoding unit PROD_A1 to decode the encoded data read from the recording medium PROD_A5 according to the recording encoding method.

[0232] Figure 2(b) is a block diagram showing the configuration of the receiver PROD_B equipped with the video decoding device 31. As shown in the figure, the receiver PROD_B includes a receiver PROD_B1 that receives a modulated signal, a demodulation unit PROD_B2 that obtains encoded data by demodulating the modulated signal received by the receiver PROD_B1, and a decoding unit PROD_B3 that obtains a video by decoding the encoded data obtained by the demodulation unit PROD_B2. The video decoding device 31 described above is used as this decoding unit PROD_B3.

[0233] The receiving device PROD_B may further include a display PROD_B4 for displaying the video, a recording medium PROD_B5 for recording the video, and an output terminal PROD_B6 for outputting the video to the outside, as recipients of the video output from the decoding unit PROD_B3. In the figure, a configuration in which the receiving device PROD_B includes all of these is shown as an example, but some may be omitted.

[0234] The recording medium PROD_B5 may be for recording unencoded video, or it may be encoded using a recording encoding method different from the transmission encoding method. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_B3 and the recording medium PROD_B5, which encodes the video acquired from the decoding unit PROD_B3 according to the recording encoding method.

[0235] The transmission medium for transmitting the modulated signal may be wireless or wired. Furthermore, the transmission method for transmitting the modulated signal may be broadcasting (referring here to a transmission method where the destination is not predetermined) or communication (referring here to a transmission method where the destination is predetermined). In other words, the transmission of the modulated signal may be achieved by wireless broadcasting, wired broadcasting, wireless communication, or wired communication.

[0236] For example, a terrestrial digital broadcasting station (broadcasting equipment, etc.) / receiving station (television receiver, etc.) is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals wirelessly. Similarly, a cable television broadcasting station (broadcasting equipment, etc.) / receiving station (television receiver, etc.) is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via wired broadcasting.

[0237] Furthermore, servers (such as workstations) and clients (such as television sets, personal computers, and smartphones) for internet-based VOD (Video On Demand) services and video sharing services are examples of transmitting devices PROD_A and receiving devices PROD_B that transmit and receive modulated signals via communication (typically, in a LAN, either wireless or wired transmission is used as the transmission medium, while in a WAN, wired transmission is used). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Smartphones also include multi-function mobile phone terminals.

[0238] Furthermore, the video sharing service client has the function of decrypting encoded data downloaded from the server and displaying it on the screen, as well as the function of encoding video images captured by the camera and uploading them to the server. In other words, the video sharing service client functions as both a transmitting device PROD_A and a receiving device PROD_B.

[0239] Next, with reference to Figure 3, we will explain how the aforementioned video encoding device 11 and video decoding device 31 can be used for recording and playing back video.

[0240] Figure 3(a) is a block diagram showing the configuration of the recording device PROD_C equipped with the video encoding device 11 described above. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that obtains encoded data by encoding video, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 to the recording medium PROD_M. The video encoding device 11 described above is used as this encoding unit PROD_C1.

[0241] The recording medium PROD_M may be (1) a type built into the recording device PROD_C, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), (2) a type connected to the recording device PROD_C, such as an SD memory card or USB (Universal Serial Bus) flash memory, or (3) a type loaded into a drive device (not shown) built into the recording device PROD_C, such as a DVD (Digital Versatile Disc: registered trademark) or BD (Blu-ray Disc: registered trademark).

[0242] Furthermore, the recording device PROD_C may also include a camera PROD_C3 for capturing moving images, an input terminal PROD_C4 for receiving moving images from an external source, a receiving unit PROD_C5 for receiving moving images, and an image processing unit PROD_C6 for generating or processing images, as sources of moving images to be input to the encoding unit PROD_C1. In the figure, a configuration in which the recording device PROD_C includes all of these is shown as an example, but some may be omitted.

[0243] The receiving unit PROD_C5 may receive unencoded video footage, or it may receive encoded data encoded using a transmission encoding scheme different from the recording encoding scheme. In the latter case, it is preferable to interpose a transmission decoding unit (not shown) between the receiving unit PROD_C5 and the encoding unit PROD_C1 to decode the encoded data encoded using the transmission encoding scheme.

[0244] Examples of such recording devices PROD_C include DVD recorders, BD recorders, and HDD (Hard Disk Drive) recorders (in this case, the input terminal PROD_C4 or the receiver PROD_C5 is the main source of moving images). Camcorders (in this case, the camera PROD_C3 is the main source of moving images), personal computers (in this case, the receiver PROD_C5 or the image processing unit C6 is the main source of moving images), and smartphones (in this case, the camera PROD_C3 or the receiver PROD_C5 is the main source of moving images) are also examples of such recording devices PROD_C.

[0245] Figure 3(b) is a block diagram showing the configuration of the playback device PROD_D equipped with the video decoding device 31 described above. As shown in the figure, the playback device PROD_D includes a reading unit PROD_D1 that reads encoded data written to the recording medium PROD_M, and a decoding unit PROD_D2 that obtains a video by decoding the encoded data read by the reading unit PROD_D1. The video decoding device 31 described above is used as this decoding unit PROD_D2.

[0246] The recording medium PROD_M may be (1) a type built into the playback device PROD_D, such as an HDD or SSD; (2) a type connected to the playback device PROD_D, such as an SD memory card or USB flash memory; or (3) a type loaded into a drive device (not shown) built into the playback device PROD_D, such as a DVD or BD.

[0247] Furthermore, the playback device PROD_D may also include a display PROD_D3 for displaying the video, an output terminal PROD_D4 for outputting the video externally, and a transmission unit PROD_D5 for transmitting the video, as recipients of the video output from the decoding unit PROD_D2. In the figure, a configuration in which the playback device PROD_D includes all of these is shown as an example, but some may be omitted.

[0248] The transmitting unit PROD_D5 may transmit unencoded video, or it may transmit encoded data encoded using a transmission encoding scheme different from the recording encoding scheme. In the latter case, it is preferable to interpose an encoding unit (not shown) between the decoding unit PROD_D2 and the transmitting unit PROD_D5 that encodes the video using the transmission encoding scheme.

[0249] Examples of such playback devices PROD_D include DVD players, BD players, and HDD players (in this case, the output terminal PROD_D4 to which a television receiver is connected becomes the main destination for the video). Other examples of such playback devices PROD_D include television receivers (in this case, the display PROD_D3 becomes the main destination for the video), digital signage (also called electronic billboards or electronic display boards, etc., where the display PROD_D3 or the transmitter PROD_D5 becomes the main destination for the video), desktop PCs (in this case, the output terminal PROD_D4 or the transmitter PROD_D5 becomes the main destination for the video), laptop or tablet PCs (in this case, the display PROD_D3 or the transmitter PROD_D5 becomes the main destination for the video), and smartphones (in this case, the display PROD_D3 or the transmitter PROD_D5 becomes the main destination for the video).

[0250] (Hardware and Software Implementation) Furthermore, each block of the video decoding device 31 and video encoding device 11 described above may be implemented in hardware by logic circuits formed on an integrated circuit (IC chip), or in software using a CPU (Central Processing Unit).

[0251] In the latter case, each of the above devices includes a CPU that executes program instructions to realize each function, a ROM (Read Only Memory) that stores the program, a RAM (Random Access Memory) that loads the program, and a storage device (recording medium) such as memory that stores the program and various data. Furthermore, the objective of the embodiment of the present invention can also be achieved by supplying each of the above devices with a recording medium on which the program code (executable program, intermediate code program, source program) of the control program of each of the above devices, which is software that realizes the above functions, is recorded in a way that can be read by a computer, and the computer (or CPU or MPU) reads and executes the program code recorded on the recording medium.

[0252] Examples of recording media that can be used include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy disks and hard disks, optical disks such as CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical discs), MDs (Mini Discs), DVDs (Digital Versatile Discs), CD-Rs (CD Recordable), and Blu-ray Discs (Blu-ray Discs), cards such as IC cards (including memory cards) and optical cards, semiconductor memories such as mask ROMs, EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable and Programmable Read-Only Memory), and flash ROMs, or logic circuits such as PLDs (Programmable logic devices) and FPGAs (Field Programmable Gate Arrays).

[0253] Furthermore, each of the above devices may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. This communication network is not particularly limited, as long as it is capable of transmitting the program code. For example, the Internet, intranet, extranet, LAN (Local Area Network), ISDN (Integrated Services Digital Network), VAN (Value-Added Network), CATV (Community Antenna television / Cable Television) communication network, Virtual Private Network, telephone line network, mobile communication network, satellite communication network, etc., can be used. Also, the transmission medium constituting this communication network is not limited to a specific configuration or type, as long as it is capable of transmitting the program code. For example, it can be used with wired connections such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carriers, cable TV lines, telephone lines, and ADSL (Asymmetric Digital Subscriber Line) lines, as well as wireless connections such as IrDA (Infrared Data Association), infrared (like remote controls), Bluetooth®, IEEE 802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance®), mobile phone networks, satellite lines, and terrestrial digital broadcasting networks. Furthermore, embodiments of the present invention can also be realized in the form of computer data signals embedded in a carrier wave, where the above program code is embodied through electronic transmission.

[0254] The embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the claims. That is, embodiments obtained by combining technical means that have been appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

[0255] Embodiments of the present invention can be suitably applied to a video decoding device that decodes encoded data from image data, and a video encoding device that generates encoded data from image data. Furthermore, they can be suitably applied to the data structure of encoded data generated by the video encoding device and referenced by the video decoding device.

[0256] (Summary) An image decoding device according to one aspect of the present invention has a parameter decoding unit that decodes parameters for generating a predicted image, and when the regular merge flag indicates regular merge mode, it checks a flag indicating whether or not MMVD prediction notified by the sequence parameter set or the like is available, and if MMVD prediction is not available, it decodes motion vector information obtained from the merge candidate.

[0257] An image decoding device according to one aspect of the present invention is characterized in that, when the regular merge flag indicates regular merge mode, it decodes a flag sps_mmvd_enabled_flag indicating whether or not MMVD prediction notified by the sequence parameter set, etc., is available, and a flag mmvd_merge_flag indicating whether or not MMVD prediction is used on a CU basis, and when mmvd_merge_flag==0 or sps_mmvd_enabled_flag==0, and the number of merge candidates MaxNumMergeCand is greater than 1, the parameter decoding unit decodes an index merge_idx for selecting from the merge candidates as motion vector information.

[0258] An image encoding device according to one aspect of the present invention has parameter encoding for encoding parameters for generating a predicted image, and when the regular merge flag indicates regular merge mode, it checks a flag indicating whether or not MMVD prediction notified by a sequence parameter set or the like is available, and if MMVD prediction is not available, it encodes motion vector information obtained from merge candidates.

[0259] An image encoding device according to one aspect of the present invention is characterized in that, when the regular merge flag indicates regular merge mode, it encodes a flag sps_mmvd_enabled_flag indicating whether or not MMVD prediction notified by the sequence parameter set, etc., is available, and a flag mmvd_merge_flag indicating whether or not to use MMVD prediction on a CU basis, and when mmvd_merge_flag==0 or sps_mmvd_enabled_flag==0, and the number of merge candidates MaxNumMergeCand is greater than 1, the parameter encoding unit encodes an index merge_idx for selecting from the merge candidates as motion vector information.

[0260] By adopting this configuration, even when MMVD mode is prohibited in the higher-level syntax, merge mode can be selectively used, resulting in high encoding efficiency.

[0261] An image decoding device according to one aspect of the present invention is an image decoding device for decoding parameters for generating a prediction image, comprising a parameter decoding unit that decodes a regular merge flag from merge data indicating whether or not a regular merge mode is used in interprediction, wherein if the regular merge flag indicates that the regular merge mode is used in interprediction, the parameter decoding unit checks a flag notified in the sequence parameter set indicating whether or not a merge candidate motion vector is valid, and if the value of the flag is 1, it decodes an MMVD merge flag indicating whether or not the merge candidate motion vector is used to generate the interprediction parameters of the target coding unit, and uses the MMVD merge flag to decode a merge index, which is an index of the merge candidate list.

[0262] An image decoding device according to one aspect of the present invention is characterized in that the merge index indicates that the MMVD merge flag does not use the motion vector of the merge candidate to generate the interpretation parameter, and decoding is performed when the number of merge candidates is greater than 1.

[0263] An image decoding device according to one aspect of the present invention is characterized in that, when the value of the MMVD merge flag is 0, the value of the merge index is estimated to be 0.

[0264] An image encoding device according to one aspect of the present invention is an image encoding device for encoding parameters for generating a prediction image, comprising a parameter encoding unit that encodes a regular merge flag from merge data indicating whether or not a regular merge mode is used in inter-prediction, wherein if the regular merge flag indicates that the regular merge mode is used in inter-prediction, the parameter encoding unit checks a flag notified in the sequence parameter set indicating whether or not a merge candidate motion vector is valid, and if the value of the flag is 1, encodes an MMVD merge flag indicating whether or not the merge candidate motion vector is used to generate the inter-prediction parameters of the target encoding unit, and encodes a merge index, which is an index of the merge candidate list, using the MMVD merge flag.

[0265] An image decoding method according to one aspect of the present invention is an image decoding method for decoding parameters for generating a prediction image, comprising at least the steps of: decoding a regular merge flag from merge data that indicates whether or not a regular merge mode is used in interprediction; if the regular merge flag indicates that the regular merge mode is used in interprediction, checking a flag notified in the sequence parameter set that indicates whether or not a merge candidate motion vector is valid; if the value of the flag is 1, decoding an MMVD merge flag that indicates whether or not the merge candidate motion vector is used to generate the interprediction parameters of the target coding unit; and decoding a merge index, which is an index of a merge candidate list, using the MMVD merge flag.

[0266] (Cross-reference of related applications) This application claims priority to Japanese Patent Application No. 2019-135746, filed on 24 July 2019, and by reference thereto, all of its contents are included herein.

[0267] 31 Image Decoder 301 Entropy Decoder 302 Parameter Decoder 303 Inter-Prediction Parameter Derivation Unit 304 Intra-Prediction Parameter Derivation Unit 305, 107 Loop Filter 306, 109 Reference Picture Memory 307, 108 Prediction Parameter Memory 308, 101 Prediction Image Generation Unit 309 Inter-Prediction Image Generation Unit 310 Intra-Prediction Image Generation Unit 311, 105 Inverse Quantization / Inverse Transform Unit 312, 106 Addition Unit 320 Prediction Parameter Derivation Unit 11 Image Coding Device 102 Subtraction Unit 103 Transformation / Quantization Unit 104 Entropy Coding Unit 110 Coding Parameter Determination Unit 111 Parameter Coding Unit 112 Inter-Prediction Parameter Coding Unit 113 Intra-Prediction Parameter Coding Unit 120 Prediction Parameter Derivation Unit

Claims

1. An image decoding device that decodes parameters for generating a predicted image, comprising a parameter decoding unit that decodes a regular merge flag from merge data that indicates whether or not regular merge mode is used in inter prediction, wherein the parameter decoding unit: if the regular merge flag indicates that the regular merge mode is used in inter prediction, checks a flag that indicates whether or not the motion vector of a merge candidate is valid, which is notified in a sequence parameter set; if the value of the flag is 1, decodes an MMVD merge flag that indicates whether or not the motion vector of the merge candidate is used to generate inter prediction parameters for the current coding unit; and uses the MMVD merge flag to decode a merge index, which is an index of a merge candidate list.

2. The image decoding device described in claim 1, characterized in that the merge index is decoded when the MMVD merge flag indicates that the motion vector of the merge candidate is not used to generate the inter prediction parameters and the number of merge candidates is greater than 1.

3. The image decoding device according to claim 1, wherein if the value of the MMVD merge flag is 0, the value of the merge index is estimated to be 0.

4. An image coding device that encodes parameters for generating a predicted image, comprising a parameter coding unit that codes a regular merge flag indicating whether or not a regular merge mode is used in inter prediction from merge data, wherein the parameter coding unit: if the regular merge flag indicates that the regular merge mode is used in inter prediction, checks a flag that indicates whether or not a motion vector of a merge candidate is valid, which is notified in a sequence parameter set; if the value of the flag is 1, codes an MMVD merge flag that indicates whether or not the motion vector of the merge candidate is used to generate inter prediction parameters for a target coding unit; and uses the MMVD merge flag to code a merge index, which is an index of a merge candidate list.

5. An image decoding method for decoding parameters for generating a predicted image, comprising at least the steps of: decoding a regular merge flag from merge data indicating whether or not the regular merge mode is used in inter prediction; if the regular merge flag indicates that the regular merge mode is used in inter prediction, checking a flag notified in a sequence parameter set indicating whether or not the motion vector of the merge candidate is valid; if the value of the flag is 1, decoding an MMVD merge flag indicating whether or not the motion vector of the merge candidate is used to generate inter prediction parameters for the current coding unit; and decoding a merge index, which is an index of a merge candidate list, using the MMVD merge flag.

Images