Video decoding device and video encoding device

Abstract

Provided are video coding / decoding apparatus (11, 31) with enhanced coding efficiency. A video decoding apparatus (31) according to one aspect of the present invention comprises a prediction unit (30377) for deriving a prediction image by using a weight parameter derived by means of an angular variable, and a non-separable transform unit (31121) for performing a non-separable transform, the video decoding apparatus (31) characterized in that the non-separable transform unit (31121), when the inter-prediction mode is a GPM mode, derives a set number for non-separable transform (lfnstTrSetId) from GPM partition information (merge_gpm_partition_idx).

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a video decoding device and a video encoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates decoded images by decoding the encoded data.

[0003] Specific video encoding methods include, for example, H.264 / AVC and HEVC (High-Efficiency Video Coding) methods.

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing the images, coding tree units (CTUs) obtained by dividing the slices, coding units (sometimes called coding units (CUs)) obtained by dividing the coding tree units, and transform units (TUs) obtained by dividing the coding units, and are coded / decoded for each CU.

[0005] In such video coding methods, a predicted image is usually generated based on a locally decoded image obtained by encoding / decoding an input image, and the predicted image is subtracted from the input image (original image) to obtain a prediction error (sometimes called a "difference image" or "residual image"), which is then coded. Methods for generating predicted images include inter-frame prediction (inter-prediction) and intra-frame prediction (intra-prediction).

[0006] Non-Patent Document 1 also discloses a GPM (Geometric Partition Mode) mode in which a target block is divided into shapes other than a rectangle and different inter predictions are performed for each region. By dividing the target block into shapes other than a rectangle in this way, more accurate predictions can be made even for complex textures, improving coding efficiency. Non-Patent Document 1 also discloses an image coding device that performs non-separable transform for each transform unit in order to further concentrate coefficients obtained by separate transforming prediction errors in the low-frequency region. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] ITU-T Rec. H.266 Summary of the Invention [Problem to be solved by the invention]

[0008] In Non-Patent Document 1, non-separable transform and inverse non-separable transform techniques that are effective for directional prediction residuals are applied only to intra-predicted blocks, and are not adapted to inter-predicted blocks that have many diagonal directions.

[0009] An object of the present invention is to provide a video decoding device, a video encoding device, and related techniques that can more suitably apply inverse non-separable transform. [Means for solving the problem]

[0010] In order to solve the above problem, a video decoding device according to one embodiment of the present invention is a video decoding device that includes a prediction unit that derives a predicted image using weight parameters derived from angle variables, and a non-separable transformation unit that performs non-separable transformation, and is characterized in that when the inter prediction mode is GPM mode, the separation transformation unit derives a set number (lfnstTrSetId) of the non-separable transformation from GPM partition information (merge_gpm_partition_idx).

[0011] Another aspect of the present invention provides a video encoding device that includes a prediction unit that derives a predicted image using weight parameters derived from angle variables, and a non-separable transformation unit that performs non-separable transformation, wherein when the inter prediction mode is GPM mode, the separation transformation unit derives a set number (lfnstTrSetId) of the non-separable transformation from GPM partition information (merge_gpm_partition_idx). [Effects of the Invention]

[0012] According to an aspect of the present invention, it is possible to improve the coding efficiency in video coding / decoding processing. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic diagram showing the configuration of an image transmission system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing a hierarchical structure of data in an encoded stream. [Figure 3] FIG. 1 is a schematic diagram illustrating a configuration of a video decoding device. [Figure 4] 10 is a flowchart illustrating a schematic operation of the video decoding device. [Figure 5] FIG. 10 is a schematic diagram showing the configuration of an inter-prediction parameter derivation unit. [Figure 6] FIG. 10 is a schematic diagram showing the configuration of an inter-prediction image generation unit. [Figure 7] FIG. 1 is a block diagram showing a configuration of a video encoding device. [Figure 8] FIG. 10 is a schematic diagram showing the configuration of an inter-prediction parameter encoding unit. [Figure 9] FIG. 1 is a diagram illustrating GPM prediction. [Figure 10] FIG. 10 is a syntax diagram illustrating coding parameters for GPM prediction. [Figure 11]FIG. 10 is a diagram showing the structure of a table indicating the correspondence between merge_gpm_partition_idx, angleIdx, and distanceIdx in GPM prediction. [Figure 12] FIG. 10 is a diagram showing the structure of a table showing the correspondence between idx and DisLut[idx] used in the weighting coefficient derivation process and the motion vector storage process in GPM prediction. [Figure 13] 10 is a flowchart showing the flow of processing for GPM prediction. [Figure 14] 10 is a table used to select a transformation matrix for non-separable transformation in intra prediction mode. [Figure 15] This is a table used to select the transformation matrix for non-separable transformation in GPM mode. [Figure 16] This is a table used to select the transformation matrix for non-separable transformation in GPM mode. [Figure 17] This is a table used to select the transformation matrix for non-separable transformation in GPM mode. [Figure 18] This is a table used to select the transformation matrix for non-separable transformation in GPM mode. [Figure 19] This is a table used to select the transformation matrix for non-separable transformation in GPM mode. [Figure 20] FIG. 2 is a functional block diagram showing an example of the configuration of an inverse quantization and inverse transform unit. [Figure 21] 1 is an example of a syntax table showing coding parameters of a GPM non-separable transform. [Figure 22] 10 is an example of a syntax table showing flags for non-separable transforms. [Figure 23] 10 is an example of a syntax table showing flags for non-separable transforms. [Figure 24] 10 is a flowchart showing a processing flow when performing non-separable transformation in GPM. DETAILED DESCRIPTION OF THE INVENTION

[0014] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] FIG. 1 is a schematic diagram showing the configuration of an image transmission system 1 according to this embodiment.

[0016] The image transmission system 1 is a system that transmits an encoded stream obtained by encoding a target image, decodes the transmitted encoded stream, and displays the image. The image transmission system 1 includes a video encoding device (image encoding device) 11, a network 21, a video decoding device (image decoding device) 31, and a video display device (image display device) 41.

[0017] An image T is input to the video encoding device 11 .

[0018] The network 21 transmits the coded stream Te generated by the video coding device 11 to the video decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, but may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Furthermore, the network 21 may be replaced by a storage medium on which the coded stream Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

[0019] The video decoding device 31 decodes each of the coded streams Te transmitted over the network 21, and generates one or more decoded images Td.

[0020] The moving image display device 41 displays all or part of one or more decoded images Td generated by the moving image decoding device 31. The moving image display device 41 includes, for example, a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. Examples of the form of the display include stationary, mobile, and HMD. Also, when the moving image decoding device 31 has high processing power, an image with high image quality is displayed, and when it has only low processing power, an image that does not require high processing power and display ability is displayed.

[0021] <Operator> The operators used in this specification are described below.

[0022] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || represents a logical OR.

[0023] x?y:z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0024] Clip3(a, b, c) is a function that clips c to a value between a and b. When c < a, it returns a; when c > b, it returns b; and in other cases, it returns c (where a <= b).

[0025] abs(a) is a function that returns the absolute value of a.

[0026] Int(a) is a function that returns the integer value of a.

[0027] floor(a) is a function that returns the largest integer less than or equal to a.

[0028] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0029] a / d represents the division of a by d (truncating the decimal part).

[0030] <Structure of the coded stream Te> Before proceeding to a detailed description of the video encoding device 11 and video decoding device 31 according to this embodiment, the data structure of the encoded stream Te generated by the video encoding device 11 and decoded by the video decoding device 31 will be described.

[0031] Fig. 2 is a diagram showing a hierarchical structure of data in a coded stream Te. The coded stream Te illustratively includes a sequence and a plurality of pictures constituting the sequence. (a) to (f) of Fig. 2 show a coded video sequence that defines the sequence SEQ, a coded picture that defines the picture PICT, a coded slice that defines the slice S, coded slice data that defines the slice data, a coding tree unit included in the coded slice data, and a coding unit included in the coding tree unit, respectively.

[0032] (Coded Video Sequence) The coded video sequence defines a set of data that the video decoding device 31 refers to in order to decode the sequence SEQ to be processed. As shown in Fig. 2, the sequence SEQ includes a video parameter set (Video Parameter Set), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a picture PICT, and supplemental enhancement information (SEI).

[0033] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers included in the video and each individual layer.

[0034] The sequence parameter set SPS defines a set of coding parameters that the video decoding device 31 references to decode the target sequence. For example, the width and height of a picture are defined. Note that there may be multiple SPSs. In this case, one of the multiple SPSs is selected from the PPS.

[0035] The picture parameter set PPS defines a set of coding parameters that the video decoding device 31 references to decode each picture in the target sequence. For example, the picture parameter set PPS includes a reference value of the quantization width used in decoding the picture (pic_init_qp_minus26) and a flag indicating the application of weighted prediction (weighted_pred_flag). Note that there may be multiple PPSs. In this case, one of the multiple PPSs is selected for each picture in the target sequence.

[0036] (encoded picture) A coded picture defines a set of data that the video decoding device 31 refers to in order to decode a picture PICT to be processed. As shown in FIG. 2, the picture PICT includes slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0037] In the following description, when there is no need to distinguish between slices 0 to NS-1, the subscripts of the symbols may be omitted. This also applies to other data that are included in the coded stream Te described below and have subscripts.

[0038] (encoded slice) A coded slice defines a set of data that the video decoding device 31 refers to in order to decode a target slice S. As shown in Fig. 2, a slice includes a slice header and slice data.

[0039] The slice header includes a group of coding parameters that the video decoding device 31 refers to in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates the slice type is an example of a coding parameter included in the slice header.

[0040] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction during encoding, (2) a P slice that uses unidirectional prediction or intra prediction during encoding, and (3) a B slice that uses unidirectional prediction, bidirectional prediction, or intra prediction during encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, P and B slices refer to slices that include blocks that can use inter prediction.

[0041] Note that the slice header may include a reference to a picture parameter set PPS (pic_parameter_set_id).

[0042] (encoded slice data) The coded slice data defines a set of data that the video decoding device 31 references in order to decode the slice data to be processed. As shown in Fig. 2(d), the slice data includes a CTU. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0043] (coding tree unit) 2 defines a set of data that the video decoding device 31 references in order to decode a CTU to be processed. The CTU is divided into coding units (CUs), which are basic units of encoding processing, by recursive quad tree (QT) division, binary tree (BT) division, or ternary tree (TT) division. BT division and TT division are collectively called multi-tree (MT) division. A node in a tree structure obtained by recursive quad tree division is called a coding node. Intermediate nodes in a quad tree, binary tree, or ternary tree are coding nodes, and the CTU itself is defined as the top-level coding node.

[0044] The CT includes, as CT information, a CU split flag (split_cu_flag) indicating whether CT splitting is performed, a QT split flag (qt_split_cu_flag) indicating whether QT splitting is performed, an MT split direction (mtt_split_cu_vertical_flag) indicating the split direction of MT splitting, and an MT split type (mtt_split_cu_binary_flag) indicating the split type of MT splitting. split_cu_flag, qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

[0045] Different trees may be used for luminance and chrominance. The tree type is indicated by treeType. For example, if a common tree is used for luminance (Y, cIdx=0) and chrominance (Cb / Cr, cIdx=1,2), the common single tree is indicated by treeType=SINGLE_TREE. If two different trees (DUAL trees) are used for luminance and chrominance, the luminance tree is indicated by treeType=DUAL_TREE_LUMA and the chrominance tree is indicated by treeType=DUAL_TREE_CHROMA.

[0046] (encoding unit) 2 defines a set of data that the video decoding device 31 references in order to decode a coding unit to be processed. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines a prediction mode, etc.

[0047] Prediction processing may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU. If the sizes of a CU and a sub-CU are the same, there is one sub-CU in the CU. If the size of a CU is larger than the size of a sub-CU, the CU is divided into sub-CUs. For example, if the CU is 8x8 and the sub-CU is 4x4, the CU is divided into 2 parts horizontally and 2 parts vertically, into 4 sub-CUs.

[0048] Prediction types (prediction modes) include intra prediction (MODE_INTRA), inter prediction (MODE_INTER), and intra block copy (MODE_IBC). Intra prediction is prediction within the same picture, while inter prediction refers to prediction processing performed between different pictures (for example, between display times or between layer images).

[0049] The transform and quantization processes are performed in units of CU, but the quantized transform coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0050] (Prediction parameters) The predicted image is derived from prediction parameters associated with the block, which include intra-prediction and inter-prediction parameters.

[0051] (Prediction parameters for intra prediction) Next, the prediction parameters of intra prediction will be described. The intra prediction parameters are composed of a luminance prediction mode IntraPredModeY (predModeIntra) and a color difference prediction mode IntraPredModeC. There are, for example, 67 types of intra prediction modes (0 to 66). For example, there are planar prediction (0), DC prediction (1), and angular prediction (2 to 66).

[0052] (Inter prediction parameters) Prediction parameters for inter prediction will be described. The inter prediction parameters are composed of prediction list usage flags predFlagL0 and predFlagL1, reference picture indices refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. predFlagL0 and predFlagL1 are flags indicating whether or not a reference picture list (L0 list, L1 list) is used, and when the value is 1, the corresponding reference picture list is used. Note that in this specification, when the term "flag indicating whether XX is true" is used, a flag other than 0 (for example, 1) is considered to be XX, and 0 is considered to be not XX, and in logical negation, logical product, etc., 1 is treated as true and 0 is treated as false (the same applies below). However, in an actual device or method, other values ​​may be used as true and false values.

[0053] Syntax elements for deriving inter-prediction parameters include, for example, a merge flag merge_flag (general_merge_flag), a merge index merge_idx, merge_subblock_flag indicating whether to use sub-block-based inter-prediction such as affine mode, regulare_merge_flag, an intra-inter flag ciip_flag, GPM_flag indicating whether to use GPM mode (Geometric partitioning merge mode), merge_gpm_partition_idx indicating the partition shape of the GPM mode, merge_gpm_idx0 and merge_gpm_idx1 indicating GPM merge indices, an inter-prediction identifier inter_pred_idc for selecting a reference picture to be used in AMVP mode, a reference picture index refIdxLX, a prediction vector index mvp_LX_idx for deriving a motion vector, a difference vector mvdLX, and a motion vector precision mode amvr_mode. Note that GPM_flag (MergeGpmFlag) may not be explicitly coded, and may be derived as GPM_flag=1 (MergeGpmFlag=1) when ciip_flag=0.

[0054] (Reference Picture List) The reference picture list is a list of reference pictures stored in the reference picture memory 306 .

[0055] (Merge prediction and AMVP prediction) Prediction parameter decoding (encoding) methods include merge prediction mode (merge mode) and AMVP (Advanced Motion Vector Prediction, adaptive motion vector prediction) mode, and merge_flag is a flag for distinguishing between them. Merge mode is a prediction mode that omits some or all of the motion vector difference. It derives the prediction list usage flag predFlagLX, reference picture index refIdxLX, and motion vector mvLX from the encoded data, instead of including them in the encoded data, and instead derives them from prediction parameters of previously processed neighboring blocks. AMVP mode is a mode that includes inter_pred_idc, refIdxLX, and mvLX in the encoded data. Note that mvLX is encoded as mvp_LX_idx, which identifies the prediction vector mvpLX, and the difference vector mvdLX. Prediction modes that omit or simplify the motion vector difference are collectively called general merge mode, and general merge mode and AMVP prediction may be selected using general_merge_flag. Furthermore, the regular_merge_flag may be transmitted, and if the regular_merge_flag is 1, the normal merge mode or MMVD may be selected, and otherwise the CIIP mode or GPM mode may be selected.

[0056] inter_pred_idc is a value indicating the type and number of reference pictures, and takes one of the values ​​PRED_L0, PRED_L1, or PRED_BI. PRED_L0 and PRED_L1 indicate uni-prediction using one reference picture managed in the L0 list and L1 list, respectively. PRED_BI indicates bi-prediction using two reference pictures managed in the L0 list and L1 list.

[0057] The merge_idx is an index indicating which prediction parameter is to be used as the prediction parameter for the current block from among prediction parameter candidates (merge candidates) derived from blocks for which processing has been completed.

[0058] (motion vector) mvLX indicates the amount of shift between blocks on two different pictures. The predicted vector and differential vector related to mvLX are called mvpLX and mvdLX, respectively.

[0059] (Inter prediction identifier inter_pred_idc and prediction list usage flag predFlagLX) The relationship between inter_pred_idc, predFlagL0, and predFlagL1 is as follows, and they can be converted to each other.

[0060] inter_pred_idc = (predFlagL1<<1)+predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 Note that the inter prediction parameters may use a prediction list usage flag or an inter prediction identifier. Furthermore, the determination using the prediction list usage flag may be replaced with a determination using the inter prediction identifier. Conversely, the determination using the inter prediction identifier may be replaced with a determination using the prediction list usage flag.

[0061] (Configuration of video decoding device) The configuration of a video decoding device 31 (FIG. 3) according to this embodiment will be described.

[0062] The video decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generation unit (prediction image generation device) 308, an inverse quantization and inverse transform unit 311, an adder 312, and a prediction parameter derivation unit (not shown). Note that the video decoding device 31 may also be configured without including the loop filter 305, in accordance with the video coding device 11 described below.

[0063] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice headers (slice information) from the coded data. The CT information decoding unit 3021 decodes the CT from the coded data. The CU decoding unit 3022 decodes the CU from the coded data. When a prediction error is included in the TU, the TU decoding unit 3024 decodes QP update information (quantization correction value) and quantized prediction error (residual_coding) from the coded data.

[0064] The header decoding unit 3020 decodes a flag sps_lfnst_enabled_flag indicating whether or not non-separable transform is used from the SPS. Furthermore, the header decoding unit 3020 decodes ph_lfnst_enabled_flag from the picture header (PH) when sps_lfnst_enabled_flag is 1. When ph_lfnst_enabled_flag does not appear, the header decoding unit 3020 infers ph_lfnst_enabled_flag to be 0. Alternatively, when ph_lfnst_enabled_flag does not appear, the value of sps_lfnst_enabled_flag may be set as the value of ph_lfnst_enabled_flag.

[0065] The GPM non-separable transform in the present invention is an inverse non-separable transform (and non-separable transform) when inter prediction is in GPM mode. Below, we will explain a method for deriving a transform matrix or a set of transform matrices for a non-separable transform from the partition shape of GPM when applying to inter CUs.

[0066] FIG. 21 is an example of a syntax table showing coding parameters for the GPM non-separable transform.

[0067] The header decoding unit 3020 decodes the flag sps_gpm_lfnst_enabled_flag from the SPS. The sps_gpm_lfnst_enabled_flag is a flag indicating whether or not to use GPM non-separable transform in the case of GPM prediction. When the sps_lfnst_enabled_flag is 1 and the sps_gpm_enabled_flag is 1, the header decoding unit 3020 decodes the sps_gpm_lfnst_enabled_flag. In this example, the sps_gpm_lfnst_enabled_flag is not transmitted when sps_lfnst_enabled_flag==0 or sps_gpm_enabled_flag==0, which has the effect of improving coding efficiency. The sps_gpm_enabled_flag is a flag indicating whether or not the GPM prediction mode is used in the target sequence.

[0068] The TU decoding unit 3024 decodes an index mts_idx indicating a transform base from the coded data. The TU decoding unit 3024 also decodes a parameter lfnst_idx indicating whether or not a non-separable transform is used and the transform base from the coded data. Specifically, the TU decoding unit 3024 decodes lfnst_idx when the width and height of the CU are 4 or more and the prediction mode is an intra prediction mode. Note that lfnst_idx indicates that a non-separable transform is not applied when it is 0, indicates one transform matrix of a set (pair) of transform matrices (transform bases) of the non-separable transform when it is 1, and indicates the other transform matrix of the pair when it is 2.

[0069] The predicted image generation unit 308 includes an inter predicted image generation unit 309 (FIG. 6) and an intra predicted image generation unit 310.

[0070] The prediction parameter derivation unit includes an inter prediction parameter derivation unit 303 (FIG. 5) and an intra prediction parameter derivation unit.

[0071] In the following, an example will be described in which CTUs and CUs are used as processing units, but this is not limiting and processing may be performed in sub-CU units. Alternatively, CTUs and CUs may be read as blocks, and sub-CUs as sub-blocks, and processing may be performed in block or sub-block units.

[0072] The entropy decoding unit 301 performs entropy decoding on the externally input coded stream Te to decode individual codes (syntax elements). Entropy coding can be performed in two ways: one is to perform variable-length coding of syntax elements using a context (probability model) adaptively selected according to the type of syntax element and the surrounding circumstances, and the other is to perform variable-length coding of syntax elements using a predetermined table or formula.

[0073] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code is, for example, a prediction mode predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0074] (Basic flow) FIG. 4 is a flowchart illustrating the general operation of the video decoding device 31.

[0075] (S1100: Decode Parameter Set Information) The header decoder 3020 decodes parameter set information such as VPS, SPS, and PPS from the coded data.

[0076] (S1200: Decode slice information) The header decoding unit 3020 decodes the slice header (slice information) from the coded data.

[0077] Thereafter, the video decoding device 31 repeats the processes from S1300 to S5000 for each CTU included in the target picture, thereby deriving a decoded image of each CTU.

[0078] (S1300: Decode CTU Information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0079] (S1400: Decode CT Information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0080] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

[0081] (S1510: Decode CU Information) The CU decoding unit 3022 decodes CU information, prediction information, the TU split flag split_transform_flag, and CU residual flags cbf_cb, cbf_cr, and cbf_luma from the coded data.

[0082] (S1520: Decode TU information) When a prediction error is included in a TU, the TU decoding unit 3024 decodes the quantized prediction error and the transform index mts_idx from the coded data.

[0083] (S2000: Generate predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on prediction information.

[0084] (S3000: Inverse Quantization and Inverse Transformation) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing on each TU included in the target CU.

[0085] (S4000: Generate decoded image) The addition unit 312 generates a decoded image of the current CU by adding the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization and inverse transform unit 311.

[0086] (S5000: Loop filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

[0087] (Configuration of the intra-prediction parameter derivation unit) The intra prediction parameter derivation unit decodes intra prediction parameters, for example, an intra prediction mode IntraPredMode, by referring to the prediction parameters stored in the prediction parameter memory 307, based on the code input from the entropy decoding unit 301. The intra prediction parameter derivation unit outputs the decoded intra prediction parameters to the predicted image generation unit 308, and also stores them in the prediction parameter memory 307. The intra prediction parameter derivation unit may derive different intra prediction modes for luma and chroma.

[0088] The loop filter 305 is a filter provided in the encoding loop that removes block distortion and ringing distortion to improve image quality. The loop filter 305 applies filters such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to the decoded image of the CU generated by the adder 312.

[0089] The reference picture memory 306 stores the decoded image of the CU generated by the adder 312 in a predetermined location for each current picture and current CU.

[0090] The prediction parameter memory 307 stores prediction parameters at a predetermined location for each CTU or CU to be decoded. Specifically, the prediction parameter memory 307 stores the parameters decoded by the parameter decoding unit 302 and the prediction mode predMode separated by the entropy decoding unit 301.

[0091] The prediction image generation unit 308 receives input of a prediction mode predMode, prediction parameters, etc. The prediction image generation unit 308 also reads a reference picture from the reference picture memory 306. The prediction image generation unit 308 generates a prediction image of a block or sub-block using the prediction parameters and the read reference picture (reference picture block) in the prediction mode indicated by the prediction mode predMode. Here, the reference picture block is a set of pixels on the reference picture (usually rectangular, and therefore referred to as a block), and is an area referenced to generate a prediction image.

[0092] The intra-predicted image generation unit 310 generates a predicted image of the current block by referring to the read decoded pixel values ​​and the prediction mode indicated by IntraPredMode. The intra-predicted image generation unit 310 outputs the generated predicted image of the block to the adder 312.

[0093] (Configuration of the inter-prediction parameter derivation unit) The inter prediction parameter derivation unit 303 derives inter prediction parameters by referring to prediction parameters stored in the prediction parameter memory 307, based on the syntax elements input from the parameter decoding unit 302. The inter prediction parameter derivation unit 303 also outputs the inter prediction parameters to the inter prediction image generation unit 309 and the prediction parameter memory 307. The inter prediction parameter derivation unit 303 and its internal elements, the AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, the GPM prediction unit 3037, and the MV addition unit 3038, are means common to the video encoding device and the video decoding device, and therefore may be collectively referred to as a motion vector derivation unit (motion vector derivation device).

[0094] When GPM_Flag is 1, that is, when it indicates GPM prediction mode, the GPM prediction unit 30377 derives parameters for GPM prediction.

[0095] If merge_flag is 1, that is, if it indicates merge prediction mode, merge_idx is derived and output to the merge prediction parameter derivation unit 3036 .

[0096] When merge_flag is 0, that is, when it indicates the AMVP prediction mode, the AMVP prediction parameter derivation unit 3032 derives mvpLX from inter_pred_idc, refIdxLX, or mvp_LX_idx.

[0097] (MV addition section) The MV adder 3038 adds the derived mvpLX and mvdLX to derive mvLX.

[0098] (Merge prediction) The merge prediction parameter derivation unit 3036 includes a merge candidate derivation unit and a merge candidate selection unit. Merging candidates are configured to include prediction parameters (predFlagLX, mvLX, refIdxLX) and are stored in a merge candidate list. Merge candidates stored in the merge candidate list are assigned indices according to a predetermined rule.

[0099] The merge candidate derivation unit derives merge candidates by directly using the motion vectors and refIdxLX of the decoded adjacent blocks. Alternatively, the merge candidate derivation unit may apply a spatial merge candidate derivation process, a temporal merge candidate derivation process, or the like, which will be described later.

[0100] In the spatial merge candidate derivation process, the merge candidate derivation unit reads prediction parameters stored in the prediction parameter memory 307 according to a predetermined rule and sets them as merge candidates. The reference pictures are specified, for example, by prediction parameters for each of the adjacent blocks within a predetermined range from the target block (e.g., all or some of the blocks adjacent to the left A1, right B1, upper right B0, lower left A0, and upper left B2 of the target block). The merge candidates are referred to as A1, B1, B0, A0, and B2. Here, A1, B1, B0, A0, and B2 are motion information derived from blocks containing the following coordinates:

[0101] A1: (xCb - 1, yCb + cbHeight - 1) B1: (xCb + cbWidth - 1, yCb - 1) B0: (xCb + cbWidth, yCb - 1) A0: (xCb - 1, yCb + cbHeight) B2: (xCb - 1, yCb - 1) The upper left coordinates of the target block are (xCb, yCb), the width is cbWidth, and the height is cbHeight.

[0102] As a temporal merge derivation process, the merge candidate derivation unit reads the prediction parameters of the lower right CBR of the target block or the block C in the reference image including the center coordinates from the prediction parameter memory 307, sets it as a merge candidate Col, and stores it in the merge candidate list mergeCandList[ ].

[0103] The order in which the merge candidates are stored in mergeCandList[] is, for example, spatial merge candidates (B1, A1, B0, A0, B2) and then temporal merge candidates Col. Note that reference blocks that are unavailable (for example, blocks that are intra-predicted) are not stored in the merge candidate list. i = 0 if( availableFlagB1 ) mergeCandList[ i++ ] = B1 if( availableFlagA1 ) mergeCandList[ i++ ] = A1 if( availableFlagB0 ) mergeCandList[ i++ ] = B0 if( availableFlagA0 ) mergeCandList[ i++ ] = A0 if( availableFlagB2 ) mergeCandList[ i++ ] = B2 if( availableFlagCol ) mergeCandList[ i++ ] = Col Furthermore, the history merge candidate HmvpCand, the average candidate avgCand, and the zero merge candidate zeroCandm may be added to mergeCandList[] and used.

[0104] The merge candidate selection unit selects a merge candidate N indicated by merge_idx from among the merge candidates included in the merge candidate list using the following formula.

[0105] N = mergeCandList[merge_idx] Here, N is a label indicating a merge candidate, and can be A1, B1, B0, A0, B2, Col, etc. The motion information of the merge candidate indicated by label N is indicated by (mvLXN[0], mvLXN[0]), predFlagLXN, and refIdxLXN.

[0106] The selected (mvLXN[0], mvLXN[0]), predFlagLXN, and refIdxLXN are selected as the inter prediction parameters of the current block. The merging candidate selection unit stores the inter prediction parameters of the selected merging candidate in the prediction parameter memory 307 and outputs them to the inter prediction image generation unit 309.

[0107] (GPM forecast) GPM prediction will now be described. GPM prediction is a prediction used in GPM mode, and generates a predicted image as two non-rectangular prediction units that divide a target CU by a line segment. An example of a line segment is shown in FIG. 9(a). A line segment that spans the target CU is defined by an angle index angleIdx (angle variable) and a distance index distanceIdx (distance variable) shown in FIG. 9(b). angleIdx indicates the angle φ between a vertical line and the line segment. distanceIdx indicates the distance ρ from the center of the target CU to the line segment. angleIdx is an integer value assigned to the angle of the line segment. For example, when 360 degrees is roughly divided into 32 equal parts, the value shown in FIG. 9(c) is assigned. Note that in this example, as mentioned "roughly," 360 degrees is not divided into perfect equal parts, but is divided into 32 using a square scale (coordinates). Note that a configuration in which 360 degrees is divided into 64 and even finer angle indexes are sent may also be used.

[0108] In generating a predicted image for GPM prediction, two "rectangular" predicted images (temporary predicted images) including a non-rectangular prediction unit may be derived, and the two rectangular regions may be derived by weighting according to the shape of the non-rectangular prediction unit. The motion compensation unit 3091 derives two temporary predicted images for the target CU, and the GPM synthesis unit 30952 derives a predicted image by applying weighting processing to each pixel of the two temporary predicted images according to the pixel's position. This processing is called GPM synthesis processing. Processing other than prediction (e.g., transformation (inverse transformation) and quantization (inverse quantization)) is applied to the entire target CU.

[0109] The GPM prediction unit 30377 derives prediction parameters for the two non-rectangular regions and supplies them to the inter-prediction image generation unit 309. To simplify the processing, GPM prediction may be configured not to use bi-prediction. In this case, unidirectional inter-prediction parameters are derived for the non-rectangular regions.

[0110] (Decoding syntax in GPM prediction) Whether GPM prediction is on or off and the parameters when GPM prediction is on are signaled in the coded data as follows:

[0111] As shown in FIG. 10(a), sps_gpm_enabled_flag is notified by SPS and indicates whether or not the GPM prediction mode is used in the target sequence. When sps_gpm_enabled_flag is 0, it indicates that the GPM prediction mode is not used in the target sequence. When sps_gpm_enabled_flag is 1, it indicates that the GPM prediction mode is used in the target sequence. Here, it indicates that the GPM prediction mode in which the number of selectable partition patterns (partition types) is NumGPMFull (e.g., 64) is used in the target sequence. Note that sps_gpm_enabled_flag is not limited to SPS, and may be transmitted in PPS, picture header, or slice header.

[0112] 10(b) shows the syntax configuration notified in merge mode. The parameter decoding unit 302 decodes syntax elements in the encoded data, and the GPM prediction unit 30377 (inter-prediction parameter derivation unit 303) derives GPM prediction parameters according to the following rules:

[0113] In the SPS, sps_max_num_merge_cand_minus_max_num_gpm_cand is notified. sps_max_num_merge_cand_minus_max_num_gpm_cand is a parameter used to derive MaxNumGpmMergeCand, the maximum number of merge candidates for GPM prediction. The GPM prediction unit 30377 derives merge candidates MergeCand ranging from 0 to MaxNumMergeCand-1, the maximum number of merge prediction candidates.

[0114] If sps_gpm_enabled_flag is on and MaxNumMergeCand is 3 or greater, MaxNumGpmMergeCand is calculated using the following formula:

[0115] MaxNumGpmMergeCand = MaxNumMergeCand - sps_max_num_merge_cand_minus_max_num_gpm_cand If sps_gpm_enabled_flag is on and MaxNumMergeCand is 2, MaxNumGpmMergeCand is derived using the following formula:

[0116] MaxNumGpmMergeCand = 2 If none of the above applies, MaxNumGpmMergeCand is calculated using the following formula:

[0117] MaxNumGpmMergeCand = 0 When MaxNumGpmMergeCand is 0, GPM prediction is disabled.

[0118] general_merge_flag is a flag indicating whether to use merge mode, which is notified when the current block is not in skip mode. In skip mode, the inter prediction parameter derivation unit 303 sets general_merge_flag=1. merge_data() is a syntax structure for notifying parameters of merge prediction.

[0119] When ciip_flag is 0, the GPM prediction syntax elements merge_gpm_partition_idx, merge_gpm_idx0, and merge_gpm_idx1 are notified. merge_gpm_partition_idx is an index (partition index) indicating the division pattern of the GPM prediction mode. The partition index indicates a combination of angleIdx and distanceIdx, which identify a line segment spanning the target block to divide the target block into two non-rectangular regions. Merge candidates are used as motion information used to generate a predicted image for GPM prediction. merge_gpm_idx0 and merge_gpm_idx1 are merge candidate indices that indicate the motion information of the two non-rectangular regions, respectively. The number of partition index options (number of division patterns) is NumGPMFull, and merge_gpm_partition_idx takes an integer value from 0 to NumGPMFull-1 (=63).

[0120] MergeGpmFlag is a flag that indicates whether or not to perform GPM prediction on the target block. If all of the following conditions (GPM determination conditions) are met, the GPM prediction unit 30377 sets MergeGpmFlag=1 (GPM prediction on), and if not, the GPM prediction unit 30377 sets MergeGpmFlag=0. sps_gpm_enabled_flag=1 slice_type is B slice general_merge_flag=1 cbWidth>=8 and cbHeight>=8 cbWidth<8*cbHeight and cbHeight<8*cbWidth regular_merge_flag=0 merge_subblock_flag=0 ciip_flag=0 When MergeGpmFlag=1, the GPM prediction unit 30377 derives parameters necessary for generating a predicted image in the following procedure and outputs them to the GPM synthesis unit 30952.

[0121] The parameter decoding unit 302 (CU decoding unit 3022) decodes merge_gpm_partition_idx, merge_gpm_idx0, and merge_gpm_idx1. The parameter decoding unit 302 (CU decoding unit 3022) may decode merge_gpm_partition_idx and merge_gpm_idx0 in the case of !ciip_flag, and may further decode merge_gpm_idx1 in the case of MaxNumGpmMergeCand>2.

[0122] (Movement information derivation process in GPM prediction) The GPM predictor 30377 derives merge indexes m and n from the syntax elements merge_gpm_idx0 and merge_gpm_idx1 that indicate the motion information of two non-rectangular regions, as follows:

[0123] m = merge_gpm_idx0 n = merge_gpm_idx1 + (merge_gpm_idx1 >= m) ? 1 : 0 In the following, the merge candidate pointed to by merge index m is denoted as M, and the merge candidate pointed to by merge index n is denoted as N.

[0124] The merge prediction parameter derivation unit 3036 derives motion information (mvLXM, mvLXN, refIdxLXM, refIdxLXN, predFlagLXM, predFlagLXN, bcwIdx, mergeCandList, etc.) of merge candidates M and N using the method described in (Merge Prediction). The GPM prediction unit 30377 uses this motion information to set motion vectors mvA and mvB, reference indices refIdxA and refIdxB, and prediction list flags predListFlagA and predListFlagB of merge_gpm_idx0 and merge_gpm_idx1.

[0125] These motion information are referenced to generate temporal prediction images for the two non-rectangular regions.

[0126] The GPM prediction unit 30377 derives angleIdx and distanceIdx corresponding to merge_gpm_partition_idx according to the table shown in FIG. 11 and outputs them to the GPM synthesis unit 30952.

[0127] The GPM prediction unit 30377 uses the motion information (mvX, refIdxX, predListFlagX, X=A or B) to derive temporary predicted images predSamplesLA and predSamplesLB and outputs them to the GPM synthesis unit 30952. The GPM synthesis unit 30952 derives a predicted image using weighting parameters derived from angle variables. More specifically, it generates a predicted image using weighting information derived using angleIdx and distanceIdx and the temporary predicted images predSamplesLA and predSamplesLB.

[0128] (Weighted pixel prediction process in GPM prediction) The GPM synthesis unit 30952 derives the predicted pixel value pbSample in the following procedure, where nCbW = cbWidth and nCbH = cbHeight. First, the GPM synthesis unit 30952 derives bitDepth as follows for the weighting coefficient derivation process in GPM prediction and the motion vector storage process described later.

[0129] If cIdx is 0, the GPM synthesis unit 30952 sets bitDepth to the luminance pixel bit count BitDepthY.

[0130] If cIdx is 0, the GPM synthesis unit 30952 sets nW and nH to nCbW and nCbH, respectively. If cIdx is not 0, the GPM synthesis unit 30952 sets nW and nH to nCbW*SubWidthC and nCbH*SubHeightC, respectively, where SubWidthC and SubHeightC are values ​​determined in advance according to the chrominance format.

[0131] If cIdx is 0, the GPM synthesis unit 30952 sets both subW and subH to 1. If cIdx is not 0, the GPM synthesis unit 30952 sets subW and subH to SubWidthC and SubHeightC, respectively.

[0132] If cIdx is not 0, the GPM synthesis unit 30952 sets bitDepth to the number of color difference pixel bits, BitDepthC.

[0133] Next, the GPM synthesis unit 30952 derives the variables nW, nH, shift1, offset1, displacementX, displacementY, partFlip, and shiftHor as follows:

[0134] nW = (cIdx==0) ? nCbW : nCbW * SubWidthC nH = (cIdx==0) ? nCbH : nCbH * SubHeightC shift1 = Max(5, 17 - BitDepth) offset1 = 1 << (shift1 - 1) displacementX = angleIdx displacementY = (angleIdx + 8) % 32 partFlip = (angleIdx>=13 && angleIdx<=27) ? 0 : 1 shiftHor = (angleIdx%16==8 || (angleIdx%16!=0 && nH>=nW)) ? 0 : 1 When shiftHor is 0, the GPM synthesis unit 30952 derives offsetX and offsetY as follows:

[0135] offsetX = (-nW) >> 1 offsetY = ((-nH) >> 1) + (angleIdx<16? (distanceIdx * nH) >> 3 : -((distanceIdx * nH) >> 3)) When shiftHor is 1, the GPM synthesis unit 30952 derives offsetX and offsetY as follows:

[0136] offsetX = ((-nW) >> 1) + (angleIdx<16? (distanceIdx * nW) >> 3 : -((distanceIdx * nW) >> 3)) offsetY = (-nH) >> 1 The GPM synthesis unit 30952 derives the predicted pixels pbSamples according to the following steps.

[0137] 1) The GPM synthesis unit 30952 derives generalized coordinates (xL, yL) from coordinates (x, y) within the CU.

[0138] xL = (cIdx==0) ? x : x * SubWidthC yL = (cIdx==0) ? y : y * SubHeightC 2) The GPM synthesis unit 30952 uses disLut shown in FIG. 12 to calculate the weighting coefficient wValue according to the coordinates within the CU as follows:

[0139] weightIdx = (((xL + offsetX) << 1) + 1) * disLut[displacementX] + (((yL + offsetY) << 1) + 1) * disLut[displacementY] weightIdxL = partFlip? 32 + weightIdx : 32 - weightIdx wValue = Clip3(0, 8, (weightIdxL + 4) >> 3) 3) The GPM synthesis unit 30952 derives the value of pbSample as follows:

[0140] pbSamples[x][y] = Clip3(0, (1 << BitDepth) - 1, (predSamplesLA[x][y] * wValue + predSamplesLB[x][y] * (8 - wValue) + offset1) >> shift1) (Motion vector storage process in GPM prediction) The GPM prediction unit 30377 stores the motion vectors (mvA, mvB) of non-rectangular areas A and B and reference picture information (predFlagA, prefFlagB, refIdxLA, refIdxLB) in memory in 4*4 sub-block units so that they can be referenced in subsequent processing.

[0141] (GPM prediction processing flow) 13 is a flowchart showing the flow of GPM prediction processing. The following describes the flow of GPM prediction processing on the assumption that all the conditions for setting MergeGpmFlag=1 (GPM prediction on) as described above are met.

[0142] In S3501, the parameter decoding unit 302 appropriately decodes various syntax elements notified by the SPS, PPS, slice header, merge data, etc., as shown in Fig. 4, for example, and outputs these syntax elements to the inter prediction parameter derivation unit 303 (the merge prediction parameter derivation unit 3036, the GPM prediction unit 30377, etc.).

[0143] In S3502, the GPM prediction unit 30377 determines whether MergeGpmFlag is 1 or not.

[0144] If MergeGpmFlag is not 1 in S3502, the GPM prediction unit 30377 turns off GPM prediction in S3506, and the process ends.

[0145] In S3502, if MergeGpmFlag is 1, the GPM prediction unit 30377 performs GPM prediction processing in S3503 to S3505.

[0146] In S3503, the GPM prediction unit 30377 derives motion information and generates two more temporal prediction images predSamplesLA and predSamplesLB.

[0147] In S3504, the GPM prediction unit 30377 derives the weighting coefficient wValue, and the GPM synthesis unit 30952 generates a predicted image.

[0148] In S3505, the GPM prediction unit 30377 stores the motion vector in memory.

[0149] (AMVP forecast) The AMVP prediction parameter derivation unit 3032 includes a vector candidate derivation unit and a vector candidate selection unit. The vector candidate derivation unit derives prediction vector candidates from the motion vectors of decoded adjacent blocks stored in the prediction parameter memory 307 based on refIdxLX, and stores the candidates in a prediction vector candidate list mvpListLX[ ].

[0150] The vector candidate selection unit selects, as mvpLX, the motion vector mvpListLX[mvp_LX_idx] indicated by mvp_LX_idx from among the predicted vector candidates in mvpListLX[ ]. The vector candidate selection unit outputs the selected mvpLX to the MV addition unit 3038.

[0151] (MV addition section) The MV addition unit 3038 calculates mvLX by adding the mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded mvdLX. The addition unit 3038 outputs the calculated mvLX to the inter predicted image generation unit 309 and the prediction parameter memory 307.

[0152] mvLX[0] = mvpLX[0]+mvdLX[0] mvLX[1] = mvpLX[1] + mvdLX[1] (Inter-prediction image generation unit 309) When predMode indicates inter prediction mode, the inter prediction image generation unit 309 generates a prediction image of a block or sub-block by inter prediction using the inter prediction parameters input from the inter prediction parameter derivation unit 303 and the reference picture.

[0153] 6 is a schematic diagram showing the configuration of an inter-prediction image generation unit 309 included in the prediction image generation unit 308 according to this embodiment. The inter-prediction image generation unit 309 includes a motion compensation unit (prediction image generation device) 3091 and a synthesis unit 3095. The synthesis unit 3095 includes an intra-inter synthesis unit 30951, a GPM synthesis unit 30952, a BIO unit 30954, and a weight prediction unit 3094.

[0154] (motion compensation) The motion compensation unit 3091 (interpolated image generation unit 3091) generates an interpolated image (motion-compensated image) by reading a reference block from the reference picture memory 306 based on the inter-prediction parameters (predFlagLX, refIdxLX, mvLX) input from the inter-prediction parameter derivation unit 303. The reference block is a block located at a position shifted by mvLX from the position of the current block on the reference picture RefPicLX specified by refIdxLX. Here, if mvLX does not have integer precision, an interpolated image is generated by applying a filter called a motion compensation filter for generating pixels at decimal positions.

[0155] The motion compensation unit 3091 first derives the integer position (xInt, yInt) and phase (xFrac, yFrac) corresponding to the coordinates (x, y) in the prediction block using the following equations.

[0156] xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x xFrac = mvLX[0]&(MVPREC-1) yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y yFrac = mvLX[1]&(MVPREC-1) Here, (xPb, yPb) are the upper left coordinates of a bW*bH size block, where x=0...bW-1 and y=0...bH-1, and MVPREC indicates the accuracy of mvLX (1 / MVPREC pixel accuracy), e.g., MVPREC=16.

[0157] The motion compensation unit 3091 derives the temporary image temp[][] by performing horizontal interpolation on the reference picture refImg using an interpolation filter. In the following, Σ is the sum over k, where k=0..NTAP-1, shift1 is a normalization parameter that adjusts the value range, and offset1=1<<(shift1-1).

[0158] temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1 Next, the motion compensation unit 3091 derives the interpolated image Pred[][] by vertically interpolating the temporary image temp[][]. In the following, Σ is the sum for k=0..NTAP-1, shift2 is a normalization parameter that adjusts the value range, and offset2=1<<(shift2-1).

[0159] Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2 In the case of bi-prediction, the above Pred[][] is derived for each L0 list and L1 list (called interpolated images PredL0[][] and PredL1[][]), and the interpolated image Pred[][] is generated from PredL0[][] and PredL1[][].

[0160] (IntraInter synthesis processing) When ciip_mode is 1, the IntraInter synthesis unit 30951 generates a predicted image in CIIP (Combined intra prediction) mode by weighting the inter predicted image and the intra predicted image.

[0161] In CIIP mode, the predicted value for CIIP mode is calculated by taking the weighted average of the inter-predicted image derived in merge mode and the intra-predicted image derived using planar prediction. The weight of the combination is determined by the prediction mode of the adjacent coding block. If intra-prediction is used in both of the two adjacent blocks (above and to the left) of the current block, the combination ratio of inter-prediction to intra-prediction is 1:3. On the other hand, if neither of the two adjacent blocks is intra-prediction, the combination ratio of inter-prediction to intra-prediction is 3:1. In all other cases, the ratio is 2:2. The same weighting factor is applied to both luma and chroma.

[0162] (BIO forecast) Next, we will explain the details of BIO prediction (Bi-Directional Optical Flow, BDOF processing) performed by the BIO unit 30954. In bi-prediction mode, the BIO unit 30954 generates a predicted image by referring to two predicted images (a first predicted image and a second predicted image) and a gradient correction term.

[0163] (Weight prediction) The weighted prediction unit 3094 generates a predicted image of the block by multiplying the interpolated image PredLX by a weighting coefficient. When one of the prediction list usage flags (predFlagL0 or predFlagL1) is 1 (uni-prediction) and weighted prediction is not used, the weighted prediction unit 3094 performs the following equation processing to adjust PredLX (LX is L0 or L1) to the pixel bit depth bitDepth.

[0164] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredLX[x][y]+offset1)> >shift1) Here, shift1=14-bitDepth, offset1=1<<(shift1-1). Furthermore, when both prediction list usage flags (predFlagL0 and predFlagL1) are 1 (bi-prediction PRED_BI) and weighted prediction is not used, the following equation is processed to average PredL0 and PredL1 and adjust them to the pixel bit rate.

[0165] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredL0[x][y]+PredL1[x][y]+offset2)> >shift2) Here, shift2=15-bitDepth, offset2=1<<(shift2-1).

[0166] Furthermore, when uni-prediction and weighted prediction are performed, the weighted prediction unit 3094 derives a weighted prediction coefficient w0 and an offset o0 from the coded data, and performs processing according to the following equations.

[0167] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,((PredLX[x][y]*w0+2^(log2WD-1))> >log2WD)+o0) Here, log2WD is a variable indicating a predetermined shift amount.

[0168] Furthermore, when bi-prediction PRED_BI and weighted prediction are performed, the weighted prediction unit 3094 derives weighted prediction coefficients w0, w1, o0, and o1 from the coded data and performs the processing of the following equations.

[0169] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,(PredL0[x][y]*w0+PredL1[x][y]*w1+((o0+o1+1)<<log2WD))> >(log2WD+1)) The inter-prediction image generation unit 309 outputs the generated prediction image of the block to the addition unit 312 .

[0170] (Intra-prediction image generation unit) When predMode indicates an intra prediction mode, the intra prediction image generation unit performs intra prediction using the intra prediction parameters input from the intra prediction parameter derivation unit and reference pixels read from the reference picture memory 306 .

[0171] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0172] (inverse quantization and inverse transformation) 20 is a block diagram showing the configuration of the inverse quantization and inverse transform unit 311 of this embodiment. The inverse quantization and inverse transform unit 311 is made up of a scaling unit 31111, an inverse non-separable transform unit 31121, and an inverse core transform unit 31123.

[0173] The inverse quantization and inverse transform unit 311 calculates transform coefficients d[][] by scaling (inverse quantizing) the quantized transform coefficients qd[][] input from the entropy decoding unit 301 using a scaling unit 31111. These quantized transform coefficients qd[][] are coefficients obtained by performing a transform such as a DCT (Discrete Cosine Transform) or a DST (Discrete Sine Transform) on prediction errors during encoding processing and then quantizing them, or coefficients obtained by further performing a non-separable transform on the transformed coefficients. If lfnst_idx!=0, the inverse quantization and inverse transform unit 311 performs an inverse transform using an inverse non-separable transform unit 31121. Furthermore, the transform coefficients are subjected to an inverse frequency transform such as an inverse DCT or an inverse DST to calculate prediction errors. Furthermore, when lfnst_idx==0, the inverse non-separable transform unit 31121 is not performed, and the transform coefficients scaled by the scaling unit 31111 are subjected to inverse transform such as inverse DCT and inverse DST to calculate a prediction error. The inverse quantization and inverse transform unit 311 outputs the prediction error to the adder 312.

[0174] Note that, since the inverse transform and the transform are paired processes, the transform and the inverse transform may be interpreted interchangeably. Alternatively, when the inverse transform is called the transform, the transform may be called the forward transform. For example, when the inverse non-separable transform is called the non-separable transform, the non-separable transform may be called the forward non-separable transform. Furthermore, the core transform is simply called the transform.

[0175] d[x][y] is transmitted to the inverse core transform unit 31123 or the inverse non-separable transform unit 31121. The inverse non-separable transform unit 31121 applies an inverse non-separable transform to the transform coefficients d[][] after inverse quantization and before the core transform.

[0176] (non-separable transformation) The inverse non-separable transform is applied to the transform coefficients of a part or the entire area of a TU in the moving image decoding device 31. After the inverse non-separable transform is applied, an inverse separable transform (such as DCT2 and DST7) is applied to the transform coefficients after the inverse non-separable transform. Also, when a TU is divided into 4x4 sub-blocks, the non-separable transform and the inverse non-separable transform are applied only to a predetermined sub-block in the upper left corner. Examples of the size of a TU where one of the width W and height H is 4 include 4×4, 8×4, 4×8, L×4, and 4×L (L is a natural number of 16 or more).

[0177] Also, a technique for transmitting only some low-frequency components among the transform coefficients after the separable transform is called RST (Reduced Secondary Transform) transform or LFNST (Low Frequency Non-Separable-Transform). Specifically, when the number nonZeroSize of the transform coefficients of the non-separable transform to be transmitted is less than or equal to the size of the separable transform ((1<<log2StSize)x(1<<log2StSize)), it is called LFNST. In this specification, the transform applied to the error in the case of GPM prediction is called GPM non-separable transform (or GPM_LFNST). Hereinafter, the processing of the inverse non-separable transform will be described in order.

[0178] (S2000: Decoding of non-separable transform index) FIG. 22 and FIG. 23 are examples of syntax tables showing flags for non-separable transforms. FIG. 22 shows an example of notifying lfnst_idx and mtx_idx in a CU, and FIG. 23 shows an example of notifying lfnst_idx and mtx_idx in a TU.

[0179] The TU decoding unit 3024 decodes an index mts_idx indicating the transform matrix of the separable transform from the encoded data. It may be configured to decode mts_idx after lfnst_idx and decode mtx_idx only when lfnst_idx is 0. That is, a configuration may be used where a transform matrix other than DCT2 indicated by mtx_idx!=0 is used only when the non-separable transform is not used (lfnst_idx is 0).

[0180] The TU decoding unit 3024 also decodes the index lfnst_idx from the coded data. lfnst_idx is an index indicating whether a non-separable transform is used and the transform matrix. The TU decoding unit 3024 derives the flag LfnstDcOnly and the flag LfnstZeroOutSigCoeffFlag. LfnstDcOnly is a flag indicating whether the transform coefficients are DC only, and LfnstZeroOutSigCoeffFlag is a flag indicating whether a transform coefficient exists in a predetermined high-frequency region (zero-out region). The TU decoding unit 3024 decodes lfnst_idx when LfnstDcOnly==0 and LfnstZeroOutSigCoeffFlag==1. Here, LfnstDcOnly==0 indicates that there are transform coefficients other than DC coefficients. LfnstZeroOutSigCoeffFlag==1 indicates that there are no non-zero transform coefficients in the zero-out region. Before decoding the residual of a TU, set LfnstDcOnly=1 and LfnstZeroOutSigCoeffFlag=1. If the position of the last coefficient is not DC (lastSubBlock==0 && lastScanPos>0), set LfnstDcOnly=0. If the last position is in the high-frequency region, set LfnstZeroOutSigCoeffFlag=0. The last position is in the high-frequency region when, for example, (lastScanPos>7 && (log2TbWidth==2 || log2TbWidth==3) is satisfied. If lfnst_idx is not to be decoded, set lfnst_idx=0. Note that, as shown in the figure, lfnst_idx may be decoded when the intra subpartition tool is on (IntraSubPartitionsSplitType!=ISP_NO_SPLIT) or (LfnstDcOnly==0 and LfnstZeroOutSigCoeffFlag==1). Furthermore, when decoding lfnst_idx, the width and height of the CU may be limited to 4 or more, or MIP prediction in small blocks may be excluded (block size may be limited to a certain size or more).

[0181] The TU decoding unit 3024 decodes lfnst_idx when the prediction mode is intra prediction mode and sps_lfnst_enabled_flag is 1 (SYN_LFNST_INTRA), or when the prediction mode is inter and sps_gpm_lfnst_enabled_flag is 1 (SYN_GPM_INTRA). Note that when lfnst_idx is 0, it indicates that a non-separable transform is not applied, when it is 1, it indicates that one transform of a set (pair) of non-separable transform matrices is used, and when it is 2, it indicates that the other transform of the pair is used.

[0182] Also, when the prediction mode is inter and a GPM non-separable transform is used, that is, when applying a GPM non-separable transform to an inter CU, only one of the non-separable transform matrix pairs is used. The value of lfnst_idx can be limited to {0,1} or {0,2}, which reduces the cost of the flags required when applying a GPM non-separable transform to an inter CU.

[0183] Specifically, when decoding a syntax element indicating whether or not non-separable transform is used, the TU decoding unit 3024 decodes lfnst_idx consisting of three values ​​{0, 1, 2} or more if the prediction mode is intra prediction. Otherwise (if the prediction mode is inter prediction, GPM prediction in this case), lfnst_flag of two values ​​{0, 1} may be decoded. In the following processing, lfnst_idx = lfnst_flag is treated as being equal. Different contexts may be used for lfnst_idx and lfnst_flag, which are switched depending on the prediction mode.

[0184] Alternatively, the TU decoding unit 3024 may first decode lfnst_flag, and then, if the prediction mode is intra prediction, decode lfnst_set_flag. lfnst_set_flag is a flag or index indicating the type of non-separable transform. Otherwise (if the prediction mode is inter prediction, GPM prediction in this case), lfnst_set_flag is not decoded from the encoded data, and lfnst_set_flag=0 is assumed. In the following processing, lfnst_idx=lfnst_flag+lfnst_set_flag is treated as follows.

[0185] Furthermore, the TU decoding unit 3024 decodes lfnst_idx regardless of the prediction mode, but the binarization may be changed depending on the prediction mode. For example, when the prediction mode is intra prediction, a truncated rice code (TR code) with a maximum value cmax=2 and a rice parameter cRiceParam=0 may be used as binarization to decode lfnst_idx within a value range from 0 to 2. When the prediction mode is inter prediction (GPM prediction in this case), a TR code with a maximum value cmax=1 and a rice parameter cRiceParam=0 may be used as binarization to decode lfnst_idx within a value range from 0 to 1. Furthermore, a truncated unary code may be used instead of a TR code.

[0186] Furthermore, the context index cIdx (or context index difference ctxInc) of the above lfnst_idx, lfnst_flag, and lfnst_set_flag may be changed depending on the prediction mode. cIdx is an index used to store probability parameters used in arithmetic coding. For example, it is derived as ctxIdx (or ctxInc) = (CuPredMode[chType][xTb][yTb]==MODE_INTRA) ? 0 : 1.

[0187] (Deriving the transformation matrix) The inverse non-separable transform unit 31121 derives a transform matrix secTransMatrix[][] used in the transform process.

[0188] Specifically, when the prediction mode is intra prediction, the inverse non-separable transform unit 31121 derives lfnstTrSetId from the intra prediction mode IntraPredMode. When the prediction mode is inter prediction, lfnstTrSetId is derived from the block size or the GPM angle index angleIdx (or the GPM mode number merge_gpm_partition_idx). lfnstTrSetId is a number indicating a set of transform matrices for non-separable transform. Note that the transform matrix (or set of transform matrices) corresponding to lfnstTrSetId=K will be referred to as MK hereinafter.

[0189] The non-separable transform can use a plurality of predetermined transform matrices, and suitable transform can be performed by selecting an appropriate transform matrix based on the intra prediction mode or GPM mode.

[0190] Fig. 14 shows a table used for a transform matrix for non-separable transform in intra prediction mode. The inverse non-separable transform unit 31121 derives a matrix for non-separable transform, here a set number (lfnstTrSetId) of a transform matrix, from the intra prediction mode IntraPredMode using the table in Fig. 14. When the table is an array of nstSetTableIntra, lfnstTrSetId may be derived using the following lookup table.

[0191] lfnstTrSetIdx = nstSetTableIntra[IntraPredMode] nstSetTableIntra[] = {2, 2, 3, 3, 3, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 2, 2, 2,, 3, 3, 3, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 2} The following example describes the process for a lookup table.

[0192] 15 is a table used to derive a transform matrix for non-separable transform in GPM mode. When the prediction mode is inter prediction and GPM prediction, the inverse non-separable transform unit 31121 derives a set number (lfnstTrSetIdx) for non-separable transform from the angle index angleIdx using the table nstSetTableGPMangle1.

[0193] lfnstTrSetIdx = nstSetTableGPMangle1[angleIdx] nstSetTableGPMangle1[] = {2, 2, 3, 3, 3, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 2, 2, 2,, 3, 3, 3, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 2} In other words, in GPM, the range A is 33.75, equivalent to three 11.25-degree angles, and the range B is 11.25*5=56.25 degrees, and the units are assigned as A, B, A, B, A, B, A, B (in terms of matrices, M2, M3, M2, M1, M2, M3, M2, M1). 11.25 degrees is an angle obtained by dividing 360 degrees into 32. The range A is used around the horizontal and vertical (angleIdx = 0, 8, 16, 24) and uses the same transformation matrix M2. The range B is a different transformation matrix for each range shifted by 90 degrees. Here, M3 and M1 are used. The same transformation matrix is ​​used for the range shifted by 180 degrees (angles shifted by 180 degrees). In other words, if the absolute value difference of angleIdx is a difference meaning a difference of 180 degrees (16 in this case), the same value is used for lfnstTrSetIdx.

[0194] 16 shows another example of a table for deriving lfnstTrSetIdx from the angle index angleIdx in GPM mode. lfnstTrSetId is assigned for each of four angleIdx values, i.e., a range of 45 degrees, which is a combination of four 11.25-degree angles. nstSetTableGPMangle2[] = {0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1} Furthermore, the start position of the unit is not limited to angleIdx=0, but may be the following table in which the start position is shifted. nstSetTableGPMangle2[] = {0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0} nstSetTableGPMangle2[] = {0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0} nstSetTableGPMangle2[] = {0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0} In the above example, the transformation matrix is ​​the same when the angle is shifted by 90 degrees (the difference in angleIdx is 8).

[0195] As shown in another example in Figure 16, lfnstTrSetId is assigned for two angleIdx values, that is, a range of 22.5 degrees, which is the sum of two 11.25 degree angles. In this case, the transformation matrix is ​​the same even if the angle is shifted by 90 degrees. nstSetTableGPMangle3[] ={0, 0, 1, 1, 2, 2, 3, 3, 0, 0, 1, 1, 2, 2, 3, 3. 0, 0, 1, 1, 2, 2, 3, 3, 0, 0, 1, 1, 2, 2, 3, 3} Alternatively, the following table with shifted positions may be used. nstSetTableGPMangle3[] = {0, 1, 1, 2, 2, 3, 3, 0, 0, 1, 1, 2, 2, 3, 3. 0, 0, 1, 1, 2, 2, 3, 3, 0, 0, 1, 1, 2, 2, 3, 3, 0} In the above example, the transformation matrix is ​​the same when the angle is shifted by 90 degrees (the difference in angleIdx is 8).

[0196] 17 shows another example of a table for deriving lfnstTrSetIdx from the angle index angleIdx in GPM mode. lfnstTrSetId is assigned for each of four angleIdx values, i.e., a range of 45 degrees, which is the sum of four 11.25-degree angles. nstSetTableGPMangle4[] = {0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3} Furthermore, the start position of the unit is not limited to angleIdx=0, but may be the following table in which the start position is shifted. nstSetTableGPMangle4[] = {0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0} nstSetTableGPMangle4[] = {0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0} nstSetTableGPMangle4[] = {0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0, 0} In the above example, the transformation matrix is ​​the same when the angle is shifted by 180 degrees (the difference in angleIdx is 16).

[0197] As shown in another example in FIG. 17, lfnstTrSetId may be allocated in units of two angleIdx values, that is, a range of 22.5 degrees, which is the sum of two angles of 11.25 degrees. nstSetTableGPMangle5[] = {0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7. 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7} Alternatively, the following table may be used with the positions shifted: nstSetTableGPMangle5[] = {0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7. 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 0} In the above example, the transformation matrix is ​​the same when the angle is shifted by 180 degrees (the difference in angleIdx is 16).

[0198] 18 shows a table used to derive a transform matrix for non-separable transform in GPM mode. When the prediction mode is inter prediction and GPM prediction, the inverse non-separable transform unit 31121 derives a set number (lfnstTrSetId) for non-separable transform from GPM partition information merge_gpm_partition_idx using the table nstSetTableGPMangleS. Here, 0 to 3 are used as lfnstTrSetId for intra prediction, and 4 to 6 are used as lfnstTrSetId for inter prediction (GPM in this case). nstSetTableGPMangleS[] = {5, 5, 6, 6, 6, 6, 6, 5, 5, 5, 4, 4, 4, 4, 5, 5, 5, 6,6, 6, 6, 6, 6, 5, 5, 5, 4, 4, 4, 4, 5, 5} The above configuration has the advantage of enabling efficient coding by using different transform matrices depending on the mode number of intra prediction or the angle index of inter prediction.

[0199] 19 is a table used to derive a transform matrix for non-separable transform in GPM mode. When the prediction mode is inter prediction and GPM prediction, the inverse non-separable transform unit 31121 derives a set number (lfnstTrSetId) for non-separable transform from GPM partition information merge_gpm_partition_idx using the table nstSetTableGPMpart.

[0200] lfnstTrSetId = nstSetTableGPMpart[merge_gpm_partition_idx] The inverse non-separable transform unit 31121 derives the transform matrix secTranMatrix[][] from the set number of the non-separable transform (lfnstTrSetId), lfnst_idx indicating the transform matrix of the non-separable transform, and the non-separable transform size nStSize(nTrS).

[0201] The predicted image generated by the GPM has a directionality indicated by the GPM's angleIdx, and the same directionality may remain in the GPM's residual. With the above configuration, when using the GPM mode, a non-separable transform that utilizes the directionality of the GPM's division pattern can further concentrate the transform coefficients of the residual in the low-frequency region, thereby improving coding efficiency.

[0202] (S2201: Setting conversion size and input / output size) In an inverse non-separable transform, the size of the inverse non-separable transform (4x4 or 8x8), the number of output transform coefficients, the number of transform coefficients to be applied, and the number of sub-blocks to which the inverse non-separable transform is applied are derived according to the size of the TU (width W, height H). The sizes of a 4x4 and 8x8 inverse non-separable transform are denoted by nStSize=4 and 8, respectively. The sizes of a 4x4 and 8x8 inverse non-separable transform may also be referred to as RST4x4 and RST8x8, respectively. Hereinafter, nStOutSize represents the number of output transform coefficients, nonZeroSize represents the number of transform coefficients to be applied (input transform coefficients), and numStX and numStY represent the number of sub-blocks to which the inverse non-separable transform is applied.

[0203] In the inverse non-separable transform, if the TU is equal to or larger than a predetermined size, 48 transform coefficients are output through an RST8x8 inverse non-separable transform. Otherwise, 16 transform coefficients are output through an RST4x4 inverse non-separable transform. If the TU is 4x4, 16 transform coefficients are derived from 8 transform coefficients using RST4x4, and if the TU is 8x8, 48 transform coefficients are derived from 8 transform coefficients using RST8x8. Otherwise, 16 or 48 transform coefficients are output from 16 transform coefficients depending on the TU size.

[0204] If W and H are both 8 or more, log2StSize = 3, nStOutSize = 48 Otherwise, log2StSize = 2, nStOutSize=16 nStSize = 1< <log2StSize If W and H are both 4, or 8x8, nonZeroSize = 8 Otherwise, nonZeroSize = 16 The input nonZeroSize of LFNST is not limited to 8 and 16. For example, it may be 12. The output nStOutSize is also not limited to 16 and 48. It may be 32, 36, 64, or the like.

[0205] numStX = (nTbH == 4 && nTbW > 8) ? 2 : 1 numStY = (nTbW == 4 && nTbH > 8) ? 2 : 1 It is also possible to always set numStX=numStY without performing non-separable transformation on multiple sub-blocks.

[0206] (S2202: Rearrange into a one-dimensional array) The inverse non-separable transform unit 31121 first rearranges some of the transform coefficients d[][] of the TU into a one-dimensional array u[] and processes it. Specifically, the inverse non-separable transform unit 31121 derives u[] from the two-dimensional transform coefficients d[][] of the target TU indicated by the region RU by referencing the transform coefficients x = 0..nonZeroSize-1. xC and yC are positions on the TU and are derived from the array DiagScanOrder indicating the scan order and the position x of the array u[].

[0207] xC = (xSbIdx< <log2StSize) + DiagScanOrder[log2StSize][log2StSize][x][0] yC = (ySbIdx< <log2StSize) + DiagScanOrder[log2StSize][log2StSize][x][1] u[x] = d[xC][yC] The range copied to the one-dimensional array is called the area RU.

[0208] (S2203: Applying conversion processing) The inverse non-separable transform unit 31121 performs a transform using the transform matrix secTransMatrix[][] on u[] of length nonZeroSize, and derives coefficients v'[] of a one-dimensional array of length nStOutSize as output.

[0209] Furthermore, the inverse non-separable transform unit 31121 performs a multiply-and-accumulate operation between the transform matrix and the one-dimensional variable u[], as shown in the following equation.

[0210] v[i] = Clip3(CoeffMin, CoeffMax,Σ(secTransMatrix[i][j]*u[j]+64)>>7) Here, Σ is the sum from j=0 to nonZeroSize-1. Also, processing is performed for i=0 to nStSize-1. CoeffMin and CoeffMax indicate the range of values ​​of the transform coefficients.

[0211] (S2204: 2D arrangement of 1D array after conversion processing) The inverse non-separable transform unit 31121 again arranges the transformed coefficients v′[] of the one-dimensional array at predetermined positions within the TU. The arrangement method may be changed depending on predModeIntra.

[0212] Specifically, if predModeIntra<=34, the following processing may be applied.

[0213] d[x][y] = (y<4) ? v[x+(y< <log2StSize)] : ((x<4) ? v[32+x+((y-4)<<2)] : d[x][y]) Otherwise, the following formula applies:

[0214] d[x][y] = (x<4) ? v[y+(x< <log2StSize)] : ((y<4) ? v[32+y+((x-4) << 2)] : d[x][y]) The branching decision may be "predModeIntra<=34" or "predModeIntra==INTRA_LT_CCLM" or "INTRA_T_CCLM or INTRA_L_CCLM", for example.

[0215] (Details of non-separable conversion process) FIG. 24 is a flowchart showing the process flow when performing non-separable conversion with GPM.

[0216] (S2101) If a non-separable transform is to be used (lfnst_idx!=0), the process proceeds to S2102A. Otherwise, the non-linear transform is skipped and the process proceeds to S2300 to perform a separable transform.

[0217] (S2102A) In the case of intra prediction mode, the process proceeds to S2103A, where lfnstTrSetId is derived from IntraPredMode, and the transformation matrix secTranMatrix[][] is derived from lfnstTrSetId and the block size.

[0218] (S2102B) If the mode is not intra prediction mode (GPM mode), the process transitions to S2102B, derives lfnstTrSetId from angleIdx, and derives the transform matrix secTranMatrix[][] from lfnstTrSetId and the block size. Note that when performing non-separable transform (lfnst_idx!=0) only in intra prediction mode or GPM mode, it is determined that the mode is GPM mode when the mode is not intra prediction mode, so the determination of S2102B can be skipped.

[0219] (S2200) A non-separable transformation is performed using the transformation matrix secTranMatrix[][]. Specifically, the processes from S2201 to S2204 described above may be performed.

[0220] (S2300) A separable transform is further performed on the non-separable transformed transform coefficients.

[0221] The adder 312 generates a decoded image for the block by adding, for each pixel, the predicted image for the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311. The adder 312 stores the decoded image for the block in the reference picture memory 306 and also outputs it to the loop filter 305.

[0222] (Configuration of a video encoding device) Next, the configuration of the video encoding device 11 according to this embodiment will be described. Fig. 7 is a block diagram showing the configuration of the video encoding device 11 according to this embodiment. The video encoding device 11 includes a prediction image generation unit 101, a subtraction unit 102, a transformation and quantization unit 103, an inverse quantization and inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, a coding parameter determination unit 110, a parameter coding unit 111, a prediction parameter derivation unit 120, and an entropy coding unit 104.

[0223] The predicted image generation unit 101 generates a predicted image for each CU. The predicted image generation unit 101 includes the inter predicted image generation unit 309 and the intra predicted image generation unit already described, and therefore a description thereof will be omitted.

[0224] The subtraction unit 102 generates a prediction error by subtracting pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

[0225] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing them. The transform / quantization unit 103 outputs the quantized transform coefficients to the parameter coding unit 111 and the inverse quantization / inverse transform unit 105.

[0226] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 in the video decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to the adder .

[0227] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes an outline of the operation of each module.

[0228] The header encoding unit 1110 performs encoding processing of parameters such as header information, division information, prediction information, and quantized transform coefficients.

[0229] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information and the like.

[0230] The CU encoding unit 1112 encodes CU information, prediction information, division information, and the like.

[0231] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0232] The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter prediction parameters (predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX), intra prediction parameters, and quantized transform coefficients to the parameter encoding unit 111.

[0233] The entropy coding unit 104 receives the quantized transform coefficients and coding parameters (division information, prediction parameters) from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output a coded stream Te.

[0234] The prediction parameter derivation unit 120 is a means including the inter-prediction parameter coding unit 112 and an intra-prediction parameter coding unit, and derives intra-prediction parameters and intra-prediction parameters from the parameters input from the coding parameter determination unit 110. The derived intra-prediction parameters and intra-prediction parameters are output to the parameter coding unit 111.

[0235] (Configuration of Inter-Prediction Parameter Encoding Unit) 8 , the inter prediction parameter coding unit 112 includes a parameter coding control unit 1121 and an inter prediction parameter derivation unit 303. The inter prediction parameter derivation unit 303 has the same configuration as the video decoding device. The parameter coding control unit 1121 includes a merge index derivation unit 11211 and a vector candidate index derivation unit 11212.

[0236] The merge index derivation unit 11211 derives merge candidates and the like, and outputs them to the inter prediction parameter derivation unit 303. The vector candidate index derivation unit 11212 derives predicted vector candidates and the like, and outputs them to the inter prediction parameter derivation unit 303 and the parameter coding unit 111.

[0237] (Configuration of the intra-prediction parameter coding unit) The intra-prediction parameter coding unit includes a parameter coding control unit and an intra-prediction parameter derivation unit, which has the same configuration as the video decoding device.

[0238] However, unlike the video decoding device, the inputs to the inter prediction parameter derivation unit 303 and the intra prediction parameter derivation unit are the coding parameter determination unit 110 and the prediction parameter memory 108 , and output to the parameter coding unit 111 .

[0239] The adder 106 generates a decoded image by adding, for each pixel, the pixel values ​​of the predicted block input from the predicted image generation unit 101 and the prediction errors input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0240] The loop filter 107 applies deblocking filtering, SAO, and ALF to the decoded image generated by the adder 106.

[0241] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0242] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined location for each current picture and CU.

[0243] The coding parameter determination unit 110 selects one set of coding parameters from among a plurality of sets of coding parameters. The coding parameters are the above-mentioned QT, BT, or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0244] The coding parameter determination unit 110 calculates an RD cost value indicating the magnitude of the information amount and the coding error for each of the multiple sets. The RD cost value is, for example, the sum of the code amount and the value obtained by multiplying the squared error by a coefficient λ. The code amount is the information amount of the coded stream Te obtained by entropy coding the quantization error and the coding parameters. The squared error is the sum of the squares of the prediction errors calculated by the subtraction unit 102. The coefficient λ is a preset real number greater than zero. The coding parameter determination unit 110 selects the set of coding parameters that minimizes the calculated cost value. The coding parameter determination unit 110 outputs the determined coding parameters to the parameter coding unit 111 and the prediction parameter derivation unit 120.

[0245] Note that parts of the video encoding device 11 and the video decoding device 31 in the above-described embodiments, such as the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generation unit 308, the inverse quantization and inverse transform unit 311, the adder 312, the predicted image generation unit 101, the subtractor 102, the transform and quantization unit 103, the entropy encoding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the encoding parameter determination unit 110, the parameter encoding unit 111, and the prediction parameter derivation unit 120, may be implemented by a computer. In this case, a program for implementing these control functions may be recorded on a computer-readable recording medium, and the program may be read and executed by a computer system. Note that the term "computer system" used here refers to a computer system built into either the video encoding device 11 or the video decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0246] Furthermore, part or all of the video encoding device 11 and video decoding device 31 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 11 and video decoding device 31 may be individually implemented as a processor, or part or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0247] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like can be made within the scope that does not deviate from the gist of the present invention. [Industrial Applicability]

[0248] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data obtained by coding image data, and a video coding device that generates coded data obtained by coding image data, and can also be suitably applied to the data structure of coded data generated by a video coding device and referenced by the video decoding device.

[0249] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to Japanese Patent Application No. 2021-091071, filed on May 31, 2021, the entire contents of which are incorporated herein by reference. [Explanation of symbols]

[0250] 31 Image decoding device 301 Entropy Decoding Unit 302 Parameter Decoding Unit 303 Inter-prediction parameter derivation unit 30377 GPM Forecasting Department 305, 107 Loop filter 306, 109 Reference Picture Memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 309 Inter-prediction image generation unit 30952 GPM synthesis section 311, 105 Inverse quantization and inverse transform unit 312, 106 Addition section 11 Image encoding device 102 Subtraction section 103 Transformation and Quantization Unit 104 Entropy coding unit 110 Encoding parameter determination unit 111 Parameter Encoding Unit 112 Inter-prediction parameter coding unit 120 Prediction parameter derivation part

Claims

1. A video decoding device comprising: a parameter decoding unit that decodes parameters from encoded data; a predicted image generation unit that derives a predicted image according to a prediction mode; and a non-separable transform unit that performs non-separable transform on a prediction error, the parameter decoding unit decodes parameters indicating at least a prediction mode and a non-separable transform; When the prediction mode is an inter prediction mode and the GPM generates a predicted image by performing weighting processing on each pixel of two predicted images according to the position of the pixel, the parameter decoding unit decodes division information of the GPM, derives an angle variable from the division information, the predicted image generation unit generates the predicted image of the GPM using weighting parameters derived from the angle variables; When the parameter indicating the non-separable transformation indicates that the non-separable transformation is on, the non-separable transformation unit derives a non-separable transformation matrix based on the division information or the angle variable of the GPM.

2. The video decoding device described in Claim 1, characterized in that the non-separable transformation unit derives a non-separable transformation matrix based on (1) block size, (2) the division information or the angle variable of the GPM, and (3) parameters indicating the non-separable transformation.

3. The moving image decoding device described in Claim 1, characterized in that the parameter decoding unit has different possible values ​​for the parameter indicating the non-separable transformation depending on whether the prediction mode is the GPM or not.

4. The video decoding device described in Claim 1, characterized in that the parameter decoding unit has different values ​​for the context index of the parameter indicating the non-separable transformation when the prediction mode is inter prediction mode and when it is not.

5. A video encoding device comprising: a parameter encoding unit that encodes parameters; a predicted image generation unit that derives a predicted image according to a prediction mode; and a non-separable transformation unit that performs non-separable transformation on a prediction error, the parameter coding unit codes parameters indicating at least a prediction mode and a non-separable transformation; In a case where the prediction mode is an inter prediction mode and the GPM generates a predicted image by performing weighting processing on each pixel of two predicted images according to the position of the pixel, the parameter encoding unit derives an angle variable from partition information and encodes the partition information of the GPM, the predicted image generation unit generates the predicted image of the GPM using weighting parameters derived from the angle variables; When the parameter indicating the non-separable transformation indicates that the non-separable transformation is on, the non-separable transformation unit derives a non-separable transformation matrix based on the division information or the angle variable of the GPM.

6. A computer-readable recording medium having recorded thereon a program for causing a computer to decode a moving image, The program is installed on the computer. decoding parameters indicating at least a prediction mode and a non-separable transform from the encoded data; In the case where the prediction mode is an inter prediction mode and the GPM generates a predicted image by performing weighting processing on each pixel of two predicted images according to the pixel position, decoding division information of the GPM and deriving an angle variable from the division information; generating the predicted image of the GPM using weighting parameters derived from the angular variables; and If the parameter indicating the non-separable transformation indicates that the non-separable transformation is on, deriving a non-separable transformation matrix based on the segmentation information or the angle variable of the GPM; A recording medium for causing the execution of the above.

Images