Video decoding apparatus and video coding apparatus

Abstract

Provided are video coding / decoding apparatus (11, 31) with enhanced coding efficiency. A video decoding apparatus (31) according to one aspect of the present invention comprises a prediction unit (30377) for deriving a prediction image by using a weight parameter derived by means of an angular variable, and a non-separable transform unit (31121) for performing a non-separable transform, the video decoding apparatus (31) characterized in that the non-separable transform unit (31121), when the inter-prediction mode is a GPM mode, derives a set number for non-separable transform (lfnstTrSetId) from GPM partition information (merge_gpm_partition_idx).

Description

Video decoding device and video encoding device

[0001] An embodiment of the present invention relates to a video decoding device and a video encoding device.

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that encodes moving images to generate coded data, and a moving image decoding device is used that decodes the coded data to generate decoded images.

[0003] Specific video encoding methods include, for example, H.264 / AVC and HEVC (High-Efficiency Video Coding) methods.

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing images, coding tree units (CTUs) obtained by dividing slices, coding units (sometimes called coding units (CUs)) obtained by dividing coding tree units, and transform units (TUs) obtained by dividing coding units, and are coded / decoded for each CU.

[0005] In such video coding methods, a predicted image is typically generated based on a locally decoded image obtained by encoding / decoding an input image, and the predicted image is subtracted from the input image (original image) to obtain a prediction error (sometimes called a "difference image" or "residual image"), which is then coded. Methods for generating predicted images include inter-frame prediction (inter-prediction) and intra-frame prediction (intra-prediction).

[0006] Furthermore, Non-Patent Document 1 discloses a geometric partition mode (GPM) in which a target block is divided into shapes other than a rectangle and different inter predictions are performed for each region. By dividing the target block into shapes other than a rectangle in this manner, more accurate predictions can be made even for complex textures, improving coding efficiency. Non-Patent Document 1 also discloses an image coding device that performs non-separable transform for each transform unit in order to further concentrate coefficients obtained by separate transforming prediction errors in the low-frequency region.

[0007] ITU-T Rec. H.266

[0008] In Non-Patent Document 1, non-separable transform and inverse non-separable transform techniques that are effective for directional prediction residuals are applied only to intra-predicted blocks, and are not adapted to inter-predicted blocks that have many diagonal directions.

[0009] An object of the present invention is to provide a video decoding device, a video encoding device, and related techniques that can more suitably apply inverse non-separable transform.

[0010] In order to solve the above problem, a video decoding device according to one embodiment of the present invention is a video decoding device that includes a prediction unit that derives a predicted image using weight parameters derived from angle variables, and a non-separable transformation unit that performs non-separable transformation, and is characterized in that when the inter prediction mode is GPM mode, the separation transformation unit derives a set number (lfnstTrSetId) of the non-separable transformation from GPM partition information (merge_gpm_partition_idx).

[0011] Another aspect of the present invention provides a video encoding device that includes a prediction unit that derives a predicted image using weight parameters derived from angle variables, and a non-separable transformation unit that performs non-separable transformation, wherein when the inter prediction mode is GPM mode, the separation transformation unit derives a set number (lfnstTrSetId) of the non-separable transformation from GPM partition information (merge_gpm_partition_idx).

[0012] According to an aspect of the present invention, it is possible to improve the coding efficiency in video coding and decoding processing.

[0013] 1 is a schematic diagram showing the configuration of an image transmission system according to the present embodiment. FIG. 2 is a diagram showing the hierarchical structure of data of an encoded stream. FIG. 3 is a schematic diagram showing the configuration of a video decoding device. FIG. 4 is a flowchart explaining the overall operation of a video decoding device. FIG. 4 is a schematic diagram showing the configuration of an inter prediction parameter derivation unit. FIG. 5 is a schematic diagram showing the configuration of an inter prediction image generation unit. FIG. 6 is a block diagram showing the configuration of a video encoding device. FIG. 7 is a schematic diagram showing the configuration of an inter prediction parameter encoding unit. FIG. 8 is a diagram explaining GPM prediction. FIG. 9 is a syntax diagram explaining encoding parameters of GPM prediction. FIG. 10 is a diagram showing the structure of a table indicating the correspondence between merge_gpm_partition_idx, angleIdx, and distanceIdx in GPM prediction. FIG. 11 is a diagram showing the structure of a table indicating the correspondence between idx and DisLut[idx] used in the weighting coefficient derivation process and the motion vector storage process in GPM prediction. FIG. 12 is a flowchart showing the process flow of GPM prediction. FIG. 13 is a table used for selecting a transform matrix for a non-separable transform in intra prediction mode. FIG. 14 is a table used for selecting a transform matrix for a non-separable transform in GPM mode. FIG. 15 is a table used for selecting a transform matrix for a non-separable transform in GPM mode. FIG. 1 is a table used for selecting a transformation matrix for a non-separable transformation in GPM mode. FIG. 2 is a functional block diagram showing an example configuration of an inverse quantization and inverse transform unit. FIG. 3 is an example syntax table showing coding parameters for a GPM non-separable transformation. FIG. 4 is an example syntax table showing flags for a non-separable transformation. FIG. 5 is a flowchart showing the processing flow when a non-separable transformation is performed in GPM.

[0014] First Embodiment Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] FIG. 1 is a schematic diagram showing the configuration of an image transmission system 1 according to this embodiment.

[0016] The image transmission system 1 is a system that transmits an encoded stream obtained by encoding a target image, decodes the transmitted encoded stream, and displays the image. The image transmission system 1 includes a video encoding device (image encoding device) 11, a network 21, a video decoding device (image decoding device) 31, and a video display device (image display device) 41.

[0017] An image T is input to the video encoding device 11 .

[0018] The network 21 transmits the coded stream Te generated by the video coding device 11 to the video decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network 21 is not necessarily limited to a bidirectional communication network, but may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be replaced by a storage medium on which the coded stream Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

[0019] The video decoding device 31 decodes each of the coded streams Te transmitted over the network 21, and generates one or more decoded images Td.

[0020] The video display device 41 displays all or part of one or more decoded images Td generated by the video decoding device 31. The video display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. Furthermore, if the video decoding device 31 has high processing power, it displays high-quality images, and if it has only low processing power, it displays images that do not require high processing power or display power.

[0021] <Operators> The operators used in this specification are listed below.

[0022] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || represents a logical OR.

[0023] x?y:z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0024] Clip3(a,b,c) is a function that clips c to a value between a and b (inclusive). If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (where a <= b).

[0025] abs(a) is a function that returns the absolute value of a.

[0026] Int(a) is a function that returns the integer value of a.

[0027] floor(a) is a function that returns the largest integer less than or equal to a.

[0028] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0029] a / d represents the division of a by d (truncating the fractional part).

[0030] <Structure of the encoded stream Te> Prior to a detailed description of the moving image encoding device 11 and the moving image decoding device 31 according to the present embodiment, the data structure of the encoded stream Te generated by the moving image encoding device 11 and decoded by the moving image decoding device 31 will be described.

[0031] FIG. 2 is a diagram showing the hierarchical structure of the data in the encoded stream Te. The encoded stream Te illustratively includes a sequence and a plurality of pictures constituting the sequence. (a) to (f) of FIG. 2 respectively show an encoded video sequence that defines the sequence SEQ, an encoded picture that defines the picture PICT, an encoded slice that defines the slice S, encoded slice data that defines the slice data, an encoded tree unit included in the encoded slice data, and an encoded unit included in the encoded tree unit.

[0032] (Encoded Video Sequence) An encoded video sequence defines a set of data that the video decoding device 31 refers to in order to decode a sequence SEQ to be processed. As shown in Fig. 2, the sequence SEQ includes a video parameter set, a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an adaptation parameter set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0033] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers included in the video and each individual layer.

[0034] The sequence parameter set SPS defines a set of coding parameters that the video decoding device 31 references to decode the target sequence. For example, the width and height of a picture are defined. Note that there may be multiple SPSs. In this case, one of the multiple SPSs is selected from the PPS.

[0035] The picture parameter set PPS defines a set of coding parameters that the video decoding device 31 references to decode each picture in the target sequence. For example, the PPS includes a reference value for the quantization width used in decoding the picture (pic_init_qp_minus26) and a flag indicating the application of weighted prediction (weighted_pred_flag). Note that there may be multiple PPSs. In this case, one of the multiple PPSs is selected for each picture in the target sequence.

[0036] (Coded Picture) A coded picture defines a set of data that the video decoding device 31 references in order to decode a picture PICT to be processed. As shown in Fig. 2, the picture PICT includes slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0037] In the following description, when there is no need to distinguish between slices 0 to NS-1, the subscripts of the symbols may be omitted. This also applies to other data that are included in the coded stream Te described below and have subscripts.

[0038] (Encoded Slice) An encoded slice defines a set of data to be referenced by the video decoding device 31 in order to decode a slice S to be processed. As shown in Fig. 2, a slice includes a slice header and slice data.

[0039] The slice header includes a group of coding parameters that the video decoding device 31 refers to in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates the slice type is an example of a coding parameter included in the slice header.

[0040] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction during encoding, (2) a P slice that uses unidirectional prediction or intra prediction during encoding, and (3) a B slice that uses unidirectional prediction, bidirectional prediction, or intra prediction during encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, when referred to as a P or B slice, it refers to a slice including a block that can use inter prediction.

[0041] Note that the slice header may include a reference to the picture parameter set PPS (pic_parameter_set_id).

[0042] (Encoded Slice Data) The encoded slice data defines a set of data that the video decoding device 31 references in order to decode the slice data to be processed. As shown in Fig. 2(d), the slice data includes a CTU. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0043] (Coding Tree Unit) Figure 2 defines a set of data that the video decoding device 31 references to decode the target CTU. The CTU is divided into coding units (CUs), which are the basic units of the encoding process, by recursive quad tree (QT) division, binary tree (BT) division, or ternary tree (TT) division. BT division and TT division are collectively called multi-tree (MT) division. A node in the tree structure obtained by recursive quad tree division is called a coding node. The intermediate nodes of a quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is defined as the top-level coding node.

[0044] The CT includes, as CT information, a CU split flag (split_cu_flag) indicating whether CT splitting is performed, a QT split flag (qt_split_cu_flag) indicating whether QT splitting is performed, an MT split direction (mtt_split_cu_vertical_flag) indicating the split direction of MT splitting, and an MT split type (mtt_split_cu_binary_flag) indicating the split type of MT splitting. split_cu_flag, qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

[0045] Different trees may be used for luminance and chrominance. The tree type is indicated by treeType. For example, if a common tree is used for luminance (Y, cIdx=0) and chrominance (Cb / Cr, cIdx=1,2), the common single tree is indicated by treeType=SINGLE_TREE. If two different trees (DUAL trees) are used for luminance and chrominance, the luminance tree is indicated by treeType=DUAL_TREE_LUMA and the chrominance tree is indicated by treeType=DUAL_TREE_CHROMA.

[0046] 2 defines a set of data that the video decoding device 31 references in order to decode a coding unit to be processed. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header defines a prediction mode, etc.

[0047] The prediction process may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU. When the sizes of a CU and a sub-CU are the same, there is one sub-CU in the CU. When the size of a CU is larger than the size of a sub-CU, the CU is divided into sub-CUs. For example, when a CU is 8x8 and a sub-CU is 4x4, the CU is divided into four sub-CUs, divided horizontally in half and vertically in half.

[0048] Prediction types (prediction modes) include intra prediction (MODE_INTRA), inter prediction (MODE_INTER), and intra block copy (MODE_IBC). Intra prediction is prediction within the same picture, while inter prediction refers to prediction processing performed between different pictures (for example, between display times or between layer images).

[0049] The transformation and quantization processes are performed in units of CUs, but the quantized transformation coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0050] (Prediction Parameters) A predicted image is derived from prediction parameters associated with a block. Prediction parameters include intra-prediction and inter-prediction parameters.

[0051] (Prediction Parameters of Intra Prediction) The prediction parameters of intra prediction will be described. The intra prediction parameters are composed of a luminance prediction mode IntraPredModeY (predModeIntra) and a color difference prediction mode IntraPredModeC. There are, for example, 67 types of intra prediction modes (0 to 66). Examples include planar prediction (0), DC prediction (1), and angular prediction (2 to 66).

[0052] (Prediction Parameters of Inter Prediction) The prediction parameters of inter prediction will be described. The inter prediction parameters are composed of prediction list usage flags predFlagL0 and predFlagL1, reference picture indices refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. predFlagL0 and predFlagL1 are flags indicating whether or not a reference picture list (L0 list, L1 list) is used, and when the value is 1, the corresponding reference picture list is used. Note that in this specification, when the term "flag indicating whether or not XX" is used, a flag other than 0 (e.g., 1) indicates XX, and 0 indicates non-XX, and in logical negation, logical product, etc., 1 is treated as true and 0 is treated as false (the same applies below). However, in actual devices and methods, other values ​​may be used as true and false values.

[0053] Syntax elements for deriving inter-prediction parameters include, for example, a merge flag merge_flag (general_merge_flag), a merge index merge_idx, merge_subblock_flag indicating whether to use sub-block-based inter-prediction such as affine mode, regulare_merge_flag, an intra-inter flag ciip_flag, GPM_flag indicating whether to use GPM mode (Geometric partitioning merge mode), merge_gpm_partition_idx indicating the partition shape of GPM mode, merge_gpm_idx0 and merge_gpm_idx1 indicating GPM merge indices, an inter-prediction identifier inter_pred_idc for selecting a reference picture to be used in AMVP mode, a reference picture index refIdxLX, a prediction vector index mvp_LX_idx for deriving a motion vector, a difference vector mvdLX, and a motion vector precision mode amvr_mode. Note that GPM_flag (MergeGpmFlag) may not be explicitly coded, and may be derived as GPM_flag=1 (MergeGpmFlag=1) when ciip_flag=0.

[0054] (Reference Picture List) A reference picture list is a list of reference pictures stored in the reference picture memory 306.

[0055] (Merge Prediction and AMVP Prediction) Prediction parameter decoding (encoding) methods include merge prediction mode (merge mode) and AMVP (Advanced Motion Vector Prediction, adaptive motion vector prediction) mode, and merge_flag is a flag for distinguishing between them. Merge mode is a prediction mode that omits some or all of the motion vector difference, and derives the prediction list usage flag predFlagLX, reference picture index refIdxLX, and motion vector mvLX from the encoded data, instead of including them in the encoded data, from prediction parameters of previously processed neighboring blocks, etc. AMVP mode is a mode that includes inter_pred_idc, refIdxLX, and mvLX in the encoded data. Note that mvLX is encoded as mvp_LX_idx, which identifies the prediction vector mvpLX, and the difference vector mvdLX. Prediction modes that omit or simplify the motion vector difference are collectively called general merge mode, and general merge mode and AMVP prediction may be selected using general_merge_flag. Furthermore, the regular_merge_flag may be transmitted, and if the regular_merge_flag is 1, the normal merge mode or MMVD may be selected, and otherwise the CIIP mode or GPM mode may be selected.

[0056] inter_pred_idc is a value indicating the type and number of reference pictures, and takes one of the values ​​PRED_L0, PRED_L1, or PRED_BI. PRED_L0 and PRED_L1 indicate uni-prediction using one reference picture managed in the L0 list and L1 list, respectively. PRED_BI indicates bi-prediction using two reference pictures managed in the L0 list and L1 list.

[0057] The merge_idx is an index indicating which prediction parameter from among prediction parameter candidates (merge candidates) derived from blocks for which processing has been completed is to be used as the prediction parameter for the current block.

[0058] (Motion Vector) mvLX indicates the amount of shift between blocks on two different pictures. The predicted vector and differential vector related to mvLX are called mvpLX and mvdLX, respectively.

[0059] (Inter prediction identifier inter_pred_idc and prediction list use flag predFlagLX) The relationship between inter_pred_idc, predFlagL0, and predFlagL1 is as follows, and they can be converted into each other.

[0060] inter_pred_idc = (predFlagL1<<1)+predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 Note that, as the inter prediction parameters, a prediction list usage flag or an inter prediction identifier may be used. Furthermore, a determination using the prediction list usage flag may be replaced with a determination using the inter prediction identifier. Conversely, a determination using the inter prediction identifier may be replaced with a determination using the prediction list usage flag.

[0061] (Configuration of Video Decoding Device) The configuration of a video decoding device 31 (FIG. 3) according to this embodiment will be described.

[0062] The video decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generation unit (prediction image generation device) 308, an inverse quantization and inverse transform unit 311, an adder 312, and a prediction parameter derivation unit (not shown). Note that the video decoding device 31 may also be configured without including the loop filter 305, in accordance with the video coding device 11 described below.

[0063] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice headers (slice information) from the coded data. The CT information decoding unit 3021 decodes the CT from the coded data. The CU decoding unit 3022 decodes the CU from the coded data. When a prediction error is included in the TU, the TU decoding unit 3024 decodes QP update information (quantization correction value) and quantized prediction error (residual_coding) from the coded data.

[0064] The header decoding unit 3020 decodes a flag sps_lfnst_enabled_flag indicating whether or not non-separable transform is used from the SPS. Furthermore, the header decoding unit 3020 decodes ph_lfnst_enabled_flag from the picture header (PH) when sps_lfnst_enabled_flag is 1. When ph_lfnst_enabled_flag does not appear, the header decoding unit 3020 infers ph_lfnst_enabled_flag to be 0. Alternatively, when ph_lfnst_enabled_flag does not appear, the value of sps_lfnst_enabled_flag may be set as the value of ph_lfnst_enabled_flag.

[0065] The GPM non-separable transform in the present invention is an inverse non-separable transform (and non-separable transform) when inter prediction is in GPM mode. Below, a method for deriving a transform matrix or a set of transform matrices for the non-separable transform from the partition shape of GPM when applying to inter CUs will be described.

[0066] FIG. 21 is an example of a syntax table showing coding parameters for the GPM non-separable transform.

[0067] The header decoding unit 3020 decodes the flag sps_gpm_lfnst_enabled_flag from the SPS. The sps_gpm_lfnst_enabled_flag is a flag indicating whether or not GPM non-separable transform is used in the case of GPM prediction. When the sps_lfnst_enabled_flag is 1 and the sps_gpm_enabled_flag is 1, the header decoding unit 3020 decodes the sps_gpm_lfnst_enabled_flag. In this example, the sps_gpm_lfnst_enabled_flag is not transmitted when sps_lfnst_enabled_flag==0 or sps_gpm_enabled_flag==0, which has the effect of improving coding efficiency. The sps_gpm_enabled_flag is a flag indicating whether or not the GPM prediction mode is used in the target sequence.

[0068] The TU decoding unit 3024 decodes an index mts_idx indicating a transform base from the coded data. The TU decoding unit 3024 also decodes a parameter lfnst_idx indicating whether a non-separable transform is used and the transform base from the coded data. Specifically, the TU decoding unit 3024 decodes lfnst_idx when the width and height of the CU are 4 or greater and the prediction mode is an intra prediction mode. Note that lfnst_idx indicates that a non-separable transform is not applied when it is 0, indicates one transform matrix of a set (pair) of transform matrices (transform bases) of the non-separable transform when it is 1, and indicates the other transform matrix of the pair when it is 2.

[0069] The predicted image generation unit 308 includes an inter predicted image generation unit 309 (FIG. 6) and an intra predicted image generation unit 310.

[0070] The prediction parameter derivation unit includes an inter prediction parameter derivation unit 303 (FIG. 5) and an intra prediction parameter derivation unit.

[0071] In the following, an example will be described in which CTUs and CUs are used as processing units, but this is not limiting and processing may be performed in sub-CU units. Alternatively, CTUs and CUs may be read as blocks, and sub-CUs as sub-blocks, and processing may be performed in block or sub-block units.

[0072] The entropy decoding unit 301 performs entropy decoding on the externally input coded stream Te to decode individual codes (syntax elements). Entropy coding can be performed in two ways: one is to perform variable-length coding of syntax elements using a context (probability model) adaptively selected according to the type of syntax element and the surrounding circumstances, and the other is to perform variable-length coding of syntax elements using a predetermined table or formula.

[0073] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code is, for example, a prediction mode predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0074] (Basic Flow) FIG. 4 is a flowchart illustrating the general operation of the video decoding device 31.

[0075] (S1100: Decode parameter set information) The header decoder 3020 decodes parameter set information such as VPS, SPS, and PPS from the coded data.

[0076] (S1200: Decode slice information) The header decoding unit 3020 decodes the slice header (slice information) from the coded data.

[0077] Thereafter, the video decoding device 31 repeats the processes from S1300 to S5000 for each CTU included in the target picture, thereby deriving a decoded image of each CTU.

[0078] (S1300: Decode CTU information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0079] (S1400: Decode CT information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0080] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

[0081] (S1510: Decode CU information) The CU decoding unit 3022 decodes CU information, prediction information, the TU split flag split_transform_flag, the CU residual flags cbf_cb, cbf_cr, cbf_luma, and the like from the encoded data.

[0082] (S1520: Decode TU information) When a prediction error is included in a TU, the TU decoding unit 3024 decodes the quantized prediction error and the transform index mts_idx from the coded data.

[0083] (S2000: Generate predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on prediction information.

[0084] (S3000: Inverse quantization and inverse transform) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing on each TU included in the target CU.

[0085] (S4000: Decoded image generation) The addition unit 312 generates a decoded image of the target CU by adding the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization and inverse transform unit 311.

[0086] (S5000: Loop filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

[0087] (Configuration of Intra Prediction Parameter Derivation Unit) The intra prediction parameter derivation unit decodes intra prediction parameters, for example, the intra prediction mode IntraPredMode, by referencing prediction parameters stored in the prediction parameter memory 307, based on the code input from the entropy decoding unit 301. The intra prediction parameter derivation unit outputs the decoded intra prediction parameters to the predicted image generation unit 308 and also stores them in the prediction parameter memory 307. The intra prediction parameter derivation unit may derive different intra prediction modes for luma and chroma.

[0088] The loop filter 305 is a filter provided in the encoding loop that removes block distortion and ringing distortion to improve image quality. The loop filter 305 applies filters such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to the decoded image of the CU generated by the adder 312.

[0089] The reference picture memory 306 stores the decoded image of the CU generated by the adder 312 in a predetermined location for each current picture and current CU.

[0090] The prediction parameter memory 307 stores prediction parameters at a predetermined location for each CTU or CU to be decoded. Specifically, the prediction parameter memory 307 stores the parameters decoded by the parameter decoding unit 302 and the prediction mode predMode separated by the entropy decoding unit 301.

[0091] The prediction image generation unit 308 receives input of a prediction mode predMode, prediction parameters, etc. The prediction image generation unit 308 also reads a reference picture from the reference picture memory 306. The prediction image generation unit 308 generates a prediction image of a block or sub-block using the prediction parameters and the read reference picture (reference picture block) in the prediction mode indicated by the prediction mode predMode. Here, the reference picture block is a set of pixels on the reference picture (usually rectangular, and therefore referred to as a block), and is an area referenced to generate a prediction image.

[0092] The intra-predicted image generation unit 310 generates a predicted image of the current block by referring to the read decoded pixel values ​​and the prediction mode indicated by IntraPredMode. The intra-predicted image generation unit 310 outputs the generated predicted image of the block to the adder 312.

[0093] (Configuration of Inter Prediction Parameter Derivation Unit) The inter prediction parameter derivation unit 303 derives inter prediction parameters by referring to prediction parameters stored in the prediction parameter memory 307, based on the syntax elements input from the parameter decoding unit 302. The inter prediction parameter derivation unit 303 also outputs the inter prediction parameters to the inter prediction image generation unit 309 and the prediction parameter memory 307. The inter prediction parameter derivation unit 303 and its internal elements, the AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, the GPM prediction unit 3037, and the MV addition unit 3038, are means common to the video encoding device and the video decoding device, and therefore may be collectively referred to as a motion vector derivation unit (motion vector derivation device).

[0094] When GPM_Flag is 1, that is, when it indicates GPM prediction mode, the GPM prediction unit 30377 derives parameters for GPM prediction.

[0095] If merge_flag is 1, that is, if it indicates merge prediction mode, merge_idx is derived and output to the merge prediction parameter derivation unit 3036 .

[0096] When merge_flag is 0, that is, when it indicates the AMVP prediction mode, the AMVP prediction parameter derivation unit 3032 derives mvpLX from inter_pred_idc, refIdxLX, or mvp_LX_idx.

[0097] (MV Addition Unit) The MV addition unit 3038 adds the derived mvpLX and mvdLX to derive mvLX.

[0098] (Merge Prediction) The merge prediction parameter derivation unit 3036 includes a merge candidate derivation unit and a merge candidate selection unit. Merge candidates are configured to include prediction parameters (predFlagLX, mvLX, refIdxLX) and are stored in a merge candidate list. Merge candidates stored in the merge candidate list are assigned indices according to a predetermined rule.

[0099] The merge candidate derivation unit derives merge candidates by directly using the motion vectors and refIdxLX of the decoded adjacent blocks. Alternatively, the merge candidate derivation unit may apply a spatial merge candidate derivation process, a temporal merge candidate derivation process, or the like, which will be described later.

[0100] In the spatial merge candidate derivation process, the merge candidate derivation unit reads prediction parameters stored in the prediction parameter memory 307 according to a predetermined rule and sets them as merge candidates. The reference pictures are specified, for example, by prediction parameters for each of the neighboring blocks within a predetermined range from the target block (e.g., all or part of the blocks adjacent to the left A1, right B1, upper right B0, lower left A0, and upper left B2 of the target block). The merge candidates are referred to as A1, B1, B0, A0, and B2. Here, A1, B1, B0, A0, and B2 are motion information derived from blocks containing the following coordinates:

[0101] A1: (xCb - 1, yCb + cbHeight - 1) B1: (xCb + cbWidth - 1, yCb - 1) B0: (xCb + cbWidth, yCb - 1) A0: (xCb - 1, yCb + cbHeight) B2: (xCb - 1, yCb - 1) The top left coordinates of the target block are (xCb, yCb), the width is cbWidth, and the height is cbHeight.

[0102] As a temporal merge derivation process, the merge candidate derivation unit reads the prediction parameters of the lower right CBR of the target block or the block C in the reference image including the center coordinates from the prediction parameter memory 307, sets it as a merge candidate Col, and stores it in the merge candidate list mergeCandList[ ].

[0103] The order in which mergeCandList[] is stored is, for example, spatial merge candidates (B1, A1, B0, A0, B2), then temporal merge candidate Col. Reference blocks that are unavailable (for example, blocks that are intra-predicted) are not stored in the merge candidate list. i = 0 if( availableFlagB1 ) mergeCandList[ i++ ] = B1 if( availableFlagA1 ) mergeCandList[ i++ ] = A1 if( availableFlagB0 ) mergeCandList[ i++ ] = B0 if( availableFlagA0 ) mergeCandList[ i++ ] = A0 if( availableFlagB2 ) mergeCandList[ i++ ] = B2 if( availableFlagCol ) mergeCandList[ i++ ] = Col Furthermore, history merge candidate HmvpCand, average candidate avgCand, and zero merge candidate zeroCandm may be added to mergeCandList[] and used.

[0104] The merge candidate selection unit selects a merge candidate N indicated by merge_idx from among the merge candidates included in the merge candidate list using the following formula.

[0105] N = mergeCandList[merge_idx] where N is a label indicating a merge candidate, such as A1, B1, B0, A0, B2, Col, etc. The motion information of the merge candidate indicated by label N is indicated by (mvLXN[0], mvLXN[0]), predFlagLXN, and refIdxLXN.

[0106] The selected (mvLXN[0], mvLXN[0]), predFlagLXN, and refIdxLXN are selected as the inter prediction parameters for the current block. The merging candidate selection unit stores the inter prediction parameters of the selected merging candidate in the prediction parameter memory 307 and outputs them to the inter prediction image generation unit 309.

[0107] (GPM Prediction) GPM prediction will now be described. GPM prediction is a prediction used in GPM mode, and generates a predicted image as two non-rectangular prediction units that divide a target CU by a line segment. An example of a line segment is shown in FIG. 9(a). A line segment spanning the target CU is defined by an angle index angleIdx (angle variable) and a distance index distanceIdx (distance variable) shown in FIG. 9(b). angleIdx indicates the angle φ between a vertical line and the line segment. distanceIdx indicates the distance ρ from the center of the target CU to the line segment. angleIdx is an integer value assigned to the angle of the line segment. For example, if 360 degrees are roughly divided into 32 equal parts, the value shown in FIG. 9(c) is assigned. Note that in this example, as mentioned "roughly," 360 degrees are not divided into perfect equal parts, but are divided into 32 parts using a square scale (coordinates). Note that a configuration in which 360 degrees are divided into 64 parts and even finer angle indexes are sent may also be used.

[0108] In generating a predicted image for GPM prediction, two "rectangular" predicted images (temporary predicted images) including a non-rectangular prediction unit may be derived, and the two rectangular regions may be derived by weighting according to the shape of the non-rectangular prediction unit. The motion compensation unit 3091 derives two temporary predicted images for the target CU, and the GPM synthesis unit 30952 derives a predicted image by applying weighting processing to each pixel of the two temporary predicted images according to the pixel's position. This processing is called GPM synthesis processing. Processing other than prediction (e.g., transformation (inverse transformation) and quantization (inverse quantization)) is applied to the entire target CU.

[0109] The GPM prediction unit 30377 derives prediction parameters for the two non-rectangular regions and supplies them to the inter-prediction image generation unit 309. To simplify the processing, GPM prediction may be configured not to use bi-prediction. In this case, unidirectional inter-prediction parameters are derived for the non-rectangular regions.

[0110] (Decoding of Syntax in GPM Prediction) Whether GPM prediction is on or off and the parameters when GPM prediction is on are notified in the coded data as follows.

[0111] As shown in FIG. 10(a), sps_gpm_enabled_flag is notified by SPS and indicates whether or not the GPM prediction mode is used in the target sequence. When sps_gpm_enabled_flag is 0, it indicates that the GPM prediction mode is not used in the target sequence. When sps_gpm_enabled_flag is 1, it indicates that the GPM prediction mode is used in the target sequence. Here, it indicates that the GPM prediction mode in which the number of selectable partition patterns (partition types) is NumGPMFull (e.g., 64) is used in the target sequence. Note that sps_gpm_enabled_flag is not limited to SPS, and may be transmitted in PPS, picture header, or slice header.

[0112] 10(b) shows the syntax configuration notified in merge mode. The parameter decoding unit 302 decodes syntax elements in the encoded data, and the GPM prediction unit 30377 (inter-prediction parameter derivation unit 303) derives GPM prediction parameters according to the following rules:

[0113] In the SPS, sps_max_num_merge_cand_minus_max_num_gpm_cand is notified. sps_max_num_merge_cand_minus_max_num_gpm_cand is a parameter used to derive MaxNumGpmMergeCand, the maximum number of merge candidates for GPM prediction. The GPM prediction unit 30377 derives merge candidates MergeCand ranging from 0 to MaxNumMergeCand-1, the maximum number of merge prediction candidates.

[0114] If sps_gpm_enabled_flag is on and MaxNumMergeCand is 3 or greater, MaxNumGpmMergeCand is calculated using the following formula:

[0115] MaxNumGpmMergeCand = MaxNumMergeCand - sps_max_num_merge_cand_minus_max_num_gpm_cand When sps_gpm_enabled_flag is on and MaxNumMergeCand is 2, MaxNumGpmMergeCand is calculated using the following formula.

[0116] MaxNumGpmMergeCand = 2 If none of the above applies, MaxNumGpmMergeCand is calculated using the following formula.

[0117] MaxNumGpmMergeCand = 0 When MaxNumGpmMergeCand is 0, GPM predictions are prohibited.

[0118] general_merge_flag is a flag indicating whether to use merge mode, which is notified when the current block is not in skip mode. In skip mode, the inter prediction parameter derivation unit 303 sets general_merge_flag=1. merge_data() is a syntax structure for notifying parameters of merge prediction.

[0119] When ciip_flag is 0, the GPM prediction syntax elements merge_gpm_partition_idx, merge_gpm_idx0, and merge_gpm_idx1 are notified. merge_gpm_partition_idx is an index (partition index) indicating the division pattern of the GPM prediction mode. The partition index indicates a combination of angleIdx and distanceIdx, which identify a line segment spanning the target block to divide the target block into two non-rectangular regions. Merge candidates are used as motion information used to generate a predicted image for GPM prediction. merge_gpm_idx0 and merge_gpm_idx1 are merge candidate indices that indicate the motion information of the two non-rectangular regions. The number of partition index options (number of division patterns) is NumGPMFull, and merge_gpm_partition_idx takes an integer value between 0 and NumGPMFull-1 (= 63).

[0120] MergeGpmFlag is a flag that indicates whether or not to perform GPM prediction on the current block. If all of the following conditions (GPM determination conditions) are met, the GPM prediction unit 30377 sets MergeGpmFlag=1 (GPM prediction on); otherwise, the GPM prediction unit 30377 sets MergeGpmFlag=0. sps_gpm_enabled_flag=1 slice_type is B slice general_merge_flag=1 cbWidth>=8 and cbHeight>=8 cbWidth<8*cbHeight and cbHeight<8*cbWidth regular_merge_flag=0 merge_subblock_flag=0 ciip_flag=0 When MergeGpmFlag=1, the GPM prediction unit 30377 derives parameters necessary for generating a predicted image in the following procedure and outputs the parameters to the GPM synthesis unit 30952.

[0121] The parameter decoding unit 302 (CU decoding unit 3022) decodes merge_gpm_partition_idx, merge_gpm_idx0, and merge_gpm_idx1. If !ciip_flag is true, the parameter decoding unit 302 (CU decoding unit 3022) may decode merge_gpm_partition_idx and merge_gpm_idx0, and may further decode merge_gpm_idx1 if MaxNumGpmMergeCand>2.

[0122] (Motion Information Deriving Process in GPM Prediction) The GPM prediction unit 30377 derives merge indexes m and n from syntax elements merge_gpm_idx0 and merge_gpm_idx1 that indicate motion information of two non-rectangular regions, as follows.

[0123] m = merge_gpm_idx0 n = merge_gpm_idx1 + (merge_gpm_idx1 >= m) ? 1 : 0 In the following, the merge candidate pointed to by merge index m is denoted as M, and the merge candidate pointed to by merge index n is denoted as N.

[0124] The merge prediction parameter derivation unit 3036 derives motion information (mvLXM, mvLXN, refIdxLXM, refIdxLXN, predFlagLXM, predFlagLXN, bcwIdx, mergeCandList, etc.) of merge candidates M and N using the method described in (Merge Prediction). The GPM prediction unit 30377 uses this motion information to set motion vectors mvA and mvB, reference indices refIdxA and refIdxB, and prediction list flags predListFlagA and predListFlagB of merge_gpm_idx0 and merge_gpm_idx1.

[0125] These motion information are referenced to generate temporal prediction images for the two non-rectangular regions.

[0126] The GPM prediction unit 30377 derives angleIdx and distanceIdx corresponding to merge_gpm_partition_idx according to the table shown in FIG. 11 and outputs them to the GPM synthesis unit 30952.

[0127] The GPM prediction unit 30377 uses the motion information (mvX, refIdxX, predListFlagX, X=A or B) to derive temporary predicted images predSamplesLA and predSamplesLB and outputs them to the GPM synthesis unit 30952. The GPM synthesis unit 30952 derives a predicted image using weighting parameters derived from angle variables. More specifically, it generates a predicted image using weighting information derived using angleIdx and distanceIdx and the temporary predicted images predSamplesLA and predSamplesLB.

[0128] (Weighted pixel prediction process in GPM prediction) The GPM synthesis unit 30952 derives the predicted pixel value pbSample in the following procedure. Here, nCbW = cbWidth, and nCbH = cbHeight. First, the GPM synthesis unit 30952 derives bitDepth as follows for the weighting coefficient derivation process in GPM prediction and the motion vector storage process described later.

[0129] If cIdx is 0, the GPM synthesis unit 30952 sets bitDepth to the luminance pixel bit count BitDepthY.

[0130] If cIdx is 0, the GPM synthesis unit 30952 sets nW and nH to nCbW and nCbH, respectively. If cIdx is not 0, the GPM synthesis unit 30952 sets nW and nH to nCbW*SubWidthC and nCbH*SubHeightC, respectively, where SubWidthC and SubHeightC are values ​​determined in advance according to the chrominance format.

[0131] If cIdx is 0, the GPM synthesis unit 30952 sets both subW and subH to 1. If cIdx is not 0, the GPM synthesis unit 30952 sets subW and subH to SubWidthC and SubHeightC, respectively.

[0132] If cIdx is not 0, the GPM synthesis unit 30952 sets bitDepth to the number of color difference pixel bits, BitDepthC.

[0133] Next, the GPM synthesis unit 30952 derives the variables nW, nH, shift1, offset1, displacementX, displacementY, partFlip, and shiftHor as follows:

[0134] nW = (cIdx==0) ? nCbW : nCbW * SubWidthC nH = (cIdx==0) ? nCbH : nCbH * SubHeightC shift1 = Max(5, 17 - BitDepth) offset1 = 1 << (shift1 - 1) displacementX = angleIdx displacementY = (angleIdx + 8) % 32 partFlip = (angleIdx>=13 && angleIdx<=27) ? 0 : 1 shiftHor = (angleIdx%16==8 || (angleIdx%16!=0 && nH>=nW)) ? 0 : 1 If shiftHor is 0, the GPM synthesis unit 30952 derives offsetX and offsetY as follows.

[0135] offsetX = (-nW) >> 1 offsetY = ((-nH) >> 1) + (angleIdx<16? (distanceIdx * nH) >> 3 : -((distanceIdx * nH) >> 3)) When shiftHor is 1, the GPM synthesis unit 30952 derives offsetX and offsetY as follows:

[0136] offsetX = ((-nW) >> 1) + (angleIdx<16? (distanceIdx * nW) >> 3 : -((distanceIdx * nW) >> 3)) offsetY = (-nH) >> 1 The GPM synthesis unit 30952 derives predicted pixels pbSamples according to the following steps.

[0137] 1) The GPM synthesis unit 30952 derives generalized coordinates (xL, yL) from coordinates (x, y) within the CU.

[0138] xL = (cIdx==0) ? x : x * SubWidthC yL = (cIdx==0) ? y : y * SubHeightC 2) The GPM synthesis unit 30952 uses disLut shown in Figure 12 to calculate the weighting coefficient wValue according to the coordinates within the CU as follows.

[0139] weightIdx = (((xL + offsetX) << 1) + 1) * disLut[displacementX] + (((yL + offsetY) << 1) + 1) * disLut[displacementY] weightIdxL = partFlip? 32 + weightIdx : 32 - weightIdx wValue = Clip3(0, 8, (weightIdxL + 4) >> 3) 3) The GPM synthesis unit 30952 derives the value of pbSample as follows.

[0140] pbSamples[x][y] = Clip3(0, (1 << BitDepth) - 1, (predSamplesLA[x][y] * wValue + predSamplesLB[x][y] * (8 - wValue) + offset1) >> shift1) (Motion vector storage process in GPM prediction) The GPM prediction unit 30377 stores the motion vectors (mvA, mvB) of non-rectangular areas A and B and reference picture information (predFlagA, prefFlagB, refIdxLA, refIdxLB) in memory in 4*4 sub-block units so that they can be referenced in subsequent processing.

[0141] (GPM Prediction Processing Flow) Fig. 13 is a flowchart showing the GPM prediction processing flow. The following describes the GPM prediction processing flow on the assumption that all the conditions set as MergeGpmFlag = 1 (GPM prediction on) are satisfied as described above.

[0142] In S3501, the parameter decoding unit 302 appropriately decodes various syntax elements notified by the SPS, PPS, slice header, merge data, etc., as shown in Fig. 4, for example, and outputs these syntax elements to the inter prediction parameter derivation unit 303 (the merge prediction parameter derivation unit 3036, the GPM prediction unit 30377, etc.).

[0143] In S3502, the GPM prediction unit 30377 determines whether MergeGpmFlag is 1 or not.

[0144] If MergeGpmFlag is not 1 in S3502, the GPM prediction unit 30377 turns off GPM prediction in S3506, and the process ends.

[0145] In S3502, if MergeGpmFlag is 1, the GPM prediction unit 30377 performs GPM prediction processing in S3503 to S3505.

[0146] In S3503, the GPM prediction unit 30377 derives motion information and generates two more temporal prediction images predSamplesLA and predSamplesLB.

[0147] In S3504, the GPM prediction unit 30377 derives the weighting coefficient wValue, and the GPM synthesis unit 30952 generates a predicted image.

[0148] In S3505, the GPM prediction unit 30377 stores the motion vector in memory.

[0149] (AMVP Prediction) The AMVP prediction parameter derivation unit 3032 includes a vector candidate derivation unit and a vector candidate selection unit. The vector candidate derivation unit derives predictor vector candidates from the motion vectors of decoded adjacent blocks stored in the prediction parameter memory 307 based on refIdxLX, and stores the predictor vector candidates in a predictor vector candidate list mvpListLX[ ].

[0150] The vector candidate selection unit selects, as mvpLX, the motion vector mvpListLX[mvp_LX_idx] indicated by mvp_LX_idx from among the predicted vector candidates in mvpListLX[ ]. The vector candidate selection unit outputs the selected mvpLX to the MV addition unit 3038.

[0151] (MV Addition Unit) The MV addition unit 3038 calculates mvLX by adding the mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded mvdLX. The addition unit 3038 outputs the calculated mvLX to the inter predicted image generation unit 309 and the prediction parameter memory 307.

[0152] mvLX[0] = mvpLX[0] + mvdLX[0] mvLX[1] = mvpLX[1] + mvdLX[1] (Inter prediction image generation unit 309) When predMode indicates inter prediction mode, the inter prediction image generation unit 309 generates a prediction image of a block or sub-block by inter prediction using the inter prediction parameters and reference picture input from the inter prediction parameter derivation unit 303.

[0153] 6 is a schematic diagram showing the configuration of an inter-prediction image generation unit 309 included in the prediction image generation unit 308 according to this embodiment. The inter-prediction image generation unit 309 includes a motion compensation unit (prediction image generation device) 3091 and a synthesis unit 3095. The synthesis unit 3095 includes an intra-inter synthesis unit 30951, a GPM synthesis unit 30952, a BIO unit 30954, and a weight prediction unit 3094.

[0154] (Motion Compensation) The motion compensation unit 3091 (interpolated image generation unit 3091) generates an interpolated image (motion-compensated image) by reading a reference block from the reference picture memory 306 based on the inter-prediction parameters (predFlagLX, refIdxLX, mvLX) input from the inter-prediction parameter derivation unit 303. The reference block is a block located at a position shifted by mvLX from the position of the current block on the reference picture RefPicLX specified by refIdxLX. Here, if mvLX does not have integer precision, an interpolated image is generated by applying a filter called a motion compensation filter, which is used to generate pixels at decimal positions.

[0155] The motion compensation unit 3091 first derives the integer position (xInt, yInt) and phase (xFrac, yFrac) corresponding to the coordinates (x, y) in the prediction block using the following equations.

[0156] xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x xFrac = mvLX[0]&(MVPREC-1) yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y yFrac = mvLX[1]&(MVPREC-1) Here, (xPb, yPb) are the upper left coordinates of a bW*bH size block, x=0...bW-1, y=0...bH-1, and MVPREC indicates the accuracy of mvLX (1 / MVPREC pixel accuracy). For example, MVPREC=16.

[0157] The motion compensation unit 3091 derives the temporary image temp[][] by performing horizontal interpolation on the reference picture refImg using an interpolation filter. In the following, Σ is the sum over k, where k=0..NTAP-1, shift1 is a normalization parameter that adjusts the value range, and offset1=1<<(shift1-1).

[0158] temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1 Next, the motion compensation unit 3091 derives the interpolated image Pred[][] by vertically interpolating the temporary image temp[][]. In the following, Σ is the sum for k=0..NTAP-1, shift2 is a normalization parameter that adjusts the value range, and offset2=1<<(shift2-1).

[0159] Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2 In the case of bi-prediction, the above Pred[][] is derived for each L0 list and L1 list (called interpolated images PredL0[][] and PredL1[][]), and the interpolated image Pred[][] is generated from PredL0[][] and PredL1[][].

[0160] (IntraInter Synthesis Processing) When ciip_mode is 1, the intrainter synthesis unit 30951 generates a predicted image in CIIP (Combined intra prediction) mode by calculating a weighted sum of an inter predicted image and an intra predicted image.

[0161] In CIIP mode, a predicted value in CIIP mode is calculated by taking a weighted average of the inter-predicted image derived in merge mode and the intra-predicted image derived by planar prediction. The weight of the combination is determined by the prediction mode of the adjacent coding block. If intra-prediction is used in both of the two adjacent blocks (above and to the left) of the current block, the combination ratio of inter-prediction to intra-prediction is 1:3. On the other hand, if neither of the two adjacent blocks is intra-prediction, the combination ratio of inter-prediction to intra-prediction is 3:1. In all other cases, the ratio is 2:2. The same weighting factor is applied to both luma and chroma.

[0162] (BIO Prediction) Next, we will explain in detail the BIO prediction (Bi-Directional Optical Flow, BDOF processing) performed by the BIO unit 30954. In bi-prediction mode, the BIO unit 30954 generates a predicted image by referring to two predicted images (a first predicted image and a second predicted image) and a gradient correction term.

[0163] (Weighted Prediction) The weighted prediction unit 3094 generates a predicted image of a block by multiplying the interpolated image PredLX by a weighting coefficient. When one of the prediction list usage flags (predFlagL0 or predFlagL1) is 1 (uni-prediction) and weighted prediction is not used, the weighted prediction unit 3094 performs processing using the following equation to adjust PredLX (LX is L0 or L1) to the pixel bit depth bitDepth.

[0164] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredLX[x][y]+offset1)> >shift1) where shift1=14-bitDepth, offset1=1<<(shift1-1). Furthermore, when both prediction list usage flags (predFlagL0 and predFlagL1) are 1 (bi-prediction PRED_BI) and weighted prediction is not used, PredL0 and PredL1 are averaged and adjusted to the number of pixel bits according to the following equation.

[0165] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,(PredL0[x][y]+PredL1[x][y]+offset2)> >shift2) where shift2=15-bitDepth and offset2=1<<(shift2-1).

[0166] Furthermore, when uni-prediction and weighted prediction are performed, the weighted prediction unit 3094 derives a weighted prediction coefficient w0 and an offset o0 from the coded data, and performs processing according to the following equations.

[0167] Pred[x][y] = Clip3(0,(1<<bitDepth)-1,((PredLX[x][y]*w0+2^(log2WD-1))> >log2WD)+o0) where log2WD is a variable indicating a predetermined shift amount.

[0168] Furthermore, when bi-prediction PRED_BI and weighted prediction are performed, the weighted prediction unit 3094 derives weighted prediction coefficients w0, w1, o0, and o1 from the coded data and performs the processing of the following equations.

[0169] Pred[x][y] = Clip3(0,(1< <bitDepth)-1,(PredL0[x][y]*w0+PredL1[x][y]*w1+((o0+o1+1)<<log2WD))> >(log2WD+1)) The inter predicted image generation unit 309 outputs the generated predicted image of the block to the addition unit 312.

[0170] (Intra-Prediction Image Generator) When predMode indicates an intra-prediction mode, the intra-prediction image generator performs intra-prediction using the intra-prediction parameters input from the intra-prediction parameter derivation unit and reference pixels read from the reference picture memory 306 .

[0171] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0172] 20 is a block diagram showing the configuration of the inverse quantization and inverse transform unit 311 of this embodiment. The inverse quantization and inverse transform unit 311 includes a scaling unit 31111, an inverse non-separable transform unit 31121, and an inverse core transform unit 31123.

[0173] The inverse quantization and inverse transform unit 311 calculates transform coefficients d[][] by scaling (inverse quantizing) the quantized transform coefficients qd[][] input from the entropy decoding unit 301 using a scaling unit 31111. These quantized transform coefficients qd[][] are coefficients obtained by performing a transform such as a DCT (Discrete Cosine Transform) or a DST (Discrete Sine Transform) on prediction errors during encoding and then quantizing them, or coefficients obtained by further performing a non-separable transform on the transformed coefficients. If lfnst_idx!=0, the inverse quantization and inverse transform unit 311 performs an inverse transform using an inverse non-separable transform unit 31121. The inverse frequency transform such as an inverse DCT or an inverse DST is then performed on the transform coefficients to calculate prediction errors. Furthermore, when lfnst_idx==0, the inverse non-separable transform unit 31121 is not performed, and the transform coefficients scaled by the scaling unit 31111 are subjected to inverse transform such as inverse DCT and inverse DST to calculate a prediction error. The inverse quantization and inverse transform unit 311 outputs the prediction error to the adder 312.

[0174] Note that, since the inverse transform and the transform are paired processes, the transform and the inverse transform may be interpreted interchangeably. Alternatively, when the inverse transform is called the transform, the transform may be called the forward transform. For example, when the inverse non-separable transform is called the non-separable transform, the non-separable transform may be called the forward non-separable transform. Furthermore, the core transform is simply called the transform.

[0175] d[x][y] is transmitted to the inverse core transform unit 31123 or the inverse non-separable transform unit 31121. The inverse non-separable transform unit 31121 applies an inverse non-separable transform to the transform coefficients d[][] after inverse quantization and before the core transform.

[0176] (Non-separable transform) The inverse non-separable transform is applied to the transform coefficients of part or all of the TU in the moving image decoding device 31. After the inverse non-separable transform is applied, an inverse separable transform (such as DCT2 and DST7) is applied to the transform coefficients after the inverse non-separable transform. Also, when the TU is divided into 4x4 sub-blocks, the non-separable transform and the inverse non-separable transform are applied only to a predetermined sub-block in the upper left. Examples of the size of the TU where one of the width W and height H of the TU is 4 include 4×4, 8×4, 4×8, L×4, and 4×L (L is a natural number of 16 or more).

[0177] Also, a technique of transmitting only some low-frequency components among the transform coefficients after the separable transform is called RST (Reduced Secondary Transform) transform or LFNST (Low Frequency Non-Separable-Transform). Specifically, when the number nonZeroSize of the transform coefficients of the non-separable transform to be transmitted is less than or equal to the size of the separable transform ((1<<log2StSize)x(1<<log2StSize)), it is called LFNST. In this specification, the transform applied to the error in the case of GPM prediction is called GPM non-separable transform (or GPM_LFNST). Hereinafter, the processing of the inverse non-separable transform will be described in order.

[0178] (S2000: Decoding of non-separable transform index) FIGS. 22 and 23 are examples of a syntax table showing the flag of the non-separable transform. FIG. 22 shows an example of notifying lfnst_idx and mtx_idx in the CU, and FIG. 23 shows an example of notifying lfnst_idx and mtx_idx in the TU.

[0179] The TU decoding unit 3024 decodes an index mts_idx indicating the transform matrix of the separable transform from the encoded data. It may be configured to decode mts_idx after lfnst_idx and decode mtx_idx only when lfnst_idx is 0. That is, a configuration may be used in which a transform matrix other than DCT2 indicated by mtx_idx!=0 is used only when the non-separable transform is not used (lfnst_idx is 0). [[ID=,11]]

[0180] The TU decoding unit 3024 also decodes the index lfnst_idx from the coded data. lfnst_idx is an index indicating whether a non-separable transform is used and the transform matrix. The TU decoding unit 3024 derives the flag LfnstDcOnly and the flag LfnstZeroOutSigCoeffFlag. LfnstDcOnly is a flag indicating whether the transform coefficients are DC only, and LfnstZeroOutSigCoeffFlag is a flag indicating whether a transform coefficient exists in a predetermined high-frequency region (zero-out region). The TU decoding unit 3024 decodes lfnst_idx when LfnstDcOnly==0 and LfnstZeroOutSigCoeffFlag==1. Here, LfnstDcOnly==0 indicates that there are transform coefficients other than DC coefficients. LfnstZeroOutSigCoeffFlag==1 indicates that there are no non-zero transform coefficients in the zero-out region. Before decoding the residual of a TU, set LfnstDcOnly=1 and LfnstZeroOutSigCoeffFlag=1. If the position of the last coefficient is not DC (lastSubBlock==0 && lastScanPos>0), set LfnstDcOnly=0. If the last position is in the high-frequency region, set LfnstZeroOutSigCoeffFlag=0. The last position is in the high-frequency region when, for example, (lastScanPos>7 && (log2TbWidth==2 || log2TbWidth==3) is satisfied. If lfnst_idx is not to be decoded, set lfnst_idx=0. Note that, as shown in the figure, lfnst_idx may be decoded when the intra subpartition tool is on (IntraSubPartitionsSplitType!=ISP_NO_SPLIT) or (LfnstDcOnly==0 and LfnstZeroOutSigCoeffFlag==1). Furthermore, when decoding lfnst_idx, the width and height of the CU may be limited to 4 or more, or MIP prediction in small blocks may be excluded (block size may be limited to a predetermined size or more).

[0181] The TU decoding unit 3024 decodes lfnst_idx when the prediction mode is intra prediction mode and sps_lfnst_enabled_flag is 1 (SYN_LFNST_INTRA), or when the prediction mode is inter and sps_gpm_lfnst_enabled_flag is 1 (SYN_GPM_INTRA). Note that when lfnst_idx is 0, it indicates that a non-separable transform is not applied, when it is 1, it indicates that one transform of a set (pair) of non-separable transform matrices is used, and when it is 2, it indicates that the other transform of the pair is used.

[0182] Also, when the prediction mode is inter and a GPM non-separable transform is used, that is, when applying a GPM non-separable transform to an inter CU, only one of the set (pair) of non-separable transform matrices is used. The value of lfnst_idx may be limited to {0,1} or {0,2}, which reduces the cost of the flag required when applying a GPM non-separable transform to an inter CU.

[0183] Specifically, when decoding a syntax element indicating whether or not non-separable transform is used, the TU decoding unit 3024 decodes lfnst_idx consisting of three values ​​{0, 1, 2} or more if the prediction mode is intra prediction. Otherwise (if the prediction mode is inter prediction, GPM prediction in this case), lfnst_flag of two values ​​{0, 1} may be decoded. In the following processing, lfnst_idx = lfnst_flag is treated as being equal to lfnst_idx. Different contexts may be used for lfnst_idx and lfnst_flag, which are switched depending on the prediction mode.

[0184] Alternatively, the TU decoding unit 3024 may first decode lfnst_flag, and then, if the prediction mode is intra prediction, decode lfnst_set_flag. lfnst_set_flag is a flag or index indicating the type of non-separable transform. Otherwise (if the prediction mode is inter prediction, GPM prediction in this case), lfnst_set_flag is not decoded from the encoded data, and lfnst_set_flag=0 is assumed. In the following processing, lfnst_idx=lfnst_flag+lfnst_set_flag is treated as follows.

[0185] Furthermore, the TU decoding unit 3024 decodes lfnst_idx regardless of the prediction mode, but the binarization thereof may be changed depending on the prediction mode. For example, when the prediction mode is intra prediction, a truncated rice code (TR code) with a maximum value cmax=2 and a rice parameter cRiceParam=0 may be used as binarization to decode lfnst_idx within a value range from 0 to 2. When the prediction mode is inter prediction (GPM prediction in this case), a TR code with a maximum value cmax=1 and a rice parameter cRiceParam=0 may be used as binarization to decode lfnst_idx within a value range from 0 to 1. Furthermore, a truncated unary code may be used instead of a TR code.

[0186] Furthermore, the context index cIdx (or context index difference ctxInc) of the above lfnst_idx, lfnst_flag, and lfnst_set_flag may be changed depending on the prediction mode. cIdx is an index used to store probability parameters used in arithmetic coding. For example, it is derived as ctxIdx (or ctxInc) = (CuPredMode[chType][xTb][yTb]==MODE_INTRA) ? 0 : 1.

[0187] (Deriving a Transformation Matrix) The inverse non-separable transformation unit 31121 derives a transformation matrix secTransMatrix[][] used in the transformation process.

[0188] Specifically, when the prediction mode is intra prediction, the inverse non-separable transform unit 31121 derives lfnstTrSetId from the intra prediction mode IntraPredMode. When the prediction mode is inter prediction, the inverse non-separable transform unit 31121 derives lfnstTrSetId from the block size or the GPM angle index angleIdx (or the GPM mode number merge_gpm_partition_idx). lfnstTrSetId is a number indicating a set of transform matrices for non-separable transform. Note that the transform matrix (or set of transform matrices) corresponding to lfnstTrSetId=K will be referred to as MK hereinafter.

[0189] The non-separable transform can use a plurality of predetermined transform matrices, and suitable transform can be performed by selecting an appropriate transform matrix based on the intra prediction mode or GPM mode.

[0190] Fig. 14 shows a table used for a transform matrix for non-separable transform in intra prediction mode. The inverse non-separable transform unit 31121 derives a matrix for non-separable transform, here a set number (lfnstTrSetId) of a transform matrix, from the intra prediction mode IntraPredMode using the table in Fig. 14. When the table is an array of nstSetTableIntra, lfnstTrSetId may be derived using the following lookup table.

[0191] lfnstTrSetIdx = nstSetTableIntra[IntraPredMode] nstSetTableIntra[] = {2, 2, 3, 3, 3, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 2, 2, 2,, 3, 3, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 2} The following example describes the processing for a lookup table.

[0192] 15 is a table used to derive a transform matrix for non-separable transform in GPM mode. When the prediction mode is inter prediction and GPM prediction, the inverse non-separable transform unit 31121 derives a set number (lfnstTrSetIdx) for non-separable transform from the angle index angleIdx using the table nstSetTableGPMangle1.

[0193] lfnstTrSetIdx = nstSetTableGPMangle1[angleIdx] nstSetTableGPMangle1[] = {2, 2, 3, 3, 3, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 2, 2, 2,, 3, 3, 3, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 2} In other words, with a range A of 33.75 (three 11.25 degree angles in GPM) and a range B of 11.25*5=56.25 degrees as units, the allocation is A, B, A, B, A, B, A, B (or in matrix terms: M2, M3, M2, M1, M2, M3, M2, M1). 11.25 degrees is an angle obtained by dividing 360 degrees into 32 parts. Range A is used around the horizontal and vertical directions (angleIdx = 0, 8, 16, 24), and the same transformation matrix M2 is used. Range B is a different transformation matrix for each range shifted by 90 degrees. Here, M3 and M1 are used. The same transformation matrix is ​​used for the range shifted by 180 degrees (angle shifted by 180 degrees). In other words, if the absolute value difference of angleIdx is a difference meaning a difference of 180 degrees (here, 16), then lfnstTrSetIdx uses the same value.

[0194] Figure 16 shows another example of a table that derives lfnstTrSetIdx from the angle index angleIdx in GPM mode. lfnstTrSetId is assigned for each of four angleIdx values, i.e., a 45-degree range consisting of four 11.25-degree angles. nstSetTableGPMangle2[] = {0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1} Note that the start position of the unit is not limited to angleIdx = 0; the following table, which shifts the start position, may also be used. nstSetTableGPMangle2[] = {0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0} nstSetTableGPMangle2[] = {0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0} nstSetTableGPMangle2[] = {0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0} In the above, the transformation matrix is ​​the same when the angle is shifted by 90 degrees (the difference in angleIdx is 8).

[0195] As shown in another example in Figure 16, lfnstTrSetId is assigned for two angleIdx values, i.e., a range of 22.5 degrees, which is the sum of two 11.25-degree angles. In this case, the transformation matrix remains the same even when the angle is shifted by 90 degrees. nstSetTableGPMangle3[] ={0, 0, 1, 1, 2, 2, 3, 3, 0, 0, 1, 1, 2, 2, 3, 3. 0, 0, 1, 1, 2, 2, 3, 3, 0, 0, 1, 1, 2, 2, 3, 3} Alternatively, the following table, with shifted positions, may also be used. nstSetTableGPMangle3[] = {0, 1, 1, 2, 2, 3, 3, 0, 0, 1, 1, 2, 2, 3, 3. 0, 0, 1, 1, 2, 2, 3, 3, 0, 0, 1, 1, 2, 2, 3, 3, 0} In the above, the transformation matrix is ​​the same when the angle is shifted by 90 degrees (the difference in angleIdx is 8).

[0196] Figure 17 shows another example of a table that derives lfnstTrSetIdx from the angle index angleIdx in GPM mode. lfnstTrSetId is assigned for each of four angleIdx values, i.e., a 45-degree range consisting of four 11.25-degree angles. nstSetTableGPMangle4[] = {0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3} Note that the start position of the unit is not limited to angleIdx = 0; the following table, which shifts the start position, may also be used. nstSetTableGPMangle4[] = {0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0} nstSetTableGPMangle4[] = {0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0} nstSetTableGPMangle4[] = {0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 0, 0, 0} In the above, the transformation matrix is ​​the same when the angle is shifted by 180 degrees (the difference in angleIdx is 16).

[0197] As shown in another example in Figure 17, lfnstTrSetId may be assigned in units of two angleIdx values, i.e., a range of 22.5 degrees, which is the sum of two 11.25-degree angles. nstSetTableGPMangle5[] = {0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7. 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7} Alternatively, the following shifted table may be used: nstSetTableGPMangle5[] = {0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7. 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 0} In the above, the transformation matrix is ​​the same even when shifted by 180 degrees (the difference in angleIdx is 16).

[0198] 18 shows a table used to derive a transform matrix for non-separable transform in GPM mode. When the prediction mode is inter prediction and GPM prediction, the inverse non-separable transform unit 31121 derives a set number (lfnstTrSetId) for non-separable transform from the GPM partition information merge_gpm_partition_idx using the table nstSetTableGPMangleS. Here, 0 to 3 are used as lfnstTrSetId for intra prediction, and 4 to 6 are used as lfnstTrSetId for inter prediction (here, GPM). nstSetTableGPMangleS[] = {5, 5, 6, 6, 6, 6, 6, 5, 5, 5, 4, 4, 4, 4, 5, 5, 5, 6,6, 6, 6, 6, 6, 5, 5, 5, 4, 4, 4, 4, 5, 5} The above configuration has the effect of enabling efficient coding by using different transformation matrices depending on the mode number of intra prediction and the angle index of inter prediction.

[0199] 19 is a table used to derive a transform matrix for non-separable transform in GPM mode. When the prediction mode is inter prediction and GPM prediction, the inverse non-separable transform unit 31121 derives a set number (lfnstTrSetId) for non-separable transform from GPM partition information merge_gpm_partition_idx using the table nstSetTableGPMpart.

[0200] lfnstTrSetId = nstSetTableGPMpart[merge_gpm_partition_idx] The inverse non-separable transform unit 31121 derives the transform matrix secTranMatrix[][] from the set number of the non-separable transform (lfnstTrSetId), lfnst_idx indicating the transform matrix of the non-separable transform, and the non-separable transform size nStSize (nTrS).

[0201] The predicted image generated by the GPM has a directionality indicated by the GPM's angleIdx, and the same directionality may remain in the GPM's residual. With the above configuration, when using the GPM mode, a non-separable transform that utilizes the directionality of the GPM's division pattern can further concentrate the transform coefficients of the residual in the low-frequency region, thereby improving coding efficiency.

[0202] (S2201: Setting transform size and input / output size) For an inverse non-separable transform, the size of the inverse non-separable transform (4x4 or 8x8), the number of output transform coefficients, the number of transform coefficients to be applied, and the number of sub-blocks to which the inverse non-separable transform is applied are derived according to the size of the TU (width W, height H). The sizes of a 4x4 and 8x8 inverse non-separable transform are indicated by nStSize = 4 and 8, respectively. The sizes of a 4x4 and 8x8 inverse non-separable transform may also be referred to as RST4x4 and RST8x8, respectively. Hereinafter, nStOutSize represents the number of output transform coefficients, nonZeroSize represents the number of transform coefficients to be applied (input transform coefficients), and numStX and numStY represent the number of sub-blocks to which the inverse non-separable transform is applied.

[0203] In the inverse non-separable transform, when the TU is of a predetermined size or larger, 48 transform coefficients are output by the inverse non-separable transform of RST8x8. Otherwise, 16 transform coefficients are output by the inverse non-separable transform of RST4x4. When the TU is 4x4, 16 transform coefficients are derived from 8 transform coefficients using RST4x4, and when the TU is 8x8, 48 transform coefficients are derived from 8 transform coefficients using RST8x8. In other cases, 16 or 48 transform coefficients are output according to the size of the TU from 16 transform coefficients.

[0204] When both W and H are 8 or more, log2StSize = 3, nStOutSize = 48. In other cases, log2StSize = 2, nStOutSize = 16. nStSize = 1 << log2StSize. When both W and H are 4, or in the case of 8x8, nonZeroSize = 8. In other cases, nonZeroSize = 16. Note that the input nonZeroSize of LFNST is not limited to 8 and 16. For example, it may be 12 or the like. The output nStOutSize is also not limited to 16 and 48, and may be 32, 36, 64, etc.

[0205] numStX = (nTbH == 4 && nTbW > 8)? 2 : 1. numStY = (nTbW == 4 && nTbH > 8)? 2 : 1. Note that non-separable transform may not be performed on multiple sub-blocks and numStX = numStY may always be used.

[0206] (S2202: Rearrange into a one-dimensional array) The inverse non-separable transform unit 31121 processes by rearranging some of the transform coefficients d[][] of the TU into a one-dimensional array u[] once. Specifically, the inverse non-separable transform unit derives u[] by referring to the transform coefficients for x = 0..nonZeroSize - 1 from the two-dimensional transform coefficients d[][] of the target TU indicated by the region RU. xC and yC are positions on the TU and are derived from the position x of the array DiagScanOrder indicating the scan order and the array u[].

[0207] xC = (xSbIdx<<log2StSize) + DiagScanOrder[log2StSize][log2StSize][x][0] yC = (ySbIdx<<log2StSize) + DiagScanOrder[log2StSize][log2StSize][x][1] u[x] = d[xC][yC] Note that the range copied to the one-dimensional array is called region RU.

[0208] (S2203: Application of conversion processing) The inverse non-separable conversion unit 31121 performs a conversion on u[] with a length of nonZeroSize using the conversion matrix secTransMatrix[][], and derives a one-dimensional array coefficient v'[] with a length of nStOutSize as the output.

[0209] Furthermore, the inverse non-separable conversion unit 31121 performs a sum-of-products operation of the conversion matrix and the one-dimensional variable u[] as shown in the following formula.

[0210] v[i] = Clip3(CoeffMin, CoeffMax, Σ(secTransMatrix[i][j]*u[j]+64)>>7) Here, Σ is the sum from j = 0..nonZeroSize - 1. Also, i performs processing for 0..nStSize - 1. CoeffMin and CoeffMax indicate the value range of the conversion coefficient.

[0211] (S2204: Two-dimensional arrangement of the one-dimensional array after conversion processing) The inverse non-separable conversion unit 31121 arranges the coefficients v'[] of the converted one-dimensional array at a predetermined position within TU again. The arrangement method may be changed according to predModeIntra.

[0212] Specifically, when predModeIntra <= 34, the following processing may be applied.

[0213] d[x][y] = (y<4)? v[x+(y<<log2StSize)] : ((x<4)? v[32+x+((y-4)<<2)] : d[x][y]) In other cases, the following formula is applied.

[0214] d[x][y] = (x < 4)? v[y+(x << log2StSize)] : ((y < 4)? v[32 + y+((x - 4) << 2)] : d[x][y]). Also, the determination of the above branch may be "predModeIntra <= 34" or "predModeIntra == INTRA_LT_CCLM" or "INTRA_T_CCLM or INTRA_L_CCLM", etc.

[0215] (Details of non-separable transform processing) FIG. 24 is a flowchart showing the processing flow when performing non-separable transform by GPM.

[0216] (S2101) When using non-separable transform (lfnst_idx!= 0), transition to S2102A. Otherwise, skip the non-linear transform and transition to S2300 to perform separable transform.

[0217] (S2102A) In the case of intra prediction mode, transition to S2103A, derive lfnstTrSetId from IntraPredMode, and derive the transform matrix secTranMatrix[][] from lfnstTrSetId and the block size.

[0218] (S2102B) When not in the intra prediction mode (GPM mode), transition to S2102B, derive lfnstTrSetId from angleIdx, and derive the transform matrix secTranMatrix[][] from lfnstTrSetId and the block size. Note that when performing non-separable transform (lfnst_idx!= 0) is only in the intra prediction mode or GPM mode, since it is determined to be in the GPM mode when not in the intra prediction mode, the determination of S2102B can be skipped.

[0219] (S2200) Perform non-separable transform using the transform matrix secTranMatrix[][]. Specifically, the processing from S2201 to S2204 described above may be performed.

[0220] (S2300) Perform further separable transform on the non-separable transformed transform coefficients.

[0221] The adder 312 generates a decoded image for the block by adding, for each pixel, the predicted image for the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311. The adder 312 stores the decoded image for the block in the reference picture memory 306 and also outputs it to the loop filter 305.

[0222] (Configuration of Video Encoding Device) Next, the configuration of the video encoding device 11 according to this embodiment will be described. Fig. 7 is a block diagram showing the configuration of the video encoding device 11 according to this embodiment. The video encoding device 11 includes a prediction image generation unit 101, a subtraction unit 102, a transformation / quantization unit 103, an inverse quantization / inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, a coding parameter determination unit 110, a parameter coding unit 111, a prediction parameter derivation unit 120, and an entropy coding unit 104.

[0223] The predicted image generation unit 101 generates a predicted image for each CU. The predicted image generation unit 101 includes the inter predicted image generation unit 309 and the intra predicted image generation unit already described, and therefore a description thereof will be omitted.

[0224] The subtraction unit 102 generates a prediction error by subtracting the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation / quantization unit 103.

[0225] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing them. The transform / quantization unit 103 outputs the quantized transform coefficients to the parameter coding unit 111 and the inverse quantization / inverse transform unit 105.

[0226] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 in the video decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to the adder .

[0227] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes an outline of the operation of each module.

[0228] The header encoding unit 1110 performs encoding processing of parameters such as header information, division information, prediction information, and quantized transform coefficients.

[0229] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information and the like.

[0230] The CU encoding unit 1112 encodes CU information, prediction information, division information, and the like.

[0231] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0232] The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter prediction parameters (predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX), intra prediction parameters, and quantized transform coefficients to the parameter encoding unit 111.

[0233] The entropy coding unit 104 receives the quantized transform coefficients and coding parameters (division information, prediction parameters) from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output a coded stream Te.

[0234] The prediction parameter derivation unit 120 is a means including the inter-prediction parameter coding unit 112 and an intra-prediction parameter coding unit, and derives intra-prediction parameters and intra-prediction parameters from the parameters input from the coding parameter determination unit 110. The derived intra-prediction parameters and intra-prediction parameters are output to the parameter coding unit 111.

[0235] (Configuration of inter prediction parameter coding unit) As shown in Fig. 8 , the inter prediction parameter coding unit 112 includes a parameter coding control unit 1121 and an inter prediction parameter derivation unit 303. The inter prediction parameter derivation unit 303 has the same configuration as the video decoding device. The parameter coding control unit 1121 includes a merge index derivation unit 11211 and a vector candidate index derivation unit 11212.

[0236] The merge index derivation unit 11211 derives merge candidates and the like, and outputs them to the inter prediction parameter derivation unit 303. The vector candidate index derivation unit 11212 derives predicted vector candidates and the like, and outputs them to the inter prediction parameter derivation unit 303 and the parameter coding unit 111.

[0237] (Configuration of Intra Prediction Parameter Encoding Unit) The intra prediction parameter encoding unit includes a parameter encoding control unit and an intra prediction parameter derivation unit. The intra prediction parameter derivation unit has the same configuration as the video decoding device.

[0238] However, unlike the video decoding device, the inputs to the inter prediction parameter derivation unit 303 and the intra prediction parameter derivation unit are the coding parameter determination unit 110 and the prediction parameter memory 108 , and output to the parameter coding unit 111 .

[0239] The adder 106 generates a decoded image by adding, for each pixel, the pixel values ​​of the predicted block input from the predicted image generation unit 101 and the prediction errors input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0240] The loop filter 107 applies deblocking filtering, SAO, and ALF to the decoded image generated by the adder 106.

[0241] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0242] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined location for each current picture and CU.

[0243] The coding parameter determination unit 110 selects one set of coding parameters from among multiple sets of coding parameters. The coding parameters are the above-mentioned QT, BT, or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0244] The coding parameter determination unit 110 calculates an RD cost value indicating the magnitude of the information amount and the coding error for each of the multiple sets. The RD cost value is, for example, the sum of the code amount and the value obtained by multiplying the squared error by a coefficient λ. The code amount is the information amount of the coded stream Te obtained by entropy coding the quantization error and the coding parameters. The squared error is the sum of the squares of the prediction errors calculated by the subtraction unit 102. The coefficient λ is a preset real number greater than zero. The coding parameter determination unit 110 selects the set of coding parameters that minimizes the calculated cost value. The coding parameter determination unit 110 outputs the determined coding parameters to the parameter coding unit 111 and the prediction parameter derivation unit 120.

[0245] Note that a portion of the video encoding device 11 and video decoding device 31 in the above-described embodiments, such as the entropy decoding unit 301, parameter decoding unit 302, loop filter 305, predicted image generation unit 308, inverse quantization / inverse transform unit 311, adder 312, predicted image generation unit 101, subtractor 102, transform / quantization unit 103, entropy encoding unit 104, inverse quantization / inverse transform unit 105, loop filter 107, encoding parameter determination unit 110, parameter encoding unit 111, and prediction parameter derivation unit 120, may be implemented by a computer. In this case, a program for implementing this control function may be recorded on a computer-readable recording medium, and the program may be read and executed by a computer system. Note that the term "computer system" used here refers to a computer system built into either the video encoding device 11 or the video decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0246] Furthermore, part or all of the video encoding device 11 and video decoding device 31 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the video encoding device 11 and video decoding device 31 may be individually implemented as a processor, or part or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0247] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.

[0248] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data obtained by coding image data, and a video coding device that generates coded data obtained by coding image data, and can also be suitably applied to the data structure of coded data generated by a video coding device and referenced by the video decoding device.

[0249] (Cross-reference to related applications) This application claims the benefit of priority to Japanese Patent Application No. 2021-091071, filed on May 31, 2021, the entire contents of which are incorporated herein by reference.

[0250] 31 Image decoding device 301 Entropy decoding unit 302 Parameter decoding unit 303 Inter prediction parameter derivation unit 30377 GPM prediction unit 305, 107 Loop filter 306, 109 Reference picture memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 309 Inter prediction image generation unit 30952 GPM synthesis unit 311, 105 Inverse quantization / inverse transform unit 312, 106 Addition unit 11 Image coding device 102 Subtraction unit 103 Transform / quantization unit 104 Entropy coding unit 110 Coding parameter determination unit 111 Parameter coding unit 112 Inter prediction parameter coding unit 120 Prediction parameter derivation unit

Claims

1. A video decoding device comprising: a parameter decoding unit that decodes parameters from encoded data; a predicted image generating unit that derives a predicted image according to a prediction mode; and a non-separable transform unit that performs non-separable transform on a prediction error, The parameter decoding unit decodes parameters indicating at least a prediction mode and a non-separable transformation, In a case where the prediction mode is an inter prediction mode and the GPM generates a predicted image by performing weighting processing on each pixel of two predicted images according to the position of the pixel, the parameter decoding unit decodes division information of the GPM, derives an angle variable from the division information, the predicted image generator generates the predicted image of the GPM using weighting parameters derived from the angle variables; When the parameter indicating the non-separable transformation indicates that the non-separable transformation is on, the non-separable transformation unit derives a non-separable transformation matrix based on the division information or the angle variable of the GPM.

2. The video decoding device of claim 1, characterized in that the non-separable transformation unit derives a non-separable transformation matrix based on (1) block size, (2) the division information or the angle variable of the GPM, and (3) parameters indicating the non-separable transformation.

3. The video decoding device described in Claim 1, characterized in that the parameter decoding unit has different possible values ​​for the parameter indicating the non-separable transformation depending on whether the prediction mode is the GPM or not.

4. The video decoding device of claim 1, characterized in that the parameter decoding unit has a different value of the context index of the parameter indicating the non-separable transformation when the prediction mode is an inter prediction mode and when it is not.

5. A video encoding device comprising: a parameter encoding unit that encodes parameters; a predicted image generating unit that derives a predicted image according to a prediction mode; and a non-separable transform unit that performs non-separable transform on a prediction error, The parameter coding unit codes parameters indicating at least a prediction mode and a non-separable transformation, In a case where the prediction mode is an inter prediction mode and the GPM generates a predicted image by performing weighting processing on each pixel of two predicted images according to the position of the pixel, the parameter encoding unit derives an angle variable from partition information and encodes the partition information of the GPM; the predicted image generator generates the predicted image of the GPM using weighting parameters derived from the angle variables; When the parameter indicating the non-separable transformation indicates that the non-separable transformation is on, the non-separable transformation unit derives a non-separable transformation matrix based on the division information or the angle variable of the GPM.

6. A computer-readable recording medium having recorded thereon a program for causing a computer to decode a moving image, The program is configured to: Decoding parameters indicative of at least a prediction mode and a non-separable transform from the encoded data; In the case where the prediction mode is an inter prediction mode and the GPM generates a predicted image by performing weighting processing on each pixel of two predicted images according to a pixel position, a step of decoding split information of the GPM and deriving an angle variable from the split information; generating the predicted image of the GPM using weighting parameters derived from the angular variables; and if the parameter indicating the non-separable transformation indicates that the non-separable transformation is on, deriving a non-separable transformation matrix based on the segmentation information or the angle variable of the GPM; A recording medium for causing the execution of the above-mentioned.