Image decoding device, image coding device, image decoding method, and image coding method

Abstract

To provide a moving image decoding device and a moving image coding device, enabling a determination whether or not it has a processing capacity of a post filter using a neural network filter.SOLUTION: In a moving image decoding device 30 of a moving image transmission system, an image decoding device for decoding an image from a coding data comprises: a header decoding part that derives a filter strength for a filter processing and a bit depth; and a loop filter that contains a neutral network (NN) filter part deriving input image information for designating a format of an input tensor of a NN model. The NN filter part derives one part of the input tensor by using the image on the basis of the input image information, sets additional information to an another one part to the input tensor, derives an output tensor by executing a filter processing of the NN by using the input tensor, derives the image after filtering from the output tensor, and derives the additional information by using the filter strength and the bit depth.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an image decoding device, an image encoding device, an image decoding method, and an image encoding method. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates a decoded image by decoding the encoded data.

[0003] Specific examples of video encoding methods include H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such a video coding method, images (pictures) constituting a video are divided into slices obtained by dividing the images, coding tree units (CTUs) obtained by dividing the slices, coding tree units (CTUs) obtained by dividing the coding tree units, and so on. The coding unit (sometimes called a coding unit (CU)) that is to be encoded, and The coding unit is divided into transform units (TUs), which are managed in a hierarchical structure, and the coding unit is encoded / decoded for each CU.

[0005] In such a video coding method, a predicted image is usually generated based on a locally decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the predicted image from the input image (original image) is coded. Methods for generating a predicted image include inter-prediction and intra-prediction.

[0006] Moreover, Non-Patent Document 1 can be cited as a recent example of a video encoding and decoding technique.

[0007] H.274 includes a number of features that allow the transmission of image characteristics, display methods, and timing information simultaneously with the encoded data. Supplementary enhancement information (SEI) is specified for this purpose.

[0008] In Non-Patent Documents 1, 2, and 3, the SEI that transmits the topology and parameters of the neural network filter used as a post-filter is explicitly described as A method of explicitly specifying the information and a method of indirectly specifying the information as reference information are disclosed. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] B. Choi, Z. Li, W. Wang, W. Jiang, X. Xu, S. Wenger and S. Liu, “AHG9 / AHG11: SEI messages for carriage of neural network information for post-filtering,” JVET-V0091 [Non-Patent Document 2] MM Hannuksela, EB Aksu, F. Cricri, HR Tavakoli and M. Santamaria, "AHG9: On post-filter SEI", JVET-X0112 [Non-Patent Document 3] MM Hannuksela, M. Santamaria, F. Cricri, EB Aksu and HR Tavakoli, "AHG9: On post-filter SEI", JVET-Y0115 Summary of the Invention [Problem to be solved by the invention]

[0010] However, in both Non-Patent Document 1 and Non-Patent Document 2, the input and output of the post-filter processing are The problem is that it is not explicitly defined.

[0011] In Non-Patent Document 2, the output color space and chrominance subsampling are not specified, and there is a problem in that it is not possible to specify from the additional information what kind of output will be obtained.

[0012] In Non-Patent Document 1 and Non-Patent Document 2, it is possible to analyze the types of input and output tensors from the topology of the neural network, but it is not possible to specify the relationship between the tensor channels and color components. For example, it is not possible to define how the luminance channel and chrominance channel are set in the input tensor, or in what color space the output tensor after processing is to be output. Therefore, there is a problem in that it is not possible to specify and execute processing. Also, 4: Chrominance subsampling of 0:0, 4:2:0, 4:2:2, and 4:4:4 allows for the separation of the luminance and chrominance components of an image. width , but I can't specify how to process them to derive the input tensor. Also, I can't specify how to generate an image from the output tensor depending on the chrominance subsampling.

[0013] In Non-Patent Document 3, the input and output are 4:2:0 chrominance subsampling, the input tensor is 10 channels, , the channels and color components of the tensor only if the output tensor is 6-channel. However, there is a problem that the processing cannot be performed for formats other than these. Also, there is a problem that it cannot handle models that do not require the input of additional information. [Means for solving the problem]

[0014] An image decoding device according to one embodiment of the present invention is an image decoding device that decodes an image from encoded data, and includes a header decoding unit that derives a filter strength and bit depth for filtering, and a neural network filter unit that derives input image information that specifies the format of an input tensor for a neural network model, wherein the neural network filter unit derives a part of the input tensor using an image based on the input image information, sets additional information in another part of the input tensor, derives an output tensor by performing neural network filtering using the input tensor, and derives a filtered image from the output tensor, and the additional information is derived using the filter strength and the bit depth.

[0015] An image coding device according to one embodiment of the present invention is an image coding device that codes an image, and includes a header coding unit that derives a filter strength and bit depth for filtering, and a neural network filter unit that derives input image information that specifies the format of an input tensor for a neural network model, wherein the neural network filter unit derives a part of the input tensor using an image based on the input image information, sets additional information in another part of the input tensor, derives an output tensor by performing neural network filtering using the input tensor, and derives a filtered image from the output tensor, and the additional information is derived using the filter strength and the bit depth.

[0016] An image decoding method according to one embodiment of the present invention is an image decoding method for decoding an image from encoded data, which is characterized in that it derives a filter strength and bit depth for filtering, derives input image information that specifies the format of an input tensor for a neural network model, derives a part of the input tensor using an image based on the input image information, sets additional information in another part of the input tensor, derives an output tensor by performing filtering processing of a neural network using the input tensor, and derives a filtered image from the output tensor.

[0017] An image coding method according to one embodiment of the present invention is an image coding device that codes an image, which derives a filter strength and bit depth for filtering, derives input image information that specifies the format of an input tensor of a neural network model, derives a part of the input tensor using an image based on the input image information, sets additional information in another part of the input tensor, derives an output tensor by performing filtering of a neural network using the input tensor, and derives a filtered image from the output tensor, and the additional information is derived using the filter strength and the bit depth. Effect of the Invention

[0018] With this configuration, the neural network model specified by the URI can be analyzed. This makes it possible to refer to the complexity of the neural network model specified by the NN filter without having to perform a calculation using a neural network filter, which has the effect of making it easier to determine whether the video decoding device has the processing capacity of a post-filter using a neural network filter. [Brief description of the drawings]

[0019] [Figure 1] 1 is a schematic diagram showing a configuration of a video transmission system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a hierarchical structure of encoded data. [Diagram 3] 1 is a conceptual diagram of an image to be processed in the video transmission system according to the present embodiment. [Figure 4] FIG. 2 is a conceptual diagram showing an example of a reference picture and a reference picture list. [Diagram 5] FIG. 1 is a schematic diagram showing a configuration of an image decoding device. [Figure 6] 11 is a flowchart illustrating a schematic operation of an image decoding device. [Figure 7] FIG. 1 is a block diagram showing a configuration of an image encoding device. [Figure 8] 13 is a schematic diagram showing a configuration of an inter-prediction parameter encoding unit. [Figure 9] FIG. 11 is a diagram showing the configuration of a syntax table of an NN filter SEI in this embodiment. [Figure 10] 13 is a diagram showing the configuration of a syntax table of an NN filter SEI when the parameter type of a neural network model is described by dividing it into a numerical value type and a bit width. FIG. [Figure 11] 13 is a diagram showing the configuration of a syntax table of an NN filter SEI when the bit width of a parameter type of a neural network model is described in logarithmic representation. FIG. [Figure 12] FIG. 13 is a flowchart showing the process of the NN filter unit 611. [Figure 13] FIG. 13 is a diagram showing the structure of a syntax table of a post-filter purpose SEI. [Figure 14] FIG. 13 is a diagram showing the configuration of a neural network of an NN filter unit 611. [Figure 15] FIG. 1 is a diagram showing an NNR encoding device and a decoding device. [Figure 16] FIG. 13 is a diagram illustrating an example of the syntax of the NN filter SEI. [Figure 17] 1 is a table showing an input data format defined in the NN filter. [Figure 18] 1 is a table showing an output data format defined by the NN filter. [Figure 19] 11 is a table showing a method of converting a decoded image into an input data format defined by an NN filter. [Figure 20] 1 is a table showing a method of converting an output data format defined by an NN filter into an output image. [Figure 21] FIG. 13 is a diagram illustrating an example of the syntax of the NN filter SEI. [Figure 22] FIG. 1 is a diagram showing the relationship between the number of dimensions of an input tensor (the number of input channels) and the conversion method, and the number of dimensions of an output tensor (the number of output channels) and the conversion method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0021] FIG. 1 is a schematic diagram showing the configuration of a video transmission system according to this embodiment.

[0022] The video transmission system 1 encodes images having different resolutions, the resolutions of which have been converted. The video transmission system 1 comprises a video encoding device 10, a network 21, and a video It comprises an image decoding device 30 and an image display device 41 .

[0023] The video encoding device 10 is made up of a resolution conversion device (resolution conversion unit) 51, an image encoding device (image encoding unit) 11, an inverse conversion information creation device (inverse conversion information creation unit) 71, and an inverse conversion information encoding device (inverse conversion information encoding unit) 81.

[0024] The video decoding device 30 includes an image decoding device (image decoding unit) 31, a resolution inverse conversion device (resolution inverse conversion unit) 61, and an inverse conversion information decoding device (inverse conversion information decoding unit) 91.

[0025] The resolution conversion device 51 converts the resolution of an image T included in a video to generate an image of a different resolution. to the image coding device 11. The resolution conversion device 51 also supplies inverse conversion information indicating whether or not the resolution of the image is converted to the image coding device 11. If the information indicates resolution conversion, the video coding device 10 sets resolution conversion information ref_pic_resampling_enabled_flag, which will be described later, to 1, and includes it in a sequence parameter set SPS (Sequence Parameter Set) of the coded data Te for coding.

[0026] The inverse conversion information generating device 71 generates inverse conversion information based on an image T1 included in a video. The inverse conversion information is generated based on an input image T1 before resolution conversion and an image Td1 after resolution conversion, encoding, and decoding. The additional information is information that indicates what to select.

[0027] Inverse transformation information is input to the inverse transformation information encoding device 81. The inverse transformation information encoding device 81 encodes the inverse transformation information to generate encoded inverse transformation information, and transmits the encoded inverse transformation information to the network 21.

[0028] A variable resolution image T2 is input to the image encoding device 11. The image encoding device 11 encodes image size information of the input image in PPS units using the framework of RPR (Reference Picture Resampling), and transmits the encoded image size information to the image decoding device 31.

[0029] In FIG. 1, the inverse transformation information coding device 81 is not connected to the image coding device 11, but the inverse transformation information coding device 81 and the image coding device 11 may communicate necessary information as appropriate.

[0030] The network 21 transmits the coded inverse transformation information and the coded data Te to the image decoding device 31. A part or all of the coded inverse transformation information is coded as supplemental enhancement information SEI. The network 21 may be the Internet, a wide area network (WAN), a local area network (LAN), or a similar network. ) or a combination thereof. The network 21 is not necessarily limited to a two-way communication network, and may be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be replaced by a storage medium on which the encoded data Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

[0031] The image decoding device 31 decodes each of the encoded data Te transmitted by the network 21 to generate a variable resolution decoded image Td 1 and supplies it to the resolution inverse conversion device 61 .

[0032] The inverse conversion information decoding device 91 decodes the coded inverse conversion information transmitted by the network 21 to generate inverse conversion information, and supplies the generated inverse conversion information to the resolution inverse conversion device 61 .

[0033] 1, the inverse transformation information decoding device 91 is illustrated separately from the image decoding device 31, but the inverse transformation information decoding device 91 may be included in the image decoding device 31. For example, the inverse transformation information decoding device 91 may be included in the image decoding device 31 separately from each functional unit of the image decoding device 31. Also, although not connected to the image decoding device 31 in FIG. 1, the inverse transformation information decoding device 91 and the image decoding device 31 may communicate necessary information as appropriate.

[0034] When the resolution conversion information indicates resolution conversion, the resolution inverse conversion device 61 generates a decoded image of the original size by inversely converting the resolution-converted image through super-resolution processing using a neural network based on the image size information included in the encoded data.

[0035] The image display device 41 receives one or more decoded images Td2 from the resolution inverse conversion device 61. The image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. When the display device has a high processing power, it displays a high quality image, and when the display device has a lower processing power, it displays an image that does not require a high processing power or display power.

[0036] FIG. 3 is a conceptual diagram of an image to be processed in the video transmission system shown in FIG. It is a diagram showing the change in the resolution of the image over time. However, in FIG. 3 does not distinguish whether the image is encoded or not. FIG. 3 shows an example of reducing the resolution and transmitting the image to the image decoder 31 in the processing process of the moving image transmission system. As shown in FIG. 3, usually, the resolution conversion device 51 performs a conversion to reduce the resolution of the image in order to reduce the amount of information of the transmitted information.

[0037] <Operator> The operators used in this specification are described below.

[0038] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR and |= is an OR assignment operator, and || represents a logical OR.

[0039] x? y : z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0040] Clip3(a, b, c) is a function that clips c to a value between a and b (inclusive). If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (where a <= b).

[0041] abs(a) is a function that returns the absolute value of a.

[0042] Int(a) is a function that returns the integer value of a.

[0043] floor(a) is a function that returns the largest integer less than or equal to a.

[0044] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0045] a / d represents the division of a by d (truncating the decimal part). ​​

[0046] a^b represents power(a, b). When a = 2, it is equal to 1<<b.

[0047] <Structure of Encoded Data Te> Prior to the detailed description of the image encoding device 11 and the image decoding device 31 according to this embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described.

[0048] FIG. 2 is a diagram showing the hierarchical structure of data in the encoded data Te. The encoded data Te exemplarily includes a sequence and a plurality of pictures constituting the sequence. FIG. 2 shows an encoded video sequence that defines a sequence SEQ, an encoded pi cture that defines a picture PICT, an encoded slice that defines a slice S, an encoded slice slice data that defines slice data, an encoded tree unit included in the encoded slice data, and an encoded unit included in the encoded tree unit.

[0049] (Encoded Video Sequence) In the encoded video sequence, a set of data that the image decoding device 31 refers to in order to decode the sequence SEQ to be processed is defined. As shown in FIG. 2, the sequence SEQ includes a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an Adaptation Parameter Set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0050] In the video parameter set VPS, in a moving image composed of a plurality of layers, A set of coding parameters common to a plurality of video streams and a set of coding parameters associated with a plurality of layers and each individual layer contained in the video stream are defined.

[0051] The sequence parameter set SPS specifies a set of coding parameters that the image decoding device 31 refers to in order to decode the target sequence. For example, the width and height of a picture are Note that multiple SPSs may exist. In that case, one of the multiple SPSs is selected from the PPS. Select.

[0052] Here, the sequence parameter set SPS includes the following syntax elements: ref_pic_resampling_enabled_flag: A flag that specifies whether or not to use a function for varying resolution (resampling) when decoding each image included in a single sequence that references the target SPS. In other words, this flag indicates that the size of the reference picture referenced in generating a predicted image changes between each image indicated by a single sequence. When the value of this flag is 1, the resampling function is enabled. is applied, if it is 0 it is not applied. pic_width_max_in_luma_samples: The maximum width of any image in a sequence. This syntax element specifies the width of the image in luminance block units. The value of this syntax element must not be 0 and must be an integer multiple of Max(8, MinCbSizeY). Here, MinCbSizeY is a value determined by the minimum size of a luminance block. pic_height_max_in_luma_samples: This syntax element specifies the height of the image with the maximum height among the images in a single sequence, in units of luminance blocks. The value of this syntax element must not be 0 and must be an integer multiple of Max(8, MinCbSizeY). is required. sps_temporal_mvp_enabled_flag: The time motion detection flag for decoding the target sequence. A flag that specifies whether or not to use vector prediction. If the value of the flag is 1, time If the value is 0, temporal motion vector prediction is not used. In addition, by defining this flag, it is possible to prevent the coordinate position to be referenced from shifting when, for example, referring to a reference picture with a different resolution.

[0053] The picture parameter set PPS specifies the number of pictures to be decoded for each picture in the target sequence. A set of coding parameters to be referred to by the image decoding device 31 is defined. For example, the reference value of the quantization width (pic_init_qp_minus26) used in decoding a picture and the application of weighted prediction are defined. A flag (weighted_pred_flag) indicating the PPS is included. Note that there may be multiple PPSs. In this case, one of multiple PPSs is selected from each picture in the target sequence.

[0054] Here, the picture parameter set PPS includes the following syntax elements: · pic_width_in_luma_samples: A syntax element that specifies the width of the target picture. The value of this syntax element is required to be a non-zero, an integer multiple of Max(8, MinCbSizeY), and a value equal to or less than pic_width_max_in_luma_samples. pic_height_in_luma_samples: Syntax element that specifies the height of the target picture. The value of this syntax element is required to be a non-zero, an integer multiple of Max(8, MinCbSizeY), and a value equal to or less than pic_height_max_in_luma_samples. · conformance_window_flag: Conformance (cropping) window offset A flag indicating whether the parameter is to be subsequently notified and where the conformance window is to be displayed. If this flag is 1, the parameter is notified. If it is 0, the conformance window offset parameter is not present. This indicates that. conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, conf_win_bottom_offset: offset values ​​for specifying the left, right, top, and bottom positions of the picture output by the decoding process with respect to the rectangular area specified by the output picture coordinates. In addition, if the value of conformance_window_flag is 0, the values ​​of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are estimated to be 0. scaling_window_flag: A flag indicating whether or not a scaling window offset parameter exists in the target PPS, and is a flag related to the specification of the output image size. When this flag is 1, it indicates that the parameter exists in the PPS, and when this flag is 0, it indicates that the parameter does not exist in the PPS. In addition, when the value of ref_pic_resampling_enabled_flag is 0, the value of scaling_window_flag is also required to be 0. scaling_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, scaling_win_bottom_offset: Syntax elements that specify the offsets applied to the image size for scaling ratio calculation in luma samples for the left, right, top, and bottom positions of the target picture, respectively. If the value of scaling_window_flag is 0, the values ​​of scaling_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, and scaling_win_bottom_offset are inferred to be 0. It is also required that the value of scaling_win_left_offset + scaling_win_right_offset is less than pic_width_in_luma_samples, and the value of scaling_win_top_offset + scaling_win_bottom_offset is less than pic_height_in_luma_samples.

[0055] The width PicOutputWidthL and height PicOutputHeightL of the output picture are derived as follows:

[0056] PicOutputWidthL = pic_width_in_luma_samples - (scaling_win_right_offset + scaling_win_left_offset) PicOutputHeightL = pic_height_in_pic_size_units - (scaling_win_bottom_offset +scaling_win_top_offset) (Subpicture) A picture may be further divided into rectangular sub-pictures. The size of a sub-picture may be a multiple of the CTU. A sub-picture is defined as a set of an integer number of consecutive tiles in both the vertical and horizontal directions. In other words, a picture is divided into rectangular tiles, and a subpicture is defined as a set of rectangular tiles. A subpicture may be defined using the IDs of the top left tile and the bottom right tile of the subpicture. Also, the slice header contains sh_subpic_id, which indicates the ID of the subpicture. may include:

[0057] (Encoded Picture) The coded picture defines a set of data to be referenced by the image decoding device 31 in order to decode the picture PICT to be processed. As shown in FIG. 2, the picture PICT is PH, including slices 0 to NS-1 (NS is the total number of slices contained in the picture PICT) .

[0058] In the following, when there is no need to distinguish between slices 0 to NS-1, the subscripts of the signs are The same applies to other data to which subscripts are added that are included in the encoded data Te described below.

[0059] The picture header contains the following syntax elements: pic_temporal_mvp_enabled_flag: The image of the slice associated with the picture header. A flag that specifies whether or not to use temporal motion vector prediction for center prediction. If the value of the flag is 0, the syntax element If the flag has a value of 1, the decoding of the slice associated with the picture header is restricted so that temporal motion vector prediction is not used. This flag indicates that temporal motion vector prediction is used for the 3D image. If this flag is not specified, its value is assumed to be 0.

[0060] (Coded Slice) In the case of the coded slice, the image decoding device 31 refers to the coded slice S to decode the slice S to be processed. As shown in Figure 2, a slice consists of a slice header and and contains slice data.

[0061] The slice header includes a group of coding parameters to be referred to by the image decoding device 31 in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates a slice type is an example of a coding parameter included in the slice header.

[0062] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction when encoding, (2) a P slice that uses uni-prediction (L0 prediction) or intra prediction when encoding, and (3) a B slice that uses uni-prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction when encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, when referring to P or B slice, it refers to a slice including a block that can use inter prediction.

[0063] In addition, the slice header may include a reference to a picture parameter set PPS (pic_parameter_set_id).

[0064] (Encoded slice data) The coded slice data defines a set of data to be referenced by the image decoding device 31 in order to decode the slice data to be processed. The slice data is the coded slice data shown in FIG. As shown in the header, it contains a CTU, which is a fixed-size (e.g., 64x64) block that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0065] (coding tree unit) FIG. 2 shows a set of data that the image decoding device 31 refers to in order to decode a CTU to be processed. The CTU performs the coding process by recursive quad tree (QT), binary tree (BT), or ternary tree (TT) division. The BT division and TT division are collectively called multi-tree division (MT (Multi Tree) division). A node of the tree structure obtained by recursive quadtree division is called a coding node. The intermediate nodes of the tree are coding nodes, and the CTU itself is defined as the top coding node. can be.

[0066] The CT includes, as CT information, a CU split flag (split_cu_flag) indicating whether or not to perform CT splitting, a QT split The flags include a QT split flag (qt_split_cu_flag) indicating whether or not to split the MT, an MT split direction (mtt_split_cu_vertical_flag) indicating the split direction of the MT, and an MT split type (mtt_split_cu_binary_flag) indicating the split type of the MT. split_cu_flag, qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

[0067] Different trees may be used for luminance and chrominance. The type of tree is indicated by treeType. For example, when using a common tree for luminance (Y, cIdx=0) and chrominance (Cb / Cr, cIdx=1,2), the common single tree is indicated by treeType=SINGLE_TREE. When using two different trees (DUAL trees) for luminance and chrominance, the luminance tree is indicated by treeType=DUAL_TREE_LUMA and the chrominance tree is indicated by treeType=DUAL_TREE_CHROMA.

[0068] (Encoding Unit) FIG. 2 shows data to be referenced by the image decoding device 31 in order to decode the coding unit to be processed. Specifically, a CU consists of a CU header CUH, prediction parameters, and transformation parameters. The CU header includes a quantization transform coefficient, a quantization metric, etc. The CU header specifies a prediction mode, etc.

[0069] The prediction process may be performed on a CU basis, or on a sub-CU basis by further dividing a CU. If the size of a CU and a sub-CU are the same, there is one sub-CU in the CU. If the size of a CU is larger than that of a sub-CU, the CU is divided into sub-CUs. For example, if the CU is 8x8 and the sub-CU is 4x4, the CU is divided into 2 parts horizontally and 2 parts vertically, into 4 sub-CUs.

[0070] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times or between layer images).

[0071] Transformation and quantization are performed in units of CUs, but quantization coefficients are stored in units of subblocks such as 4x4. It may be entropy coded.

[0072] (Prediction parameters) The predicted image is derived from prediction parameters associated with the block, which include intra-prediction and inter-prediction parameters.

[0073] Hereinafter, prediction parameters of inter prediction will be described. Inter prediction parameters are composed of prediction list use flags predFlagL0 and predFlagL1, reference picture indexes refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. predFlagL0 and predFlagL1 are flags indicating whether or not a reference picture list (L0 list, L1 list) is used, and when the value is 1, the corresponding reference picture list is used. Note that in this specification, when "a flag indicating whether or not XX" is written, a flag other than 0 (for example, 1) is XX, 0 is not XX, and 1 is treated as true and 0 is treated as false in logical negation, logical product, etc. (similarly below). However, in an actual device or method, other values ​​can be used as true and false values.

[0074] Syntax elements for deriving inter prediction parameters include, for example, an affine flag affine_flag used in the merge mode, a merge flag merge_flag, a merge index merge_idx, an MMVD flag mmvd_flag, and an index for selecting a reference picture used in the AMVP mode. Inter prediction identifier inter_pred_idc, reference picture index refIdxLX, predicted vector index mvp_LX_idx for deriving a motion vector, difference vector mvdLX, motion vector There is a rule precision mode amvr_mode.

[0075] (See picture list) The reference picture list is a list of reference pictures stored in the reference picture memory 306. FIG. 4 is a conceptual diagram showing an example of a reference picture and a reference picture list. In the conceptual diagram of an example of a reference picture in FIG. 4, the rectangles represent pictures and the arrows represent the positions of the pictures. Reference relationship, the horizontal axis is time, and I, P, and B in the rectangle are intra picture, uni-prediction picture, and bi-prediction picture, respectively. The numbers in the predicted pictures and rectangles indicate the decoding order. As shown in the figure, the decoding order of the pictures is I0, P1, B2, B3, B4, and the display order is I0, B3, B2, B4, P1. In FIG. 4, picture B3 13 shows an example of a reference picture list of a target picture (a target picture). A reference picture list is a list indicating candidates for a reference picture, and one picture (slice) may have one or more reference picture lists. In the example shown in the figure, a target picture B3 has L0 lists RefPicList0 and Each CU has two reference picture lists: L1 list RefPicList1 and L2 list RefPicList2. refIdxLX indicates which picture in the picture list RefPicListX (X=0 or 1) is actually referenced. The figure shows an example where refIdxL0=2 and refIdxL1=0. Note that LX is a notation method used when there is no distinction between L0 prediction and L1 prediction, and hereafter, parameters for the L0 list and parameters for the L1 list will be distinguished by replacing LX with L0 and L1.

[0076] (Merge prediction and AMVP prediction) The prediction parameter decoding (encoding) method includes a merge prediction mode and an AMVP (Advanced Motion Vector Prediction) mode, and merge_flag is a flag for identifying these. The merge prediction mode is a mode in which the prediction list usage flag predFlagLX, the reference picture index refIdxLX, and the motion vector mvLX are not included in the encoded data, but are derived from the prediction parameters of the neighboring blocks that have already been processed. The AMVP mode is a mode in which inter_pred_idc, refIdxLX, and mvLX are included in the encoded data. Note that mvLX is encoded as mvp_LX_idx, which identifies the prediction vector mvpLX, and the difference vector mvdLX. In addition to the merge prediction mode, there may be an affine prediction mode and an MMVD prediction mode.

[0077] The inter_pred_idc is a value indicating the type and number of reference pictures, and takes one of the values ​​PRED_L0, PRED_L1, and PRED_BI. PRED_L0 and PRED_L1 are managed by the L0 list and the L1 list, respectively. PRED_BI is managed by the L0 list and the L1 list. 1 shows bi-prediction using two reference pictures.

[0078] merge_idx is the prediction parameter candidate (merge candidate) derived from the processed block. (comment) is to be used as a prediction parameter for the current block.

[0079] (Motion Vector) mvLX indicates the amount of shift between blocks on two different pictures. The prediction vector and difference vector related to mvLX are called mvpLX and mvdLX, respectively.

[0080] (Inter prediction identifier inter_pred_idc and prediction list usage flag predFlagLX) The relationship between inter_pred_idc, predFlagL0, and predFlagL1 is as follows, and they can be converted to each other: inter_pred_idc = (predFlagL1<<1)+predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 In addition, the inter prediction parameter may use a prediction list usage flag or an inter prediction identifier. Furthermore, the determination using the prediction list usage flag may be replaced with a determination using the inter prediction identifier. Conversely, the determination using the inter prediction identifier may be replaced with a determination using the prediction list usage flag.

[0081] (Configuration of an image decoding device) The configuration of an image decoding device 31 (FIG. 5) according to this embodiment will be described.

[0082] The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, and a prediction image A generation unit (prediction image generation device) 308, an inverse quantization and inverse transformation unit 311, an addition unit 312, a prediction parameter The image decoding device 320 is configured to include a data derivation unit 320. In some configurations, the device 31 does not include a loop filter 305 .

[0083] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022. The CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice headers (slice information) from the encoded data. The CT information decoding unit 3021 decodes the CT from the encoded data. The CU decoding unit 3022 decodes the CU from the encoded data. The TU decoding unit 3024 extracts QP update information (quantization correction value) and quantized prediction error (residual_coding) from the encoded data when a prediction error is included in the TU. Decrypt.

[0084] The TU decoding unit 3024 decodes the QP update information and the quantized prediction error from the encoded data when the mode is other than the skip mode (skip_mode==0). More specifically, when skip_mode==0, the TU decoding unit 3024 decodes the flag cu_cbp indicating whether or not the current block includes a quantized prediction error, and decodes the quantized prediction error when cu_cbp is 1. When cu_cbp does not exist in the encoded data, In this case, it is derived as 0.

[0085] The TU decoding unit 3024 decodes the index mts_idx indicating the transformation base from the encoded data. In addition, the TU decoding unit 3024 decodes an index stIdx indicating the use of a secondary transform and a transform base from the encoded data. When stIdx is 0, it indicates that a secondary transform is not applied, when it is 1, it indicates one of the transforms in a set (pair) of secondary transform bases, and when it is 2, it indicates that one of the above-mentioned pair is applied. The other transformation is shown below.

[0086] The predicted image generating unit 308 includes an inter predicted image generating unit 309 and an intra predicted image generating unit 310. It is composed of:

[0087] The prediction parameter derivation unit 320 is configured to include an inter prediction parameter derivation unit 303 and an intra prediction parameter derivation unit 304 .

[0088] The entropy decoding unit 301 performs entropy decoding on the encoded data Te input from the outside. The decoding process is then performed to decode each code (syntax element). There are two types of entropy coding: one is to use a context (probability model) that is adaptively selected according to the type of syntax element and the surrounding circumstances to code the syntax element as a variable length code, and the other is to use a predefined table or formula to code the syntax element as a variable length code. The former, CABAC (Context Adaptive Binary Arithmetic Coding), uses the CABAC state (dominant symbol) of the context to code the syntax element as a variable length code. The type of ball (0 or 1) and the probability state index (pStateIdx) that specifies the probability are stored in memory. The entropy decoding unit 301 initializes all CABAC states at the beginning of a segment (tile, CTU row, slice). The entropy decoding unit 301 converts the syntax elements into a binary string (Bin String) and decodes each bit of the Bin String. When a context is used, a context index ctxInc is derived for each bit of the syntax element, the bit is decoded using the context, and the CABAC state of the used context is updated. Bits that do not use text are decoded with equal probability (EP, bypass) and used to derive ctxInc and CABAC states. is omitted. The decoded syntax elements include prediction information for generating a predicted image, a prediction error for generating a difference image, and the like.

[0089] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code is, for example, a prediction mode predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. Which code is to be decoded is controlled by the parameter decoding unit 302. This is done based on instructions from the meter decoder 302 .

[0090] (Basic flow) FIG. 6 is a flowchart illustrating a schematic operation of the image decoding device 31.

[0091] (S1100: Decode Parameter Set Information) The header decoding unit 3020 decodes parameter set information such as the VPS, SPS, and PPS from the encoded data.

[0092] (S1200: Decode slice information) The header decoding unit 3020 decodes slice header information from the encoded data. Decode (slice information).

[0093] Hereinafter, the image decoding device 31 performs the steps S1300 to S5000 for each CTU included in the target picture. The process is repeated to derive a decoded image for each CTU.

[0094] (S1300: Decode CTU Information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0095] (S1400: Decode CT Information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0096] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data. Issued.

[0097] (S1510: Decode CU information) The CU decoding unit 3022 decodes CU information, prediction information, and TU division from the encoded data. The flag split_transform_flag, the CU residual flags cbf_cb, cbf_cr, cbf_luma, etc. are decoded.

[0098] (S1520: Decode TU information) When a TU includes a prediction error, the TU decoding unit 3024 decodes the encoded TU information. The QP update information, the quantization prediction error, and the transform index mts_idx are decoded from the data. Note that the QP update information is derived from the quantization parameter predicted value qPpred, which is the predicted value of the quantization parameter QP. This is the difference value.

[0099] (S2000: Generation of predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on the prediction information.

[0100] (S3000: Inverse Quantization and Inverse Transformation) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing for each TU included in the target CU.

[0101] (S4000: Decoded image generation) The adder 312 receives a predicted image from the predicted image generator 308 and The prediction error supplied from the inverse quantization and inverse transform unit 311 is added to the A decoded image is generated.

[0102] (S5000: Loop Filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

[0103] (Configuration of inter-prediction parameter derivation unit) The inter-prediction parameter derivation unit 303 (motion vector derivation device) derivates the prediction parameter stored in the prediction parameter memory 307 based on the syntax elements input from the parameter decoding unit 302. The inter prediction parameter derivation unit 303 and the AMVP prediction parameter derivation unit 3032, which are its internal elements, and the merge The prediction parameter derivation unit 3036, the affine prediction unit 30372, the MMVD prediction unit 30373, the GPM unit 30377, the DMVR unit 30537, and the MV addition unit 3038 are means common to the image encoding device and the image decoding device. These may be collectively referred to as a motion vector derivation unit (motion vector derivation device).

[0104] The header decoding unit 3020 and the scale parameter derivation unit 30378 included in the header encoding unit 1110 , derive the horizontal scaling ratio of the reference picture RefPicScale[i][j][0], the vertical scaling ratio of the reference picture RefPicScale[i][j][1], and RefPicIsScaled[i][j], which indicates whether the reference picture is scaled or not, where i is the reference picture. Let j denote whether the list is an L0 list (i=0) or an L1 list (i=1), and let j be the value of the L0 reference picture list or the L1 reference picture list (reference picture), then we derive it as follows: RefPicScale[i][j][0] = ((fRefWidth << 14)+(PicOutputWidthL >> 1)) / PicOutputWidthL RefPicScale[i][j][1] = ((fRefHeight << 14)+(PicOutputHeightL >> 1)) / PicOutputHeightL RefPicIsScaled[i][j] = (RefPicScale[i][j][0] != (1<<14)) || (RefPicScale[i][j][1] != (1<<14)) Here, the variable PicOutputWidthL is the horizontal scale when the coded picture is referenced. The variable PicOutputHeightL is a value used when calculating the scaling ratio, and is calculated by subtracting the left and right offset values ​​from the number of pixels in the horizontal direction of the luminance of the coded picture. The variable PicOutputHeightL is a value used when calculating the vertical scaling ratio when the coded picture is referenced, and is calculated by subtracting the top and bottom offset values ​​from the number of pixels in the vertical direction of the luminance of the coded picture. The variable fRefWidth Let fRefHight be the value of PicOutputWidthL of reference picture list value j of list i, and let variable fRefHight be the value of PicOutputHeightL of reference picture list value j of list i.

[0105] (MV addition section) The MV addition unit 3038 calculates mvLX by adding the mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded mvdLX. The addition unit 3038 outputs the calculated mvLX to the inter predicted image generation unit 309. and output to the predicted parameter memory 307: mvLX[0] = mvpLX[0] + mvdLX[0] mvLX[1] = mvpLX[1] + mvdLX[1] The loop filter 305 is a filter provided in the coding loop, and is used to remove block distortion and ringing. The loop filter 305 is a filter that removes distortion and improves image quality. The loop filter 305 applies a deblocking filter, a sample adaptive offset (SAO), and an adaptive filter to the decoded image of the CU generated by the adder 312. Apply a filter such as an automatic loop filter (ALF).

[0106] The DF unit 601 is a bS deriving unit that derives the strength bS of the deblocking filter in units of pixels, boundaries, and lines. The output unit 602 performs deblocking filter processing to reduce block noise. It is composed of a data section 602.

[0107] The DF unit 601 derives an edge degree edgeIdc indicating whether there is a partition boundary, a prediction block boundary, or a transform block boundary in the input image resPicture before NN processing (processing of the NN filter unit 601) and a maximum filter length maxFilterLength of the deblocking filter. Furthermore, the strength bS of the deblocking filter is calculated from edgeIdc, the transform block boundary, and the coding parameters. The coding parameters include, for example, a prediction mode CuPredMode, a BDPCM prediction mode intra_bdpcm_luma_flag, a flag indicating whether the prediction mode is an IBC prediction mode, a motion vector, a reference picture, These are flags such as tu_y_coded_flag and tu_u_coded_flag that indicate whether a non-zero coefficient exists in a transform block. edgeIdc and bS may take values ​​of 0, 1, and 2, or may take other values.

[0108] The reference picture memory 306 stores the decoded image of the CU in a predetermined Store it in position.

[0109] The prediction parameter memory 307 stores prediction parameters at a predetermined location for each CTU or CU. Specifically, the prediction parameter memory 307 stores parameters decoded by the parameter decoding unit 302, parameters derived by the prediction parameter derivation unit 320, and the like.

[0110] The predicted image generation unit 308 receives the parameters derived by the prediction parameter derivation unit 320. The predicted image generation unit 308 also reads a reference picture from the reference picture memory 306. The predicted image generation unit 308 generates a prediction image based on the parameters and the reference picture (reference picture A reference picture block (a reference block) is used to generate a predicted image of a block or sub-block. Here, a reference picture block is a set of pixels on a reference picture (usually rectangular, so called a block), and is an area to be referenced in order to generate a predicted image.

[0111] When predMode indicates an inter prediction mode, the inter prediction image generating unit 309 Using the inter-prediction parameters and the reference pictures input from the prediction parameter derivation unit 303, A predicted image of the block or sub-block is generated by inter prediction.

[0112] (Motion Compensation) The motion compensation unit 3091 (interpolated image generation unit 3091) receives the input from the inter prediction parameter derivation unit 303. Based on the input inter prediction parameters (predFlagLX, refIdxLX, mvLX), an interpolated image (motion compensated image) is generated by reading a reference block from the reference picture memory 306. The reference block is a block shifted by mvLX from the position of the target block on the reference picture RefPicLX specified by refIdxLX. If mvLX does not have integer precision, an interpolated image is generated by applying a filter called a motion compensation filter for generating pixels at decimal positions.

[0113] The motion compensation unit 3091 first derives an integer position (xInt, yInt) and a phase (xFrac, yFrac) corresponding to coordinates (x, y) in the prediction block using the following formula: xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x xFrac = mvLX[0]&(MVPREC-1) yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y yFrac = mvLX[1]&(MVPREC-1) Here, (xPb, yPb) are the upper left coordinates of a block of size bW*bH, where x=0...bW-1, y=0...bH-1, and MVPREC indicates the accuracy of mvLX (1 / MVPREC pixel accuracy), e.g., MVPREC=16.

[0114] The motion compensation unit 3091 derives a temporary image temp[][] by performing horizontal interpolation processing on the reference picture refImg using an interpolation filter. In the following, Σ is the sum for k=0..NTAP-1, shift1 is a normalization parameter that adjusts the value range, and offset1=1<<(shift1-1): temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1 Next, the motion compensation unit 3091 performs vertical interpolation on the temporary image temp[][] to generate an interpolated image Pred [][] is derived. The Σ below is the sum for k=0..NTAP-1, and shift2 adjusts the range of values. R Normalization parameter, offset2=1<<(shift2-1): Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2 In the case of bi-prediction, the above Pred[][] is derived for each L0 list and L1 list (called interpolated images PredL0[][] and PredL1[][]), and an interpolated image Pred[][] is generated from PredL0[][] and PredL1[][].

[0115] The motion compensation unit 3091 uses the reference picture derived by the scale parameter derivation unit 30378. The interpolation unit 102 has a function of scaling the interpolated image according to the horizontal scaling ratio RefPicScale[i][j][0] of the reference picture 102 and the vertical scaling ratio RefPicScale[i][j][1] of the reference picture 102.

[0116] When predMode indicates an intra prediction mode, the intra prediction image generation unit 310 The intra prediction unit 303 performs intra prediction using the intra prediction parameters input from the prediction parameter derivation unit 304 and reference pixels read from the reference picture memory 306 .

[0117] The inverse quantization and inverse transform unit 311 (residual decoding unit) inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0118] The inverse quantization and inverse transform unit 311 converts the quantized transform coefficients qd[] [] is scaled (dequantized) by a scaling unit 31111 to obtain the transform coefficients d[][]. .

[0119] The scaling unit 31111 uses the quantization parameter and scaling factor derived in the parameter decoding unit 302 to scale the transform coefficients decoded by the TU decoding unit using coefficient-by-coefficient weighting.

[0120] Here, the quantization parameter qP is derived as follows using the color component cIdx of the target transform coefficient and the joint chrominance residual coding flag tu_joint_cbcr_flag.

[0121] qP = qPY (cIdx==0) qP = qPCb (cIdx==1 && tu_joint_cbcr_flag==0) qP = qPCr (cIdx==2 && tu_joint_cbcr_flag==0) qP = qPCbCr (tu_joint_cbcr_flag!=0) The scaling unit 31111 derives a value rectNonTsFlag relating to the size or shape from the size (nTbW, nTbH) of the target TU.

[0122] rectNonTsFlag = (((Log2(nTbW)+Log2(nTbH)) & 1)==1 && transform_skip_flag[xTbY] [yTbY]==0) The transform_skip_flag is a flag indicating whether or not to skip the transform.

[0123] The scaling unit 31111 is derived in the scaling list decoding unit 3026 (not shown). The following process is performed using the ScalingFactor[][].

[0124] The scaling unit 31111 performs the following when the scaling list is not enabled (scaling_list_enabled_flag==0) or when transform skip is used (transform_skip_flag==1). , m[x][y]=16. In other words, uniform quantization is performed. A flag that indicates whether the queuing list is valid.

[0125] Otherwise (i.e., when scaling_list_enabled_flag==1 and transform_skip_flag==0), the scaling unit 31111 uses the scaling list. Here, m[][] is set as follows.

[0126] m[x][y] = ScalingFactor[Log2(nTbW)][Log2(nTbH)][matrixId][x][y] Here, matrixId is set based on the prediction mode (CuPredMode) of the target TU, the color component index (cIdx), and whether or not a non-separable transform is applied (lfnst_idx).

[0127] The scaling unit 31111 sets the scaling factor ls[x][y] when sh_dep_quant_used_flag is 1. In this case, it is derived using the following formula.

[0128] ls[x][y] = (m[x][y]*quantScale[rectNonTsFlag][(qP+1)%6]) << ((qP+1) / 6) Otherwise (sh_dep_quant_used_flag=0), it may be derived using the following formula:

[0129] ls[x][y] = (m[x][y]*quantScale[rectNonTsFlag][qP%6]) << (qP / 6) Here, quantScale[] = {{ 40, 45, 51, 57, 64, 72}, {57, 64, 72, 80, 90, 102}}. sh_dep_quant_used_flag is a flag that is set to 1 when dependent quantization is performed and set to 0 when not. The value of quantScale is derived from the value of x (x=0..6) using the following formula.

[0130] quantScale[x] = RoundInt(2^(6 / (x-qsoffset))) qsoffset = rectNonTsFlag==0 ? 4 : 2 If qP has a value of 4, then quantScale is 64, where RoundInt is a rounding constant (e.g. 0. 5) and then truncate to an integer.

[0131] The scaling unit 31111 performs inverse quantization by deriving dnc[][] from the product of the scaling factors ls[][] and the decoded transform coefficients TransCoeffLevel.

[0132] dnc[x][y] = (TransCoeffLevel[xTbY][yTbY][cIdx][x][y]*ls[x][y]+bdOffset1) >> bdShift1 where bdOffset1 = 1<<(bdShift1-1) Finally, the scaling unit 31111 clips the dequantized transform coefficients to derive d[x][y].

[0133] d[x][y] = Clip3(CoeffMin, CoeffMax, dnc[x][y]) (formula CLIP-1) CoeffMin and CoeffMax are the minimum and maximum values ​​of clipping and are derived by the following formula:

[0134] CoeffMin = -(1 << log2TransformRange) CoeffMax = (1 << log2TransformRange) - 1 Here, log2TransformRange is a value indicating the range of transform coefficients derived by a method to be described later.

[0135] d[x][y] is transmitted to the inverse core transform unit 31123 or the inverse non-separation transform unit 31121. The transform unit 31121 applies an inverse non-separable transform to the transform coefficients d[][] after inverse quantization and before the core transform.

[0136] The adder 312 adds, for each pixel, the predicted image of the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311 to generate a decoded image of the block. The adder 312 stores the decoded image of the block in the reference picture memory 306 , and also outputs it to the loop filter 305 .

[0137] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0138] The adder 312 adds, for each pixel, the predicted image of the block input from the predicted image generation unit 308 and the prediction error input from the inverse quantization and inverse transform unit 311 to generate a decoded image of the block. The adder 312 stores the decoded image of the block in the reference picture memory 306 , and also outputs it to the loop filter 305 .

[0139] (Example of the configuration of the NN filter unit 611) FIG. 14 shows an interpolation filter using a neural network filter unit (NN filter unit 611). 1 is a diagram illustrating an example of the configuration of a post filter, an interpolation filter, a loop filter, and a post filter.

[0140] The post-processing unit 61 after the video decoding device includes an NN filter unit 611. When outputting an image from the reference picture memory 306, the NN filter unit 611 performs a filter process and outputs the image to the outside. The image may be displayed, written to a file, re-encoded (transcoded), transmitted, etc. The NN filter unit 611 filters the input image using a neural network model. At the same time, it may also perform reduction or enlargement at a constant or rational number factor.

[0141] Here, the neural network model (hereafter referred to as NN model) means the elements and connections (topology) of the neural network, and the parameters (weights, biases) of the neural network. Note that the topology may be fixed, and only the parameters of the neural network model may be switched.

[0142] (Details of the NN filter unit 611) The NN filter uses the input image inSamples and input parameters (e.g., QP, bS, etc.) to Filtering is performed using a neural network model. The input image may be an image for each component, or an image with multiple components as channels. In addition, the input parameters may be assigned to a channel different from the image.

[0143] The NN filter unit may repeatedly apply the following process.

[0144] The NN filter part performs convolution operation (conv, convolution) on inSamples with kernel k[m][i][j]. Then, derive the output image outSamples by adding bias, where nn=0..n-1, xx=0..width-1, and yy=0..height-1.

[0145] outSamples[nn][xx][yy]=ΣΣΣ(k[mm][i][j]*inSamples[mm][xx+i-of][yy+j-of]+bias [nn]) In the case of 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0. In this case, set of=0. In the case of 3x3 Conv, Σ represents the sum of mm=0..m-1, i=0..2, j=0..2. In this case, of=1 is set. n is the number of channels in outSamples, m is the number of channels in inSamples, width is the width of inSamples and outSamples, and height is the height of inSamples and outSamples. of is the padding area around inSamples to make inSamples and outSamples the same size. In the following, when the output of the NN filter is not an image but a value (correction value), the output is represented as corrNN instead of outSamples.

[0146] Note that if you write inTensor and outTensor in CHW format instead of inSamples and outSamples in CWH format, it is equivalent to the following process.

[0147] outTensor[nn][yy][xx]=ΣΣΣ(k[mm][i][j]*inTensor[mm][yy+j-of][xx+i-of]+bias[nn]) Alternatively, a process called Depth wise Conv may be performed, which is expressed by the following formula: Here, nn=0..n-1, xx=0..width-1, and yy=0..height-1.

[0148] outSamples[nn][xx][yy]=ΣΣ(k[nn][i][j]*inSamples[nn][xx+i-of][yy+j-of]+bias[nn]) Σ represents the sum for i and j, respectively. n is the number of channels in outSamples and inSamples, width is the width of inSamples and outSamples, and height is the height of inSamples and outSamples.

[0149] Also, a nonlinear process called Activate, for example, ReLU, may be used. ReLU(x) = x >= 0 ? x : 0 Alternatively, leakyReLU shown in the following formula may be used.

[0150] leakyReLU(x) = x >= 0 ? x : a * x Here, a is a predetermined value, for example, 0.1 or 0.125. In order to perform integer arithmetic, all the values ​​of k, bias, and a above may be integers, and a right shift may be performed after conv.

[0151] In ReLU, 0 is always output for values ​​less than 0, and the input value is output as is for values ​​greater than or equal to 0. On the other hand, in leakyReLU, linear processing is performed with the gradient set by a for values ​​less than 0. In ReLU, the gradient for values ​​less than 0 disappears, so learning may not progress smoothly. In leakyReLU, the gradient for values ​​less than 0 is left, making the above problem less likely to occur. In addition, among the above leakyReLU(x), PReLU, which uses a parameterized value of a, may be used.

[0152] (SEI for neural network model complexity reference) FIG. 9 is a diagram showing the structure of a syntax table of the NN filter SEI of this embodiment. , which contains information on the neural network model complexity. ·nnrpf_id: The identification number of the NN filter. nnrpf_mode_idc: The mode index that indicates how to specify the neural network model to be used for the NN filter. If the value is 0, the NN filter associated with nnrpf_id is is not specified in this SEI message. A value of 1 indicates that the NN filter associated with nnrpf_id is a neural network model identified by the given Uniform Resource Identifier (URI). A URI can be a logical or physical resource. Note that the URI does not need to contain actual data. If the value is 2, the NN associated with the nnrpf_id is used. The filter is represented by an ISO / IEC 15938-17 bit stream contained in this SEI message. A value of 3 indicates that the model is a neural network model associated with nnrpf_id. Indicates that the NN filter being decoded is the neural network model identified in the NN filter SEI message used in the previous decoding and updated with the ISO / IEC 15938-17 bit stream contained in this SEI message. nnrpf_persistence_flag: Specifies the persistence of this SEI message for the current layer. A flag. A value of 0 indicates that this SEI message applies only to the current decoded picture. A value of 1 indicates that this SEI message applies to the current decoded picture and all subsequent pictures. This indicates to the CSS that they should be applied in output order. nnrpf_uri[i]: URI of the neural network model to be used as the NN filter The i is the i-th byte of the UTF-8 string terminated with NULL. When nnrpf_mode_idc==1, the header encoding unit 1110 and the header decoding unit 3020 decode nnrpf_uri, which is a URI indicating the neural network model to be used as the NN filter. A neural network model corresponding to the character string shown in the figure is used for a video encoding device or a video The data is read from a memory provided in the decoding device, or from an external device via a network. ·nnrpf_payload_byte[i]: Indicates the i-th byte of a bitstream conforming to ISO / IEC 15938-17.

[0153] The NN filter SEI is a neural network model complexity information (network model complexity The following syntax elements are included as miscellaneous information: nnrpf_parameter_type_idc: An index indicating the variable type included in the parameters of the NN model. If the value is 0, the NN model uses only integer types. If the value is 1, the NN model uses either floating point types or integer types. nnrpf_num_parameters_idc: An index indicating the number of parameters of the NN model used in the post filter. If the value is 0, it indicates that the number of parameters of the NN model is not defined. If the value is not 0, the following process is performed using nnrpf_num_parameters_idc to estimate the performance of the NN model. Derive the number of parameters.

[0154] The header encoding unit 1110 and the header decoding unit 3020 derive the maximum number of parameters of the NN model, MaxNNParameters, based on nnrpf_num_parameters_idc as follows, and The model complexity information may be encoded and decoded.

[0155] MaxNNParameters = (UNITPARAM << nnrpf_num_parameters_idc) - 1 Here, UNITPARAM is a predetermined constant, and may be UNITPARAM=2048=2^11. Note that the shift operation is equivalent to the exponent and can be less than or equal to it.

[0156] MaxNNParameters = 2 ^(nnrpf_num_parameters_idc+11) - 1 Also, the unit of the parameter number is not in double units, but a combination of double units and 1.5 units as shown below. You may also combine them.

[0157] MaxNNParameters = (nnrpf_num_parameters_idc & 1) ? (UNITPARAM2 << nnrpf_num_parameters_idc)-1 : (UNITPARAM << nnrpf_num_parameters_idc)-1 Here, UNITPARAM2 may be a predetermined constant, which is UNITPARAM*1.5. For example, if UNITPARAM=2048, then UNITPARAM2=3072. The following is also acceptable.

[0158] MaxNNParameters = (nnrpf_num_parameters_idc & 1) : 2^(nnrpf_num_parameters_idc+11)*1.5-1 : 2^(nnrpf_num_parameters_idc+11)-1 That is, in the header encoding unit 1110, the value of nnrpf_num_parameter_idc is The header decoding unit 3020 sets a value such that the number of parameters is equal to or less than MaxNNParameters. The encoded data set as above is decoded.

[0159] In addition, a linear expression may be used to derive MaxNNParameters.

[0160] MaxNNParameters = (UNITPARAM * nnrpf_num_parameters_idc) -1 In this case, UNITPARAM may be 10000. UNITPARAM is preferably a value of 1000 or more, and is preferably a multiple of 10. nnrpf_num_kmac_operations_idc: The number of operations required for postfilter processing. The header encoding unit 1110 and the header decoding unit 3020 obtain MaxNNOperations based on nnrpf_num_kmac_operations_idc as follows: MaxNNOperations is a value indicating the scale. This is the maximum number of operations required to process a stream filter.

[0161] MaxNNOperations = nnrpf_num_kmac_operations_idc * 1000 * pictureWidth * pictureHeight Here, pictureWidth and pictureHeight are the width of the picture input to the post filter. It is the height.

[0162] That is, the video coding apparatus uses the value of nnrpf_num_kmac_operations_idc as a post-filling Set this according to the number of operations required to process the filter.

[0163] In the above, the syntax of the network model complexity information related to the amount of processing defined in units of a specific constant is transmitted, or encoded, or decoded, which has the effect of enabling the complexity to be transmitted concisely. Furthermore, using a multiple of 10 has the effect of making the value easy for humans to understand. Furthermore, if the value is 1000 or more, it becomes possible to appropriately express the scale of the model with a small number of divisions, and transmission can be performed efficiently.

[0164] That is, the syntax indicating the network model complexity information indicates an upper limit on the number of parameters or the number of operations, and the number of parameters or the number of operations is defined in units of an exponential power of 2. Alternatively, the number of parameters or the number of operations may be defined in units of an exponential power of 2 or 1.5 times an exponential power of 2. Also, the number of parameters or the number of operations may be defined in units of a multiple of 10.

[0165] The above further has the effect of enabling efficient transmission of complexity using shorter codes by transmitting, encoding, or decoding the syntax of network model complexity information relating to the processing amount defined by a shift representation or an exponential representation. nnrpf_alignment_zero_bit: A bit for byte alignment. The header encoding unit 1110 and the header decoding unit 3020 encode and decode the code “0” one bit at a time until the bit position reaches a byte boundary.

[0166] nnrpf_operation_type_idc: An index that indicates the restriction of elements or the restriction of topology used in the NN model of the post filter. Depending on the value of the index, for example, the following processing may be performed.

[0167] If the value is 3, the element or topology is limited to the maximum size of the kernel. The size is 5x5, the maximum number of channels is 32, the activation function can be leaky ReLU or only ReLU, and the The maximum level of branching is 3 (excluding skip connections).

[0168] If the value is 2, in addition to the above restrictions, it prohibits the use of leaky Relu in the activation function and skips Prohibit branching other than preconnection (e.g., do not perform U-Net or grouping convolutions).

[0169] A value of 1 further restricts the above, and also disables spatial-to-channel mappings (e.g., Pixel Shuffler) and global average pooling.

[0170] A value of 0 means that the element or topology is not restricted. <Another configuration example 1> The parameter nnrpf_parameter_type_idc indicating network model complexity information may be defined as follows: nnrpf_parameter_type_idc: An index indicating the parameter type of the neural network model. For example, the parameter type may be determined according to the parameter value as follows: If the value is 0, it is defined as 8-bit unsigned integer type, if the value is 1, it is defined as 16-bit unsigned integer type, if the value is 2, it is defined as 32-bit unsigned integer type, if the value is 3, it is defined as 16-bit floating point type (bfloat16), if the value is 4, it is defined as 16-bit floating point type (half precision), if the value is 5, it is defined as 32-bit floating point type (single precision). <Another configuration example 2> FIG. 10 shows the structure of the syntax table of the NN filter SEI with network model complexity information. In this example, the parameter type of the neural network model is defined by dividing it into a numerical value type and a bit width.

[0171] In this configuration example, the definition of the following syntax information in the SEI in FIG. 9 is changed. nnrpf_parameter_type_idc: An index that indicates the numeric type of the neural network model. For example, the numeric type may be determined according to the parameter value as follows: A value of 0 defines an integer type, and a value of 1 defines a floating-point type.

[0172] In addition to the syntax of the SEI in Figure 9, the following syntax information is included: nnrpf_parameter_bit_width_idc: An index indicating the bit width of the neural network model. For example, the bit width may be determined according to the parameter value as follows: A value of 0 is defined as 8 bits, a value of 1 is defined as 16 bits, and a value of 2 is defined as 32 bits. <Another configuration example 3> FIG. 11 shows the structure of the syntax table of the NN filter SEI with network model complexity information. Here, the bit width of the parameter type of the neural network model is defined in logarithmic representation.

[0173] In this configuration example, the following syntax information is used instead of nnrpf_parameter_bit_width_idc. Includes. nnrpf_log2_parameter_bit_width_minus3: Neural network model parameters Based on nnrpf_log2_parameter_bit_width_minus3, the parameter bit width, parameterBitWidth, is calculated as follows: parameterBitWidth = 1 << ( nnrpf_log2_parameter _bit_width_minus3 + 3 ) (SEI decoding and post-filtering) The header decoding unit 3020 decodes the network model complexity information from the SEI message defined in Figures 9 to 11. The SEI is additional information for processing related to decoding, display, and the like.

[0174] 12 is a diagram showing a flowchart of the process of the NN filter unit 611. The NN filter unit 611 performs the following process in accordance with the parameters of the above SEI message. S6001: Read the processing amount and accuracy from the network model complexity information of SEI. S6002: End if the complexity exceeds the processable level of the NN filter unit 611. If not, proceed to S6003. S6003: End if the accuracy exceeds the processable accuracy of the NN filter unit 611. If not, proceed to S6004. S6004: Identify the network model from the SEI and set the topology of the NN filter unit 611 . S6005: Derive network model parameters from the updated information of SEI. S6006: The parameters of the derived network model are read into the NN filter unit 611. S6007: The NN filter unit 611 executes the filtering process and outputs the result to the outside. However, the SEI is not necessarily required to construct luma and chroma samples in the decoding process. I can't.

[0175] (SEI for neural network model data format reference) FIG. 16 is a diagram showing another configuration of the syntax table of the NN filter SEI of this embodiment. This SEI includes information in the form of a neural network model (NN model) data. The same syntax elements as those in the SEI including the information on the complexity of the neural network model already explained are described below. Omit the details nnrpf_input_format_idc: Input tensor identification parameters. NN used in NN filters. This indicates the format of the input data (input tensor) of the model. The header decoding unit 3020 derives the format of the input data (number of channels (NumInChannels), data format) based on the value of nnrpf_input_format_idc, as shown in FIG.

[0176] If nnrpf_input_format_idc==0, the input data format is 1-channel (luminance) 3D data. The luminance channel of the decoded image is used as input data for the NN filter. In this embodiment, the three dimensions of the three-dimensional data are (C, H, W). Although this is defined as an order, the order of dimensions is not limited to this. For example, the order of dimensions is (H, W, C). In this case, since there is only one channel, the data is two-dimensional (H,W) and You may do so.

[0177] If nnrpf_input_format_idc==1, the input data format is 2-channel (color difference) 3D data. The two chrominance channels (U and V) of the decoded image are the inputs of the NN filter. This indicates that it is used as force data.

[0178] If nnrpf_input_format_idc==2, the input data format is 3-channel (luminance and 2 This shows that the luma and two chroma channels of the decoded image are used as input data to the NN filter in 4:4:4 format.

[0179] When nnrpf_input_format_idc==3, the input data format is 6-channel (4 luminance and 2 chrominance) 3D data (3D tensor). It indicates that 4 channels derived from the luminance channel of the decoded image in 4:2:0 format and 2 chrominance channels are used as input data for the NN filter.

[0180] nnrpf_output_format_idc: Output tensor identification parameter. Indicates the format of the output data (NN output data, output tensor) of the NN model used in the NN filter. Header Decoding Unit 3020 The format of the output data (channel) is determined based on the value of nnrpf_output_format_idc, as shown in Figure 18. Derive the number of channels (NumOutChannels, data format).

[0181] If nnrpf_output_format_idc==0, the format of the NN output data is 3D data (3D tensor) with one channel (luminance). This indicates that the output data of the NN filter is used as the luminance channel of the output image.

[0182] If nnrpf_output_format_idc==1, the format of the NN output data is 2-channel (chrominance) 3D data (3D tensor). It indicates that the output data of the NN filter will be used as the two chrominance channels (U and V) of the output image.

[0183] When nnrpf_output_format_idc==2, the format of the NN output data is three-channel (luminance and two chrominance) three-dimensional data (3D tensor). The output data of the NN filter is the luminance of the output image. This indicates that it is used in the 4:4:4 format as a 32-bit D-sub and two chrominance channels.

[0184] When nnrpf_output_format_idc==3, the format of the NN output data is 3D data (3D tensor) with 6 channels (4 luminance and 2 chrominance). The output data of the NN filter is 6-channel One luminance channel and two chrominance channels are derived by integrating four channels of the indicates that it will be used in 4:2:0 format.

[0185] (Post-filter SEI processing) The header encoding unit 1110 and the header decoding unit 3020 process the post filter SEI. The syntax values ​​and variables of the decoding device may be set as variables for filtering. PicWidthInLumaSamples = pps_pic_width_in_luma_samples PicHeightInLumaSamples = pps_pic_height_in_luma_samples ChromaFormatIdc = sps_chroma_format_idc BitDepthY= BitDepthC = BitDepth ComponentSample[cIdx] is a two-dimensional array that stores the cIdx-th decoded pixel value of the decoded image. where pps_pic_width_in_luma_samples, pps_pic_height_in_luma_samples, and sps_chroma_format_idc are syntax values ​​indicating the subsampling of the image width, height, and color components, and BitDepth is the bit depth of the image.

[0186] The header encoding unit 1110 and the header decoding unit 3020 derive the following variables according to ChromaFormatIdc: Put out. SubWidthC = 1, SubHeghtC = 1 (ChromaFormatIdc == 0) SubWidthC = 2, SubHeghtC = 2 (ChromaFormatIdc == 1) SubWidthC = 2, SubHeghtC = 1 (ChromaFormatIdc == 2) SubWidthC = 1, SubHeghtC = 1 (ChromaFormatIdc == 3) The header encoding unit 1110 and the header decoding unit 3020 derive the width and height of the luminance image and the width and height of the chrominance image to be filtered using the following variables. LumaWidth = PicWidthInLumaSamples LumaHeight = PicHeightInLumaSamples ChromaWidth = PicWidthInLumaSamples / SubWidthC ChromaHeight = PicHeightInLumaSamples / SubHeithtC SW = SubWidthC SH = SubHeightC SubWidthC(=SW) and SubHeightC(=SH) indicate the subsampling of color components. is a variable that represents the ratio of color difference resolution to luminance resolution.

[0187] Furthermore, the header encoding unit 1110 and the header decoding unit 3020 may derive the output image width outWidth and height outHeight according to a scale value indicating a ratio as follows. outLumaWidth = LumaWidth * scale outLumaHeight = LumaHeight * scale outChromaWidth = LumaWidth * scale / outSW outChromaHeight = LumaHeight * scale / outSH outSW and outSH are the chrominance subsampling values ​​of the output image.

[0188] (Conversion of post-filter input data to NN) When inputting image data to the NN filter, the NN filter unit 611 converts the decoded image into NN input data inputTensor[][][], which is a three-dimensional array, based on the value of nnrpf_input_format_idc, as shown in FIG. 19 and below. In the following, x and y represent the coordinates of the luminance pixel. For example, the range of x and y in ComponentSample is x=0..LumaWidth-1, y=LumaHeight-1. cx and cy represent the coordinates of the chrominance pixel. The ranges are cx=0..ChromaWidth-1, and cy=ChromaHeight-1. The following NN filter unit 611 processes this range. When nnrpf_input_format_idc is 0 (pfp_component_idc==0), the NN filter unit 611 derives the inputTensor as follows.

[0189] inputTensor[0][y][x] = ComponentSample[0][x][y] If nnrpf_input_format_idc is 1 (pfp_component_idc==1), derive inputTensor as follows: inputTensor[0][cy][cx] = ComponentSample[1][cx][cy] inputTensor[1][cy][cx] = ComponentSample[2][cx][cy] Or: inputTensor[0][y / SH][x / SW] = ComponentSample[1][x / SW][y / SH] inputTensor[1][y / SH][x / SW] = ComponentSample[2][x / SW][y / SH] If nnrpf_input_format_idc is 2 (pfp_component_idc==2), derive inputTensor as follows: inputTensor[0][y][x] = ComponentSample[0][x][y] ChromaOffset = 1<<(BitDepthC-1) inputTensor[1][y][x] = ChromaFormatIdc==0? ChromaOffset : ComponentSample[1][x / SW][y / SH] inputTensor[2][y][x] = ChromaFormatIdc==0? ChromaOffset : ComponentSample [2][x / SW][y / SH] When nnrpf_input_format_idc is 3, inputTensor is derived as follows: inputTensor[0][cy][cx] = ComponentSample[0][cx*2 ][cy*2 ] inputTensor[1][cy][cx] = ComponentSample[0][cx*2+1][cy*2 ] inputTensor[2][cy][cx] = ComponentSample[0][cx*2 ][cy*2+1] inputTensor[3][cy][cx] = ComponentSample[0][cx*2+1][cy*2+1] ChromaOffset = 1<<(BitDepthC-1) inputTensor[4][cy][cx] = ChromaFormatIdc==0? ChromaOffset : ComponentSample[1][cx][cy] inputTensor[5][cy][cx] = ChromaFormatIdc==0? ChromaOffset : ComponentSample[2][cx][cy] If ChromaFormatIdc is 0, the ComponentSample is an image with only the luminance channel. In this case, the NN filter unit 611 may set a constant ChromaOffset derived from the bit depth to the chrominance data portion of the inputTensor. ChromaOffset may be another value such as 0. Alternatively, As shown in the above, the NN input data may be derived according to pfp_component_idc, which will be described later.

[0190] FIG. 22 is a diagram showing the relationship between the number of channels of a tensor and the conversion process.

[0191] Furthermore, the NN filter unit 611 may derive the NN input data according to the number of channels NumInChannels (numTensors) of the input tensor as follows. If NumInChannels==1, derive inputTensor as follows:

[0192] inputTensor[0][y][x] = ComponentSample[0][x][y] If NumInChannels==2, derive inputTensor as follows:

[0193] inputTensor[0][cy][cx] = ComponentSample[1][cx][cy] inputTensor[1][cy][cx] = ComponentSample[2][cx][cy] Or: inputTensor[0][y / SH][x / SW] = ComponentSample[1][x / SW][y / SH] inputTensor[1][y / SH][x / SW] = ComponentSample[2][x / SW][y / SH] If NumInChannels==3, derive inputTensor as follows:

[0194] inputTensor[0][y][x] = ComponentSample[0][x][y] ChromaOffset = 1<<(BitDepthC-1) inputTensor[1][y][x] = ChromaFormatIdc==0? ChromaOffset : ComponentSample[1][x / SW][y / SH] inputTensor[2][y][x] = ChromaFormatIdc==0? ChromaOffset : ComponentSample[2][x / SW][y / SH] When NumInChannels == 6, derive inputTensor as follows.

[0195] inputTensor[0][cy][cx] = ComponentSample[0][cx*2 ][cy*2 ] inputTensor[1][cy][cx] = ComponentSample[0][cx*2+1][cy*2 ] inputTensor[2][cy][cx] = ComponentSample[0][cx*2 ][cy*2+1] inputTensor[3][cy][cx] = ComponentSample[0][cx*2+1][cy*2+1] ChromaOffset = 1<<(BitDepthC-1) inputTensor[4][cy][cx] = ChromaFormatIdc==0? ChromaOffset : ComponentSample[1][cx][cy] inputTensor[5][cy][cx] = ChromaFormatIdc==0? ChromaOffset : ComponentSample[2][cx][cy] As described above, the video decoding device includes a predicted image derivation unit that decodes a predicted image and a residual decoding unit that decodes a residual, and has the effect of reliably and easily deriving an input tensor by deriving an input tensor from a parameter that specifies the number of channels of the input tensor and output tensor of a neural network model, or by deriving an image from the output tensor.

[0196] The NN filter unit 611 separates a one-channel luminance image into four channels according to pixel positions. The NN filter unit 611 may be derived as follows. , 4:2:0, 4:2:2, 4:4:4 differences are absorbed by the variable indicating the chrominance subsampling. Can be processed regardless of sample differences: inputTensor[0][cy][cx] = ComponentSample[0][cx*2 ][cy*2 ] inputTensor[1][cy][cx] = ComponentSample[0][cx*2+1][cy*2 ] inputTensor[2][cy][cx] = ComponentSample[0][cx*2 ][cy*2+1] inputTensor[3][cy][cx] = ComponentSample[0][cx*2+1][cy*2+1] ChromaOffset = 1<<(BitDepthC-1) inputTensor[4][cy][cx] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSample[1][cx*2 / SW][cy*2 / SH] inputTensor[5][cy][cx] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSample[2][cx*2 / SW][cy*2 / SH] In addition, since cx = x / 2 and cy = y / 2, the NN filter unit 611 performs the following for the above-mentioned range of x and y: may be derived as follows: inputTensor[0][y / 2][x / 2] = ComponentSample[0][x / 2*2 ][y / 2*2 ] inputTensor[1][y / 2][x / 2] = ComponentSample[0][x / 2*2+1][y / 2*2 ] inputTensor[2][y / 2][x / 2] = ComponentSample[0][x / 2*2 ][y / 2*2+1] inputTensor[3][y / 2][x / 2] = ComponentSample[0][x / 2*2+1][y / 2*2+1] ChromaOffset = 1<<(BitDepthC-1) inputTensor[4][y / 2][x / 2] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSample[1][x / SW][y / SH] inputTensor[5][y / 2][x / 2] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSample[2][x / SW][y / SH] (Conversion from post-filter NN output data) The NN filter unit 611 derives the output image outSamples from the NN output data outputTensor[][][], which is the output data of the NN filter, in a three-dimensional array based on the value of nnrpf_output_format_idc. Specifically, as shown in FIG. 20 and below, the NN filter unit 611 derives an image as follows based on the value of nnrpf_output_format_idc and the chrominance subsampling values ​​outSW and outSH of the output image. It should be noted that for outSW and outSH, values ​​derived based on OutputChromaFormatIdc decoded from the encoded data are used. Below, x and y represent the coordinates of the luminance pixel of the output image. For example, the ranges of x and y in outputTensor are x = 0..outLumaWidth-1, y = outLumaHeight-1, respectively. cx and cy represent the coordinates of the chrominance pixel of the output image, and the ranges of cx and cy are cx = 0..outChromaWidth-1, cy = 0..outChromaHeight-1, respectively. The following NN filter unit 611 processes this range. outSamplesL, outSamplesCb, and outSamplesCr represent the luminance pixel of the output image, respectively. The three channels represent the intensity channel, the chrominance (Cb) channel, and the color difference (Cr) channel.

[0197] When nnrpf_output_format_idc is 0, the NN filter unit 611 derives outSamplesL as follows.

[0198] outSamplesL[x][y] = outputTensor[0][y][x] When nnrpf_output_format_idc is 1, outSamplesCb and outSamplesCr are derived as follows.

[0199] outSamplesCb[cx][cy] = outputTensor[0][cy][cx] outSamplesCr[cx][cy] = outputTensor[1][cy][cx]. When nnrpf_output_format_idc is 2, outSamplesL, outSamplesCb, and outSamplesC are derived as follows.

[0200] outSamplesL[x][y] = outputTensor[0][y][x] outSamplesCb[x / outSW][y / outSH] = outputTensor[1][y][x] outSamplesCr[x / outSW][y / outSH] = outputTensor[2][y][x] Alternatively, outSamplesCb and outSamplesCr may be derived as follows.

[0201] outSamplesCb[cx][cy] = outputTensor[1][cy*outSH][cx*outSW] outSamplesCr[cx][cy] = outputTensor[2][cy*outSH][cx*outSW] If nnrpf_output_format_idc is 3, outSamplesL is derived as follows: outSamplesL[x / 2*2 ][y / 2*2 ] = outputTensor[0][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2 ] = outputTensor[1][y / 2][x / 2] outSamplesL[x / 2*2 ][y / 2*2+1] = outputTensor[2][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2+1] = outputTensor[3][y / 2][x / 2] Or outSamplesL may be derived as follows: outSamplesL[cx*2 ][cy*2 ] = outputTensor[0][cy][cx] outSamplesL[cx*2+1][cy*2 ] = outputTensor[1][cy][cx] outSamplesL[cx*2 ][cy*2+1] = outputTensor[2][cy][cx] outSamplesL[cx*2+1][cy*2+1] = outputTensor[3][cy][cx] Furthermore, if nnrpf_output_format_idc is 3 and the output image is in 4:2:0 format (the ChromaFormatIdc of the output image is 1, SW=SH=2), outSamplesCb and outSamplesCr are derived as follows: outSamplesCb[cx][cy] = outputTensor[4][cy][cx] outSamplesCr[cx][cy] = outputTensor[5][cy][cx] Or, if the output image is in 4:2:2 format (the ChromaFormatIdc of the output image is 2, SW=2, SH=1), For example, outSamplesCb and outSamplesCr are derived as follows: outSamplesCb[cx][cy / 2*2 ] = outputTensor[4][cy][cx] outSamplesCb[cx][cy / 2*2+1] = outputTensor[4][cy][cx] outSamplesCr[cx][cy / 2*2 ] = outputTensor[5][cy][cx] outSamplesCr[cx][cy / 2*2+1] = outputTensor[5][cy][cx] Alternatively, if the output image is in 4:4:4 format (output image ChromaFormatIdc is 3, SW=SH=1), outSamplesCb, outSamplesCr are derived as follows: outSamplesCb[cx / 2*2 ][cy / 2*2 ] = outputTensor[4][cy][cx] outSamplesCb[cx / 2*2+1][cy / 2*2+1] = outputTensor[4][cy][cx] outSamplesCr[cx / 2*2 ][cy / 2*2 ] = outputTensor[5][cy][cx] outSamplesCr[cx / 2*2+1][cy / 2*2+1] = outputTensor[5][cy][cx] Alternatively, it may be derived as follows: for (j=0; j <outSH; j++) for (i=0; i <outSW; i++) outSamplesCb[cx / outSW*outSW+i][cy / outSH*outSH+j] = outputTensor[4][cy][cx] outSamplesCr[cx / outSW*outSW+i][cy / outSH*outSH+j] = outputTensor[5][cy][cx] In addition, in the case of the YUV4:2:0 format, cx = x / 2 and cy = y / 2, so the NN filter unit 611 may derive outSamplesL, outSamplesCb, and outSamplesCr as follows: outSamplesL[x / 2*2 ][y / 2*2 ] = outputTensor[0][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2 ] = outputTensor[1][y / 2][x / 2] outSamplesL[x / 2*2 ][y / 2*2+1] = outputTensor[2][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2+1] = outputTensor[3][y / 2][x / 2] outSamplesCb[x / outSW][y / outSH] = outputTensor[4][y / 2][x / 2] outSamplesCr[x / outSW][y / outSH] = outputTensor[5][y / 2][x / 2] Incidentally, outSW and outSH may be set to the same as the color difference sampling of the input data and processed.

[0202] outSW = SW outSW = SH FIG. 22 is a diagram showing the relationship between the number of channels of a tensor and the conversion process.

[0203] Furthermore, as shown below, the NN filter unit 611 may derive NN input data as follows depending on the number of channels NumOutChannels(numTensors) of the output tensor.

[0204] If NumOutChannels is 1, then outSamplesL is derived as follows:

[0205] outSamplesL[x][y] = outputTensor[0][y][x] When NumOutChannels is 2, outSamplesCb and outSamplesCr are derived as follows:

[0206] outSamplesCb[cx][cy] = outputTensor[0][cy][cx] outSamplesCr[cx][cy] = outputTensor[1][cy][cx] When NumOutChannels is 3, outSamplesL, outSamplesCb, and outSamplesCr are derived as follows:

[0207] outSamplesL[x][y] = outputTensor[0][y][x] outSamplesCb[x / outSW][y / outSH] = outputTensor[1][y][x] outSamplesCr[x / outSW][y / outSH] = outputTensor[2][y][x] Alternatively, outSamplesCb and outSamplesCr may be derived as follows.

[0208] outSamplesCb[cx][cy] = outputTensor[1][cy*outSH][cx*outSW] outSamplesCr[cx][cy] = outputTensor[2][cy*outSH][cx*outSW] When NumOutChannels is 6, outSamplesL, outSamplesCb, and outSamplesCr are derived as follows.

[0209] outSamplesL[x / 2*2 ][y / 2*2 ] = outputTensor[0][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2 ] = outputTensor[1][y / 2][x / 2] outSamplesL[x / 2*2 ][y / 2*2+1] = outputTensor[2][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2+1] = outputTensor[3][y / 2][x / 2] outSamplesCb[x / outSW][y / outSH] = outputTensor[4][y / 2][x / 2] outSamplesCr[x / outSW][y / outSH] = outputTensor[5][y / 2][x / 2] <Second embodiment> FIG. 21 is a diagram showing another embodiment of a syntax table of the NN filter SEI. In this embodiment, a syntax element nnrpf_additional_input_idc is used in addition to the syntax element nnrpf_io_idc in Non-Patent Document 3. This provides the advantage that the conversion method can be flexibly specified even if the number of input / output types and combinations increases. Note that a description of the same parts as in the first embodiment will be omitted.

[0210] As shown in FIG. 21, when nnrpf_mode_idc is 1 or 2 (indicating that the SEI is new post-filter data), the header decoding unit 3020 decodes the input / output image information nnrpf_io_idc and the additional input information nnrpf_additional_info_idc.

[0211] The header decoder 3020 derives the format of the model's input and output tensors based on the value of nnrpf_io_idc as follows: If nnrpf_io_idc==0: numInChannels = numOutChannels = 1 If nnrpf_io_idc==1: numInChannels = numOutChannels = 2 If nnrpf_io_idc==2: numInChannels = numOutChannels = 3 If nnrpf_io_idc==3: numInChannels = numOutChannels = 4 If nnrpf_io_idc==4: numInChannels = numOutChannels = 6 In this embodiment, the values ​​of numInChannels and numOutChannels are derived to be equal. However, the correspondence between nnrpf_io_idc, numInChannles, and numOutChannels is not limited to this. Alternatively, other combinations may be used. Also, the number of channels may be directly decoded without using nnrpf_io_idc. Also, the same variables may be used without distinguishing between numInChannels and numOutChannels. The conversion process may be performed using the numbers numInOutChannels and numTensors.

[0212] Furthermore, the header decoder 3020 performs input / output based on the value of nnrpf_additional_info_idc. The values ​​of variables useSliceQPY, usebSY, and usebSC, which indicate the use of input channels other than the common image components, are derived. SliceQPY, bSY, bSCb, and bSCr are flags indicating whether or not to use them as additional input channels. In this embodiment, one flag usebSC controls both bSCb and bSCr. Note that SliceQPY is a luminance quantization parameter in a slice to which a certain coordinate belongs. bSY, bSCb, and bSCR are arrays that store the values ​​of the strength (block strength) of the deblocking filter at a certain coordinate in the luminance and chrominance (Cb, Cr) channels of the decoded image, respectively. The above flags are derived from nnrpf_additional_info_idc in the following manner: For example, use the following: If nnrpf_additional_info_idc==0: useSliceQPY = 0, usebSY = 0, usebSC = 0 If nnrpf_additional_info_idc==1: useSliceQPY = 1, usebSY = 0, usebSC = 0 If nnrpf_additional_info_idc==2: useSliceQPY = 0, usebSY = 1, usebSC = 0 If nnrpf_additional_info_idc==3: useSliceQPY = 1, usebSY = 1, usebSC = 0 If nnrpf_additional_info_idc==4: useSliceQPY = 0, usebSY = 0, usebSC = 1 If nnrpf_additional_info_idc==5: useSliceQPY = 1, usebSY = 0, usebSC = 1 If nnrpf_additional_info_idc==6: useSliceQPY = 0, usebSY = 1, usebSC = 1 If nnrpf_additional_info_idc==7: useSliceQPY = 1, usebSY = 1, usebSC = 1 Note that the correspondence between nnrpf_additional_info_idc and useSliceQPY, usebSY, and usebSC is not limited to this, and other combinations may be used.

[0213] Or, use the following to set each bit of nnrpf_additional_indo_idc to a corresponding flag: It may be derived by calculation. useSliceQPY = nnrpf_additional_info_idc & 1 usebSY = (nnrpf_additional_info_idc >> 1) & 1 usebSC = (nnrpf_additional_info_idc >> 2) & 1 Alternatively, the header decoding unit 3020 may not encode nnrpf_additional_indo_idc, but may decode the flags useSliceQPY, usebSY, and usebSC from the encoded data.

[0214] Note that the types of flags are not limited to those described above, and flags may be derived for other information as well. For example, a slice quantization parameter SliceQPC for chrominance, a reference image, a predicted image, or a QP value for each coding block may be used.

[0215] The header decoder 3020 further decodes the size of the processing unit (patch) of the model specified by the SEI. Decode nnrpf_patch_size_minus1, which indicates (number of pixels in the horizontal and vertical directions)-1. The variable patchSize, which represents the size of the patch, is derived using the following formula: patchSize = nnrpf_patch_size_minus1 + 1 Furthermore, the header decoding unit 3020 decodes nnrpf_overlap. nnrpf_overlap indicates the width of the area adjacent to the patch and input to the model together with the patch. In other words, the size of the height x width of the tensor input to the model is (nnrpf_overlap*2+patchSize) x (nnrpf_overlap*2+patchSize) It is a pixel.

[0216] (Another example of details of the NN filter unit 611) In this embodiment, the NN filter unit 611 performs the following steps: STEP 1: Extract input tensors from an image ComponentSample. The method performs three steps: STEP 1: converting the inputTensor into an inputTensor, STEP 2: applying the neural network filtering process postProcessingFilter to the inputTensor and outputting the outputTensor, and STEP 3: deriving an image from the derived outputTensor.

[0217] SliceQPY is a luminance quantization parameter qP in a slice to which a certain coordinate belongs. bSY, bSCb, and bsCR are arrays that store the value of the strength (block strength, bS) of the deblocking filter at a certain coordinate in the luminance and chrominance (Cb, Cr) channels of the decoded image, respectively. bS may be a value derived from the prediction mode of the block, the motion vector, the difference value of the pixel value, etc. in the deblocking filter process.

[0218] (Pseudocode) The NN filter unit 611 may perform the processes in STEP 1, STEP 2, and STEP 3 in units of a predetermined patch size from the upper left coordinate of the image. The following pseudo code is an overview of the process in units of a patch. for(cTop=0; cTop <PicHeightInLumaSamples; cTop+=patchSize*rH) { for (cLeft = 0; cLeft < PicWidthInLumaSamples; cLeft += patchSize * rW) { <STEP1: Conversion to Input Tensor of Image and Additional Information> <STEP2: Application of Filter Processing> <STEP3: Conversion of Output Tensor to Image> } } Here, rH and rW are variables that determine the increment amounts of loop variables related to chrominance subsampling. Also, rH and rW are vertical and horizontal magnification factors for deriving the size of the tensor patch on the image.

[0219] The NN filter unit 611 may be derived as follows using chrominance subsampling. rH = SH rW = SW Alternatively, the NN filter unit 611 may be derived as follows using the number of input channels of the tensor as well. rH = numInChannles == 6? 2 : 1 rW = (numInChannles == 4 || numInChannels == 6)? 2 : 1 Alternatively, the NN filter unit 611 may use the chrominance subsampling inside the tensor and the chrominance subsampling of the input image to derive rW and rH, for example, by the following formula. rH = numInChannels == 2? SH : numInChannles == 6? 2 : 1 rW = numInChannels == 2? SW : (numInChannles == 4 || numInChannels == 6)? 2 : 1 rH and rW indicate that a tensor with a patch size (height x width) of patchSize x patchSize corresponds to a region of patchSize * rW x patchSize * rH on the input luminance image (width x height). An example of processing from STEP 1 to STEP 3 will be described below.

[0220] (STEP 1) The NN filter unit 611 derives patches that include overlaps. If included, the size of the input data is patchSize+nnrpf_overlap*2 x patchSize+nnrpf_overlap*2 pixels in both length and width. The NN filter unit 611 derives each element of one patch including overlap in the input tensor as shown in the following pseudo code.

[0221] for (yP = -nnrpf_overlap; yP <patchSize+nnprf_overlap; yP++) { for (xP = -nnrpf_overlap; xP <patchSize+nnprf_overlap; xP++) { yP1 = yP+nnrpf_overlap xP1 = xP+nnrpf_overlap yT = Clip3(0, PicHeightInLumaSamples-1, cTop+yP*rH) yL = Clip3(0, PicWidthInLumaSamples-1, cLeft+xP*rW) yB = Clip3(0, PicHeightInLumaSamples-1, yT+rH-1) xR = Clip3(0, PicWidthInLumaSamples-1, yL+rW-1) cy = yT / SH cx = yL / SW <Deriving the input tensor: transforming the input image> <Deriving the input tensor: Transforming additional information> } } yP and xP are the height (H) and width (W) loop variables of the tensor. yP1 and xP1 are subscripts offset by the overlap size to reference the elements of the input tensor that contain the overlap region. yT, yL, yB, xR are the yP and cy, cx are the coordinate values ​​on the luminance component corresponding to xP and xP. These variables may be derived as necessary during each process described below. Next, the conversion process of the input image will be described.

[0222] (Transformation of input image) In the loop of STEP 1, the NN filter unit 611 performs the following pseudo code: Transform the input image according to the value of and derive the values ​​of each channel corresponding to the tensor coordinates yP, xP. do: if (numInChannels==1) { inputTensor[0][yP1][xP1] = ComponentSamples[0][yT][yL] ch_pos = 1 } else if (numInChannels==2) { inputTensor[0][yP1][xP1] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSamples[1][cx][cy] inputTensor[1][yP1][xP1] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSamples[2][cx][cy] ch_pos = 2 } else if (numInChannels==3) { inputTensor[0][yP1][xP1] = ComponentSamples[0][yT][yL] inputTensor[1][yP1][xP1] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSamples[1][cx][cy] inputTensor[2][yP1][xP1] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSamples[2][cx][cy] ch_pos = 3 } else if (numInChannels==4) { inputTensor[0][yP1][xP1] = ComponentSamples[0][yL][yT] inputTensor[1][yP1][xP1] = ComponentSamples[0][yL][yB] inputTensor[2][yP1][xP1] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSamples[1][cx][cy] inputTensor[3][yP1][xP1] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSamples[2][cx][cy] ch_pos = 4 } else if (numInChannels==6) { inputTensor[0][yP1][xP1] = ComponentSamples[0][yL][yT] inputTensor[1][yP1][xP1] = ComponentSamples[0][xR][yT] inputTensor[2][yP1][xP1] = ComponentSamples[0][yL][yB] inputTensor[3][yP1][xP1] = ComponentSamples[0][xR][yB] inputTensor[4][yP1][xP1] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSamples[1][cx][cy] inputTensor[5][yP1][xP1] = ChromaFormatIdc==0 ? ChromaOffset : ComponentSamples[2][cx][cy] ch_pos = 6 } In the above, ch_pos = numInChannels may be derived.

[0223] If numInChannels==1, one channel of the input tensor is extracted from the luminance component of the decoded image. The image data of the channel is derived.

[0224] If numInChannels==2, we extract the two chrominance components of the decoded image from the input tensor Two-channel image data is derived.

[0225] If numInChannels==3, 4:4:4 format image data represented by three channels is derived from the luminance component and chrominance component of the decoded image and set to the input tensor.

[0226] If numInChannels==4, the 4:2:2 format image data represented by a 2-channel luminance image and a 2-channel chrominance image is generated from the luminance and chrominance components of the decoded image. Derive and set it to the input tensor.

[0227] When numInChannels==6, the 4:2:0 format image data represented by a 4-channel luminance image and a 2-channel chrominance image is generated from the luminance and chrominance components of the decoded image. Derive and set it to the input tensor.

[0228] In either case, if the decoded image does not have a chrominance component (ChromaFormatIdc==0), the NN filter unit 611 sets ChromaOffset in the chrominance channel. In addition, ch_pos is a subscript that indicates the next available channel in the input tensor. When adding data to the input tensor, it is recommended to use the channel after the value of ch_pos.

[0229] (Derivation of additional information) Next, the NN filter unit 611 adds additional information to the inputTensor based on the additional input information (flag). For example, if useSliceQPY is true, the value of SliceQPY is set to the next channel (inputTensor[ch_pos]) of the inputTensor. Note that when a value is set to the channel, ch_pos Then, if usebSY is true, the value of inputTenso is increased according to the number of channels used. Set the value of bSY at the position corresponding to the patch area to the next channel of r. Furthermore, if usebSC is true and the decoded image has a chrominance component, set the next channel of inputTensor to The bSCb and bSCr values ​​at the position corresponding to the patch area are set. At this time, the NN filter unit 611 may set the SliceQPY, bSY, bSCb, and bSCr values ​​as they are, or may set values ​​that have been converted. An example of deriving the channel of the additional information is shown below: if (useSliceQPY) { inputTensor[ch_pos][yP1][xP1] = 2^((SliceQPY-42) / 6) ch_pos += 1 } if (usebSY) { inputTensor[ch_pos][yP1][xP1] = Clip3(0,(1< <BitDepthY)-1, bSY[yL][yT]< <(BitDepthY-2)) ch_pos += 1 } if (usebSC && InputChromaFormatIdc!=0) { inputTensor[ch_pos][yP1][xP1] = Clip3(0, (1< <BitDepthC)-1, bSCb[cx][cy] << (BitDepthC-2)) ch_pos += 1 inputTensor[ch_pos][yP1][xP1] = Clip3(0, (1< <BitDepthC)-1, bSCr[cx][cy] << (BitDepthC-2)) ch_pos += 1 } As additional information other than the above, the NN filter unit 611 inputs a second image different from the input image. Here is an example of adding to a tensor. The following example uses the flag useRefPicY to add to the reference image refPicY. Here is the process for adding a channel: if (useRefPicY) { inputTensor[ch_pos][yP1][xP1] = refPicY[yL][yT] ch_pos += 1 } The flag useRefPicY, like other additional information flags such as useSliceQPY, may be derived from nnrpf_additional_info_idc or may be decoded from the encoded data. In addition to the reference image, you can also add a channel for the predicted image predPicY for one frame. Similarly: if (usePredPicY) { inputTensor[ch_pos][yP1][xP1] = predPicY[yL][yT] ch_pos += 1 } The way to provide additional information to the post-filter is not limited to the above. Alternatively, the calculated values ​​may be derived as parameters and input to the post-filtering process.

[0230] (STEP 2) In STEP 2, the NN filter unit 611 performs post-filter processing using the derived inputTensor as an input. Then, run the filter and get the output tensor outputTensor that is the result of the filtering: outputTensor = postProcessingFilter(inputTensor) In this embodiment, the height (H) x width (W) size of the output tensor outputTensor is patchSize x patchSize, and does not include the overlapping area. However, Alternatively, you can set the output tensor including the overlap region to a size of patchSize x patchSize. It is acceptable to crop the image to the desired size.

[0231] (STEP 3) In STEP 3, the NN filter unit 611 derives image data from the output tensor outputTensor. The NN filter unit 611 derives each pixel value of the components outSamplesL, outSamplesCb, and outSamplesCr of the output image from the data of the output tensor outputTensor. At this time, as shown in the following pseudo code, the image data is derived based on the number of channels (data format) of the output tensor and the chrominance subsampling of the output image. The pseudo code shown below is the processing of STEP 3. For example: for(yP=0, ySrc=cTop; yP <patchSize; yP++, ySrc+=rH) for(xP=0, xSrc=cLeft; xP<patchSize; xP++, xSrc+=rW) { if (numOutChannels==1) { / / rW=1, rH=1 if (pfp_component_idc!=1) { outSamplesL[xSrc][ySrc] = outputTensor[0][yP][xP] } else { outSamplesL[xSrc][ySrc] = ComponentSamples[0][xSrc][ySrc] } if (OutputChromaFormatIdc!=0 && pfp_component_idc!=0) { cyOut = ySrc / outSH cxOut = xSrc / outSW outSamplesCb[cxOut][cyOut] = ComponentSamples[1][ySrc / SH][xSrc / SW] outSamplesCr[cxOut][cyOut] = ComponentSamples[2][ySrc / SH][xSrc / SW] } } else if (numOutChannels==2) { / / rW=SW,rH=SH for (dy=0; dy<rH, dy++) { for (dx=0; dx<rW, dx++) { outSamplesL[ySrc+dy][xSrc+dx] = ComponentSamples[0][ySrc+dy][xSrc+dx] if (OutputChromaFormatIdc!=0) { cyOut = (ySrc+dy) / outSH cxOut = (xSrc+dx) / outSW if (pfp_component_idc!=0) { outSamplesCb[cxOut][cyOut] = outputTensor[0][yP][xP] outSamplesCr[cxOut][cyOut] = outputTensor[1][yP][xP] } else { outSamplesCb[cxOut][cyOut] = ComponentSamples[1][(ySrc+dy) / SH] [ (xSrc+dx) / SW ] outSamplesCr[cxOut][cyOut] = ComponentSamples[2][(ySrc+dy) / SH] [ (xSrc+dx) / SW ] } } } } } else if (numOutChannels==3) { / / rW=1, rH=1 if (pfp_component_idc!=1) { outSamplesL[xSrc][ySrc] = outputTensor[0][yP][xP] } else { outSamplesL[xSrc][ySrc] = ComponentSamples[0][xSrc][ySrc] } if (OutputChromaFormatIdc!=0) { cyOut = ySrc / outSH cxOut = xSrc / outSW if (pfp_component_idc!=0) { outSamplesCb[cxOut][cyOut] = outputTensor[1][yP][xP] outSamplesCr[cxOut][cyOut] = outputTensor[2][yP][xP] } else { outSamplesCb[cxOut][cyOut] = ComponentSamples[1][xSrc / SW][ySrc / SH] outSamplesCr[cxOut][cyOut] = ComponentSamples[2][xSrc / SW][ySrc / SH] } } } else if (numOutChannels==4) { / / rW=2, rH=1 if (pfp_component_idc!=1) { outSamplesL[xSrc ][ ySrc ] = outputTensor[0][yP][xP] outSamplesL[xSrc+1][ ySrc ] = outputTensor[1][yP][xP] } else { outSamplesL[xSrc ][ ySrc ] = ComponentSamples[0][xSrc ][ ySrc ] outSamplesL[xSrc+1][ ySrc ] = ComponentSamples[0][xSrc+1][ ySrc ] } if (OutputChromaFormatIdc!=0) { for (dx=0; dx<rW; dx++) { cyOut = (ySrc+dy) / outSH cxOut = (xSrc+dx) / outSW if (pfp_component_idc!=0) { outSamplesCb[cxOut][cyOut] = outputTensor[2][yP][xP] outSamplesCr[cxOut][cyOut] = outputTensor[3][yP][xP] } else { outSamplesCb[cxOut][cyOut] = ComponentSamples[1][(xSrc+dx) / SW] [ (ySrc+dy) / SH] outSamplesCr[cxOut][cyOut] = ComponentSamples[2][(xSrc+dx) / SW] [ (ySrc+dy) / SH] } } } } else if (numOutChannels==6) { / / rW=2, rH=2 if (pfp_component_idc!=1) { outSamplesL[xSrc ][ySrc ] = outputTensor[0][yP][xP] outSamplesL[xSrc+1][ySrc ] = outputTensor[1][yP][xP] outSamplesL[xSrc ][ySrc+1] = outputTensor[2][yP][xP] outSamplesL[xSrc+1][ySrc+1] = outputTensor[3][yP][xP] } else { outSamplesL[xSrc ][ySrc ] = ComponentSamples[0][xSrc ][ySrc ] outSamplesL[xSrc+1][ySrc ] = ComponentSamples[0][xSrc+1][ySrc ] outSamplesL[xSrc ][ySrc+1] = ComponentSamples[0][xSrc ][ySrc+1] outSamplesL[xSrc+1][ySrc+1] = ComponentSamples[0][xSrc+1][ySrc+1] } if (OutputChromaFormatIdc!=0) { for (dy=0; dy <rH; dy++) { for (dx=0; dx <rW; dx++) { cyOut = (ySrc+dy) / outSH cxOut = (xSrc+dx) / outSW if (pfp_component_idc != 0) { outSamplesCb[cxOut][cyOut] = outputTensor[4][yP][xP] outSamplesCr[cxOut][cyOut] = outputTensor[5][yP][xP] } else { outSamplesCb[cxOut][cyOut] = ComponentSamples[1][(xSrc+dx) / SW] [(ySrc+dy) / SH] outSamplesCr[cxOut][cyOut] = ComponentSamples[2][(xSrc+dx) / SW] [(ySrc+dy) / SH] } } } } } } } If numOutChannels is 1, the NN filter unit 611 outputs using one channel of the outputTensor. The NN filter unit 611 derives the luminance component of the image. If the output image includes a chrominance component, the NN filter unit 611 copies and uses the chrominance component of the input image.

[0232] When numOutChannels is 2, the NN filter unit 611 outputs using two channels of outputTensor. The NN filter unit 611 derives two chrominance components of the image, and uses the luminance component of the input image as the luminance component of the output image.

[0233] When numOutChannels is 3, the NN filter unit 611 derives the luminance component and two chrominance components of the output image from the outputTensor.

[0234] When numOutChannels is 4, the NN filter unit 611 uses two channels of the outputTensor. The first channel is used to derive the luminance component of the output image, and the other two channels are used to derive the two chrominance components.

[0235] When numOutChannels is 6, the NN filter unit 611 uses four channels of the outputTensor. The first channel is used to derive the luminance component of the output image, and the other two channels are used to derive the two chrominance components.

[0236] Regardless of numOutChannels, if the output image does not have a chrominance component, the chrominance channels of the output tensor are not processed. Also, if numOutChannels is 2 or more, the chrominance channels If numOutChannels is 2, convert the chrominance component format of the output image to match the chrominance component format of the output image using the values ​​outSH and outSW derived from the chrominance subsampling of the output image. The values ​​SH(SubHightC) and SW(SubWidthC) derived from the chrominance subsampling of the input image are also referenced, and the values ​​are converted to conform to the chrominance component format of the output image.

[0237] In addition, if numOutChannels is not 2 and pfp_component_idc is 1, Instead of updating the luminance component of the output tensor, use the luminance component of the input image. Similarly, if numOutChannels is not 1 and pfp_component_idc is 0, the output image The chrominance components of the input image are used instead of updating them with the output tensor.

[0238] In this way, the NN filter unit 611 determines the number of channels of the output tensor and the chrominance sub-sub-sub-sampling of the output image. Based on the sampling, the pixel value of each component of the output image is derived. 611 derives rW*rH pixels on the output image corresponding to one pixel of the output tensor represented by yP and xP. This can be done in a loop (dy, dx) as in the pseudocode above when numOutChannels is 2 or 6, or the loop can be omitted or expanded as in the other cases. may be set to always loop twice for both dy and dx.

[0239] In the above example, the NN filter unit 611 performs multiple inductions for the same color difference coordinates of the output image. However, if the coordinates overlap, the second and subsequent derivations may be omitted. do not have.

[0240] Furthermore, in the above example, cTop and cLeft are configured to increment in increments of patchSize*rH and patchSize*rW as coordinate values ​​on the image, but this is not limited to this. It may be configured as a loop that increments by patchSize*SH and patchSize*SW using subsampling, or it may be a loop that always increments by 1 or 2. Similarly, ySrc and xSrc are configured to increment by rH and rW, respectively, as coordinate values ​​on the image, but this is not limited to this. It may be configured as a loop that increments by SH and SW using chrominance subsampling of the input image, or a loop that increments by outSH and outSW using chrominance subsampling of the output image, or it may be a loop that always increments by 1.

[0241] As described above, a video decoding device having a predicted image derivation unit that decodes a predicted image and a residual decoding unit that decodes a residual decodes input / output image information that specifies the number of channels of an input / output tensor of a neural network model and additional input information, deriving a part of the input tensor from a first image using the input / output image information, and further deriving another part of the input tensor using a second image different from the first using the additional input information, or encoding information related to the derivation of the predicted image or the decoding of the residual, thereby achieving the effect of transmitting the properties and characteristics of the input image in more detail to the neural network model and enhancing the filtering effect.

[0242] Furthermore, as described above, the additive value of the loop variable is changed using the color difference subsampling parameters, and loop processing is performed in a raster scan manner. Within the above loop, an input tensor is derived from the image, and a deep learning filter is applied to the input tensor to derive an output tensor, thereby achieving the effect of enabling the input tensor to be derived using the same processing even in the case of different color sampling.

[0243] (summary) The present application may be configured to decode encoded data including an input tensor identification parameter that specifies a correspondence between a channel of an input tensor of a neural network model and a color component.

[0244] The present application may also be configured such that a relational expression for deriving an input tensor from an input image is defined in accordance with an input tensor discrimination parameter and chrominance subsampling of the input image.

[0245] Furthermore, the input tensor identification parameters of the present application may be configured to specify one of 1 channel, 2 channels, 3 channels, and 6 channels.

[0246] The present invention may also include a means for deriving an input tensor from the input image in response to an input tensor discrimination parameter.

[0247] The present application relates to a configuration for decoding coded data including an output tensor discrimination parameter that specifies a correspondence between a channel of an output tensor of a neural network model and a color component. That's fine too.

[0248] The present application may be configured such that a relational equation for deriving an output image from an output tensor is defined according to an output tensor identification parameter and color difference subsampling of the output image.

[0249] The output tensor discrimination parameters of the present application may include specifying one channel, two channels, three channels, or six channels.

[0250] The present application may include means for deriving an output image from the output tensor in response to an output tensor discrimination parameter.

[0251] In this way, the SEI is the input data format to the NN filter and the output data format of the NN filter. This provides an advantage in that it is possible to easily select a method for appropriately converting a decoded image into input data for an NN filter, or a method for appropriately converting output data of an NN filter into an output image, without loading and analyzing a model.

[0252] (Configuration for deriving input tensor identification parameters and output tensor parameters) The NN filter unit 611 receives input tensor discrimination parameters and outputs tensor discrimination parameters Instead of decoding from additional data, the encoded data is decoded, or the URI is used. The topology of the NN model identified by the

[0253] The NN filter unit 611 calculates the number of channels NumInChannels of the input data inputTensor of the NN model. Derive nnrpf_input_format_idc accordingly as follows: For 1 channel, nnrpf_input_format_idc=0 For 2 channels, nnrpf_input_format_idc=1 For 3 channels, nnrpf_input_format_idc=2 For 6 channels, nnrpf_input_format_idc=3 The NN filter unit 611 calculates the number of channels NumOutChannels of the output data outputTensor of the NN model. Derive nnrpf_output_format_idc as follows depending on the For 1 channel, nnrpf_output_format_idc=0 For 2 channels, nnrpf_output_format_idc=1 For 3 channels, nnrpf_output_format_idc=2 For 6 channels, nnrpf_output_format_idc=3 According to the above configuration, the NN filter unit 611 receives the NN transmitted or specified in the encoded data. The number of dimensions of the model's input data and output data is analyzed, and the input image is converted to an input tensor, and the output tensor is converted to an output image according to the analysis results (input tensor identification parameters, output tensor identification parameters). This has the effect of identifying the relationship between color components and channels that are not specified in the NN model itself, preparing NN input data, and obtaining an output image from the NN output data. Work model complexity information may also be included.

[0254] (Configuration of Image Encoding Device) Next, the configuration of the image encoding device 11 according to this embodiment will be described. 1 is a block diagram showing a configuration of an image encoding device 11 according to the present embodiment. The image encoding device 11 includes a predicted image generating unit 101, a subtraction unit 102, a transform / quantization unit 103, an inverse quantization / inverse transform unit 105, an addition unit 106, a rule A loop filter 107, a prediction parameter memory (a prediction parameter storage unit, a frame memory) 108, a reference picture memory (a reference image storage unit, a frame memory) 109, and an encoding parameter determination unit. The image forming apparatus includes a prediction parameter derivation unit 110, a parameter coding unit 111, a prediction parameter derivation unit 120, and an entropy coding unit 104.

[0255] The predicted image generation unit 101 generates a predicted image for each CU. The predicted image generation unit 101 includes the inter predicted image generation unit 309 and the intra predicted image generation unit 310, which have already been described, and therefore a description thereof will be omitted.

[0256] The subtraction unit 102 generates a prediction error by subtracting the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

[0257] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction error input from the subtraction unit 102, and derives quantized transform coefficients by quantizing the prediction error. The quantized transform coefficients are output to the parameter coding unit 111 and the inverse quantization and inverse transform unit 105 .

[0258] The inverse quantization and inverse transform unit 105 corresponds to the inverse quantization and inverse transform unit 311 (FIG. 5) in the image decoding device 31. The calculated prediction error is output to the adder 106.

[0259] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The outline of the operation of each module is explained below.

[0260] The header encoding unit 1110 performs encoding processing of parameters such as header information, division information, prediction information, and quantized transform coefficients.

[0261] The CT information encoding unit 1111 encodes the QT, MT (BT, TT) division information and the like.

[0262] The CU encoding unit 1112 encodes the CU information, prediction information, division information, and so on.

[0263] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0264] The CT information encoding unit 1111 and the CU encoding unit 1112 transmit syntax elements such as inter prediction parameters (predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX), intra prediction parameters (intra_luma_mpm_flag, intra_luma_mpm_idx, intra_luma_mpm_reminder, intra_chroma_pred_mode), and quantized transform coefficients to the parameter encoding unit 111. Supply.

[0265] The entropy coding unit 104 receives the quantized transform coefficients and coding parameters (division information, prediction parameters) from the parameter coding unit 111. These are then entropy-encoded to generate encoded data Te, which is then output.

[0266] The prediction parameter derivation unit 120 is a means including an inter-prediction parameter coding unit 112 and an intra-prediction parameter coding unit 113, and derives intra-prediction parameters and intra-prediction parameters from the parameters input from the coding parameter determination unit 110. The derived intra-prediction parameters and intra-prediction parameters are output to the parameter coding unit 111. can be.

[0267] (Configuration of the inter-prediction parameter encoding unit) As shown in FIG. 8, the inter prediction parameter encoding unit 112 includes a parameter encoding control unit 1121 and an inter prediction parameter derivation unit 303. The parameter encoding control unit 1121 is a merge-in unit. The vector candidate index derivation unit 11211 and the vector candidate index derivation unit 11212 .

[0268] The merge index derivation unit 11211 derives merge candidates and the like, and calculates inter prediction parameters The vector candidate index derivation unit 11212 derives predicted vector candidates and the like, and outputs them to the inter prediction parameter derivation unit 303 and the parameter coding unit 111.

[0269] (Configuration of the intra-prediction parameter encoding unit 113) The intra-prediction parameter coding unit 113 is a parameter coding control unit 1131 and an intra-prediction The image decoding device includes a parameter derivation unit 304. The intra prediction parameter derivation unit 304 has the same configuration as the image decoding device.

[0270] The parameter coding control unit 1131 derives IntraPredModeY and IntraPredModeC. Furthermore, it determines intra_luma_mpm_flag by referring to mpmCandList[ ]. These prediction parameters are output to the intra prediction parameter derivation unit 304 and the parameter coding unit 111.

[0271] However, unlike the image decoding device, the inter prediction parameter derivation unit 303 and the intra prediction parameter derivation unit 304 are The input to the parameter derivation unit 304 is the coding parameter determination unit 110 and the prediction parameter memory 108. and outputs it to parameter coding section 111.

[0272] The adder 106 generates a decoded image by adding, for each pixel, the pixel value of the predicted block input from the predicted image generation unit 101 and the prediction error input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0273] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily include the above three types of filters. For example, the filter may be configured with only a deblocking filter.

[0274] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0275] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined position for each current picture and CU.

[0276] The encoding parameter determination unit 110 determines one of the multiple sets of encoding parameters. The coding parameters are the above-mentioned QT, BT or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generating unit 101 generates a predicted image using these coding parameters.

[0277] The coding parameter determination unit 110 determines the size of the amount of information and the coding parameter for each of the plurality of sets. The RD cost value indicating the error is calculated. The RD cost value is, for example, the sum of the code amount and the value obtained by multiplying the squared error by a coefficient λ. The code amount is the information amount of the coded data Te obtained by entropy coding the quantization error and the coding parameters. The squared error is calculated in the subtraction unit 102. The coefficient λ is a preset real number greater than zero. The coding parameter determination unit 110 determines a set of coding parameters that minimizes the calculated cost value. The encoding parameter determination unit 110 selects the determined encoding parameters as a parameter code. The resultant signal is output to the image forming unit 111 and the prediction parameter derivation unit 120.

[0278] Note that a part of the image encoding device 11 and the image decoding device 31 in the above-described embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generating unit 308, the inverse quantization and inverse transform unit 311, the adder 312, the prediction parameter derivation unit 320, the predicted image generating unit 101, the subtractor 102, the transform and quantization unit 103, the entropy coding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the coding parameter determination unit 110, the parameter coding unit 111, the prediction parameter The data derivation unit 120 may be realized by a computer. In that case, the control function is performed by Alternatively, the image coding apparatus 10 may be realized by recording a program for implementing the image coding apparatus 10 in a computer-readable recording medium, and having the computer system read and execute the program recorded in the recording medium. Note that the "computer system" referred to here is a computer system built into either the image coding apparatus 11 or the image decoding apparatus 31, and includes hardware such as an OS and peripheral devices. Also, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a computer system. The term "computer-readable recording medium" refers to a storage device such as a hard disk built into a computer system. Furthermore, the term "computer-readable recording medium" may also include a device that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or a device that stores a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such a case. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0279] In addition, a part or the whole of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually made into a processor, or a part or the whole may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used.

[0280] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0281] (NNR) Neural Network Coding and Representation (NNR) is a method for representing neural networks (NNs) This is an international standard for efficient compression. Compressing trained NNs makes it possible to store and transmit them more efficiently.

[0282] The following provides an overview of the NNR encoding and decoding processes.

[0283] FIG. 15 is a diagram showing an NNR encoding device and a decoding device.

[0284] The NN coding device 801 includes a preprocessing unit 8011, a quantization unit 8012, and an entropy coding unit 8013. The NN encoding device 801 receives the uncompressed NN model O, and the quantization unit 8012 quantizes the NN model O. In the NN coding device 801, before quantization, a pre-processing unit 8011 may repeatedly apply a parameter reduction method such as pruning or sparsification. After that, an entropy coding unit 8013 performs entropy coding on the quantization model Q. This is applied to obtain a bitstream S for storing and transmitting the NN model.

[0285] The NN decoding device 802 includes an entropy decoding unit 8021, a parameter restoration unit 8022, and a post-processing unit 8023. The NN decoding device 802 first receives the transmitted bit stream S, and the entropy decoding unit 8021 performs entropy decoding of S to obtain an intermediate model RQ. Operation of the NN model If the environment supports inference using the quantized representation used in the RQ, the RQ may be output and used for inference. If not, the parameters of the RQ are restored to their original representation in the parameter restoration unit 8022, and an intermediate model RP is obtained. The sparse tensor representation used is the operating environment of the NN model. If it can be processed by, the RP is output and can be used for inference. If not, the NN model O and A reconstruction NN model R that does not contain different tensors or structural representations is obtained and output.

[0286] The NNR standard specifies the decoding procedures for the numeric representation of specific NN parameters, such as integers and floating point. There is law.

[0287] The decoding method NNR_PT_INT decodes models with integer-valued parameters. The decoding method NNR_PT_FLOAT extends NNR_PT_INT by adding a quantization step size delta. This delta is multiplied by the integer value to produce a scaled integer. delta is the integer quantization parameter. From the meter qp and the delta granularity parameter qp_density, it is derived as follows:

[0288] mul = 2^(qp_density) + (qp & (2^(qp_density)-1)) delta = mul * 2^((qp >> qp_density)-qp_density) (Format of trained NN) The representation of a trained NN consists of two elements: a topological representation, such as the size of layers and the connections between layers, and a parameter representation, such as weights and biases.

[0289] Topological representation is covered in native formats such as Tensorflow and PyTorch. However, to improve interoperability, exchange formats such as Open Neural Network Exchange Format (ONNX) and Neural Network Exchange Format (NNEF) exist.

[0290] The NNR standard also transports topology information, nnr_topology_unit_payload, as part of the NNR bitstream that contains the compressed parameter tensors. This allows for the exchange of This enables interoperability not only with existing data mats, but also with topology information expressed in native formats.

[0291] (Post-filter purpose SEI) The postfilter purpose SEI indicates the purpose of the postfilter processing. Describe the input and output information according to the purpose. Figure 13 shows an example of the syntax of this SEI. do.

[0292] First, as the input of this SEI, the InputChromaFormatIdc (ChromaFormatIdc) of the input image is This value is assigned to the sps_chroma_format_idc value of the encoded data.

[0293] The value of pfp_id indicates the identification number of the post-filter process specified by another mechanism. In this embodiment, it is linked to the NN filter SEI. This SEI message is applied to the current decoded image and all subsequent decoded images in output order in the current layer until a new CLVS (Coded Layer Video Sequence) starts or the bitstream ends.

[0294] pfp_id contains an identification number used to identify the post-filter process. The identification number shall take values ​​from 0 to 2^20-1, with values ​​from 2^20 to 2^21-1 reserved for future use. do.

[0295] pfp_purpose indicates the purpose of the post-filter process identified by pfp_id. The value of pfp_purpose is in the range of 0 to 2^32-2. Other values ​​of pfp_purpose are reserved for future use. Note that the decoder of the additional information ignores the post_filter_purpose SEI message that contains a reserved value of pfp_purpose.

[0296] A value of 0 for pfp_purpose indicates improved visual quality, i.e. post-filtering is applied, which performs image restoration without image resolution conversion.

[0297] A value of 1 for pfp_purpose specifies the width or height of the cropped output image, which means that post-filtering involving image resolution conversion is applied.

[0298] When the value of pfp_purpose is 1, the syntax elements pfp_pic_width_in_luma_samples and pfp_pic_height_in_luma_samples exist.

[0299] pfp_pic_width_in_luma_samples is the width of the sampled output image identified by pfp_id. Specifies the width of the luminance pixel array that results from applying post-processing filters.

[0300] pfp_pic_height_in_luma_samples specifies the height of the luma pixel array resulting from applying the post-processing filter identified by pfp_id to the cropped output image.

[0301] In the examples of Non-Patent Documents 1 and 2, resolution conversion and It was not possible to describe the information on the inverse transformation well. In this embodiment, the above problem is solved by clarifying the information on the input and output.

[0302] If the value of pfp_purpose is 2, there are syntax elements that indicate the color components to which postfiltering is applied and the chrominance format information of the output image. In other words, this means that postfiltering related to chrominance format conversion is applied.

[0303] When the value of pfp_purpose is 2, the syntax elements pfp_component_idc and pfp_output_diff_chroma_format_idc are present.

[0304] pfp_component_idc specifies the color component to which postfiltering should be applied.

[0305] A value of 0 for pfp_component_idc indicates that post-filtering should be applied to the luma component only.

[0306] A value of 1 for pfp_component_idc applies post-filtering to the two chrominance components. This indicates that.

[0307] A value of 2 in pfp_component_idc applies post-filtering to all three color components. This indicates that the

[0308] If the syntax element pfp_component_idc is not present, the value of pfp_component_idc is 2. It is estimated as follows.

[0309] pfp_output_diff_chroma_format_idc is the filter update information, and is used for post-filter processing. pfp_output_diff_chroma_format_idc indicates the difference between the identification value of the chrominance format output by the processor and the identification value of the input chrominance format. The value of pfp_output_diff_chroma_format_idc must be in the range of 0 to 2. The variable OutputChromaFormatIdc, which is an identification value of the chrominance format output by the post-filtering process, is derived as follows.

[0310] OutputChromaFormatIdc = InputChromaFormatIdc + pfp_output_diff_chroma_format_idc Here, InputChromaFormatIdc is written in the SPS of the encoded data. The value of sps_chroma_format_idc identifies the chroma format of the decoded image. A value of 0 indicates monochrome (4:0:0), a value of 1 indicates 4:2:0, a value of 2 indicates 4:2:2, and a value of 3 indicates 4:4:4. The variable OutputChromaFormatIdc, which is the identification value of the color difference format output by the data processing, is As with ChromaFormatIdc, a value of 0 indicates monochrome (4:0:0), a value of 1 indicates 4:2:0, a value of 2 indicates 4:2:2, and a value of 3 indicates 4:4:4.

[0311] In this embodiment, in order to derive OutputChromaFormatIdc, The OutputChromaFormatIdc value was set to be equal to or greater than the InputChromaFormatIdc value using the difference between the identification value of the format and the identification value of the input color difference format. Alternatively, OutputChromaFormatIdc may be used directly as a syntax element without using the difference value. In this case, the value of OutputChromaFormatIdc must be defined independently of the value of InputChromaFormatIdc. In this case, the syntax element OutputChromaFormatIdc is present. If not present, it is assumed to be the same as the value of InputChromaFormatIdc.

[0312] When the update information php_output_diff_chroma_format_idc does not exist in the encoded data, the NN filter unit 611 estimates php_output_diff_chroma_format_idc to be 0. As a result, when php_output_diff_chroma_format_idc does not exist in the encoded data, the NN filter unit 611 may set it as follows. OutputChromaFormatIdc = ChromaFormatIdc As described above, the video decoding device includes a prediction image derivation unit that decodes a prediction image and a residual decoding unit that decodes a residual. When there is no filter update information, the header decoding unit 3020 estimates the output chrominance format for chrominance subsampling from the input chrominance format, thereby achieving the effect of enabling accurate updating by filter processing even when filter update information is not specified.

[0313] The NN filter unit 611 derives a variable indicating the chrominance subsampling of the output image as follows: do.

[0314] outSubWidthC = outSW = 1, outSubHeightC = outSH = 1 (OutputChromaFormatIdc == 0) outSubWidthC = outSW = 2, outSubHeightC = outSH = 2 (OutputChromaFormatIdc == 1) outSubWidthC = outSW = 2, outSubHeightC = outSH = 1 (OutputChromaFormatIdc == 2) outSubWidthC = outSW = 1, outSubHeightC = outSH = 1 (OutputChromaFormatIdc == 3) In this manner, by defining the input components and output format of the post-filter processing of the color difference format conversion, it is possible to clarify the input and output data of the post-filter processing of the color difference format conversion.

[0315] Although pfp_component_idc, which specifies the color component to which the above post-filtering is applied, distinguishes between luminance and chrominance, it may simply indicate the number of components. may have the following semantics:

[0316] A value of 0 for pfp_component_idc means to apply post-filtering for one component. This indicates that:

[0317] A value of 1 for pfp_component_idc means that post-filtering is applied to two components. This indicates that:

[0318] A value of 2 for pfp_component_idc applies post-filtering to all three components. This indicates that.

[0319] The NN filter unit 611 may switch the NN model depending on pfp_component_idc.

[0320] When pfp_component_idc==0: The NN filter unit 611 converts a 1-channel 3D tensor into A NN model that derives a one-channel three-dimensional tensor is selected and used for filtering.

[0321] When pfp_component_idc==1: The NN filter unit 611 extracts 2 A NN model is selected to derive the 3D tensor of the channel and perform filtering.

[0322] When pfp_component_idc==2: The NN filter unit 611 converts a 3-channel 3-dimensional tensor into a 3-dimensional tensor. A NN model is selected to derive the 3D tensor of the channel and perform filtering.

[0323] As a result, the amount of processing can be reduced by selecting an appropriate NN model depending on the color component to be applied.

[0324] The NN filter unit 611 selects one channel for one component, two channels for two components, and so on, depending on pfp_component_idc. The nnrpf_input_format_idc may be derived as follows to create a 2-channel NN model for components, and a 3-channel NN model for 3 components. nnrpf_input_format_idc = 0 (pfp_component_idc==0) nnrpf_input_format_idc = 1 (pfp_component_idc==1) nnrpf_input_format_idc = 2 (pfp_component_idc==2) i.e. nnrpf_input_format_idc = pfp_component_idc As another example, the NN filter unit 611 may derive nnrpf_input_format_idc as follows, depending on pfp_component_idc, so as to obtain a NN model with 1 channel for 1 component, 2 channels for 2 components, and 6 channels for 3 components. nnrpf_input_format_idc = 0 (pfp_component_idc==0) nnrpf_input_format_idc = 1 (pfp_component_idc==1) nnrpf_input_format_idc = 3 (pfp_component_idc==2) That is, nnrpf_input_format_idc = pfp_component_idc < 2 ? pfp_component_idc : 3 As a result, it is possible to select an appropriate tensor format for the NN model depending on the color component to be applied and perform processing.

[0325] The NN filter unit 611 directly derives the inputTensor according to pfp_component_idc as described above. Alternatively, the NN filter unit 611 may switch between the value of the input image ComponentSample and the value of the NN output data outTensor in accordance with pfp_component_idc, and derive the output image by the following process. If pfp_component_idc==0 outSamplesL[x][y] = outTensor[0][y][x] outSamplesCb[x*2 / outSW][y*2 / outSH] = ComponentSample[1][x*2 / SW][y*2 / SH] outSamplesCr[x*2 / outSW][y*2 / outSH] = ComponentSample[2][x*2 / SW][y*2 / SH] If pfp_component_idc==1 outSamplesL[x][y] = ComponentSample[0][x][y] outSamplesCb[x*2 / outSW][y*2 / outSH] = outTensor [0][x*2 / SW][y*2 / SH] outSamplesCr[x*2 / outSW][y*2 / outSH] = outTensor [1][x*2 / SW][y*2 / SH] If pfp_component_idc==2 outSamplesL[x][y] = outTensor[0][x][y] outSamplesCb[x*2 / outSW][y*2 / outSH] = outTensor[1][x*2 / SW][y*2 / SH] outSamplesCr[x*2 / outSW][y*2 / outSH] = outTensor[2][x*2 / SW][y*2 / SH] In the above example, the identification value of the color difference format output by the post-filtering process is represented as a difference value between the identification value of the input color difference format, but it may be directly described as a syntax element.

[0326] In this embodiment, the postfilter objective SEI is defined independently of the NN postfilter SEI to define the input and output of the postfilter processing. may be defined, and the problem can be solved in a similar manner.

[0327] The present embodiment will be described with reference to FIG. 1. The video decoding device is characterized by having an image decoding device, a resolution inverse conversion device that converts the resolution of an image decoded by the image decoding device, and an inverse conversion information decoding device that decodes color component information to be input to the resolution inverse conversion device and color difference format information to be output.

[0328] The present invention also provides a moving image coding device, characterized by having an image coding device that codes an image, and an inverse conversion information coding device that codes color component information to be input to a resolution inverse conversion device that converts the resolution of the coded image, and color difference format information to be output.

[0329] A video decoding device according to one aspect of the present invention is a video decoding device including a predicted image derivation unit that decodes a predicted image and a residual decoding unit that decodes a residual, The method is characterized by deriving an input tensor from parameters that specify the number of channels of the input tensor and output tensor of the neural network model, or deriving an image from the output tensor.

[0330] A video decoding device including a predicted image derivation unit that decodes a predicted image and a residual decoding unit that decodes a residual, The method is characterized in that input / output image information specifying the number of channels of the input / output tensor of the neural network model and additional input information are decoded, and a part of the input tensor is derived from a first image using the input / output image information, and further, a second image different from the first is derived using the additional input information, or another part of the input tensor is derived using coding information related to the derivation of the predicted image or the decoding of the residual.

[0331] The method is characterized in that the additive value of the loop variable is changed using the parameters of the color difference subsampling, loop processing is performed on the raster scan, and within the loop, an input tensor is derived from the image, and a deep learning filter is applied to the input tensor to derive an output tensor.

[0332] A video decoding device including a predicted image derivation unit that decodes a predicted image and a residual decoding unit that decodes a residual, wherein the header unit estimates an output chrominance format for chrominance subsampling from an input chrominance format when there is no filter update information.

[0333] A video encoding device according to one aspect of the present invention is a video encoding device including a predicted image derivation unit that decodes a predicted image and a residual encoding unit that decodes a residual, The method is characterized by deriving an input tensor from parameters that specify the number of channels of the input tensor and output tensor of the neural network model, or deriving an image from the output tensor.

[0334] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0335] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data in which image data is coded, and a video coding device that generates coded data in which image data is coded, and can also be suitably applied to the data structure of coded data that is generated by a video coding device and referenced by the video decoding device. [Explanation of symbols]

[0336] 1. Video transmission system 30 Video Decoding Device 31 Image Decoding Device 301 Entropy Decoding Unit 302 Parameter Decoding Unit 303 Inter-prediction parameter derivation unit 304 Intra prediction parameter derivation unit 305, 107 Loop Filter 306, 109 Reference Picture Memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 309 Inter-prediction image generation unit 310 Intra-prediction image generation unit 311, 105 Inverse quantization and inverse transformation unit 312, 106 Addition section 320 Prediction Parameter Derivation Unit 10 Video Encoding Device 11 Image encoding device 102 Subtraction section 103 Transformation and Quantization Section 104 Entropy coding unit 110 Encoding parameter determination unit 111 Parameter Encoding Unit 112 Inter-prediction parameter coding unit 113 Intra prediction parameter coding unit 120 Prediction parameter derivation part 71 Reverse conversion information creation device 81 Inverse transformation information coding device 91 Inverse transformation information decoding device 611 NN filter section

Claims

1. An image decoding device that decodes an image from encoded data, comprising: a header decoding unit for deriving a filter strength and a bit depth for the filtering process; a neural network filter unit that derives input image information that specifies a format of an input tensor for a neural network model; the neural network filter unit derives a part of the input tensor using an image based on the input image information, sets additional information in another part of the input tensor, derives an output tensor by performing a neural network filtering process using the input tensor, and derives a filtered image from the output tensor; An image decoding device, characterized in that the additional information is derived using the filter strength and the bit depth.

2. An image encoding device that encodes an image, comprising: a header encoder for deriving a filter strength and a bit depth for the filtering process; a neural network filter unit that derives input image information that specifies a format of an input tensor for a neural network model; the neural network filter unit derives a part of the input tensor using an image based on the input image information, sets additional information in another part of the input tensor, derives an output tensor by performing a neural network filtering process using the input tensor, and derives a filtered image from the output tensor; An image encoding device characterized in that the additional information is derived using the filter strength and the bit depth.

3. 1. An image decoding method for decoding an image from encoded data, comprising the steps of: Derive a filter strength and a bit depth for the filtering process; Derive input image information that specifies a format of an input tensor for the neural network model; deriving a portion of the input tensor using an image based on the input image information; Set additional information in another part of the input tensor; Deriving an output tensor by performing a neural network filtering process using the input tensor; An image decoding method comprising: deriving a filtered image from the output tensor.

4. 1. An image encoding method for encoding an image, comprising the steps of: Derive a filter strength and a bit depth for the filtering process; Derive input image information that specifies a format of an input tensor for the neural network model; deriving a portion of the input tensor using an image based on the input image information; Set additional information in another part of the input tensor; Deriving an output tensor by performing a neural network filtering process using the input tensor; Deriving a filtered image from the output tensor; An image coding method, characterized in that the additional information is derived using the filter strength and the bit depth.

Images