Video encoding device and video decoding device

Abstract

To solve a problem in which a conventional SEI that does not include information that simply identifies whether a neural network model is executable, a relationship between a target image and an input tensor of the neural network model and a relationship between an output tensor and an output image is undefined, and a model alone cannot be processed, and processing cannot be performed when the color component of the output image and the color components of the target image are different.SOLUTION: A video decoding device according to an embodiment of the present invention includes a header portion that decodes additional information, a predicted image derivation unit that decodes a predicted image, and a video decoding device provided with a transform portion for decoding residuals, and the header portion decodes additional information about a postfilter, and the additional information about the postfilter includes a syntax indicating network model complexity information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a video encoding device and a video decoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding moving images, and a moving image decoding device is used that generates decoded images by decoding the encoded data.

[0003] Specific examples of video encoding methods include H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such a video coding method, images (pictures) constituting a video are divided into slices obtained by dividing the image, coding tree units (CTUs) obtained by dividing the slices, and coding tree units (CTUs) obtained by dividing the coding tree units. The coding unit (sometimes called a coding unit (CU)) that is used for encoding, and The coding unit is divided into transform units (TUs) and managed in a hierarchical structure, and is coded / decoded for each CU.

[0005] In such video coding methods, a predicted image is usually generated based on a locally decoded image obtained by encoding / decoding an input image, and the predicted image is subtracted from the input image (original image) to obtain a prediction error (sometimes called a "difference image" or "residual image"), which is then coded. Methods for generating predicted images include inter-frame prediction (inter-prediction) and intra-frame prediction (intra-prediction).

[0006] Furthermore, Non-Patent Document 1 is an example of a recent video encoding and decoding technique.

[0007] H.274 includes a function that transmits information such as image characteristics, display method, and timing simultaneously with the encoded data. Supplemental enhancement information (SEI) is specified for this purpose.

[0008] In Non-Patent Document 1 and Non-Patent Document 2, the SEI that transmits the topology and parameters of the neural network filter used as a post-filter is explicitly defined. The method and the method indirectly specified as reference information are disclosed. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] B. Choi, Z. Li, W. Wang, W. Jiang, X. Xu, S. Wenger and S. Liu, “AHG9 / AHG11: SEI messages for carriage of neural network information for post-filtering,” JVET-V0091 [Non-patent document 2] MM Hannuksela, EB Aksu, F. Cricri, HR Tavakoli and M. Santamaria, "AHG9: On post-filter SEI", JVET-X0112 Summary of the Invention [Problem to be solved by the invention]

[0010] However, both Non-Patent Document 1 and Non-Patent Document 2 have the problem that they do not include information that simply specifies whether a neural network model is executable, and therefore require analysis of the neural network model. In Non-Patent Document 1, it is necessary to analyze the explicitly defined topology. In Non-Patent Document 2, the neural network model specified based on the URI is also not executable. However, there is a problem in that the complexity cannot be obtained without analyzing the topology of the network model.

[0011] Furthermore, both Non-Patent Document 1 and Non-Patent Document 2 have the problem that the input and output of the post-filtering process are not explicitly defined.

[0012] In Non-Patent Document 2, the output color space and color difference sampling are not specified, and there is a problem in that it is not possible to specify what kind of output will be obtained from the additional information.

[0013] Non-Patent Documents 1 and 2 can analyze the types of input and output tensors from the topology of a neural network, but they cannot identify the relationship between tensor channels and color components. For example, they cannot define how to set the luma and chroma channels in the input tensor or the color space in which the processed output tensor is output. This poses a problem: it is not possible to identify and execute processing. Furthermore, the width and height of luma and chroma vary depending on the color sampling (4:0:0, 4:2:0, 4:2:2, and 4:4:4), but it is not possible to identify how to process and derive the input tensor. Furthermore, it is not possible to identify how to generate an image from the output tensor depending on the color sample. [Means for solving the problem]

[0014] A video decoding device according to one aspect of the present invention includes a header decoding unit that decodes additional information, a predicted image derivation unit that decodes a predicted image, and a transform unit that decodes a residual, The header portion decodes additional information related to a post filter, and the additional information related to the post filter includes a syntax indicating network model complexity information.

[0015] The syntax indicating the network model complexity information is an index, and when the value is 0, coded data indicating no designation is decoded.

[0016] The syntax indicating the network model complexity information is characterized by indicating an upper limit on the number of parameters or the number of operations.

[0017] The syntax indicating the network model complexity information is characterized by indicating the type of parameters of the network model.

[0018] A video encoding device comprising: a header encoding unit that encodes additional information; a predicted image derivation unit that encodes a predicted image; and a transform unit that encodes a residual, The header portion encodes additional information related to a post filter, and the additional information related to the post filter includes a syntax indicating network model complexity information.

[0019] The method is characterized by decoding coded data including input tensor identification parameters that specify the correspondence between the channels of the input tensor of the neural network model and the color components.

[0020] The method is characterized in that a relational expression for deriving an input tensor from an input image is defined according to an input tensor discrimination parameter and color difference sampling of the input image.

[0021] The method is characterized by including means for deriving an input tensor from an input image in response to an input tensor identification parameter.

[0022] The method is characterized by decoding coded data including an output tensor identification parameter that specifies the correspondence between the channels of the output tensor of the neural network model and the color components.

[0023] The relational expression for deriving the input tensor from the output image according to the output tensor identification parameter is defined according to color difference sampling of the output image.

[0024] The method is characterized by including means for deriving an output image from the output tensor in response to the output tensor discrimination parameter.

[0025] The method is characterized in that the encoded data includes an input tensor identification parameter that specifies the correspondence between the channels of the input tensor of the neural network model and the color components. [Effects of the Invention]

[0026] By configuring it like this, you can analyze the neural network model specified by the URI. This makes it possible to refer to the complexity of the neural network model specified by the NN filter without having to perform a post-filtering operation, which has the effect of making it easier to determine whether the video decoding device has the processing capacity for a post-filter using a neural network filter. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a schematic diagram showing the configuration of a video transmission system according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing the configuration of a transmitting device equipped with a video encoding device according to this embodiment, and a receiving device equipped with a video decoding device, in which PROD_A indicates the transmitting device equipped with the video encoding device, and PROD_B indicates the receiving device equipped with the video decoding device. [Figure 3] 1 is a diagram showing the configuration of a recording device equipped with a video encoding device according to this embodiment, and a playback device equipped with a video decoding device, in which PROD_C indicates a recording device equipped with a video encoding device, and PROD_D indicates a playback device equipped with a video decoding device. [Figure 4] FIG. 2 is a diagram showing a hierarchical structure of encoded data. [Figure 5] 1 is a conceptual diagram of an image to be processed in a video transmission system according to the present embodiment. [Figure 6] FIG. 1 is a conceptual diagram illustrating an example of a reference picture and a reference picture list. [Figure 7] FIG. 1 is a schematic diagram illustrating a configuration of an image decoding device. [Figure 8] 10 is a flowchart illustrating a schematic operation of the image decoding device. [Figure 9] FIG. 1 is a block diagram showing a configuration of an image encoding device. [Figure 10] FIG. 10 is a schematic diagram showing the configuration of an inter-prediction parameter encoding unit. [Figure 11] FIG. 10 is a diagram showing the configuration of a syntax table of an NN filter SEI in this embodiment. [Figure 12] FIG. 10 is a diagram showing the structure of a syntax table of an NN filter SEI when the parameter type of a neural network model is described by dividing it into a numeric value type and a bit width. [Figure 13] FIG. 10 is a diagram showing the configuration of a syntax table of an NN filter SEI when the bit width of the parameter type of the neural network model is described in logarithmic representation. [Figure 14] FIG. 10 is a flowchart showing the processing of the NN filter unit 611. [Figure 15] FIG. 10 is a diagram showing the structure of a syntax table of a post-filter purpose SEI. [Figure 16] FIG. 10 is a diagram showing the configuration of a neural network of an NN filter unit 611. [Figure 17] FIG. 1 is a diagram illustrating an NNR encoding device and decoding device. [Figure 18] FIG. 10 is a diagram illustrating an example of the syntax of an NN filter SEI. [Figure 19] 1 is a table showing an input data format defined in the NN filter. [Figure 20] 10 is a table showing an output data format defined by the NN filter. [Figure 21] 10 is a table showing a method of converting a decoded image into an input data format defined by an NN filter. [Figure 22] 10 is a table showing a method of converting an output data format defined by an NN filter into an output image. DETAILED DESCRIPTION OF THE INVENTION

[0028] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0029] FIG. 1 is a schematic diagram showing the configuration of a video transmission system according to this embodiment.

[0030] The video image transmission system 1 encodes images of different resolutions after resolution conversion. The video transmission system 1 comprises a video encoding device 10, a network 21, and a video signal. It consists of an image decoding device 30 and an image display device 41 .

[0031] The video encoding device 10 is composed of a resolution conversion device (resolution conversion unit) 51, an image encoding device (image encoding unit) 11, an inverse conversion information creation device (inverse conversion information creation unit) 71, and an inverse conversion information encoding device (inverse conversion information encoding unit) 81.

[0032] The video decoding device 30 includes an image decoding device (image decoding unit) 31, a resolution inverse conversion device (resolution inverse conversion unit) 61, and an inverse conversion information decoding device (inverse conversion information decoding unit) 91.

[0033] The resolution conversion device 51 converts the resolution of an image T included in a moving image, and generates an image of a different resolution. The resolution conversion device 51 supplies the variable resolution video T2 including the above to the image coding device 11. The resolution conversion device 51 also supplies inverse conversion information indicating whether or not the resolution of the image is to be converted to the image coding device 11. If the information indicates resolution conversion, the video coding device 10 sets resolution conversion information ref_pic_resampling_enabled_flag, which will be described later, to 1, and includes it in a sequence parameter set SPS (Sequence Parameter Set) of the coded data Te for coding.

[0034] The inverse conversion information creation device 71 creates inverse conversion information based on an image T1 included in a video. The inverse conversion information is derived or selected from the relationship between the input image T1 before resolution conversion and the image T17 after resolution conversion, encoding, and decoding. The auxiliary information is information indicating what to select.

[0035] Inverse transformation information is input to the inverse transformation information coding device 81. The inverse transformation information coding device 81 codes the inverse transformation information to generate coded inverse transformation information, and sends the coded inverse transformation information to the network 21.

[0036] A variable resolution image T2 is input to the image encoding device 11. The image encoding device 11 encodes the image T2 into the RPR frame. The image size information of the input image is encoded in PPS units using the set, and sent to the image decoding device 31. .

[0037] In FIG. 1, the inverse transformation information coding device 81 is not connected to the image coding device 11, but the inverse transformation information coding device 81 and the image coding device 11 may communicate necessary information as appropriate.

[0038] The network 21 transmits the coded inverse transformation information and the coded data Te to the image decoding device 31. A part or all of the coded inverse transformation information is coded as supplemental enhancement information SEI. The network 21 may be the Internet, a wide area network (WAN), a local area network (LAN), or the like. ) or a combination thereof. The network 21 is not necessarily limited to a two-way communication network. The network 21 may be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting, satellite broadcasting, etc. The network 21 may also be replaced by a storage medium on which encoded data Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

[0039] The image decoding device 31 decodes each of the coded data Te transmitted over the network 21 , generates a variable resolution decoded image, and supplies it to the resolution inverse conversion device 61 .

[0040] The inverse conversion information decoding device 91 decodes the coded inverse conversion information transmitted by the network 21 to generate inverse conversion information, and supplies the generated inverse conversion information to the resolution inverse conversion device 61 .

[0041] 1, the inverse transformation information decoding device 91 is illustrated separately from the image decoding device 31, but the inverse transformation information decoding device 91 may be included in the image decoding device 31. For example, the inverse transformation information decoding device 91 may be included in the image decoding device 31 separately from each functional unit of the image decoding device 31. Furthermore, although not connected to the image decoding device 31 in FIG. 1, the inverse transformation information decoding device 91 and the image decoding device 31 may communicate necessary information as appropriate.

[0042] When the resolution conversion information indicates resolution conversion, the resolution inverse conversion device 61 generates a decoded image of the original size by inversely converting the resolution-converted image through super-resolution processing using a neural network based on the image size information included in the encoded data.

[0043] The image display device 41 receives one or more decoded images Td2 from the resolution inverse conversion device 61. The image display device 41 displays all or part of the image. The image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. If the image decoding device 31 has high processing power, it displays high-quality images, and if it has only low processing power, it displays images that do not require high processing power or display power.

[0044] FIG. 5 is a conceptual diagram of an image to be processed in the moving image transmission system shown in FIG. 1, and is a diagram showing the change in the resolution of the image as time elapses. However, in FIG. 5, it is not distinguished whether the image is encoded or not. FIG. 5 shows an example in which the resolution is reduced and the image is transmitted to the image decoder 31 in the processing process of the moving image transmission system. As shown in FIG. 5, usually, the resolution conversion device 51 performs conversion to reduce the resolution of the image in order to reduce the amount of information of the transmitted information.

[0045] <Operator> The operators used in this specification are described below.

[0046] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, and | is a bitwise OR , |= is an OR assignment operator, and || indicates a logical OR.

[0047] x? y : z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0) is.

[0048] Clip3(a, b, c) is a function that clips c to a value between a and b, returns a when c < a, returns b when c > b, and returns c in other cases (however, a <= b).

[0049] abs(a) is a function that returns the absolute value of a.

[0050] Int(a) is a function that returns the integer value of a.

[0051] floor(a) is a function that returns the largest integer less than or equal to a.

[0052] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0053] a / d represents the division of a by d (rounding down the decimal part).

[0054] a^b represents power(a,b) and is equal to a<<b.

[0055] <Structure of Encoded Data Te> Prior to the detailed description of the image encoding apparatus 11 and the image decoding apparatus 31 according to the present embodiment, the data structure of the encoded data Te generated by the image encoding apparatus 11 and decoded by the image decoding apparatus 31 will be described.

[0056] FIG. 4 is a diagram showing the hierarchical structure of the data in the encoded data Te. The encoded data Te exemplarily includes a sequence and a plurality of pictures constituting the sequence. FIG. 4 shows an encoded video sequence that defines a sequence SEQ, an encoded pi cture that defines a picture PICT, an encoded slice that defines a slice S, an encoded slice slice data that defines slice data, an encoded tree unit included in the encoded slice data, and an encoded unit included in the encoded tree unit.

[0057] (Encoded Video Sequence) In the encoded video sequence, a set of data that the image decoding device 31 refers to in order to decode the sequence SEQ to be processed is defined. As shown in FIG. 4, the sequence SEQ includes a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an Adaptation Parameter Set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0058] In the video parameter set VPS, in a moving image composed of a plurality of layers, A set of coding parameters common to a plurality of video streams and a set of coding parameters associated with a plurality of layers included in the video stream and with each individual layer are defined.

[0059] The sequence parameter set SPS defines a set of coding parameters that the image decoding device 31 refers to in order to decode the target sequence. For example, the width and height of a picture are There may be multiple SPSs. In that case, one of the multiple SPSs can be selected from the PPS. Select.

[0060] Here, the sequence parameter set SPS includes the following syntax elements: ref_pic_resampling_enabled_flag: A flag that specifies whether or not to use a function that changes the resolution (resampling) when decoding each image included in a single sequence that references the target SPS. In other words, this flag indicates that the size of the reference picture referenced in generating a predicted image changes between each image indicated by a single sequence. When the value of this flag is 1, the resampling function is used. is applied, and 0 is not applied. pic_width_max_in_luma_samples: The maximum width of an image in a single sequence. This syntax element specifies the width of the image in luminance block units. The value of this syntax element must not be 0 and must be an integer multiple of Max(8, MinCbSizeY). Here, MinCbSizeY is a value determined by the minimum size of a luminance block. pic_height_max_in_luma_samples: Maximum height of an image in a single sequence This is a syntax element that specifies the height of an image having the following value in units of luminance blocks. The value of this syntax element must not be 0 and must be an integer multiple of Max(8, MinCbSizeY). is required. sps_temporal_mvp_enabled_flag: Enables temporal motion detection when decoding the target sequence. This is a flag that specifies whether vector prediction is used. If the value of this flag is 1, time A value of 0 indicates that temporal motion vector prediction is used, while a value of 0 indicates that temporal motion vector prediction is not used. Furthermore, by defining this flag, it is possible to prevent the coordinate position to be referenced from shifting when, for example, a reference picture with a different resolution is referenced.

[0061] The picture parameter set PPS requires the following parameters to be used to decode each picture in the target sequence: A set of coding parameters to be referred to by the image decoding device 31 is defined. For example, the reference value of the quantization width (pic_init_qp_minus26) used for decoding a picture and the application of weighted prediction are defined. It should be noted that there may be multiple PPSs. In this case, one of multiple PPSs is selected from each picture in the target sequence.

[0062] Here, the picture parameter set PPS includes the following syntax elements: · pic_width_in_luma_samples: A syntax element that specifies the width of the target picture. The value of this syntax element must not be 0, must be an integer multiple of Max(8, MinCbSizeY), and must be less than or equal to pic_width_max_in_luma_samples. pic_height_in_luma_samples: This syntax element specifies the height of the target picture. The value of this syntax element must be a non-zero integer multiple of Max(8, MinCbSizeY) and equal to or less than pic_height_max_in_luma_samples. conformance_window_flag: Conformance (cropping) window offset This flag indicates whether the parameter will be notified subsequently and indicates where the conformance window should be displayed. If this flag is 1, the parameter will be notified. If it is 0, the conformance window offset parameter is not present. This indicates that... conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, conf_win_bottom_offset: Offset values ​​for specifying the left, right, top, and bottom positions of the picture output by the decoding process, relative to the rectangular area specified by the output picture coordinates. If the value of conformance_window_flag is 0, the values ​​of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are presumed to be 0. scaling_window_flag: A flag indicating whether the scaling window offset parameter exists in the target PPS, and is a flag related to the specification of the output image size. If this flag is 1, it indicates that the parameter exists in the PPS, and if this flag is 0, it indicates that the parameter does not exist in the PPS. In addition, if the value of ref_pic_resampling_enabled_flag is 0, the value of scaling_window_flag is also required to be 0. scaling_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, scaling_win_bottom_offset: These syntax elements specify the offsets (in luma samples) applied to the image size for scaling ratio calculation at the left, right, top, and bottom positions of the target picture. If the value of scaling_window_flag is 0, the values ​​of scaling_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, and scaling_win_bottom_offset are inferred to be 0. It is also required that the value of scaling_win_left_offset + scaling_win_right_offset is less than pic_width_in_luma_samples, and the value of scaling_win_top_offset + scaling_win_bottom_offset is less than pic_height_in_luma_samples.

[0063] The width PicOutputWidthL and height PicOutputHeightL of the output picture are derived as follows:

[0064] PicOutputWidthL = pic_width_in_luma_samples - (scaling_win_right_offset + scaling_win_left_offset) PicOutputHeightL = pic_height_in_pic_size_units - (scaling_win_bottom_offset + scaling_win_top_offset) (Subpicture) A picture may be further divided into rectangular sub-pictures. The size of a sub-picture may be a multiple of a CTU. A sub-picture is defined as a set of an integer number of consecutive tiles in both the vertical and horizontal directions. That is, a picture is divided into rectangular tiles, and a subpicture is defined as a set of rectangular tiles. A subpicture may be defined using the IDs of the top-left tile and the bottom-right tile of the subpicture. The slice header may also include sh_subpic_id, which indicates the ID of the subpicture.

[0065] (encoded picture) The coded picture defines a set of data that the image decoding device 31 refers to in order to decode the picture PICT to be processed. As shown in FIG. 4, the picture PICT is Picture PH includes slices 0 to NS-1 (NS is the total number of slices included in the picture PICT) .

[0066] Hereinafter, when there is no need to distinguish between slices 0 to NS-1, the subscripts of the signs are used. The same applies to other data that are included in the encoded data Te described below and have subscripts.

[0067] The picture header contains the following syntax elements: pic_temporal_mvp_enabled_flag: The image of the slice associated with the picture header. This flag specifies whether temporal motion vector prediction is used for center prediction. If the value of this flag is 0, the syntax element of the slice associated with this picture header is The flag restricts the use of temporal motion vector prediction in decoding the slice. If the flag has a value of 1, it indicates that temporal motion vector prediction is used in decoding the slice associated with the picture header. If the flag is not specified, the value is assumed to be 0.

[0068] (encoded slice) In the coded slice, the image decoding device 31 refers to the slice S to be processed in order to decode the slice S. As shown in Figure 4, a slice consists of a slice header and and contains slice data.

[0069] The slice header includes a group of coding parameters that the image decoding device 31 refers to in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates the slice type is an example of a coding parameter included in the slice header.

[0070] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction during encoding, (2) a P slice that uses uni-prediction (L0 prediction) or intra prediction during encoding, and (3) a B slice that uses uni-prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction during encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, the terms P and B slices refer to slices that include blocks that can use inter prediction.

[0071] Note that the slice header may include a reference to a picture parameter set PPS (pic_parameter_set_id).

[0072] (encoded slice data) The coded slice data defines a set of data that the image decoding device 31 refers to in order to decode the slice data to be processed. As shown in the header, it contains a CTU, which is a fixed-size (e.g., 64x64) block that makes up a slice and is sometimes called a Largest Coding Unit (LCU).

[0073] (coding tree unit) 4 defines a set of data that the image decoding device 31 references in order to decode a CTU to be processed. The CTU is divided into coding units (CUs), which are basic units of encoding processing, by recursive quad tree (QT) division, binary tree (BT) division, or ternary tree (TT) division. BT division and TT division are collectively called multi-tree (MT) division. A node in a tree structure obtained by recursive quad tree division is called a coding node. Intermediate nodes in a quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is defined as the top-level coding node.

[0074] The CT includes, as CT information, a CU split flag (split_cu_flag) indicating whether to split the CT, a QT split flag (split_cu_flag) indicating whether to split the CT, and The flags include a QT split flag (qt_split_cu_flag) indicating whether splitting is performed, an MT split direction (mtt_split_cu_vertical_flag) indicating the split direction of the MT split, and an MT split type (mtt_split_cu_binary_flag) indicating the split type of the MT split. split_cu_flag, qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each encoding node.

[0075] Different trees may be used for luminance and chrominance. The tree type is indicated by treeType. For example, if a common tree is used for luminance (Y, cIdx=0) and chrominance (Cb / Cr, cIdx=1,2), the common single tree is indicated by treeType=SINGLE_TREE. If two different trees (DUAL trees) are used for luminance and chrominance, the luminance tree is indicated by treeType=DUAL_TREE_LUMA and the chrominance tree is indicated by treeType=DUAL_TREE_CHROMA.

[0076] (encoding unit) FIG. 4 shows data to be referenced by the image decoding device 31 in order to decode the coding unit to be processed. Specifically, a CU consists of a CU header CUH, prediction parameters, and transformation parameters. The CU header contains information such as a prediction mode.

[0077] Prediction processing may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU. If the sizes of a CU and a sub-CU are the same, there is one sub-CU in the CU. If the size of a CU is larger than the size of a sub-CU, the CU is divided into sub-CUs. For example, if the CU is 8x8 and the sub-CU is 4x4, the CU is divided into four sub-CUs, divided horizontally by two and vertically by two.

[0078] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times or between layer images).

[0079] The transformation and quantization process is performed in units of CU, but the quantized transformation coefficients are stored in units of sub-blocks such as 4x4. It may be entropy coded.

[0080] (Prediction parameters) The predicted image is derived from prediction parameters associated with the block, which include intra-prediction and inter-prediction parameters.

[0081] Hereinafter, prediction parameters of inter prediction will be described. The inter prediction parameters are composed of prediction list usage flags predFlagL0 and predFlagL1, reference picture indices refIdxL0 and refIdxL1, and motion vectors mvL0 and mvL1. predFlagL0 and predFlagL1 are flags indicating whether or not a reference picture list (L0 list, L1 list) is used, and when the value is 1, the corresponding reference picture list is used. Note that in this specification, when a "flag indicating whether XX is true" is used, a flag other than 0 (for example, 1) is considered to be XX, and 0 is considered to be not XX, and in logical negation, logical product, etc., 1 is treated as true and 0 is treated as false (the same applies below). However, in an actual device or method, other values ​​may be used as true and false values.

[0082] The syntax elements for deriving inter prediction parameters include, for example, an affine flag affine_flag used in merge mode, a merge flag merge_flag, a merge index merge_idx, an MMVD flag mmvd_flag, and an index for selecting a reference picture used in AMVP mode. Inter prediction identifier inter_pred_idc, reference picture index refIdxLX, predicted vector index mvp_LX_idx for deriving a motion vector, difference vector mvdLX, motion vector There is a precision mode amvr_mode.

[0083] (Reference Picture List) The reference picture list is a list of reference pictures stored in the reference picture memory 306. FIG. 6 is a conceptual diagram showing an example of reference pictures and reference picture lists. In the conceptual diagram showing an example of reference pictures in FIG. 6, rectangles represent pictures, arrows represent picture reference relationships, the horizontal axis represents time, I, P, and B in the rectangles represent intra-pictures, uni-predictive pictures, and bi-predictive pictures, respectively, and the numbers in the rectangles represent decoding order. As shown in the diagram, the decoding order of pictures is I0, P1, B2, B3, and B4, and the display order is I0, B3, B2, B4, and P1. FIG. 6 shows an example of a reference picture list for picture B3 (target picture). A reference picture list is a list indicating candidate reference pictures, and one picture (slice) may have one or more reference picture lists. In the example shown in the diagram, target picture B3 has two reference picture lists: an L0 list RefPicList0 and an L1 list RefPicList1. In each CU, refIdxLX specifies which picture in the reference picture list RefPicListX (X=0 or 1) to actually reference. The diagram shows an example where refIdxL0=2 and refIdxL1=0. Note that LX is a notation method used when there is no distinction between L0 prediction and L1 prediction; hereinafter, parameters for the L0 list and parameters for the L1 list will be distinguished by replacing LX with L0 or L1.

[0084] (Merge prediction and AMVP prediction) Prediction parameter decoding (encoding) methods include merge prediction mode and AMVP (Advanced Motion Vector Prediction) mode, and merge_flag is a flag for distinguishing between them. Merge prediction mode is a mode in which the prediction list usage flag predFlagLX, reference picture index refIdxLX, and motion vector mvLX are not included in the encoded data, but are derived from prediction parameters of already processed neighboring blocks, etc. AMVP mode is a mode in which inter_pred_idc, refIdxLX, and mvLX are included in the encoded data. Note that mvLX is encoded as mvp_LX_idx, which identifies the prediction vector mvpLX, and a difference vector mvdLX. In addition to merge prediction mode, affine prediction mode and MMVD prediction mode may also be used.

[0085] inter_pred_idc is a value indicating the type and number of reference pictures, and takes one of the values ​​PRED_L0, PRED_L1, and PRED_BI. PRED_L0 and PRED_L1 are managed by the L0 list and the L1 list, respectively. PRED_BI is managed by the L0 list and the L1 list. This shows bi-prediction using two reference pictures.

[0086] merge_idx is the prediction parameter candidate (merge candidate) derived from the processed block. (Auxiliary) is an index indicating which prediction parameter is to be used as the prediction parameter for the current block.

[0087] (motion vector) mvLX indicates the amount of shift between blocks on two different pictures. The predicted vector and differential vector related to mvLX are called mvpLX and mvdLX, respectively.

[0088] (Inter prediction identifier inter_pred_idc and prediction list usage flag predFlagLX) The relationship between inter_pred_idc, predFlagL0, and predFlagL1 is as follows, and they can be converted to each other: inter_pred_idc = (predFlagL1<<1)+predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 Note that the inter prediction parameters may use a prediction list usage flag or an inter prediction identifier. Furthermore, the determination using the prediction list usage flag may be replaced with a determination using the inter prediction identifier. Conversely, the determination using the inter prediction identifier may be replaced with a determination using the prediction list usage flag.

[0089] (Configuration of image decoding device) The configuration of an image decoding device 31 (FIG. 7) according to this embodiment will be described.

[0090] The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, and a prediction image A generation unit (prediction image generation device) 308, an inverse quantization and inverse transformation unit 311, an addition unit 312, a prediction parameter The image decoding device 11 is configured to include a data derivation unit 320. In some configurations, the device 31 does not include a loop filter 305 .

[0091] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit. The CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice headers (slice information) from the coded data. The CT information decoding unit 3021 decodes the CT from the coded data. The CU decoding unit 3022 decodes the CU from the coded data. When a prediction error is included in the TU, the TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the coded data.

[0092] The TU decoding unit 3024 decodes the QP update information and the quantized prediction error from the coded data when the mode is other than the skip mode (skip_mode==0). More specifically, when skip_mode==0, the TU decoding unit 3024 decodes the flag cu_cbp indicating whether or not the current block contains a quantized prediction error, and decodes the quantized prediction error when cu_cbp is 1. When cu_cbp does not exist in the coded data, In this case, it is derived as 0.

[0093] The TU decoding unit 3024 decodes the index mts_idx indicating the transformation base from the coded data. The TU decoding unit 3024 also decodes an index stIdx, which indicates the use of a secondary transform and the transform base, from the coded data. When stIdx is 0, it indicates that no secondary transform is applied, when it is 1, it indicates one of the transforms in a set (pair) of secondary transform bases, and when it is 2, it indicates one of the transforms in the pair. The other transformation is shown below.

[0094] The predicted image generating unit 308 includes an inter predicted image generating unit 309 and an intra predicted image generating unit 310. It is composed of:

[0095] The prediction parameter derivation unit 320 includes an inter prediction parameter derivation unit 303 and an intra prediction parameter derivation unit 304 .

[0096] The entropy decoding unit 301 performs entropy decoding on the externally input coded data Te. The entropy decoder 301 performs decoding to decode individual codes (syntax elements). Entropy coding can be divided into two types: one is a method that performs variable-length coding of syntax elements using a context (probability model) adaptively selected according to the type of syntax element and surrounding circumstances, and the other is a method that performs variable-length coding of syntax elements using a predetermined table or formula. The former, CABAC (Context Adaptive Binary Arithmetic Coding), stores the CABAC state of the context (probability state index pStateIdx, which specifies the type of most-probable symbol (0 or 1) and its probability) in memory. The entropy decoder 301 initializes all CABAC states at the beginning of a segment (tile, CTU row, slice). The entropy decoder 301 converts the syntax elements into a binary string (bin string) and decodes each bit of the bin string. When a context is used, a context index ctxInc is derived for each bit of the syntax element, the bit is decoded using the context, and the CABAC state of the used context is updated. Bits without context are decoded with equal probability (EP, bypass), and the ctxInc derivation and CABAC state are omitted. The decoded syntax elements include prediction information for generating a predicted image and prediction error for generating a difference image.

[0097] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. The decoded code is, for example, a prediction mode predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0098] (Basic flow) FIG. 8 is a flowchart illustrating the general operation of the image decoding device 31.

[0099] (S1100: Decode Parameter Set Information) The header decoder 3020 decodes parameter set information such as VPS, SPS, and PPS from the coded data.

[0100] (S1200: Decode slice information) The header decoding unit 3020 decodes the slice header from the encoded data. Decode (slice information).

[0101] Hereinafter, the image decoding device 31 performs the steps S1300 to S5000 for each CTU included in the target picture. By repeating the process, a decoded image of each CTU is derived.

[0102] (S1300: Decode CTU Information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0103] (S1400: Decode CT Information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0104] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data. Issued.

[0105] (S1510: Decode CU information) The CU decoding unit 3022 decodes CU information, prediction information, and TU division information from the encoded data. The flag split_transform_flag, the CU residual flags cbf_cb, cbf_cr, cbf_luma, etc. are decoded.

[0106] (S1520: TU information decoding) When a TU includes a prediction error, the TU decoding unit 3024 The QP update information, the quantization prediction error, and the transform index mts_idx are decoded from the data. Note that the QP update information is derived from the quantization parameter predicted value qPpred, which is the predicted value of the quantization parameter QP. This is the difference value.

[0107] (S2000: Generate predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on prediction information.

[0108] (S3000: Inverse Quantization and Inverse Transformation) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing on each TU included in the target CU.

[0109] (S4000: Generate decoded image) The adder 312 generates a decoded image using the predicted image supplied from the predicted image generator 308. , and the prediction error supplied from the inverse quantization and inverse transform unit 311 are added to obtain the target CU. A decoded image is generated.

[0110] (S5000: Loop filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

[0111] (Configuration of the inter-prediction parameter derivation unit) The inter-prediction parameter derivation unit 303 (motion vector derivation device) decodes the prediction parameters stored in the prediction parameter memory 307 based on the syntax elements input from the parameter decoding unit 302. The inter prediction parameter derivation unit 303 derives inter prediction parameters by referring to the measured parameters. The inter prediction parameters are output to the inter prediction image generation unit 309 and the prediction parameter memory 307. The inter prediction parameter derivation unit 303 and its internal elements, namely, the AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, the affine prediction unit 30372, the MMVD prediction unit 30373, the GPM unit 30377, the DMVR unit 30537, and the MV addition unit 3038, are means common to the image encoding device and the image decoding device, and therefore may be collectively referred to as a motion vector derivation unit (motion vector derivation device).

[0112] The scale parameter derivation unit 30378 included in the header decoding unit 3020 and the header encoding unit 1110 , derive the horizontal scaling ratio of the reference picture RefPicScale[i][j][0], the vertical scaling ratio of the reference picture RefPicScale[i][j][1], and RefPicIsScaled[i][j], which indicates whether the reference picture is scaled or not, where i indicates whether the reference picture list is an L0 list or an L1 list, and j is the value of the L0 reference picture list or the L1 reference picture list, as follows: RefPicScale[i][j][0] = ((fRefWidth << 14)+(PicOutputWidthL >> 1)) / PicOutputWidthL RefPicScale[ i ][ j ]

[0001] = ((fRefHeight << 14)+(PicOutputHeightL >> 1)) / PicOutputHeightL RefPicIsScaled[i][j] = (RefPicScale[i][j][0] != (1<<14)) || (RefPicScale[i][j][1] != (1<<14)) Here, the variable PicOutputWidthL is the horizontal scale when the coded picture is referenced. The variable PicOutputHeightL is the value used when calculating the scaling ratio, and is the number of pixels in the horizontal direction of the luminance of the picture to be coded minus the left and right offset values. The variable PicOutputHeightL is the value used when calculating the vertical scaling ratio when the picture to be coded is referenced, and is the number of pixels in the vertical direction of the luminance of the picture to be coded minus the top and bottom offset values. The variable fRefWidth is the value of PicOutputWidthL of the reference picture with reference list value j in list i, and variable fRefHight is the value of PicOutputHeightL of the reference picture with reference picture list value j in list i.

[0113] (MV addition section) The MV addition unit 3038 calculates mvLX by adding the mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded mvdLX. The addition unit 3038 outputs the calculated mvLX to the inter predicted image generation unit 309. and output to the predicted parameter memory 307: mvLX[0] = mvpLX[0]+mvdLX[0] mvLX[1] = mvpLX[1] + mvdLX[1] The loop filter 305 is a filter provided in the encoding loop, and is used to remove block distortion and ringing. The loop filter 305 is a filter that removes distortion and improves image quality. The loop filter 305 applies a deblocking filter, a sample adaptive offset (SAO), and an adaptive filter to the decoded image of the CU generated by the adder 312. Apply a filter such as an adaptive loop filter (ALF).

[0114] The reference picture memory 306 stores the decoded image of the CU in a predetermined format for each target picture and target CU. Store in the location.

[0115] The prediction parameter memory 307 stores prediction parameters at a predetermined location for each CTU or CU. Specifically, the prediction parameter memory 307 stores parameters decoded by the parameter decoding unit 302, parameters derived by the prediction parameter derivation unit 320, and the like.

[0116] The predicted image generation unit 308 receives the parameters derived by the prediction parameter derivation unit 320. The predicted image generation unit 308 also reads a reference picture from the reference picture memory 306. The predicted image generation unit 308 generates a prediction image based on the parameters and the reference picture (reference picture A reference picture block is a set of pixels (usually rectangular, hence the name block) on the reference picture, and is the area referenced to generate a predicted image.

[0117] When predMode indicates an inter prediction mode, the inter predicted image generation unit 309 Using the inter-prediction parameters and reference pictures input from the prediction parameter derivation unit 303, A predicted image of the block or sub-block is generated by inter-prediction.

[0118] (motion compensation) The motion compensation unit 3091 (interpolated image generation unit 3091) receives the input from the inter prediction parameter derivation unit 303. Based on the input inter prediction parameters (predFlagLX, refIdxLX, mvLX), an interpolated image (motion compensated image) is generated by reading reference blocks from the reference picture memory 306. The reference block is a block located at a position shifted by mvLX from the position of the current block on the reference picture RefPicLX specified by refIdxLX. If mvLX does not have integer precision, an interpolated image is generated by applying a filter called a motion compensation filter, which generates pixels at decimal positions.

[0119] The motion compensation unit 3091 first derives the integer position (xInt, yInt) and phase (xFrac, yFrac) corresponding to the coordinates (x, y) in the prediction block using the following formula: xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x xFrac = mvLX[0]&(MVPREC-1) yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y yFrac = mvLX[1]&(MVPREC-1) Here, (xPb, yPb) are the upper left coordinates of a bW*bH size block, x=0...bW-1, y=0...bH-1, and MVPREC indicates the accuracy of mvLX (1 / MVPREC pixel accuracy), e.g., MVPREC=16.

[0120] The motion compensation unit 3091 derives the temporary image temp[][] by performing horizontal interpolation on the reference picture refImg using an interpolation filter. In the following, Σ is the sum over k=0..NTAP-1, shift1 is a normalization parameter that adjusts the value range, and offset1=1<<(shift1-1): temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1 Next, the motion compensation unit 3091 derives the interpolated image Pred[][] by vertically interpolating the temporary image temp[][]. In the following, Σ is the sum over k=0..NTAP-1, shift2 is a normalization parameter that adjusts the value range, and offset2=1<<(shift2-1): Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2 In the case of bi-prediction, the above Pred[][] is derived for each L0 list and L1 list (called interpolated images PredL0[][] and PredL1[][]), and the interpolated image Pred[][] is generated from PredL0[][] and PredL1[][].

[0121] The motion compensation unit 3091 uses the reference picture derived by the scale parameter derivation unit 30378. The interpolated image is scaled according to the horizontal scaling ratio RefPicScale[i][j][0] of the reference picture and the vertical scaling ratio RefPicScale[i][j][1] of the reference picture.

[0122] If predMode indicates an intra prediction mode, the intra predicted image generation unit 310 Intra prediction is performed using the intra prediction parameters input from the prediction parameter derivation unit 304 and reference pixels read from the reference picture memory 306 .

[0123] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0124] The adder 312 adds, for each pixel, the predicted image of the block input from the predicted image generator 308 and the prediction error input from the inverse quantization and inverse transformer 311 to generate a decoded image of the block. The adder 312 stores the decoded image of the block in the reference picture memory 306 and also outputs it to the loop filter 305 .

[0125] The inverse quantization and inverse transform unit 311 inverse quantizes the quantized transform coefficients input from the parameter decoding unit 302 to obtain transform coefficients.

[0126] The adder 312 adds, for each pixel, the predicted image of the block input from the predicted image generator 308 and the prediction error input from the inverse quantization and inverse transformer 311 to generate a decoded image of the block. The adder 312 stores the decoded image of the block in the reference picture memory 306 and also outputs it to the loop filter 305 .

[0127] (Configuration example of NN filter unit 611) FIG. 16 shows a post-filtering process using a neural network filter unit (NN filter unit 611). 10 illustrates an example of the configuration of a filter.

[0128] The post-processing unit 61 after the video decoding device includes an NN filter unit 611. When outputting an image from the reference picture memory 306, the NN filter unit 611 processes the image and outputs it to the outside. The output image may be displayed, written to a file, re-encoded (transcoded), transmitted, etc. The NN filter unit 611 is a means for performing filtering on the input image using a neural network model. At the same time, reduction or enlargement may be performed at the same magnification or a rational number magnification.

[0129] Here, a neural network model (hereafter referred to as an NN model) refers to the elements and connections (topology) of a neural network, as well as the parameters (weights and biases) of the neural network. Note that the topology may be fixed, and only the parameters of the neural network model may be switched.

[0130] (Details of the NN filter unit 611) The NN filter uses the input image inSamples and input parameters (e.g., QP, bS, etc.), Filtering is performed using a neural network model. The input image may be an image for each component, or multiple components may be Alternatively, the input parameters may be assigned to channels different from the image.

[0131] The NN filter unit may repeatedly apply the following process.

[0132] As shown in the following equation, the kernel k[m][i][j] is convolved (conv, convolution) with inSamples and bias is added to derive the output image outSamples, where nn=0..n-1, xx=0..width-1, yy=0..height-1.

[0133] outSamples[nn][xx][yy]=ΣΣΣ(k[mm][i][j]*inSamples[mm][xx+i-of][yy+j-of]+bias[nn]) In the case of 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0. In this case, set of=0. In the case of 3x3 Conv, Σ represents the sum of mm=0..m-1, i=0..2, j=0..2, respectively. In this case, of=1 is set. n is the number of channels in outSamples, m is the number of channels in inSamples, width is the width of inSamples and outSamples, and height is the height of inSamples and outSamples. of is the size of the padding area placed around inSamples to make the sizes of inSamples and outSamples the same. In the following, when the output of the NN filter section is a value (corrected value) rather than an image, the output will be represented as corrNN instead of outSamples.

[0134] Alternatively, a process called Depth-wise Conv may be performed, which is expressed by the following equation: where nn=0..n-1, xx=0..width-1, and yy=0..height-1.

[0135] outSamples[nn][xx][yy]=ΣΣ(k[nn][i][j]*inSamples[nn][xx+i-of][yy+j-of]+bias[nn]) Σ represents the sum for i and j, respectively. n is the number of channels in outSamples and inSamples, width is the width of inSamples and outSamples, and height is the height of inSamples and outSamples.

[0136] Alternatively, a nonlinear process called Activate, such as ReLU, may be used. ReLU(x) = x >= 0 ? x : 0 Alternatively, leakyReLU shown in the following formula may be used. leakyReLU(x) = x >= 0 ? x : a * x Here, a is a predetermined value, for example, 0.1 or 0.125. In order to perform integer arithmetic, all of the above values ​​of k, bias, and a may be integers, and a right shift may be performed after conv.

[0137] In ReLU, values ​​less than 0 are always output as 0, and values ​​greater than or equal to 0 are output as the input value. On the other hand, in leakyReLU, values ​​less than 0 are processed linearly with the gradient set by a. In ReLU, the gradient for values ​​less than 0 disappears, which may make it difficult to progress in learning. In leakyReLU, gradients for values ​​less than 0 are left, making the above problem less likely to occur. Also, among the above leakyReLU(x), PReLU, which uses a parameterized value of a, may be used.

[0138] (SEI for neural network model complexity reference) 11 is a diagram showing the structure of a syntax table of the NN filter SEI of this embodiment. This SEI includes information on the complexity of the neural network model. ·nnrpf_id: The identification number of the NN filter. nnrpf_mode_idc: An index indicating the mode of specifying the neural network model used for the NN filter. A value of 0 indicates that the NN filter associated with nnrpf_id is not specified in this SEI message. A value of 1 indicates that the NN filter associated with nnrpf_id is a neural network model identified by a specific URI (Uniform Resource Identifier). A URI is an identification string that indicates a logical or physical resource. Note that actual data does not need to exist at the location indicated by the URI; it is sufficient if the string can identify the resource. A value of 2 indicates that the NN filter associated with nnrpf_id is a neural network model represented by the ISO / IEC 15938-17 bit stream included in this SEI message. A value of 3 indicates that the NN filter associated with nnrpf_id is a neural network identified in the NN filter SEI message used in the previous decoding and updated by the ISO / IEC 15938-17 bit stream included in this SEI message. nnrpf_persistence_flag: Specifies the persistence of this SEI message for the current layer. A value of 0 indicates that this SEI message applies only to the current decoded picture. A value of 1 indicates that this SEI message applies to the current decoded picture and all subsequent pictures. This indicates to the chat that they are applied in output order. nnrpf_uri[i]: Reference URI of the neural network model to be used as the NN filter The string to store i, where i is the i-th byte of the NULL-terminated UTF-8 string. ·nnrpf_payload_byte[i]: Indicates the i-th byte of the bitstream conforming to ISO / IEC 15938-17. When nnrpf_mode_idc == 1, the header encoding unit 1110 and the header decoding unit 3020 use the NN filter The neural network model corresponding to the character string indicated by nnrpf_uri is read from the memory of the video encoding device or video decoding device, or read from an external device via the network.

[0139] The NN filter SEI is a neural network model complexity information (network model complexity). The following syntax elements are included as miscellaneous information: nnrpf_parameter_type_idc: An index indicating the variable type included in the parameters of the NN model. If the value is 0, the NN model uses only integer types. If the value is 1, the NN model uses either floating-point or integer types. nnrpf_num_parameters_idc: An index indicating the number of parameters of the NN model used in the post filter. If the value is 0, it means that the number of parameters of the NN model is not defined. If the value is not 0, the following process is performed using nnrpf_num_parameters_idc to calculate the performance of the NN model. Derive the number of parameters.

[0140] The header encoding unit 1110 and the header decoding unit 3020 derive the maximum value MaxNNParameters of the number of parameters of the NN model based on nnrpf_num_parameters_idc as follows, and The model complexity information may be encoded and decoded.

[0141] MaxNNParameters = (UNITPARAM << nnrpf_num_parameters_idc) - 1 Here, UNITP ARAM is a predetermined constant, and UNITP ARAM=2048=2^11 may be used. Note that the shift operation is equivalent to the exponent and can be the following:

[0142] MaxNNParameters = 2 ^ (nnrpf_num_parameters_idc+11) - 1 Also, the unit of the number of parameters is not doubled, but is a combination of doubled and 1.5x units as shown below. You can also combine them.

[0143] MaxNNParameters = (nnrpf_num_parameters_idc & 1) ? (UNITPARAM2 << nnrpf_num_parameters_idc) - 1 : (UNITPARAM << nnrpf_num_parameters_idc) - 1 Here, UNITP ARAM2 may be a predetermined constant that is UNITP ARAM * 1.5. For example, if UNITP ARAM = 2048, then UNITP ARAM2 = 3072. The following is also possible:

[0144] MaxNNParameters = (nnrpf_num_parameters_idc & 1) : 2 ^ (nnrpf_num_parameters_i dc+11)*1.5 - 1 : 2 ^ (nnrpf_num_parameters_idc+11) - 1 That is, the header encoding unit 1110 sets the value of nnrpf_parameter_type_idc so that the number of parameters of the actual NN model is equal to or less than MaxNNParameters. The coded data set as above is decoded.

[0145] Note that a linear expression may be used to derive MaxNNParameters.

[0146] MaxNNParameters = (UNITPARAM * nnrpf_num_parameters_idc) -1 In this case, UNITPARAM may be 10000. UNITPARAM is preferably a value of 1000 or more, and is preferably a multiple of 10. nnrpf_num_kmac_operations_idc: Number of operations required for post-filter processing The header encoding unit 1110 and the header decoding unit 3020 calculate MaxNNOperations based on the above nnrpf_num_kmac_operations_idc as follows: MaxNNOperations is the maximum number of operations required for post-filter processing.

[0147] MaxNNOperations = nnrpf_num_kmac_operations_idc * 1000 * picture width * picture height Here, picture width and picture height are the width of the picture input to the post filter. , height.

[0148] That is, the video encoder uses the value of nnrpf_num_kmac_operations_idc as a post-filter. The number of operations required to process the filter is set according to the value of nnrpf_num_kmac_operations. .

[0149] The above method achieves the effect of transmitting, encoding, or decoding the syntax of network model complexity information related to the amount of processing defined in units of a predetermined constant, thereby enabling the transmission of complexity concisely. Furthermore, using multiples of 10 has the effect of making the value easier for humans to understand. Furthermore, using a value of 1000 or more makes it possible to appropriately express the scale of the model with a small number of divisions, allowing for efficient transmission.

[0150] That is, the syntax indicating the network model complexity information indicates an upper limit on the number of parameters or the number of operations, and the number of parameters or the number of operations is defined in units of exponential powers of two. The number of parameters or the number of operations may be defined in units of an exponential power of 2 or 1.5 times an exponential power of 2. The number of parameters or the number of operations may also be defined in units of a multiple of 10.

[0151] The above further has the effect of enabling efficient transmission of complexity using short codes by transmitting, encoding, or decoding the syntax of network model complexity information related to the processing amount defined by shift representation or exponential representation. nnrpf_alignment_zero_bit: A bit for byte alignment. The header encoding unit 1110 and the header decoding unit 3020 encode and decode the code "0" one bit at a time until the bit position reaches a byte boundary. nnrpf_operation_type_idc: An index that indicates the element restriction or topology restriction used in the post-filter NN model. Depending on the index value, for example, The process may be carried out as follows:

[0152] If the value is 3, the element or topology is limited to the maximum kernel size. The size is 5x5, the maximum number of channels is 32, and the activation function can be leaky ReLU or only ReLU. The maximum branch level is 3 (excluding skip connections).

[0153] If the value is 2, in addition to the above, it uses leaky Relu as the activation function and skip connections. Branching other than the above is prohibited (e.g., U-Net and grouped convolutions are not performed).

[0154] If the value is 1, in addition to the above, spatial to channel mapping (e.g., Pixel Shuffler) and global average pooling are prohibited.

[0155] If the value is 0, the element or topology is not restricted. <Another configuration example 1> The parameter nnrpf_parameter_type_idc indicating the network model complexity information may be defined as follows: nnrpf_parameter_type_idc: An index indicating the parameter type of the neural network model. For example, the parameter type may be determined according to the parameter value as follows: If the value is 0, it is defined as an 8-bit unsigned integer type, if the value is 1, it is defined as a 16-bit unsigned integer type, if the value is 2, it is defined as a 32-bit unsigned integer type, if the value is 3, it is defined as a 16-bit floating-point type (bfloat16), if the value is 4, it is defined as a 16-bit floating-point type (half precision), and if the value is 5, it is defined as a 32-bit floating-point type (single precision). <Another configuration example 2> Figure 12 shows the structure of the syntax table of the NN filter SEI with network model complexity information. In this example, the parameter types of the neural network model are defined as numerical types and bit widths.

[0156] In this configuration example, in addition to the above SEI syntax, the following syntax information is included. nnrpf_parameter_type_idc: An index indicating the numeric type of the neural network model. For example, the numeric type may be determined according to the parameter value as follows: If the value is 0, it is defined as an integer type, and if the value is 1, it is defined as a floating-point type. nnrpf_parameter_bit_width_idc: The bit width of the neural network model. For example, the bit width may be determined as follows depending on the parameter value: A value of 0 is defined as 8 bits, a value of 1 is defined as 16 bits, and a value of 2 is defined as 32 bits. <Another configuration example 3> Figure 13 shows the structure of the syntax table of the NN filter SEI with network model complexity information. Here, the bit width of the parameter type of the neural network model is defined in logarithmic notation.

[0157] In this configuration example, instead of nnrpf_parameter_bit_width_idc, use the following syntax information: Includes: nnrpf_log2_parameter_bit_width_minus3: Neural network model parameter Based on nnrpf_log2_parameter_bit_width_minus3, the parameter bit width, parameterBitWidth, is calculated as follows: parameterBitWidth = 1 << ( nnrpf_log2_parameter _bit_width_minus3 + 3 ) (SEI decoding and post-filtering) The header decoder 3020 derives the network model complexity from the SEI message defined in FIG. The SEI is auxiliary information for processing related to decoding, display, etc.

[0158] 14 is a flowchart showing the processing of the NN filter unit 611. The NN filter unit 611 performs the following processing in accordance with the parameters of the SEI message. S6001: Read the processing amount and accuracy from the network model complexity information of SEI. S6002: End if the complexity exceeds the processable level of the NN filter unit 611. If not, proceed to S6003. S6003: If the accuracy exceeds the processing capability of the NN filter unit 611, the process ends. If not, the process proceeds to S6004. S6004: A network model is identified from the URI in the SEI, and the topology of the NN filter unit 611 is set. S6005: Derive parameters of the network model from the updated information of the SEI. S6006: The derived parameters of the network model are read into the NN filter unit 611. S6007: The NN filter unit 611 executes filtering and outputs the result to the outside. However, the SEI is not required for constructing luma and chroma samples during decoding. .

[0159] (SEI for neural network model data format reference) FIG. 18 is a diagram showing another configuration of the syntax table of the NN filter SEI of this embodiment. This SEI contains information in the neural network model data format. Explanation of the same information as in the SEI containing the information on the neural network model complexity already explained will be omitted. nnrpf_input_format_idc: Input tensor identification parameter. Indicates the format of input data (input tensor) of the NN model used in the NN filter. The header decoding unit 3020 derives the format of the input data based on the value of nnrpf_input_format, as shown in FIG. 19 .

[0160] If nnrpf_input_format_idc == 0, the input data format is 3D with 1 channel (luminance). The luminance channel of the decoded image is used as input data to the post-filter. In this embodiment, the three dimensions of the three-dimensional data are defined as (C, H, W) in order, but the order of the dimensions is not limited to this. For example, the data may be stored in the order of (H, W, C). In this case, since there is one channel, the data may be two-dimensional data (H, W).

[0161] If nnrpf_input_format_idc == 1, the input data format is 2-channel (chrominance) 3D The two chrominance channels (U and V) of the decoded image are This indicates that the data is used as input data to the

[0162] If nnrpf_input_format_idc == 2, the input data format is 3 channels (luminance and 2 This is a 3D tensor of the luminance and two chrominance channels of the decoded image. It indicates that the luminance and two chrominance channels of the decoded image are used as input data to the post filter in YUV4:4:4 format.

[0163] If nnrpf_input_format_idc == 3, the input data format is 6 channels (4 luminance and This indicates that four channels and two chrominance channels derived from the luminance channel of a decoded image in YUV4:2:0 format are used as input data to the post filter.

[0164] nnrpf_output_format_idc: Output tensor identification parameter. Indicates the format of the output data (NN output data, output tensor) of the NN model used in the NN filter. Header Decoding Unit 3020 derives the format of the input data based on the value of nnrpf_output_format, as shown in Figure 20. do.

[0165] If nnrpf_output_format_idc == 0, the format of the NN output data is 3D data (3D tensor) with one channel (luminance). The output data from the post filter is used as the luminance channel of the output image.

[0166] If nnrpf_output_format_idc == 1, the format of the NN output data is 2-channel (color difference) 3D data (3D tensor). The output data from the post filter is the two color Indicates that it is used as the difference channel (U and V).

[0167] If nnrpf_output_format_idc == 2, the format of the NN output data is 3-channel (luminance and two chrominance) 3D data (3D tensor). The output data from the post filter is The input image is shown to be used in YUV4:4:4 format as the luminance and two chrominance channels.

[0168] If nnrpf_output_format_idc == 3, the format of the NN output data is 3D data (3D tensor) with 6 channels (4 luminance and 2 chrominance). The output data from the post filter is used in YUV4:2:0 format as 1 luminance channel and 2 chrominance channels derived by integrating 4 of the 6 channels.

[0169] (Post-filter SEI processing) The header encoding unit 1110 and the header decoding unit 3020 perform the following in the processing of the post filter SEI: As described above, syntax values ​​and variables of the image decoding device may be set as variables for filtering. PicWidthInLumaSamples = pps_pic_width_in_luma_samples PicHeightInLumaSamples = pps_pic_height_in_luma_samples ChromaFormatIdc = sps_chroma_format_idc BitDepthY= BitDepthC = BitDepth ComponentSample[ cIdx ] is a two-dimensional array that stores the cIdxth decoded sample pixel value of the decoded image. Here, pps_pic_width_in_luma_samples, pps_pic_height_in_luma_samples, and sps_chroma_format_idc are syntax values ​​indicating the image width, height, and color component subsamples, and BitDepth is the bit depth of the image.

[0170] The header encoding unit 1110 and the header decoding unit 3020 derive the following variables according to ChromaFormatIdc (sps_chroma_format_idc). SubWidthC = 1, SubHeghtC = 1 (ChromaFormatIdc == 0) SubWidthC = 2, SubHeghtC = 2 (ChromaFormatIdc == 1) SubWidthC = 2, SubHeghtC = 1 (ChromaFormatIdc == 2) SubWidthC = 1, SubHeghtC = 1 (ChromaFormatIdc == 3) The header encoding unit 1110 and the header decoding unit 3020 derive the image width and height of the luminance and the screen width and height of the chrominance to be filtered using the following variables. LumaWidth = PicWidthInLumaSamples LumaHeight = PicHeightInLumaSamples ChromaWidth = PicWidthInLumaSamples / SubWidthC ChromaHeight = PicHeightInLumaSamples / SubHeithtC SW = SubWidthC SW = SubHeightC SubWidthC(=SW) and SubHeightC(=SH) indicate the subsampling of the color components. is a variable that represents the ratio of color difference resolution to luminance resolution.

[0171] Furthermore, the header encoding unit 1110 and the header decoding unit 3020 may derive the output image width outWidth and height outHeight according to a scale value indicating a ratio as follows: outLumaWidth = LumaWidth * scale outLumaHeight = LumaHeight * scale outChromaWidth = LumaWidth * scale / outSW outChromaHeight = LumaHeight * scale / outSH (Conversion of post-filter input data to NN) When inputting image data to the NN filter, the NN filter unit 611 converts the decoded image into NN input data inputTensor[][][], which is a three-dimensional array, based on the value of nnrpf_input_format_idc, as shown in FIG. 21 and below. In the following, x and y represent the coordinates of the luminance pixel. For example, in ComponentSample, the range of x and y is x = 0..LumaWidth-1, y = LumaHeight-1 respectively. cx and cy represent the coordinates of the chrominance pixel, and the range of cx and cy is x = 0..ChromaWidth-1, y = ChromaHeight-1 respectively. The following NN filter The filter unit 611 processes this range. If nnrpf_input_format_idc is 0 (pfp_component_idc==0), the inputTensor is derived as follows: inputTensor[0][y][x] = ComponentSample[0][x][y] If nnrpf_input_format_idc is 1 (pfp_component_idc==1), the inputTensor is derived as follows: inputTensor[0][cy][cx] = ComponentSample[1][cx][cy] inputTensor[1][cy][cx] = ComponentSample[2][cx][cy] Or: inputTensor[0][y / SH][x / SW] = ComponentSample[1][x / SW][y / SH] inputTensor[1][y / SH][x / SW] = ComponentSample[2][x / SW][y / SH] If nnrpf_input_format_idc is 2 (pfp_component_idc==2), inputTensor is derived as follows: inputTensor[0][y][x] = ComponentSample[0][x][y] ChromaOffset = 1<<(BitDepthC-1) inputTensor[1][y][x] = ChromaFormatIdc == 0 ? ChromaOffset : ComponentSample[1][x / SW][y / SH] inputTensor[2][y][x] = ChromaFormatIdc == 0 ? ChromaOffset : ComponentSample[2][x / SW][y / SH] If nnrpf_input_format_idc is 3, the inputTensor is derived as follows: inputTensor[0][cy][cx] = ComponentSample[0][cx*2 ][cy*2 ] inputTensor[1][cy][cx] = ComponentSample[0][cx*2+1][cy*2 ] inputTensor[2][cy][cx] = ComponentSample[0][cx*2 ][cy*2+1] inputTensor[3][cy][cx] = ComponentSample[0][cx*2+1][cy*2+1] ChromaOffset = 1<<(BitDepthC-1) inputTensor[4][cy][cx] = ChromaFormatIdc == 0 ? ChromaOffset : ComponentSample[1][cx][cy] inputTensor[5][cy][cx] = ChromaFormatIdc == 0 ? ChromaOffset : ComponentSample[2][cx][cy] If ChromaFormatIdc is 0, the ComponentSample is an image with only the luminance channel. Then, the NN filter unit 611 sets a constant ChromaOffset derived from the bit depth to the chrominance data portion of the inputTensor. ChromaOffset may be set to another value such as 0. As shown in parentheses, the NN input data may be derived according to pfp_component_idc, which will be described later. stomach.

[0172] The NN filter unit 611 separates a one-channel luminance image into four channels according to pixel positions. The NN filter unit 611 may be derived as follows: , 4:2:0, 4:2:2, 4:4:4 differences are absorbed by the variable indicating the color subsample. It can be handled regardless of the format: inputTensor[0][cy][cx] = ComponentSample[0][cx*2 ][cy*2 ] inputTensor[1][cy][cx] = ComponentSample[0][cx*2+1][cy*2 ] inputTensor[2][cy][cx] = ComponentSample[0][cx*2 ][cy*2+1] inputTensor[3][cy][cx] = ComponentSample[0][cx*2+1][cy*2+1] ChromaOffset = 1<<(BitDepthC-1) inputTensor[4][cy][cx] = ChromaFormatIdc == 0 ? ChromaOffset : ComponentSample[1][cx*2 / SW][cy*2 / SH] inputTensor[5][cy][cx] = ChromaFormatIdc == 0 ? ChromaOffset : ComponentSample[2][cx*2 / SW][cy*2 / SH] Furthermore, since cx = x / 2 and cy = y / 2, the NN filter unit 611 calculates the following for the range of x and y described above: and may be derived as follows: inputTensor[0][y / 2][x / 2] = ComponentSample[0][x / 2*2 ][y / 2*2 ] inputTensor[1][y / 2][x / 2] = ComponentSample[0][x / 2*2+1][y / 2*2 ] inputTensor[2][y / 2][x / 2] = ComponentSample[0][x / 2*2 ][y / 2*2+1] inputTensor[3][y / 2][x / 2] = ComponentSample[0][x / 2*2+1][y / 2*2+1] ChromaOffset = 1<<(BitDepthC-1) inputTensor[4][y / 2][x / 2] = ChromaFormatIdc == 0 ? ChromaOffset : ComponentSample[1][x / SW][y / SH] inputTensor[5][y / 2][x / 2] = ChromaFormatIdc == 0 ? ChromaOffset : ComponentSample[2][x / SW][y / SH] As will be described later, the above processing may be performed in accordance with numInChannels.

[0173] (Conversion from post-filter NN output data) Based on the value of nnrpf_output_format_idc, the NN filter unit 611 derives the output image outSampless from the NN output data outputTensor[][][], which is a three-dimensional array that is the output data of the NN filter. Specifically, as shown in FIG. 22 and below, the NN filter unit 611 derives the image as follows based on the value of nnrpf_output_format_idc and the chrominance subsample values ​​outSW and outSH of the output image. Note that outSW and outSH use values ​​obtained by decoding OutputChromaFormatID from the encoded data (described below). Below, x and y represent the coordinates of the luminance pixel of the output image. For example, in outputSample, the ranges of x and y are x = 0..outLumaWidth-1 and y = outLumaHeight-1, respectively. cx and cy represent the coordinates of the chrominance pixel of the output image, and the ranges of cx and cy are x = 0..outChromaWidth-1 and y = 0..outChromaHeight-1, respectively. The following NN filter unit 611 processes this range: outSamplesL, outSamplesCb, and outSamplesCr represent the luminance channel, chrominance (Cb) channel, and chrominance (Cr) channel of the output image, respectively.

[0174] If nnrpf_output_format_idc is 0, outSamplesL is derived as follows: outSamplesL[x][y] = outputTensor[0][y][x] If nnrpf_output_format_idc is 1, outSamplesCb and outSamplesCr are derived as follows: outSamplesCb[cx][cy] = outputTensor[0][cy][cx] outSamplesCr[cx][cy] = outputTensor[1][cy][cx] If nnrpf_output_format_idc is 2, derive outSamplesL as follows: outSamplesL[x][y] = outputTensor[0][y][x] In this case, outSamplesCb and outSamplesCr are derived as follows: outSamplesCb[x / outSW][y / outSH] = outputTensor[1][y][x] outSamplesCr[x / outSW][y / outSH] = outputTensor[2][y][x] Alternatively, outSamplesCb and outSamplesCr may be derived as follows: outSamplesCb[cx][cy] = outputTensor[1][cy*outSH][cx*outSW] outSamplesCr[cx][cy] = outputTensor[2][cy*outSH][cx*outSW] If nnrpf_output_format_idc is 3, derive outSamplesL as follows: outSamplesL[x / 2*2 ][y / 2*2 ] = outputTensor[0][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2 ] = outputTensor[1][y / 2][x / 2] outSamplesL[x / 2*2 ][y / 2*2+1] = outputTensor[2][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2+1] = outputTensor[3][y / 2][x / 2] In this case, if the output image is in YUV4:2:0 format (ChromaFormatIdc of the output image is 1, SW=SH=2), For example, outSamplesCb and outSamplesCr are derived as follows: outSamplesCb[cx][cy] = outputTensor[4][cy][cx] outSamplesCr[cx][cy] = outputTensor[5][cy][cx] In this case, if the output image is in YUV4:2:2 format (ChromaFormatIdc of the output image is 2, SW=2, SH=1), outSamplesCb and outSamplesCr are derived as follows: outSamplesCb[cx][cy / 2*2 ] = outputTensor[4][cy][cx] outSamplesCb[cx][cy / 2*2+1] = outputTensor[4][cy][cx] outSamplesCr[cx][cy / 2*2 ] = outputTensor[5][cy][cx] outSamplesCr[cx][cy / 2*2+1] = outputTensor[5][cy][cx] In this case, if the output image is in YUV4:4:4 format (ChromaFormatIdc of the output image is 3, SW=SH=1), For example, outSamplesCb and outSamplesCr are derived as follows: outSamplesCb[cx / 2*2 ][cy / 2*2 ] = outputTensor[4][cy][cx] outSamplesCb[cx / 2*2+1][cy / 2*2+1] = outputTensor[4][cy][cx] outSamplesCr[cx / 2*2 ][cy / 2*2 ] = outputTensor[5][cy][cx] outSamplesCr[cx / 2*2+1][cy / 2*2+1] = outputTensor[5][cy][cx] When nnrpf_output_format_idc is 3, outSamplesCb and outSamplesCr may be derived as follows: for (j=0; j <outSH; j++) for (i=0; i <outSW; i++) outSamplesCb[cx / outSW*outSW+i][cy / outSH*outSH+j] = outputTensor[4][cy][cx] outSamplesCr[cx / outSW*outSW+i][cy / outSH*outSH+j] = outputTensor[5][cy][cx] Furthermore, in the case of the YUV4:2:0 format, cx = x / 2 and cy = y / 2, so the NN filter unit 611 may derive outSamplesL as follows: outSamplesL[x / 2*2 ][y / 2*2 ] = outputTensor[0][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2 ] = outputTensor[1][y / 2][x / 2] outSamplesL[x / 2*2 ][y / 2*2+1] = outputTensor[2][y / 2][x / 2] outSamplesL[x / 2*2+1][y / 2*2+1] = outputTensor[3][y / 2][x / 2] outSamplesCb[x / outSW][y / outSH] = outputTensor[4][y / 2][x / 2] outSamplesCr[x / outSW][y / outSH] = outputTensor[5][y / 2][x / 2] As will be described later, the above processing may be performed in accordance with numOutChannels.

[0175] In addition, set outSW and outSH to the same as the color component sampling of the input data. outSW = SW outSW = SH It may be treated as (summary) The present application may be configured to decode coded data including an input tensor identification parameter that specifies a correspondence between a channel of an input tensor of a neural network model and a color component.

[0176] The present invention may also be configured such that a relational expression for deriving an input tensor from an input image is defined according to an input tensor identification parameter and color difference sampling of the input image.

[0177] Furthermore, the input tensor identification parameters of the present application may be configured to include designation of one channel, two channels, three channels, or six channels.

[0178] The present invention may also include a means for deriving an input tensor from an input image in accordance with an input tensor identification parameter.

[0179] The present application may be configured to decode coded data including an output tensor identification parameter that specifies the correspondence between the channels of the output tensor of the neural network model and the color components.

[0180] The present invention may be configured such that a relational expression for deriving the input tensor from the output image is defined according to the output tensor identification parameter and color difference sampling of the output image.

[0181] The output tensor identification parameters of the present application may include specifying one channel, two channels, three channels, or six channels.

[0182] The present invention may include means for deriving an output image from the output tensor in response to an output tensor identification parameter.

[0183] In this way, this SEI can determine the input data format to the post-filter and the output data format from the post-filter. This has the effect of making it possible to easily select a method for appropriately converting a decoded image into input data for a post filter, or a method for appropriately converting output data of a post filter into an output image, without having to load and analyze a model.

[0184] (Configuration for deriving input tensor identification parameters and output tensor parameters) The NN filter unit 601 receives input tensor identification parameters and outputs tensor identification parameters Instead of decoding from encoded data such as additional data, it may be transmitted in the encoded data or derived from the topology of the NN model identified by a URI or the like.

[0185] The NN filter unit 601 calculates the number of channels numInChannels of the input data inputTensor of the NN model. Derive nnrpf_input_format_idc accordingly as follows: For 1 channel, nnrpf_input_format_idc=0 For 2 channels, nnrpf_input_format_idc=1 For 3 channels, nnrpf_input_format_idc=2 For 6 channels, nnrpf_input_format_idc=3 The NN filter unit 601 calculates the number of channels numOutChannels of the output data outputTensor of the NN model. Derive nnrpf_output_format_idc as follows depending on the For 1 channel, nnrpf_output_format_idc=0 For 2 channels, nnrpf_output_format_idc=1 For 3 channels, nnrpf_output_format_idc=2 For 6 channels, nnrpf_output_format_idc=3 According to the above configuration, the NN filter unit 601 receives the NN transmitted or specified in the coded data. The number of dimensions of the model's input data and output data is analyzed, and the input image is converted to an input tensor, and the output tensor is converted to an output image according to the analysis results (input tensor identification parameters, output tensor identification parameters). This has the effect of identifying the relationship between color components and channels that are not specified in the NN model itself, preparing NN input data, and obtaining an output image from NN output data. The SEI may also include information on the complexity of the neural network model.

[0186] (Configuration of image encoding device) Next, the configuration of the image encoding device 11 according to this embodiment will be described. 1 is a block diagram showing the configuration of an image encoding device 11 according to the present embodiment. The image encoding device 11 includes a predicted image generating unit 101, a subtraction unit 102, a transform / quantization unit 103, an inverse quantization / inverse transform unit 105, an addition unit 106, a ruler 108, a filter 109, a filter 1101, a filter 1112, a filter 1113, a filter 1114, a filter 1115, a filter 1116, a filter 1117, a filter 1118, a filter 1119, a filter 1120, a filter 1121, a filter 1122, a filter 1123, a filter 1124, a filter 1125, a filter 1126, a filter 1127, a filter 1128, a filter 1129, a filter The image processing system includes a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determination unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

[0187] The predicted image generation unit 101 generates a predicted image for each CU. The predicted image generation unit 101 includes the inter predicted image generation unit 309 and the intra predicted image generation unit 310, which have already been described, and therefore further description thereof will be omitted.

[0188] The subtraction unit 102 generates a prediction error by subtracting pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

[0189] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing the prediction errors. The quantized transform coefficients are output to the parameter coding unit 111 and the inverse quantization and inverse transform unit 105 .

[0190] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (FIG. 7) in the image decoding device 31. The calculated prediction error is output to the adder 106.

[0191] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes an outline of the operation of each module.

[0192] The header encoding unit 1110 performs encoding processing of parameters such as header information, division information, prediction information, and quantized transform coefficients.

[0193] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information and the like.

[0194] The CU encoding unit 1112 encodes CU information, prediction information, division information, and the like.

[0195] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0196] The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter-prediction parameters (predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX), intra-prediction parameters (intra_luma_mpm_flag, intra_luma_mpm_idx, intra_luma_mpm_reminder, intra_chroma_pred_mode), and quantized transform coefficients to the parameter encoding unit 111.

[0197] The entropy coding unit 104 receives the quantized transform coefficients and coding parameters (division information, prediction parameters) from the parameter coding unit 111. These are then entropy-encoded to generate and output encoded data Te.

[0198] The prediction parameter derivation unit 120 is a means including an inter-prediction parameter coding unit 112 and an intra-prediction parameter coding unit 113, and derives intra-prediction parameters and intra-prediction parameters from the parameters input from the coding parameter determination unit 110. The derived intra-prediction parameters and intra-prediction parameters are output to the parameter coding unit 111.

[0199] (Configuration of Inter-Prediction Parameter Encoding Unit) 10 , the inter prediction parameter encoding unit 112 includes a parameter encoding control unit 1121 and an inter prediction parameter derivation unit 303. The inter prediction parameter derivation unit 303 has the same configuration as the image decoding device. The parameter encoding control unit 1121 includes a merge index derivation unit 11211 and a vector candidate index derivation unit 11212.

[0200] The merge index derivation unit 11211 derives merge candidates and the like, and calculates inter prediction parameters The vector candidate index derivation unit 11212 derives predicted vector candidates and outputs them to the inter prediction parameter derivation unit 303 and the parameter coding unit 111.

[0201] (Configuration of the intra-prediction parameter encoding unit 113) The intra-prediction parameter coding unit 113 performs intra-prediction parameter coding with a parameter coding control unit 1131. The image decoding device includes a parameter derivation unit 304. The intra-prediction parameter derivation unit 304 has the same configuration as the image decoding device.

[0202] The parameter coding control unit 1131 derives IntraPredModeY and IntraPredModeC. Furthermore, it determines intra_luma_mpm_flag by referring to mpmCandList[ ]. These prediction parameters are output to the intra prediction parameter derivation unit 304 and the parameter coding unit 111.

[0203] However, unlike the image decoding device, the inter prediction parameter derivation unit 303 and the intra prediction parameter derivation unit 304 are The input to the parameter derivation unit 304 is the coding parameter determination unit 110 and the prediction parameter memory 108. The parameter encoding unit 111 outputs the parameter.

[0204] The adder 106 generates a decoded image by adding, for each pixel, the pixel values ​​of the predicted block input from the predicted image generation unit 101 and the prediction errors input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0205] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily include the above three types of filters. For example, the filter may be configured with only a deblocking filter.

[0206] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0207] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined location for each current picture and CU.

[0208] The encoding parameter determination unit 110 determines one set of encoding parameters from among a plurality of sets of encoding parameters. The coding parameters are the above-mentioned QT, BT or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0209] The encoding parameter determination unit 110 determines the size of the information amount and the encoding parameter for each of the plurality of sets. The RD cost value indicating the error is calculated. The RD cost value is, for example, the sum of the code amount and the value obtained by multiplying the squared error by a coefficient λ. The code amount is the information amount of the coded data Te obtained by entropy coding the quantization error and the coding parameters. The squared error is calculated in the subtraction unit 102. The coefficient λ is a preset real number greater than zero. The encoding parameter determination unit 110 determines the set of encoding parameters that minimizes the calculated cost value. The encoding parameter determination unit 110 selects the determined encoding parameters as a parameter code. The result is output to the derivation unit 111 and the prediction parameter derivation unit 120.

[0210] Note that a part of the image encoding device 11 and the image decoding device 31 in the above-described embodiments, such as the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generation unit 308, the inverse quantization and inverse transform unit 311, the adder 312, the prediction parameter derivation unit 320, the predicted image generation unit 101, the subtractor 102, the transform and quantization unit 103, the entropy encoding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the encoding parameter determination unit 110, the parameter encoding unit 111, and the prediction parameter derivation unit 120, may be implemented by a computer. In this case, a program for implementing this control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer system. Note that the "computer system" referred to here refers to a computer system built into either the image encoding device 11 or the image decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0211] Furthermore, part or all of the image encoding device 11 and image decoding device 31 in the above-described embodiments may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually implemented as a processor, or part or all of them may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used.

[0212] One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.

[0213] (NNR) Neural Network Coding and Representation (NNR) is a method for representing neural networks (NNs) This is an international standard for efficient compression. Compressing trained neural networks makes it possible to store and transmit them more efficiently.

[0214] The following provides an overview of the NNR encoding and decoding process.

[0215] FIG. 17 is a diagram showing an NNR encoding device and decoding device.

[0216] The NN coding device 801 includes a preprocessing unit 8011, a quantization unit 8012, and an entropy coding unit 8013. The NN encoding device 801 receives the uncompressed NN model O as input, and the quantization unit 8012 quantizes the NN model O. The NN encoding device 801 performs quantization to obtain a quantization model Q. Before quantization, the preprocessing unit 8011 may repeatedly apply parameter reduction techniques such as pruning and sparsification. After that, the entropy coding unit 8013 performs entropy coding on the quantization model Q. This is applied to obtain a bitstream S for storing and transmitting the NN model.

[0217] The NN decoding device 802 includes an entropy decoding unit 8021, a parameter restoration unit 8022, and a post-processing unit 8023. The NN decoding device 802 first receives the transmitted bit stream S, and the entropy decoding unit 8021 performs entropy decoding on S to obtain an intermediate model RQ. Operation of the NN model If the environment supports inference using the quantized representation used in the RQ, the RQ may be output and used for inference. If not, the parameters of the RQ are restored to their original representation in the parameter restoration unit 8022, and an intermediate model RP is obtained. If the sparse tensor representation used can be processed in the operating environment of the NN model, the RP may be output and used for inference. If not, the NN model O and A reconstructed NN model R that does not contain different tensors or structural representations is obtained and output.

[0218] The NNR standard specifies decoding methods for the numeric representation of specific NN parameters, such as integers and floating-point numbers. There is law.

[0219] The decoding method NNR_PT_INT decodes models with integer-valued parameters. The decoding method NNR_PT_FLOAT extends NNR_PT_INT by adding a quantization step size delta. This delta is multiplied by the integer value to produce a scaled integer. Delta is derived from the integer quantization parameter qp and the delta granularity parameter qp_density as follows:

[0220] mul = 2^(qp_density) + (qp & (2^(qp_density)-1)) delta = mul * 2^((qp >> qp_density)-qp_density) (Trained NN format) The representation of a trained NN consists of two elements: a topological representation such as the size of layers and the connections between layers, and a parameter representation such as weights and biases.

[0221] Topological representation is covered in native formats such as Tensorflow and PyTorch. However, to improve interoperability, exchange formats such as Open Neural Network Exchange Format (ONNX) and Neural Network Exchange Format (NNEF) exist.

[0222] The NNR standard also transmits topology information as part of the NNR bitstream containing the compressed parameter tensor, enabling interoperability with topology information expressed in native formats as well as interchange formats.

[0223] (Post-filter purpose SEI) The post-filter purpose SEI indicates the purpose of the post-filter processing. Describe the input and output information according to the purpose. Figure 15 shows an example of the syntax of this SEI. do.

[0224] First, define the InputChromaIdc of the input image as the input of this SEI. This value is the sign The value of sps_chroma_format_idc of the encoded data is assigned.

[0225] The value of pfp_id indicates the identification number of the post-filtering process specified by another mechanism. In this embodiment, it is associated with the NN filter SEI.

[0226] This SEI message is applied to the current decoded picture and all subsequent decoded pictures in output order at the current layer until a new Coded Layer Video Sequence (CLVS) starts or the bitstream ends.

[0227] pfp_id contains an identification number used to identify the post-filter process. The alternate number shall take values ​​from 0 to 2^20-1, with values ​​from 2^20 to 2^21-1 reserved for future use. do.

[0228] pfp_purpose indicates the purpose of the post-filtering process identified by pfp_id. The value of pfp_purpose ranges from 0 to 2^32-2. Other values ​​of pfp_purpose are reserved for future use. Note that a side information decoder shall ignore a post_filter_purpose SEI message that contains a reserved value of pfp_purpose.

[0229] A value of 0 for pfp_purpose indicates enhanced visual quality, meaning that post-filtering is applied with image restoration without image resolution conversion.

[0230] A value of 1 for pfp_purpose specifies the width or height of the cropped decoded output image, which means that post-filtering with image resolution conversion is applied.

[0231] If the value of pfp_purpose is 1, the syntax elements pfp_pic_width_in_luma_samples and pfp_pic_height_in_luma_samples exist.

[0232] pfp_pic_width_in_luma_samples specifies the post-processing filter identified by pfp_id. Specifies the width of the luminance pixel array resulting from applying this filter to the decoded output image.

[0233] pfp_pic_height_in_luma_samples specifies the height of the luma pixel array of the image resulting from applying the post-processing filter identified by pfp_id to the cropped decoded output image.

[0234] In the examples of Non-Patent Documents 1 and 2, resolution conversion and It was not possible to properly describe the information for inverse transformation. In this embodiment, the above problem is solved by clarifying the information for input and output.

[0235] If the value of pfp_purpose is 2, there are syntax elements that indicate the color components to which postfiltering is applied and the information on the output chrominance format. In other words, this means that postfiltering related to chrominance format conversion is applied.

[0236] If the value of pfp_purpose is 2, the syntax elements pfp_component_idc and pfp_output_diff_chroma_format_idc are present.

[0237] pfp_component_idc specifies the color component to which post-filtering is applied.

[0238] A value of 0 for pfp_component_idc indicates that post-filtering should be applied to the luma component only.

[0239] A value of 1 for pfp_component_idc applies post-filtering to the two chrominance components. This indicates that

[0240] A value of 2 for pfp_component_idc applies post-filtering to all three color components. Indicates that it is used.

[0241] pfp_output_diff_chroma_format_idc is the chrominance format output by post-filter processing. This variable indicates the difference between the identification value of the input chrominance format and the identification value of the input chrominance format. The value of ph_output_diff_chroma_format_idc must be in the range of 0 to 2. The variable OutputChromaFormatIdc, which is the identification value of the chrominance format output by post-filtering, is derived as follows:

[0242] OutputChromaFormatIdc = InputChromaFormatIdc + php_output_diff_chroma_format_idc Here, InputChromaFormatIdc is the value of sps_chroma_format_idc described in the SPS of the encoded data, and is the identification value of the chrominance format of the decoded image. A value of 0 indicates monochrome (4:0:0), a value of 1 indicates 4:2:0, a value of 2 indicates 4:2:2, and a value of 3 indicates 4:4:4. The variable OutputChromaFormatIdc, which is the identification value of the chrominance format output by post-filtering, is similar to InputChromaFormatIdc, with a value of 0 indicating monochrome (4:0:0), a value of 1 indicating 4:2:0, a value of 2 indicating 4:2:2, and a value of 3 indicating 4:4:4.

[0243] The NN filter unit 611 derives variables indicating color subsamples of the output image as follows.

[0244] outSubWidthC = outSW = 1, outSubHeightC = outSH = 1 (OutputChromaFormatIdc == 0) outSubWidthC = outSW = 2, outSubHeightC = outSH = 2 (OutputChromaFormatIdc == 1) outSubWidthC = outSW = 2, outSubHeightC = outSH = 1 (OutputChromaFormatIdc == 2) outSubWidthC = outSW = 1, outSubHeightC = outSH = 1 (OutputChromaFormatIdc == 3) In this way, by defining the input components and output format of the post-filtering process of the color difference format conversion, it is possible to clarify the input and output data of the post-filtering process of the color difference format conversion.

[0245] Although pfp_component_idc, which specifies the color component to which the post-filtering process is applied, distinguishes between luminance and chrominance, it may simply indicate the number of components. Specifically, the following semantics may be used:

[0246] A value of 0 for pfp_component_idc means to apply post-filtering for one component. Indicates that.

[0247] A value of 1 for pfp_component_idc means that post-filtering is applied to two components. Indicates that.

[0248] A value of 2 for pfp_component_idc applies post-filtering to all three components. Indicates that.

[0249] The NN filter unit 611 may switch the NN model according to pfp_component_idc. When pfp_component_idc==0: The NN filter unit 611 extracts 1 A NN model is selected to derive the 3D tensor of the channel and perform filtering. When pfp_component_idc==1: The NN filter unit 611 converts a two-channel three-dimensional tensor into a two-channel A NN model that derives a 3D tensor of the channel is selected and filtering is performed. When pfp_component_idc==2: The NN filter unit 611 performs a 3-channel 3D tensor transformation. A NN model that derives a 3D tensor of the channel is selected and filtering is performed.

[0250] This has the effect of reducing the amount of processing because the appropriate NN model is selected depending on the color component to be applied.

[0251] The NN filter unit 611 uses one channel for one component and two channels for two components according to pfp_component_idc. n so that it becomes a NN model with two channels for components and three channels for three components. nrpf_input_format_idc may be derived as follows: nnrpf_input_format_idc = 0 (pfp_component_idc==0) nnrpf_input_format_idc = 1 (pfp_component_idc==1) nnrpf_input_format_idc = 2 (pfp_component_idc==2) i.e. nnrpf_input_format_idc = pfp_component_idc As another example, the NN filter unit 611 may derive nnrpf_input_format_idc as follows, depending on pfp_component_idc, so as to obtain an NN model with one channel for one component, two channels for two components, and six channels for three components. nnrpf_input_format_idc = 0 (pfp_component_idc==0) nnrpf_input_format_idc = 1 (pfp_component_idc==1) nnrpf_input_format_idc = 3 (pfp_component_idc==2) That is, nnrpf_input_format_idc = pfp_component_idc < 2 ? pfp_component_idc : 3 This has the effect of enabling the processing to be performed by selecting the tensor format of an appropriate NN model depending on the color component to be applied.

[0252] The NN filter unit 611 directly converts the inputTensor into a The NN filter unit 611 may derive the output image by switching the value of the input image ComponentSamples and the value of the NN output data outTensor according to pfp_component_idc, and performing the following processing. If pfp_component_idc==0 outSamplesL[x][y] = outTensor[0][y][x] outSamplesCb[x*2 / outSW][y*2 / outSH] = ComponentSamples[1][x*2 / SW][y*2 / SH] outSamplesCr[x*2 / outSW][y*2 / outSH] = ComponentSamples[2][x*2 / SW][y*2 / SH] If pfp_component_idc==1 outSamplesL[x][y] = ComponentSamples[0][x][y] outSamplesCb[x*2 / outSW][y*2 / outSH] = outTensor [0][x*2 / SW][y*2 / SH] outSamplesCr[x*2 / outSW][y*2 / outSH] = outTensor [1][x*2 / SW][y*2 / SH] If pfp_component_idc==2 outSamplesL[x][y] = outTensor[0][x][y] outSamplesCb[x*2 / outSW][y*2 / outSH] = outTensor[1][x*2 / SW][y*2 / SH] outSamplesCr[x*2 / outSW][y*2 / outSH] = outTensor[2][x*2 / SW][y*2 / SH] In the above example, the identification value of the color difference format output from the post-filtering process is expressed as a difference value between the identification value of the input color difference format, but it may be written directly as a syntax element.

[0253] In this embodiment, the postfilter objective SEI is defined independently of the NN postfilter SEI to define the input and output of the postfilter processing. However, the NN postfilter SEI can also be defined using a similar syntax. may be defined, and the problem can be solved in the same way.

[0254] This embodiment will be described with reference to FIG. 1. A moving image decoding device is characterized by having an image decoding device, a resolution inverse conversion device that converts the resolution of an image decoded by the image decoding device, and an inverse conversion information decoding device that decodes color component information input to the resolution inverse conversion device and color difference format information that is output.

[0255] The video coding device is characterized by having an image coding device that codes an image, and an inverse conversion information coding device that codes color component information to be input to a resolution inverse conversion device that converts the resolution of the coded image and color difference format information to be output.

[0256] [Application example] The above-described video encoding device 10 and video decoding device 30 can be mounted and used in various devices that transmit, receive, record, and play back video. The video may be a natural video captured by a camera or the like, or an artificial video (including CG and GUI) generated by a computer or the like.

[0257] First, it will be explained with reference to FIG. 2 that the above-described video encoding device 10 and video decoding device 30 can be used for transmitting and receiving videos.

[0258] PROD_A in FIG. 2 is a block diagram showing the configuration of a transmission device PROD_A equipped with a video encoding device 10. As shown in the figure, the transmitting device PROD_A includes an encoding unit PROD_11 that encodes a video image to obtain encoded data, a modulation unit PROD_12 that modulates a carrier wave with the encoded data obtained by the encoding unit PROD_11 to obtain a modulated signal, and a transmitting unit PROD_13 that transmits the modulated signal obtained by the modulation unit PROD_12. The above-described video encoding device 10 is used as this encoding unit PROD_11.

[0259] The transmitting device PROD_A captures moving images as a supply source of moving images to be input to the encoding unit PROD_11. The apparatus further includes a camera PROD_14 for recording moving images, a recording medium PROD_A5 for recording moving images, an input terminal PROD_A6 for inputting moving images from the outside, and an image processing unit A7 for generating or processing images. In the figure, the transmitting device PROD_A is shown as having all of these components, but some of them may be omitted.

[0260] The recording medium PROD_A5 may also be one that records unencoded moving images. Alternatively, the recording medium PROD_A5 may be a recording medium that has been coded using a coding method for recording that is different from the coding method for transmission. In the latter case, a decoding unit ( It is advisable to use a device (not shown) between the two.

[0261] PROD_B in FIG. 2 is a block diagram showing the configuration of a receiving device PROD_B equipped with a video decoding device 30. As shown in the figure, the receiving device PROD_B includes a receiving unit PROD_B1 that receives a modulated signal, The video decoding device 30 includes a demodulation unit PROD_B2 that obtains coded data by demodulating the modulated signal received by the receiving unit PROD_B1, and a decoding unit PROD_B3 that obtains video by decoding the coded data obtained by the demodulation unit PROD_B2. The video decoding device 30 is used as the decoding unit PROD_B3.

[0262] The receiving device PROD_B is a supply destination of the video output from the decoding unit PROD_B3, and displays the video. The device may further include a display PROD_B4 for recording moving images, a recording medium PROD_B5 for recording moving images, and an output terminal PROD_B6 for outputting moving images to the outside. Although the receiving device PROD_B is illustrated as having the above components, some of these may be omitted.

[0263] The recording medium PROD_B5 is for recording unencoded moving images. In the latter case, a signal from the decoder PROD_B3 to the recording medium PROD_B5 is inserted between the decoder PROD_B3 and the recording medium PROD_B5. It is preferable to interpose an encoding unit (not shown) that encodes the acquired moving images according to an encoding method for recording.

[0264] The transmission medium for transmitting the modulated signal may be wireless or wired. The transmission mode for transmitting the modulated signal may be broadcast (here, this refers to a transmission mode in which the destination is not specified in advance) or communication (here, this refers to a transmission mode in which the destination is specified in advance). In other words, the transmission of the modulated signal may be realized by any of wireless broadcasting, wired broadcasting, wireless communication, and wired communication.

[0265] For example, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for terrestrial digital broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via wireless broadcasting. Also, a broadcasting station (such as a broadcasting facility) / receiving station (such as a television receiver) for cable television broadcasting is an example of a transmitting device PROD_A / receiving device PROD_B that transmits and receives modulated signals via cable broadcasting.

[0266] Furthermore, a server (such as a workstation) / client (such as a television receiver, personal computer, or smartphone) of an Internet-based VOD (Video On Demand) service or video sharing service is an example of a transmitter PROD_A / receiver PROD_B that transmits and receives modulated signals via communication (usually, a LAN uses either a wireless or wired transmission medium, while a WAN uses a wired transmission medium). Here, personal computers include desktop PCs, laptop PCs, and tablet PCs. Smartphones also include multi-function mobile phone terminals.

[0267] The client of the video hosting service has the function of decoding the encoded data downloaded from the server and displaying it on a display, as well as the function of encoding the video images captured by a camera and uploading them to the server. In other words, the client of the video hosting service functions as both the transmitting device PROD_A and the receiving device PROD_B.

[0268] Next, it will be explained with reference to FIG. 3 that the above-described video encoding device 10 and video decoding device 30 can be used for recording and reproducing video.

[0269] PROD_C in FIG. 3 shows the configuration of a recording device PROD_C equipped with the above-described video encoding device 10. As shown in the figure, the recording device PROD_C includes an encoding unit PROD_C1 that encodes a video image to obtain encoded data, and a writing unit PROD_C2 that writes the encoded data obtained by the encoding unit PROD_C1 onto a recording medium PROD_M. The above-described video encoding device 10 is used as this encoding unit PROD_C1.

[0270] The recording medium PROD_M may be (1) a type that is built into the recording device PROD_C, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), (2) a type that is connected to the recording device PROD_C, such as an SD memory card or USB (Universal Serial Bus) flash memory, or (3) a type that is loaded into a drive device (not shown) built into the recording device PROD_C, such as a DVD (Digital Versatile Disc: registered trademark) or BD (Blu-ray Disc: registered trademark).

[0271] The recording device PROD_C also receives the video as a video source to be input to the encoding unit PROD_C1. The recording device PROD_C may further include a camera PROD_C3 for capturing images, an input terminal PROD_C4 for inputting moving images from the outside, a receiving unit PROD_C5 for receiving moving images, and an image processing unit PROD_C6 for generating or processing images. In the figure, the recording device PROD_C is shown as having all of these components, but some of them may be omitted.

[0272] The receiving unit PROD_C5 may receive unencoded video. Alternatively, the receiving unit PROD_C5 may receive coded data coded by a coding method for transmission that is different from the coding method for recording. In the latter case, it is preferable to interpose a decoding unit for transmission (not shown) between the receiving unit PROD_C5 and the coding unit PROD_C1, which decodes the coded data coded by the coding method for transmission.

[0273] Examples of such a recording device PROD_C include a DVD recorder, a BD recorder, and an HDD (Hard Disk Drive) recorder (in this case, the input terminal PROD_C4 or the receiving unit PROD_C5 is the main source of moving images).Other examples of such a recording device PROD_C include a camcorder (in this case, the camera PROD_C3 is the main source of moving images), a personal computer (in this case, the receiving unit PROD_C5 or the image processing unit C6 is the main source of moving images), and a smartphone (in this case, the camera PROD_C3 or the receiving unit PROD_C5 is the main source of moving images).

[0274] FIG. 3 is a block diagram showing the configuration of a playback device PROD_D incorporating the above-described video decoding device 30. As shown in the figure, the playback device PROD_D includes a reading unit PROD_17 that reads coded data written on a recording medium PROD_M, and a decoding unit PROD_D2 that obtains video by decoding the coded data read by the reading unit PROD_17. The above-described video decoding device 30 is used as this decoding unit PROD_D2.

[0275] The recording medium PROD_M may be (1) a type that is built into the playback device PROD_D, such as an HDD or SSD, or (2) a type that is not built into the playback device PROD_D, such as an SD memory card or a USB flash memory. (3) DVD, BD, etc. As shown in the figure, the disc may be loaded into a drive device (not shown) built into the playback device PROD_D.

[0276] Furthermore, the playback device PROD_D receives the video output from the decoding unit PROD_D2 and sends the video to The image processing device may further include a display PROD_D3 for displaying the moving image, an output terminal PROD_D4 for outputting the moving image to the outside, and a transmission unit PROD_D5 for transmitting the moving image. Although the configuration of the playback device PROD_D is illustrated, some of the components may be omitted.

[0277] The transmission unit PROD_D5 may transmit unencoded video. Alternatively, the decoder PROD_D2 may transmit coded data coded by a coding method for transmission that is different from the coding method for recording. In the latter case, it is preferable to interpose a coding unit (not shown) between the decoder PROD_D2 and the transmitter PROD_D5, which codes the video by the coding method for transmission.

[0278] Examples of such a playback device PROD_D include a DVD player, a BD player, and an HDD player (in this case, the output terminal PROD_D4 to which a television receiver or the like is connected operates). Other examples of such a playback device PROD_D include a television receiver (in this case, the display PROD_D3 is the main destination of the moving images), a digital signage (also called an electronic billboard or electronic bulletin board, and the display PROD_D3 or the transmission unit PROD_D5 is the main destination of the moving images), a desktop PC (in this case, the output terminal PROD_D4 or the transmission unit PROD_D5 is the main destination of the moving images), a laptop or tablet PC (in this case, the display PROD_D3 or the transmission unit PROD_D5 is the main destination of the moving images), and a smartphone (in this case, the display PROD_D3 or the transmission unit PROD_D5 is the main destination of the moving images).

[0279] (hardware and software implementations) Furthermore, each block of the video decoding device 30 and the video encoding device 10 may be realized in hardware by a logic circuit formed on an integrated circuit (IC chip), or may be realized by a CPU. This may be realized in software using a Central Processing Unit (Central Processing Unit).

[0280] In the latter case, each of the above devices includes a CPU that executes instructions from a program to realize each function, ROM (Read Only Memory) stores the program, and RAM (Random Access Memory) expands the program. The object of the embodiment of the present invention is to provide a computer-readable storage device (recording medium) for storing the program code (executable program, intermediate code program, source program) of the control program for each of the above devices, which is software for realizing the above-mentioned functions. This can also be achieved by supplying the recording medium to each of the above devices, and having the computer (or CPU or MPU) read and execute the program code recorded on the recording medium.

[0281] Examples of the recording medium that can be used include tapes such as magnetic tape and cassette tape, magnetic disks such as floppy disks (registered trademark) and hard disks, disks including optical disks such as CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical discs), MDs (Mini Discs), DVDs (Digital Versatile Discs: registered trademark), CD-Rs (CD Recordable), and Blu-ray Discs (registered trademark), cards such as IC cards (including memory cards) and optical cards, semiconductor memories such as mask ROMs, EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable and Programmable Read-Only Memory: registered trademark), and flash ROMs, and logic circuits such as PLDs (Programmable logic devices) and FPGAs (Field Programmable Gate Arrays).

[0282] Furthermore, each of the above devices may be configured to be connectable to a communications network, and the program code may be supplied via the communications network. This communications network may be any network capable of transmitting the program code. For example, the Internet, an intranet, an extranet, a local area network (LAN), an integrated services digital network (ISDN), a value-added network (VAN), a community antenna television / cable television (CATV) communications network, a virtual private network, a telephone line network, a mobile communications network, a satellite communications network, etc. Furthermore, the transmission media constituting this communications network may be any medium capable of transmitting the program code, and are not limited to any particular configuration or type. For example, the present invention can be used in wired communication such as IEEE (Institute of Electrical and Electronic Engineers) 1394, USB, power line carrier, cable TV line, telephone line, or ADSL (Asymmetric Digital Subscriber Line) line, or in wireless communication such as IrDA (Infrared Data Association), infrared such as that used in remote controls, Bluetooth (registered trademark), IEEE 802.11 wireless, HDR (High Data Rate), NFC (Near Field Communication), DLNA (Digital Living Network Alliance: registered trademark), mobile phone networks, satellite lines, or terrestrial digital broadcasting networks. Note that the present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

[0283] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention. [Industrial Applicability]

[0284] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data obtained by coding image data, and a video coding device that generates coded data obtained by coding image data, and can also be suitably applied to the data structure of coded data generated by a video coding device and referenced by the video decoding device. [Explanation of symbols]

[0285] 1. Video transmission system 30 Video decoding device 31 Image decoding device 301 Entropy Decoding Unit 302 Parameter Decoding Unit 303 Inter-prediction parameter derivation unit 304 Intra prediction parameter derivation unit 305, 107 Loop filter 306, 109 Reference Picture Memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 309 Inter-prediction image generation unit 310 Intra-prediction image generation unit 311, 105 Inverse quantization and inverse transform unit 312, 106 Addition section 320 Prediction parameter derivation part 10 Video Encoding Device 11 Image encoding device 102 Subtraction section 103 Transformation and Quantization Unit 104 Entropy coding unit 110 Encoding parameter determination unit 111 Parameter Encoding Unit 112 Inter-prediction parameter coding unit 113 Intra prediction parameter coding unit 120 Prediction parameter derivation part 71 Reverse conversion information creation device 81 Inverse transformation information coding device 91 Inverse transformation information decoding device 611 NN filter section

Claims

1. A video decoding device for decoding encoded data, comprising: a header decoding unit for decoding input tensor identification parameters to derive a form of an input tensor for a neural network; The video decoding device is characterized in that the input tensor identification parameters are parameters for specifying color components.

2. A video decoding device as described in Claim 1, characterized in that a relational equation for deriving the input tensor is defined depending on the input tensor identification parameter and color difference sampling of the input image.

3. The header decoding unit decodes the output tensor identification parameters, 2. The video decoding device according to claim 1, wherein the output tensor identification parameter is a parameter for specifying the format of the output tensor.

4. A video decoding device as described in Claim 3, characterized in that a relational equation for deriving the output tensor is defined depending on the output tensor identification parameter and the color difference subsample value of the output image.

5. The header decoding unit decodes the network model complexity information, 2. The video decoding device according to claim 1, wherein the network model complexity information includes a first syntax element indicating the number of parameters of a neural network model for a post filter.

6. The header decoding unit decodes the network model complexity information, 2. The video decoding device of claim 1, wherein the network model complexity information includes a second syntax element indicating a parameter type for the neural network.

7. The header decoding unit decodes the network model complexity information, 2. The video decoding device according to claim 1, wherein the network model complexity information includes a third syntax element indicating a bit width of parameters for the neural network.

8. The video decoding device described in Claim 5, characterized in that the header decoding unit derives the maximum value of the number of parameters based on the first syntax element.

9. The header decoding unit decodes the network model complexity information, 2. The video decoding device according to claim 1, wherein the network model complexity information includes a fourth syntax element indicating the scale of the number of operations for a post filter.

10. The video decoding device of claim 1, wherein the color components are luminance and chrominance.

11. A video encoding device for generating encoded data, comprising: a header encoding unit that encodes an input tensor identification parameter for deriving a form of an input tensor for a neural network; The video encoding device is characterized in that the input tensor identification parameters are parameters for specifying color components.

Images