Video encoding device, video decoding device, and integrated circuit

Abstract

To provide a video encoding device and a video decoding device that can efficiently and accurately execute neural network processing.SOLUTION: In a video transmission system 1, a video decoding device 30 includes an image decoding device that decodes encoded data to generate a decoded image, and a resolution inverse conversion device using a neural network that converts a decoded image to a specified resolution using inverse conversion information. The resolution inverse conversion device decodes information specifying resolution and information indicating a unit of inverse conversion processing. The values of information specifying the resolution and the information indicating the unit of inverse conversion processing have the same proportional relationship.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to a moving image encoding device and a moving image decoding device.

Background Art

[0002] In order to efficiently transmit or record a moving image, a moving image encoding device that generates encoded data by encoding the moving image, and a moving image decoding device that generates a decoded image by decoding the encoded data are used.

[0003] Specific moving image encoding methods include, for example, the H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding) methods.

[0004] In such a moving image encoding method, an image (picture) constituting a moving image is managed by a hierarchical structure composed of a slice obtained by dividing the image, a coding tree unit (CTU: Coding Tree Unit) obtained by dividing the slice, a coding unit (sometimes called a Coding Unit: CU) obtained by dividing the coding tree unit, and a transform unit (TU: Transform Unit) obtained by dividing the coding unit, and is encoded / decoded for each CU. and [[ID=e23]] a prediction image is usually generated based on a local decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the prediction image from the input image (original image) is encoded. Examples of the method for generating the prediction image include inter-picture prediction (inter prediction) and intra-picture prediction (intra prediction).

[0005] In addition, Non-Patent Document 1 is cited as a recent technology for moving image encoding and decoding.

[0006] In addition, Non-Patent Document 1 is cited as a recent technology for moving image encoding and decoding.

[0007] H.274 transmits image properties, display methods, and timing along with the encoded data. Supplemental Enhancement Information (SEI) messages are defined for this purpose.

[0008] Non-Patent Documents 1, 2, and 3 describe the SEI that transmits the topology and parameters of a neural network filter used as a post-filter. Methods of specifying explicitly and methods of specifying indirectly as reference information are disclosed. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] B. Choi, Z. Li, W. Wang, W. Jiang, X. Xu, S. Wenger and S. Liu, “AHG9 / AHG11: SEI messages for carriage of neural network information for post-filtering,” JVET-V0091 [Non-Patent Document 2] MM Hannuksela, EB Aksu, F. Cricri, HR Tavakoli and M. Santamaria, "AHG9: On post-filter SEI", JVET-X0112 [Non-Patent Document 3] MM Hannuksela, M. Santamaria, F. Cricri, EB Aksu and HR Tavakoli, "AHG9: On post-filter SEI", JVET-Y0115 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] However, in Non-Patent Documents 1, 2, and 3, the neural network There was a problem in that the relationship between the patch size, which is the unit of processing in the program, and the size of the input and output screens was not clearly defined.

[0011] Furthermore, Non-Patent Documents 1, 2, and 3 had the problem that the relationship between the data types of the input and output tensor values ​​of a neural network and the bit length of the pixel values ​​in the decoded image was not clearly defined.

[0012] [Means for solving the problem]

[0013] A video decoding device according to one aspect of the present invention is: An image decoding device that decodes encoded data and generates a decoded image, The device includes a resolution inverse conversion device that uses a neural network to convert the decoded image to a specified resolution using inverse conversion information. In the resolution inverse conversion device, information specifying the resolution and information indicating the unit of the inverse conversion process are decoded. The information specifying the resolution and the information indicating the unit of the inverse transformation process have the same proportional relationship.

[0014] Furthermore, a video decoding device according to one aspect of the present invention is An image decoding device that decodes encoded data and generates a decoded image, The device includes a resolution inverse conversion device that uses a neural network to convert the decoded image to a specified resolution using inverse conversion information. The resolution inverse transformer is characterized by using the data types of the input and output tensor values ​​of the neural network and the bit length of the pixel values ​​of the decoded image to convert between the pixel values ​​of the image and the input / output values ​​of the tensor.

[0015] A video encoding device according to one aspect of the present invention is: An image encoding device that encodes an image to generate encoded data, An inverse transformation information generation device that generates inverse transformation information for inverse-transforming the resolution of the decoded image obtained by decoding the encoded data, Having an inverse transformation information encoding device that encodes the inverse transformation information as auxiliary extension information, The inverse transformation information is characterized by generating inverse transformation information in which the values of information specifying the resolution and information indicating the unit of inverse transformation processing have the same proportional relationship.

[0016] Also, a moving image encoding device according to one aspect of the present invention, An image encoding device that encodes an image to generate encoded data, An inverse transformation information generation device that generates inverse transformation information for inverse-transforming the resolution of the decoded image obtained by decoding the encoded data, Having an inverse transformation information encoding device that encodes the inverse transformation information as auxiliary extension information, Using the data type of the values of the input tensor and output tensor of the neural network in the resolution inverse transformation device and the bit length of the pixel values of the encoded image, inverse transformation information for converting the pixel values of the image and the values of the input / output of the tensor into each other is generated.

Advantages of the Invention

[0017] By adopting such a configuration, it becomes possible to efficiently and accurately execute the processing of the neural network.

[0018]

Brief Description of the Drawings

[0019] [Figure 1] It is a schematic diagram showing the configuration of a moving image transmission system according to this embodiment. [Figure 2] It is a diagram showing the hierarchical structure of encoded data. [Figure 3] It is a conceptual diagram of an image to be processed in the moving image transmission system according to this embodiment. [Figure 4]This is a conceptual diagram showing an example of a reference picture and a reference picture list. [Figure 5] This is a schematic diagram showing the configuration of an image decoding device. [Figure 6] This is a flowchart illustrating the general operation of the image decoding device. [Figure 7] This is a block diagram showing the configuration of an image encoding device. [Figure 8] This is a schematic diagram showing the configuration of the interprediction parameter coding unit. [Figure 9] This figure shows the SEI syntax for post-filtering in this embodiment. [Figure 10] This figure illustrates the input and output processing of image data to the neural network in this embodiment. [Figure 11] This diagram illustrates the process of inputting image data into an input tensor. [Figure 12] This diagram illustrates the process of outputting data from an output tensor. This diagram also illustrates the process of inputting image data into a tensor. [Figure 13] This figure shows another example 1 of the SEI syntax for post-filtering in this embodiment. [Figure 14] This figure shows another example 2 of the SEI syntax for post-filtering in this embodiment. [Figure 15] This figure shows another example 3 of the SEI syntax for post-filtering in this embodiment. [Figure 16] This figure shows another example 4 of the SEI syntax for post-filtering in this embodiment. [Figure 17] This figure shows another example 5 of the SEI syntax for post-filtering in this embodiment. [Figure 18] This diagram shows a flowchart of the processing performed by the NN filter unit 611. [Figure 19] This diagram shows the neural network configuration of the NN filter section 611. [Figure 20]This diagram shows the encoding and decoding devices for NNR.

[0020] [Modes for carrying out the invention]

[0021] (First Embodiment) Embodiments of the present invention will be described below with reference to the drawings.

[0022] Figure 1 is a schematic diagram showing the configuration of the video transmission system according to this embodiment.

[0023] The video transmission system 1 encodes images of different resolutions whose resolutions have been converted. This system transmits data, decodes the transmitted encoded data, and converts the image back to its original resolution for display. The video transmission system 1 consists of a video encoding device 10 and a network 21. It consists of an image decoding device 30 and an image display device 41.

[0024] The video encoding device 10 consists of a resolution conversion device (resolution conversion unit) 51, an image encoding device (image encoding unit) 11, an inverse conversion information creation device (inverse conversion information creation unit) 71, and an inverse conversion information encoding device (inverse conversion information encoding unit) 81.

[0025] The video decoding device 30 includes an image decoding device (image decoding unit) 31 and a resolution inverse conversion device (resolution inverse conversion It consists of a conversion unit 61 and an inverse conversion information decoding device (inverse conversion information decoding unit) 91.

[0026] The resolution conversion device 51 converts the resolution of the image T contained in the moving image, and converts the image to a different resolution. The variable-resolution video T2, which includes the above, is supplied to the image encoding device 11. The resolution conversion device 51 also supplies inverse conversion information indicating whether or not the image resolution has been converted to the image encoding device 11. If this information indicates a resolution conversion, the video encoding device 10 sets the resolution conversion information ref_pic_resampling_enabled_flag (described later) to 1 and includes it in the sequence parameter set SPS (Sequence Parameter Set) of the encoded data Te before encoding.

[0027] The inverse transformation information generation device 71 creates inverse transformation information based on image T1 contained in the video. The inverse transformation information is derived or selected from the relationship between the input image T1 before resolution transformation and the image Td1 after resolution transformation, encoding, and decoding. The additional information indicates what to select.

[0028] The inverse transform information encoding device 81 receives the inverse transform information as input. The inverse transform information encoding device 81 encodes the inverse transform information to generate encoded inverse transform information and sends it to the network 21.

[0029] The image encoding device 11 receives a variable-resolution image T2 as input. The image encoding device 11 uses the RPR (Reference Picture Resampling) framework to encode the image size information of the input image in PPS units and sends it to the image decoding device 31.

[0030] In Figure 1, the inverse transform information encoding device 81 is not connected to the image encoding device 11, but the inverse transform information encoding device 81 and the image encoding device 11 may communicate necessary information as appropriate.

[0031] Network 21 transmits the encoded inverse transform information and encoded data Te to the image decoding device 31. Part or all of the encoded inverse transform information is encoded as auxiliary extended information SEI. It may be included in the data Te. Network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. Network 21 is not necessarily limited to a bidirectional communication network; it may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, Network 21 may be replaced by a storage medium that records encoded data Te, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blu-ray Disc: registered trademark).

[0032] The image decoding device 31 decodes each of the encoded data Te transmitted by the network 21, generates a variable resolution decoded image Td1, and supplies it to the resolution inverse conversion device 61.

[0033] The inverse conversion information decoding device 91 decodes the encoded inverse conversion information transmitted by the network 21 to generate inverse conversion information and supplies it to the resolution inverse conversion device 61.

[0034] In Figure 1, the inverse transformation information decoding device 91 is shown separately from the image decoding device 31, but the inverse transformation information decoding device 91 may be included in the image decoding device 31. For example, the inverse transformation information decoding device 91 may be included in the image decoding device 31 separately from the various functional parts of the image decoding device 31. Also, although the inverse transformation information decoding device 91 and the image decoding device 31 are not connected in Figure 1, they may communicate necessary information as appropriate.

[0035] When the resolution conversion information indicates a resolution conversion, the resolution inverse conversion device 61 generates a decoded image of the original size by inversely converting the resolution-converted image through post-filtering processing such as super-resolution processing using a neural network, based on the image size information contained in the encoded data.

[0036] Furthermore, the resolution inverse converter 61, when the resolution conversion information has a resolution of equal size, the neural A post-filtering process using a network may be performed, followed by an inverse resolution transformation to restore the input image T1, and finally, the decoded image Td2 may be generated.

[0037] The image display device 41 receives one or more decoded images Td2 from the resolution inverse converter 61. The image display device 41 may display all or part of the image. The image display device 41 may include a display device such as a liquid crystal display or an organic EL (electroluminescence) display. The display may take the form of a stationary, mobile, or HMD. Furthermore, if the image decoding device 31 has high processing power, it will display a high-quality image, and if it has lower processing power, it will display an image that does not require high processing power or display power.

[0038] Figure 3 is a conceptual diagram of the image to be processed in the video transmission system shown in Figure 1. This figure shows the change in the resolution of the image over time. However, in Figure 3... This does not distinguish whether the image is encoded or not. Figure 3 shows the processing of a video transmission system. In the processing step, an example is shown in which the resolution is reduced and the image is transmitted to the image decoding device 31. As shown in Figure 3, normally the resolution conversion device 51 reduces the amount of information transmitted. Therefore, a conversion is performed to make the image resolution equal to or less than the resolution of the input image.

[0039] <operators> The operators used in this specification are listed below.

[0040] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, and | is a bitwise OR. |= is the OR assignment operator, and || represents logical disjunction.

[0041] x ? y : z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0). ru.

[0042] Clip3(a, b, c) is a function that clips c to a value between a and b. If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (where a <= b).

[0043] abs(a) is a function that returns the absolute value of a.

[0044] Int(a) is a function that returns the integer value of a.

[0045] floor(a) is a function that returns the largest integer less than or equal to a.

[0046] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0047] a / d represents the division of a by d (truncating the decimal part).

[0048] a ^ b represents power(a, b). When a = 2, it is equal to 1 << b.

[0049] <Structure of the encoded data Te> Prior to the detailed description of the image encoding device 11 and the image decoding device 31 according to this embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described.

[0050] FIG. 2 is a diagram showing the hierarchical structure of the data in the encoded data Te. The encoded data Te illustratively includes a sequence and a plurality of pictures constituting the sequence. In FIG. 2 an encoded video sequence that specifies the sequence SEQ, an encoded pi cture that specifies the picture PICT, an encoded slice that specifies the slice S, an encoded slice [[ID=4�]] slice data that specifies the slice data, an encoded tree unit included in the encoded slice data, and an encoded unit included in the encoded tree unit are shown.

[0051] (Encoded video sequence) In encoded video sequences, an image decoding machine is used to decode the sequence (SEQ) to be processed. The set of data referenced by location 31 is defined. As shown in Figure 2, the sequence SEQ includes the video parameter set VPS, the sequence parameter set SPS, the picture parameter set PPS, the adaptation parameter set (APS), the picture PICT, and the supplemental enhancement information SEI.

[0052] In the video parameter set VPS, for video images composed of multiple layers, A set of encoding parameters common to multiple video images, as well as sets of encoding parameters associated with multiple layers contained in a video image and each individual layer, are defined.

[0053] The Sequence Parameter Set (SPS) defines a set of encoding parameters that the image decoding device 31 references to decode the target sequence. For example, the width and height of the picture are defined. It is defined. Note that there may be multiple SPS. In that case, one of the multiple SPS from the PPS Select.

[0054] Here, the sequence parameter set SPS includes the following syntax elements: ·ref_pic_resampling_enabled_flag: This flag specifies whether or not to use the resampling function, which allows for variable resolution, when decoding each image in a single sequence that references the target SPS. In other words, this flag indicates that the size of the reference picture referenced in the generation of the predicted image changes between each image represented by the single sequence. If the value of this flag is 1, the above resampling is enabled. If it applies, and is 0, it does not apply. • pic_width_max_in_luma_samples: The maximum width of images in a single sequence. This syntax element specifies the width of the image in units of luminance blocks. Furthermore, the value of this syntax element is required to be non-zero and an integer multiple of Max(8, MinCbSizeY). Here, MinCbSizeY is a value determined by the minimum size of the luminance block. • pic_height_max_in_luma_samples: This syntax element specifies the height of the tallest image in a single sequence, in units of luminance blocks. Furthermore, the value of this syntax element must not be 0 and must be an integer multiple of Max(8, MinCbSizeY). It is required. · sps_temporal_mvp_enabled_flag: When decoding the target sequence, time movement This flag determines whether or not to use vector prediction. If the value of this flag is 1, time Motion vector prediction is used, and if the value is 0, time motion vector prediction is not used. Furthermore, by defining this flag, it is possible to prevent the referenced coordinate position from shifting when referencing reference pictures of different resolutions, etc.

[0055] In the Picture Parameter Set (PPS), to decode each picture in the target sequence... A set of encoding parameters referenced by the image decoding device 31 is defined. For example, the reference value for the quantization width used for decoding the picture (pic_init_qp_minus26) and the application of weighted prediction are defined. It includes the flag (weighted_pred_flag). Note that there may be multiple PPS entries. In this case, select one of several PPS images from each picture in the target sequence.

[0056] Here, the picture parameter set PPS includes the following syntax elements: · pps_pic_width_in_luma_samples: A syntax element that specifies the width of the target picture. The value of the syntax element is not 0, but an integer multiple of Max(8, MinCbSizeY), and The value must be less than or equal to sps_pic_width_max_in_luma_samples. • pps_pic_height_in_luma_samples: This syntax element specifies the height of the target picture. The value of this syntax element is not 0, but an integer multiple of Max(8, MinCbSizeY). Furthermore, it is required that the value be less than or equal to sps_pic_height_max_in_luma_samples. • conformance_window_flag: Conformance (cropping) window offset This flag indicates whether the parameter will be notified next, and also indicates where to display the conformance window. If this flag is 1, the parameter will be notified. If known and is 0, the conformance window offset parameter does not exist. This indicates that. ·conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, conf_win_bottom_offset: These are offset values ​​used to specify the left, right, top, and bottom positions of the picture output during the decoding process, relative to the rectangular area specified by the picture coordinates for output. Also, if the value of conformance_window_flag is 0, it is assumed that the values ​​of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are 0.

[0057] The width PicWidthInLumaSamples and height PicHeightInLumaSamples of the output picture are derived as follows:

[0058] PicWidthInLumaSamples = pps_pic_width_in_luma_samples - SubWidthC * (conf_win_right_offset + conf_win_left_offset) PicHightInLumaSamples = pps_pic_height_in_luma_samples - SubHightC * (conf_win_bottom_offset + conf_win_top_offset) Here, the chroma format variable ChromaFormatIdc is the value of sps_chroma_format_id. The variables SubWidthC and SubHightC are values ​​determined by ChromaFormatIdc, and In the 360 ​​format, SubWidthC and SubHightC are both 1; in the 4:2:0 format, SubWidthC and SubHightC are both 2; in the 4:2:2 format, SubWidthC is 2 and SubHightC is 1; and in the 4:4:4 format, SubWidthC and SubHightC are both 1. • scaling_window_flag: This flag indicates whether the scaling window offset parameter exists in the target PPS and is a flag related to the specification of the output image size. If this flag is 1, it indicates that the parameter exists in the PPS, and if this flag is 0, it indicates that the parameter does not exist in the PPS. Also, if the value of ref_pic_resampling_enabled_flag is 0, the value of scaling_window_flag is also required to be 0. · scaling_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, scaling_win_bottom_offset: These are syntax elements that specify the offsets applied to the image size for scaling ratio calculation, in units of luminance pixels, for the left, right, top, and bottom positions of the target picture, respectively. Also, if the value of scaling_window_flag is 0, the values ​​of scaling_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, and scaling_win_bottom_offset are assumed to be 0. Furthermore, the value of scaling_win_left_offset + scaling_win_right_offset is required to be less than pic_width_in_luma_samples, and the value of scaling_win_top_offset + scaling_win_bottom_offset is required to be less than pic_height_in_luma_samples.

[0059] The width PicScaleWidthL and height PicScaleHeightL of the scaling picture are derived as follows: ru.

[0060] PicScaleWidthL = pic_width_in_luma_samples - SubWidthC * (scaling_win_right_offset + scaling_win_left_offset) PicScaleHeightL = pic_height_in_luma_samples - SubHightC * (scaling_win_bottom _offset + scaling_win_top_offset) (Sub-picture) A picture may be further divided into rectangular subpictures. The size of a subpicture may be a multiple of CTU. A subpicture is defined as a set of tiles that are an integer number of consecutive tiles in the vertical and horizontal directions. This means that the picture is divided into rectangular tiles, and a subpicture is defined as a collection of these rectangular tiles. A subpicture may also be defined using the ID of its top-left tile and the ID of its bottom-right tile. Alternatively, the slice header may include sh_subpic_id, which indicates the ID of the subpicture.

[0061] (Encoded picture) In the encoded picture, a set of data that the image decoding device 31 references to decode the picture PICT to be processed is defined. As shown in Figure 2, the picture PICT is a picture header PH includes slices 0 to NS-1 (NS is the total number of slices included in the picture PICT). .

[0062] In the following cases where it is not necessary to distinguish between slices 0 to NS-1, the sign subscript is used. The subscript may be omitted in some cases. The same applies to other data included in the encoded data Te described below that has a subscript.

[0063] The picture header contains the following syntax elements: • pic_temporal_mvp_enabled_flag: The slice associated with the picture header. This flag specifies whether or not to use time motion vector prediction for center prediction. If the value of this flag is 0, the syntax element of the slice associated with the picture header is... This restricts the decoding of the slice from using time-motion vector prediction. A value of 1 for this flag indicates that time-motion vector prediction is used to decode the slice associated with the picture header. If this flag is not specified, its value is assumed to be 0.

[0064] (Encoded slice) In the encoded slice, the image decoding device 31 references the slice S to be processed in order to decode it. A set of data is defined. As shown in Figure 2, the slice has a slice header, It also includes slice data.

[0065] The slice header contains a set of encoding parameters that the image decoding device 31 references to determine the decoding method for the target slice. The slice type specification information (slice_type), which specifies the slice type, is an example of the encoding parameters included in the slice header.

[0066] The slice types that can be specified by the slice type specification information include: (1) I slices that use only intra prediction during encoding; (2) P slices that use single prediction (L0 prediction) or intra prediction during encoding; and (3) B slices that use single prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction during encoding. Note that inter-prediction is not limited to single or bi-prediction, and a prediction image may be generated using more reference pictures. Hereinafter, when referring to P slices and B slices, we mean slices that contain blocks on which inter-prediction can be used.

[0067] The slice header may also include a reference to the picture parameter set PPS (pic_parameter_set_id).

[0068] (Encoded slice data) In encoded slice data, an image decoding device is used to decode the slice data to be processed. The set of data referenced by 31 is defined. The slice data is the encoded slice shown in Figure 2. As shown in the header, it includes CTUs. A CTU is a fixed-size (e.g., 64x64) block that makes up a slice, and is sometimes called the Largest Coding Unit (LCU).

[0069] (Code tree unit) Figure 2 defines the set of data that the image decoding device 31 references to decode the CTU to be processed. The CTU is divided into coding units CU, which are the basic units of encoding processing, by recursive quad tree partitioning (QT), binary tree partitioning (BT), or ternary tree partitioning (TT). BT and TT partitioning together are called multi-tree partitioning (MT). The nodes of the tree structure obtained by recursive quad tree partitioning are called coding nodes. The intermediate nodes of quad trees, binary trees, and ternary trees are coding nodes, and the CTU itself is defined as the top-level coding node.

[0070] CT information includes a CU splitting flag (split_cu_flag) indicating whether or not to perform CT splitting, and QT minutes. It includes a QT splitting flag (qt_split_cu_flag) indicating whether or not to perform a split, an MT splitting direction (mtt_split_cu_vertical_flag) indicating the splitting direction of the MT split, and an MT splitting type (mtt_split_cu_binary_flag) indicating the splitting type of the MT split. split_cu_flag, qt_split_cu_flag, mtt_split_cu_vertical_flag, and mtt_split_cu_binary_flag are transmitted for each coding node.

[0071] Different trees may be used for luminance and chrominance. The type of tree is indicated by treeType. For example, if a common tree is used for luminance (Y, cIdx=0) and chrominance (Cb / Cr, cIdx=1,2), the common single tree is indicated by treeType=SINGLE_TREE. If two different trees (DUAL trees) are used for luminance and chrominance, the luminance tree is indicated by treeType=DUAL_TREE_LUMA and the chrominance tree is indicated by treeType=DUAL_TREE_CHROMA.

[0072] (Encoding Unit) Figure 2 shows the data that the image decoding device 31 references in order to decode the encoding unit to be processed. A set of parameters is defined. Specifically, CU consists of a CU header (CUH), prediction parameters, and transformation parameters. It consists of meters, quantization conversion coefficients, etc. The CU header defines the prediction mode, etc.

[0073] Prediction processing can be performed at the CU (Unit) level or at the subCU level, which is a further division of the CU. If the size of the CU and the subCU are equal, there is one subCU within the CU. If the CU is larger than the size of the subCU, the CU is divided into subCUs. For example, if the CU is 8x8 and the subCU is 4x4, the CU will be divided into four subCUs, each consisting of two horizontal and two vertical divisions.

[0074] There are two types of predictions (prediction modes): intra-prediction and inter-prediction. Intra-prediction is prediction within the same picture, while inter-prediction refers to prediction processing performed between different pictures (for example, between display times or between layer images).

[0075] The conversion and quantization processes are performed in units of CUs, but the quantization conversion coefficients are in sub-block units such as 4x4. Entropy coding can also be used.

[0076] (Prediction parameters) The predicted image is derived from the prediction parameters associated with the block. These prediction parameters include intra-prediction and inter-prediction parameters.

[0077] The following describes the prediction parameters for interpretation. The interpretation parameters are the prediction list usage flags predFlagL0 and predFlagL1, and the reference picture index refIdxL0 and re fIdxL1 is composed of motion vectors mvL0 and mvL1. predFlagL0 and predFlagL1 are flags that indicate whether or not a reference picture list (L0 list, L1 list) is used; if the value is 1, the corresponding reference picture list is used. In this specification, when referring to a "flag indicating whether or not XX is true," a flag other than 0 (e.g., 1) is considered true, and 0 is considered false. In logical negation, logical AND, etc., 1 is treated as true and 0 as false (the same applies below). However, in actual devices and methods, other values ​​may be used as true and false values.

[0078] Syntax elements for deriving interpretation parameters include, for example, the affine flag affine_flag, merge flag merge_flag, merge index merge_idx, MMVD flag mmvd_flag used in merge mode, and the element for selecting the reference picture used in AMVP mode. Interpretation identifier inter_pred_idc, reference picture index refIdxLX, prediction vector index mvp_LX_idx for deriving motion vector, difference vector mvdLX, motion vector There is a precision mode called amvr_mode.

[0079] (Reference picture list) The reference picture list consists of reference pictures stored in the reference picture memory 306. This is a stock. Figure 4 is a conceptual diagram showing an example of a reference picture and a reference picture list. In the conceptual diagram showing an example of a reference picture in Figure 4, the rectangle represents the picture, and the arrow represents the picture. Reference relationship, horizontal axis is time, I, P, and B in the rectangle are intrapicture, single predictive picture, and bidirectional picture, respectively. In the predicted picture, the numbers in the rectangle indicate the decoding order. As shown in the figure, the decoding order of the pictures is I0, P1, B2, B3, B4, and the display order is I0, B3, B2, B4, P1. Figure 4 shows an example of the reference picture list for picture B3 (the target picture). A reference picture list is a list that represents candidate reference pictures, and a single picture (slice) may have one or more reference picture lists. In the example in the figure, the target picture B3 has two reference picture lists: L0 list RefPicList0 and L1 list RefPicList1. In each CU, refIdxLX specifies which picture in the reference picture list RefPicListX (X=0 or 1) to actually reference. The figure shows an example where refIdxL0=2 and refIdxL1=0. Note that LX is a notation used when L0 predictions and L1 predictions are not distinguished. Hereafter, we will distinguish between parameters for the L0 list and parameters for the L1 list by replacing LX with L0 and L1.

[0080] (Merge prediction and AMVP prediction) There are two methods for decoding (encoding) prediction parameters: merge prediction mode and AMVP (Advanced Motion Vector Prediction) mode. merge_flag is a flag used to distinguish between these modes. In merge prediction mode, the prediction list usage flag predFlagLX, reference picture index refIdxLX, and motion vector mvLX are not included in the encoded data, but are derived from the prediction parameters of already processed neighboring blocks. In AMVP mode, inter_pred_idc, refIdxLX, and mvLX are included in the encoded data. Note that mvLX is encoded as mvp_LX_idx, which identifies the prediction vector mvpLX, and the difference vector mvdLX. In addition to merge prediction mode, affine prediction mode and MMVD prediction mode may also be available.

[0081] inter_pred_idc is a value that indicates the type and number of reference pictures, and can take one of the values ​​PRED_L0, PRED_L1, or PRED_BI. PRED_L0 and PRED_L1 are managed in the L0 list and L1 list, respectively. This demonstrates a single prediction using one reference picture. PRED_BI is managed by L0 list and L1 list. This demonstrates a biprediction method using two reference pictures.

[0082] merge_idx is a candidate for the predicted parameters (merge candidate) derived from the block that has been processed. This is an index indicating which of the following prediction parameters will be used as the prediction parameter for the target block.

[0083] (Motion vector) mvLX represents the amount of shift between blocks in two different pictures. The prediction vector and difference vector for mvLX are called mvpLX and mvdLX, respectively.

[0084] (Interpretation identifier inter_pred_idc and prediction list usage flag predFlagLX) The relationship between inter_pred_idc, predFlagL0, and predFlagL1 is as follows, and they are mutually convertible: inter_pred_idc = (predFlagL1<<1)+predFlagL0 predFlagL0 = inter_pred_idc & 1 predFlagL1 = inter_pred_idc >> 1 The inter-prediction parameters may use either the prediction list usage flag or the inter-prediction identifier. Furthermore, the determination using the prediction list usage flag may be replaced with the determination using the inter-prediction identifier. Conversely, the determination using the inter-prediction identifier may be replaced with the determination using the prediction list usage flag.

[0085] (Configuration of the image decoding device) The configuration of the image decoding device 31 (Figure 5) according to this embodiment will be described below.

[0086] The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (predictive image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, and a predicted image Generation unit (predictive image generation device) 308, inverse quantization / inverse transformation unit 311, and addition unit 312, prediction parameter It is configured to include a data output unit 320. Furthermore, in accordance with the image encoding device 11 described later, it is configured to perform image decoding. There are also configurations in which the device 31 does not include the loop filter 305.

[0087] The parameter decoding unit 302 further comprises a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit. The system includes a unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These can be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice headers (slice information) from the encoded data. The CT information decoding unit 3021 decodes the CT from the encoded data. The CU decoding unit 3022 decodes the CU from the encoded data. The TU decoding unit 3024 decodes the QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data if the TU contains a prediction error.

[0088] The TU decoding unit 3024 decodes the QP update information and quantization prediction error from the encoded data when the skip mode is not active (skip_mode==0). More specifically, when skip_mode==0, the TU decoding unit 3024 decodes the flag cu_cbp, which indicates whether or not the target block contains a quantization prediction error, and decodes the quantization prediction error if cu_cbp is 1. If cu_cbp does not exist in the encoded data... In this case, the answer is derived as 0.

[0089] The TU decoding unit 3024 decodes the mts_idx index, which indicates the transformation basis, from the encoded data. Furthermore, the TU decoding unit 3024 decodes an index stIdx from the encoded data that indicates the use of secondary transformations and transformation basis. If stIdx is 0, it indicates that the secondary transformation is not applied; if it is 1, it indicates one of the transformations in the set (pair) of secondary transformation basis; and if it is 2, it indicates the use of the above pair. This shows the other conversion.

[0090] The predictive image generation unit 308 connects the inter predictive image generation unit 309 and the intra predictive image generation unit 310. It is composed of including.

[0091] The prediction parameter derivation unit 320 is configured to include an inter-prediction parameter derivation unit 303 and an intra-prediction parameter derivation unit 304.

[0092] The entropy decoding unit 301 calculates the entropy of the encoded data Te input from the outside. Decoding is performed to decode individual codes (syntax elements). Entropy coding has two methods: one that uses a context (probability model) adaptively selected according to the type of syntax element and the surrounding circumstances to encode syntax elements in a variable length, and another that uses a predetermined table or calculation formula to encode syntax elements in a variable length. The former, CABAC (Context Adaptive Binary Arithmetic Coding), stores the CABAC state of the context (a probability state index pStateIdx that specifies the type (0 or 1) and probability of the dominant symbol) in memory. The entropy decoding unit 301 initializes all CABAC states at the beginning of a segment (tile, CTU row, slice). The entropy decoding unit 301 converts the syntax elements into a binary string and decodes each bit of the Bin String. When a context is used, a context index ctxInc is derived for each bit of the syntax element, the bits are decoded using the context, and the CABAC state of the context used is updated. Bits without context are decoded with equal probability (EP, bypass), and the ctxInc derivation and CABAC state are omitted. The decoded syntax elements include prediction information for generating the predicted image and prediction errors for generating the difference image.

[0093] The entropy decoding unit 301 outputs the decoded codes to the parameter decoding unit 302. The decoded codes include, for example, the prediction mode predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX, amvr_mode, etc. The control of which codes to decode is performed based on the instructions of the parameter decoding unit 302.

[0094] (Basic Flow) Figure 6 is a flowchart illustrating the schematic operation of the image decoding device 31.

[0095] (S1100: Parameter set information decoding) The header decoding unit 3020 decodes parameter set information such as VPS, SPS, and PPS from the encoded data.

[0096] (S1200: Slice information decoding) The header decoding unit 3020 decodes the slice header from the encoded data. Decode the (slice information).

[0097] The image decoding device 31 then performs the following steps for each CTU included in the target picture, from S1300 to S5000. By repeating the process, the decoded image of each CTU is derived.

[0098] (S1300: CTU information decoding) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0099] (S1400: CT information decoding) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0100] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data. To designate.

[0101] (S1510: CU information decoding) The CU decoding unit 3022 decodes the encoded data into CU information, prediction information, and TU partitioning. Decode the flag split_transform_flag, CU residual flags cbf_cb, cbf_cr, cbf_luma, etc.

[0102] (S1520: TU information decoding) The TU decoding unit 3024 encodes the TU if it contains a prediction error. The QP update information, quantization prediction error, and transformation index mts_idx are decoded from the data. Note that the QP update information is derived from the quantization parameter prediction value qPpred, which is the predicted value of the quantization parameter QP. This is the difference value.

[0103] (S2000: Predictive image generation) The predictive image generation unit 308 generates a predictive image for each block included in the target CU. It generates a predicted image based on the predictive information.

[0104] (S3000: Inverse Quantization / Inverse Transform) The inverse quantization / inverse transform unit 311 performs inverse quantization / inverse transform processing for each TU included in the target CU.

[0105] (S4000: Decoded image generation) The addition unit 312 combines the predicted image supplied from the predicted image generation unit 308 with the predicted image. By adding the prediction error supplied from the inverse quantization / inverse conversion unit 311, the target CU is obtained. Generate a decoded image.

[0106] (S5000: Loop filter) Loop filter 305 applies loop filters such as a deblocking filter, SAO, and ALF to the decoded image to generate a decoded image.

[0107] (Configuration of the Interpretation Parameter Derivation Unit) The interpretation parameter derivation unit 303 (motion vector derivation device) uses the syntax elements input from the parameter decoding unit 302 to store the prediction parameters in the prediction parameter memory 307. Interpretation parameters are derived by referring to the measured parameters. The interpretation parameters are also output to the interpretation image generation unit 309 and the prediction parameter memory 307. The interpretation parameter derivation unit 303 and its internal elements, the AMVP prediction parameter derivation unit 3032, the merge prediction parameter derivation unit 3036, the affine prediction unit 30372, the MMVD prediction unit 30373, the GPM unit 30377, the DMVR unit 30537, and the MV addition unit 3038, are common means in the image encoding device and the image decoding device, so they may be collectively referred to as the motion vector derivation unit (motion vector derivation device).

[0108] The scale parameter derivation unit 30378 provided by the header decoding unit 3020 and the header encoding unit 1110 is We derive the horizontal scaling ratio of the referenced picture, RefPicScale[i][j][0], the vertical scaling ratio of the referenced picture, RefPicScale[i][j][1], and RefPicIsScaled[i][j], which indicates whether the referenced picture is scaled or not. Here, i indicates whether the referenced picture list is an L0 list (i=0) or an L1 list (i=1), and j is the value (referenced picture) of the L0 referenced picture list or L1 referenced picture list, and we derive them as follows: RefPicScale[i][j][0] = ((fRefWidth << 14)+(PicScaleWidthL >> 1)) / PicScaleWidthL RefPicScale[i][j][1] = ((fRefHeight << 14)+(PicScaleHeightL >> 1)) / PicScaleHeightL RefPicIsScaled[i][j] = (RefPicScale[i][j][0] != (1<<14)) || (RefPicScale[i][j][1] != (1<<14)) Here, the variable PicScaleWidthL is the value used to calculate the horizontal scaling ratio when the encoded picture is referenced, and it is obtained by subtracting the left and right offset values ​​from the horizontal number of pixels of the luminance of the encoded picture. The variable PicScaleHeightL is used when the encoded picture is referenced This is the value used when calculating the vertical scaling ratio during encoding, and it is obtained by subtracting the vertical offset value from the vertical number of pixels in the brightness of the encoded picture. The variable fRefWidth is The value of PicScaleWidthL is the reference list value j of list i, and the variable fRefHight is the reference list value of list i. This value is set to the PicScaleHeightL value of the picture list value j.

[0109] (MV addition section) The MV summing unit 3038 calculates mvLX by adding the mvpLX input from the AMVP prediction parameter derivation unit 3032 and the decoded mvdLX. The summing unit 3038 then processes the calculated mvLX into the interprediction image generation unit 309 And output to the prediction parameter memory 307: mvLX[0] = mvpLX[0]+mvdLX[0] mvLX[1] = mvpLX[1]+mvdLX[1] Loop filter 305 is a filter placed within the encoding loop to correct block distortion and ringing. This is a filter that removes distortion and improves image quality. The loop filter 305 is generated by the summing unit 312. For the decoded CU image, a deblocking filter, sample-adaptive offset (SAO), and appropriate Apply filters such as an automatic loop filter (ALF).

[0110] The DF unit 601 derives the intensity bS of the deblocking filter on a pixel, boundary, or line segment basis. Output section 602, DF filter that performs deblocking filter processing to reduce block noise. It consists of section 602.

[0111] The DF unit 601 derives edgeIdc, which indicates whether the input image resPicture before NN (Neural Network) processing (processing by the NN filter unit 601) has partition boundaries, prediction block boundaries, and transformation block boundaries, and maxFilterLength, which is the maximum filter length of the deblocking filter. Furthermore, it derives the intensity bS of the deblocking filter from edgeIdc, the transformation block boundaries, and the coding parameters. The coding parameters include, for example, the prediction mode CuPredMode, the BDPCM prediction mode intra_bdpcm_luma_flag, a flag indicating whether it is IBC prediction mode, a motion vector, a reference picture, and flags tu_y_coded_flag and tu_u_coded_flag indicating whether non-zero coefficients exist in the transformation blocks. edgeIdc and bS may take values ​​of 0, 1, or 2, or other values.

[0112] The reference picture memory 306 stores the decoded image of the CU in a predetermined format for each target picture and target CU. Store in location.

[0113] The prediction parameter memory 307 stores prediction parameters at predetermined locations for each CTU or CU. Specifically, the prediction parameter memory 307 stores parameters decoded by the parameter decoding unit 302 and parameters derived by the prediction parameter derivation unit 320, etc.

[0114] The predictive image generation unit 308 receives the parameters derived by the predictive parameter derivation unit 320. The predictive image generation unit 308 also reads a reference picture from the reference picture memory 306. The predictive image generation unit 308 uses the prediction mode indicated by predMode to input the parameters and the reference picture (reference picture A block or subblock prediction image is generated using a reference picture block. Here, a reference picture block is a set of pixels on a reference picture (usually rectangular, hence called a block), and is the region referenced to generate the prediction image.

[0115] If predMode indicates interprediction mode, the interprediction image generation unit 309 performs interprediction. Using the interprediction parameters input from the prediction parameter derivation unit 303 and the reference picture Interpretation generates predicted images of blocks or subblocks.

[0116] (Motion compensation) The motion compensation unit 3091 (interpolation image generation unit 3091) receives input from the interpretation parameter derivation unit 303. Based on the predicted interpretation parameters (predFlagLX, refIdxLX, mvLX), an interpolated image (motion-compensated image) is generated by reading a reference block from the reference picture memory 306. The image is generated. The reference block is the block located at a position shifted by mvLX from the position of the target block on the reference picture RefPicLX specified by refIdxLX. If mvLX is not integer precision, a filter called a motion compensation filter is applied to generate pixels at decimal positions, and an interpolated image is generated.

[0117] The motion compensation unit 3091 first derives the integer position (xInt, yInt) and phase (xFrac, yFrac) corresponding to the coordinates (x, y) within the prediction block using the following formula: xInt = xPb+(mvLX[0]>>(log2(MVPREC)))+x xFrac = mvLX[0]&(MVPREC-1) yInt = yPb+(mvLX[1]>>(log2(MVPREC)))+y yFrac = mvLX[1]&(MVPREC-1) Here, (xPb, yPb) are the top-left coordinates of a block of size bW*bH, x=0...bW-1, y=0...bH-1, and MVPREC represents the precision of mvLX (1 / MVPREC pixel precision). For example, MVPREC = 16.

[0118] The motion compensation unit 3091 derives a temporary image temp[][] by performing horizontal interpolation on the reference picture refImg using an interpolation filter. The following Σ is the sum with respect to k of k=0..NTAP-1, shift1 is a normalization parameter that adjusts the range of values, and offset1=1<<(shift1-1): temp[x][y] = (ΣmcFilter[xFrac][k]*refImg[xInt+k-NTAP / 2+1][yInt]+offset1)>>shift1 Next, the motion compensation unit 3091 derives the interpolated image Pred[][] from the temporary image temp[][] by vertical interpolation. The following Σ is the sum with respect to k of k=0..NTAP-1, shift2 is a normalization parameter that adjusts the range of values, and offset2=1<<(shift2-1): Pred[x][y] = (ΣmcFilter[yFrac][k]*temp[x][y+k-NTAP / 2+1]+offset2)>>shift2 In the case of biprediction, the above Pred[][] is derived for each L0 list and L1 list (referred to as interpolated images PredL0[][] and PredL1[][]), and the interpolated image Pred[][] is generated from PredL0[][] and PredL1[][].

[0119] Furthermore, the motion compensation unit 3091 uses the reference picture derived by the scale parameter derivation unit 30378. It has the function to scale the interpolated image according to the horizontal scaling ratio RefPicScale[i][j][0] of the reference picture and the vertical scaling ratio RefPicScale[i][j][1] of the reference picture.

[0120] The intra predictive image generation unit 310, when predMode indicates intra predictive mode, Intra-prediction is performed using the intra-prediction parameters input from the prediction parameter derivation unit 304 and the reference pixels read from the reference picture memory 306.

[0121] The inverse quantization / inverse transformation unit 311 (residual decoding unit) inversely quantizes the quantization transformation coefficients input from the parameter decoding unit 302 to obtain the transformation coefficients.

[0122] The inverse quantization / inverse transformation unit 311 scales (inverse quantizes) the quantization transformation coefficients qd[][] input from the entropy decoding unit 301 using the scaling unit 31111 to obtain the transformation coefficients d[][].

[0123] The scaling unit 31111 uses the quantization parameters and scaling factor derived in the parameter decoding unit 302 to scale the transformation coefficients decoded by the TU decoding unit using weights in units of coefficients.

[0124] Here, the quantization parameter qP is derived using the color component cIdx of the target transformation coefficient and the joint color difference residual coding flag tu_joint_cbcr_flag as follows.

[0125] qP = qPY (cIdx==0) qP = qPCb (cIdx==1 && tu_joint_cbcr_flag==0) qP = qPCr (cIdx==2 && tu_joint_cbcr_flag==0) qP = qPCbCr (tu_joint_cbcr_flag!=0) The scaling unit 31111 derives a value rectNonTsFlag related to size or shape from the size (nTbW, nTbH) of the target TU.

[0126] rectNonTsFlag = (((Log2(nTbW)+Log2(nTbH)) & 1)==1 && transform_skip_flag[xTbY][yTbY]==0) `transform_skip_flag` is a flag that indicates whether or not to skip the transformation.

[0127] The scaling unit 31111 is derived in the scaling list decoding unit 3026 (not shown). The following process is performed using the selected ScalingFactor[][].

[0128] The scaling unit 31111 performs the following actions when the scaling list is not enabled (scaling_list_enabled_flag==0) or when transformation skipping is used (transform_skip_flag==1): Set m[x][y]=16. In other words, uniform quantization is performed. scaling_list_enabled_flag is This flag indicates whether the kaling list is enabled or disabled.

[0129] Otherwise (i.e., when scaling_list_enabled_flag==1 and transform_skip_flag==0), the scaling unit 31111 uses a scaling list. Here, m[][] is set as follows.

[0130] m[x][y] = ScalingFactor[Log2(nTbW)][Log2(nTbH)][matrixId][x][y] Here, matrixId is set by the prediction mode (CuPredMode), color component index (cIdx), and whether or not non-separated transformation is applied (lfnst_idx) of the target TU.

[0131] The scaling unit 31111 sets the scaling factor ls[x][y] to 1 if sh_dep_quant_used_flag is 1 In this case, the following formula is used for derivation.

[0132] ls[x][y] = (m[x][y]*quantScale[rectNonTsFlag][(qP+1)%6]) << ((qP+1) / 6) Otherwise (sh_dep_quant_used_flag=0), it can be derived using the following formula.

[0133] ls[x][y] = (m[x][y]*quantScale[rectNonTsFlag][qP%6]) << (qP / 6) Here, quantScale[] = {{ 40, 45, 51, 57, 64, 72}, {57, 64, 72, 80, 90, 102}}. sh_dep_quant_used_flag is a flag that is set to 1 if dependent quantization is performed, and 0 if it is not. The value of quantScale is derived using the following formula based on the value of x (x=0..6).

[0134] quantScale[x] = RoundInt(2^(6 / (x-qsoffset))) qsoffset = rectNonTsFlag==0 ? 4 : 2 If the value of qP is 4, then quantScale is 64. Here, RoundInt is a function that adds a rounding constant (e.g., 0.5) and then truncates the decimal part to convert it to an integer.

[0135] The scaling unit 31111 performs inverse quantization by deriving dnc[][] from the product of the scaling factor ls[][] and the decoded transformation coefficient TransCoeffLevel.

[0136] dnc[x][y] = (TransCoeffLevel[xTbY][yTbY][cIdx][x][y]*ls[x][y]+bdOffset1) >> bdShift1 Here, bdOffset1 = 1 << (bdShift1 - 1) Finally, the scaling unit 31111 clips the inversely quantized transformation coefficients to derive d[x][y].

[0137] d[x][y] = Clip3(CoeffMin, CoeffMax, dnc[x][y]) (formula CLIP-1) CoeffMin and CoeffMax are the minimum and maximum values ​​of the clipping, and are derived using the following formulas.

[0138] CoeffMin = -(1 << log2TransformRange) CoeffMax = (1 << log2TransformRange) - 1 Here, log2TransformRange is a value that indicates the range of the transformation coefficients derived using the method described later.

[0139] d[x][y] is transmitted to the inverse core conversion unit 31123 or the inverse non-separated conversion unit 31121. The transformation unit 31121 performs an inverse non-separable transformation on the transformation coefficients d[][] after inverse quantization and before core transformation. Apply.

[0140] The addition unit 312 adds the predicted image of the block input from the prediction image generation unit 308 and the prediction error input from the inverse quantization / inverse transform unit 311 pixel by pixel to generate a decoded image of the block. The summing unit 312 stores the decoded image of the block in the reference picture memory 306 and also outputs it to the loop filter 305.

[0141] The inverse quantization / inverse transformation unit 311 inversely quantizes the quantization transformation coefficients input from the parameter decoding unit 302 to obtain the transformation coefficients.

[0142] The addition unit 312 adds the predicted image of the block input from the prediction image generation unit 308 and the prediction error input from the inverse quantization / inverse transform unit 311 pixel by pixel to generate a decoded image of the block. The summing unit 312 stores the decoded image of the block in the reference picture memory 306 and also outputs it to the loop filter 305.

[0143] (SEI for post-filtering based on neural networks) Figure 9 shows the syntax of SEI for neural network-based post-filtering. • nnrpf_id: This is the identification number of the neural network. ·nnrpf_mode_idc: Specifies the neural network model to be used for post-filtering. This is the mode index indicating the method. If the value is 0, it is the NN associated with nnrpf_id. (Nueral Network) Indicates that the filter is not specified in this SEI message. Value If the value is 1, it indicates that the NN filter associated with nnrpf_id is a neural network model identified by a given URI (Uniform Resource Identifier). A URI is an identifying string that indicates a logical or physical resource. Note that the actual data does not need to exist at the location indicated by the URI; it is sufficient that the string identifies the resource. If the value is 2, it indicates that the NN filter associated with nnrpf_id is a neural network model represented by the ISO / IEC 15938-17 bitstream included in this SEI message. If the value is 3, it indicates that the NN filter associated with nnrpf_id is a neural network model identified in the NN filter SEI message used in the previous decoding and updated by the ISO / IEC 15938-17 bitstream included in this SEI message. • nnrpf_purpose indicates the purpose of post-filtering. If the value of nnrpf_purpose is 0, This indicates the purpose of improving image quality through post-filtering. A value of nnrpf_purpose of 1 indicates that a color difference format conversion will be performed. Specifically, this refers to a resolution conversion of the color difference signal, such as converting a 4:2:0 format to a 4:4:4 format. A value of nnrpf_purpose of 2 indicates that the image resolution will be converted through post-filtering to increase the image size. ·nnrpf_out_sub_c_idc specifies the difference between the chroma format display value of the output image and the ChromaFormatIdc of the input image. The value must be in the range of 0 to 3 -ChromaFormatIdc. For chroma sampling over luminance sampling of the output image, the chroma difference format display variable OutputChromaFormatIDC is derived as follows:

[0144] OutputChromaFormatIdc = ChromaFormatIdc + nnrpf_out_sub_c_idc If the value of OutputChromaFormatIdc is 0, the output is a monochrome image, and the variables outSubWidthC and outSubHeightC will contain: outSubWidthC = 1 outSubHeightC = 1 This is substituted.

[0145] If the value of OutputChromaFormatIdc is 1, the output is a 4:2:0 formatted image, and the variables outSubWidthC and outSubHeightC will contain: outSubWidthC = 2 outSubHeightC = 2 This is substituted.

[0146] If the value of OutputChromaFormatIdc is 2, the output is a 4:2:2 format image, and the variables outSubWidthC and outSubHeightC will contain: outSubWidthC = 2 outSubHeightC = 1 This is substituted.

[0147] If the value of OutputChromaFormatIdc is 3, the output will be a 4:4:4 formatted image, and the variables outSubWidthC and outSubHeightC will contain: outSubWidthC = 1 outSubHeightC = 1 This is substituted. ·nnrpf_patch_size_minus1 + 1 is the horizontal and Specify the number of pixels in the vertical direction. ·nnrpf_overlap * 2 + nnrpf_patch_size_minus1 + 1 is calculated for each input in the post-filtering process. Specify the horizontal and vertical pixel counts for each tensor. The value of nnrpf_overlap must be in the range of 0 to 16383. A patch is a block that divides the screen, and pixels are input to the input tensor with overlaps on the left, right, top, and bottom by the value of nnrpf_overlap. Where nnrpf_pic_width_in_luma_samples and nnrpf_pic_height_in_luma_samples exist The combined value shows the width and height of the luminance pixel array in the decoded image, respectively, after applying the post-filtering process identified by nnrpf_id to the decoded image.

[0148] Non-patent document 2 states that when resolution conversion is performed by post-filtering, the image after resolution conversion The size is defined. Also, Non-Patent Document 3 defines the unit of processing in a neural network. Although a patch size is defined, its relationship to the input and output image sizes was unclear. As a result, problems sometimes occurred in the input and output of neural network processing. Therefore, in this embodiment, each variable is set as follows.

[0149] The variables patchWidth and patchHeight represent the horizontal and vertical dimensions of the input patch, The variables outPatchWidth and outPatchHight, which represent the horizontal and vertical size of the output patch, and the variables outPatchCWidth and outPatchCHeight, which indicate the horizontal and vertical size of the output color difference signal patch, and the variable overlapSize, which indicates the size of the overlap, are derived as follows.

[0150] patchWidth = nnrpf_patch_size_minus1 + 1 patchHeight = nnrpf_patch_size_minus1 + 1 outPatchWidth = (nnrpf_pic_width_in_luma_samples * patchWidth) / PicWidthInLumaSamples outPatchHeight = (nnrpf_pic_height_in_luma_samples * patchHeight) / (PicHeightInLumaSamples) outPatchCWidth = outPatchWidth * InpSubWidthC / outSubWidthC outPatchCHeight = outPatchHeight * InpSubHeightC / outSubHeightC hiddenClass = nnrpf_overlap Here, the value of outPatchWidth * PicWidthInLumaSamples is set to be equal to the value of nnrpf_pic_width_in_luma_samples * patchWidth. Also, the value of outPatchHight * PicHightInLumaSamples is set to be equal to the value of nnrpf_pic_hight_in_luma_samples * patchHight.

[0151] This constraint is equivalent to making the ratio of PicWidthInLumaSamples to nnrpf_pic_width_in_luma_samples equal to the ratio of patchWidth to outPatchWidth, and also equivalent to making the ratio of PicHeightInLumaSamples to nnrpf_pic_height_in_luma_samples equal to the ratio of patchHeight to outPatchHeight.

[0152] By imposing constraints on the values ​​of the variables described above, and by applying these constraints to the size of the decoded image, the size of the image after post-filtering, and the size of the patch, it is possible to process the image without degradation even when resolution conversion is performed during post-filtering. nnrpf_io_order_idc indicates how the decoded image is input and output to a tensor for post-filtering in a neural network.

[0153] If the value of nnrpf_io_order_idc is 0, only one channel of luminance component is input to and output from the input and output tensors.

[0154] If the value of nnrpf_io_order_idc is 1, only 2-channel color difference components will be input to and output from the input and output tensors.

[0155] If the value of nnrpf_io_order_idc is 2, then the input and output tensors will have a 1-channel luminance component and a 2-channel chrominance component.

[0156] If the value of nnrpf_io_order_idc is 3, then, as shown in Figure 10, the input and output tensors have 4 luminance channels, 2 chrominance channels, and a channel for the quantization parameter. Input and output.

[0157] Figure 10 illustrates the input and output processing of image data to a neural network that performs post-filtering.

[0158] When the value of nnrpf_io_order_idc is 0, as shown in the code in Figure 10, first the brightness image Based on the image size, the input tensor `inputTensor` is input for each patch size by calling `InputTensors()`, which receives only the luminance component of one channel. Next, `PostProcessingFilter(inputTensor)` is executed to perform post-filtering. Finally, the output tensor `outputTensor` of the luminance component of one channel is output by `OutputTensors(OutputTensor)` to output the image. Output as follows. Here, the values ​​of the variables cTop and cLeft are the top-left vertical direction of the brightness image data. The coordinates of the point and the horizontal coordinates are shown.

[0159] When the value of nnrpf_io_order_idc is 1, as shown in the code in Figure 10, first the brightness image Based on the image size of the color difference obtained by dividing the image size by InpSubHeightC or InpSubWidth, the input tensor `inputTensor` is input for each patch size by calling `InputTensors()`, which takes input from the 2-channel color difference component. Next, `PostProcessingFilter(inputTensor)` is executed to perform post-filtering. Finally, the output tensor `outputTensor` of the 2-channel color difference component is output as the output image using `OutputTensors()`. Here, the values ​​of the variables `cTop` and `cLeft` indicate the vertical and horizontal coordinates of the top-left corner of the color difference image data.

[0160] When the value of nnrpf_io_order_idc is 2, as shown in the code in Figure 10, first the brightness image Based on the image size, the input tensor `inputTensor` for each patch size calls `InputTensors()`, which receives input from a 1-channel luminance component and a 2-channel chrominance component. Input the data. Then, execute PostProcessingFilter(inputTensor) to perform post-filtering. Finally, the output tensor outputTensor, which consists of a 1-channel luminance component and a 2-channel chrominance component, is output as the output image using OutputTensors(). Here, the variable c The Top and cLeft values ​​indicate the vertical and horizontal coordinates of the top-left corner of the luminance image data.

[0161] When the value of nnrpf_io_order_idc is 3, as shown in the code in Figure 10, first the brightness image Based on the image size, the inputTensor generates 4 channels of luminance components at intervals of twice the patch size. The InputTensors() function takes input from the color difference component and the two-channel color difference component. Next, we execute PostProcessingFilter(inputTensor) to perform post-filtering. Finally, the output tensor outputTensor, which consists of a 4-channel luminance component and a 2-channel chrominance component, is output as the output image using OutputTensors(). Here, the values ​​of the variables cTop and cLeft represent the vertical and horizontal coordinates of the top-left corner of the luminance image data.

[0162] Figure 11 illustrates the process of InputTensors(), which inputs image data into the input tensor of a neural network that performs post-filtering.

[0163] When the value of nnrpf_io_order_idc is 0, the input tensor inputTensor receives only one channel of luminance components, as shown in the code in Figure 11. Here, the horizontal size of the input tensor inputTensor is patchWidth + 2 * overlapSize, and the vertical size is patchHeight + 2 * overlapSize, so that pixel values ​​overlap by overlapSize are input to adjacent patches in all directions (up, down, left, and right).

[0164] In this case, function InpY is a function that converts the pixel values ​​of the luminance signal of the decoded image to the type of the variable in the input tensor. Function InpC is a function that converts the pixel values ​​of the chrominance signal of the decoded image to the type of the variable in the input tensor.

[0165] When the value of nnrpf_io_order_idc is 1, the input tensor inputTensor receives two-channel color difference components, as shown in the code in Figure 11. Here, the horizontal size of the input tensor inputTensor is patchWidth + 2 * overlapSize, and the vertical size is patchHeight + 2 * overlapSize, so that pixel values ​​equal to overlapSize are input with an overlap to adjacent patches in all directions (top, bottom, left, and right).

[0166] When the value of nnrpf_io_order_idc is 2, the input tensor inputTensor receives one channel of luminance and two channels of chrominance, as shown in the code in Figure 11. Here, the horizontal size of the input tensor inputTensor is patchWidth + 2 * overlapSize, and the vertical size is patchHeight + 2 * overlapSize, so that pixel values ​​equal to overlapSize are input with an overlap to adjacent patches in all directions (top, bottom, left, and right).

[0167] In the 4:2:0 format, the color difference signal is horizontal and vertical to the luminance signal. In this case, the number of pixels is halved, and both the InpSubWidthC and InpSubHeight values ​​become 2. The input tensor takes a color difference pixel corresponding to the luminance pixel position, and if there is no corresponding pixel, it takes the color difference pixel of the nearest neighbor to the left or above as input.

[0168] In the 4:2:2 format, the color difference signal has half the number of pixels horizontally compared to the luminance signal. Therefore, the value of InpSubWidthC is 1 and the value of InpSubHeight is 2. In this case, the chrominance pixel corresponding to the luminance pixel position is input, and if there is no corresponding pixel, the chrominance pixel of the nearest neighbor to the left is input to the input tensor.

[0169] In the 4:4:4 format, the chrominance signal has the same number of pixels as the luminance signal, and both InpSubWidthC and InpSubHeight values ​​are 1. In this case, the chrominance pixels corresponding to the luminance pixel positions are input to the input tensor.

[0170] When the value of nnrpf_io_order_idc is 3, the input tensor inp utTensor takes as input values ​​obtained by transforming 4 channels of luminance components, 2 channels of chrominance components, and quantization parameters. Here, the horizontal direction of the input tensor inputTensor The size is patchWidth + 2 * overlapSize, and the vertical size is patchHeight + 2 * overlapSize. Pixel values ​​equal to overlapSize are input with the adjacent patches in all directions (top, bottom, left, and right) to create an overlap.

[0171] In the 4:2:0 format, the luminance signal is sampled horizontally and vertically for each pixel, divided into four parts, and then converted to four channels. Alternatively, the color difference signal may be sampled vertically for each pixel to create a 4-channel color difference component.

[0172] Figure 12 illustrates the process of OutputTensors(), which outputs image data that has undergone post-filtering from the output tensor of a neural network that has undergone post-filtering. This is a diagram to clarify.

[0173] In this case, function OutY is a function that converts the type of the variables in the output tensor to the pixel values ​​of the luminance signal. Function OutC is a function that converts the type of the variables in the output tensor to the pixel values ​​of the chrominance signal.

[0174] When the value of nnrpf_io_order_idc is 0, the output tensor outputTensor outputs a 1-channel luminance component, as shown in the code in Figure 12. Here, the horizontal size of the output tensor outputTensor is outPatchWidth, and the vertical size is outPatchHeight. The post-filtered luminance signal within the range of width nnrpf_pic_width_in_luma_samples and height nnrpf_pic_height_in_luma_samples of the luminance image resulting from applying post-filtering to the decoded image is output to the output image buffer FilteredYPic.

[0175] When the value of nnrpf_io_order_idc is 1, the output tensor outputTensor outputs a 2-channel color difference component, as shown in the code in Figure 12. Here, the horizontal size of the output tensor outputTensor is outPatchWidth, and the vertical size is outPatchHeight. The post-filtered color difference signal, within the range of width nnrpf_pic_width_in_luma_samples / outSubWidthC and height nnrpf_pic_height_in_luma_samples / outSubHightC of the color difference image resulting from applying post-filtering to the decoded image, is output to the output image buffer FilteredCPic.

[0176] When the value of nnrpf_io_order_idc is 2, the output tensor outputTensor outputs a 1-channel luminance component and a 2-channel chrominance component, as shown in the code in Figure 12. Here, the horizontal size of the output tensor outputTensor is outPatchWidth, and the vertical size is outPatchHeight. The output image buffer FilteredYPic contains the post-filtered luminance signal within the range of width nnrpf_pic_width_in_luma_samples and height nnrpf_pic_height_in_luma_samples of the luminance image resulting from applying post-filtering to the decoded image. It outputs the chrominance signal and, at the same time, outputs the chrominance signal output image buffer FilteredCPic after post-filtering.

[0177] When the value of nnrpf_io_order_idc is 3, the output tensor outputTensor outputs a 4-channel luminance component and a 2-channel chrominance component, as shown in the code in Figure 12. Here, the horizontal size of the output tensor outputTensor is outPatchWidth, and the vertical size is outPatchHeight. The output image buffer FilteredYPic of the luminance signal after post-filtering is output within the range of width nnrpf_pic_width_in_luma_samples and height nnrpf_pic_height_in_luma_samples of the luminance image as a result of applying post-filtering to the decoded image, and at the same time, the output image buffer FilteredCPic of the chrominance signal after post-filtering is output. • nnrpf_reserved_zero_bit must be 0. The SEI bitstream is byte-based. This is entered to align. ·nnrpf_uri[i] is an i-byte UTF-8 character code defined in ITEF Internet Standard 63. The character set contains i-byte characters terminated by a null character. The UTF-8 character code sequence is specified in IETF Internet Standard 66 to identify the neural network used for post-filtering. This includes URIs with defined syntax and semantics. ·nnrpf_payload_byte[i] contains the i-th bitstream in accordance with ISO / IEC15938-17. The byte sequence of nnrpf_payload_byte[i] conforms to ISO / IEC 15938-17. It is a bitstream.

[0178] (Another example of SEI for neural network-based post-filtering 1) Another embodiment of SEI for post-filtering based on neural networks is shown. vinegar.

[0179] Figure 13 shows another synth of SEI for neural network-based post-filtering. This shows example task 1. Below, we will omit explanations of syntax elements that are identical to those already explained in Figure 9. ·nnrpf_component_last_flag is the input and output tensors of the post-filter process. This flag indicates whether each channel is stored in the last dimension. If nnrpf_component_last_flag is 0, the channels are stored in the second dimension of the tensor. If the value is 1, store it in the last dimension. • nnrpf_inp_sample_idc indicates how to convert the pixel values ​​of the decoded image into input values ​​for post-filtering. When nnrpf_inp_sample_idc is 0, 1, 2, or 3, the input values ​​for post-filtering are... These are binary16, binary32, binary64, and binary128, respectively. These are floating-point numbers defined in IEEE 754-2019. In this case, the functions InpY, InpC, and InpQP are defined as follows: InpY( x ) = x ÷ ( ( 1 << BitDepthY ) - 1 ) InpC( x ) = x ÷ ( ( 1 << BitDepthC ) ) - 1 ) InpQP( x ) = 2^( ( x - 42 ) / 6 ) Note that the operator ÷ represents division with decimal precision (the decimal part of the quotient is not truncated). BitDepthY and BitDepthC are the bit lengths of the luminance component and chrominance component of the decoded image, respectively.

[0180] When nnrpf_inp_sample_idc is 4, 5, or 6, the input values to the post-filtering process are 8-bit unsigned integer, 16-bit unsigned integer, and 32-bit unsigned integer respectively. At this time, the functions InpY, InpC, and InpQP are defined as follows: When inpTensorBitDepth >= BitDepthY, InpY(x) = x << (inpTensorBitDepth - BitDepthY) Otherwise, InpY(x) = Clip3(0, (1<<inpTensorBitDepth) - 1, (x + (1 << (shift - 1))) >> shift) Here, shift = BitDepthY - inpTensorBitDepth When inpTensorBitDepth >= BitDepthC, InpC(x) = x << (inpTensorBitDepth - BitDepthC) Otherwise, InpC(x) = Clip3(0, (1<<inpTensorBitDepth) - 1, (x + (1 << (shift - 1))) >> shift) Here, shift = BitDepthC - inpTensorBitDepth InpQP(x) = x inpTensorBitDepth is the bit length of pixel values in the input tensor. The functions InpY, InpC, and InpQP are used to set input values ​​for the input tensor. Figure 11 includes examples of using InpY and InpC. By switching between the input value transformation functions InpY and InpC depending on the data type, the range of the input value can be appropriately transformed and input to the post-filter process. In the case of nnrpf_io_order_idc=3 shown in Figure 11, the QP value is input using the input tensor. If InpQP is used as the transformation function in this case, a transformation formula corresponding to the type of the input tensor can be applied. For example, the formula in Figure 11 inputTensor[0][6][yP+overlapSize][xP+overlapSize] = 2 (SliceQPY - 42) / 6 Instead inputTensor[0][6][yP+overlapSize][xP+overlapSize] = InpQP(SliceQPY) That would be a good idea.

[0181] Alternatively, instead of performing a rounded right shift as in the example above, you can simply perform a right shift as follows: If inpTensorBitDepth >= BitDepthY, InpY( x ) = x << ( inpTensorBitDepth - BitDepthY ) Otherwise, InpY( x ) = x >> ( BitDepthY - inpTensorBitDepth ) If inpTensorBitDepth >= BitDepthC, InpC( x ) = x << ( inpTensorBitDepth - BitDepthC ) Otherwise, InpC( x ) = x >> ( BitDepthC - inpTensorBitDepth ) ·nnrpf_inp_tensor_bitdepth_minus8 + 8 represents the bit length of the input value in the integer input tensor. inpTensorBitDepth is calculated as follows: inpTensorBitDepth = nnrpf_inp_tensor_bitdepth_minus8 + 8 nnrpf_inp_tensor_bitdepth_minus8 encodes the input tensor if its input value is an integer type larger than an 8-bit unsigned integer. If the input tensor's input value is an 8-bit unsigned integer (nnrpf_inp_sample_idc is 4), set nnrpf_inp_tensor_bitdepth_minus8 = 0. If the input tensor's input value is a 16-bit unsigned integer or a 32-bit unsigned integer (nnrpf_inp_sample_idc is 5 or 6), the range of nnrpf_inp_tensor_bitdepth_minus8 is 0 to 8 or 0 to 24, respectively.

[0182] Furthermore, you may use other values ​​such as minus1 or minus4 instead of minus8. For example, if you want the minimum bit length to be X bits, you can use nnrpf_inp_tensor_bitdepth_minusX as the syntax element and calculate inpTensorBitDepth as follows: inpTensorBitDepth = nnrpf_inp_tensor_bitdepth_minusX + X nnrpf_inp_tensor_bitdepth_minusX encodes the input tensor if its input value is an integer type greater than X bits of an unsigned integer. nnrpf_inp_order_idc is used to distribute the pixel array of the decoded image as input to the post-filtering process. This shows how to place it. • nnrpf_out_sample_idc indicates the type of the output value of the post-filter. When nnrpf_inp_sample_idc is 0, 1, 2, or 3, the input values ​​to the post-filter are binary16, binary32, binary64, and binary128, respectively. These are floating-point numbers as defined by IEEE 754-2019. In this case, the functions OutY and OutC are defined as follows: OutY(x) = Clip3(0, (1< <BitDepthY)-1, Round( x * ((1<<BitDepthY)-1) ) ) OutC(x) = Clip3(0, (1< <BitDepthC)-1, Round( x * ((1<<BitDepthC)-1) ) ) BitDepthY and BitDepthC are the bit lengths of the luminance component and chrominance component of the decoded image, respectively.

[0183] If nnrpf_inp_sample_idc is 4, 5, or 6, the input values ​​for post-filtering are 8, respectively. These are 16-bit unsigned integers, 16-bit unsigned integers, and 32-bit unsigned integers. In this case, the functions InpY, InpC, and InpQP are defined as follows: If outTensorBitDepth >= BitDepthY, OutY( x ) = x << ( outTensorBitDepth - BitDepthY ) Otherwise, OutY(x) = Clip3(0, (1< <outTensorBitDepth) -1, ( x + ( 1 << (shift-1) ) ) >> shift ) Here, shift = BitDepthY - outTensorBitDepth If outTensorBitDepth >= BitDepthC, OutC( x ) = x << ( outTensorBitDepth - BitDepthC ) Otherwise, OutC(x) = Clip3(0, (1< <outTensorBitDepth) -1, ( x + ( 1 << (shift-1) ) ) >> shift ) Here, shift = BitDepthC - outTensorBitDepth outTensorBitDepth is the bit length of the output value in the output tensor.

[0184] The functions OutY and OutC are used to obtain output values ​​from the output tensor. Figure 12 includes an example of using OutY and OutC. By switching the processing of the functions OutY and OutC, which convert output values, depending on the data type, the range of the output value can be appropriately converted and obtained. ·nnrpf_out_tensor_bitdepth_minus8 + 8 represents the bit length of the output value in the integer output tensor. outTensorBitDepth is calculated as follows: outTensorBitDepth = nnrpf_out_tensor_bitdepth_minus8 + 8 If the output value of the output tensor is an 8-bit unsigned integer (nnrpf_out_sample_idc is 4), nnrpf_out_tensor_bitdepth_minus8 is not encoded and is set to nnrpf_out_tensor_bitdepth_minus8 = 0. If the output value of the output tensor is a 16-bit unsigned integer or a 32-bit unsigned integer (nnrpf_out_sample_idc is 5 or 6), the range of the value of nnrpf_out_tensor_bitdepth_minus8 is 0 to 8 or 0 to 24, respectively.

[0185] Note that instead of using nnrpf_inp_tensor_bitdepth_minus8 and nnrpf_out_tensor_bitdepth_minus8, the following can be derived from a single syntax element nnrpf_tensor_bitdepth_minus8: It is also acceptable: inpTensorBitDepth = nnrpf_tensor_bitdepth_minus8 + 8 outTensorBitDepth = nnrpf_tensor_bitdepth_minus8 + 8 At this time, if either the input tensor or the output tensor is an integer type and either is of a type larger than 8-bit unsigned integer, then nnrpf_tensor_bitdepth_minus8 is encoded. Also, similar to the input bit length, another value indicating the minimum bit length such as minus1 or minus4 instead of minus8 may be used. · nnrpf_out_order_idc indicates how the output values of the post-filtering process are arranged. That is. · nnrpf_constant_patch_size_flag indicates whether the patch (processing unit) of the post-filtering process has a fixed size. When nnrpf_constant_patch_size_flag is 1, patchWidth and patchHeight representing the width and height of the processing unit of the post-filtering process are set to the width and height specified by the following two syntax elements: patchWidth = nnrpf_patch_width_minus1 + 1 patchWidth = nnrpf_patch_width_minus1 + 1 patchHeight = nnrpf_patch_height_minus1 + 1 If nnrpf_constant_patch_size_flag is 0, patchWidth and patchHeight are set to values ​​arbitrarily determined by the device (NN filter unit 611) that performs post-filtering. For example, the input tensor width - 2 * overlapSize, input tensor height - 2 * overlapSize, and overlapSize = 8 used by the NN filter unit 611 may be used. ·nnrpf_patch_width_minus1 + 1 indicates the width if the patch is a fixed size. ·nnrpf_patch_height_minus1 + 1 indicates the height when the patch is a fixed size.

[0186] Note that the values ​​of nnrpf_inp_sample_idc and nnrpf_out_sample_idc may be assigned to values ​​other than those described above. Furthermore, it may be possible to select types not included in the above description. For example, nnrpf_inp_sample_idc=7 and nnrpf_out_sample_idc=7 may be assigned to 64-bit unsigned integers, or to signed integers.

[0187] By doing so, the bit length can be appropriately converted between the input pixels and input tensor to the neural network, the output tensor, and the pixel values ​​of the output image after post-filtering.

[0188] Furthermore, as a variation, regardless of the values ​​of the syntax variables, patchWidth, patchHeight, and overlapSize are arbitrarily determined by the device (NN filter unit 611) that performs post-filtering. It may also be a different value. The value is as explained above.

[0189] As an alternative, the output image may be output without converting the bit length of the output value of the output tensor. In this case, the functions OutY and OutC are as follows: OutY(x) = x OutC(x) = x Alternatively, you can omit the function call.

[0190] By doing so, it is possible to obtain an output image that preserves the bit length of the output value.

[0191] (Another example of SEI for neural network-based post-filtering, part 2) Another embodiment of SEI for post-filtering based on neural networks is shown. vinegar.

[0192] Figure 14 shows another synth of SEI for neural network-based post-filtering. This shows example 2 of the task. Below, the same syntax elements as those already explained are used. Therefore, I will omit the explanation. ·nnrpf_inp_tensor_bitdepth_delta indicates the bit length of the input values ​​in the integer input tensor. inpTensorBitDepth is calculated as follows: inpTensorBitDepth = (nnrpf_inp_sample_idc == 4) ? nnrpf_inp_tensor_bitdepth_delta + 1 : (nnrpf_inp_sample_idc == 5) ? nnrpf_inp_tensor_bitdepth_delta + 9 : (nnrpf_inp_sample_idc == 6) ? nnrpf_inp_tensor_bitdepth_delta + 17 Alternatively, you may ask like this: inpTensorBitDepth = nnrpf_inp_tensor_bitdepth_delta + 1 + (nnrpf_inp_sample_idc-4) : (nnrpf_inp_sample_idc == 5) ? nnrpf_inp_tensor_bitdepth_delta + 9 : (nnrpf_inp_sample_idc == 6) ? nnrpf_inp_tensor_bitdepth_delta + 17 When nnrpf_inp_sample_idc is 4, 5, or 6, the range of nnrpf_inp_tensor_bitdepth_delta is 0 to 7, 0 to 15, and 0 to 15, respectively. The minimum possible bit length is changed depending on the integer type of the input value, and the difference from the minimum value is encoded. This allows for a wide range of bit lengths to be used while reducing the amount of code required to represent the bit length. ·nnrpf_out_tensor_bitdepth_delta indicates the bit length of the output value in the integer output tensor. outTensorBitDepth is calculated as follows: outTensorBitDepth = (nnrpf_out_sample_idc == 4) ? nnrpf_out_tensor_bitdepth_delta + 1 : (nnrpf_out_sample_idc == 5) ? nnrpf_out_tensor_bitdepth_delta + 9 : (nnrpf_out_sample_idc == 6) ? nnrpf_out_tensor_bitdepth_delta + 17 When nnrpf_out_sample_idc is 4, 5, or 6, the range of nnrpf_out_tensor_bitdepth_delta is 0 to 7, 0 to 15, and 0 to 15, respectively. This allows for a wide range of bit lengths to be used to represent the bit length, similar to nnrpf_out_tensor_bitdepth_delta. The amount of code can be reduced.

[0193] (Another example of SEI for neural network-based post-filtering, part 3) Another embodiment of SEI for post-filtering based on neural networks is shown. vinegar.

[0194] Figure 15 shows another synth of SEI for neural network-based post-filtering. This shows example 3 of the task. Below, the same syntax elements as those already explained are used. Therefore, I will omit the explanation. ·nnrpf_inp_tensor_bitdepth_luma_minus8 and nnrpf_inp_tensor_bitdepth_luma_minus8, like nnrpf_inp_tensor_bitdepth_minus8, indicate the bit length in an integer input tensor, but they only show the bit length of the luminance signal value and the chrominance signal value, respectively. The bit lengths of the luminance signal value and chrominance signal value in the input tensor, inpTensorBitDepthY and inpTensorBitDepthC, are calculated as follows: inpTensorBitDepthY = nnrpf_inp_tensor_bitdepth_luma_minus8 + 8 inpTensorBitDepthC = nnrpf_inp_tensor_bitdepth_chroma_minus8 + 8 In this case, the functions InpY and InpC for integer input tensors are defined as follows: If inpTensorBitDepthY >= BitDepthY, InpY( x ) = x << ( inpTensorBitDepthY - BitDepthY ) Otherwise, InpY(x) = Clip3(0, (1< <inpTensorBitDepthY) -1, ( x + ( 1 << (shift-1) ) ) >> shift ) Here, shift = BitDepthY - inpTensorBitDepthY If inpTensorBitDepthC >= BitDepthC, InpC( x ) = x << ( inpTensorBitDepthC - BitDepthC ) Otherwise, InpC(x) = Clip3(0, (1< <inpTensorBitDepthC) -1, ( x + ( 1 << (shift-1) ) ) >> shift ) Here, shift = BitDepthC - inpTensorBitDepthC · nnrpf_out_tensor_bitdepth_luma_minus8 and nnrpf_out_tensor_bitdepth_luma_minus8, like nnrpf_out_tensor_bitdepth_minus8, indicate the bit length in the integer output tensor, but they only indicate the bit length of the luminance signal value and the chrominance signal value, respectively. The bit lengths of the luminance signal value and chrominance signal value in the output tensor, outTensorBitDepthY and outTensorBitDepthC, are calculated as follows: outTensorBitDepthY = nnrpf_out_tensor_bitdepth_luma_minus8 + 8 outTensorBitDepthC = nnrpf_out_tensor_bitdepth_chroma_minus8 + 8 In this case, the functions OutY and OutC for the integer output tensor are defined as follows: If outTensorBitDepthY >= BitDepthY, OutY( x ) = x << ( outTensorBitDepthY - BitDepthY ) Otherwise, OutY(x) = Clip3(0, (1< <outTensorBitDepthY) -1, ( x + ( 1 << (shift-1) ) ) >> shift ) Here, shift = BitDepthY - outTensorBitDepthY As described above, it is possible to accommodate embodiments where the bit lengths of the luminance signal value and the chrominance signal value are different.

[0195] (Another example of SEI for neural network-based post-filtering, part 4) Another embodiment of SEI for post-filtering based on neural networks is shown. This embodiment is an example in which the data type and bit length of the input tensor and the output tensor are the same.

[0196] Figure 16 shows another synth of SEI for post-filtering based on neural networks. This shows example 4 of the task. Below, the same syntax elements as those already explained are used. Therefore, I will omit the explanation. ·nnrpf_sample_idc indicates the data types of the input and output tensors. The data types indicated by the values ​​are the same as those of nnrpf_inp_sample_idc, etc., which have already been explained. Set the value of nnrpf_sample_idc to nnrpf_inp_sample_idc and nnrpf_out_sample_idc. ·nnrpf_tensor_bitdepth_minus8 + 8 indicates the bit length of the input values ​​of the input and output tensors. inpTensorBitdepth and outTensorBitDepth are set as follows: inpTensorBitDepth = nnrp_tensor_bitdepth_minis8 + 8 outTensorBitDepth = nnrp_tensor_bitdepth_minis8 + 8 As described above, the encoding efficiency can be improved in embodiments where the input tensor and output tensor types are the same.

[0197] (Another example of SEI for neural network-based post-filtering 5) Another embodiment of SEI for post-filtering based on neural networks is shown. This embodiment is an example in which the data types of the input tensor and the output tensor are the same as the bit length.

[0198] Figure 17 shows another synth of SEI for neural network-based post-filtering. This shows example task 5. Below, the same syntax elements as those already explained are used. Therefore, I will omit the explanation. ·nnrpf_inp_sample_idc indicates how to convert the sample values ​​of the decoded image into input values ​​for post-filtering. When nnrpf_inp_sample_idc is 0, 1, 2, or 3, the input for post-filtering is... The values ​​are binary16, binary32, binary64, and binary128, respectively. These are floating-point numbers defined in IEEE 754-2019. The functions InpY, InpC, and InpQP are defined as follows: InpY( x ) = x ÷ ( ( 1 << BitDepthY ) - 1 ) InpC( x ) = x ÷ ( ( 1 << BitDepthC ) ) - 1 ) InpQP( x ) = 2^( ( x - 42 ) / 6 ) BitDepthY and BitDepthC are the bit lengths of the luminance component and chrominance component of the decoded image, respectively. For example, when nnrpf_inp_sample_idc is between 4 and 28... In total, the input values ​​for post-filtering are (nnrpf_imp_sample_idc-4+8) bit unsigned integers. At this time, the bit length of the input tensor, inpTensorBitDepth, is as follows: To stipulate: inpTensorBitDepth = nnrpf_inp_sample_idc - 4 + 8 The functions InpY and InpC were explained in the other example 1 above.

[0199] By doing so, the syntax can be made simpler.

[0200] (SEI decryption and post-filtering) The header decoding unit 3020 extracts network model complexity information from the defined SEI message. Decode it. SEI is additional information related to processing such as decoding and display.

[0201] Figure 18 is a flowchart showing the processing of the NN filter unit 611. The NN filter unit 611 Then, the following processing is performed according to the parameters of the above SEI message.

[0202] S6001: Reads processing volume and accuracy from the network model complexity information of SEI.

[0203] S6002: If the complexity exceeds what the NN filter unit 611 can process, the process terminates. If it does not exceed this limit, proceed to S6003.

[0204] S6003: If the accuracy exceeds the processing capacity of the NN filter unit 611, the process terminates. If it does not exceed the capacity, proceed to S6004.

[0205] S6004: Identify the network model from the SEI and set the topology of the NN filter unit 611. ru.

[0206] S6005: Derive network model parameters from SEI update information.

[0207] S6006: The parameters of the derived network model are loaded into the NN filter unit 611.

[0208] S6007: The NN filter unit 611 performs filtering and outputs the result externally.

[0209] However, SEI is not always necessary for constructing luminance samples and color difference samples in the decoding process. It will not be done.

[0210] (Example configuration of NN filter unit 611) Figure 19 shows interpolation filtering using the neural network filter unit (NN filter unit 611). This diagram shows an example configuration of a post-filter, loop filter, and post-filter. The following explanation will focus on a post-filter example, but interpolation filters or loop filters can also be used.

[0211] The post-processing unit 61 after the video decoding device includes an NN filter unit 611. The NN filter unit 611 performs filtering on the image from the reference picture memory 306 before outputting it to the outside. The image may be displayed, exported to a file, re-encoded (transcoded), transmitted, etc. The NN filter unit 611 applies a neural network model to the input image. This is a means of performing filtering. Simultaneously, scaling down or up by the same or a rational number of times may be performed.

[0212] Here, a neural network model (hereinafter referred to as an NN model) refers to the elements and connections (topology) of a neural network, as well as the parameters (weights, biases) of the neural network. Note that the topology can be fixed, and only the parameters of the neural network model can be changed.

[0213] (Details of NN filter section 611) The NN filter unit uses the input image inputTensor and input parameters (e.g., QP, bS, etc.) The image is then filtered using a neural network model. The input image may be a separate image for each component, or an image with multiple components, each serving as a channel. Furthermore, the input parameters may be assigned to channels different from those in the image.

[0214] The NN filter section may repeatedly apply the following processes.

[0215] The NN filter unit performs a convolution operation (conv, convolution) on the input Tensor with the kernel k[m][i][j] and adds a bias to derive the output image output Tensor. Here, nn = 0..n-1, xx = 0..width-1, and yy = 0..height-1.

[0216] outputTensor[nn][xx][yy]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][xx+i-of][yy+j-of]+bias[nn]) In the case of a 1x1 convolution, Σ represents the sum of mm=0,.m-1, i=0, and j=0. In this case, of=0. It is determined that in the case of a 3x3 convolution, Σ represents the sum of mm=0..m-1, i=0..2, and j=0..2 respectively. Set of=1. n is the number of channels in outSamples, m is the number of channels in inputTensor, width is the width between inputTensor and outputTensor, and height is the height between inputTensor and outputTensor. of is the size of the padding region around inputTensor to make the sizes of inputTensor and outputTensor the same. Below, if the output of the NN filter is a value (correction value) instead of an image, the output will be represented by corrNN instead of outputTensor.

[0217] Note that if you write the inputTensor and outputTensor in CHW format instead of CWH format, the following process is equivalent.

[0218] outputTensor[nn][yy][xx]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn]) Alternatively, a process called Depth-wise Conv may be performed, as shown by the following formula, where nn = 0..n-1, xx = 0..width-1, and yy = 0..height-1.

[0219] outputTensor[nn][xx][yy]=ΣΣ(k[nn][i][j]*inputTensor[nn][xx+i-of][yy+j-of]+bias[nn]) Σ represents the sum for i and j respectively. n is the number of channels in outputTensor and inputTensor, and width `width` is the width between the inputTensor and outputTensor, and `height` is the height between the inputTensor and outputTensor.

[0220] Alternatively, a nonlinear process called Activate, such as ReLU, may be used. ReLU(x) = x >= 0 ? x : 0 Alternatively, leakyReLU, as shown in the following formula, may be used.

[0221] leakyReLU(x) = x >= 0 ? x : a * x Here, a is a predetermined value, for example, 0.1 or 0.125. Alternatively, to perform integer arithmetic, all of the above values ​​of k, bias, and a may be treated as integers, and a right shift may be performed after the conv operation.

[0222] In ReLU, values ​​less than 0 are always output as 0, and values ​​greater than or equal to 0 are output as is. On the other hand, in leakyReLU, values ​​less than 0 are processed linearly with a gradient set by a. In ReLU, the gradient disappears for values ​​less than 0, which can sometimes hinder learning. In leakyReLU, the gradient for values ​​less than 0 is preserved, making the above problem less likely to occur. Alternatively, PReLU can be used, which parameterizes the value of a in leakyReLU(x).

[0223] (NNR) Neural Network Coding and Representation (NNR) is a method of using neural networks (NNs) This is an international standard for efficient compression. By compressing trained neural networks (NNs), it becomes possible to improve the efficiency of storing and transmitting NNs.

[0224] The following is an overview of the encoding and decoding process for NNR.

[0225] Figure 20 shows the encoding and decoding devices for NNR.

[0226] The NN coding device 801 includes a preprocessing unit 8011, a quantization unit 8012, and an entropy coding unit 8013. The NN encoding device 801 takes the uncompressed NN model O as input and the quantization unit 8012 processes the amount of the NN model O. The quantization is performed to obtain the quantization model Q. Before quantization, the NN encoding device 801 performs the preprocessing unit 8011 on Parameter reduction techniques such as pruning and sparsification may be repeatedly applied. After that, the entropy coding unit 8013 applies entropy coding to the quantization model Q. Apply the solution to obtain the bitstream S for saving and transmitting the NN model.

[0227] The NN decoding device 802 comprises an entropy decoding unit 8021, a parameter restoration unit 8022, and a post-processing unit 8023. The NN decoder 802 receives the initially transmitted bitstream S as input, and the entropy decoder 8021 performs entropy decoding of S to obtain the intermediate model RQ. (NN model operation) If the environment supports inference using the quantized representation used in RQ, RQ may be output and used for inference. Otherwise, the parameter recovery unit 8022 recovers the parameters of RQ to their original representation and obtains the intermediate model RP. If the sparse tensor representation to be used can be processed in the operating environment of the NN model, RP may be output and used for inference. Otherwise, the NN model O and This program finds and outputs a reconstructed neural network model R that does not include different tensors or structural representations.

[0228] The NNR standard specifies decoding methods for numerical representations of specific NN parameters, such as integers and floating-point numbers. A law exists.

[0229] The decoding method NNR_PT_INT decodes a model consisting of integer parameters. The decoding method NNR_PT_FLOAT extends NNR_PT_INT by adding a quantization step size delta. This delta is multiplied by the above integer value to generate a scaled integer. delta is derived from the integer quantization parameter qp and the granularity parameter qp_density of delta as follows:

[0230] mul = 2^(qp_density) + (qp & (2^(qp_density)-1)) delta = mul * 2^((qp >> qp_density)-qp_density) (Format of pre-trained neural network) The representation of a trained neural network consists of two elements: a topological representation, which includes layer sizes and connections between layers, and a parameter representation, which includes weights and biases.

[0231] Topological representations are covered by native formats such as Tensorflow and PyTorch. However, to improve interoperability, exchange formats such as Open Neural Network Exchange Format (ONNX) and Neural Network Exchange Format (NNEF) exist.

[0232] Furthermore, the NNR standard transmits topology information nnr_topology_unit_payload as part of the NNR bitstream, which includes a compressed parameter tensor. This allows for exchange of data. - It enables interoperability not only with matte but also with topology information expressed in native formats.

[0233] (Configuration of the image encoding device) Next, the configuration of the image encoding device 11 according to this embodiment will be described. Figure 7 shows the configuration of the image encoding device 11 according to this embodiment. This is a block diagram showing the configuration of the image coding device 11 related to the state. The image coding device 11 consists of a predictive image generation unit 101, a subtraction unit 102, a conversion / quantization unit 103, an inverse quantization / inverse conversion unit 105, an addition unit 106, and It is composed of a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determination unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

[0234] The prediction image generation unit 101 generates a prediction image for each CU. The prediction image generation unit 101 includes the inter-prediction image generation unit 309 and the intra-prediction image generation unit 310, which have already been described, so their description will be omitted.

[0235] The subtraction unit 102 subtracts the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of image T to generate a prediction error. The subtraction unit 102 converts the prediction error to the conversion / quantization unit 103. Output to [this location].

[0236] The conversion / quantization unit 103 calculates conversion coefficients by frequency conversion for the prediction error input from the subtraction unit 102, and derives quantized conversion coefficients by quantization. The quantization conversion coefficients are output to the parameter coding unit 111 and the inverse quantization / inverse conversion unit 105.

[0237] The inverse quantization / inverse transformation unit 105 is in the same configuration as the inverse quantization / inverse transformation unit 311 (Figure 5) in the image decoding device 31. They are the same, so the explanation is omitted. The calculated prediction error is output to the summing unit 106.

[0238] The parameter coding unit 111 comprises a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (predictive mode coding unit). The CU coding unit 1112 further comprises a TU coding unit 1114. The general operation of each module is described below.

[0239] The header encoding unit 1110 performs encoding processing on parameters such as header information, partitioning information, prediction information, and quantization conversion coefficients.

[0240] The CT information encoding unit 1111 encodes QT, MT (BT, TT) division information, etc.

[0241] The CU encoding unit 1112 encodes CU information, prediction information, division information, etc.

[0242] The TU encoding unit 1114 encodes the QP update information and the quantization prediction error when the TU contains a prediction error.

[0243] The CT information coding unit 1111 and the CU coding unit 1112 supply syntax elements such as inter-prediction parameters (predMode, merge_flag, merge_idx, inter_pred_idc, refIdxLX, mvp_LX_idx, mvdLX), intra-prediction parameters (intra_luma_mpm_flag, intra_luma_mpm_idx, intra_luma_mpm_reminder, intra_chroma_pred_mode), and quantization conversion coefficients to the parameter coding unit 111.

[0244] The entropy coding unit 104 receives quantization conversion coefficients and coding parameters (partitioning information, prediction parameters) from the parameter coding unit 111. These are entropy-encoded to generate encoded data Te, which is then output.

[0245] The prediction parameter derivation unit 120 is a means that includes an inter-prediction parameter coding unit 112 and an intra-prediction parameter coding unit 113, and derives intra-prediction parameters and intra-prediction parameters from parameters input from the coding parameter determination unit 110. The derived intra-prediction parameters and intra-prediction parameters are output to the parameter coding unit 111.

[0246] (Configuration of the interpredictive parameter coding unit) As shown in Figure 8, the interprediction parameter coding unit 112 includes a parameter coding control unit 1121 and an interprediction parameter derivation unit 303. The interprediction parameter derivation unit 303 has the same configuration as the image decoding device. The parameter coding control unit 1121 includes a merge index derivation unit 11211 and a vector candidate index derivation unit 11212.

[0247] The merge index derivation unit 11211 derives merge candidates, etc., and inter prediction parameters The output is sent to the derivation unit 303. The vector candidate index derivation unit 11212 derives prediction vector candidates, etc., and outputs them to the inter-prediction parameter derivation unit 303 and the parameter coding unit 111.

[0248] (Configuration of the intra-predictive parameter coding unit 113) The intra prediction parameter coding unit 113 consists of a parameter coding control unit 1131 and an intra prediction parameter coding unit 1131. It includes a parameter derivation unit 304. The intra-prediction parameter derivation unit 304 has the same configuration as the image decoding device.

[0249] The parameter coding control unit 1131 derives IntraPredModeY and IntraPredModeC. Furthermore, it determines intra_luma_mpm_flag by referring to mpmCandList[]. These prediction parameters are output to the intra prediction parameter derivation unit 304 and the parameter coding unit 111.

[0250] However, unlike the image decoding device, the interprediction parameter derivation unit 303 and the intraprediction parameter The input to the meter derivation unit 304 is processed by the coding parameter determination unit 110 and the prediction parameter memory 108. Yes, it outputs to the parameter coding unit 111.

[0251] The addition unit 106 generates a decoded image by adding the pixel values ​​of the prediction blocks input from the prediction image generation unit 101 and the prediction error input from the inverse quantization / inverse transform unit 105 for each pixel. The addition unit 106 stores the generated decoded image in the reference picture memory 109.

[0252] The loop filter 107 applies a deblocking filter, SAO, and ALF to the decoded image generated by the summing unit 106. Note that the loop filter 107 does not necessarily include the three types of filters mentioned above. It is not necessary; for example, a configuration consisting only of a deblocking filter is acceptable.

[0253] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in predetermined locations for each target picture and CU.

[0254] The reference picture memory 109 stores the decoded images generated by the loop filter 107 at predetermined locations for each target picture and CU.

[0255] The coding parameter determination unit 110 selects one set from among multiple sets of coding parameters. Select the appropriate option. The coding parameters are the QT, BT, or TT segmentation information, prediction parameters, or parameters that are generated in relation to these and are subject to coding. The prediction image generation unit 101 generates a prediction image using these coding parameters.

[0256] The coding parameter determination unit 110 determines the amount of information and coding for each of the multiple sets. The RD cost value, which indicates the error, is calculated. The RD cost value is, for example, the sum of the code amount and the squared error multiplied by a coefficient λ. The code amount is the information content of the encoded data Te obtained by entropy encoding the quantization error and encoding parameters. The squared error is calculated in the subtraction unit 102. This is the sum of the squares of the prediction errors. The coefficient λ is a preset real number greater than zero. The coding parameter determination unit 110 determines the set of coding parameters that minimizes the calculated cost value. Select the option. The coding parameter determination unit 110 uses the determined coding parameters as the parameter code. The output is sent to the processing unit 111 and the prediction parameter derivation unit 120.

[0257] Furthermore, some parts of the image coding device 11 and image decoding device 31 in the above-described embodiment, such as the entropy decoding unit 301, parameter decoding unit 302, loop filter 305, prediction image generation unit 308, inverse quantization / inverse transformation unit 311, addition unit 312, prediction parameter derivation unit 320, prediction image generation unit 101, subtraction unit 102, transformation / quantization unit 103, entropy coding unit 104, inverse quantization / inverse transformation unit 105, loop filter 107, coding parameter determination unit 110, parameter coding unit 111, and prediction parameter derivation unit 120, may be implemented using a computer. In that case, a program for realizing this control function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Hereinafter, "computer system" refers to a computer system built into either the image coding device 11 or the image decoding device 31, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. In addition, "computer-readable recording media" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. Moreover, the above-mentioned programs may be for the purpose of realizing some of the functions described above, and may also be programs that can realize the aforementioned functions in combination with programs already recorded in the computer system.

[0258] Furthermore, some or all of the image encoding device 11 and image decoding device 31 in the above-described embodiment may be implemented as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and image decoding device 31 may be individually implemented as a processor, or some or all of them may be integrated into a single processor. In addition, the method of implementing the integrated circuit is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. Furthermore, if an integrated circuit technology that can replace LSIs emerges due to advances in semiconductor technology, an integrated circuit using that technology may be used.

[0259] Although one embodiment of this invention has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes can be made without departing from the spirit of this invention.

[0260] This embodiment will be described with reference to Figure 1, where encoded data is decoded to generate a decoded image. The motion image decoding device comprises an image decoding device and a resolution inverse conversion device using a neural network that converts the decoded image to a specified resolution using inverse conversion information, wherein the resolution inverse conversion device decodes information specifying the resolution and information indicating the unit of the inverse conversion process, and the values ​​of the information specifying the resolution and the information indicating the unit of the inverse conversion process have the same proportional relationship.

[0261] Furthermore, the motion image decoding device comprises an image decoding device that decodes encoded data to generate a decoded image, and a resolution inverse transform device using a neural network that converts the decoded image to a specified resolution using inverse transform information, and is characterized in that the pixel values ​​of the image and the input / output values ​​of the tensor are converted to each other using the data types of the input and output tensor values ​​of the neural network in the resolution inverse transform device and the bit length of the pixel values ​​of the decoded image.

[0262] Furthermore, the motion image encoding device comprises an image encoding device that encodes an image and generates encoded data, an inverse conversion information generation device that generates inverse conversion information for inversely converting the resolution of a decoded image obtained by decoding the encoded data, and an inverse conversion information encoding device that encodes the inverse conversion information as auxiliary extended information, wherein the inverse conversion information generates inverse conversion information in which the values ​​of information specifying the resolution and information indicating the unit of the inverse conversion process have the same proportional relationship.

[0263] Furthermore, the motion image encoding device comprises an image encoding device that encodes an image and generates encoded data, an inverse transformation information generation device that generates inverse transformation information for inversely transforming the resolution of a decoded image obtained by decoding the encoded data, and an inverse transformation information encoding device that encodes the inverse transformation information as auxiliary extension information, and is characterized in that it generates inverse transformation information that converts the pixel values ​​of the image and the input / output values ​​of the tensor to each other using the data types of the input tensor and output tensor values ​​of the neural network in the resolution inverse transformation device and the bit length of the pixel values ​​of the encoded image.

[0264] The embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the claims. That is, embodiments obtained by combining technical means that have been appropriately modified within the scope of the claims are also included in the technical scope of the present invention. [Industrial applicability]

[0265] Embodiments of the present invention can be suitably applied to a video decoding device that decodes encoded data from image data, and a video encoding device that generates encoded data from image data. Furthermore, they can be suitably applied to the data structure of encoded data generated by the video encoding device and referenced by the video decoding device. [Explanation of symbols]

[0266] 1. Video transmission system 30. Video Decoder 31 Image Decoder 301 Entropy Decoder 302 Parameter Decoding Unit 303 Interpretation parameter derivation section 304 Intra Prediction Parameter Derivation Unit 305, 107 Loop Filter 306, 109 Reference picture memory 307, 108 Prediction parameter memory 308, 101 Predictive Image Generation Unit 309 Interpretation Image Generation Unit 310 Intra Predictive Image Generation Unit 311, 105 Inverse Quantization / Inverse Transformation Section 312, 106 Addition section 320 Prediction parameter derivation section 10. Video Encoding Device 11 Image encoding device 102 Subtraction Unit 103 Conversion / Quantization Section 104 Entropy coding unit 110 Encoding parameter determination unit 111 Parameter coding section 112 Interpretation Parameter Coding Unit 113 Intra Prediction Parameter Coding Unit 120 Prediction parameter derivation section 71 Inverse Transformation Information Creation Device 81 Inverse Transform Information Encoding Device 91 Inverse Transformation Information Decoder 611 NN filter section

Claims

1. In a video decoding device that decodes encoded data and outputs a decoded image, A resolution inverse conversion unit that performs post-filtering using a neural network identified by resolution conversion information, The system comprises a neural network filter unit that derives a variable relating to the size of the luminance samples in the decoded image, a variable relating to the size of the luminance samples in the image to which post-filter processing has been applied, a variable indicating the input patch size for post-filter processing, and a variable indicating the output patch size derived using the variable indicating the input patch size. A video decoding device characterized in that the product of a variable relating to the size of the luminance samples in the decoded image and a variable indicating the output patch size is equal to the product of a variable relating to the size of the luminance samples in the image to which the post-filtering process has been applied and a variable indicating the input patch size.

2. In a video encoding device that encodes images and generates encoded data, A resolution inverse conversion unit that performs post-filtering using a neural network identified by resolution conversion information, The system comprises a neural network filter unit that derives a variable relating to the size of luminance samples in an encoded image, a variable relating to the size of luminance samples in an image to which post-filter processing has been applied, a variable indicating the input patch size for post-filter processing, and a variable indicating the output patch size derived using the variable indicating the input patch size. A video encoding device characterized in that the product of a variable relating to the size of the luminance samples in the encoded image and a variable indicating the output patch size is equal to the product of a variable relating to the size of the luminance samples in the image to which the post-filtering process has been applied and a variable indicating the input patch size.

3. A process that performs post-filtering using a neural network identified by resolution conversion information, The process involves deriving at least a variable relating to the size of the luminance samples in the decoded image, a variable relating to the size of the luminance samples in the image to which post-filtering has been applied, a variable indicating the input patch size for post-filtering, and a variable indicating the output patch size derived using the variable indicating the input patch size. An integrated circuit characterized in that the product of a variable relating to the size of the luminance samples in the decoded image and a variable indicating the output patch size is equal to the product of a variable relating to the size of the luminance samples in the image to which the post-filtering process has been applied and a variable indicating the input patch size.

Images