Video decoding device, video encoding device, and recording medium

Abstract

To solve the problem of video encoding and decoding methods in which image quality which is deteriorated due to encoding distortion at low transmission rate can be improved by post-filtering processing, however values of elements in a cross correlation matrix defined as post-filter hint information are too small, resulting in reduction in accuracy.SOLUTION: A video decoding apparatus includes: an image decoding apparatus which decodes encoded data to generate a decoded image; a post-filter processing apparatus which performs post-filter processing on the decoded image; and a supplemental enhancement information decoding apparatus which decodes supplemental enhancement information for use in the post-filter processing apparatus. The supplemental enhancement information decoding apparatus decodes cross correlation matrix information of an original image and a decoded image with fixed bit accuracy which does not depend on a pixel bit length of the decoded image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a video encoding device and a video decoding device. [Background technology]

[0002] In order to efficiently transmit or record moving images, a moving image encoding device is used that generates encoded data by encoding the moving images, and a moving image decoding device is used that generates a decoded image by decoding the encoded data.

[0003] Specific examples of video encoding methods include H.264 / AVC and H.265 / HEVC (High-Efficiency Video Coding).

[0004] In such a video coding method, images (pictures) constituting a video are managed in a hierarchical structure consisting of slices obtained by dividing images, coding tree units (CTUs) obtained by dividing slices, coding units (sometimes called coding units: CUs) obtained by dividing coding tree units, and transform units (TUs) obtained by dividing coding units, and are coded / decoded for each CU.

[0005] In such a video coding method, a predicted image is usually generated based on a locally decoded image obtained by encoding / decoding an input image, and a prediction error (sometimes called a "difference image" or "residual image") obtained by subtracting the predicted image from the input image (original image) is coded. Methods for generating a predicted image include inter-prediction and intra-prediction.

[0006] Non-Patent Document 1 is an example of a recent video encoding and decoding technology. Non-Patent Document 1 specifies a supplemental enhancement information (SEI) message for transmitting image properties, display method, timing, and the like simultaneously with encoded data. It also discloses an SEI that transmits the topology and parameters of a neural network filter used as a post-filter in units that allow random access. It also discloses an SEI that transmits whether or not to perform post-filter processing on a picture-by-picture basis.

[0007] Non-Patent Document 1 describes, as an improvement method, a method of defining bit lengths separately for luminance and chrominance when the input tensor of a neural network is an integer value.

[0008] Non-Patent Document 1 also discloses SEI that transmits hint information for post-filter processing. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Improvements under consideration for neural network post filter SEI messages, JVET-AC2032, 2023-02-01. Summary of the Invention [Problem to be solved by the invention]

[0010] The method disclosed in Non-Patent Document 1 has a problem in that when a cross-correlation matrix is ​​defined as post-filter hint information, the values ​​of the elements of the matrix become too small, causing a problem in accuracy.

[0011] Furthermore, in the method disclosed in the non-patent document, when performing chrominance upsampling processing, if the input is in 4:2:2 format, the chrominance upsampling will always be in 4:4:4 format, which poses the problem of redundant flag information. [Means for solving the problem]

[0012] A video decoding device according to one embodiment of the present invention comprises an image decoding device that decodes encoded data to generate a decoded image, a post-filter processing device that performs post-filter processing on the decoded image, and an auxiliary extension information decoding device that decodes auxiliary extension information used in the post-filter processing device, wherein the auxiliary extension information decoding device decodes cross-correlation matrix information between a decoded image and an original image with a constant bit precision regardless of the pixel bit length of the decoded image.

[0013] In addition, a moving image coding device according to one embodiment of the present invention has an image coding device that codes an input image, an auxiliary extension information generating device that generates auxiliary extension information for post-filter processing, and an auxiliary extension coding device that codes the auxiliary extension information, wherein the auxiliary extension information generating device generates cross-correlation matrix information between a locally decoded image and an original image with a constant bit precision regardless of the pixel bit length of the decoded image, and is characterized in that it has an auxiliary extension information coding device that codes the cross-correlation information.

[0014] A video decoding device according to one embodiment of the present invention includes an image decoding device that decodes encoded data to generate a decoded image, a postfilter processing device that performs postfilter processing on the decoded image, and an auxiliary extension information decoding device that decodes auxiliary extension information used in the postfilter processing device, wherein when the postfilter processing device performs upsampling processing of chrominance pixels of a 4:2:2 format decoded image, the auxiliary extension information decoding device does not decode flag information for the output format.

[0015] A moving image coding device according to one embodiment of the present invention comprises an image coding device for coding an input image, an auxiliary extension information generating device for generating auxiliary extension information for post-filter processing, and an auxiliary extension coding device for coding the auxiliary extension information, wherein the auxiliary extension generating device generates auxiliary extension information for upsampling processing of chrominance pixels of a 4:2:2 format decoded image, and the auxiliary extension information coding device does not code flag information for the output format. Effect of the Invention

[0016] By adopting such a configuration, it is possible to solve the problem that in the method disclosed in the non-patent document, when a cross-correlation matrix is ​​defined as post-filter hint information, the values ​​of each element of the matrix become too small, resulting in accuracy problems.

[0017] In addition, in the method disclosed in the non-patent document, when performing chrominance upsampling processing, if the input is in 4:2:2 format, the chrominance upsampling will always be in 4:4:4 format, which solves the problem of redundant flag information. [Brief description of the drawings]

[0018] [Figure 1] 1 is a schematic diagram showing a configuration of a video transmission system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a hierarchical structure of encoded data. [Diagram 3] FIG. 13 is a diagram showing a hierarchical structure of encoded data in units of PUs. [Figure 4] FIG. 1 is a schematic diagram showing a configuration of an image decoding device. [Diagram 5] 11 is a flowchart illustrating a schematic operation of an image decoding device. [Figure 6] FIG. 1 is a block diagram showing a configuration of an image encoding device. [Figure 7] FIG. 13 is a diagram showing the syntax of the neural network post-filter characteristic (NNPFC) SEI. [Figure 8]A diagram showing the syntax of Neural Network Post Filter Activation (NNPFA) SEI. [Figure 9] A diagram showing the syntax of a postfilter hint SEI. [Figure 10] FIG. 1 is a diagram showing an equation for determining a cross-correlation matrix in a non-patent document. [Figure 11] FIG. 4 is a diagram showing an equation for determining a cross-correlation matrix in the present embodiment. [Figure 12] FIG. 11 is a diagram showing an equation for calculating another cross-correlation matrix in the present embodiment. [Figure 13] A diagram showing the syntax of an SEI payload, which is a container for an SEI message. [Figure 14] FIG. 13 is a flowchart showing the processing of the post-filter processor 61. [Figure 15] FIG. 1 is a diagram showing an NNC encoding device and a decoding device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0020] FIG. 1 is a schematic diagram showing the configuration of another moving image transmission system according to the present embodiment.

[0021] The video transmission system 1 is a system that transmits coded data obtained by coding an image, and decodes and displays the coded data. The video transmission system 1 is made up of a video coding device 10, a network 21, a video decoding device 30, and an image display device 41.

[0022] The video encoding device 10 is composed of an image encoding device (image encoding unit) 11, an auxiliary extension information creation device (auxiliary extension information creation unit) 71, an auxiliary extension information encoding device (auxiliary extension information encoding unit) 81, and a pre-filter processing device (pre-filter processing unit) 1001.

[0023] The video encoding device 10 creates a prefiltered image T2 from an input video T1 using a prefilter processing device 51, compresses and encodes the image using an image encoding device 11, and analyzes the input video T1 and a locally decoded image T3 from the image encoding device 11 to generate auxiliary extension information to be input to a postfilter processing device 61 using an auxiliary extension information creation device 51, encodes the information using an auxiliary extension upward encoding device 81, generates encoded data Te, and transmits it to a network 21.

[0024] The video decoding device 30 includes an image decoding device (image decoding unit) 31, an auxiliary extension information decoding device (auxiliary extension information decoding unit) 91, and a post-filter processing device (host filter processing unit) 61.

[0025] The video decoding device 30 decodes the encoded data Te received from the network 21 using the image decoding device 31 and the auxiliary extension information decoding device 91, and performs post-filter processing on the decoded image Td1 using the auxiliary extension information in the post-filter processing device 61, and outputs the post-filter decoded image Td2 to the image display device 41.

[0026] The postfilter processor 61 may output the decoded image Td1 as is without performing postfiltering on the auxiliary enhancement information.

[0027] The image display device 41 displays all or a part of the post-filter image Td2 output from the post-filter processing device 1002. The image display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. Examples of the display form include a stationary display, a mobile display, and an HMD. When the image decoding device 31 has high processing capability, it displays a high-quality image, and when it has only low processing capability, it displays an image that does not require high processing capability or display capability.

[0028] The network 21 transmits the encoded auxiliary extension information and the encoded data Te to the image decoding device 31. A part or all of the encoded auxiliary extension information may be included in the encoded data Te as auxiliary extension information SEI. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be replaced by a storage medium on which the encoded data Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blue-ray Disc: registered trademark).

[0029] As an example of a specific embodiment, in pre-filtering, the input image is reduced, and as auxiliary enhancement information, in post-filtering, auxiliary enhancement information for neural network processing to enlarge the decoded image by super-resolution processing based on a neural network may be encoded and decoded.

[0030] As another example of a specific embodiment, in the pre-filtering process, no special processing is performed, and in the post-filtering process, auxiliary extension information is encoded and decoded for neural network processing to restore the decoded image to the input video image by image restoration processing based on a neural network.

[0031] In such a configuration, a framework is provided that enables the auxiliary extension information to be efficiently encoded and decoded.

[0032] <operator> The operators used in this specification are listed below.

[0033] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is the OR assignment operator, and || indicates logical OR.

[0034] x? y : z is a ternary operator that takes y when x is true (non-zero) and z when x is false (0).

[0035] Clip3(a, b, c) is a function that clips c to a value between a and b, returning a if c < a, b if c > b, and c otherwise (where a <= b).

[0036] abs(a) is a function that returns the absolute value of a.

[0037] Int(a) is a function that returns the integer value of a.

[0038] Floor(a) is a function that returns the largest integer less than or equal to a.

[0039] ceil(a) is a function that returns the smallest integer greater than or equal to a.

[0040] a / d represents the division of a by d (truncating the decimal part).

[0041] a÷d and a over d represent the division of a by d (without rounding).

[0042] (Structure of the encoded data Te) Prior to the detailed description of the image encoding device 11 and the image decoding device 31 according to the present embodiment, the data structure of the encoded data Te generated by the image encoding device 11 and decoded by the image decoding device 31 will be described with reference to FIGS. 2 and 3.

[0043] The coded data Te is a bitstream consisting of multiple CVS (Coded Video Sequence) and EoB (End of Bitstream) NAL units as shown in FIG. 2. The CVS consists of multiple AU (Access Unit) and EoS (End of Sequence) NAL units. The AU at the beginning of the CVS is called the CVSS (Coded Video Sequence Start) AU. The unit obtained by dividing the CVS into layers is called the CLVS (Coded Layer Video Sequence). The AU consists of one or more layered PUs (Picture Units) with the same output time. If the multilayer coding method is not adopted, the AU consists of one PU. The PU is a unit of coded data for one decoded picture consisting of multiple NAL units. The CLVS consists of PUs of the same layer, and the PU at the beginning of the CLVS is called the CLVSS (Coded Layer Video Sequence Start) PU. The CLVSS PU is limited to PUs consisting of randomly accessible IRAP (Intra Random Access Pictures) and GDR (Gradual Decoder Refresh Picture). A NAL unit consists of a NAL unit header and RBSP (Raw Byte Sequence Payload) data. The NAL unit header consists of 2 bits of 0 data, followed by a 6-bit nuh_layer_id that indicates the layer value, a 5-bit nuh_unit_type that indicates the NAL unit type, and a 3-bit nuh_temporal_id_plus1 that is the Temporal ID value plus 1.

[0044] Fig. 3 is a diagram showing a hierarchical structure of data in the coded data Te in units of PU. The coded data Te illustratively includes a sequence and a plurality of pictures constituting the sequence. Fig. 3 shows a diagram showing a coded video sequence that defines the sequence SEQ, a coded picture that defines the picture PICT, a coded slice that defines the slice S, coded slice data that defines the slice data, a coding tree unit included in the coded slice data, and a coding unit included in the coding tree unit.

[0045] In the coded video sequence, a set of data to be referred to by the image decoding device 31 in order to decode the sequence SEQ to be processed is defined. As shown in Fig. 3, the sequence SEQ includes a video parameter set VPS (Video Parameter Set), a sequence parameter set SPS (Sequence Parameter Set), a picture parameter set PPS (Picture Parameter Set), an adaptation parameter set (APS), a picture PICT, and supplemental enhancement information SEI (Supplemental Enhancement Information).

[0046] The video parameter set VPS specifies a set of coding parameters common to multiple videos composed of multiple layers, as well as a set of coding parameters related to multiple layers and each individual layer included in the video.

[0047] The sequence parameter set SPS specifies a set of coding parameters that the image decoding device 31 refers to in order to decode the target sequence. For example, the width and height of a picture are specified. Note that there may be multiple SPSs. In that case, one of the multiple SPSs is selected from the PPS.

[0048] Here, the sequence parameter set SPS includes the following syntax elements:

[0049] pic_width_max_in_luma_samples: A syntax element that specifies the width of the image with the largest width among the images in a single sequence, in units of luminance blocks. The value of this syntax element must not be 0 and must be an integer multiple of Max(8, MinCbSizeY), where MinCbSizeY is a value determined by the minimum size of a luminance block.

[0050] pic_height_max_in_luma_samples: This syntax element specifies the height of the image with the maximum height among the images in a single sequence, in units of luminance blocks. The value of this syntax element is required to be a non-zero integer multiple of Max(8, MinCbSizeY).

[0051] The picture parameter set PPS defines a set of coding parameters that the image decoding device 31 refers to in order to decode each picture in the target sequence. Note that there may be multiple PPSs. In that case, one of the multiple PPSs is selected for each picture in the target sequence.

[0052] Here, the picture parameter set PPS includes the following syntax elements:

[0053] pps_pic_width_in_luma_samples: This syntax element specifies the width of the target picture. The value of this syntax element must be a non-zero integer multiple of Max(8, MinCbSizeY) and equal to or less than sps_pic_width_max_in_luma_samples. InpPicWidthInLumaSamples, described below, is set equal to pps_pic_width_in_luma_samples.

[0054] pps_pic_height_in_luma_samples: This syntax element specifies the height of the target picture. The value of this syntax element must be a non-zero integer multiple of Max(8, MinCbSizeY) and equal to or less than sps_pic_height_max_in_luma_samples. InpPicHeightInLumaSamples, described below, is set equal to pps_pic_height_in_luma_samples.

[0055] pps_conformance_window_flag: a flag indicating whether conformance (cropping) window offset parameters are to be signaled subsequently, and where the conformance window is to be displayed. If this flag is 1, the parameters are to be signaled, if it is 0, the conformance window offset parameters are not present.

[0056] sps_chroma_format_id is a parameter that indicates the chrominance format. ChromaFormatIdc, described below, is set equal to sps_chroma_format_id.

[0057] · pps_init_qp_minus26 is information for deriving the quantization parameter SliceQpY of the slice referenced by the PPS.

[0058] (Encoded Picture) A coded picture defines a set of data to be referenced by the image decoding device 31 in order to decode a picture PICT to be processed. As shown in Fig. 3, the picture PICT includes a picture header PH and slices 0 to NS-1 (NS is the total number of slices included in the picture PICT).

[0059] (Coded Slice) A coded slice defines a set of data to be referenced by the image decoding device 31 in order to decode a target slice S. As shown in Fig. 3, a slice includes a slice header and slice data.

[0060] The slice header includes a group of coding parameters to be referred to by the image decoding device 31 in order to determine a decoding method for the current slice. Slice type designation information (slice_type) that designates the slice type is an example of a coding parameter included in the slice header.

[0061] Slice types that can be specified by the slice type specification information include (1) an I slice that uses only intra prediction when encoding, (2) a P slice that uses uni-prediction (L0 prediction) or intra prediction when encoding, and (3) a B slice that uses uni-prediction (L0 prediction or L1 prediction), bi-prediction, or intra prediction when encoding. Note that inter prediction is not limited to uni-prediction or bi-prediction, and a predicted image may be generated using more reference pictures. Hereinafter, when referring to P or B slice, it refers to a slice including a block that can use inter prediction.

[0062] In addition, the slice header may include a reference to a picture parameter set PPS (pic_parameter_set_id).

[0063] (Encoded slice data) The coded slice data specifies a set of data to be referenced by the image decoding device 31 in order to decode the slice data to be processed. The slice data includes a CTU, as shown in the coded slice header in Fig. 3. A CTU is a block of a fixed size (e.g., 64x64) that constitutes a slice, and is also called a Largest Coding Unit (LCU).

[0064] (coding tree unit) 3 specifies a set of data that the image decoding device 31 refers to in order to decode a CTU to be processed. The CTU is divided into coding units CU, which are basic units of encoding processing, by recursive quad tree division (QT (Quad Tree) division), binary tree division (BT (Binary Tree) division), or ternary tree division (TT (Ternary Tree) division). A node of a tree structure obtained by recursive quad tree division is called a coding node. Intermediate nodes of the quad tree, binary tree, and ternary tree are coding nodes, and the CTU itself is specified as the top coding node.

[0065] (Encoding Unit) 3 specifies a set of data to be referenced by the image decoding device 31 in order to decode a coding unit to be processed. Specifically, a CU is composed of a CU header CUH, prediction parameters, transformation parameters, quantization transformation coefficients, etc. The CU header specifies a prediction mode, etc.

[0066] The prediction process may be performed in units of CUs, or in units of sub-CUs obtained by further dividing a CU.

[0067] There are two types of prediction (prediction modes): intra prediction and inter prediction. Intra prediction is a prediction within the same picture, while inter prediction refers to a prediction process performed between different pictures (for example, between display times or between layer images).

[0068] The transform and quantization processes are performed in units of CUs, but the quantized transform coefficients may be entropy coded in units of sub-blocks such as 4x4.

[0069] In this specification, when "a flag indicating whether or not XX" is used, a flag other than 0 (for example, 1) indicates XX and 0 indicates non-XX, and in logical negation, logical product, etc., 1 is treated as true and 0 as false (same below). However, in an actual device or method, other values ​​may be used as true and false values.

[0070] (Configuration of an image decoding device) The configuration of an image decoding device 31 (FIG. 4) according to this embodiment will be described.

[0071] The image decoding device 31 includes an entropy decoding unit 301, a parameter decoding unit (prediction image decoding device) 302, a loop filter 305, a reference picture memory 306, a prediction parameter memory 307, a prediction image generating unit (prediction image generating device) 308, an inverse quantization and inverse transform unit 311, an adder 312, and a prediction parameter derivation unit 320. Note that, in accordance with the image encoding device 11 described below, the image decoding device 31 may also be configured not to include the loop filter 305.

[0072] The parameter decoding unit 302 further includes a header decoding unit 3020, a CT information decoding unit 3021, and a CU decoding unit 3022 (prediction mode decoding unit), and the CU decoding unit 3022 further includes a TU decoding unit 3024. These may be collectively referred to as a decoding module. The header decoding unit 3020 decodes parameter set information such as VPS, SPS, PPS, and APS, and slice header (slice information) from the encoded data. The CT information decoding unit 3021 decodes the CT from the encoded data. The CU decoding unit 3022 decodes the CU from the encoded data. The TU decoding unit 3024 decodes QP update information (quantization correction value) and quantization prediction error (residual_coding) from the encoded data.

[0073] The predicted image generating unit 308 includes an inter predicted image generating unit 309 and an intra predicted image generating unit 310 .

[0074] The entropy decoding unit 301 performs entropy decoding on the encoded data Te input from the outside, and decodes each code (syntax element).

[0075] The entropy decoding unit 301 outputs the decoded code to the parameter decoding unit 302. Control of which code to decode is performed based on an instruction from the parameter decoding unit 302.

[0076] (Basic flow) FIG. 5 is a flowchart illustrating a schematic operation of the image decoding device 31.

[0077] (S1100: Decode Parameter Set Information) The header decoding unit 3020 decodes parameter set information such as the VPS, SPS, and PPS from the encoded data.

[0078] (S1200: Decode slice information) The header decoding unit 3020 decodes the slice header (slice information) from the encoded data.

[0079] Thereafter, the image decoding device 31 repeats the processes from S1300 to S5000 for each CTU included in the target picture to derive a decoded image of each CTU.

[0080] (S1300: Decode CTU Information) The CT information decoding unit 3021 decodes the CTU from the encoded data.

[0081] (S1400: Decode CT Information) The CT information decoding unit 3021 decodes the CT from the encoded data.

[0082] (S1500: CU Decoding) The CU decoding unit 3022 performs S1510 and S1520 to decode the CU from the encoded data.

[0083] (S1510: Decode CU information) The CU decoding unit 3022 decodes the CU information, prediction information, and the like from the encoded data.

[0084] (S1520: Decode TU information) The TU decoding unit 3024 decodes the QP update information and the quantization prediction error, etc. from the encoded data. Note that the QP update information is a difference value from the quantization parameter predicted value qPpred, which is a predicted value of the quantization parameter QP.

[0085] (S2000: Generation of predicted image) The predicted image generation unit 308 generates a predicted image for each block included in the current CU based on the prediction information.

[0086] (S3000: Inverse Quantization and Inverse Transformation) The inverse quantization and inverse transform unit 311 executes inverse quantization and inverse transform processing for each TU included in the target CU.

[0087] (S4000: Generate decoded image) The adder 312 adds the predicted image supplied from the predicted image generation unit 308 and the prediction error supplied from the inverse quantization and inverse transform unit 311 to generate a decoded image of the current CU.

[0088] (S5000: Loop Filter) The loop filter 305 applies a loop filter such as a deblocking filter, SAO, or ALF to the decoded image to generate a decoded image.

[0089] (Neural network post-filter characteristics (NNPFC) SEI Figure 7 shows an outline of the syntax of the neural network post-filter characteristic (NNPFC) SEI message of the non-patent document. The auxiliary enhancement information decoding device 91 decodes the syntax elements (e.g., nnpfc_auxiliary_inp_idc, nnpfc_inp_order_idc, nnpfc_inp_format_idc, etc.) shown in Figure 7. The NNPFC SEI message specifies the neural network to be applied as a post-filter process. The application of a specific post-filter process to a specific picture is indicated by a neural network post-filter activation SEI message (Figure 8).

[0090] To apply this SEI message, the following variables must be defined:

[0091] The width InpPicWidthInLumaSamples and height InpPicHeightInLumaSamples in luminance pixels of the picture decoded by the image decoding device 31.

[0092] The picture luma pixel array CroppedYPic[idx] and chrominance pixel arrays CroppedCbPic[idx] and CroppedCrPic[idx], with idx ranging from 0 to numInputPics-1, that are used as input for postfiltering.

[0093] · Pixel bit depth BitDepthY of the luminance pixel array.

[0094] BitDepthC: the pixel bit length of the chrominance pixel array.

[0095] The variables SubWidthC and SubHeightC indicate the chrominance format indicated by ChromaFormatIdc of the picture. When the format is 4:2:0, the variables SubWidthC and SubHeightC are both 2, when the format is 4:2:2, the variables SubWidthC and SubHeightC are both 2 and 1, when the format is 4:4:4, the variables SubWidthC and SubHeightC are both 1.

[0096] nnpfc_auxiliary_inp_idc indicates that auxiliary input data is present in the input tensor of the neural network post filter. If the value is equal to 1, the strength control value StrengthControlVal (strength control information), which is a real number ranging from 0 to 1, is input as auxiliary input data. Note that the real number may be a decimal point number or a floating point number. The strength control value may be a value obtained in the process of generating a decoded image, such as a quantization parameter.

[0097] nnpfc_purpose indicates the purpose of the process. If the value of nnpfc_purpose is 0, it may be used as determined by the application.

[0098] If nnnpfc_purpose > 0 and the value of (nnpfc_purpose & 0x01) is 0, it indicates that the purpose is not to improve general visual quality. If the value of (nnpfc_purpose & 0x01) is not 0, it indicates that the purpose is to improve general visual quality.

[0099] If nnnpfc_purpose > 0 and the value of (nnpfc_purpose & 0x02) is 0, it indicates that there is no chrominance upsampling (from 4:2:0 format to 4:2:2 or 4:4:4 format, or from 4:2:2 format to 4:4:4 format). If the value of (nnpfc_purpose & 0x02) is not 0, it indicates that there is chrominance upsampling.

[0100] If nnnpfc_purpose > 0 and the value of (nnpfc_purpose & 0x04) is 0, it indicates that there is no resolution upsampling (increasing the picture width or height). If the value of (nnpfc_purpose & 0x04) is not 0, it indicates that there is resolution upsampling.

[0101] If nnnpfc_purpose > 0 and the value of (nnpfc_purpose & 0x08) is 0, it indicates that no picture rate upsampling processing is performed. If the value of (nnpfc_purpose & 0x08) is not 0, it indicates that picture rate upsampling processing is performed.

[0102] If nnnpfc_purpose > 0 and the value of (nnpfc_purpose & 0x10) is 0, it indicates that there is no pixel bit length expansion (increase in the pixel bit length of the luminance or chrominance). If the value of (nnpfc_purpose & 0x10) is not 0, it indicates that there is pixel bit length expansion.

[0103] If nnnpfc_purpose > 0 and the value of (nnpfc_purpose & 0x20) is 0, it indicates that colorization (conversion from 4:0:0 format to 4:2:0, 4:2:2, or 4:4:4 format) is not performed. If the value of (nnpfc_purpose & 0x20) is not 0, it indicates that colorization is performed.

[0104] Note that if ChromaFormatIdc is 3, the value of (nnpfc_purpose & 0x02) must be 0. Also, if the value of ChromaFormatIdc or (nnpfc_purpose & 0x02) is not 0, (nnpfc_purpose & 0x20) must be 0.

[0105] nnpfc_id indicates an identification number that can be used to identify the post-filter process.

[0106] If the NNPFC SEI message is the first NNPFC SEI message, in decoding order, with a particular nnpfc_id value within the current CLVS, the following applies:

[0107] Indicates that this SEI message is a basic post-filter process.

[0108] · This SEI message pertains to the current decoded picture and all subsequent decoded pictures of the current layer until the end of the current CLVS. If an NNPFC SEI message is a repetition of a previous NNPFC SEI message in the current CLVS in decoding order, the semantics of the succeeding messages apply as if this SEI message was the only NNPFC SEI message with the same content in the current CLVS.

[0109] If the NNPFC SEI message is not the first NNPFC SEI message, in decoding order, with a particular nnpfc_id value within the current CLVS, the following applies:

[0110] Indicates that this SEI message is an update relative to the previous base post-filter in decoding order that uses the same nnpfc_id value.

[0111] This SEI message concerns the current decoded picture and all subsequent decoded pictures of the current layer until the end of the current CLVS or until the next NNPFC SEI message with the particular nnpfc_id value in the current layer.

[0112] nnpfc_mode_idc equals 0 to indicate that this SEI message contains an ISO / IEC 15938-17 bit stream that specifies post-filter processing, or that this SEI message is an update related to a base post-processing filter with the same nnpfc_id value.

[0113] When nnpfc_mode_idc is 1, it indicates that the post-filtering associated with the nnpfc_id value is the neural network identified by the URI indicated by nnpfc_uri.

[0114] nnpfc_reserved_zero_bit_a indicates 0.

[0115] nnpfc_tag_uri contains a tag URI with syntax and semantics specified in IETF RFC4151. It is used to store format and related information about the neural network used as the base postfilter, or for updates related to postfilters with the same nnpfc_id value. Note that nnpfc_tag_uri allows the format of the neural network data specified by nnpfc_uri to be uniquely identified without the need for a registration authority. If nnpfc_tag_uri is "tag:iso.org,2023:15938-17", it indicates that the neural network data identified by nnpfc_uri is ISO / IEC 15938-17 compliant and encoded with NNC (Neural Network Coding).

[0116] The nnpfc_uri contains a URI with syntax and semantics specified in IETF Internet Standard 66 to identify the neural network to be used as a postfilter or update associated with a postfilter with the same nnpfc_id value.

[0117] A value of 1 for nnpfc_property_present_flag indicates the presence of syntax elements related to the filter's purpose, input format, output format, and complexity. A value of 0 for nnpfc_property_present_flag indicates the absence of syntax elements related to the filter's purpose, input format, output format, and complexity. The value of nnpfc_property_present_flag shall be 1 if this SEI message is the first NNPFC SEI message, in decoding order, with a particular nnpfc_id value in the current CLVS.

[0118] nnpfc_out_sub_c_flag indicates the values ​​of the variables outSubWidthC and outSubHeightC if the value of (nnpfc_purpose & 0x02) is not 0. If the value of nnpfc_out_sub_c_flag is 1, then the value of outSubWidthC is 1 and the value of outSubHeightC is 1, indicating 4:4:4 format. If the value of nnpfc_out_sub_c_flag is 0, then the value of outSubWidthC is 2 and the value of outSubHeightC is 1, indicating 4:2:2 format. If the value of ChromaFormatIdc is 2 and nnpfc_out_sub_c_flag is present, then the value of nnpfc_out_sub_c_flag must be 1.

[0119] In Non-Patent Document 1, when the input is in 4:2:2 format, the chrominance upsampling always becomes 4:4:4 format, which causes a problem that the flag information becomes redundant. Specifically, when the value of ChromaFormatIdc is 2 and the value of (nnpfc_purpose & 0x02) is not 0, the value of nnpfc_out_sub_c_flag must always be 1, which causes a problem that the bits of nnpfc_out_sub_c_flag are redundant.

[0120] Therefore, in this embodiment, when the value of (nnpfc_purpose & 0x02) is not 0 as a syntax and when the value of ChromaFormatIdc is 1, nnpfc_out_sub_c_flag indicates the values ​​of the variables outSubWidthC and outSubHeightC.

[0121] By using such a syntax, when the value of ChromaFormatIdc is 2, redundant flag information is not sent, making it possible to reduce redundancy.

[0122] Alternatively, nnpfc_out_sub_c_flag may indicate, as a syntax, the values ​​of the variables outSubWidthC and outSubHeightC when the value of (nnpfc_purpose & 0x02) is not 0 and when the value of SubHeightC is 2.

[0123] As an alternative method, as semantics, if the value of ChromaFormatIdc is 2 and the value of (nnpfc_purpose & 0x02) is not 0, then as syntax, nnpfc_out_sub_c_flag is not sent, the value of outSubWidthC is 1, and the value of outSubHeightC is 1, i.e., the format is 4:4:4.

[0124] According to this embodiment, when the value of ChromaFormatIdc is 2, redundant flag information is not sent, and redundancy can be reduced.

[0125] nnpfc_out_colour_format_idc specifies the output colour format if the value of nnpfc_purpose & 0x20 is not 0, and therefore specifies the values ​​of the variables outSubWidthC and outSubHeightC.

[0126] A value of 1 for nnpfc_out_colour_format_idc indicates that the NNPF output colour format is 4:2:0 and that the values ​​of outSubWidthC and outSubHeightC are both 2.

[0127] A value of 2 for nnpfc_out_colour_format_idc indicates that the NNPF output colour format is 4:2:2, outSubWidthC has a value of 2, and outSubHeightC has a value of 1.

[0128] A value of 3 for nnpfc_out_colour_format_idc indicates that the NNPF output colour format is 4:4:4 and that the values ​​of outSubWidthC and outSubHeightC are both 1.

[0129] The value of nnpfc_out_colour_format_idc shall not be 0.

[0130] If the values ​​of nnpfc_purpose & 0x02 and nnpfc_purpose & 0x20 are both 0, the values ​​of outSubWidthC and outSubHeightC must be equal to SubWidthC and SubHeightC, respectively.

[0131] nnpfc_inp_order_idc is the type of input tensor that indicates how the pixel array of the decoded image is ordered as input to the postfilter process. The semantics of nnpfc_inp_order_idc, in the range 0 to 3, inclusive, specifies the process of deriving the input tensor inputTensor for each value of nnpfc_inp_order_idc.

[0132] If nnpfc_inp_order_idc has a value of 0, then if nnpfc_auxiliary_inp_idc has a value of 0, then there is one luma matrix in the input tensor for each input image, and the number of channels is 1. Otherwise (nnpfc_auxiliary_inp_idc has a value of 1), there is one luma matrix and one auxiliary input matrix, and the number of channels is 2.

[0133] If nnpfc_inp_order_idc has a value of 1, if nnpfc_auxiliary_inp_idc has a value of 0, there are two chrominance matrices in the input tensor and the number of channels is 2. Otherwise (nnpfc_auxiliary_inp_idc has a value of 1), there are two chrominance matrices and one auxiliary input matrix, and the number of channels is 3.

[0134] If nnpfc_inp_order_idc has a value of 2, if nnpfc_auxiliary_inp_idc has a value of 0, there is one luma matrix and two chroma matrices in the input tensor and the number of channels is 3. Otherwise (nnpfc_auxiliary_inp_idc has a value of 1), there is one luma matrix, two chroma matrices and one auxiliary input matrix and the number of channels is 4.

[0135] If the value of nnpfc_inp_order_idc is 3, if the value of nnpfc_auxiliary_inp_idc is 0, there are 4 luma matrices and 2 chroma matrices in the input tensor and the number of channels is 6. Otherwise (nnpfc_auxiliary_inp_idc is 1), there are 4 luma matrices, 2 chroma matrices and 1 auxiliary input matrix and the number of channels is 7. This nnpfc_inp_order_idc can only be used if the input format is 4:2:0.

[0136] A value of nnpfc_auxiliary_inp_idc greater than 0 indicates that auxiliary input data is present in the input tensor. A value of 0 for nnpfc_auxiliary_inp_idc indicates that auxiliary input data is not present in the input tensor. If the value of nnpfc_auxiliary_inp_idc is 1, the variable strengthControlScaledVal is derived as follows:

[0137] if (nnpfc_inp_format_idc == 1) strengthControlScaledVal = Floor(StrengthControlVal * ((1 << inpTensorBitDepthY)-1)) else strengthControlScaledVal = StrengthControlVal In Non-Patent Document 1, when pixel values ​​input to the neural network for post-filter processing are unsigned integers, the bit length of the luminance pixel value inpTensorBitDepthY and the bit length of the chrominance pixel value inpTensorBitDepthC are defined independently. Therefore, when the value of inputTensor is an integer value, there is a problem in that the definition of the variable strengthControlScaledVal is unclear.

[0138] Therefore, the auxiliary enhancement information decoding device 91 may change whether to use the luma bit depth or the chroma bit depth for deriving StrengthControlVal according to the input tensor format nnpfc_inp_order_idc. Specifically, if the condition (nnpfc_inp_order_idc==0 || nnpfc_inp_order_idc==2 || nnpfc_inp_order_idc==3) is true, the luma pixel bit length may be used for derivation (luminance-priority configuration). Otherwise, the chroma bit depth may be used for deriving StrengthControlVal.

[0139] if (nnpfc_inp_format_idc==1) { if (nnpfc_inp_order_idc==0 || nnpfc_inp_order_idc==2 || nnpfc_inp_order_idc==3) strengthControlScaledVal=Floor(StrengthControlVal*((1 << inpTensorBitDepthY)-1)) else if (nnpfc_inp_order_idc == 1) strengthControlScaledVal=Floor(StrengthControlVal*((1 << inpTensorBitDepthC)-1)) } Alternatively, if the condition (nnpfc_inp_order_idc==1) is true, the chrominance pixel bit length may be used for the derivation (chrominance-first configuration), otherwise the luma bit depth may be used for the derivation of StrengthControlVal.

[0140] if (nnpfc_inp_format_idc==1) { if (nnpfc_inp_order_idc==1) strengthControlScaledVal=Floor(StrengthControlVal*((1 << inpTensorBitDepthC)-1)) else strengthControlScaledVal=Floor(StrengthControlVal*((1 << inpTensorBitDepthY)-1)) } Note that the variable strengthControlScaledVal is input as one channel of the input tensor as an input of the auxiliary input matrix.

[0141] (Neural Network Post Filter Activation (NNPFA) SEI) 8 shows the syntax of the Neural Network Postfilter Activation (NNPFA) SEI message. The Neural Network Postfilter Activation NNPFA SEI message activates or deactivates the application of a target neural network postfiltering identified by nnpfa_target_id for postfiltering of a sequence of pictures.

[0142] nnpfa_target_id indicates the target picture neural network post-filtering. It identifies one or more NNPFC SEI messages with nnpfc_id equal to nnpfa_target_id for the current picture.

[0143] An NNPFA SEI message with a particular value of nnpfa_target_id should not be present on the current PU unless one or both of the following conditions are true:

[0144] ·There is an NNPFC SEI message in the current CLVS with nnpfc_id equal to a specific value of nnpfa_target_id present in a PU preceding the current PU in decoding order.

[0145] · There is an NNPFC SEI message with nnpfc_id equal to a specific value of nnpfa_target_id for the current PU.

[0146] If a PU contains both an NNPFC SEI message with a particular value of nnpfc_id and an NNPFA SEI message with nnpfa_target_id equal to a particular value of nnpfc_id, the NNPFC SEI message shall precede the NNPFA SEI message in decoding order.

[0147] nnpfa_cancel_flag, when set to 1, indicates that the continuity of the target neural network postfiltering set by the previous NNPFA SEI message with the same nnpfa_target_id as the current SEI message is cancelled, i.e. the target neural network postfiltering is not performed.

[0148] If a message has the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag is 0, the targeted neural network postfilter will not be used unless activated by another NNPFA SEI message. If nnpfa_cancel_flag is 0, nnpfa_persistence_flag indicates that it will persist.

[0149] nnpfa_persistence_flag indicates that the target neural network postfilter processing for the current layer should persist in display output order.

[0150] If nnpfa_persistence_flag is 0, it indicates that the target neural network postfiltering applies to postfiltering of the current picture only.

[0151] nnpfa_persistence_flag, when set to 1, indicates that the target neural network postfiltering should be applied to the current image and all subsequent pictures of the current layer until one or more of the following conditions become true:

[0152] - A new CLVS for the current layer is started.

[0153] Bitstream finished.

[0154] · Images in the current layer associated with an NNPFA SEI message that has the same nnpfa_target_id as the current SEI message and has nnpfa_cancel_flag equal to 1 are output after the current image in the display output order.

[0155] Note that neural network postfiltering is not applied to subsequent pictures of the current layer associated with the NNPFA SEI message if they have the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag is 1.

[0156] Furthermore, the auxiliary extension information creating device 71, the auxiliary extension information encoding device 81, and the auxiliary extension information decoding device 91 may commonly hold generic network parameters. The auxiliary extension information creating device 71 uses a framework such as a neural network postfilter characteristic SEI to create network parameters for partially updating the commonly held generic network as auxiliary extension information. The auxiliary extension information may then be encoded by the auxiliary extension information encoding device 81 and decoded by the auxiliary extension information decoding device 91. With this configuration, the amount of code for the auxiliary extension information can be reduced, and auxiliary extension information corresponding to the input image T can be created, encoded, and decoded.

[0157] In addition, in order to support multiple formats as the transmission format of the network parameters, a parameter (identifier) ​​indicating the format may be sent. Furthermore, the actual supplementary extension information following the identifier may be transmitted as a byte string.

[0158] The auxiliary extension information of the network parameters decoded by the auxiliary extension information decoder 91 is input to the postfilter processor 61 .

[0159] The postfilter processor 61 performs post-image processing using a neural network by using the decoded auxiliary enhancement information (neural network postfilter characteristics SEI, neural network postfilter activation SEI) to restore the decoded video Td.

[0160] The auxiliary extension information encoding device 81 encodes the auxiliary extension information based on the syntax tables in Fig. 7 and Fig. 8. The auxiliary extension information is encoded as auxiliary extension information SEI, multiplexed into the encoded data Te output by the image encoding device 11, and output to the network 21.

[0161] The auxiliary extension information decoding device 91 decodes the auxiliary extension information from the encoded data Te based on the syntax tables in Figures 7 and 8, and sends the decoded result to the post-filter processing device 61 and the image recognition device 51. The auxiliary extension information decoding device 91 decodes the auxiliary extension information encoded as auxiliary extension information SEI.

[0162] The post-filtering device 61 performs post-image processing on the decoded video image Td using the decoded video image Td and the auxiliary extension information to generate a post-image processing image To.

[0163] Furthermore, general-purpose network parameters may be held in common by the auxiliary extension information creating device 71, the auxiliary extension information encoding device 81, and the auxiliary extension information decoding device 91. In the auxiliary extension information creating device 71, network parameters for partially updating the commonly held general-purpose network may be created as auxiliary extension information, which may then be encoded by the auxiliary extension information encoding device 81 and decoded by the auxiliary extension information decoding device 91. With this configuration, the amount of code for the auxiliary extension information can be reduced, and auxiliary extension information corresponding to the input image T can be created, encoded, and decoded.

[0164] In addition, in order to support multiple formats as the transmission format of the network parameters, a parameter (identifier) ​​indicating the format may be sent. Furthermore, the actual supplementary extension information following the identifier may be transmitted as a byte string.

[0165] The auxiliary extension information of the network parameters decoded by the auxiliary extension information decoder 91 is input to the postfilter processor 61 .

[0166] In addition, in the example of this embodiment, the syntax in SEI is shown, but it is not limited to SEI, and syntax such as SPS, PPS, APS, slice header, etc. may be used.

[0167] In this embodiment, in such a configuration, a method is provided that can improve image quality even when the transmission rate is low, and can efficiently code and decode supplemental extension information regardless of the prediction structure.

[0168] (Postfilter Hints SEI) Figure 9 shows the syntax of the Post Filter Hint SEI described in Non-Patent Document. This SEI message provides post filter coefficients or correlation information for post filter design. After a picture is decoded and output, it is possible to perform post processing on the picture to improve the image quality.

[0169] To use the Post-Filter Hints SEI message, define the following variables:

[0170] - Variable CroppedWidth that represents the width value and variable CroppedHeight that represents the height value of the decoded image of the output luminance pixels.

[0171] ChromaFormatIdc is a variable that indicates the chrominance format. A value of 0 indicates monochrome, a value of 1 indicates 4:2:0 format, a value of 2 indicates 4:2:2 format, and a value of 3 indicates 4:4:4 format.

[0172] BitDepthY variable indicating the pixel bit length of the luminance pixels of the output decoded image.

[0173] Variable BitDepthC that indicates the pixel bit length of the chrominance pixels of the output decoded image.

[0174] The pixel array CroppedSampleArray[cIdx] of the cIdx-th component of the decoded image to be output. When the value of cIdx is 0, it indicates the luminance Y component, when the value of cIdx is 1, it indicates the chrominance Cb component, and when the value of cIdx is 2, it indicates the chrominance Cr component.

[0175] The variables SubWidthC and SubHeightC are derived by the value of ChromaFormatIdc. If ChromaFormatIdc has a value of 0, i.e. monochrome, then both SubWidthC and SubHeightC have the same value. If ChromaFormatIdc has a value of 1, i.e. 4:2:0 format, then both SubWidthC and SubHeightC have the same value. If ChromaFormatIdc has a value of 2, i.e. 4:2:2 format, then SubWidthC has the same value and SubHeightC has the same value. If ChromaFormatIdc has a value of 3, i.e. 4:4:4 format, then both SubWidthC and SubHeightC have the same value.

[0176] filter_hint_cancel_flag, when equal to 1, indicates that the SEI message cancels the continuity of the previous post-filter hint SEI message in the output order applied to the current layer.

[0177] filter_hint_cancel_flag equals 0 to indicate that post-filter hint information follows.

[0178] filter_hint_persistence_flag specifies the persistence of post-filter hints SEI messages for the current layer.

[0179] filter_hint_persistence_flag equal to 0 indicates that the postfilter hint SEI applies only to the current decoded image.

[0180] filter_hint_persistence_flag equal to 1 indicates that the postfilter SEI message is applied to the current decoded picture and persists for all subsequent pictures of the current layer in output order until one or more of the following conditions become true:

[0181] A new CLVS for the current hierarchy is started.

[0182] The bitstream ends.

[0183] · The picture of the current layer of the AU associated with the Post-Filter Hints SEI message is the output that follows the current picture in output order.

[0184] filter_hint_size_y specifies the vertical size of the filter coefficient or correlation array. The value of filter_hint_size_y ranges from 1 to 15.

[0185] filter_hint_size_x specifies the horizontal size of the filter coefficient or correlation array. The value of filter_hint_size_x ranges from 1 to 15.

[0186] filter_hint_type identifies the format of the post-filter hint sent. A value of 0 for filter_hint_type indicates that the filter coefficients are a 2-D FIR filter. A value of 1 for filter_hint_type indicates that the filter coefficients are a 1-D FIR filter. A value of 2 for filter_hint_type indicates a cross-correlation matrix. The value of filter_hint_type MUST be in the range 0 to 2. The value of filter_hint_type is reserved for future use. If filter_hint_type is 3, the post-filter hint SEI message SHALL be ignored.

[0187] A value of 1 in filter_hint_chroma_coeff_present_flag indicates that chrominance filter coefficients are present, and a value of 0 in filter_hint_chroma_coeff_present_flag indicates that chrominance filter coefficients are not present.

[0188] filter_hint_value[cIdx][cy][cx] indicates the filter coefficient or cross-correlation matrix element between the original signal and the decoded signal with 16-bit accuracy. The value of filter_hint_value[cIdx][cy][cx] ranges from minus 2^31 plus 1 to 2^31 plus 1. cIdx indicates the component, and a value of 0 for cIdx indicates the element for luma pixel, 1 for chroma pixel Cb, and 2 for chroma pixel Cr. cy indicates the vertical counter and cx indicates the horizontal counter. Depending on the value of filter_hint_type, the following applies:

[0189] If the value of filter_hint_type is 0, the coefficients of a two-dimensional Finite Impulse Response (FIR) filter of size filter_hint_size_y*filter_hint_size_x are transmitted.

[0190] If the value of filter_hint_type is 1, the filter coefficients of two 1D FIR filters are transmitted. In this case, the value of filter_hint_size_y must be 2. A value of 0 for index cy indicates the filter coefficients of the horizontal filter, and a value of 1 for cy indicates the filter coefficients of the vertical filter. In the filtering process, the horizontal filter is applied first, and the result is filtered by the vertical filter.

[0191] A value of 2 for filter_hint_type indicates that the transmitted hint information is a cross-correlation matrix between the original signal s and the decoded signal s', where the normalized cross-correlation matrix of the associated component identified by cIdx and having size filter_hint_size_y*filter_hint_size_x is defined by the formula shown in Fig. 10.

[0192] Here, s represents an array of pixels of component cIdx of the original image, and s' represents CroppedSampleArray[cIdx]. h is the value of CroppedHeight when the value of cIdx is 0, and the value of CroppedHeight / SubHeightC when the value of cIdx is greater than 0. w is the value of CroppedWeight when the value of cIdx is 0, and the value of CroppedWidth / SubWidthC when the value of cIdx is greater than 0.

[0193] bitDepth shall be the value of BitDepthY when the value of cIdx is 0, and the value of BitDepthC when the value of cIdx is greater than 0.

[0194] When cIdx is greater than 0, OffsetY is (filter_hint_size_y>>1), OffsetX is (filter_hint_size_x>>1), and 0 <= cy < filter_hint_size_y and 0 <= cx < filter_hint_size_x.

[0195] Note that the auxiliary extended information decoder 91 can derive a Wiener filter from the cross-correlation matrix of the original signal and the decoded signal, and the autocorrelation matrix of the decoded signal.

[0196] In Non-Patent Document 1, when the value of filter_hint_type is 0 or 1, the filter coefficient itself is transmitted by filter_hint_value, but there is a problem that the pixel bit length of the filtered image is unclear.

[0197] Therefore, when the value of filter_hint_type is 0 or 1, the pixel bit length of the filtered image is derived by the following formula using the total value filtSum for m = 0..h - 1, n = 0..w - 1 of filter_hint_value[m][n].

[0198] filtSum = bitDepth+Ceil(Log2(filtSum)) In Non-Patent Document 1, in the formula in Fig. 10 that defines the cross-correlation matrix when the value of filter_hint_type is 2, the values ​​of the cross-correlation matrix are normalized by dividing them by the product of the squared value of 2 to the power of (8+bitDepth) minus 1 and the number of pixels. This causes a problem in that the values ​​of each element of the matrix become very small.

[0199] Furthermore, the processing of the edge portions of the screen is not taken into consideration, and the definition equation results in accessing the outside of the screen of the decoded image s'.

[0200] Therefore, in this embodiment, instead of the value obtained by squaring the value of 2 to the power of (8+bitDepth) minus 1 as in the formula shown in Fig. 11(a), a value of 4 to the power of (bitDepth-8) is used. This makes it possible to suppress the value of each element of the matrix to 16-bit precision regardless of the pixel bit length. Alternatively, instead of division, an operation of performing a right shift of (2*bitDepth-16) bits may be used as in the formula shown in Fig. 11(b).

[0201] When the value of each element of the matrix is ​​to have 32-bit precision, a method that eliminates division of values ​​related to bitDepth, that is, a formula that normalizes only the number of pixels h*w, may be used.

[0202] Also, clipping processing is added so that only pixels within the screen are accessed. In Clip3(0,h-1,m+cy-OffsetY), pixel positions that are less than the top edge of the screen or greater than the bottom edge are replaced with pixel positions at the top edge of the screen or the bottom edge of the screen, respectively (padding processing is performed). Similarly, in Clip3(0,w-1,n+cx-OffsetX), pixel positions that are less than the left edge of the screen or greater than the right edge of the screen are replaced with pixel positions at the left edge of the screen or the right edge of the screen, respectively (padding processing is performed).

[0203] As another method, instead of clipping, the cross-correlation matrix may be calculated only within the screen of the decoded image s' as shown in the formula in Fig. 12. In this case, m is set to (filter_hint_size_y>>1) to h-1-(filter_hint_size_y>>1) instead of 0 to h-1, and n is set to (filter_hint_size_x>>1) to w-1-(filter_hint_size_x>>1) instead of 0 to w-1. Therefore, the number of pixels used in the calculation is (h-2*(filter_hint_size_y>>1))*(w-2*(filter_hint_size_x>>1)).

[0204] Also, there are several methods other than clipping processing that assumes that pixels on the edge of the screen are extended off the screen as they are. For example, reflection padding may be used, which assumes that pixels are extended from the edge of the screen in a mirror-like manner. Wrap-around padding may be used, which assumes that pixels are continuous with pixels on the opposite edge of the screen. Fixed padding may be used, which pads with a fixed value, for example, a value of (1<<(BitDepth-1)) for the pixel bit length BitDepth. A method of padding with a value of 0 may also be used.

[0205] [Application example] The above-mentioned video encoding device 10 and video decoding device 30 can be mounted and used in various devices that transmit, receive, record, and play back videos. Note that the video may be a natural video captured by a camera or the like, or an artificial video (including CG and GUI) generated by a computer or the like.

[0206] (SEI payload) FIG. 13 is a diagram showing the syntax of the SEI payload, which is a container of the SEI message.

[0207] This is called when nal_unit_type is PREFIX_SEI_NUT. PREFIX_SEI_NUT indicates that the SEI is located before the slice data.

[0208] When payloadType is 210, the neural network postfilter feature SEI is called.

[0209] When payloadType is 211, the neural network post-filter activation SEI is called.

[0210] (SEI decoding and post-filtering) The header decoding unit 3020 reads the SEI payload, which is a container of the SEI message, and decodes the neural network post-filter characteristics SEI message. For example, the header decoding unit 3020 decodes nnpfc_id, nnpfc_mode_idc, nnpfc_formatting_and_purpose_flag, nnpfc_purpose, nnpfc_reserved_zero_bit_a, nnpfc_uri_tag[i], nnpfc_uri[i], nnpfc_reserved_zero_bit_b, and nnpfc_payload_byte[i].

[0211] 14 is a flowchart showing the processing of the post-filter processor 61. The post-filter processor 61 performs the following processing in accordance with the parameters of the SEI message.

[0212] S6001: Read processing volume and accuracy from SEI.

[0213] S6002: If the amount of post processing exceeds the processable complexity, the process ends. If not, the process proceeds to S6003.

[0214] S6003: End if the accuracy exceeds the processable accuracy of the post-filter processor 61. If not, proceed to S6004.

[0215] S6004: A network model is identified from the SEI, and the topology of the post-filter processing device 61 is set.

[0216] S6005: Derive network model parameters from the updated information of SEI.

[0217] S6006: The derived network model parameters are loaded into the post-filter processor 61.

[0218] S6007: The post-filter processor 61 executes filtering and outputs the result to the outside.

[0219] However, the SEI is not necessarily required to construct luma and chroma samples in the decoding process.

[0220] (Details of the post-filter processing device 61) The NN filter unit performs filtering using a neural network model, using the input image inputTensor and input parameters (e.g., QP, bS, etc.). The input image may be an image for each component, or an image with multiple components as channels. In addition, the input parameters may be assigned to a channel different from the image.

[0221] The NN filter unit may repeatedly apply the following process.

[0222] The NN filter part derives the output image outputTensor by convolving the inputTensor with the kernel k[m][i][j] (conv, convolution) and adding bias. Here, nn=0..n-1, xx=0..width-1, yy=0..height-1, and Σ represents the summation for mm, i, and j, respectively.

[0223] outputTensor[nn][xx][yy]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][xx+i-of][yy+j-of]+bias[nn]) In the case of 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0. In this case, of=0 is set. In the case of 3x3 Conv, Σ represents the sum of mm=0..m-1, i=0..2, j=0..2. In this case, of=1 is set. n is the number of channels of outSamples, m is the number of channels of inputTensor, width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor. of is the size of the padding area placed around inputTensor to make the sizes of inputTensor and outputTensor the same. In the following, when the output of the NN filter part is a value (correction value) rather than an image, the output is represented as corrNN instead of outputTensor.

[0224] Note that if you write inputTensor and outputTensor in CHW format instead of CWH format, it is equivalent to the following process.

[0225] outputTensor[nn][yy][xx]=ΣΣΣ(k[mm][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn]) In addition, a process called Depth-wise Conv, shown in the following formula, may be performed. Here, nn=0..n-1, xx=0..width-1, yy=0..height-1, and Σ represents the summation for i and j, respectively. n is the number of channels of outputTensor and inputTensor, width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor.

[0226] outputTensor[nn][xx][yy]=ΣΣ(k[nn][i][j]*inputTensor[nn][xx+i-of][yy+j-of]+bias[nn]) Also, a nonlinear process called Activate, for example, ReLU, may be used.

[0227] ReLU(x) = x >= 0 ? x : 0 Alternatively, leakyReLU shown in the following formula may be used.

[0228] leakyReLU(x) = x >= 0 ? x : a * x Here, a is a predetermined value, for example, 0.1 or 0.125. In order to perform integer arithmetic, all the values ​​of k, bias, and a above may be integers, and a right shift may be performed after conv.

[0229] With ReLU, 0 is always output for values ​​less than 0, and the input value is output as is for values ​​greater than or equal to 0. On the other hand, with leakyReLU, linear processing is performed for values ​​less than 0 with the gradient set by a. With ReLU, the gradient for values ​​less than 0 disappears, which can make it difficult for learning to progress. With leakyReLU, the gradient for values ​​less than 0 remains, making the above problem less likely to occur. Of the above leakyReLU(x), PReLU, which uses a parameterized value of a, may also be used.

[0230] (NNC) Neural Network Coding (NNC) is an international standard ISO / IEC15938-17 for efficiently compressing neural networks (NNs). Compressing trained NNs makes it possible to store and transmit NNs more efficiently.

[0231] The following provides an overview of the encoding and decoding processes of the NNC.

[0232] FIG. 15 is a diagram showing an encoding device and a decoding device of the NNC.

[0233] The NN coding device 801 has a pre-processing unit 8011, a quantization unit 8012, and an entropy coding unit 8013. The NN coding device 801 receives an uncompressed NN model O, and the quantization unit 8012 quantizes the NN model O to obtain a quantized model Q. The NN coding device 801 may repeatedly apply a parameter reduction method such as pruning or sparsification before quantization in the pre-processing unit 8011. Thereafter, the entropy coding unit 8013 applies entropy coding to the quantized model Q to obtain a bit stream S for storing and transmitting the NN model.

[0234] The NN decoding device 802 includes an entropy decoding unit 8021, a parameter restoration unit 8022, and a post-processing unit 8023. The NN decoding device 802 first inputs the transmitted bit stream S, and the entropy decoding unit 8021 performs entropy decoding of S to obtain an intermediate model RQ. If the operating environment of the NN model supports inference using the quantized representation used in RQ, the RQ may be output and used for inference. If not, the parameter restoration unit 8022 restores the parameters of RQ to their original representation to obtain an intermediate model RP. If the sparse tensor representation used can be processed in the operating environment of the NN model, the RP may be output and used for inference. If not, a reconstructed NN model R that does not include a tensor or structural representation different from the NN model O is obtained and output.

[0235] The NNC standard has decoding methods for certain numeric representations of NN parameters, including integers and floating point.

[0236] The decoding method NNR_PT_INT decodes a model with integer-valued parameters. The decoding method NNR_PT_FLOAT extends NNR_PT_INT by adding a quantization step size delta, which is multiplied by the integer value to produce a scaled integer. Delta is derived from the integer quantization parameter qp and the granularity parameter qp_density of delta as follows:

[0237] mul = 2^(qp_density) + (qp & (2^(qp_density)-1)) delta = mul * 2^((qp >> qp_density)-qp_density) (Format of trained NN) The representation of a trained NN consists of two elements: a topological representation, such as the size of layers and the connections between layers, and a parameter representation, such as weights and biases.

[0238] Topological representation is covered by native formats such as TensorFlow (registered trademark) and PyTorch (registered trademark), but to improve interoperability, exchange formats such as Open Neural Network Exchange Format (ONNX, registered trademark) and Neural Network Exchange Format (NNEF) exist.

[0239] The NNC standard also transports the topology information nnr_topology_unit_payload as part of the NNC bitstream containing the compressed parameter tensors, enabling interoperability with topology information expressed in native formats as well as interchange formats.

[0240] (Configuration of Image Encoding Device) Next, the configuration of the image encoding device 11 according to this embodiment will be described. Fig. 6 is a block diagram showing the configuration of the image encoding device 11 according to this embodiment. The image encoding device 11 includes a prediction image generating unit 101, a subtraction unit 102, a transformation and quantization unit 103, an inverse quantization and inverse transformation unit 105, an addition unit 106, a loop filter 107, a prediction parameter memory (prediction parameter storage unit, frame memory) 108, a reference picture memory (reference image storage unit, frame memory) 109, an encoding parameter determining unit 110, a parameter encoding unit 111, a prediction parameter derivation unit 120, and an entropy encoding unit 104.

[0241] The predicted image generating unit 101 generates a predicted image for each CU.

[0242] The subtraction unit 102 generates a prediction error by subtracting the pixel values ​​of the predicted image of the block input from the predicted image generation unit 101 from the pixel values ​​of the image T. The subtraction unit 102 outputs the prediction error to the transformation and quantization unit 103.

[0243] The transform / quantization unit 103 calculates transform coefficients by frequency transforming the prediction errors input from the subtraction unit 102, and derives quantized transform coefficients by quantizing the prediction errors. The transform / quantization unit 103 outputs the quantized transform coefficients to the parameter coding unit 111 and the inverse quantization / inverse transform unit 105.

[0244] The inverse quantization and inverse transform unit 105 is the same as the inverse quantization and inverse transform unit 311 (FIG. 6) in the image decoding device 31, and a description thereof will be omitted. The calculated prediction error is output to the addition unit .

[0245] The parameter coding unit 111 includes a header coding unit 1110, a CT information coding unit 1111, and a CU coding unit 1112 (prediction mode coding unit). The CU coding unit 1112 further includes a TU coding unit 1114. The following describes an outline of the operation of each module.

[0246] The header encoding unit 1110 performs encoding processing of parameters such as header information, division information, prediction information, and quantized transform coefficients.

[0247] The CT information encoding unit 1111 encodes the QT, MT (BT, TT) division information and the like.

[0248] The CU encoding unit 1112 encodes the CU information, prediction information, division information, and so on.

[0249] When a prediction error is included in a TU, the TU encoding unit 1114 encodes the QP update information and the quantized prediction error.

[0250] The CT information encoding unit 1111 and the CU encoding unit 1112 supply syntax elements such as inter prediction parameters and quantized transform coefficients to the parameter encoding unit 111.

[0251] The entropy coding unit 104 receives the quantized transform coefficients and the coding parameters from the parameter coding unit 111. The entropy coding unit 104 entropy codes these to generate and output coded data Te.

[0252] The prediction parameter derivation unit 120 derives inter prediction parameters and intra prediction parameters from the parameters input from the encoding parameter determination unit 110. The derived inter prediction parameters and intra prediction parameters are output to the parameter encoding unit 111.

[0253] The adder 106 generates a decoded image by adding, for each pixel, the pixel value of the predicted block input from the predicted image generation unit 101 and the prediction error input from the inverse quantization and inverse transform unit 105. The adder 106 stores the generated decoded image in a reference picture memory 109.

[0254] The loop filter 107 performs deblocking filtering, SAO, and ALF on the decoded image generated by the adder 106. Note that the loop filter 107 does not necessarily have to include the above three types of filters, and may be configured, for example, as only a deblocking filter.

[0255] The prediction parameter memory 108 stores the prediction parameters generated by the coding parameter determination unit 110 in a predetermined location for each current picture and CU.

[0256] The reference picture memory 109 stores the decoded image generated by the loop filter 107 at a predetermined position for each current picture and CU.

[0257] The coding parameter determination unit 110 selects one set from among a plurality of sets of coding parameters. The coding parameters are the above-mentioned QT, BT or TT division information, prediction parameters, or parameters to be coded that are generated in relation to these. The predicted image generation unit 101 generates a predicted image using these coding parameters.

[0258] In addition, a part of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment, for example, the entropy decoding unit 301, the parameter decoding unit 302, the loop filter 305, the predicted image generating unit 308, the inverse quantization and inverse transform unit 311, the addition unit 312, the prediction parameter derivation unit 320, the predicted image generating unit 101, the subtraction unit 102, the transform and quantization unit 103, the entropy coding unit 104, the inverse quantization and inverse transform unit 105, the loop filter 107, the coding parameter determination unit 110, the parameter coding unit 111, and the prediction parameter derivation unit 120 may be realized by a computer. In this case, a program for realizing this control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into and executed by a computer system. In addition, the "computer system" referred to here is a computer system built into either the image encoding device 11 or the image decoding device 31, and includes hardware such as an OS and peripheral devices. In addition, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include devices that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and devices that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0259] In addition, a part or the whole of the image encoding device 11 and the image decoding device 31 in the above-mentioned embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image encoding device 11 and the image decoding device 31 may be individually made into a processor, or a part or the whole may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the progress of semiconductor technology, an integrated circuit based on that technology may be used.

[0260] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0261] This embodiment will be described with reference to FIG. 1. A video decoding device 30 includes an image decoding device 31 that decodes encoded data to generate a decoded image, a post-filter processing device 61 that performs post-filter processing on the decoded image, and an auxiliary extension information decoding device 91 that decodes auxiliary extension information used in the post-filter processing device, and is characterized in that the auxiliary extension information decoding device 91 decodes cross-correlation matrix information between a decoded image and an original image with a constant bit precision regardless of the pixel bit length of the decoded image.

[0262] The video coding device 10 comprises an image coding device 11 which codes an input image, an auxiliary extension information generating device 71 which generates auxiliary extension information for post-filter processing, and an auxiliary extension coding device 81 which codes the auxiliary extension information, and is characterized in that the auxiliary extension information generating device 71 generates cross-correlation matrix information between a locally decoded image and an original image with a constant bit precision regardless of the pixel bit length of the decoded image, and comprises an auxiliary extension information coding device 81 which codes the cross-correlation information.

[0263] This embodiment will be described with reference to FIG. 1. A video decoding device 30 includes an image decoding device 31 that decodes encoded data to generate a decoded image, a postfilter processing device 61 that performs postfilter processing on the decoded image, and an auxiliary extension information decoding device 91 that decodes auxiliary extension information used in the postfilter processing device 61. The postfilter processing device 61 is characterized in that when upsampling processing is performed on chrominance pixels of a 4:2:2 format decoded image, the auxiliary extension information decoding device 91 does not decode flag information for the output format.

[0264] The video encoding device 10 includes an image encoding device 11 that encodes an input image, an auxiliary extension information generating device 71 that generates auxiliary extension information for post-filter processing, and an auxiliary extension encoding device 81 that encodes the auxiliary extension information, and is characterized in that the auxiliary extension generating device 71 generates auxiliary extension information for upsampling processing of chrominance pixels of a 4:2:2 format decoded image, and the auxiliary extension information encoding device 81 does not encode flag information for the output format.

[0265] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0266] The embodiments of the present invention can be suitably applied to a video decoding device that decodes coded data in which image data is coded, and a video coding device that generates coded data in which image data is coded, and can also be suitably applied to the data structure of coded data that is generated by a video coding device and referenced by the video decoding device. [Explanation of symbols]

[0267] 1. Video transmission system 30 Video Decoding Device 31 Image Decoding Device 301 Entropy Decoding Unit 302 Parameter Decoding Unit 305, 107 Loop Filter 306, 109 Reference Picture Memory 307, 108 Prediction parameter memory 308, 101 Prediction image generation unit 311, 105 Inverse quantization and inverse transformation unit 312, 106 Addition section 320 Prediction Parameter Derivation Unit 10 Video Encoding Device 11 Image encoding device 102 Subtraction section 103 Transformation and Quantization Section 104 Entropy coding unit 110 Encoding parameter determination unit 111 Parameter Encoding Unit 120 Prediction parameter derivation part 41 Image display device 51 Pre-filter processing device 61 Post-filter processing device 71 Auxiliary extension information creation device 81 Supplementary extension information coding device 91 Auxiliary extension information decoding device

Claims

1. A video decoding device for decoding coded data including supplemental extension information, comprising: a supplemental extension information decoding unit that decodes the supplemental extension information; the auxiliary enhancement information includes a post-filter hint SEI message that provides coefficients of a post-filter or correlation information for designing a post-filter; the post-filter hint SEI message includes: a first syntax element indicating a type of a filter hint; and a second syntax element indicating filter coefficients or elements of a cross-correlation matrix between an original signal and a decoded signal with 16-bit precision; A video decoding device, characterized in that when the value of the first syntax element is 2, the cross-correlation matrix is ​​defined using values ​​of 4 to the power of (bitDepth-8).

2. The video decoding device described in Claim 1, characterized in that the cross-correlation matrix is ​​derived by replacing pixel positions outside the screen of the decoded image signal with pixel positions at the edge of the screen.

3. A video decoding device for decoding coded data including supplemental extension information, a supplemental extension information decoding unit that decodes the supplemental extension information; the auxiliary enhancement information includes a post-filter hint SEI message that provides coefficients of a post-filter or correlation information for designing a post-filter; the post-filter hint SEI message includes: a first syntax element indicating a type of a filter hint; and a second syntax element indicating filter coefficients or elements of a cross-correlation matrix between an original signal and a decoded signal with 16-bit precision; A video decoding device, characterized in that when the value of the first syntax element is 2, the cross-correlation matrix is ​​defined using values ​​of 4 to the power of (bitDepth-8).

4. A recording medium for recording a signal that has been subjected to encoding processing, the signal includes supplemental enhancement information; the auxiliary enhancement information includes a post-filter hint SEI message that provides coefficients of a post-filter or correlation information for designing a post-filter; the post-filter hint SEI message includes: a first syntax element indicating a type of a filter hint; and a second syntax element indicating filter coefficients or elements of a cross-correlation matrix between an original signal and a decoded signal with 16-bit precision; 2. A recording medium comprising: a first syntax element having a value of 2; a cross-correlation matrix defined using a value of 4 to the power of (bitDepth-8);

Images