Message passing parameters for neural network post-filtering in image and video coding

Abstract

Methods, systems, and bitstream syntax are described for transmitting neural network topologies and parameters related to neural network based post-filtering (NNPF) in image and video coding. Examples of NNPF SEI messaging applicable to the MPEG video picture coding standard are described at the sequence layer and the picture layer.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 328,131 filed on April 6, 2022 and U.S. Provisional Patent Application No. 63 / 354,549 filed on June 22, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] This document relates generally to image and video coding. More specifically, embodiments of the present invention relate to message passing information related to message passing parameters related to neural network post filtering in image and video coding. Background Art

[0004] In 2020, the MPEG expert group in the International Organization for Standardization (ISO) and the International Telecommunication Union (ITU) jointly released the first version of the Universal Video Coding standard (VVC), also known as H.266 (reference [3]). Recently, the MPEG expert group has been working on the development of the next generation coding standard, whose coding performance is improved compared to existing video coding technologies. As part of this research, coding technologies based on artificial intelligence and deep learning are also investigated. As used herein, the term "deep learning" refers to a neural network (NN) having at least three layers, and preferably more than three layers.

[0005] Neural network post filtering (NNPF) and neural network loop filtering (NNLF) have been shown to improve coding efficiency for image and video coding. Although MPEG-7 Part 17 (ISO / IEC 15938-17) (Reference

[11] ) describes a method for compressing neural network representations, the efficiency of this method is quite low under the bit rate constraints of image and video coding. As the inventors herein appreciate, what is needed is improved techniques for transmission of neural network topologies and parameters associated with NNPF in image and video coding, and these techniques are described herein.

[0006] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, issues identified with respect to one or more approaches should not be assumed to qualify as prior art based on this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0008] Figure 1 An example processing pipeline of a neural network post-filter (NNPF) according to an embodiment of the present invention is shown;

[0009] Figure 2 An example packing format of a luma channel in a YUV420 signal according to an embodiment of the present invention is shown;

[0010] Figure 3 An example of a luminance-chrominance dependency is shown;

[0011] Figure 4 An example of zero padding of a frame is shown;

[0012] Figure 5 An example process of processing an SEI message for NNPF processing at a coding sequence layer according to an embodiment of the present invention is shown; and

[0013] FIG. 6 shows an example process of processing an SEI message for NNPF processing at a picture layer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] Example embodiments involving transmission of neural network topologies and parameters associated with NNPF in image and video coding are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the invention. However, it is apparent that various embodiments of the invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in detail in order to avoid unnecessarily obscuring, obscuring, or confusing embodiments of the invention.

[0015] Overview

[0016] Example embodiments described herein relate to transmitting neural network topologies and parameters associated with NNPF in image and video encoding. In an embodiment, a processor receives a decoded image and NNPF metadata associated with processing the decoded image using NNPF. The processor:

[0017] Parsing syntax parameters in the NNPF metadata to perform NNPF based on one or more neural network models, associated NNPF data, and NNPF parameters; and

[0018] Performing an NNPF operation on the decoded image according to the syntax parameters to generate an output image, wherein the syntax parameters in the NNPF metadata include a first set of NNPF message passing parameters that persist until the decoding of the encoded video sequence is completed and a second set of NNPF message passing parameters that persist until the NN post-filtering of the decoded image is completed.

[0019] In another embodiment, a processor receives an image or a video sequence containing pictures. The processor:

[0020] encoding the image or the video sequence into a coded bitstream; and

[0021] generating neural network post-filtering (NNPF) metadata to allow a decoder of the encoded bitstream to perform NNPF according to one or more neural network models, associated NNPF data, and NNPF parameters; and

[0022] Generate an output comprising the encoded bitstream and the NNPF metadata, wherein the syntax parameters in the NNPF metadata include a first set of NNPF message passing parameters that persist until the decoding of the encoded video sequence is completed and a second set of NNPF message passing parameters that persist until the NN post-filtering of a single decoded image is completed.

[0023] Example model of neural network post-filtering

[0024] Figure 1 An example process (100) of a neural network post-filter (NNPF) according to an embodiment is shown. Given a decoded input 102, the NNPF pipeline includes a preprocessing (130), actual NNPF processing, and a post-processing stage. The preprocessing stage (130) includes software / hardware initialization (105), data preparation (110), and NNPF model loading (115). Software / hardware initialization configures the receiver's computing environment (such as a graphics processing unit (GPU)) and specified software libraries (such as Tensorflow, PyTorch, etc.). After initialization, the computing platform can be used. Data preparation (110) converts the decoded frame (102) into a format that can be directly processed by the corresponding NN model. For example, the decoded frame is typically partitioned into patches (rectangular image blocks), which are converted into the data input format of the NN model, such as YUV444, etc., and are organized into patches before input. At the same time, in step 115, a specific model is selected based on the image type and other flags and loaded for use. The above three processes can be performed in parallel. The NNPF stage (120) performs the actual NN post-filtering operations (e.g., upscaling, filtering, etc.) based on the specific model, data, and platform inputs from the pre-processing stage (130). Finally, in the post-processing stage, in step 125, the NNPF output (122) will be converted to a data format suitable for display as output 127, while in step 130, the NNPF model may be unloaded so that the NNPF pipeline (100) is ready for other operations. Note that process 100 can be easily extended to other NN-based post-processing, such as super-resolution and denoising.

[0025] Metadata signaling for NNPF (e.g., via SEI messaging) has been proposed in past JVET meetings (references [4-10]). Previous proposals have focused more on how to signal NN topology and NN parameters by carrying an NNR (Neural Network Compression and Representation) bitstream (references [11-13]) or using external links (reference [4]) such as given Uniform Resource Identifiers (URIs), with syntax and semantics as described in IETF Internet Standard 66. Some proposals have also addressed issues related to NN input or output interfaces and NN complexity (references [7-9]).

[0026] Despite the use of compression, NNR bitstreams can still be quite large, affecting bandwidth utilization. In addition, when using NNR, the decoder needs to be compliant with and able to decode another standard. As the inventors appreciate, the NNPF metadata must be lightweight, but still provide the decoder with the necessary information to check whether it can apply NNPF, and if so, access the required parameters to perform the NNPF processing (100) described above.

[0027] Although neural networks can also be applied to loop filtering and other applications, the embodiments described herein focus on (but are not limited to) NNPF for two main reasons: 1) NNPF is independent of decompression, so the implementation can have greater freedom and can be used for any image or video codec. 2) It is performed outside the encoding loop (which usually includes transform processing, quantization and loop filtering (deblocking)), so fixed-point implementation is not required to avoid drift problems. Therefore, floating-point implementations commonly used in NNs can be applied.

[0028] Since NNPF is performed outside the decoding loop, NNPF does not have the potential drift problem of NNLF (loop filter) processing. For NNLF, if the filtering result of a certain frame or a certain block is not good (this is possible because NN may not be robust enough for all frame data), this will result in poor quality of the current decoded frame, and this frame may be used as a reference frame for subsequent frames. Therefore, errors and artifacts may be accumulated and propagated to other frames as drift phenomena. In another example, most NNs are implemented using floating points, which may have different results on different machines / platforms / operating systems. This may cause the encoder and decoder of a certain frame to not match, and if the mismatched frame is used as a reference, the error may cause drift problems in subsequent decoded frames.

[0029] Two levels of NNPF-related messaging are proposed: 1) at the CLVS (Coding Layer Video Sequence) layer (where NNPF operations persist until the end of the video sequence), and 2) at the picture level (where NNPF operations persist only until the end of the current picture). This allows per-picture NNPF messaging and filtering without repeating certain filter characteristics applicable to the entire video sequence. Although the proposed messaging is described using the notation and syntax commonly used to describe MPEG's SEI messaging (references [1-3]), the proposed metadata messaging can be carried using a variety of other suitable messaging formats, such as those used in AV1 and other proprietary or standard-based coding formats. The proposed messaging approach can also be applied to other MPEG-based standards, such as AVC and HEVC. The proposed SEI messages help NNPF exploit coding features by providing information that is not available to a standalone post-filter, thereby further improving the performance of the post-filter.

[0030] In an example embodiment, the proposed CLVS NNPF SEI aims to provide information to assist efficient implementation of the NNPF pipeline, such as initialization, preprocessing, model loading / unloading, and postprocessing. The picture-level NNPF SEI aims to achieve picture-level adaptation to further improve NNPF coding efficiency.

[0031] CLVS layer NNPF SEI

[0032] The scope of the CLVS layer NNPF SEI is for the entire coded sequence. The purpose is to signal it together with the first picture of the CLVS and it should not change during the entire CLVS process. It should be able to assist the decoder in being ready to apply NNPF to the decoded pictures after the bitstream is decoded. More specifically, when the NNPF SEI message exists in any picture of the CLVS of a particular layer, the NNPF SEI message should appear in the first picture of the CLVS. For the current layer, the NNPF SEI message persists from the current picture in decoding order until the end of the CLVS. All NNPF SEI messages applicable to the same CLVS should have the same content. In an example embodiment, the CLVS NNPF SEI includes the following information.

[0033] 1) Network topology and model parameters

[0034] For the NNPF SEI message, it is desirable that the SEI message carries only necessary information, so that the size of the SEI message is not too large. Otherwise, the encoder can only reduce the quantization (QP) value to improve the quality of the encoded sequence, but at the expense of increasing the bitrate. The size of the detailed network topology (e.g., using a graph to describe the topology) and its corresponding parameter values ​​(weights and biases in the case of a convolutional neural network (CNN)) can be relatively large, such as in the range of kilobytes, megabytes, or even gigabytes. It is not practical to carry all the information in the SEI bitstream. The model can be compressed (such as NNR in reference

[11] ), but its size is still not negligible. One way to signal the detailed NN model information is to use explicit links or some external means, such as cross-references to URIs (IETF Internet Standard 66) as discussed in reference [4]. Another way is to set a fixed model standardization 0 or external reference link for the base model, and the bitstream only carries incremental information (reference

[14] ), such as updated biases or weights, whether for the complete NN or a small part of the NN.

[0035] In addition to the topology and model parameters, it is also important to let the decoder know the following information so that it can help the decoder achieve fast initialization or quickly decide whether to implement NNPF or bypass it.

[0036] -NN storage / interchange formats: The most popular formats currently include ONNX, NNEF, PyTorch, and TensorFlow, but others can be added as needed

[0037] - Complexity indicators of NNPF: computation and memory. The most commonly used indicators are: NN parameter precision values: floating point (FP64, FP32 or FP16) and integer (e.g. INT8); number of NN model parameters; number of multiply-accumulate (MAC) operations per pixel, in thousands (kMac / pixel) or millions (mMac / pixel), and floating point operations per second (FLOPS). It is worth noting that the multiplication of NN parameter precision and number of NN parameters gives the memory size of the model. Other parameters (such as latency, throughput, power consumption) are also good indicators. Complexity indicators can help the decoder skip or bypass NNPF processing when there are not enough computational resources.

[0038] - Number of models: The NN models can be different based on various parameters, such as the signal encoded in the bitstream, QP value, slice / picture type, content type, and device type. For example, if a GBR (RGB) signal is encoded directly, a joint model is generally used. If a YUV signal is encoded, a joint model can be used (reference

[16] ) or separate models can be used for Y and U / V components (reference

[15] ). The bitstream can contain different slice types, such as intra (I) slices and inter (P / B) slices. The same model can be used for each slice, or different models can be used for intra and inter slices (references [15-16, 18]). The sequence can be standard dynamic range (SDR) or high dynamic range (HDR), natural content or screen capture content (SCC), etc., and each such variation may also require a different model. If the bitstream is decoded in various displays, the model may depend on the display type (e.g., TV or mobile device) to cope with the computational power of the decoder or the perceived visual quality on the display. Different quality problems may require different models, such as models that vary with QP (reference

[17] ).

[0039] As an example, Table 1 describes an example of syntax parameters for a NNPF topology and model parameter information for a single model. The syntax includes the NN topology and explicit link parameters (if present) or updated parameters, NN storage and exchange formats, and NN complexity indications. Multiple models should loop through this SEI. It is worth noting that multiple models are likely to use the same storage and exchange formats, so an alternative solution is to move this information out and signal it only once in the core NNPF SEI message.

[0040] Table 1. Examples of NNPF topology and model parameters

[0041]

[0042] nnpf_model_exter_link_flag equal to 1 indicates that the NNPF model is stored in the external link. nnpf_model_exter_link_flag equal to 0 indicates that the NNPF model is not stored in the external link.

[0043] nnpf_exter_uri[i] contains the i-th byte of a NULL-terminated UTF-8 string indicating the URI (IETF Internet Standard 66) that specifies the neural network to be used as a post-processing filter.

[0044] nnpf_model_upd_param_present_flag equal to 1 indicates that the model parameters have been updated. nnpf_model_upd_param_present_flag equal to 0 indicates that the model parameters have not been updated.

[0045] Note: For more updated parameter syntax and semantics, see reference [4].

[0046] nnpf_model_storage_form_idc indicates the storage and exchange format of the NNPF model, as shown in Table 2. Values ​​0 to 3 correspond to ONNX, NNEF, Tensorflow, and PyTorch, respectively. Values ​​4 to 7 are reserved for future extensions.

[0047] Table 2. nnpf_model_storage_form_idc explanation example

[0048] nnpf_model_storage_form_idc NNPF storage and exchange format 0 ONNX l NNE 2 Tensorflow 3 PyTorch 4..7 reserve

[0049] nnpf_model_complexity_ind_present_flag equal to 1 indicates that the model complexity indicator is present in the SEI message. If nnpf_model_complexity_ind_present_flag is equal to 0, the model complexity indicator is not present in the SEI message. Unless otherwise specified, the inferred value of all the following syntaxes should be 0. In this case, "0" can be interpreted as "NULL" (meaning not present) or "can be ignored".

[0050] nnpf_param_prec_idc represents the NNPF model parameter precision, as shown in Table 3. When not present, the syntax value of nnpf_param_prec_idc is inferred to be 5.

[0051] Table 3. nnpf_param_prec_idc explanation example

[0052] nnpf_param_prec_idc NNPF model parameter accuracy values 0 int8 l int16 2 int32 3 float16 4 float32 5 float64 6..15 reserve

[0053] Note: For the number of parameters, the following method can be used to represent them: c = 1.a*2^b, where "a" represents the decimal part of 1, and "b" (integer) is a power of 2 (for example, if a = 5 and b = 2, then c = 1.5*2^2 = 6).

[0054] nnpf_num_param_frac is a fraction used to represent the total number of model parameters

[0055] log2_prec_denorm is the base 2 logarithm of the denominator of the fraction used to represent the total number of model parameters.

[0056] log2_nnpf_num_param_minus11 plus 11 is the base 2 logarithm used to represent the total number of model parameters.

[0057] The variable tot_num_params is obtained as follows:

[0058] tot_num_params=(int64)(1.0+(float64)nnpf_num_param_frac /

[0059] (float64)(1<<(log2_prec_denorm))*(float64)

[0060] (1<<(log2_nnpf_num_param_minus11+11))

[0061] The total number of parameters of the NNPF model should not be greater than the value of tot_num_params.

[0062] When the above three syntax elements do not exist, the value of tot_num_params is inferred to be 0, that is, "NULL".

[0063] nnpf_num_ops multiplied by 1,000 specifies the maximum number of MAC (multiply-accumulate) operations per pixel for NNPF.

[0064] Note: A more precise definition of this parameter can use 1.a*2^b as tot_num_params.

[0065] nnpf_latency_idc specifies the latency indication of the NNPF model, as shown in Table 4. It indicates the latency of the NNPF model on the benchmark GPU (e.g., defined as Nvidi a The resolution and frame rate combinations that the NNPF model can support while ensuring real-time decoding and keeping consistent with the decoder without delay, if RTX 1080Ti) is available.

[0066] Table 4. nnpf_latency_idc explanation example

[0067] nnpf_latency_idc Resolutions and frame rates supported by the NNPF model 0 No requirement 1 720p, 30fps 2 720p, 60fps 3 1080p, 30fps 4 1080p, 60fps 5 1080p, 120fps 6 4k, 30fps 8 4k, 60fps 8 4k, 120fps 9 4k, 15fps 10-15 reserve

[0068] It is worth noting that the NN storage / interchange format or complexity indication can be generated by downloading the model and using a separate analyzer. Therefore, a "present flag" (such as nnpf_model_complexity_ind_present_flag) is used to provide this option as a complexity indication.

[0069] 2) Input and output colorimetric and data formats

[0070] The data input to the NN may not be in the same format as the decoded one. In order to correctly apply the NNPF, the following information may be included in the bitstream.

[0071] - Input and output data formats

[0072] -Data accuracy

[0073] -Tensor format

[0074] - Input and output patch sizes, boundary overlap indication and overlap size, image size, and padding method. It is worth noting that in NN training, the size of the input patch is very important to ensure the generalization and robustness of the model. Video frames can have a wide range of resolutions, so the scale of objects, textures, and artifacts can vary greatly. In addition to including various patch sizes when training one model, another efficient way is to use different models to handle different patch sizes. Patch size is also one of the important factors affecting training speed. Therefore, it is very important to indicate the patch size in the SEI.

[0075] An example of SEI message transmission data information is shown in Table 5.

[0076] Table 5. Example of NNPF data information

[0077]

[0078]

[0079] The semantics of input_chroma_format_idc are the same as specified for syntax sps_chroma_format_idc.

[0080] The semantics of output_chroma_format_idc are the same as specified for the syntax sps_chroma_format_idc.

[0081] The semantics of vui_matrix_coeffs are the same as specified for syntax vui_matrix_coeffs

[0082] packing_format_idc indicates the packing format of the luma channel, as shown in Table 6. The purpose is to make all input channels have the same dimension. Figure 2 FIG. 2 shows the case where packing_format_idc is equal to 0 for YUV420. Figure 2 In YUV420, one luma channel / plane is interleaved into 4 luma channels to have the same dimensions as the chroma channels U and V. So YUV420 becomes 6 channels. The case of YUV422 uses similar packing: YUV422 becomes 4 channels: one luma channel is interleaved into 2 luma channels to have the same dimensions as U and V.

[0083] Table 6. packing_format_idc explanation example

[0084] packing_format_idc value 0 staggered 1-7 reserve

[0085] chroma_luma_dependency_flag equal to 1 indicates that for the chroma NNPF model, the chroma channels are dependent on the luma channels of the NNPF input. chroma_luma_dependency_flag equal to 0 indicates that the chroma channels are independent of the luma channels of the NNPF input. Figure 3 An example of this concept is illustrated.

[0086] In alternative examples, more cases can be supported. luma_chroma_dependency_idc specifies the luma and chroma dependencies of the inputs to the luma model and chroma model, as shown in Table 7

[0087] Table 7. luma_chroma_dependency_idc explanation example

[0088]

[0089] The semantics of precision_format_idc is identical to the syntax of nnpf_param_prec_idc.

[0090] tensor_format_idc represents the tensor format of the input and output tensors, as shown in Table 8.

[0091] Table 8. tensor_format_idc explanation example

[0092] tensor format idc Format 0 NCHW l NHWC 2-3 reserve

[0093] In Table 8, variables N, C, H, and W represent:

[0094] variable meaning N Number of images / patches C Number of channels H high W width

[0095] log2_patch_size_minus6 specifies the base 2 logarithm of the luma patch size plus 6. The value of log2_patch_size_minus6 should be in the range of 0 to 6, inclusive.

[0096] Variable PatchSiz e The definition is as follows:

[0097] PatchSize=1<<(log2_patch_size_minus6+6).

[0098] Note: PatchSiz e Indicates the height and width of the patch. In another embodiment, the patch width and patch height can be specified separately.

[0099] picture_padding_type indicates the picture padding type, as shown in Table 9. Figure 4 The figure shows the situation when picture_padding_type is set to 0.

[0100] Table 9. picture_padding_type_idc explanation example

[0101] picture_padding_type Fill format 0 Zero (constant) padding 1 Copy Fill 2 Reflection Fill 3 reserve

[0102] When the width and height of the picture are not a multiple of patchSize, padding is required based on picture_padd_type. Padding is performed on the bottom and / or right side of the picture. The width and height of the decoded output picture are expressed in luma samples as PicWidthInLumaSamples and PicHeightInLumaSamples, respectively. The width and height of the filtered picture are expressed in luma samples as FilterPicWidthInLumaSamples and FilterPicHeightInLumaSamples, respectively. The derivation is as follows:

[0103] FilterPicWidthInLumaSamples=PicWidthInLumaSamples+patchSize-(PicWidthInLumaSamples%patchSize)

[0104] FilterPicHeightInLumaSamples=PicHeightInLumaSamples+patchSize-(PicHeightInLumaSamples%patchSize)

[0105] patch_boundary_overlap_flag equal to 1 specifies that patches overlap on the boundary. patch_boundary_overlap_flag equal to 0 specifies that patches do not overlap on the boundary.

[0106] log2_boundary_overlap_minus3 plus 3 specifies the base 2 logarithm of the boundary overlap between the horizontal patch and the vertical patch. The boundary overlap value in units of luma samples is derived to be equal to (1<<(log2_boundary_overlap_minus3+3). The value of log2_boundary_overlap_minus3 shall be in the range of 0 to 2, inclusive.

[0107] It is worth noting that the final input patch size of NNPF is set equal to

[0108] PatchSize+(patch_boundary_overlap_flag==0)?

[0109] 0:2*(1<<(log2_boundary_overlap_minus3+3))

[0110] 3) Auxiliary input information prompt

[0111] One of the advantages of using NNPF SEI messaging instead of pure NNPF is that NNPF SEI messaging is generated during encoding. This allows information related to bitstream characteristics to be included in SEI: such as QP information, picture / slice type information, partition information, inter / intra mapping information, classification information, temporal neighboring pictures, etc., as input to NNPF. In order to prepare the device to receive such auxiliary input, the auxiliary input information hint message can be indicated in the CLVS layer NNPF SEI, and more detailed information can be carried in the picture layer SEI. An example of auxiliary input hint information is shown in Table 10.

[0112] Table 10. NNPF auxiliary input prompt example

[0113]

[0114] nnpf_auxi_input_id contains a possible identifier number that can be used to identify the NNPF auxiliary input information. nnpf_auxi_input_id equal to 0 indicates that there is no auxiliary input for NNPF in CLVS. nnpf_auxi_input_id is interpreted as follows:

[0115] -The variable QpFlag (bit 0) is set equal to (nnpf_auxi_input_id & 0x01). QpFlag equal to 1 specifies that the QP map may be an auxiliary input to the NNPF in the current CLVS. QpFlag equal to 0 specifies that the QP map is not an auxiliary input to the NNPF in the current CLVS. (Note: "&" means bitwise AND)

[0116] - The variable PartitionFlag (bit 1) is set equal to ((nnpf_auxi_input_id & 0x02) >> 1). PartitionFlag equal to 1 specifies that the partition map may be an auxiliary input to the NNPF in the current CLVS. PartitionFlag equal to 0 specifies that the partition map is not an auxiliary input to the NNPF in the current CLVS.

[0117] - The variable ClassificationFlag (bit 2) is set equal to ((nnpf_auxi_input_id & 0x04) >> 2). ClassificationFlag equal to 1 specifies that the classification map may be an auxiliary input to the NNPF in the current CLVS. ClassificationFlag equal to 0 specifies that the classification map is not an auxiliary input to the NNPF in the current CLVS.

[0118] - The variable TemporalPicFlag (bit 3) is set equal to ((nnpf_auxi_input_id & 0x08) >> 3). TemporalPicFlag equal to 1 specifies that the temporal neighboring picture in the current CLVS may be an auxiliary input to the NNPF. TemporalPicFlag equal to 0 specifies that the temporal neighboring picture in the current CLVS is not an auxiliary input to the NNPF.

[0119] - The remaining bits (from bit 4 to bit 7) are reserved for future use by ITU-T | ISO / IEC.

[0120] Table 11 shows an example of CLVS layer NNPF SEI. The semantics follows the syntax table. In this example, multiple NNPF models are looped over picture types and device types.

[0121] Table 11. Example CLVS layer NNPF SEI message

[0122]

[0123]

[0124] nnpf_purpose indicates the purpose of the post-processing filter, as shown in Table 12. The value of nnpf_purpose should be between 0 and 2. 32 The nnpf_purpose value is in the range -2, inclusive. Values ​​of nnpf_purpose not appearing in Table 12 are reserved for future specification by ITU-T|ISO / IEC and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​of nnpf_purpose (reference [4]).

[0125] Table 12. nnpf_purpose explanation example

[0126] value explain 0 Visual quality improvements 1 Super Resolution 2 Denoising 3 Display Mapping other reserve

[0127] NOTE – When ITU-T | ISO / IEC uses the reserved value of nnrpf_purpose in the future, the syntax of this SEI message may be extended by a syntax element whose presence depends on whether nnrpf_purpose is equal to the reserved value.

[0128] The syntax and semantics of nnpf_purpose are taken from [4]. The allowed range may be too large for post-filtering purposes.

[0129] nnpf_model_info_present_flag equal to 1 specifies that nnpf model information is present in the SEI message. nnpf_model_info_present_flag equal to 0 specifies that nnpf model information is not present in the SEI message.

[0130] NOTE – When the NNPF model information is not present in the SEI message, the NNPF model shall be accessed by other means not specified in this specification.

[0131] nnpf_joint_model_flag equal to 1 specifies that NNPF uses the same model for all color components. nnpf_joint_model_flag equal to 0 specifies that NNPF uses separate models for luma and chroma components. When not present, the value of nnpf_joint_model_flag is inferred to be equal to 0.

[0132] NOTE: When nnpf_joint_model_flag is equal to 0, the external link should contain one model for the luma component and one model for the chroma components.

[0133] It is worth noting that when counting the number of models, in one embodiment, the luma component and the chroma component can be counted as one model. Therefore, even if separate models are used for luma and chroma, since only these two models can be used to complete a picture with both luma and chroma components, they can be counted as one model. Therefore, num_nnpf_models = (nnpf_num_pic_type_minus1+1)*(nnpf_num_device_type_minus1+1). In another embodiment, the models of the luma component and the chroma component can be counted as separate counts. Therefore, if the luma component and the chroma component use separate models, they are counted as two models. Therefore, num_nnpf_models = (nnpf_joint_model_flag = = 0?2:1)*(nnpf_num_pic_type_minus1+1)*

[0134] (nnpf_num_device_type_minus1+1). In Table 11, the latter method is used.

[0135] nnpf_num_pic_type_minus1 plus 1 indicates the number of picture types supported by the nnpf picture type based model. When not present, the value of nnpf_num_pic_type_minus1 is inferred to be equal to 0. This value should be in the range of 0 to 3 (inclusive).

[0136] nnpf_num_device_type_minus1 plus 1 indicates the number of device types supported by the nnpf device type based model. When not present, the value of nnpf_num_device_type_minus1 is inferred to be equal to 0. This value should be in the range of 0 to 15 (inclusive).

[0137] nnpf_mode_id[i] contains the identifier number that can be used to identify the i-th NNPF model. When not present, the value of nnpf_mode_id is inferred to be equal to 0. The value of nnpf_mode_id[i] should be in the range of 0 to 255 (inclusive). nnpf_model_id is interpreted as follows:

[0138] - The variable CompType (bit 0) is set equal to (nnpf_model_id[i] & 0x01), as shown in Table 13

[0139] - The variable PicType (bit 1) is set equal to ((nnpf_model_id[i]&0x02)>>1) as shown in Table 14

[0140] - The variable DeviceType (bits 2, 3, 4, 5) is set equal to ((nnpf_model_id[i] & 0x1C) >> 2). The variable displayType is set equal to (DeviceType & 0x03) as shown in Table 15. The display types are sorted in ascending order based on the display size. The variable complexityType is set equal to ((DeviceType & 0x0C) >> 2) as shown in Table 16. The complexityType is sorted in ascending order based on the complexity.

[0141] Table 13. CompType explanation example

[0142]

[0143] Table 14. PicType explanation example

[0144]

[0145] Note: Pictures in VVC may contain multiple slices, which may have different slice types. Since SEI is defined at the picture level, for pictures with mixed slice types, the encoder can decide which PicType the current picture belongs to. For example, if more than a certain percentage of blocks in a picture are encoded using the intra model, the picture can be considered an intra picture.

[0146] Table 15. displayType explanation example

[0147] displayType Display Type 0 All (using the same model) 1 Mobile phone 2 Mobile tablet / laptop 3 TV / Computer Monitor

[0148] In another example, a QualityType indication may also be added. Thus, different decoding qualities may use different models. The quality may be determined by the picture level QP.

[0149] - The variable QualityType (bits 6 and 7) is set equal to ((nnpf_model_id[i]&0xC0)>>6). QualityType is indicated in descending order. 0 indicates the highest quality and 3 indicates poor quality.

[0150] NOTE: It is also possible to define an association between QualityType and QP-based information. An example is given in Table 16.

[0151] Table 16. QualityType explanation example

[0152] qualityType baseQP 0 ＜＝27 1 27<baseQP<32 2 22<baseQP<37 3 >37

[0153] num_of_ckpts_minus1[nnpf_model_id[i]] plus 1 specifies the number of checkpoints for nnpf_model_id[i]. The index of each checkpoint is in increasing order from 0 to num_of_ckpts_minus1[nnpf_model_id[i]] (inclusive).

[0154] In NN literature, checkpoints (ckpts) are used to save model parameters, such as weights and biases in CNN. In this application, ckpts means using the same model topology. The difference between ckpts is the value of the model parameters.

[0155] nnpf_data_info_present_flag is equal to 1, indicating that nnpf_data_info() is present in the SEI message. nnpf_data_info_present_flag is equal to 0, indicating that nnpf_data_info() is not present in the SEI message.

[0156] In an alternative example, nnpf_data_info() and nnpf_auxiliary_input_info() can be associated with nnpf_model_id to gain more flexibility.

[0157] Table 17. Alternative example of CLVS layer NNPF SEI message

[0158]

[0159] It is worth noting that in another embodiment, the number of nnpf models can be specified using only the syntax num_nnpf_models_minus1, and the index i is assigned to nnpf_mode_id[i]. The disadvantage of this method is that nnpf_model_id[i] has no specific meaning, and the decoder will blindly use the nnpf model. The advantage is that the bitstream can carry as many models as you like. In addition, there is no need to strictly distinguish between checkpoints and models. For example, the bitstream can carry two different checkpoints for the same picture type, even though only one checkpoint will be used for any given picture.

[0160] num_nnpf_models_minus1plus1 specifies the number of NNPF models.

[0161] Model indices are in increasing order from 0 to num_nnpf_models_minus1 (inclusive).

[0162] Image layer NNPF SEI

[0163] One benefit of using picture-level NNPF SEI (denoted as nnpf_pic_adapt_SEI()) instead of independent NNPF is that SEI can carry adaptation information for each picture. This information may include the following parameters: picture layer, luminance / chrominance component and CTU layer NNPF on / off flag, picture / slice type, picture / slice QP, block-level QP, picture / slice / block-level classification, picture / slice-level inter / intra mapping, etc.

[0164] To save bit overhead, nnpf_pic_adapt_SEI() can reference CLVS level nnpf_sei() for high-level control.

[0165] The duration scope of nnpf_pic_adapt_SEI() is the current picture.

[0166] As for the signal transmission of nnpf_pic_model_id, several methods of Table 11 can be used: 1) nnpf_pic_model_id in nnpf_sei() can be explicitly signaled in nnpf_pic_adapt_SEI() at the cost of ue(v) bits. This explicit model is the base model. Bit0 should always be 0 to indicate that nnpf_pic_model_id represents a brightness model. The base model can tell PicType, DeviceType, or QualityType. If the model has a deviceType option, the user can select a different model based on displayType and complexityType. 2) nnpf_pic_model_id is inferred from other syntax in nnpf_pic_adapt_SEI accordingly. If the model has a deviceType option, the user can select the correct model based on displayType and complexityType. If an implicit model is used, nnpf_pic_type needs to be signaled to select a model from the pool.

[0167] The additional nnpf_pic_mode_id_chroma of the chroma component may be decided based on the nnpf_joint_model_flag derived as follows.

[0168]

[0169] In Table 17, if nnpf_joint_model_flag is equal to 0, then nnpf_pic_mode_id and nnpf_pic_model_id_chroma only need to be explicitly signaled.

[0170] For region related information, the region size can be implied to be the same as PatchSize in nnpf_sei(), or explicitly signaled if the size is different from PatchSize. In general, the region size should not be smaller than PatchSize, and preferably a multiple of PatchSize. For QP maps, classification maps, or partition maps used inside the region to generate auxiliary inputs, smaller units can be used, but the trade-off between accuracy and bit overhead needs to be considered.

[0171] Auxiliary input information should be generated from picture-level information or region-level information. For example, picture-level QP or region-based QP information can be used to generate a QP map. Region-based intra / inter information can be used to generate a classification map. Region-based partition information can be used to generate a partition map.

[0172] Table 18 shows an example of nnpf_pic_adapt_SEI(). In this example, for simplicity, the corresponding nnpf_mode_id is sent directly. It allows turning on / off the picture level and CTU level. The region size is inferred to be the same as the patchSize defined in nnpf_SEI().

[0173] Table 18. Picture layer NNPF SEI messaging example

[0174]

[0175]

[0176] nnpf_pic_enabled_flag equal to 1 specifies that nnpf is applied to the current picture. nnpf_pic_enabled_flag equal to 0 specifies that nnpf is not applied to the current picture. When not present, the value of nnpf_pic_enabled_flag is inferred to be equal to 0.

[0177] nnpf_pic_luma_enabled_flag equal to 1 specifies that nnpf is applied to the luma component of the current picture. nnpf_pic_luma_enabled_flag equal to 0 specifies that nnpf is not applied to the luma component of the current picture. When not present, the value of nnpf_pic_luma_enabled_flag is inferred to be equal to 0.

[0178] nnpf_pic_chroma_enabled_flag equal to 1 specifies that nnpf is applied to the chroma components of the current picture. nnpf_pic_chroma_enabled_flag equal to 0 specifies that nnpf is not applied to the chroma components of the current picture. When not present, the value of nnpf_pic_chroma_enabled_flag is inferred to be equal to 0.

[0179] nnpf_pic_model_id specifies the nnpf_mode_id for the current picture. nnpf_pic_ckpt_idx specifies the checkpoint index for nnpf_pic_model_id. The value of nnpf_pic_ckpt_idx is in the range from 0 to num_ckpts_minus1[nnpf_pic_model_id], inclusive.

[0180] nnpf_qp_info_present_flag equal to 1 specifies that the current SEI contains QP information. nnpf_qp_info_present_flag equal to 0 specifies that the current SEI does not contain QP information. When not present, the value of nnpf_qp_info_present_flag is inferred to be equal to 0.

[0181] nnpf_region_info_present_flag equal to 1 specifies that the current SEI contains region information. nnpf_region_info_present_flag equal to 0 specifies that the current SEI does not contain region information. When not present, the value of nnpf_region_info_present_flag is inferred to be equal to 0.

[0182] nnpf_region_qp_present_flag equal to 1 specifies that the current SEI contains region-based QP information. nnpf_region_qp_present_flag equal to 0 specifies that the current SEI does not contain region-based QP information. When not present, the value of nnpf_region_qp_present_flag is inferred to be equal to 0.

[0183] nnpf_region_ptt_present_flag equal to 1 specifies that the current SEI contains region-based partition information. nnpf_region_ptt_present_flag equal to 0 specifies that the current SEI does not contain region-based partition information. When not present, the value of nnpf_region_ptt_present_flag is inferred to be equal to 0.

[0184] nnpf_region_clfc_present_flag equal to 1 specifies that the current SEI contains region-based classification information. nnpf_region_clfc_present_flag equal to 0 specifies that the current SEI does not contain region-based classification information. When not present, the value of nnpf_region_clfc_present_flag is inferred to be equal to 0.

[0185] NOTE: nnpf_region_qp / ptt / clfc / _present_flag can also be implicitly inferred from nnpf_pic_model_id, i.e. this region level information is only needed when picType=Intra.

[0186] nnpf_region_enabled_flag[i] equal to 1 specifies that nnpf is enabled for the i-th region. nnpf_region_enabled_flag[i] equal to 0 specifies that nnpf is not enabled for the i-th region. When not present, the value of nnpf_region_enabled_flag[i] is inferred to be equal to 0.

[0187] The semantics of qp_delta_abs_map[i] are the same as those specified for cu_qp_delta_abs. The semantics of qp_delta_sign_map_flag[i] are the same as those specified for cu_qp_delta_sign_flag.

[0188] ptt_map[i] specifies the partition map of the ith region. The partition map is represented using the same interpretation as MaxMttDepthY. The value is in the range of 0 to log2(PatchSize)-3 (inclusive).

[0189] clfc_map[i] specifies the classification map of the i-th region.

[0190] In one example, the classification map only indicates intra or inter.

[0191] -If PicType is intra, clfc_map[i] equals 0 to specify that the classification map of the i-th region is intra, clfc_map[i] equals 1 to specify that the classification map of the i-th region is inter with no residual, and clfc_map[i] equals 2 to specify that the classification map of the i-th region is inter with residual.

[0192] -Otherwise, clfc_map[i] equals 0 to specify that the classification map of the i-th region is inter-frame and has no residual, clfc_map[i] equals 1 to specify that the classification map of the i-th region is inter-frame and has residual, and clfc_map[i] equals 0 to specify that the classification map of the i-th region is intra-frame.

[0193] The CLVS layer NNPF SEI messaging of Table 11 (which may carry the data defined in Tables 1-15) may require metadata information that is considered too large or unnecessary in some applications. To reduce the payload size, an example of an alternative simplified CLVS NNPF SEI message is illustrated in Table 19. To generate the syntax of Table 19, some previously defined parameters are deleted, as explained below.

[0194] The parameter nnpf_num_device_type_minus1 was skipped due to lack of experimental support for NNPF across multiple devices. The parameter nnpf_model_upd_param_present_flag was skipped because it comes from reference [4] and there is no demonstrated need for it. The parameter nnpf_latency_idc was skipped. This is also because it would need to be tested at too many different resolution and framerate configurations. Even if results were available, they would only be based on a baseline GPU. In practice, devices may use a variety of GPU architectures, making this metric less accurate or less useful. The parameters input_chroma_format_idc and output_chroma_format_idc have been merged into one: nnpf_chroma_format_idc, since in practice it is unlikely that the input and output of an NNPF will have different chroma formats. The parameter precision_format_idc was skipped because its function of indicating precision could be considered duplicated with the previously defined nnpf_param_prec_idc value. The parameter tensor_format_idc is skipped because it is highly related to the previously defined nnpf_model_storage_form_idc value. Storage formats (such as ONNX) usually specify tensor formats as well. The patch_boundary_overlap_flag is skipped because a deblocking filter is usually applied in the bitstream. Therefore, for NNPF, overlap is most likely not needed.

[0195] Table 19. Simplified NNPF SEI messaging example

[0196]

[0197]

[0198] Given the above syntax, an example of how to apply the NNPF SEI message in Table 17 is as follows. Assuming that NNPF is used to improve visual quality, nnpf_purpose is set to 0. Considering that information related to the NNPF model needs to be transmitted by signal, nnpf_model_info_present_flag is set to 1. If different models are used for luma and chroma, nnpf_joint_model_flag is set to 0. Different models are applied to intra and inter pictures, so nnpf_num_pic_type_minus1 is set to 1. num_of_nnpf_models is set to 4 (luminance / chroma and inter / inter). Given these four models, the value of nnpf_model_id[0] is set to 0 for luma components and intra pictures; the value of nnpf_model_id[1] is set to 1 for chroma components and intra pictures; the value of nnpf_model_id[2] is set to 2 for luma components and inter pictures; the value of nnpf_model_id[3] is set to 3 for chroma components and inter pictures. The number of checkpoints provided for each model is set to 1, so num_of_ckpts_minus1[0] / [1] / [2] / [3] are all set to 0. An external web link can be provided for both model IDs. Therefore nnpf_model_exter_link_flag[0] / [1] is set to 1. The web link is encoded using IETF Internet Standard 66. For all models, Pytorch is used, so nnpf_model_storage_form_idc[0] / [1] is set to 3. To indicate model complexity, nnpf_model_complexity_ind_present_flag[0] / [1] is set to 1. The model uses single precision floating point format. The value of nnpf_param_prec_idc[0] / [1] is set to 4. The number of model parameters per id is 214k = 1.6327*2^17. Therefore, the value of log2_nnpf_num_param_minus11[0] / [1] is set to 6. log2_prec_denom[0] / [1] is set to 5 and nnpf_num_param_frac[0] / [1] is set to 21. Therefore the maximum number of parameters is set to 217k. The number of operations in kMacs / pixel is 33.6k. The value of nnpf_num_op[0] / [1] is set to 34.

[0199] Continuing with signal data formation information, nnpf_data_info_present_flag is set to 1. The input and output of NNPF are both YUV420, so nnpf_chroma_format_idc is set to 1 (420 format). vui_matrix_coeffs is set to 1 or 9 (YUV). Since separate models are used for the luma and chroma components, nnpf_joint_model_flag is 0, so there is no need to transmit packing_format_idc with the signal. The chroma model also uses luma information, so chroma_luma_dependency_flag is set to 1. The patch size is 128, so the value of log2_patch_size_minus6 is set to 1. Assuming the picture size is 4k, padding needs to be added. For copy padding, the value of picture_padding_type is set to 1. Since deblocking is used in the bitstream, patch overlap is not used. For auxiliary input information, the QP map is used and the value of nnpf_auxi_input_id is set to 1.

[0200] The picture-level NNPF SEI messaging of Table 18 may require region-level metadata that may be too large or rarely used in many applications. To reduce the overall payload size and focus on the QP mapping SEI information, an example of an alternative simplified picture-level NNPF SEI message is illustrated in Table 20. To generate this syntax, some previously defined parameters are deleted, as will be explained below.

[0201] Table 20. Simplified NNFP picture layer SEI messaging example

[0202]

[0203]

[0204] in:

[0205] nnpf_pic_model_id_chroma specifies the index of the model used for the chrominance components of the current picture. For this version of this specification, the value of nnpf_pic_model_id_chroma shall be in the range of 0 to nnpfc_max_num_models, inclusive. When not present, the value of nnpf_pic_model_id_chroma is inferred to be equal to nnpf_pic_model_id.

[0206] nnpf_pic_ckpt_idx_chroma specifies the index of the checkpoint of the model used for the chroma components of the current picture. The value of nnpf_pic_ckpt_idx_chroma shall be in the range of 0 to nnpfc_max_num_ckpts_minus1[nnpf_mode_id_chroma], inclusive. When not present, the value of nnpf_pic_ckpt_idx_chroma is inferred to be equal to nnpf_pic_ckpt_idx.

[0207] Parameters related to region-level messaging are all removed and the redundancy created by said parameters is eliminated. More specifically, nnpf_region_info_present_flag is considered unnecessary and redundant due to the use of nnpf_qp_info_present_flag. Similarly, nnpf_region_ptt_present_flag, ptt_map and clfc_map are not required if region-level partitioning is not available.

[0208] Presence of auxiliary data in neural network tensors

[0209] In reference

[21] , auxiliary input data is present in the neural network input tensor only when the value of nnpfc_inp_order_idc is equal to 3, that is, when the input tensor is configured as four interleaved luma channels and two chroma channels. Currently, auxiliary input data cannot be present in input tensors configured as luma-only, chroma-only, and 3-channel luma and chroma, that is, nnpfc_inp_order_idc is equal to 0, 1, and 2, respectively. It is claimed that auxiliary input data may be beneficial for all input tensor configurations.

[0210] As mentioned previously (e.g., see Table 10 and the syntax parameter nnpf_auxi_input_id), it is proposed to add the syntax element nnpfc_auxiliary_input_idc and the corresponding semantics in the NNPF CLVSSEI message, which is denoted as NNPFC SEI in reference

[21] , so that for each allowed configuration of the input tensor (i.e., each value of nnpfc_inp_order_idc), auxiliary data can be present in the input tensor. As in the current draft of the VSEI amendment in reference

[21] , it is proposed to restrict the auxiliary input data to be from the luma quantization parameter SliceQp Y The derived signal. The parameter nnpfc_auxiliary_input_idc was also proposed in reference

[22] .

[0211] Color description of neural network tensors

[0212] It is not possible to use the current text of reference

[21] to signal the color description information of the neural network tensor. It is claimed that the color description information of the neural network tensor may be beneficial. For example, when applying a neural network post-filter to an HDR WCG signal, IC T C P May be preferred.

[0213] It is proposed to add syntax elements nnpfc_separate_colour_description_present_flag, nnpfc_colour_primaries, nnpfc_transfer_characteristic and nnpfc_matrix_coeffs and corresponding semantics in the NNPFC SEI message. It is proposed that these syntax and semantics be modeled on the syntax and semantics of the film grain characteristics SEI message.

[0214] Additionally, when nnpfc_matrix_coeffs is equal to 0 (which is typically used for GBR (RGB) and YZX 4:4:4 chroma formats), the following constraints are imposed on nnpfc_purpose, nnpfc_inp_order_idc, and nnpfc_out_order_idc:

[0215] 1.nnpfc_purpose must not be equal to 2 (chroma upsampled to 4:4:4 chroma format) or 4 (increase the width or height of the cropped decoded output picture and upsample the chroma format)

[0216] 2.nnpfc_inp_order_idc must not be equal to 1 (there are two chroma channels and no luma channels in the input tensor) or 3 (there are four interleaved luma channels and two chroma channels in the input tensor)

[0217] 3.nnpfc_out_order_idc must not be equal to 1 (only two chroma channels in the output tensor) or 3 (there are four interleaved luma channels and two chroma channels in the output tensor)

[0218] Indicates dependencies between multiple activations of neural network post filters

[0219] It is claimed that when multiple neural network post filters are activated for the current picture, it may be beneficial to apply the neural network post filters in a specific order. For example, the output tensor of only the luma neural network post filter can be used to derive the input tensor of the luma chroma neural network post filters. As another example, the output tensor of a neural network post filter used to increase the width or height of a decoded picture (nnpfc_purpose equals 2, 3, or 4) can be used to derive the input tensor of a neural network post filter to improve video quality (nnpfc_purpose equals 1).

[0220] It is proposed to add three syntax elements and corresponding semantics in the NNPFA SEI message, as follows:

[0221] 1.nnpfa_independent_flag indicates the preference that the neural network post-filter signaled in the SEI is either independent of other neural network post-filters that may also be used for the current picture, or dependent on the output of one or more such neural network post-filters

[0222] 2.nnpfa_num_dependencies_minus1 indicates the number of neural network post filters that the current neural network post filter may depend on

[0223] 3.nnpfa_dependency_nnpfa_id[i] specifies the identification number nnpfa_id of the i-th neural network post-processing filter that the current neural network filter may depend on

[0224] Neural Network Post Filter Characteristic (NNPFC) SEI Message

[0225] In view of these proposed new syntax elements, the following table presents the revised NNPF CLVS or NNPFC SEI message. Changes compared to reference

[21] are indicated in italic font.

[0226] Table 21. NNPFC SEI message syntax modification example

[0227]

[0228]

[0229]

[0230] Neural network post-filter characteristics SEI message semantics

[0231] Compared with the original text and semantics of NNPFC, the following revisions are proposed.

[0232] This SEI message specifies a neural network that can be used as a post-processing filter. The neural network post-filter activation SEI message indicates the use of a specified post-processing filter for a particular picture.

[0233] The following variables need to be defined to use this SEI message:

[0234] - The width and height of the cropped decoded output picture, in units of luma samples, denoted in this document as InpPicWidthInLumaSamples and InpPicHeightInLumaSamples, respectively.

[0235] - The luma sample array CroppedYPic[y][x] and the chroma sample arrays CroppedCbPic[y][x] and CroppedCrPic[y][x] (if any) at vertical coordinate y and horizontal coordinate x of the cropped decoded output picture, where the coordinate y of the upper left corner of the sample array is equal to 0 and x is equal to 0.

[0236] - Bit depth BitDepthY of the luma sample array of the cropped decoded output picture.

[0237] - The bit depth BitDepthC of the chroma sample array (if any) of the cropped decoded output picture.

[0238] - Chroma subsampling ratio relative to luma, expressed as InpSubWidthC and InpSubHeightC.

[0239]

[0240] When this SEI message specifies a neural network that can be used as a post-processing filter, these semantics specify the derivation of the luma sample array FilteredYPic[y][x] and the chroma sample arrays FilteredCbPic[y][x] and FilteredCrPic[y][x], as indicated by the value of nnpfc_out_order_idc, which contain the outputs of the post-processing filters.

[0241]

[0242]

[0243] Table 22. Proposed amendment to Table 21 of reference

[21] - Illustrative description of nnpfc_inp_order_idc values

[0244]

[0245] Due to the proposed new syntax, Table 23 in reference

[21] can be updated as follows.

[0246] Table 23. Modified example of Table 23 in reference

[21] : For the upper left corner of the sample patch included in the specified input tensor

[0247] The input tensor is derived by given the vertical sample point coordinates cTop and the horizontal sample point coordinates cLeft of the corner sample point positions.

[0248] InputTensor process

[0249]

[0250]

[0251]

[0252] Neural Network Post Filter Activation (NNPFA) SEI Message

[0253] In reference

[21] , the picture layer NNPF message is denoted as NNPFA SEI message. The proposed modifications to the existing syntax are indicated in italics in Table 24.

[0254] Neural Network Post-Filter Activation SEI Message Syntax

[0255] Table 24. Proposed modifications to the NNPFA SEI messaging in

[21]

[0256]

[0257] Neural network post-filter activation SEI message semantics

[0258] The SEI message specifies the neural network post-processing filters that can be used for post-processing filtering of the current picture, and conveys information about the dependencies of other neural network post-filters (if any) that may exist for the current picture.

[0259] Neural network post-processing filter activation SEI messages persist only for the current picture.

[0260] NOTE - There may be multiple neural network post-processing filter activation SEI messages for the same image, for example, when the post-processing filters serve different purposes or filter different color components.

[0261] nnpfa_id specifies that the neural network post-processing filters specified by one or more neural network post-processing filter characteristics SEI messages belonging to the current picture and with nnpfc_id equal to nnfpa_id can be used for post-processing filtering of the current picture.

[0262]

[0263]

[0264] Figure 5 An example of data flow for processing CLVS layer NNPF SEI message delivery is depicted. The data flow follows the syntax of Table 11. For the picture layer NNPF SEI message depicted in Table 16, its corresponding data flow processing example is shown in Figure 6.

[0265] As discussed in reference

[19] , in some applications it may be necessary to define a priority order for executing multiple SEI messages. For example, when considering the SEI messages for FGC (Film Grain Characteristics) and CTI (Color Transform Information), priority is very important. In HEVC and AVC, the Post Filter Hint, Tone Mapping Information, and Chroma Resampling Filter Hint SEI messages are additional examples of SEI messages that need to be considered to define their processing order. The processing order of NNPF SEI messaging should also be considered. The specific order needs to be determined based on the user's situation and can be transmitted with the bitstream as recommended in the Proposed Processing Order SEI (reference

[19] ). For example, assume that the bitstream carries SDR (Standard Dynamic Range) video and FGC, CTI, and NNPF SEI messaging, where the CTI SEI is used to convert the SDR video to HDR video and the NNPF SEI is used to improve the quality of the SDR decoded video. In an embodiment, the proposed order may be: NNPF SEI first (improves the quality of the decoded video), followed by CTI SEI (converts SDR to HDR), and finally FGC SEI (adds film grain effect for final display). For example, if applied earlier, the added film grain noise may be amplified during the SDR to HDR conversion process.

[0266] References

[0267] Each of the references listed herein is incorporated by reference in its entirety. The term JVET refers to the Joint Video Experts Group of ITU-TSG 16 WP 3 and ISO / IEC JTC 1 / SC 29.

[0268] [1] Advanced Video Coding, Rec. ITU-T H.264, May 2019.

[0269] [2] High Efficiency Video Coding, Rec. ITU-T H.265, November 2019.

[0270] [3]Versatile Video Coding, Rec. ITU-T H.266, August 2020.

[0271] [4] M. Hannuksela, M. Santamaria, F. Cricri, E. B. Aksu, H. R. Tavakoli, “AHG9: On post-filter SEI,” JVET-Y0115, online conference, January 2022

[0272] [5] M.M. Hannuksela, E.B. Aksu, F. Cricri, H.R. Tavalkoli, and M. Santamaria, “AHG9: On post-filter SEI,” JVET-X0112, online conference, October 2021.

[0273] [6] M.M. Hannuksela, E.B. Aksu, F. Cricri, H.R. Tavalkoli, “AHG9: On post-filter SEI,” JVET-V0058, online conference, April 2021.

[0274] [7] T. Chujoh, Y. Yasugi, K. Takada, T. Ikai, “AHG9: Colour component description for post-filter purpose SEI message,” JAN-Y0073, online conference, January 2022

[0275] [8] Y. Yasugi, T. Chujoh, K. Takada, T. Ikai, “AHG9: Data conversion description for NNR post-filter SEI message,” JVET-Y0074, online conference, January 2022.

[0276] [9] K. Takada, Y. Yasugi, T. Chujoh, T. Ikai, “AHG9: Complexity description for NNR post-filter SEI message,” JVET-Y0075, online conference, January 2022

[0277]

[10] B. Choi, Z. Li, W. Wang, W. Jiang, Xu, S. Wenger, and S. Liu, “AHG9 / AHG11: SEI messages for carriage of neural network information for post-filtering,” JVET-V0091, online conference, April 2021.

[0278]

[11] MPEG-7:Compression of Neural Networks for Multimedia Content Description and analysis:ISO / IEC 15938-17.

[0279]

[12] “White Paper on Neural Network Coding”, MPEG document N00057, ISO / IEC JTC 1 / SC 29 / WG 04, January 2022.

[0280]

[13] H. Kirchhoffer et al., “Overview of the Neural Network Compression and Representation (NNR) Standard,” IEEE Transactions on Circuits and Systems for Video Technology, doi:10.1109 / TCSVT.2021.3095970.

[0281]

[14] Maria Santamaria, Jani Lainema, Francesco Cricri, Ramin G. Youvalari, Zhang Honglei, Alireza Zare, Goutham Rangu, Hamed R. Tavakoli, Homayun Afrandpey, Miska Hannuksela, “AHG11: MPEG NNR compressed bias update for the CNN based post-filter of EE1-1.1,” JVET-X0111, October 2021.

[0282]

[15] Y. Li, K. Zhang, L. Zhang, H. Wang, J. Chen, K. Reuze, A. M. Kora, and M. Karczewicz, “EE1-1.6: Combined Test of EE1-1.2and EE1-1.4,” JVET-X0066, online conference, October 2021.

[0283]

[16] H. Wang, J. Chen, K. Reuzé, A. M. Kotra, and M. Karczewicz, “EE1-1.4: Tests on Neural Network-based In-Loop Filter with constrained computational complexity,” JVET-X0140, online conference, October 2021.

[0284]

[17] Y. Li, K. Zhang, L. Zhang, “AHG11: Deep In-Loop Filter with Adaptive Model Selection and External Attention,” JVET-W0100, online conference, July 2021.

[0285]

[18] L. Wang, X.,Xu, and S. Liu, “EE1-1.1: neural network based in-loop filter with constrained storage and low complexity,” JVET-Y0078, online conference, January 2022.

[0286]

[19] P. Yin et al., “Signaling of priority processing order for metadata messaging in video coding,” U.S. provisional patent application serial number 63 / 216,318, filed on June 29, 2021.

[0287]

[20] M.M. Hannuksela et al., “AHG9: NN post-filter SEI,” JVET-Z0244, online conference, April 20–29, 2022.

[0288]

[21] S. McCarthy et al., “Additional SEI messages for VSEI (Draft 1)”, JVET-Z2006, April 2022 online meeting output document, June 2022.

[0289]

[22] S. McCarthy et al., “AHG9: Neural-network post filtering SEI message,” JVET-Z0121, online conference, April 20–29, 2022.

[0290] Example Computer System Implementation

[0291] Embodiments of the present invention may be implemented using a computer system, a system configured with electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices, or components. The computer and / or IC may perform, control, or execute instructions related to transmitting neural network topologies and parameters associated with NNPF in image and video encoding, such as those described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to transmitting neural network topologies and parameters associated with NNPF in image and video encoding described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0292] Certain embodiments of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. can implement methods related to the neural network topology and parameters associated with the transmission and NNPF in image and video coding as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention can also be provided in the form of a program product. The program product may include any non-transient tangible medium that carries a set of computer-readable signals including instructions, which, when executed by a data processor, cause the data processor to perform the method of the present invention. The program product according to the present invention may take any of a variety of non-transient and tangible forms. The program product may include, for example, physical media, such as magnetic data storage media including floppy disks, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAMs, etc. The computer-readable signals on the program product may be optionally compressed or encrypted.

[0293] Where components (e.g., software modules, processors, components, devices, circuits, etc.) are mentioned above, unless otherwise indicated, references to the components (including references to "means") should be interpreted to include any components that perform the functions of the described components as equivalents of the components (e.g., functionally equivalent), including components that are not structurally equivalent to the disclosed structures that perform the functions in the illustrated example embodiments of the invention.

[0294] Equivalents, extensions, alternatives and miscellaneous

[0295] Thus, example embodiments involving transmission of neural network topologies and parameters associated with NNPFs in image and video coding are described. In the foregoing description, embodiments of the invention have been described with reference to many specific details that may vary from implementation to implementation. Therefore, the sole and exclusive indicator of the invention and the applicant's inventive intent is the set of claims issued in the specific form of this application, wherein such claim issuance includes any subsequent corrections. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Therefore, limitations, elements, properties, features, advantages or attributes that are not expressly recited in a claim should not limit the scope of such claim in any way. Therefore, the specification and drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A method for processing one or more pictures in a coded video sequence using a neural network post-filter (NNPF), the method comprising: Receiving a decoded image and NNPF metadata associated with processing the decoded image using the NNPF; Parsing syntax parameters in the NNPF metadata to perform NNPF according to one or more neural network models, associated NNPF data, and NNPF parameters; as well as NNPF is performed on the decoded image according to the syntax parameters to generate an output image, wherein the syntax parameters in the NNPF metadata include a first set of NNPF message passing parameters that persist until the decoding of the encoded video sequence is completed and a second set of NNPF message passing parameters that persist until the NN post-filtering of the decoded image is completed.

2. The method of claim 1, wherein: The first set of NNPF messaging parameters includes one or more of the following: NNPF model information existence flag, indicating that NNPF model information exists in the NNPF metadata; NNPF joint model flag (nnpf_joint_model_flag), which indicates whether NNPF applies the same neural network model to the luma component and the chroma component; NNPF picture type number parameter (nnpf_num_pic_type_minus1), indicating the number of different picture types supported by NNPF; NNPF model ID array (nnpf_model_id[i]), used to identify each NNPF model; The first parameter is related to the neural network topology and model information; a second parameter related to data information in the decoded image; and The third parameter is related to the NNPF auxiliary information.

3. The method of claim 1, wherein: The first parameters related to the neural network topology and model information include one or more of the following: Flag indicating whether to use external links to provide detailed information about the NN models used in NNPF; NNPF storage and exchange data format parameters; NNPF arithmetic precision parameter; NNPF model quantity parameters; as well as NNPF delay estimation parameters.

4. The method of claim 1, wherein: The second parameter related to the data information in the decoded image includes one or more of the following: Input chroma format parameters; Packing format parameters; Chroma-dependent format parameters; Input tensor format parameters; Image fill parameters; and Temporal picture flag indicating the presence of temporal neighboring pictures as auxiliary input.

5. The method of claim 4, wherein: The picture filling parameters include: 0, indicating zero padding; 1, for copy fill; and 2, indicating reflective fill.

6. The method of claim 4, wherein: The second parameter related to the data information further includes one or more of the following items: a flag indicating whether auxiliary input data exists in the input tensor of the NNPF metadata; and Flag indicating that there are different combinations of color primaries, transfer characteristics, and matrix coefficients for NNPF metadata.

7. The method of claim 1, wherein: The third parameter associated with the NNPF assistance information includes a NNPF auxiliary input identifier indicating availability of an auxiliary input, the auxiliary input including one or more of the following: QP graph; Zoning map; and Classification chart.

8. The method of claim 1, wherein: The second set of NNPF messaging parameters includes a NNPF picture model ID specifying a NN post-filter to be used for the decoded image.

9. The method of claim 8, wherein: The second set of NNPF messaging parameters further includes one or more of the following: Image QP related metadata; Metadata related to image partitions; Metadata related to image classification; a dependency flag indicating whether the signaled NN post-filter is independent of or dependent on other NN post-filters, and if the dependency flag indicates dependence on other NN post-filters, further comprising: A number-before variable indicating how many NN post-filters should precede the current NNPF specified by the picture-level NNPF identifier variable in the processing order; An array of NNPF identifier variables of the NN post filters that should precede the current NNPF in the processing order.

10. The method of claim 9, wherein: The picture QP related metadata includes one or more of the following: NNPF QP information existence flag indicating the existence of QP information; NNPF area information flag indicating the presence of area information; an NNPF region QP existence flag indicating the existence of region-based QP information; and if the NNPF QP information existence flag is set, further including QP information of at least one region.

11. The method of claim 9, wherein: The image partition related metadata includes: an NNPF region partition existence flag indicating the existence of NNPF region partition information; and if the NNPF region partition existence flag is set, further including at least one picture partition map.

12. The method of claim 9, wherein: The image classification related metadata includes one or more of the following: an NNPF picture classification existence flag indicating the existence of picture classification information; and if the NNPF picture classification existence flag is set, further including a picture classification of at least one area.

13. A method for encoding an image or an encoded video sequence using a processor, the method comprising: receiving an image or a video sequence containing images; encoding the image or the video sequence into a coded bit stream; as well as generating neural network post-filtering (NNPF) metadata to allow a decoder of the encoded bitstream to perform NNPF according to one or more neural network models, associated NNPF data, and NNPF parameters; as well as Generate an output comprising the encoded bitstream and the NNPF metadata, wherein the syntax parameters in the NNPF metadata include a first set of NNPF message passing parameters that persist until the decoding of the encoded video sequence is completed and a second set of NNPF message passing parameters that persist until the NN post-filtering of a single decoded image is completed.

14. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for executing the method of any one of claims 1 to 13 using one or more processors.

15. An apparatus comprising a processor and configured to perform any of the methods of claims 1 to 13.

Images