Messaging Parameters for Neural Network Post-Filtering in Image and Video Coding

Abstract

A method, system and bitstream syntax for conveying neural network topology and parameters related to Neural Network Based Post Filtering (NNPF) in image and video coding is described. An example of NNPF SEI messaging applicable to the MPEG standard for coding video pictures is described at the sequence layer and picture layer.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 63 / 328,131, filed April 6, 2022, and U.S. Provisional Patent Application No. 63 / 354,549, filed June 22, 2022, the entire contents of each of which are incorporated by reference.

[0002] [Technical field] This document relates generally to image and video coding. More specifically, embodiments of the present invention relate to messaging information related to messaging parameters associated with neural network post-filtering in image and video coding. [Background technology]

[0003] In 2020, the MPEG group of the International Standardization Organization (ISO), in collaboration with the International Telecommunications Union (ITU), released the first version of the Versatile Video coding Standard (VVC), also known as H.266 (Reference [3]). More recently, the same group has been working on developing a next-generation coding standard that will provide improved coding performance over existing video coding techniques. As part of this work, coding techniques based on artificial intelligence and deep learning are also being considered. As used herein, the term "deep learning" refers to a neural network (NN) with at least three layers, preferably more than three layers.

[0004] Neural-networks post filtering (NNPF) and neural-networks loop filtering (NNLF) have been shown to improve coding efficiency in image and video coding. MPEG-7, part 17 (ISO / IEC 15938-17) (Reference

[11] ) describes methods for compression of the representation of neural networks, but they are quite inefficient under bit-rate constraints in image and video coding. As recognized by the inventors, improved techniques for the conveyance of neural network topology and parameters associated with NNPF in image and video coding are desired, and these are described herein.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art by virtue of their mere inclusion in the present section. Similarly, it should not be assumed, based on this section, that problems identified with one or more approaches have been recognized in any prior art, unless otherwise indicated. [Brief description of the drawings]

[0006] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements and in which: [Figure 1] 1 illustrates an exemplary processing pipeline for neural network post filtering (NNPF) according to an embodiment of the present invention. [Diagram 2] 1 illustrates an exemplary packing format for a luma channel in a YUV420 signal in accordance with an embodiment of the present invention. [Diagram 3] 1 shows an example of luma-chroma dependency. [Figure 4] 13 shows an example of frame zero padding. [Diagram 5] 1 illustrates an exemplary process for processing SEI messages for NNPF processing at a coded sequence layer in accordance with an embodiment of the present invention. [Figure 6A] 1 illustrates an exemplary process for processing SEI messages for NNPF processing at the picture layer, in accordance with an embodiment of the present invention. [Figure 6B] 1 illustrates an exemplary process for processing SEI messages for NNPF processing at the picture layer, in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Exemplary embodiments of conveying neural network topology and parameters associated with NNPF in image and video coding are described herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the present invention. However, it will be apparent that various embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid unnecessarily obscuring, obscuring, or obfuscating the embodiments of the present invention.

[0008] [overview] Exemplary embodiments described herein relate to conveying neural network topology and parameters associated with an NNPF in image and video coding. In an embodiment, a processor receives a decoded image and NNPF metadata associated with processing the decoded image with the NNPF. The processor comprises: Parsing syntax parameters in the NNPF metadata to execute the NNPF according to the one or more neural network models, the associated NNPF data, and the NNPF parameters; Executing the NNPF on the decoded image according to the syntax parameters to generate an output image, the syntax parameters in the NNPF metadata including a first set of NNPF messaging parameters that persist until the end of decoding of the coded video sequence and a second set of NNPF messaging parameters that persist until the end of NN post-filtering of the decoded image.

[0009] In another embodiment, a processor receives an image or video sequence including a picture. The processor comprises: Encoding an image or video sequence into a coded bitstream; generating neural network post-filtering (NNPF) metadata to enable a decoder of the coded bitstream to perform NNPF in accordance with the one or more neural network models, associated NNPF data, and NNPF parameters; Generate an output including a coded bitstream and NNPF metadata, where syntax parameters in the NNPF metadata include a first set of NNPF messaging parameters that persist until the end of decoding of the coded video sequence and a second set of NNPF messaging parameters that persist until the end of NN post-filtering of the single decoded image.

[0010] [An exemplary model for neural network post-filtering] FIG. 1 illustrates an exemplary process (100) for neural-network post filtering (NNPF) according to an embodiment. Given a decoded input 102, the NNPF pipeline includes pre-processing (130), the actual NNPF processing, and post-processing stages. The pre-processing stage (130) includes software / hardware initialization (105), data preparation (110), and loading of the NNPF model (115). The software / hardware initialization configures the receiver's computing environment, such as a graphical processing unit (GPU), and specified software libraries, such as Tensorflow, PyTorch, etc. After initialization, a ready-to-use computing platform is available. Data preparation (110) converts the decoded frames (102) into a format that can be directly processed by the corresponding NN model. For example, the decoded frames are typically partitioned into patches (rectangular image blocks), which are converted into the data input format of the NN model, such as YUV444, and organized into batches before input. Meanwhile, in step 115, a particular model is selected and loaded for use based on the picture type and other flags. The above three steps can be performed in parallel. The NNPF stage (120) performs the actual NN post-filtering operations (e.g., upscaling, filtering, etc.) based on the particular model, data and platform input from the pre-processing stage (130). Finally, in the post-processing stage, in step 125, the NNPF output (122) is converted to a data format suitable for display as output 127, while in step 130, the NNPF model may be unloaded so that the NNPF pipeline (100) is ready for other operations. It is noted that the process 100 can be easily extended to other NN-based post-processing such as super-resolution and denoising.

[0011] Metadata signaling for NNPF, e.g., via SEI messaging, has been proposed in several JVET conferences in the past (refs. [4-10]). Previous proposals have focused more on how to signal NN topology and NN parameters by carrying the Neural Network Compression and Representation (NNR) bitstream (refs. [11-13]) or by using external links such as Uniform Resource Identifiers (URIs) with syntax and semantics specified in IETF Internet Standard 66 (ref. [4]). Some of the proposals have also addressed issues related to NN input or output interfaces and NN complexity (refs. [7-9]).

[0012] Despite the use of compression, NNR bitstreams can still be quite large, thus impacting bandwidth utilization. Furthermore, when using NNR, the decoder must be capable of decoding in compliance with yet another standard. As recognized by the inventors, the NNPF metadata must be lightweight, yet provide the information necessary for the decoder to check whether NNPF is applicable, and if so, access the parameters necessary to perform the NNPF process (100) as described above.

[0013] Although neural nets may also be applied to loop filtering and other applications, the embodiments described herein focus, without limitation, on NNPF for two main reasons: 1) Since NNPF is decoupled from decompression, the implementation has more degrees of freedom and can be used for any image or video codec; 2) Since it is outside the coding loop (which typically includes transform processing, quantization and loop filtering (deblocking)), it does not require a fixed-point implementation to avoid drift problems; therefore, the floating-point implementation commonly used in NNs is applicable.

[0014] Since the NNPF is executed outside the decoding loop, the NNPF does not have the potential drift problem of the NNLF (loop filter) processing. With respect to the NNLF, if there is a bad filtering result for one frame or one block, which is possible because the NN may not be robust enough for all frame data, this will result in a low quality of the currently decoded frame, which may be used as a reference frame for the following frames. Thus, errors and artifacts may accumulate as drift phenomena and propagate to other frames. In another example, most NNs are implemented using floating point, which may have different results on different machines / platforms / operating systems. This may cause a mismatch between the encoder and the decoder for one frame, and if the mismatched frame is used as a reference, the errors may cause drift problems for the subsequent decoded frames.

[0015] Two levels of NNPF-related messaging are proposed: 1) the Coded Layer Video Sequence (CLVS) layer (NNPF operations persist until the end of the video sequence), and 2) the picture layer (NNPF operations persist only until the end of the current picture). This allows picture-by-picture NNPF messaging and filtering without repeating specific filter characteristics that are applied to the entire video sequence. The proposed messaging is described using notations and syntax commonly used to describe MPEG's SEI messaging (References [1-3]), but the proposed metadata messaging may be carried using a variety of other suitable messaging formats, e.g., as used in AV1 and other proprietary or standards-based coding formats. The proposed messaging is also applicable to other MPEG-based standards, such as AVC and HEVC. The proposed SEI messages help the NNPF to exploit coding characteristics by providing information that is not available to a standalone postfilter, thus further improving postfilter performance.

[0016] In an exemplary embodiment, the proposed CLVS NNPF SEI aims to provide information to support efficient implementation of the NNPF pipeline, such as initialization, pre-processing, model loading / unloading, and post-processing. The picture layer NNPF SEI aims to enable picture-level adaptation to further improve NNPF coding efficiency.

[0017] [NNPF SEI for CLVS layer] The scope of the NNPF SEI for a CLVS layer is for the entire coded sequence. Its purpose is to signal it together with the first picture of the CLVS and should not be changed throughout the CLVS. It should be able to assist the decoder to prepare to apply the NNPF to the decoded picture after bitstream decoding. More specifically, when an NNPF SEI message is present for any picture of the CLVS of a particular layer, the NNPF SEI message shall be present for the first picture of the CLVS. The NNPF SEI message persists for the current layer from the current picture in decoding order to the end of the CLVS. All NNPF SEI messages that apply to the same CLVS shall have the same content. In an exemplary embodiment, the CLVS NNPF SEI includes the following information:

[0018] [1] Network topology and model parameters For the NNPF SEI message, it is desirable to have the SEI message carry only the necessary information, so that the size of the SEI message is not too large. Otherwise, the encoder can simply reduce the quantization (QP) value to improve the quality of the coded sequence at the expense of a higher bit rate. The size of the detailed network topology (e.g., using a graph to describe the topology) and its corresponding parameter values ​​(weights and biases in the case of a convolutional neural network (CNN)) can be relatively large, e.g., in the range of kilobytes, megabytes, or even gigabytes. It is not practical to carry all the information in the SEI bitstream. Compression can be applied to the model (e.g., NNR in

[11] ), but the size is still not negligible. One way to signal detailed NN model information is to use explicit links or some external means, such as cross-references to URIs (IETF Internet Standard 66) as discussed in [4]. Another approach is to standardize a fixed model to zero or external reference links for the base model, and the bitstream carries only incremental information, such as updated biases or weights, for the complete NN or a small subset of the NN (Reference

[14] ).

[0019] In addition to the topology and model parameters, it is important to also inform the decoder of the following information, which can help it achieve fast initialization or quickly decide whether it can implement an NNPF or bypass it: -NN storage / exchange formats: currently the most popular include ONNX, NNEF, PyTorch and TensorFlow, but more formats can be added as needed. - NNPF computational and memory complexity indication: The most frequently used indicators are the NN parameter precision value, i.e. floating point (FP64, FP32 or FP16) and integer (e.g. INT8), and the number of NN model parameters, i.e. the number of multiply-and-accumulations (MACs) and floating-point operations per second (FLOPS) in units such as thousands (kMac / pixel) or millions (mMac / pixel). Note that the multiplication of the NN parameter precision and the number of NN parameters can give the memory size of the model. Other parameters such as latency, throughput, and power usage are also good indicators. The complexity indicator may help the decoder to skip or bypass the NNPF processing if there are not enough computing resources. - Number of models: The NN models can be different based on various parameters such as the signal coded in the bitstream, the QP value, slice / picture type, content type and device type. For example, if a GBR (RGB) signal is coded directly, a joint model is generally used. If a YUV signal is coded, it can have either a joint model (NPL

[16] ) or separate models (NPL

[15] ) for the Y and U / V components. The bitstream can contain different slice types such as intra (I) and inter (P / B) slices. It is possible to use the same model for each slice or to use different models for intra and inter slices (References [15-16, 18]). The sequences can be standard-dynamic range (SDR) or high dynamic range (HDR), natural content or screen-captured content (SCC), etc., and each such variant may also require different models. If the bitstream is decoded on various displays, the model may depend on the display type (e.g., TV or mobile device) to address the decoder computing capabilities or the perceived visual quality on the display. Different quality issues may also require different models, such as the QPvaried model

[17] .

[0020] As an example, Table 1 shows an example of syntax parameters of NNPF topology and model parameter information for a single model. The syntax includes NN topology and parameters of explicit links (if any) or updated parameters, NN storage and exchange format, and NN complexity index. Multiple models should loop over this SEI. Note that since multiple models most likely use the same storage and exchange format, an alternative solution is to move this information out and signal it only once in the core NNPF SEI message. [Table 1] nnpf_model_exter_link_flag equal to 1 indicates that the NNPF model is stored in the external link. nnpf_model_exter_link_flag equal to 0 indicates that the NNPF model is not stored in the external link. nnpf_exter_uri[i] contains the i-th byte of a null-terminated UTF-8 string indicating a URI (IETF Internet Standard 66) that specifies the neural network to be used as a post-processing filter. nnpf_model_upd_param_present_flag equal to 1 indicates that the model parameters are updated. nnpf_model_upd_param_present_flag equal to 0 indicates that the model parameters are not updated. NOTE: For further updated parameter syntax and semantics, see reference [4]. nnpf_model_storage_form_idc indicates the storage and exchange format for NNPF models as specified in Table 2. Values ​​0-3 correspond to ONNX, NNEF, Tensorflow, and PyTorch, respectively. Values ​​4-7 are reserved for future extensions. [Table 2] nnpf_model_complexity_ind_present_flag equal to 1 indicates that the model complexity indicator is present in the SEI message. When nnpf_model_complexity_ind_present_flag is equal to 0, the model complexity indicator is not present in the SEI message. The estimates of all the following syntax should be 0 unless otherwise specified. "0" can be interpreted in this context as "null" (meaning not present) or "negligible". nnpf_param_prec_idc indicates the NNPF model parameter precision as specified in Table 3. When not present, the syntax value of nnpf_param_prec_idc is inferred to be 5. [Table 3] Note: For many parameters, the following method may be used to represent them: c=1.a*2^b, where "a" represents the fractional part of 1 and "b" (an integer) is a power of 2 (e.g., if a=5 and b=2, then c=1.5*2^2=6). nnpf_num_param_frac is a fraction representing the total number of model parameters. log2_prec_denorm is the base 2 logarithm of the denominator of the fraction representing the total number of model parameters. log2_nnpf_num_param_minus11 plus 11 is the base 2 logarithm of the total number of model parameters. The variable tot_num_params is derived as follows: tot_num_params=(int64)(1.0+(float64)nnpf_num_param_frac / (float64)(1<<(log2_prec_denorm))*(float64)(1<<(log2_nnpf_num_param_minus11+11)) The total number of parameters for an NNPF model must not be greater than the value of tot_num_params. When the above three syntax elements are not present, the value of tot_num_params is inferred to be 0 for "NULL". nnpf_num_ops times 1,000 specifies the maximum number of MAC (multiply-accumulate) operations per pixel for the NNPF. NOTE: A more precise definition of this parameter could be to use 1.a*2^b as tot_num_params. nnpf_latency_idc specifies the latency indication of the NNPF model as specified in Table 4. It indicates the combination of resolution and frame rate that can be supported by the NNPF model to ensure real-time decoding and no delay in matching with the decoder, using an available baseline GPU (e.g., defined as an Nvidia RTX 1080Ti). [Table 4]

[0021] Note that the NN storage / exchange format or complexity indication can be generated by downloading the model and using a standalone analyzer. Therefore, a "present flag" such as nnpf_model_complexity_ind_present_flag is used to provide this option as a complexity indication.

[0022] [2) Input / output chrominance format and data format] The data input to the NN may differ from the decoded format. For the correct application of the NNPF, the following information may be included in the bitstream: - Input and output data formats -Accuracy of data -Tensor format -Input and output patch sizes, boundary overlap indication and overlap size, picture size and padding method. Note that in NN training, the input patch size is very important to ensure the generalization and robustness of the model. Video frames can have a wide range of resolutions, and therefore the scales of objects, textures and artifacts can be very different. Another efficient way other than including various patch sizes in training one model is to use different models to handle different patch sizes. Patch size is also one of the important factors that affect the training speed. Therefore, it is very important to indicate the patch size in the SEI. An example of SEI messaging data information is shown in Table 5. [Table 5] input_chroma_format_idc has the same semantics as specified for the syntax sps_chroma_format_idc. output_chroma_format_idc has the same semantics as specified for the syntax sps_chroma_format_idc. vui_matrix_coeffs has the same semantics as specified for the syntax vui_matrix_coeffs. packing_format_idc indicates the packing format for the luma channel as specified in Table 6. The purpose is to allow all input channels to have the same dimensions. Figure 2 shows the case where packing_format_idc is equal to 0 for the case of YUV420. In Figure 2, one luma channel / plane is interleaved into four luma channels to have the same dimensions as the chroma channels U and V. Thus, YUV420 becomes six channels. Similar packing is applied in the case of YUV422. YUV422 becomes four channels, one luma channel is interleaved into two luma channels to have the same dimensions as U and V. [Table 6] A chroma_luma_dependency_flag equal to 1 specifies that for a chroma NNPF model, the chroma channels depend on the luma channel for the input of the NNPF. A chroma_luma_dependency_flag equal to 0 specifies that the chroma channels are independent of the luma channel for the input of the NNPF. Figure 3 shows an example of the concept.

[0023] In the alternative, more cases may be supported. luma_chroma_dependency_idc specifies the luma and chroma dependencies for the luma and chroma model inputs as specified in Table 7. [Table 7] precision_format_idc has the same semantics as the syntax nnpf_param_prec_idc. tensor_format_idc indicates the tensor format of the input and output tensors as specified in Table 8. [Table 8-1] In Table 8, the variables N, C, H, and W represent the following: [Table 8-2] log2_patch_size_minus6 plus 6 specifies the base 2 logarithm of the luma patch size. The value of log2_patch_size_minus6 shall be in the range 0 to 6, inclusive. The variable PatchSize is defined as follows: PatchSize=1<<(log2_patch_size_minus6+6) NOTE: PatchSize indicates both the patch height and width. In alternative embodiments, patch width and patch height can be specified separately. picture_padding_type indicates the picture padding type as specified in Table 9. Figure 4 shows the case where picture_padding_type is set to 0. [Table 9]

[0024] When the picture width and height are not multiples of patchSize, padding is required based on picture_padd_type. Padding occurs at the bottom and / or right of the picture. The width and height of the decoded output picture are in units of luma samples and are denoted by PicWidthInLumaSamples and PicHeightInLumaSamples, respectively. The width and height of the filtered picture are in units of luma samples and are denoted by FilterPicWidthInLumaSamples and FilterPicHeightInLumaSamples, respectively. The derivation is as follows: FilterPicWidthInLumaSamples=PicWidthInLumaSamples+patchSize-(PicWidthInLumaSamples%patchSize) FilterPicHeightInLumaSamples=PicHeightInLumaSamples+patchSize-(PicHeightInLumaSamples%patchSize) patch_boundary_overlap_flag equal to 1 specifies that the patches overlap within the boundary. patch_boundary_overlap_flag equal to 0 specifies that the patches do not overlap at the boundary. log2_boundary_overlap_minus3 plus 3 specifies the base 2 logarithm of the boundary overlap between horizontal and vertical patches. The value of the boundary overlap in units of luma samples is derived to be equal to (1<<(log2_boundary_overlap_minus3+3). The value of log2_boundary_overlap_minus3 shall be in the range 0 to 2, inclusive. Note that the final input patch size to the NNPF is set equal to PatchSize+(patch_boundary_overlap_flag==0)?0:2*(1<<(log2_boundary_overlap_minus3+3)).

[0025] [3) Auxiliary input information hints] One advantage of using NNPF SEI messaging over a pure NNPF is that the NNPF SEI messaging is generated during encoding. This allows information related to bitstream characteristics such as QP information, picture / slice type information, partition information, inter / intra map information, classification information, and temporally adjacent pictures to be included in the SEI as input to the NNPF. To make the device ready for such auxiliary inputs, an auxiliary input information hint message can be indicated in the NNPF SEI of the CLVS layer and more detailed information can be carried in the picture layer SEI. An example of auxiliary input hint information is shown in Table 10. [Table 10] nnpf_auxi_input_id contains an identifier number that may be used to identify the possible presence of NNPF auxiliary input information. An nnpf_auxi_input_id equal to 0 infers that no auxiliary input is used for the NNPF in the CLVS. nnpf_auxi_input_id is interpreted as follows: - The variable QpFlag (bit 0) is set equal to (nnpf_auxi_input_id&0x01). QpFlag equal to 1 specifies that the QP map can be an auxiliary input of the NNPF for the current CLVS. QpFlag equal to 0 specifies that the QP map is not an auxiliary input of the NNPF for the current CLVS. (Note: "&" indicates bitwise AND.) - The variable PartitionFlag (bit 1) is set equal to ((nnpf_auxi_input_id&0x02)>>1). PartitionFlag equal to 1 specifies that the partition map can be an auxiliary input of the NNPF for the current CLVS. PartitionFlag equal to 0 specifies that the partition map is not an auxiliary input of the NNPF for the current CLVS. - The variable ClassificationFlag (bit 2) is set equal to ((nnpf_auxi_input_id&0x04)>>2). ClassificationFlag equal to 1 specifies that the classification map can be an auxiliary input of the NNPF for the current CLVS. ClassificationFlag equal to 0 specifies that the classification map is not an auxiliary input of the NNPF for the current CLVS. - The variable TemporalPicFlag (bit 3) is set equal to ((nnpf_auxi_input_id & 0x08) >> 3). TemporalPicFlag equal to 1 specifies that the temporally adjacent picture can be an auxiliary input of the NNPF for the current CLVS. TemporalPicFlag equal to 0 specifies that the temporally adjacent picture is not an auxiliary input of the NNPF for the current CLVS. - The remaining bits (bits 4 to 7) are reserved for future use by ITU-T | ISO / IEC. An example of the NNPF SEI for the CLVS layer is shown in Table 11. Semantics follow the syntax table. In this example, several NNPF models are looped over picture types and device types. [Table 11] nnpf_purpose indicates the purpose of the post-processing filter as specified in Table 12. The value of nnpf_purpose must be between 0 and 2. 32 nnpf_purpose SHALL be in the range -2 or less. Values ​​of nnpf_purpose not appearing in Table 12 are reserved for future specification by ITU-T|ISO / IEC and SHALL NOT be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification SHALL ignore SEI messages containing reserved values ​​of nnpf_purpose (Reference [4]). [Table 12] NOTE – When reserved values ​​of nnrpf_purpose are used in the future by ITU-T|ISO / IEC, the syntax of this SEI message may be extended with syntax elements whose presence is conditioned by nnrpf_purpose equal to that value. The syntax and semantics of nnpf_purpose can be taken from [4]. The tolerance range may be too large for postfilter purposes in some cases. nnpf_model_info_present_flag equal to 1 specifies that the nnpf model information is present in the SEI message. nnpf_model_info_present_flag equal to 0 specifies that the nnpf model information is not present in the SEI message. NOTE – When the nnpf model information is not present in the SEI message, the NNPF model should be accessed by some other means not specified in this specification. nnpf_joint_model_flag equal to 1 specifies that the NNPF uses the same model for all color components. nnpf_joint_model_flag equal to 0 specifies that the NNPF uses separate models for the luma and chroma components. When not present, the value of nnpf_joint_model_flag is inferred to be equal to 0. Note: When nnpf_joint_model_flag is equal to 0, the external link should contain one model for the luma component and one model for the chroma component.

[0026] Note that when counting the number of models, in one embodiment, one model can be counted for both luma and chroma components. Thus, even if luma and chroma use separate models, they are counted as one model because only one picture with both luma and chroma components can be completed by using both models. Thus, num_nnpf_models=(nnpf_num_pic_type_minus1+1)*(nnpf_num_device_type_minus1+1). In another embodiment, luma and chroma component models can be counted as individual counts. Thus, if luma and chroma components use separate models, they are counted as two models. Thus, num_nnpf_models=(nnpf_joint_model_flag==0?2:1)*(nnpf_num_pic_type_minus1+1)*(nnpf_num_device_type_minus1+1). In Table 11, the latter method is used. nnpf_num_pic_type_minus1 plus 1 indicates the number of picture types supported in the nnpf picture type based model. When not present, the value of nnpf_num_pic_type_minus1 is inferred to be equal to 0. The value shall be in the range 0 to 3, inclusive. nnpf_num_device_type_minus1 plus 1 indicates the number of device types supported in the nnpf device type based model. When not present, the value of nnpf_num_device_type_minus1 is inferred to be equal to 0. The value shall be in the range 0 to 15, inclusive. nnpf_mode_id[i] contains an identifier number that may be used to identify the i-th NNPF model. When not present, the value of nnpf_mode_id is inferred to be equal to 0. The value of nnpf_mode_id[i] shall be in the range 0 to 255, inclusive. nnpf_model_id is interpreted as follows: The variable CompType (bit 0) is set equal to (nnpf_model_id[i] & 0x01) as specified in Table 13. The variable PicType (bit 1) is set equal to ((nnpf_model_id[i]&0x02)>>1), as specified in Table 14. - The variable DeviceType (bits 2, 3, 4, 5) is set equal to ((nnpf_model_id[i]&0x1C)>>2). The variable displayType is set equal to (DeviceType&0x03) as specified in Table 15. The display types are arranged in ascending order based on display size. The variable complexityType is set equal to ((DeviceType&0x0C)>>2) as specified in Table 16. The complexityType are arranged in ascending order based on complexity. [Table 13] [Table 14]

[0027] Note: A picture in VVC can contain multiple slices that may have different slice types. Since the SEI is defined at the picture layer, the encoder can determine which PicType the current picture belongs to for such a picture with mixed slice types. For example, if more than a certain percentage of blocks in a picture are coded with the intra model, the picture can be considered an intra picture. [Table 15]

[0028] In another example, a QualityType indication can be added, so that different decoding qualities can use different models. The quality can be determined by the picture level QP. - The variable QualityType (bits 6 and 7) is set equal to ((nnpf_model_id[i]&0xC0)>>6). The QualityTypes are listed in descending order, with 0 meaning the best quality and 3 meaning worse quality. Note: The association between QualityType and base QP information can also be defined. An example is shown in Table 16. [Table 16] num_of_ckpts_minus1[nnpf_model_id[i]] plus 1 specifies the number of checkpoints for nnpf_model_id[i]. The indices of each checkpoint are in ascending order from 0 to num_of_ckpts_minus1[nnpf_model_id[i]] inclusive. In the NN literature, checkpoints (ckpts) are used to save model parameters such as weights and biases in CNNs. In this application, a ckpt means that the same model topology is used. The difference between ckpts is the value of the model parameter. nnpf_data_info_present_flag equal to 1 indicates that nnpf_data_info() is present in the SEI message. nnpf_data_info_present_flag equal to 0 indicates that nnpf_data_info() is not present in the SEI message. In the alternative, nnpf_data_info() and nnpf_auxiliary_input_info() can be tied to nnpf_model_id for greater flexibility. [Table 17] Note that in another embodiment, one can simply specify the number of nnpf models using the syntax num_nnpf_models_minus1 and assign index i to nnpf_mode_id[i]. The drawback of this method is that nnpf_model_id[i] has no specific meaning and the decoder blindly uses the nnpf models. The advantage is that the bitstream can carry as many models as one likes. Furthermore, there is no need to strictly distinguish checkpoints from models. For example, the bitstream can carry two different checkpoints for the same picture type, even if only one checkpoint is used for any given picture. num_nnpf_models_minus1 plus 1 specifies the number of NNPF models. The model indices are in ascending order from 0 to num_nnpf_models_minus1 inclusive.

[0029] [NNPF SEI of picture layer] One advantage of using the picture layer NNPF SEI (denoted as nnpf_pic_adapt_SEI()) instead of a standalone NNPF is that the SEI can carry adaptation information on a per-picture basis. The information can include parameters such as NNPF on / off flags for picture layer, luma / chroma components and CTU layer, picture / slice type, picture / slice QP, block level QP, picture / slice / block level classification, picture / slice level inter / intra map, etc. To save bits overhead, nnpf_pic_adapt_SEI() can reference the CLVS-level nnpf_sei() for high-level control. The persistence scope of nnpf_pic_adapt_SEI() is for the current picture.

[0030] Regarding signaling nnpf_pic_model_id, several methods can be used in Table 11. 1) The nnpf_pic_model_id from nnpf_sei() can be explicitly signaled in nnpf_pic_adapt_SEI() at the expense of the ue(v) bit. This explicit model is the base model. Bit 0 should always be 0 to indicate that nnpf_pic_model_id represents a luma model. The base model can indicate PicType, DeviceType, or QualityType. If the model has a deviceType option, the user can select other models based on the displayType and complexityType. 2) The nnpf_pic_model_id is inferred accordingly from other syntax in nnpf_pic_adapt_SEI. If the model has a deviceType option, the user can select the correct model based on the displayType and complexityType. If an implicit model is used, nnpf_pic_type needs to be signaled to select a model from the pool.

[0031] The further nnpf_pic_mode_id_chroma for the chroma components can be determined based on nnpf_joint_model_flag, which is derived as follows: If nnpf_joint_model_flag==0, nnpf_pic_mode_id_chroma=nnpf_pic_mode_id+1 If not, nnpf_pic_mode_id_chroma=nnpf_pic_mode_id In Table 17, if nnpf_joint_model_flag is equal to 0, nnpf_pic_mode_id and nnpf_pic_model_id_chroma can only be explicitly signaled.

[0032] For region-related information, the region size can be implied to be the same as PatchSize in nnpf_sei(), or it can be explicitly signaled if the size is different from PatchSize. Region size should generally not be smaller than PatchSize, and in some cases is best to be a multiple of PatchSize. For QP maps, classification maps, or partition maps in the regions used to generate auxiliary input, smaller units can be used, but the tradeoff between precision and bit overhead must be considered.

[0033] The auxiliary input information should be generated by either picture level information or region level information. For example, the QP map can be generated using picture level QP or region-based QP information. The classification map can be generated using region-based inter / intra information. The partition map can be generated using region-based partition information.

[0034] Table 18 shows an example of nnpf_pic_adapt_SEI(). In this example, for simplicity, we send the corresponding nnpf_mode_id directly. This allows to switch on / off the picture level and CTU level. The region size is assumed to be the same as the patchSize defined in nnpf_SEI(). [Table 18] nnpf_pic_enabled_flag equal to 1 specifies that nnpf applies to the current picture. nnpf_pic_enabled_flag equal to 0 specifies that nnpf does not apply to the current picture. When not present, the value of nnpf_pic_enabled_flag is inferred to be equal to 0. nnpf_pic_luma_enabled_flag equal to 1 specifies that nnpf applies to the luma component of the current picture. nnpf_pic_luma_enabled_flag equal to 0 specifies that nnpf does not apply to the luma component of the current picture. When not present, the value of nnpf_pic_luma_enabled_flag is inferred to be equal to 0. nnpf_pic_chroma_enabled_flag equal to 1 specifies that nnpf applies to the chroma components of the current picture. nnpf_pic_chroma_enabled_flag equal to 0 specifies that nnpf does not apply to the chroma components of the current picture. When not present, the value of nnpf_pic_chroma_enabled_flag is inferred to be equal to 0. nnpf_pic_model_id specifies the nnpf_mode_id used for the current picture. nnpf_pic_ckpt_idx specifies the checkpoint index used for nnpf_pic_model_id. The value of nnpf_pic_ckpt_idx is in the range 0 to num_ckpts_minus1[nnpf_pic_model_id] inclusive. nnpf_qp_info_present_flag equal to 1 specifies that the current SEI contains QP information. nnpf_qp_info_present_flag equal to 0 specifies that the current SEI does not contain QP information. When not present, the value of nnpf_qp_info_present_flag is inferred to be equal to 0. nnpf_region_info_present_flag equal to 1 specifies that the current SEI contains region information. nnpf_region_info_present_flag equal to 0 specifies that the current SEI does not contain region information. When not present, the value of nnpf_region_info_present_flag is inferred to be equal to 0. nnpf_region_qp_present_flag equal to 1 specifies that the current SEI contains region-based QP information. nnpf_region_qp_present_flag equal to 0 specifies that the current SEI does not contain region-based QP information. When not present, the value of nnpf_region_qp_present_flag is inferred to be equal to 0. nnpf_region_ptt_present_flag equal to 1 specifies that the current SEI contains region-based partition information. nnpf_region_ptt_present_flag equal to 0 specifies that the current SEI does not contain region-based partition information. When not present, the value of nnpf_region_ptt_present_flag is inferred to be equal to 0. nnpf_region_clfc_present_flag equal to 1 specifies that the current SEI contains region-based classification information. nnpf_region_clfc_present_flag equal to 0 specifies that the current SEI does not contain region-based classification information. When not present, the value of nnpf_region_clfc_present_flag is inferred to be equal to 0. Note: nnpf_region_qp / ptt / clfc / _present_flag can also be implicitly inferred from nnpf_pic_model_id, e.g., the region level information is only needed when picType=Intra. nnpf_region_enabled_flag[i] equal to 1 specifies that nnpf is enabled for the i-th region. nnpf_region_enabled_flag[i] equal to 0 specifies that nnpf is not enabled for the i-th region. When not present, the value of nnpf_region_enabled_flag[i] is inferred to be equal to 0. qp_delta_abs_map[i] has the same semantics as specified for cu_qp_delta_abs. qp_delta_sign_map_flag[i] has the same semantics as specified for cu_qp_delta_sign_flag. ptt_map[i] specifies the partition map for the i-th region. The partition map is represented using the same interpretation as MaxMttDepthY. This value is in the range 0 to log2(PatchSize)-3 inclusive. clfc_map[i] specifies the classification map for the i-th region. In one example, the classification map only indicates intra or inter. -If PicType is intra, clfc_map[i] equal to 0 specifies that the classification map for the i-th region is intra, clfc_map[i] equal to 1 specifies that the classification map for the i-th region is inter without residual, otherwise clfc_map[i] equal to 2 specifies that the classification map for the i-th region is inter with residual. - Otherwise, clfc_map[i] equal to 0 specifies that the classification map for the i-th region is inter without residual, clfc_map[i] equal to 1 specifies that the classification map for the i-th region is inter with residual, else clfc_map[i] equal to 0 specifies that the classification map for the i-th region is intra.

[0035] The CLVS layer NNPF SEI messaging in Table 11 (which may be loaded with data as defined in Tables 1-15) may require metadata information that may be too large or deemed unnecessary in some applications. To reduce payload size, an example of an alternative simplified CLVS NNPF SEI message is shown in Table 19. To generate the syntax of Table 19, some of the previously defined parameters are removed as described below.

[0036] The parameter nnpf_num_device_type_minus1 is skipped due to lack of experimental support for NNPF across multiple devices. The parameter nnpf_model_upd_param_present_flag is skipped because it is from reference [4] and has no demonstrated need. The parameter nnpf_latency_idc is skipped because it would also require testing under too many different resolution and frame rate configurations. Even if results are available, they may only be based on the baseline GPU. In practice, devices may use different GPU architectures that make this metric less accurate or useful. Since it is considered unlikely that the input and output of an NNPF will have different chroma formats in practice, the parameters input_chroma_format_idc and output_chroma_format_idc are merged into one nnpf_chroma_format_idc. The parameter precision_format_idc is skipped because its function of indicating precision may be considered to overlap with the previously defined nnpf_param_prec_idc value. The parameter tensor_format_idc is skipped because it is highly correlated with the previously defined nnpf_model_storage_form_idc value. Storage formats like ONNX usually specify the tensor format as well. patch_boundary_overlap_flag is skipped because deblocking filters are typically applied in the bitstream. Therefore, for NNPF, overlapping is likely not needed. [Table 19-1] [Table 19-2]

[0037] Given the above syntax, an example of how to apply the NNPF SEI messages in Table 17 is shown below: Assuming that the NNPF is used to improve visual quality, nnpf_purpose is set to 0. Assuming that NNPF model related information needs to be signaled, nnpf_model_info_present_flag is set to 1. If luma and chroma use different models, nnpf_joint_model_flag is set to 0. Since different models are applied to intra and inter pictures, nnpf_num_pic_type_minus1 is set to 1. num_of_nnpf_models is set to 4 (luma / chroma and inter / inter). Given these four models, the value of nnpf_model_id[0] is set to 0, which is used for luma components and intra pictures, the value of nnpf_model_id[1] is set to 1, which is used for chroma components and intra pictures, the value of nnpf_model_id[2] is set to 2, which is used for luma components and inter pictures, and the value of nnpf_model_id[3] is set to 3, which is used for chroma components and inter pictures. The number of checkpoints provided per model is set to 1, so num_of_ckpts_minus1[0] / [1] / [2] / [3] are all set to 0. External web links can be provided for two model IDs. Therefore, nnpf_model_exter_link_flag[0] / [1] is set to 1. The web links are coded using IETF Internet Standard 66. For all models, Pytorch is used, so nnpf_model_storage_form_idc[0] / [1] is set to 3. nnpf_model_complexity_ind_present_flag[0] / [1] is set to 1 to indicate the model complexity. This model uses single precision floating point format. The value of nnpf_param_prec_idc[0] / [1] is set to 4. The number of model parameters for each id is 214k=1.6327*2^17.Therefore, the value of log2_nnpf_num_param_minus11[0] / [1] is set to 6. log2_prec_denom[0] / [1] is set to 5, and nnpf_num_param_frac[0] / [1] is set to 21. Therefore, the max number parameter is set equal to 217k. The number of operations as kMac / pixel is 33.6k. The value of nnpf_num_op[0] / [1] is set to 34.

[0038] Following the signal data formation information, nnpf_data_info_present_flag is set to 1. Since the input and output of the NNPF is YUV420, nnpf_chroma_format_idc is set to 1 (420 format). vui_matrix_coeffs is set to 1 or 9 (YUV). Since separate models are used for the luma and chroma components, nnpf_joint_model_flag is 0, so there is no need to signal packing_format_idc. The chroma model also uses luma information, so chroma_luma_dependency_flag is set to 1. Since the patch size is 128, the value of log2_patch_size_minus6 is set to 1. Assuming the picture size is 4k, padding needs to be added. For duplication padding, the value of picture_padding_type is set to 1. Since deblocking is used in the bitstream, duplication for patches is not used. For auxiliary input information, the QP map is used and the value of nnpf_auxi_input_id is set to 1.

[0039] The picture-level NNPF SEI messaging of Table 18 may require region-level metadata that may be too large or rarely used in many applications. To reduce the overall payload size and focus on the QP mapping SEI information, an alternative simplified picture-level NNPF SEI message example is shown in Table 20. To generate the syntax, some of the previously defined parameters are removed as described below.

[0040] [Table 20] nnpf_pic_model_id_chroma specifies the index of the model used for the current picture for a chroma component. The value of nnpf_pic_model_id_chroma shall be in the range 0 to nnpfc_max_num_models inclusive for this version of this specification. When not present, the value of nnpf_pic_model_id_chroma is inferred to be equal to nnpf_pic_model_id. nnpf_pic_ckpt_idx_chroma specifies the index of the checkpoint to use with the current picture model for a chroma component. The value of nnpf_pic_ckpt_idx_chroma shall be in the range 0 to nnpfc_max_num_ckpts_minus1[nnpf_mode_id_chroma]. When not present, the value of nnpf_pic_ckpt_idx_chroma is inferred to be equal to nnpf_pic_ckpt_idx.

[0041] All parameters related to region-level messaging are removed, and the redundancy created by these parameters is also removed. More specifically, nnpf_region_info_present_flag is considered unnecessary and redundant due to the use of nnpf_qp_info_present_flag. Similarly, nnpf_region_ptt_present_flag, ptt_map, and clfc_map are not required if region-level partitions are not available.

[0042] [Existence of auxiliary data in neural network tensors] In reference

[21] , auxiliary input data can be present in the neural network input tensor only when the value of nnpfc_inp_order_idc is equal to 3, i.e., when the input tensor is configured as four interleaved luma channels and two chroma channels. Currently, auxiliary input data cannot be present in the input tensor for luma-only configurations, chroma-only configurations, and three-channel luma and chroma configurations, i.e., nnpfc_inp_order_idc equal to 0, 1, and 2, respectively. It is argued that auxiliary input data can be beneficial for all input tensor configurations.

[0043] As suggested above (see, e.g., Table 10 and the syntax parameter nnpf_auxi_input_id), it is proposed to add the syntax element nnpfc_auxiliary_input_idc and corresponding semantics to the NNPF CLVS SEI message, denoted as NNPFC SEI in

[21] , to allow auxiliary data to be present in the input tensor for all allowed configurations of the input tensor, i.e., for all values ​​of nnpfc_inp_order_idc. As in the current draft of the VSEI amendment

[21] , the auxiliary input data is added to the luma quantization parameter SliceQp YIt is proposed that the input signal be limited to signals derived from nnpfc_auxiliary_input_idc. The parameter nnpfc_auxiliary_input_idc was also previously proposed in

[22] .

[0044] [Instructions for color description of neural network tensors] The color description information of neural network tensors cannot be signaled using the current text in

[21] . It is argued that the color description information of neural network tensors can be useful. For example, IC T C P could be preferable when applying neural network post-filters to HDR WCG signals.

[0045] Syntax Elements nnpfc_separate_colour_description_present_flag , nnpfc_colour_primaries , nnpfc_transfer_characteristic and nnpfc_matrix_coeffs It is proposed to add the film grain feature SEI message syntax and semantics to the NNPFC SEI message. It is proposed that the syntax and semantics are modeled based on the film grain feature SEI message syntax and semantics.

[0046] Additionally, the following constraints are proposed for nnpfc_purpose, nnpfc_inp_order_idc and nnpfc_out_order_idc when nnpfc_matrix_coeffs is equal to 0. This is typically used for GBR(RGB) and YZX4:4:4 chroma formats. 1. nnpfc_purpose shall not be equal to 2 (chroma upsampling to 4:4:4 chroma format) or 4 (increasing the width or height of the cropped decoded output picture and upsampling chroma format). 2. nnpfc_inp_order_idc shall not be equal to 1 (2 chroma channels and 0 luma channel in the input tensor) or 3 (4 interleaved luma channels and 2 chroma channels in the input tensor). 3. nnpfc_out_order_idc shall not be equal to 1 (only 2 chroma channels in the output tensor) or 3 (4 interleaved luma channels and 2 chroma channels in the output tensor).

[0047] [Dependency Indication for Multiple Activation Neural Network Postfilters] It is argued that it may be beneficial to apply neural network postfilters in certain sequences when more than one neural network postfilter is activated for the current picture. For example, the output tensor of a luma-only neural network postfilter can be used to derive the input tensor of a luma-chroma neural network postfilter. As another example, the output tensor of a neural network postfilter for increasing the width or height of the decoded picture (nnpfc_purpose equal to 2, 3 or 4) can be used to derive the input tensor of a neural network postfilter for improving the video quality (nnpfc_purpose equal to 1).

[0048] It is proposed to add three syntax elements and corresponding semantics to the NNPFA SEI message as follows: 1. nnpfa_independent_flag to indicate a preference that the neural network post filter signaled in the SEI should either be independent of other neural network post filters that may also be used for the current picture, or should depend on the output of one or more such neural network post filters. 2. nnpfa_num_dependencies_minus1 to indicate the number of neural network postfilters on which the current neural network postfilter may depend. 3. nnpfa_dependency_nnpfa_id[i] to specify the identification number nnpfa_id of the i-th neural network post-processing filter on which the current neural network filter may depend.

[0049] [Neural network post-filter characteristics (NNPFC) SEI message] Given these proposed new syntax elements, the following table represents the revised NNPF CLVS or NNPFC SEI message. Changes to reference

[21] are indicated using italics (or underlines). [Table 21-1-1] [Table 21-1-2] [Table 21-2]

[0050] [Semantics of Neural Network Post-Filter Characteristics SEI Message] Compared to the original text and semantics of the NNPFC, the following modifications are proposed:

[0051] This SEI message specifies a neural network that may be used as a post-processing filter. The use of a specified post-processing filter for a particular picture is indicated with a Neural Network Post Filter Activation SEI message. Use of this SEI message requires the definition of the following variables: The width and height of the cropped decoded output picture in units of luma samples, denoted here by InpPicWidthInLumaSamples and InpPicHeightInLumaSamples, respectively. - The luma sample array CroppedYPic[y][x] and chroma sample arrays CroppedCbPic[y][x] and CroppedCrPic[y][x] (if present) of the cropped decoded output picture, for vertical coordinate y and horizontal coordinate x, where the top-left corner of the sample arrays has coordinate y equal to 0 and coordinate x equal to 0. - The bit depth BitDepthY for the luma sample array of the cropped decoded output picture. - The bit depth, if any, of the chroma sample array of the cropped decoded output picture, BitDepthC. -Chroma to luma subsampling ratio, shown as InpSubWidthC and InpSubHeightC. - When nnpfc_auxiliary_input_idc is equal to 1, SliceQp Y denotes the initial luma quantization parameter value.

[0052] When this SEI message specifies a neural network that can be used as a post-processing filter, the semantics specify the derivation of a luma sample array, FilteredYPic[y][x], and chroma sample arrays, FilteredCbPic[y][x] and FilteredCrPic[y][x], as indicated by the value of nnpfc_out_order_idc, that contain the output of the post-processing filter. nnpfc_auxiliary_input_idc not equal to 0 specifies that the auxiliary input data is present in the neural network postfilter input tensor. nnpfc_auxiliary_input_id equal to 0 indicates that the auxiliary input data is not present in the input tensor. nnpfc_auxiliary_input_idc equal to 1 specifies that the auxiliary input data is derived as specified in Table 23. Values ​​of nnpfc_auxiliary_input_id greater than 1 are reserved for future specifications by ITU-T|ISO / IEC and shall not be present in bitstreams conforming to this version of this specification. Decoders conforming to this version of this specification shall ignore SEI messages containing reserved values ​​of nnpfc_inp_order_idc. nnpfc_separate_colour_description_present_flag equal to 1 indicates that the syntax of the Neural Network Postfilter Features SEI message contains separate combinations of colour primaries, transfer characteristics and matrix coefficients for the Neural Network Postfilter Features specified in the SEI message. nnfpc_separate_colour_description_present_flag equal to 0 indicates that the combinations of colour primaries, transfer characteristics and matrix coefficients for the Film Grain Features specified in the SEI message are the same as those indicated in the VUI parameters for CLVS. nnpfc_colour_primaries has the same semantics as specified in clause 7.3 of [3] for the vui_colour_primaries syntax element, except as follows: -nnpfc_colour_primaries specifies the primaries of the neural network postfilter characteristics specified in the SEI message, rather than the primaries used for CLVS. - When nnpfc_colour_primaries is not present in the Neural Network Postfilter Characteristics SEI message, the value of nnpfc_colour_primaries is inferred to be equal to vui_colour_primaries. The nnpfc_transfer_characteristics has the same semantics as specified in Section 7.3 of [3] for the vui_transfer_characteristics syntax element, except for the following: -nnpfc_transfer_characteristics specifies the transfer characteristics of the neural network postfilter characteristics specified in the SEI message instead of the transfer characteristics used for CLVS. When -nnpfc_transfer_characteristics is not present in the Neural Network Postfilter Characteristics SEI message, the value of nnpfc_transfer_characteristics is inferred to be equal to vui_transfer_characteristics. nnpfc_matrix_coeffs has the same semantics as specified in Section 7.3 of [3] for the vui_matrix_coeffs syntax element, except as follows: -nnpfc_matrix_coeffs specifies the matrix coefficients of the neural network postfilter characteristics specified in the SEI message, not the matrix coefficients used for CLVS. When -nnpfc_matrix_coeffs is not present in the Neural Network Postfilter Characteristics SEI message, the value of fg_matrix_coeffs is inferred to be equal to vui_matrix_coeffs. The allowed values ​​for -nnpfc_matrix_coeffs are not constrained by the chroma format of the decoded video picture indicated by the value of ChromaFormatIdc for the semantics of the VUI parameters. - when nnpfc_matrix_coeffs is equal to 0, nnpfc_purpose shall not be equal to 2 or 4, nnpfc_inp_order_idc shall not be equal to 1 or 3, and nnpfc_out_order_idc shall not be equal to 1 or 3. [Table 22]

[0053] Due to the proposed new syntax, Table 23 in

[21] may be updated as follows: [Table 23-1] [Table 23-2] [Table 23-3] [Neural-network post-filter activation (NNPFA) SEI message In reference

[21] , the NNPF messages for the picture layer are denoted as NNPFA SEI messages. The proposed modifications to the existing syntax are shown in italics (or underlined) in Table 24. [Neural network postfilter activation SEI message syntax] [Table 24]

[0054] Semantics of Neural Network Post-Filter Activation SEI Messages This SEI message specifies the neural network post-processing filters that may be used for post-processing filtering for the current picture and carries information about dependencies, if any, on other neural network post-filters that may exist for the current picture.

[0055] The neural network post-processing filter activation SEI message only persists for the current picture.

[0056] NOTE − There may be several neural network post-processing filter activation SEI messages for the same picture, e.g., when the post-processing filters are meant for different purposes or are meant to filter different color components. nnpfa_id specifies that the neural network post-processing filters specified by one or more Neural Network Post-Processing Filter Properties SEI messages associated with the current picture and having nnpfc_id equal to nnfpa_id may be used for post-processing filtering of the current picture. nnpfa_independent_flag equal to 0 indicates a preference that the input to the neural network post-processing filter with nnfpa_id should depend on the output of one or more other neural network post-processing filters that are related to the current picture and have nnpfc_id not equal to nnpfa_id. nnpfa_independent_flag equal to 1 indicates no preference. The value of nnpfa_independent_flag should be equal to 1 when there is only one neural network post-filter activation SEI message for the current picture. nnpfa_num_preceding_nnpfa_ids_minus1 plus 1 specifies the number of neural network post-processing filters related to the current picture that should precede the neural network post-processing filter specified by nnpfa_id in the processing order. nnpfa_preceding_nnpfa_id[i] specifies that the neural network post-processing filter specified by nnpfc_id equal to nnpfa_preceding_nnpfa_id[i] should precede the neural network post-processing filter specified by nnpfa_id in the processing order.

[0057] Figure 5 shows an example of data flow for processing CLVS layer NNPF SEI messaging. The data flow follows the syntax of Table 11. For the picture layer NNPF SEI messages shown in Table 16, corresponding data flow processing examples are shown in Figures 6A and 6B.

[0058] As discussed in

[19] , certain applications may require defining a priority order for how multiple SEI messages can be executed. As an example, priority is important when considering the SEI messages for Film Grain Characteristics (FGC) and Colour Transform Information (CTI). In HEVC and AVC, the Post Filter Hints, Tone Mapping Information and Chroma Sampling Filter Hints SEI messages are further examples of SEI messages that need to be considered to define their processing order. The processing order of the NNPF SEI messaging should also be considered. The specific order needs to be determined by the user case and can be transmitted together with the bitstream as suggested in the proposed processing order SEI (

[19] ). As an example, assume that the bitstream carries standard dynamic range (SDR) video and FGC, CTI and NNPF SEI messaging. Here, the CTI SEI is used to convert the SDR video to HDR video and the NNPF SEI is used for quality improvement to the SDR decoded video. In an embodiment, the proposed order may be first the NNPF SEI (to improve the decoded video quality), then the CTI SEI (to convert SDR to HDR), and finally the FGC SEI (to add film grain effect for final display). For example, if applied earlier, the added film grain noise may be amplified during the conversion from SDR to HDR.

[0059] [References] Each of the references cited herein is incorporated by reference in its entirety. The term JVET refers to the Joint Video Experts Team of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29. [1] Advanced Video Coding, Rec. ITU-T H.264, May 2019. [2] High Efficiency Video Coding, Rec. ITU-T H.265, November 2019. [3] Versatile Video Coding, Rec. ITU-T H.266, August 2020. [4] M. M. Hannuksela, M. Santamaria, F. Cricri, E. B. Aksu, H. R. Tavakoli, “AHG9: On post-filter SEI,” JVET-Y0115, online meeting, Jan 2022 [5] M. M. Hannuksela, E. B. Aksu, F. Cricri, H. R. Tavakoli, M. Santamaria,“AHG9: On post-filter SEI,” JVET-X0112, online meeting, Oct 2021. [6] M. M. Hannuksela, E. B. Aksu, F. Cricri, H. R. Tavakoli,“ AHG9: On post-filter SEI”, JVET-V0058, online meeting, April, 2021. [7] T. Chujoh, Y. Yasugi, K. Takada, T. Ikai, “AHG9: Colour component description for post-filter purpose SEI message,” JVET-Y0073, online meeting, Jan 2022 [8] Y. Yasugi, T. Chujoh, K. Takada, T. Ikai, “AHG9: Data conversion description for NNR post-filter SEI message,” JVET-Y0074, online meeting, Jan 2022. [9] K. Takada, Y. Yasugi, T. Chujoh, T. Ikai, “AHG9: Complexity description for NNR post-filter SEI message,” JVET-Y0075, online meeting, Jan 2022

[10] B. Choi, Z. Li, W. Wang, W. Jiang, X. Xu, S. Wenger, S. Liu,“ AHG9 / AHG11: SEI messages for carriage of neural network information for post-filtering,” JVET-V0091, online meeting, April, 2021.

[11] MPEG-7: Compression of Neural Networks for Multimedia Content Description and analysis: ISO / IEC 15938-17.

[12] “White Paper on Neural Network Coding,” MPEG document N00057, ISO / IEC JTC 1 / SC 29 / WG 04, Jan. 2022.

[13] H. Kirchhoffer et al., "Overview of the Neural Network Compression and Representation (NNR) Standard," in IEEE Transactions on Circuits and Systems for Video Technology, doi: 10.1109 / TCSVT.2021.3095970.

[14] Maria Santamaria, Jani Lainema, Francesco Cricri, Ramin G. Youvalari, Honglei Zhang, Alireza Zare, Goutham Rangu, Hamed R. Tavakoli, Homayun Afrabandpey, Miska Hannuksela," AHG11: MPEG NNR compressed bias update for the CNN based post-filter of EE1-1.1", JVET-X0111, Oct. 2021.

[15] Y. Li, K. Zhang, L. Zhang, H. Wang, J. Chen, K. Reuze, A.M. Kotra, M. Karczewicz, “EE1-1.6: Combined Test of EE1-1.2 and EE1-1.4,” JVET-X0066, online meeting, Oct. 2021.

[16] H. Wang, J. Chen, K. Reuze, A. M. Kotra, M. Karczewicz, “EE1-1.4: Tests on Neural Network-based In-Loop Filter with constrained computational complexity,” JVET-X0140, online meeting, Oct. 2021.

[17] Y. Li, K. Zhang, L. Zhang, “AHG11: Deep In-Loop Filter with Adaptive Model Selection and External Attention,” JVET-W0100, online meeting, July 2021.

[18] L. Wang, X. Xu, S. Liu, “EE1-1.1: neural network based in-loop filter with constrained storage and low complexity,” JVET-Y0078, online meeting, Jan 2022.

[19] P. Yin et al., “Signaling of priority processing order for metadata messaging in video coding,” U.S. Provisional Patent Application, Ser. No. 63 / 216,318, filed on June 29, 2021.

[20] M. M. Hannuksela et al., “AHG9: NN post-filter SEI,” JVET-Z0244, online meeting, 20-29 April 2022.

[21] S. McCarthy et al., “Additional SEI messages for VSEI (Draft 1),” JVET-Z2006, output document of April 2022 online meeting, June 2022.

[22] S. McCarthy et al., “AHG9: Neural-network post filtering SEI message,” JVET-Z0121, online meeting, 20-29 April 2022.

[0060] [Exemplary Computer System Implementation] The embodiments of the present invention may be implemented using a computer system, a system comprised of electronic circuits and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or an apparatus including one or more of such systems, devices or components. The computer and / or IC may execute, control or carry out instructions related to the delivery of neural network topologies and parameters associated with NNPFs in image and video coding, such as those described herein. The computer and / or IC may calculate any of various parameters or values ​​related to the delivery of neural network topologies and parameters associated with NNPFs in image and video coding, such as those described herein. The image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

[0061] A particular implementation of the present invention includes a computer processor executing software instructions that cause the processor to perform the method of the present invention. For example, one or more processors in a display, encoder, set-top box, transcoder, etc. may perform the method of conveying neural network topology and parameters associated with NNPF in image and video coding as described above by executing software instructions in a program memory accessible to the processor. An embodiment of the present invention may also be provided in the form of a program product. The program product may include any non-transitory tangible medium that carries a set of computer-readable signals that include instructions that, when executed by a data processor, cause the data processor to perform the method of an embodiment of the present invention. A program product according to an embodiment of the present invention may be in any of a wide variety of non-transitory tangible forms. The program product may include a physical medium, such as, for example, a magnetic data storage medium including a floppy disk, a hard disk drive, an optical data storage medium including a CD ROM, a DVD, an electronic data storage medium including a ROM, a flash RAM, etc. The computer-readable signals on the program product may optionally be compressed or encrypted. When a component (e.g., a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, the reference to that component (including the reference to a "means") should be interpreted as including any component that performs the function of the described component (e.g., is functionally equivalent) as an equivalent of that component, including components that are not structurally equivalent to the disclosed structures that perform that function in the illustrated exemplary embodiments of the present invention.

[0062] [Equivalents, extensions, alternatives and others] Exemplary embodiments of the conveyance of neural network topology and parameters associated with NNPF in image and video coding have been described above. In the above specification, the embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the only and exclusive indication of what the invention is and what is intended by the applicant to be the invention is the set of claims issued from this application, in the particular form in which such claims are issued, including any subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Thus, any limitations, elements, properties, features, advantages or attributes not expressly set forth in the claims should in no way limit the scope of such claims. Thus, the specification and drawings are to be regarded in an illustrative and not restrictive sense.

Claims

1. 1. A method for processing one or more pictures in a coded video sequence with neural network post-filtering (NNPF), comprising: receiving a decoded image and NNPF metadata related to processing the decoded image using an NNPF; parsing syntax parameters in the NNPF metadata to perform NNPF according to one or more neural network models, associated NNPF data, and NNPF parameters; performing NNPF on the decoded image in accordance with the syntax parameters to generate an output image; Including, The method of claim 1, wherein the syntax parameters in the NNPF metadata include a first set of NNPF messaging parameters that persist until the end of decoding of the coded video sequence and a second set of NNPF messaging parameters that persist until the end of NN post-filtering of the decoded image.

2. The first set of NNPF messaging parameters comprises: an NNPF model information presence flag indicating that NNPF model information exists in the NNPF metadata; an NNPF joint model flag (nnpf_joint_model_flag) indicating whether the NNPF applies the same neural network model to both the luma and chroma components; The NNPF picture type count parameter (nnpf_num_pic_type_minus1) indicates the number of different picture types supported by the NNPF; An array of NNPF model IDs (nnpf_model_id[i]) to identify each NNPF model, First parameters related to neural network topology and model information; a second parameter related to data information in the decoded image; The third parameter is related to the NNPF auxiliary information. The method of claim 1 , comprising one or more of:

3. The first parameter, which is related to the neural network topology and model information, is A flag indicating whether detailed information about the NN model used in the NNPF is provided using an external link; NNPF storage and exchange data format parameters; NNPF arithmetic precision parameters; NNPF model number parameters, NNPF latency estimation parameters and The method of claim 1 , comprising one or more of:

4. A second parameter related to data information in the decoded image is input chroma format parameters; Packing format parameters; Chroma-dependent format parameters; The input tensor format parameters, A picture padding parameter; A temporal picture flag indicating the presence of a temporally adjacent picture as an auxiliary input; The method of claim 1 , comprising one or more of:

5. The picture padding parameters are: 0 for zero padding, 1 for duplicate padding, and 2 and about reflective padding The method of claim 4 , comprising:

6. The second parameter relating to the data information is a flag indicating whether auxiliary input data is present in the input tensor of the NNPF metadata; a flag indicating that there is a distinct combination of primaries, transfer characteristics, and matrix coefficients for the NNPF metadata; The method of claim 4 , further comprising one or more of:

7. The third parameter, related to the NNPF auxiliary information, is QP map and The partition map, Classification map and 2. The method of claim 1, further comprising: an NNPF auxiliary input identifier indicating availability of auxiliary inputs including one or more of:

8. The method of claim 1 , wherein the second set of NNPF messaging parameters includes an NNPF Picture Model ID that specifies an NN postfilter to be used for the decoded image.

9. The second set of NNPF messaging parameters comprises: Picture QP related metadata; Picture partition related metadata; Picture classification related metadata; A dependency flag indicating whether the signaled NN post-filtering is independent or dependent on other NN post-filters; Includes one or more of the following: If the dependency flag indicates a dependency on another NN postfilter, a number of predecessors variable indicating how many NN post-filters should precede in processing order the current NNPF specified by the picture layer NNPF identification variable; an array of NNPF identification variables for an NN postfilter that should precede the current NNPF in the processing order; The method of claim 8 further comprising:

10. The picture QP related metadata is An NNPF QP information presence flag indicating the presence of QP information; an NNPF region information flag indicating the presence of region information; An NNPF Region QP Present flag indicating the presence of region-based QP information; Includes one or more of the following:

10. The method of claim 9, further comprising: if the NNPF QP information present flag is set, then including QP information for at least one region.

11. The picture partition related metadata includes: An NNPF area partition presence flag indicating the presence of NNPF area partition information; 10. The method of claim 9, further comprising: if the NNPF region partition present flag is set, then at least one picture partition map.

12. The picture classification related metadata includes: including an NNPF picture classification present flag indicating the presence of picture classification information; 10. The method of claim 9, further comprising: a picture classification for at least one region if the NNPF picture classification present flag is set.

13. 1. A method for encoding an image or a coded video sequence in a processor, comprising the steps of: encoding said image or said video sequence into a coded bitstream; generating Neural Network Post-Filtering (NNPF) metadata to enable a decoder of the coded bitstream to perform NNPF according to one or more neural network models, associated NNPF data, and NNPF parameters; generating an output comprising the coded bitstream and the NNPF metadata; Including, The method of claim 1, wherein syntax parameters in the NNPF metadata include a first set of NNPF messaging parameters that persist until the end of decoding of the coded video sequence and a second set of NNPF messaging parameters that persist until the end of NN post-filtering of a single decoded image.

14. A non-transitory computer readable storage medium having stored thereon computer executable instructions for performing the method of any one of claims 1 to 13 using one or more processors.

15. Apparatus comprising a processor and configured to carry out the method of any one of claims 1 to 13.

Images