Carriage of neural-network post-filter

Abstract

Various embodiments describe an apparatus, a method, and a computer program product. An example apparatus includes at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: storing coded video data in a video track of a file; obtaining a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message comprising a neural network representation (NNR) bitstream; storing the NNR bitstream as an item of the file; storing the NNPFC SEI message excluding the NNR bitstream in the file; and associating, in the file, the item and the NNPFC SEI message excluding the NNR bitstream.

Description

CARRIAGE OF NEURAL-NETWORK POST-FILTERTECHNICAL FIELD

[0001] The examples and non-limiting embodiments relate generally to neural networks , and more particularly, to method, apparatus, and computer program product for carriage of neural-network post-filter.BACKGROUND

[0002] It is known to use neural network for image or video coding.SUMMARY

[0003] An example apparatus includes at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: storing coded video data in a video track of a file; obtaining a neural-network postfilter characteristics (NNPFC) supplemental enhancement information (SEI) message comprising a neural network representation (NNR) bitstream; storing the NNR bitstream as an item of the file; storing the NNPFC SEI message excluding the NNR bitstream in the file; and associating, in the file, the item and the NNPFC SEI message excluding the NNR bitstream.

[0004] The example apparatus may further include, wherein the NNR bitstream stored in the item specifies at least one of a base filter or a filter update.

[0005] The example apparatus may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored as an item property associated with an NNR item, and wherein the NNR item is associated with an NNPFC SEI item property through an item property association.

[0006] The example apparatus may further include, wherein the NNPFC SEI item property comprises a box type, a property type, and a container.

[0007] The example apparatus may further include, wherein the NNR item and the item comprising the NNPFC SEI message are associated with each other with an item reference.

[0008] The example apparatus may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored as a sample group description entry.

[0009] The example apparatus may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored as a NAL-unit-like structure within a sample.

[0010] The example apparatus may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored in a sample entry.

[0011] The example apparatus may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored in a sample of a timed metadata track.

[0012] The example apparatus may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored as sample auxiliary information.

[0013] The example apparatus may further include, wherein a sample-to-item sample group is used to map samples of a track with the NNR item.

[0014] The example apparatus may further include, wherein a sample group is extended from a sample-to-item sample group defined in ISO / IEC 14496-12 and is used to map samples of a track with NNR items.

[0015] The example apparatus may further include, wherein to the sample group description entry extends a sample-to-item sample group description entry.

[0016] The example apparatus may further include, wherein a sample group description entry indicates when the NNR item is referenced.

[0017] The example apparatus may further include, wherein a track reference from a track comprising the NNPFC SEI message excluding the NNR bitstream to the item is included in the file.

[0018] The example apparatus may further include, wherein a ISOBMFF comprises an NNR item referenced flag, wherein when NNR item referenced flag equal to one specifies that this sample group description entry does not include an ISO / IEC 15938-17 bitstream and that the NNPFC SEI message is formed by concatenating an associated NNR item after the NNPFC SEI message data bytes of the sample group description entry, and wherein when the NNR item referenced flag equal to zeroindicates that the ISO / IEC 15938-17 bitstream is included in the sample group description entry or that no ISO / IEC 15938-17 bitstream is needed to form the NNPFC SEI message.

[0019] The example apparatus may further include, wherein the apparatus is further caused to perform: setting a sample-to-item sample group NNPFC identifier; and mapping a sample group description of the sample-to-item sample group to a first sample of each run of samples mapped to same NNPFC sample group description entry of same NNPFC identifier value.

[0020] The example apparatus may further include, wherein a grouping type parameter of the sample-to-item sample group comprises a filter update flag and a filter identifier, and wherein the filter update flag equal to one indicates that all the sample group description entries referenced by a sample to group box includes the NNR bitstream for the NNPFC SEI message that provides an update on top of a base post-processing filter, and wherein the filter update flag equal to zero indicates that all the sample group description entries referenced by the sample to group box includes the NNR bitstream for the NNPFC SEI message that specifies a base post-processing filter, and wherein the filter identifier indicates that all the sample group description entries referenced by the sample to group box are associated with the NNPFC SEI message that has NNPFC identifier equal to the filter identifier.

[0021] The example apparatus may further include, wherein a SampleToMetadataltemEntry is further extended to include a stm_grouping_type and a stm_grouping_type_parameter and wherein when the stm_grouping_type is present indicates that the sample group description entry associates with a sample group with a grouping-type equal to the stm_grouping_type, and when stm_grouping_type is not present, no association to a sample group is provided with the sample group description entry, and wherein when the stm_grouping_type_parameter is present, indicates that this sample group description entry associates with a sample group with the grouping- type equal to the stm_grouping_type and grouping_type_value equal to stm_grouping_type_value, and wherein when the stm_grouping_type_parameter is not present, no association to a particular grouping_type_parameter value is provided with the sample group description entry.

[0022] The example apparatus may further include, wherein NNR data units specifying either the base filter, the filter update, or both with a specific NNPFC identifier are stored in 'imda' boxes.

[0023] The example apparatus may further include, wherein the NNR data units are referenced using data reference entry of DataEntryltemlmdaBox, wherein the DataEntryltemlmdaBox identifies an IdentifiedMediaDataBox comprising the item data accessed through the data_reference_index corresponding to the DataEntryltemlmdaBox, when the data_reference_index comprised in anItemLocationBox refers to the DataEntryltemlmdaBox, a respective item is included in an IdentifiedMediaDataBox comprising imda_identifier equal to item_ID of the item, and base_offset and extent_offset are relative to a first byte of a payload of the IdentifiedMediaDataBox,.

[0024] Another example apparatus includes at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: reading, from a file, coded video data of a video track; reading a neural network representation (NNR) bitstream from an item of the file; reading a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message excluding the NNR bitstream from the file; decoding coded video data from the video track for generating a decoded video data; reconstructing a neural network from the NNR bitstream; forming input data to the neural network according to the NNPFC SEI message excluding the NNR bitstream; and filtering the decoded video data with a post-filter defined by the neural network using input data.

[0025] The example apparatus may further include, wherein the apparatus is further caused to perform: concatenating an NNR item identified by a sample group description entry after the nnpfc sei data byte array comprised in the sample group description entry to form the NNPFC SEI message.

[0026] The example apparatus may further include, wherein the apparatus is further caused to perform: determining, based on one or more NNPFC sample groups, which NNPFC sample groups are processed; and using groupi ng_type and grouping_type_parameter values of the NNPFC sample groups that are processed to find items with the same stm_grouping_type and stm_grouping_type_parameter values in order to obtain or otherwise process the NNR item.

[0027] The example apparatus may further include, wherein the apparatus is further caused to perform: receiving an NNPFC SEI message comprising the NNR bitstream; storing the NNR bitstream as the item in the file; and storing the NNPFC SEI message apart from the NNR bitstream separately from the NNR item in the file.

[0028] The example apparatus may further include, wherein the apparatus is further caused to perform: concluding whether the NNR item is referenced by or associated with the sample group description entry; wherein when the NNR item is not referenced by or is not associated with the sample group description entry, the NNPFC SEI message is formed from the NNPFC SEI bitstream including the NNR bitstream; and wherein the NNR item is referenced by or associated with the sample group description entry, the NNPFC SEI message is formed from the NNPFC SEI bitstream excluding the NNR bitstream followed by the item data of the NNR item.

[0029] An example method includes: storing coded video data in a video track of a file; obtaining a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message comprising a neural network representation (NNR) bitstream; storing the NNR bitstream as an item of the file; storing the NNPFC SEI message excluding the NNR bitstream in the file; and associating, in the file, the item and the NNPFC SEI message excluding the NNR bitstream.

[0030] The example method may further include, wherein the NNR bitstream stored in the item specifies at least one of a base filter or a filter update.

[0031] The example method may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored as an item property associated with an NNR item, and wherein the NNR item is associated with an NNPFC SEI item property through an item property association.

[0032] The example method may further include, wherein the NNPFC SEI item property comprises a box type, a property type, and a container.

[0033] The example method may further include, wherein the NNR item and the item comprising the NNPFC SEI message are associated with each other with an item reference.

[0034] The example method may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored as a sample group description entry.

[0035] The example method may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored as a NAL-unit-like structure within a sample.

[0036] The example method may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored in a sample entry.

[0037] The example method may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored in a sample of a timed metadata track.

[0038] The example method may further include, wherein the NNPFC SEI message excluding the NNR bitstream is stored as sample auxiliary information.

[0039] The example method may further include, wherein a sample-to-item sample group is used to map samples of a track with the NNR item.

[0040] The example method may further include, wherein a sample group is extended from a sample-to-item sample group defined in ISO / IEC 14496-12 and is used to map samples of a track with NNR items.

[0041] The example method may further include, wherein to the sample group description entry extends a sample-to-item sample group description entry.

[0042] The example method may further include, wherein a sample group description entry indicates when the NNR item is referenced.

[0043] The apparatus of any of the claims 1 to 10, wherein a track reference from a track comprising the NNPFC SEI message excluding the NNR bitstream to the item is included in the file.

[0044] The example method may further include, wherein a ISOBMFF comprises an NNR item referenced flag, wherein when NNR item referenced flag equal to one specifies that this sample group description entry does not include an ISO / IEC 15938-17 bitstream and that the NNPFC SEI message is formed by concatenating an associated NNR item after the NNPFC SEI message data bytes of the sample group description entry, and wherein when the NNR item referenced flag equal to zero indicates that the ISO / IEC 15938-17 bitstream is included in the sample group description entry or that no ISO / IEC 15938-17 bitstream is needed to form the NNPFC SEI message.

[0045] The example method may further include: setting the sample-to-item sample group NNPFC identifier; and mapping a sample group description of the sample-to-item sample group to a first sample of each run of samples mapped to same NNPFC sample group description entry of same NNPFC identifier value.

[0046] The example method may further include, wherein a grouping type parameter of the sample-to-item sample group comprises a filter update flag and a filter identifier, and wherein the filter update flag equal to one indicates that all the sample group description entries referenced by a sample to group box includes the NNR bitstream for the NNPFC SEI message that provides an update on top of a base post-processing filter, and wherein the filter update flag equal to zero indicates that all the sample group description entries referenced by the sample to group box includes the NNR bitstream for the NNPFC SEI message that specifies a base post-processing filter, and wherein the filer identifierindicates that all the sample group description entries referenced by the sample to group box are associated with the NNPFC SEI message that has NNPFC identifier equal to the filter identifier.

[0047] The example method may further include, wherein a SampleToMetadataltemEntry is further extended to include a stm_grouping_type and a stm_grouping_type_parameter and wherein when the stm_grouping_type is present indicates that the sample group description entry associates with a sample group with a grouping-type equal to the stm_grouping_type, and when stm_grouping_type is not present, no association to a sample group is provided with the sample group description entry, and wherein when the stm_grouping_type_parameter is present, indicates that this sample group description entry associates with a sample group with the grouping- type equal to the stm_grouping_type and grouping_type_value equal to stm_grouping_type_value, and wherein when the stm_grouping_type_parameter is not present, no association to a particular grouping_type_parameter value is provided with the sample group description entry.

[0048] The example method may further include, wherein NNR data units specifying either the base filter, the update filters, or both with a specific NNPFC identifier are stored in 'imda' boxes.

[0049] The example method may further include, wherein the NNR data units are referenced using data reference entry of DataEntrylmdaBox, wherein the DataEntryltemlmdaBox identifies an IdentifiedMediaDataBox comprising the item data accessed through the data_reference_index corresponding to the DataEntryltemlmdaBox, when the data_reference_index comprised in an ItemLocationBox refers to the DataEntryltemlmdaBox, a respective item is included in an IdentifiedMediaDataBox comprising imda_identifier equal to item_ID of the item, and base_offset and extent_offset are relative to a first byte of a payload of the IdentifiedMediaDataBox.

[0050] Another example method includes: reading, from a file, coded video data of a video track; reading a neural network representation (NNR) bitstream from an item of the file; reading a neural- network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message excluding the NNR bitstream from the file; decoding coded video data from the video track for generating a decoded video data; reconstructing a neural network from the NNR bitstream; forming input data to the neural network according to the NNPFC SEI message excluding the NNR bitstream; and filtering the decoded video data with a post-filter defined by the neural network using the input data.

[0051] The example method may further include: concatenating an NNR item identified by a sample group description entry after the he nnpfc sei data byte array comprised in the sample group description entry to form the NNPFC SEI message.

[0052] The example method may further include: determining, based on one or more NNPFC sample groups, which NNPFC sample groups are processed; and using grouping type and grouping_type_parameter values of the NNPFC sample groups that are processed to find items with the same stm_grouping_type and stm_grouping_type_parameter values in order to obtain or otherwise process the NNR item.

[0053] The example method may further include: concluding whether the NNR item is referenced by or associated with the sample group description entry; wherein when the NNR item is not referenced by or is not associated with the sample group description entry, the NNPFC SEI message is formed from the NNPFC SEI bitstream including the NNR bitstream; and wherein the NNR item is referenced by or associated with the sample group description entry, the NNPFC SEI message is formed from the NNPFC SEI bitstream excluding the NNR bitstream followed by the item data of the NNR item.

[0054] A non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform the methods as described in any of the previous paragraphs.

[0055] An apparatus comprising means for performing methods as described in any of the previous paragraphs.BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The foregoing embodiments and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0057] FIG. 1 shows schematically an electronic device employing embodiments of the examples described herein.

[0058] FIG. 2 shows schematically a user equipment suitable for employing embodiments of the examples described herein.

[0059] FIG. 3 shows a block diagram of a general structure of a video encoder.

[0060] FIG. 4 is a block diagram showing the interface between an encoder implementing neural network based encoding, and a decoder implementing neural network based decoding in accordance with the examples described herein.

[0061] FIG. 5 depicts an example system that includes a source of media data and associated metadata.

[0062] FIG. 6. depicts an apparatus in accordance with an example embodiment.

[0063] FIG. 7 is an example apparatus, which may be implemented in hardware, and is caused to, implement reduced header mode for image file format based on the examples described herein.

[0064] FIG. 8 is an example method to implement the embodiments described herein, in accordance with an embodiment.

[0065] FIG. 9 is another example method to implement the embodiments described herein, in accordance with another embodiment.

[0066] FIG. 10 is a block diagram of one possible and non-limiting system in which the example embodiments may be practiced.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0067] The following acronyms and abbreviations that may be found in the specification and / or the drawing figures are defined as follows:4CC four character code5G fifth generation cellular network technology5GC 5G core network a.k.a. also known asAVC advanced video codingCU central unitDSP digital signal processorDU distributed uniteNB (or eNodeB) evolved Node B (for example, an LTE base station)EN-DC E-UTRA-NR dual connectivity en-gNB or En-gNB node providing NR user plane and control plane protocol terminations towards the UE, and acting as secondary node in EN-DCE-UTRA evolved universal terrestrial radio access, for example, the LTE radio access technologyFl or Fl-C interface between CU and DU control interface gNB (or gNodeB) base station for 5G / NR, for example, a node providing NR user plane and control plane protocol terminations towards the UE, and connected via the NG interface to the 5GCIEC International Electrotechnical Commission loT internet of thingsISO International Organization for StandardizationISOBMFF ISO base media file formatJPEG joint photographic experts groupLTE long-term evolution mdat MediaDataBoxMME mobility management entity moov MovieBoxMP4 file format for MPEG-4 Part 14 filesMPEG moving picture experts groupMPEG-2 H.222 / H.262 as defined by the ITUMPEG-4 audio and video coding standard for ISO / IEC 14496 ng or NG new generation ng-eNB or NG-eNB new generation eNBNR new radio (5G radio)N / W or NW networkPDCP packet data convergence protocolPHY physical layerPNG portable network graphicsRAN radio access networkRFC request for commentsREC radio link controlRRC radio resource controlRRH remote radio headRU radio unitRx receiverSDAP service data adaptation protocolSGW serving gatewaySMF session management functionSPS sequence parameter setSVC scalable video codingSI interface between eNodeBs and the EPC trak TrackBoxTx transmitterUE user equipmentUICC Universal Integrated Circuit CardUPF user plane functionURL uniform resource locatorX2 interconnecting interface between two eNodeBs in LIE networkXn interface between two NG-RAN nodes

[0068] Some embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments are shown. Indeed, various embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout. As used herein, the terms ‘data,’ ‘content,’ ‘information,’ and similar terms may be used interchangeably to refer to data capable of being transmitted, received and / or stored in accordance with embodiments of the present invention. Thus, use of any such terms should not be taken to limit the spirit and scope of embodiments.

[0069] Additionally, as used herein, the term ‘circuitry’ refers to (a) hardware-only circuit implementations (e.g., implementations in analog circuitry and / or digital circuitry); (b) combinations of circuits and computer program product(s) comprising software and / or firmware instructions stored on one or more computer readable memories that work together to cause an apparatus to perform one or more functions described herein; and (c) circuits, such as, for example, a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation even if the software or firmware is not physically present. This definition of ‘circuitry’ applies to all uses of this term herein, including in any claims. As a further example, as used herein, the term ‘circuitry’ also includes animplementation comprising one or more processors and / or portion(s) thereof and accompanying software and / or firmware. As another example, the term ‘circuitry’ as used herein also includes, for example, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, other network device, and / or other computing device.

[0070] As defined herein, a ‘computer-readable storage medium,’ which refers to a non-transitory physical storage medium (e.g., volatile or non-volatile memory device), can be differentiated from a ‘computer-readable transmission medium,’ which refers to an electromagnetic signal.

[0071] A method, apparatus and computer program product are provided in accordance with example embodiments for carriage of neural-network post-filter, for example, carriage of neural- network post-filter supplemental enhancement information (SEI) with ISO base media file format (ISOBMFF) .

[0072] In an example, the following describes in detail suitable apparatus and possible mechanisms for carriage of neural-network post-filter, for example, carriage of neural-network postfilter SEI with ISOBMFF. In this regard reference is first made to FIG. 1 and FIG. 2, where FIG. 1 shows an example block diagram of an apparatus 50. The apparatus may be an internet of things (loT) apparatus configured to perform various functions, for example, gathering information by one or more sensors, receiving or transmitting information, analyzing information gathered or received by the apparatus, or the like. The apparatus may comprise a video coding system, which may incorporate a codec. FIG. 2 shows a layout of an apparatus according to an example embodiment. The elements of FIG. 1 and FIG. 2 will be explained next.

[0073] The apparatus 50, may for example be, a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or a lower power device. However, it would be appreciated that embodiments of the examples described herein may be implemented within any electronic device or apparatus which may process data by neural networks.

[0074] The apparatus 50 may comprise a housing 30 for incorporating and protecting the device. The apparatus 50 may further comprise a display 32, for example, in the form of a liquid crystal display, light emitting diode display, organic light emitting diode display, and the like. In other embodiments of the examples described herein the display may be any suitable display technology suitable to display media or multimedia content, for example, an image or a video. The apparatus 50 may further comprise a keypad 34. In other embodiments of the examples described herein any suitable data or user interfacemechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

[0075] The apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analogue signal input. The apparatus 50 may further comprise an audio output device which in embodiments of the examples described herein may be any one of: an earpiece 38, speaker, or an analogue audio or digital audio output connection. The apparatus 50 may also comprise a battery (or in other embodiments of the examples described herein the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise a camera 42 capable of recording or capturing images and / or video. The apparatus 50 may further comprise an infrared port for short range line of sight communication to other devices. In other embodiments the apparatus 50 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB / firewire wired connection.

[0076] The apparatus 50 may comprise a controller 56, a processor or a processor circuitry for controlling the apparatus 50. The controller 56 may be connected to a memory 58 which in embodiments of the examples described herein may store both data in the form of an image, audio data and video data, and / or may also store instructions for implementation on the controller 56. The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and / or decoding of audio, image and / or video data or assisting in coding and / or decoding carried out by the controller.

[0077] The apparatus 50 may further comprise a card reader 48 and a smart card 46, for example, a universal integrated circuit card (UICC) and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.

[0078] The apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals, for example, for communication with a cellular communications network, a wireless communications system or a wireless local area network. The apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and / or for receiving radio frequency signals from other apparatus(es).

[0079] The apparatus 50 may comprise a camera 42 capable of recording or detecting individual frames which are then passed to the codec circuitry 54 or the controller for processing. The apparatus may receive the video image data for processing from another device prior to transmission and / orstorage. The apparatus 50 may also receive either wirelessly or by a wired connection the image for coding / decoding. The structural elements of apparatus 50 described above represent examples of means for performing a corresponding function.

[0080] Fundamentals of video / image coding

[0081] Video codec consists of an encoder that transforms the input video into a compressed representation suited for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form, for example, at lower bitrate.

[0082] Typical hybrid video codecs, for example ITU-T H.263 and H.264, encode the video information in two phases. Firstly, pixel values in a certain picture area (or ‘block’) are predicted, for example, by motion compensation means or circuits (by finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means or circuit (by using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, e.g. the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g. discrete cosine transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder may control the balance between the accuracy of the pixel representation (e.g., picture quality) and size of the resulting coded video representation (e.g., file size or transmission bitrate).

[0083] In temporal prediction, the sources of prediction are previously decoded pictures (e.g., reference pictures). In intra block copy (IBC; also known as intra-block-copy prediction and current picture referencing), prediction is applied similarly to temporal prediction, but the reference picture is the current picture and only previously decoded samples can be referred in the prediction process. Interlayer or inter-view prediction may be applied similarly to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter prediction may refer to temporal prediction only, while in other cases inter prediction may refer collectively to temporal prediction and any of intra block copy, inter-layer prediction, and inter-view prediction provided that they are performed with the same or similar process than temporal prediction. Inter prediction or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction.

[0084] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, for example, either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra-coding, where no inter prediction is applied.

[0085] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently when they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction.

[0086] Scalable video coding refers to coding structure where one bitstream can include multiple representations of the content e.g. at different bitrates, resolutions or frame rates. In these cases, the receiver can extract the desired representation depending on its characteristics (e.g. resolution that matches best the display device). Alternatively, a server or a network element can extract the portions of the bitstream to be transmitted to the receiver depending on e.g. the network characteristics or processing capabilities of the receiver.

[0087] Scalable video coding may be realized through multi-layered coding. Multi-layered coding is a concept wherein an un-encoded visual representation of a scene is, by processes such as transformation and filtering, mapped into multiple dependent or independent representations (called layers). One or more encoders are used to encode a layered visual representation. When the layers include redundancies, the use of a single encoder may, by using inter-layer prediction techniques, encode with a significant gain in coding efficiency. Layered video coding is typically used to provide some form of scalability in services, e.g., quality scalability, spatial scalability, temporal scalability, and view scalability.

[0088] A portion of a scalable video bitstream that provides a certain decoded representation, such as a base quality video or a depth map video for a bitstream that also contains texture video, and is independently decodable from other portions of the scalable video bitstream, may be referred to as an independent layer. A scalable video bitstream may comprise multiple independent layers, e.g., a texture video layer, a depth video layer, and an alpha map video layer. A portion of a scalable video bitstream that provides a certain decoded representation or enhancement, such as a quality enhancement to aparticular fidelity or a resolution enhancement to a certain picture width and height in samples, and requires decoding of one or more other layers (e.g., reference layers) in the scalable video bitstream due to inter-layer prediction may be referred to as a dependent layer or a predicted layer.

[0089] In some scenarios, a scalable bitstream includes a "base layer", which may provide a basic representation, such as the lowest quality video available, and one or more enhancement layers. In order to improve coding efficiency for an enhancement layer, the coded representation of that layer may depend on one or more of the lower layers, e.g., inter-layer prediction may be applied. For example, the motion and mode information of the enhancement layer may be predicted from lower layers. Similarly, the pixel data of the lower layers may be used to create prediction for the enhancement layer. The term enhancement layer may refer to enhancing one or more aspects of reference layer(s), such as quality or resolution. A portion of the bitstream that remains after removal of all enhancement layers may be referred to as the base layer.

[0090] It needs to be understood that the term layer may be conceptual, e.g., the bitstream syntax might not include signaling of layers or the signaling of layers is not in use in a scalable bitstream that conceptually comprises several layers. The term scalability layer may be used interchangeably with the term layer.

[0091] FIG. 3 shows a block diagram of a general structure of a video encoder. FIG. 3 presents an encoder for two layers, but it would be appreciated that presented encoder could be similarly extended to encode more than two layers. FIG. 3 illustrates a video encoder comprising a first encoder section 500 for a base layer and a second encoder section 502 for an enhancement layer. Each of the first encoder section 500 and the second encoder section 502 may comprise similar elements for encoding incoming pictures. The encoder sections 500, 502 may comprise a pixel predictor 302, 402, prediction error encoder 303, 403 and prediction error decoder 304, 404. FIG. 3 also shows an embodiment of the pixel predictor 302, 402 as comprising an inter-predictor 306, 406, an intra-predictor 308, 408, a mode selector 310, 410, a filter 316, 416, and a reference frame memory 318, 418. The pixel predictor 302 of the first encoder section 500 receives base layer picture(s) / image(s) 300 of a video stream to be encoded at both the inter-predictor 306 (which determines the difference between the image and a motion compensated reference frame) and the intra-predictor 308 (which determines a prediction for an image block based only on the already processed parts of current frame or picture). The output of both the inter-predictor and the intra-predictor are passed to the mode selector 310. The intra-predictor 308 may have more than one intra-prediction modes. Hence, each mode may perform the intra-prediction and provide the predicted signal to the mode selector 310. The mode selector 310 also receives a copy of the base layer image(s) 300. Correspondingly, the pixel predictor 402 of the second encoder section 502receives enhancement layer picture(s) / images(s) 400 of a video stream to be encoded at both the interpredictor 406 (which determines the difference between the image and a motion compensated reference frame) and the intra-predictor 408 (which determines a prediction for an image block based only on the already processed parts of current frame or picture). The output of both the inter-predictor and the intra- predictor are passed to the mode selector 410. The intra-predictor 408 may have more than one intraprediction modes. Hence, each mode may perform the intra-prediction and provide the predicted signal to the mode selector 410. The mode selector 410 also receives a copy of the enhancement layer pictures 400.

[0092] Depending on which encoding mode is selected to encode the current block, the output of the inter-predictor 306, 406 or the output of one of the optional intra-predictor modes or the output of a surface encoder within the mode selector is passed to the output of the mode selector 310, 410. The output of the mode selector 310, 410 is passed to a first summing device 321, 421. The first summing device may subtract the output of the pixel predictor 302, 402 from the base layer image(s) 300 / enhancement layer image(s) 400 to produce a first prediction error signal 320, 420 which is input to the prediction error encoder 303, 403.

[0093] The pixel predictor 302, 402 further receives from a preliminary reconstructor 339, 439 the combination of the prediction representation of the image block 312, 412 and the output 338, 438 of the prediction error decoder 304, 404. The preliminary reconstructed image 314, 414 may be passed to the intra-predictor 308, 408 and to the filter 316, 416. The filter 316, 416 receiving the preliminary representation may filter the preliminary representation and output a final reconstructed image 340, 440 which may be saved in the reference frame memory 318, 418. The reference frame memory 318 may be connected to the inter-predictor 306 to be used as the reference image against which a future base layer image 300 is compared in inter-prediction operations. Subject to the base layer being selected and indicated to be source for inter-layer sample prediction and / or inter-layer motion information prediction of the enhancement layer according to some embodiments, the reference frame memory 318 may also be connected to the inter-predictor 406 to be used as the reference image against which a future enhancement layer image(s) 400 is compared in inter-prediction operations. Moreover, the reference frame memory 418 may be connected to the inter-predictor 406 to be used as the reference image against which the future enhancement layer image(s) 400 is compared in inter -prediction operations.

[0094] Filtering parameters from the filter 316 of the first encoder section 500 may be provided to the second encoder section 502 subject to the base layer being selected and indicated to be source for predicting the filtering parameters of the enhancement layer according to some embodiments.

[0095] The prediction error encoder 303, 403 comprises a transform unit 342, 442 and a quantizer 344, 444. The transform unit 342, 442 transforms the first prediction error signal 320, 420 to a transform domain. The transform is, for example, the DCT transform. The quantizer 344, 444 quantizes the transform domain signal, for example, the DCT coefficients, to form quantized coefficients.

[0096] The prediction error decoder 304, 404 receives the output from the prediction error encoder 303, 403 and performs the opposite processes of the prediction error encoder 303, 403 to produce a decoded prediction error signal 338, 438 which, when combined with the prediction representation of the image block 312, 412 at the second summing device 339, 439, produces the preliminary reconstructed image 314, 414. The prediction error decoder may be considered to comprise a dequantizer 346, 446, which dequantizes the quantized coefficient values, for example, DCT coefficients, to reconstruct the transform signal and an inverse transformation unit 348, 448, which performs the inverse transformation to the reconstructed transform signal wherein the output of the inverse transformation unit 348, 448 includes reconstructed block(s). The prediction error decoder may also comprise a block filter which may filter the reconstructed block(s) according to further decoded information and filter parameters.

[0097] The entropy encoder 330, 430 receives the output of the prediction error encoder 303, 403 and may perform a suitable entropy encoding / variable length encoding on the signal to provide a compressed signal. The outputs of the entropy encoders 330, 430 may be inserted into a bitstream, for example, by a multiplexer 508.

[0098] FIG. 4 is a block diagram showing the interface between an encoder 401 implementing neural network based encoding 407, and a decoder 409 implementing neural network based decoding 405 in accordance with the examples described herein. The encoder 401 may embody a device, a software method or a hardware circuit. The encoder 401 has the goal of compressing an input data 411 (for example, an input video) to a compressed data 412 (for example, a bitstream) such that the bitrate measuring the size of compressed data 412 is minimized, and the accuracy of an analysis or processing algorithm is maximized. To this end, the encoder 401 uses an encoder or compression algorithm, for example to perform neural network based encoding 407, e.g., encoding the input data by using one or more neural networks.

[0099] The general analysis or processing algorithm may be part of the decoder 409. The decoder 409 uses a decoder or decompression algorithm, for example, to perform the neural network based decoding 405 (e.g., decoding by using one or more neural networks) to decode the compressed data 412(for example, compressed video) which was encoded by the encoder 401. The decoder 409 produces decompressed data 413 (for example, reconstructed data).

[0100] The encoder 401 and decoder 409 may be entities implementing an abstraction, may be separate entities or the same entities, or may be part of the same physical device.

[0101] The analysis / processing algorithm may be any algorithm, traditional or learned from data. In the case of an algorithm which is learned from data, in some embodiments it is assumed that this algorithm can be modified or updated, for example, by using optimization via gradient descent. An example of the learned algorithm is a neural network.

[0102] An out-of-band transmission, signaling, or storage may refer to the capability of transmitting, signaling, or storing information in a manner that associates the information with a video bitstream. The out-of-band transmission may use a more reliable transmission mechanism compared to the protocols used for carrying coded video data, such as slices. The out-of-band transmission, signaling or storage can additionally or alternatively be used e.g. for ease of access or session negotiation. For example, a sample entry of a track in a file conforming to the ISO Base Media File Format may comprise parameter sets, while the coded data in the bitstream is stored elsewhere in the file or in another file. Another example of out-of-band transmission, signaling, or storage comprises including information, such as NN and / or NN updates in a file format track that is separate from track(s) including coded video data.

[0103] The phrase along the bitstream (e.g. indicating along the bitstream) or along a coded unit of a bitstream (e.g. indicating along a coded tile) may be used in claims and described embodiments to refer to transmission, signaling, or storage in a manner that the ‘out-of-band’ data is associated with, but not included within, the bitstream or the coded unit, respectively. The phrase decoding along the bitstream or along a coded unit of a bitstream or alike may refer to decoding the referred out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) that is associated with the bitstream or the coded unit, respectively. For example, the phrase along the bitstream may be used when the bitstream is included in a container file, such as a file conforming to the ISO Base Media File Format, and certain file metadata is stored in the file in a manner that associates the metadata to the bitstream, such as boxes in the sample entry for a track including the bitstream, a sample group for the track including the bitstream, or a timed metadata track associated with the track including the bitstream. In another example, the phrase along the bitstream may be used when the bitstream is made available as a stream over a communication protocol and a media description, such as a streaming manifest, is provided to describe the stream.

[0104] A bitstream may be defined as a sequence of bits or a sequence of syntax structures. A bitstream format may constrain the order of syntax structures in the bitstream.

[0105] A syntax element may be defined as an element of data represented in a bitstream. A syntax structure may be defined as zero or more syntax elements present together in a bitstream in a specified order.

[0106] An elementary unit for the output of a video encoder and the input of a video decoder, respectively, may be a network abstraction layer (NAL) unit. For transport over packet-oriented networks or storage into structured files, NAL units may be encapsulated into packets or similar structures. A bytestream format encapsulating NAL units may be used for transmission or storage environments that do not provide framing structures. The bytestream format may separate NAL units from each other by attaching a start code in front of each NAL unit. To avoid false detection of NAL unit boundaries, encoders may run a byte-oriented start code emulation prevention algorithm, which may add an emulation prevention byte to the NAL unit payload when a start code would have occurred otherwise. In order to enable straightforward gateway operation between packet and stream-oriented systems, start code emulation prevention may be performed regardless of whether the bytestream format is in use or not. A NAL unit may be defined as a syntax structure including an indication of the type of data to follow and bytes including that data in the form of a raw byte sequence payload interspersed as necessary with emulation prevention bytes. A raw byte sequence payload (RBSP) may be defined as a syntax structure including an integer number of bytes that is encapsulated in a NAL unit. An RBSP is either empty or has the form of a string of data bits including syntax elements followed by an RBSP stop bit and followed by zero or more subsequent bits equal to 0.

[0107] A bitstream may be defined to logically include a syntax structure, such as a NAL unit, when the syntax structure is transmitted along the bitstream but may be included in the bitstream according to the bitstream format. A bitstream may be defined to natively comprise a syntax structure, when the bitstream includes the syntax structure.

[0108] In some coding formats or standards, a bitstream may be in the form of a network abstraction layer (NAL) unit stream or a byte stream, that forms the representation of coded pictures and associated data forming one or more coded video sequences.

[0109] In some coding formats or standards, the end of a bitstream may be indicated by a specific NAL unit, which may be referred to as the end of bitstream (EOB) NAL unit and which is the last NAL unit of the bitstream.

[0110] In some formats or standards, a first bitstream may be followed by a second bitstream in the same logical channel, such as in the same file or in the same connection of a communication protocol. An elementary stream (in the context of video coding) may be defined as a sequence of one or more bitstreams.

[0111] In some coding formats, such as AVI, a bitstream may comprise a sequence of open bitstream units (OBUs). An OBU comprises a header and a payload, wherein the header identifies a type of the OBU. Furthermore, the header may comprise a size of the payload in bytes.

[0112] In some coding standards, NAL units include of a header and payload. The NAL unit header indicates the type of the NAL unit. In some coding standards, the NAL unit header indicates a scalability layer identifier (e.g. called nuh_layer_id in H.265 / HEVC and H.266 / VVC), which could be used e.g. for indicating spatial or quality layers, views of a multiview video, or auxiliary layers (such as depth maps or alpha planes). In some coding standards, the NAL unit header includes a temporal sublayer identifier, which may be used for indicating temporal subsets of the bitstream, such as a 30- frames-per-second subset of a 60-frames-per-second bitstream.

[0113] NAL units may be categorized into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are typically coded slice NAL units.

[0114] A non-VCL NAL unit may be, for example, one of the following types: a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence NAL unit, an end of bitstream NAL unit, or a filler data NAL unit. Parameter sets may be needed for the reconstruction of decoded pictures, whereas many of the other non-VCL NAL units are not necessary for the reconstruction of decoded sample values.

[0115] Some coding formats specify parameter sets that may carry parameter values needed for the decoding or reconstruction of decoded pictures. A parameter may be defined as a syntax element of a parameter set. A parameter set may be defined as a syntax structure that includes parameters and that can be referred to from or activated by another syntax structure, for example, using an identifier.

[0116] Some types of parameter sets are briefly described in the following, but it needs to be understood, that other types of parameter sets may exist and that embodiments may be applied, but are not limited to, the described types of parameter sets.

[0117] Parameters that remain unchanged through a coded video sequence may be included in a sequence parameter set. Alternatively, an SPS may be limited to apply to a layer that references the SPS, e.g. an SPS may remain valid for a coded layer video sequence. In addition to the parameters that may be needed by the decoding process, the sequence parameter set may optionally include video usability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation.

[0118] A picture parameter set includes such parameters that are likely to be unchanged in several coded pictures. A picture parameter set may include parameters that can be referred to by the VCL NAL units of one or more coded pictures.

[0119] A video parameter set (VPS) may be defined as a syntax structure including syntax elements that apply to zero or more entire coded video sequences and may include parameters applying to multiple layers. The VPS may provide information about the dependency relationships of the layers in a bitstream, as well as many other information that are applicable to all slices across all layers in the entire coded video sequence.

[0120] A video parameter set RBSP may include parameters that can be referred to by one or more sequence parameter set RBSPs.

[0121] The relationship and hierarchy between a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS) may be described as follows. A VPS resides one level above an SPS in the parameter set hierarchy and in the context of scalability. The VPS may include parameters that are common for all slices across all layers in the entire coded video sequence. The SPS includes the parameters that are common for all slices in a particular layer in the entire coded video sequence, and may be shared by multiple layers. The PPS includes the parameters that are common for all slices in a particular picture and are likely to be shared by all slices in multiple pictures.

[0122] An adaptation parameter set (APS) may be specified in some coding formats, such as H.266 / VVC. An APS may be applied to one or more image segments, such as slices. In H.266 / VVC, an APS may be defined as a syntax structure including syntax elements that apply to zero or more slices as determined by zero or more syntax elements found in slice headers or in a picture header. An APSmay comprise a type (aps_params_type in H.266 / VVC) and an identifier (aps_adaptation_parameter_set_id in H.266 / VVC). The combination of an APS type and an APS identifier may be used to identify a particular APS. H.266 / VVC comprises three APS types: an adaptive loop filtering (ALF), a luma mapping with chroma scaling (LMCS), and a scaling list APS types. The ALF APS(s) are referenced from a slice header (thus, the referenced ALF APSs can change slice by slice), and the LMCS and scaling list APS(s) are referenced from a picture header (thus, the referenced LMCS and scaling list APSs can change picture by picture). In H.266 / VVC, the APS RBSP has the following syntax:

[0123] Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or alike. Some video coding specifications include SEI NAL units, and some video coding specifications include both prefix SEI NAL units and suffix SEI NAL units. A prefix SEI NAL unit can start a picture unit or alike; and a suffix SEI NAL unit can end a picture unit or alike. Hereafter, an SEI NAL unit may equivalently refer to a prefix SEI NAL unit or a suffix SEI NAL unit. An SEI NAL unit includes one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, post-processing of decoded pictures, rendering, error detection, error concealment, and resource reservation.

[0124] Several SEI messages are specified in H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards, and the user data SEI messages enable organizations and companies to specify SEI messages for specific use. The standards may include the syntax and semantics for the specified SEI messages but a process for handling the messages in the recipient might not be defined. Consequently, encoders may be required to follow the standard specifying a SEI message when they create SEI message(s), and decoders might not be required to process SEI messages for output order conformance. One of the reasons to include the syntax and semantics of SEI messages in standards is to allow different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified.

[0125] Some video coding specifications enable metadata OBUs. A metadata OBU comprises a type field, which specifies the type of metadata.

[0126] A coded video sequence (CVS) may be defined as a sequence of coded pictures in decoding order that is independently decodable and is followed by another coded video sequence or the end of the bitstream.

[0127] A coded layer video sequence (CLVS) may be defined as a sequence of pictures and associated other data within the same scalable layer (e.g., with the same value of nuh_layer_id in VVC) that is decodable independently of other pictures in the same layer.

[0128] An identifier may be defined as a syntax element that identifies a syntax structure. A value of the identifier may for example differ in different instances of the same syntax structure, such as a parameter set. A particular instance of the syntax structure may be referenced through its identifier value. For example, a parameter set that is referenced by the (de)coding of a coded video slice may be identified by providing the identifier value of the parameter set in a header of the coded video slice.

[0129] An indicator (ide) may be defined as a syntax element whose value indicates a selection among more than two values (for which semantics have been specified). An indicator syntax element may have _idc postfix in its name.

[0130] A uniform resource identifier (URI) may be defined as a string of characters used to identify a name of a resource. Such identification enables interaction with representations of the resource over a network, using specific protocols. A URI is defined through a scheme specifying aconcrete syntax and associated protocol for the URL The uniform resource locator (URL) and the uniform resource name (URN) are forms of URL A URL may be defined as a URI that identifies a web resource and specifies the means of acting upon or obtaining the representation of the resource, specifying both its primary access mechanism and network location. A URN may be defined as a URI that identifies a resource by name in a particular namespace. A URN may be used for identifying a resource without implying its location or how to access it.

[0131] Internet media types, also known as multipurpose internet mail extension (MIME) types, are used by various applications to identify the type of a resource or a file. MIME types include a media type (e.g., ‘image’ in the case of still images), a subtype, and zero or more optional parameters.

[0132] The MIME is an extension to an email protocol which makes it possible to transmit and receive different kinds of data files on the Internet, for example video, audio, images, and software. An internet media type is an identifier used on the Internet to indicate the type of data that a file includes. Such internet media types may also be called as content types. Several MIME type / subtype combinations exist that may include different media formats. Content type information may be included by a transmitting entity in a MIME header at the beginning of a media transmission. A receiving entity thus may need to examine the details of such media content to determine when the specific elements may be rendered given an available set of codecs. Especially, when the end system has limited resources, or the connection to the end system has limited bandwidth, it may be helpful to know from the content type alone if the content can be rendered.

[0133] One of the original motivations for MIME is the ability to identify the specific media type of a message part. However, due to various factors, it is not always possible from looking at the MIME type and subtype to know which specific media formats are included in the body part or which codecs are indicated in order to render the content. Optional media parameters may be provided in addition to the MIME type and subtype to provide further details of the media content.

[0134] An optional ‘codecs’ MIME parameter is specified to be used with various MIME types or type / subtype combinations to allow for unambiguous specification of the codecs employed by the media formats included within the overall container format.

[0135] By labelling content with the specific codecs indicated to render the included media, receiving systems may determine when the codecs are supported by the end system, and when not, may take appropriate action (such as rejecting the content, sending notification of the situation, transcodingthe content to a supported type, fetching and installing the required codecs, further inspection to determine when it may be sufficient to support a subset of the indicated codecs, and the like).

[0136] For file formats derived from the ISOBMFF, the codecs parameter may be considered to comprise a comma-separated list of one or more list items.

[0137] When a list item of the codecs parameter represents a track of an ISOBMFF compliant file, the list item may comprise a four-character code of the sample entry of the track. For NAL unit structured video, the format of the list item is specified in ISO / IEC 14496-15.

[0138] The method and apparatus of an example embodiment may be utilized in a wide variety of systems, including systems that rely upon the compression and decompression of media data and possibly also the associated metadata. In at least an embodiment, however, the method and apparatus are configured to train or finetune a decoder-side neural network. In this regard, FIG. 5 depicts an example of such a system 500 that includes a source 502 of media data and associated metadata. The source 502 may be, in an embodiment, a server. However, the source may be embodied in other manners when desired. The source 502 is configured to stream the media data and associated metadata to a client device 504. The client device may be embodied by a media player, a multimedia system, a video system, a smart phone, a mobile telephone or other user equipment, a personal computer, a tablet computer or any other computing device configured to receive and decompress the media data and process associated metadata. In the illustrated embodiment, media data and metadata are streamed via a network 506, such as any of a wide variety of types of wireless networks and / or wireline networks. The client device is configured to receive structured information including media, metadata and any other relevant representation of information including the media and the metadata and to decompress the media data and process the associated metadata (e.g. for proper playback timing of decompressed media data).

[0139] An apparatus 600 is provided in accordance with an example embodiment as shown in FIG. 6. In an embodiment, the apparatus of FIG. 6 may be embodied by the source 502, such as a file writer which, in turn, may be embodied by a server, that is configured to stream a compressed representation of the media data and associated metadata. In an alternative embodiment, the apparatus may be embodied by the client device 504, such as a file reader which may be embodied, for example, by any of the various computing devices described above. In either of these embodiments and as shown in FIG. 6, the apparatus of an example embodiment includes, is associated with or is in communication with a processing circuitry 602, one or more memory devices 604, a communication interface 606 and optionally a user interface.

[0140] The processing circuitry 602 may be in communication with the memory device 604 via a bus for passing information among components of the apparatus 600. The memory device may be non- transitory and may include, for example, one or more volatile and / or non-volatile memories. In other words, for example, the memory device may be an electronic storage device (e.g., a computer readable storage medium) comprising gates configured to store data (e.g., bits) that may be retrievable by a machine (e.g., a computing device like the processing circuitry). The memory device may be configured to store information, data, content, applications, instructions, or the like for enabling the apparatus to carry out various functions in accordance with an example embodiment of the present disclosure. For example, the memory device could be configured to buffer input data for processing by the processing circuitry. Additionally or alternatively, the memory device could be configured to store instructions for execution by the processing circuitry.

[0141] The apparatus 600 may, in some embodiments, be embodied in various computing devices as described above. However, in some embodiments, the apparatus may be embodied as a chip or chip set. In other words, the apparatus may comprise one or more physical packages (e.g., chips) including materials, components and / or wires on a structural assembly (e.g., a baseboard). The structural assembly may provide physical strength, conservation of size, and / or limitation of electrical interaction for component circuitry included thereon. The apparatus may therefore, in some cases, be configured to implement an embodiment of the present disclosure on a single chip or as a single ‘system on a chip.’ As such, in some cases, a chip or chipset may constitute means for performing one or more operations for providing the functionalities described herein.

[0142] The processing circuitry 602 may be embodied in a number of different ways. For example, the processing circuitry may be embodied as one or more of various hardware processing means such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing element with or without an accompanying DSP, or various other circuitry including integrated circuits such as, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like. As such, in some embodiments, the processing circuitry may include one or more processing cores configured to perform independently. A multi-core processing circuitry may enable multiprocessing within a single physical package. Additionally or alternatively, the processing circuitry may include one or more processors configured in tandem via the bus to enable independent execution of instructions, pipelining and / or multithreading.

[0143] In an example embodiment, the processing circuitry 602 may be configured to execute instructions stored in the memory device 604 or otherwise accessible to the processing circuitry.Alternatively or additionally, the processing circuitry may be configured to execute hard coded functionality. As such, whether configured by hardware or software methods, or by a combination thereof, the processing circuitry may represent an entity (e.g., physically embodied in circuitry) capable of performing operations according to an embodiment of the present disclosure while configured accordingly. Thus, for example, when the processing circuitry is embodied as an ASIC, FPGA or the like, the processing circuitry may be specifically configured hardware for conducting the operations described herein. Alternatively, as another example, when the processing circuitry is embodied as an executor of instructions, the instructions may specifically configure the processing circuitry to perform the algorithms and / or operations described herein when the instructions are executed. However, in some cases, the processing circuitry may be a processor of a specific device (e.g., an image or video processing system) configured to employ an embodiment of the present invention by further configuration of the processing circuitry by instructions for performing the algorithms and / or operations described herein. The processing circuitry may include, among other things, a clock, an arithmetic logic unit (ALU) and logic gates configured to support operation of the processing circuitry.

[0144] The communication interface 606 may be any means such as a device or circuitry embodied in either hardware or a combination of hardware and software that is configured to receive and / or transmit data, including video bitstreams. In this regard, the communication interface may include, for example, an antenna (or multiple antennas) and supporting hardware and / or software for enabling communications with a wireless communication network. Additionally or alternatively, the communication interface may include the circuitry for interacting with the antenna(s) to cause transmission of signals via the antenna(s) or to handle receipt of signals received via the antenna(s). In some environments, the communication interface may alternatively or also support wired communication. As such, for example, the communication interface may include a communication modem and / or other hardware / software for supporting communication via cable, digital subscriber line (DSL), universal serial bus (USB) or other mechanisms.

[0145] In some embodiments, the apparatus 600 may optionally include a user interface that may, in turn, be in communication with the processing circuitry 602 to provide output to a user, such as by outputting an encoded video bitstream and, in some embodiments, to receive an indication of a user input. As such, the user interface may include a display and, in some embodiments, may also include a keyboard, a mouse, a joystick, a touch screen, touch areas, soft keys, a microphone, a speaker, or other input / output mechanisms. Alternatively or additionally, the processing circuitry may comprise user interface circuitry configured to control at least some functions of one or more user interface elements such as a display and, in some embodiments, a speaker, ringer, microphone and / or the like. The processing circuitry and / or user interface circuitry comprising the processing circuitry may beconfigured to control one or more functions of one or more user interface elements through computer program instructions (e.g., software and / or firmware) stored on a memory accessible to the processing circuitry (e.g., memory device, and / or the like).

[0146] SEI prefix indication SEI message

[0147] The SEI prefix indication SEI message has been specified, for example, in HEVC and VVC. The SEI prefix indication SEI message carries one or more SEI prefix indications for SEI messages of a particular value of SEI payload type (payloadType). Each SEI prefix indication is a bit string that follows the SEI payload syntax of that value of payloadType and includes a number of complete syntax elements starting from the first syntax element in the SEI payload.

[0148] Each SEI prefix indication for an SEI message of a particular value of payloadType indicates that one or more SEI messages of this value of payloadType are expected or likely to be present in the coded video sequence (CVS), and to start with the provided bit string. A starting bit string would typically include a true subset of an SEI payload of the type of SEI message indicated by the payloadType, and may include a complete SEI payload.

[0149] SEI prefix indications should provide sufficient information for indicating what type of processing is needed or what type of content is included. The former (type of processing) indicates decoder-side processing capabilities, e.g., whether some type of frame unpacking is needed. The latter (type of content) indicates, for example, whether the bitstream includes subtitle captions in a particular language.

[0150] SEI processing order SEI message

[0151] The SEI processing order SEI message has been described, for example, in document JVET-AA2027. The SEI processing order SEI message carries information indicating a preferred processing order, as determined by the encoder (e.g., the content producer), for different types of SEI messages that may be present in the bitstream. When an SEI processing order SEI message is present, it is present in the first access unit of the coded video sequence. The SEI processing order SEI message persists in decoding order from the current access unit until the end of the CVS. The SEI processing order SEI message comprises a list of pairs, each pair comprising a SEI payload type value po_sei_payload_type[ i ] and a processing order value po_sei_processing_order[ i ]. po_sei_payload_type[ i ] specifies a value of payloadType for the i-th SEI message for which information is provided in the SEI processing order SEI message. po_sei_processing_order[ i ] indicatesthe preferred order of processing any SEI message with payloadType equal to po_sei_payload_type[ i ]. po_sei_processing_order[ m ] greater than 0 and less than po_sei_processing_order[ n ] indicates any SEI message with payloadType equal to po_sei_payload_type[ m ], when present, should be processed before any SEI message with payloadType equal to po_sei_payload_type[ n ]. po_sei_processing_order[ m ] greater than 0 and equal to po_sei_processing_order[ n ] indicates that the preferred order of processing of SEI messages with payloadTypes equal to po_sei_payload_type[ m ] and po_sei_payload_type[ n ] is unknown, unspecified, or determined by external means. po_sei_processing_order[ i ] equal to 0 specifies that the preferred order of processing SEI messages with payloadType equal to po_sei_payload_type[ i ] is unknown, unspecified, determined by external means.

[0152] Neural-network post-filter characteristics (NNPFC) and neural- network post-filter activation (NNPFA) SEI messages

[0153] The neural-network post-filter characteristics (NNPFC) SEI message and the neural- network post-filter activation (NNPFA) SEI message have been described, for example, JVET- AD2006, which specifies a draft amendment to the versatile supplemental enhancement information (VSEI) standard.

[0154] The syntax structure specifying the NNPFC SEI message may be called nn_post_filter_characteristics. The syntax structure specifying the NNPFA SEI message may be called nn_po st_filter_acti vation .

[0155] An NNPFC SEI message identifies an applicable post-processing filter associated with the nnpfc_id value. The use of applicable post-processing filters with different values of nnpfc_id for specific pictures is indicated with neural-network post-filter activation (NNPFA) SEI messages.

[0156] An NNPFC SEI message either specifies a base post-processing filter or a neural network update. A base post-processing filter is defined by the first NNPFC SEI message, in decoding order, that has a particular nnpfc_id value within a CLVS. When there is no subsequent NNPFC SEI message that has the same nnpfc_id value as the base post-processing filter, the applicable post-processing filter is the same as the base post-processing filter. Otherwise, the applicable post-processing filter is obtained by applying the update defined by a subsequent NNPFC SEI message relative to the base postprocessing filter.

[0157] The NNPFC SEI message comprises the nnpfc_id syntax element, which includes an identifying number that may be used to identify a post-processing filter. A base post-processing filter is the filter that is contained in or identified by the first NNPFC SEI message, in decoding order, that has a particular nnpfc_id value within a coded layer video sequence (CLVS). If an NNPFC SEI message is neither the first NNPFC SEI message, in decoding order, in the current CLVS nor a repetition of the first NNPFC SEI message, in decoding order, that has a particular nnpfc_id value within the current CLVS, the NNPFC SEI message defines an update relative to the base post-processing filter, and the update relative to the base post-processing filter is applied to obtain a post-processing filter associated with the nnpfc_id value. The update may be obtained by decoding the coded neural network bitstream in the second NNPFC SEI message (when nnpfc_mode_idc is equal to 0) or through the Uniform Resource Identifier defining the update (when nnpfc_mode_idc is equal to 1). Otherwise (i.e., when there is no update defined by an NNPFC SEI message), the post-processing filter associated with the nnpfc_id value is assigned to be the same as the base post-processing filter.

[0158] The NNPFC SEI message comprises nnpfc_mode_idc syntax element, the semantics of which may be defined as follows: nnpfc_mode_idc equal to 0 indicates that this SEI message includes an ISO / IEC 15938-17 bitstream that specifies the base post-processing filter or updates the base post-processing filter with the same nnpfc_id value. The ISO / IEC 15938-17 bitstream may be at the end of the NNPFC SEI message. In other words, no syntax elements may follow the ISO / IEC 15938-17 bitstream within the NNPFC SEI message. nnpfc_mode_idc equal to 1 specifies that the base post-processing filter or the update relative to the base post-processing filter associated with the nnpfc_id value is a neural network identified by the uniform resource identifier (URI) nnpfc_uri with the format identified by the tag URI nnpfc_tag_uri.

[0159] The NNPFC SEI message may also comprise:Purpose of the post-processing filter, which may comprise one or more of the following:- Visual quality improvement.- Chroma upsampling from the 4:2:0 chroma format to the 4:2:2 or 4:4:4 chroma format, or from the 4:2:2 chroma format to the 4:4:4 chroma format.- Resolution resampling, i.e., changing the width or height of the cropped decoded output picture without changing the chroma format- Bit depth upsampling- ColorizationFormatting of the input tensors that are given as input to the neural network inference.Formatting of the output tensors that are resulting from the neural network inference. Characterization of the complexity of the neural network.

[0160] The NNPFA SEI message specifies the neural-network post-processing filter that may be used for post-processing filtering for the current picture, or for post-processing filtering for the current picture and one or more other pictures. The NNPFA SEI message comprises the nnpfa_target_id syntax element, which specifies that the neural -network post-processing filter with nnpfc_id equal to nnfpa_target_id may be used for post-processing filtering for the indicated persistence. The indicated persistence may be the current picture only (indicated by nnpfa_persistence_flag equal to 0). Alternatively, the NNPF activation may be indicated to be persistent by nnpfa_persistence_flag equal to 1, in which case the persistence of the NNPF activation may last until the end of the current CLVS or the next picture, in output order, in the current layer associated with a NNPFA SEI message with the same nnpfa_target_id as the current SEI.

[0161] The NNPFA SEI message syntax may comprise a syntax element indicative if the base post-processing filter or the latest post-processing filter is activated, where the latest post-processing filter is defined by the base post-processing filter relative to which the latest filter update, if any, has been applied. The syntax element may be called nnpfa_target_base_flag. nnpfa_target_base_flag equal to 1 specifies that the target NNPF is the base NNPF with nnpfc_id equal to nnpfa_target_id. nnpfa_target_base_flag equal to 0 specifies that the target NNPF is the NNPF specified by the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id that precedes the first VCL NAL unit of the current picture in decoding order and is not a repetition of the NNPFC SEI message that contains the base NNPF.

[0162] ISO base media file format

[0163] Available media file format standards include International Standards Organization (ISO) base media file format (ISO / IEC 14496-12, which may be abbreviated ISOBMFF), Moving Picture Experts Group (MPEG)-4 file format (ISO / IEC 14496-14, also known as the MP4 format or file format for MPEG-4 Part 14 files), file format for network abstraction ayer (NAL) unit structured video (ISO / IEC 14496-15) and High Efficiency Video Coding standard (HEVC or H.265 / HEVC).

[0164] Some concepts, structures, and specifications of ISOBMFF are described below as an example of a container file format, based on which some embodiments may be implemented. The features described in various embodiments are not limited to ISOBMFF, but rather the description is given for one possible basis on top of which at least some embodiments may be partly or fully realized.

[0165] A basic building block in the ISO base media file format is called a box. Each box has a header and a payload. The box header indicates the type of the box and the size of the box in terms of bytes. Box type is typically identified by an unsigned 32-bit integer, interpreted as a four character code (4CC). A box may enclose other boxes, and the ISO file format specifies which box types are allowed within a box of a certain type. Furthermore, the presence of some boxes may be mandatory in each file, while the presence of other boxes may be optional. Additionally, for some box types, it may be allowable to have more than one box present in a file. Thus, the ISO base media file format may be considered to specify a hierarchical structure of boxes.

[0166] In files conforming to the ISO base media file format, the media data may be provided in one or more instances of MediaDataBox (‘mdat‘) and the MovieBox (‘moov’) may be used to enclose the metadata for timed media. In some cases, for a file to be operable, both of the ‘mdat’ and ‘moov’ boxes may be required to be present. The ‘moov’ box may include one or more tracks, and each track may reside in one corresponding TrackBox (‘trak’). Each track is associated with a handler, identified by a four-character code, specifying the track type. Video, audio, and image sequence tracks may be collectively called media tracks, and they include an elementary media stream. Other track types comprise hint tracks and timed metadata tracks.

[0167] Tracks comprise samples, such as audio or video frames. For video tracks, a media sample may correspond to a coded picture or an access unit.

[0168] A media track refers to samples (which may also be referred to as media samples) formatted according to a media compression format (and its encapsulation to the ISO base media file format). A hint track refers to hint samples, including cookbook instructions for constructing packets for transmission over an indicated communication protocol. A timed metadata track may refer to samples describing referred media and / or hint samples.

[0169] The 'trak' box includes in its hierarchy of boxes the SampleDescriptionBox, which gives detailed information about the coding type used, and any initialization information needed for that coding. The SampleDescriptionBox includes an entry-count and as many sample entries as the entrycount indicates. The format of sample entries is track-type specific but derived from generic classes (e.g., VisualSampleEntry, AudioS ampleEntry). Which type of sample entry form is used for derivation of the track-type specific sample entry format is determined by the media handler of the track.

[0170] The track reference mechanism may be used to associate tracks with each other. The TrackReferenceBox includes box(es), each of which provides a reference from the containing track to a set of other tracks. These references are labeled through the box type (e.g., the four-character code of the box) of the included box(es).

[0171] The ISO Base Media File Format includes three mechanisms for timed metadata that may be associated with particular samples: sample groups, timed metadata tracks, and sample auxiliary information. A derived specification may provide similar functionality with one or more of these three mechanisms.

[0172] A sample grouping in the ISO base media file format and its derivatives, such as the advanced video coding (AVC) file format and the scalable video coding (SVC) file format, may be defined as an assignment of each sample in a track to be a member of one sample group, based on a grouping criterion. A sample group in a sample grouping is not limited to being contiguous samples and may include non-adjacent samples. As there may be more than one sample grouping for the samples in a track, each sample grouping may have a type field to indicate the type of grouping. Sample groupings may be represented by two linked data structures: (1) a SampleToGroupBox (sbgp box) represents the assignment of samples to sample groups; and (2) a SampleGroupDescriptionBox (sgpd box) includes a sample group entry for each sample group describing the properties of the group. There may be multiple instances of the SampleToGroupBox and SampleGroupDescriptionBox based on different grouping criteria. These may be distinguished by a type field used to indicate the type of grouping. SampleToGroupBox may comprise a grouping_type_parameter field that may be used, for example, to indicate a sub-type of the grouping.

[0173] In ISOMBFF, an edit list provides a mapping between the presentation timeline and the media timeline. Among other things, an edit list provides for the linear offset of the presentation of samples in a track, provides for the indication of empty times and provides for a particular sample to be dwelled on for a certain period of time. The presentation timeline may be accordingly modified to provide for looping, such as for the looping videos of the various regions of the scene. One example of the box that includes the edit list, the EditListBox, is provided below: aligned(8) class EditListBox extends FullBox(‘elst’, version, flags) { unsigned int(32) entry _count; or (i=l ; i <= entry_count; i++) { if (version==l) { unsigned int(64) segment_duration;int(64) media lime;} else { / / version==0 unsigned int(32) segment_duration; int(32) media lime;} int(16) media_rate_integer; int(16) media_rate_fraction = 0;}}

[0174] In ISOBMFF, an EditListBox may be included in EditBox, which is included in TrackBox ('trak').

[0175] In this example of the edit list box, flags specifies the repetition of the edit list. By way of example, setting a specific bit within the box flags (the least significant bit, e.g., flags & 1 in ANSI-C notation, where & indicates a bit-wise AND operation) equal to 0 specifies that the edit list is not repeated, while setting the specific bit (.g., flags & 1 in ANSI-C notation) equal to 1 specifies that the edit list is repeated. The values of box flags greater than 1 may be defined to be reserved for future extensions. As such, when the edit list box indicates the playback of zero or one samples, (flags & 1) shall be equal to zero. When the edit list is repeated, the media at time 0 resulting from the edit list follows immediately the media having the largest time resulting from the edit list such that the edit list is repeated seamlessly.

[0176] In ISOBMFF, a Track group enables grouping of tracks based on certain characteristics or the tracks within a group have a particular relationship. Track grouping, however, does not allow any image items in the group.

[0177] The syntax of TrackGroupBox in ISOBMFF is as follows; aligned(8) class TrackGroupBox extends Box('trgr') {} aligned(8) class TrackGroupTypeBox(unsigned int(32) track_group_type) extends FullBox(track_group_type, version = 0, flags = 0){ unsigned int(32) track_group_id; / / the remaining data may be specified for a particular track_group_type}

[0178] track_group_type indicates the grouping_type and shall be set to one of the following values, or a value registered, or a value from a derived specification or registration:

[0179] 'msrc' indicates that this track belongs to a multi-source presentation. The tracks that include the same value of track_group_id within a TrackGroupTypeBox of track_group_type 'msrc' are mapped as being originated from the same source. For example, a recording of a video telephony call may have both audio and video for both participants, and the value of track_group_id associated with the audio track and the video track of one participant differs from value of track_group_id associated with the tracks of the other participant.

[0180] The pair of track_group_id and track_group_type identifies a track group within the file. The tracks that include a particular TrackGroupTypeBox having the same value of track_group_id and track_group_type belong to the same track group.

[0181] The Entity grouping is similar to track grouping but enables grouping of both tracks and image items in that same group.

[0182] The syntax of EntityToGroupBox in ISOBMFF is as follows: aligned(8) class EntityToGroupBox(grouping_type, version, flags) extends FullBox(grouping_type, version, flags) { unsigned int(32) group_id; unsigned int(32) num_entities_in_group; for(i=0; i<num_entities_in_group; i++) unsigned int(32) entity _id;}

[0183] group_id is a non-negative integer assigned to the particular grouping that shall not be equal to any group_id value of any other EntityToGroupBox, any item_ID value of the hierarchy level (file, movie, or track) that includes the GroupsListBox, or any track_ID value (when the GroupsListBox is included in the file level).

[0184] num_entities_in_group specifies the number of entity _id values mapped to this entity group.

[0185] entity _id is resolved to an item, when an item with item_ID equal to entity _id is present in the hierarchy level (file, movie, or track) that includes the GroupsListBox, or to a track, when a track with track_ID equal to entity _id is present and the GroupsListBox is included in the file level.

[0186] Files conforming to the ISOBMFF may include any non-timed objects, referred to as items, meta items, or metadata items, in a meta box (four-character code: ‘meta’ ). While the name of the meta box refers to metadata, items may generally include metadata or media data. The meta box may reside at the top level of the file, within a movie box (four-character code: ‘moov’), and within a track box (four-character code: ‘trak’), but at most one meta box may occur at each of the file level, movie level, or track level. The meta box may be required to include a ‘ hd I r’ box indicating the structure or format of the ‘meta’ box contents. The meta box may list and characterize any number of items that can be referred and each one of them can be associated with a file name and are uniquely identified with the file by item identifier (item_id) which is an integer value. The metadata items may be, for example, stored in the 'idat' box of the meta box or in an 'mdaf box or reside in a separate file. When the metadata is located external to the file then its location may be declared by the DatalnformationBox (four- character code: ‘dinf’). In the specific case that the metadata is formatted using extensible Markup Language (XML) syntax and is required to be stored directly in the MetaBox, the metadata may be encapsulated into either the XMLBox (four-character code: ‘xml’) or the BinaryXMLBox (four- character code: ‘bxml’). An item may be stored as a contiguous byte range, or it may be stored in several extents, each being a contiguous byte range. In other words, items may be stored fragmented into extents, e.g. to enable interleaving. An extent is a contiguous subset of the bytes of the resource. The resource may be formed by concatenating the extents.

[0187] A common base structure is used to include general untimed metadata. This structure is called the MetaBox as it was originally designed to carry metadata, e.g., data that is annotating other data. However, it is now used for a variety of purposes including the carriage of data that is not annotating other data, especially when present at ‘file level’ .

[0188] The MetaBox is required to include a HandlerBox indicating the structure or format of the MetaBox contents.

[0189] All other included boxes are specific to the format specified by the HandlerBox.

[0190] The other boxes defined here may be defined as optional or mandatory for a given format. When they are used, then they shall take the form specified here. These optional boxes include a DatalnformationBox, which documents other files in which metadata values (e.g., pictures) are placed,and an ItemLocationBox, which documents where in those files each item is located (e.g. in the common case of multiple pictures stored in the same file).

[0191] At most one MetaBox may occur at each of the file level, segment, movie level, or track level.

[0192] When an ItemProtectionBox occurs, then some or all of the metadata, including possibly the primary resource, may have been protected and be un-readable unless the protection system is taken into account.

[0193] The MetaBox is unusual in that it is a container box yet extends a FullBox, and not a Box.

[0194] Metadata items are identified by item_ID. Within a given MetaBox, a given item_ID shall uniquely refer to a single item. When an item is updated in movie fragments, the item_ID refers to the latest received version.

[0195] Derived specifications may further restrict the criteria for uniqueness: unique among the item_IDs in both file and movie-level boxes, or unique within that set extended with the track_ID of the tracks in a movie box. The item_ID value of 0 should not be used, and shall not be used when the set is extended to include track_IDs.

[0196] There are three scopes for item_IDs: file and segments; MovieBox and MovieFragmentBox; and TrackBox and TrackFragmentBox. In other words, there shall be only one item with a given item_ID within a given scope (e.g., in the TrackBox and all TrackFragmentBox with the same track_ID).

[0197] Following is an example syntax for byte alignment in ISO / IEC 14496 part 1: aligned(8) class MetaBox (handler_type) extends FullBoxfmeta', version = 0, 0) {HandlerBox(handler_type) theHandler;Primary ItemB ox primary _resource: / / optionalD atalnformationB ox file_locations; / / optionalItemLocationBox item_locations; / / optionalItemProtectionBox protections; / / optionalItemlnfoBox item_infos; / / optionalIPMPControlBox IPMP_control; / / optionalItemReferenceBox item_refs; / / optionalItemDataBox item_data: / / optionalBox other_boxes[]; / / optional}

[0198] The structure or format of the metadata is declared by the handler. In the case that the primary data is identified by a primary item, and that primary item has an item information entry with an item_type, the handler type may be the same as the item_type.

[0199] The ItemPropertiesBox enables the association of any item with an ordered set of item properties. Item properties may be regarded as small data records. The ItemPropertiesBox includes two parts: ItemPropertyContainerBox that includes an implicitly indexed list of item properties, and one or more ItemProperty AssociationBox(es) that associate items with item properties.

[0200] Dynamic Adaptive Streaming with HTTP (MPEG-DASH)

[0201] Recently, Hypertext Transfer Protocol (HTTP) has been widely used for the delivery of real-time multimedia content over the Internet, such as in video streaming applications. Unlike the use of the Real-time Transport Protocol (RTP) over the User Datagram Protocol (UDP), HTTP is easy to configure and is typically granted traversal of firewalls and network address translators (NAT), which makes it attractive for multimedia streaming applications.

[0202] Chunked HTTP delivery enables servers to respond to an HTTP GET request in multiple parts. However, chunked HTTP delivery does not remove the inherent encoding and encapsulation delay caused by creating self-standing movie fragments. Chunked HTTP delivery is specified in IETF RFC 7230.

[0203] Several commercial solutions for adaptive streaming over HTTP, such as Microsoft® Smooth Streaming, Apple® Adaptive HTTP Live Streaming and Adobe® Dynamic Streaming, have been launched as well as standardization projects have been carried out. Adaptive HTTP streaming (AHS) was first standardized in Release 9 of 3rd Generation Partnership Project (3GPP) packet- switched streaming (PSS) service (3GPP TS 26.234 Release 9: ‘Transparent end-to-end packet- switched streaming service (PSS); protocols and codecs’). MPEG took 3GPP AHS Release 9 as astarting point for the MPEG DASH standard (ISO / IEC 23009-1: ‘Dynamic adaptive streaming over HTTP (DASH)-Part 1: Media presentation description and segment formats,’ International Standard, 2nd Edition, 2014). 3GPP continued to work on adaptive HTTP streaming in communication with MPEG and published 3GP-DASH (Dynamic Adaptive Streaming over HTTP; 3GPP TS 26.247: ‘Transparent end-to-end packet-switched streaming Service (PSS); Progressive download and dynamic adaptive Streaming over HTTP (3GP-D ASH)’. MPEG DASH and 3GP-DASH are technically close to each other and may therefore be collectively referred to as DASH. Streaming systems like MPEG- DASH include for example HTTP Live Streaming (a.k.a. HLS), specified in the IETF RFC 8216. For a detailed description of said adaptive streaming system, all providing examples of a video streaming system, wherein the embodiments may be implemented, a reference is made to the above standard documents. Various embodiments of the invention are not limited to the above standard documents but rather the description is given for an example possible basis on top of which various embodiments of the invention may be partly or fully realized.

[0204] In DASH, the multimedia content may be stored on an HTTP server and may be delivered using HTTP. The content may be stored on the server in two parts: Media Presentation Description (MPD), which describes a manifest of the available content, its various alternatives, their URL addresses, and other characteristics; and segments, which includes the actual multimedia bitstreams in the form of chunks, in a single or multiple files. The MDP provides the necessary information for clients to establish a dynamic adaptive streaming over HTTP. The MPD includes information describing media presentation, such as an HTTP -uniform resource locator (URL) of each Segment to make GET Segment request. To play the content, the DASH client may obtain the MPD e.g. by using HTTP, email, thumb drive, broadcast, or other transport methods. By parsing the MPD, the DASH client may become aware of the program timing, media-content availability, media types, resolutions, minimum and maximum bandwidths, and the existence of various encoded alternatives of multimedia components, accessibility features and required digital rights management (DRM), media-component locations on the network, and other content characteristics. Using this information, the DASH client may select the appropriate encoded alternative and start streaming the content by fetching the segments using e.g. HTTP GET requests. After appropriate buffering to allow for network throughput variations, the client may continue fetching the subsequent segments and monitor the network bandwidth fluctuations. The client may decide how to adapt to the available bandwidth by fetching segments of different alternatives (with lower or higher bitrates) to maintain an adequate buffer.

[0205] In the context of DASH, the following definitions may be used: A media content component or a media component may be defined as one continuous component of the media content with an assigned media component type that can be encoded individually into a media stream. Mediacontent may be defined as one media content period or a contiguous sequence of media content periods. Media content component type may be defined as a single type of media content such as audio, video, or text. A media stream may be defined as an encoded version of a media content component.

[0206] In DASH, a hierarchical data model may be used to structure media presentation as follows. A media presentation includes a sequence of one or more periods, each period includes one or more groups, each group includes one or more adaptation sets, each adaptation set includes one or more representations, each representation includes one or more segments. A Group may be defined as a collection of adaptation sets that are not expected to be presented simultaneously. An adaptation set may be defined as a set of interchangeable encoded versions of one or several media content components. A representation is one of the alternative choices of the media content or a subset thereof typically differing by the encoding choice, e.g. by bitrate, resolution, language, codec, and the like. The Segment includes certain duration of media data, and metadata to decode and present the included media content. A Segment is identified by a URI and may typically be requested by, e.g., a HTTP GET request. A segment may be defined as a unit of data associated with an HTTP-URL and optionally a byte range that are specified by an MPD.

[0207] An initialization segment may be defined as a Segment including metadata that is necessary to present the media streams encapsulated in Media Segments. In ISOBMFF based segment formats, an initialization segment may comprise the Movie Box ('moov') which might not include metadata for any samples, e.g., any metadata for samples is provided in 'moof boxes.

[0208] A media segment may include certain duration of media data for playback at a normal speed, such duration is referred as media segment duration or segment duration. The content producer or service provider may select the segment duration according to the desired characteristics of the service. For example, a relatively short segment duration may be used in a live service to achieve a short end-to-end latency. The reason is that Segment duration is typically a lower bound on the end-to-end latency perceived by a DASH client since a segment is a discrete unit of generating media data for DASH. Content generation is typically done such a manner that a whole Segment of media data is made available for a server. Furthermore, many client implementations use a segment as the unit for GET requests. Thus, in typical arrangements for live services a Segment can be requested by a DASH client only when the whole duration of Media Segment is available as well as encoded and encapsulated into a Segment. For on-demand service, different strategies of selecting segment duration may be used.

[0209] A segment may be further partitioned into subsegments e.g. to enable downloading segments in multiple parts. The subsegments may be required to include complete access units. Thesubsegments may be indexed by a segment index box, which includes information to map presentation time range and byte range for each subsegment. The segment index box may also describe subsegments and stream access points in the segment by signaling their durations and byte offsets. A DASH client may use the information obtained from segment index box(es) to make a HTTP GET request for a specific subsegment using byte range HTTP request. When a relatively long segment duration (e.g., a segment of duration > 7,000 milliseconds) is used, then subsegments may be used to keep the size of HTTP responses reasonable and flexible for bitrate adaptation. The indexing information of a segment may be put in the single box at the beginning of that segment or spread among many indexing boxes in the segment. Different methods of spreading are possible, such as hierarchical, daisy chain, and hybrid. This technique may avoid adding a large box at the beginning of the segment and therefore may prevent a possible initial download delay.

[0210] DASH supports rate adaptation by dynamically requesting media segments from different representations within an adaptation set to match varying network bandwidth. When a DASH client switches up / down representation, coding dependencies within the representation have to be taken into account. A representation switch may only happen at a random access point (RAP), which is typically used in video coding techniques such as H.264 / AVC. In DASH, a more general concept named stream access point (SAP) is introduced to provide a codec-independent solution for accessing a representation and switching between representations. In DASH, a SAP is specified as a position in a representation that enables playback of a media stream to be started using only the information included in representation data starting from that position onwards (preceded by initializing data in the initialization Segment, if any). Hence, representation switching may be performed in SAP.

[0211] DASH provides a preselection concept which allows to group a subset of media component in a bundle that are expected to be consumed jointly. A bundle is a set of media components which may be consumed jointly by a single decoder instance. Elements are addressable and separable components of a bundle and may be selected or deselected dynamically by the application, either directly or indirectly by using preselections. Media components are mapped to adaptation sets by either a one-to-one mapping or by the inclusion of multiple media components in a single adaptation set. Furthermore, representations in one adaptation set may include multiple media components that are multiplexed on elementary stream level or on file container level. In the multiplexing case each media component is mapped to a Media Content component. Each media component in the bundle is therefore identified and referenced by the ‘@id’ of a Media Content component, or, if only a single media component is included in the adaptation set, by the ‘@id’ of an adaptation set.

[0212] Each bundle includes a main media component that includes the decoder specific information and bootstraps the decoder. The adaptation set that includes the main media component is referred to as main adaptation set. The main media component shall always be included in any Preselection that is associated to a bundle. In addition, each bundle may include one or multiple partial adaptation sets. The partial adaptation sets may only be processed in combination with the main adaptation set.

[0213] A Preselection defines a subset of media component in a bundle that are expected to be consumed jointly. A preselection is identified by a unique tag towards the decoder. Multiple preselection instances can refer to the same set of streams in a bundle. Only media components of the same bundle can contribute to the decoding and rendering of a preselection.

[0214] An end-to-end system for DASH may be described as follows. The media content is provided by an origin server, which may be a conventional web (HTTP) server. The origin server may be connected with a content delivery network (CDN) over which the streamed content is delivered to and stored in edge servers. The MPD allows signaling of multiple base URLs for the content, which may be used to announce the availability of the content in different edge servers. Alternatively, the content server may be directly connected to the Internet. Web proxies may reside on the path of routing the HTTP traffic between the DASH clients and the origin or edge server from which the content is requested. The web proxies cache HTTP messages and hence can serve clients' requests with the cached content. They are commonly used by network service providers, since they reduce the required network bandwidth from the proxy towards origin or edge servers. For end-user’s HTTP caching provides shorter latency. DASH clients are connected to the Internet through an access network, such as a mobile cellular network. The mobile network may comprise mobile edge servers or mobile edge cloud, operating similarly to a CDN edge server and / or web proxy.

[0215] Fundamentals of neural networks

[0216] A neural network (NN) is a computation graph consisting of several layers of computation. Each layer consists of one or more units, where each unit performs a computation. A unit is connected to one or more other units, and a connection may be associated with a weight. The weight may be used for scaling the signal passing through an associated connection. Weights are learnable parameters, for example, values which can be learned from training data. There may be other learnable parameters, such as those of batch-normalization layers.

[0217] Couple of examples of architectures for neural networks are feed-forward and recurrent architectures. Feed-forward neural networks are such that there is no feedback loop, each layer takes input from one or more of the previous layers, and provides its output as the input for one or more of the subsequent layers. Also, units inside a certain layer take input from units in one or more of preceding layers and provide output to one or more of following layers.

[0218] Initial layers, those close to the input data, extract semantically low-level features, for example, edges and textures in images, and intermediate and final layers extract more high-level features. After the feature extraction layers there may be one or more layers performing a certain task, for example, classification, semantic segmentation, object detection, denoising, style transfer, superresolution, and the like. In recurrent neural networks, there is a feedback loop, so that the neural network becomes stateful, for example, it is able to memorize information or a state.

[0219] Neural networks are being utilized in an ever-increasing number of applications for many different types of devices, for example, mobile phones, chat bots, loT devices, smart cars, voice assistants, and the like. Some of these applications include, but are not limited to, image and video analysis and processing, social media data analysis, device usage data analysis, and the like.

[0220] One of the properties of neural networks, and other machine learning tools, is that they are able to learn properties from input data, either in a supervised way or in an unsupervised way. Such learning is a result of a training algorithm, or of a meta-level neural network providing the training signal.

[0221] In general, the training algorithm consists of changing some properties of the neural network so that its output is as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the neural network can be used to derive a class or category index which indicates the class or category that the object in the input image belongs to. Training usually happens by minimizing or decreasing the output error, also referred to as the loss. Examples of losses are mean squared error, cross-entropy, and the like. In recent deep learning techniques, training is an iterative process, where at each iteration the algorithm modifies the weights of the neural network to make a gradual improvement in the network’s output, for example, gradually decrease the loss.

[0222] Training a neural network is an optimization process, but the final goal is different from the typical goal of optimization. In optimization, the only goal is to minimize a function. In machine learning, the goal of the optimization or training process is to make the model learn the properties ofthe data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset in order to learn to generalize to previously unseen data, for example, data which was not used for training the model. This is usually referred to as generalization. In practice, data is usually split into at least two sets, the training set and the validation set. The training set is used for training the network, for example, to modify its learnable parameters in order to minimize the loss. The validation set is used for checking the performance of the network on data, which was not used to minimize the loss, as an indication of the final performance of the model. In particular, the errors on the training set and on the validation set are monitored during the training process to understand the following:- when the network is learning at all - in this case, the training set error should decrease, otherwise the model is in the regime of underfitting.- when the network is learning to generalize - in this case, also the validation set error needs to decrease and be not too much higher than the training set error. For example, the validation set error should be less than 20% higher than the training set error. If the training set error is low, for example 10% of its value at the beginning of training, or with respect to a threshold that may have been determined based on an evaluation metric, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, the model is in the regime of overfitting. This means that the model has just memorized properties of the training set and performs well only on that set, but performs poorly on a set not used for training or tuning of its parameters.

[0223] Lately, neural networks have been used for compressing and de-compressing data such as images. The most widely used architecture for such task is the auto-encoder, which is a neural network consisting of two parts: a neural encoder and a neural decoder. In various embodiments, these neural encoder and neural decoder would be referred to as encoder and decoder, even though these refer to algorithms which are learned from data instead of being tuned manually. The encoder takes an image as an input and produces a code, to represent the input image, which requires less bits than the input image. This code may have been obtained by a binarization or quantization process after the encoder. The decoder takes in this code and reconstructs the image which was input to the encoder.

[0224] Such encoder and decoder are usually trained to minimize a combination of bitrate and distortion, where the distortion may be based on one or more of the following metrics: mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), or the like. These distortion metrics are meant to be correlated to the human visual perception quality, so that minimizing or maximizing one or more of these distortion metrics results into improving the visual quality of the decoded image as perceived by humans.

[0225] In various embodiments, terms ‘model’, ‘neural network’, ‘neural net’ and ‘network’ may be used interchangeably, and also the weights of neural networks may be sometimes referred to as learnable parameters or as parameters.

[0226] Neural Network Representation (NNR)

[0227] ISO / IEC 15938-17 (Compression of Neural Networks for Multimedia Content Description and Analysis) is also known as neural network representation (NNR) or neural network compression (NNC). NNR specifies a compressed representation of the parameters and / or weights of a trained neural network and a decoding process for the compressed representation. NNR complements the description of the network topology in existing neural network exchange formats. NNR is independent of a particular neural network exchange format and is interoperable with common neural network exchange formats.

[0228] NNR establishes a toolbox of compression methods, specifying (where applicable) the resulting elements of the compressed bitstream. All of these tools may be applied to the compression of entire neural networks, and some of them may also be applied to the compression of differential updates of neural networks with respect to a base network. Such differential updates are, for example, useful when models are redistributed after fine-tuning or transfer learning, or when providing versions of a neural network with different compression ratios. The support for incremental compression of updates of neural networks respective to a base model will be included in the 2ndedition of NNR, which is currently being standardized.

[0229] NNR comprises the syntax format, semantics, associated decoding process requirements, parameter sparsification, parameter transformation methods, parameter quantization, entropy coding method and integration / signaling within existing exchange formats.

[0230] An NNR bitstream may conform to ISO / IEC 15938-17. NNR bitstream or NNR data in a channel may comprise a sequence of NNR Units. An NNR Unit may be regarded as a basic high-level syntax structure in an NNR bitstream, and may include three syntax elements or structures: NNR Unit Size, NNR unit header, and NNR unit payload.

[0231] Neural network based filtering

[0232] In some video codecs, a neural network may be used as filter in the decoding loop, and itmay be referred to as neural network loop filter, or neural network in-loop filter. The NN loop filter may replace all other loop filters of an existing video codec, or may represent an additional loop filter with respect to the already present loop filters in an existing video codec.In the context of image and video enhancement, a neural network may be used as post-processing filter, for example applied to the output of an image or video decoder in order to remove or reduce coding artifacts.

[0233] Some of the embodiments related to the architecture of neural networks used as part of the decoding operations (such as a NN loop filter, or an intra-frame prediction NN, or an inter-frame prediction NN) or as part of post-processing operations (a NN post-processing filter). Also, it concerns the signaling of information related to those NNs, where the information is signaled from an encoder to a decoder.

[0234] The following example system will be used in several embodiments to illustrate or describe the idea. The example system comprises a codec that comprises one or more NN loop filters. For example, the codec could comprise a modified VVC / H.266 compliant codec (e.g., a VVC / H.266 compliant codec that has been modified so that it would comprise one or more NN loop filters). The input to the one or more NN loop filters may comprise at least a reconstructed block or frames (simply referred to as reconstruction) or data derived from a reconstructed block or frame (e.g., the output of a conventional loop filter). The reconstruction may be obtained based on predicting a block or frame (e.g., by means of intra-frame prediction or inter-frame prediction) and performing residual compensation. The one or more NN loop filters (may be referred to simply as NN filters in some of the embodiments) may enhance the quality of at least one of their input, so that a rate-distortion loss is decreased. The rate may indicate a bitrate (estimate or real) of the encoded video. The distortion may indicate a pixel fidelity distortion such as the following:- Mean- squared error (MSE)- Mean absolute error (MAE)- Mean Average Precision (mAP) computed based on the output of a task NN (such as an object detection NN) when the input is the output of the post-processing NN.- Other machine task-related metric, for tasks such as object tracking, video activity classification, video anomaly detection, etc.

[0235] The enhancement may result into a coding gain, which can be expressed for example in terms of BD-rate, BD-PSNR, or BD-mAP.

[0236] However at least some of the embodiments described herein are applicable to a NN filterwhich is not a loop filter of a codec. For example, the NN filter may be a NN post-processing filter, whose input may comprise one or more outputs of a video codec. In this case, the filter may be used only for increasing a quality metric of at least one of its inputs, where the quality metric may be, for example, peak signal-to-noise ratio (PSNR), mAP for object detection, MOTA for object tracking, and the like.

[0237] Storage of neural network as an item in ISO base media file

[0238] Following paragraphs describe the storage of NNR coded data as NAL units in a track that is linked to a media track.

[0239] An example method for storing NNR coded data includes defining a metadata box for a neural network representation (NNR) item data, wherein the NNR item data comprises an NNR bitstream; and defining an association between the NNR item data and an NNR configuration by using a configuration item property, wherein the NNR configuration item property comprises information about stored NNR item data.

[0240] A new item type called ‘NNR item’ may be defined. NNR item may be referred to as NNR item data in some embodiments. The metadata for NNR items may be included in a meta box like the metadata for any other items. NNR items may be stored in an ISOBMFF file or in an external file similarly to any other items as described earlier. For example, NNR items may be stored in the ItemDataBox of the meta box of an ISOBMFF file. Such storage could be at file level, movie box level, or track level. In track level storage, a media track may be dependent on the non-timed neural network to process its samples. The process may be associated to visual enhancement of decoded media data, decoding of the media data in the sample, or alike. In another example, NNR items may be stored in one or more media data boxes (e.g. MediaDataBox) of an ISOBMFF file, while the metadata for items may be included in a meta box, which may be stored at file level, movie box level, or track level.

[0241] Storage of NNR data as NAL units in a separate track

[0242] Following paragraphs describe the storage of NNR coded data as NAL units in a track that is linked to a media track.

[0243] In an example, NN tracks could have time-aligned samples with the samples of associated media tracks.Global NN weight updates could be stored e.g. in a new box in the track-level, a sample entry, sample group entries, or samples. Storing them in sample entry or sample group entry may be beneficial over the other alternatives e.g. for unicast streaming services.- NN weight updates may be aligned with the Group of Picture structures of media data and media track. In such a case the NN data could be stored in the samples of NN track.

[0244] In another example, NNR coded data may be stored in conjunction with media data as follows:- There could be NAL units which are NNR compressed and defined in audio / video codec scope. Such NAL units could be collected in NN tracks as samples of this track then linked to the video track- The same NN track may be linked or referenced by multiple audio / video tracks. This could be especially useful when there are multiple representations of the same media track which can utilize the same NN in its samples.

[0245] In yet another example, such media samples may include further NN updates in their bitstreams as specially marked data structures (e.g. NN specific NAL units).

[0246] In still another example, there could be NAL units which are NNR compressed and defined in an audio / video codec context. Such NAL units may be aggregated in NNR tracks as samples of NNR track and then linked to the media tracs that they apply. Time alignment of such media tracks may be done via the sample composition timing mechanisms.

[0247] In an example, a method includes defining an NNR track sample including one or more NNR units. The NNR track sample is linked to an NNR unit parameter via one or more of a sample entry, a sample group, or a non-timed NNR item data. In another example, the method includes defining a network abstraction layer (NAL) unit, which is aggregated in NNR tracks as a sample of NNR track and linked to the associated media track. In an example, the NNR sample track is may be independently decodable or dependent of previous sample for decoding. The NNR sample track may be used by other media tracks for decoding media samples of the other media tracks. A track referencing mechanism may be used to associate the NNR sample track with the other media tracks for decoding the media samples of the other media tracks.

[0248] The method may further include comprising defining a network abstraction layer (NAL) unit, wherein the NAL unit is aggregated in NNR tracks as a sample of NNR track and linked to the associated media track.

[0249] In an example an apparatus may be caused to define a network abstraction layer (NAL) unit, wherein the NAL unit is aggregated in NNR tracks as a sample of NNR track and linked to the associated media track.

[0250] Usage of SEI prefix indication SEI message with post-filter SEIs for media presentation characteristics signalling

[0251] In an example, an entity makes one or more SEI prefix indications available along a video bitstream, e.g., in a media description, such as SDP or DASH MPD. An SEI prefix indication may comprise a SEI prefix indication SEI message or may comprise an initial part or an entire syntax structure of one or more SEI messages, such as a post-filter related SEI messages. An SEI prefix indication may be, but is not limited to, one or more of the following:One or more MIME media parameters. The MIME media parameter(s) may be encapsulated in an SDP parameter or in an attribute of a streaming manifest (e.g. DASH MPD) or alike. Separate or same MIME media parameter(s) may be used for declarative bitstream properties, encoding capabilities, and / or preferences or requirements for bitstream to be decoded;One or more attributes, parameters, or alike of the media description. An attribute may for example be an attribute in DASH MPD; orOne or more syntax elements or syntax structures in a file encapsulating a bitstream.

[0252] Support for carriage of NN SEI message in ISOBMFF

[0253] In an example, a file writing method is provided. The file writing method includes: storing coded video data in a video track of a file; storing neural-network post-filter (NNPF) supplemental enhancement information (SEI) in the video track or in a second track associated with the video track; indicating, in the file, the presence of the NNPF SEI in the video track or the second track with at least one of a scheme type of a restricted video sample entry type and an essential sample group for the NNPF SEI. NNPF SEI may be defined as a collective term for NNPFC SEI message(s) and / or NNPFA SEI message(s) and / or SEI NAL unit(s) including NNPFC SEI message(s) and / or NNPFA SEI message(s).

[0254] In another example, a file parsing method is provided. The file parsing method includes: reading, from a file, the presence of the NNPF SEI in a video track or a second track from at least one of a scheme type of a restricted video sample entry type and an essential sample group for the NNPF SEI; in response to supporting NNPF SEI, processing the NNPF SEI in the video track or the secondtrack; decoding coded video data from the video track; filtering the decoded video data with a postfilter defined by the NNPF SEI.

[0255] Both of the following approaches are possible for carriage of NNPFC and NNPFA SEI messages:- NNPFC and NNPFA SEI messages are included in a track that also includes coded video data. There might be multiple tracks with the same coded video data. One track might not include NNPFC and NNPFA SEI messages and is intended for players and decoders that do not support NNPFC and NNPFA SEI messages. Other tracks may include or reference different post-filters.- NNPFC and NNPFA SEI messages are included in a track that is separate from the coded video data. For example, NNPFC and NNPFA SEI messages are included in a VVC non-VCE track. Players and decoders need to access the track with the coded video data and additionally players and decoders may access the track with the NNPFC and NNPFA SEI messages.

[0256] Indicating parameters of the neural network

[0257] It may be beneficial for a player or reader or decoder to get to know parameters of the neural network, such as complexity parameters, to determine when is possible to use it. Examples for indicating parameters of the neural network are described below.

[0258] In an example, a neural-network post-filter information box (e.g., with 4CC equal to 'nnfi') is defined to carry syntax elements of an NNPFC SEI message, but without NNR bitstream, or any information similar to what is carried in an NNPFC SEI message. The container for the neural- network post-filter information box may be the SchemelnformationBox. The syntax may be specified as follows:

[0259] aligned(8) class NNPFInformationBox extends FullBox('nnfi',0,0) { unsigned int(8) nnpfc_sei_data_byte[];}

[0260] The byte array nnpfc_sei_data_byte[ ] includes a bit string that starts an NNPFC SEI message.

[0261] In an example, a SEI prefix indication box (e.g. with 4CC equal to 'seip') is defined to include a SEI prefix indication SEI message. The container for the SEI prefix indication box may be the SchemelnformationBox. The syntax may be specified as follows: aligned(8) class SEIPrefixIndicationBox extends FullBox('seip',0,0) { unsigned int(8) sei_prefix_sei_data_byte[];}

[0262] The byte array sei_prefix_sei_data_byte[ ] includes a SEI prefix indication SEI message.

[0263] It may be advantageous for a player or reader or decoder to get to know the order of SEI-defined post-processing steps to determine when it is possible to perform the intended postprocessing. In an embodiment, a SEI processing order box (e.g., with 4CC equal to 'seio') is defined to include a SEI processing order SEI message. The container for the SEI prefix indication box may be the SchemelnformationBox. The syntax may be specified as follows: aligned(8) class SEIProcessingOrderBox extends FullBox('seio',0,0) { unsigned int(8) sei_processing_order_sei_data_byte[];}

[0264] The byte array sei_processing_order_data_byte[ ] includes a SEI processing order SEI message.

[0265] Item property for associating an nnpfc_id value to a neural network item in ISOBMFFs

[0266] A neural network may be stored as an item. The neural network data may, for example, comply with NNR. The item data may reside in a standalone file, which may be identified with a URL. The URL may be provided in a DataEntryUrlBox as an entry in the DataReferenceBox, and the data reference entry may then be associated with an item in an ItemLocationBox. Alternatively, the item data may reside in the ISO base media file, for example in ItemDataBox.

[0267] When the neural network data resides in a standalone file which is identified by a URL, a client needs to fetch the resource pointed to be the URL only once, as opposed to fetching it again and again, e.g., in the case of storing the neural network data in the video bitstream.

[0268] In an example, a specific handler, such as 'nrnw', that is associated with an item including a neural network (for example, through HandlerBox or HandlerProperty) may indicate that the item comprises a neural network. The item type may indicate the format of the item. For example, the item type 'nnrl' or 'nncl' may indicate that the item data conforms to ISO / IEC 15938-7.

[0269] In an example, a new item property is defined to provide an identifier value, such as an nnpfc_id value, which associates the neural network stored in the item data to an identifier that may be used to associate NN characteristics provided in a video bitstream (e.g. in a NNPFC SEI message) and / or activate a NN for one or more pictures in a video bitstream (e.g. through a NNPFA SEI message). This item property may be present for a neural network item that is present in a track that includes NNPFC and / or NNPFA SEI messages. The item property may have the following syntax: aligned(8) class NNPostFilterProperty(property_type, version, flags) extends ItemFullProperty('nnfp', 0, 0){ unsigned int(32) nnpf_id;}

[0270] nnpf_id indicates that the track includes an NNPFC SEI message with nnpfc_mode_idc equal to 0 (the base post-processing filter associated with the nnpfc_id value is determined by external means) and nnpfc_id is equal to nnpf_id. The neural network included in the item data serves as the base post-processing filter determined by external means.

[0271] In an example, NNPostFilterProperty may comprise syntax elements of NNPFC SEI message or any information similar to what is included theNNPFC SEI message.

[0272] Carriage of NNPFC and NNPFA SEI in ISOBMFF

[0273] The document, WD of 14496-15 6th edition AMD 3 Support for neural-network postfilter supplemental enhancement information and other improvements (MDS22617_WG03_N00875) specifies the storage and carriage of NNPF SEI messages as follows.

[0274] Neural-network post-filter characteristics sample group

[0275] Definition

[0276] The neural-network post-filter characteristics (NNPFC) SEI message is specified in ISO / IEC 23002-7. NNPFC SEI messages may be included in a VVC bitstream.

[0277] An NNPFC SEI message includes the nnpfc_id syntax element, which is an identifying number that may be used to identify the post-processing filter that the NNPFC SEI message concerns.

[0278] An NNPFC SEI message identifies an applicable post-processing filter associated with the nnpfc_id value. The use of applicable post-processing filters with different values of nnpfc_id for specific pictures is indicated with neural-network post-filter activation (NNPFA) SEI messages.

[0279] An NNPFC SEI message either specifies a base post-processing filter or includes a neural network update. A base post-processing filter is identified by the first NNPFC SEI message, in decoding order, that has a particular nnpfc_id value within a CLVS. If there is no subsequent NNPFC SEI message that has the same nnpfc_id value as the base post-processing filter, the applicable postprocessing filter is the same as the base post-processing filter. Otherwise, the applicable postprocessing filter is obtained by applying the update provided as an ISO / IEC 15938-17 bitstream in a subsequent NNPFC SEI message on top of the base post-processing filter.

[0280] All instances of the SampleToGroupBox for the NNPFC sample group shall include grouping_type_parameter. The grouping_type_parameter field is specified for the NNPFC sample group as follows:{ unsigned int(l) filter_update_flag; unsigned int(31) filter_id;}

[0281] filter_update_flag equal to 1 indicates that all the sample group description entries referenced by this SampleToGroupBox includes an NNPFC SEI message that provides an update on top of a base post-processing filter. filter_update_flag equal to 0 indicates that all the sample group description entries referenced by this SampleToGroupBox includes an NNPFC SEI message that specifies a base post-processing filter.

[0282] filter_id indicates that all the sample group description entries referenced by this SampleToGroupBox includes an NNPFC SEI message that has nnpfc_id equal to filter_id.

[0283] As a consequence of the grouping_type_parameter definition, the post-processing filters for different nnpfc_id values are specified in different instances of the SampleToGroupBox. Furthermore, one SampleToGroupBox specifies the base post -processing filter(s) for a particular nnpfc_id value, while another SampleToGroupBox, if any, specifies the filter updates for the same nnpfc_id value. It is therefore possible to indicate that the base post-processing filter persists over a longer period than any of the filter updates.

[0284] When a sample is not mapped to NnpfcSeiEntry in a SampleToGroupBox including filter_update_flag equal to 0 and a particular filter_id, the sample shall not be mapped to an NnpfcSeiEntry in a SampleToGroupBox having filter_update_flag equal to 1 and the same filter_id.

[0285] When a track includes an NNPFC sample group, no NNPFC SEI messages shall be present within the samples of the track. When a VVC track has an associated VVC non-VCL track that includes an NNPFC sample group, no NNPFC SEI messages shall be present within the samples of the VVC track.

[0286] When a reader supports the NNPFC sample group, it shall perform the following implicit insertion of prefix SEI NAL units as a part of the bitstream reconstruction:— When a sample is mapped to at least one NnpfcSeiEntry with filter_update_flag equal to 0 and the sample is— a sync sample, or— the first sample of a sequence of samples associated with the same sample entry, or— the first sample of a sequence of samples mapped to the same NnpfcSeiEntry with filter_update_flag equal to 0 and a particular filter_id value filterldBase, then the sample implicitly includes a prefix SEI NAL unit for each layer included in the track and each filter_id value mapped to the sample, and the prefix SEI NAL unit includes the NNPFC SEI message from the NnpfcSeiEntry with filter_update_flag equal to 0, followed by the NNPFC SEI message from the NnpfcSeiEntry with filter_update_flag equal to 1 and filter_id equal to filterldBase that is mapped to the sample, when any.— When a sample is the first sample in a sequence of samples mapped to the same NnpfcSeiEntry with filter_update_flag equal to 1 and a particular filter_id value filterldUpdate and the sample is: not a sync sample; not the first sample of a sequence of samples associated with the same sample entry; andnot the first sample in a sequence of samples mapped to the same NnpfcSeiEntry with filter_update_flag equal to 0 and filter_id equal to filterldUpdate,— then the sample implicitly includes a prefix SEI NAL unit for each layer included in the track and each filter_id value mapped to the sample, and the prefix SEI NAL unit includes the NNPFC SEI message from the NnpfcSeiEntry with filter_update_flag equal to 1.

[0287] Syntax aligned(8) class NnpfcSeiEntryQ extends VisualSampleGroupEntry('nfcs') { unsigned int(8) nnpfc_sei_data_byte[]; }

[0288] Semantics

[0289] nnpfc_sei_data_byte[] is a byte array that shall include exactly one complete NNPFC SEI message as specified in ISO / IEC 23002-7.

[0290] Neural-network post-filter activation sample group

[0291] Definition

[0292] The neural-network post-filter activation (NNPFA) SEI message is specified in ISO / IEC 23002-7. NNPFA SEI messages may be included in a VVC bitstream.

[0293] An NNPFA SEI message includes the nnpfa_target_id syntax element, which is an identifying number that may be used to identify the post-processing filter that the NNPFA SEI message concerns.

[0294] An NNPFA SEI message indicates that the applicable post-processing filter with nnpfc_id equal to nnpfa_target_id may be used to filter the picture including the NNPFA SEI message.

[0295] Instances of the SampleToGroupBox for the NNPFA sample group shall not include grouping_type_parameter.

[0296] When a track includes an NNPFA sample group, no NNPFA SEI messages shall be present within the samples of the track.

[0297] When a reader supports the NNPFA sample group, it shall perform the following implicit insertion of prefix SEI NAL units as a part of the bitstream reconstruction:- When a sample is mapped to at least one NnpfaSeiEntry, the sample implicitly includes a prefix SEI NAL unit for each layer included in the track, and the prefix SEI NAL unit includes the NNPFA SEI message from the NnpfaSeiEntry.- When a reader processes an NNPFA sample group, it shall also process the NNPFC sample groups of the same track. When a VVC track has an associated VVC non- VCL track that includes an NNPFA sample group, no NNPFA SEI messages shall be present within the samples of the VVC track.- When an NNPFC sample group is an essential sample group and an NNPFA sample group is present in the same track, the NNPFA sample group shall be an essential sample group and the 'esgh' sample group shall list 'nfcs' and 'nfas' in subsequent entries of the sample_group_description_type array.

[0298] Syntax aligned(8) class NnpfaSeiEntryQ extends VisualSampleGroupEntry('nfas'){ do { unsigned int(8) nnpfa_sei_len; if (nnpfa_sei_len > 0) unsigned int(8) nnpfa_sei_data_byte[nnpfa_sei_len];} while (nnpfa_sei_len > 0)}

[0299] Semantics

[0300] nnpfa_sei_len greater than 0 is the number of bytes in the following byte array nnpfa_sei_data_byte[nnpfa_sei_len]. At least the first instance of nnpfa_sei_len shall be greater than 0. nnpfa_sei_len equal to 0 specifies that no further byte arrays follow in this NnpfaSeiEntry.

[0301] nnpfa_sei_data_byte[nnpfa_sei_len] is a byte array that shall include exactly one complete NNPFA SEI message as specified in ISO / IEC 23002-7.

[0302] ISOBMFF

[0303] ISOBMFF defines sample-to-item sample group, where samples of a track can be linked to one more metadata item using the sample-to-item sample grouping. The MetaBox including the referred items is resolved as specified in the semantics below.

[0304] The sample-to-item sample grouping is allowed for any types of tracks, and its syntax and semantics are unchanged regardless of the track handler type.

[0305] In the absence of this sample group, the entire track-level MetaBox, if any, is applicable to every sample.

[0306] The syntax of sample-to-item sample group is given below. class SampleToMetadataltemEntryO extends SampleGroupDescriptionEntry('stmi') { unsigned int(32) meta_box_handler_type; unsigned int(32) num_items; for(i = 0; i < num_items; i++) { unsigned int(32) item_id[i];}}

[0307] where,

[0308] meta_box_handler_type informs about the type of metadata schema used by the MetaBox which is referenced by the items in this sample group. When there are multiple MetaBoxes with the same handler types, the MetaBox referred to in this sample group entry is the first MetaBox fulfilling one of the following ordered constraints:- A MetaBox included in the current track, with handler lype equal to meta_box_handler_type.- A MetaBox included in MovieBox, with handler_type equal to meta_box_handler_type.- A MetaBox included in the root level of the file, with handler lype equal to meta_box_handler_type.- num_items counts the number of items referenced by this sample group.- item_id[i] specifies the item_ID value of an item that applies to or is valid for the sample mapped to this sample group description entry.

[0309] Identified media data box

[0310] The identified media data box or IdentifiedMediaDataBox is described in patent application US16 / 077576, published as US20190052937, and is herein incorporated by reference.

[0311] An example identified media data box mays have the same semantics as a MediaDataBox has but it additionally includes an identifier that is used in setting up data references to the included media data. The identifier may for example be the first element included by the identified media data box. The syntax of an identified media data box may be specified as follows, where imda_identifier is the identifier of the box. It is noted that while imda_identifier of type 32-bit unsigned integer is used in the syntax, other field lengths and other basic data types (e.g. string) are similarly possible. aligned(8) class IdentifiedMediaDataBox extends Box('imda') { unsigned int(32) imda_identifier; bit(8) data[]; / / until the end of the box}

[0312] imda_identifier shall differ from the imda_identifier values of the other IdentifiedMediaDataBoxes of the file.

[0313] Track data layout structures

[0314] The ISOBMFF defines the DatalnformationBox with the four-character code ‘dinf which includes objects that declare the location of the media information in a track.

[0315] The data reference object includes a table of data references (normally URLs) that declare the location(s) of the media data used within the presentation. The data reference index in the sample description ties entries in this table to the samples in the track. A track may be split over several sources in this way.

[0316] When the flag is set indicating that the data is in the same file as this box, then no string (not even an empty string) shall be supplied in the entry field.

[0317] The entry _count in the DataReferenceBox shall be 1 or greater.

[0318] When a file that has data entries with the flag set indicating that the media data is in the same file, is split into segments for transport, the value of this flag does not change, as the file is (logically) reassembled after the transport operation.

[0319] The DataEntrylmdaBox identifies the IdentifiedMediaDataBox including the media data accessed through the data_reference_index corresponding to this DataEntrylmdaBox. The DataEntrylmdaBox includes the value of imda_identifier of the referred IdentifiedMediaDataBox. The media data offsets are relative to the first byte of the payload of the referred IdentifiedMediaDataBox. In other words, media data offset 0 points to the first byte of the payload of the referred IdentifiedMediaDataBox.

[0320] The DataEntrySeqNumlmdaBox identifies the IdentifiedMediaDataBox including the media data accessed through the data_reference_index corresponding to this DataEntrySeqNumlmdaBox. When a data_reference_index included in a sample entry refers to DataEntrySeqNumlmdaBox, each sample referring to the sample entry shall be included in a movie fragment, and media data offset 0 points to the first byte of the payload of the IdentifiedMediaDataBox that has imda_identifier equal to sequence_number of the MovieFragmentHeaderBox of the movie fragment including the sample. aligned(8) class DataEntryBaseBox(entry_type, bit(24) flags) extends FullBox(entry_type, version = 0, flags) {} aligned(8) class DataEntryUrlBox (bit(24) flags) extends DataEntryBaseBoxfurl ', flags) { utf8string location;} aligned(8) class DataEntryUrnBox (bit(24) flags) extends DataEntryBaseBoxfurn ', flags) { utf8string name; utf8string location;}aligned(8) class DataEntrylmdaBox (bit(24) flags) extends DataEntryBaseBoxfimdt', flags) { unsigned int(32) imda_ref_identifier;} aligned(8) class DataEntrySeqNumlmdaBox (bit(24) flags) extends DataEntryBaseBox ('snim', flags) {} aligned(8) class DataReferenceBox extends FullBoxfdref , version = 0, 0) { unsigned int(32) entry _count; for (i= 1; i <= entry _count; i++) {DataEntryBaseBox(entry_type, entry_flags) data_entry;}}

[0321] version is an integer that specifies the version of this box.

[0322] entry _count is an integer that counts the actual entries.

[0323] entry _flags is a 24-bit integer with flags; one flag is defined (xOOOOOl) which means that the media data is in the same file as the Box including this data reference. If this flag is set, the DataEntryUrlBox shall be used and no string is present; the box terminates with the entry -flags field.

[0324] data_entry is an instance of a class derived from DataEntryBaseBox.

[0325] name is a URN, and is required in a URN entry

[0326] location is a URL, and is required in a URL entry and optional in a URN entry, where it gives a location to find the resource with the given name. The URL type should be of a service that delivers a file (e.g., URLs of type file, http, ftp, and the like.), and which services ideally also permit random access. Relative URLs are permissible and are relative to the file that includes this data reference.

[0327] imda_ref_identifier identifies the IdentifiedMediaDataBox including the media data accessed through the data_reference_index corresponding to this DataEntrylmdaBox. The referred IdentifiedMediaDataBox includes imda_identifier that is equal to imda_ref_identifier.

[0328] Example features of identified media data boxes are described in the following paragraphs have been introduced patent application US16 / 972318, published as US20210250617, and is herein incorporated by reference.

[0329] According to an example, instead of estimating the size of the segment header based on the segment duration, identified media data boxes are used. An identifier value for the identified media data box of the segment is determined and that identifier value is provided as the data reference basis for the media data of the segment.

[0330] According to an example, which may be applied independently of or together with other features, a template scheme for the identifier for the identified media data box is defined to be used as a data reference for sample data, e.g., in DataReferenceBox. The template scheme may be based on, but is not limited to, the movie fragment sequence number (such as the sequence_number field of the MovieFragmentHeaderBox) or track fragment decode time (such as the baseMediaDecodeTime field of TrackFragmentBaseMediaDecodeTimeBox). It needs to be understood that any identifier provided for a movie fragment or a track fragment may be appropriate for the template scheme in addition to or instead of those described above. In an example, the following syntax may be used for referencing an identified media data box using a template for deriving the identifier: aligned(8) class DataEntryTfdtBasedlmdaBox (bit(24) flags) extends FullBox('imdt', version = 0, flags) { }

[0331] The DataEntryTfdtBasedlmdaBox identifies the IdentifiedMediaDataBox including the media data accessed through the data_reference_index corresponding to this DataEntryTfdtBasedlmdaBox. Media data offset 0 points to the first byte of the payload of the IdentifiedMediaDataBox that has imda_identifier equal to baseMediaDecodeTime of the TrackFragmentBaseMediaDecodeTimeBox. 64-bit imda_identifier values are used in order to carry the 64-bit value of baseMediaDecodeTime. If 32-bit baseMediaDecodeTime values are in use, the most- significant bits of the 64-bit imda_identifier may be set to 0. For self-contained movie fragments, the imda_identifier of the IdentifiedMediaDataBox is required to be equal to the baseMediaDecodeTime of TrackFragmentBaseMediaDecodeTimeBox, when the referenced data reference entry is of type DataEntryTfdtBasedlmdaBox.

[0332] Thus, the size of the MovieFragmentBox need not be known at the time of determining the base data offset(s) of the track(s) of the movie fragment, and consequently the child boxes of the MovieFragmentBox (e.g., TrackFragmentHeaderBox and TrackRunBoxes) can be authored "progressively" before all coded media data for the movie fragment is available. Moreover, the content encapsulator does not need to estimate the size of the segment header correctly and has the flexibility of some dynamic variability of segment durations.

[0333] Chunk offset box

[0334] The chunk offset table gives the index of each chunk into the containing file. There are two variants, permitting the use of 32-bit or 64-bit offsets. The latter is useful when managing very large presentations. At most one of these variants will occur in any single instance of a sample table.

[0335] When the referenced data reference entry is not DataEntrylmdaBox or DataEntrySeqNumlmdaBox, offsets are file offsets, not the offset into any box within the file (e.g., MediaDataBox). This permit referring to media data in files without any box structure. It does also mean that care must be taken when constructing a self-contained ISO file with its structure-data (MovieBox) at the front, as the size of the MovieBox will affect the chunk offsets to the media data.

[0336] When the referenced data reference entry is DataEntrylmdaBox or DataEntrySeqNumlmdaBox, offsets are relative to the first byte of the payload of the IdentifiedMediaDataBox corresponding to the data reference entry. This permits reordering file-level boxes and receiving a subset of file-level boxes but could require traversing the file-level boxes until the referenced IdentifiedMediaDataBox is found.Syntax aligned(8) class ChunkOffsetBox extends FullBoxfstco', version = 0, 0) { unsigned int(32) entry _count; for (i= 1; i <= entry _count; i++) { unsigned int(32) chunk_offset;}} aligned(8) class ChunkLargeOffsetBox extends FullBox('co64', version = 0, 0) { unsigned int(32) entry _count; for (i= 1; i <= entry _count; i++) {unsigned int(64) chunk_offset;} }

[0337] Semantics

[0338] version is an integer that specifies the version of this box.

[0339] entry _count is an integer that gives the number of entries in the following table.

[0340] chunk_offset is a 32 or 64 bit integer that gives the offset of the start of a chunk. If the referenced data reference entry is DataEntrylmdaBox or DataEntrySeqNumlmdaBox, the value of chunk_offset is relative to the first byte of the payload of the IdentifiedMediaDataBox corresponding to the data reference entry. Otherwise, the value of chunk_offset is relative to the start of the containing media file.

[0341] Item location box

[0342] The ItemLocationBox provides a directory of resources in this or other files, by locating their container, their offset within that container, and their length. Using byte offsets and lengths enables common handling of this data, even by systems which do not understand the particular metadata system (handler) used. For example, a system might integrate all the externally referenced metadata resources into one place, re-adjusting offsets and references accordingly.

[0343] The box starts with three or four values, specifying the size in bytes of the offset field, length field, base_offset field, and, in versions 1 and 2 of this box, the item_reference_index fields, respectively. These values shall be from the set {0, 4, 8}.

[0344] The construction_method field indicates the ‘construction method’ for the item:- file_offset: by absolute byte offsets into the file or the payload of IdentifiedMediaDataBox referenced by data_reference_index; (construction_method == 0)- idat_offset: by byte offsets into the ItemDataBox in the same MetaBox; neither the data_reference_index nor item_reference_index fields are used; (construction_method == 1)- item_offset: by byte offset into the items indicated by the item_reference_index field, which is only used (currently) by this construction method. (construction_method == 2).

[0345] The item_reference_index is only used for the method item_offset; it indicates the 1 -based index of the item reference with referenceType 'iloc' linked from this item. If index_size is 0, then the value 1 is implied; the value 0 is reserved.

[0346] Items may be stored fragmented into extents, e.g., to enable interleaving. An extent is a contiguous subset of the bytes of the resource; the resource is formed by concatenating the extents in the order specified in this box. When only one extent is used (extent_count = 1) then either or both of the offset and length may be implied:- When the offset is not identified (the field has a length of zero), then the beginning of the source (offset 0) is implied.- When the length is not specified, or specified as zero, then the entire length of the source is implied. References into the same file as this structure-data, or items divided into more than one extent, should have an explicit offset and length, or use a MIME type requiring a different interpretation of the file, to avoid infinite recursion.

[0347] The size of the item is the sum of the extent lengths.

[0348] Extents can be interleaved with the chunks defined by the sample tables of tracks.

[0349] The offsets are relative to a data origin. That origin is determined as follows:- when the MetaBox is in a Movie Fragment, and the construction_method specifies a file offset, and the data reference indicates ‘same file’ , the data origin is the first byte of the enclosing MovieFragmentBox (as for the default -base-is-moof flag in the TrackFragmentHeaderBox);- when the construction_method specifies a file offset and the data reference indicates DataEntrylmdaBox or DataEntrySeqNumlmdaBox, the data origin is the first byte of the payload of the corresponding IdentifiedMediaDataBox;- in all other cases when the construction_method specifies a file offset, the data origin is the beginning of the file identified by the data reference;- when the construction_method specifies offsets into the ItemDataBox, the data origin is the beginning of data[] in the ItemDataBox;- when the data reference specifies another item, the data origin is the first byte of the concatenated data (of all the extents) of that item.

[0350] There are offset calculations in other parts of this file format based on the beginning of a box header; in contrast, item data offsets are calculated relative to the box payload.

[0351] The data_reference_index may take the value 0, indicating a reference into the same file as this structure-data, or an index into the data references in the DatalnformationBox in the containing MetaBox, with value 1 indicating the first entry in the data reference list.

[0352] Some referenced data may itself use offset / length techniques to address resources within it (e.g., an MP4 file might be ‘included’ in this way). Normally such offsets in the item itself are relative to the beginning of the containing file. The field ‘base offset’ provides an additional offset for offset calculations within that included data. For example, when an MP4 file is included within a file formatted to this document, then normally data-offsets within that MP4 section are relative to the beginning of file; the base offset adds to those offsets.

[0353] When an item is constructed from other items, and those source items are protected, the offset and length information apply to the source items after they have been de-protected. That is, the target item data is formed from unprotected source data.

[0354] For maximum compatibility, version 0 of this box should be used in preference to version 1 with constrnction_method==0, or version 2 when possible. Similarly, version 2 of this box should only be used when support for large item_ID values (exceeding 65535) is required or expected to be required.

[0355] When construction_method 2 is used and one item needs to have an offset of 0 into another item, the base_offset field is set to 0.

[0356] Syntax aligned(8) class ItemLocationBox extends FullBoxfiloc', version, 0) { unsigned int(4) offset_size; unsigned int(4) length_size; unsigned int(4) base_offset_size; if ((version == 1) II (version == 2)) { unsigned int(4) index_size;} else {unsigned int(4) reserved;} if (version < 2) { unsigned int( 16) item_count;} else if (version == 2) { unsigned int(32) item_count;} for (i=0; i<item_count; i++) { if (version < 2) { unsigned int(16) item_ID;} else if (version == 2) { unsigned int(32) item_ID; } if ((version == 1) II (version == 2)) { unsigned int(12) reserved = 0; unsigned int(4) construction_method;} unsigned int(16) data_reference_index; unsigned int(base_offset_size*8) base_offset; unsigned int( 16) extent_count; for (j =0; j<extent_cou +) { if (((version =(version == 2)) && (index_size > 0)) { unsigned int(index_size*8) item_reference_index;} unsigned int(offset_size*8) extent_offset; unsigned int(length_size*8) extent_length;}}}

[0357] Semantics

[0358] offset_size is taken from the set {0, 4, 8} and indicates the length in bytes of the offset field.

[0359] length_size is taken from the set {0, 4, 8} and indicates the length in bytes of the length field.

[0360] base_offset_size is taken from the set {0, 4, 8} and indicates the length in bytes of the base_offset field.

[0361] index_size is taken from the set {0, 4, 8} and indicates the length in bytes of the item_reference_index field.

[0362] item_count counts the number of resources in the following array.

[0363] item_ID is an arbitrary integer ‘name’ for this resource which can be used to refer to it (e.g., in a URL).

[0364] construction_method is taken from the set 0 (file), 1 (idat) or 2 (item)

[0365] data-reference-index is either zero ('this file') or an index, with value 1 indicating the first entry, into the data references in the DatalnformationBox.

[0366] base_offset provides a base value for offset calculations within the referenced data. When base_offset_size is 0, base_offset takes the value 0, e.g., it is unused.

[0367] extent_count provides the count of the number of extents into which the resource is fragmented; it shall have the value 1 or greater.

[0368] item_reference_index provides an index as defined for the construction method.

[0369] extent_offset provides the absolute offset, in bytes from the data origin of the container, of this extent data. When offset_size is 0, extent_offset takes the value 0.

[0370] extent_length provides the absolute length in bytes of this metadata item extent. When length_size is 0, extent_length takes the value 0. When the value is 0, then length of the extent is the length of the entire referenced container.

[0371] The 'relo' brand

[0372] Requirements for files

[0373] Files having the 'relo' brand among compatible_brands of a TypeCombinationBox associated with the FileTypeBox shall be constrained as follows:- When construction_method equal to 0 (file_offset) is in use for an item in ItemLocationBox, data_reference_index for the item shall point to a DataEntrylmdaBox.- Each data_reference_index value in the sample entries of all tracks shall point to DataEntrylmdaBox or DataEntrySeqNumlmdaBox.

[0374] By including 'relo' in the compatible_brands of TypeCombinationBox associated with a TrackTypeBox or included in ExtendedTypeBox used as an item property, it is possible to limit the scope of the constraints above to individual tracks or items.

[0375] A file having the 'relo' brand in the compatible_brands of a TypeCombinationBox associated with the FileTypeBox shall be accompanied by a BoxFilelndexBox that indexes the file and is provided separately from the file, for example using the fileindex MIME parameter with the file. The BoxFilelndexBox shall include FrontPartBox including imda_identifier for each IdentifiedMediaDataBox. BoxFilelndexBox, FrontPartBox, and the fileindex MIME parameter are specified in ISO / IEC 23001-14.

[0376] Rather than traversing the top-level boxes of the file, parsers may use the BoxFilelndexBox to conclude the byte locations of each IdentifiedMediaDataBox referred to by the DataReferenceBox(es).

[0377] Requirements for readers

[0378] When a data reference refers to an IdentifiedMediaDataBox of the containing file but no IdentifiedMediaDataBox with imda_identifier equal to the imda_ref_identifier given in the data reference is present in the file, readers shall omit the processing of the associated track or item. Furthermore, readers shall omit the processing of any tracks or items depending on such an associated track or item, for example through a track reference or an item reference.

[0379] When the 'relo' brand is in use, removal and addition of tracks and items can be done as described below.

[0380] When a track or item is added into a file complying with the 'relo' brand, its media data can be included in a new IdentifiedMediaDataBox. The MovieBox is appended with a new TrackBox or item information. The location offset information of the tracks or items that were already present in the file does not have to be updated, because the offsets are relative to the IdentifiedMediaDataBox that carries the respective media data. The location offset information of the added track or item is relative to the newly added IdentifiedMediaDataBox and is hence also straightforward to compute. It is easy to locate the amended MovieBox into the beginning of the file, because no location offsets depend on the size of the boxes preceding the IdentifiedMediaDataBoxes in the file order.

[0381] When a track or item is removed from a file complying with the 'relo' brand, it can be checked when no other media data than that of the removed track or item is present in the IdentifiedMediaDataBox. When so, the IdentifiedMediaDataBox can be removed without affecting the location offsets of the remaining tracks or items. Likewise, the TrackBox or the item information can be removed from the file without a need to re-compute the location offsets of the remaining tracks and items.

[0382] A file editor can maintain a file index that is a data structure including the box types and either the sizes or byte ranges of all the file-level boxes and the imda_identifier values of all instances of IdentifiedMediaDataBox. The file editor should be able to output the file index data structure formatted as a BoxFilelndexBox.

[0383] Partial image file reception

[0384] In the file writing, the following two steps are carried out:A file with 'relo' brand and with image items as specified in ISO / IEC 23008-12 (HEIF) is created, and each image item is located in its own IdentifiedMediaDataBox.A BoxFilelndexBox for the HEIF file is created as a sequence of BoxIndexBoxes and has the following structure:BoxIndexBox { indexed_box_type = 'ftyp'; indexed_box_size = XI;}BoxIndexBox { indexed_box_type = 'etyp'; indexed_box_size = X2;}BoxIndexBox { indexed_box_type = 'meta'; indexed_box_size = X3;}BoxIndexBox { indexed_box_type = 'imda'; indexed_box_size = X4;FrontPartBox { box_content[4]; / / containing imda_identifier}}... / / one BoxIndexBox per IdentifiedMediaDataBoxBoxIndexBox { indexed_box_type = 'imda'; indexed_box_size = X5;FrontPartBox { box_content[4]; / / containing imda_identifier }}

[0385] The file is made available for downloading, e.g., in a web page, by referring to its HTTP URL and providing its MIME media type, including the fileindex parameter.

[0386] Reception of a subset of the image items is enabled by: parsing the file index from the fileindex parameter, concluding from the file index the byte ranges for the FileTypeBox, the ExtendedTypeBox, and the root-level MetaBox, fetching the FileTypeBox, the ExtendedTypeBox, and the root-level MetaBox with a byte range request, determining which image items are of interest based on the item information in the MetaBox, resolving from the file index the byte ranges of the IdentifiedMediaDataBoxes including the image items of interest, fetching the concluded byte ranges, creating a local file by concatenating all the fetched byte ranges, and when the reader expects a file index as input, pruning the parsed file index by keeping only the top-level BoxIndexBoxes of the received top-level boxes.

[0387] The local file is a compliant HEIF file with the 'relo' brand.

[0388] An example file writing method comprises: storing coded video data in a video track of a file; obtaining a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message including a neural network representation (NNR) bitstream; storing the NNR bitstream as an item of the file; storing the NNPFC SEI message excluding the NNR bitstream in the file; and associating, in the file, the item and the NNPFC SEI message excluding the NNR bitstream.

[0389] An example file parsing method comprises: reading, from a file, coded video data of a video track; reading a neural network representation (NNR) bitstream from an item of the file; reading a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message excluding the NNR bitstream from the file; decoding coded video data from the video track; reconstructing a neural network from the NNR bitstream; forming input data to the neural network according to the NNPFC SEI message excluding the NNR bitstream; and filtering the decoded video data with a post-filter defined by the neural network using the input data.

[0390] Storage of NNR bitstream as an NNR item

[0391] A neural network may be stored as an item. The neural network data may, for example, comply with NNR. The MetaBox describing the NNR items is present at the root-level of the file or at the MovieBox level or at the track-level.

[0392] The NNR item data may reside in a standalone file, which may be identified with a URL. The URL may be provided in a DataEntryUrlBox as an entry in the DataReferenceBox, and the data reference entry may then be associated with an item in an ItemLocationBox. Alternatively, the NNR item data may reside in the ISO base media file, for example, in ItemDataBox, or in MediaDataBox, or in IdentifiedMediaDataBox.

[0393] The NNR bitstream stored as an item may specify the base filter or a filter update or both.

[0394] Storage of NNPFC SEI message associated with the NNR item

[0395] In an embodiment, an NNPFC SEI message that includes an NNR bitstream is stored as an item.

[0396] In an embodiment, an NNPFC SEI message apart from the NNR bitstream (e.g., the NNPFC SEI message may be decoupled from the associated NNR bitstream) is stored in a first extent of the item, and the NNR bitstream is stored in a second extent of the item.

[0397] In an embodiment, a file writer receives an NNPFC SEI message that includes an NNR bitstream, stores the NNR bitstream as an item in a file, and stores the NNPFC SEI message apart from the NNR bitstream separately from the NNR item in the file. The item including the NNR bitstream may be referred to as the NNR item.

[0398] In an embodiment, an NNPFC SEI message apart from the NNR bitstream is stored as an item property associated with the NNR item. An NNR item is associated with the NNPFC SEI item property through an item property association.

[0399] In an example embodiment, the NNPFC SEI item property is defined as below.Box type: 'nncs'Property type: Descriptive item propertyContainer: ItemPropertyContainerBox

[0400] The NNpfcSEIItemProperty with 4cc ‘nncs’ (any other 4cc may also be used) informs about the NNPFC SEI message related with the associated NNR item.

[0401] The example syntax of NNpfcSEIItemProperty aligned(8) class NNpfcSEIItemProperty extends ItemFullPropertyfnncs', version = 0, flags = 0){ unsigned int(8) nnpfc_sei_data_byte[]; / / NNPFC SEI messageThe byte array nnpfc_sei_data_byte[ ] includes an NNPFC SEI message.

[0402] In an embodiment, an NNPFC SEI message apart from the NNR bitstream is stored as in item. The NNR item and the item comprising the NNPFC SEI message are associated with each other with an item reference.

[0403] In an embodiment, an NNPFC SEI message apart from the NNR bitstream is stored as a sample group description entry.

[0404] In an embodiment, an NNPFC SEI message apart from the NNR bitstream is stored as a NAL-unit-like structure within a sample.

[0405] In an embodiment, an NNPFC SEI message apart from the NNR bitstream is stored in a sample entry.

[0406] In an embodiment, an NNPFC SEI message apart from the NNR bitstream is stored in a sample of a timed metadata track.

[0407] In an embodiment, an NNPFC SEI message apart from the NNR bitstream is stored as sample auxiliary information.

[0408] Association of an NNR item with samples and / or NNPFC SEI messages apart from the NNR bitstream

[0409] In an embodiment, the sample-to-item sample group defined in ISO / IEC 14496-12 is used to map the samples of the track with NNR items.

[0410] In an embodiment, a sample group is extended from the sample-to-item sample group defined in ISO / IEC 14996-12 and used to map the samples of the track with NNR items.

[0411] The NnpfcSeiEntry defined in MDS22617_WG03_N00875 may be redefined to extend the SampleToMetadataltemEntry and include the additional parameters required for defining the NNPFC SEI message.

[0412] In an example embodiment, the syntax of in ISOBMFF is shown below. In this example, it may be considered that the NNR item comprises NNPFC SEI message comprising an NNR bitstream.

[0413] Syntax aligned(8) class NnpfcSeiEntryO extends SampleToMetadataltemEntry ('nfcs'){ / / optional data describing the SEI message can be added}

[0414] In an example embodiment, the syntax of in ISOBMFF is shown below. In this example, it may be considered that the NNR item comprises an NNR bitstream and does not comprise any other parts of an NNPFC SEI message.

[0415] Syntax aligned(8) class NnpfcSeiEntryO extends SampleToMetadataltemEntry ('nfcs'){ unsigned int(8) nnpfc_sei_data_byte[] ;}

[0416] nnpfc_sei_data_byte[] is a byte array that includes exactly one complete NNPFC SEI message as specified in ISO / IEC 23002-7, apart from the NNR bitstream, when any (e.g., apart from the nnpfc_payload_byte[ i ] syntax elements).

[0417] In an embodiment, a file reader concatenates the NNR item identified by the sample group description entry after the the nnpfc_sei_data_byte[] byte array included in the sample group description entry to form an NNPFC SEI message.

[0418] In an embodiment, a file reader concludes if an NNR item is referenced by or associated with the sample group description entry. If no NNR item is referenced by or associated with the sample group description entry, an NNPFC SEI message is formed from the nnpfc_sei_data_byte[] byte array. Otherwise (i.e., when an NNR item is referenced by or associated with the sample group description entry), an NNPFC SEI message is formed from the nnpfc_sei_data_byte[] byte array followed by the item idata of the NNR item.

[0419] In an embodiment, a file reader resolves NnpfcSeiEntry to an NNPFC SEI message as follows: If num_items is equal to 0, the NNPFC SEI message is made of the nnpfc_sei_data_byte[]array. Otherwise (num_items is equal to 1), the NNPFC SEI message is made of the nnpfc_sei_data_byte[] array followed by the item data of the item with item_ID equal to item_id[0].

[0420] In an embodiment, an NNPFC sample group description entry indicates when an NNR item is referenced. In an example embodiment, the syntax of in ISOBMFF is shown below. In this example, it may be considered that the NNR item comprises an NNR bitstream and does not comprise any other parts of an NNPFC SEI message. aligned(8) class NnpfcSeiEntryQ extends SampleToMetadataltemEntry ('nfcs'){ unsigned int(l) nnr_item_referenced_flag; bit(7) reserved = 0; unsigned int(8) nnpfc_sei_data_byte[];}

[0421] Syntax

[0422] nnr_item_referenced_flag equal to 1 specifies that this sample group description entry does not include an ISO / IEC 15938-17 bitstream and that an NNPFC SEI message is formed by concatenating an associated NNR item after the NNPFC SEI message data bytes of the sample group description entry. nnr_item_referenced_flag equal to 0 indicates that an ISO / IEC 15938-17 bitstream is included in the sample group description entry or that no ISO / IEC 15938-17 bitstream is needed to form an NNPFC SEI message.

[0423] In an embodiment, the grouping_type_parameter of the sample-to-item sample group is set equal to nnpfc_id and a file writer authors the sample-to-item sample group to map its sample group description entry to the first sample of each run of samples mapped to the same NNPFC sample group description entry of the same nnpfc_id value.

[0424] In an embodiment, the grouping_type_parameter of the sample-to-item sample group is set equal to a value using the same format as for the NNPFC sample group, e.g., as follows:{ unsigned int(l) filter_update_flag; unsigned int(31) filter_id;}

[0425] filter_update_flag equal to 1 indicates that all the sample group description entries referenced by this SampleToGroupBox includes an NNR bitstream for an NNPFC SEI message that provides an update on top of a base post-processing filter. filter_update_flag equal to 0 indicates that all the sample group description entries referenced by this SampleToGroupBox includes an NNR bitstream for an NNPFC SEI message that specifies a base post-processing filter.

[0426] filter_id indicates that all the sample group description entries referenced by this SampleToGroupBox are associated with an NNPFC SEI message that has nnpfc_id equal to filter_id.

[0427] In an embodiment, the SampleToMetadataltemEntry is optionally extended with a stm_grouping_type value and, when any, stm_grouping_type_parameter value. For example, the following syntax may be used: class SampleToMetadataltemEntryO extends SampleGroupDescriptionEntry('stmi'){ unsigned int(32) meta_box_handler_type; unsigned int(32) num_items; unsigned int(32) item_id[num_i terns]; unsigned int(32) stm_grouping_type; / / optional unsigned int(32) stm_grouping_type_parameter; / / optional}

[0428] The presence of stm_grouping_type and stm_grouping_type_parameter may be controlled by the default_length and / or description_length parameters of SampleGroupDescriptionBox.

[0429] stm_grouping_type, when present, indicates that this sample group description entry associates with a sample group with grouping type equal to stm_grouping_type. When stm_grouping_type is not present, no association to a sample group is provided with this sample group description entry.

[0430] stm_grouping_type_parameter, when present, indicates that this sample group description entry associates with a sample group with grouping_type equal to stm_grouping_type and grouping_type_value equal to stm_grouping_type_value. When stm_grouping_type_parameter is notpresent, no association to a particular grouping_type_parameter value is provided with this sample group description entry.

[0431] In an embodiment, a file writer sets stm_grouping_type to indicate an NNPFC sample group.

[0432] In an embodiment, a file writer sets stm_grouping_type_parameter according to the grouping_type_parameter of the NNPFC sample group.

[0433] In an embodiment, a file reader determines, based on one or more NNPFC sample groups, which NNPFC sample groups are processed and uses their grouping type and grouping_type_parameter values to find items with the same stm_grouping_type and stm_grouping_type_parameter values in order to obtain or otherwise process (e.g., load to the neural network inference engine) the NNR items.

[0434] In an embodiment, a new track reference, which may for example be assigned a 4CC 'item', is defined to indicate that the track containing the track reference depends on the item(s) having ID values listed in the track reference (e.g., within the box having the box type 'item'). Alternatively, a new track reference, which may for example be assigned a 4CC 'nnpf, is defined to indicate that an NNPFC sample group of the track containing the track reference depends on the item having the ID value in the track reference (i.e., within the box having the box type 'nnpf ). It may be required that when the 'item' or 'nnpf track reference is used, the 'unif' brand is indicated. ISOBMFF defines the 'unif' brand indicate unified handling of identifiers, for example across tracks, track groups, entity groups, and filelevel MetaBoxes (including the item_ID values therein). Consequently, a track reference may be resolved to an item, since the same ID value is not in use as any other ID, such as track ID, track group ID, or entity group ID.

[0435] In an embodiment, a file writer includes an NNPFC sample group for a track according to any other embodiment. A file writer associates at least one of the sample group description entries of the NNPFC sample group with an NNR item according to any other embodiment. The file writer additionally includes an 'item' or 'nnpf track reference into the track, and includes the item ID of the NNR item in the 'item' or 'nnpf track reference.

[0436] In an embodiment, an NNPFC sample group description entry may include a flag indicating that the item data of the item referred to by the 'item' or 'nnpf' track reference is used in resolving the sample group description entry to an NNPFC SEI message. Alternatively, an NNPFCsample group description entry may include a track reference index referring to an 'item' or 'nnpf track reference indicating which of the listed item IDs therein is used in resolving the sample group description entry to an NNPFC SEI message.

[0437] In an embodiment, a file reader parses an NNPFC sample group for a track according to any other embodiment. A file reader parses an association of at least one of the sample group description entries of the NNPFC sample group with an NNR item according to any other embodiment. The file reader additionally parses an 'item' or 'nnpf' track reference from the track, and parses the item ID of the NNR item from the 'item' or 'nnpf track reference, and reconstructs an NNPFC SEI message from the sample group description entry and the item data of the NNR item.

[0438] Storage of NNR data units in 'imda' boxes

[0439] The neural network data may, for example, comply with NNR. The NNR data units which specify either the base filter or the update filters or both with a specific nnpfc_id are stored in 'imda' boxes.

[0440] In an embodiment, the NNR data units are referenced using data reference entry of DataEntrylmdaBox.

[0441] In an embodiment, a file writer includes item data of one and only one item into an IdentifiedMediaDataBox. A DataEntryltemlmdaBox is defined as follows: aligned(8) class DataEntryltemlmdaBox (bit(24) flags) extends DataEntryBaseBox fitim', flags){}

[0442] The DataEntryltemlmdaBox identifies the IdentifiedMediaDataBox including the item data accessed through the data_reference_index corresponding to this DataEntryltemlmdaBox. When a data_reference_index included in an ItemLocationBox refers to DataEntryltemlmdaBox, the respective item is included in an IdentifiedMediaDataBox that has imda_identifier equal to item_ID of the item, and base_offset and extent_offset are relative to the first byte of the payload of the IdentifiedMediaDataBox.

[0443] In an embodiment, a new data reference entry may be defined called the DataEntryNNPFImdaBox with 4cc ‘nnim’ or any other name and 4cc may be used. The DataEntryNNPFImdaBox refers to a NnpfcSeiEntry.

[0444] The DataEntryNNPFImdaBox identifies the IdentifiedMediaDataBox including the NNPF SEI data accessed through the data_reference_index corresponding to this DataEntryNNPFImdaBox.

[0445] When a data_reference_index included in a ItemLocationBox refers to DataEntryNNPFImdaBox, the respective item is included in an IdentifiedMediaDataBox that has imda_identifier equal to nnpfc_sei_identifier of the NnpfcSeiEntry and base_offset and extent_offset are relative to the first byte of the payload of the IdentifiedMediaDataBox.. aligned(8) class DataEntryNNPFImdaBox (bit(24) flags) extends FullBox(’nnim’, version = 0, flags) { }

[0446] In an embodiment, NnpfcSeiEntry sample group entry includes the unique identifier called the nnpfc_sei_identifier. aligned(8) class NnpfcSeiEntryQ extends VisualSampleGroupEntry('nfcs'){ unsigned int(32) nnpfc_sei_identifier[];} nnpfc_sei_identifier is a unique identifier.

[0447] These embodiments enable downloading some of the IdentifiedMediaDataBoxes, while the file reconstructed with only some of the IdentifiedMediaDataBoxes remains conforming to ISOBMFF, since the byte offsets, e.g., in the ItemLocationBox, are relative to the IdentifiedMediaDataBoxes and need not be rewritten when a file is reconstructed with some of the filelevel boxes.

[0448] Neural network post-filter SEI offsets

[0449] In an embodiment, a NNPF SEI offset box is defined as below. The box, for example, represents a table of offset values into each SEI message.

[0450] In an embodiment, the NNPF SEI offset table gives the index of each NNPF SEI into the including file. There are two variants, permitting the use of 32-bit or 64-bit offsets. The latter is useful, for example, when managing very large (e.g., larger than the 32-bit range size)NNPF SEI or NNR data. At most one of these variants will occur in any single instance.

[0451] When the referenced data reference entry is not DataEntrylmdaBox or DataEntrySeqNumlmdaBox or DataEntryNNPFImdaBox, offsets are file offsets, not the offset into any box within the file (e.g. MediaDataBox). This permits referring to NNR data in files without any box structure. It does also mean that care must be taken when constructing a self-included ISO file with its structure-data (MovieBox) at the front, as the size of the MovieBox may affect the NNPF SEI offsets to the NNR data.

[0452] When the referenced data reference entry is DataEntrylmdaBox or DataEntrySeqNumlmdaBox or DataEntryNNPFImdaBox, offsets are relative to the first byte of the payload of the IdentifiedMediaDataBox corresponding to the data reference entry. This permits reordering file-level boxes and receiving a subset of file-level boxes but could require traversing the file-level boxes until the referenced IdentifiedMediaDataBox is found.

[0453] In an example embodiment, the NNPF SEI message data may be present in the file without any box structure. In such a case the offset to each of the SEI message is specified by the NNPFSEI offset boxes, which defines a NNPFC SEI table as defined below. When the referenced data reference entry is not DataEntrylmdaBox or DataEntrySeqNumlmdaBox, offsets are file offsets, not the offset into any box within the file (e.g., MediaDataBox). When the referenced data reference entry is DataEntrylmdaBox or DataEntrySeqNumlmdaBox, offsets are relative to the first byte of the payload of the IdentifiedMediaDataBox corresponding to the data reference entry. There are two variants, permitting the use of 32-bit or 64-bit offsets

[0454] Definition

[0455] Box Type: ‘nnfo’ ‘no64’

[0456] Container: ?

[0457] Mandatory: Yes

[0458] Quantity: Exactly one variant shall be present

[0459] Syntax aligned(8) class NNPFSEIOffsetBox extends FullBoxfnnfo', version = 0, 0) { unsigned int(32) entry _count; for (i=l; i <= entry _count; i++) { unsigned int(32) nnpf_sei_offset;}} aligned(8) class NNPFSEILargeOffsetBox extends FullBox('no64', version = 0, 0) { unsigned int(32) entry _count; for (i=l; i <= entry _count; i++) { unsigned int(64) nnpf_sei_offset;}} aligned(8) class NnpfcSeiEntryQ extends VisualSampleGroupEntry('nfcs'){NNPFSEIOffsetBox nnpf_sei_offset;}

[0460] Semantics

[0461] version is an integer that specifies the version of this box.

[0462] entry _count is an integer that gives the number of entries in a table providing the offsets to each SEI message in the file.

[0463] nnpf_sei_offset is a 32 or 64 bit integer that gives the offset of the start of a NNR data used in NNPF SEI. When the referenced data reference entry is DataEntrylmdaBox or DataEntrySeqNumlmdaBox, the value of nnpf_sei_offset is relative to the first byte of the payload of the IdentifiedMediaDataBox corresponding to the data reference entry. Otherwise, the value of nnpf_seis_offset is relative to the start of the including media file.

[0464] FIG. 7 is an example apparatus 700, which may be implemented in hardware, caused to implement carriage of neural-network post-filter, for example, carriage of neural-network post-filter SEI with ISOBMFF. The apparatus 700 comprises at least one processor 702, at least one non-transitorymemory 704 including computer program code 705, wherein the at least one memory 704 and the computer program code 705 are configured to, with the at least one processor 702, cause the apparatus 700 to implement carriage of neural-network post-filter, for example, carriage of neural-network postfilter SEI with ISOBMFF.

[0465] The apparatus 700 optionally includes a display 708 that may be used to display content during rendering. The apparatus 700 optionally includes one or more network (NW) interfaces (I / F(s)) 710. The NW I / F(s) 710 may be wired and / or wireless and communicate over the Internet / other network(s) via any communication technique. The NW I / F(s) 710 may comprise one or more transmitters and one or more receivers. The N / W I / F(s) 710 may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder / decoder circuitry(ies) and one or more antennas.

[0466] The apparatus 700 may be a remote, virtual or cloud apparatus. The apparatus 700 may be either a coder or a decoder, or both a coder and a decoder. The at least one memory 704 may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The at least one memory 704 may comprise a database for storing data. The apparatus 700 need not comprise each of the features mentioned, or may comprise other features as well. The apparatus 700 may correspond to or be another embodiment of the apparatus 50 shown in FIG. 1 to FIG. 6. The apparatus 700 may correspond to or be another embodiment of the apparatuses shown in FIG. 10, including UE 110, RAN node 170, or network element(s) 190.

[0467] FIG. 8 is an example method 800 to implement the examples described herein, in accordance with an embodiment. At 802, the method 800 includes storing coded video data in a video track of a file. At 804, the method 800 includes obtaining a neural -network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message comprising a neural network representation (NNR) bitstream. At 806, the method 800 includes storing the NNR bitstream as an item of the file. At 808, the method 800 includes storing the NNPFC SEI message excluding the NNR bitstream in the file. At 810, the method 800 includes associating, in the file, the item and the NNPFC SEI message excluding the NNR bitstream.

[0468] The method 800 may be performed with an apparatus described herein, for example, the any apparatus of FIG. 1 to FIG. 7, any apparatus of FIG. 10, or any other apparatus described herein.

[0469] FIG. 9 is an example method 900 to implement the examples described herein, in accordance with an embodiment. At 902, the method 900 includes reading, from a file, coded video data of a video track. At 904, the method 900 includes reading a neural network representation (NNR) bitstream from an item of the file. At 906, the method 900 includes reading a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message excluding the NNR bitstream from the file. At 908, the method 900 includes decoding coded video data from the video track for generating a decoded video data. At 910, the method 900 includes reconstructing a neural network from the NNR bitstream. At 912, the method 900 includes forming input data to the neural network according to the NNPFC SEI message excluding the NNR bitstream. At 914, the method 900 includes filtering the decoded video data with a post-filter defined by the neural network using the input data.

[0470] The method 900 may be performed with an apparatus described herein, for example, the any apparatus of FIG. 1 to FIG. 7, any apparatus of FIG. 10, or any other apparatus described herein.

[0471] Referring to FIG. 10, this figure shows a block diagram of one possible and non-limiting example in which the examples may be practiced. A user equipment (UE) 110, radio access network (RAN) node 170, and network element(s) 190 are illustrated. In the example of FIG. 1, the user equipment (UE) 110 is in wireless communication with a wireless network 100. A UE is a wireless device that can access the wireless network 100. The UE 110 includes one or more processors 120, one or more memories 125, and one or more transceivers 130 interconnected through one or more buses 127. Each of the one or more transceivers 130 includes a receiver, Rx, 132 and a transmitter, Tx, 133. The one or more buses 127 may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. The one or more transceivers 130 are connected to one or more antennas 128. The one or more memories 125 include computer program code 123. The UE 110 includes a module 140, comprising one of or both parts 140-1 and / or 140-2, which may be implemented in a number of ways. The module 140 may be implemented in hardware as module 140-1, such as being implemented as part of the one or more processors 120. The module 140-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the module 140 may be implemented as module 140-2, which is implemented as computer program code 123 and is executed by the one or more processors 120. For instance, the one or more memories 125 and the computer program code 123 may be configured to, with the one or more processors 120, cause the user equipment 110 to perform one or more of the operations as described herein. The UE 110 communicates with a radio access network (RAN) node 170 via a wireless link 111.

[0472] The RAN node 170 in this example is a base station that provides access by wireless devices such as the UE 110 to the wireless network 100. The RAN node 170 may be, for example, a base station for fifth generation cellular network technology (5G), also called New Radio (NR). In 5G, the RAN node 170 may be a NG-RAN node, which is defined as either a gNB (e.g., base station for 5G / NR, for example, a node providing NR user plane and control plane protocol terminations towards the UE, and connected via the NG interface to the 5GC) or an ng (new generation)-eNB. A gNB is a node providing NR user plane and control plane protocol terminations towards the UE, and connected via the NG interface to a 5G core network (5GC) (such as, for example, the network element(s) 190). The ng-eNB, is a node providing evolved universal terrestrial radio access (E-UTRA), for example, the LTE radio access technology, user plane and control plane protocol terminations towards the UE, and connected via the NG interface to the 5GC. The NG-RAN node may include multiple gNBs, which may also include a central unit (CU) (gNB-CU) 196 and distributed unit(s) (DUs) (gNB-DUs), of which DU 195 is shown. Note that the DU may include or be coupled to and control a radio unit (RU). The gNB- CU is a logical node hosting radio resource control (RRC), service data adaptation protocol (SDAP) and PDCP protocols of the gNB or RRC and packet data convergence protocol (PDCP) protocols of the en-gNB (e.g., node providing NR user plane and control plane protocol terminations towards the UE, and acting as secondary node in E-UTRA-NR dual connectivity (EN-DC)) that controls the operation of one or more gNB-DUs. The gNB-CU terminates the interface between CU and DU control interface (Fl or Fl-C) interface connected with the gNB-DU. The Fl interface is illustrated as reference 198, although reference 198 also illustrates a link between remote elements of the RAN node 170 and centralized elements of the RAN node 170, such as between the gNB-CU 196 and the gNB-DU 195. The gNB-DU is a logical node hosting radio link control (REC), MAC and physical layer (PHY) layers of the gNB or en-gNB, and its operation is partly controlled by gNB-CU. One gNB-CU supports one or multiple cells. One cell is supported by only one gNB-DU. The gNB-DU terminates the Fl interface 198 connected with the gNB-CU. Note that the DU 195 is considered to include the transceiver 160, for example, as part of a RU, but some examples of this may have the transceiver 160 as part of a separate RU, for example, under control of and connected to the DU 195. The RAN node 170 may also be an eNB (evolved NodeB) base station, for example, long term evolution (LTE), or any other suitable base station or node.

[0473] The RAN node 170 includes one or more processors 152, one or more memories 155, one or more network interfaces (N / W I / F(s)) 161, and one or more transceivers 160 interconnected through one or more buses 157. Each of the one or more transceivers 160 includes a receiver, Rx, 162 and a transmitter, Tx, 163. The one or more transceivers 160 are connected to one or more antennas 158. The one or more memories 155 include computer program code 153. The CU 196 may include theprocessor(s) 152, memories 155, and network interfaces 161. Note that the DU 195 may also include its own memory / memories and processor(s), and / or other hardware, but these are not shown.

[0474] The RAN node 170 includes a module 150, comprising one of or both parts 150-1 and / or 150-2, which may be implemented in a number of ways. The module 150 may be implemented in hardware as module 150-1, such as being implemented as part of the one or more processors 152. The module 150-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the module 150 may be implemented as module 150-2, which is implemented as computer program code 153 and is executed by the one or more processors 152. For instance, the one or more memories 155 and the computer program code 153 are configured to, with the one or more processors 152, cause the RAN node 170 to perform one or more of the operations as described herein. Note that the functionality of the module 150 may be distributed, such as being distributed between the DU 195 and the CU 196, or be implemented solely in the DU 195.

[0475] The one or more network interfaces 161 communicate over a network such as via the links 176 and 131. Two or more gNBs 170 may communicate using, for example, link 176. The link 176 may be wired or wireless or both and may implement, for example, an Xn interface for 5G, an X2 interface for LTE, or other suitable interface for other standards.

[0476] The one or more buses 157 may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, wireless channels, and the like. For example, the one or more transceivers 160 may be implemented as a remote radio head (RRH) 195 for LTE or a distributed unit (DU) 195 for gNB implementation for 5G, with the other elements of the RAN node 170 possibly being physically in a different location from the RRH / DU, and the one or more buses 157 could be implemented in part as, for example, fiber optic cable or other suitable network connection to connect the other elements (for example, a central unit (CU), gNB-CU) of the RAN node 170 to the RRH / DU 195. Reference 198 also indicates those suitable network link(s).

[0477] It is noted that description herein indicates that ‘cells’ perform functions, but it should be clear that equipment which forms the cell may perform the functions. The cell makes up part of a base station. That is, there can be multiple cells per base station. For example, there could be three cells for a single carrier frequency and associated bandwidth, each cell covering one-third of a 360 degree area so that the single base station’s coverage area covers an approximate oval or circle. Furthermore, each cell can correspond to a single carrier and a base station may use multiple carriers. So when there are three 120 degree cells per carrier and two carriers, then the base station has a total of 6 cells.

[0478] The wireless network 100 may include a network element or elements 190 that may include core network functionality, and which provides connectivity via a link or links 181 with a further network, such as a telephone network and / or a data communications network (for example, the Internet). Such core network functionality for 5G may include access and mobility management function(s) (AMF(S)) and / or user plane functions (UPF(s)) and / or session management function(s) (SMF(s)). Such core network functionality for LTE may include MME (Mobility Management Entity ) / SGW (Serving Gateway) functionality. These are merely example functions that may be supported by the network element(s) 190, and note that both 5G and LTE functions might be supported. The RAN node 170 is coupled via a link 131 to the network element 190. The link 131 may be implemented as, for example, an NG interface for 5G, or an SI interface for LTE, or other suitable interface for other standards. The network element 190 includes one or more processors 175, one or more memories 171, and one or more network interfaces (N / W I / F(s)) 180, interconnected through one or more buses 185. The one or more memories 171 include computer program code 173. The one or more memories 171 and the computer program code 173 are configured to, with the one or more processors 175, cause the network element 190 to perform one or more operations.

[0479] The wireless network 100 may implement network virtualization, which is the process of combining hardware and software network resources and network functionality into a single, softwarebased administrative entity, a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to software containers on a single system. Note that the virtualized entities that result from the network virtualization are still implemented, at some level, using hardware such as processors 152 or 175 and memories 155 and 171, and also such virtualized entities create technical effects.

[0480] The computer readable memories 125, 155, and 171 may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The computer readable memories 125, 155, and 171 may be means for performing storage functions. The processors 120, 152, and 175 may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi-core processor architecture, as non-limiting examples. The processors 120, 152, and 175 may be means for performing functions, such as controlling the UE 110, RAN node 170, network element(s) 190, and other functions as described herein.

[0481] In general, the various embodiments of the user equipment 110 can include, but are not limited to, cellular telephones such as smart phones, tablets, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback appliances having wireless communication capabilities, Internet appliances permitting wireless Internet access and browsing, tablets with wireless communication capabilities, as well as portable units or terminals that incorporate combinations of such functions.

[0482] One or more of modules 140-1, 140-2, 150-1, and 150-2 may be configured to implement carriage of neural-network post-filter, for example, carriage of neural-network post-filter SEI with ISOBMFF. Computer program code 173 may also be implement carriage of neural -network post-filter, for example, carriage of neural-network post-filter SEI with ISOBMFF.

[0483] As described above, FIGs. 8 and 9 include flowcharts of an apparatus (e.g. 50, 700, or any other apparatuses described herein), method, and computer program product according to certain example embodiments. It will be understood that each block of the flowcharts, and combinations of blocks in the flowcharts, may be implemented by various means, such as hardware, firmware, processor, circuitry, and / or other devices associated with execution of software including one or more computer program instructions. For example, one or more of the procedures described above may be embodied by computer program instructions. In this regard, the computer program instructions which embody the procedures described above may be stored by a memory (e.g. 58, 125, or 704) of an apparatus employing an embodiment of the present invention and executed by processing circuitry (e.g. 56, 120, or 702) of the apparatus. As will be appreciated, any such computer program instructions may be loaded onto a computer or other programmable apparatus (e.g., hardware) to produce a machine, such that the resulting computer or other programmable apparatus implements the functions specified in the flowchart blocks. These computer program instructions may also be stored in a computer-readable memory that may direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture, the execution of which implements the function specified in the flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide operations for implementing the functions specified in the flowchart blocks.

[0484] A computer program product is therefore defined in those instances in which the computer program instructions, such as computer-readable program code portions, are stored by at least one non- transitory computer-readable storage medium with the computer program instructions, such as the computer-readable program code portions, being configured, upon execution, to perform the functions described above, such as in conjunction with the flowchart(s) of FIGs. 8 and 9. In other embodiments, the computer program instructions, such as the computer-readable program code portions, need not be stored or otherwise embodied by a non-transitory computer-readable storage medium, but may, instead, be embodied by a transitory medium with the computer program instructions, such as the computer- readable program code portions, still being configured, upon execution, to perform the functions described above.

[0485] Accordingly, blocks of the flowcharts support combinations of means for performing the specified functions and combinations of operations for performing the specified functions for performing the specified functions. It will also be understood that one or more blocks of the flowcharts, and combinations of blocks in the flowcharts, may be implemented by special purpose hardware-based computer systems which perform the specified functions, or combinations of special purpose hardware and computer instructions.

[0486] In some embodiments, certain ones of the operations above may be modified or further amplified. Furthermore, in some embodiments, additional optional operations may be included. Modifications, additions, or amplifications to the operations above may be performed in any order and in any combination.

[0487] In the above, some example embodiments have been described with the help of syntax of the bitstream. It needs to be understood, however, that the corresponding structure and / or computer program may reside at the encoder for generating the bitstream and / or at the decoder for decoding the bitstream.

[0488] In the above, where example embodiments have been described with reference to an encoder, it needs to be understood that the resulting bitstream and the decoder have corresponding elements in them. Likewise, where example embodiments have been described with reference to a decoder, it needs to be understood that the encoder has structure and / or computer program for generating the bitstream to be decoded by the decoder.

[0489] Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the inventions are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe example embodiments in the context of certain example combinations of elements and / or functions, it should be appreciated that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and / or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0490] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

[0491] References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single / multi-processor architectures and sequential (Von Neumann) / parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGA), application specific circuits (ASIC), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device such as instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device, and the like.

[0492] As used herein, the term ‘circuitry’ may refer to any of the following: (a) hardware circuit implementations, such as implementations in analog and / or digital circuitry, and (b) combinations of circuits and software (and / or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s) / software including digital signal processor(s), software, and memory(ies) that work together to cause an apparatus to perform various functions, and (c) circuits, such as amicroprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present. This description of ‘circuitry’ applies to uses of this term in this application. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and / or firmware. The term ‘circuitry’ would also cover, for example and if applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device.

[0493] Circuitry or Circuit: As used in this application, the term ‘circuitry’ or ‘circuit’ may refer to one or more or all of the following:(a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); and(b) combinations of hardware circuits and software, such as (as applicable):(i) a combination of analog and / or digital hardware circuit(s) with software / firmware; and(ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and(c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

[0494] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

Claims

CLAIMSWhat is claimed is:

1. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: storing coded video data in a video track of a file; obtaining a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message comprising a neural network representation (NNR) bitstream; storing the NNR bitstream as an item of the file; storing the NNPFC SEI message excluding the NNR bitstream in the file; and associating, in the file, the item and the NNPFC SEI message excluding the NNR bitstream.

2. The apparatus of claim 1, wherein the NNR bitstream stored in the item specifies at least one of a base filter or a filter update.

3. The apparatus of claim 1 , wherein the NNPFC SEI message excluding the NNR bitstream is stored as an item property associated with an NNR item, and wherein the NNR item is associated with an NNPFC SEI item property through an item property association.

4. The apparatus of claim 3, wherein the NNPFC SEI item property comprises a box type, a property type, and a container.

5. The apparatus of claim 1, wherein the NNR item and the item comprising the NNPFC SEI message are associated with each other with an item reference.

6. The apparatus of claim 1 , wherein the NNPFC SEI message excluding the NNR bitstream is stored as a sample group description entry.

7. The apparatus of claim 1, wherein the NNPFC SEI message excluding the NNR bitstream is stored as a NAL-unit-like structure within a sample.

8. The apparatus of claim 1, wherein the NNPFC SEI message excluding the NNR bitstream is stored in a sample entry.

9. The apparatus of claim 1, wherein the NNPFC SEI message excluding the NNR bitstream is stored in a sample of a timed metadata track.

10. The apparatus of claim 1, wherein the NNPFC SEI message excluding the NNR bitstream is stored as sample auxiliary information.

11. The apparatus of claim 1 or 6, wherein a sample-to-item sample group is used to map samples of a track with the NNR item.

12. The apparatus of any of the claims 1 to 10, wherein a sample group is extended from a sample-to-item sample group defined in ISO / IEC 14496-12 and is used to map samples of a track with NNR items.

13. The apparatus of claim 6, wherein to the sample group description entry extends a sample-to-item sample group description entry.

14. The apparatus of claim 12, wherein the sample group description entry indicates when the NNR item is referenced.

15. The apparatus of any of the claims 1 to 10, wherein a track reference from a track comprising the NNPFC SEI message excluding the NNR bitstream to the item is included in the file.

16. The apparatus of claim 1, wherein a ISOBMFF comprises an NNR item referenced flag, wherein when the NNR item referenced flag equal to one specifies that this sample group description entry does not include an ISO / IEC 15938-17 bitstream and that the NNPFC SEI message is formed by concatenating an associated NNR item after the NNPFC SEI message data bytes of the sample group description entry, and wherein when the NNR item referenced flag equal to zero indicates that the ISO / IEC 15938-17 bitstream is included in the sample group description entry or that no the ISO / IEC 15938-17 bitstream is needed to form the NNPFC SEI message.

17. The apparatus of claim 1, wherein the apparatus is further caused to perform:setting a sample-to-item sample group NNPFC identifier; and mapping a sample group description of a sample-to-item sample group to a first sample of each run of samples mapped to same NNPFC sample group description entry of same NNPFC identifier value.

18. The apparatus of claim 17, wherein a grouping type parameter of the sample-to-item sample group comprises a filter update flag and a filter identifier, and wherein the filter update flag equal to one indicates that all sample group description entries referenced by a sample to group box includes the NNR bitstream for the NNPFC SEI message that provides an update on top of a base post-processing filter, and wherein the filter update flag equal to zero indicates that all the sample group description entries referenced by the sample to group box includes the NNR bitstream for the NNPFC SEI message that specifies the base post-processing filter, and wherein the filter identifier indicates that all the sample group description entries referenced by the sample to group box are associated with the NNPFC SEI message that has NNPFC identifier equal to the filter identifier.

19. The apparatus of claim 1, wherein a SampleToMetadataltemEntry is further extended to include a stm_grouping_type and a stm_grouping_type_parameter and wherein when the stm_grouping_type is present indicates that the sample group description entry associates with a sample group with a grouping lype equal to the stm_grouping_type, and when the stm_grouping_type is not present, no association to the sample group is provided with the sample group description entry, and wherein when the stm_grouping_type_parameter is present, indicates that the sample group description entry associates with the sample group with the grouping lype equal to the stm_grouping_type and grouping_type_value equal to stm_grouping_type_value, and wherein ,when the stm_grouping_type_parameter is not present, no association to a particular grouping_type_parameter value is provided with the sample group description entry.

20. The apparatus of claim 2, wherein NNR data units specifying either the base filter, the filter update, or both with a specific NNPFC identifier are stored in 'imda' boxes.

21. The apparatus of claim 20, wherein the NNR data units are referenced using data reference entry of a DataEntryltemlmdaBox, wherein the DataEntryltemlmdaBox identifies an IdentifiedMediaDataBox comprising an item data accessed through a data_reference_index corresponding to the DataEntryltemlmdaBox, when the data_reference_index comprised in an ItemLocationBox refers to the DataEntryltemlmdaBox, a respective item is included in theIdentifiedMediaDataBox comprising imda_identifier equal to item_ID of the item, and base_offset and extent_offset are relative to a first byte of a payload of the IdentifiedMediaDataBox.

22. An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: reading, from a file, coded video data of a video track; reading a neural network representation (NNR) bitstream from an item of the file; reading a neural -network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message excluding the NNR bitstream from the file; decoding coded video data from the video track for generating a decoded video data; reconstructing a neural network from the NNR bitstream; forming input data to the neural network according to the NNPFC SEI message excluding the NNR bitstream; and filtering the decoded video data with a post-filter defined by the neural network using input data.

23. The apparatus of claim 22, wherein the apparatus is further caused to perform: concatenating an NNR item identified by a sample group description entry after an nnpfc sei data byte array comprised in the sample group description entry to form the NNPFC SEI message.

24. The apparatus of any of claims 22 or 23, wherein the apparatus is further caused to perform: determining, based on one or more NNPFC sample groups, which NNPFC sample groups are processed; and using grouping type and grouping_type_parameter values of the NNPFC sample groups that are processed to find items with the same stm_grouping_type and stm_grouping_type_parameter values in order to obtain or otherwise process the NNR item.

25. The apparatus of any of the claims 23 or 24, wherein the apparatus is further caused to perform: concluding whether the NNR item is referenced by or associated with the sample group description entry;wherein when the NNR item is not referenced by or is not associated with the sample group description entry, the NNPFC SEI message is formed from an NNPFC SEI bitstream including the NNR bitstream; and wherein the NNR item is referenced by or associated with the sample group description entry, the NNPFC SEI message is formed from the NNPFC SEI bitstream excluding the NNR bitstream followed by an item data of the NNR item.

26. A method comprising: storing coded video data in a video track of a file; obtaining a neural-network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message comprising a neural network representation (NNR) bitstream; storing the NNR bitstream as an item of the file; storing the NNPFC SEI message excluding the NNR bitstream in the file; and associating, in the file, the item and the NNPFC SEI message excluding the NNR bitstream.

27. The method of claim 26, wherein the NNR bitstream stored in the item specifies at least one of a base filter or a filter update.

28. The method of claim 26, wherein the NNPFC SEI message excluding the NNR bitstream is stored as an item property associated with an NNR item, and wherein the NNR item is associated with an NNPFC SEI item property through an item property association.

29. The method of claim 28, wherein the NNPFC SEI item property comprises a box type, a property type, and a container.

30. The method of claim 26, wherein the NNR item and the item comprising the NNPFC SEI message are associated with each other with an item reference.

31. The method of claim 26, wherein the NNPFC SEI message excluding the NNR bitstream is stored as a sample group description entry.

32. The method of claim 26, wherein the NNPFC SEI message excluding the NNR bitstream is stored as a NAL-unit-like structure within a sample.

33. The method of claim 26, wherein the NNPFC SEI message excluding the NNR bitstream is stored in a sample entry.

34. The method of claim 26, wherein the NNPFC SEI message excluding the NNR bitstream is stored in a sample of a timed metadata track.

35. The method of claim 26, wherein the NNPFC SEI message excluding the NNR bitstream is stored as sample auxiliary information.

36. The method of claim 26 or 31, wherein a sample-to-item sample group is used to map samples of a track with the NNR item.

37. The method of any of the claims 26 to 35, wherein a sample group is extended from a sample-to-item sample group defined in ISO / IEC 14496-12 and is used to map samples of a track with NNR items.

38. The method of claim 31, wherein to the sample group description entry extends a sample- to-item sample group description entry.

39. The method of claim 37, wherein a sample group description entry indicates when the NNR item is referenced.

40. The method of any of the claims 26 to 35, wherein a track reference from a track comprising the NNPFC SEI message excluding the NNR bitstream to the item is included in the file.

41. The method of claim 26, wherein a ISOBMFF comprises an NNR item referenced flag, wherein when the NNR item referenced flag equal to one specifies that this sample group description entry does not include an ISO / IEC 15938-17 bitstream and that the NNPFC SEI message is formed by concatenating an associated NNR item after the NNPFC SEI message data bytes of the sample group description entry, and wherein when the NNR item referenced flag equal to zero indicates that the ISO / IEC 15938-17 bitstream is included in the sample group description entry or that no the ISO / IEC 15938-17 bitstream is needed to form the NNPFC SEI message.

42. The method of claim 26 further comprising:setting a sample-to-item sample group NNPFC identifier; and mapping a sample group description of a sample-to-item sample group to a first sample of each run of samples mapped to same NNPFC sample group description entry of same NNPFC identifier value.

43. The method of claim 42, wherein a grouping type parameter of the sample-to-item sample group comprises a filter update flag and a filter identifier, and wherein the filter update flag equal to one indicates that all sample group description entries referenced by a sample to group box includes the NNR bitstream for the NNPFC SEI message that provides an update on top of a base post-processing filter, and wherein the filter update flag equal to zero indicates that all the sample group description entries referenced by the sample to group box includes the NNR bitstream for the NNPFC SEI message that specifies the base post-processing filter, and wherein the filter identifier indicates that all the sample group description entries referenced by the sample to group box are associated with the NNPFC SEI message that has NNPFC identifier equal to the filter identifier.

44. The method of claim 26, wherein a SampleToMetadataltemEntry is further extended to include a stm_grouping_type and a stm_grouping_type_parameter and wherein when the stm_grouping_type is present indicates that the sample group description entry associates with a sample group with a grouping lype equal to the stm_grouping_type, and when the stm_grouping_type is not present, no association to the sample group is provided with the sample group description entry, and wherein when the stm_grouping_type_parameter is present, indicates that the sample group description entry associates with the sample group with the grouping lype equal to the stm_grouping_type and grouping_type_value equal to stm_grouping_type_value, and wherein when the stm_grouping_type_parameter is not present, no association to a particular grouping_type_parameter value is provided with the sample group description entry.

45. The method of claim 27, wherein NNR data units specifying either a base filter, the filter update, or both with a specific NNPFC identifier are stored in 'imda' boxes.

46. The method of claim 45, wherein the NNR data units are referenced using data reference entry of a DataEntryltemlmdaBox, wherein the DataEntryltemlmdaBox identifies an IdentifiedMediaDataBox comprising an item data accessed through a data_reference_index corresponding to the DataEntryltemlmdaBox, when the data_reference_index comprised in an ItemLocationBox refers to the DataEntryltemlmdaBox, a respective item is included in theIdentifiedMediaDataBox comprising imda_identifier equal to item_ID of the item, and base_offset and extent_offset are relative to a first byte of a payload of the IdentifiedMediaDataBox.

47. A method comprising: reading, from a file, coded video data of a video track; reading a neural network representation (NNR) bitstream from an item of the file; reading a neural -network post-filter characteristics (NNPFC) supplemental enhancement information (SEI) message excluding the NNR bitstream from the file; decoding coded video data from the video track for generating a decoded video data; reconstructing a neural network from the NNR bitstream; forming input data to the neural network according to the NNPFC SEI message excluding the NNR bitstream; and filtering the decoded video data with a post-filter defined by the neural network using input data.

48. The method of claim 47 further comprising: concatenating an NNR item identified by a sample group description entry after an nnpfc sei data byte array comprised in the sample group description entry to form the NNPFC SEI message.

49. The method of any of claims 47 or 48 further comprising: determining, based on one or more NNPFC sample groups, which NNPFC sample groups are processed; and using grouping type and grouping_type_parameter values of the NNPFC sample groups that are processed to find items with the same stm_grouping_type and stm_grouping_type_parameter values in order to obtain or otherwise process the NNR item.

50. The method of any of the claims 48 or 49, further comprising: concluding whether the NNR item is referenced by or associated with the sample group description entry; wherein when the NNR item is not referenced by or is not associated with the sample group description entry, the NNPFC SEI message is formed from an NNPFC SEI bitstream including the NNR bitstream; and wherein the NNR item is referenced by or associated with the sample group description entry, the NNPFC SEI message is formed from the NNPFC SEI bitstream excluding the NNR bitstream followed by an item data of the NNR item.

51. A non -transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform the methods as claimed in any of the claims 26 to 50.

52. An apparatus comprising means for performing methods as claimed in any of the claims 26 to 50.

Images