Method and apparatus for encapsulating and parsing media files including neural network-based post-filter information

Abstract

The present invention relates to a method of encapsulating media data into an ISOBMFF-based media file, the media data comprising media data units and at least one post-filter data unit, the method comprising: generating a first track of a sequence of encapsulated samples, each sample comprising at least one of the media data units, the at least one post-filter data unit is applied to at least one of the samples of the first track; generating a data portion of an entity comprising at least a portion of one of the post-filter data units; generating post-filter information describing at least one post-filter data unit; and encapsulating the first track, the entity, and the post-filter information in the media file, the post-filter information being encapsulated in a metadata portion of the media file.

Description

Technical Field

[0001] This disclosure relates to methods and apparatus for encapsulating media data in ISOBMFF-based media files and parsing such media files. More particularly, it relates to a method for encapsulating neural network-based post-filter information associated with the encapsulated media data. Background Technology

[0002] The ISO Basic Media File Format (ISOBMFF, also known as the file format) is the general format upon which many other, more specific file formats are built. ISOBMFF is standardized by the International Organization for Standardization as ISO / IEC 14496-12. This format contains characteristics of media data, such as the timing, size, or media information of the timing sequence of media data (such as audiovisual presentation). These characteristics are referred to as metadata (or structure data) in contrast to media data (or data). Structure data is contained in metadata boxes (e.g., MovieBox 'moov', MovieFragmentBox 'moof', or MetaBox 'meta'), while data is contained in media data boxes (e.g., MediaDataBox 'mdat', ItemDataBox ('idat'), or the identified media data box 'imda'). An ISO Basic Media File (which we will later refer to as a media file, animation file, or media presentation) can appear as a single file containing the entire presentation (potentially containing animation clips). ISO base media files can also appear as segmented files, each segmented file being an ISO base media file containing one or more segments. A segment is part of an animation file, consisting of a MovieBox and its associated media data (if present) and other associated boxes, or one or more MovieFragmentBoxes and their associated media data and other associated boxes. ISO base media files are constructed as “boxes.” In the file format, the overall presentation is called animation. It is logically divided into tracks; each track represents a timing sequence of media (e.g., video frames). Within each track, each timing unit is called a sample. Each track has one or more sample descriptions; each sample within a track is bound to the description by reference. All structured data or metadata, including those defining the placement and timing of the media, is contained within the structured boxes. Media data (e.g., video frames) is referenced by this structured data or metadata. The total duration of each track is defined in the structured data. Each sample has a defined duration. The exact decoding timestamp of a sample is defined by summing the durations of the previous samples.

[0003] The V-SEI (Multipurpose SEI) specification defines the syntax and semantics of Video Availability Information (VUI) parameters and Supplemental Enhancement Information (SEI) messages. The defined VUI parameters and SEI messages are designed to be communicated within the encoded video bitstream in the manner specified in the video coding specification, or by other means determined by the specification of the system utilizing such an encoded video bitstream. This can be applied to encoded video bitstreams (such as VVC (ISO / IEC 23090-3)) (although the intent is sufficiently general to allow for use with other types of encoded video bitstreams (e.g., AVC, HEVC, etc.)).

[0004] VUI parameters and SEI messages can assist in processing related to decoding, display, or other purposes. However, unless otherwise specified in the reference specification, the interpretation and use of VUI parameters and SEI messages specified in this document are not essential functions of the video decoder or receiving video system. Although semantics are specified for VUI parameters and SEI messages, decoders and receiving video systems may simply ignore the content of VUI parameters and SEI messages, or may use the content of VUI parameters and SEI messages in a manner slightly different from that specified in the specification. However, there are means in the file format to indicate to the application or media player that certain SEI messages are considered important from the application's perspective (this can be parameterized by the content creator and indicated during encapsulation) and should be considered by the player to improve the user experience when rendering media presentation.

[0005] In addition to the main specification, revisions to V-SEI are also underway within the JVET group to define additional SEI messages, particularly the following two: NNPFA for neural network post-filter activation and NNPFC for neural network post-filter characteristics. The Neural Network Post-Filter Characteristics (NNPFC) SEI message specifies the neural network that can be used as a post-processing filter (or, as an abbreviation, post-filter). The Neural Network Post-Filter Activation SEI message indicates the use of a specified post-processing filter for a particular image. The Neural Network Post-Filter Activation (NNPFA) SEI message activates or deactivates the possible use of the target neural network post-processing filter, identified by nnpfa_target_id, for post-processing filtering of a set of images. It should be noted that, for example, several NNPFA SEI messages may exist for the same image when the post-processing filter is intended for different purposes or to filter different color components.

[0006] Typically in ISOBMFF, depending on the sample entry type, SEI messages are stored either within the sample entry (in the metadata section) or within the sample (in the media data section). There is a revision starting with a NAL unit-based file format (ISO / IEC 14496-15) that recommends storing NNPFC SEI messages as sample groups, i.e., within the metadata section of the media file. This breaks with the usual design and can introduce a large amount of data into the file's metadata section. This is due to the fact that NNPFC SEI messages can embed a payload that provides a compressed representation of a neural network model for post-filtering operations. This payload can be encoded according to the MPEG-7 Part 17 specification (ISO / IEC 15938-17) and can represent megabytes of data. Placing such a payload in the metadata section can cause a startup delay when starting playback of the media file or while searching for the file, introduced by the time it takes to download the initialization segments or metadata associated with the synchronization sample or stream access point. Furthermore, the current revision to Part 15 proposes simply copying the SEINAL unit into the sample group description entry. Embedding the payload in the sample group description does not provide the reader with any information (such as the purpose of the filters or whether the payload in the file needs to be obtained from an external file). Furthermore, having NNPFA or NNPFC SEI payloads in the sample group description entry does not allow the application to identify active filters and skip filters that are useless to the application (e.g., based on the purpose of the filters or based on complexity). This would require the parser to support decoding SEI messages and would not be supported by general media players. Summary of the Invention

[0007] The present invention has been designed to address one or more of the aforementioned problems. The present invention allows post-filter information to be declared in the media track, or in some embodiments in an external entity of the media track, in a manner that allows the parser to decide whether to use or skip post-filter information and does not introduce delay when starting playback of a media file containing the media track or searching for that media file.

[0008] To achieve this, when post-filter information is represented as an SEI message for a video track, the neural network-based SEI message is split into two parts: a descriptive part and a payload part. The descriptive part is stored in the media track description or in a dedicated entity (e.g., another track, item, or entity group) as part of the media file's metadata. The payload (if present) is stored (or carried) in the data portion of the media track, in the media data box, or in the data portion of an external entity associated with the media track. This should keep the size used to describe the post-filter information acceptable.

[0009] According to a first aspect of the present invention, a method is provided for encapsulating media data into an ISOBMFF-based media file, the media data including media data units and at least one post-filter data unit, the method comprising:

[0010] A first track is generated to form a sequence of encapsulated samples, each sample including at least one of the media data units, and the at least one post-filter data unit is applied to at least one of the samples in the first track;

[0011] Generate a data portion of an entity that includes at least a portion of one of the post-filter data units;

[0012] Generate post-filter information describing the at least one post-filter data unit; and

[0013] The first track, the entity, and the post-filter information are encapsulated in the media file, and the post-filter information is encapsulated in the metadata portion of the media file.

[0014] In this embodiment, the entity is the first track.

[0015] In this embodiment, an entity is an item associated with the first track.

[0016] In this embodiment, the entity is the second track associated with the first track.

[0017] In an embodiment, the post-filter information includes general information indicating the presence of post-filter information.

[0018] In this embodiment, the general information indicating the presence of post-filter information is a proprietary brand.

[0019] In an embodiment, the post-filter information also includes information indicating the purpose of the post-filter information.

[0020] In an embodiment, the post-filter information also includes information indicating the complexity of the post-filter information.

[0021] In one embodiment, post-filter information is provided in a sub-box of the first track metadata.

[0022] In this embodiment, post-filter information is provided in the sample entries of the first track metadata.

[0023] In one embodiment, post-filter information is provided in a sample group of the first track metadata.

[0024] In this embodiment, post-filter information is provided in the metadata section of the item.

[0025] In this embodiment, post-filter information is provided in the metadata section of the second track.

[0026] According to another aspect of the present invention, a method for generating a bitstream of media data from an ISOBMFF-based media file is provided, the method comprising:

[0027] The first track of the sequence of encapsulated samples is obtained from the media file, each sample including at least one media data unit;

[0028] Post-filter information describing at least one post-filter data unit is obtained from the metadata portion of the media file, the at least one post-filter data unit being applied to at least one of the samples of the first track;

[0029] At least a portion of one of the post-filter data units is obtained from the data portion of the entity in the media file; and

[0030] The bitstream of the media data is generated based on the first track, the entity, and the post-filter information in the media file.

[0031] According to another aspect of the invention, a computer program product for a programmable device is provided, the computer program product comprising a sequence of instructions for implementing the method according to the invention when loaded into and executed by the programmable device.

[0032] According to another aspect of the invention, a computer-readable storage medium is provided for storing instructions for implementing a computer program according to the method of the invention.

[0033] According to another aspect of the invention, a computer program is provided that causes the method of the invention to be performed when executed.

[0034] According to another aspect of the invention, an apparatus is provided for encapsulating media data into an ISOBMFF-based media file, the media data including media data units and at least one post-filter data unit, the apparatus including a processor configured to:

[0035] A first track is generated to form a sequence of encapsulated samples, each sample including at least one of the media data units, and the at least one post-filter data unit is applied to at least one of the samples in the first track;

[0036] Generate a data portion of an entity that includes at least a portion of one of the post-filter data units;

[0037] Generate post-filter information describing the at least one post-filter data unit; and

[0038] The first track, the entity, and the post-filter information are encapsulated in the media file, and the post-filter information is encapsulated in the metadata portion of the media file.

[0039] According to another aspect of the invention, an apparatus is provided for generating a bitstream of media data from an ISOBMFF-based media file, the apparatus comprising a processor configured to:

[0040] The first track of the sequence of encapsulated samples is obtained from the media file, each sample including at least one media data unit;

[0041] Post-filter information describing at least one post-filter data unit is obtained from the metadata portion of the media file, the at least one post-filter data unit being applied to at least one of the samples of the first track;

[0042] At least a portion of one of the post-filter data units is obtained from the data portion of the entity in the media file; and

[0043] The bitstream of the media data is generated based on the first track, the entity, and the post-filter information in the media file.

[0044] At least a portion of the method according to the invention can be implemented by a computer. Therefore, the invention can take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which are generally referred to herein as “circuit,” “module,” or “system.” Furthermore, the invention can take the form of a computer program product embodied in any tangible medium having computer-usable program code embodied therein. The invention can be embedded in cameras, smartphones, devices for recording, packaging, or transmitting video. The parsing portion can be embedded in smartphones, televisions, head-mounted displays, etc.

[0045] Because this invention can be implemented in software, it can be embodied as computer-readable code for provision to a programmable device on any suitable carrier medium. Tangible non-transient carrier media may include storage media such as floppy disks, CD-ROMs, hard disk drives, magnetic tape devices, or solid-state storage devices. Transient carrier media may include signals such as electrical signals, electronic signals, optical signals, acoustic signals, magnetic signals, or electromagnetic signals (e.g., microwave or RF signals). Attached Figure Description

[0046] Embodiments of the invention will now be described by way of example only and with reference to the following figures, in which:

[0047] Figure 1a , Figure 1b and Figure 1cAn overview of methods for encapsulating or storing multimedia presentations, exemplified by some embodiments of the present invention;

[0048] Figure 2 The main steps of the packaging process according to an embodiment of the present invention are illustrated;

[0049] Figure 3 This illustrates the main steps of the parsing or reading process according to an embodiment of the present invention;

[0050] Figure 4a The example metadata structure that is available for the filter information following the instruction, for a track in a media file;

[0051] Figure 4b The following embodiment illustrates an example in which, in order to encapsulate filter data units (FDUs) and media data units (MDUs), the media file includes a metadata portion that includes boxes that can be part of a general description or a V-SEI-specific mapping.

[0052] Figure 4c The following embodiment illustrates an example in which, in order to encapsulate filter data units (FDUs) and media data units (MDUs), a media file includes a metadata portion that includes boxes that may include a portion of the filter data units (e.g., syntactic elements describing post-filter information).

[0053] Figure 5 An example is given where the filter information is stored outside the media track to which it is applied;

[0054] Figure 6 This example illustrates an implementation where the filter information is encapsulated in a dedicated track.

[0055] Figure 7 This is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention;

[0056] Figure 8 Another embodiment of the present invention is illustrated. Detailed Implementation

[0057] Figure 1a , Figure 1b and Figure 1cAn overview of methods for encapsulating or storing multimedia presentations according to some embodiments of the present invention is provided. The media data to be encapsulated or stored, denoted as 105, can be video, a spatial sub-part of video, audio, volumetric visual media, subtitles, or a combination of such basic streams. They are encapsulated into a media file or media segment file 125 in a manner that allows adaptive streaming over HTTP (potentially with low latency). As shown, server 100 includes an encapsulation module 120 (also referred to as an ISOBMFF writer or simply a writer) and an optional streaming manifest generation module (not shown). Server 100 can be connected to a communication network 130 via a network interface not shown, and communication network 130 is also connected to a client 150 via a network interface not shown. Examples of clients are media players on PCs, tablets, TVs, smartphones, or virtual reality headsets.

[0058] Server 100 processes media data 105 in preparation for streaming or storage; this is called encapsulation. It primarily involves adding metadata describing the media data, such as data type, codecs used, size, data offset, and timing. As mentioned above, media data 105 can correspond to either audio or video presentation and may include subtitles or timed text. Media data 105 can be raw data captured by sensors or generated by content creators or editing tools. Media data 105 may be available as compressed or encoded media data 115 depending on the different encoding versions. This means that encoding or compression can be performed by the server itself using encoder modules such as encoder module 110 (possibly one encoder module per media type), or remotely relative to the server. Compression can be performed in real-time encoding (and encapsulation).

[0059] Media data 105 may also be encapsulated media data, which is encapsulated in a manner unsuitable for low-latency streaming (e.g., as a non-fragmented track). In the latter case, according to some embodiments of the invention, the encapsulation module (e.g., encapsulation module 120) further includes a reader portion for decapsulating the media data and then encapsulating the decapsulated media data.

[0060] According to the example shown, the encapsulation module 120 of server 100 is used to encapsulate media data into animation segments, for example, according to ISOBMFF and its extensions (e.g., CMAF, NAL-based file formats, etc.). Server 100 then generates a media file or one or more segment files 125, labeled 125. Server 100 may optionally generate streaming playlists, such as DASH MPD or HLS playlists (not shown). The generated file, segment file, or playlist may be stored on a remote storage device (e.g., a remote storage device accessible via network 140) for redistribution via on-demand or real-time streaming.

[0061] According to some embodiments of the present invention, the encapsulation module 120 generates encapsulated files (or segments) that allow low-latency and adaptive streaming over HTTP.

[0062] Client 150 is used to process data received from communication network 130 or data read from (local or remote) storage devices, such as media files or media segments 125. Data can be streamed to the client, thus involving a streaming module (not shown) responsible for parsing the streaming manifest, determining the request to obtain the media stream, and adapting the transmission according to the instructions in the manifest and client parameters (such as available bandwidth, CPU, application requirements, or user preferences).

[0063] The received data is decapsulated in the decapsulation module 160 (also known as an ISOBMFF parser, ISOBMFF reader, or simply a parser or reader). The decapsulated data (or parsed data) can be decoded by the decoding module for storage and display or for output to an application or (one or more) user. The decoder module (possibly one or more decoder modules per media type) can be part of the client, an external module, or dedicated hardware. The decapsulated data can correspond to the encoded media data 165 (e.g., video bitstream, audio bitstream, etc.).

[0064] Decapsulation, decoding, and rendering can be performed, for example, by processing data chunks of individual media streams in parallel and synchronously, and by performing real-time manipulation of the media file as soon as it is received, so as to minimize the latency between the recorded multimedia presentation (such as media data 105) and its visualization by (one or more) users as media data 175, for example, on the screen of his / her device.

[0065] The client or server can be a user device, or it can be a network node operating on media files being transmitted or stored. The server or client may consist of only encapsulation and decapsulation components, respectively.

[0066] It should be noted that media file 125 can be communicated to the client or reader 150 in different ways. Specifically, server or writer (or packer) 100 can generate media file 125 with a media description (e.g., DASH MPD) and communicate (or stream) media file 125 directly to client 150 upon receiving a request from client 150. Media file 125 can also be downloaded by client 150 once or incrementally and stored on client 150.

[0067] For illustrative purposes, media file 125 may encapsulate media data into boxes according to, for example, ISO Basic Media File Formats (ISOBMFF, ISO / IEC 14496-12) and their derived specifications (e.g., NAL unit structured video carrying in ISOBMFF, ISO / IEC 14496-15, or Common Media Application Format (CMAF) ISO / IEC 23001-19). In this case, media file 125 may correspond to one or more media files (indicated in the FileTypeBox ('ftyp') or SegmentTypeBox ('styp') box). According to ISOBMFF, media file 125 may include two types of boxes: one or more "media data boxes" (e.g., 'mdat', 'idat', or 'imda' boxes) containing media data; and "metadata boxes" (e.g., 'moov' or 'moof' boxes) containing metadata that defines the placement and timing of the media data and, more generally, provides descriptive information about the media data. Media files or segmented files contain animation clips. One or more media data boxes contain all the data of media data 105 or encoded media data 115. A single media data box may exist that reuses media data, but there may also be one or more media data boxes, for example, one media data box per media type, and possibly one or more media data boxes per animation clip. In system configuration, especially... Figure 1b or Figure 1c The depicted package side may have variations. Figure 1b The encapsulation module can encapsulate the post-filtering information 105-2 within the media file 125' after the initial media data has been encapsulated. This post-filtering information 105-2 can be calculated, for example, after the media data 105-1 has been encoded or compressed by the encoder module 110. Figure 1cOn the above, there may be several encapsulation modules that encapsulate media data 105-1 and post-filter information 105-2 in a media file 125''. Still for this example, the server or writer for media data 105-1 and the server or writer for post-filter information 105-2 may be different entities (e.g., software modules on different devices), each entity encapsulating the media data and post-filter information into media file 125'' respectively. Note that server 100-2 may also include an encoder or compression module (not depicted), such as an MPEG-7 Part 17 encoder, to produce an efficient representation of the neural network model for post-filtering. Media file 125' may also include a concatenation of media files generated by writer 100-1 on one hand and media files generated by writer 100-2 on the other hand. According to some embodiments, as indicated by reference numeral 180, the two writers may share common setup information, such as knowledge of information written into the 'moov' box, such as the initial track or sample configuration.

[0068] Figure 2 The main steps of the packaging process according to an embodiment of the present invention are illustrated. This is achieved through... Figure 1a The packaging module 120 or Figure 1c This is accomplished by encapsulation modules 120-1 and 120-2. The process begins in step 200 via configuration server 100 and optionally server 100-1. Configuration may involve encapsulation modules 120, 120', or 120" and encoder module 110 or (one or more) encapsulation modules only when encoder settings cannot be controlled, such as when configuration is hard-coded. For illustration, the configuration step may include indicating whether media files will be included in a single file or multiple segmented files. Configuration may include defining the granularity of random access in rendering (e.g., every second or every 2 seconds…), and for streaming applications, may include switching points and addressable objects in rendering (e.g., DASH media segments, CMAF segments, CMAF fragments, or CMAF blocks). Configuration may include indicating whether post-filter information is stored within a media track or in another entity (e.g., an item, track, group of tracks, or group of entities).

[0069] At step 201, the encapsulation module receives compressed or encoded media data, a media bitstream. Then, at step 202, the encapsulation module checks whether the bitstream contains or embeds post-filter information. This may include finding the SEI NAL cell type in the bitstream and parsing the payload type of the detected SEI NAL cells. If the payload type corresponds to a post-filter SEI message, the encapsulation module proceeds to step 204. Otherwise, encapsulation includes encapsulating the media data in a conventional manner at step 211. As an alternative to the SEI message in the bitstream, as explained in step 203, post-filter information can be provided by other means (e.g., as a separate bitstream or as configuration information…) (as described in 105-2). At step 204, based on the SEI or based on external post-filter information, the encapsulation module determines a filter description. The description may include, for example, the filter type (e.g., based on a neural network or a simple coefficient matrix or any other representation), the filter purpose (e.g., increasing image resolution, increasing temporal resolution, improving signal quality…). Next, step 205 includes checking whether the filter description contains a payload. For example, the NNPFC SEI message can be embedded in an ISO / IEC 15938-17 bitstream (e.g., when the nnpfc_mode_idc parameter is equal to 0). This determination can depend on the representation format of the post-filter information, or it can be provided in the post-filter information received in 203. If a payload exists (test 205 is true), the encapsulation module stores this information in the filter description in 206. Otherwise, in step 211, the encapsulation module stores its payload outside the file in the filter description. Following step 206 is step 207, during which the encapsulation module stores the filter payload in the data portion of the media file. (For example, ...) Figure 5 The data items above, such as Figure 4a , Figure 4b and Figure 4c or Figure 6 (The track samples, sample entries, sample groups, etc.) Different strategies for storing this payload will be described later. Step 208 includes generating a filter description and storing it in the metadata section of the media file. It should be noted that steps 206 and 207 can be performed in any order. Figure 1c In some cases, these can even be handled by different encapsulation modules. Then, media samples are encapsulated in 209, and optionally, in 210, filter information can be associated with the samples. By default, post-filtering information can be associated at the track level rather than the sample level. These operations are iterated over the samples present in the media bitstream, and finally, the media file is published for later use (storage, streaming, playback, sending, etc.).

[0070] The media files published according to this invention differ from classic ISO-based media files in that they can contain additional metadata indicating that media tracks can be further post-filtered to, for example, improve their rendering. Without this invention, media tracks can be considered, for example, video tracks or audio tracks, and applications may be unaware of the existence of post-filter instructions that may provide improved rendering of the media tracks, or conversely allow applications to decide to skip post-filtering by ultimately discarding the post-filter information when reconstructing the media bitstream.

[0071] Although described for a single track, encapsulation can be applied to multiple bitstreams, each with its own post-filter information, or some bitstreams sharing common post-filter information. Encapsulation can also be applied to bitstreams containing multiple post-filter information.

[0072] Figure 3 The main steps of the parsing or reading process according to an embodiment of the present invention are illustrated. The client may also be referred to as an "application" or a "media player" (or renderer). The process begins in step 300 with player initialization. The player receives a media file or a media segment file, possibly with a manifest file, generated according to an embodiment of the present invention. The player is initialized with information obtained by requesting or reading the initial segment or start (e.g., the first byte) of the media file. When the media file is stored on a remote server, the URL or byte range for the initialization data can be obtained from a manifest file such as a DASH MPD or a playlist for HTTP live streaming. Depending on the media file, whether or not it contains an indication for post-filtering (test 301), the player can know the type of media present in the media presentation, the codecs used, etc., and select the version that it can decode and render. Step 301 includes obtaining a filter description. The filter description can be obtained from a dedicated box in the track or sample description (as will be explained later). Figure 4a , Figure 4b and Figures 4c to 6(Further description above). When a media file contains post-filter information, the parser needs to check whether the post-filter has been indicated as mandatory by the content creator when packaging the media for rendering, or whether it has been set to be optional for the rendered media track (this could also be a configuration set by the wrapper module from the application server). This places some demands on the player: if it cannot recognize or support the post-filter, it can not select a track and indicate to the application that it does not support this media format. Conversely, if the player can support and process the post-filter information, the player can select a track and indicate to the application that it supports the media type for that track. When a track is selected, the parser begins processing the sample at 302. In 303, the parser checks whether the current sample is associated with post-filter information. This can be determined from the metadata structure of the media file, which can be included in the track description, sample description, or dedicated metadata structure. When the post-filter information is not stored within a media track, it may also be included in another entity. In the case where no post-filter information is associated with a sample (test 303 is false), the parser extracts the data corresponding to the sample and appends it to the reconstructed bitstream in step 308. When no more samples need processing (test 309 is false), the reconstructed bitstream is provided to the entity (typically the media decoder) that will process it in 310. When a sample is associated with post-filter information (test 303 is true), the parser can append a filter description to the reconstructed bitstream of the track in 304. A filter configuration can be added in 304 if the filter configuration defines a new configuration, if the sample is a synchronous or random access sample (e.g., allowing decoding to begin from that sample), or if different content is spliced ​​into the first initial content. The parser then checks if the post-filter information contains a payload in the media file (test 305). If not, data for the sample, which may precede the filter description, is appended to the bitstream in 308. In the case of a video bitstream with dedicated SEI messages for post-filter information, this can be done by generating appropriate SEI NAL units and inserting these SEI NAL units into the video bitstream. If a payload exists, the parser obtains it in step 306 (e.g., from a media data box such as 'mdat') and can append the payload to the bitstream in 307. Preferably, when present, the payload is appended after the filter description. Steps 302 through 308 are iterated until no more samples can be read from the track being parsed.

[0073] Although described for a single track, the parsing process can be applied to multiple tracks in a media file, each with its own post-filter information or some tracks sharing common post-filter information. It can also be applied to situations where a track contains multiple post-filter information.

[0074] General indication of post-filter information

[0075] Post-filter information may exist in different encapsulation modes: within the media track to which it is applied, or in an external entity (such as a dedicated track or item, or a group of tracks or entities). Regardless of the encapsulation, there may be a generic description indicated in the media file to inform the player (or reader or parser) that one or more media tracks appear with the post-filter information. This generic information may be generated during encapsulation, for example, based on the SEI content in the case of a video bitstream, or based on external post-filter information 105-2. These indications allow the media player to detect the presence of post-filter information that may have a purpose or complexity and can help the player decide whether to process or skip post-filtering.

[0076] Through brand-specific general instructions

[0077] Defining a specific brand to use in the 'ftyp' or 'styp' box allows the parser to be notified of the presence of additional requirements related to post-filtering to support the media file. This specific brand value can also be defined in the TrackTypeBox 'ttyp' to impose additional requirements on the post-processing of a specific track. Within the scope of this invention, for example, a brand referred to as "Post-Filter Information Present" and represented by the four-character code 'pfip' will inform the parser that some media tracks in the media file (or segment file) may contain or be associated with post-filter information, and that there may be alternative reconstructions for that media track: applying or not applying the post-filter information. This brand can be declared in a list of compatible brands. The brand can be combined with a metadata structure used to indicate the post-filter information associated with the media track. This tag can also inform the parser that additional decoders may be needed for a decoder corresponding to the media type of the track to decode the post-filter information for that track.

[0078] Hereinafter, different embodiments of the invention are described. In all these embodiments, post-filter information is stored in the metadata portion of the media file, while the payload (when provided), such as an NNPFC or NNPFA SEI message describing the model, is stored in the data portion of the media file. According to a first embodiment, post-filter information is stored in the metadata portion of the media track to which the post-filter information is applied. According to a first variation of the first embodiment, post-filter information is provided in a static track sub-box or in a sample entry. According to a second variation of the first embodiment, post-filter information in a static track sub-box or in a sample entry is completed with codec-specific information. According to a third variation, post-filter information is provided in a sample group box describing the sample group to which the post-filter information is applied. In a second embodiment, post-filter information is provided as a metadata item associated with the media track to which the post-filter information is applied. In a third embodiment, post-filter information is provided in a metadata track associated with the media track to which the post-filter information is applied.

[0079] Generic post-filter information in the metadata of the media track

[0080] According to a first variation of the first embodiment, general information regarding post-filter information may first indicate that the track contains or is associated with post-filter information. For example, we call this indication `post_filter_presence`. It can be a binary value. Then, a second field or parameter may indicate whether the filter data or filter payload is provided in a media file or an external file. For example, we call this indication `post_filter_without_payload` or `post_filter_external_payload`. Optionally, additional parameters may provide the type of payload, such as a four-character code or Uniform Resource Identifier (URI) or Uniform Resource Name (URN) in a representation format of post-filter data that is explicitly identified (e.g., recorded in a registry). Optionally, further description of the payload may be indicated, such as its size in bytes. Depending on this size, considering latency aspects (e.g., fast startup or lookup), the parser may decide whether to retrieve the payload if it is external or use it if it is internal to a file. When applied to NNPFC SEI messages as defined in the Multipurpose SEI specification, this may correspond to the `nnpfc_tag_uri` parameter. This parameter can be set even if the payload is not in the media file. Another parameter for post-filter information can be the purpose of the filter. For example, it can improve the quality of the media track, increase the resolution of the frame sequence in the video track, or increase the frame rate. For example, we call this parameter `post_filter_purpose`. There can be another parameter indicating whether the post-filter information comes from the same bitstream as the bitstream representing the media data. For example, we call this parameter `post_filter_same_origin` or `post_filter_same_bitstream`. Another possible parameter can be an indication of whether the filter description changes over time (i.e., whether it remains the same along the track duration) (or in other words, whether it is dynamic (can be updated) or static). For example, we call this parameter `post_filter_update`. Yet another parameter can be defined in the general description of the post-filter information: `post_filter_identifier`. This parameter can be useful when the media track is associated with several filters, either along time or at different parts of the sample. This identifier allows a sample to be associated with one or more filters or samples when needed. For example, in a video track, a first filter can be applied to the luminance component of a decoded sample, and a second filter can be applied to the chroma component of that decoded sample. Another parameter could be an indication of the intended use, such as improving the user's rendering? Or improving the machine processing of the media track? Both? Predefined values ​​can be used to indicate the intended use of the array of filtered output samples produced by the post-processing filters.Another parameter can provide a description of the requirements in terms of the complexity of the post-filter, such as memory requirements, the latency it may introduce in terms of samples, display latency, the number of operations to be performed, etc.

[0081] These parameters can be grouped into boxes specifically for post-filter description. For example:

[0082] The post-filter information box provides a description of the post-filter information associated with the track. This box is defined by the four-character code 'pfif' (e.g., 'post-filter information') used for "post-filter information". Figure 4a The 414) identifier. This box can be included in the sample entry as an optional box. When it is not present, it means that the sample referencing this sample entry does not have any associated post-filter information. Alternatively, when the encapsulation module can determine that the box is static information, the box can be included in the track description, such as Figure 4a The 'trak' box 410 is a sub-box. When it is absent, it means that no sample of that track has associated post-filter information.

[0083] Examples of box definitions can be summarized as follows:

[0084] Box type: 'pfif'

[0085] Container: Sample Entries, TrackBox, or MetaBox

[0086] Number: zero or more than zero

[0087] The PostFilterInformationBox contains a description of the post-filter information that can be applied to the samples of the track. It should be noted that a subset of the possible parameters listed above is illustrated in the examples below, and any of these parameters can be included in the metadata structure presented below.

[0088] The syntax of a box can be represented as follows (the parameter names and the four-character code 'pfif' are just examples):

[0089] syntax

[0090] aligned(8) class PostFilterInformationBox extends Box('pfif')

[0091] {

[0092] unsigned int(1) post_filter_presence;

[0093] unsigned int(7) reserved_bits;

[0094] if (post_filter_presence) {

[0095] unsigned int(1) post_filter_without_payload;

[0096] unsigned int(1) post_filter_same_bitstream;

[0097] unsigned int(1) post_filter_update;

[0098] unsigned int(5) post_filter_purpose;

[0099] utf8string post_filter_identifier;

[0100] }

[0101] }

[0102] In variations where the presence or absence of this box can be used to determine whether some samples in the track may have associated post-filter information or not, the parameter of `post_filter_presence` can be omitted, thus simplifying the syntax as follows:

[0103] aligned(8) class PostFilterInformationBox extends Box('pfif')

[0104] {

[0105] unsigned int(1) post_filter_without_payload;

[0106] unsigned int(1) post_filter_same_bitstream;

[0107] unsigned int(1) post_filter_update;

[0108] unsigned int(5) post_filter_purpose;

[0109] utf8string post_filter_identifier;

[0110] }

[0111] Although the above syntactic examples are defined as inheriting from Box, they can also inherit from FullBox, thus providing additional versioning and flag parameters for possible extensions or parameterization. It should be noted that if a long predefined list exists registered in some registry (e.g., the mp4 registry) or standard specifications for post-filter descriptions (e.g., V-SEI), more bits can be allocated to fields indicating the purpose of the post-filter.

[0112] In variations handling both static and dynamic configurations with the same metadata structure, the post-filter information is indicated as follows: Figure 4a The entries in the sample group description box ('sgpd') shown in box 415 above. For example, for post-filter information applied to a video track, the sample group description entries that may be contained in a sample group description box with grouping_type='pfif' can be defined as follows, in which case it is not necessary to specify parameters indicating static or dynamic post-filter information. The SampleToGroupBox('sbgp') box 416 with the same grouping_type='pfif' can be used to map samples to different entries in the 'sgpd' box:

[0113] class PostFilterSampleGroupEntry extends VisualSampleGroupEntry ('pfif')

[0114] {

[0115] unsigned int(1) post_filter_without_payload;

[0116] unsigned int(1) post_filter_same_bitstream;

[0117] unsigned int(5) post_filter_purpose;

[0118] unsigned int(1) reserved;

[0119] utf8string post_filter_identifier;

[0120] }

[0121] The semantics of the parameters that describe the filter information in a general way are as follows:

[0122] When `post_filter_presence` equals 1, it indicates that some samples referencing a sample entry containing the box have associated post-filter information. When used in a sample group description entry, this parameter applies to samples mapped to that entry. When equal to 0, this parameter indicates that samples referencing a sample entry containing the box, or samples mapped to a sample group description entry, do not contain post-filter information. A default value of 1 can be assumed.

[0123] When `post_filter_without_payload` equals 1, it indicates that no payload describing the post-filter data is available for some samples that reference the sample entry containing the box, or for some samples mapped to the description of the sample group. When it equals 0, this parameter indicates that a payload describing the post-filter information is available in the data portion of the media file for some samples that reference the sample entry containing the box, or for some samples mapped to the description entry of the sample group. A default value of 1 can be assumed. When applied to a video track with an NNPFC SEI message, the value of this parameter can correspond to the value of the `nnpfc_mode_idc` parameter.

[0124] When `post_filter_same_bitstream` equals 1, it indicates that for some samples referencing sample entries containing that box, or for some samples mapped to that sample group description entry, a post-filter description exists, along with its payload (depending on the value of `post_filter_without_payload`) as a data unit for the media track (e.g., a NAL unit for an MPEG video codec). In other words, this parameter indicates that if the parser or reader determines to render a post-filtered version of the media presentation, the parser or reader will include post-filter information in the reconstructed bitstream of those samples. When equal to 0, this parameter indicates that no post-filter description is included in the data units of samples referencing sample entries containing that box or for some samples mapped to that sample group description entry. A default value of 1 can be assumed. For example, in... Figure 4a In this context, media data mixes data units related to post-filter information (e.g., SEI NAL units) from 421-1 and 423-1 with data units specific to the media sample (e.g., VCL NAL units 421-2, 421-3, or 423-2, etc.) within a given sample (e.g., 421 or 423). Some samples, such as 422, may not contain any data units corresponding to post-filter data or payloads.

[0125] `post_filter_purpose` is a value indicating the purpose of the post-filter applied to samples that reference sample entries containing the box or to samples mapped to description entries of the sample group. When applied to a video track with an NNPFC SEI message, the value of this parameter can correspond to the `nnpfc_purpose` parameter. The predefined list of values ​​to be used in this parameter can be media-specific, such as a list of values ​​for each processor type (video, audio, text, volume, etc.).

[0126] `post_filter_identifier` is the identifier for the post-filter. When applied to a video track with an NNPFC SEI message, the value of this parameter can correspond to the `nnpfc_id` parameter. Although represented as a string in the above structure, this parameter can also be represented by an integer value, such as a 32-bit value, or a universally unique identifier (UUID) on a 128-bit scale. This parameter can identify a single filter or filter bank to be applied to a sample or group of samples, or to an item or group of items.

[0127] In this embodiment, the sample group method is preferred for the metadata structure that describes post-filter information within the media track because it allows for handling the default, static, or dynamic configuration of the post-filter information by setting appropriate values ​​in the flags or versions of the sample group descriptor box ('sgpd') with a grouping type (e.g., 'pfif' grouping_type) for the post-filter indication.

[0128] Post-filter information, which may contain codec-specific information, is found in the metadata of the media track.

[0129] According to a second variation of the first embodiment, the metadata structure defined in the previous embodiment for embedding post-filter information in the media file can be enriched by additional parameters, thereby allowing the post-filter information to be considered according to a specific representation format (such as NNPFC SEI messages of the Multipurpose SEI specification). The first variation may include allocating a predefined number of bits to convey codec-specific parameters, as follows:

[0130] aligned(8) class PostFilterInformationBox extends Box('pfif')

[0131] {

[0132] unsigned int(1) post_filter_without_payload;

[0133] unsigned int(1) post_filter_same_bitstream;

[0134] unsigned int(1) post_filter_update;

[0135] unsigned int(5) post_filter_purpose;

[0136] unsigned int(32) post_filter_codec_specific;

[0137] }

[0138] Codec-specific parameters can be controlled by flags indicating their presence (not shown in the syntax above) to save 32 bits when no codec-specific information is available. Another variation would include an indication of the size of the codec-specific information, which is then indicated as a sequence of bytes.

[0139] aligned(8) class PostFilterInformationBox extends Box('pfif')

[0140] {

[0141] unsigned int(1) post_filter_without_payload;

[0142] unsigned int(1) post_filter_same_bitstream;

[0143] unsigned int(1) post_filter_update;

[0144] unsigned int(5) post_filter_purpose;

[0145] unsigned int(16) post_filter_codec_specific_length;

[0146] int size = 0;

[0147] while (size <post_filter_codec_specific_length){

[0148] unsigned int(8) post_filter_codec_byte;

[0149] size++;

[0150] }

[0151] }

[0152] These two variations can be applied to the 'pfif' box or to the sample group description entry 'pfif'.

[0153] The semantics corresponding to the post_filter_codec_specific parameter or the post_filter_codec_byte parameter are defined as follows:

[0154] This parameter provides additional information related to codec-specific post-filtering (i.e., additional specifications that a generic ISOBMFF parser might not understand and needs to support). For example, for a video track embedded SEI message (one or more) targeting neural network-based post-filtering (e.g., NNPFC or NNPFA), the encapsulation module can copy some elements of the syntax from the NNPFC SEI message in this codec-specific parameter. It should be noted that even if post_filter_without_payload equals 0, elements only from the syntax can be copied in post_filter_codec_specific or post_filter_codec_byte, but not the data corresponding to the payload. This is to avoid having a significant description size for the metadata structure. This means that the SEI message providing post-filter information can be split into two parts: a description part and a payload part, and only the description part can be stored in the codec-specific part indicating the metadata structure of the post-filter information. In this case, the following about the data part of the media file... Figure 4a The described data units 421-1 and 423-1 may correspond only to the post-filter payload (and no longer to the complete SEI data unit). This also means that when the post_filter_same_bitstream parameter is equal to 1, the parser can use a portion of the SEI syntax from the metadata structure indicating the post-filter information, and further, when post_filter_without_payload is equal to 0, it can use a portion of the data corresponding to the payload of the SEI message to reconstruct the bitstream.

[0155] The post_filter_codec_specific_length indicates the number of bytes in the metadata structure that indicate codec-specific parameters for post-filter information.

[0156] The post-filter SEI message is mapped in the metadata of the media track.

[0157] According to a third variation of the first embodiment, a metadata structure is created by the encapsulation module to indicate information in the media file about post-filter information to be used with the media data. These indications allow the media player to detect the presence of post-filter information that may have a purpose or complexity and can help the player decide whether to process or skip post-filtering. Some indications can constrain the player to process media tracks with post-filter instructions, thus affecting track selection on the player side. For example, in the case of a VVC encoded video bitstream, NNPFC and NNPFA SEI messages can also be embedded in the VVC bitstream. Revisions to the NAL unit-based file format are considering using sample groups to convey NNPFC and NNPFA messages. However, their current definitions still embed the NNPFC payload (when stored) in the SEI message. Furthermore, the proposed sample groups simply copy the SEI NAL unit for the NNPFC and NNPFA SEI messages in the sample group description entry. This does not provide any indication to a general parser about post-filtering (purpose, in the file or externally, etc.) and requires the parser to implement the parsing of these NNPFC and NNPFA SEI messages. The mappings for NNPFC and NNPFA differ here because a high-level and general description of the post-filter is provided (without needing to parse the SEI message), and it is proposed to keep the payload portion within the data portion of the media file. This avoids having too much data in the metadata portion of the media file by keeping the SEI within the data. Post-filter information can be provided, for example, as a sample group of description entries containing a description of the post-filter rather than (one or more) NNPFC or NNPFA SEI NAL units.

[0158] The current design of the neural network post-filter activation sample set is limited when several updates are available. The current activation SEI allows activation, deactivation, or reactivation of the update corresponding to the most recently encountered update. In practice, the activation only contains the target identifier of the NNPFC, but not an indication of the update version. The NNPFA SEI message can then be improved to include an update index in addition to the identifier of the target neural network post-processing filter, thus allowing the reactivation of a given update using that index.

[0159] Example syntax for handling different updated versions of NNPFA SEI messages could be:

[0160]

[0161] The `nnpfa_update_version` directive indicates the index of the update to the NNPFC model identified by `nnpfa_target_id`. The default value is 0 when applied to the base or when no update is available. This can improve the NNPFA SEI message by allowing any configuration of a given post-filter (base or any update), while the initial current design appears to consider the latest update (if it exists).

[0162] Sample sets (NNPFC sample set) for conveying information related to the characteristics of neural network-based post-filters and sample sets (NNPFA sample set) for conveying information related to the activation of neural network-based post-filters are proposed.

[0163] All instances of SampleToGroupBox for NNPFC sample groups should include the grouping_type_parameter. The grouping_type_parameter field is specified for NNPFC sample groups as follows:

[0164] {

[0165] unsigned int(1) filter_update_flag;

[0166] unsigned int(31) filter_id;

[0167] }

[0168] A filter_update_flag value of 1 indicates that all sample group description entries referenced by the SampleToGroupBox contain updated NNPFC SEI messages provided on top of the underlying post-processing filter. A filter_update_flag value of 0 indicates that all sample group description entries referenced by the SampleToGroupBox contain NNPFC SEI messages specifying the underlying post-processing filter.

[0169] filter_id indicates that all sample group description entries referenced by the SampleToGroupBox contain NNPFC SEI messages with nnpfc_id (the identifier of the neural network model used) equal to filter_id.

[0170] When a sample is not mapped to an NnpfcSeiSampleGroupEntry in SampleToGroupBox that has a filter_update_flag equal to 0 and a specific filter_id, the sample should not be mapped to an NnpfcSeiSampleGroupEntry in SampleToGroupBox that has a filter_update_flag equal to 1 and the same filter_id.

[0171] When a track contains an NNPFC sample group, an NNPFC SEI message should not exist within the sample entries of that track. When a VVC track has an associated VVC non-VCL track containing an NNPFC sample group, an NNPFC SEI message should not exist within the samples or sample entries of the VVC track, and an NNPFC SEI message should not exist within the sample entries of the VVC non-VVL track.

[0172] When the reader supports NNPFC sample groups, it should perform the following insertion of the prefix SEI NAL cell as part of the bitstream reconstruction:

[0173] - If a sample is mapped to at least one NnpfcSeiSampleGroupEntry with a filter_update_flag equal to 0 and the sample is:

[0174] o Synchronous samples, or

[0175] o The first sample in the sequence of samples associated with the same sample entry, or

[0176] o maps to the first sample in the same NnpfcSeiSampleGroupEntry sample group that has a filter_update_flag equal to 0 and a specific filter_id value filterIdBase.

[0177] The sample implicitly contains a prefix SEI NAL unit for each layer contained in the track and each filter_id value mapped to the sample, and the prefix SEI NAL unit contains an NNPFC SEI message for the data of the first sample mapped sample with filter_update_flag equal to 0, followed by an NNPFC SEI message (if present) for an NnpfcSeiSampleGroupEntry mapped to the sample with filter_update_flag equal to 1 and filter_id equal to filterIdBase.

[0178] - If the sample is the first sample in a sequence of samples mapped to the same NnpfcSeiSampleGroupEntry with a filter_update_flag equal to 1 and a specific filter_id value filterIdUpdate, and that sample:

[0179] o is not a synchronous sample, and

[0180] o is not the first sample in the sequence of samples associated with the same sample entry, and

[0181] o is the first sample in the sequence of samples mapped to the same NnpfcSeiSampleGroupEntry, which has a filter_update_flag equal to 0 and a filter_id equal to filterIdUpdate.

[0182] The sample implicitly contains a prefix SEI NAL unit for each layer contained in the track and each filter_id value mapped to the sample, and the prefix SEI NAL unit contains an NNPFC SEI message from an NnpfcSeiSampleGroupEntry with a filter_update_flag equal to 1.

[0183] The syntax for describing entries in the NNPFC sample group can be as follows:

[0184] aligned(8) class NnpfcSeiSampleGroupEntry() extendsVisualSampleGroupEntry('nfcs')

[0185] {

[0186] unsigned int(8) nnpfc_sei_id; / / to map authorized values ​​for nnpfc_id, could be described on 32 bits to align to NNPFC SEI

[0187] unsigned int(1) nnpfc_update; / / if not first occurrence of this id

[0188] unsigned int(1) nnpfc_external_payload;

[0189] unsigned int(1) nnpfc_purpose;

[0190] unsigned int(1) nnpfc_complexity_info_present_flag;

[0191] if (nnpfc_complexity_info_present_flag) {

[0192] unsigned int(32) nnpfc_num_kmac_operations_idc;

[0193] unsigned int(32) nnpfc_total_kilobyte_size;

[0194] }

[0195] }

[0196] It has the following semantics:

[0197] `nnpfc_sei_id` contains an identifier that can be used to identify the post-processing filter. The value of `nnpfc_sei_id` should be in the range of 0 to 255 (inclusive). This parameter can be omitted in the sample group description entry because it can be inferred from the value of `grouping_type_parameter` of the corresponding 'sbgp' box with grouping type 'nfcs'.

[0198] `nnpfc_update`, when set to 1, indicates that the corresponding post-processing filter is updated. When set to 0, it indicates the base model. This parameter can be omitted in the sample group description entry because it can be inferred from the value of `grouping_type_parameter` of the corresponding 'sbgp' box with grouping type 'nfcs'. This parameter should be present if the sample grouping used to describe post-filter information does not use or define `grouping_type_parameter`.

[0199] When nnpfc_external_payload is set to 1, it indicates that the payload for the post-processing filter is outside the file. When set to 0, the payload exists in the data portion of the file.

[0200] nnpfc_purpose indicates the purpose of the post-processing filter as specified in ISO / IEC 23002-7.

[0201] The nnpfc_complexity_info_present_flag specifies whether one or more syntactic elements of the complexity of the post-processing filter associated with the nnpfc_id are present (when set to 1) or absent (when set to 0).

[0202] nnpfc_num_kmac_operations_idc indicates the maximum number of multiply-accumulate operations per sample as specified in ISO / IEC 23002-7.

[0203] nnpfc_total_kilobyte_size indicates the total size in kilobytes required to store the uncompressed parameters of a neural network as specified in ISO / IEC 23002-7.

[0204] The sample set can then be activated using the definition described below for the post-filter based on the neural network.

[0205] The Neural Network Post-Filter Activation (NNPFA) SEI message is specified in ISO / IEC 23002-7. The NNPFA SEI message can be included in the VVC bitstream.

[0206] The NNPFA SEI message contains the nnpfa_target_id syntax element, which is an identifier that can be used to identify the post-processing filter involved in the NNPFA SEI message.

[0207] The NNPFA SEI message indicates that an applicable post-processing filter with an nnpfc_id equal to nnpfa_target_id can be used to filter images containing the NNPFA SEI message.

[0208] An instance of SampleToGroupBox for an NNPFA sample group should include a grouping_type_parameter. The grouping_type_parameter field is specified for an NNPFA sample group as follows (the number of bits for each parameter can differ from 16 bits, depending on the possible range of authorized values ​​for each parameter, provided the total remains equal to 32 bits):

[0209] {

[0210] unsigned int(16) filter_update_index;

[0211] unsigned int(16) target_filter_id;

[0212] }

[0213] `filter_update_index` is an integer indicating that all sample group description entries referenced by the `SampleToGroupBox` contain an NNPFA SEI message relating to an update of `filter_update_index-th` on top of the underlying post-processing filter. A `filter_update_index` value of 0 indicates that all sample group description entries referenced by the `SampleToGroupBox` contain an NNPFA SEI message applied to the underlying post-processing filter.

[0214] target_filter_id indicates that all sample group description entries referenced by the SampleToGroupBox contain NNPFA SEI messages with target_nnpfa_id equal to target_filter_id.

[0215] When a track contains NNPFA sample groups, there should be no NNPFA SEI messages in the sample entries on the track.

[0216] When the reader supports NNPFA sample groups, it should perform the following insertion of the prefix SEI NAL cell as part of the bitstream reconstruction:

[0217] - When a sample is mapped to at least one NnpfaSeiSampleGroupEntry, the sample contains a prefix SEI NAL unit for each layer contained in the track, and the prefix SEI NAL unit contains an NNPFA SEI message from the sample data.

[0218] When the reader processes an NNPFA sample group, it should also process an NNPFC sample group on the same track. When a VVC track has an associated VVC non-VCL track containing an NNPFA sample group, there should be no NNPFA SEI message in the sample or sample entry of the VVC track, and there should be no NNPFA SEI message in the sample entry of the VVC non-VCL track.

[0219] When the NNPFC sample group is a required sample group and the NNPFA sample group exists in the same track, the NNPFA sample group should be a required sample group and the 'esgh' sample group should be listed as 'nfcs' and 'nfas' in the subsequent entries of the sample_group_description_type array.

[0220] The syntax for describing entries in the NNPFC sample group can be as follows:

[0221] aligned(8) class NnpfaSeiSampleGroupEntry() extendsVisualSampleGroupEntry('nfas')

[0222] {

[0223] unsigned int(1) nnpfa_cancel_flag;

[0224] if (!nnpfa_cancel_flag) {

[0225] unsigned int(1) nnpfa_persistence_flag;

[0226] unsigned int(6) reserved;

[0227] }

[0228] else

[0229] unsigned int(7) reserved;

[0230] }

[0231] It has the following semantics:

[0232] `nnpfa_cancel_flag` has the same semantics as defined in ISO / IEC 23002-7. This means that when it equals 1, it indicates that the persistence of the target neural network post-processing filter established by any previous NNPFA SEI message with the same `nnpfa_target_id` as the current SEI message is canceled; that is, the target neural network post-processing filter is no longer used unless it is activated by another NNPFA SEI message with the same `nnpfa_target_id` as the current SEI message and `nnpfa_cancel_flag` equals 0. `nnpfa_cancel_flag` equal to 0 indicates adherence to `nnpfa_persistence_flag`.

[0233] The nnpfa_persistence_flag has the same semantics as defined in ISO / IEC 23002-7. It specifies the persistence of the target neural network post-processing filter for the current layer.

[0234] Setting nnpfa_persistence_flag to 0 specifies that the target neural network post-processing filter can only be used for post-processing filtering of the current image.

[0235] The `nnpfa_persistence_flag` value of 1 specifies that the target neural network post-processing filter can be used for post-processing filtering of the current image of the current layer and all subsequent images in output order, until one or more of the following conditions are true:

[0236] - A new CLVS begins in the current layer.

[0237] - End of stream.

[0238] - Outputs the image associated with the NNPFA SEI message in the current layer that has the same nnpfa_target_id as the current SEI message and an nnpfa_cancel_flag equal to 1, appended to the current image in the output order.

[0239] The target neural network post-processing filter is not applied to the subsequent image in the current layer associated with an NNPFA SEI message that has the same nnpfa_target_id as the current SEI message and an nnpfa_cancel_flag equal to 1.

[0240] Optionally, the identifier of the target post-filter and the parameters corresponding to the indication of the base or newer version can be inserted into the 'nfas' sample group description entry, especially if the grouping_type_parameter is not used or defined for the sample group for post-filter information.

[0241] In embodiments that map NNPFC or NNPFA SEI messages to VVC tracks, sample groups can be used to indicate samples containing SEI NAL units for these SEI messages. This means that samples are not mapped based on filter activation: depending on the application, which samples use which filter is determined based on the samples after the filter is activated or deactivated. This embodiment makes bitstream reconstruction easier for the parser while providing sufficient information to infer the configuration of the activated filter. To this end, sample groups for neural network-based post-processing filter characteristics and activation are described below.

[0242] A sample mapped to an NnpfcSeiSampleGroupEntry contains one or more SEI NAL units corresponding to one or more NNPFC SEI messages. For each contained NNPFC SEI message, the sample group description indicates whether the sample contains a basic post-processing filter or an update of a post-processing filter. The grouping_type_parameter field may not be specified for the NNPFC sample group.

[0243] When a sample is not mapped to an NnpfcSeiSampleGroupEntry, the sample does not contain any SEI NAL unit for NNPFC SEI messages.

[0244] When a track contains NNPFC sample groups, there should be no NNPFC SEI message in the sample entries of that track. When a VVC track has an associated VVC non-VCL track containing NNPFC sample groups, there should be no NNPFC SEI message in the samples or sample entries of the VVC track, and there should be no NNPFC SEI message in the sample entries of the VVC non-VCL track.

[0245] A synchronization sample should contain an SEI NAL unit corresponding to an NNPFC SEI message that is active for at least one sample from this synchronization sample to the next synchronization sample. The synchronization sample should be mapped to an NnpfcSeiSampleGroupEntry.

[0246] When the reader supports NNPFC sample groups, it should insert prefix SEI NAL units corresponding to the NNPFC SEI messages for the mapped samples as part of the bitstream reconstruction. The proposed syntax for describing neural network-based post-filter characteristics in a video track can be defined as follows:

[0247] aligned(8) class NnpfcSeiSampleGroupEntry() extendsVisualSampleGroupEntry('nfcs')

[0248] {

[0249] unsigned int(8) nnpfc_entries;

[0250] for (int i=0; i < nnpfc_entries; i++) {

[0251] unsigned int(32) nnpfc_sei_id;

[0252] unsigned int(1) nnpfc_update;

[0253] unsigned int(1) nnpfc_external_payload;

[0254] unsigned int(5) nnpfc_purpose;

[0255] unsigned int(1) nnpfc_complexity_info_present_flag;

[0256] if (nnpfc_complexity_info_present_flag) {

[0257] unsigned int(32) nnpfc_num_kmac_operations_idc;

[0258] unsigned int(32) nnpfc_total_kilobyte_size;

[0259] }

[0260] }

[0261] }

[0262] It has the following semantics:

[0263] nnpfc_entry indicates the number of NNPFC SEI messages contained in the sample mapped to this entry.

[0264] nnpfc_sei_id contains an identifier that can be used to identify post-processing filters as specified in ISO / IEC 23002-7.

[0265] `nnpfc_update` indicates whether the sample is a base model containing post-processing filters or an update. A value of 0 indicates a base model, and a value of 1 indicates an update.

[0266] When nnpfc_external_payload is set to 1, it indicates that the payload for the post-processing filter is outside the file. When set to 0, the payload exists in the data portion of the file.

[0267] nnpfc_purpose indicates the purpose of the post-processing filter as specified in Table 20 of ISO / IEC 23002-7.

[0268] The nnpfc_complexity_info_present_flag specifies whether one or more syntactic elements of the complexity of the post-processing filter associated with the nnpfc_id are present (when set to 1) or absent (when set to 0).

[0269] nnpfc_num_kmac_operations_idc indicates the maximum number of multiply-accumulate operations per sample as specified in ISO / IEC 23002-7.

[0270] nnpfc_total_kilobyte_size indicates the total size in kilobytes required to store the uncompressed parameters of a neural network as specified in ISO / IEC 23002-7.

[0271] In a variation, instead of looping through the number of NNPFC SEI messages, the grouping_type_parameter can still be used to map a given sample onto different sample group description entries, each entry containing a post-filter description (possibly a set or subset of the parameters in the for loop in the above syntax). For example, the identifier and the update indication can be repeated (when they are available in the grouping_type_parameter).

[0272] The sample group used to support NNPFA SEI messages can be redefined as follows:

[0273] The samples mapped to NnpfaSeiSampleGroupEntry contain one or more SEI NAL units corresponding to NNPFA SEI messages.

[0274] Samples not mapped to NnpfaSeiSampleGroupEntry do not contain any SEI NAL units corresponding to NNPFA SEI messages.

[0275] Synchronization samples can contain SEI NAL units corresponding to the NNPFA SEI messages of the active configuration prior to retaining the synchronization sample. Synchronization samples can be mapped to NnpfaSeiSampleGroupEntry.

[0276] An instance of SampleToGroupBox for an NNPFA sample group should include the grouping_type_parameter. The grouping_type_parameter field is specified for an NNPFA sample group as follows:

[0277] {

[0278] unsigned int(16) filter_update_index;

[0279] unsigned int(16) target_filter_id;

[0280] }

[0281] Where `filter_update_index` is an integer indicating that all sample group description entries referenced by the `SampleToGroupBox` contain an NNPFA SEI message relating to an update of `filter_update_index-th` on top of the base post-processing filter. `filter_update_index` equal to 0 indicates that all sample group description entries referenced by the `SampleToGroupBox` contain an NNPFA SEI message applied to the base post-processing filter. In a variation, this parameter could simply include a flag distinguishing between base and update, but having this parameter instead of an index improves support for NNPFA SEI messages by allowing any configuration of a given post-filter (base or any update, while the current design appears to consider the latest update (if any)).

[0282] target_filter_id indicates that all sample group description entries referenced by the SampleToGroupBox contain NNPFA SEI messages with target_nnpfa_id equal to target_filter_id.

[0283] In a variation, instead of `filter_update_index`, `grouping_type_parameter` can simply use a flag to indicate whether the sample group description corresponds to the base or the latest (assuming then last updated). Alternatively, a parameter corresponding to the identifier of the target post-filter and an indication of the base or latest version can be inserted into the 'nfas' sample group description entry, especially if `grouping_type_parameter` is not used or defined for sample groups targeting post-filter information.

[0284] When a track contains NNPFA sample groups, there should be no NNPFA SEI messages in the sample entries on the track.

[0285] When the reader supports NNPFA sample groups, it should insert the prefix SEI NAL cell corresponding to the NNPFA SEI message for the mapped sample as part of the bitstream reconstruction.

[0286] When a sample is mapped to at least one NnpfaSeiSampleGroupEntry, the sample contains a prefix SEI NAL unit for each layer contained in the track, and the prefix SEI NAL unit contains an NNPFA SEI message from the sample data.

[0287] When the reader processes an NNPFA sample group, it should also process an NNPFC sample group on the same track. When a VVC track has an associated VVC non-VCL track containing an NNPFA sample group, there should be no NNPFA SEI message in the sample or sample entry of the VVC track, and there should be no NNPFA SEI message in the sample entry of the VVC non-VCL track.

[0288] When the NNPFC sample group is a necessary sample group and the NNPFA sample group exists in the same track, the NNPFA sample group should be a necessary sample group, and the 'esgh' sample group should be listed as 'nfcs' and 'nfas' in subsequent entries of the sample_group_description_type array. The syntax of the description entry for the activated sample group of the described filter can be defined as follows:

[0289] aligned(8) class NnpfaSeiSampleGroupEntry() extendsVisualSampleGroupEntry('nfas')

[0290] {unsigned int(1) nnpfa_cancel_flag;

[0291] if (!nnpfa_cancel_flag) {

[0292] unsigned int(1) nnpfa_persistence_flag;

[0293] unsigned int(6) reserved;

[0294] }

[0295] else

[0296] unsigned int(7) reserved;

[0297] }

[0298] It has the following semantics:

[0299] nnpfa_cancel_flag has the same semantics as defined in ISO / IEC 23002-7.

[0300] nnpfa_persistence_flag has the same semantics as defined in ISO / IEC 23002-7.

[0301] In a variation, instead of using the `grouping_type_parameter`, the number of NNPFA SEI messages can be included in a cycle in the sample group description entry of type `nfas` in the same way as in the `nfcs` sample group description entry. For example, the parameter `nnpfa_entries` indicates the number of NNPFA SEI messages contained in the samples mapped to that entry.

[0302] As another variation, the `grouping_type_parameter` can be used to map samples to post-filter activation or deactivation, for example, by assigning bits from `grouping_type_parameter` to indicate deactivation / activation, or by assigning two bits to reflect possible values ​​for deactivation and persistence. In this variation, the sample group description entry may not repeat deactivation or persistence information. If the `grouping_type_parameter` is only used for deactivation / activation indication, a `post_filter` identifier or an indication of `post_filter` updates can be declared in the sample group description entry 'nfas'.

[0303] To facilitate the parser's location of the NAL cell corresponding to the NNPFC or NNPFA SEI message within the mapped samples, additional parameters can be provided in the sample group description entries nfcs and nfas, thus providing the index or position of the NAL cell corresponding to the NNPFC SEI message and NNPFA SEI message, respectively. The indexing starts at the sample offset, meaning the first NAL cell of the sample has index 1.

[0304] Alternatively, or in any variant of providing post-filter information such as a sample group with grouping_type 'nfcs' or 'nfas', the NNPFC or NNPFA SEI message may be included in the list of important SEI messages from an application perspective, for example, using the 'seii' box.

[0305] Indication of post-filter information within the media track

[0306] Figure 4a This example demonstrates the sample metadata structure available for the filter information following the indication, specific to a track in the media file. This corresponds to the details of step 208 in the encapsulation process. The sample metadata structure can be used... Figure 3 The parsing process described herein, for example, at step 303, can exist in different ways in which the indication track in the media file has relevant post-filter information, depending on the dynamics of this information and the granularity of the information.

[0307] Figure 4aAn example of a media file 400 containing media track 410 with associated post-filter information is shown. The media file may contain multiple media tracks, but only one is illustrated for clarity. Track 410 may be fragmented, but this is not illustrated here. The media file has a metadata portion represented by 'moov' box 401 and its sub-boxes 410, etc., and a data portion represented by 'mdat' box 402. This data portion contains data units (e.g., NAL units for MPEG video codecs) for a sequence of samples such as 421, 422, or 423. Samples may contain data units 421-1 or 423-1 for post-filter information and data units 421-2, 421-3, or 423-2 for media data, etc. The media file also contains a 'ftyp' box 403, which may contain a brand indicating that a certain track has present post-filter information, such as a 'pfip' compatible brand. Alternatively, this specific brand may be indicated in a TrackTypeBox such as 413. Box 411 contains a sample table box ('stbl'), which provides a sample description and declares sample entries in box 412 ('stsd'). For example, 'stsd' 412 indicates that the track is a video track of type 'vvc1'. Within the sample description, there may be specific boxes such as 414 or dedicated sample groups such as 415 and 416 to indicate post-filter information. The content of 414 may correspond to one of the variations in the previously described embodiments, with the PostFilterInformationBox as an optional box in the sample entries. Boxes 416 and 415 may correspond to a variation in the previously described embodiments, i.e., having a specific grouping_type ( ) that maps samples to post-filter descriptions. Figure 4a The sample group of 'pfif' (or 'nfcs' or 'nfas' used to map NNPFC and NNPFA SEI messages in the metadata section of the media track). Figure 4a As shown, the description of the post-filter information can be in the metadata section, but the payload remains in the data section represented by the NAL unit in the case of 421-1 or 423-1 data units or MPEG video tracks. This avoids extracting data units from the bitstream during encapsulation and re-inserting data units into the bitstream during parsing. The description section should allow the parser to determine whether to retain or skip post-filter-related data units for its bitstream reconstruction. A smart parser that allows for fast startup can decide to shift the insertion of data units related to the post-filter. For example, the parser can wait for a few samples to insert the post-filter-related data units into the reconstructed bitstream instead of inserting them into the first sample or synchronization sample. The player can then begin rendering the first video sample while obtaining the post-filter information and applying it to subsequent samples. This allows video quality to gradually improve with fast startup or fast loading or searching.

[0308] It should be noted that combining metadata describing post-filter information at both the sample entry or track level with the sample group description can be advantageous. The former allows the player to be notified of potential additional processing for media decoding. This can be useful for instantiating the appropriate decoder and allocating resources (CPU or memory). For example, the type or purpose of the post-filter and its availability within the file (as it may cause latency when external) can be useful. This can be described using a subset of the parameters of the PostFilterInformationBox. The sample group description level provides more granular information about which samples can be post-filtered and how they are post-filtered (to use the payload).

[0309] When sharing descriptions and payloads used to indicate post-filter information in media tracks, different approaches may exist when considering the encapsulation of filter data units, such as... Figure 4b and Figure 4c As shown:

[0310] Figure 4b The following embodiment illustrates an example where, to encapsulate Filter Data Unit 451 (FDU) and Media Data Unit (MDU), Media File 450 includes a metadata portion comprising a box 452, which may be part of the general description described above or a V-SEI-specific mapping (e.g., a dedicated box or sample set). Media File 450 includes a payload 453 in its data portion, which comprises the Filter Data Unit 451 without any modifications. It should be noted that there may be some overlap between the metadata in 452 and the FDU in 453, meaning duplicated parameters.

[0311] Figure 4c As illustrated in the following embodiment, in order to encapsulate Filter Data Unit 461 (FDU) and Media Data Unit (MDU), Media File 460 includes a metadata portion comprising boxes 462, which may include a portion of Filter Data Unit 461 (e.g., syntactic elements describing post-filter information). Media File 460 includes a payload 463 in its data portion, which may include the payload portion of the Filter Data Unit (if present) or the remainder of the syntactic elements describing post-filter information plus the payload (if present). The data portion may mix Media Data Units and Filter Data Units. It should be noted that there may be some overlap between the description data in the metadata portion FDU_part1 462 and the payload and / or remaining syntactic elements in the data portion FDU_part2 463, meaning duplicate parameters.

[0312] It should be noted that Figure 4b and Figure 4c The two possibilities shown can be applied to all embodiments.

[0313] When post-filter information is provided in a sample group, and when a post-filter information is applied to the set of samples or all samples of a track, default sample grouping can be used to avoid mapping the set of samples or all samples to sample group description entries.

[0314] A static sample group can be used when post-filter information is provided in the sample group, and when all post-filter information is known at the start of encapsulation, or when it can be determined that no updates to post-filter information are anticipated along the media presentation (e.g., system configuration or settings).

[0315] Indication of post-filter information as an information item

[0316] Figure 5An example is illustrated where post-filter information is stored outside the media track to which it is applied. One advantage of this embodiment is that it allows for the interoperability of post-filter information across multiple media tracks (if desired). In this embodiment, media file 500 may contain an 'ftyp' box 501, which may be embedded in a list of compatible brands indicating the presence of post-filter information. Alternatively, the brand can be indicated in a TrackTypeBox within a 'trak' box of a media track (e.g., 503) using post-filter information. Media file 500 contains a metadata portion comprising MovieBox 502 and its sub-boxes (e.g., 503 and MetaBox 504), their own box hierarchy describing the media present in file 500 and its associated post-filter information. Media file 500 also contains a media data portion comprising one or more MediaDataBoxes 505 to store data for track samples or items. As a reminder, tracks have been defined to encapsulate timed media data, where the media data consists of samples (e.g., video sequences) to be rendered under time constraints. Items are defined to encapsulate non-timing media data (e.g., still images). Similar to tracks, items consist of a metadata section that includes a description of the item stored in metadata section 504 and associated with the media data section in data section 505 of the media file. A metadata item is a specific item characterized as the data section of an item stored in data section 505 of the media file actually including metadata. In this embodiment, MediaDataBox 505 can mix data for post-filter information (e.g., 551-1 or 551-2) and data for media samples (e.g., 552-1, 552-2, 552-3, etc.). For example, when applied to an MPEG video codec (e.g., VVC) and a neural network-based post-filter SEI message, 551-1 or 551-2 can be a NAL unit for an NNPFC SEI message or an NNPFA SEI message. These messages can provide descriptions and data for an underlying neural network model or for an update to a neural network model. Each neural network model (underlying or updated) can be described as a metadata item in 'meta' box 504 within item information box 540. For example, metadata item 541 is the base model for the neural network post-filter with filter identifier 0. Furthermore, 542 is the base model for the neural network post-filter with filter identifier 1. The following metadata items 541-1 or 541-2 correspond to updates to the neural network post-filter with filter identifier 0. Similarly, metadata items 542-1 or 542-2 correspond to updates to the neural network post-filter with filter identifier 1. These metadata items are indicated by the encapsulation module and identified by the parser module through a specific item_type.For example, the specific item_type is 'cnnr' when they correspond to a compressed neural network representation as specified in MPEG-7 Part 17; or more generally, the specific item_type is 'pfii' for post-filter information items when the actual representation format is specified elsewhere (e.g., using an 'iinf' box for item_uri_type). Arrows between metadata items for post-filter information indicate decoding dependencies from the metadata item corresponding to the update to the metadata item corresponding to the base. Decoding dependencies can be described using item reference boxes with specific item reference types (e.g., 'nnrd' for "neural network representation dependency" or 'pfid' for "post-filter information dependency" (4CC and names are merely examples)). Alternatively, item references can reuse existing item reference types to decode dependencies between items. ItemLocationBox('iloc') 545 indicates the location of the data corresponding to the metadata item in MediaDataBox 505. For example, when applied to MPEG video codecs (such as VVC) and neural network-based post-filter SEI messages, 541 or 542 may contain a description of the post-filter information, where the actual payload from the VVC bitstream is located in data units 551-1 (e.g., NNR units according to MPEG-7 Part 17) and 551-2 (data units may be MPEG-7 NNR units or NAL units for NNPFC SEI messages or NNPFA SEI messages), respectively. Metadata items for the post-filter information can be associated with item attributes described in the 'ipco' box (within the 'iprp' box 546) as shown in 547. Arrows between metadata items 541 or 542 and attributes in the 'ipco' box can be described within the 'ipma' box. A specific item attribute 'nnpf' is defined to store a description of the post-filter metadata item. One item attribute for the post-filter information can be associated with several metadata items for the post-filter information. The content of this attribute can be the same set or subset of parameters described for the general post-filter information in the previously described embodiments (e.g., parameters providing filter identifiers, filter purpose, complexity, or indications related to internal or external payloads, whether it is the same original bitstream, etc.). In the specific case of MPEG video codecs such as VVC and associated NNPFC and NNPFASEI messages, the 'nnpf' attribute can contain a general description obtained from analysis or copying of a portion of the syntactic elements for these SEI messages. The payload of the SEI message (if present) will be maintained in the data portion of the media file (e.g., MediaDataBox as in 505).

[0317] Finally, the media samples described in sample table box 530 can be associated with metadata items 541, 542, ... via 'stmi' sample groups, which allow associations between sample groups and one or more items. This association indicates which filter(s) is applied to which sample or sample group. This association is represented by ellipses 531 to 534 and dashed arrows from the ellipses to one or more metadata items.

[0318] An example of a general term property for post-filtered information can be described as follows:

[0319] aligned(8) class PostFilteringProperty

[0320] extends ItemFullProperty('pfip', version = 0, flags = 0) {

[0321] unsigned int(1) post_filter_without_payload;

[0322] unsigned int(1) post_filter_same_bitstream;

[0323] unsigned int(1) post_filter_update;

[0324] unsigned int(5) post_filter_purpose;

[0325] utf8string post_filter_identifier;

[0326] }

[0327] It has the same semantics as in the embodiment for general post-filter information.

[0328] Examples of more specific term properties for neural network-based post-filtering can be described as follows:

[0329] aligned(8) class NNPostFilteringProperty

[0330] extends ItemFullProperty('nnpf', version = 0, flags = 0) {

[0331] unsigned int(1) nnpfc_external_payload;

[0332] unsigned int(1) nnpfc_purpose;

[0333] unsigned int(1) nnpfc_complexity_info_present_flag;

[0334] if (nnpfc_complexity_info_present_flag) {

[0335] unsigned int(32) nnpfc_num_kmac_operations_idc;

[0336] unsigned int(32) nnpfc_total_kilobyte_size;

[0337] }

[0338] }

[0339] Or in a variant that is byte-aligned and supports the same range of filter objectives as in V-SEI:

[0340] aligned(8) class PostFilteringProperty

[0341] extends ItemFullProperty('pfip', version = 0, flags = 0) {

[0342] unsigned int(1) post_filter__external_payload;

[0343] unsigned int(10) post_filter__purpose;

[0344] unsigned int(1) post_filter__complexity_info_present_flag;

[0345] unsigned int(4) reserved;

[0346] if (nnpfc_complexity_info_present_flag) {

[0347] unsigned int(32) post_filter__num_kmac_operations_idc;

[0348] unsigned int(32) post_filter__total_kilobyte_size;

[0349] }

[0350] }

[0351] Or in another variation, providing a URI for accessing external post-filter information and an identifier for post_filter:

[0352] aligned(8) class PostFilteringProperty

[0353] extends ItemFullProperty('pfip', version = 0, flags = 0) {

[0354] unsigned int(32) post_filter_id;

[0355] unsigned int(1) post_filter_external_payload;

[0356] unsigned int(10) post_filter_purpose;

[0357] unsigned int(1) post_filter_complexity_info_present_flag;

[0358] unsigned int(4) reserved;

[0359] if (nnpfc_external_payload) {

[0360] utf8string post_filter_uri;

[0361] utf8string post_filter_tag_ uri; / / optional

[0362] }

[0363] if (nnpfc_complexity_info_present_flag) {

[0364] unsigned int(32) post_filter_num_kmac_operations_idc;

[0365] unsigned int(32) post_filter_total_kilobyte_size;

[0366] }

[0367] }

[0368] It should be noted that the same variations can be applied to post-filters described by NNPFC SEI messages or neural network-based post-filters (e.g., NNPostFilteringProperty), which have the same semantics as those described in the variations of the embodiments for mapping post-filter SEI messages in the metadata of the media track (except for the mapping being performed in the metadata items here). The semantics of the additional parameters in the variations are as follows:

[0369] The `post_filter_id` indicates the identifier of the post-processing filter. For example, when the post-filter is a neural network-based post-filter, this identifier can correspond to the `nnpfc_id` specified in ISO / IEC 23002-7.

[0370] The `post_filter_uri` contains a URI with syntax and semantics as specified in IETF Internet Standard 66, which identifies the underlying post-processing filter or an updated neural network relative to that underlying post-processing filter using an identifier such as in the `npfc_id` field. For example, when the post-filter is a neural network-based post-filter, `post_filter_uri` could correspond to `nnpfc_uri` as specified in ISO / IEC 23002-7.

[0371] The `post_filter_tag_uri` provides a unique identifier for the format of the post-filter. For example, when the post-filter is a neural network-based post-filter, `post_filter_tag_uri` can correspond to `nnpfc_tag_uri` as specified in ISO / IEC 23002-7.

[0372] For mapping NNPFC or NNPFA SEI messages, the 'stmi' sample group associates a sample with one or more metadata items of post-filter information applied to the mapped sample. It should be noted that metadata items can be associated with image items, for example, through a specific item reference type. Still using the 'stmi' sample group, metadata items can also be associated with samples from image sequence tracks.

[0373] Back Figure 5In the example above, the 'stmi' sample group will include a SampleToGroupBox('sbgp') with a grouping_type equal to 'stmi' and a SampleGroupDescriptionBox('sgpd') with the same grouping_type value 'stmi'. The first entry 531 in the 'sbgp' box maps samples S1 and S2 to a first group description entry with an item_ID represented on the metadata item indicated on 542-1. This entry depends on metadata item 542. Item 542-1 is the update, while item 542 is the base model for the post-filter. Another entry in the sbgp box 532 maps samples S4 and S5 to another sample group description entry indicating two item_IDs: one for metadata item 541-1 and one for metadata item 541-2, both of which depend on metadata item 541. When samples are mapped to several metadata items for a post-filter, as in 'sbgp' entry 532, the order of items in the 'stmi' sample group description entry should follow the order of the post-filter operations. Although entry 532 links samples to items describing the post-filter corresponding to updates of the same base model 541, the same mechanism applies to updates from different base models. For example, the 'stmi' sample group can map one or more later samples in the sequence (not illustrated) to items 541-2 and 542-2, which are updates to base models 541 and 542, respectively. The third entry in 'sbgp' (533) maps samples S6 and S7 to a sample group description entry containing an item_ID corresponding to the metadata item denoted as 541-2. Each entry in the 'sgpd' box contains a parameter called meta_box_handler_type, which informs the type of metadata scheme used by the MetaBox referenced by the items in the sample group. Its value can be set to the generic value 'meta' to indicate that the item provides metadata for the sample. It is appropriate and safe to exclude other MetaBoxes once they are in the track that conveys the 'stmi' sample group or in the moov box that has a handler type also set to 'meta'. To avoid any ambiguity, specific values ​​(such as 4CC) can be used to indicate the MPEG-7 compressed neural network representation (e.g., 'cnnr'), such as item_type; or HandlerProperty can be included in the 'ipco' box 547 associated with the metadata items (541, 542, 541-1...) for post-filter information.In this HandlerProperty, if (one or more) the post-filter is a compressed neural network representation, the handler_type value can be set to 'cnnr', or for general post-filter information, the handler_type value can be set to 'pfif'. Once all metadata items describing post-filter information in the file have the same item_type, this handler attribute can be shared across these metadata items. The item_ID of the metadata item describing the post-filter information can have a value different from the nnpfc_id parameter communicated in the NNPFC SEI message or different from the post-filter identifier. In some very specific configurations (where items in the media file only include the NN-based post-filter underlying model), the item_ID can be the post-filter identifier. However, in most cases, and for greater flexibility, the post-filter identifier appears as an attribute of the metadata item describing the post-filter and can be part of the metadata structure providing the post-filter information, such as the 'nnpf' item attribute contained in box 547. In a variation of using metadata items to describe post-filter information and linking samples to the post-filter information using 'stmi' sample groups, the 'stmi' sample groups can use the `grouping_type_parameter` (of a `SampleToGroupBox` with `grouping_type='stmi'`) to map samples to either the base filter or the updated filter, depending on the value of that `grouping_type_parameter`. This parameter can also be used to indicate the version of the post-filter, instead of a base or updated indication. For example, still based on... Figure 5Entries 531, 532, 533, and 534 in the SBGP box with grouping_type='stmi' can have a value of grouping_type_parameter indicating an update or version of metadata item 542-1. Samples using the underlying model such as item 541 or 542 will be mapped into an SBGP box with grouping_type='stmi' and a grouping_type_parameter indicating the underlying post-filter. Using this variation, the indication of the underlying or updated filter in the metadata structure providing post-filter information (e.g., an item attribute such as 'nnpf' contained in box 547) can become optional. It should be noted that the version indication (possibly as a version number or version identifier) ​​can depend on information from the bitstream representation of the neural network model used for post-filtering. For example, in an MPEG NNR representation, the bitstream can provide an explicit list of versions of the neural network models contained within the bitstream. This list can be a single list at the beginning of the stream or an initial list that can be supplemented throughout the stream. Versions can have explicit identifiers (recommended for this use case) or implicit identifiers (i.e., the identifier is "only" an index of the version in the version list). This list can be available in the base model (and thus in one item), or in an initial list in the base model (one item) followed by supplementation in the updated model (one or more other items). The wrapper module can decide to embed several versions of the same model in a single item or multiple items describing the post-filter. In the case of NNPFC and NNPFA SEI messages, it is convenient to use one item for each version of the model: one item payload will correspond to one NAL unit for the NNPFC SEI message. When one or more versions or base plus update are stored in one item, the dependency indication may not be necessary if the individual items are self-contained (i.e., containing all the data for a complete version of the neural network model). The list of versions embedded in that item can then be indicated as an attribute of that item (e.g., in the 'nnpf' attribute in 'ipco' box 547, rather than in the 'stmi' sample-to-group box grouping_type_parameter).

[0374] Figure 5This describes the use of metadata items to describe post-filter information (such as NNPFA SEI messages) and the use of 'stmi' sample groups to associate samples with metadata items describing the (one or more) post-filters applied to those samples. The activation of the filter on a specific video frame is the target of the NNPFA SEI message. This conversion of NNPFA SEI messages to 'stmi' sample groups requires the encapsulation module to interpret the NNPFA SEI messages to detect when a given post-filter is active and when it is inactive. This can be done by parsing the body of (one or more) NNPFA SEI messages (e.g., the value of target_id, cancel_flag, and persistence flags (if present), which is quite straightforward. This is performed, for example, during step 204. When cancel_flag equals 0, the sample (containing (one or more) NNPFA SEI messages) must be mapped to an entry in the stmi SampleGroupDescriptionBox. The encapsulation module must determine whether an existing entry can be reused or whether a new entry should be created. To this end, the encapsulation module lists (one or more) nnpfa_target_ids for the current samples, searches for metadata items associated with these filter identifiers, and creates a new entry in the stmiSampleGroupDescriptionBox if the determined list of metadata items is new. If the stmiSampleGroupDescriptionBox already contains an entry with the determined list of metadata items, the sample is mapped to that sample group description index in the stmiSampleToGroupBox. Then, if persistence_flag is set for one or more NNPFASEI messages, subsequent samples are mapped to the same sample group description entry until the encapsulation module encounters an NNPFA SEI message with cancel_flag=1 in step 204. Once the number of NNPFAs changes or at least one filter is canceled, the encapsulation module reconsiders the mapping of the current samples. From the parser side, there is an equivalent processing method for operations that occur when the NNPFASEI message has already been converted to an stmi sample group during encapsulation. This can be done at steps 303 and 304. If a sample is mapped to a sample group description entry of type stmi, then step 303 determines that the sample is actually associated with post-filter information.If the sample is the first sample mapped to a given 'stmi' entry, the parser prepares NAL units of type NNPFA SEI messages as many as indicated in the stmi sample group description entry, where nnpfa_target_id equals the post-filter identifier associated with the relevant metadata item (indicated by item_ID), cancel_flag is set to 0, and persistence is set to 1 if subsequent samples are mapped to the same sample group description entry. If only one sample is mapped, persistence is set to 0. The same operation occurs for the last sample mapped to the stmi sample group description entry: the parser creates NAL units of type NNPFASEI messages as many as indicated in the stmi sample group description entry, where nnpfa_target_id equals the post-filter identifier associated with the metadata item (indicated by item_ID), and cancel_flag is set to 1. These NAL units are then appended to the reconstructed bitstream for the post-filtered video. Furthermore, storing them as items provides some flexibility for reconstructing NNPFC SEI messages. In practice, these messages can be accessed in the media file when needed or on demand. For example, these messages can be obtained from ItemLocationBox 545, which provides byte offsets in the media data box. When there are no constraints on startup delay, these messages can be inserted at the beginning of the reconstructed bitstream simply by copying the byte range of the post-filter metadata item (since the byte range corresponds to the NNPFC SEI NAL unit). When there are constraints on startup delay from an application perspective, the NAL unit corresponding to the post-filter metadata item can be inserted into the reconstructed bitstream after some samples. The NAL unit corresponding to the NNPFC SEI message should be delayed in the bitstream accordingly (i.e., it should not exist before the target NNPFC SEI message). Furthermore, the NAL unit corresponding to the base model should be inserted before the NAL unit for the updated model. In general, data corresponding to metadata items describing post-filter operation or information can be inserted into the bitstream at the beginning of data corresponding to samples using the post-filter. In the case of random access, the item information (541, 542...) and item location box 545 provide a byte range for requesting or reading post-filter information on demand. Therefore, it is not necessary to copy data for the post-filter in the synchronized samples.When encapsulating a media sample that may use a post-filter as a synchronous or random access sample (e.g., as indicated in the NNPFA SEI message), the sample should be mapped in 'stmi' to an appropriate metadata item that correctly indicates the activation of the post-filter (i.e., consistent with the content of the NNPFA SEI message) and allows the media player to extract or reconstruct (one or more) NNPFA SEI NAL units for that synchronous or random access sample. Encapsulating the NNPFC SEI message in steps 207 and 208 involves obtaining the NAL units for the NNPFC message, inserting them into the media data, describing the byte offsets in the media data in ItemLocationBox 545, and associating them with metadata items created as item information entries. The encapsulation module may extract some parameters from the NNPFC SEI message (or obtain parameters from application settings or through the user interface) and store the parameters as item attributes in ipco box 547 or a dedicated structure to store general post-filter information.

[0375] In cases where several STMI sample groups are used to link samples to metadata items, the specific link used for the post-filter can be indicated by using the `grouping_type_parameter` (e.g., set to 'cnnr' or 'pfif') for different purposes. However, when used for this purpose, the `grouping_type_parameter` cannot be used to indicate the base, update, or version of the post-filter (these indications can then be described in the item attributes described above or by other means, such as dedicated boxes or sample group descriptions).

[0376] Indication of post-filter information as a combination of sample groups and information items

[0377] Having an NNPFC payload within a video track sample will inevitably lead to duplication of that data when the media file is configured for random access (e.g., with synchronized samples). In fact, it is necessary to repeat the payload across each synchronized or random access sample. On the other hand, having an NNPFC payload in the media file's metadata can cause startup delays when the model size reaches megabytes and does not help the player determine if a post-filter is available and for what purpose.

[0378] Figure 8 This embodiment is illustrated by the following:

[0379] - As a sample group of media (e.g., video, audio, volumetric...) track 803, it is used to expose parameters describing NNPFC, but

[0380] - Store the NAL unit associated with the NPFC SEI message as a metadata item in the data section 805 of the media file 800.

[0381] - Using ISOBMFF's sample-to-item sample group ('stmi') (831, 832, 833), media samples are mapped to their active (one or more) NNPFCs.

[0382] - Keep the NAL cell corresponding to the NNPFA SEI message in sample data 805.

[0383] This embodiment offers several advantages:

[0384] - It is not necessary to repeat the NNPFC payload for each synchronization sample (if it exists).

[0385] - Specific signaling notifications for NNPFA SEI messages that do not require handling by the 'stmi' sample group.

[0386] - The ability to apply skipped or non-skipped post-filtering based on the NNPFC description in the media track.

[0387] - Retain fast startup and fast media search; metadata items are not necessarily downloaded at startup, but rather when the filter is activated.

[0388] - Access the payload for the post-processing filter via metadata items (if present).

[0389] According to this embodiment, it is proposed to use 'stmi' sample groups, describe NNPFC as a sample group and metadata item, and describe NNPFA as a link between media samples and (one or more) metadata items.

[0390] Each neural network model (base or updated) is described as a metadata item (841, 842, 842-1) identifying the post-filter information by the item_type value 'pfii' (for post-filter information items, any dedicated 4CC can be used as long as it does not conflict with existing and registered information). The actual format and description of the post-processing filters are specified in the 'nfcs' sample group (830-2 and possibly 830-1) describing the post-processing filters as defined below. The NAL unit corresponding to the NNPFC SEI message should be stored in the data of the metadata item (851-1, 851-2, 851-3).

[0391] Samples mapped to NnpfcSeiSampleGroupEntry contain descriptions of one or more NNPFC SEI messages (within 830-1). For each contained NNPFC SEI message, the sample group description (830-1) indicates whether the sample contains a base post-processing filter (841 or 842) or an update of a post-processing filter (841-1, 841-2, or 842-1). When a sample is not mapped to NnpfcSeiSampleGroupEntry, the sample does not contain any descriptions of NNPFC SEI messages.

[0392] In this embodiment, an instance of SampleToGroupBox for NNPFC sample groups may include a grouping_type_parameter. The grouping_type_parameter field is specified for NNPFC sample groups as follows:

[0393] {

[0394] unsigned int(1) filter_update_flag;

[0395] unsigned int(31) filter_id;

[0396] }

[0397] A `filter_update_flag` value of 1 indicates that all sample group description entries referenced by the `SampleToGroupBox` contain updated NNPFC SEI messages provided on top of the underlying post-processing filter. A `filter_update_flag` value of 0 indicates that all sample group description entries referenced by the `SampleToGroupBox` contain NNPFC SEI messages specifying the underlying post-processing filter. This is particularly useful when a metadata item does not have updates to the blended post-filter and the underlying version. Otherwise, a list of `filter_id`s can be provided in a loop within the sample group description entries, and the `grouping_type_parameter` should not be used.

[0398] filter_id indicates that all sample group description entries referenced by the SampleToGroupBox contain NNPFC SEI messages with nnpfc_id equal to filter_id.

[0399] When a track (803) contains NNPFC sample groups (830-2 and possibly 830-1), the NAL unit for the NNPFC SEI message should not exist in the sample entry or within the sample of the track.

[0400] Preferably, the synchronized samples are mapped to NnpfcSeiSampleGroupEntry.

[0401] When the reader supports NNPFC sample groups, it should also support metadata items for post-filter information, and it should insert the prefix SEI NAL cell corresponding to the NNPFC SEI message for the mapped sample as part of the bitstream reconstruction. The ItemLocationBox('iloc') 845 associated with the metadata item for post-filter information indicates the location of the NAL cell (851-1, 851-2, and 851-3) corresponding to the NNPFC SEI message in the media data box 805 (or the item data box, or the identified media data box).

[0402] According to this embodiment, the sample group description entry (within 830-2) describing the post-processing filter can be expressed as follows:

[0403] aligned(8) class NnpfcSeiSampleGroupEntry() extendsVisualSampleGroupEntry('nfcs')

[0404] {

[0405] unsigned int(32) nnpfc_item_ID;

[0406] unsigned int(1) nnpfc_external_payload;

[0407] unsigned int(5) nnpfc_purpose;

[0408] unsigned int(1) nnpfc_complexity_info_present_flag;

[0409] unsigned int(1) reserved;

[0410] if (nnpfc_complexity_info_present_flag) {

[0411] unsigned int(32) nnpfc_num_kmac_operations_idc;

[0412] unsigned int(32) nnpfc_total_kilobyte_size;

[0413] }

[0414] }

[0415] It has the following semantics:

[0416] When nnpfc_external_payload is set to 1, it indicates that the payload for the post-processing filter is outside the file. When set to 0, the payload exists in the data section of the file, in the metadata item for post-filter information.

[0417] nnpfc_item_ID indicates the item identifier of the metadata item storing the SEINAL unit of the post-processing filter described in the entry for this sample group description.

[0418] nnpfc_purpose indicates the purpose of the post-processing filter as specified in Table 20 of ISO / IEC 23002-7.

[0419] The nnpfc_complexity_info_present_flag specifies whether one or more syntactic elements indicating the complexity of the post-processing filter associated with filter_id are present (when set to 1) or absent (when set to 0).

[0420] nnpfc_num_kmac_operations_idc indicates the maximum number of multiply-accumulate operations per sample as specified in ISO / IEC 23002-7.

[0421] nnpfc_total_kilobyte_size indicates the total size in kilobytes required to store the uncompressed parameters of a neural network as specified in ISO / IEC 23002-7.

[0422] It should be noted that the above sample group description entries may contain a general description from the post-filter information or other parameters from the syntax of the NNPFC SEI message (when useful for the application).

[0423] The samples to which post-processing filters are activated should be mapped to a sample-to-item ('stmi') sample group. The 'stmi' sample group associates the samples with one or more metadata items (as shown in 831, 832, and 833) of the post-filter information activated for the mapped samples. The order in which post-filters are applied to samples is their order within the 'stmi' sample group, unless the post-filter description contains parameters for an order or relationship different from the order (e.g., parallel or alternative). The scheme_type within the 'stmi' sample group should be set to 'meta'. The NAL unit corresponding to the NNPFA SEI message is stored in sample 805 of the video track. Optionally, metadata items (841, 842, 842-1, 841-1, and 841-2) can be associated with item attributes, such as attributes further describing the post-filters, via 'ipma' boxes 844.

[0424] In another embodiment that combines sample groups and metadata items, such as Figure 5 The metadata items on the page are used to describe the post-filter information and the location of its payload. However, the same metadata items are still used as described in the original text. Figure 4aThe post-filters depicted above (as any variation of box 414 or sample groups 415 and 416) describe in the media track the properties of a subset of the parameters exposing the NNPFC SEI message (as a possible list of parameters in the general post-filter information). It should be noted that box 416 may not exist when all samples are mapped to the same post-filter information, i.e., when using the default sample grouping or sample groups with a static mapping for the description of the post-filter information. The parser can then infer from this sample group (415 and 416 or only 415, or from box 414) which samples are likely associated with one or more post-filters. If the parser or application decides to use and apply a post-filter, it can access the payload (when present in the file, as indicated in a parameter in the post-filter description) through metadata items for the post-filter information (e.g., 541, 542, 541-1…). As illustrated in the previous embodiments, in order to identify the relevant metadata item (i.e., which post-filter(s) are applied to which samples), the parser can use the 'stmi' sample group in the media track. Once identified, the item location box provides a byte range of the payload for the (one or more) post-filters used, which can be copied to the reconstructed bitstream. In the case of NNPFC and NNPFA SEI messages indicating post-filter information, the payload may include NAL units for the NNPFC SEI message and / or NAL units for the NNPFA SEI message. Alternatively, as illustrated in the embodiment “Indication of Post-Filter Information as Information Items”, the units for the NNPFA SEI message may not be described as data for the metadata item, but rather inferred from the 'stmi' sample group description entries.

[0425] In an alternative where a first sample group is used for filter description and a second sample group is used to link media samples to metadata items, a new sample group can be used together with a sample group description entry, which, in addition to the post-filter description, includes an indication of an item identifier describing the post-filter payload. Then, for cases where the bitstream does not contain any payload for the post-filter (e.g., nnpfc_mode_idc is different from 0), the item identifier (e.g., nnpfc_item_ID) is optional. The presence or absence of this item identifier can be indicated in the general description of the post-filter by post_filter_without_payload or post_filter_external_payload. For cases where the bitstream does contain a payload for the post-filter, post_filter_without_payload or post_filter_external_payload can be set to false, and an item identifier can be provided to locate the payload in the data. Using this alternative, items may not have any associated item attributes that expose post-filter information, as these attributes are described in the new sample group description entry (e.g., the 'nfcs' sample group in the media track). However, when the base and update models are stored in different items, dependencies between metadata items describing post-filter information (e.g., as item references of type 'nnrd' or 'pfid') can be useful for locating the update and its base model in the data (e.g., in a MediaDataBox).

[0426] Indication of post-filter information as inference information

[0427] Possible representations of post-filter information as metadata items can rely on HEIF inference terms (instead of, for example, based on...). Figure 5The inferred information is a type of metadata item that describes 'cnnr'. Inferred information is image-related metadata that is processed by image analysis or generated manually. Image context analysis, object detection, or face detection are examples of algorithms that generate inferred information. Inferred information can be stored as an opaque data block in an item with an item type equal to "uri". Here, opaque means a specific format that a generic ISOBMFF or HEIF reader might not be able to handle. The item_uri_type of the ItemInfoEntry box should contain an absolute URI that uniquely identifies the specification (the format of the opaque data block) of the inferred information. To use an inferred item to describe post-filter information, this URI can be set to nnpfc_uri as indicated in the NNPFC SEI message when the post-filter appears as defined in the V-SEI specification. Inferred items can also be extended with tag_uri to support nnpfc_tag_uri, or associated with item attributes (e.g., the 'nnpf' item attribute on 547 (or the more generic 'pfif' item attribute)) that provide more parameters describing the post-filter (parameters or subsets of parameters listed in the generic post-filter information). Compared to existing designs that only allow association to 'uuid' attributes, associating inference terms with post-filter attributes improves interoperability. These are user-defined and therefore not interoperable in open systems where the writer and parser are not controlled by the same application provider or developer. By reusing 'stmi' sample sets as described in previous embodiments, inference for post-filters can be chained to samples of its application to handle post-filter activation and deactivation. The SEI NAL unit corresponding to the NNPFA SEI message can be stored as item data (as in the embodiment "Indication of Post-Filter Information as a Combination of Sample Sets and Information Items"), or it can be left unstored and reconstructed on demand when a sample mapping to a metadata item describing the post-filter information is reached (as in the embodiment "Indication of Post-Filter Information as an Information Item"). Inference terms for updates to post-filters can be referenced using a general term reference of type 'dpnd' or a specific reference type for post-filter dependencies (such as 'nnrd' for 'neural network representation dependency' or 'pfid' (4CC, and the names are just examples) for 'post-filter information dependency') to refer to the inference terms describing their underlying post-filters. When post-filter information appears as an SEI as specified in the V-SEI specification, the data of the inference terms can correspond to the NAL unit of the NNPFC SEI message. The processing will differ from current cascaded inference terms as defined in HEIF, where cascading or dependency will involve applying updates to the basis and then applying the results to the associated samples. This will not be a cascade as currently specified in HEIF.To avoid confusion, another reference type can then be used to indicate the combination of inferred terms before they are applied to associated terms or samples.

[0428] One advantage of describing post-filters as metadata items is that when a media track has one or more associated items (e.g., cover images or thumbnails or previews of video tracks), post-filter information can also be applied to these associated items to render improved versions of those items (e.g., enhanced thumbnails or previews). It should be noted that different, more or less generic `item_type`s can be used to describe metadata items that describe post-filter information. The more generic the `item_type`, the more exact the format for describing the post-filter information, the additional parameters required for the purpose (within the item description, e.g., in 'infe'), or the more additional attributes are available. Possible item_types, ranging from less general to more general, could be, for example: 'cnnr' for explicitly indicating a post-filter based on MPEG-7 NNR, 'pfii' or 'pfif' for a general post-filter (where associated attributes provide purpose, format, etc.), 'uri' which may have parameters indicating the format of the post-filter and possibly associated item attributes, or 'mime' which requires the correct value to be set in the content_encoding parameter (e.g., or in the associated item attributes).

[0429] Another advantage of storing post-filter data in items is that this data can be requested, loaded, read, or processed "on demand" by the reader or media player. This means the reader already knows the location of the different post-filter data from the file's top-level metadata. Then, when the reader decides to apply post-filtering, it can request or process this data only when needed (in cases where the file is exchanged or streamed over a communication network) or locally. For example, the reader can append the NAL unit representing the post-filter immediately before the sample where it decides to begin applying post-filtering. As long as the order between the base and updated filters ensures that the update always occurs after its base, there is no need to append this data from the beginning of the reconstructed bitstream.

[0430] Indication of post-filter information for a specific track

[0431] Figure 6An example is illustrated where post-filter information is encapsulated in a dedicated track 630. When a sample corresponds to one or more NNR units as specified in MPEG-7 Part 17, the track defines a specific sample entry type, such as 'cnnr' for compressed neural network representations. If the data corresponds to other data unit types (e.g., SEI NAL units from the V-SEI specification), the sample entry may have a different type. Then, for neural network-based post-filter samples, the sample entry could be 'nnpf'. The MediaData box 630 contains samples for media track 610 and for post-filter track 630. Track 610 may be a classic video track (e.g., a 'vvc1' track) plus a specific track reference indicating its association with the post-filter track. This can be indicated by a new track reference type such as 'pfdc' used for post-filter descriptions. In variations, track associations can use the same track reference type or a specific track reference type that does not conflict with other registered tracks from the post-filter track to the media track. Post-filter tracks have specific sample groups used to describe post-filter information, such as using sample grouping mechanisms as shown in 635 and 636. These sample groups have specific grouping_types, such as 'pfif', which provides a description of the post-filter information (similar to the 'nnpf' item attribute), or 'nfas' and 'nfcs' grouping_types when the track carries SEI NAL units for neural network-based post-filters with the same parameters as in the NFCS and NFAS sample group description entries. To associate media samples from track 610 with samples in post-filter track 630, the samples in track 630 can be time-aligned with the media samples, such that the samples in the media track are applied to the time-aligned samples in the media track. Several post-filter tracks and several time-aligned samples with media samples can exist, which can be interpreted as multiple post-filter operations on a given media sample. The order of filter operations can follow the order of track references from post-filter track 630 to media track 610.

[0432] It should be noted that, depending on the manner and type of information available to the encapsulation module at step 204, the encapsulation module may choose one of the above embodiments, for example, considering what information is provided in the media bitstream:

[0433] 1. If a post-filter description is not provided in the bitstream (test 202 is false), but a post-filter description can be provided out-of-band (step 203), the encapsulation module may preferably describe the post-filter information outside the media track using the method of the second or third embodiment.

[0434] 2. If a post-filter description is provided in the bitstream (test 202 is true), the encapsulation module may prefer the method of the first or second embodiment, and when the bitstream is a VVC bitstream with NNPFC and NNPFA SEI messages, the encapsulation module may prefer a variant with NFCS and NFAS sample grouping or an item-based method.

[0435] In the above embodiments, it should be noted that media track 410, 503, or 610 can be a video track, an audio track, a volumetric track (which may be a video-based track, such as a geometry or property track), or any media track that can be improved by a post-filtering operation. For example, media tracks encapsulating volumetric media (such as one or more video tracks (e.g., MPEG V-PCC or V-DMC)) can benefit from post-filtering operations such as converting decoded video frames to a nominal format (e.g., nominal resolution, bit depth, chroma format, etc.). This can be performed before the decoded frames are provided to the volumetric media for reconstruction processing, according to reference... Figure 4a , 4b and 4c or Figure 5 or Figure 6 In one of the described embodiments, individual video tracks or subsets of one or more video tracks of the encapsulated volume media can be associated with post-filter information. Furthermore, a post-filter identifier is specified as an identifier for the post-filter. However, if the bitstream provides identifiers for filter groups to be applied to the same set of samples, the above description also applies; the post-filter identifier is the identifier for the filter group. For example, when used with NNPFC and NNPFA SEI messages, the identifier for a post-filter group can be declared in the NNPFC SEI message and used in some NNPFA SEI messages to activate or deactivate that post-filter set. When post-filters are grouped, there may be additional parameters describing the characteristics of the filter set, such as complexity or processing requirements. This can be considered in the general description of the post-filter information as a parameter that then describes the complexity of the group, or as a loop for the individual post-filters of the group to provide the complexity for the individual post-filters of the group. Some processing rules can be described in the NNPFC SEI message, such as whether the filters are applied sequentially, in parallel, or should be considered alternatives. These are parameters, for example, in the NNPFC SEI message, which can also be provided in the general description of the post-filter information.

[0436] Figure 7 This is a schematic block diagram of a computing device 700 for implementing one or more embodiments of the present invention. The computing device 700 may be a device such as a microcomputer, workstation, or lightweight portable device. The computing device 700 includes a communication bus 702 connected to:

[0437] - Central Processing Unit (CPU) 704, such as a microprocessor;

[0438] - Random access memory (RAM) 708, for storing executable code of the methods of embodiments of the present invention and registers adapted to record variables and parameters required for implementing methods for encapsulating, indexing, decapsulating and / or accessing data, the memory capacity of which can be expanded, for example, by connecting to an optional RAM via an expansion port;

[0439] - Read-only memory (ROM) 706 for storing computer programs for implementing embodiments of the present invention;

[0440] - A network interface 712, which is typically connected to a communication network 714, transmits or receives digital data to be processed on the communication network 714. The network interface 712 can be a single network interface or a set of different network interfaces (e.g., wired and wireless interfaces, or different types of wired or wireless interfaces). Under the control of a software application running in the CPU 704, data is written to the network interface for transmission or read from the network interface for reception.

[0441] - User Interface (UI) 716, used to receive input from the user or display information to the user;

[0442] - Hard disk (HD) 710; and / or

[0443] - I / O module 718 is used to receive / send data from / to external devices such as video sources or displays.

[0444] The executable code can be stored in read-only memory 706, hard disk 710, or removable digital media (such as a disk). According to a variation, the executable code of the program can be received via network interface 712 through a communication network and stored in one of the storage components (such as hard disk 710) of the communication device 700 before being executed.

[0445] The central processing unit 704 can be adapted to control and direct the execution of instructions or portions of software code of one or more programs according to embodiments of the present invention, these instructions being stored in one of the aforementioned storage components. For example, upon power-up, the CPU 704 is capable of executing software application-related instructions from the main RAM memory 708 after instructions have been loaded from the program ROM 706 or the hard disk (HD) 710. Such a software application, when executed by the CPU 704, causes the steps of the flowchart shown in the previous figures to be performed.

[0446] In this embodiment, the device is a programmable device that implements the invention using software. However, alternatively, the invention can be implemented in hardware (e.g., in the form of an application-specific integrated circuit or ASIC).

[0447] Although the invention has been described above with reference to specific embodiments, the invention is not limited to the specific embodiments, and modifications falling within the scope of the appended claims will be apparent to those skilled in the art.

[0448] Many further modifications and variations will arise for those skilled in the art when referring to the foregoing illustrative embodiments. These embodiments are given by way of example only and are not intended to limit the scope of the invention, which is defined only by the appended claims. In particular, different features from different embodiments may be interchanged where appropriate.

[0449] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude plural. The fact that different features are recited only in mutually different dependent claims does not indicate that a combination of these features cannot be used advantageously.

Claims

1. A method for encapsulating media data into an ISOBMFF-based media file, the media data comprising media data units and at least one post-filter data unit, the method comprising: A first track is generated to form a sequence of encapsulated samples, each sample including at least one of the media data units, and the at least one post-filter data unit is applied to at least one of the samples in the first track; Generate a data portion of an entity that includes at least a portion of one of the post-filter data units; Generate post-filter information describing the at least one post-filter data unit; as well as The first track, the entity, and the post-filter information are encapsulated in the media file, and the post-filter information is encapsulated in the metadata portion of the media file.

2. The method according to claim 1, wherein, The entity is the first track.

3. The method according to claim 1, wherein, The entity is an item associated with the first track.

4. The method according to claim 1, wherein, The entity is the second track associated with the first track.

5. The method according to claim 1, wherein, The post-filter information includes general information indicating the presence of post-filter information.

6. The method according to claim 5, wherein, The general information regarding the presence of filter information after indication is that it is a proprietary brand.

7. The method according to claim 5, wherein, The post-filter information also includes information indicating the purpose of the post-filter information.

8. The method according to claim 5, wherein, The post-filter information also includes information indicating the complexity of the post-filter information.

9. The method according to claim 2, wherein, Post-filter information is provided in the sub-box of the first track metadata.

10. The method according to claim 2, wherein, Post-filter information is provided in the sample entries of the first track metadata.

11. The method according to claim 2, wherein, Post-filter information is provided in the sample group of the first track metadata.

12. The method according to claim 3, wherein, Post-filter information is provided in the metadata section of the item.

13. The method according to claim 4, wherein, Post-filter information is provided in the metadata section of the second track.

14. A method for generating a bitstream of media data from an ISOBMFF-based media file, the method comprising: The first track of the sequence of encapsulated samples is obtained from the media file, each sample including at least one media data unit; Post-filter information describing at least one post-filter data unit is obtained from the metadata portion of the media file, the at least one post-filter data unit being applied to at least one of the samples of the first track; At least a portion of one of the post-filter data units is obtained from the data portion of the entity in the media file; as well as The bitstream of the media data is generated based on the first track, the entity, and the post-filter information in the media file.

15. A computer program product for a programmable device, the computer program product comprising a sequence of instructions for implementing the method according to any one of claims 1 to 14 when loaded into and executed by the programmable device.

16. A computer-readable storage medium storing instructions of a computer program for implementing the method according to any one of claims 1 to 14.

17. A computer program that, when executed, causes the method according to any one of claims 1 to 14 to be performed.

18. An apparatus for encapsulating media data into an ISOBMFF-based media file, the media data including media data units and at least one post-filter data unit, the apparatus including a processor configured to: A first track is generated to form a sequence of encapsulated samples, each sample including at least one of the media data units, and the at least one post-filter data unit is applied to at least one of the samples in the first track; Generate a data portion of an entity that includes at least a portion of one of the post-filter data units; Generate post-filter information describing the at least one post-filter data unit; as well as The first track, the entity, and the post-filter information are encapsulated in the media file, and the post-filter information is encapsulated in the metadata portion of the media file.

19. An apparatus for generating a bitstream of media data from an ISOBMFF-based media file, the apparatus comprising a processor configured to: The first track of the sequence of encapsulated samples is obtained from the media file, each sample including at least one media data unit; Post-filter information describing at least one post-filter data unit is obtained from the metadata portion of the media file, the at least one post-filter data unit being applied to at least one of the samples of the first track; At least a portion of one of the post-filter data units is obtained from the data portion of the entity in the media file; and The bitstream of the media data is generated based on the first track, the entity, and the post-filter information in the media file.

Images