Processing a multilayer video stream

Description

1 / 68 “PROCESSING A MULTILAYER VIDEO STREAM” TECHNICAL FIELD

[0001] The present invention relates to the processing of a multilayer video stream. BACKGROUND

[0002] Multilayer video coding schemes have existed for several years, but have faced problems with widespread adoption. Much of the video content on the Internet is still encoded using H.264 or MPEG-4 Part 10, Advanced Video Coding (MPEG-4 AVC), this format being used by between 80% and 90% of online video content. This content is typically provided to decoding devices as a single video stream that has a one-to-one relationship with available hardware and / or software video decoders, for example, a single stream is received, analyzed and decoded by a single video decoder to produce a reconstructed video signal. Many video decoder implementations are then developed according to this structure.

[0003] Existing multi-layer coding schemes include the Scalable Video Coding (SVC) extension for H.264, Scalable extensions for H.265 or High Efficiency Video Coding (SHVC) MPEG-H Part 2, and more recent standards such as Low Complexity Enhancement Video Coding (LCEVC) MPEG-5 Part 2. While H.265 is a development of the coding framework used by H.264, LCEVC takes a different approach to scalable video. SVC and SHVC operate by creating different coding layers and feeding each one with a different spatial resolution. Each layer encodes the input according to a normal AVC or HEVC encoder with the possibility of leveraging information generated by the lower coding layers. LCEVC, on the other hand, generates one or more layers of enhancement residue compared to a base coding, where the base coding may have a lower spatial resolution.

[0004] One reason for the slow adoption of schemes Petition 870250110555, dated 02 / 12 / 2025, p. 9 / 85 2 / 68 Multilayer encoding has been a challenge in adapting new and existing decoders to process encoded multilayer streams. As discussed above, video streams are typically single data streams that have a one-to-one pairing with a suitable decoder, whether implemented in hardware or software or a combination of both. Client devices and media players, including internet browsers, are therefore built to receive a data stream, determine which video encoding the stream uses, and then pass the stream to an appropriate video decoder. Within this framework, multilayer schemes such as SVC and SHVC are generally packaged as larger single video streams containing multiple layers, where these streams can be detected as “SVC” or “SHVC” and passed atomically to an SVC or SHVC decoder for reconstruction.This approach often diminishes some of the benefits of multilayer encoding. Consequently, many developers and engineers have concluded that multilayer encoding schemes are too complicated and instead revert to a multicast of single H.264 video streams.

[0005] It is therefore desirable to obtain an improved method and system for decoding multilayer video data that overcomes some of the disadvantages discussed above and that allows more benefits of multilayer encoding schemes to be realized.

[0006] The article “The Scalable Video Coding Extension of the H.264 / AVC Standard”, by Heiko Schwarz and Mathias Wien, published in IEEE Signal Processing Magazine 135, March 2008, provides an overview of the SVC extension.

[0007] The article “Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard”, by Jill Boyce, Yan Ye, Jianle Chen, and Adarsh ​​K. Ramasubramonian, as published in IEEE Transactions on Circuits and Systems for Video Technology, VOL. 26, No. 1, January 2016, provides an overview of SHVC extensions.

[0008] The decoding technology for LCEVC is Petition 870250110555, dated 02 / 12 / 2025, page 10 / 85 3 / 68 defined in the Preliminary Text of the ISO / IEC FDIS 23094-2 document, as published at the 129th MPEG Meeting in Brussels in January 2020, as well as in the Approved Final Text and in document WO 2020 / 188273 A1. Figure 29B of document WO 2020 / 188273 A1 describes a hypothetical reference decoder where a demultiplexer provides a base bitstream to a base decoder and an enhancement bitstream to an enhancement decoder.

[0009] Document WO 2020 / 016562 A1 describes methods for decoding a bitstream. In some examples, a version of an original signal is encoded using a base encoding standard, and one or more layers of a residual stream can be encoded using an enhancement encoding. In a first example, the encoded residual stream or streams are encapsulated in a SEI (Supplemental Enhancement Information) message. In a second example, the encoded residual stream or streams can be transmitted as a series of Network Abstraction Layer (NAL) units interleaved with NAL units of an encoded basestream.

[0010] With multilayer streams, there is also a general stream management problem. Different layers of a multilayer stream can be generated together or separately, and can be provided together or separately. It is desirable to have improved methods and systems for decoding these multilayer streams. For example, it is desirable to allow content distributors to easily and flexibly modify video quality by adding additional layers in a multilayer scheme. It is also desirable to be able to flexibly remultiplex multilayer video streams without interrupting downstream single-layer or multilayer decoding.

[0011] In a way, a “single-track” solution can reduce the number of flow management problems compared to a multi-track solution (e.g., dual-track). However, it is difficult to provide a single-track solution that is compatible with older devices. This is related to the problem of providing multi-layered flows as formats of Petition 870250110555, dated 02 / 12 / 2025, page 11 / 85 4 / 68 Static files. For example, video streams can be read from fixed or portable media, such as solid-state devices or portable disks, or downloaded and stored as a file for later viewing. Supporting the transport of multi-layered video with existing file formats is difficult because these file formats typically assume a one-to-one mapping with media content and decoding settings, whereas multi-layered streams can use different decoding settings for different layers. Changes to file formats generally don't work in practice, as they require hardware and software decoding updates and can affect the decoding of older formats.

[0012] All publications listed above should be incorporated into this document by reference. SUMMARY OF THE INVENTION

[0013] Aspects of the present invention are presented in the appended independent claims. Variations of these aspects are presented in the appended dependent claims.

[0014] According to a first aspect, a method can be provided comprising: adding an enhancement sample entry to a sample description contained in a media data, such that: an older player is configured to play back database data associated with a database sample entry in said sample description; and an enhanced player is configured to play back enhancement data associated with said enhancement sample entry and database data associated with said database sample entry.

[0015] The method can be a method for storing multimedia data. The method can be a method for describing the format of multimedia data. The sample description can be a type of data structure called a box. The sample description can be an example of a data structure. The sample description can be an example of a box-type data structure. The sample enhancement input can be an example of a structure of Petition 870250110555, dated 02 / 12 / 2025, page 12 / 85 5 / 68 box-type data. The sample augmentation input can be a secondary box structure.

[0016] The method involves adding an enhancement sample entry to an existing data structure, in the form of a sample description box, to provide backward compatibility with single-track multimedia files.

[0017] The method may also include adding the base sample entry to the sample description. Thus, the base sample entry is added to the existing data structure (which is the sample description box). In this way, the base sample entry and the enhancement sample entry are added to the same data structure, that is, the sample description box. The sample description box may include an entry_count data field, and the method may also include increasing the entry count of the sample description box by a number equal to the number of enhancement flows in the multimedia data.

[0018] The entry_count data field can have a value that corresponds to a number of secondary data structures within the sample description box. Each of the secondary data structures can be a sample entry box. The method can include defining a value for the entry_count data field equal to the number of separate bitstreams within the multimedia data, wherein the multimedia data comprises a base bitstream and / or at least one enhancement bitstream. In this way, the number of secondary data structures (i.e., sample entry boxes) in the sample description box is the same as the number of bitstreams in the multimedia data.

[0019] When the multimedia data includes a base stream and a single LCEVC stream, the method may further understand the definition of the entry_count value as 2. When the media data comprises a base stream and enhancement streams, the method may understand the definition of the entry_count value as n+1, where n is an integer greater than or equal to 0. The sample description may understand n enhancement sample input boxes. Petition 870250110555, dated 02 / 12 / 2025, page 13 / 85 6 / 68

[0020] The method may also include adding the base sample entry to the sample description before adding the enhancement sample entry.

[0021] The method may further comprise adding a secondary box structure to the base sample input, preferably when the secondary box structure is an MPEG4 Extension Descriptor box.

[0022] The method may also include filling the MPEG4 extension descriptor box with the enhancement sample entry. In this way, the enhancement sample entry is added to an existing data structure, in the form of the MPEG4 extension descriptor box, to provide backward compatibility with one-way multimedia files.

[0023] The sample enhancement entry can be a descriptor for more than one enhancement stream associated with the media data.

[0024] According to another aspect of the invention, a data stream may be provided comprising a sample description comprising one or more enhancement sample entries.

[0025] The sample description may include a base sample entry.

[0026] The sample description may include an entry_count data field, and an entry_count value may be greater than one. The entry_count value may be equal to a number that is the sum of a number from one or more enhancement sample entries plus a number of base sample entries included in the sample description.

[0027] When the entry_count value is equal to 2, the sample description may include a single enhanced sample entry and a single base sample entry. When the entry_count value is greater than 2, the sample description may include more than one enhanced sample entry and a single base sample entry.

[0028] The basic sample entry may include Petition 870250110555, dated 02 / 12 / 2025, page 14 / 85 7 / 68 an MPEG4 extension descriptor.

[0029] The MPEG4 extension descriptor comprises said one or more enhancement sample entries.

[0030] In another example of implementing the present disclosure, a method may be provided comprising: adding an enhancement sample entry to a data structure contained in a media data, such that: an existing player is configured to play back database data associated with a base sample entry in said data structure; and an enhanced player is configured to play back enhancement data associated with said enhancement sample entry and database data associated with said base sample entry. The data structure may be a sample box or a sample description.

[0031] In another example of implementing the present disclosure, a method can be provided for generating a single-track multilayer video stream, wherein the video stream comprises a base bitstream and at least one enhancement bitstream, the method comprising: inserting an enhancement-level decoder configuration into a box containing the base decoder configuration, wherein the enhancement-level decoder configuration is associated with the base decoder configuration.

[0032] The box can be a data structure.

[0033] The box might be a sample description box, SampleDescriptionBox. The box might be a box for an MPEG file format, for example, an mp4 file format, and in another example, according to ISO / IEC 14496-15.

[0034] The box may contain an entry_count variable that indicates the number of sample entry boxes within the box.

[0035] Before the step of inserting the enhancement-level decoder configuration, the method may involve adding a base sample input box within the box, where the input box Petition 870250110555, dated 02 / 12 / 2025, page 15 / 85 8 / 68 base sample includes the base decoder configuration.

[0036] The method may involve setting the value of the entry_count variable equal to the number of separate bitstreams within the video stream.

[0037] When the video stream comprises an enhancement bitstream, the method may include setting the value of the entry_count variable to 2.

[0038] When the video stream comprises n bit streams of enhancement, the method may understand the definition of the value of the entry_count variable as n+1, where n is greater than 1.

[0039] The insertion may comprise increasing a value of the entry_count variable and adding an enhancement sample input box within the box, wherein the enhancement sample input box comprises the enhancement level decoder setting.

[0040] The insertion may involve adding the enhanced sample input box to the box after the base sample input box.

[0041] The method may also include adding an extension box to the input box of the base sample.

[0042] The extension box may be a description of the MPEG4 extension.

[0043] The method may further comprise adding an enhancement sample input box to the extension box, wherein the enhancement sample input box comprises the enhancement level decoder configuration.

[0044] The insertion may comprise the addition of an extension box to the base sample input box and the addition of an enhancement sample input box to the extension box, wherein the enhancement sample input box comprises the decoder configuration. Petition 870250110555, dated 02 / 12 / 2025, page 16 / 85 9 / 68 enhancement level.

[0045] In another example of an implementation of the present disclosure, a data stream may be provided comprising a box, wherein the box comprises a base decoder configuration and at least one enhancement-level decoder configuration, wherein the enhancement-level decoder configuration is associated with the base decoder configuration.

[0046] The box can be a sample description box, SampleDescriptionBox.

[0047] The box may contain an entry_count variable that indicates a number of sample entry boxes within the box.

[0048] When the value of the entry_count variable is 1, the box may contain a basic sample entry box, where the basic sample entry box contains the basic decoder configuration.

[0049] A value of the entry_count variable can be defined as being equal to the number of separate bit streams within the data stream.

[0050] When the data stream comprises an enhancement bit stream, the value of the entry_count variable can be set to 2.

[0051] When the video stream comprises n enhancement bitstreams, the value of the entry_count variable can be set to n+1, where n is greater than 1.

[0052] When the entry_count variable is 2, the box may comprise an enhancement sample input box, wherein the enhancement sample input box comprises the enhancement level decoder configuration.

[0053] The enlargement sample input box may be after the base sample input box.

[0054] The basic sample input box may comprise an extension box.

[0055] The extension box can be a description of Petition 870250110555, dated 02 / 12 / 2025, page 17 / 85 10 / 68 MPEG4 extension.

[0056] The extension box may comprise at least one enhancement sample input box, wherein the enhancement sample input box comprises the enhancement level decoder configuration.

[0057] The enhancement sample input box can be a descriptor for more than one enhancement bitstream associated with the datastream.

[0058] According to another aspect of the invention, a device configured to perform the method described above can be provided.

[0059] According to another aspect of the invention, a device configured to decode the data stream as described above can be provided.

[0060] According to another aspect of the invention, a computer-readable means can be provided comprising instructions which, when executed, cause a processor to execute the method as described above. BRIEF DESCRIPTION OF THE FIGURES

[0061] Figure 1 is a schematic diagram showing an early example of a multilayer video stream comprising interleaved NAL units.

[0062] Figure 2 is a schematic diagram showing a second example of a multilayer video stream comprising base-layer NAL units and enhancement SEI messages.

[0063] Figure 3 is a flow diagram that shows an example method for analyzing a multilayer video stream.

[0064] Figure 4 shows a set of bit configurations for NAL header data according to an example.

[0065] Figures 5 and 6 are schematic diagrams that show, respectively, an example of a multilayer encoder and decoder configuration. Petition 870250110555, dated 02 / 12 / 2025, page 18 / 85 11 / 68

[0066] Figure 7 is a schematic diagram showing certain data processing operations performed by an example of a multilayer encoder.

[0067] Figure 8 is a schematic diagram showing an example of data flow comprising a sample description with one or more enhanced sample entries.

[0068] Figure 9 is a schematic diagram showing an example of an additional data flow comprising a sample description comprising one or more enhancement sample entries. DETAILED DESCRIPTION

[0069] Certain examples described in this document allow decoding devices to be easily adapted to handle multilayer video encoding schemes. Certain examples are described with reference to an LCEVC multilayer video stream, but the general concepts can be applied to other multilayer video schemes.

[0070] Different examples are presented. In one set of examples, a method for analyzing a multilayer video stream is described. The method can be seen as a way to adapt an existing analyzer to handle a multilayer video stream. The existing analyzer may initially be configured to analyze a layerless video stream, for example, a conventional video encoding where a single stream of NAL units contains all the data for decoding a video signal. In the examples, the analyzer is modified to handle a multilayer video stream where the data for decoding a video signal is received in different streams of NAL units or through different transport mechanisms. Thus, the examples allow multilayer video streams to be stored and / or transmitted as if they were conventional layerless video streams.

[0071] The current examples allow the transport of a base video bitstream and an enhancement video bitstream in Petition 870250110555, dated 02 / 12 / 2025, page 19 / 85 12 / 68 a single “track”, for example, in a data structure that resembles a bitstream for only the base video bitstream. For example, a single bitstream containing both a base video bitstream and an enhancement video bitstream may have a single Packet Identifier (PID) in a transport stream or a single Track Identifier (Track ID) in a file format – thus resembling conventional non-enhanced bitstreams with a single PID or TrackID.

[0072] The current examples provide an alternative approach to the so-called “dual track” transport of base and enhancement bitstreams. In the “dual track” case, a base video bitstream and an enhancement video bitstream can be transmitted and / or stored as two separate bitstreams, for example, with separate PIDs or on separate tracks.

[0073] The present examples can be applied to multilayer video bitstreams configured according to the specifications of an MPEG-2 Transport Stream (TS) and / or an MPEG-4 File Format (FF). Reference to a multilayer video bitstream may include reference to a multilayer video bitstream that is suitable for transport via any of the following: MPEG-2 TS, MPEG-4 FF, Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH), or Common Media Application Format (CMAF).

[0074] The present examples can be applied to an enhancement bitstream that conforms to the ISO / IEC 23094-2 standard specification – the so-called MPEG-5 Low Complexity Enhancement Video Codec (LCEVC). In general, the enhancement bitstream is configured to enhance a standalone “base” bitstream, where the base bitstream is a video encoding that can be independently decoded (and viewed, if desired) by a base decoder. The enhancement bitstream may include so-called residual bitstream, wherein the frames reconstructed from the enhancement bitstream comprise Petition 870250110555, dated 02 / 12 / 2025, page 20 / 85 13 / 68 residual frames (i.e., sets of residual frames) are combined with frames decoded from the base bitstream to enhance the quality of the base bitstream. In certain cases, specific layers of residual frames can be combined with reconstructed signals with higher sampling. For example, in LCEVC, there is a first enhancement layer, which may be at the resolution of the reconstructed decoded baseframe and is capable of correcting encoding artifacts, and a second enhancement layer, which is at a higher resolution than the reconstructed decoded baseframe, where a residual frame at the higher resolution can be combined after higher sampling of a reconstructed decoded baseframe that is corrected with a residual frame from the first enhancement layer. This process is described in more detail with reference to Figures 5 to 7 below.

[0075] In enhancement specifications, such as LCEVC, an enhancement bitstream is defined that improves a separately defined video bitstream (the base). A process is defined to combine the two bitstreams to produce an enhanced video output. Since the two bitstreams can be decoded independently, the transport and storage of the base and enhancement bitstreams can be performed using a dual-track approach with two separate streams (PIDs or tracks) that can be linked by a dependency mechanism. The use of this approach and mechanism can be facilitated by existing dependency mechanisms, such as those present in MPEG4 FF and already used for the transport of other standard formats.The dual-track approach can facilitate the transport of a base bitstream and an enhancement bitstream over two separate physical channels (e.g., the base bitstream via an overhead channel and the enhancement bitstream via a broadband channel). However, the dual-track approach may not be suitable for all implementations. For example, there are many commercial media player implementations, especially for video streaming, where the media player only supports playback of a single stream. Furthermore, the packaging of the stream... Petition 870250110555, dated 02 / 12 / 2025, page 21 / 85 14 / 68 bits as a single stream can simplify the design of future media players and allow for wider adoption of multi-layered video streams. Therefore, there are technical advantages to supporting both single-track and dual-track approaches. Thus, the present examples consider technical solutions that can support single-track approaches.

[0076] In certain examples described in this document, a base bitstream and an enhancement bitstream, which may be called a multilayer video bitstream, may be transported and / or stored as a single bitstream. For example, this may comprise a single MPEG2 TS transport stream with a single PID or a single MPEG4 FF track. The layers of this multilayer bitstream may comprise the “base” as the first layer and one or more enhancement layers (LCEVC has the option of up to two enhancement layers). In some examples, the base bitstream is generated according to one of the following MPEG video encoding specifications: Advanced Video Coding (AVC - also known as ISO / IEC 14496-10 or H.264), High Efficiency Video Coding (HEVC - also known as ISO / IEC 23008-2 or H.265), and Versatile Video Coding (VVC - also known as ISO / IEC 23090-3 or H.266).Other past and future video encoding specifications may also be supported due to the independence of the enhancement layers – they can be generated based on any input frame from a base layer decoding. As an example, a single “track” multilayer video stream may comprise a base bitstream (e.g., VVC) plus an LCEVC bitstream as a single track in terms of MPEG2 TS PID or MPEG4 FF track.

[0077] Figures 1 and 2 show, respectively, two alternative implementations of a single-track multilayer video stream. Both implementations provide a complete multilayer video stream comprising a base bitstream and an enhancement bitstream. Figure 1 shows an example with interleaved NAL units. Figure 2 shows an example where the enhancement bitstream is carried in a set Petition 870250110555, dated 02 / 12 / 2025, page 22 / 85 15 / 68 of SEI messages.

[0078] . In example 100 of Figure 1, the NAL units for the base bitstream 104 and the enhancement bitstream 106 (in this case, LCEVC) are interleaved to provide a single-track multilayer video stream 102. Because enhancement bitstreams can be generated for a variety of different base video coding specifications, each base video coding specification may define its own NAL unit syntax, and this syntax may vary between specifications. The NAL units for the enhancement bitstream 106 may be NAL units as defined in section 7.3.2 of ISO / IEC 23094-2.

[0079] In the examples presented, the format of the enhancement NAL units can be specified to allow unambiguous detection, even when parsed according to the NAL unit syntax of a base bitstream, where the base bitstream can vary according to a number of different base specifications (e.g., base NAL units can be formatted and processed according to any of the NAL unit syntaxes AVC, HEVC, or VVC). This property allows a single “interleaved” base-plus-enhancement stream, where the enhancement NAL units are inserted between the base NAL units, within the same sequence of NAL units (e.g., as shown in Figure 1).In the resulting interleaved single stream, each Access Unit, defined as the set of NAL units that result in each decoded image, will contain the enhancement NAL units (for example, an LCEVC access unit contains only one LCEVC NAL unit) and the base NAL units relevant to the specific Access Unit.

[0080] Figure 2 shows an alternative to the interleaved approach of Figure 1. In example 200 of Figure 2, the enhancement data is encapsulated as NAL units, for example, as defined in section 7.3.2 of ISO / IEC 23094-2, which is then encapsulated with 216 metadata information for base 214 NAL units to generate a video bitstream. Petition 870250110555, dated 02 / 12 / 2025, page 23 / 85 16 / 68 single-track multilayer 102. In specific examples, metadata information 216 may comprise SEI messages, for example, as defined by any of the AVC, HEVC, and VVC. Therefore, in Figure 2, NAL units are employed as base data units, and the NAL unit type identified as Supplemental Enhancement Information (SEI) is used to incorporate a stream of enhancement NAL units. SEI messages can be full-size or large-size. In both cases of Figure 1 and Figure 2, the bitstream encapsulation can be within the MPEG2 TS “.ts” and MPEG4 FF “.mp4” data structures.

[0081] In certain examples described in this document, a base bitstream parser is modified to process a multilayer video bitstream, such as one of the bitstream examples shown in Figures 1 and 2. A parser may comprise one or more software and hardware components arranged to process data received from one or more file and network sources. For example, the parser may receive bits read from a data storage device via a file system on a computing device and / or bits emitted by a network processing stack configured to process data received via a network connection (e.g., wired or wireless). The parser may be located at the beginning of a media processing pipeline before one or more decoding components. The one or more decoding components may comprise implementations of encoding-decoding components—that is, codecs.These codecs can be implemented through computer program code processed by a processor and / or through specifically configured electronic hardware (such as Application-Specific Integrated Circuits - ASICs - and Field-Programmable Gate Arrays - FPGAs). Often, codecs can integrate hardware-accelerated calculations, that is, calculations that use specially configured electronic chips to speed up the operations performed during decoding. In certain cases, an analyzer may understand part of a specific codec (for example, an AVC analyzer may understand one). Petition 870250110555, dated 02 / 12 / 2025, page 24 / 85 17 / 68 first stage in an AVC codec); in other cases, an analyzer may be provided separately.

[0082] In a comparative case, a comparative analyzer (e.g., “unmodified”) for video coding that receives bits of data structured according to the syntax of a given video coding standard. For example, an AVC analyzer might form an initial component or the first component in an AVC decoding pipeline. This comparative analyzer might receive NAL units for a single video coding specification (e.g., AVC, HEVC, or VVC NAL units) and process these units to extract encoded data to be transmitted to specific decoding components. For example, a comparative analyzer might be configured to process NAL units that comprise data encoded according to a first video coding specification, such as AVC, HEVC, or VVC.The analyzer may comprise a memory for storing NAL unit header data belonging to a video stream and a processor configured to extract a NAL unit type value from the header data for a given NAL unit. In response to the NAL unit type value being within a specified range for the first video encoding specification, the processor may be configured to instruct the decoding of a given NAL unit payload according to the first video encoding specification. For example, in normal operation, an AVC, HEVC, or VVC analyzer receives AVC, HEVC, or VVC NAL units, inspects the header, and performs checks according to the relevant video encoding specification before passing the encoded data to a decoder that implements the video encoding specification.In this comparative case, if the comparative analyzer finds a NAL unit with a NAL unit type that is outside a predefined range (for example, as discussed later), the analyzer is configured to discard the NAL unit (for example, to not pass the payload data for decoding according to the encoding specification). Petition 870250110555, dated 02 / 12 / 2025, page 25 / 85 18 / 68 video from the analyzer). In specific examples, the preset range is a set of values ​​defined as “unspecified” in each video encoding specification.

[0083] In the present examples, a modified parser is presented, where the parser is modified or adapted to process a multilayer video stream. In these examples, the parser may comprise a parser that was previously configured to process a single-track video stream for a single, layerless video encoding specification, for example, an AVC, HEVC, or VVC parser. In other examples, a parser may comprise a newly provided parser that is capable of handling a multilayer video stream comprising multiple video encoding specifications (for example, in addition to conventional single-layer video streams). In both cases, the result is a parser configured to process a multilayer video stream comprising multiple video encoding specifications.

[0084] In general, in the current examples, an analyzer is configured to analyze the data associated with the received NAL units. The analyzer is configured to identify whether the NAL unit relates to a first video encoding specification (e.g., “base”) or whether the NAL unit relates to a further enhancement video encoding specification. This can be performed by inspecting the values ​​with the NAL unit metadata. In some examples, the NAL units for the enhancement bitstream are configured to indicate a metadata value within an unspecified range for the base layer specification for a plurality of different base layer specifications, for example, as unspecified for two or more AVC, HEVC, and VVC.Thus, if an unmodified analyzer receives a single-track multilayer video stream, only the data from the NAL units for the base bitstream will be decoded; the data from the NAL units for the enhancement bitstream will be discarded in accordance with the first video coding specification. Petition 870250110555, dated 02 / 12 / 2025, page 26 / 85 19 / 68 The bitstream can therefore be decoded and viewed as a normal base bitstream. However, a modified parser is configured to additionally pass the data from the NAL units to the enhancement bitstream to an enhancement decoder for decoding (e.g., in parallel) with data for the first video coding specification. In this way, the modified parser allows the decoding of both the base and enhancement bitstreams and the eventual viewing of a decoded and enhanced video output. In specific examples, different values ​​within an unspecified range for the first video coding specification may have different meanings or uses for the enhancement video coding specification.Therefore, a check can be made to determine if the “unspecified” values ​​correspond to one or more values ​​defined in the enhancement video coding specification. In this way, the decoding of the NAL unit by an enhancement decoder can occur in response to the fact that the NAL unit type value is within a predetermined part of a range not specified for the first video coding specification (seen as a base layer specification in relation to the enhancement video coding specification).

[0085] Figure 3 shows an example method 300 of analyzing a multilayer video stream. In this example, the multilayer video stream encodes a video signal and comprises a base layer and one or more enhancement layers. The base layer is encoded using a first layer or base layer specification (e.g., AVC, HEVC, or VVC). The one or more enhancement layers are encoded using an enhancement layer specification. The one or more enhancement layers may comprise one or more frame residue sets, wherein a frame residue set comprises a difference between a reconstruction derived from a decoded base layer frame and a frame derived from an original video signal at a given quality level. For example, one or more enhancement layers may comprise layers of Petition 870250110555, dated 02 / 12 / 2025, page 27 / 85 20 / 68 LCEVC, as described in Figures 5 to 7 below.

[0086] The example method 300 in Figure 3 can be executed as part of an analysis operation performed by a modified parser for the base layer. The analysis operation can comprise the analysis of header data for NAL units belonging to the multilayer video stream. For example, the analysis operation can be performed for each NAL unit 104, 106 in Figure 1 as they are received at a decoding device. The analysis operation can also be applied to the base NAL unit 214 and to an enhancement NAL unit that is extracted from an SEI message 216. The analysis operation can be performed iteratively on each received NAL unit. Figure 3 shows the sub-operations that can be performed on each NAL unit as part of this analysis operation.

[0087] In Figure 3, at step 302, a NAL unit type value is extracted from the header data of a given NAL unit (i.e., a currently parsed NAL unit). Syntax examples for NAL unit types for AVC, HEVC, and VVC are presented in more detail below. The header data may comprise a NAL header. NAL headers may comprise one or two bytes. The extraction of the NAL unit type value may depend on the base layer syntax, and different base layers may implement different syntaxes. In many base video specifications, the NAL unit type is represented by a defined nal_unit_type field, which is represented by a plurality of bits in the NAL header.

[0088] In step 304, the extracted value of the NAL unit type is compared to one or more predefined value ranges. In the base layer specification, values ​​can be defined as “specified” or “unspecified”. A “specified” value range indicates that these values ​​are used (and have meaning within) the base layer specification. Therefore, in Figure 3, in step 306, in response to the fact that the NAL unit type value Petition 870250110555, dated 02 / 12 / 2025, page 28 / 85 21 / 68 being within a range specified for the base layer specification, the decoding of a given NAL unit payload is instructed in accordance with the base layer specification (e.g., as per the comparative case described above).

[0089] In an unmodified case, if the nal_unit_type value is not within a specified range for the base layer specification (for example, it is “unspecified”), then the data for the NAL unit will be discarded and a next NAL unit will be obtained for analysis. However, in the current modifications, the indication of an unspecified NAL unit type prompts further analysis of the NAL unit. In particular, in step 308, a determination is made to verify whether the NAL unit type value indicates a NAL unit from the enhancement layer. For example, this may involve checking whether the NAL unit type value is within a predetermined part of an unspecified range for the base layer specification, i.e., whether it has one or more predefined “unspecified” values.In the case of a stroke analyzer, these one or more predefined "unspecified" values ​​can be 25 or 27; in the case of a HEVC analyzer, these one or more predefined "unspecified" values ​​can be 60 or 61; in the case of a VVC analyzer, the one or more predefined "unspecified" values ​​can be 31.

[0090] If the determination is negative, that is, if the NAL unit type value does not indicate an enhancement layer NAL unit, the NAL unit (and its data) can be discarded in step 310, as per the unmodified analysis. However, if the determination is positive, that is, if the NAL unit type value does in fact indicate an enhancement layer NAL unit, for example, by being within a specific set of unspecified values, method 300 proceeds to step 312, where an enhancement layer NAL unit type is determined. In certain cases, different enhancement layer NAL unit types are associated with different unspecified NAL unit type values. The type Petition 870250110555, dated 02 / 12 / 2025, page 29 / 85 The 22 / 68 NAL unit of the enhancement layer can indicate whether the given NAL unit is associated with an instant decoding update image for the enhancement layer. For example, an instant decoding update (IDR) image might comprise an image (e.g., data for a set of color component frames) for which a NAL unit contains a global configuration data block and does not refer to any other image for the operation of the decoding process of that image, and for which no subsequent image in the decoding order refers to any image preceding it in the decoding order. In some cases, an IDR image must occur at least when an IDR image occurs in the base bitstream.

[0091] After step 312, method 300 proceeds to step 314, where the decoding of the determined NAL unit payload is instructed according to the enhancement layer specification. The decoding is performed based on the determined enhancement layer NAL unit type. This may involve passing the NAL unit payload to an enhancement decoder or codec. In one case, the enhancement layer NAL unit type may be transmitted along with the payload, but without the header data. In another case, the complete NAL unit (e.g., header and payload) may be passed to the enhancement decoding process.In response to the nal_unit_type of the enhancement layer indicating an instant decoding update image, global configuration data can be extracted from the payload of the determined NAL unit, and the global configuration data can be used to decode a plurality of residual frames, where the data for the plurality of residual streams is distributed across multiple NAL units in the multilayer video stream.

[0092] In certain implementations, for example, for VVC, additional base-layer metadata may be checked to process the NAL unit, after a positive determination that the type of Petition 870250110555, dated 02 / 12 / 2025, page 30 / 85 23 / 68 NAL unit falls under one or more values ​​not specified for the base video encoding specification, but specified for the enhancement video encoding specification. For example, as described later, if the NAL unit type is specified as VVC “unspecified”, value “31”, a VVC layer identifier (the additional metadata of the base layer) defined in the NAL header can be examined to verify if the layer identifier values ​​are in a given set of values, such as “57” or “59” in the example below. If the layer identifier is not “57” or “59”, the NAL unit can be discarded.In certain cases, an additional check is performed on a reserved zero flag value before instructing the decoding of the determined NAL unit payload according to the enhancement layer specification; for example, the layer identifier is only checked if the reserved zero flag is 1 and the value of the unspecified NAL unit type is 31. Thus, the modified parser is configured to discard a determined NAL unit in response to values ​​within the header data that fall outside a predefined set of metadata values.

[0093] To complete, the following paragraphs provide examples of NAL unit syntax and examples of syntax implementations. These are provided only as examples and should not be considered limiting - actual implementations may vary in practice based on context.

[0094] The AVC NALU header is defined in the ISO / IEC 14496-10 standard, Section 7.3.1, with the following syntax: Syntax Descriptor nal_unit_header() { forbidden_zero_bit u(1) nal_ref_idc u(2) nal_unit_type u(5) Petition 870250110555, dated 02 / 12 / 2025, p. 31 / 85 24 / 68 } TABLE 1 - SYNTAX OF THE NALU STROKES HEADER

[0095] The values ​​and semantics of the NALU type for AVC are specified in Table 7 to 1 of the AVC specification (IS 14496-10). Table 2 summarizes the use of AVC NALU types. Since the AVC NALU type is a 5-bit field, the possible values ​​are from 0 to 31. In this case, values ​​21 to 23 are “reserved” and values ​​24 to 31 are “unspecified”. nal_unit_type Name of the nal_unit_type NAL unit content and syntax structure RBSP NAL unit type class 0-5 (...) (...) VCL 6-20 (...) (...) Not VCL 21-23 RSV Reserved Not VCL 24-31 UNSPEC Not specified Not VCL TABLE 2 - TYPES OF STROKE NALU

[0096] The HEVC NALU header is defined in the standard ISO / IEC 23008-2, Section 7.3.1.2, with the following syntax: Syntax Descriptor nal_unit_header() { forbidden_zero_bit f(1) nal_unit_type u(6) nuh_layer_id u(6) nuh_temporal_id_plus1 u(3)} TABLE 3 - HEVC NALU HEADER SYNTAX

[0097] The NALU type values ​​and semantics for HEVC are specified in Table 7 to 1 of the HEVC specification (IS 23008-2). Table 4 summarizes the usage of HEVC NALU types. Since the HEVC NALU type is a 6-bit field, the possible values ​​are from 0 to 63. In this case, values ​​41 to 47 are “reserved” and values ​​48 to 63 are “unspecified”. Petition 870250110555, dated 02 / 12 / 2025, p. 32 / 85 25 / 68 nal_unit_type Name of the nal_unit_type NAL unit content and syntax structure RBSP NAL unit type class 0-21 (...) (...) VCL 22-31 RSV Reserved VCL 32-40 (...) (...) No VCL 41-47 RSV Reserved No VCL 48-63 UNSPEC Not specified No VCL TABLE 4 - TYPES OF NALU HEVC

[0098] The VVC NALU header is defined in the standard ISO / IEC 23090-3, Section 7.3.1.2, with the following syntax: Syntax Descriptor nal_unit_header( ) { forbidden_zero_bit f(1) nuh_reserved_zero_bit u(1) nuh_layer_id u(6) nal_unit_type u(5) nuh_temporal_id_plus1 u(3)} TABLE 5 - VVC NALU UNIT HEADER SYNTAX

[0099] The values ​​and semantics of the NALU type for VVC are specified in Table 5 of the VVC specification (IS 23090-3). Table 6 summarizes the use of VVC NALU types. Since the VVC NALU type is a 5-bit field, the possible values ​​are from 0 to 31. In this case, values ​​26 to 27 are “reserved” and values ​​28 to 31 are “unspecified”. It can be observed in these tables that each video encoding specification has a different range of “unspecified” values. nal_unit_type Name of the nal_unit_type Content of the NAL unit and RBSP syntax structure NAL unit type class 0-11 (...) (...) VCL Petition 870250110555, dated 02 / 12 / 2025, page 33 / 85 26 / 68 12-25 (...) (...) Not VCL 26-27 RSV Reserved Not VCL 28-31 UNSPEC Not specified Not VCL TABLE 6 - VVC NALU TYPES

[0100] The LCEVC NALU header is defined in the standard ISO / IEC 23094-2, Section 7.3.2, with the following syntax: Syntax Descriptor nal_unit_header( ) { forbidden_zero_bit u(1) forbidden_one_bit u(1) nal_unit_type u(5) reserved_flag u(9)} TABLE 7 - LCEVC NALU UNIT HEADER

[0101] The NALU type values ​​and semantics for LCEVC are specified in Table 17 of the LCEVC specification (IS 23094-2). nal_unit_type Name of the nal_unit_type Content of the NAL unit and RBSP syntax structure NAL unit of class type 0-27 UNSPEC Not specified Not VCL 28-29 (...) (...) VCL 30 RSV Reserved Not VCL 31 UNSPEC Not specified Not VCL

[0102] Table 8 summarizes the use of LCEVC NALU types. Since the LCEVC NALU type is a 5-bit field, the possible values ​​are from 0 to 31. nal_unit_type Name of the nal_unit_type NAL unit content and RBSP syntax structure Class type NAL unit 0-27 UNSPEC Not specified Not specified VCL 28-29 (...) (...) VCL Petition 870250110555, dated 02 / 12 / 2025, page 34 / 85 27 / 68 30 RSV Reserved Not VCL 31 UNSPEC Not specified Not VCL TABLE 8 - TYPES OF NALU FROM LCEVC

[0103] As shown in the example in Figure 2, when the base encoding is an MPEG standard, the elementary stream is a NAL unit stream. In this case, encapsulation of the Enhancement Data Access Units as metadata can be implemented using SEI messages specific to each base encoding specification. For example, AVC, HEVC, and VVC each have a different NAL unit format (e.g., with different NAL unit headers), NALU type divisions, and payloads. However, all these base encoding specifications have the option of using SEI messages. SEI messages can comprise a NAL unit identified with a specific NAL unit type field (usually called nal_unit_type in the base encoding specifications). An example of NAL unit types for SEI messages for AVC, HEVC, and VVC is presented in the table below, where RBSP stands for raw byte sequence payload: MPEG Standard SEI Unit Type NAL Corresponding NAL Unit Payload AVC 6 sei_rbsp() HEVC 39* sei_rbsp() VVC 23* sei_rbsp() * SEI prefix TABLE 9 - TYPE OF SEI NALU

[0104] Raw Byte Sequence Payload (RBSP) of SEI and its internal syntax, for enhancement encapsulation purposes, is the same across all MPEG standards considered above. In one case, the NAL units for an enhancement bitstream can be configured to use an ITU-T T.35 registered user data type (user_data_registered_itu_t_t35 type) SEI payload. For example, the general syntax of Petition 870250110555, dated 02 / 12 / 2025, page 35 / 85 28 / 68 a unit of SEI NAL is: sei_rbsp( ) { Descriptor Value of { sei_message( )} while( more_rbsp_data( ) ) rbsp_ trailing_ bits( )} TABLE 10 - GENERAL FORMAT OF SEI RBSP

[0105] A sintaxe de uma sei_message usada para transportar uma carga útil ITU-T T.35 pode ser definido como mostrado em sei_message( ) { Descritor Valor payload_type_byte u(8) 0x04 payloadType = payload_type_byte payloadSize = 0 do { payload_size_byte u(8) payloadSize += payload_size_byte} while(payload_size_byte = = 0xFF) sei_payload(payloadType = 0x04, payloadSize)} TABELA 11: sei_message() { Descritor Valor payload_type_byte u(8) 0x04 payloadType = payload_type_byte payloadSize = 0 do { payload_size_byte u(8) payloadSize += payload size byte} while(payload_size_byte = = 0xFF) Petição 870250110555, de 02 / 12 / 2025, pág. 36 / 85 29 / 68 sei_payload(payloadType = 0x04, payloadSize)} TABELA 11 - SINTAXE DA MENSAGEM SEI

[0106] A sintaxe de user_data_registered_itu_t_t35 pode seguir a estrutura como mostrado em user data registered itu t t35(payloadSize) { Descritor Valor itu_t_t35_country_code b(8) if(itu_t_t35_country_code != 0xFF) i = 1 else { itu_t_t35_country_code_extension_byte b(8) i = 2} do { itu_t_t35_payload_byte b(8) i++} while(i < payloadSize)} TABELA 12. user_data_registered_itu_t_t35(payloadSize) { Descritor Valor itu_t_t35_country_code b(8) if(itu_t_t35_country_code != 0xFF) i = 1 else { itu_t_t35_country_code_extension_byte b(8) i = 2} do { itu_t_t35_payload_byte b(8) i++ Petição 870250110555, de 02 / 12 / 2025, pág. 37 / 85 30 / 68 } while(i < payloadSize)} TABELA 12 - SINTAXE ITU-T.T35

[0107] In one case, for example, when LCEVC is used as the enhancement layer specification, a manufacturer code can be used to identify a specific form of SEI message. For example, using a manufacturer code, a payload might have sei_payload(0x04, payloadSize) = user_data_registered_itu_t_t35 (payloadSize), with the following format: user_data_registered_itu_t_t35( payloadSize ) { Descriptor Value itu_t_t35_country_code b(8) 0xB4 itu_t_t35_payload_header { t35_uk_country_code_second_octet b(8) 0x00 t35_uk_manufacturer_code_fi rst_octet b(8) 0x50 t35_uk_manufacturer_code_second_octet b(8) 0x00} i = 4 do { itu_t_t35_payload_byte b(8) i++} while (i < payloadSize)} TABLE 13 - ITU-T T. 35 SYNTAX WITH VALUES SPECIFIED BY AN EXAMPLE MANUFACTURER CODE

[0108] The NAL unit header specifications set out in the tables above are visually illustrated in example 400 of Figure 4. The top of the Figure shows the various sections of the NAL unit header for AVC, HEVC, and VVC. In general, the header of a NAL unit can be one or two bytes long, and the bits of these bytes are illustrated in 402 in the Figure. As can be seen, the NAL unit header for AVC is one byte where the field Petition 870250110555, dated 02 / 12 / 2025, page 38 / 85 In 31 / 68, the NAL unit type is carried by 5 bits – from bit 3 to bit 7 (inclusive). For HEVC, the NAL unit type is 6 bits carried from bit 1 to bit 6 (inclusive). In HEVC, bits 7 to 12 contain a 6-bit layer identifier, and bits 13 to 15 contain a temporal identifier. The layer and temporal identifier are internal HEVC fields used for decoding according to that specification. VVC has a similar field set to HEVC, but the layer identifier is 6 bits ranging from bit 2 to bit 7 (inclusive). The NAL unit type is a 5-bit field ranging from bit 8 to bit 12. Bits 12 to 15 contain a 3-bit temporal identifier.

[0109] Given the variety of NAL header configurations shown at the top of Figure 4, a set of NAL header configurations for the enhancement bitstream compatible with all three base video specifications (e.g., AVC, HEVC, and VVC) and the methods described herein are shown at the bottom of the Figure. Line 404 shows possible bit values ​​for an enhancement bitstream NAL header that allows the NAL header to be of the “unspecified” type for all three base video specifications, despite each base video specification having a different NAL header specification. In particular, in line 404, the first bit is 0, bits 1 to 4 and 7 to 15 are 1, and bits 5 and 6 are either 0 or 1.Line 406 shows bit values ​​for an enhancement NAL unit where the NAL unit type is identified as 28 for LCEVC, and line 408 shows bit values ​​for an enhancement NAL unit where the NAL unit type is identified as 29 for LCEVC. Therefore, for LCEVC, the two NALU type values ​​used to identify the video encoding layer NAL units are 28 (= 0x1 C = 0b1.1100), which specifies a non-IDR image, and 29 (= 0x1 D = 0b1.1101), which specifies an IDR image. The two NAL unit header bytes for the two LCEVC video encoding layer NAL unit types can be as follows:

[0110] 0111.1001:1111.1111 (LCEVC NALU type 28 No IDR) Petition 870250110555, dated 02 / 12 / 2025, p. 39 / 85 32 / 68

[0111] 0111.1011:1111.1111 (LCEVC NALU type 29 IDR)

[0112] A summary of how a parser for each of the AVC, HEVC, and VVC would interpret the bits of lines 406 and 408 is defined in the table below (where the first value refers to a parsing of line 406 and the second value refers to a parsing of line 408 - for example, 406|408): NAL ID unit type Temporal layer ID AVC Analyzer 25|27 HEVC Analyzer 60|61 63 7 VVC Analyzer 31|31 57|59 7 TABLE 14 - VALUES ANALYZED FOR THE DATA FROM FIGURE 4

[0113] As can be seen by comparing the table above and the specifications for AVC, HEVC, and VVC, the NAL unit types are all within the “unspecified” ranges for each of the AVC, HEVC, and VVC. For the AVC analyzer, the LCEVC NAL unit types (interpreted as 25 or 27) fall within the “unspecified” range of 24 to 31. For the HEVC analyzer, the LCEVC NALU types (interpreted as 60 or 61) fall within the “unspecified” range of 48 to 63. For the VVC analyzer, the LCEVC NALU types (interpreted as 31) fall within the “unspecified” range of 28 to 31. In this way, the enhancement NAL unit types are configured to avoid overlaps with the NAL unit types of a base video coding specification used.In all three cases, the NAL enhancement unit header, as parsed by a base decoder, is detected as a NAL unit type in a range defined as “unspecified”. Therefore, a base parser that is unaware of an enhancement video coding specification, such as LCEVC, would continue to classify enhancement NAL units as “unspecified”. To be aware of an enhancement bitstream, a base parser can be modified as described in this document. Petition 870250110555, dated 02 / 12 / 2025, page 40 / 85 33 / 68

[0114] In particular, by applying method 300 from Figure 3 to the example NAL header configurations in Figure 4, the operation of a CVA, HEVC, and VVC analyzer can be modified as follows to correctly recognize the enhancement NAL units. For example, the behavior of a CVA analyzer with LCEVC awareness can be modified to execute the following logic:

[0115] if nal_unit_type = 25: treat as LCEVC_NON_IDR;

[0116] otherwise, if nal_unit_type = 27: treat as LCEVC_IDR;

[0117] otherwise: treat as per stroke (ISO / IEC 14496-10 standard).

[0118] Here, LCEVC_NON_IDR is a flag indicating a non-IDR LCEVC image, and LCEVC_IDR is a flag indicating an IDR LCEVC image. Similarly, the behavior of an LCEVC-aware HEVC parser can be modified to execute the following logic:

[0119] if nal_unit_type = 60: treat as LCEVC_NON_IDR;

[0120] otherwise, if nal_unit_type = 61: treat as LCEVC_IDR;

[0121] otherwise: treat in accordance with HEVC (ISO / IEC 23008-2 standard).

[0122] For VVC, the behavior of an LCEVC-aware VVC parser can be modified to execute the following logic:

[0123] if nuh_reserved_zero_bit = 1 and nal_unit_type = 31:

[0124] if nuh_layer_id = 57: treat as LCEVC_NON_IDR;

[0125] otherwise, if nuh_layer_id = 59: treat as Petition 870250110555, dated 02 / 12 / 2025, p. 41 / 85 34 / 68 LCEVC_IDR;

[0126] otherwise, discard;

[0127] otherwise: treat in accordance with VVC (ISO / IEC 23090-3 standard)

[0128] In this case, the method is adapted so that a given NAL unit is discarded in response to additional base-layer metadata values ​​within the header data that are outside a predefined range, where the base-layer metadata comprises the layer identifier. An additional check is also made on a reserved zero flag value before instructing the decoding of the given NAL unit payload, according to the enhancement layer specification.

[0129] Some general information regarding examples of enhancement coding schemes will now be described. This information provides examples of specific multilayer coding schemes.

[0130] It should be noted that examples are presented in this document with reference to a signal as a sequence of samples (i.e., two-dimensional images, video frames, video fields, sound frames, etc.). For simplicity, the non-limiting examples illustrated in this document often refer to signals that are displayed as 2D plans of definitions (e.g., 2D images in a suitable color space), such as, for example, a video signal. In a preferred case, the signal comprises a video signal. An example of a video signal is described in more detail with reference to Figure 7.

[0131] The terms “photo”, “frame” or “field” are used interchangeably with the term “image” to indicate a sample in time of the video signal: any concepts and methods illustrated for video signals made of frames (progressive video signals) can also be easily applied to video signals made of fields (interlaced video signals) and vice versa. Although the examples illustrated in this document focus on image and video signals, the Petition 870250110555, dated 02 / 12 / 2025, page 42 / 85 Those skilled in the art can easily understand that the same concepts and methods are also applicable to any other types of multidimensional signals (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.). Although examples of image or video encoding are provided, the same approaches can be applied to signals with dimensions smaller than two (e.g., audio or sensor streams) or larger than two (e.g., volumetric signals).

[0132] In the description, the terms “image”, “photo” or “plane” (intended with the broader meaning of “hyperplane”, i.e., a set of elements with any number of dimensions and a given sampling grid) will often be used to identify the digital rendering of a sample of the signal along the sequence of samples, where each plane has a given resolution for each of its dimensions (e.g., X and Y), and comprises a set of planar elements (or “element”, or “pel”, or display element for two-dimensional images often called “pixel”, for volumetric images often called “voxel”, etc.A signal is characterized by one or more “values” or “definitions” (for example, by way of non-limiting examples, color definitions in a suitable color space, definitions indicating density levels, definitions indicating temperature levels, definitions indicating audio pitch, definitions indicating amplitude, definitions indicating depth, definitions indicating alpha channel transparency level, etc.). Each planar element is identified by a suitable set of coordinates, indicating the integer positions of said element in the image sampling grid. The signal dimensions may include only spatial dimensions (for example, in the case of an image) or also a temporal dimension (for example, in the case of a signal evolving over time, such as a video signal).In one case, a frame of a video signal can be viewed as comprising a two-dimensional matrix with three color component channels or a three-dimensional matrix with two spatial dimensions (for example, of a given resolution - with lengths equal to the respective height and width of the frame) and one dimension of... Petition 870250110555, dated 02 / 12 / 2025, page 43 / 85 36 / 68 color component (for example, with length 3). In certain cases, the processing described in this document is performed individually for each value plane of the color components that make up the frame. For example, pixel value planes representing each of the Y, U, and V color components can be processed in parallel using the methods described in this document.

[0133] Certain examples described in this document utilize a scalability framework that uses a base coding and an enhancement coding. The video coding systems described in this document operate based on a decoding received from a base coding (e.g., frame-by-frame or full base coding) and add one or more spatial, temporal, or other quality enhancements via an enhancement layer. The base coding may be generated by a base layer, which may use a coding scheme that differs from the enhancement layer and, in certain cases, may comprise an older or comparative (e.g., older) coding standard.

[0134] Figures 5 to 7 show a spatially scalable encoding scheme that uses a downsampled source signal encoded with a base codec, adds a first level of correction or enhancement data to the decoded output of the base codec to generate a corrected photo, and then adds an additional level of correction or enhancement data for a upsampled version of the corrected photo. Thus, the spatially scalable encoding scheme can generate an enhancement stream with two spatial resolutions (higher and lower), which can be combined with a base stream at the lower spatial resolution.

[0135] In the spatially scalable coding scheme, the methods and devices can be based on a general algorithm that is built upon an existing encoding and / or decoding algorithm (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, etc., as well as non-standard algorithms such as VP9, ​​AV1, and others) that serve as a baseline for a Petition 870250110555, dated 02 / 12 / 2025, page 44 / 85 37 / 68 Enhancement layer. The enhancement layer operates according to a different encoding and / or decoding algorithm. The general idea behind the algorithm is to hierarchically encode / decode the video frame, rather than using block-based approaches as is done in the MPEG family of algorithms. Hierarchical encoding of a frame includes generating residues for the complete frame and then a reduced or deciphered frame, and so on.

[0136] Figure 5 shows a system configuration for an example of a spatially scalable 500 coding system. The coding process is divided into two halves, as shown by the dashed line. Each half can be implemented separately. Below the dashed line is a base level and above the dashed line is the enhancement level, which can be usefully implemented in software. The 500 coding system can comprise only the enhancement level processes or a combination of base level and enhancement level processes, as needed. The topology of the 500 coding system at a general level is as follows. The 500 coding system comprises an input I to receive an input signal 501. The input I is connected to a reduced sampler 505D. The reduced sampler 505D outputs to a base encoder 520E at the base level of the 500 coding system.The reduced sampler 505D also outputs to a residual generator 510-S. An encoded basestream is created directly by the baseencoder 520E and can be quantized and entropy-encoded as needed according to the baseencoding scheme. The encoded basestream can be the baselayer as described above, for example, a lower layer in a multilayer encoding scheme.

[0137] Above the dashed line is a series of enhancement-level processes to generate an enhancement layer of a multilayer encoding scheme. In the present example, the enhancement layer comprises two sublayers. In another example, one or more sublayers may be provided. In Figure 5, to generate an encoded sublayer 1 enhancement stream, the encoded base stream is decoded. Petition 870250110555, dated 02 / 12 / 2025, page 45 / 85 38 / 68 is achieved through a decoding operation applied in a 520D base decoder. In preferred examples, the 520D base decoder may be a decoding component complementing an encoding component in the form of the 520E base encoder within a base codec. In other examples, the 520D base decoding block may instead be part of the enhancement level. Through the 510-S residual generator, a difference between the output of the decoded basestream from the 520D base decoder and the downsampled input video is created (i.e., a 510-S subtraction operation is applied to one frame of the downsampled input video and one frame of the decoded basestream to generate a first set of residuals). Here, the residuals represent the error or differences between a reference signal or frame and a desired signal or frame.The residues used in the first level of enhancement can be considered a correction signal, as they are capable of "correcting" a frame of a future decoded base stream. This is useful because it can correct particularities or other peculiarities of the base codec. These include, among others, motion compensation algorithms applied by the base codec, quantization and entropy encoding applied by the base codec, and block adjustments applied by the base codec.

[0138] In Figure 5, the first set of residues is transformed, quantized, and entropy-encoded to produce the encoded flow of the enhancement layer, sublayer 1. In Figure 5, a transformation operation 510-1 is applied to the first set of residues; a quantization operation 520-1 is applied to the transformed set of residues to generate a quantized set of residues; and an entropy encoding operation 530-1 is applied to the quantized set of residues to generate the encoded flow of the enhancement layer, sublayer 1 (e.g., at a first level of enhancement). However, it should be noted that in other examples only the quantization step 520-1 may be performed, or only the transformation step 510-1. Entropy encoding may not be used or may optionally be used in addition to one or both of the transformation steps 510-1 and the quantization step 520-1. Petition 870250110555, dated 02 / 12 / 2025, page 46 / 85 39 / 68 The entropy encoding operation can be any suitable type of entropy encoding, such as a Huffmann encoding operation or a run-length encoding (RLE) operation, or a combination of a Huffmann encoding operation and an RLE operation (e.g., RLE then Huffmann or prefix encoding).

[0139] To generate the encoded enhancement layer stream, sublayer 2, an additional level of enhancement information is created by producing and encoding an additional set of residues by means of the 500-S residue generator. The additional set of residues is the difference between an enhanced-sampled version (through the enhanced-sampler 505U) of a corrected version of the decoded base stream (the reference signal or frame) and the 501 input signal (the desired signal or frame).

[0140] To achieve a reconstruction of the corrected version of the decoded basestream as it would be generated in a decoder (for example, as shown in Figure 6), at least some of the sublayer 1 encoding operations are inverted to mimic the decoder processes, and to account for at least some losses and peculiarities of the transformation and quantization processes. For this, the first set of residues is processed by a decoding pipeline comprising an inverse quantization block 520-1i and an inverse transformation block 510-1i. The first quantized set of residues is inversely quantized in the inverse quantization block 520-1i and inversely transformed in the inverse transformation block 510-1i in the coding system 500 to regenerate a decoder-side version of the first set of residues.The decoded basestream from the 520D decoder is then combined with the decoder-side version of the first set of residues (i.e., a 510-C summation operation is performed on the decoded basestream and the decoder-side version of the first set of residues). The 510-C summation operation generates a downsampled reconstruction of the input video as it would most likely be generated in the decoder – i.e., a reconstructed basecodec video. The reconstructed basecodec video is then... Petition 870250110555, dated 02 / 12 / 2025, page 47 / 85 40 / 68 increased sampling by the 505U increased sampler. Processing in this example is typically performed frame by frame. Each color component of a frame can be processed as shown in parallel or in series.

[0141] The upsampled signal (i.e., reference signal or frame) is then compared with the input signal 501 (i.e., desired signal or frame) to create the additional set of residues (i.e., a difference operation is applied by the residual generator 500-S to the recreated upsampled frame to generate an additional set of residues). The additional set of residues is then processed via an encoding pipeline that mirrors the one used for the first set of residues to become an encoded stream of the enhancement layer, sublayer 2 (i.e., an encoding operation is then applied to the additional set of residues to generate the encoded additional enhancement stream). In particular, the additional set of residues is transformed (i.e., a 510-0 transformation operation is performed on the additional set of residues to generate a transformed additional set of residues).The transformed residues are then quantized and the entropy encoded in the manner described above with respect to the first set of residues (i.e., a 520-0 quantization operation is applied to the transformed set of residues to generate an additional set of quantized residues; and, a 530-0 entropy encoding operation is applied to the additional quantized set of residues to generate an encoded stream of the enhancement layer, sublayer 2 containing the additional level of enhancement information). In certain cases, the operations can be controlled, for example, so that only the 520-1 quantization step can be performed, or only the transformation and quantization step. Entropy encoding can optionally be used additionally.Preferably, the entropy encoding operation can be a Huffmann encoding operation or a run-length encoding (RLE) operation, or both (e.g., RLE followed by Huffmann encoding). The transformation applied to both blocks 510-1 and 510-0 can be a Hadamard transformation that is applied to 2x2 or 4x4 blocks of residues. Petition 870250110555, dated 02 / 12 / 2025, page 48 / 85 41 / 68

[0142] The encoding operation in Figure 5 does not result in dependencies between local blocks of the input signal (for example, compared to many known encoding schemes that apply inter- or intra-macroblock prediction and thus introduce macroblock dependencies). Consequently, the operations shown in Figure 5 can be performed in parallel on 4x4 or 2x2 blocks, which greatly increases encoding efficiency on multicore central processing units (CPUs) or graphics processing units (GPUs).

[0143] As illustrated in Figure 5, the output of the spatially scalable encoding process is one or more enhancement streams for an enhancement layer that preferably comprises a first enhancement level and an additional enhancement level. This is then combinable (e.g., via multiplexing or otherwise) with a base stream at a base level. The first enhancement level (sublayer 1) can be considered to allow for corrected video at a base level, i.e., for example, to correct encoder quirks. The second enhancement level (sublayer 2) can be considered as an additional enhancement level that is usable to convert the corrected video back into the original input video or a close approximation thereof.For example, the second level of enhancement can add fine details that are lost during downsampling and / or help correct errors that are introduced by one or more of the 510-1 transformation operation and the 520-1 quantization operation.

[0144] Figure 6 shows an example of a corresponding decoding system 600 for the example of a spatially scalable encoding scheme. In Figure 6, the encoded basestream is decoded in the basedecoder 620 in order to produce a base reconstruction of the input signal 501. This base reconstruction can be used in practice to provide a visualizable rendering of the signal 501 at a lower quality level. However, the main objective of this base reconstruction signal is to provide a basis for a higher quality rendering of the input signal 501. To this end, the stream of Petition 870250110555, dated 02 / 12 / 2025, page 49 / 85 The decoded 42 / 68 base is provided for processing by the enhancement layer, sublayer 1 (i.e., sublayer 1 decoding). Sublayer 1 processing in Figure 6 comprises an entropy decoding process 6301, an inverse quantization process 620-1, and an inverse transformation process 610-1. Optionally, only one or more of these steps may be performed depending on the operations performed on the corresponding block 500-1 in the encoder. By executing these corresponding steps, a decoded stream of the enhancement layer, sublayer 1, comprising the first set of residues is made available in the decoding system 600.The first set of residues is combined with the decoded basestream from the 620 base decoder (i.e., a 610-C summation operation is performed on one frame of the decoded basestream and one frame from the first set of decoded residues to generate a downsampled reconstruction of the input video—i.e., the reconstructed basecodec video). One frame of the reconstructed basecodec video is then upsampled by the 605U upsampled sampler.

[0145] In addition, and optionally in parallel, the encoded stream of the enhancement layer, sublayer 2, is processed to produce an additional set of decoded residues. Similar to the processing of sublayer 1, the processing of sublayer 2 of the enhancement layer comprises an entropy decoding process 630-0, an inverse quantization process 620-0, and an inverse transformation process 610-0. Of course, these operations will correspond to those performed on block 500-0 in the 500 encoding system, and one or more of these steps may be omitted as needed. Block 600-0 produces a decoded enhancement layer, sublayer 2, comprising the additional set of residues, and these are summed in operation 600-C with the output of the enhanced sampler 605U to create an enhancement layer, reconstruction of sublayer 2 of the input signal 501, which can be provided as the output of the decoding system 800.Thus, as illustrated in Figures 5 and 6, the output of the decoding process can comprise up to three outputs: a base reconstruction, a resolution signal, and more. Petition 870250110555, dated 02 / 12 / 2025, page 50 / 85 43 / 68 low corrected and a reconstruction of the original signal for the multilayer encoding scheme at a higher resolution.

[0146] In general, the examples described in this document operate within encoding and decoding pipelines that comprise at least one transformation operation. The transformation operation may comprise the DCT or a variation of the DCT, a Fast Fourier Transform (FFT), or, in preferred examples, a Hadamard transformation as implemented by LCEVC. The transformation operation may be applied block by block. For example, an input signal may be segmented into a number of different consecutive signal portions or blocks, and the transformation operation may comprise a matrix multiplication (i.e., linear transformation) that is applied to the data of each of these blocks (e.g., as represented by a 1D vector).In this description and technique, it can be said that a transformation operation results in a set of values ​​for a predefined number of data elements, for example, representing positions in a resulting vector after the transformation. These data elements are known as transformed coefficients (or sometimes simply "coefficients").

[0147] As described in this document, where enhancement data comprises residual data, a reconstructed set of coefficient bits may comprise transformed residual data, and a decoding method may further comprise the instruction of a combination of residual data obtained from further decoding of the reconstructed set of coefficient bits with a reconstruction of the input signal generated from a representation of the input signal with a lower quality level to generate a reconstruction of the input signal with a first quality level.The representation of the input signal with a lower quality level can be a decoded base signal, and the decoded base signal can optionally be augmented before being combined with residual data obtained from further decoding of the reconstructed set of coefficient bits, the residual data being at a first quality level (e.g., a first resolution). The decoding can still... Petition 870250110555, dated 02 / 12 / 2025, page 51 / 85 44 / 68 understand receiving and decoding residual data associated with a second sublayer, for example, obtaining an output from the inverse transform and inverse quantization component and combining it with data derived from the aforementioned reconstruction of the input signal at the first quality level. This data may comprise data derived from an enlarged version of the reconstruction of the input signal at the first quality level, i.e., a scale-up to the second quality level.

[0148] Additional details and examples of a two-sublayer enhanced encoding and decoding system can be found in the published LCEVC documentation. Although examples have been described with reference to a layered hierarchical encoding scheme in the form of LCEVC, the methods described in this document can also be applied to another layered hierarchical encoding scheme, such as VC-6: SMPTE VC-6 ST-2117 as described in document PCT / GB2018 / 053552 and / or the associated published standard document, both of which are incorporated herein by reference.

[0149] Figure 7 shows an example 700 of how a video signal can be decomposed into different components and then encoded. In the example in Figure 7, a video signal 702 is encoded. The video signal 702 comprises a plurality of frames or pictures 704, for example, where the plurality of frames represents action over time. In this example, each frame 704 is composed of three color components. The color components can be in any known color space. In Figure 7, the three color components 706 are Y (luma), U (a first chroma opponent color), and V (a second chroma opponent color). Each color component can be considered a plane 708 of values. The plane 708 can be decomposed into a set of n by n signal data blocks 710. For example, in LCEVC, n can be 2 or 4; in other video encoding technologies, n can be 8 to 32.

[0150] In LCEVC and some other encoding technologies, a video signal fed into a base layer is a version Petition 870250110555, dated 02 / 12 / 2025, page 52 / 85 45 / 68 scaled down the input video signal, for example, 501. In this case, the signal fed into both sublayers of the enhancement layer comprises a residual signal containing residual data. A residual data plane can also be organized into sets of n by n 710 signal data blocks. Residual data can be generated by comparing data derived from the input signal being encoded, for example, the 501 video signal, and data derived from a reconstruction of the input signal, this reconstruction being generated from a representation of the input signal with a lower quality level. The comparison can involve subtracting the reconstruction from the downsampled version. The comparison can be performed frame by frame (and / or block by block).The comparison can be performed at the first quality level; if the base quality level is lower than the first quality level, a reconstruction from the base quality level may be performed before comparison. Similarly, the input signal for the second sublayer, for example, the input for the transformation and quantization component of the second sublayer, may comprise residual data resulting from a comparison of the input video signal 501 at the second quality level (which may comprise an original full-quality version of the video signal) with a reconstruction of the video signal at the second quality level. As before, the comparison can be performed frame by frame (and / or block by block) and may involve subtraction.Video signal reconstruction can comprise a reconstruction generated from the decoded version of the encoded base bitstream and a decoded version of the residual data stream from the first sublayer. The reconstruction can be generated at the first quality level and can be upsampled to the second quality level.

[0151] Therefore, a 708 data plane for the first sublayer may comprise residual data that is arranged in n by n signal blocks. One such 2 by 2 signal blocks is shown in more detail in Figure 7 (n is selected as 2 for ease of explanation), where for a color plane the block may have 712 values ​​with a defined bit length (e.g., Petition 870250110555, dated 02 / 12 / 2025, page 53 / 85 46 / 68 or 16 bits). Each n by n signal block can be represented as a flattened vector 714 of length n² representing the signal data blocks. To perform the transformation operation, the flattened vector 714 can be multiplied by a transformation matrix 716 (i.e., the resulting dot product). This then generates another vector 718 of length n² representing different transformed coefficients for a given signal block 710. Figure 7 shows an example similar to LCEVC where the transformation matrix 716 is a 4 by 4 Hadamard matrix, resulting in a transformed coefficient vector 718 having four elements with respective values. These elements are sometimes referred to by the letters A, H, V, and D, as they can represent a mean, horizontal difference, vertical difference, and diagonal difference. Such a transformation operation can also be called directional decomposition.When n = 4, the transformation operation can use a 16 by 16 matrix and be called a directional square decomposition.

[0152] As shown in Figure 7, the set of values ​​for each data element across the complete set of signal blocks 710 for the plane 708 can be represented as a plane or surface of coefficient values ​​720. For example, values ​​for the data elements “H” for the set of signal blocks can be combined into a single plane, where the original plane 708 is then represented as four separate coefficient planes 722. For example, the illustrated coefficient plane 722 contains all the “H” values. These values ​​are stored with a predefined bit length, for example, a bit length B, which can be 8, 16, 32, or 64 depending on the bit depth. A 16-bit example is considered below, but this is not limiting.As such, the 722 coefficient plane can be represented as a sequence (e.g., in memory) of 16-bit or 2-byte 724 values ​​representing the values ​​of a data element of the transformed coefficients. These can be referred to as coefficient bits. These coefficient bits can be quantized and then entropy-encoded, as discussed, to then generate the encoded enhancement layer data, as described above. Petition 870250110555, dated 02 / 12 / 2025, page 54 / 85 47 / 68

[0153] Although base transport and LCEVC on a Single Track, based on interleaved NALUs or Embedded SEI Messages (see Figures 1 and 2, respectively), provide many advantages, these implementations still leave open the possibility of an older player discarding the track completely without any playback.

[0154] Figures 8 and 9 show two implementations of a single-track multilayer video stream that aim to reduce, and preferably eliminate entirely, the possibility of an older player discarding the entire track without any playback. The goal of these embodiments is to present a solution to provide a “Single Track” transport of a base bitstream plus an enhancement bitstream (e.g., LCEVC) that is backward compatible with “older” players.

[0155] The “Single Track” transport of a Base bitstream plus an Enhancement bitstream generally refers to combining the two bitstreams into a single track or file. The Base bitstream contains the base video or audio content, while the Enhancement bitstream contains additional information that improves the quality or adds new features to the content (such as an LCEVC enhancement). By combining the base and enhancement bitstreams into a single track, it becomes easier to manage and transport the media content. The combination can be achieved by using a multiplexer to combine the two bitstreams or by embedding the data from the enhancement bitstream into the base bitstream.

[0156] As described earlier, the modalities described in Figures 1 and 2 (i.e., base transport and LCEVC on a single track, respectively, based on Interleaved NALUs or Embedded SEI Messages) of the present application provide several advantages as they provide for LCEVC and Base on a single track. However, a potential danger of any single-track approach is that an older player might discard the entire track if it does not recognize a part of the track (e.g., the enhancement part), which could result in no playback (this contrasts with some approaches of Petition 870250110555, dated 02 / 12 / 2025, p. 55 / 85 48 / 68 multiple tracks, where the older player can recognize and play a base track, but simply ignores the enhancement track, which ultimately results in some kind of playback, albeit at the base quality level.

[0157] This danger is realized in a “comparative solution,” whereby: two new “Boxes” (in NALU FF terminology, IS 14496-15) are used (e.g., the LCEVCMixedSampleEntry() boxes extend VisualSampleEntry('Ivms') and BaseConfigurationBox() extends Box('bicf')) to associate an LCEVC bitstream with a relevant base bitstream (e.g., AVC, HEVC, VVC, EVC). This comparative solution is based on an Aggregator structure, as defined in clause A.2 of 14496-15 (NALU FF). As a disadvantage, this comparative example makes NALU FF inherently backward compatible. This is because an "older" player (for example, a device that has not been updated to decode LCEVC) would discard both the base and the enhancement due to the two "new" boxes, resulting in no playback.

[0158] This comparative example helps illustrate a potential problem with single-track approaches: the question of how to transport two different codecs (e.g., via DecoderConfigurationBoxes) on a single track while preserving backward compatibility with existing players (e.g., MP4). This is important because, for a player to decode a stream, it needs to be informed about the decoder(s) to be used. In other words, without any enhancement configuration box, the player would not be able to handle the enhancement stream. However, including the enhancement configuration box could compromise the entire stream, resulting in no playback.

[0159] Therefore, it would be advantageous if such “single-track” schemes were designed to be “backward compatible,” so that “clients” (i.e., decoding devices, players, and so on, such as MP4 players) that do not support an enhancement layer and / or enhancement codec (e.g., such as LCEVC) could still Petition 870250110555, dated 02 / 12 / 2025, page 56 / 85 49 / 68 reproduce the Base bitstream (such as AVC, HEVC, VVC) and the “clients” (i.e., decoding devices, players, and so on, such as MP4 players) that support an enhancement layer and / or enhancement codec (e.g., LCEVC) could reproduce both the Base and the Enhancement. Backward compatibility is achieved by reusing existing constructs in the current specifications. Figures 8 and 9 illustrate embodiments of a concept that solves this problem. Figures 8 and 9 are two alternative solutions that can be used to handle single-track transport of a base bitstream plus an enhanced bitstream. The solution shown in Figure 8 reuses the existing SampleDescriptionBox specification for a generic base bitstream.The solution shown in Figure 9 reuses the SampleEntry specifications for different bitstream bases and reuses the existing mechanism for extending VisualSampleEntry. Although these solutions are alternatives, both allow the possibility of transporting two different DecoderConfigurationBoxes on a single track, preserving backward compatibility.

[0160] In short, instead of defining two new “Boxes” that make NALU FF inherently non-backward compatible, embodiments of the present invention reuse (and add to) existing “Boxes.” To complete, a “box” refers to a structural component used in the file format specification for storing multimedia data. The file format is designed to be a flexible container for various data types, including video, audio, metadata, etc. A box is a hierarchical data structure comprising a header and a payload. The payload may include actual data as well as nested additional boxes. The box data structure enables efficient analysis, editing, and streaming of multimedia data in the MPEG-4 file format and provides a useful way to organize and encapsulate different data types in a single file container.

[0161] For example, as mentioned above, Figure 8 illustrates an embodiment where the “existing” SampleDescriptionBox is used. For example, as mentioned above, Figure 9 illustrates an embodiment where the Petition 870250110555, dated 02 / 12 / 2025, page 57 / 85 The 50 / 68 “existing” MPEG4ExtensionDescriptorsBox is used. These modes provide backward compatibility. These modes utilize the feature that any compliant player will ignore and discard boxes it does not understand (i.e., cannot compute). Thus, an “old” player (e.g., a device that has not been updated to decode LCEVC) would still understand the “base” (i.e., old) box and would only ignore and discard the enhancement box, thus resulting in the playback of at least the base data.

[0162] In general, the enhancement configuration box can be added at different levels, for example, at a descriptor level.

[0163] The embodiments of the present invention have been implemented to edit an existing MP4 file with AVC or HEVC video and to insert the LCEVC Decoder Configuration. It has been found that the edited MP4 file is indeed compatible with existing players.

[0164] Advantageously, embodiments of the present invention can be used to provide backward compatibility to any single-track transport delivery of a multi-layer video scheme.

[0165] One embodiment, as shown in Figure 8, uses the existing SampleDescriptionBox specification for a generic base bitstream SampleDescriptionBox (found in ISO / IEC 14496-12, clause 8.5.2.2).

[0166] The diagram in Figure 8 shows the data 800 comprising the sample table (“stbl”) 801. The sample table ('stbl') 801 comprises the sample description ('stsd') 803. The Sample Description box 803 is a sub-box of the Sample Table box 81 and provides a detailed description of the media samples on a specific track. In general, the purpose of the Sample Description box 803 is to store information about the encoding and data format used for the samples on a track. The Sample Description box 803 contains one or more Petition 870250110555, dated 02 / 12 / 2025, page 58 / 85 51 / 68 Sample Input boxes, each describing a specific encoding type and providing relevant parameters for decoding and rendering the samples. The Sample Description box 803 contains a data field called entry_count that specifies the number of Sample Input boxes contained in the Sample Description box 803. Each Sample Input box describes a specific encoding format and provides the parameters needed to decode and render samples of that encoding type. The entry_count field tells a parser or media player how many Sample Input boxes are present in the Sample Description box, allowing it to correctly read and process each one. The value of entry_count determines the number of Sample Input boxes that follow in the Sample Description box structure.Generally, the entry_count value is set during the creation of the MPEG-4 file, based on the number of different encoding formats used for the media samples in a given track. It is derived from the actual content of the Sample Description box and the number of Sample Entry boxes included within it. To increase or decrease the entry_count value, Sample Entry boxes are added to or removed from the Sample Description box. In this mode shown in Figure 8, Sample Description (“stsd”) 803 has an input count of 2, indicating that there are two Sample Input boxes in Sample Description box 803. We can see in Figure 8 that Sample Description ('stsd') 803 includes a Base Sample Input 805 (equal to 'avc1' if the base codec is AVC, equal to 'hvc1' if the base codec is HEVC, equal to 'vvc1' if the base codec is VVC) 805 and an Enhancement Sample Input 807 (equal to 'Ivc1' if the enhancement codec is LCEVC).The base sample entry 805 and the enlarged sample entry 807 are examples of sample entry boxes.

[0167] With regard to Figure 8, we describe a method for generating 800 data, an encoder configured to generate 800 data, a decoder configured to decode 800 data, and a bit stream comprising 800 data. The Sample Table ('stbl') 801, the Sample Description ('stsd') 803, the Base Sample Input 805 and the Sample Input of Petition 870250110555, dated 02 / 12 / 2025, page 59 / 85 52 / 68 enhancement 807 can be referred to as SampleTable box ('stbl') 801, Sample Description box ('stsd') 803, Base Sample Entry box 805, and Enhancement Sample Entry box 807, respectively. Alternatively, SampleTable ('stbl') 801, Sample Description ('stsd') 803, Base Sample Entry 805, and Enhancement Sample Entry 807 can be referred to as SampleTable 801, SampleDescriptionBox 803, Base SampleEntry 805, and Enhancement SampleEntry 807, respectively.

[0168] ISO / IEC 14496-12 refers to Part 12 of ISO / IEC 14496-12, called the “ISO base media file format”. This generally relates to a file format for storing time-based media content. It is a general format that forms the basis for several other more specific file formats (e.g., 3GP, Motion JPEG 2000, MPEG-4 Part 14, MPEG Part 15). It is technically identical to ISO / IEC 15444-12 (JPEG 2000 Picture Coding System - Part 12). MPEG-4 Part 12, also known as ISO / IEC 14496-12, is a standard that defines the file format for multimedia content of the MPEG-4 family. It specifies the structure of the MP4 file format, including how audio, video, and other multimedia data are stored and interleaved in the file. MPEG-4 Part 12 is widely used in various applications, such as digital broadcasting, streaming media, and mobile devices.

[0169] The overall purpose of Clause 8.5.2 concerns the storage of metadata in MP4 files, related to the Sample Description. This clause specifies the requirements for storing metadata boxes, which are generally used to convey detailed additional information about the encoding type used and any initialization information required for that encoding. The clause defines the syntax and semantics of the metadata boxes, including their box type, size, and structure, as well as the encoding of the metadata values. In general, Clause 8.5.2 is important for ensuring the interoperability of MP4 files across different systems and applications by providing a standard mechanism for storing and retrieving metadata associated with multimedia content. Petition 870250110555, dated 02 / 12 / 2025, pages 60 / 85 53 / 68

[0170] As previously discussed, the Sample Description Box is a box structure defined in clause 8.5.2.2 of ISO / IEC 14496-12, also known as the MP4 file format specification. The Sample Description Box contains information about the format and characteristics of the media samples in a track, such as video frames or audio samples. The purpose of the sample description box is to provide the necessary information for a media player or decoder to correctly interpret and decode the media samples in a track. The box includes fields describing the codec used to encode the samples, the bit depth of the samples, the frame rate, the video dimensions, and other parameters necessary for proper decoding and playback. The Sample Description Box is an essential part of an MP4 file, as it allows for efficient and accurate decoding and rendering of the media samples in the file.Typically, the box is located within the Sample Table Box, which is located within the Media Information Box, which is located within the Media Box, which, in turn, is located within the Track Box. In short, the Sample Description Box in ISO / IEC 14496-12 has the important function of providing information about the format and characteristics of the media samples on a track, allowing for efficient and accurate decoding and playback of the media content in an MP4 file.

[0171] This mode allows the insertion of an LCEVC Decoder Configuration associated with the Base Decoder configuration using a SampleDescriptionBox (e.g., “same”) that contains the Base Decoder configuration. To complete, a decoder configuration refers to a set of parameters that describe the capabilities and requirements of a decoder for a specific encoding format and provides essential information that allows the decoder to correctly process and decode the encoded media data. Thus, the base decoder configuration provides information to allow the decoder to decode the base stream, and the LCEVC decoder configuration provides information to allow the decoder to decode the enhancement stream. Petition 870250110555, dated 02 / 12 / 2025, pp. 61 / 85 54 / 68 Each decoder configuration is encapsulated in a data structure contained within a Sample Input box. For example, the Base Decoder configuration is contained in the Base Sample Input box, and the LCEVC Decoder configuration is contained in the Enhancement Sample Input box. Because the Base Sample Input box and the Enhancement Sample Input box are contained within the Sample Description box (in particular, within the same Sample Description box), it is possible to see that both the base decoder configuration and the LCEVC decoder configuration are contained within the same Sample Description box.An LCEVC Decoder Configuration associated with the Base Decoder configuration means that, for a multilayer video bitstream comprising a base bitstream and an enhancement bitstream, the LCEVC Decoder Configuration comprises parameters for the enhancement bitstream that belong to the same multilayer video bitstream whose base bitstream parameters are included in the Base Decoder configuration. A given multilayer video bitstream may comprise more than one enhancement bitstream, and therefore more than one LCEVC decoder configuration may be associated with a given Base Decoder configuration. BASEFF 8.5.2.2

[0172] In relation to ISO / IEC 14496, BASEFF 8.5.2 refers to the base file format specification for audio and video files in the MPEG-4 standard. Clause 8.5.2 of ISO / IEC 14496-12 defines the SampleDescriptionBox, which specifies the format and encoding of audio and video samples in the file. BASEFF (Base File Format) refers to the baseline specification for the MPEG-4 file format, which defines the minimum set of features and capabilities that must be supported by all compatible MPEG-4 file formats. The BASEFF specification includes the basic file structure, such as the hierarchy of boxes and bins, as well as the media base types and their encodings. In the context of clause 8.5.2 of ISO / IEC 14496-12, BASEFF 8.5.2 refers to the baseline specification for SampleDescriptionBox. Petition 870250110555, dated 02 / 12 / 2025, page 62 / 85 55 / 68 which includes the base format and encoding information for audio and video samples in the file. This base specification should be supported by all compatible MPEG-4 file formats and can be extended or modified by additional boxes and extensions to meet specific needs and requirements. SampleDescriptionBox

[0173] aligned(8) class SampleDescriptionBox () extends FullBox ('stsd', version, 0) {

[0174] int i;

[0175] unsigned int(32) entry_count;

[0176] for (i = 1 ; i <= entry_count; i++) {

[0177] SampleEntry(); / / an instance of a class derived from SampleEntry

[0178] }

[0179] }

[0180] The SampleDescriptionBox allows the allocation of one or more SampleEntry, depending on the entry_count field. Thus, the LCEVC Sample Entry can be inserted after the Base Sample Entry in the same SampleDescriptionBox.

[0181] The information stored in Sample Entries after counting entries is specific to the track type and may have variations within the same track type (for example, different encodings may use different specific information after some common fields, even within a video track).

[0182] ISO / IEC 14496 defines a standard for multimedia file formats, including the MP4 file format. In the MP4 file format, SampleEntry specifies the format and encoding of a media sample. The standard defines SampleEntry specifications for different bitstreams, including video, audio, and timed text. For video, SampleEntry can specify several codecs, including MPEG 2 Video, MPEG-4 Visual, and MPEG-4 AVC / H.264, HEVC / H.265, VVC / H.266. The specifications include information such as Petition 870250110555, dated 02 / 12 / 2025, page 63 / 85 56 / 68 the video frame size, frame rate, bit rate, and color space. For audio, SampleEntry can specify several different codecs, including MPEG-2 Audio, MPEG-4 Audio, and MPEG-4 AAC. Specifications include information such as audio sample rate, bit depth, and channel configuration. For timed text, SampleEntry can specify the format of the text data, such as the DFXP (Distribution Format Change Profile) format, and include style and layout information. In general, the SampleEntry specifications defined in ISO / IEC 14496 provide a standardized way to describe the format and encoding of different types of media samples in multimedia files. This standardization enables interoperability between different systems and applications, allowing multimedia files to be reliably processed and played back on a wide range of devices and platforms.

[0183] In other words, a method might include increasing entry_count (for example, by one) and adding an enhancement sample entry (for example, it could be an LCEVC sample entry) to a SampleDescriptionBox. Where SampleDescriptionBox comprises a Base Sample Entry. In other words, an enhancement sample entry (for example, an LCEVC sample entry) is added to the same SampleDescriptionBox that contains the base sample entry. Preferably, the enhancement sample entry is added after the base sample entry.

[0184] This approach is particularly advantageous because, in principle, no change to ISO / IEC 14496-12, clause 8.5.2.2, would be strictly necessary, since the current specification allows the insertion of a first Sample Entry for the Base and a second Sample Entry for the Enhancement of the LCEVC. For greater clarity and completeness, the specific case of a second Sample Entry for LCEVC could be described in the text or in a note to the standard specification.

[0185] It was discovered that if an old device reads that the Petition 870250110555, dated 02 / 12 / 2025, pages 64 / 85 If the 57 / 68 entry_count is 2, but it cannot understand the second descriptor (for example, the LCEVC configuration box), the old device will simply ignore the second descriptor. This allows the old device to still play back database data, even if it cannot understand the second descriptor.

[0186] As discussed, an additional (i.e., alternative) mode, as shown in Figure 9, reuses the existing SampleEntry specifications for the different base bitstreams, using the existing mechanism for extending VisualSampleEntry: MPEG4ExtensionDescriptorsBox (14496-15, clauses 5.4.2.1.2, 8.4.1.1.2, 11.3.1.2, 12.4.1.2). This mode allows the insertion of an LCEVC Decoder Configuration associated with the Base Decoder configuration using an MPEG4ExtensionDescriptionBox present in the base decoder sample input.

[0187] class AVCSampleEntry() extends VisualSampleEntry ('avc1 or 'avc3') {

[0188] AVCConfigurationBox config;

[0189] MPEG4ExtensionDescriptorsBox(); / / optional

[0190] }

[0191] The diagram in Figure 9 shows the data 900 comprising the sample table ('stbl') 901. The Sample Table ('stbl') 901 comprises the Sample Description ('stsd') 903. In this embodiment, the Sample Description ('stsd') 903 has an entry count of 1, meaning there is only one Sample Entry box within the Sample Description box. As seen in Figure 9, the Sample Description ('stsd') 903 comprises a Base Sample Entry (equal to 'avc1' if the base codec is AVC, equal to 'hvc1' if the base codec is HEVC, equal to 'vvc1' if the base codec is VVC) 905. The Base Sample Entry 905 comprises the MPEG4 Extension Descriptor 906, which is a secondary box within the Sample Entry box. Generally, the MPEG4 Extension Descriptor 906 is a box that includes additional information, such as metadata and configuration information, to help decoders to... Petition 870250110555, dated 02 / 12 / 2025, pages 65 / 85 58 / 68 interpret and process encoded media data. Additional boxes or data structures can be added to the MPEG4 Extension Descriptor box, allowing the inclusion of additional data within the existing box structure. As we can see in Figure 9, MPEG4 Extension Descriptor 906 comprises Enhancement Sample Input 907 (equal to 'Ivc1' if the enhancement codec is LCEVC), which is a secondary box within the MPEG4 Extension Descriptor box. The figure refers to “Ivc1 - descr n°1”. It can also refer to “Ivc1 - input n°1” interchangeably.

[0192] With respect to Figure 9, we describe a method for generating data 900, an encoder configured to generate data 900, a decoder configured to decode data 900, and a bitstream comprising data 900. Sample Table ('stbl') 901, Sample Description ('stsd') 903, Base Sample Input 905, MPEG4 Extension Descriptor 906, and Enhancement Sample Input 907 can be referred to as Sample Table ('stbl') box 901, Sample Description box ('stsd') 903, Base Sample Input box 905, MPEG4 Extension Descriptor box 906, and Enhancement Sample Input box 907, respectively. Furthermore, the MPEG4 Extension Descriptor 906 can be called MPEG4ExtensionDescriptor 906.

[0193] In ISO / IEC 14496, the VisualSampleEntry specifies the format and encoding of a video sample. The standard provides a mechanism to extend the VisualSampleEntry through the use of the SampleEntry extension mechanism. The SampleEntry extension mechanism is defined in clause 8.5.2 of ISO / IEC 14496-12, which specifies the file format for MPEG-4 audio and video. This clause defines the SampleDescriptionBox, which contains the SampleEntry boxes that describe the format and encoding of the audio and video samples. The VisualSampleEntry can be extended through the use of the MPEG4ExtensionDescriptorsBox, which is defined in ISO / IEC 14496-15. This box allows additional metadata about the visual content to be included in the sample description. In general, the extension mechanisms defined in ISO / IEC 14496 provide a flexible and extensible framework for describing the format and encoding of the sample. Petition 870250110555, dated 02 / 12 / 2025, pages 66 / 85 59 / 68 video sample encoding, enabling interoperability and efficient processing of multimedia files across a wide range of systems and platforms. The embodiments of the present invention utilize these features for a novel purpose. NALUFF 5.4.2.1.3

[0194] Regarding ISO / IEC 14496-15, NALUFF 5.4.2.1.3 refers to the specification of the byte stream format used for the AVC (Advanced Video Coding) video codec in the MPEG-4 file format. Clause 5.4.2.1 of ISO / IEC 14496-15 defines the byte stream format of the NAL (Network Abstraction Layer) unit used for storing and transporting AVC video data. The NAL unit format is used to encapsulate individual AVC video frames and includes a header that specifies the nal_unit_type and its length. NALUFF (NAL Unit File Format) refers to the specific file format used to store video data (e.g., AVC, HEVC, etc.) in the MPEG-4 file format. This format uses the NAL unit byte stream format defined in clause 5.4.2.1.3 of ISO / IEC 14496-15 to encapsulate individual video frames, which are then stored in a series of SampleEntry boxes in the MPEG-4 file. In general, NALUFF 5.4.2.1 specifies the Sample Input for video data (e.g., AVC) in the MPEG-4 file format, providing a standardized mechanism for storing and transporting video data that can be implemented across a wide range of systems and platforms. MPEG4 EXTENSION DESCRIBERS BOX

[0195] Descr is a descriptor that must be placed in the ElementaryStreamDescriptor when this stream is used in an MPEG-4 system context. This does not include SLConfigDescriptor or DecoderConfigDescriptor, but it does include the other descriptors that should be placed after SLConfigDescriptor.

[0196] In ISO / IEC 14496-15, “Descr” refers to a general descriptor for MPEG-4 systems. The overall purpose of “Descr” is to provide additional information about an object in the MPEG-4 bitstream, such as a media stream or a scene description. The “Descr” descriptor is used in several clauses of the Petition 870250110555, dated 02 / 12 / 2025, pages 67 / 85 The present invention relates to the 60 / 68 ISO / IEC 14496-15 standard, such as clause 5.4.2.1, which defines the "Descr" box, and clause 8.5.2.2, which specifies the use of "Descr" in the SampleEntry box to provide additional information about the codec configuration, and clause 8.6.4, which explains how "Descr" can be used to signal the presence of a specific object in the bitstream. The "Descr" box can contain various types of descriptors, such as object descriptor, ES descriptor, and decoder configuration descriptor. The embodiments of the present invention utilize the flexibility of the 'Descr' descriptor to include metadata and additional information about an object in the MPEG-4 bitstream, ultimately allowing for more accurate and efficient decoding and processing of media streams due to the backward compatibility of the generated streams.

[0197] DecoderConfigDescriptor is defined in 14496-1, clause 7.2.6.6. In ISO / IEC 14496-1, the DecoderConfigDescriptor is a box used to describe the configuration and capabilities of a decoder for a specific media stream. The purpose of this box is to provide information that can be used by a client application or player to ensure that the decoder is able to correctly decode the stream. The DecoderConfigDescriptor box is defined in clause 7.2.6.6 of ISO / IEC 14496-1 and is used in conjunction with other boxes, such as VisualSampleEntry or AudioSampleEntry, to fully describe the format and encoding of a media stream. The DecoderConfigDescriptor box itself contains information such as decoder-specific information, buffer size, and maximum bitrate.When a client application or player receives a media stream, it can use the information in the associated DecoderConfigDescriptor box to determine if its decoder is capable of handling the stream. If the decoder does not meet the requirements described in the box, the client application or player can choose to transcode the stream or notify the user that the stream cannot be played. In short, the purpose of the DecoderConfigDescriptor box in ISO / IEC 14496-1 is to describe the configuration and capabilities of a decoder for a media stream, while its function is to allow client applications or players to ensure that the decoder is capable of correctly decoding the stream. Petition 870250110555, dated 02 / 12 / 2025, pages 68 / 85 61 / 68 flow.

[0198] SLConfigDescriptor (Synchronization Layer Configuration Descriptor) is defined in 14496-1, clause 7.3.2.3. In ISO / IEC 14496-1, the SLConfigDescriptor is a box used to describe the synchronization layer configuration for a given media stream. The overall purpose of this box is to provide information about the synchronization layer of the stream, including the target device and the type of service provided. The SLConfigDescriptor box is defined in clause 7.2.6.8 of ISO / IEC 14496-1 and is used in conjunction with other boxes, such as VisualSampleEntry or AudioSampleEntry, to maintain temporal synchronization within and between elementary streams and to fully describe the format and encoding of a media stream. The SLConfigDescriptor box itself contains information such as the service type, the service provider, the target device, and the maximum bitrate.When a client or player receives a media stream, it can use the information from the associated SLConfigDescriptor box to determine the stream's service level. This information can be used to optimize playback settings, such as screen resolution and bitrate, to ensure the stream plays correctly on the target device. In short, the purpose of the SLConfigDescriptor box in ISO / IEC 14496 is to describe the service level configuration for a media stream, while its function is to allow client or playback applications to optimize playback settings based on the information provided in the box.

[0199] Since the specific ConfigurationBox for any base codec of interest (AVCConfigurationBox, HEVCConfigurationBox, EVCConfigurationBox, VVCConfigurationBox, and so on) is not derived from the DecoderConfigDescriptor, it is consistent with the existing specification 14496-15, clause 5.4.2.1.3, to include an LCEVCConfigurationBox in the MPEG4ExtensionDescriptorsBox, thus providing backward compatibility.

[0200] The update to ISO / IEC 14496-15, clause 5.4.2.1.3, at the end of the Descr semantics, is described to add the following text: “In particular, the descriptor for an LCEVC decoder may be present in the field Petition 870250110555, dated 02 / 12 / 2025, pages 69 / 85 62 / 68 Description: In this case, one or more LCEVCConfigurationBoxes should be considered as the descriptor(s) for one or more associated LCEVC decoder(s) as an enhancement to the Base decoder identified by the ConfigurationBox preceding the MPEG4EXtensionDescriptorsBox.

[0201] Thus, the ISO / IEC 14496-15 standard, clause 5.4.2.1.3, at the end of Descr, the semantics would read that Descr is a descriptor that must be placed in the ElementaryStreamDescriptor when that stream is used in an MPEG-4 system context. This does not include the SLConfigDescriptor or the DecoderConfigDescriptor, but includes the other descriptors to be placed after the SLConfigDescriptor. In particular, the descriptor of an LCEVC decoder may be present in the Descr field: in this case, one or more LCEVCConfigurationBoxes will be considered as descriptor(s) of one or more LCEVC decoder(s) associated as an enhancement to the Base decoder identified by the ConfigurationBox field preceding the MPEG4ExtensionDescriptorsBox.

[0202] Advantageously, this mode is backward compatible. Furthermore, this mode offers the added advantage of possibly associating more than one enhancement bitstream (e.g., LCEVC) with the same base bitstream without the need for duplication. This is because additional Sample Input boxes can be added to the MPEG4 906 Extension Descriptor box for each additional enhancement bitstream associated with a given base bitstream. This contrasts with the “comparative solution,” which implies that when there are multiple LCEVC bitstreams for the same Base bitstream, a different Aggregator containing LCEVC plus Base must be defined for each one.

[0203] Although reference has been made to LCEVC, it should be understood that these references are still valid for any other enhancement-type coding schemes (e.g., known and / or future).

[0204] In summary, Figures 8 and 9 show two implementations of a single-track multilayer video stream that is compatible Petition 870250110555, dated 02 / 12 / 2025, pages 70 / 85 63 / 68 with older versions of older players, so that the older player will not discard the entire video stream without any playback. Both implementations achieve this by reusing and adding to existing data structures in the file format, rather than defining new structures that cannot be recognized. Specifically, the existing Sample Description box is reused. In particular, with respect to Figure 8, an additional Sample Input box is added to the existing Sample Description box, so that the enhancement decoder configuration can be contained in this additional Sample Input box. With respect to Figure 9, a Sample Input box is added to the existing MPEG4 Extension Descriptor box in the Sample Description box, so that the enhancement decoder configuration can be contained in this new Sample Input box.Both implementations allow "clients" that do not support an enhancement layer and / or enhancement codec (e.g., LCEVC) to still play back the base bitstream, and allow "clients" that do support an enhancement layer and / or enhancement codec (e.g., LCEVC) to play back both the base and enhancement bitstreams.

[0205] In certain cases, the method in example 300 or any other of the examples described in this document may be implemented by means of instructions retrieved from a computer-readable medium. These may be executed by a processor of a decoding system, such as a client device. The techniques described in this document may be implemented in software or hardware, or may be implemented using a combination of software and hardware. They may include configuring an apparatus to perform and / or support any or all of the techniques described in this document. The above examples are to be understood as illustrative. Other examples are provided. It should be understood that any feature described in relation to any example may be used alone, or in combination with other described features, and may also be used in combination with one or more features of any of the other examples, or any combination of any of the other examples.In addition. Petition 870250110555, dated 02 / 12 / 2025, pp. 71 / 85 64 / 68 In addition, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the appended claims.

[0206] We describe a series of clauses:

[0207] 1. Method for analyzing a multilayer video stream, the multilayer video stream encoding a video signal and comprising a base layer and one or more enhancement layers, the base layer being encoded using a base layer specification, the one or more enhancement layers being encoded using an enhancement layer specification and comprising one or more frame residue sets, a frame residue set comprising a difference between a reconstruction derived from a decoded base layer frame and a frame derived from an original video signal at a given quality level, the method comprising:

[0208] In a modified parser for the base layer, parse header data for Network Abstraction Layer (NAL) units belonging to the multilayer video stream, including, for a given NAL unit:

[0209] extract a NAL unit type value from the header data for the given NAL unit;

[0210] in response to the fact that the NAL unit type value is within a range specified for the base layer specification, instruct the decoding of a determined NAL unit payload in accordance with the base layer specification; and

[0211] in response to the fact that the NAL unit type value is within a predetermined part of a range not specified for the base layer specification,

[0212] determine a type of enhancement layer NAL unit from the header data, and

[0213] instruct the decoding of the payload of the unit of Petition 870250110555, dated 02 / 12 / 2025, pp. 72 / 85 65 / 68 NAL determined according to the enhancement layer specification based on the NAL unit type of the determined enhancement layer.

[0214] 2. Method, according to clause 1, wherein the NAL unit type of the enhancement layer indicates whether the given NAL unit is associated with an instant decoding update image.

[0215] 3. Method, according to clause 2, wherein the method further comprises:

[0216] in response to the nal_unit_type of the enhancement layer indicating an instant decoding update image, extract global configuration data from the payload of the given enhancement layer unit NAL and using the global configuration data to decode a plurality of residual frames, data for the plurality of residual streams being distributed across multiple NAL units in the multilayer video stream.

[0217] 4. Method, according to any of the preceding clauses, wherein the base layer specification is one of Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), and the enhancement layer specification is Low Complexity Enhancement Video Coding (LCEVC).

[0218] 5. Method, according to any of the preceding clauses, in which NAL units for the base layer are interleaved with NAL units for one or more enhancement layers within the multilayer video stream.

[0219] 6. Method, according to any of the preceding clauses, wherein the header data comprises a 16-bit sequence, in which the bit values ​​for NAL units belonging to one or more enhancement layers are configured to indicate a value that is Petition 870250110555, dated 02 / 12 / 2025, pp. 73 / 85 66 / 68 falls within the unspecified range for base layer specification for a plurality of different base layer specifications.

[0220] 7. Method, according to any of the preceding clauses, comprising:

[0221] discard the determined NAL unit in response to additional base-layer metadata values ​​within the header data that are outside of a predefined set of metadata values.

[0222] 8. Method, according to clause 7, wherein the base layer specification comprises Versatile Video Coding (VVC) and the additional base layer metadata comprises a layer identifier.

[0223] 9. Method, according to clause 8, in which an additional check is made on a reserved zero flag value before instructing the decoding of the NAL unit payload determined according to the enhancement layer specification.

[0224] 10. Analyzer for analyzing a video stream, wherein the analyzer is configured to process Network Abstraction Layer (NAL) units comprising data encoded according to a first video encoding specification, the analyzer comprising:

[0225] a memory for storing Network Abstraction Layer (NAL) unit header data belonging to a video stream; and

[0226] a processor configured for:

[0227] extract a NAL unit type value from the header data for a given NAL unit; and in response to the fact that the NAL unit type value is within a specified range for the first video encoding specification, instruct the decoding of a payload of the given NAL unit according to the first video encoding specification,

[0228] in which the parser is modified to process Petition 870250110555, dated 02 / 12 / 2025, pages 74 / 85 67 / 68 a multilayer video stream, so that the processor is configured to, in response to the NAL unit type value, remain within a predetermined portion of an unspecified range for the first video encoding specification:

[0229] determine a type of enhancement layer NAL unit from the header data, and

[0230] instruct the decoding of the payload of the determined NAL unit according to an enhancement layer specification based on the determined enhancement layer NAL unit type,

[0231] wherein a reconstruction of an original video signal associated with the multilayer video stream is generated by combining data derived from frames decoded according to the first video coding specification with data derived from residual frames decoded according to the enhancement layer specification, wherein each residual frame provides a quality improvement over a frame decoded according to the first video coding specification.

[0232] 11. Analyzer, according to clause 10, where the NAL unit type of the enhancement layer indicates whether the given NAL unit is associated with an instant decoding update image.

[0233] 12. Analyzer, according to clause 11, wherein the processor is further configured to, in response to the type of NAL unit of the enhancement layer indicating an instantaneous decoding update image, instruct the extraction of global configuration data from the payload of the determined NAL unit and instruct the use of the global configuration data to decode a plurality of residual frames, wherein the data for the plurality of residual streams are distributed across multiple NAL units in the multilayer video stream.

[0234] 13. Analyzer, according to any of clauses 10 to 12, where the first video encoding specification is a Petition 870250110555, dated 02 / 12 / 2025, pages 75 / 85 68 / 68 among Advanced Video Coding (AVC), High-Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), and the enhancement layer specification is Low-Complexity Enhanced Video Coding (LCEVC).

[0235] 14. Analyzer, according to any of clauses 10 to 13, wherein the NAL units for the first video encoding specification are interleaved with NAL units for the enhancement layer specification in the multilayer video stream.

[0236] 15. Parser, according to any of clauses 10 to 14, wherein the header data comprise a 16-bit sequence, wherein the bit values ​​for NAL units belonging to one or more enhancement layers are set to indicate a value that falls within the unspecified range for the first video encoding specification for a plurality of different first video encoding specifications.

[0237] 16. Parser, according to any of clauses 10 to 14, wherein the parser is configured to discard the determined NAL unit in response to additional base-layer metadata values ​​within the header data that are outside a predefined range.

[0238] 17. Analyzer, according to clause 16, wherein the first video coding specification comprises Versatile Video Coding (VVC) and the additional base layer metadata comprises a layer identifier.

[0239] 18. Analyzer, according to clause 17, in which an additional check is made on a reserved zero flag value before instructing the decoding of the NAL unit payload determined according to the enhancement layer specification.

[0240] 19. Computer-readable media comprising instructions which, when executed, cause a processor to execute the method as defined in any of clauses 1 to 9. Petition 870250110555, dated 02 / 12 / 2025, pages 76 / 85

Claims

1 / 3 CLAIMS 1. A method characterized in that it comprises: adding an enhancement sample entry (807, 907) to a sample description (803, 903) contained in a media data (800, 900), such that: an older player is configured to play back database data associated with a database sample entry in said sample description; and an enhanced player is configured to play back enhancement data associated with said enhancement sample entry and database data associated with said database sample entry.

2. Method according to claim 1, characterized in that it further comprises adding the entry of the base sample to the sample description.

3. A method according to claim 1 or 2, characterized in that the sample description comprises an entry_count data field, and the method further comprises increasing the entry count of the sample description by a number equal to the number of enhancement flows in the media data.

4. Method according to claim 3, characterized in that, when the media data comprises a base stream and a single LCEVC stream, the method further comprises setting the value of entry_count to 2.

5. Method, according to claim 3, characterized in that, when the media data comprises a base stream of enhancement streams, the definition of the entry_count value is n+1, where n is an integer greater than or equal to 0.

6. Method according to claim 2, characterized in that it further comprises adding the base sample entry to the data structure before adding the enhancement sample entry.

7. Method according to claim 2, characterized in that it further comprises adding a secondary box structure to the base sample input, preferably wherein the secondary box structure is an MPEG4 (906) extension descriptor.

8. Method according to claim 7, characterized in that it further comprises filling the MPEG4 extension descriptor with the enhancement sample input.

9. A method, according to any previous claim, characterized in that the enhancement sample input is a descriptor for more than one enhancement stream associated with the media data.

10. Data flow, characterized in that it comprises a sample description (803, 903) comprising one or more enhanced sample entries (807, 907).

11. Data flow according to claim 10, characterized in that the sample description comprises a base sample entry.

12. Data flow according to claim 10 or 11, characterized in that the sample description comprises an entry_count data field and wherein a value of entry_count is greater than one.

13. Data flow, according to claim 12, characterized in that the value of entry_count is equal to a number equal to the sum of a number from one or more enhancement sample entries plus a number of base sample entries included in the sample description.

14. Data flow according to claim 11, characterized in that when the value of entry_count is equal to 2, the sample description comprises a single enhancement sample entry and a single base sample entry.

15. Data stream according to claim 10, characterized in that the base sample input comprises an MPEG4 extension descriptor.

16. Data stream, according to claim 15, characterized in that said MPEG4 extension descriptor comprises said Petition 870250110541, dated 02 / 12 / 2025, page 8 / 17 3 / 3 one or more enhancement sample entries.

17. Device, characterized in that it is configured to execute the method as defined in any one of claims 1 to 9.

18. Device, characterized in that it is configured to decode the data stream as defined in any one of claims 10 to 16.

19. Computer-readable medium, characterized in that it comprises instructions which, when executed, cause a processor to execute the method as defined in any one of claims 1 to 9. Petition 870250110541, dated 02 / 12 / 2025, page 9 / 17

Images