Processing a multi-layer video stream

Abstract

A method comprising: adding an enhancement sample entry (807, 907) to a sample description (803, 903) contained in a media data (800) such that: a legacy player is configured to playback base data associated with a base sample entry in said sample description; and an enhanced player is configured to playback enhancement data associated with said enhancement sample entry and base data associated with said base sample entry.

Description

PROCESSING A MULTI-LAYER VIDEO STREAMTechnical Field

[0001] The present invention relates to the processing of a multi-layer video stream.Background

[0002] Multi-layer video coding schemes have existed for a number of years but have experienced problems with widespread adoption. Much of the video content on the Internet is still encoded using H.264 or MPEG-4 Part 10, Advanced Video Coding (MPEG- 4 AVC), with this format being used for between 80-90% of online video content. This content is typically supplied to decoding devices as a single video stream that has a one- to-one relationship with available hardware and / or software video decoders, e.g. a single stream is received, parsed and decoded by a single video decoder to output a reconstructed video signal. Many video decoder implementations are thus developed according to this framework.

[0003] Existing multi-layer coding schemes include the Scalable Video Coding (SVC) extension to H.264, Scalable extensions to H.265 or MPEG-H Part 2 High Efficiency Video Coding (SHVC), and newer standards such as MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC). While H.265 is a development of the coding framework used by H.264, LCEVC takes a different approach to scalable video. SVC and SHVC operate by creating different encoding layers and feeding each of these with a different spatial resolution. Each layer encodes the input according to a normal AVC or HEVC encoder with the possibility of leveraging information generated by lower encoding layers. LCEVC, on the other hand, generates one or more layers of enhancement residuals as compared to a base encoding, where the base encoding may be of a lower spatial resolution.

[0004] One reason for the slow adoption of multi-layer coding schemes has been the difficulty adapting existing and new decoders to process multi-layer encoded streams. As discussed above, video streams are typically single streams of data that have a one-to- one pairing with a suitable decoder, whether implemented in hardware or software or a combination of the two. Client devices and media players, including Internet browsers, are thus built to receive a stream of data, determine what video encoding the stream uses, and then pass the stream to an appropriate video decoder. Within this framework, multi-layer schemes such as SVC and SHVC have typically been packaged as largersingle video streams containing multiple layers, where these streams may be detected as “SVC” or “SHVC” and passed atomically to an SVC or SHVC decoder for reconstruction. This approach though often mitigates some of the benefits of multi-layer encodings. Hence, many developers and engineers have concluded that multi-layer coding schemes are too cumbersome and return instead to a multicast of single H.264 video streams.

[0005] It is thus desired to obtain an improved method and system for decoding multilayer video data that overcomes some of the disadvantages discussed above and that allows more of the benefits of multi-layer coding schemes to be realised.

[0006] The paper “The Scalable Video Coding Extension of the H.264 / AVC Standard” by Heiko Schwarz and Mathias Wien, as published in IEEE Signal Processing Magazine 135, March 2008, provides an overview of the SVC extension.

[0007] The paper “Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard” by Jill Boyce, Yan Ye, Jianle Chen, and Adarsh K. Ramasubramonian, as published in IEEE Transactions on Circuits and Systems for Video Technology, VOL. 26, NO. 1 , January 2016, provides an overview of the SHVC extensions.

[0008] The decoding technology for LCEVC is set out in the Draft Text of ISO / IEC FDIS 23094-2 as published at Meeting 129 of MPEG in Brussels in January 2020, as well as the Final Approved Text, and WO 2020 / 188273 A1. Figure 29B of WO 2020 / 188273 A1 describes a hypothetical reference decoder where a demuxer provides a base bitstream to a base decoder and an enhancement bitstream to an enhancement decoder.

[0009] WO 2020 / 016562 A1 describes methods of decoding a bitstream. In examples, a version of an original signal is encoded using a base coding standard and one or more layers of a residual stream may be encoded using an enhancement encoding. In a first example, the encoded residual stream or streams are encapsulated within a Supplemental Enhancement Information (SEI) message. In a second example, the encoded residual stream or streams may be transmitted as a series of Network Abstraction Layer (NAL) units that are interleaved with NAL units of an encoded base stream.

[0010] With multi-layer streams there is also a general problem of stream management. Different layers of a multi-layer stream may be generated together or separately, and may be supplied together or separately. It is desired to have improved methods and systems for decoding of these multi-layer streams. For example, it is desiredto allow content distributors to easily and flexibility modify video quality by adding additional layers in a multi-layer scheme. It is also desired to be able to flexibly remultiplex multi-layer video streams without breaking downstream single-layer or multilayer decoding.

[0011] In some ways a ‘single track’ solution may reduce the number of stream management problems compared to a multi-track (e.g. dual-track) solution. However, it is difficult to provide a single track solution that is backwards compatible with old devices. This is related to the problem of supplying multi-layer streams as static file formats. For example, video streams may be read from fixed or portable media, such as solid-state devices or portable disks, or downloaded and stored as a file for later viewing. It is difficult to support the carriage of multi-layer video with existing file formats, as these file formats typically assume a one-to-one mapping with media content and decoding configurations, whereas multi-layer streams may use different decoding configurations for different layers. Changes in file formats often do not work practically, as they require updates to decoding hardware and software and may affect the decoding of legacy formats.

[0012] All of the above publications set out above are to be incorporated by reference herein.Summary of the Invention

[0013] Aspects of the present invention are set out in the appended independent claims. Variations of these aspects are set out in the appended dependent claims.

[0014] According to a first aspect three may be provided a method comprising: adding an enhancement sample entry to a sample description contained in a media data such that: a legacy player is configured to playback base data associated with a base sample entry in said sample description; and an enhanced player is configured to playback enhancement data associated with said enhancement sample entry and base data associated with said base sample entry.

[0015] The method may be a method for storing multimedia data. The method may be a method for describing the format of multimedia data. The sample description may be a type of data structure referred to as a box. The sample description may be an example of a data structure. The sample description may be an example of a box type data structure. The enhancement sample entry may be an example of a box type data structure. The enhancement sample entry may be a child box structure.

[0016] The method involves adding an enhancement sample entry to an existing data structure, in the form of the sample description box, to provide for backward compatibility of single-track multi-media files.

[0017] The method may further comprise adding the base sample entry to the sample description. Thus, the base sample entry is added to the existing data structure (which is the sample description box). In this way, the base sample entry and the enhancement sample entry are added to the same data structure, i.e. the sample description box. The sample description box may comprise a data field entry_count, and the method may further comprise increasing an entry count of the sample description box by a number equal to a number of enhancement streams in the multimedia data.

[0018] The data field entry_count may have a value that corresponds to a number of child data structures within the sample description box. Each of the child data structures may be a sample entry box. The method may comprise setting a value of the data field entry_count equal to the number of separate bitstreams within the multimedia data, wherein the multimedia data comprises a base bitstream and I or at least one enhancement bitstream. In this way the number of child data structures (i.e. sample entry boxes) in the sample description box is the same as the number of bitstreams in the multimedia data.

[0019] When the multimedia data comprises a base stream and a single LCEVC stream, the method may further comprise setting the value of entry_count to 2. When the media data comprises a base steam and n enhancement streams, the method may comprise setting the value of entry_count to n+1 , wherein n is an integer greater than or equal to 0. The sample description may comprise n enhancement sample entry boxes.

[0020] The method may further comprise adding the base sample entry to the sample description before adding the enhancement sample entry.

[0021] The method may further comprise adding a child box structure to the base sample entry, preferably wherein the child box structure is an MPEG4 Extension Descriptor box.

[0022] The method may further comprise populating the MPEG4 extension descriptor box with the enhancement sample entry. In this way, the enhancement sample entry is added to an existing data structure, in the form of the MPEG4 extension descriptor box, to provide for backward compatibility of single-track multi-media files.

[0023] The enhancement sample entry may be a descriptor for more than one enhancement stream associated with the media data.

[0024] According to another aspect of the invention there may be provided a data stream comprising a sample description comprising one or more enhancement sample entries.

[0025] The sample description may comprise a base sample entry.

[0026] The sample description may comprises a data field entry_count, and a value of entry_count may be greater than one. The value of entry_count may be equal to a number that is the sum of a number of the one or more enhancement sample entries plus a number of the base sample entries comprised in the sample description.

[0027] When the value of entry_count is equal to 2, the sample description may comprise a single enhancement sample entry and a single base sample entry. When the value of entry_count is greater than 2, the sample description may comprise more than one enhancement sample entry and a single base sample entry.

[0028] The base sample entry may comprise an MPEG4 extension descriptor.

[0029] The MPEG4 extension descriptor comprises said one or more enhancement sample entries.

[0030] In a further example implementation of the present disclosure, there may be provided a method comprising: adding an enhancement sample entry to a data structure contained in a media data such that: a legacy player is configured to playback base data associated with a base sample entry in said data structure; and an enhanced player is configured to playback enhancement data associated with said enhancement sample entry and base data associated with said base sample entry. The data structure may be a box or a sample description.

[0031] In a further example implementation of the present disclosure, there may be provided a method of generating a single track multilayer video stream wherein the video stream comprises a base bitstream and at least one enhancement bitstream, the method comprising: inserting an enhancement level decoder configuration into a box containing base decoder configuration wherein the enhancement level decoder configuration is associated with the base decoder configuration.

[0032] The box may be a data structure.

[0033] The box maybe a sample description box, SampleDescriptionBox. The box may be a box of an MPEG File Format, for example, an mp4 file format and in a further example, according to ISO / IEC 14496-15.

[0034] The box may comprise a variable entry_count which indicates a number of sample entry boxes within the box.

[0035] Before the step of inserting the enhancement level decoder configuration the method may comprise adding a base sample entry box within the box, wherein the base sample entry box comprises the base decoder configuration.

[0036] The method may comprise setting a value of the variable entry_count equal to the number of separate bitstreams within the video stream.

[0037] When the video stream comprises one enhancement bitstream, the method may comprise setting the value of the variable entry_count to 2.

[0038] When the video stream comprises n enhancement bitstreams, the method may comprise setting the value of the variable entry_count to n+1 , wherein n is greater than 1.

[0039] The inserting may comprise increasing a value of the variable entry_count and adding an enhancement sample entry box within the box, wherein the enhancement sample entry box comprises the enhancement level decoder configuration.

[0040] The inserting may comprise adding the enhancement sample entry box into the box after the base sample entry box.

[0041] The method may further comprise adding an extension box to the base sample entry box.

[0042] The extension box may be an MPEG4 extension description.

[0043] The method may further comprise adding an enhancement sample entry box to the extension box, wherein the enhancement sample entry box comprises the enhancement level decoder configuration.

[0044] The inserting may comprise adding an extension box to the base sample entry box and adding an enhancement sample entry box into the extension box, and wherein the enhancement sample entry box comprises the enhancement level decoder configuration.

[0045] In a further example implementation of the present disclosure, a data stream may be provided comprising a box wherein the box comprises a base decoder configuration and at least one enhancement level decoder configuration, wherein the enhancement level decoder configuration is associated with the base decoder configuration.

[0046] The box may be a sample description box, SampleDescriptionBox.

[0047] The box may comprisee a variable entry_count which indicates a number of sample entry boxes within the box.

[0048] When the value of the variable entry_count is 1 , the box may comprise a base sample entry box, wherein the base sample entry box comprises the base decoder configuration.

[0049] A value of the variable entry_count may be set to be equal to the number of separate bitstreams within the data stream.

[0050] When the data stream comprises one enhancement bitstream, the value of the variable entry_count may be set to 2.

[0051] When the video stream comprises n enhancement bitstreams, the value of the variable entry_count may be set to n+1 , wherein n is greater than 1 .

[0052] When the variable entry_count is 2, the box may comprise an enhancement sample entry box, wherein the enhancement sample entry box comprises the enhancement level decoder configuration.

[0053] The enhancement sample entry box may be after the base sample entry box.

[0054] The base sample entry box may comprise an extension box.

[0055] The extension box may be a MPEG4 extension description.

[0056] The extension box may comprise at least one enhancement sample entry box, wherein the enhancement sample entry box comprises the enhancement level decoder configuration.

[0057] The enhancement sample entry box may be a descriptor for more than one enhancement bitstream associated with the data stream.

[0058] According to another aspect of the invention there may be provided a device configured to perform the method as described above.

[0059] According to another aspect of the invention there may be provided a device configured to decode the data stream as described above.

[0060] According to another aspect of the invention there may be provided a computer-readable medium comprising instructions which when executed cause a processor to perform the method as described above.Brief Description of the Figures

[0061] Figure 1 is a schematic diagram showing a first example of a multi-layer video stream comprising interleaved NAL units.

[0062] Figure 2 is a schematic diagram showing a second example of a multi-layer video stream comprising base layer NAL units and enhancement SEI messages.

[0063] Figure 3 is a flow diagram showing an example method of parsing a multi-layer video stream.

[0064] Figure 4 shows a set of bit configurations for NAL header data according to an example.

[0065] Figures 5 and 6 are schematic diagrams respectively showing an example multi-layer encoder and decoder configuration.

[0066] Figure 7 is a schematic diagram showing certain data processing operations performed by an example multi-layer encoder.

[0067] Figure 8 is a schematic diagram showing an example of a data stream comprising a sample description comprising one or more enhancement sample entries.

[0068] Figure 9 is a schematic diagram showing an example of a further data stream comprising a sample description comprising one or more enhancement sample entries.Detailed Description

[0069] Certain examples described herein allow decoding devices to be easily adapted to handle multi-layer video coding schemes. Certain examples are described with reference to an LCEVC multi-layer video stream, but the general concepts may be applied to other multi-layer video schemes.

[0070] Different examples are presented. In one set of examples, a method of parsing a multi-layer video stream is described. The method may be seen as a way of adapting an existing parser to handle a multi-layer video stream. The existing parser may be initially configured to parse a non-layered video stream, e.g. a conventional video encoding whereby a single stream of NAL units contains all the data for a decoding of a video signal. In the examples, the parser is modified to handle a multi-layer video stream where data for decoding a video signal is received across different streams of NAL units or via different transport mechanisms. The examples thus allow multi-layer video streams to be stored and / or transmitted as if they were conventional non-layered video streams.

[0071] The present examples enable the carriage of a base video bitstream and an enhancement video bitstream within a single “track”, e.g. within a data structure that resembles a bitstream for just the base video bitstream. For example, a single bitstream carrying both a base video bitstream and an enhancement video bitstream may have a single Packet Identifier (PID) within a transport stream or a single Track Identifier (Track ID) within a file format - thus resembling conventional non-enhanced bitstreams with a single PID or TracklD.

[0072] The present examples provide an alternative approach to so-called “dual track” carriage of base and enhancement bitstreams. In the “dual track” case, a base video bitstream and an enhancement video bitstream may be transmitted and / or stored as two separate bitstreams, e.g. with separate PIDs or in separate tracks.

[0073] The present examples may be applied to multi-layer video bitstreams that are configured according to the specifications of an MPEG-2 Transport Stream (TS) and / or an MPEG-4 File Format (FF). Reference to a multi-layer video bitstream may comprise reference to a multi-layer video bitstream that is suitable for carriage via any one of: MPEG-2 TS, MPEG-4 FF, Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH), or Common Media Application Format (CMAF).

[0074] The present examples may be applied to enhancement bitstream that conforms to standard specification ISO / IEC 23094-2 - so-called MPEG-5 Low Complexity Enhancement Video Codec (LCEVC). In general, the enhancement bitstream is configured to enhance a standalone “base” bitstream, where the base bitstream is a video encoding that may be independently decoded (and viewed if desired) by a base decoder. The enhancement bitstream may comprise a so-called residual bitstream where frames reconstructed from the enhancement bitstream comprises residual frames (i.e., sets of frame residuals) that are combined with frames decoded from the base bitstream to enhance the quality of the base bitstream. In certain cases, certain layers of residual frames may be combined with upsampled reconstructed signals. For example, in LCEVC there is a first enhancement layer, which may be at the resolution of the reconstructed decoded base frame and is able to correct for encoding artifacts, and a second enhancement layer, which is at a higher resolution than the reconstructed decoded base frame, where a residual frame at the higher resolution may be combined following the upsampling of a reconstructed decoded base frame that is corrected with a residual frame from the first enhancement layer. This process is described in more detail with reference to Figures 5 to 7 below.

[0075] In enhancement specifications such as LCEVC, an enhancement bitstream is defined that enhances a separately defined video bitstream (the base). A process is defined to combine the two bitstreams to produce an enhanced video output. As the two bitstreams may be independently decoded, carriage and storage of base and enhancement bitstreams may be performed by a dual-track approach with two separate streams (PIDs or tracks) which can be linked together with a dependency mechanism. The use of such an approach and mechanism may be facilitated by existing dependencymechanisms, for example those already present in MPEG4 FF and already used for carriage of other standard formats. The dual-track approach may facilitate transport of a base bitstream and an enhancement bitstream in two separate physical channels (e.g., the base bitstream via an over-the-air channel and the enhancement bitstream via a broadband channel). However, the dual-track approach may not be suitable for all implementations. For example, there are many commercial media player implementations, particularly for video streaming, where the media player can only support playback of a single stream. Additionally, packaging the bitstream as a single stream may simplify the design of future media players and allow for more widespread adoption of multi-layer video streams. Hence, there are technical benefits to supporting both single and dual-track approaches. The present examples thus consider technical solutions that may support single track approaches.

[0076] In certain examples described herein, a base bitstream and an enhancement bitstream, which may be referred to as a multi-layer video bitstream, is able to be carried and / or stored as a single bitstream. For example, this may comprise a single MPEG2 TS transport stream with a single PID or a single MPEG4 FF track. The layers of such a multilayer bitstream may comprise the “base” as a first layer and one or more enhancement layers (LCEVC has an option of up to two enhancement layers). In certain examples, the base bitstream is generated according to one of the following MPEG video coding specifications: Advanced Video Coding (AVC - also referred to as ISO / IEC 14496-10 or H.264), High Efficiency Video Coding (HEVC - also referred to as ISO / IEC 23008-2 or H.265), and Versatile Video Coding (WC - also referred to as ISO / IEC 23090-3 or H.266). Other past and future video coding specifications may also be supported due to the independence of the enhancement layers - they may be generated based on any input frame from a base layer decoding. As one example, a single “track” multi-layer video stream may comprise a base bitstream (e.g., WC) plus an LCEVC bitstream as a single- track in terms of MPEG2 TS PID or MPEG4 FF track.

[0077] Figures 1 and 2 respectively show two alternative implementations of a single- track multi-layer video stream. Both implementations provide a complete multi-layer video stream comprising a base bitstream and an enhancement bitstream. Figure 1 shows an example with interleaved NAL units. Figure 2 shows an example where the enhancement bitstream is carried in a set of SEI messages.

[0078] In the example 100 of Figure 1 , NAL units for the base bitstream 104 and the enhancement bitstream 106 (in this case LCEVC) are interleaved to provide a single-track multi-layer video stream 102. As enhancement bitstreams may be generated for a variety of different base video coding specifications, each base video coding specification may define its own NAL unit syntax and this syntax may vary between specifications. The NAL units for the enhancement bitstream 106 may be NAL units as defined in section 7.3.2 of ISO / IEC 23094-2.

[0079] In present examples, the format of the enhancement NAL units may be specified to allow for unambiguous detection even when parsed according to the NAL unit syntax of a base bitstream, where the base bitstream may vary according to a number of different base specifications (e.g., the base NAL units may be formatted and processed according to any one of the AVC, HEVC or WC NAL unit syntaxes). This property allows for an “interleaved” single stream of base plus enhancement, where enhancement NAL units are inserted among base NAL units, within the same NAL unit sequence (e.g. as shown in Figure 1 ). In the resulting interleaved single stream, each Access Unit, defined as the set of NAL units that result in each decoded picture, will contain both the enhancement NAL units (e.g., an LCEVC Access Unit contains only one LCEVC NAL unit) and the base NAL units relevant for the specific Access Unit.

[0080] Figure 2 shows an alternative to the interleaved approach of Figure 1. In the example 200 of Figure 2, enhancement data is encapsulated as NAL units, e.g. as defined in section 7.3.2 of ISO / IEC 23094-2, which is then further encapsulated with metadata information 216 for base NAL units 214 to generate a single-track multi-layer video bitstream 102. In specific examples, the metadata information 216 may comprise SEI messages, e.g. as defined by any one of AVC, HEVC and WC. Hence, in Figure 2 NAL units are employed as basic data units and the type of NAL unit identified as Supplemental Enhancement Information (SEI) is used to embed an enhancement NAL unit stream. The SEI messages may be regular size or large size SEI messages. In both cases of Figure 1 and Figure 2, the encapsulation of the bitstream may be within MPEG2 TS “.ts” and MPEG4 FF “.mp4” data structures.

[0081] In certain examples described herein a parser for a base bitstream is modified to process a multi-layer video bitstream, such as one of the bitstream examples shown in Figures 1 and 2. A parser may comprise one or more of software and hardware arranged to process data received from one or more of a file source and a network source. For example, the parser may receive bits read from a data storage device via a file system of a computing device and / or bits output by a network processing stack configured to process data received over a network connection (e.g., wired or wireless). The parsermay be located at the beginning of a media processing pipeline before one or more decoding components. The one or more decoding components may comprise implementations of coding-decoding components - i.e. codecs. These codecs may be implemented via computer program code being processed by a processor and / or via specifically configured electronic hardware (such as Application Specific Integrated Circuits - ASICs - and Field Programmable Gate Arrays - FPGAs). Often codecs may integrate hardware-accelerated computations, i.e. computations that use specially configured electronic chips to speed up operations performed during decoding. In certain cases, a parser may comprise part of a specific codec (e.g., an AVC parser may comprise a first stage in an AVC codec); in other cases, a parser may be provided separately.

[0082] In a comparative case, a comparative (e.g., “unmodified”) parser for video coding receives bits of data structured according to the syntax of a particular video coding standard. For example, an AVC parser may form an early or first component in an AVC decoding pipeline. This comparative parser may receive NAL units for a single video coding specification (e.g., AVC, HEVC or WC NAL units) and process these units to extract encoded data to pass to specific decoding components. For example, a comparative parser may be configured to process NAL units comprising data encoded according to a first video coding specification such as AVC, HEVC, or WC. The parser may comprise a memory to store header data from NAL units belonging to a video stream and a processor configured to extract a NAL unit type value from the header data for a given NAL unit. Responsive to the NAL unit type value falling within a specified range for first video coding specification, the processor may be configured to instruct the decoding of a payload of the given NAL unit according to the first video coding specification. For example, in normal operation, an AVC, HEVC or WC parser receives AVC, HEVC or WC NAL units, inspects the header and performs checks according to the relevant video coding specification before passing encoded data to a decoder that implements the video coding specification. In this comparative case, if the comparative parser encounters a NAL unit with a NAL unit type that is outside of a predefined range (e.g., as discussed later below), the parser is configured to drop or discard the NAL unit (e.g., to not pass the payload data for decoding according to the parser video coding specification). In particular examples, the predefined range is a set of values which are defined as “unspecified” within each video coding specification.

[0083] In the present examples, a modified parser is presented wherein the parser is modified or adapted to process a multi-layer video stream. In these examples, the parsermay comprise a parser that was previously configured to process a single-track video stream for a single non-layered video coding specification, e.g. an AVC, HEVC or WC parser. In other examples, a parser may comprise a newly provided parser that is able to manage a multi-layer video stream comprising multiple video coding specifications (e.g., in addition to conventional single-layer video streams). In both cases, a result is a parser that is configured to process a multi-layer video stream comprising multiple video coding specifications.

[0084] In general, in the present examples, a parser is configured to parse data associated with received NAL units. The parser is configured to identify whether the NAL unit relates to a first video coding (e.g. “base”) specification or whether the NAL unit relates to an additional enhancement video coding specification. This may be performed by inspecting values with the NAL unit metadata. In certain examples, the NAL units for the enhancement bitstream are configured to indicate a metadata value falling within an unspecified range for the base layer specification for a plurality of different base layer specifications, e.g. as unspecified for two or more of AVC, HEVC, and WC. In this manner, if an unmodified parser receives a single-track multi-layer video stream, only data from the NAL units for the base bitstream are decoded; data from the NAL units for the enhancement bitstream are discarded as per the first video coding specification - the bitstream may thus be decoded and viewed as just a normal base bitstream. However, a modified parser is configured to additionally pass the data from the NAL units for the enhancement bitstream to an enhancement decoder for decoding (e.g., in parallel) with data for the first video coding specification. In this manner, the modified parser enables the decoding of both the base and enhancement bitstreams and the eventual viewing of a decoded and enhanced video output. In particular examples, different values within an unspecified range for the first video coding specification may have different meanings or uses for the enhancement video coding specification. Hence, a check may be made to determine if the “unspecified” values match one or more values that are defined within the enhancement video coding specification. In this manner, decoding of the NAL unit by an enhancement decoder may occur responsive to the NAL unit type value falling within a predetermined portion of an unspecified range for the first video coding specification (seen as a base layer specification relative to the enhancement video coding specification).

[0085] Figure 3 shows an example method 300 of parsing a multi-layer video stream. In this example, the multi-layer video stream encodes a video signal and comprises abase layer and one or more enhancement layers. The base layer is encoded using a first or base layer specification (e.g., AVC, HEVC, or WC). The one or more enhancement layers are encoded using an enhancement layer specification. The one or more enhancement layers may comprise one or more sets of frame residuals, a set of frame residuals comprising a difference between a reconstruction derived from a decoded base layer frame and a frame derived from an original video signal at a given level of quality. For example, the one or more enhancement layers may comprise LCEVC layers as described with reference to Figures 5 to 7 below.

[0086] The example method 300 of Figure 3 may be performed as part of a parsing operation that is performed by a modified parser for the base layer. The parsing operation may comprise parsing header data for NAL units belonging to the multi-layer video stream. For example, the parsing operation may be performed for each NAL unit 104, 106 in Figure 1 as they are received at a decoding device. The parsing operation may also be applied to the base NAL unit 214 and an enhancement NAL unit that is extracted from an SEI message 216. The parsing operation may be performed iteratively on each received NAL unit. Figure 3 shows the sub-operations that may be performed on each NAL unit as part of this parsing operation.

[0087] In Figure 3, at step 302, a NAL unit type value is extracted from the header data for a given NAL unit (i.e. , a currently parsed NAL unit). Examples of syntax for NAL unit types for AVC, HEVC, and WC are set out in more detail later below. The header data may comprise a NAL header. NAL headers may comprise one or two bytes. The extraction of the NAL unit type value may depend on the syntax for the base layer and different base layers may implement different syntax. In many base video specifications, the NAL unit type is represented by a defined nal_unit_type field that is represented by a plurality of bits in the NAL header.

[0088] At step 304, the extracted NAL unit type value is compared with one or more ranges of predefined values. In the base layer specification, values may be defined as “specified” or “unspecified”. A range of “specified” values indicates that these values are used (and have meaning within) the base layer specification. Hence, in Figure 3, at step 306, responsive to the NAL unit type value falling within a specified range for the base layer specification, the decoding of a payload of the given NAL unit is instructed according to the base layer specification (e.g., as per the comparative case described above).

[0089] In a non-modified case, if the NAL unit type value does not fall within a specified range for the base layer specification (e.g., is “unspecified”), then the data forthe NAL unit is discarded and a next NAL unit is obtained for parsing. However, in the present modifications, indication of an unspecified NAL unit type instigates further parsing of the NAL unit. In particular, at step 308 a determination is made to check whether the NAL unit type value indicates an enhancement layer NAL unit. For example, this may comprise checking whether the NAL unit type value falls within a predetermined portion of an unspecified range for the base layer specification, i.e. has one or more predefined “unspecified” values. In a case of an AVC parser, these one or more predefined “unspecified” values may be 25 or 27; in the case of an HEVC parser, these one or more predefined “unspecified” values may be 60 or 61 ; in the case of a WC parser, the one or more predefined “unspecified” values may be 31 .

[0090] If the determination is negative, i.e. the NAL unit type value does not indicate an enhancement layer NAL unit, then the NAL unit (and its data) may be discarded at step 310 as per unmodified parsing. However, if the determination is positive, i.e. the NAL unit type value does indicate an enhancement layer NAL unit, e.g. by falling within a particular set of unspecified values, then the method 300 proceeds to step 312 where an enhancement layer NAL unit type is determined. In certain cases, different enhancement layer NAL unit types are associated with different unspecified NAL unit type values. The enhancement layer NAL unit type may indicate whether the given NAL unit is associated with an instantaneous decoding refresh picture for the enhancement layer. For example, an instantaneous decoding refresh (IDR) picture may comprise a picture (e.g., data for a set of colour component frames) for which a NAL unit contains a global configuration data block and does not refer to any other picture for operation of the decoding process of this picture and for which no subsequent pictures in decoding order refer to any picture that precedes it in decoding order. In certain cases, an IDR picture shall occur at least when an IDR picture occurs within the base bitstream.

[0091] Following step 312, the method 300 proceeds to step 314 where the decoding of the payload of the given NAL unit is instructed according to the enhancement layer specification. The decoding is performed based on the determined enhancement layer NAL unit type. This may comprise passing the NAL unit payload to an enhancement decoder or codec. In one case, the enhancement layer NAL unit type may be passed along with the payload but without the header data. In another case, the complete NAL unit (e.g., header and payload) may be passed to the enhancement decoding process. Responsive to the enhancement layer NAL unit type indicating an instantaneous decoding refresh picture, global configuration data may be extracted from the payload ofthe given NAL unit, and the global configuration data may be used to decode a plurality of residual frames, where data for the plurality of residual streams are distributed across multiple NAL units in the multi-layer video stream.

[0092] In certain implementations, e.g. for WC, additional base layer metadata may be checked to process the NAL unit, following a positive determination that the NAL unit type falls within one or more values that are unspecified for the base video coding specification but specified for the enhancement video coding specification. For example, as described later below, if the NAL unit type is specified as WC “unspecified” value “31”, then a WC layer identifier (the additional base layer metadata) that is defined in the NAL header may be examined to see whether values for the layer identifier fall within a particular set of values, such as “57” or “59” in the example below. If the layer identifier is not “57” or “59” the NAL unit may be discarded. In certain cases, an additional check is made on a value of a reserved zero flag prior to instructing the decoding of the payload of the given NAL unit according to the enhancement layer specification, e.g. the layer identifier is only checked if the reserved zero flag is 1 and the unspecified NAL unit type value is 31. Hence, the modified parser is configured to discard a given NAL unit responsive to values within the header data falling outside of a predefined set of metadata values.

[0093] For completeness, in the following paragraphs examples of NAL unit syntax and example syntax implementations are provided. These are provided as examples only and should not be considered limiting - actual implementations may vary in practice based on context.

[0094] The AVC NALU header is defined in ISO / IEC 14496-10, Sec. 7.3.1 , with the following syntax:Table 1 - AVC NALU header syntax

[0095] The NALU type values and semantics for AVC are specified in Table 7-1 of the AVC specification (IS 14496-10). Table 2 summarizes the usage of the AVC NALU types.Since the AVC NALLI type is a field of 5 bits, the possible values are from 0 to 31 . In this case, values of 21 to 23 are “reserved” and values of 24 to 31 are “unspecified”.Table 2 - AVC NALU types

[0096] The HEVC NALU header is defined in ISO / IEC 23008-2, Sec. 7.3.1 .2, with the following syntax:Table 3 - HEVC NALU header syntax

[0097] The NALU type values and semantics for HEVC are specified in Table 7-1 of the HEVC specification (IS 23008-2). Table 4 summarizes the usage of the HEVC NALU types. Since the HEVC NALU type is a field of 6 bits, the possible values are from 0 to63. In this case, values of 41 to 47 are “reserved” and values of 48 to 63 are “unspecified”.Table 4 - HEVC NALU types

[0098] The WC NALU header is defined in ISO / IEC 23090-3, Sec. 7.3.1 .2, with the following syntax:Table 5 - VVC NAL unit header syntax

[0099] The NALLI type values and semantics for WC are specified in Table 5 of the WC specification (IS 23090-3). Table 6 summarizes the usage of the WC NALLI types. Since the WC NALLI type is a field of 5 bits, the possible values are from 0 to 31. In this case, values of 26 to 27 are “reserved” and values of 28 to 31 are “unspecified”. It may be seen from these tables that each video coding specification has a different range of“unspecified” values.Table 6 - VVC NALU types

[0100] The LCEVC NALU header is defined in ISO / IEC 23094-2, Sec. 7.3.2, with the following syntax:Table 7 - LCEVC NAL unit header

[0101] The NALU type values and semantics for LCEVC are specified in Table 17 of the LCEVC specification (IS 23094-2).

[0102] Table 8 summarizes the usage of the LCEVC NALLI types. Since the LCEVC NALLI type is a field of 5 bits, the possible values are from 0 to 31 .Table 8 - LCEVC NALU types

[0103] As shown in the example of Figure 2, when the base encoding is an MPEG standard, the elementary stream is a NAL unit stream. In this case, the encapsulation of enhancement data Access Units as metadata may be implemented using the SEI messages specific for each base coding specification. For example, AVC, HEVC and WC each a different NAL unit format (e.g., with different NAL unit headers), breakdown of NALU types and payloads. However, all of these base coding specifications have an option of using SEI messages. SEI messages may comprise a NAL unit that is identified with a particular NAL unit type field (often referred to as nal_unit_type in the base coding specifications). An example of NAL unit types for SEI messages for AVC, HEVC, and WC is set out in the table below, where RBSP stands for raw byte sequence payload:Table 9 - SEI NALU type

[0104] The SEI raw byte sequence payload (RBSP) and its inner syntax, for the purpose of enhancement encapsulation, is the same across the MPEG standards considered above. In one case, NAL units for an enhancement bitstream may be configured to, use a ITU-T T.35 user data registered type (user_data_registered_itu_t_t35 type) of SEI payload. For example, the general syntax of an SEI NAL unit is:Table 10 - General SEI RBSP format

[0105] The syntax of a sei_message used to carry a ITU-T T.35 payload may be defined as shown in

[0106] Table 1 T.Table 11 - SEI message syntax

[0107] The syntax of a user_data_registered_itu_t_t35 may follow the structure shown in

[0108] Table 12.Table 12 - ITU-T T.35 syntax

[0109] In one case, for example where LCEVC is used as the enhancement layer specification, a manufacturer code may be used to identify a particular form of SEI message. For example, using a manufacturer code, a payload may have sei_payload( 0x04, payloadSize ) = user_data_registered_itu_t_t35 ( payloadSize ), with the following format:Table 13 - ITU-T T.35 syntax with the values specified by an example manufacturer code

[0110] The specifications of the NAL unit header set out in the tables above are illustrated visually in the example 400 of Figure 4. The top of the Figure shows the various sections of the NAL unit header for AVC, HEVC, and WC. In general, a NAL unit header may be one or two bytes and the bits of these bytes are illustrated at 402 across the Figure. As can be seen, the NAL unit header for AVC is one byte where the NAL unit type field is carried by 5 bits - from bit 3 to bit 7 (inclusive). For HEVC, the NAL unit type is 6 bits carried from bit 1 to bit 6 (inclusive). In HEVC, bits 7 to 12 then carry a 6-bit layer identifier and bits 13 to 15 carry a temporal identifier. The layer and temporal identifier are internal fields for HEVC that are used for decoding according to that specification. WC has a similar set of fields to HEVC, but the layer identifier is 6 bits that runs from bit 2 to but 7 (inclusive). The NAL unit type is a 5-bit field that runs from bit 8 to bit 12. Bits 12 to 15 then carry a 3-bit temporal identifier.

[0111] Given the variety of configurations of the NAL header shown in the upper part of Figure 4, a set of configurations of the NAL header for the enhancement bitstream that are compatible with all three base video specifications (e.g., AVC, HEVC, and WC) and the methods described herein are shown in the lower part of the Figure. Row 404 shows possible bit values for an enhancement bitstream NAL header that allow the NAL header to be of an “unspecified” type for all three base video specifications, despite each base vide specification having a different NAL header specification. In particular, in row 404, the first bit is 0, bits 1 to 4 and 7 to 15 are 1 and bits 5 and 6 are either 0 or 1 . Row 406 shows bit values for an enhancement NAL unit where the NAL unit type is identified as 28 for LCEVC and row 408 shows bit values for an enhancement NAL unit where the NAL unit type is identified as 29 for LCEVC. Hence, for LCEVC, the two NALU type values used to identify video coding layer NAL units are 28 (= 0x1 C = 0b1 .1100), which specifies a non-IDR picture and 29 (= 0x1 D = 0b1 .1101 ), which specifies an IDR picture. The two NAL unit header bytes for two video coding layer NAL unit types of LCEVC may thus be as follows:0111 .1001 : 1111 .1111 (LCEVC NALU type 28 Non-IDR) 0111.1011 :1111.1111 (LCEVC NALU type 29 IDR)

[0112] A summary of how a parser for each of AVC, HEVC, and WC would interpret the bits of rows 406 and 408 is set out in the table below (where the first value relates to a parsing of row 406 and the second value relates to a parsing of row 408 - e.g. 406|408):Table 14 - Parsed Values for Data in Figure 4

[0113] As may be seen by comparing the table above and the specifications for AVC, HEVC, and WC, the NAL unit types all fall within the “unspecified” ranges for each of AVC, HEVC, and WC. For the AVC parser, the LCEVC NAL unit types (interpreted as 25 or 27) fall in the “unspecified” range from 24 to 31 . For the HEVC parser, the LCEVC NALU types (interpreted as 60 or 61 ) fall in the “unspecified” range from 48 to 63. For the WC parser, the LCEVC NALU types (interpreted as 31 ) fall in the “unspecified” range from 28 to 31 . As such the enhancement NAL unit types are configured to avoid overlaps with NAL unit types of a used base video coding specification. In all three cases, the enhancement NAL unit header as parsed by a base decoder is detected as a NAL unit type in a defined “unspecified” range. In this way, a base parser which is unaware of an enhancement video coding specification such as LCEVC would continue classifying the enhancement NAL units as “unspecified”. To be aware of an enhancement bitstream, a base parser may be modified as described herein.

[0114] In particular, applying the method 300 of Figure 3 to the example NAL header configurations of Figure 4, the operation of an AVC, HEVC and WC parser may be modified in the following manner to correctly recognize enhancement NAL units. For example, the behaviour of an LCEVC-aware AVC parser may be modified to perform the following logic: if nal_unit_type = 25: treat as LCEVC_NON_IDR; else if nal_unit_type = 27: treat as LCEVCJDR; else: treat as per AVC (ISO / IEC 14496-10 standard).

[0115] Here, LCEVC_NON_IDR is a flag indicating an LCEVC non-IDR picture and LCEVCJDR is a flag indicating an LCEVC IDR picture. Similarly, the behaviour of an LCEVC-aware HEVC parser may be modified to perform the following logic: if nal_unit_type = QC. treat as LCEVC_NON_IDR; else if nal_unit_type = 61 : treat as LCEVCJDR; else: treat as per HEVC (ISO / IEC 23008-2 standard).

[0116] For WC, the behaviour of an LCEVC-aware WC parser may be modified to perform the following logic: if nuh_reserved_zero_bit = 1 and nal_unit_type = 31 : if nuh_layer_id = 57: treat as LCEVC_NON_IDR; else if nuh_layer_id = 59: treat as LCEVCJDR; else discard; else: treat as per WC (ISO / IEC 23090-3 standard)

[0117] In this case, the method is adapted such that a given NAL unit is discarded responsive to values for additional base layer metadata within the header data falling outside of a predefined range, where the base layer metadata comprises the layer identifier. An additional check is also made on a value of a reserved zero flag prior to instructing the decoding of the payload of the given NAL unit according to the enhancement layer specification.

[0118] Certain general information relating to example enhancement coding schemes will now be described. This information provides examples of specific multi-layer coding schemes.

[0119] It should be noted that examples are presented herein with reference to a signal as a sequence of samples (i.e., two-dimensional images, video frames, video fields, sound frames, etc.). For simplicity, non-limiting examples illustrated herein often refer to signals that are displayed as 2D planes of settings (e.g., 2D images in a suitable colour space), such as for instance a video signal. In a preferred case, the signal comprises a video signal. An example video signal is described in more detail with reference to Figure 7.

[0120] The terms “picture”, “frame” or “field” are used interchangeably with the term “image”, so as to indicate a sample in time of the video signal: any concepts and methods illustrated for video signals made of frames (progressive video signals) can be easily applicable also to video signals made of fields (interlaced video signals), and vice versa. Despite the focus of examples illustrated herein on image and video signals, people skilled in the art can easily understand that the same concepts and methods are also applicable to any other types of multidimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, plenoptic signals, point clouds, etc.). Although image or video coding examples are provided, the same approaches may be applied to signals with dimensions fewer than two (e.g., audio or sensor streams) or greater than two (e.g., volumetric signals).

[0121] In the description the terms “image”, “picture” or “plane” (intended with the broadest meaning of “hyperplane”, i.e. , array of elements with any number of dimensions and a given sampling grid) will be often used to identify the digital rendition of a sample of the signal along the sequence of samples, wherein each plane has a given resolution for each of its dimensions (e.g., X and Y), and comprises a set of plane elements (or “element”, or “pel”, or display element for two-dimensional images often called “pixel”, for volumetric images often called “voxel”, etc.) characterized by one or more “values” or “settings” (e.g., by ways of non-limiting examples, colour settings in a suitable colour space, settings indicating density levels, settings indicating temperature levels, settings indicating audio pitch, settings indicating amplitude, settings indicating depth, settings indicating alpha channel transparency level, etc.). Each plane element is identified by a suitable set of coordinates, indicating the integer positions of said element in the sampling grid of the image. Signal dimensions can include only spatial dimensions (e.g., in the case of an image) or also a time dimension (e.g., in the case of a signal evolving over time, such as a video signal). In one case, a frame of a video signal may be seen to comprise a two-dimensional array with three colour component channels or a three-dimensional array with two spatial dimensions (e.g., of an indicated resolution - with lengths equal to the respective height and width of the frame) and one colour component dimension (e.g., having a length of 3). In certain cases, the processing described herein is performed individually to each plane of colour component values that make up the frame. For example, planes of pixel values representing each of Y, II, and V colour components may be processed in parallel using the methods described herein.

[0122] Certain examples described herein use a scalability framework that uses a base encoding and an enhancement encoding. The video coding systems described herein operate upon a received decoding of a base encoding (e.g., frame-by-frame or complete base encoding) and add one or more of spatial, temporal, or other quality enhancements via an enhancement layer. The base encoding may be generated by a base layer, which may use a coding scheme that differs from the enhancement layer, and in certain cases may comprise a legacy or comparative (e.g., older) coding standard.

[0123] Figure 5 to 7 show a spatially scalable coding scheme that uses a down- sampled source signal encoded with a base codec, adds a first level of correction or enhancement data to the decoded output of the base codec to generate a corrected picture, and then adds a further level of correction or enhancement data to an up-sampled version of the corrected picture. Thus, the spatially scalable coding scheme may generatean enhancement stream with two spatial resolutions (higher and lower), which may be combined with a base stream at the lower spatial resolution.

[0124] In the spatially scalable coding scheme, the methods and apparatuses may be based on an overall algorithm which is built over an existing encoding and / or decoding algorithm (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, etc. as well as nonstandard algorithms such as VP9, AV1 , and others) which works as a baseline for an enhancement layer. The enhancement layer works accordingly to a different encoding and / or decoding algorithm. The idea behind the overall algorithm is to encode / decode hierarchically the video frame as opposed to using block-based approaches as done in the MPEG family of algorithms. Hierarchically encoding a frame includes generating residuals for the full frame, and then a reduced or decimated frame and so on.

[0125] Figure 5 shows a system configuration for an example spatially scalable encoding system 500. The encoding process is split into two halves as shown by the dashed line. Each half may be implemented separately. Below the dashed line is a base level and above the dashed line is the enhancement level, which may usefully be implemented in software. The encoding system 500 may comprise only the enhancement level processes, or a combination of the base level processes and enhancement level processes as needed. The encoding system 500 topology at a general level is as follows. The encoding system 500 comprises an input I for receiving an input signal 501 . The input I is connected to a down-sampler 505D. The down-sampler 505D outputs to a base encoder 520E at the base level of the encoding system 500. The down-sampler 505D also outputs to a residual generator 510-S. An encoded base stream is created directly by the base encoder 520E, and may be quantised and entropy encoded as necessary according to the base encoding scheme. The encoded base stream may be the base layer as described above, e.g. a lowest layer in a multi-layer coding scheme.

[0126] Above the dashed line is a series of enhancement level processes to generate an enhancement layer of a multi-layer coding scheme. In the present example, the enhancement layer comprises two sub-layers. In other example, one or more sub-layers may be provided. In Figure 5, to generate an encoded sub-layer 1 enhancement stream, the encoded base stream is decoded via a decoding operation that is applied at a base decoder 520D. In preferred examples, the base decoder 520D may be a decoding component that complements an encoding component in the form of the base encoder 520E within a base codec. In other examples, the base decoding block 520D may instead be part of the enhancement level. Via the residual generator 510-S, a difference betweenthe decoded base stream output from the base decoder 520D and the down-sampled input video is created (i.e., a subtraction operation 510-S is applied to a frame of the down-sampled input video and a frame of the decoded base stream to generate a first set of residuals). Here, residuals represent the error or differences between a reference signal or frame and a desired signal or frame. The residuals used in the first enhancement level can be considered as a correction signal as they are able to ‘correct’ a frame of a future decoded base stream. This is useful as this can correct for quirks or other peculiarities of the base codec. These include, amongst others, motion compensation algorithms applied by the base codec, quantisation and entropy encoding applied by the base codec, and block adjustments applied by the base codec.

[0127] In Figure 5, the first set of residuals are transformed, quantised and entropy encoded to produce the encoded enhancement layer, sub-layer 1 stream. In Figure 5, a transform operation 510-1 is applied to the first set of residuals; a quantisation operation 520-1 is applied to the transformed set of residuals to generate a set of quantised residuals; and, an entropy encoding operation 530-1 is applied to the quantised set of residuals to generate the encoded enhancement layer, sub-layer 1 stream (e.g., at a first level of enhancement). However, it should be noted that in other examples only the quantisation step 520-1 may be performed, or only the transform step 510-1. Entropy encoding may not be used, or may optionally be used in addition to one or both of the transform step 510-1 and quantisation step 520-1. The entropy encoding operation can be any suitable type of entropy encoding, such as a Huffmann encoding operation or a run-length encoding (RLE) operation, or a combination of both a Huffmann encoding operation and a RLE operation (e.g., RLE then Huffmann or prefix encoding).

[0128] To generate the encoded enhancement layer, sub-layer 2 stream, a further level of enhancement information is created by producing and encoding a further set of residuals via residual generator 500-S. The further set of residuals are the difference between an up-sampled version (via up-sampler 505U) of a corrected version of the decoded base stream (the reference signal or frame), and the input signal 501 (the desired signal or frame).

[0129] To achieve a reconstruction of the corrected version of the decoded base stream as would be generated at a decoder (e.g., as shown in Figure 6), at least some of the sub-layer 1 encoding operations are reversed to mimic the processes of the decoder, and to account for at least some losses and quirks of the transform and quantisation processes. To this end, the first set of residuals are processed by a decoding pipelinecomprising an inverse quantisation block 520-1 i and an inverse transform block 510-1 i. The quantised first set of residuals are inversely quantised at inverse quantisation block 520-1 i and are inversely transformed at inverse transform block 510-1 i in the encoding system 500 to regenerate a decoder-side version of the first set of residuals. The decoded base stream from decoder 520D is then combined with the decoder-side version of the first set of residuals (i.e., a summing operation 510-C is performed on the decoded base stream and the decoder-side version of the first set of residuals). Summing operation 510- C generates a reconstruction of the down-sampled version of the input video as would be generated in all likelihood at the decoder - i.e. a reconstructed base codec video). The reconstructed base codec video is then up-sampled by up-sampler 505LI. Processing in this example is typically performed on a frame-by-frame basis. Each colour component of a frame may be processed as shown in parallel or in series.

[0130] The up-sampled signal (i.e., reference signal or frame) is then compared to the input signal 501 (i.e., desired signal or frame) to create the further set of residuals (i.e., a difference operation is applied by the residual generator 500-S to the up-sampled recreated frame to generate a further set of residuals). The further set of residuals are then processed via an encoding pipeline that mirrors that used for the first set of residuals to become an encoded enhancement layer, sub-layer 2 stream (i.e., an encoding operation is then applied to the further set of residuals to generate the encoded further enhancement stream). In particular, the further set of residuals are transformed (i.e., a transform operation 510-0 is performed on the further set of residuals to generate a further transformed set of residuals). The transformed residuals are then quantised, and entropy encoded in the manner described above in relation to the first set of residuals (i.e., a quantisation operation 520-0 is applied to the transformed set of residuals to generate a further set of quantised residuals; and, an entropy encoding operation 530-0 is applied to the quantised further set of residuals to generate the encoded enhancement layer, sublayer 2 stream containing the further level of enhancement information). In certain cases, the operations may be controlled, e.g. such that, only the quantisation step 520-1 may be performed, or only the transform and quantisation step. Entropy encoding may optionally be used in addition. Preferably, the entropy encoding operation may be a Huffmann encoding operation or a run-length encoding (RLE) operation, or both (e.g., RLE then Huffmann encoding). The transformation applied at both blocks 510-1 and 510-0 may be a Hadamard transformation that is applied to 2x2 or 4x4 blocks of residuals.

[0131] The encoding operation in Figure 5 does not result in dependencies between local blocks of the input signal (e.g., in comparison with many known coding schemes that apply inter or intra prediction to macroblocks and thus introduce macroblock dependencies). Hence, the operations shown in Figure 5 may be performed in parallel on 4x4 or 2x2 blocks, which greatly increases encoding efficiency on multicore central processing units (CPUs) or graphical processing units (GPUs).

[0132] As illustrated in Figure 5, the output of the spatially scalable encoding process is one or more enhancement streams for an enhancement layer which preferably comprises a first level of enhancement and a further level of enhancement. This is then combinable (e.g., via multiplexing or otherwise) with a base stream at a base level. The first level of enhancement (sub-layer 1 ) may be considered to enable a corrected video at a base level, that is, for example to correct for encoder quirks. The second level of enhancement (sub layer 2) may be considered to be a further level of enhancement that is usable to convert the corrected video to the original input video or a close approximation thereto. For example, the second level of enhancement may add fine detail that is lost during the downsampling and / or help correct from errors that are introduced by one or more of the transform operation 510-1 and the quantisation operation 520-1 .

[0133] Figure 6 shows a corresponding example decoding system 600 for the example spatially scalable coding scheme. In Figure 6, the encoded base stream is decoded at base decoder 620 in order to produce a base reconstruction of the input signal 501 . This base reconstruction may be used in practice to provide a viewable rendition of the signal 501 at the lower quality level. However, the primary purpose of this base reconstruction signal is to provide a base for a higher quality rendition of the input signal 501 . To this end, the decoded base stream is provided for enhancement layer, sub-layer 1 processing (i.e. , sub-layer 1 decoding). Sub-layer 1 processing in Figure 6 comprises an entropy decoding process 630-1 , an inverse quantisation process 620-1 , and an inverse transform process 610-1. Optionally, only one or more of these steps may be performed depending on the operations carried out at corresponding block 500-1 at the encoder. By performing these corresponding steps, a decoded enhancement layer, sublayer 1 stream comprising the first set of residuals is made available at the decoding system 600. The first set of residuals is combined with the decoded base stream from base decoder 620 (i.e., a summing operation 610-C is performed on a frame of the decoded base stream and a frame of the decoded first set of residuals to generate a reconstruction of the down-sampled version of the input video - i.e. the reconstructedbase codec video). A frame of the reconstructed base codec video is then up-sampled by up-sampler 60511.

[0134] Additionally, and optionally in parallel, the encoded enhancement layer, sublayer 2 stream is processed to produce a decoded further set of residuals. Similar to sublayer 1 processing, enhancement layer, sub-layer 2 processing comprises an entropy decoding process 630-0, an inverse quantisation process 620-0 and an inverse transform process 610-0. Of course, these operations will correspond to those performed at block 500-0 in encoding system 500, and one or more of these steps may be omitted as necessary. Block 600-0 produces a decoded enhancement layer, sub-layer 2 stream comprising the further set of residuals, and these are summed at operation 600-C with the output from the up-sampler 605LI in order to create an enhancement layer, sub-layer 2 reconstruction of the input signal 501 , which may be provided as the output of the decoding system 800. Thus, as illustrated in Figure 5 and 6, the output of the decoding process may comprise up to three outputs: a base reconstruction, a corrected lower resolution signal and an original signal reconstruction for the multi-layer coding scheme at a higher resolution.

[0135] In general, examples described herein operate within encoding and decoding pipelines that comprises at least a transform operation. The transform operation may comprise the DCT or a variation of the DCT, a Fast Fourier Transform (FFT), or, in preferred examples, a Hadamard transform as implemented by LCEVC. The transform operation may be applied on a block-by-block basis. For example, an input signal may be segmented into a number of different consecutive signal portions or blocks and the transform operation may comprise a matrix multiplication (i.e., linear transformation) that is applied to data from each of these blocks (e.g., as represented by a 1 D vector). In this description and in the art, a transform operation may be said to result in a set of values for a predefined number of data elements, e.g. representing positions in a resultant vector following the transformation. These data elements are known as transformed coefficients (or sometimes simply “coefficients”).

[0136] As described herein, where the enhancement data comprises residual data, a reconstructed set of coefficient bits may comprise transformed residual data, and a decoding method may further comprise instructing a combination of residual data obtained from the further decoding of the reconstructed set of coefficient bits with a reconstruction of the input signal generated from a representation of the input signal at a lower level of quality to generate a reconstruction of the input signal at a first level ofquality. The representation of the input signal at a lower level of quality may be a decoded base signal and the decoded base signal may be optionally upscaled before being combined with residual data obtained from the further decoding of the reconstructed set of coefficient bits, the residual data being at a first level of quality (e.g., a first resolution). Decoding may further comprise receiving and decoding residual data associated with a second sub-layer, e.g. obtaining an output of the inverse transformation and inverse quantisation component, and combining it with data derived from the aforementioned reconstruction of the input signal at the first level of quality. This data may comprise data derived from an upscaled version of the reconstruction of the input signal at the first level of quality, i.e. an upscaling to the second level of quality.

[0137] Further details and examples of a two sub-layer enhancement encoding and decoding system may be obtained from published LCEVC documentation. Although examples have been described with reference to a tier-based hierarchical coding scheme in the form of LCEVC, the methods described herein may also be applied to other tierbased hierarchical coding scheme, such as VC-6: SMPTE VC-6 ST-2117 as described in PCT / GB2018 / 053552 and / or the associated published standard document, which are both incorporated by reference herein.

[0138] Figure 7 shows an example 700 of how a video signal may be decomposed into different components and then encoded. In the example of Figure 7, a video signal 702 is encoded. The video signal 702 comprises a plurality of frames or pictures 704, e.g. where the plurality of frames represent action over time. In this example, each frame 704 is made up of three colour components. The colour components may be in any known colour space. In Figure 7, the three colour components 706 are Y (luma), II (a first chroma opponent colour) and V (a second chroma opponent colour). Each colour component may be considered a plane 708 of values. The plane 708 may be decomposed into a set of n by n blocks of signal data 710. For example, in LCEVC, n may be 2 or 4; in other video coding technologies n may be 8 to 32.

[0139] In LCEVC and certain other coding technologies, a video signal fed into a base layer is a downscaled version of the input video signal, e.g. 501 . In this case, the signal that is fed into both sub-layers of the enhancement layer comprises a residual signal comprising residual data. A plane of residual data may also be organised in sets of n-by- n blocks of signal data 710. The residual data may be generated by comparing data derived from the input signal being encoded, e.g. the video signal 501 , and data derived from a reconstruction of the input signal, the reconstruction of the input signal beinggenerated from a representation of the input signal at a lower level of quality. The comparison may comprise subtracting the reconstruction from the downsampled version. The comparison may be performed on a frame-by-frame (and / or block-by-block) basis. The comparison may be performed at the first level of quality; if the base level of quality is below the first level of quality, a reconstruction from the base level of quality may be upscaled prior to the comparison. In a similar manner, the input signal to the second sublayer, e.g. the input for the second sub-layer transformation and quantisation component, may comprise residual data that results from a comparison of the input video signal 501 at the second level of quality (which may comprise a full-quality original version of the video signal) with a reconstruction of the video signal at the second level of quality. As before, the comparison may be performed on a frame-by-frame (and / or block-by-block) basis and may comprise subtraction. The reconstruction of the video signal may comprise a reconstruction generated from the decoded decoding of the encoded base bitstream and a decoded version of the first sub-layer residual data stream. The reconstruction may be generated at the first level of quality and may be upsampled to the second level of quality.

[0140] Hence, a plane of data 708 for the first sub-layer may comprise residual data that is arranged in n-by-n signal blocks 710. One such 2 by 2 signal block is shown in more detail in Figure 7 (n is selected as 2 for ease of explanation) where for a colour plane the block may have values 712 with a set bit length (e.g., 8 or 16-bit). Each n-by-n signal block may be represented as a flattened vector 714 of length n2representing the blocks of signal data. To perform the transform operation, the flattened vector 714 may be multiplied by a transform matrix 716 (i.e. , the dot product taken). This then generates another vector 718 of length n2representing different transformed coefficients for a given signal block 710. Figure 7 shows an example similar to LCEVC where the transform matrix 716 is a Hadamard matrix of size 4 by 4, resulting in a transformed coefficient vector 718 having four elements with respective values. These elements are sometimes referred to by the letters A, H, V and D as they may represent an average, horizontal difference, vertical difference and diagonal difference. Such a transform operation may also be referred to as a directional decomposition. When n = 4, the transform operation may use a 16 by 16 matrix and be referred to as a directional decomposition squared.

[0141] As shown in Figure 7, the set of values for each data element across the complete set of signal blocks 710 for the plane 708 may themselves be represented as a plane or surface of coefficient values 720. For example, values for the “H” data elementsfor the set of signal blocks may be combined into a single plane, where the original plane 708 is then represented as four separate coefficient planes 722. For example, the illustrated coefficient plane 722 contains all the “H” values. These values are stored with a predefined bit length, e.g. a bit length B, which may be 8, 16, 32 or 64 depending on the bit depth. A 16-bit example is considered below but this is not limiting. As such, the coefficient plane 722 may be represented as a sequence (e.g., in memory) of 16-bit or 2- byte values 724 representing the values of one data element from the transformed coefficients. These may be referred to as coefficient bits. These coefficient bits may be quantised and then entropy encoded as discussed to then generate the encoded enhancement layer data as described above.

[0142] Whilst carriage of Base and LCEVC in a Single Track, based on Interleaved NALUs or Embedded SEI Messages (see Figures 1 and 2 respectively) provide many advantages these implementations still leave open the possibility that a legacy player discards the track completely without any playback.

[0143] Figures 8 and 9 show two implementations of a single-track multi-layer video stream which aim to reduce, and preferably remove entirely, the possibility that a legacy player will discard the entire track without playback of any sort. The aim of these embodiments is to present a solution to provide a “Single Track” carriage of a Base bitstream plus an enhancement (e.g. LCEVC) bitstream that is backward compatible with ‘legacy’ players.

[0144] "Single Track" carriage of a Base bitstream plus an enhancement bitstream generally refers to combining the two bitstreams into a single track or file. The Base bitstream contains the basic video or audio content, while the enhancement bitstream contains additional information that improves the quality or adds new features to the content (such as an LCEVC enhancement). By combining the Base and enhancement bitstreams into a single track, it becomes easier to manage and transport the media content. The combination can be achieved by using a multiplexer to combine the two bitstreams or by embedding the enhancement bitstream data within the Base bitstream.

[0145] As previously described, the embodiments described in figures 1 and 2 (i.e. carriage of Base and LCEVC in a Single Track, respectively based on Interleaved NALUs or Embedded SEI Messages) of the present application provide multiple advantages as they provide for LCEVC and Base in a single track. However, one potential danger of any single track approach is that a legacy player may discard the entire track if it does not recognise a part of the track (e.g. the enhancement part), potentially resulting in noplayback whatsoever (this is in contrast with some multi track approaches, where the legacy player may recognise and playback a base track but merely ignores the enhancement track, which ultimately results in playback of some sort, albeit at the base level of quality).

[0146] Such a danger is realised in a ‘comparative solution’, whereby: two new “Boxes” (in NALLI FF, IS 14496-15, terminology) are utilised (for example the boxes LCEVCMixedSampleEntry ( ) extends VisualSampleEntry ( 'Ivms ' ) andBaseConf igurationBox ( ) extends Box ( ' bicf ) ) to associate an LCEVC bitstream to a relevant Base bitstream (e.g. AVC, HEVC, WC, EVC). This comparative solution is based on an Aggregator structure, as defined in clause A.2 of 14496-15 (NALLI FF). Disadvantageously, this comparative example makes the NALLI FF inherently nonbackward compatible. This is because a ‘legacy’ player (e.g. a device that has not been updated to decode LCEVC) would discard both the base and enhancement because of the 2 ‘new’ boxes, resulting in no playback.

[0147] This comparative example helps to illustrate a potential issue with single track approaches: the question of how to carry two different codecs (e.g. via DecoderConfigurationBoxes) in a Single Track, whilst preserving backward compatibility with existing (e.g. MP4) players. This is important because in order for a player to decode a stream, the player needs to be told what decoder(s) to use. In other words, without any enhancement configuration box, the player wouldn’t be able to handle the enhancement stream. However, inclusion of the enhancement configuration box may jeopardise the entire stream, resulting in no playback whatsoever.

[0148] It would therefore be advantageous if such ‘single track’ schemes are designed to be “backward compatible”, so that ‘clients’ (i.e. decoding devices, players and so forth, such as MP4 Players) not supporting an enhancement layer and / or enhancement codec (e.g. such as LCEVC) are still able to play the Base bitstream (such as AVC, HEVC, WC), and ‘clients’ (i.e. decoding devices, players and so forth, such as MP4 Players) supporting an enhancement layer and / or enhancement codec (e.g. LCEVC) are able to play both Base and Enhancement. The backwards compatibility is achieved by re-using existing constructions within current specifications. Figures 8 and 9 illustrate embodiments of a concept that solves such an issue. Figures 8 and 9 are two alternative solutions which can be used to handle single track carriage of a base bitstream plus an enhancement bitstream. The solution shown in Figure 8 re-uses the existing SampleDescriptionBox specification for a generic base bitstream. The solution shown inFigure 9 re-uses the SampleEntry specifications for different base bitstreams and re-uses the existing mechanism for extension of the VisualSampleEntry. Although these solutions are alternatives, they both allow the possibility of carrying two different DecoderConfigurationBoxes in a single track, preserving backward compatibility.

[0149] In summary, rather than defining two new “Boxes” that make the NALLI FF inherently non-backward compatible, embodiments of the present invention reuse (and add to) existing “Boxes”. For completeness, a “box” refers to a structural component used in the file format specification for storing multimedia data. The file format is designed to be a flexible container for various types of data including video, audio, metadata etc. A box is a hierarchical data structure comprising a header and a payload. The payload can include actual data as well as further nested boxes. The box data structure allows for efficient parsing, editing, and streaming of multimedia data in the MPEG-4 file format, and provides a useful way to organize and encapsulate different types of data within a single file container.

[0150] For example, as mentioned above, figure 8 illustrates an embodiment where the ‘existing’ box SampieDes criptionBox is utilised. For example, as mentioned above, figure 9 illustrates an embodiment where the ‘existing’ box MPEG4ExtensionDes criptorsBox is utilised. Such embodiments provide backward compatibility. Such embodiments utilise the feature that any compliant player will ignore and discard boxes which they don’t understand (i.e. cannot compute). Thus a ‘legacy’ player (e.g. a device that has not been updated to decode LCEVC) would still comprehend the ‘base’ (i.e. legacy) box and would only ignore and discard the enhancement box, thus resulting in playback at least of the base data.

[0151] In general, the enhancement configuration box could be added at different levels, for example at a level of descriptor.

[0152] Embodiments of the present invention have been implemented to edit an existing MP4 file with AVC or HEVC video, and to insert the LCEVC Decoder Configuration. It was found that the edited MP4 file is indeed backward compatible with existing players.

[0153] Advantageously, embodiments of the present invention can be used to provide backward compatibility to any single track carnage delivery of a multi-layer video scheme.

[0154] One embodiment, as shown in figure 8, utilises the existing SampieDescriptionBox specification for a generic Base bitstream SampieDes criptionBox (found in ISO / IEC 14496-12, clause 8.5.2.2).

[0155] The schematic in figure 8 shows data 800 comprising Sample Table (‘stbl’) 801. Sample Table (‘stbl’) 801 comprises Sample Description (‘stsd’) 803. The Sample Description box 803 is a child box of the Sample Table box 81 and provides a detailed description of the media samples within a specific track. In general, the purpose of Sample Description Box 803 is to store information about the coding and data format used for the samples in a track. Sample Description box 803 contains one or more Sample Entry boxes, each describing a specific coding type and providing relevant parameters for decoding and rendering the samples. The Sample Description box 803 comprises a data field called entry_count which specifies the number of Sample Entry boxes contained within the Sample Description box 803. Each Sample Entry box describes a specific coding format and provides the necessary parameters for decoding and rendering the samples of that coding type. The entry_count field tells a parser or media player how many Sample Entry boxes are present in the Sample Description box, allowing it to correctly read and process each one. The value of entry_count determines the number of Sample Entry boxes that follow in the Sample Description box structure. Generally, the entry_count value is set during the creation of the MPEG-4 file, based on the number of different coding formats used for the media samples within a particular track. It is derived from the actual contents of the Sample Description box and the number of Sample Entry boxes included within it. To increase or decrease the entry_count value, Sample Entry boxes are added or removed from the Sample Description box. In this embodiment in Figure 8, Sample Description (‘stsd’) 803 has an entry count of 2 indicating that there are two Sample Entry boxes within the Sample Description box 803. We can see in Figure 8 that Sample Description (‘stsd’) 803 comprises a Base Sample Entry 805 (equal to ‘avcT if the base codec is AVC, equal to ‘hvcT if the base codec is HEVC, equal to ‘vvcT if the base codec is WC) 805 and enhancement Sample Entry 807 (equal to ‘IvcT if the enhancement codec is LCEVC). The base sample entry 805 and the enhancement sample entry 807 are both examples of Sample Entry boxes.

[0156] In relation to Figure 8, we describe a method of generating data 800, an encoder configured to generate data 800, a decoder configured to decode data 800 and a bitstream comprising data 800. Sample Table (‘stbl’) 801 , Sample Description (‘stsd’) 803, Base Sample Entry 805 and enhancement Sample Entry 807 may be referred to as Sample Table (‘stbl’) box 801 , Sample Description (‘stsd’) box 803, Base Sample Entry box 805 and enhancement Sample Entry box 807 respectively. In addition or alternatively, Sample Table (‘stbl’) 801 , Sample Description (‘stsd’) 803, Base Sample Entry 805 andenhancement Sample Entry 807 may be referred to as SampleTable 801 , SampleDescriptionBox 803, Base SampleEntry 805 and enhancement SampleEntry 807 respectively.

[0157] ISO / IEC 14496-12 refers to Part 12 of ISO / IEC 14496-12, named “ISO base media file format”. This generally relates to a file format for storing time-based media content. It is a general format forming the basis for a number of other more specific file formats (e.g. 3GP, Motion JPEG 2000, MPEG-4 Part 14, MPEG Part 15). It is technically identical to ISO / IEC 15444-12 (JPEG 2000 image coding system - Part 12). MPEG-4 Part 12, also known as ISO / IEC 14496-12, is a standard that defines the file format for multimedia content in the MPEG-4 family. It specifies the structure of the MP4 file format, including how audio, video, and other multimedia data are stored and interleaved within the file. MPEG-4 Part 12 is widely used in various applications, such as digital broadcasting, streaming media, and mobile devices.

[0158] The general purpose of Clause 8.5.2 pertains to the storage of metadata in MP4 files, related to Sample Description. This clause specifies the requirements for storing metadata boxes, which are usually used to convey detailed additional information about the coding type used, and any initialization information needed for that coding. The clause defines the syntax and semantics of the metadata boxes, including their box type, size, and structure, as well as the encoding of the metadata values. Overall, Clause 8.5.2 is important for ensuring the interoperability of MP4 files across different systems and applications, by providing a standard mechanism for storing and retrieving metadata associated with the multimedia content.

[0159] As has been discussed, the Sample Description Box is a box structure defined in clause 8.5.2.2 of ISO / IEC 14496-12, also known as the MP4 file format specification. The Sample Description Box contains information about the format and characteristics of the media samples in a track, such as video frames or audio samples. The purpose of the Sample Description Box is to provide the necessary information for a media player or decoder to correctly interpret and decode the media samples in a track. The box includes fields that describe the codec used to encode the samples, the bit depth of the samples, the frame rate, the video dimensions, and other parameters that are needed for proper decoding and playback. The Sample Description Box is an essential part of an MP4 file, as it enables the efficient and accurate decoding and rendering of the media samples in the file. The box is typically located within the Sample Table Box, which is located in the Media Information Box, which is located within the Media Box, which in turn is locatedwithin the Track Box. In summary, the Sample Description Box in ISO / IEC 14496-12 serves the important function of providing information about the format and characteristics of media samples in a track, enabling efficient and accurate decoding and playback of media content in an MP4 file.

[0160] This embodiment enables insertion of a LCEVC Decoder Configuration associated to the Base Decoder configuration by using a (e.g. ‘same’) SampleDes criptionBox containing the Base Decoder configuration. For completeness, a decoder configuration refers to a set of parameters that describe the capabilities and requirements of a decoder for a specific coding format, and it provides essential information that allows the decoder to correctly process and decode the encoded media data. Thus, the base decoder configuration provides information to allow the decoder to decode the base stream and the LCEVC decoder configuration provides information to allow the decoder to decode the enhancement stream. Each decoder configuration is encapsulated within a data structure contained within a Sample Entry box. For example, the Base Decoder configuration is contained within the base Sample Entry box and the LCEVC Decoder configuration is contained within the enhancement Sample Entry box. Since both the base Sample Entry box and the enhancement Sample Entry box are contained within the Sample Description box (in particular, within the same Sample Descriptoin box), it can be seen that both the Base Decoder configuration and the LCEVC Decoder configuration are contained within the same Sample Description box. A LCEVC Decoder Configuration associated to the Base Decoder configuration means that for a multi-layer video bitstream comprising a base bitstream and an enhancement bitstream, the LCEVC Decoder Configuration comprises parameters for the enhancement bitstream that belongs to the same multi-layer video bitstream whose base bitstream parameters are comprised in the Base Decoder configuration. A given multi-layer video bitstream may comprise more than one enhancement bitstream and so more than one LCEVC Decoder Configuration can be associated with a given Base Decoder configuration.

[0161] BASEFF 8.5.2.2

[0162] In relation to ISO / IEC 14496, BASEFF 8.5.2 refers to the base file format specification for audio and video files in the MPEG-4 standard. Clause 8.5.2 of ISO / IEC 14496-12 defines the SampleDescriptionBox, which specifies the format and encoding of audio and video samples in the file. The BASEFF (Base File Format) refers to the baseline specification for the MPEG-4 file format, which defines the minimum set of features andcapabilities that must be supported by all compliant MPEG-4 file formats. The BASEFF specification includes the basic structure of the file, such as the boxes and boxes hierarchy, as well as the basic media types and their encodings. In the context of clause 8.5.2 of ISO / IEC 14496-12, BASEFF 8.5.2 refers to the baseline specification for the SampleDescriptionBox, which includes the basic format and encoding information for audio and video samples in the file. This baseline specification must be supported by all compliant MPEG-4 file formats, and can be extended or modified by additional boxes and extensions to meet specific needs and requirements, embodiments of the present invention make use of such functionality.

[0163] SampleDescriptionBox aligned(8) class SampleDescriptionBox () extends FullBox ('stsd', version, 0) { int i; unsigned int(32) entry_count; for (i = 1 ; i <= entry_count ; i++) {SampleEntryQ; / / an instance of a class derived from SampleEntry}}

[0164] The SampleDescriptionBox allows the allocation of one or more SampleEntry, depending on the field entry_count. Thus, the LCEVC Sample Entry can be inserted after the Base Sample Entry in the same SampleDescriptionBox.

[0165] The information stored in the Sample Entries after the entry-count is track-type specific, and can have variants within the same track type (e.g. different codings may use different specific information after some common fields, even within a video track).

[0166] ISO / IEC 14496 defines a standard for multimedia file formats, including the MP4 file format. Within the MP4 file format, the SampleEntry specifies the format and encoding of a media sample. The standard defines SampleEntry specifications for the different Base bitstreams, including video, audio, and timed text. For video, the SampleEntry can specify several codecs, including MPEG 2 Video, MPEG-4 Visual, and MPEG-4 AVC / H.264, HEVC / H.265, WC / H.266. The specifications include information such as the video frame size, frame rate, bit rate, and color space. For audio, the SampleEntry can specify several different codecs, including MPEG-2 Audio, MPEG-4 Audio, and MPEG-4 AAC. The specifications include information such as the audio sample rate, bit depth, and channel configuration. For timed text, the SampleEntry canspecify the format of the text data, such as the DFXP (Distribution Format Exchange Profile) format, and include styling and layout information. Overall, the SampleEntry specifications defined in ISO / IEC 14496 provide a standardized way to describe the format and encoding of different types of media samples within multimedia files. This standardization enables interoperability between different systems and applications, allowing multimedia files to be reliably processed and played back on a wide range of devices and platforms.

[0167] In other words, a method may comprise increasing entry_count (e.g. by one), and adding an enhancement sample entry (e.g. can be LCEVC Sample Entry) to a SampleDescriptionBox. Wherein the SampleDescriptionBox comprises a Base Sample Entry. In other words, an enhancement sample entry (e.g. LCEVC Sample Entry) is added to the same SampleDescriptionBox that contains the Base Sample Entry. Preferably, the enhancement sample entry is added after the Base Sample Entry.

[0168] This embodiment is particularly advantageous because, in principle, no amendment to ISO / IEC 14496-12, clause 8.5.2.2, would be strictly needed, since the current specification allows the insertion of a first Sample Entry for the Base and a second Sample Entry for the LCEVC Enhancement. For clarity and completeness, the specific case of a second Sample Entry for LCEVC could be described in the text or a note to the standard specification.

[0169] It has been found that if a legacy device reads that the entry_count is 2, but cannot comprehend the second descriptor (e.g. the LCEVC Configuration Box), then the legacy device will merely ignore the second descriptor. This enables the legacy device to still play the base data even though it cannot comprehend the second descriptor.

[0170] As discussed, a further (i.e. an alternative) embodiment, as shown in figure 9 reuses the existing SampleEntry specifications for the different Base bitstreams, using the existing mechanism for extension of the VisualSampleEntry: MPEG4ExtensionDes criptorsBox (14496-15, clauses 5.4.2.1.2, 8.4.1.1.2, 11.3.1.2, 12.4.1.2). This embodiment enables insertion of a LCEVC Decoder Configuration associated with the Base Decoder configuration by using a MPEG4ExtensionDescriptionBox present in the sample entry of the base decoder. class AVCSampleEntryO extends VisualSampleEntry ('avcT or 'avc3') { AVCConfigurationBox config;MPEG4ExtensionDescriptorsBox (); / / optional}

[0171] The schematic in figure 9 shows data 900 comprising Sample Table (‘stbl’) 901. Sample Table (‘stbl’) 901 comprises Sample Description (‘stsd’) 903. In this embodiment, Sample Description (‘stsd’) 903 has an entry count of 1 meaning that there is only one Sample Entry box within the Sample Description box. As seen in Figure 9, Sample Description (‘stsd’) 903 comprises a Base Sample Entry (equal to ‘avcT if the base codec is AVC, equal to ‘hvcT if the base codec is HEVC, equal to ‘vvcT if the base codec is WC) 905. Base Sample entry 905 comprises MPEG4 Extension Descriptor 906, which is a child box within the Sample Entry box. Generally, MPEG4 Extension Descriptor 906 is a box which includes additional information, such as metadata and configuration information, to help decoders interpret and process encoded media data. Additional boxes or data structures can be added to the MPEG4 Extension Descriptor box which allows for the addition of additional data within the existing box structure. As we can see in Figure 9, MPEG4 Extension Descriptor 906 comprises enhancement Sample Entry 907 (equal to ‘IvcT if the enhancement codec is LCEVC) which is a child box within the MPEG4 Extension Descriptor box. The figure refers to “Ivc1 - descr#1”. It may also refer to “Ivc1 - entry #1” interchangeably.

[0172] In relation to Figure 9, we describe a method of generating data 900, an encoder configured to generate data 900, a decoder configured to decode data 900 and a bitstream comprising data 900. Sample Table (‘stbl’) 901 , Sample Description (‘stsd’) 903, Base Sample Entry 905, MPEG4 Extension Descriptor 906, and enhancement Sample Entry 907 may be referred to Sample Table (‘stbl’) box 901 , Sample Description (‘stsd’) box 903, Base Sample Entry box 905, MPEG4 Extension Descriptor box 906, and enhancement Sample Entry box 907 respectively. In addition, MPEG4 Extension Descriptor 906 may be referred to as MPEG4ExtensionDescriptor 906.

[0173] In ISO / IEC 14496, the VisualSampleEntry specifies the format and encoding of a video sample. The standard provides a mechanism for extending the VisualSampleEntry through the use of the SampleEntry extension mechanism. The SampleEntry extension mechanism is defined in clause 8.5.2 of ISO / IEC 14496-12, which specifies the file format for MPEG-4 audio and video. This clause defines the SampleDescriptionBox, which contains the SampleEntry boxes that describe the format and encoding of the audio and video samples. The VisualSampleEntry can be extended through the use of the MPEG4ExtensionDescriptorsBox, which is defined in ISO / IEC 14496-15. This box allows additional metadata about the visual content to be included in the sample description. Overall, the extension mechanisms defined in ISO / IEC 14496provide a flexible and extensible framework for describing the format and encoding of video samples, enabling interoperability and efficient processing of multimedia files across a wide range of systems and platforms. Embodiments of the present invention utilise these features for a new purpose.

[0174] NALUFF 5.4.2.1.3

[0175] In relation to ISO / IEC 14496-15, NALUFF 5.4.2.1.3 refers to the specification for the byte stream format used for the AVC (Advanced Video Coding) video codec in the MPEG-4 file format. Clause 5.4.2.1 of ISO / IEC 14496-15 defines the NAL (Network Abstraction Layer) unit byte stream format used for the storage and transport of AVC video data. The NAL unit format is used to encapsulate individual AVC video frames, and includes a header that specifies the type of NAL unit and its length. The NALUFF (NAL Unit File Format) refers to the specific file format used to store (e.g. AVC, HEVC ,and so forth) video data in the MPEG-4 file format. This format uses the NAL unit byte stream format defined in clause 5.4.2.1.3 of ISO / IEC 14496-15 to encapsulate individual video frames, which are then stored in a series of SampleEntry boxes in the MPEG-4 file. Overall, NALUFF 5.4.2.1 generally specifies the Sample Entry for (e.g. AVC) video data in the MPEG-4 file format, providing a standardized mechanism for the storage and transport of video data that can be implemented across a wide range of systems and platforms.

[0176] MPEG4ExtensionDescriptorsBox

[0177] Descr is a descriptor that should be placed in the ElementaryStreamDescriptor when this stream is used in an MPEG-4 systems context. This does not include SLConf igDescriptor or DecoderConf igDescriptor, but includes the other descriptors in order to be placed after the SLConf igDescriptor.

[0178] In ISO / IEC 14496-15, 'Descr' refers to a general descriptor for MPEG-4 systems. The general purpose of 'Descr' is to provide additional information about an object in the MPEG-4 bitstream, such as a media stream or a scene description. The 'Descr' descriptor is used in multiple clauses of the ISO / IEC 14496-15 standard, such as clause 5.4.2.1 which defines the 'Descr' box, and clause 8.5.2.2 which specifies the use of 'Descr' in the SampleEntry box to provide additional information about the codec configuration, and clause 8.6.4 which explains how 'Descr' can be used to signal thepresence of a specific object in the bitstream. The 'Descr' box may contain various types of descriptors, such as object descriptor, ES descriptor, and decoder configuration descriptor. Embodiments of the present invention utilise the flexibility of the 'Descr' descriptor for including additional metadata and information about an object in the MPEG- 4 bitstream, ultimately allowing for more accurate and efficient decoding and processing of media streams due to the backward compatibility of the generated streams.

[0179] DecoderConf igDescriptor (Decoder Configuration Descriptor) is defined in 14496-1 , clause 7.2.6.6. In ISO / IEC 14496-1 , the DecoderConfigDescriptor is a box that is used to describe the configuration and capabilities of a decoder for a particular media stream. The purpose of this box is to provide information that can be used by a client or player application to ensure that the decoder is capable of properly decoding the stream. The DecoderConfigDescriptor box is defined in clause 7.2.6.6 of ISO / IEC 14496- 1 , and it is used in conjunction with other boxes, such as the VisualSampleEntry or AudioSampleEntry, to fully describe the format and encoding of a media stream. The DecoderConfigDescriptor box itself contains information such as the decoder specific information, the buffer size, and the maximum bit rate. When a client or player application receives a media stream, it can use the information in the associated DecoderConfigDescriptor box to determine if its decoder is capable of handling the stream. If the decoder does not meet the requirements described in the box, the client or player application can choose to either transcode the stream or notify the user that the stream cannot be played. In summary, the purpose of the DecoderConfigDescriptor box in ISO / IEC 14496-1 is to describe the configuration and capabilities of a decoder for a media stream, while its function is to enable client or player applications to ensure that the decoder is capable of properly decoding the stream.

[0180] SLConf igDescriptor (Sync Layer Configuration Descriptor) is defined in 14496-1 , clause 7.3.2.3. In ISO / IEC 14496-1 , the SLConfigDescriptor is a box that is used to describe the sync layer configuration for a particular media stream. The general purpose of this box is to provide information about the sync layer of the stream, including the target device and the type of service provided. The SLConfigDescriptor box is defined in clause 7.2.6.8 of ISO / IEC 14496-1 , and it is used in conjunction with other boxes, such as the VisualSampleEntry or AudioSampleEntry, to maintain temporal synchronisation within and among elementary streams and to fully describe the format and encoding of a media stream. The SLConfigDescriptor box itself contains information such as the service type, service provider, target device, and the maximum bit rate. When a client or playerapplication receives a media stream, it can use the information in the associated SLConfigDescriptor box to determine the service level of the stream. This information can be used to optimize playback settings, such as the display resolution and bitrate, to ensure that the stream is played back correctly on the target device. In summary, the purpose of the SLConfigDescriptor box in ISO / IEC 14496 is to describe the Service Level configuration for a media stream, while its function is to enable client or player applications to optimize playback settings based on the information provided in the box.

[0181] Since the specific Conf igurationBox for any Base codec of interest (AVCConf igurationBox, HEVCConf igurationBox, EVCConf igurationBox, wcconf igurationBox , and so forth) is not derived from the DecoderConf igDescriptor, it is consistent with the existing 14496-15 specification, clause 5.4.2.1.3, to include in the MPEG4Extens ionDescriptorsBox an LCEVCConf igurationBox, thus providing for backward compatibility.

[0182] We describe updating ISO / IEC 14496-15, clause 5.4.2.1.3, at the end of the Descr semantics, to add the following text: “In particular, the descriptor for an LCEVC decoder can be present in the Descr field: in this case, one or more LCEVCConf igurationBox shall be considered as the descriptor(s) for one or more LCEVC decoder(s) associated as enhancement to the Base decoder identified by the Conf igurationBox preceding the MPEG4Extens ionDescriptorsBox.”

[0183] Thus, ISO / IEC 14496-15, clause 5.4.2.1 .3, at the end of the Descr semantics would read Descr is a descriptor that should be placed in the ElementaryStreamDescriptor when this stream is used in an MPEG-4 systems context. This does not include SLConf igDescriptor or DecoderConf igDescriptor, but includes the other descriptors in order to be placed after the SLConf igDescriptor. In particular, the descriptor for an LCEVC decoder can be present in the Descr field: in this case, one or more LCEVCConf igurationBox shall be considered as the descnptor(s) for one or more LCEVC decoder(s) associated as enhancement to the Base decoder identified by the Conf igurationBox preceding the MPEG4Ext ens ionDe script or sBox.

[0184] Advantageously, this embodiment is backwards compatible. Further advantageously, this embodiment further provides the advantage of possibly associating more than one enhancement (e.g. LCEVC) bitstream to the same Base bitstream, without the need for duplication. This is because additional Sample Entry boxes can be added tothe MPEG4 Extension Descriptor box 906 for each additional enhancement stream that is associated with a given base bitstream. This is in contrast with the ‘comparative solution’ which implies that, when there are several LCEVC bitstreams for the same Base bitstream, for each of them a different Aggregator containing LCEVC plus Base should be defined.

[0185] Although reference has been made to LCEVC, it is to be understood that such references are still valid for any other (e.g. known and / or future) enhancement type coding schemes.

[0186] In summary Figures 8 and 9 show two implementations of a single-track multilayer video stream that is backward compatible with legacy players such that the legacy player will not discard the entire video stream without any playback. The two implementations achieve this by re-using, and adding to, existing data structures within the file format rather than defining new structures which manot be recognised. Specifically, the existing Sample Description box is reused. In particular, in relation to Figure 8, an additional Sample Entry box is added to the existing Sample Description box such that the enhancement decoder configuration can be contained within this additional Sample Entry box. In relation to Figure 9, a Sample Entry box is added to the existing MPEG4 Extension Descriptor box within the Sample Description box such that the enhancement decoder configuration can be contained within this new Sample Entry box. Both of these implementations enable ‘clients’ not supporting an enhancement layer and / or enhancement codec (e.g. such as LCEVC) to still play the base bitstream, and ‘clients’ supporting an enhancement layer and / or enhancement codec (e.g. LCEVC) to play both the base and enhancement bitstreams.

[0187] In certain cases, example method 300 or any other of the examples described herein may be implemented via instructions retrieved from a computer-readable medium. These may be executed by a processor of a decoding system, such as a client device. The techniques described herein may be implemented in software or hardware, or may be implemented using a combination of software and hardware. They may include configuring an apparatus to carry out and / or support any or all of techniques described herein. The above examples are to be understood as illustrative. Further examples are envisaged. It is to be understood that any feature described in relation to any one example may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the examples, or any combination of any other of the examples. Furthermore, equivalents and modificationsnot described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.

[0188] We describe a number of clauses:1. A method of parsing a multi-layer video stream, the multi-layer video stream encoding a video signal and comprising a base layer and one or more enhancement layers, the base layer being encoded using a base layer specification, the one or more enhancement layers being encoded using an enhancement layer specification and comprising one or more sets of frame residuals, a set of frame residuals comprising a difference between a reconstruction derived from a decoded base layer frame and a frame derived from an original video signal at a given level of quality, the method comprising: at a modified parser for the base layer, parsing header data for Network Abstraction Layer (NAL) units belonging to the multi-layer video stream, including, for a given NAL unit: extracting a NAL unit type value from the header data for the given NAL unit; responsive to the NAL unit type value falling within a specified range for the base layer specification, instructing the decoding of a payload of the given NAL unit according to the base layer specification; and responsive to the NAL unit type value falling within a predetermined portion of an unspecified range for the base layer specification, determining an enhancement layer NAL unit type from the header data, and instructing the decoding of the payload of the given NAL unit according to the enhancement layer specification based on the determined enhancement layer NAL unit type.2. The method of clause 1 , wherein the enhancement layer NAL unit type indicates whether the given NAL unit is associated with an instantaneous decoding refresh picture.3. The method of clause 2, wherein the method further comprises: responsive to the enhancement layer NAL unit type indicating an instantaneous decoding refresh picture, extracting global configuration data from the payload of thegiven NAL unit, and using the global configuration data to decode a plurality of residual frames, data for the plurality of residual streams being distributed across multiple NAL units in the multi-layer video stream.4. The method of any one of the previous clauses, wherein the base layer specification is one of Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (WC), and the enhancement layer specification is Low Complexity Enhancement Video Coding (LCEVC).5. The method of any one of the previous clauses, wherein NAL unitsfor the base layer are interleaved with NAL units for the one or more enhancement layers within the multi-layer video stream.6. The method of any one of the previous clauses, wherein the header data comprises a 16-bit sequence, wherein bit values for NAL units belonging to the one or more enhancement layers are configured to indicate a value falling within the unspecified range for the base layer specification for a plurality of different base layer specifications.7. The method of any one of the previous clauses, comprising: discarding the given NAL unit responsive to values for additional base layer metadata within the header data falling outside of a predefined set of metadata values.8. The method of clause 7, wherein the base layer specification comprises Versatile Video Coding (WC) and the additional base layer metadata comprises a layer identifier.9. The method of clause 8, wherein an additional check is made on a value of a reserved zero flag prior to instructing the decoding of the payload of the given NAL unit according to the enhancement layer specification.10. A parser for parsing a video stream, the parser being configured to process Network Abstraction Layer (NAL) units comprising data encoded according to a first video coding specification, the parser comprising: a memory to store header data from Network Abstraction Layer (NAL) units belonging to a video stream; anda processor configured to: extract a NAL unit type value from the header data for a given NAL unit; and responsive to the NAL unit type value falling within a specified range for first video coding specification, instruct the decoding of a payload of the given NAL unit according to the first video coding specification, wherein the parser is modified to process a multi-layer video stream, such that the processor is configured to, responsive to the NAL unit type value falling within a predetermined portion of an unspecified range for the first video coding specification: determine an enhancement layer NAL unit type from the header data, and instruct the decoding of the payload of the given NAL unit according to an enhancement layer specification based on the determined enhancement layer NAL unit type, wherein a reconstruction of an original video signal associated with the multi-layer video stream is generated by combining data derived from frames decoded according to the first video coding specification with data derived from residual frames decoded according to the enhancement layer specification, each residual frame providing a quality improvement to a frame decoded according to the first video coding specification.11 . The parser of clause 10, wherein the enhancement layer NAL unit type indicates whether the given NAL unit is associated with an instantaneous decoding refresh picture.12. The parser of clause 11 , wherein the processor is further configured to, responsive to the enhancement layer NAL unit type indicating an instantaneous decoding refresh picture, instruct the extraction of global configuration data from the payload of the given NAL unit and instruct the use of the global configuration data to decode a plurality of residual frames, data for the plurality of residual streams being distributed across multiple NAL units in the multi-layer video stream.13. The parser of any one of clauses 10 to 12, wherein the first video coding specification is one of Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (WC), and the enhancement layer specification is Low Complexity Enhancement Video Coding (LCEVC).14. The parser of any one of clauses 10 to 13, wherein NAL units for the first video coding specification are interleaved with NAL units for the enhancement layer specification within the multi-layer video stream.15. The parser of any one of clauses 10 to 14, wherein the header data comprises a 16-bit sequence, wherein bit values for NAL units belonging to one or more enhancement layers are configured to indicate a value falling within the unspecified range for the first video coding specification for a plurality of different first video coding specifications.16. The parser of any one of clauses 10 to 14, wherein the parser is configured to discard the given NAL unit responsive to values for additional base layer metadata within the header data falling outside of a predefined range.17. The parser of clause 16, wherein the first video coding specification comprises Versatile Video Coding (WC) and the additional base layer metadata comprises a layer identifier.18. The parser of clause 17, wherein an additional check is made on a value of a reserved zero flag prior to instructing the decoding of the payload of the given NAL unit according to the enhancement layer specification.19. A computer-readable medium comprising instructions which when executed cause a processor to perform the method of any of clauses 1 to 9.

Claims

CLAIMS1 . A method comprising: adding an enhancement sample entry (807, 907) to a sample description (803, 903) contained in a media data (800,900) such that: a legacy player is configured to playback base data associated with a base sample entry in said sample description; and an enhanced player is configured to playback enhancement data associated with said enhancement sample entry and base data associated with said base sample entry.

2. The method of claim 1 , further comprising adding the base sample entry to the sample description.

3. The method of claim 1 or claim 2, wherein the sample description comprises a data field entry_count, and the method further comprises increasing an entry count of the sample description by a number equal to a number of enhancement streams in the media data.

4. The method of claim 3, wherein when the media data comprises a base stream and a single LCEVC stream, the method further comprises setting the value of entry_count to 2.

5. The method of claim 3, wherein when the media data comprises a base steam and n enhancement streams, setting the value of entry_count to n+1 , wherein n is an integer greater than or equal to 0.

6. The method of claim 2, further comprising adding the base sample entry to the data structure before adding the enhancement sample entry.

7. The method of claim 2, further comprising adding a child box structure to the base sample entry, preferably wherein the child box structure is an MPEG4 extension descriptor (906).

8. The method of claim 7, comprising populating the MPEG4 extension descriptor with the enhancement sample entry.SUBSTITUTE SHEET (RULE 26)9. The method of any preceding claim, wherein the enhancement sample entry is a descriptor for more than one enhancement stream associated with the media data.

10. A data stream comprising a sample description (803,903) comprising one or more enhancement sample entries (807, 907).

11. The data stream of claim 10, wherein said sample description comprises a base sample entry.

12. The data stream of claim 10 or 11 , wherein said sample description comprises a data field entry_count, and wherein a value of entry_count is greater than one.

13. The data stream of claim 12, wherein the value of entry_count is equal to a number equal to the sum of a number of the one or more enhancement sample entries plus a number of the base sample entries comprised in the sample description.

14. The data stream of claim 11 , wherein when the value of entry_count is equal to 2, the sample description comprises a single enhancement sample entry and a single base sample entry.

15. The data stream of claim 10, wherein the base sample entry comprises an MPEG4 extension descriptor.

16. The data stream of claim 15, wherein said MPEG4 extension descriptor comprises said one or more enhancement sample entries.

17. A device configured to perform the method of any one of claims 1 to 9.

18. A device configured to decode the data stream of any one of claims 10 to 16.

19. A computer-readable medium comprising instructions which when executed cause a processor to perform the method of any of claims 1 to 9.SUBSTITUTE SHEET (RULE 26)

Images