Implementing enhancement coding schemes in streaming applications

Abstract

There may be provided a method of streaming multimedia content encoded using an enhancement coding scheme and a method of packaging multimedia content encoded using an enhancement coding scheme for streaming, comprising retrieving an constructing a manifest file, the manifest file comprising metadata for constructing a locator to segments of the encoded multimedia content, the metadata comprising: a first representation pointing to a first version of the multimedia content encoded using a base coding scheme; and, location metadata for a first enhancement layer of the encoded multimedia content associated with the first version. A client device, packager, system and computer readable medium are also provided.

Description

[0001] IMPLEMENTING ENHANCEMENT CODING SCHEMES IN STREAMING APPLICATIONS

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to methods, systems, computer programs and computer readable media for implementing enhancement coding schemes in multimedia streaming applications, in particular for video streaming applications.

[0004] BACKGROUND

[0005] Streaming applications allow video content to be delivered remotely to users quickly and efficiently. Generally, video content is pre-encoded using a traditional coding scheme in different resolutions to ensure that the video content is compatible with a wide range of user devices whilst providing the best possible experience.

[0006] As modem players capable of displaying higher resolution video content become more common, and are commonly connected via wireless connections, various techniques must be employed to improve bandwidth utilisation so that delivery of video content is not only reliable, but also provides a seamless experience for the end user.

[0007] Much of the video content on the internet is encoded using H.264 (also known as MPEG-4 Part 10, Advanced Video Coding — MPEG-4 AVC). This content is typically supplied to decoding devices as a single video stream that has a one-to- one relationship with available hardware and / or software video decoders, for example a single stream is received, parsed, and decoded by a single video decoder to output a reconstructed video signal. Many video decoder implementations are thus developed according to this framework.

[0008] To support different streaming conditions, client devices are generally configured with a switching mechanism that is driven based on metadata identifying a stream format. One example is Adaptive Bit-Rate streaming, or ABR, and a typical, standardised mechanism implementing this is Dynamic Adaptive Streaming over HTTP (DASH), also known as MPEG-DASH. Other common HTTP based streaming protocols include HTTP Live Stream (HLS), and Real-Time Messaging Protocol (RTMP), which both function in a similar manner to HTTP (DASH), although RTMP is not based on HTTP but is rather TCP / UDP based.

[0009] In ABR streaming, a video is coded into multiple resolutions or bitrate combinations, and each is typically referred to as a ‘rendition’ (or a representation, see below in the context of manifest files). Client devices determine the available bandwidth and retrieve the appropriate rendition according to their available bandwidth and in a format they can support. The collection of renditions is typically referred to as a ladder or a profile.

[0010] To implement this, in DASH, each client device retrieves a manifest file, known as an MPEG-DASH MPD (Media Presentation Description) file, which comprises metadata to enable the client device to construct a uniform resource locator (URL) to retrieve the appropriate stream based on available conditions (such as device capability, resolution, available bandwidth, state of the buffer, etc). Within each manifest file is a set of representations, each describing one of the versions of the content. The manifest file describes how the video has been split into pieces by a packager and how it is arranged so that the client device can request chunks of the video for subsequent playback. Generally, the representations include specific video and audio tracks for describing a version of the media stream, bitrates, resolution, and other information associated with the media content. The manifest may also be referred to as a playlist in other formats.

[0011] Existing enhancement coding schemes include the Scalable Video Coding (SVC) extension to H.264, Scalable extensions to H.265 (MPEG-H Part 2 High Efficiency Video Coding — SHVC), and newer standards such as MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) and SMPTE ST-2117 (VC-6). SVC and SHVC operate by creating different encoding layers and feeding each of these with a different spatial resolution. Each layer encodes the input according to a normal AVC or HEVC encoder with the possibility of leveraging information generated by lower encoding layers. LCEVC, on the other hand, generates one or more layers of enhancement residuals compared to a decoded version of a base encoding, where the base coding may typically be of a lower spatial resolution.

[0012] Unlike traditional coding schemes, LCEVC encodes a lower or base quality version of a source image using an existing codec, and an enhancement layer is encoded that can be used to increase, and bring, the decoded base version to a higher quality. The enhancement layer adds an additional degree of freedom in how a video stream can be implemented and provides new features such as extending the compression capability of the base codec, lowering the encoding and decoding complexity, and providing a platform for additional future enhancements.

[0013] For ABR streaming (and DASH) to accommodate different user devices, and adapt to changing network conditions, content must be encoded and stored at different bitrates and resolutions so that each device can retrieve an appropriate stream. Similarly, while some devices may be able to stream content encoded using a particular coding scheme, legacy devices may not. If content is to be streamed by as many users as possible, it must be encoded and stored using schemes available to those user devices. This generally means that there must be at least one representation of the content for every possible device setup, bitrate, and resolution combination. For example, advanced setups with high available bandwidth are able to stream media with high bitrate, frame rate or resolution while basic setups, or those with poor bandwidth, are able to stream media in low quality.

[0014] Therefore, to support ABR streaming of multimedia content encoded using enhancement coding schemes, there is a significant increase in the number of versions encoded and stored for delivery and a significant increase in the number of representations available in each manifest file. The increased flexibility of enhancement coding schemes comes with an inherent compromise, that is, the proliferation of representations for each variant, combination, and permutation balanced against the inherent flexibility and improved quality for the available conditions resulting from the ability to stream those representations. In short, a difficulty with implementing enhancement coding schemes is that managing encoders, packagers and decoders is much trickier than managing the same in relation to a traditional ‘single layer’ coding scheme. This is because, once an enhancement coding scheme (which results in enhancement coding layer) is introduced, there are many more combinations, permutations available to encoders, packagers and decoders.

[0015] SUMMARY OF INVENTION

[0016] According to an aspect of the invention there may be provided a method of streaming multimedia content encoded using an enhancement coding scheme. The method may comprise: retrieving a manifest file, the manifest file comprising metadata for constructing a locator to segments of the encoded multimedia content, the metadata comprising: a first representation pointing to a first version of the multimedia content encoded using a base coding scheme; and, location metadata for a first enhancement layer of the encoded multimedia content associated with the first version; and, parsing the manifest file to construct a locator for retrieval of the first representation and segments of the enhancement layer; downloading segments of the first representation and segments of the enhancement layer for reconstructing the encoded multimedia content.

[0017] According to an aspect of the invention there may be provided a method of packaging multimedia content encoded using an enhancement coding scheme for streaming, the method comprising: constructing a manifest file, the manifest file comprising metadata for constructing a locator to segments of the encoded multimedia content, the metadata comprising: a first representation pointing to a first version of the multimedia content encoded using a base coding scheme; and, location metadata for a first enhancement layer of the encoded multimedia content associated with the first version.

[0018] In this way, the number of variants of encoded multimedia content that need to be maintained and supported can be reduced. By presenting the enhancement layer and base layer of the encoded content separately, it is possible for a client device to retrieve the base layer only depending on available bandwidth and device conditions, but where appropriate retrieve both the base layer and enhancement data for combination and reconstruction even though they are stored and presented separately. Legacy devices can retrieve legacy coded data without the associated enhancement data thereby utilising all available bandwidth while that legacy coded data can be used by enhancement enabled devices as the base coded data for reconstruction of higher quality content using the separately presented enhancement data.

[0019] The encoded multimedia content may be a video stream. The enhancement coding scheme may be LCEVC and the base coding scheme may be any ‘single layer’ coding scheme such as AVC, HEVC, AV1 or WC. The manifest file may comprise a media presentation description (MPD). The locator may be a link or uniform resource locator (URL). The first version of the multimedia content encoded using a base coding scheme may the base layer of the multimedia content encoded using the enhancement encoding scheme, that is, the base layer associated with the particular first enhancement layer. Segments may be downloaded according to guidelines set out in MPEG-DASH, HTTP Live Streaming (HLS) or Microsoft Smooth Streaming. In other words, MPEG-DASH and other streaming protocols are guidelines suitable to indicate the order and retrieval of segments. The downloading itself is done on the player side and hence it is different for different players based on configurable parameters that control the downloads of the segments based on, for example bandwidth or quality and many more such parameters.

[0020] While the term manifest is used, as it is the official term for streaming protocols, it will be understood this may be considered interchangeable with the term playlist depending on the format used. In some circumstances, the manifest may be passed on as a playlist.

[0021] Concepts according to the present invention facilitate adaptive bitrate streaming of a base layer of encoded multimedia content separately from its enhancement layer, where a client device performs the function of deciding which bit rate segments to download, based on the current state of the network. The second representation may be at a higher level of quality than the first representation. By level of quality we mean resolution, bitrate, colour depth and file size, among others. In the sense of the representation, we mean that the higher level of quality likely requires a more advanced client device support or better conditions (e.g. network or device such as buffer capacity).

[0022] The method may be performed by a client device, player or decoder.

[0023] Preferably, the metadata comprises a second representation pointing to the first enhancement layer of the encoded multimedia content. The location metadata may be comprised in the second representation. Accordingly, the manifest file presents multiple representations for selection by the client device with each representation corresponding to a different layer of the enhancement coding scheme. By downloading both layers according to network conditions, they can be subsequently combined for rendering the multimedia content at the client device. In implementations, each representation corresponds to a single track. Through use of the second representation, the base layer and enhancement layer may be provided separately with an external link associating the two together.

[0024] More preferably, the metadata further comprises a dependency attribute associating the second representation with the first representation. The dependency attribute may be a dependency ID attribute of an MPEG-DASH manifest file. The dependency attribute provides for an association of the second representation with the first representation so that they can be provided separately with an external link, i.e. the dependency. The dependency attribute provides for utilising an existing mechanism for a hitherto unforeseen purpose, thus providing an immediate path to implementation without significant modification of existing standards and devices. The dependency attribute means that any implementation can be rolled out without affecting devices currently deployed and without requiring software update for those devices to function.

[0025] Together with a dependency attribute, the representation may also comprise a resolution of the encoded multimedia content. The second representation may comprise a resolution of the combined base and enhancement layers when constructed. Further the representation may also comprise a bandwidth attribute, minimum, maximum or average, for retrieving the representation. The bandwidth attribute of the second representation may correspond to the bandwidth required to retrieve the enhancement layer only or to retrieve both the representation of the enhancement layer and its associated representation of the base layer.

[0026] The method may further comprise determining a dependency between the second representation and the first representation. In other words, setting up the second representation based on the dependency. The method may further comprise parsing the dependency attribute to identify the association between the second representation and the first representation. In the context of MPEG DASH terminology the dependency may be a dependencyld.

[0027] Preferably the first and second representations are comprised in respective Adaptation Sets. By using respective Adaptation Sets, an ABR algorithm of client devices supporting one or both of the base coding scheme and enhancement scheme can traverse an ABR ladder of that scheme. An ABR ladder can comprise representations of the enhancement layer while retrieving representations of the base layer through the use of the dependency. The dependency may associate representations between different Adaptation Sets or within the same Adaptation Set. Alternatively the first and second representations may be comprised in the same Adaptation Set for selection by one ABR algorithm traversing one ABR ladder within the same Adaptation Set. Optionally the first representation may be repeated across different Adaptations Sets, the different Adaptation sets being associated with different MIME types or supported codecs.

[0028] In optional implementations, the metadata may further comprise, for each representation, a bandwidth for retrieving the respective representation, and the method may further comprise: determining a total bandwidth for streaming the multimedia content by summing the bandwidth associated with the second representation and the bandwidth of its respective associated first representation. In this way the client device can select an appropriate representation according to available network conditions using different representations. As above, the representation may also comprise a bandwidth associated with the total bandwidth for retrieving the second representation and its associated first representation.

[0029] Preferably, the first representation comprises an attribute indicating the base coding scheme; and, the second representation comprises an attribute indicating the enhancement coding scheme. The attribute may comprise a ‘codecs’ attribute. Examples include “avc1.64001 f” and “Ivc1”. The attribute may also comprise a MIME type attribute. Examples include video / h264. Setting attributes in this way allows client devices to select ABR profiles according to codecs supported by the devices.

[0030] The metadata may further comprise: a third representation pointing to a second version of the multimedia content encoded using the base coding scheme, the first and second versions being versions of the multimedia content at a different level of quality; and, a fourth representation pointing to a second enhancement layer associated with the second version of the encoded multimedia content. The different level of quality may be a higher level of quality as set out elsewhere herein.

[0031] The first representation may be 340p with an associated second representation at 720p and the third representation may be 540p with an associated fourth representation at 1080p. In this way, four representations may be provided from only two encoded and stored videos. This example demonstrates how multiple representations can be presented using few encoded and stored videos and tracks by using base layers as both low-quality representations in the ABR ladder and in different base layers of the enhancement layers at higher quality levels in the ABR ladder.

[0032] The following are preferred implementations of the above method of streaming multimedia content encoded using an enhancement coding scheme.

[0033] The method may further comprise: building a set of virtual representations from the first and second representations and the dependency attribute, wherein the virtual representations comprise: a first virtual representation comprising the first representation; and, a second virtual representation comprising the first and second representations. The method may further comprise selecting a virtual representation for retrieval according to ABR rules. The virtual representation allows for an ABR algorithm to traverse the virtual representations according to available conditions, where the virtual representations comprise representations of the base and enhancement layers presented separately in the manifest file.

[0034] The method may further comprise: sensing available bandwidth and / or device conditions; and, downloading segments of the first representation or segments of both the first representation and segments of the enhancement layer for reconstructing the encoded multimedia content according to the available conditions. The method may also comprise selecting a representation for download according to available bandwidth and / or device conditions. Bandwidth conditions may be thought of as network conditions. Device conditions includes examples such as buffer status, power status, decoder capability or hardware capability. In this way, the representations can be selected and ABR profiles traversed according to a set of predefined rules.

[0035] The method may also comprise: passing downloaded segments of the enhancement layer to an enhancement decoder to generate decoded enhancement data; and, reconstructing multimedia content from the decoded enhancement data and a decoded version of the video stream encoded using the base coding scheme decoded from the downloaded segments of the first representation. Optionally the downloaded segments of the base layer may be passed to a base decoder directly or via the enhancement decoder. The base decoder may then operate under the control of the enhancement decoder or independently of it.

[0036] The following are preferred implementations of the above method of packaging multimedia content encoded using an enhancement coding scheme for streaming.

[0037] Preferably, the method may further comprise: receiving an encoded version of the multimedia content encoded using the enhancement coding scheme comprising a base layer and an enhancement layer; generating a first set of segments representing the base layer; and, generating a second set of segments representing the enhancement layer, wherein the first set of segments are independent of the second set of segments. Independent may be considered as separate from. In this way, the packager may store the segments separately for subsequent retrieval. The segments may be stored in separate files or one respective file with defined byte ranges. Alternatively the segments of both layers may be stored in one file with defined byte ranges, with the packager generating the segments for the manifest file to comprise locations for separate representations to present retrieval of those segments separately.

[0038] The first set of segments and second set of segments may be generated as respective single track representations.

[0039] The constructing the manifest file may comprise: inserting an association between the second set of segments and the first set of segments; obtaining a location of the first and second set of segments; and, inserting metadata for constructing a locator to the location of the first and second set of segments.

[0040] Embodiments herein may generally be player and protocol agnostic, for example, different containers may be used such as ISO base media file format (e.g. MP4 file format) or MPEG-2 Transport Stream as well as different clients such as those based on HTML5 or other transport stream mechanisms. Theoretically if DASH dependencyld is implemented, embodiments presented herein will work on any platform that supports LCEVC.

[0041] According to aspects of the present invention there is provided a method comprising parsing a file (e.g. a manifest file) and choosing from a set of renditions based on a determined available bandwidth.

[0042] We describe a method comprising parsing an attribute to determine a dependency between a representation of a low level of a hierarchical coding scheme (e.g. such as base layer representation, e.g. AVC, HEVC, WC, and so forth) and a higher level of said hierarchical coding scheme (e.g. such as an enhancement layer representation, such as LCEVC). Determining may be referred to as setting-up. A manifest file may comprise said attribute. The parsing may be parsing of a manifest file.

[0043] The method may further comprise ma”ntai’lng a list of representations. The list may comprise one or more representations comprising a base layer (e.g. and no enhancement layer). The list may comprise one or more representations comprising a base layer and an (e.g. associated) enhancement layer. The list may comprise a first representation comprising a first version of a base layer. The list may comprise a second representation comprising a second version of a base layer. The list may comprise a third representation comprising the first version of a base layer associated with a first version of an enhancement layer. The list may comprise a fourth representation comprising the second version of a base layer associated with a second version of an enhancement layer. The second versions may be higher quality (e.g. higher resolution, higher file size, or so forth) than the first versions of the base and / or enhancement layers. Each version of the enhancement layer may comprise a dependency identifier configured to identify a base representation that the version of the enhancement layer is dependent on.

[0044] The method may comprise determining a total bandwidth associated with each representation of the list of representations. The method may comprise determining the total bandwidth associated with each representation of the list of representations by summing a bandwidth associated with each of the constituent elements of the representation. For example, the method may comprise determining a total bandwidth associated with the third representation by combining (e.g. summing) a bandwidth associated with the first version of a base layer and a bandwidth associated with the first version of an enhancement layer. For example, the method may comprise determining a total bandwidth associated with the fourth representation by combining (e.g. summing) a bandwidth associated with the second first version of a base layer and a bandwidth associated with second first version of an enhancement layer.

[0045] The method may comprise determining available bandwidth. The method may comprise determining a coding attribute of a device associated with (e.g. performing) the method. The coding attribute may be an indication that the device supports a coding scheme associated with the enhancement layer (and / or the base layer). The aforementioned ‘available bandwidth’ may be the available bandwidth between the device (i.e. the player, the decoder, and so forth) and the streaming device (e.g. content server).

[0046] The method may comprise selecting a representation of the list of representations based on the determined available bandwidth. The method may comprise selecting a representation of the list of representations based on the determination of the coding attribute of a device associated with (e.g. performing) the method. The method may comprise selecting a representation associated (e.g. having, comprising, and so forth) with a bandwidth matching the determined bandwidth.

[0047] The method may comprise downloading segments (e.g. a version of a base layer and, if applicable a version of the enhancement layer associated with the selected representation) associated with the selected representation. The method may comprise downloading the segments in a synchronous mode.

[0048] The method may further comprise sending the segment(s) to the appropriate decoder. The sending may be performed via an interface. For example, a segment associated with the version of a base layer may be sent to a base decoder. For example, a segment associated with the version of the enhancement layer may be sent to an enhancement decoder.

[0049] The method may comprise selecting a representation that does not comprise (and / or associated with) an enhancement layer segment (e.g. the selected representation may only comprise associated with a base layer segment and / or the selected representation is only associated with a base layer segment), such a selection may occur due to a low determined bandwidth and / or that the codec attribute of the device does not correspond with the coding scheme associated with the enhancement layer. The method may comprise, in response to said selection (i.e. selecting a representation that only comprises and / or is associated with a base layer segment) downloading only the base layer segment. The method may comprise sending these base layer segments to a base decoder. The method may comprise processing decoded segments to enable a playback of content. The method may comprise combining decoded base segment with an associated decoded enhancement segment, to generate an output / display video.

[0050] The method may be performed by a decoder, a player’, a client device, and so forth.

[0051] According to an aspect of the invention there may be provided a client device for streaming multimedia content encoded using an enhancement coding scheme configured to perform the method of any of the above aspects.

[0052] According to an aspect of the invention there may be provided a packager for packaging multimedia content encoded using an enhancement coding scheme for streaming configured to perform the method of any of the above aspects.

[0053] According to an aspect of the invention there may be provided a system comprising a client device and a packager of any of the above aspects. The system may further comprise: an encoder configured to encode the multimedia content using an enhancement coding scheme; and, a content delivery network configured to serve the segments and the manifest file.

[0054] According to a further aspect there may be provided a computer readable medium comprising instructions which when executed by a computer cause the computer to perform the method of any of the above aspects of the invention.

[0055] BRIEF DESCRIPTION OF DRAWINGS

[0056] Examples of systems and methods in accordance with the invention will now be described with reference to the accompanying drawings, in which:

[0057] Figure 1 shows a known, high-level schematic of an LCEVC encoding, decoding and transport process;

[0058] Figure 2 shows a known high-level schematic of an ABR process of implementing enhancement coding;

[0059] Figure 3 shows an example of a known ABR profile for enhancement coding; Figure 4 shows an example of a known approach to encoding and storing representations for ABR;

[0060] Figure 5 shows an example of an approach to encoding and storing representations for ABR according to principles of the present disclosure;

[0061] Figure 6 shows an example of an approach to encoding, storing, presenting and retrieving representations for ABR according to principles of the present disclosure;

[0062] Figure 7 shows a high-level schematic of an ABR process of implementing enhancement coding according to principles of the present disclosure;

[0063] Figure 8 shows a high-level approach to encapsulation according to the present disclosure; and,

[0064] Figure 9 is an example process to be performed at a client device according to principles of the present disclosure.

[0065] DETAILED DESCRIPTION

[0066] For context, we will begin by describing an enhancement coding scheme, LCEVC, suitable for use with the concepts of the present disclosure. LCEVC will be described in the context of Figure 1. Throughout the description the terms hierarchical coding and enhancement coding may be used interchangeably and while examples are described in the context of LCEVC, it will be understood that the concepts described may be suitable to any similar hierarchical or enhancement coding scheme.

[0067] LCEVC adopts a multi-layer approach where any base codec (e.g. h.264, HEVC, AV1 and others), is enhanced via an additional low bitrate stream. LCEVC’s data stream structure is defined by two component streams: a base stream decodable by a hardware decoder; and, an enhancement stream consisting of one or two enhancement layers suitable for software processing implementation with sustainable power consumption. The enhancement provides improved compression efficiency to existing codecs, and reduces encoding and decoding complexity, for on demand and live streaming applications. Figure 1 below illustrates how LCEVC operates on both encoding and decoding pipelines. The base encoding (whether H.264, HEVC or others) is performed on a down-scaled input at a lower resolution, typically a quarter of the desired output resolution. LCEVC enhancement data is calculated at the two resolutions providing two levels of correction and enhancement. The LCEVC encoder generates the enhancement stream from two inputs: the base encoding and the original uncompressed full resolution video, effectively correcting the quality gap between the two. The LCEVC data can be packaged together with the base elementary stream (for example as Supplemental Enhancement Information (SEI) of the Network Abstraction Layer, NAL), as frame metadata in a WebM container or in an additional data Packet Identifier (PID) in a MPEG-2 TS stream.

[0068] The LCEVC decoder works at an individual video frame level. As input it takes the decoded low-resolution picture from the base video decoder, which is typically provided by a hardware decoder on the device, and the LCEVC enhancement decoded in software to produce a full-resolution picture ready for rendering on the display view. Example implementations of decoding LCEVC are set out in WO2022 / 023739 and WO2023 / 118851 which are incorporated by reference.

[0069] As illustrated in figure 1 , in the encoder 100, an input full resolution video, i.e. a source video 102, is processed to generate various encodings. A first encoding (base encoding 110) is produced by feeding a base encoder 106 (e.g., AVC, HEVC, VP9, or any other codec) with a down-sampled version of the input video, which is produced by down-sampling 104 the input video 102. In the example shown, the downsampling is to a quarter resolution but this is optional as will be elaborated on elsewhere. The base encoding 110 may be referred to as a base layer.

[0070] A second encoding (level 1 encoding 112, an example of an enhancement encoding) is produced to create first level corrections 116 by applying an encoding operation to the residuals obtained by taking the difference between a reconstructed base codec video and the down-sampled version of the input video. The reconstructed base codec video is obtained by decoding the output of the base encoder 106 with a base decoder. In typical implementations, the level 1 encoding 112 is optional. This level 1 encoding 112 may be referred to as a first enhancement layer.

[0071] A third encoding (level 2 encoding 114, another example of an enhancement encoding) is produced to create first level corrections 120 by processing the residuals obtained by taking the difference between an up-sampled version (i.e. normative upsampling 118) of a corrected version of the reconstructed base coded video and the input video 102. This level 2 encoding 114 may be referred to as a second enhancement layer.

[0072] The enhancement layer(s) and the base layer are typically combined (i.e. as illustrated by mux 122) and the full resolution video is encoded in layers. This is then typically transmitted using standard packaging and transmission protocols, an example of which will be described below in the context of Figure 2.

[0073] While often the corrections forming the enhancement layer involve an upsampling to higher resolution, generally any improvement in quality may be provided by the enhancement layer. For example, resolution, visual quality (VQ), bit depth (e.g. 8 to 10b) or colour space (e.g. HDR).

[0074] At the decoder 140, the encoded video is separated (i.e. as illustrated by demux 142) into an ancillary data stream 144 and a video stream 146. The decoder receives the layers (a base encoding, an optional level 1 encoding and a level 2 encoding) together with headers containing further decoding information. The base encoding, i.e. in video stream 146, is decoded by a base decoder 148 corresponding to the base decoder used in the encoder. At an enhancement decoder 150, which receives headers and the enhancement layers in the ancillary data stream 144, its output is combined with the decoded residuals obtained by decoding the level 1 encoding (if present). The combined video is up-sampled and further combined with the decoded residuals obtained by applying a decoding operation to the level 2 encoding to output the full resolution video 152.

[0075] Figure 2 illustrates a schematic example of implementing enhancement coding using standard packaging and transmission protocols. The source video 102 is encoded by the encoder 100 to produce the full resolution video encoded in layers. An example of how the encoder 100 may be implemented using an encoder integration layer is described in WO2022 / 023747 which is incorporated herein by reference. Any suitable container format may be used for the video, but here we illustrate an mp4 container format containing the base layer and enhancement layer.

[0076] The encoder will be instructed to output a set of videos in different qualities, referred to here as bitrate variants, depending on the desired configurations. For example, the set of files may include videos at different resolutions (e.g. 1080p, 720p, etc.) and different qualities, frame rates or bitrates at the same resolution (e.g. 720p, 60 fps and 720p, 30fps etc.).

[0077] LCEVC is designed so that legacy devices are still able to decode and play the base video, even if the stream with both the enhancement and base layers is retrieved. In other words, LCEVC is inherently backwards-compatible since devices that have”t yet been updated to decode LCEVC happily play the underlying base codec. However, for those legacy devices, in this situation they are streaming data they are not using and therefore underutilising their available bandwidth.

[0078] To address this, as well as video files of the full resolution video encoded in layers, in order to support legacy devices, the encoder will also typically be instructed to output variants encoded using the base codec only. This enables those files to be retrieved by legacy client devices without the accompanying enhancement data and thereby utilising more of their available bandwidth on viewable video data.

[0079] The set of variants, i.e. the set of encodes or renditions, are packaged by a packaging service referred to here as a packager 202. An example packager is the Shaka packager. The packager 202 splits each rendition into chunks of a specified duration (e.g. 2 or 4 seconds) and records how the video is split and the order of the chunks in a media presentation description file (MPD), otherwise referred to as a manifest file 205. The packaged videos and manifest files are then placed into storage 204. Once stored, the chunks of the video may be referred to as segments and the segments and manifest file are made available to retrieve over a network 206.

[0080] The manifest file is a document, typically in Extensible Markup Language (XML), that contains metadata required by a client device to construct appropriate uniform resource locators (URLs) to access the segments. The manifest is structured hierarchically. The media presentation contains a sequence of one or more periods; a period contains one or more Adaptation Sets; an Adaptation Set contains one or more representations; a representation contains one or more segments; and, segments carry the locations of the actual media data and associated metadata.

[0081] The Media Presentation (i.e. the overall hierarchy) contains information about all the different media types in the content. The most common media types are video, audio, and closed captioning data. At the top level, the MPD contains information including the MPD Profile, minimum buffer time, presentation duration, and maximum segment duration, and title. The video into structured into multiple Periods, for example for advert insertion or chapters. An Adaptation Set catalogues the available representations of the media. For example, as shown in the schematic in Figure 2, there may be an Adaptation Set for the base video and an Adaptation Set for the LCEVC video. There may also be an Adaptation Set for a subtitle track and an audio track, for example. Within each Adaptation Set, the player will only select one Representation for a particular Segment, but the Representation can change from one Segment to the next. Video Adaptation Sets typically contain multiple Representations, one for each resolution / bitrate, allowing the media player to select the best available quality without buffering. If the video is available in more than one codec, each codec will be in a different Adaptation Set. Within an Adaptation Set, a Representation describes one of the versions of the content. Each Representation typically includes the average bandwidth of the Representation to allow for the client device to choose which representation to retrieve. Each Representation includes a set of Segments which contain the information required to construct the URLs to download the content. The MPD can either provide a list of the segment URLs or a template that the player uses to build the URLs dynamically.

[0082] The structure of an implementation of MPEG DASH will be well understood to the skilled person and so we provide this description for context only. Further information may be found in the ISO / IEC 23009 standards documentation.

[0083] Figure 2 illustrates two client devices 208, 210. To help demonstrate the principles of the present invention, we will assume that client device #1 208 may be an LCEVC capable device and client device #2 210 may be a legacy device not capable of decoding LCEVC video. Where we refer to functions of the client device, functionality may be embodied in a player on the device, a decoder module or any other module on the device, or functionality may be split over multiple modules such as an ABR handler, Data handler, Media Handler, Player and decoder, etc. The client devices 208, 210 may first retrieve (i.e. download) the manifest file 205 over the network 206, for example as served by a content delivery network CDN. The manifest file is parsed by the client device to understand the mechanism of serving the video. The client device 208, 210 senses its available bandwidth and selects an available representation according to the available bandwidth and the representation supported by the device. The client device then requests chunks from one of the video’s bitrate variants depending on its buffer size and available bandwidth. In other words, the client device selects segments for download according to the available bandwidth and the device status (e.g. video supported and buffer conditions). In the example of Figure 2, client device #1 208 supports LCEVC and has 2Mbps available. It accordingly parses the manifest file to identify that the most appropriate segments for download are those according to the biei representation and from the metadata in the manifest file it constructs an appropriate URL to retrieve the segments to fill its buffer. The retrieved segments are passed to an LCEVC decoder for decoding. The client device 208 continuously senses its bandwidth conditions and buffer levels and adaptively streams content. According to ABR principles, if bandwidth conditions change or buffer levels change for example, the device can retrieve segments from a different representation, e.g. within the same Adaptation Set. To conclude the example, we noted above that client device #2 210 does not support LCEVC. Let’s suppose it senses the same available bandwidth as client device #1 208. Accordingly, it would choose from the AVC Adaptation Set, since it does not support LCEVC and would choose to retrieve segments from Representation b2since that indicates an average bandwidth of 2Mbps and would then construct an appropriate URL from the metadata.

[0084] The Representations within an Adaptation Set may be referred to as an ABR ladder or ABR profile. Figure 3 illustrates a sample ABR profile, from a report written by industry analyst Jan Ozer in April 2020. The H.264 ABR Ladder profiles, encoded using x264 (medium preset), are in line what recommended by Apple (RTM) in the HLS Authoring Specification. The LCEVC H.264 ABR Ladder profiles have been optimised to achieve both a quality uplift and a reduction in the total average bitrate streamed. As shown, an ABR ladder using LCEVC balances quality and efficiency. In other words, there may be a lower bitrate for the top profile at a similar or higher quality as x264, while improving quality & resolution of lower profiles. To achieve maximum efficiency, it may be possible to reduce data rate as much as possible while matching x264 quality on all profiles to decrease costs (e.g., transcoding, CDN) and maximise network capacity (e.g., number of simultaneous users).

[0085] Figure 4 illustrates an alternative view of a traditional approach, with or without LCEVC. As shown, each resolution is encoded to require a set bitrate to be able to appropriately retrieve that encoded video stream. As described above, a difficulty with implementing enhancement coding schemes is that managing encoders, packagers and decoders is much more complex than managing the same in relation to a traditional ‘single layer’ coding scheme. This is because, once an enhancement coding scheme, which results in enhancement coding layer, is introduced, there are many more combinations, permutations available to encoders, packagers and decoders.

[0086] While the use of representations works well for the majority of mainstream setups, it is evident that it may be detrimental to the interoperability of uncommon or older setups. As each representation defines a specific bitrate and resolution, the manifest must present representations for as many common setups as possible. However, as each representation is stored separately and independently of one another, the amount of storage required increases exponentially with the number of representations.

[0087] It would not be feasible to store representations for every possible permutation and, to strike a balance, uncommon or older setups are likely to be left out and are therefore forced to correspond to a specific representation. Similarly, players are unable to change or modify the encoded representations and must simply select from what is already available.

[0088] Examples of the present disclosure describe methods and systems that can benefit from the scalable capabilities of enhancement coding schemes (such as MPEG-5 LCEVC) in streaming applications. According to the principles of the present disclosure, the base and enhancement layers are stored and provided separately to one another. The representations are also presented separately to one another in theABR ladder, i.e. the manifest file. In an optional implementation, a dependency is introduced between representations across Adaptation Sets. This dependency creates an association between the segments and frames of the base layer and the enhancement layer. In other words, the enhancement layer is associated with the base layer it is configured to enhance through the use of the dependency, specifically in examples the dependencyld of MPEG DASH.

[0089] This concept and those described herein may be referred to as Scalable Mode. When used in Scalable Mode the encoder is configured such that the Base Layer, which corresponds to an existing delivery method such as AVC, HEVC, AV1 , may be forced to correspond to an existing standard profile of the ABR ladder (e.g., 540p AVC, 1080p HEVC). The rate control mode may be chosen to match the setting of the pre-existing system, e.g. CBR, capped-CRF, capped-VBR etc., and to ensure that the target bitrates for both the base layer and the total base layer plus enhancement layer are hit.

[0090] The LCEVC enhancement layer on top may then be used to efficiently compress a higher resolution (e.g., 1080p AVC, 2160p HEVC). The enhancement chunk will then be combined with the base to produce the higher resolution video at the decoder.

[0091] As an example, assuming target bitrate lanes of 2700 Kbps and 1900 Kbps, LCEVC can be encoded at 1080p (with 540p BL) or 720p (with 360p BL) — or other resolutions in-between — in CBR targeting 2700 Kbps, with a -75% base proportion target, so to serve two profiles with a single encode.

[0092] This mode can be used for distribution over HTTP with LCEVC-enabled players capable of choosing whether to add the LCEVC enhancement or not. Legacy players will only select and decode the base file. The legacy players will not need to download the enhancement file.

[0093] In this document, the following terminology will be used: BL = Base Layer (e.g. AVC, HEVC, WC); and, EL = Enhancement Layer (LCEVC).

[0094] Figure 5 illustrates a comparison with Figure 4 in which an advantageous implementation of the Scalable Mode may be illustrated. In the illustrated implementation, the splitting of the base and enhancement can be leveraged to present a condensed set of variants in an approach termed LCEVC-powered multiscale. A first set of representations are optionally presented as unaffected base profiles, e.g. unaffected AVC profiles. In this example, videos encoded solely using base codecs are neither stored nor presented. Shown here as the 360p and 540p profiles. Instead, the encoder produces LCEVC enhanced videos and the base layer from those LCEVC enhanced videos are provided separately as the profiles for the lower bandwidths. In other words, the base layer of the LCEVC encoding process is provided as a separate representation in the ABR ladder.

[0095] In the specific example shown, a further benefit is provided by encoding the LCEVC video using different base settings. This facilitates an overall increase in the available representations without significantly increasing the number of encodings and stored videos. The higher resolutions can be encoded using different base settings so that more base representations can be presented without increasing the encodings and stored videos. In this example, four representations can be derived from two LCEVC encodings, with each LCEVC encoding using different base settings. In the specific example of figure 5, the enhancement layer adds a small bitrate on top of the 540p and 360p profiles to present 720p and 1080p representations. These re-use the bitrate of the base layer, specifically, re-using the approx. 1.9 Mbps (1.1 Mbps) base layer of 1080p as the 540p profile and the base layer of the 720p LCEVC encoding as the 360p representation. This allows 4 much smaller representations to be derived from 2 base and 2 enhancement layers. The specific example shown in the comparison of Figures 4 and 5 leads to an approx. 54% reduction in total bandwidth required to present all profiles. Note that 720p would typically be derived from the 360p base because of the normative scaling modes in the LCEVC standard, i.e. x2 scaling horizontally and vertically. In other words, two enhancement encoded videos can be presented as four representations for ABR logic to move between.

[0096] It may be worth noting at this point that ABR typically has rules around which representation to consider. For example, based on codec support or maximum resolution. The client device typically navigates the ladder in a predetermined pattern based on the manifest and aspects such as startup speed are considered to optimise user experience. The configuration proposed herein facilitates the improvement of those rules by allowing for easy traversal of the ladder to improve user experience.

[0097] The following pseudo-code is an example of this implementation: trackSelectorParameters = new DefaultTrackSelector.ParametersBuilder() setMaxVideoSize(maxVideoWidth, maxVideoHeight)

[0098] .setMaxVideoBitrate(maxVideoBitrate) .setForceHighestSupportedBitrate(false) .buildQ;

[0099] Figure 6 illustrates how the above example implementation of how Scalable Mode can be implemented at the different points of the pipeline. The encoder 100 is configured to produce two LCEVC enhanced encoded videos of a source video 102. In a specific example, this includes 2 fmp4 files, each with 2x tracks, 1x ‘acvcl (i.e. a video encoded using AVC) and 1x ‘IvcT (i.e. an enhancement layer providing an LCEVC enhancement to a respective base layer that is a respective AVC encoded video). In the example, this is a 720p LCEVC enhanced video and a 1080p LCEVC enhanced video. These videos are provided to a modified packager 602.

[0100] The packager 602, as above, curates the manifest file and presents each track of the two enhancement encoded videos as separate independent profiles. The decoder (i.e. the client device or player) will then select the appropriate representation according to the packager defined profile. In the example, an LCEVC enabled client device 608 will download both a base file and an enhancement file and will combine them to recreate the video (or combine them to create the representation, in optional implementations). A legacy client device 210 (decoder or player) will download only the base segments, avoiding having to download the enhancement layer they are not able to process (or avoiding having to have packager present two representations including the base for both the legacy and LCEVC enabled devices). By legacy we mean non-LCEVC enabled.

[0101] Figure 7 illustrates this concept using the pipeline of Figure 2. As described here, delivering LCEVC encoded streams in Scalable Mode can be done using MP4 ISO BMFF carriage and MPEG-DASH delivery. Other mechanisms are considered within the scope of the present disclosure, but examples are presented using MPEG-DASH for brevity, in examples, modifications may be made at the encoder, packager and decoder to adapt them to the delivery method.

[0102] As before, a source video 102 is encoded using the encoder 100 to produce a plurality of variants of the source video encoded using the enhancement codec. Here we illustrate the use of LCEVC encoding the video in an mp4 file. Depending on the implementation of the LCEVC encoder, modifications may or may not be necessary. What is important is that that enhancement layer and base layer can be separated. Therefore in some embodiments, the encoder may be configured to output an encoded version of the source video, encoded using an enhancement codec comprising separate base and enhancement tracks.

[0103] The LCEVC enhancement layer may be encoded “in loop” with the base layer, downscaling the picture before base encode and upscaling the reconstructed image to allow calculation of the EL residual information which is then encoded and transmitted. The LCEVC encoder will produce LCEVC NALUs according to the standard (ISO / IEC23094-2). LCEVC NAL unit types can have the following meaning: It is expected that for every I DR picture in the base the corresponding LCEVC picture is also IDR.

[0104] Alternative approaches are contemplated to facilitate synchronisation of segments (and frames) of enhancement layer data and base layer data sent separately. For example, an Instantaneous Decoder Refresh picture of the base layer and enhancement layer should ideally be synchronised to facilitate combination of the residuals of the enhancement data. Techniques exist within known multi-layer codecs such as SVC and SHVC. Frames of the base layer video and enhancement layer video may also be kept synchronised by techniques set out in WO2023 / 118851 which is incorporated herein by reference. This technique is relevant for HTML element implementations where the player may not have access to information in the base layer. It is described there how a packetised elementary stream (PES) may be constructed of a plurality of T ransport Stream (TS) packets. A presentation time stamp (PTS) is supplied in packets for the base stream. The enhancement stream uses a presentation time stamp in the enhancement stream to synchronise frames of the two streams during HTML reconstructions. In other words, the frames of the enhancement stream are indexed according to the PTS and this is compared with timing metadata such as a “media time” variable or a current playback time provided in the base layer to pair the frame of the enhancement layer with the frame of the base layer. In other words, in HTML5 the player may be provided with RGB data and timing metadata (but not timestamp information) which can be used to figure out timestamp information and combine as appropriate with the enhancement data.

[0105] Alternative approaches include the use of SEI messages to carry timing information which can then be used to synchronise enhancement and base.

[0106] In order to preserve timing information and ensure synchronisation, an example of encapsulation of the LCEVC within the MP4 file format is provided. The encapsulation of an LCEVC bitstream within the MP4 file format can be summarised in Figure 8.

[0107] The two main concepts used to link the LCEVC enhancement to its corresponding XVC base (where X stands for "whatever coding format", e.g. AVC, HEVC, EVC, WC) are:

[0108] (1) link the LCEVC enhancement to the XVC base by means of an "sbas" box identifying which is the corresponding base;

[0109] (2) use the same time base for the "stts" boxes in the enhancement, as they are used in the base, to perform picture to picture synchronization. A process 800 is shown in Figure 8 in which an LCEVC NALU 802 is linked to the base NALU 804 using an «sbas>> box. THE LCEVC NALU 802 is turned into an LCEVC sample with an «stts> box the same as the <stts> time to sample as the Base sample 808 created from the Base NALU. These samples can then be combined into the LCEVC picture 810.

[0110] Example encoder implementations may be configured to return two separate outputs, one for the BL NALUs and one for the EL NALUs with the necessary synchronisation between BL and El. The packaging into the carriage mechanism by the encoder muxer may then generate either: a single mp4 file with the BL and EL in two separate tracks, or two files, one with BL and one with EL, both in a single track.

[0111] In an example implementation, the former can be produced, i.e. a single mp4 with BL and EL in separate tracks. The separation of the tracks into different independent chunks may then be left to the packager.

[0112] Returning to Figure 7, a modified packager 702 is configured to separate the encoded variants into base and enhancement layers for retrieval as a set of Segments and construct a manifest file comprising metadata suitable for constructing URLs to retrieve those segments. As usual, that metadata may comprise a modified URL or a template URL to be combined with information for each segment to construct a URL to retrieve that segment. The separated segments are then stored in storage 204 for later retrieval as usual. Each profile created by the packager may also comprise a suitable indication of average bandwidth and other available metadata usable by an ABR handler or ABR rules to identify and retrieve the appropriate segments.

[0113] In an implementation, the packager will generate segments that are represented by separate MP4 files (or a single file with byte ranges), those only containing BL and those containing EL as a single track representation within. The BL and EL are linked at an elementary stream by the encoder with the method described above. The packager optionally provides a way for the player to associate the BL MP4 chunks and the EL MP4 chunks. This can be done at manifest level. In the implementations described here, this association is using a dependency attribute in the manifest file.

[0114] In order to allow for smooth playback, the packager may signal chunks in the BL and EL at the same start and end times / frames. In certain implementations, this means that the packager will have to recognise the LCEVC I DR NALUs (nal_unit_type = 29) and cut the segment in correspondence of the equivalent AVC / HEVC / WC I DR.

[0115] Matching I DR frames is preferably ensured by the encoder, as explained above.

[0116] As explained elsewhere in this document, a dependency attribute is one possible implementation used to create an association between the EL segments and the BL segments that they are enhancing. A dependency attribute may be set by the packager 702 in the manifest file 705. In the example shown in Figure 7, this is shown as ‘dID’ and links the representation in the Adaptation Set for the enhancement layer to a representation in the Adaptation Set for the base layer. The bandwidth for the enhancement Adaptation Set may represent the bandwidth of the combined representation of the BL and EL or optionally the bandwidth of the enhancement layer only for the client device to combine it with the bandwidth requirement of the base to identify the total bandwidth. The latter may be preferable but that may depend on the standards and how they are designed.

[0117] Contemporaneous use of the dependency ID attribute of Adaptation Sets has been only within a single Adaptation Set for purposes such as HDR. There has hitherto been no proposed and documented use or mechanism for the combination of two video Adaptation Sets. Contemporary players understand how to combine segments retried from multiple Adaptation Sets, for example for video and audio or subtitles, but players have not been adapted for the combination of two Adaptation Sets comprising video representations.

[0118] In certain embodiments, the Adaptation Set comprising the enhancement layer may be labelled as Data rather than video to facilitate retrieval and combination by players and client devices. The present disclosure presents the concept of separating the base and enhancement layers for storage and presentation and provides mechanisms to allow ABR logic to move between representations representing an encoded video stream using the combined base and enhancement layers as well as base encoded video alone. Use of a dependency ID attribute to link enhancement layer segments with respective base layer segments is just one possible preferred implementation.

[0119] In this exemplary scenario illustrated, we expect one single Adaptation Set but with separate Representations. Other adaptations sets may of course be included in the MPD, but for the sake of explanation here, there are multiple representations within each Adaptation Set for each MIME type. Each Representation points to an MP4 Segment which contains a single track, for the BL only the legacy bitstream, e.g., AVC; for the EL only the enhancement encoded in LCEVC. The link from EL to BL is external, from the EL MP4 to the BL MP4, and is provided using the @DependencylD attribute in MPEG-DASH.

[0120] This mechanism is described in DASH-IF Interoperability - Guidelines for Implementations, Annex G: Sample Dual-Layer MPD. The example reported in the guidelines is shown as follows:

[0121] <Period>

[0122] <!- Video ->

[0123] <AdaptationSet mimeType="video / mp4" subsegmentAlignment- 'true" subsegmentStartsWithSAP="1 " frameRate="24">

[0124] Representation id="base-layer" codecs="hev1.2.4.H153.bO" bandwidth="29200362" width="3840" height="2160">

[0125] <BaseURL>BL_Dashlf_BL-HDR10_UHD.mp4< / BaseURL> <SegmentBase indexRange="687-862"> clnitialization range="0-686" / >

[0126] < / SegmentBase>

[0127] < / Representation>

[0128] Representation id="enhancement-layer" dependencyld- 'base-layer" codecs="dvhe.07.03" bandwidth="9320458" width="1920" height="1080"> <BaseURL>EL_Dashlf_BL-HDR10_UHD.mp4< / BaseURL> <SegmentBase indexRange="723-898"> clnitialization range="0-722" / >

[0129] < / SegmentBase>

[0130] < / Representation>

[0131] < / AdaptationSet>

[0132] < / Period>

[0133] < / MPD>

[0134] In the case of an LCEVC enhancement, the only attribute changing is the 4cc for codecs, which for LCEVC is “Ivc1”. A sample manifest for LCEVC is set out as follows:

[0135] <?xml version="1.0" encoding="utf-8"?>

[0136] <MPD xmlns:xsi=http: / / www.w3.org / 2001 / XMLSchema-instance xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xlink=http: / / www.w3.org / 1999 / xlink xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 http: / / standards.iso.org / ittf / PubliclyAvailableStandards / MPEG-

[0137] DASH_schema_files / DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" type="static" mediaPresentationDuration="PT 10.0S" maxSegmentDuration="PT5.0S" minBufferTime="PT20.0S">

[0138] <Programlnformation>

[0139] < / Programlnformation>

[0140] <ServiceDescription id="0">

[0141] < / ServiceDescription>

[0142] <!- In this Period the LCEVC + Base stream is split into two representations: BL and EL ->

[0143] <!- In this example 4 representations are used to inform the player of the existence of 4x profiles, 2 of which are derived from the combination of base and enhancement ->

[0144] <!- the linking mechanism is dependency! D->

[0145] <Period id="0" start="PT0.0S"> <AdaptationSet id="0" contentType- 'video" startWithSAP="1" segmentAlignment- 'true" bitstreamSwitching- 'true" frameRate="25 / 1 " maxWidth- ' 1920" maxHeight="1080" par="16:9">

[0146] <!- Independent Representation (BL) of profile 0: 360p-> Representation id="0" mimeType="video / mp4" codecs="avc1 .64001 f" bandwidth="1100000" width="640" height="360" sar="1 :1"> <SegmentTemplate timescale="12800" initialization- 'init- stream$RepresentationlD$.m4s" media="chunk-stream$RepresentationlD$- $Number%05d$.m4s" startNumber="1 "> <SegmentTimeline>

[0147] <S t="1024" d="128000" / >

[0148] <S d="1024" / > < / SegmentTimeline> < / SegmentTemplate> < / Representation>

[0149] <!- Independent Representation (BL) of profile 1 : 540p-> Representation id="1" mimeType="video / mp4" codecs="avc1 .64001 f" bandwidth="1400000" width="960" height="540" sar="1 :1"> <SegmentTemplate timescale="12800" initialization- 'init- stream$RepresentationlD$.m4s" media="chunk-stream$RepresentationlD$- $Number%05d$.m4s" startNumber="1 "> <SegmentTimeline>

[0150] <S t="1024" d="128000" / > <S d="1024" / > < / SegmentTimeline> < / SegmentTemplate> < / Representation>

[0151] <!- Dependent Representation (EL) of profile 2: combines with pO to produce 720p->

[0152] Representation id="2" mimeType="video / mp4" codecs="lvc1" bandwidth="800000" width="1280" height="720" sar="1 :1" dependencyld="O"> <SegmentTemplate timescale="12800" initialization- 'init- stream$RepresentationlD$.m4s" media="chunk-stream$RepresentationlD$- $Number%05d$.m4s" startNumber="1 "> <SegmentTimeline>

[0153] <S t="1024" d="129024" / > < / SegmentTimeline> < / SegmentTemplate> < / Representation>

[0154] <!- Dependent Representation (EL) of profile 3: combines with p1 to produce 1080p— >

[0155] Representation id="3" mimeType="video / mp4" codecs="lvc1" bandwidth="900000" width="1920" height="1080" sar="1 :1" dependencyld="1"> <SegmentTemplate timescale="12800" initialization- 'init- stream$RepresentationlD$.m4s" media="chunk-stream$RepresentationlD$- $Number%05d$.m4s" startNumber="1 "> <SegmentTimeline>

[0156] <S t="1024" d="129024" / > < / SegmentTimeline> < / SegmentTemplate> < / Representation>

[0157] < / AdaptationSet>

[0158] < / Period>

[0159] < / MPD> The concept of Dependent Representation (which is the case for an LCEVC EL, depending on its corresponding BL), is introduced in 23009-1 , clause 5.3.5.1.

[0160] The following is a quote from 23009-1 , clause 5:

[0161] “Dependent Representations are described by a Representation element that contains a @dependencyld attribute. Dependent Representations are regular Representations except that they depend on a set of complementary Representations for decoding and / or presentation. The @dependencyld contains the values of the @id attribute of all the complementary Representations, i.e. Representations that are necessary to present and / or decode the media content components contained in this dependent Representation.”

[0162] @codingDependency is defined in 23009-1 , clause 5.3.7.2 (Semantics), and clause 5.3.7.3 (Syntax), as a common attribute for AdaptationSet and Representation.

[0163] Further relevant information can be found in 230090 - Dynamic adaptive streaming over HTTP (DASH), Part 3: Implementation guidelines, Section 10. Mapping of ISOBMFF’s track reference type to DASH MPD. The following is a quote from 23009-3, clause 10:

[0164] “Dependencies (See definition of “Dependent Representations” in Section 5.3.5.1 . of DASH Part-1):

[0165] For a given track reference type (one row of the Table), when the “Mapping to DASH” column of the Table indicates “Dependency”, it means that a Representation@dependencyld should be set from the Representation corresponding to the track declaring the track reference of the given type to Representation(s) corresponding to the track(s) having their track_ID listed in the ‘tref box. The list of Representation ids in the @dependencyld attribute shall match the track D order from the ‘tref’ box.”

[0166] In a further possible implementation detail, concerning the MIME types for the Base Layer and the Enhancement Layer, the Base Track will have the Sample Description Box according to the base codec (e.g. "avc1", "hvc1", "evs1", "vvc1"...) while the latter will have consistently its Sample Description Box (e.g. "Ivc1").

[0167] As stated above, the link between the EL and its corresponding BL may be implemented by means of the "sbas" box. Note in other embodiments set out herein, the current time from the media element is used to compare pts on enhancement side and so this example is not needed.

[0168] Returning to Figure 7, in the known manner, in order to retrieve a video stream, a client device will first retrieve the manifest file. In the known manner, a legacy device 210, i.e. a non-LCEVC enabled device, will retrieve and parse the manifest file and will retrieve a representation according to the Adaptation Set which they support by constructing a URL to the base layer encoded video and downloading segments of the video to fill their available buffer according to present conditions. References to client devices here may equally refer to a player or decoder.

[0169] A modified client device according to principles of the present invention will retrieve the manifest file and parse the manifest file to identify a representation of the enhancement layer and from the dependency attribute may identify an associated representation of the base layer. In a first example, the client device will determine the appropriate representation from the enhancement Adaptation Set and will the retrieve the associated base layer segments as appropriate. In a second example, the client device may build a set of virtual representations from the manifest file, the virtual representations indicating the combination of the base layer and enhancement layer indicated by the dependency attribute.

[0170] References to client devices here may equally refer to a player or decoder. An example player to be modified is dash.js.

[0171] The client device will retrieve the base layer and enhancement layer segments and pass these to an appropriate enhancement decoder for decoding and combination for rendering by the client device.

[0172] In an example process, a client device referred to here as a decoder 708, will have to parse the manifest file and choose from a set of renditions based on the bandwidth it has available. It is expected for the player (i.e. the player on the client device or decoder such as dash.js) to build a combined virtual set of representations where an enhanced profile will have a bandwidth equal to the sum of EL and BL. The example process may include the following:

[0173] 1. The DASH manifest parser should parse the @DependencylD attribute to set up the dependency between BL and EL Representations.

[0174] 2. A DASH ABR engine may maintain its list of Representations considering BL as a Representation, EL togetherwith its dependency BL as another Representation. In the manifest example of Figures 5 and 6 above, the DASH ABR engine would maintain 4 Representations: a. Representation 1 with 360p BL only, with bandwidth=1100000 b. Representation 2 with 540p BL only, with bandwidth= 1400000 c. Representation 3 with 720p EL with its dependency 360p BL , with bandwidth=1900000, which is sum of 720p EL bandwidth(Representation id 2) and 360p BL bandwidth (Representation id 0). d. Representation 4 with 1080p EL with tits dependency 720p BL , with bandwidth=2300000, which is sum of 1080p EL bandwidth (Representation id 3) and 540p BL bandwidth (Representation id 1).

[0175] 3. When the DASH ABR engine detects enough bandwidth and codec ability (i.e. LCEVC support) to be able to choose the bitrate matching the bandwidth for the Representation containing the EL: a. The DASH downloader may download the BL segment and its related EL segment. The downloader may make sure BL and EL segments are downloaded in asynchronous mode as much as possible so all required EL and BL data are available to be able to decode a full picture. In other words, the segments are to be downloaded together as opposed to synchronous download which would mean one after the other. Synchronous mode is also contemplated. b. The playback pipeline may feed the BL data to the base decoder through the available interface (e.g. mediacodec in Android) and EL data to the LCEVC Decoder to decode the full picture. Further details may be provided in WO2022 / 023739 which is incorporated by reference, which describes a Decoder Integration Layer (DIL).

[0176] 4. If the ABR engine detects a bandwidth or a codec ability matching the BL stream only, only BL data may be downloaded and fed into base decoder as it is currently done in contemporary processes.

[0177] A simplified example of this process 900 from the perspective of the client device is illustrated in Figure 9. At step 902, the modified manifest file is obtained. The manifest file is then parsed at step 904. Based on available conditions, a Representation for retrieval is identified at step 906 and a dependency identified from the dependency attribute at step 908. At step 910, the base layer segment is downloaded and its related enhancement layer segment based on the dependency. At step 912 these are passed to a decoder for decoding and reconstruction of the enhancement video.

[0178] As indicated above, concepts set out herein may be implemented at a client device, player on the device or decoder. Similarly, concepts may be embodied by modifications to an encoder, packager and / or content delivery network. At each of these entities, methods and processes described herein can be embodied as code (e.g., software code) and / or data. The functionality may be implemented in hardware or software as is well-known in the art of data compression and video streaming. For example, hardware acceleration using a specifically programmed Graphical Processing Unit (GPU) or a specifically designed Field Programmable Gate Array (FPGA) may provide certain efficiencies. For completeness, such code and data can be stored on one or more computer-readable media, which may include any device or medium that can store code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on a computer-readable medium, the computer system performs the methods and processes embodied as data structures and code stored within the computer-readable storage medium. In certain embodiments, one or more of the steps of the methods and processes described herein can be performed by a processor (e.g., a processor of a computer system or data storage system). Generally, any of the functionality described in this text or illustrated in the figures can be implemented using software, firmware (e.g., fixed logic circuitry), programmable or nonprogrammable hardware, or a combination of these implementations. The terms “component” or “function” as used herein generally represents software, firmware, hardware or a combination of these. For instance, in the case of a software implementation, the terms “component” or “function” may refer to program code that performs specified tasks when executed on a processing device or devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

Claims

CLAIMS1. A method of streaming multimedia content encoded using an enhancement coding scheme, the method comprising: retrieving a manifest file, the manifest file comprising metadata for constructing a locator to segments of the encoded multimedia content, the metadata comprising: a first representation pointing to a first version of the multimedia content encoded using a base coding scheme; and, location metadata for a first enhancement layer of the encoded multimedia content associated with the first version; and, parsing the manifest file to construct a locator for retrieval of the first representation and segments of the enhancement layer; downloading segments of the first representation and segments of the enhancement layer for reconstructing the encoded multimedia content.

2. A method according to claim 1 , wherein the metadata comprises a second representation pointing to the first enhancement layer of the encoded multimedia content.

3. A method according to claim 2, wherein the metadata further comprises a dependency attribute associating the second representation with the first representation.

4. A method according to claim 2 or 3, wherein the first and second representations are comprised in respective Adaptation Sets.

5. A method according to any of claims 2 to 4, wherein the metadata further comprises, for each representation, a bandwidth for retrieving the respective representation, and the method further comprises: determining a total bandwidth for streaming the multimedia content by summing the bandwidth associated with the second representation and the bandwidth of its respective associated first representation.

6. A method according to claim 2 to 5, wherein:the first representation comprises an attribute indicating the base coding scheme; and, the second representation comprises an attribute indicating the enhancement coding scheme.

7. A method according to any of claims 2 to 6, wherein the metadata further comprises: a third representation pointing to a second version of the multimedia content encoded using the base coding scheme, the first and second versions being versions of the multimedia content at a different level of quality; and, a fourth representation pointing to a second enhancement layer associated with the second version of the encoded multimedia content.

8. A method according to any of claims 3 to 7, wherein the method further comprises: building a set of virtual representations from the first and second representations and the dependency attribute, wherein the virtual representations comprise: a first virtual representation comprising the first representation; and, a second virtual representation comprising the first and second representations.

9. A method according to any preceding claim, further comprising: sensing available bandwidth and / or device conditions; and, downloading segments of the first representation or segments of both the first representation and segments of the enhancement layer for reconstructing the encoded multimedia content according to the available conditions.

10. A method according to any preceding claim, comprising: passing downloaded segments of the enhancement layer to an enhancement decoder to generate decoded enhancement data; and,reconstructing multimedia content from the decoded enhancement data and a decoded version of the video stream encoded using the base coding scheme decoded from the downloaded segments of the first representation.

11. A method of packaging multimedia content encoded using an enhancement coding scheme for streaming, the method comprising: constructing a manifest file, the manifest file comprising metadata for constructing a locator to segments of the encoded multimedia content, the metadata comprising: a first representation pointing to a first version of the multimedia content encoded using a base coding scheme; and, location metadata for a first enhancement layer of the encoded multimedia content associated with the first version.

12. A method according to claim 11 , wherein the metadata comprises a second representation pointing to the first enhancement layer of the encoded multimedia content.

13. A method according to claim 12, wherein the metadata further comprises a dependency attribute associating the second representation with the first representation.

14. A method according to claim 12 or 13, wherein the first and second representations are comprised in respective Adaptation Sets.

15. A method according to any of claims 12 to 14, wherein the metadata further comprises, for each representation, a bandwidth for retrieving the respective representation, and the method further comprises: determining a total bandwidth for streaming the multimedia content by summing the bandwidth associated with the second representation and the bandwidth of its respective associated first representation.

16. A method according to claim 12 to 15, wherein: the first representation has a type indicating the base coding scheme; and,the second representation has a type indicating the enhancement coding scheme.

17. A method according to any of claims 12 to 16, wherein the metadata further comprises: a third representation pointing to a second version of the multimedia content encoded using the base coding scheme, the first and second versions being versions of the multimedia content at a different level of quality; and, a fourth representation pointing to a second enhancement layer associated with the second version of the encoded multimedia content.

18. A method according to any of claims 11 to 17, further comprising: receiving an encoded version of the multimedia content encoded using the enhancement coding scheme comprising a base layer and an enhancement layer; generating a first set of segments representing the base layer; and, generating a second set of segments representing the enhancement layer, wherein the first set of segments are independent of the second set of segments.

19. A method according to any of claims 11 to 17, wherein the first set of segments and second set of segments are generated as respective single track representations.

20. A method according to any of claims 11 to 19, wherein the constructing the manifest file comprises: inserting an association between the second set of segments and the first set of segments; obtaining a location of the first and second set of segments; and, inserting metadata for constructing a locator to the location of the first and second set of segments.

21. A client device for streaming multimedia content encoded using an enhancement coding scheme configured to perform the method of any of claims 1 to 10.

22. A packager for packaging multimedia content encoded using an enhancement coding scheme for streaming configured to perform the method of any of claims 1 to 20.

23. A system comprising a client device according to claim 21 and a packager according to claim 22.

24. A system according to claim 23 further comprising: an encoder configured to encode the multimedia content using an enhancement coding scheme; and, a content delivery network configured to serve the segments and the manifest file.

25. A computer readable medium claim comprising instructions which when executed by a computer cause the computer to perform the method of any of claims 1 to 10 or 11 to 20.

Images