Implementation of an extended coding scheme in streaming applications

Abstract

Implementation of an extended coding scheme in streaming applications Methods for streaming multimedia content encoded using an extended coding scheme and methods for packaging multimedia content encoded using an extended coding scheme for streaming may be provided. These include obtaining and constructing a manifest file, the manifest file containing metadata for constructing locators to segments of the encoded multimedia content, the metadata including a first representation pointing to a first version of the multimedia content encoded using a base coding scheme, and location metadata for a first extension layer of the encoded multimedia content associated with the first version. Client devices, packagers, systems, and computer-readable media are also provided.

Description

Technical Field

[0001] The present invention relates to a method, system, computer program, and computer-readable medium for implementing an extended coding scheme in multimedia streaming applications, particularly video streaming applications.

Background Art

[0002] Streaming applications enable the rapid and efficient remote distribution of video content to users. Generally, video content is pre-encoded at various resolutions using conventional coding schemes, while ensuring that the video content is compatible with a wide range of user devices while providing the best possible experience.

[0003] As newer players capable of displaying higher resolution video content become widespread and are generally connected via wireless connections, various techniques must be employed to improve bandwidth utilization and ensure that the delivery of video content is reliable and provides a seamless experience to end-users.

[0004] Most video content on the Internet is encoded using H.264 (also known as MPEG-4 Part 10, Advanced Video Coding - MPEG-4 AVC). This content is typically supplied to a decoding device as a single video stream that has a one-to-one relationship with the available hardware and / or software video decoders. For example, a single stream is received, parsed, and decoded by a single video decoder to output a reconstructed video signal. Therefore, many video decoder implementations are developed according to this framework.

[0005] To support different streaming conditions, client devices are typically configured with a switching mechanism driven by metadata that identifies the stream format. One example is Adaptive Bitrate Streaming (or ABR), and a common standardized mechanism for implementing this is Dynamic Adaptive Streaming over HTTP (DASH) (also known as MPEG-DASH). Other common HTTP-based streaming protocols include HTTP Live Stream (HLS) and Real-Time Messaging Protocol (RTMP). Both function similarly to HTTP (DASH), although RTMP is not HTTP-based, but rather TCP / UDP-based.

[0006] In ABR streaming, video is coded into multiple resolution or bitrate combinations, each typically referred to as a "rendation" (or representation, see the context of the manifest file below). Client devices determine their available bandwidth and obtain appropriate rendations in formats they support, depending on their available bandwidth. A collection of rendations is typically called a ladder or profile.

[0007] To implement this, DASH requires each client device to obtain a manifest file known as an MPEG-DASH MPD (Media Presentation Description) file. This file contains metadata that allows the client device to construct a Uniform Resource Locator (URL) and retrieve the appropriate stream based on available conditions (e.g., device capabilities, resolution, available bandwidth, buffer status, etc.). Within each manifest file is a set of representations, each describing one version of the content. The manifest file describes how the video has been divided into pieces by the packager and how they are arranged, allowing the client device to request chunks of video for subsequent playback. Generally, a representation includes specific video and audio tracks to describe the version of the media stream, bitrate, resolution, and other information associated with the media content. A manifest is sometimes referred to as a playlist in other formats.

[0008] Existing extension coding schemes include the Scalable Video Coding (SVC) extension to H.264, the Scalable Extension to H.265 (MPEG-H Part 2 High Efficiency Video Coding - SHVC), and newer standards such as MPEG-5 Part 2 Low Complexity Extension Video Coding (LCEVC) and SMPTE ST-2117 (VC-6). SVC and SHVC work by creating different encoding layers, each providing a different spatial resolution. Each layer encodes the input according to a normal AVC or HEVC encoder, but may also leverage information generated by the lower encoding layers. LCEVC, on the other hand, generates one or more layers of extension residuals compared to the decoded version of the base encoding. The base encoding can typically have lower spatial resolution.

[0009] Unlike traditional coding methods, LCEVC encodes a lower-quality or base-quality version of the source image using existing codecs. An enhancement layer is encoded that can be used to improve the decoded base version and make it higher quality. The enhancement layer adds more flexibility to how the video stream can be implemented and provides new features (e.g., enhanced compression capabilities of the base codec, reduced encoding and decoding complexity, and a platform for further future enhancements).

[0010] For ABR streaming (and DASH) to be compatible with various user devices and adapt to changing network conditions, content must be encoded and stored at different bitrates and resolutions so that each device can obtain the appropriate stream. Similarly, some devices can stream content encoded using a specific coding scheme, while legacy devices may not. If content is to be streamed by as many users as possible, it must be encoded and stored using a scheme available to those user devices. This generally means that there must be at least one representation of the content for every possible combination of device setup, bitrate, and resolution. For example, an advanced setup with high bandwidth can stream media at high bitrates, frame rates, or resolutions, while a basic setup or one with insufficient bandwidth can stream media of lower quality.

[0011] Therefore, supporting ABR streaming of multimedia content encoded using the extended coding scheme significantly increases the number of versions encoded and stored for delivery, and the number of available representations in each manifest file. The increased flexibility of the extended coding scheme comes with inherent compromises: the surge in representations for each variant, combination, and permutation is balanced by the inherent flexibility and quality improvements to the available conditions resulting from the ability to stream those representations.

[0012] Simply put, the difficulty in implementing extended coding schemes lies in the fact that managing encoders, packagers, and decoders becomes far more complex than managing the same things with traditional "single-layer" coding schemes. This is because the introduction of extended coding schemes (which introduce extended coding layers) results in a much greater number of possible combinations and permutations for encoders, packagers, and decoders. [Overview of the project]

[0013] According to aspects of the present invention, a method for streaming multimedia content encoded using an extended coding scheme may be provided. The method may include obtaining a manifest file, the manifest file containing metadata for constructing locators to segments of encoded multimedia content, the metadata including a first representation pointing to a first version of the multimedia content encoded using a base coding scheme, and location metadata for a first extended layer of the encoded multimedia content associated with the first version; parsing the manifest file to construct locators for obtaining the first representation and the segments of the extended layer; and downloading the segments of the first representation and the segments of the extended layer for reconstruction of the encoded multimedia content.

[0014] According to aspects of the present invention, a method may be provided for packaging multimedia content encoded using an extended coding scheme for streaming, the method comprising constructing a manifest file, the manifest file comprising metadata for constructing locators to segments of the encoded multimedia content, the metadata comprising a first representation pointing to a first version of the multimedia content encoded using a base coding scheme, and location metadata for a first extended layer of the encoded multimedia content associated with the first version.

[0015] In this way, the number of variations of encoded multimedia content that need to be maintained and supported can be reduced. By presenting the augmentation and base layers of the encoded content separately, client devices can obtain only the base layer depending on the available bandwidth and device conditions, but can also obtain both the base layer and augmentation data for combination and reconstruction when appropriate, even though they are stored and presented separately. Legacy devices can obtain legacy encoded data without associated augmentation data, thus utilizing all available bandwidth. Meanwhile, their legacy encoded data can be used as base encoded data by augmentation-enabled devices to reconstruct higher-quality content using separately presented augmentation data.

[0016] Encoded multimedia content can be a video stream. The extended encoding scheme can be LCEVC, and the base encoding scheme can be any “single-layer” encoding scheme (e.g., AVC, HEVC, AV1, or VVC). The manifest file may contain a Media Presentation Description (MPD). The locator can be a link or a Uniform Resource Locator (URL). The first version of multimedia content encoded using the base encoding scheme can be the base layer of multimedia content encoded using the extended encoding scheme (i.e., the base layer associated with a particular first extended layer). Segments can be downloaded according to the guidelines outlined in MPEG-DASH, HTTP Live Streaming (HLS), or Microsoft Smooth Streaming. In other words, MPEG-DASH and other streaming protocols are good guidelines for indicating the order and retrieval of segments. Since the download itself is done on the player side, it will differ for different players based on configurable parameters that control the download of segments, such as bandwidth or quality and many more such parameters.

[0017] The term "manifest" is used because it is the official term for streaming protocols, but it is understood that, depending on the format used, it may be considered interchangeable with the term "playlist." In some situations, a manifest may be passed as a playlist.

[0018] The concept according to the present invention facilitates adaptive bitrate streaming of the base layer of encoded multimedia content, separate from its extension layer, and the client device performs the function of determining which bitrate segment to download based on the current state of the network.

[0019] The second representation may be at a higher quality level than the first representation. The quality level refers to things like resolution, bitrate, color depth, and file size. In terms of representation, a higher quality level may mean that support for more advanced client devices or better conditions (e.g., network or device, such as buffer capacity) are required.

[0020] This method can be performed by a client device, player, or decoder.

[0021] Preferably, the metadata includes a second representation that points to a first extension layer of the encoded multimedia content. Location metadata may be included in the second representation. Thus, the manifest file presents multiple representations (each corresponding to a different layer of the extension coding scheme) for selection by the client device. By downloading both layers according to the network conditions, they can then be combined and made available for rendering the multimedia content on the client device. In the implementation, each representation corresponds to a single track. By using the second representation, the base layer and the extension layer may be provided with separate external links relating the two to each other.

[0022] More preferably, the metadata further includes dependency attributes that associate a second representation with a first representation. These dependency attributes may be dependency ID attributes in the MPEG-DASH manifest file. The dependency attributes provide an association between the second and first representations and can separately provide external links (i.e., dependencies) to them. The dependency attributes provide the use of existing mechanisms for previously unforeseen purposes, thereby providing an immediate path to implementation without significant changes to existing standards and devices. The dependency attributes mean that any implementation can be deployed without affecting currently deployed devices and without requiring software updates for those devices to function.

[0023] Along with dependency attributes, the representation may also include the resolution of the encoded multimedia content. The second representation may include the resolution of the combined base and extension layers when constructed. Furthermore, the representation may also include bandwidth attributes, minimum, maximum, or average, for obtaining the representation. The bandwidth attributes of the second representation may correspond to the bandwidth required to obtain only the extension layer, or the bandwidth required to obtain both the representation of the extension layer and the associated base layer representation.

[0024] This method may further include determining the dependency between the second representation and the first representation. In other words, it sets the second representation based on the dependency. This method may further include parsing the dependency attribute to identify the association between the second representation and the first representation. In the context of MPEG DASH terminology, the dependency may be a dependencyId.

[0025] Preferably, the first and second representations are contained within their respective adaptation sets. By using each adaptation set, an ABR algorithm in a client device supporting one or both of the base coding scheme and / or extension scheme can traverse the ABR ladder of that scheme. The ABR ladder can include the extension layer representations while retrieving the base layer representations by using dependencies. Dependencies can associate representations between different adaptation sets or within the same adaptation set. Alternatively, the first and second representations may be contained within the same adaptation set and offered for selection by a single ABR algorithm traversing one ABR ladder within the same adaptation set. Optionally, the first representation may be repeated across different adaptation sets, and the different adaptation sets may be associated with different MIME types or supported codecs.

[0026] In an arbitrary implementation, metadata may further include, for each representation, a bandwidth for obtaining each representation, and the method may further include determining a total bandwidth for streaming multimedia content by summing the bandwidth associated with the second representation and the bandwidth of each associated first representation. In this way, the client device can use different representations to select an appropriate representation according to the available network state. As described above, a representation may also include a bandwidth associated with the total bandwidth for obtaining the second representation and the first representation related thereto.

[0027] Preferably, the first representation includes an attribute indicating a base coding method, and the second representation includes an attribute indicating an extended coding method. The attribute may include a "codec" attribute. Examples include "avc1.64001f" and "lvc1". The attribute may also include a MIME type attribute. Examples include video / h264. By setting the attributes in this way, the client device can select an ABR profile according to the codec supported by the device.

[0028] The metadata may further include a third representation that refers to a second version of the multimedia content encoded using the base coding method, where the first and second versions are versions of the multimedia content at different quality levels, and a fourth representation that refers to a second extension layer associated with the second version of the encoded multimedia content. The different quality levels may be higher quality levels as described elsewhere in this specification.

[0029] The first representation can be 340p, the associated second representation can be 720p, the third representation can be 540p, and the associated fourth representation can be 1080p. In this way, four representations can be provided from only two encoded and stored videos. In this example, the base layer is used as the low-quality representation in the ABR ladder, and by using it in different base layers of the enhancement layers at higher quality levels in the ABR ladder, a way to present multiple representations using a small number of encoded and stored videos and tracks is shown.

[0030] The following is a preferred implementation of the above method for streaming multimedia content encoded using an extended coding scheme.

[0031] The method may include constructing a set of virtual representations from the first and second representations and dependent attributes, the virtual representations further including a first virtual representation including the first representation and a second virtual representation including the first and second representations. The method may further include selecting a virtual representation for acquisition according to the ABR rules. The virtual representation enables the ABR algorithm to traverse the virtual representation according to available conditions, and the virtual representation includes the representations of the base layer and the enhancement layer presented separately in the manifest file.

[0032] The method may further include detecting the available bandwidth and / or device conditions and downloading segments of the first representation or both segments of the first representation and the enhancement layer to be used for reconstructing the multimedia content encoded according to the available conditions. The method may also include selecting a representation for downloading according to the available bandwidth and / or device conditions. The state of the bandwidth may be considered as the network state. Examples of device conditions include the buffer status, power status, decoder capabilities, or hardware capabilities. In this way, a representation can be selected and the ABR profile can be traversed according to a set of predetermined rules.

[0033] This method may also include passing the downloaded segments of the extension layer to the extension decoder to generate decoded extension data, and reconstructing multimedia content from the decoded extension data and a decoded version of the video stream encoded using the base coding scheme and decoded from the downloaded segments of the first representation. Optionally, the downloaded segments of the base layer may be passed directly to the base decoder or via the extension decoder. The base decoder may then operate under the control of the extension decoder or independently thereof.

[0034] The following is a preferred implementation of a method for packaging multimedia content encoded using an extended coding scheme for streaming.

[0035] Preferably, the method may further include receiving an encoded version of multimedia content encoded using an extended coding scheme that includes a base layer and an extension layer, generating a first set of segments representing the base layer, and generating a second set of segments representing the extension layer, wherein the first set of segments is independent of the second set of segments. Independent can be thought of as distinct. In this way, the packager can store the segments separately for subsequent retrieval. The segments may be stored in separate files, or in a single file with a defined byte range for each. Alternatively, the segments of both layers may be stored in a single file with a defined byte range, and the packager generates segments for a manifest file, including their locations for separate representations, and presents the retrieval of those segments separately.

[0036] The segments of the first set and the segments of the second set can be generated as individual track representations, respectively.

[0037] Constructing a manifest file may involve inserting associations between the second set of segments and the first set of segments, obtaining the locations of the first and second sets of segments, and inserting metadata for constructing locators at the locations of the first and second sets of segments.

[0038] The embodiments described herein may generally be player and protocol independent and may use, for example, various containers (e.g., ISO-based media file formats (e.g., MP4 file format) or MPEG-2 transport streams) and various clients (e.g., those based on HTML5 or other transport stream mechanisms). Theoretically, if DASH dependencyId is implemented, the embodiments presented herein will work on any platform that supports LCEVC.

[0039] According to aspects of the present invention, a method is provided which includes parsing a file (e.g., a manifest file) and selecting from a set of renditions based on the determined available bandwidth.

[0040] A method is described that involves parsing attributes to determine dependencies between lower-level representations of a hierarchical coding scheme (e.g., base layer representations, e.g., AVC, HEVC, VVC, etc.) and higher levels of the hierarchical coding scheme (e.g., extended layer representations, e.g., LCEVC). This determination may be referred to as setup. A manifest file may contain these attributes. Parsing may involve parsing the manifest file.

[0041] The method may further include maintaining a list of representations. The list may include one or more representations containing a base layer (e.g., no extension layer). The list may include one or more representations containing a base layer and (e.g., associated) extension layers. The list may include a first representation containing a first version of the base layer. The list may include a second representation containing a second version of the base layer. The list may include a third representation containing a first version of the base layer associated with a first version of the extension layer. The list may include a fourth representation containing a second version of the base layer associated with a second version of the extension layer. The second version may be of higher quality (e.g., higher resolution, larger file size, etc.) than the base layer and / or the first version of the extension layer. Each version of the extension layer may include a dependency identifier configured to identify the base representation on which the version of the extension layer depends.

[0042] The method may include determining the total bandwidth associated with each representation in a list of representations. The method may include determining the total bandwidth associated with each representation in a list of representations by summing the bandwidths associated with each component of the representation. For example, the method may include determining the total bandwidth associated with a third representation by combining (e.g., summing) the bandwidth associated with the first version of the base layer and the bandwidth associated with the first version of the extension layer. For example, the method may include determining the total bandwidth associated with a fourth representation by combining (e.g., summing) the bandwidth associated with the second first version of the base layer and the bandwidth associated with the second first version of the extension layer.

[0043] This method may include determining available bandwidth. This method may include determining coding attributes of a device associated with this method (for example, a device running this method). Coding attributes may indicate that the device supports a coding scheme associated with the extension layer (and / or base layer). The aforementioned "available bandwidth" may be the available bandwidth between the device (i.e., a player, decoder, etc.) and the streaming device (e.g., a content server).

[0044] This method may include selecting an expression from a list of expressions based on the determined available bandwidth. This method may include selecting an expression from a list of expressions based on the determination of coding attributes of the device associated with this method (for example, the device on which this method is performed). This method may include selecting an expression associated with bandwidth that matches the determined bandwidth (for example, has, includes, etc.).

[0045] This method may include downloading the segments associated with the selected representation (for example, the base layer version and, if applicable, the extension layer version associated with the selected representation). This method may include downloading the segments in synchronous mode.

[0046] This method may further include sending a segment(s) to the appropriate decoder. The transmission may be performed via an interface. For example, a segment associated with a base layer version may be sent to the base decoder. For example, a segment associated with an extension layer version may be sent to the extension decoder.

[0047] The method may include selecting representations that do not include (and / or are not associated with) extension layer segments (for example, the selected representations may include only those associated with base layer segments, and / or the selected representations may be associated only with base layer segments). Such selections may be made due to low determined bandwidth and / or the device's codec attributes not corresponding to coding schemes associated with extension layers. In response to the selection (i.e., selecting representations that include only base layer segments and / or are associated only with base layer segments), the method may include downloading only base layer segments. The method may include sending these base layer segments to the base decoder.

[0048] This method may include processing the decoded segment to enable content playback. This method may include combining the decoded base segment with an associated decoded extended segment to generate an output / display video.

[0049] This method can be performed by decoders, players, client devices, etc.

[0050] According to aspects of the present invention, a client device for streaming multimedia content encoded using an extended coding scheme may be provided, configured to perform any of the methods described above.

[0051] According to aspects of the present invention, a packager is provided for packaging multimedia content encoded using an extended coding scheme for streaming, configured to perform any of the methods described above.

[0052] According to aspects of the present invention, a system may be provided that includes a client device and a packager as described above. The system may further include an encoder configured to encode multimedia content using an extended coding scheme, and a content distribution network configured to provide segments and manifest files.

[0053] In a further embodiment, a computer-readable medium may be provided that, when executed by a computer, contains instructions causing the computer to perform any of the methods described above in the present invention.

[0054] Next, examples of systems and methods according to the present invention will be described with reference to the accompanying drawings. [Brief explanation of the drawing]

[0055] [Figure 1] This is a known high-level schematic diagram of the LCEVC encoding, decoding, and transport processes. [Figure 2] This is a known high-level schematic diagram of the ABR process that implements extended coding. [Figure 3] This figure shows an example of a known ABR profile for extended coding. [Figure 4] This figure shows an example of a known approach to encoding and storing representations for ABR. [Figure 5] This figure shows an example of an approach to encode and store representations for ABRs using the principles of this disclosure. [Figure 6] This figure shows an example of an approach to encode, store, present, and retrieve representations for ABRs using the principles of this disclosure. [Figure 7] This figure shows a high-level schematic diagram of an ABR process that implements extended coding according to the principles of this disclosure. [Figure 8] This figure shows a high-level approach to encapsulation as described in this disclosure. [Figure 9]This is an example of a process that should be executed on a client device according to the principles of this disclosure. [Modes for carrying out the invention]

[0056] We will begin by describing LCEVC, an extended coding scheme that is suitable for use in conjunction with the concepts of this disclosure in this context. LCEVC is described in the context of Figure 1. Throughout this description, the terms hierarchical coding and extended coding may be used interchangeably, and while examples are given in the context of LCEVC, it will be understood that the concepts described may be applicable to any similar hierarchical coding scheme or extended coding scheme.

[0057] LCEVC employs a multi-layer approach where any base codec (e.g., h.264, HEVC, AV1, etc.) is extended by an additional low-bitrate stream. The LCEVC data stream structure is defined by two component streams: a base stream that can be decoded by a hardware decoder, and an extension stream consisting of one or two extension layers suitable for software processing implementation with sustainable power consumption. This extension improves compression efficiency compared to existing codecs and reduces the complexity of encoding and decoding for on-demand and live streaming applications.

[0058] Figure 1 below illustrates how LCEVC operates on both the encoding and decoding pipelines. The base encoding (H.264, HEVC, or something else) is performed on the input, which has been downscaled to a lower resolution (typically one-quarter of the desired output resolution). The LCEVC augmentation data is computed at two resolutions, providing two levels of correction and augmentation. The LCEVC encoder generates an augmentation stream from the two inputs (the base encoding and the original uncompressed full-resolution video), effectively correcting the quality gap between the two. The LCEVC data can be packaged together with the base elementary stream (for example, as Auxiliary Augmentation Information (SEI) in the Network Abstraction Layer NAL), as frame metadata within a WebM container, or as frame metadata within an Additional Data Packet Identifier (PID) in an MPEG-2 TS stream.

[0059] The LCEVC decoder operates at the individual video frame level. It takes a low-resolution image decoded from a base video decoder (typically provided by a hardware decoder on the device) and a software-decoded LCEVC extension as input, forming a full-resolution image ready for rendering on the display view. Implementation examples for decoding LCEVC are described in WO2022 / 023739 and WO2023 / 118851, which are incorporated by reference.

[0060] As shown in Figure 1, the encoder 100 processes the input full-resolution video (i.e., source video 102) to generate various encodings. The first encoding (base encoding 110) is generated by supplying the base encoder 106 (e.g., AVC, HEVC, VP9, ​​or any other codec) with a downsampled version of the input video, which is produced by downsampling the input video 102 104. In the illustrated example, the downsampling is performed to one-quarter resolution, but this is optional and will be discussed in more detail elsewhere. The base encoding 110 can be referred to as the base layer.

[0061] A second encoding (level 1 encoding 112, an example of an extended encoding) is generated to create the first level correction 116. This is done by applying an encoding operation to the residual obtained by taking the difference between the reconstructed base codec video and the downsampled version of the input video. The reconstructed base codec video is obtained by decoding the output of the base encoder 106 with the base decoder. In a typical implementation, the level 1 encoding 112 is optional. This level 1 encoding 112 may be called the first extension layer.

[0062] A third encoding (level 2 encoding 114, an example of another extended encoding) is generated to create the first level correction 120. This is done by processing the residual obtained by taking the difference between the upsampled version of the corrected version of the reconstructed base-coded video (i.e., the normative upsampling 118) and the input video 102. This level 2 encoding 114 may be called the second extension layer.

[0063] The extension layer(s) and base layer are typically combined (i.e., as shown by mux122), and the full-resolution video is encoded in the layer. This is then typically transmitted using standard packaging and transmission protocols. An example of this will be discussed later in the context of Figure 2.

[0064] The corrections that form the extended layer often involve upsampling to a higher resolution, but generally, the extended layer can bring about some kind of quality improvement. For example, resolution, visual quality (VQ), bit depth (e.g., 8-10b), or color space (e.g., HDR).

[0065] In decoder 140, the encoded video is separated into an auxiliary data stream 144 and a video stream 146 (i.e., as shown by demux 142). The decoder receives the layers (base encoding, optional level 1 encoding and level 2 encoding) along with a header containing further decoding information. The base encoding (i.e., in video stream 146) is decoded by base decoder 148, which corresponds to the base decoder used in the encoder. In extension decoder 150 (which receives the header in auxiliary data stream 144 and the extension layer), its output is combined with the decoded residual obtained by decoding the level 1 encoding (if any). The combined video is upsampled and further combined with the decoded residual obtained by applying a decoding operation to the level 2 encoding to output the full-resolution video 152.

[0066] Figure 2 shows a schematic example of implementing extended coding using standard packaging and transmission protocols. The source video 102 is encoded by the encoder 100 to produce a layer-encoded full-resolution video. An example of how the encoder 100 can be implemented using an encoder integration layer is described in WO2022 / 023747, which is incorporated herein by reference. Any suitable container format can be used for the video, but here we show the mp4 container format, which includes a base layer and an extended layer.

[0067] The encoder is instructed to output a set of videos at different qualities (referred to here as bitrate variants) depending on the desired configuration. For example, a set of files might include videos at different resolutions (e.g., 1080p, 720p, etc.) and videos at the same resolution but at different qualities, frame rates, or bitrates (e.g., 720p, 60fps, and 720p, 30fps, etc.).

[0068] LCEVC is designed so that legacy devices can still decode and play the base video even when a stream with both the extension and base layers is received. In other words, LCEVC is inherently backward compatible because devices that have not yet been updated to decode LCEVC will still play the underlying base codec properly. However, for these legacy devices, in such situations they are not making full use of the available bandwidth because they are streaming data that is not being used.

[0069] To address this, and to support legacy devices, the encoder is typically instructed to output variants encoded using only the base codec, as well as video files of full-resolution video encoded in the layer. This allows legacy client devices to retrieve these files without the accompanying extension data, allowing more available bandwidth to be utilized for viewable video data.

[0070] A set of variants (i.e., a set of encodings or renditions) is packaged by a packaging service (referred to here as the packager 202). An example of a packager is the Shaka packager. The packager 202 divides each rendition into chunks of a specified duration (e.g., 2 or 4 seconds) and records how the video is divided and the order of the chunks in a Media Presentation Description File (MPD) (also known as a manifest file 205). The packaged video and manifest file are then placed in storage device 204. Once stored, the video chunks may be called segments, and the segments and manifest file become available for retrieval via the network 206.

[0071] A manifest file is typically an Extensible Markup Language (XML) document containing metadata requested by client devices to construct the appropriate Uniform Resource Locators (URLs) for accessing segments. A manifest is hierarchically structured. A media presentation contains a sequence of one or more time periods. Each time period contains one or more adaptive sets. Each adaptive set contains one or more representations. Each representation contains one or more segments. Each segment carries the location of the actual media data and associated metadata.

[0072] The media presentation (i.e., the overall hierarchy) includes information about all different media types within the content. The most common media types are video, audio, and closed captioning data. At the highest level, the MPD includes information (e.g., MPD profile, minimum buffer time, presentation time, and maximum segment time, and title). Video is structured into multiple time segments (e.g., for ad insertions or chapters). Adaptive sets are catalogs of available representations of media. For example, as shown in the schematic diagram in Figure 2, there may be adaptive sets for base video and adaptive sets for LCEVC video. There may also be adaptive sets for subtitle tracks and audio tracks, for example. Within each adaptive set, the player selects only one representation for a particular segment, but the representation can be changed per segment. Video adaptive sets typically contain multiple representations, one per resolution / bitrate, so that the media player can select the best available quality without buffering. If video is available in multiple codecs, each codec will have a different adaptive set. Within an adaptation set, a representation describes one version of the content. Each representation typically includes the average bandwidth of the representation, allowing the client device to select which representation to retrieve. Each representation includes a set of segments containing the information necessary to construct a URL for downloading the content. The MPD can provide a list of segment URLs or a template that the player uses to dynamically construct the URLs.

[0073] The structure of the MPEG DASH implementation is well understood by those skilled in the art, so this explanation is provided for context only. Further information can be found in the ISO / IEC 23009 standard document.

[0074] Figure 2 shows two client devices 208 and 210. To help illustrate the principles of the present invention, we assume that client device #1 208 may be an LCEVC-enabled device, and client device #2 210 may be a legacy device that cannot decode LCEVC video. When referring to the functionality of a client device, the functionality may be embodied in a player, decoder module, or any other module on the device, or the functionality may be divided into multiple modules (e.g., ABR handler, data handler, media handler, player, and decoder). Client devices 208 and 210 may first obtain (i.e., download) a manifest file 205 via network 206, for example, provided by a content delivery network (CDN). The manifest file is parsed by the client device to understand the mechanism for providing video. Client devices 208 and 210 detect their available bandwidth and select an available representation according to the available bandwidth and the representations supported by the device. Next, the client device requests a chunk from one of the video's bitrate variants, depending on its buffer size and available bandwidth. In other words, the client device selects a segment to download based on its available bandwidth and device status (e.g., supported video and buffer status). In the example in Figure 2, client device #1 208 supports LCEVC and has 2 Mbps available. Accordingly, it parses the manifest file to identify the most appropriate segment for download, which conforms to the b1e1 representation, constructs the appropriate URL from the metadata in the manifest file, and retrieves the segment to fill its buffer. The retrieved segment is passed to the LCEVC decoder for decoding. Client device 208 continuously monitors its bandwidth status and buffer level and adaptively streams the content.According to the ABR principle, for example, if the bandwidth state changes or the buffer level changes, the device can retrieve segments from different representations, for example, within the same adaptive set. To conclude this example, it was mentioned earlier that client device #2 210 does not support LCEVC. We assume it detects the same available bandwidth as client device #1 208. Therefore, since it does not support LCEVC, it selects from the AVC adaptive set and chooses to retrieve a segment from representation b2, which indicates an average bandwidth of 2 Mbps, and then constructs the appropriate URL from the metadata.

[0075] The representation within the adaptive set can be called an ABR ladder or ABR profile. Figure 3 shows a sample ABR profile from a report written by industry analyst Jan Ozer in April 2020. The H.264 ABR ladder profile is encoded using x264 (medium preset) and conforms to what Apple (RTM) recommends in the HLS authoring specification. The LCEVC H.264 ABR ladder profile is optimized to achieve both improved quality and a reduction in the overall average bitrate being streamed. As illustrated, the ABR ladder using LCEVC balances quality and efficiency. In other words, it can achieve lower bitrates for higher-level profiles at quality equivalent to or higher than x264, while improving the quality and resolution of lower-level profiles. To achieve maximum efficiency, it is possible to reduce the data rate as much as possible while matching x264 quality on all profiles, thereby lowering costs (e.g., transcoding, CDN) and maximizing network capacity (e.g., number of concurrent users).

[0076] Figure 4 shows alternative views of the conventional approach, with and without LCEVC. As illustrated, each resolution is encoded to require a set bitrate so that its encoded video stream can be properly obtained. As previously mentioned, the difficulty in implementing extended coding schemes lies in the fact that managing the encoder, packager, and decoder is far more complex than managing the same thing with conventional "single-layer" coding schemes. This is because the introduction of extended coding schemes (and the resulting extended coding layer) results in a much greater number of combinations and permutations available for the encoder, packager, and decoder.

[0077] While the use of representations works well for most mainstream setups, it is clear that it can negatively impact interoperability with less common or older setups. Since each representation specifies a particular bitrate and resolution, the manifest must provide representations for as many common setups as possible. However, because each representation is stored independently of the others, the amount of memory required increases exponentially with the number of representations.

[0078] It is not feasible to memorize representations for every possible permutation, and to maintain balance, uncommon or older setups may be excluded, thus forcing the system to support specific representations. Similarly, the player cannot change or modify the encoded representation; they must simply select from those already available.

[0079] The examples in this disclosure describe ways and systems in which streaming applications can benefit from the scalable capabilities of an extended coding scheme (e.g., MPEG-5 LCEVC). According to the principles of this disclosure, the base layer and the extended layer are stored and provided separately from each other. The representations are also presented separately from each other in the ABR ladder (i.e., manifest file). In an optional implementation, dependencies are introduced between representations across the adaptive set. These dependencies create associations between segments and frames in the base layer and the extended layer. In other words, the extended layer is associated with the base layer. The base layer is configured to extend using dependencies (specifically, in the examples, the dependencyId of MPEG DASH).

[0080] This concept, and the concept described herein, may be referred to as scalable mode. When used in scalable mode, the encoder is configured such that the base layer (corresponding to existing distribution methods (e.g., AVC, HEVC, AV1)) can be forced to correspond to existing standard profiles of the ABR ladder (e.g., 540p AVC, 1080p HEVC). Rate control mode may be selected to match the settings of an existing system (e.g., CBR, capped CRF, capped VBR, etc.) and ensure that the target bitrate is achieved for both the base layer and the base layer plus extension layer combined.

[0081] Next, a higher LCEVC extension layer can be used to efficiently compress higher resolutions (e.g., 1080p AVC, 2160p HEVC). Then, the extension chunks are combined with the base to produce higher resolution video in the decoder.

[0082] As an example, assuming target bitrate lanes of 2700kbps and 1900kbps, LCEVC can provide two profiles with a single encoding, as it encodes at 1080p (in 540p BL) or 720p (in 360p BL) (or other intermediate resolutions) in CBR targeting 2700kbps with a base ratio target of approximately 75%.

[0083] This mode can be used for HTTP delivery by LCEVC-compatible players that allow you to choose whether or not to add the LCEVC extension. Legacy players will only select and decode the base file. Legacy players do not need to download the extension file.

[0084] This document uses the following technical terms: BL = Base Layer (e.g., AVC, HEVC, VVC), and EL = Extension Layer (LCEVC).

[0085] Figure 5 shows a comparison with Figure 4, illustrating a potentially advantageous implementation of scalable mode. The illustrated implementation utilizes a base-to-extension split to present a condensed set of variants in an approach called LCEVC-driven multiscale. A first set of representations is optionally presented as an unaffected base profile (e.g., an unaffected AVC profile). In this example, video encoded using only the base codec is neither stored nor presented. Here, 360p and 540p profiles are shown. Instead, the encoder generates LCEVC-extended video, and the base layer from these LCEVC-extended videos is provided separately as a profile for lower bandwidths. In other words, the base layer of the LCEVC encoding process is provided as a separate representation in the ABR ladder.

[0086] In the specific example illustrated, further advantages are offered by encoding LCEVC video using different base settings. This makes it easier to increase the overall number of available representations without significantly increasing the number of videos encoded and stored. Higher resolutions can be encoded using different base settings, allowing more base representations to be presented without increasing the number of videos encoded and stored. In this example, four representations can be derived from two LCEVC encodings, each using a different base setting. In the specific example in Figure 5, the extension layer presents 720p and 1080p representations by adding a small bitrate on top of the 540p and 360p profiles. These reuse the bitrate of the base layer. Specifically, the base layer of approximately 1.9 Mbps (1.1 Mbps) for 1080p is reused as the 540p profile, and the base layer of the 720p LCEVC encoding is reused as the 360p representation. This allows four much smaller representations to be derived from two base layers and two extension layers. In the specific example shown in the comparison in Figures 4 and 5, the total bandwidth required to present all profiles is reduced by approximately 54%. Note that 720p is typically derived from 360p because it is the normative scaling mode in the LCEVC standard (i.e., x2 scaling change in the horizontal and vertical directions). In other words, two augmented encoded videos can be presented as four representations for ABR logic to move between them.

[0087] At this point, it is worth noting that ABR typically has rules regarding which representations should be considered, for example, based on codec support or maximum resolution. Client devices typically navigate the ladder in a predetermined pattern based on the manifest, with aspects such as startup speed considered to optimize the user experience. The configuration proposed herein facilitates improvements to these rules by enabling easy ladder traversal and enhancing the user experience.

[0088] The following pseudocode is an example of this implementation. trackSelectorParameters = new DefaultTrackSelector.ParametersBuilder() setMaxVideoSize(maxVideoWidth, maxVideoHeight) .setMaxVideoBitrate(maxVideoBitrate) .setForceHighestSupportedBitrate(false) .build();

[0089] Figure 6 shows how the scalable mode can be implemented at various points in the pipeline using the above implementation example.

[0090] The encoder 100 is configured to produce two LCEVC-enhanced encoded videos from the source video 102. In a specific example, this would include two fmp4 files, each having 2x tracks, 1x "acvc1" (i.e., video encoded using AVC), and 1x "lvc1" (i.e., an extension layer that provides LCEVC enhancement to each base layer, which is the respective AVC-enhanced video). In this example, these are a 720p LCEVC-enhanced video and a 1080p LCEVC-enhanced video. These videos are provided to the modified packager 602.

[0091] As described above, packager 602 curates the manifest file and presents each track of the two extended encoded videos as a separate, independent profile. The decoder (i.e., client device or player) then selects the appropriate representation according to the profile defined by the packager. In this example, LCEVC-enabled client device 608 downloads both the base and extension files and combines them to recreate the video (or, in an optional implementation, combines them to create a representation). Legacy client device 210 (decoder or player) downloads only the base segment, avoiding the need to download the extension layer which they cannot process (or, avoiding the need for the packager to present two representations, including the base, to both legacy and LCEVC-enabled devices). Legacy means non-LCEVC-enabled.

[0092] Figure 7 illustrates this concept using the pipeline from Figure 2. As described here, delivering an LCEVC-encoded stream in scalable mode can be done using an MP4 ISO BMFF carriage and MPEG-DASH delivery. Other mechanisms are conceivable within the scope of this disclosure, but for simplicity, we present an example using MPEG-DASH. In the example, modifications may be made to the encoder, packager, and decoder, and they may be adapted to the delivery method.

[0093] As described above, the encoder 100 is used to encode the source video 102, and multiple variants of the encoded source video are created using the extended codec. Here, we describe encoding video in an mp4 file using LCEVC. Depending on the implementation of the LCEVC encoder, modifications may or may not be necessary. The key is that the extended layer and the base layer can be separated. Therefore, in some embodiments, the encoder may be configured to output an encoded version of the source video (encoded using an extended codec that includes separate base and extended tracks).

[0094] The LCEVC extension layer can be encoded "in the loop" together with the base layer, downscaling the image before base encoding, upscaling the reconstructed image to enable the calculation of EL residual information, and then encoding and transmitting. The LCEVC encoder generates LCEVC NALU according to the standard (ISO / IEC 23094-2). The LCEVC NAL unit type can have the following meanings: [Table 1]

[0095] For all IDR images in the base dataset, the corresponding LCEVC images are also expected to be IDRs.

[0096] Alternative approaches can be considered to facilitate the synchronization of segments (and frames) between separately transmitted augmented layer data and base layer data. For example, instantaneous decoder refresh images of the base layer and augmented layer should ideally be synchronized to facilitate the combination of residuals in the augmented data. Techniques also exist for known multilayer codecs (e.g., SVC and SHVC).

[0097] Furthermore, frames between the base layer video and the extended layer video can be synchronized by the technique described in WO2023 / 118851, which is incorporated herein by reference. This technique relates to the implementation of HTML elements where the player may not have access to information in the base layer. It describes how a packetized elementary stream (PES) can be constructed from multiple transport stream (TS) packets. A presentation timestamp (PTS) is provided to the packets for the base stream. The extended stream uses the presentation timestamp within the extended stream to synchronize the frames of the two streams during HTML reconstruction. In other words, frames in the extended stream are indexed according to the PTS, which is compared to timing metadata (e.g., the “media time” variable or current playback time provided to the base layer) to pair the frames in the extended layer with those in the base layer. In other words, in HTML5, the player may be provided with RGB data and timing metadata (but not timestamp information), which can be used to retrieve the timestamp information and combine it with the extended data as needed.

[0098] Alternative approaches include using SEI messages to convey timing information and then using that to synchronize the extension and the base.

[0099] This example demonstrates encapsulating LCEVC within the MP4 file format to preserve timing information and ensure synchronization. Figure 8 summarizes the process of encapsulating the LCEVC bitstream within the MP4 file format.

[0100] The two main concepts used to link the LCEVC extension to its corresponding XVC base (where X means "whatever the coding format", e.g., AVC, HEVC, EVC, VVC) are as follows: (1) The LCEVC extension is linked to the XVC base by the "sbas" box which identifies the corresponding base. (2) Use the same time base used in the base in the "stts" box within the extension to synchronize the images.

[0101] < <sbas>>Figure 8 shows the process in which LCEVC NALU802 is linked to base NALU804 using a box. LCEVC NALU802 is created from base sample 808 made from base NALU <stts>Same as sampling time< <stts>These samples can be transformed into LCEVC samples with boxes. Then, these samples can be combined to create LCEVC image 810.

[0102] An example encoder implementation could be configured to return two separate outputs (one for BL NALU and one for EL NALU), with the necessary synchronization between BL and EL. The encoder muxer then packages these into a carriage mechanism, which can produce either a single mp4 file (BL and EL on two separate tracks) or two files (one for BL and one for EL, both on a single track).

[0103] In the implementation example, the former, i.e., a single mp4 (with BL and EL on separate tracks), can be generated. The packager can then be left to separate the tracks into different, independent chunks.

[0104] Returning to Figure 7, the modified packager 702 is configured to separate the encoded variant into a base layer and an extension layer, retrieve them as a set of segments, and construct a manifest file containing metadata suitable for constructing URLs to retrieve those segments. As usual, this metadata may include modified URLs or template URLs. These should be combined with information for each segment to construct URLs to retrieve that segment. The separated segments are then stored in the storage device 204 for later retrieval, as usual. Each profile created by the packager may also include a suitable display of the average bandwidth and other available metadata that can be used by the ABR handler or ABR rule to identify and retrieve the appropriate segments.

[0105] In the implementation, the packager generates segments (some containing only BL, and others containing EL as a single track representation) that are represented by separate MP4 files (or a single file with a byte range). The BL and EL are linked in the elementary stream by the encoder using the method described above. The packager optionally provides a way for the player to associate the BL MP4 chunks with the EL MP4 chunks. This can be done at the manifest level. In the implementation described here, this association uses dependency attributes in the manifest file.

[0106] To enable smooth playback, the packager may signal chunks within the BL and EL to have the same start and end times / frames. In some implementations, this means the packager must recognize the LCEVC IDR NALU (nal_unit_type=29) and cut out the segments corresponding to the equivalent AVC / HEVC / VVC IDR.

[0107] IDR frame matching is preferably ensured by the encoder, as described above.

[0108] As described elsewhere in this document, dependency attributes are one possible implementation used to create an association between an EL segment and the BL segment it extends. Dependency attributes can be set by the packager 702 in manifest file 705. In the example shown in Figure 7, this is shown as "dID" and links the representation in the adaptation set for the extension layer to the representation in the adaptation set for the base layer. The bandwidth for the extension adaptation set may represent the bandwidth of the combined representation of the BL and EL, or, optionally, may represent only the bandwidth of the extension layer, allowing the client device to combine it with the base bandwidth requirement to determine the total bandwidth. The latter may be preferable in some cases, but this may depend on the standards and how they are designed.

[0109] The simultaneous use of dependency ID attributes for adaptive sets has been done only within a single adaptive set, for purposes such as HDR. To date, there have been no proposed or documented usages or mechanisms for combining two video adaptive sets. While modern players understand how to combine retried segments from multiple adaptive sets, such as video and audio or subtitles, players are not adapted to combinations of two adaptive sets that include video representations.

[0110] In one embodiment, the adaptive set, including the extension layer, may be labeled as data rather than video to facilitate acquisition and combination by player and client devices.

[0111] This disclosure presents a concept of separating the base layer and extension layer for storage and presentation, and provides a mechanism that allows ABR logic to move between representations of an encoded video stream using only the combined base and extension layers, and the base-encoded video. Linking extension layer segments to their respective base layer segments using dependency ID attributes is just one possible preferred implementation.

[0112] In this typical scenario illustrated, we assume a single set of adaptations, but with distinct representations. Naturally, other sets of adaptations can also be included in MPD, but for the sake of explanation, there are multiple representations within each adaptation set for each MIME type. Each representation refers to an MP4 segment containing a single track; for BL, only legacy bitstreams (e.g., AVC), and for EL, only extensions encoded in LCEVC. The link from EL to BL is external, from EL MP4 to BL MP4, provided using the @DependencyID attribute in MPEG-DASH.

[0113] This mechanism is described in DASH-IF Interoperability-Guidelines for Implementations, Annex G: Sample Dual-Layer MPD. An example reported in this guideline is shown below. <period> <!--ビデオ--> <adaptationset mimetype="video / mp4" subsegmentalignment="true" subsegmentstartswithsap="1" framerate="24"> <representation id="base-layer" codecs="hev1.2.4.H153.b0" bandwidth="29200362" width="3840" height="2160"> <baseurl> BL#DashIf#BL-HDR10#UHD.mp4< / baseurl> <segmentbase indexrange="687-862"> <initialization range="0-686" / > < / segmentbase> < / representation> <representation id="enhancement-layer" dependencyid="base-layer" codecs="dvhe.07.03" bandwidth="9320458" width="1920" height="1080"> <baseurl> EL#DashIf#BL-HDR10#UHD.mp4< / baseurl> <segmentbase indexrange="723-898"> <initialization range="0-722" / > < / segmentbase> < / representation> < / adaptationset> < / period>

[0114] In the case of the LCEVC extension, the only attribute that changes is 4cc for the codec, which is "lvc1" in the case of LCEVC. A sample manifest for LCEVC is written as follows: <?xml version="1.0" encoding="utf-8"?> <MPD xmlns:xsi=http: / / www.w3.org / 2001 / XMLSchema-instance xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xlink=http: / / www.w3.org / 1999 / xlink xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 http: / / standards.iso.org / ittf / PubliclyAvailableStandards / MPEG-DASH#schema#files / DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" type="static" mediaPresentationDuration="PT10.0S" maxSegmentDuration="PT5.0S" minBufferTime="PT20.0S"> <programinformation> < / programinformation> <servicedescription id="0"> < / servicedescription> <--!During this period, the LCEVC+ base stream is split into two representations: BL and EL--> <!--In this example, four representations are used to inform the player of the existence of a 4x profile, two of which are derived from the combination of base and extension--> <!--The link mechanism is the dependencyID--> <period id="0" start="PT0.0S"> <adaptationset id="0" contenttype="video" startwithsap="1" segmentalignment="true" bitstreamswitching="true " framerate="25 / 1" maxwidth="1920" maxheight="1080" par="16:9"> <!--Independent representation (BL) of Profile 0: 360p--> <representation id="0" mimetype="video / mp4" codecs="avc1.64001f" bandwidth="1100000" width="640" height="360" sar="1:1"> <segmenttemplate timescale="12800" initialization="init-stream$RepresentationID$.m4s" media="chunk-stream$RepresentationID$-$Number%05d$.m4s" startnumber="1"> <segmenttimeline> <s t="1024" d="128000">< / s> <s d="1024">< / s> < / segmenttimeline> < / segmenttemplate> < / representation> <!--Independent representation (BL) of Profile 1: 540p--> <representation id="1" mimetype="video / mp4" codecs="avc1.64001f" bandwidth="1400000" width="960" height="540" sar="1:1"> <segmenttemplate timescale="12800" initialization="init-stream$RepresentationID$.m4s" media="chunk-stream$RepresentationID$-$Number%05d$.m4s" startnumber="1"> <segmenttimeline> <s t="1024" d="128000">< / s> <s d="1024">< / s> < / segmenttimeline> < / segmenttemplate> < / representation> <!--Dependent representation (EL) of Profile 2: Generates 720p in combination with p0--> <representation id="2" mimetype="video / mp4" codecs="lvc1" bandwidth="800000" width="1280" height="720" sar="1:1" dependencyid="0"> <segmenttemplate timescale="12800" initialization="init-stream$RepresentationID$.m4s" media="chunk-stream$RepresentationID$-$Number%05d$.m4s" startnumber="1"> <segmenttimeline> <s t="1024" d="129024">< / s> < / segmenttimeline> < / segmenttemplate> < / representation> <!--Dependent representation (EL) of Profile 3: Generates 1080p in combination with p1--> <representation id="3" mimetype="video / mp4" codecs="lvc1" bandwidth="900000" width="1920" height="1080" sar="1:1" dependencyid="1"> <segmenttemplate timescale="12800" initialization="init-stream$RepresentationID$.m4s" media="chunk-stream$RepresentationID$-$Number%05d$.m4s" startnumber="1"> <segmenttimeline> <s t="1024" d="129024">< / s> < / segmenttimeline> < / segmenttemplate> < / representation> < / adaptationset> < / period>

[0115] The concept of dependent representation (in the case of LCEVC EL, which depends on its corresponding BL) is introduced in section 5.3.5.1 of 23009-1.

[0116] The following is a quote from section 5 of 23009-1.

[0117] "A dependent representation is described by a representation element that includes the @dependencyId attribute. A dependent representation is a normal representation, except that it depends on a set of complementary representations for decoding and / or presentation. @dependencyId includes the values of the @id attributes of all complementary representations, i.e., the representations necessary to present and / or decode the media content components included in this dependent representation."

[0118] @codingDependency is defined as a common attribute for adaptation sets and representations in sections 5.3.7.2 (semantics) and 5.3.7.3 (syntax) of 23009-1.

[0119] Further relevant information can be found in 230090 - Dynamic Adaptive Streaming over HTTP (DASH), Part 3: Implementation Guidelines, Section 10. Mapping of ISOBMFF Track Reference Types to DASH MPD. The following is a quote from section 10 of 23009-3.

[0120] "Dependencies (see the definition of "dependency expression" in Section 5.3.5.1 of DASH Part 1):" For a given track reference type (a row in the table), if the "Mapping to DASH" column in the table indicates a "Dependency," it means that the expression @dependencyId should be set from the expression corresponding to the track declaring the track reference of the given type to the expression(s) corresponding to the track(s) with those track_IDs listed in the "tref" box. The list of expression IDs in the @dependencyId attribute must match the order of the track_IDs from the "tref" box.

[0121] In further possible implementation details, with respect to MIME types for the base layer and extension layer, the base track has sample description boxes corresponding to the base codec (e.g., "avc1", "hvc1", "evs1", "vvc1", etc.), while the latter consistently has its sample description box (e.g., "lvc1").

[0122] As mentioned above, the link between the EL and its corresponding BL can be achieved by a "sbas" box. Note that in other embodiments described herein, this example is not necessary because the current time from the media element is used for comparison with the extension's pts.

[0123] Returning to Figure 7, in the known method, to obtain a video stream, the client device first obtains a manifest file. In the known method, legacy device 210 (i.e., a non-LCEVC-enabled device) obtains and parses the manifest file and obtains a representation according to the set of adaptations they support. This is done by constructing a URL to the video encoded in the base layer and downloading segments of the video to fill their available buffers according to the current state. Where we refer to a client device here, we may similarly refer to a player or decoder.

[0124] A client device modified according to the principles of the present invention may obtain a manifest file, parse the manifest file to identify the extension layer representation, and identify the associated base layer representation from the dependency attributes. In the first example, the client device determines the appropriate representation from the extension adaptation set and obtains the associated base layer segment as needed. In the second example, the client device may construct a set of virtual representations from the manifest file, the virtual representations representing the combination of base and extension layers indicated by the dependency attributes.

[0125] When referring to a client device here, it can similarly refer to a player or decoder. An example of a player that should be modified is dash.js.

[0126] The client device obtains segments from the base layer and extension layer, passes them to the appropriate extension decoder for decoding, combines them, and renders them on the client device.

[0127] In the example process, the client device (referred to here as decoder 708) needs to parse the manifest file and select from a set of renditions based on the available bandwidth. The player (i.e., the player on the client device, or decoder, e.g., dash.js) is expected to construct a combined virtual set of representations (where the bandwidth of the extended profile is equal to the sum of EL and BL). The example process may include: 1. The DASH manifest parser needs to parse the @DependencyID attribute to set up the dependencies between the BL and EL representations. 2. The DASH ABR engine may maintain a list of its representations by considering BL as a representation and EL as another representation along with its dependency BL. In the manifest examples in Figures 5 and 6 above, the DASH ABR engine maintains four representations: a. Expression 1, 360p BL only, bandwidth = 1,100,000. b. Representation 2, 540p BL only, bandwidth = 1,400,000. c. Representation 3, 720p EL, its dependency 360p BL, bandwidth = 1,900,000 (this is the sum of the 720p EL bandwidth (representation id2) and the 360p BL bandwidth (representation id0)). d. Representation 4, 1080p EL, its dependency 720p BL, bandwidth = 2,300,000 (this is the sum of the 1080p EL bandwidth (representation id3) and the 540p BL bandwidth (representation id1)). 3. When the DASH ABR engine detects sufficient bandwidth and codec capability (i.e., LCEVC support) to select a bitrate that matches the bandwidth for the representation including EL, the following occurs: a. The DASH downloader can download BL segments and their associated EL segments. The downloader can ensure that the BL and EL segments are downloaded in asynchronous mode whenever possible, so that all necessary EL and BL data are available for decoding the complete image. In other words, the segments are downloaded in batches, not synchronously (meaning one by one). A synchronous mode is also possible. b. The playback pipeline may decode the complete image by supplying BL data to the base decoder and EL data to the LCEVC decoder via an available interface (e.g., mediacodec in Android®). Further details may be provided in WO2022 / 023739, incorporated by reference, which describes the Decoder Integration Layer (DIL). 4. If the ABR engine detects bandwidth or codec capability that matches only the BL stream, it may download only the BL data and feed it to the base decoder, as is currently done in modern processes.

[0128] A simplified example of this process 900 from the perspective of a client device is shown in Figure 9. In step 902, the modified manifest file is retrieved. The manifest file is then parsed in step 904. Based on the available conditions, the representation for retrieval is identified in step 906, and dependencies are identified from the dependency attributes in step 908. In step 910, the base layer segment is downloaded, and its associated extension layer segments are downloaded based on the dependencies. In step 912, these are passed to the decoder for decoding and reconstruction of the extension video.

[0129] As previously stated, the concepts described herein may be implemented in client devices, players on devices, or decoders. Similarly, the concepts may be embodied by modifications to encoders, packagers, and / or content distribution networks. In each of these entities, the methods and processes described herein may be embodied as code (e.g., software code) and / or data. This functionality may be implemented in hardware or software, as is well known in the technical fields of data compression and video streaming. For example, a certain degree of efficiency may be achieved through hardware acceleration using a specially programmed graphical processing unit (GPU) or a specially designed field-programmable gate array (FPGA). For completeness, such code and data may be stored on one or more computer-readable media. These media may include any devices or media capable of storing code and / or data for use by a computer system. When a computer system reads and executes code and / or data stored on a computer-readable media, the computer system executes the methods and processes embodied as data structures and code stored in the computer-readable storage medium. In some embodiments, one or more steps of the methods and processes described herein can be performed by a processor (for example, a processor in a computer system or a data storage system).

[0130] In general, any functionality described in this document or shown in the figures may be implemented using software, firmware (e.g., fixed logic circuits), programmable or non-programmable hardware, or a combination of these implementations. The terms “component” or “function,” as used herein, generally refer to software, firmware, hardware, or a combination thereof. For example, in the case of a software implementation, the terms “component” or “function” may refer to a processing device or program code that performs a specified task when executed on the device. The exemplary separation of components and functions into separate units may reflect any actual or conceptual physical grouping and assignment of such software and / or hardware to tasks.< / stts> < / stts> < / sbas>

Claims

1. A method for streaming multimedia content encoded using an extended coding scheme, wherein the method is: Obtaining a manifest file, wherein the manifest file includes metadata for constructing locators to segments of the encoded multimedia content, and the metadata is A first representation referring to a first version of the multimedia content encoded using a base coding scheme, Acquisition includes, the location metadata for the first extension layer of the encoded multimedia content associated with the first version, The process involves parsing the manifest file and constructing a locator for obtaining the first representation and the segment of the extension layer, A method comprising downloading the first representation segment and the extension layer segment and providing them for reconstruction of the encoded multimedia content.

2. The method according to claim 1, wherein the metadata includes a second representation that points to the first extension layer of the encoded multimedia content.

3. The method according to claim 2, wherein the metadata further includes dependency attributes that associate the second representation with the first representation.

4. The method according to claim 2 or 3, wherein the first and second expressions are included in their respective adaptation sets.

5. The metadata further includes, for each representation, bandwidth for obtaining each representation, and the method The method according to any one of claims 2 to 4, further comprising determining the total bandwidth for streaming the multimedia content by summing the bandwidth associated with the second representation and the bandwidth of each associated first representation.

6. The first representation includes an attribute indicating the base coding scheme, The method according to claims 2 to 5, wherein the second expression includes an attribute indicating the extended coding scheme.

7. The aforementioned metadata is A third representation that refers to a second version of the multimedia content encoded using the base coding scheme, wherein the first and second versions are versions of the multimedia content at different quality levels, The method according to any one of claims 2 to 6, further comprising: a fourth representation referring to a second extension layer associated with the second version of the encoded multimedia content.

8. The aforementioned method, Further comprising constructing a set of virtual representations from the first and second representations and the dependent attributes, the virtual representations are A first virtual representation including the first representation, The method according to any one of claims 3 to 7, comprising a second virtual representation including the first and second representations.

9. Detecting available bandwidth and / or device conditions, The method according to any prior claim, further comprising downloading a segment of the first representation, or a segment of both the first representation and the extension layer, and providing it for reconstruction of the encoded multimedia content according to the available conditions.

10. The downloaded segment of the aforementioned extension layer is passed to the extension decoder to generate decoded extension data, The method according to any prior claim, comprising: reconstructing multimedia content from the decoded extended data and a decoded version of the video stream encoded using the base coding scheme and decoded from the downloaded segment of the first representation.

11. A method for packaging multimedia content encoded using an extended coding scheme for streaming, wherein the method is: The process involves constructing a manifest file, the manifest file including metadata for constructing locators to segments of the encoded multimedia content, and the metadata includes: A first representation referring to a first version of the multimedia content encoded using a base coding scheme, A method including constructing a first version of the encoded multimedia content, including location metadata for a first extension layer of the encoded multimedia content associated with the first version.

12. The method according to claim 11, wherein the metadata includes a second representation that refers to the first extension layer of the encoded multimedia content.

13. The method according to claim 12, wherein the metadata further includes dependency attributes that associate the second representation with the first representation.

14. The method according to claim 12 or 13, wherein the first and second expressions are included in their respective adaptation sets.

15. The metadata further includes, for each representation, the bandwidth required to retrieve each representation, and the method further, The method according to any one of claims 12 to 14, further comprising determining the total bandwidth for streaming the multimedia content by summing the bandwidth associated with the second representation and the bandwidth of each associated first representation.

16. The first representation has a type indicating the base coding scheme, The method according to claims 12 to 15, wherein the second expression has the type of indicating the extended coding scheme.

17. The aforementioned metadata is A third representation that refers to a second version of the multimedia content encoded using the base coding scheme, wherein the first and second versions are versions of the multimedia content at different quality levels, The method according to any one of claims 12 to 16, further comprising: a fourth representation referring to a second extension layer associated with the second version of the encoded multimedia content.

18. Receiving an encoded version of the multimedia content encoded using the extended coding scheme which includes a base layer and an extension layer, To generate a first set of segments that represent the base layer, The method according to any one of claims 11 to 17, further comprising generating a second set of segments representing the extension layer, wherein the first set of segments is independent of the second set of segments.

19. The method according to any one of claims 11 to 17, wherein the segments of the first set and the segments of the second set are generated as single track representations, respectively.

20. Constructing the aforementioned manifest file means Inserting an association between the segments of the second set and the segments of the first set, The method according to any one of claims 11 to 19, comprising obtaining the locations of the first and second sets of segments, and inserting metadata for constructing a locator at the locations of the first and second sets of segments.

21. A client device for streaming multimedia content encoded using an extended coding scheme, configured to perform the method described in any one of claims 1 to 10.

22. A packager for packaging multimedia content encoded using an extended coding scheme for streaming, configured to perform the method according to any one of claims 1 to 20.

23. A system comprising the client device described in claim 21 and the packager described in claim 22.

24. It is a system, An encoder configured to encode the multimedia content using an extended coding scheme, The system according to claim 23, further comprising a content distribution network configured to provide the segments and the manifest file.

25. A computer-readable medium claim that, when executed by a computer, includes instructions causing the computer to perform the method according to any one of claims 1 to 10 or 11 to 20.

Images