Method for decoding video stream

Abstract

A method for decoding a video stream is described. The method includes: receiving a video stream at a server application, the video stream being encoded according to a multi-layer coding scheme; transmitting an encoded base stream contained in the video stream to a base decoder to obtain a decoded base stream; and delivering an encoded enhancement stream included in the video stream to an enhancement decoder for decoding, wherein an original input video can be reconstructed from the decoded base stream and the decoded enhancement stream.

Description

Technical Field

[0001] The present invention relates to a method, a system, a computer program and a computer-readable medium for decoding a video stream. Background Art

[0002] The encoding and decoding of video content is a consideration in many known systems. Video content can be encoded for transmission, for example, over a data communication network. When decoding such video content, it may be necessary to increase the quality level of the video and / or recover as much information as possible contained in the original video. Many video decoding formats and their associated codecs have been developed that attempt to achieve these desired characteristics, but often require significant software updates and / or hardware upgrades at the operating system level. In addition, in order to increase the quality of the decoded video content, it is usually necessary to increase the complexity of the encoding and decoding procedures, which may increase power usage and increase the delay in delivering the video content.

[0003] Hybrid backward-compatible decoding techniques have been previously proposed in, for example, WO 2014 / 170819 A1 and WO 2018 / 046940 A1, the contents of which are incorporated herein by reference. Another example of a layer-based decoding format includes ISO / IEC MPEG-5 Part 2 LCEVC (hereinafter referred to as "LCEVC"). LCEVC is described in WO 2020 / 188273 A1 and associated standard specification documents, including the ISO / IEC DIS 23094-2 Low Complexity Enhancement Video Coding draft published at the MPEG 129 meeting held in Brussels from Monday, January 13, 2020 to Friday, January 17, 2020, both of which are incorporated herein by reference in their entirety.

[0004] In these coding formats, the signal is broken down into multiple "tiers" of data (also called "hierarchical layers"), each corresponding to a "quality level," from the highest tier to the lowest tier at the sampling rate of the original signal. The lowest tier is typically a low-quality reproduction of the original signal, and the other tiers contain information about the corrections applied to the reconstructed reproduction in order to produce the final output.

[0005] LCEVC adopts this multi-layered approach, where any base codec, such as Advanced Video Coding (AVC, also known as H.264) or High Efficiency Video Coding (HEVC, also known as H.265), can be enhanced via an additional low-bitrate stream. LCEVC is defined by two component streams: a base stream, which is typically decodable by a hardware decoder, and an enhancement stream, which consists of one or more enhancement layers suitable for software processing implementation with sustainable power consumption. The enhancements provide improved compression efficiency over existing codecs and reduce encoding and decoding complexity.

[0006] Since LCEVC and similar coding formats utilize existing decoders and are inherently backward compatible, there is a need for efficient and effective integration with existing video coding implementations without requiring a complete redesign. Examples of known video coding implementations include the software tool FFmpeg, which is used by the simple media player FFplay.

[0007] LCEVC is not limited to known codecs and can theoretically utilize codecs that have not yet been developed. Thus, any LCEVC implementation should be able to integrate with any hitherto known or yet to be developed codec implemented in hardware or software without introducing decoding complexity.

[0008] Red5 Pro, August 18, 2020, "9 Reasons Why LCEVC Circumvents the Codec Wars." Red5 Pro (https: / / www.red5pro.com / blog / 9-reasons-why-lcevc-circumvents-the-codec-wars) disclosed live streaming involving the Low Complexity Enhancement Video Coding (LCEVC) codec. Jan Ozer, March 12, 2020, "How to Encode with LCEVC." Streaming Media (https: / / www.streamingmedia.com / Articles / ReadArticle.aspx?ArticleID=139705) disclosed encoding with LCEVC. ITU Journal: ICT Discoveries, Volume 3(1), June 8, 2020; Florian Maurer et al.; “MPEG-5 Part 2 — Overview of Low Complexity Enhancement Video Coding (LCEVC)” (https: / / www.itu.int / pub / S-JOURNAL-ICTS.V3I1-2020-12) discloses the outline of the LCEVC encoder and decoder coding architecture, tools, and an overview of the performance of LCEVC.

[0009] It may be desirable to integrate LCEVC implementations with existing frameworks for handling and decoding video streams, such as those widely used in operating systems such as iOS (RTM). What is needed is a method to use a multi-layer decoding structure such as LCEVC in conjunction with an operating system and its native base decoder in order to achieve the aforementioned efficiency gains without requiring additional hardware and with minimal necessary changes to the software. Summary of the Invention

[0010] Various aspects and variations of the present invention are set forth in the appended claims. Certain non-claimed aspects are further set forth in the following detailed description.

[0011] According to one aspect, there is provided a method comprising:

[0012] receiving, at a server application, a video stream, the video stream being encoded according to a multi-layer coding scheme;

[0013] Passing the encoded elementary stream included in the video stream to an elementary decoder to obtain a decoded elementary stream; and

[0014] The encoded enhancement stream included in the video stream is passed to an enhancement decoder for decoding, wherein the original input video can be reconstructed from the decoded base stream and the decoded enhancement stream.

[0015] The present inventors have recognized that a server application can be advantageously used to handle encoded video streams, particularly video streams encoded according to a multi-layer scheme. For example, instead of a native player or decoder forming part of an existing framework within a conventional operating system, such a server application is employed to receive an incoming video stream at the application and pass the encoded base component of the stream to the decoder while separately handling the enhancement stream, enabling efficient video decoding methods to be integrated with a device and its operating system through a straightforward integration process.

[0016] The video stream can be considered as input data. The server application typically runs on the same terminal device as the base decoder and / or decoder integration layer (DIL). A video stream encoded according to a multi-layer decoding scheme can be considered as a video stream comprising a coded base stream and a coded enhancement stream.

[0017] The method may involve delivering the encoded elementary stream directly or indirectly to the basic decoder. The encoded elementary stream may be considered as the video stream or second input data included in the input data. Delivering the encoded elementary stream to the basic decoder preferably involves instructing the basic decoder, via a server application or other means, to obtain a decoded elementary stream from the encoded elementary stream included in the video stream. The encoded elementary stream delivered or transmitted to the basic decoder is typically suitable or adapted to be used to obtain a decoded elementary stream therefrom.

[0018] Similarly, the coded enhancement stream can also be delivered directly or indirectly to the enhancement decoder. The method generally comprises obtaining a decoded enhancement stream from the video stream and / or obtaining a coded base stream from the video stream. These streams are preferably obtained by a server application.

[0019] Passing the encoded enhancement stream to the enhancement decoder preferably involves instructing a decoder integration layer (DIL) to decode the encoded enhancement stream contained in the video stream. This preferably further includes instructing the DIL to reconstruct the original input video. Although instructions to either or both the base decoder and the DIL can be sent by a suitably configured device or an application instance running thereon, in some embodiments, the instructions or a subset of those instructions can be sent directly or indirectly from a server application.

[0020] It will be appreciated that in preferred embodiments the decoded base stream is typically the base stream decoded by the base decoder and the decoded enhancement stream is the enhancement stream decoded by the DIL.

[0021] The method can be advantageously applied to devices using the iOS operating system. The ability to integrate and handle multiple layers of encoded streams on such devices is a key objective of some embodiments. Therefore, in some embodiments, the base decoder is a native decoder of the iOS operating system. Preferably, as described above, the encoded enhancement stream is passed to the DIL. In some embodiments, the decoder integration layer controls the operation of the enhancement decoder accordingly.

[0022] Typically, the method further includes receiving, by the server application, a request for a data asset sent by the base decoder. This reception can be considered as intercepting the request. However, typically, the request from the base decoder has been modified so that it is sent to the application server, rather than being sent directly to the data asset source from which the request was received. This modification is typically performed by the application or module that includes the base decoder. In such embodiments, the method typically further includes sending the intercepted request to such data asset source, and receiving, by the server application, a data asset including a video stream in response to the request. The data asset can be referred to as a video asset. The data asset can also be referred to as video data or a video stream. The request for the data asset is typically sent based on or in response to instructions from the client application. In some embodiments, the request can be sent by a module that includes the base decoder or communicates with the base decoder. Typically, the module or application is composed of the Apple AVFoundation framework and typically includes an AVPlayer instance. The method disclosed herein is suitable for use with AVPlayer and helps devices that utilize AVPlayer and its specific modified versions to efficiently handle enhanced video.

[0023] The data asset source is typically an external server and may be referred to as a source server. Typically, a data asset source, such as a video on demand server, returns a response to an entity or network address from which a request for the asset or data contained in the response is received. However, in some embodiments, the method further comprises modifying the request before the server application sends the request so as to indicate the server application as the recipient of the response to the request. Generally speaking, the server application operates on a device, such as an iOS device, on which an instance of a base decoder (which may include an AVPlayer instance) runs. That is, the server application is not typically executed on a device that is remote from the decoder. Preferably, the server application and the base decoder and preferably the decoder integration layer execute on the same device, which is preferably an iOS device.

[0024] In a preferred embodiment, the base decoder is part of a framework or application that supports adaptive bitrate (ABR) video streaming. For example, it will be understood that streaming protocols such as the Apple HTTP Live Streaming (HLS) media streaming communication protocol are implemented as part of various operating systems. Traditionally, video handling configurations (such as those implemented in the AVFoundation framework for video streams such as video on demand and live content) support ABR protocols such as HLS. For example, existing AVPlayer implementations are suitable for receiving a video asset stream that is divided into multiple video chunks, typically MPEG2-TS files with different bit rates and predetermined durations. Typically, an index file is provided that serves as a playlist file for those received video segments. Generally speaking, each playlist corresponds to a corresponding bitrate level and contains location information or URLs of video segments with relevant bitrate information. Client devices and applications that utilize AVPlayer are typically configured to request the appropriate playlist based on the available bandwidth. In a preferred embodiment of the method, for example, when bandwidth changes or other factors cause ABR changes, an appropriate bitrate level is selected from the available profiles in a manner that ensures that both the base stream and the enhanced stream are decoded using the appropriate profile, and the appropriate bitrate level can be considered to correspond to the bitrate profile. In this way, the enhanced decoder (which can be an enhanced iOS or AVPlayer compatible decoder) is able to select and decode the appropriate video segment while the native AVPlayer decoder or other basic decoder is changing the bitrate profile. Therefore, in a preferred embodiment, the method further includes instructing the basic decoder to obtain the decoded base stream according to the predetermined bitrate parameters set according to the encoded base stream. As described above, multiple buffer queues are usually created when receiving video data, and these queues correspond to different video bit rates. These queues can be used for decoding, and decoding is usually performed on the appropriate queue by selecting the appropriate bitrate profile. This can be understood as a profile corresponding to the current bitrate, and if the bitrate changes, it is the next active queue after the change.

[0025] It will be appreciated that the method may additionally or alternatively comprise instructing the enhancement decoder to decode the enhancement stream according to a predetermined bitrate profile. A bitrate parameter set may comprise one or more parameters. The parameter set may be considered a bitrate profile or an indication thereof. It may be considered to represent the bitrate of the encoded video stream. The bitrate parameter set may specifically correspond to a current frame and / or one or more subsequent frames of the encoded base stream.

[0026] According to such methods, the method may further comprise detecting a change in a predetermined set of bitrate parameters corresponding to a change in the video bitrate. It will be appreciated that an ABR profile may be modified during the reception or playback of a video stream. Identifying a parameter set or the need to change to a different parameter set may comprise directly identifying a buffer queue corresponding to a particular ABR profile, and / or may comprise identifying a profile parameter set from which an appropriate bitrate for a video segment may be selected or inferred. The change is typically an imminent change. This may occur for any current or subsequent frame, or concurrently with any current or subsequent frame.

[0027] Detecting changes preferably includes detecting a notification of a bitrate parameter set change. As described in more detail later in this disclosure, such notifications are typically generated by a native decoder of the iOS operating system, particularly an AVPlayer instance. In some embodiments, the notification includes notification data recorded in a log, and detecting the notification includes monitoring or analyzing the data recorded in the log. The log is typically saved in a log file, or in a storage unit or resource that records log data. Typically, a log is an instance of an object that accumulates key metrics about network playback and typically presents these metrics in the form of a collection of event object instances. For example, where the method involves AVPlayer, the log can be an AVPlayerItemAccessLog instance. Generally speaking, logs are accessed via an application program interface (API). The monitoring and analysis can be performed in addition to or instead of each other.

[0028] Detection of a change, such as by notification as described above, does not in itself typically provide an indication of the specific time at which the change occurred. Preferably, detection may comprise detecting when a rate change occurs. Thus, the method may further comprise obtaining an indication of the time at which a bit rate change occurs or has occurred, or in other words, may comprise determining the time. This may be by directly indicating an absolute time or a relative time relative to some time stamp associated with video decoding, reception, or playback. For example, it may be a timestamp, frame number, or ID.

[0029] In some embodiments, obtaining an indication includes detecting a change in an identifier associated with the video stream. The identifier may typically be a track ID for the input data or video asset. In iOS embodiments, it may be an AVAsset. An identifier change typically indicates a profile change, and thus a time (e.g., absolute time or relative to an event such as a notification) may typically be obtained or inferred from the change in the track ID. Typically, in addition to changes for ABR profile changes, track IDs for actions and operation types such as image searches and find actions also change. Thus, preferably, this is combined with obtaining an indication or detecting changes in conjunction with notification criteria and, optionally, additionally, or alternatively, frame resolution as described below.

[0030] The association with the video stream may include that the identifier is an identifier of a current video asset, the current video asset being, for example, a data asset being decoded and / or played at the current time.

[0031] Preferably, obtaining the indication includes identifying the difference in frame resolution between a first frame in a video stream and a second immediately subsequent frame in the video stream, and obtaining the indication based on the time at which the second frame appeared. In the context of digital images composed of video frames, resolution can be understood as, or at least indicates or represents, the pixel count of a frame. That is, it can correspond to the number of pixels comprising or contained in an image. Additionally or alternatively, it can include a count and / or ratio of the number of rows and columns of pixels in a video frame, such as an aspect ratio, or an indication of resolution by number of rows, such as 1080p or 2160p. The indication can also be obtained by including determining the time at which the first frame appeared. This makes it easier to identify changes. However, a change in bitrate does not necessarily mean a change in resolution, and therefore, where appropriate, it is preferable to additionally use the aforementioned identifier or track ID change. Using this additional criterion allows the overall process of identifying bitrate changes to be performed more efficiently. A frame present can be understood as a frame present in the stream and / or a frame currently being decoded and / or displayed.

[0032] Preferably, the method further comprises identifying a changed bitrate parameter set. That is, in preferred embodiments, the bitrate profile to be modified is identified. In some embodiments, a bitrate change can be identified based on a resolution change without monitoring or detecting a notification. The issuance or monitoring of bitrate change notifications may be delayed. Therefore, advantageously, in some embodiments, if a resolution change is detected or identified between frames, this can itself be considered to indicate an ABR change.

[0033] The changed bitrate parameter set can be one of a plurality of bitrate parameter sets. For example, the plurality of bitrate parameter sets can be listed or recorded in a manifest. This typically indicates the available bitrate profiles according to which the video stream is decoded, and in some embodiments, can be indicated in the aforementioned playlist. The bitrate parameter set of the video stream before the change is also typically included in the plurality of sets. Therefore, identifying the changed bitrate parameter set can be performed based on determining whether any one of the plurality of bitrate parameter sets corresponds to the same frame resolution as the frame resolution corresponding to another bitrate parameter set in the plurality of bitrate parameter sets. In some embodiments, this determination can be pre-determined, or in other embodiments, the determination can be performed as part of the method. In other words, the method can involve determining whether all available profiles correspond to different video resolutions, or whether any of those profiles are identical in terms of frame resolution. If all profiles correspond to different resolutions, identifying the changed profile can be performed more efficiently, and thus this determination improves the overall efficiency of the method. Typically, if the determination is negative, identifying the changed bitrate parameter set comprises comparing the frame resolution of the video stream frame occurring after the change with the frame resolution corresponding to each of one or more of the plurality of bitrate parameter sets, and determining that the changed parameter set is the one of the plurality of bitrate parameter sets corresponding to the frame resolution of the video stream frame. Thus, if all of the plurality of bitrate parameter sets correspond to different frame resolutions, the simplicity of the case can be exploited where the base frame has a different resolution than the previous frame received, and the frame resolution itself can be used to identify the new bitrate profile. That is, such a match can be used to implement a simple and efficient selection of the next active buffer queue video stream, as it can be ensured that there is no resolution degradation in the available queues.

[0034] In some embodiments, the first plurality of bitrate parameter sets may include a second plurality of bitrate parameter sets. The second plurality may be identical to the first plurality. However, it preferably includes fewer sets than the first plurality. Selection can be performed in a variety of ways, as will now be described. Generally, if the above determination is affirmative, the variable bitrate parameter set can be identified by selecting a bitrate parameter set from the second plurality of bitrate parameter sets that corresponds to the maximum bitrate or a bitrate corresponding to the second plurality of parameter sets. Performing the steps described herein is advantageous when there is degradation in the resolution of the available profiles. However, the method steps can also be performed in the absence of such conditions, that is, in some embodiments, the method steps can be performed regardless of the presence of such degradation, including when all profiles correspond to different resolutions. A positive determination can be understood to mean that any two sets in the plurality of bitrate parameter sets correspond to the same frame resolution. The variable bitrate parameter set can accordingly be identified as one of the second plurality of bitrate parameter sets corresponding to the highest video bitrate. In a preferred embodiment, in the case of the current image, if there are multiple notifications corresponding to the same resolution, the method may further include checking or otherwise examining or analyzing one or more recently downloaded video segments, and typically also identifying the sub-playlist containing or responding to the segment. If there are multiple options for finding the next stream, typically indicating that segments of multiple sub-playlists have been downloaded, the one with the highest bit rate may be selected, as described above.

[0035] The ABR profile selection process can be made more computationally efficient by pre-selecting candidate ABR profiles prior to the selection. This may be based primarily on the temporal range of the video segments. In some embodiments, if the temporal range-based selection (typically in combination with the resolution-based selection) allows for the identification of a single matching profile parameter set, bitrate matching may be abandoned in some cases. Preferably, the first plurality of bitrate parameter sets comprises a third plurality of bitrate parameter sets, and the second plurality of bitrate parameter sets is selected by selecting, from the third plurality of bitrate parameter sets, or preferably from any of the bitrate parameter sets, bitrate parameter sets corresponding to time periods containing indicated times at which bitrate changes occur (preferably indicated as described above). The third plurality may be the same as the first plurality, or may be a selective subset thereof. Thus, filtering based on time period is preferably performed by reference to a data set indicating the time period to which each ABR profile corresponds.

[0036] By first selecting from available ABR profiles based on video resolution, the selection process can be made more efficient. Resolution in this context should be understood to have the meaning set forth earlier in this disclosure, preferably in terms of pixel or row / column count. If only one profile matches the video resolution, the parameter set can be selected based solely on that resolution. Thus, the third plurality of bitrate parameters can preferably be obtained by selecting from the first plurality of bitrate parameter sets those bitrate parameter sets that correspond to the same frame resolution as the frame resolution of the video stream frames occurring after, and preferably immediately after, a change in the predetermined bitrate parameter set. Any of the first, second, and third bitrate parameter sets can be the same as and / or different from any other bitrate parameter set in the first, second, and third plurality of bitrate parameter sets. Selection based on matching frame resolution can typically be performed before selection based on time range or bitrate. Therefore, if a single matching profile parameter set can be identified based on resolution, the subsequent selection step can be foregone and only that resolution can be used. In other cases, efficiency gains can still be achieved by performing a preselection or series of preselections to identify the appropriate profile.

[0037] The method generally further comprises instructing the rendering module to render the video frame using the reconstructed first rendition at a relatively high quality level.As with the aforementioned instructing steps, in some embodiments, these steps may be performed by or via a server application.

[0038] Typically, a video stream includes first input data based on a set of spatiotemporally correlated elements and second input data based on a first time sample of a signal reproduced at a relatively lower quality level in a hierarchical hierarchy having a plurality of quality levels, wherein the set of spatiotemporally correlated elements indicates a degree of spatial correlation between a plurality of residual elements in a set of residual elements and a degree of temporal correlation between first reference data based on the first reproduction and second reference data based on a reproduction of a second time sample of the signal. Typically, instructing the decoder integration layer includes instructing the decoder integration layer to perform the reconstruction by using the set of spatiotemporally correlated elements to obtain a set of residual elements, the set of residual elements being usable to reconstruct a first reproduction of the first time sample of the signal at a relatively higher quality level using a second reproduction of the first time sample of the signal at a relatively higher quality level in the hierarchical hierarchy, the second reproduction being based on the reproduction at the relatively lower quality level.

[0039] Typically, instructing the base decoder to obtain the decoded base stream includes instructing the base decoder to obtain a second reproduction, typically of a relatively lower quality level. Typically, instructing the decoder integration layer to reconstruct the original input video includes instructing the decoder integration layer to reconstruct the first reproduction at a relatively higher quality level using the second reproduction and the set of residual elements.

[0040] The first data typically has a relatively high quality level. The second data typically has a relatively high quality level. The second time sample of the signal is typically an earlier time sample than the first time sample of the signal. Typically, the first reference data includes a first set of spatially correlated elements associated with the first time sample of the signal, the first set of spatially correlated elements indicating a degree of spatial correlation between a plurality of residual elements in the residual element set.

[0041] Typically, the second reference data comprises a second set of spatially correlated elements associated with a second time sample of the signal, the second set of spatially correlated elements indicating a degree of spatial correlation between multiple residual elements in another set of residual elements associated with the second time sample, and wherein the another set of residual elements can be used to reconstruct a reproduction of the second time sample of the signal at a relatively higher quality level using data based on a reproduction of the second time sample of the signal at a relatively lower quality level.

[0042] Typically, the first reference data comprises a first reproduction of a first time sample of the signal. Typically, the second reference data comprises a reconstructed reproduction of a second time sample of the signal at a relatively higher quality level. Typically, the input data comprises data indicating a result of selecting between the set of spatiotemporally correlated elements and the first set of spatially correlated elements for inclusion in the input data.

[0043] Typically, the data indicating the result of the selection is included in a bit sequence associated with a given data element in the input data, the given data element indicating a degree of horizontal correlation between a plurality of residual elements in the set of residual elements.

[0044] Typically, the first input data comprises a set of spatiotemporally correlated elements and a quantized element selected from the first set of spatially correlated elements for inclusion in the input data. Typically, the method comprises: retrieving first buffer data from a buffer; and reconstructing a first reconstruction using the first buffer data. Typically, the first buffer data comprises second reference data. Typically, the method comprises: generating second buffer data using the set of spatiotemporally correlated elements; and updating the buffer by overwriting the first buffer data with the second buffer data.

[0045] Typically, the method comprises generating second buffer data by combining a set of spatiotemporally correlated elements with first buffer data. Typically, the first buffer data comprises at least one zero value. Typically, the second rendition is obtained by performing a sampling operation on a rendition having a relatively low quality level. Typically, the signal is a video signal. The input data is typically received from at least one other device via one or more data communication networks.

[0046] Preferably, the enhanced decoder is an LCEVC decoder, so that the decoder integration layer, one or more plug-ins and the enhanced decoder together provide an LCEVC decoding software solution. The LECVC decoding software stack can be implemented in one or more LCEVC decoder libraries and thus provide an optimized software library for decoding MPEG-5 enhancement streams. The decoder integration layer can also include controls for upward scaling operations to scale the decoded video signal from the base coding layer so that one or more residual data layers can be applied to the decoded video signal from the base coding layer. The decoder can be easily implemented on popular media players across platforms such as iOS.

[0047] One or more decoder plug-ins can be configured to indicate the corresponding basic decoder through library function calls or operating system function calls. Therefore, different basic decoding implementation schemes can be easily supported, including native implementation schemes within the operating system and hardware accelerated decoding.

[0048] The decoder integration layer can be configured to apply one or more residual data layers from the enhancement coding layer to the decoded video signal from the base coding layer to produce a decoded reconstruction of the original input video signal. In some cases, the decoder integration layer can instruct a plugin from the decoder plugin set to apply one or more residual data layers; in other cases, the decoder integration layer can obtain the decoded output from the base coding layer using the decoder plugin instructions and combine this with the output of the enhancement decoder. Preferably, the residual data layers can be applied during playback.

[0049] In some embodiments, the decoder integration layer is configured to receive: one or more input buffers containing the encoded video signal and the encoded enhancement signal in encoding order, wherein the one or more input buffers also feed the base decoder; and one or more base decoded frames of the decoded video signal from the base coding layer in presentation order. In this way, minimal processing is required on the client side, and the integration is responsible for client operation. The same input buffer can be passed to both the base decoding layer and the enhancement decoding layer to increase simplicity.

[0050] In a particularly preferred embodiment, the control interface includes an output type configuration parameter, wherein the decoder integration layer is configured to change how a decoded reconstruction of the original input video signal is output based on a value of the output type configuration parameter. The value of the output type configuration parameter may be stored in a configuration data structure that is retrieved by the decoder integration layer after initialization.

[0051] In one example of configured outputs, the decoder integration layer is configured to output a decoded reconstruction of the original input video signal as one or more buffers. In another example, the decoder integration layer is configured to output a decoded reconstruction of the original input video signal as one or more on-screen surfaces. Alternatively, the decoder integration layer is configured to output a decoded reconstruction of the original input video signal as one or more off-screen textures. Each of these three example outputs can be selected by the output type configuration parameter.

[0052] According to another aspect of the present invention, there is provided a computer program comprising instructions which, when executed, cause a device to perform the method according to the above aspect.

[0053] According to a further aspect of the present invention, there is provided a computer-readable medium comprising the computer program according to the immediately preceding aspect.

[0054] According to a further aspect of the present invention, there is provided a device configured to:

[0055] receiving, at a server application, a video stream, the video stream being encoded according to a multi-layer coding scheme;

[0056] Passing the encoded elementary stream included in the video stream to an elementary decoder to obtain a decoded elementary stream; and

[0057] The encoded enhancement stream included in the video stream is passed to an enhancement decoder for decoding, wherein the original input video can be reconstructed from the decoded base stream and the decoded enhancement stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Examples of the method according to the present invention will now be described with reference to the accompanying drawings, in which like reference numerals indicate like features, and in which:

[0059] FIG1 shows a flow chart illustrating a method for decoding a video stream according to the prior art;

[0060] Figure 2a and Figure 2b Flowcharts are respectively shown, which illustrate an example method and an example pipeline for decoding an enhanced video stream according to an embodiment of the present invention;

[0061] Figure 3 shows a flow chart illustrating an example method including a TS pipeline according to an embodiment of the present invention;

[0062] Figure 4 shows a flow chart illustrating an example method including an MP4 pipeline according to an embodiment of the present invention;

[0063] Figure 5 shows a flow chart illustrating an example method including a TS pipeline according to an embodiment of the present invention;

[0064] Figure 6 shows a flow chart illustrating an example method including an MP4 pipeline according to an embodiment of the present invention;

[0065] Figure 7 is a flow chart depicting another example of a method including detecting ABR profile changes using base frame resolution according to an embodiment of the present invention;

[0066] Figure 8 is a flowchart depicting another example of a method including detecting an ABR profile change using a VideoAssetTrack trackId identifier associated with a video stream according to an embodiment of the present invention;

[0067] Figure 9 is a flowchart depicting an example method including identifying a varying ABR profile when all profile resolutions are different, according to an embodiment of the present invention;

[0068] Figure 10 is a flowchart depicting an example method including identifying a changed ABR profile when two or more profiles have the same resolution according to an embodiment of the present invention;

[0069] Figure 11 shows a flow chart illustrating an example method including creating a new MultiVideoQueue queue according to an embodiment of the present invention;

[0070] Figure 12 A flow chart is shown illustrating an example method including activating a MultiVideoQueue queue according to an embodiment of the present invention; and

[0071] Figure 13 A flow chart illustrating an example method including cache cleaning according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0072] The present disclosure relates to implementations of hybrid backward-compatible coding techniques integrated with existing decoders, optionally via software updates, and in particular describes implementations of integrating such methods into enhanced players suitable for devices that traditionally use existing base decoders and players. In a non-limiting example, the present disclosure relates to implementations and integrations of enhanced players that employ an enhanced decoder using MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC), as described in WO 2022 / 023739 A1, the contents of which are incorporated herein by reference. LCEVC is a hybrid backward-compatible coding technique that is a flexible, adaptable, efficient, and computationally inexpensive coding format that combines different video coding formats, base codecs (i.e., encoder-decoder pairs such as AVC / H.264, HEVC / H.265, or any other current or future codecs, as well as non-standard algorithms such as VP9, ​​AV1) with coded data of one or more enhancement layers. An example LCEVC implementation is described below that may be integrated as part of the enhanced player and methods described subsequently.

[0073] The example hybrid backward compatible decoding technique uses a downsampled source signal that is encoded using a base codec to form a base stream. Enhancement streams are formed using coded sets that correct or enhance the residuals of the base stream, for example, by increasing the resolution or by increasing the frame rate. Multiple levels of enhancement data may exist in the hierarchical structure. In some arrangements, the base stream may be decoded by a hardware decoder, while the enhancement stream may be suitable for processing using a software implementation. Therefore, the stream is considered to be a base stream and one or more enhancement streams, wherein two enhancement streams may typically exist but one enhancement stream is often used. It is noteworthy that typically the base stream may be decoded by a hardware decoder, while the enhancement stream may be suitable for a software processing implementation with suitable power consumption.

[0074] Compared to the block-based approach used in the MPEG family of algorithms, video frames are encoded in a hierarchical manner. Encoding frames in a hierarchical manner includes generating a residual for a full frame, and then generating a residual for a reduced or decimated frame, etc. In the examples described herein, the residual can be considered as an error or difference at a specific quality level or resolution.

[0075] The detailed structure of LCEVC is known and is set forth in an approved draft standard specification. Based on the general description of LCEVC presented in Figure 1 of WO 2022 / 023739A1, those skilled in the art will understand how the examples described herein are also applicable to other multi-layer decoding schemes (e.g., those using base layer and enhancement layer). The LCEVC decoder operates at a separate video frame level. It takes decoded low-resolution pictures and LCEVC enhancement data from a base (H.264 or other) video decoder as input to produce decoded full-resolution pictures ready for rendering on a display view. LCEVC enhancement data is typically received in the Supplemental Enhancement Information (SEI) of the H.264 Network Abstraction Layer (NAL) or in an additional track or data packet identifier (PID) and is separated from the base coded video by a demultiplexer. Therefore, the base video decoder receives a demultiplexed coded base stream and the LCEVC decoder receives a demultiplexed coded enhancement stream, which is decoded by the LCEVC decoder to generate a set of residuals for combining with the decoded low-resolution pictures from the base video decoder.

[0076] Additional PID means additional tracks or PIDs. This means not only transport streams (PIDs) but also ISO base media file formats and WebM as container types.

[0077] Throughout this specification, the present invention may be described in the context of NAL units. However, it should be understood that NAL units in this context may equally and more generally refer to elementary stream input buffers or equivalent. That is, LCEVC is also capable of supporting non-MPEG base codecs that do not typically use NAL encapsulation, namely VP8 / VP9 and AV1. Therefore, where the term NAL unit is used, the term may be read to mean elementary stream input buffers depending on the underlying codec being utilized.

[0078] LCEVC can be quickly implemented in existing decoders with software updates and is inherently backward compatible since devices that have not yet been updated to decode LCEVC can play video using the basic base codec, further simplifying deployment.

[0079] Decoder implementations that integrate decoding and rendering with existing systems and devices that perform basic decoding can be used as part of an extension to existing player and decoding libraries, as described later in this disclosure. The integration is easy to deploy.

[0080] The terms LCEVC and enhancement may be used interchangeably in some examples, eg, an enhancement layer may contain one or more enhancement streams, ie, residual data of the LCEVC enhancement data.

[0081] WO 2022 / 023739 A1 Figure 2a An unmodified video pipeline is shown. In this conceptual pipeline, obtained or received Network Abstraction Layer (NAL) units are input to a base decoder. Depending on the operating system, the base decoder can be, for example, a low-level media codec accessed using a mechanism such as MediaCodec (e.g., as found in the Android (RTM) operating system), VTDecompression Session (e.g., as found in the iOS (RTM) operating system), or Media Foundation Transform (MFT—e.g., as found in the Windows (RTM) family of operating systems). The output of the pipeline is a surface representing the decoded raw video signal (e.g., frames of such a video signal, with the order of successful frames showing rendered video).

[0082] WO 2022 / 023739 A1 Figure 2b The proposed video pipeline using the LCEVC decoder integration layer is conceptually described. As with the aforementioned comparative video decoder pipeline, NAL units are obtained or received and processed by the LCEVC decoder to provide a surface of reconstructed video data. By using the LCEVC decoder, the quality of the surface can be higher than the quality of the comparative surface, or the quality of the surface can be the same as the quality of the comparative surface but require less processing and / or network resources.

[0083] As mentioned above, when referring to NAL units herein, this refers to the underlying stream input buffer or equivalent, depending on the underlying codec used.

[0084] In WO 2022 / 023739 A1 Figure 2b In the embodiment of the present invention, the LCEVC decoder is implemented together with the base decoder. The base decoder can be provided by a variety of mechanisms, including operating system functions as discussed above (for example, MediaCodec, VTDecompression Session or MFT interfaces or commands can be used). The base decoder 26 can be hardware accelerated, for example, using a dedicated processing chip to implement operations for a specific codec. The base decoder can be the same as the above-mentioned base decoder and used for other non-LCCEVC video decoding, for example, it can include a pre-existing base decoder.

[0085] In WO 2022 / 023739 A1 Figure 2bIn the present invention, the LCEVC decoder is implemented using a decoder integration layer (DIL). The decoder integration layer is used to provide a control interface for the LCEVC decoder so that client applications can use the LCEVC decoder in a similar manner to the previous base decoder, for example as a complete solution from buffer to output. The decoder integration layer is used to control the operation of the decoder plug-in (DPI) and the enhancement decoder to produce a decoded reconstruction of the original input video signal. In some variations, as shown in the figure, the decoder integration layer can also control GPU functions such as GPU shaders to reconstruct the original input video signal from the decoded base stream and the decoded enhancement stream.

[0086] NAL units containing the coded video signal along with associated enhancement data may be provided in one or more input buffers. The input buffers may be fed by similar non-MPEG base stream input buffers, such as those used in VP8 / VP9 or AV1. The input buffers may be fed to (or made available to) the base decoder and the decoder integration layer, particularly the enhancement decoder controlled by the decoder integration layer. In some examples, the coded video signal may comprise a coded base stream and be received separately from a coded enhancement stream comprising enhancement data; in other preferred examples, the coded video signal comprising the coded base stream may be received together with the coded enhancement stream, for example as a single multiplexed coded video stream. In the latter case, the same buffer may be fed to (or made available to) both the base decoder and the decoder integration layer. In this case, the base decoder may retrieve the coded video signal comprising the coded base stream and ignore any enhancement data in the NAL units. For example, the enhancement data may be carried in SEI messages for the base stream of video data, which may be ignored by the base decoder if the enhancement data is not suitable for processing the custom SEI message data. In this case, the base decoder may operate according to the base decoder in the figure, but in some cases the base video stream may be at a lower resolution than the comparative case.

[0087] After receiving an encoded video signal comprising an encoded elementary stream, the base decoder is configured to decode the encoded video signal and output it as one or more base decoded frames. This output can then be received or accessed by the decoder integration layer for enhancement. In one example set, the base decoded frames are passed as input to the decoder integration layer in presentation order.

[0088] The decoder integration layer extracts the LCEVC enhancement data from the input buffer and decodes the enhancement data. Decoding of the enhancement data is performed by an enhancement decoder that receives the enhancement data from the input buffer as an encoded enhancement signal and extracts the residual data by applying the enhancement decoding pipeline to one or more streams of encoded residual data. For example, the enhancement decoder may implement an LCEVC standard decoder as set forth in the LCEVC specification.

[0089] The decoder plug-in is provided at the decoder integration layer to control the function of the basic decoder. In some cases, the decoder plug-in can handle the reception and / or access of the basic decoded video frame, and preferably during playback, the LCEVC enhancement is applied to these frames. In other cases, the decoder plug-in can be arranged to be accessible by the decoder integration layer with the output of the basic decoder, and the decoder integration layer is then arranged to control the interpolation of the residual output from the enhancement decoder to produce an output surface. Once integrated in the decoding device, the LCEVC decoder is just realized by decoding and the playback of the video of the LCEVC enhancement coding. The rendering of the decoded reconstructed video signal can be supported by one or more GPU functions (as the GPU shader controlled by the decoder integration layer).

[0090] Typically, the decoder integration layer controls the operation of one or more decoder plug-ins and enhancement decoders to produce a decoded reconstruction of the original input video signal using a decoded video signal from a base coding layer (i.e., as implemented by a base decoder) and one or more residual data layers from an enhancement coding layer (i.e., as implemented by an enhancement decoder). The decoder integration layer provides a control interface for the video decoder, such as to an application within a client device.

[0091] Depending on the configuration, the decoder integration layer can output the surface of decoded data in different ways. For example, as a buffer, as an off-screen texture, or as an on-screen surface. The output format used can be set in the configuration settings provided after creating an instance of the decoder integration layer, as explained further below.

[0092] In some embodiments, in the event that no enhancement data is found in the input buffer, e.g., in the event that the NAL unit does not contain enhancement data, the decoder integration layer may fall back to delivering the video signal at a lower resolution to the output, i.e., the output of the base decoding layer as implemented by the base decoder. In this case, the LCEVC decoder may operate according to the video decoder pipeline.

[0093] The decoder integration layer can be used for both application integration and operating system integration, for example, for use by both client applications and operating systems. The decoder integration layer can be used to control operating system functions, such as function calls to the hardware-accelerated basic codec, without requiring the client application to understand these functions. In some cases, multiple decoder plug-ins can be provided, wherein each decoder plug-in provides a wrapper for different basic codecs. It is also possible for a common basic codec to have multiple decoder plug-ins. This can be the situation where there are different implementations of the basic codec, such as a GPU accelerated version, a native hardware accelerated version, and an open source software version.

[0094] A decoder plugin can be viewed as integrated with a base decoder or alternatively as a wrapper around the base decoder. The decoder integration layer conceptually includes functionality to extract enhancement data from NAL units, functionality to communicate with the decoder plugin and apply the enhancement decoded data to the base decoded data, and one or more GPU functions.

[0095] The collection of decoder plugins is configured to present a common interface (i.e., a common command set) to the decoder integration layer, so that the decoder integration layer can operate without knowledge of the specific commands or functionality of each base decoder. Plugins thus allow base codec specific commands (such as MediaCodec, VTDecompression Session, or MFT) to be mapped to a plugin command set accessible by the decoder integration layer (e.g., multiple different decoding function calls can be mapped to a single common plugin "Decode(...)" function).

[0096] Since the decoder integration layer effectively contains a 'residual engine', ie a library that generates sets of correction planes of different quality levels from LCEVC coded NAL units, the layer can behave as a complete decoder (ie the same as the decoder) by controlling the base decoder.

[0097] For simplicity, the indicated entity will be referred to herein as a client, but it will be understood that the client can be considered any application layer or functional layer, and the decoder integration layer can be simply and easily integrated into a software solution. The terms client, application layer, and user are used interchangeably herein.

[0098] In application integration, the decoder integration layer can be configured to render directly onto a screen surface of arbitrary size (typically different from the content resolution) provided by the client. For example, even if the base decoded video may be standard definition (SD), the decoder integration layer may use enhancement data to render the surface in high definition (HD), ultra high definition (UHD), or a custom resolution. Further details of a non-standard method of upward scaling and post-processing that can be applied to LCEVC decoded video streams are found in PCT / GB2020 / 052420, which is incorporated herein by reference. An example application integration includes, for example, a media player using an LCEVC decoder, the media player being configured to decode an LCEVC video stream by using the LCEVC decoder "in the background", wherein the computer program code of the media player function is configured to use and call commands provided by the decoder integration layer (i.e., the control interface of the LCEVC decoder), as will be described in more detail later in this disclosure.

[0099] In operating system integration, the decoder integration layer can be configured to decode the buffer or draw on an off-screen texture of the same size of the content's final resolution. In this case, the decoder integration layer can be configured so that it does not handle the final rendering to displays such as display devices. In these cases, the final rendering can be handled by the operating system, and therefore, the operating system can use the control interface provided by the decoder integration layer to provide LCEVC decoding as part of the operating system call. In these cases, the operating system can implement additional operations on LCEVC decoding, such as YUV to RGB conversion, and / or adjust the size of the destination surface before the final rendering on the display device. Examples of operating system integration include integration with the MFT decoder (or backend) for Microsoft Windows (RTM) operating system or with the Open Media Acceleration (OpenMAX-OMX) decoder (or backend), which is a C-based programming interface set (e.g., at the kernel level) for low-power and embedded systems, including smartphones, digital media players, game consoles, and set-top boxes.

[0100] These integration modes can be set by the client device or application, and the mechanisms for selection and configuration are described in more detail below.

[0101] The configuration and use of the decoder integration layer allow LCEVC decoding and rendering to be integrated with many different types of existing traditional (i.e., basic) decoder implementations. For example, the configuration can be considered a modification of the configuration, as can be found on a computing device. Another example of integration includes an LCEVC decoding library available in common video decoding tools such as FFmpeg and FFplay. For example, FFmpeg is often used as a basic video decoding tool in client applications. By configuring the decoder integration layer as a plug-in or patch for FFmpeg, an LCEVC-enabled FFmpeg decoder can be provided so that the client application can use the known functionality of FFmpeg and FFplay to decode LCEVC (i.e., enhanced) video streams. For example, an LCEVC-enabled FFmpeg decoder can provide video decoding operations such as playback, decoding to YUV and running metrics (e.g., peak signal-to-noise ratio (PSNR) or video multi-method assessment fusion (VMAF) metrics) without first decoding to YUV. This can be achieved by a plug-in or patch computer program code for FFmpeg call functions provided by the decoder integration layer.

[0102] An example embodiment is now described in which LCEVC enhanced video decoding functionality is added to an iOS application based on native AVPlayer. It will be understood that AVPlayer is a controller object used with devices running the iOS operating system, such as for managing playback and timing of media assets.

[0103] Figure 1 shows a flow diagram illustrating the interactions in a traditional platform where an application within the AVFoundation framework 101 obtains, decodes, and plays a video asset. As will be appreciated by those skilled in the art, AVFoundation is an existing framework for handling time-based audiovisual media on iOS, macOS, watchOS, and tvOS operating systems. A command or request 1 is received at an AVPlayer instance 102 running on an iOS device, signaling that a media asset, in this example, a video asset, is to be played. In response, AVPlayer transmits an HTTP request 2 to an origin server 107, such as a server of a video streaming platform. The video asset is transferred 3 from the origin server to the AVFoundation internal cache 104 for playback on the iOS device.

[0104] Figure 2aAn example interaction that may form part of the present method is depicted. In this example, a local network server application 203 is provided on an iOS device and is configured to intercept asset requests sent by AVFoundation. The request 1 received by AVFoundation 201 includes additional information to enable this modified interaction. In particular, the request to play a video asset includes an indication of a modified host. Therefore, the AVFoundation application then sends an HTTP request for the video content, which is not transmitted directly to the original server 207, but is received by the local network server application. The network server then sends a request for the video asset to the source. In response, the source transmits the requested encoded stream to the entity that received the request, i.e. the network server. The received video stream includes an encoded base stream and an enhancement stream encoded according to the above-mentioned LCEVC scheme. LCEVC encoded video includes additional information that can be used by the LCEVC enhancer 209 to improve the quality and resolution of the image after rendering. The encoded base stream is returned from the local network server to AVPlayer 202 within the AVFoundation framework 201 and decoded according to the traditional base decoding scheme used by the AVPlayer base decoder. The enhancement stream is passed to the demultiplexer 205, from which the demultiplexed video stream is passed to the DIL for decoding. Subsequently, the decoded base frame 210 is transmitted to the DIL, and the renderer plays the combined base and enhancement streams, allowing the reconstructed video to be played by the associated iOS application. Figure 2b The receiving and processing of encoded video data 206 when delivered from a local server application is shown. The result of the enhancement process is a new frame 208 to be displayed. In this example, the rendering of the frame is handled by the LCEVCAVPlayerKit library, which provides a custom UI view for rendering video.

[0105] An example implementation of integrating this enhancement process with the AVFoundation framework and AVPlayer may involve the following components. The LCEVC process is encapsulated in a single class for easy and quick integration. LCEVCAVPlayer is a subtype of AVPlayer that adds the interceptions and processes required to generate enhanced video. This class is compatible with standard components that utilize AVPlayer and adds:

[0106] outputVideoView: A read-only property that allows obtaining a reference to the view containing the enhanced video at any time. This view is maintained and created by the player and can be easily integrated into any existing view.

[0107] renderIn: This function initializes and configures the outputVideoView as a subview of the provided parent view, creating the constraints necessary for it to fill its parent view. If the client application has custom layout management, this call can be skipped.

[0108] Width: A read-only property that returns the width of the enhanced video. Typically, this property is only used for debugging purposes, but it can be important if the client application needs to manipulate the application's layout.

[0109] Height: A read-only property that returns the width of the enhanced video. Typically, this property is only used for debugging purposes, but it can be important if the client application needs to manipulate the application's layout.

[0110] Figure 5 and 6 A more detailed flow corresponding to the above-described interception and decoding process is shown, which also illustrates the architecture of an example embodiment.

[0111] The example described herein includes an extension of the Apple AVFoundation library VTDecompressionSession (VTDS), called LCEVCDecompressionSession. iOS players can use LCEVCDecompressionSession instead of VTDS to parse streams and decode when LCEVC occurs. In the case of these embodiments, LCEVC decoding can be used on iOS devices by simply patching the VLCKit infrastructure and the LCEVC library. Advantageously, LCEVC can be easily integrated into VLCKit and other iOS players.

[0112] In these examples, LCEVCDecompressionSession uses DIL to extract LCEVC enhancement data and apply it to the base video in preparation for compliance with Apple's decoder module specification. A favorable modification is made to the native AVPlayer, called LCEVC AVPlayer. LCEVC AVPlayer is a subclass of the native AVPlayer and inherits the same core functionality and behavior, such as ABR handling, which is described in detail later in this disclosure.

[0113] As a result, any iOS application that uses the native AVPlayer can be quickly updated to incorporate the LCEVC decoding library with minor code modifications. The LCEVC AVPlayer can be used with both the native control (AVPlayerViewController) or a custom control.

[0114] Now we will describe an example process of integrating LCEVC AVPlayer on an iOS device.

[0115] The first integration method uses AVPlayerViewController. The LCEVCAVPlayerKit library is compatible with applications that use native iOS player controls provided by AVFoundation through AVPlayerViewController.

[0116] First, import the framework. AVPlayerViewController requires the corresponding import of Apple's AVKit framework. In this example, this is called LCEVCAVPlayerKit. Then, create an instance of LCEVCAVPlayer, providing the remote URL of the asset to play. This is a subclass of AVPlayer and can be used in place of a traditional AVPlayer instance.

[0117] Once an LCEVCAVPlayer instance is created, it can be provided to the AVPlayerViewController using its player property, and the renderIn method of LCEVCAVPlayer can be called. This renders the enhanced video onto the provided superlayer. This results in the LCEVC enhanced video being displayed using the native iOS player controls provided by AVFoundation.

[0118] The second example integration method uses LCEVCAVPlayer directly. This implementation is suitable if you need greater control over the playback UI controls or further customize the player.

[0119] First, add a UIView instance to the relevant app. This will be used to store the augmented video. This can be done programmatically or using a storyboard. Then, import the LCEVCAVPlayerKit framework.

[0120] An LCEVCAVPlayer instance is created and maintained via the ViewController, providing the URL of the remote stream to be played. The player displays the functionality of a standard AVPlayer. Once the view is created, for example when the viewDidLoad trigger is called, the addObserver function can be added to the player to monitor and react to different status changes by using the key path AVPlayer.status. This causes the callback to be triggered every time the player state changes. At this point, the LCEVCAVPlayer is equipped with the required rendering target. The targetView parameter refers to the UIView into which the LCEVCAVPlayer is inserted into the output rendering target. It is configured to extend so as to fill its parent view. If you need to manage the layout manually, you can access the view using the outputVideoView property.

[0121] This way, the implementation is configured and able to handle enhanced video data. The previously defined observer can be used to trigger playback by capturing the readyToPlay state, in which case it is safe to use AVPlayer's default play method to start a playback session.

[0122] As mentioned above, in this example, the LCEVC AVPlayer utilizes a local network server to intercept requests executed by AVFoundation. This approach is secure because the included server is configured to reject any requests that do not originate from a local device. The example implementation of the enhancer internally intercepts and modifies the requested URL in order to extract the LCEVC enhancement data.

[0123] Figure 3-12 is a flowchart showing an example of an implementation scheme involved in AVPlayer. Figure 3 and 4 As shown, AVPlayer 302 initially issues an HTTP request 2 to the origin 307 to request a resource to play. The URL of this request has been modified so that the origin can return the result 4 of the request to the local network server 303 instead of reporting directly to AVPlayer in response to request 3.

[0124] Once the response is generated, it is dispatched to both the AVPlayer and the ByteRangeBasedCache. An example AVPlayer pipeline and an MP4 demuxer pipeline are shown in the above figures.

[0125] The AVPlayer pipeline operates as follows. AVPlayer performs basic decoding. The frames of the requested video asset to be displayed on screen can be retrieved via AVPlayerItemVideoOutput, more specifically using CADisplayLinkCallback. This callback is called periodically and synchronized with the playback stream's frame rate. The frames are then passed to DIL 311 to generate the enhancement frames.

[0126] In the MP4 demux pipeline, the MP4 demuxer requests a specific byterange of data in order to perform demultiplexing. This data will be provided by the ByteRangeBasedCache 412. Once the demuxer completes demultiplexing, it generates access units which are fed to the DIL using the DIL_AddNalUnitData API.

[0127] There is a maximum number of LCEVC packets that can be stored in the DIL (20 in this example). The DIL is typically configured to request more packets once the number of stored packets falls below this value. The reason for this constraint is that adding the product of the demuxer directly to the DIL could cause the DIL to allocate a large amount of memory (typically over 1GB).

[0128] When feeding LCEVC packets to the DIL, PTS is also added to a buffer to keep track of the PTS currently contained in the DIL, called videoBuffersFedTimeList. In order to produce the final enhancement frame, the DIL needs the PTS of the base frame (generated as explained above) and the LCEVC data paired with the base frame.

[0129] Since the time information of the base frame corresponds to the screen timestamp, not the actual presentation timestamp, when pairing a base frame with an LCEVC chunk, we look for the LCEVC chunk whose PTS is closest to the time value reported on the base frame. The delta is calculated when the base frame and LCEVC packet are first paired, so taking this into account, we can be confident that the time information reported on the base frame plus the calculated delta will correspond one-to-one with the PTS on one of the LCEVC chunks extracted by the MP4 demuxer.

[0130] Once the base frame and the PTS of the corresponding LCEVC chunk are found, these two pieces of information are passed to the DIL_Decode API to produce the enhanced output.

[0131] As mentioned above, the first part of the pipeline related to WebProxyServer is common to both MP4 and TS. TS does not have a ByteRangeBasedCache, and therefore the data received from the HTTP request is instead passed directly to the TS demultiplexer, which produces access units. The access units are then stored in a cache (TimeBasedCacheManager), which is responsible for keeping the NAL units sorted according to their PTS. The TimeBasedCacheManager will then provide the next frame to be fed to the DIL when requested via the variable nextBuffer.

[0132] Before being fed to the DIL, the NAL units are stored in LcevcVideoBufferQueue (LVBQ) objects, which are held inside LcevcMultiVideoBufferQueue (LMVBQ) instances, which can hold one or more LVBQ class instances simultaneously.

[0133] The number of LVBQ instances saved by LMVBQ is determined by the number of configuration files in the playlist manifest (TS / FMP4).

[0134] Every time new data is received, LMVBQ checks whether an LVBQ for the specific URL has already been created. If so, the data is added to the existing LVBQ; if not, another one is created, such as Figure 11 What is shown.

[0135] Each of these LVBQs holds a local instance of the corresponding cache (ByteRangeBasedCache for MP4 and imeBasedCacheManger for TS) and demuxer, and has a unique ID generated using the URL (handleVideoDataURLRequested contains the logic to create the ID). In this example, there can only be one active LVBQ at any given time, representing the profile currently playing on the screen.

[0136] Detection of ABR configuration file changes is done in Figure 9 and 10 In the example depicted in . In the latter process, some profiles correspond to the same video resolution. As previously mentioned, in the present disclosure, AVPlayer provides a notification when a profile switch is about to occur. Unfortunately, however, the notification does not indicate the exact point in time when the switch occurs, only that a change is about to occur. Figure 7-9As shown, in order to provide a more reliable way to know when such a switch actually occurs, three different situations are considered: the above-mentioned AVPlayer notification, the underlying resolution, and the track ID of the currently playing AVAsset.

[0137] When using resolution to determine if a profile has changed, simply check each extracted base frame and determine if the resolution matches the previously extracted resolution. If the resolution does not match, then the current LVBQ needs to be changed.

[0138] This procedure is Figure 7 . The AVPlayer 702 instance within the LCEVCPlayer layer outputs the decoded base frame 710 via the display link and the AVPlayer project video output. The base frame is passed to the media streaming service 715, where it is received at the resolution handler 713. It is then determined at step 717 whether the frame resolution of the base frame 710 is different from the previous base frame. If it is determined that the resolution is not different, the service accordingly determines that no ABR profile change has occurred 719. Alternatively, if a frame resolution difference 721 is detected, the service switches to the next queue. Subsequently, the multi-video buffer queue layer (including the ABR change handler 723) determines a match for the ABR profile, thereby instructing the video queue map 725 and activating the corresponding video queue 727.

[0139] In the case where the resolutions of the video substreams of the current stream are all different, changes can be detected by simply tracking the resolution changes. For example, this approach can be used given the following example manifest and corresponding substreams:

[0140] manifest.m3u8

[0141] -lcevc2.m3u8-1920x1080

[0142] -lcevc3.m3u8-1280x720

[0143] -lcevc4.m3u8-960x540

[0144] On the other hand, if the stream has at least two video substreams that share the same resolution, the track ID can additionally be used to detect changes. An example of such a set of substreams is:

[0145] manifest.m3u8

[0146] -lcevc2.m3u8-1920x1080-5K

[0147] -lcevc3.m3u8-1920x1080-3K

[0148] -lcevc4.m3u8-1280x720-1K

[0149] Because track IDs can change during both profile switching and seek operations, this standard alone cannot determine whether a profile has changed. For this task, it is also important to consider the aforementioned notifications that AVPlayer uses to notify that a profile change is imminent. Once such a notification is received, it is stored, and when the track ID changes, the profile is also changed. Typically, there is no relationship between the base image and the track ID. Instead, the track ID information is provided as part of the AVPlayerItem structure. However, generally speaking, the track ID is sent to the track ID change handler with each frame update.

[0150] Profile change detection via track ID identifier associated with the video stream Figure 8 As shown in FIG. As in the previous example, in this process, the output of the base frame 810 from the AVPlayer instance 802 is performed in a similar manner. It is then evaluated by the track ID change handler in the media streaming service layer 815. The track ID change handler 829 evaluates whether the track ID has changed compared to the track ID associated with the previous frame 831 based on the video associated with the received base frame 810 as the set track ID. If it is determined that no such change has occurred 832, it is also determined that the ABR profile has not changed.

[0151] Based on the register access log entry notification 833, a similar determination is made by the media streaming service 815 further monitoring the log entries therein and evaluating whether any new entries 835 signal an impending bitrate change 837. If no event is determined at step 839 via the ABR change event array, then a determination is made that no ABR change has occurred 832.

[0152] In the event that an ABR change event is determined to have occurred, or if it is determined at step 831 that the track ID has changed, then it is determined that the ABR profile has changed 841. A switch is then made to the next queue, and the ABR change handler 823 similarly initiates the matching of the appropriate video queue 827 and its activation.

[0153] Once you've determined when to change profiles, you need to select the next profile to change. When making this selection, two different scenarios are envisioned that could apply to this example. The first scenario involves a playlist containing only profiles with different resolutions, while the second scenario involves a playlist containing several profiles that share the same resolution.

[0154] The first scenario where all profiles have different resolutions is represented by Figure 9 The process is shown in . Once the manifest passes through WebProxyServer, MediaStreamService parses it and determines how many different profiles are included in the playlist. Using this information, when base frames with different resolutions are received, the next active LVBQ can be detected by comparing the resolutions and identifying the base frame that matches the resolution of the new base.

[0155] In the depicted example, decoded base frames 910 corresponding to the resolution shown are output from the AVPlayer instance 902. The resolution handler 913 analyzes the frame resolution and determines the resolution difference, and accordingly determines that the ABR profile has changed 940. Based on the available ABR profile parameter sets 941 obtained from the media stream 943 in the manifest file 945 in the media streaming service 915, it can be determined that all available corresponding ABR profiles are associated with different resolutions 947. Based on this, a match can be found 949 with an appropriate media stream having the correct resolution and switched to that media stream.

[0156] Some profiles have a second scene with the same resolution in Figure 10 As shown in the figure, when the underlying resolution changes, the list is first checked to see if there are multiple matching resolutions. If no multiple matches are found, the next active LVBQ can be selected similarly to the previous scenario. However, in the case of multiple matching allocation rates, some other factors must also be considered to find the next active queue.

[0157] Even though the underlying resolution has not changed when the second notification is detected, it is envisaged that more than one notification may be detected indicating that a profile change is to occur. In this case, a check is made as to which LVBQ contains the time range that includes the switch. It is accordingly selected as the next active queue. In the event of more than one match, the profile with the highest bitrate is selected. In the example shown, a check is made as to which LVBQ contains the time range that includes the switch. The LVBQ is not associated with any downloaded content and is associated with the MediaStreamService. In particular, a segment download listener is provided in this example, which is configured to store downloaded segment information for use in the functionality of finding the next queue. The queue identification logic for deciding on ABR changes and the next profile is provided within the MediaStreamsService.

[0158] like Figure 10As shown, the ABR decision handler 1051 analyzes the base frame 1010 with corresponding resolution, time, and track ID information as shown from the AVPlayer instance 1002. As described in conjunction with the previous example, a determination is made as to whether the resolutions of the available ABR profiles are all different 1047. If a positive determination is made and there is no degradation in the available resolutions, the resolution determined by the detected ABR change 1040 can be used to match the appropriate media stream 1053, as described previously.

[0159] However, if the determination is negative, meaning that at least two available ABR profiles correspond to the same resolution, further selection is necessary. First, it is advantageous to filter the available profiles by resolution 1055. If a single result 1057 is generated, the appropriate media stream and ABR profile 1053 can be clearly identified. Otherwise, if filtering 1055 results in multiple results, a further selection is made at step 1061 based on whether there are multiple matching events in the ABR change event array 1059. In this way, the multiple results filtered by resolution can be cross-referenced with the information in the aforementioned event log, and if a single matching event is generated at step 1061, the appropriate ABR profile parameters 1053 can be identified.

[0160] Alternatively, if further checking does not yield a single result, multiple matching segments can be further filtered based on time range at step 1063. The information based on which this determination is based can be obtained from the media streaming service 1015, where the manifest file 1052 provides the media stream 1054, from which the available resolutions 1056 can be obtained, based on which the filtering at step 1055 is performed. The video segments 1058 can then be used in the time range-based filtering at step 1063. The timing information can be used to find the correct segment for the player to subsequently play. For example, if the time is 27.7 seconds, then if the segments are regular and each segment is ten seconds long, in this step, a segment can be searched for in the range of 20.0-30.0.

[0161] Therefore, in the case of resolution degradation, multiple resolution matches, and multiple matching events causing ambiguity at this stage, the appropriate ABR profile can be found based on the matching of the time range 1053. In the case where multiple matching segments 1065 are identified, the segment corresponding to the highest bit rate is selected 1067 and determined to correspond to the appropriate ABR profile, which is switched at 1053 as described above.

[0162] Figure 13An example demonstrating cache cleanup is shown. In this case, data can only be received through requests from the AVPlayer instance. Therefore, a method for simulating the internal cache of AVPlayer is required. In order to understand when the cache needs to be pruned, the loadedTimeRange notification of AVPlayer is used, from which it can be determined when the time range loaded internally by AVPlayer changes.

[0163] The notification is then processed to find the safest drop ranges for fMP4 (findSafestBytaRanges) and TS (findSafestTimeRanges). The safest ranges are generated, usually with some margin added to the start and end of the ranges. Once this is done, all data outside the safest ranges is discarded. This is done safely because AVPlayer only keeps specific ranges in memory, which ensures that if part of the discarded stream needs to be played in the future (for example after a seek operation), AVPlayer will request that data again because it has not stored it. This means that the data will pass through the WebProxyServer and then through the demuxer (TS or MP4) again.

Claims

1. A method comprising: receiving, at a server application, a video stream, the video stream being encoded according to a multi-layer coding scheme; Passing the encoded elementary stream included in the video stream to an elementary decoder to obtain a decoded elementary stream; as well as Passing the encoded enhancement stream included in the video stream to an enhancement decoder for decoding, The original input video can be reconstructed from the decoded base stream and the decoded enhancement stream. 2 . The method according to claim 1 , wherein the base decoder is a native decoder of the iOS operating system.

3. The method of claim 1, wherein a decoder integration layer controls the operation of the enhancement decoder.

4. The method according to claim 1, further comprising receiving, by the server application, a request for a data asset sent by the base decoder, Sending the intercepted request to the data asset source; A data asset comprising the video stream is received by the server application in response to the request.

5. The method of claim 4, further comprising modifying the request before the server application sends the request to indicate the server application as a recipient of a response to the request.

6. The method of claim 1, further comprising instructing the elementary decoder to obtain the decoded elementary stream according to a predetermined set of bitrate parameters corresponding to the encoded elementary stream.

7. The method of claim 6, further comprising detecting a change in the predetermined set of bitrate parameters corresponding to a change in video bitrate.

8. The method of claim 7, wherein the detecting the change comprises detecting a bit rate parameter set change notification.

9. The method of claim 8, wherein the notification comprises notification data recorded in a log, and detecting the notification comprises monitoring or analyzing data recorded in the log.

10. The method of claim 7, comprising obtaining an indication of a time when the bit rate change occurs. The method of claim 10 , wherein the obtaining the indication comprises detecting a change in an identifier associated with the video stream.

12. The method of claim 10, wherein the obtaining the indication comprises identifying a difference in frame resolution between a first frame in the video stream and a second immediately subsequent frame in the video stream, and obtaining the indication based on a time at which the second frame occurs.

13. The method of claim 7, further comprising identifying a varying bitrate parameter set.

14. The method of claim 13 , wherein the changed bitrate parameter set is one of a plurality of bitrate parameter sets, and wherein identifying the changed bitrate parameter set is performed based on a determination of whether any one of the plurality of bitrate parameter sets corresponds to the same frame resolution as a frame resolution corresponding to another bitrate parameter set of the plurality of bitrate parameter sets.

15. The method according to claim 14, wherein If the determination is negative, identifying the changed bitrate parameter set includes comparing the frame resolution of the video stream frame appearing after the change with the frame resolution of each bitrate parameter set corresponding to one or more bitrate parameter sets of the multiple bitrate parameter sets, and determining that the changed bitrate parameter set is the one bitrate parameter set in the multiple bitrate parameter sets that corresponds to the frame resolution of the video stream frame.

16. The method of claim 14, wherein the first plurality of bitrate parameter sets comprises a second plurality of bitrate parameter sets, and if the determination is affirmative, identifying the changed bitrate parameter set by selecting, from the second plurality of bitrate parameter sets, a bitrate parameter set corresponding to a maximum bitrate among all bitrates corresponding to the second plurality of parameter sets.

17. The method of claim 16, wherein the first plurality of bitrate parameter sets comprises a third plurality of bitrate parameter sets, and wherein the second plurality of bitrate parameter sets are obtained by selecting from the third plurality of bitrate parameter sets those bitrate parameter sets that correspond to a time period containing an indication of when the bitrate change occurs.

18. A method according to claim 17, wherein the third plurality of bitrate parameter sets are obtained by selecting from the first plurality of bitrate parameter sets those bitrate parameter sets that correspond to a frame resolution that is the same as a frame resolution of video stream frames occurring after the change of the predetermined bitrate parameter set.

19. The method of claim 1, further comprising instructing a rendering module to render a video frame using the reconstructed first rendition at a relatively high quality level.

20. A non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause a device to perform the method of claim 1.

21. A device configured to: receiving, at a server application, a video stream, the video stream being encoded according to a multi-layer coding scheme; Passing the encoded elementary stream included in the video stream to an elementary decoder to obtain a decoded elementary stream; and Passing the encoded enhancement stream included in the video stream to an enhancement decoder for decoding, The original input video can be reconstructed from the decoded base stream and the decoded enhancement stream.

Images