Decoder integrated for layered video coding

Abstract

A video decoder is provided having one or more decoder plugins providing wrappers for one or more respective base decoders to implement a base decoding layer to decode an encoded video signal, each wrapper implementing an interface for data exchange with a corresponding base decoder, an enhancement decoder to implement an enhancement decoding layer, the enhancement decoder configured to receive an encoded enhancement signal and decode the encoded enhancement signal to obtain one or more layers of residual data generated based on a comparison of data derived from a decoded video signal and data derived from an original input video signal, and a decoder integration layer to control operation of the one or more decoder plugins and the enhancement decoder to generate a decoded reconstruction of the original input video signal using the one or more layers of residual data from the enhancement encoding layer, wherein the decoder integration layer provides a control interface for the video decoder. A system, method, and computer readable medium are also provided.

Description

Technical Field

[0001] This invention relates to a method for decoding signals, such as video signals, using a layered coding format, as well as a decoder and decoding system. More particularly, this invention relates to a video decoder for integrating a layered decoder, preferably an LCEVC decoder, into an application or client. Background Technology

[0002] Hybrid backward-compatible coding techniques have previously been proposed in, for example, WO 2014 / 170819 and WO 2018 / 046940 (the contents of which are incorporated herein by reference). Other examples of layer-based coding formats include ISO / IEC MPEG-5 Part 2 LCEVC (hereinafter referred to as "LCEVC"). LCEVC is described in WO 2020 / 188273A1 and related standard specification documents, the latter being the ISO / IEC DIS 23094-2 Low Complexity Enhanced Video Coding Draft, published at the MPEG 129 meeting held in Brussels from Monday, January 13, 2020 to Friday, January 17, 2020, both of which are incorporated herein by reference in their entirety.

[0003] In these encoding formats, the signal is broken down into multiple data "tiers" (also called "hierarchical layers"), each corresponding to a "quality level," ranging from the highest to the lowest sampling rate of the original signal. The lowest tier typically represents a low-quality reproduction of the original signal, while the other tiers contain information about corrections applied to the reconstructed reproduction to produce the final output.

[0004] LCEVC employs this multi-layered approach, where any underlying codec (e.g., Advanced Video Coding-AVC, also known as H.264, or High Efficiency Video Coding-HEVC, also known as H.265) can be enhanced via additional low-bit-rate streams. LCEVC is defined by two component streams: the underlying stream, which is typically decoded by a hardware decoder; and the enhancement stream, which consists of one or more enhancement layers suitable for software processing implementations with sustainable power consumption. These enhancements provide improved compression efficiency for existing codecs and reduce encoding and decoding complexity.

[0005] Because LCEVC and similar encoding formats fully utilize existing decoders and are inherently backward compatible, efficient and effective integration with existing video coding implementations is possible without a complete redesign. Examples of known video coding implementations include the software tool FFmpeg, used by the simple media player FFplay.

[0006] Furthermore, LCEVC is not limited to known codecs and can theoretically make full use of codecs yet to be developed. Therefore, any LCEVC implementation should be able to integrate with any known or yet to be developed codecs implemented in hardware or software without introducing coding complexity. Summary of the Invention

[0007] Various aspects and variations of the invention are set forth in the appended claims. Certain unclaimed aspects are further set forth in the following detailed description.

[0008] According to one aspect, a video decoder is provided, comprising: one or more decoder plug-ins that provide wrappers for one or more corresponding base decoders to implement a base decoding layer for decoding an encoded video signal, each wrapper implementing an interface for data exchange with the corresponding base decoder; an enhancement decoder configured to: receive an encoded enhancement signal; and decode the encoded enhancement signal to obtain one or more layers of residual data, the one or more layers of residual data being generated based on a comparison of data derived from the decoded video signal and data derived from the original input video signal; and a decoder integration layer configured to control the operation of the one or more decoder plug-ins and the enhancement decoder to generate a decoded reconstruction of the original input video signal using the decoded video signal from the base coding layer and the one or more layers of residual data from the enhancement coding layer, wherein the decoder integration layer provides a control interface for the video decoder.

[0009] Preferably, the enhancement decoder is an LCEVC decoder, such that the decoder integration layer, one or more plugins, and the enhancement decoder together provide an LCEVC decoding software solution. The LCEVC decoding software stack can be implemented in one or more LCEVC decoder libraries and thus provides an optimized software library for decoding MPEG-5 enhancement streams.

[0010] LCEVC decoding is very lightweight, often freeing up resources and matching or reducing battery power consumption compared to native base codec decoding. These aspects enable rapid deployment of LCEVC on all platforms, including implementations supporting different base codecs and decoders.

[0011] The decoder integration layer may also include operations for controlling an upscaling operation to upscale the decoded video signal from the base coding layer, such that the one or more layers of residual data can be applied to the decoded video signal from the base coding layer.

[0012] The decoder can be easily implemented on popular media players on platforms such as iOS (RTM), Android (RTM), and Windows (RTM).

[0013] One or more decoder plugins can be configured to instruct the corresponding underlying decoder via library function calls or operating system function calls. For example, depending on the operating system, function calls may include Android (RTM) mediacodec, VTDecompressionSession, and MFT. Therefore, it is easy to support different underlying decoding implementations, including native implementations within the operating system and hardware-accelerated decoding.

[0014] The decoder integration layer can be configured to apply one or more layers of residual data from the enhancement coding layer to the decoded video signal from the base coding layer to produce a decoded reconstruction of the original input video signal. In some cases, the decoder integration layer may instruct plugs from a set of decoder plugs to apply one or more layers of residual data; in other cases, the decoder integration layer may obtain the decoded output from the base coding layer using the instructions of the decoder plugs and combine this with the output of the enhancement decoder. Preferably, the residual data layer can be applied during playback.

[0015] In some embodiments, the decoder integration layer is configured to receive: one or more input buffers comprising, in coded order, coded video signals and coded enhancement signals, wherein one or more input buffers are also fed to the base decoder; and one or more base decoded frames of the decoded video signals from the base coding layer in presentation order. In this way, the client requires minimal processing, and the integration handles the client's operations. The same input buffers can be passed to both the base decoding layer and the enhancement decoding layer to further simplify the process.

[0016] In a particularly preferred embodiment, the control interface includes an output type configuration parameter, wherein the decoder integration layer is configured to change how the decoded reconstruction of the original input video signal is output based on the value of the output type configuration parameter. The value of the output type configuration parameter may be stored in a configuration data structure retrieved by the decoder integration layer after initialization.

[0017] In one example of configured output, the decoder integration layer is configured to output the decoded reconstruction of the original input video signal as one or more buffers. In another example, the decoder integration layer is configured to output the decoded reconstruction of the original input video signal as one or more on-screen surfaces. Alternatively, the decoder integration layer is configured to output the decoded reconstruction of the original input video signal as one or more off-screen textures. Each of these three example outputs can be selected by the output type configuration parameter.

[0018] When the output is selected as one or more off-screen textures, the control interface may include a display instruction, and the decoder integration layer can be configured to display the off-screen texture when it receives the display instruction. This is particularly useful when the client wants fine-grained management of the display timing of each frame and may want to keep the queue of decoded frames ready to be displayed at the correct time. For this use, a separate display function, namely the display instruction, is provided.

[0019] The control interface may include pipeline mode parameters, wherein the decoder integration layer is configured to control the stages of the enhancement layer to be executed on the central processing unit (CPU) or graphics processing unit (GPU) based on the values ​​of the pipeline mode parameters. For example, in one pipeline mode, all LCEVC stages may be executed in the CPU, while the GPU is only used for possible color component (e.g., YUV / RGB) conversions. Similarly, in another mode, most LCEVC stages, including color component (e.g., YUV / RGB) conversions, may be executed in the GPU using graphics library (GL) shaders, while the CPU may only be used to generate the LCEVC residual plane. The decoder configuration of this invention allows for efficient allocation of processing across CPU / GPU, and is configured for this purpose via the decoder integration layer.

[0020] The decoder integration layer can be configured to backtrack to pass the output of the base decoding layer as a decoded reconstruction of the original input video signal, where the encoded enhancement signal was not received. This is particularly advantageous because a video signal can still be output, albeit at a lower resolution than when the enhancement signal was successfully received.

[0021] The control interface may include frame skipping instructions, and the decoder integration layer may be configured to control operation in response to receiving a frame skipping instruction to not decode frames of the encoded augmented signal and / or not decode frames of the encoded video signal. When a client skips a frame, for example due to a search in the timeline or discards a frame due to frame 'delay', it can use an appropriate function to alert the decoder integration layer. If a skip instruction is received, the decoder integration layer falls back to a 'no operation' state. This alert can be used internally to perform minimal frame decoding to keep the reference decoder buffer consistent or to fall back to no operation.

[0022] One or more decoder plugins can provide a basic control interface to the basic decoder layer to call functions of the corresponding basic decoder. The plugins thus provide an application programming interface (API) to control operation and exchange information.

[0023] The control interface may include a set of predetermined encoding options, wherein the decoder integration layer is configured to retrieve a configuration data structure comprising a set of decoding settings corresponding to the predetermined set of decoding options. The configuration data structure may be retrieved by the decoder integration layer after initialization. Examples of decoding settings include: graphics library version (e.g., use of graphics library functions such as OpenGL major and minor versions or for embedded systems, such as OpenGL ES); bit depth, such as the use of an 8-bit or 16-bit LCEVC residual plane; use of hardware buffers; user interface (UI) configuration (e.g., enabling an on-screen UI for statistics and real-time configuration); and recording (e.g., enabling dumping of statistics and / or raw output frames to local storage).

[0024] In some embodiments, the decoder integration layer may be configured to receive, via a control interface, an indication of the mode in which the decoder integration layer should control the operation of one or more decoder plugins and enhancement decoders. In synchronous mode, the decoder integration layer may be configured to block calls to the decoding function until decoding is complete; and in asynchronous mode, the decoder integration layer may be configured to return after (e.g., immediately) a decoding function call and then call back when decoding is complete. Therefore, the decoder integration layer may optionally be used in synchronous or asynchronous mode by implementing the decoding function in either mode.

[0025] For client applications, the use of the decoder integration layer is simplified because the control interface operates at a relatively high level, has a small number of commands, and hides additional complexity. The control interface may include a set of functions that indicate the appropriate operational phase of the decoder integration layer, including one or more of the following: a creation function, in response to creating an instance of the decoder integration layer; a destruction function, in response to destroying an instance of the decoder integration layer; a decoding function, in response to the decoder integration layer controlling the operation of one or more decoder plug-ins and enhancement decoders to generate a decoded reconstruction of the original input video signal using one or more layers of residual data from the enhancement coding layer; an input feed function, which passes an input buffer including the encoded video signal and the encoded enhancement signal to the video decoder; and a callback function, which the decoder integration layer will call back in response to generating the decoded reconstruction of the original input video signal. The callback can be thought of as a registered alert indicating to the client that decoding is complete.

[0026] According to another aspect, a method for generating a decoded reconstruction of the original input video signal using a video encoder according to any of the above aspects may be provided, the method comprising: initializing an instance of a decoder integration layer; feeding an input including an encoded video signal and an associated encoded enhancement signal to a video decoder; instructing the decoder integration layer to generate the decoded reconstruction; and destroying an instance of the decoder integration layer. The method may be performed by a client or an application.

[0027] According to other aspects of the invention, the video decoder and method may be provided by a computer-readable medium including instructions that, when executed by a processor, cause the processor to perform the functionality of the video encoder or the steps of the method.

[0028] According to another aspect, a video decoding system may be provided, comprising: a video decoder according to any of the first aspects; one or more base decoders; and a client that provides one or more calls to the video decoder via a control interface to instruct the video decoder to generate a decoded reconstruction of the original input video signal. Combined with base decoders typically provided by an operating system, the video decoder described herein provides a complete solution from buffer output. Examples of one or more base codecs include, for example, AVC, HEVC, VP9, ​​EVC, AV1, and may be implemented in software or hardware common in the art.

[0029] According to another aspect of the invention, a decoder integration layer can be provided for controlling the operation of one or more decoder plugins and enhancement decoders to generate a decoded reconstruction of the original input video signal using one or more layers of decoded video signal from a base coding layer and residual data from an enhancement coding layer. The decoder integration layer provides a control interface for the video decoders, wherein one or more decoder plugins provide wrappers for one or more corresponding base decoders to implement the base decoding layer for decoding the encoded video signal, each wrapper implementing an interface for data exchange with the corresponding base decoder; and wherein the enhancement decoder implements an enhancement decoding layer configured to: receive the encoded enhancement signal; and decode the encoded enhancement signal to obtain one or more layers of residual data, said layers of residual data being generated based on a comparison of data derived from the decoded video signal and data derived from the original input video signal. In this manner, the decoder integration layer, along with a suitably configured client, communicates with a simple API.

[0030] In another illustrative aspect, this disclosure provides a decoder for decoding input data comprising multiple data layers in a hierarchical structure, wherein the multiple layers comprise base layer data and at least one enhancement layer data, wherein the at least one enhancement layer data is available to enhance the reproduction of the base layer data at a first quality level to an enhanced reproduction at a second quality level, the second quality level being higher than the first quality level. The decoder is characterized by comprising a basic functional arrangement (e.g., a base decoder) for processing the base layer data and an enhancement layer arrangement (e.g., an enhancement decoder) for processing at least one enhancement layer data. The decoder further comprises a plug-in system implemented in software to intersect between the basic functional arrangement and the enhancement layer arrangement, and an application layer arrangement (e.g., a functional layer) for executing one or more software applications executable on computing hardware to control the operation of the decoder. The decoder further comprises an orchestration unit (e.g., a decoder integration layer) for adapting or selecting one or more plug-ins of the plug-in system for use when transferring data between the enhancement layer arrangement and the basic functional arrangement. The orchestration unit, in operation, reconfigures the decoder via the plug-in system to at least adapt to changes in the operational characteristics of the basic functional arrangement and the enhancement layer arrangement.

[0031] The advantage of this is that combining the plug-in system with the orchestration unit allows the decoder to be reconfigured and adapted to changes in at least one of the basic function layout and enhancement layer layout.

[0032] Optionally, in the encoder, the orchestration unit monitors changes in the operational characteristics of the basic function layout and the enhancement layer layout while in operation, and reconfigures the plug-in system according to the changes in operational characteristics.

[0033] Optionally, in the decoder, orchestration units (e.g., decoder integration layers) are arranged in operation to apply test data to the basic function arrangement and enhancement layer arrangement to determine their operational characteristics, and to implement the selection or adaptation of the plug-in system based on the operational characteristics. More optionally, in the decoder, the selection or adaptation is implemented using at least one of the following: machine learning, artificial intelligence (AI) algorithms.

[0034] Optionally, the decoder includes a parsing unit for parsing the input data to divide the input data into layer data for basic function arrangement and at least one enhancement layer data for enhancement layer arrangement.

[0035] Optionally, in the decoder, the application layer layout can be updated with enhancements that provide additional functionality to be provided by the decoder.

[0036] Optionally, in the decoder, the basic functional layout implements a base layer decoder that conforms to industry-recognized coding standards.

[0037] According to another illustrative aspect, a method is provided for controlling a decoder for decoding input data (i.e., a method for controlling a decoder for decoding input data), the input data comprising multiple data layers in a hierarchical structure, wherein the multiple layers comprise base layer data and at least one enhancement layer data, wherein the at least one enhancement layer data is available to enhance the reproduction of the base layer data at a first quality level to an enhanced reproduction at a second quality level, the second quality level being higher than the first quality level, characterized in that the method comprises: (a) arranging the decoder to include a basic functional arrangement for processing the base layer data and an enhancement layer arrangement for processing at least one enhancement layer data; (b) arranging the decoder to include a plug-in system implemented in software to intersect between the basic functional arrangement and the enhancement layer arrangement and an application layer arrangement for executing one or more software applications executable on computing hardware for controlling the operation of the decoder; (c) arranging the decoder to include an orchestration unit for adapting or selecting one or more plug-ins of the plug-in system for use when transferring data between the enhancement layer arrangement and the basic functional arrangement; and (d) arranging the orchestration unit to reconfigure the decoder via the plug-in system during operation to at least adapt to changes in the operational characteristics of the basic functional arrangement and the enhancement layer arrangement. Attached Figure Description

[0038] Examples of systems and methods according to the invention will now be described with reference to the accompanying drawings, wherein:

[0039] Figure 1 A known high-level diagram illustrating the LCEVC decoding process;

[0040] Figure 2a and 2b A schematic diagram of a comparative basic decoder and a schematic diagram of a decoder integration layer in a video pipeline are shown respectively, according to examples of this disclosure;

[0041] Figure 3 A flowchart showing the API calls of the decoder integration layer;

[0042] Figure 4 A computer-readable medium demonstrating instructions for implementing a sample decoder integration layer; and

[0043] Figure 5 This is a schematic diagram of a decoder according to the present disclosure. Detailed Implementation

[0044] This disclosure describes implementations for integrating hybrid backward-compatible coding techniques with existing decoders, optionally via software updates. In a non-limiting example, this disclosure relates to implementations and integration of MPEG-5 Part 2 Low Complexity Enhanced Video Coding (LCEVC). LCEVC is a hybrid backward-compatible coding technique that is a flexible, adaptable, efficient, and computationally inexpensive coding format that combines different video coding formats, underlying codecs (i.e., encoder-decoder pairs, such as AVC / H.264, HEVC / H.265, or any other current or future codecs, and non-standard algorithms such as VP9, ​​AV1, etc.) with encoded data of one or more enhancement layers.

[0045] Example hybrid backward-compatible coding techniques use a downsampled source signal encoded with a base codec to form the base stream. Enhanced streams are formed using, for example, an encoded set of residuals from the base stream, either by increasing resolution or by improving frame rate correction. Multiple levels of enhanced data can exist in a hierarchical structure. In some arrangements, the base stream can be decoded by a hardware decoder, while the enhanced streams are suitable for processing using a software implementation. Thus, the stream is viewed as a base stream and one or more enhanced streams, where typically two enhanced streams may exist, but often only one is used. Notably, the base stream is typically decoded by a hardware decoder, while the enhanced streams are suitable for software processing implementations with appropriate power consumption.

[0046] Compared to the block-based approach used in the MPEG algorithm family, video frames are encoded in a layered manner. Layered frame encoding involves generating residuals for the entire frame, and then generating residuals for the reduced or extracted frames, etc. In the examples described in this paper, residuals can be considered as errors or differences at a specific quality level or resolution.

[0047] For context purposes only, since the detailed structure of LCEVC is known and described in the approved draft standard specification, Figure 1 The logical flow explains how LCEVC operates on the decoding side, assuming H.264 as the underlying codec. Those skilled in the art will understand how the examples described herein also rely on references. Figure 1 The general description of LCEVC presented applies to other multi-layer coding schemes (e.g., those using a base layer and enhancement layers). [Go to...] Figure 1The LCEVC decoder 10 operates at the individual video frame level. It takes as input the decoded low-resolution image from the base (H.264 or other) video decoder 11 and LCEVC enhancement data to produce a decoded full-resolution image ready to be displayed in the display view. The LCEVC enhancement data is typically received in the Supplemental Enhancement Information (SEI) of the H.264 Network Abstraction Layer (NAL) or in additional tracks or Data Packet Identifiers (PIDs) and separated from the base encoded video by the demultiplexer 12. Therefore, the base video decoder 11 receives the demultiplexed encoded base stream and the LCEVC decoder 10 receives the demultiplexed encoded enhancement stream, which is decoded by the LCEVC decoder 10 to produce a residual set for combination with the decoded low-resolution image from the base video decoder 11.

[0048] Additional PID means additional track or PID. This not only means transport stream (PID) but also ISO base media file format and WebM as container type.

[0049] Throughout this specification, the invention may be described in the context of NAL units. However, it should be understood that NAL units in this context may refer equivalently and more generally to a basic stream input buffer or its equivalent. That is, LCEVC is also capable of supporting non-MPEG underlying codecs that do not typically use NAL encapsulation, namely VP8 / VP9 and AV1. Therefore, when using the term NAL unit, the term may be read as meaning a basic stream input buffer depending on the underlying codec used.

[0050] LCEVC can be quickly implemented in existing decoders with software updates and is essentially backward compatible, as devices that have not yet been updated to decode LCEVC can play video using the basic codec, which further simplifies deployment.

[0051] In this context, this paper proposes a decoder implementation to integrate decoding and display with existing systems and devices that perform the underlying decoding. The integration is easy to deploy. It also enables support for a wide range of encoding and player vendors and can be easily updated to support future systems.

[0052] The proposed decoder implementation is available through an optimized software library for decoding MPEG-5 LCEVC enhanced streams, providing a simple yet powerful control interface or API. This allows developers the flexibility to deploy LCEVC at any level, from low-level command-line tools to software stacks integrated with common open-source encoders and players.

[0053] The terms LCEVC and enhancement are used interchangeably in this document. For example, an enhancement layer may include one or more enhancement streams, i.e., residual data of LCEVC enhancement data.

[0054] Figure 2a This describes the unmodified video pipeline 20. In this conceptual pipeline, acquired or received Network Abstraction Layer (NAL) units are input to the base decoder 22. Depending on the operating system, the base decoder 22 may be a low-level media codec accessed using mechanisms such as MediaCodec (e.g., found in Android (RTM) operating systems), VTDecompression Session (e.g., found in iOS (RTM) operating systems), or Media Platform Transformer (MFT - e.g., found in Windows (RTM) family of operating systems). The pipeline output is a surface 23 representing the decoded raw video signal (e.g., frames of this video signal, with the successful frames displayed sequentially in the video output).

[0055] Figure 2b This conceptually illustrates the proposed video pipeline using the LCEVC decoder integration layer. (Similar to...) Figure 2a In a comparative video decoder pipeline, NAL unit 24 is acquired or received and processed by LCEVC decoder 25 to provide surface 28 with reconstructed video data. By using LCEVC decoder 25, surface 28 can have a higher resolution than... Figure 2a The comparison surface 23 has a higher quality, or surface 28 may have the same quality as comparison surface 23 but require less processing and / or network resources.

[0056] As mentioned above, when NAL units are mentioned here, they refer to the basic stream input buffer or equivalent, depending on the underlying codec used.

[0057] Figure 5 Similar alternative descriptions of the examples described in this article will be provided below, and will be described in more detail using similar alternative terms.

[0058] exist Figure 2b In this context, the LCEVC decoder 25 is implemented in conjunction with the base decoder 26. The base decoder 26 can be provided by various sources and includes operating system functions as discussed above (e.g., using MediaCodec, VTDecompressionSession, or MFT interfaces or commands). The base decoder 26 can be hardware-accelerated, for example, using a dedicated processing chip to implement operations specific to a particular codec. The base decoder 26 can be used in conjunction with... Figure 2a The base decoder shown as 22 and used for other non-LCEVC video decoding may be the same as the base decoder, for example, it may include a pre-existing base decoder.

[0059] exist Figure 2b In this implementation, the LCEVC decoder 25 is implemented using a decoder integration layer (DIL) 27. The decoder integration layer 27 provides a control interface for the LCEVC decoder 25, enabling client applications to interact with it. Figure 2a The basic decoder 22 shown uses an LCEVC decoder 25 in a similar manner, for example, as a complete solution from buffer to output. The decoder integration layer 27 is used to control the operation of decoder plug-in (DPI) 27a and enhancement decoder 27b to produce a decoded reconstruction of the original input video signal. In some variations, such as... Figure 2b As shown, the decoder integration layer can also control GPU functions 27c, such as GPU shaders, to reconstruct the original input video signal from the decoded base stream and the decoded enhancement stream.

[0060] NAL units 24, including the encoded video signal along with associated enhancement data, can be provided in one or more input buffers. These input buffers can be fed from similar non-MPEG elementary stream input buffers, such as those used in VP8 / VP9 or AV1. The input buffers can be fed to (or made available to) the base decoder 26 and the decoder integration layer 27, particularly the enhancement decoder controlled by the decoder integration layer 27. In some instances, the encoded video signal may include the encoded elementary stream and be received separately from the encoded enhancement stream including the enhancement data; in other preferred instances, the encoded video signal including the encoded elementary stream may be received together with the encoded enhancement stream, for example, as a single multiplexed encoded video stream. In the latter case, the same buffer can be fed to (or made available to) both the base decoder 26 and the decoder integration layer 27. In this case, the base decoder 26 can retrieve the encoded video signal including the encoded elementary stream and ignore any enhancement data in the NAL units. For example, the enhanced data can be carried in the SEI message of the base stream used for video data. If the enhanced data is not suitable for processing custom SEI message data, then the SEI message can be ignored by the base decoder 26. In this case, the base decoder 26 can... Figure 2a The underlying decoder 22 operates, but in some cases, the underlying video stream may be at a lower resolution than in the comparison case.

[0061] Upon receiving the encoded video signal, which includes the encoded underlying stream, the underlying decoder 26 is configured to decode the encoded video signal and output it as one or more underlying decoded frames. This output can then be received or accessed by the decoder integration layer 27 for enhancement. In one example set, the underlying decoded frames are passed as input to the decoder integration layer 27 in presentation order.

[0062] Decoder integration layer 27 extracts LCEVC enhancement data from the input buffer and decodes the enhancement data. Decoding of the enhancement data is performed by enhancement decoder 27b, which receives the enhancement data from the input buffer as an encoded enhancement signal and extracts the residual data by applying the enhancement decoding pipeline to one or more streams of encoded residual data. For example, enhancement decoder 27b may implement an LCEVC standard decoder as described in the LCEVC specification.

[0063] Decoder plugins provide functions at the decoder integration layer to control the base decoder. In some cases, decoder plugin 27a can handle the reception and / or access of base decoded video frames, and preferably applies LCEVC enhancements to these frames during playback. In other cases, the decoder plugin can arrange the output of the base decoder 26 to be accessible by the decoder integration layer 27, which is then arranged to control the addition of residual outputs from the enhancement decoder to produce output surface 28. Once integrated into the decoding device, LCEVC decoder 25 enables the decoding and playback of video encoded with LCEVC enhancements. The display of the decoded and reconstructed video signal can be supported by one or more GPU functions 27c (e.g., GPU shaders controlled by decoder integration layer 27).

[0064] Generally, the decoder integration layer 27 controls the operation of one or more decoder plug-ins and enhancement decoders to produce a decoded reconstruction of the original input video signal 28 using one or more layers of decoded video signal from the base coding layer (i.e., as implemented by the base decoder 26) and residual data from the enhancement coding layer (i.e., as implemented by the enhancement decoder). The decoder integration layer 27 provides a control interface to the video decoder 25, for example, to an application within a client device.

[0065] Depending on the configuration, the decoder integration layer can output the decoded data to surface 28 in different ways. For example, as a buffer, as an off-screen texture, or as the top surface of the screen. The output format can be configured in the settings provided after creating an instance of the decoder integration layer 27, as explained further below.

[0066] In some implementations, if no enhancement data is found in the input buffer, for example, if NAL unit 24 does not contain enhancement data, the decoder integration layer 27 can back off to pass the video signal to the output at a lower resolution, i.e., the output of the base decoding layer implemented by base decoder 26. In this case, LCEVC decoder 25 can... Figure 2a The video decoder pipeline 20 operates.

[0067] Decoder integration layer 27 can be used for both application integration and operating system integration, for example, for use by both client applications and the operating system. Decoder integration layer 27 can be used to control operating system functions, such as function calls to the hardware-accelerated underlying codec, without requiring the client application to be aware of these functions. In some cases, multiple decoder plugins may be provided, where each decoder plugin provides a wrapper for a different underlying codec. It is also possible for a common underlying codec to have multiple decoder plugins. This could be in cases where different implementations of the underlying codec exist, such as a GPU-accelerated version, a native hardware-accelerated version, and an open-source software version.

[0068] When viewing Figure 2b In the schematic diagram, the decoder plugin can be considered as integrated with the base decoder 26 or alternatively as a wrapper around the base decoder 26. Effectively, Figure 2b It can be considered a visualization of stacking. Figure 2b The decoder integration layer 27 in the middle conceptually includes functionality 27b for extracting enhanced data from the NAL unit, functionality 27a for communicating with the decoder plugin and applying the enhanced decoded data to the base decoded data, and one or more GPU functions 27c.

[0069] The collection of decoder plugins is configured to present a common interface (i.e., a common set of commands) to the decoder integration layer 27, allowing the decoder integration layer 27 to operate without knowing the specific commands or functionality of each underlying decoder. Plugins thus allow underlying codec-specific commands, such as MediaCodec, VTDecompression Session, or MFT, to map to a set of plugin commands accessible by the decoder integration layer 27 (e.g., multiple different decoding function calls can be mapped to a single common plugin "Decode(...)" function).

[0070] Since the decoder integration layer 27 effectively includes a 'residual engine,' i.e., a library of correction plane sets at different quality levels generated from LCEVC via the encoded NAL unit, the layer can behave as a complete decoder (i.e., the same as decoder 22) by controlling the underlying decoder.

[0071] For simplicity, the entity referred to herein will be called the client, but it should be understood that the client can be considered as any application layer or functional layer, and the decoder integration layer 27 can be easily and readily integrated into the software solution. The terms client, application layer, and user are used interchangeably herein.

[0072] In application integration, decoder integration layer 27 can be configured to be displayed directly on a screen surface of any size (typically different from the content resolution) provided by the client. For example, even if the underlying decoded video is standard definition (SD), decoder integration layer 27 can use enhanced data to display the surface at high definition (HD), ultra-high definition (UHD), or custom resolutions. Further details regarding non-standard methods for upscaling and post-processing LCEVC decoded video streams that can be applied are found in PCT / GB2020 / 052420, the content of which is incorporated herein by reference. Example application integration includes the use of LCEVC decoder 25, for example, by ExoPlayer (an application-level media player for Android) or VLCKit (a target C wrapper for the libVLC media framework). In these cases, VLCKit and / or ExoPlayer can be configured to decode LCEVC video streams "behind the scenes" using LCEVC decoder 25, wherein the computer program code for VLCKit and / or ExoPlayer functionality is configured to use and invoke commands provided by decoder integration layer 27, i.e., the control interface of LCEVC decoder 25. VLCKit integration can be used to provide LCEVC display on iOS devices, and ExoPlayer integration can be used to provide LCEVC display on Android devices.

[0073] In operating system integration, decoder integration layer 27 can be configured to decode the buffer or draw on an off-screen texture of the same size as the final resolution of the content. In this case, decoder integration layer 27 can be configured such that it does not handle the final display to, for example, a display device. In these cases, the final display can be handled by the operating system, and therefore the operating system can use the control interface provided by decoder integration layer 27 to provide LCEVC decoding as part of the operating system call. In these cases, the operating system can implement additional operations regarding LCEVC decoding, such as YUV to RGB conversion, and / or resizing the destination surface before final display on the display device. Examples of operating system integration include integration with an MFT decoder (or backend) for Microsoft Windows (RTM) operating systems or with an Open Media Acceleration (OpenMAX-OMX) decoder (or backend), OMX being a set of C-language-based programming interfaces (e.g., at the kernel level) for low-power and embedded systems, including smartphones, digital media players, game consoles, and set-top boxes.

[0074] These integration modes can be set by the client device or application, and the mechanisms for selection and configuration will be described in more detail below.

[0075] Figure 2bThe configuration and use of the decoder integration layer allow LCEVC decoding and rendering to integrate with many different types of existing legacy (i.e., underlying) decoder implementations. For example, Figure 2b The configuration can be regarded as a... Figure 2a Modifications to the configuration can be found on computing devices. Other examples of integration include LCEVC decoding libraries available within common video encoding tools such as FFmpeg and FFplay. For example, FFmpeg is often used as the basic video encoding tool within client applications. By configuring the decoder integration layer as a plugin or patch for FFmpeg, an LCEVC-enabled FFmpeg decoder can be provided, allowing client applications to use the known functionality of FFmpeg and FFplay to decode LCEVC (i.e., enhanced) video streams. For example, an LCEVC-enabled FFmpeg decoder can provide video decoding operations such as playback, decoding of YUV and runtime metrics (e.g., Peak Signal-to-Noise Ratio - PSNR or Video Multi-Method Evaluation Fusion - VMAF) without first decoding the YUV. This can be achieved through plugin or patch computer program code provided by the decoder integration layer for calling FFmpeg functions.

[0076] As described above, in order to integrate the LCEVC decoder (e.g., 25) into the client (i.e., the application or operating system), the decoder integration layer (e.g., 27) provides a control interface or API to receive instructions and configuration and exchange information. Figure 3 This section describes example methods that use the set of main API calls. These API calls can be provided to instruct the operation of one implementation of the decoder integration layer. Of course, additional API calls may exist, but these five instructions indicate the simplicity of the relatively high-level API used for the LCEVC decoder.

[0077] At step 30, a first API call (e.g., "Create(...)") is used to create an instance of the decoder integration layer. The term "instance" is used here according to its normal usage in the field of computer programming; for example, an instance could be an instance of a defined software class or object. If successful, the call returns a success indication. When creating the instance, the decoder integration layer may retrieve a configuration data structure that includes a set of decoding settings. Details of the configuration system will be described below, but the decoder integration layer may not be able to initialize an instance of the layer if the settings are incorrect or not properly retrieved. The layer is initialized according to the values ​​of the settings specified in the configuration data structure.

[0078] Figure 3The corresponding API call (e.g., Destroy(...)) used in step 33 to subsequently destroy the instance of the decoder integration layer is also shown. This releases the memory once the received LCEVC stream or the accessed LCEVC file has been displayed.

[0079] The main phases of the decoder integration layer operation are provided by two types of API calls: one type is shown in Figure 3 In step 31, the input data is fed to the decoder integration layer (e.g., AddNALData(...)); and in another type, in step 32, the input data is decoded to produce output data for display (e.g., Decode(...)). In some embodiments, the input API call may inform the decoder integration layer how to handle the input buffer and certain important attributes of the input buffer. For example, this could be in NAL format or some necessary information for extracting augmented data from the input buffer, such as instructions for profiling the input buffer or retrieving attributes (including the presentation timestamp or the length of the buffer) from metadata.

[0080] Step 34 illustrates the fifth type of API call. This API call (e.g., `SetCallBack(...)`) can be used to register a callback from the decoder integration layer to the client, so that the client is notified by the decoder integration layer once a specific event has occurred. For example, a callback can be registered based on the output of the final (e.g., highest) resolution data from the LCEVC decoding process or a surface change event. The callback can be considered an alert from the decoder integration layer and can then be used by the client to perform post-processing simultaneously with or before the displayed decoded data. For example, a callback can allow the client to override the content resolution of the base decoding (e.g., LCEVC decoding may produce a surface at a higher resolution than the default base decoding, and this may need to be handled by the client). The client can register various callbacks, such as when the surface has changed or when decoding completes in asynchronous mode, as described below.

[0081] The API calls to the decoder integration layer can occur in different modes (synchronous or asynchronous), where each mode is signaled by using a specific API call to the decoding mode. It should be noted that, for each mode, instead of signaling these modes in the use of a specific function or API call, the selection of the decoding mode can be signaled in the configuration data structure retrieved after the initialization of the decoder integration layer.

[0082] In synchronous mode, API calls to the calling program (e.g., control interfaces to the decoder integration layer) are blocked until the entire decoding process is complete. In asynchronous mode, the call returns immediately, and then the decoder integration layer calls back the client's registered callback function when decoding is complete. Asynchronous mode allows the calling program (i.e., the client) to perform other operations while waiting for the decoded data.

[0083] As mentioned above, after creating an instance of the decoder integration layer, the layer can retrieve a data structure that includes a set of configuration settings set by the client or populated by default. The data structure can be a program code data structure, such as a C / C++ "structure," or a collection of passed values, such as a simple string, a JavaScript Object Notation (JSON) string, or any suitable structure of Extensible Markup Language (XML) data.

[0084] The decoder integration layer can be configured to work with different types of internal pipelines via a configuration setting. For example, a specific internal pipeline can control how the various stages of the decoding operation are performed. In one case, different types of internal pipelines can distribute computation across one or more central processing units (CPUs) and / or graphics processing units (GPUs). In one case, two types of internal pipelines can be provided. A first example type may involve CPU boot operations, where the LCEVC stages (e.g., all stages) are performed in the CPU of the computing device running the LCEVC decoder. The CPU boot mode can use only Single Instruction Multiple Data (SIMD) acceleration, such as an implementation based solely on the decoder plugin. For this first example type, the GPU can be used only for possible YUV / RGB conversion. The first example type may not use... Figure 2b GPU functionality 27c. The second example type may involve GPU boot operations, where one or more GPUs of a device running an LCEVC decoder perform the LCEVC phase (e.g., a majority or a predefined set). The second example type may use, for example... Figure 2b The second example type can use GL shaders that include YUV / RGB conversion, while the CPU is only used to generate LCEVC residual planes at various enhancement levels.

[0085] Various configurations can be set by passing or setting configuration data after creating an instance of the decoder integration layer. Other non-limiting examples of client configuration in the decoder integration layer include:

[0086] Which of the major and minor versions of OpenGL should be used (or the decoder integration layer can be configured to use auto-configuration for the highest supported version);

[0087] The use of OpenGL ES;

[0088] The use of an 8-bit LCEVC residual plane, for example, instead of a 16-bit one;

[0089] The use of hardware buffers, for example in Android (RTM);

[0090] Enable the on-screen UI for statistics and real-time configuration;

[0091] Enables the dumping of statistical data to local storage devices; and

[0092] Enables dumping the raw output frames to local storage.

[0093] As mentioned, in Figure 3 The five main types of API calls listed in the context are non-exhaustive. Additional calls (not shown) can be used to instruct the decoder integration layer to display the decoded reconstruction of the input video signal that has been stored as an off-screen texture. That is, by setting the output type to "Texture Output" after creating an instance of the decoder integration layer, the client can potentially decouple decoding (e.g., for off-screen textures) from display. This is particularly useful when the client wants fine-grained management of the display time of each frame and can potentially keep a queue of decoded frames ready to be displayed at the correct time.

[0094] Another API call (e.g., Skip(...)) can be defined and used to allow clients to instruct the decoder integration layer that processing of certain frames should be skipped. The call can instruct the decoder integration layer to skip the base layer's decoding of the encoded video signal or skip the enhancement layer's decoding of the enhancement data, i.e., to pass the input video signal decoded by the base decoding layer at low resolution as output.

[0095] In some cases, LCEVC streams can be configured to use a time-based mode for encoded data. In this scenario, if the LCEVC content encoding has time features, the LCEVC decoder may need to process each frame to maintain the correctness of its internal time reference data (e.g., as used by the current decoder plugin). Therefore, this skip call can be used by the client to alert the decoder integration layer when, for example, the client skips a frame due to a search in the timeline or drops a frame because of frame "delay." The decoder integration layer can then know whether the time features are enabled and can therefore fall back to a "no-operation" state. For example, the decoder integration layer can ensure proper processing in time-based mode through decoder plugins but provide "skip" behavior in the output provided to the client.

[0096] Figure 4The demonstration includes a non-transitory computer-readable storage medium containing instructions for at least one processor. These instructions may include functions as defined in computer program code, which may be integrated into application computer program code (e.g., as part of a library) and executed as API calls within the application computer program code. Defined API commands may form a control interface for integration into a decoder integration layer as described above. It is important to note that the client may be an application within the application layer, and the decoder integration layer may be integrated into the application.

[0097] Using initialization command 40, the client can create and populate a setup data structure. Using instance creation command 41, the client can create an instance of the decoder initialization layer. Instance creation command 41 can pass the setup data structure generated via initialization command 40. For example, the client can configure the decoder integration layer to use CPU mode for its internal pipeline and output it to a buffer.

[0098] After successful initialization, for example via commands 40 and 41, the decoder integration layer instance is now ready to receive input buffers. Clients can pass input buffers or input buffer data to the decoder integration layer instance via buffer disposal command 42. This command may include information about the decoder integration layer's NAL format and how to identify the enhanced encoded data from the input buffers (i.e., from the input buffer's metadata, attributes, or parsing).

[0099] And such as Figure 3 The decoding operations shown in the diagram, the decoder integration layer also provides a set of commands for general operations related to the base and / or LCEVC decoders. Figure 4 Within this framework, there exists a base frame processing command 43 that allows the client to instruct the decoder integration layer to process (i.e., control) the base frame processing generated from the decoded base video stream from the base decoder. In some cases, the data structure for the LCEVC frame can be defined by generating a picture data structure that inherits the data attributes of the base frame. This can be performed via a picture padding command 44. The LCEVC frame can be used as part of the decoding command that passes the defined decoder integration layer instance and an empty version of the data structure for the LCEVC frame; the decoding command then pads the data structure with the decoded LCEVC data.

[0100] Figure 4 Two additional commands are shown. A destruction command 45 can be defined, which can be used to destroy the instance of the decoder integration layer once the client has finished using the LCEVC decoder. A callback registration command 46 is also shown to register callbacks from the decoder integration layer to receive alerts when the surface has changed.

[0101] In some implementations described above, a decoder plugin corresponding to the base codec used for the encoded base video stream can be selected (via configuration data passed to the decoder integration layer and / or via the determination of the encoded video signal configuration data). This decoder plugin can be selected from multiple available decoder plugins, each forming an interface to a different base decoder implementation. The decoder plugin is configured to receive commands from the decoder and issue function calls to the base decoder (e.g., using specific commands for the determined base decoder implementation). When using a hardware-accelerated base decoder, the decoder plugin can make system calls to the operating system functions that provide this base decoding. The decoder integration layer also controls the enhancement layer decoding of the enhancement (i.e., LCEVC enhancement) data. This can be performed in parallel with the base decoding. Finally, the decoder integration layer controls the combination of the decoded base and enhancement data to produce a reconstructed output frame. The reconstruction of the video signal, such as the combination of the decoded base and residual data of the LCEVC data stream, can be performed by one or more of the decoder plugins, enhancement decoders, and the decoder integration layer.

[0102] It should be noted that in the case of software applications that provide various functions when executed on computing hardware, it is necessary to be able to modify the behavior of these functions for various reasons, including tuning or enhancing their performance. These functions are easily modified and include video and audio processing, metadata extraction, image processing, error correction, data security, detection and identification, etc. The pipeline described in this article addresses these challenges.

[0103] It should also be noted that the aforementioned decoder is advantageously configured to enhance the performance of underlying functionalities (e.g., feature sets) within the application layer with minimal or no disruption to the system, enabling the deployment of enhancements on already used systems. This advantage is achieved through a combination of a plug-in system and an enhancement integration layer.

[0104] In addition to controlling the decoding and reconstruction of the original input signal, at the decoder integration layer, it is advantageous to create multiple plugins for various functions of the basic functionality. This allows enhancements to be applied to selected functional subsets of the basic functionality simply by deploying new plugins to perform this creation. New plugins can also be developed to improve the performance of additional underlying systems provided in the decoder by enhancing them in appropriate locations.

[0105] Advantageously, capability extensions are optional, minor modifications that can be applied to the functional layer for:

[0106] (i) Fully utilize enhanced specific features and capabilities; or

[0107] (ii) Maintain both the given basic function X and the functionality of the enhanced basic function X available in the enhanced integration layer.

[0108] Without modifying the basic functionality, enhancements become proxies for both the basic functionality and the enhanced basic functionality.

[0109] When the decoder is modified, the enhanced integration layer, plug-in system, and basic functionality can be distributed separately at different times through instances of the relevant owners of the software components to be incorporated into the decoder. Advantageously, the additions provided to the decoder are not disruptive, but rather provide seamless improvements to the decoder's performance.

[0110] Additional functionalities that can be easily added to the decoder include at least one of the following: dynamic bit rate control, various upsampling functions, various downsampling functions, dynamic adaptive quantization (e.g., in the encoder), dynamic adaptive dequantization (e.g., in the decoder), various implementations of motion compensation, various high dynamic range color enhancements, and error correction.

[0111] Advantageously, in embodiments of the decoder, the enhancement and basic functions can be arranged to provide combined outputs, such as encoded or decoded video data.

[0112] Enhancements applied to the decoder may optionally be implemented using one or more software modules introduced or swapped within the decoder. For example, the one or more software modules advantageously include an orchestration (i.e., decoder integration) module and one or more core enhancement (i.e., enhanced decoder) modules.

[0113] The orchestration (i.e., decoder integration) module advantageously uses one or more machine learning algorithms (e.g., in the form of artificial intelligence, AI) to characterize the performance and functionality of one or more functions of the base function and the characteristics of the enhancement integration layer, for example, in terms of input / output requirements, execution performance (e.g., latency, compression efficiency, data format requirements, etc.), and selects and / or tunes the plug-in system accordingly based on such characterization. For example, such characterization optionally involves the orchestration (i.e., decoder integration) module applying test data to the user function and / or to the enhancement decoder layer to determine how it operates and whether its operation has been temporarily altered (e.g., due to software upgrades or the replacement of physical hardware). When a given computational task arises (e.g., image compression, object analysis in an image, image enhancement), the orchestration module advantageously selects how much computational task to implement in the base function and how much to implement through the enhancement decoder layer, so that the decoder operates as efficiently as possible. The orchestration module advantageously includes, for example, a priori taught deep learning neural networks to teach the configured plugins of the decoder's plugin system how to respond to various situations that arise within the basic functionality and / or enhancement integration layer, for example by selecting and / or configuring the plugins of the plugin system.

[0114] Another example following this one is illustrated in Figure 5 In this context, it should be understood that each of the features described herein is not mutually exclusive with the examples above, but may be composable or integrated. Elements may optionally possess similar functionality in cases where the terminology is similar but may differ.

[0115] An example of a decoder is shown; the decoder is generally indicated by 500. Decoder 500 includes, for example, basic functions 510 that implement the aforementioned base layer decoder. Optionally, basic functions 510 may include input signals S provided to decoder 500. in The decoder 500 comprises multiple distinct base layer decoders selected based on their properties. Optionally, the base functions 510 are implemented at least partially in hardware, for example, by employing a custom decoder integrated circuit or a field-programmable gate array (FPGA). The decoder 500 further includes an application layer 520 for managing specific functionalities, such as controlling the graphical user interface (GUI) for the decoder 500, performing object recognition in video data, video editing, etc. The application layer 520 is conveniently implemented using one or more software products executable on computing hardware. The application layer 520 is easily upgraded periodically with additional functionalities referred to as 'capability extensions', as indicated by 530; the application layer 520 can therefore be reconfigured over time to accommodate new requirements and functionalities.

[0116] Decoder 500 also includes an enhancement integration layer 540, which, for example, implements the enhancement layer decoder as described above. Enhancement integration layer 540 is advantageously implemented, at least in part, using one or more software products executable on computing hardware. However, it should be understood that as improved methods for enhancing the output from basic function 510 emerge, enhancement integration layer 540 may also need to be upgraded from time to time. In the event that both basic function 510 and enhancement integration layer 540 are temporarily changed through software upgrades, technical problems may arise regarding the compatibility between enhancement integration layer 540 and basic function 510.

[0117] To address these technical challenges, decoder 500 further includes a plug-in system 550 containing one or more software plug-ins that provide a flexible interface between basic functionality 510 and enhanced integration layer 540. Therefore, changes occur over time as modifications in plug-in system 150 adapt to adjustments made to basic functionality 510 and enhanced integration layer 540.

[0118] Therefore, in the case of software applications that provide various functions when executed on computing hardware, there is a need to be able to modify the behavior of those functions for various reasons, including adapting or enhancing their performance. These functions are easily modified and include video and audio processing, metadata extraction, image processing, error correction, data security, detection and identification, etc.

[0119] Typically, the application layer 520, such as a given program run by a given user, and the basic functionality 510 are tightly coupled and belong to the same given entity, such as a given software programmer or a given software company that builds the program. However, currently, this tight coupling no longer exists, and both open-source and proprietary software allow for additional functionality through various methods, including contributions to the decoder 500, plugins, and extensions. However, the functionality of the basic functionality 510 remains within the domain of its builder, who may not always be encouraged or able to improve its performance. This example addresses such issues concerning the maintenance and upgrades of the software related to the decoder 500.

[0120] During operation, the enhancement integration layer 540 improves the performance of one or more functions of the basic function 510 by modifying one or more of its inputs, outputs, and controls (e.g., via modification of the plug-in system 550). It should be understood that the enhancements provided in the enhancement integration layer 540 are optionally very similar to, and sometimes equivalent to, the enhanced functions of the basic function 510.

[0121] In other words, in decoder 500, a basic function plug-in is employed to provide an interface between enhancement integration layer 540 and basic function 510, ensuring that the enhancements provided thereby improve the performance of basic function 510, even if basic function 510 is unaware of the existence of the enhancements. The basic function plug-in is designed to dynamically link and load given functions of basic function 510 so that decoder 500 performs well even if the given functions of basic function 510 are absent. This approach provides a clear separation of operation between enhancements and basic function 510, which maintains the ability for enhancement integration layer 540 and basic function 510 to be separately allocated, priced, maintained, and updated, for example, by different vendors. At the decoder, for example, implemented in a client device or a client device decoding from a data repository, the methods and processes described herein can be embodied in code (e.g., software code) and / or data. The decoder can be implemented in hardware or software, as is well known in the field of data compression. For example, hardware acceleration using a specially programmed graphics processing unit (GPU) or a specially designed field-programmable gate array (FPGA) can provide some efficiency. For integrity purposes, such code and data may be stored on one or more computer-readable media, which may contain any means or medium capable of storing code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on the computer-readable medium, the computer system executes methods and processes embodied in the data structures and code stored within the computer-readable storage medium. In some embodiments, one or more steps of the methods and processes described herein may be executed by a processor (e.g., a processor of a computer system or a data storage system).

[0122] Generally, any of the functionalities described in this text or illustrated in the diagrams may be implemented using software, firmware (e.g., a fixed logic circuit system), programmable or non-programmable hardware, or a combination of these embodiments. Generally, as used herein, the terms "component" or "function" refer to software, firmware, hardware, or a combination of these. For example, in the case of a software embodiment, the terms "component" or "function" may refer to program code that performs a specified task when executed on one or more processing devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

Claims

1. A video decoder, comprising: One or more decoder plugins, each decoder plugin providing a wrapper for one or more corresponding base decoders to implement a base decoding layer for decoding encoded video signals, each decoder plugin providing a base control interface for the corresponding base decoding layer to call functions of each corresponding base decoder, each decoder plugin providing an application programming interface to control operations, and each wrapper implementing an interface for exchanging data with each corresponding base decoder. An enhancement decoder, which implements the enhancement decoding layer, is configured to: Receive encoded enhanced signals; and Decoding the encoded enhanced signal to obtain one or more layers of residual data, said one or more layers of residual data being generated based on a comparison of data derived from the decoded video signal and data derived from the original input video signal, and A decoder integration layer controls the operation of the one or more decoder plugins and the enhancement decoder to generate a decoded reconstruction of the original input video signal using the one or more layers of decoded video signal from the base coding layer and residual data from the enhancement coding layer. The decoder integration layer provides a control interface for the video decoder.

2. The video decoder of claim 1, wherein the one or more decoder plugins are configured to instruct the corresponding underlying decoder via library function calls or operating system function calls.

3. The video decoder of claim 1, wherein the decoder integration layer is configured to apply one or more layers of residual data from the enhanced coding layer to the decoded video signal from the base coding layer to produce the decoded reconstruction of the original input video signal.

4. The video decoder of claim 1, wherein the decoder integration layer is configured to obtain: Data from one or more input buffers, in coded order, comprising the coded video signal and the coded enhancement signal, wherein the one or more input buffers are further fed to the base decoder; and One or more base decoded frames of the decoded video signal from the base coding layer, presented in the order of presentation.

5. The video decoder of claim 1, wherein the control interface includes an output type configuration parameter, wherein the decoder integration layer is configured to change how the decoded and reconstructed original input video signal is output based on the value of the output type configuration parameter.

6. The video decoder of claim 1, wherein the decoder integration layer is configured to output the decoded reconstructed original input video signal as one or more buffers.

7. The video decoder of claim 1, wherein the decoder integration layer is configured to output the decoded reconstructed original input video signal as one or more on-screen top surfaces.

8. The video decoder of claim 1, wherein the decoder integration layer is configured to output the decoded reconstruction of the original input video signal as one or more off-screen textures.

9. The video decoder of claim 8, wherein the control interface includes a display instruction, and when the decoder integration layer receives the display instruction, the decoder integration layer is configured to display the one or more off-screen textures.

10. The video decoder of claim 1, wherein the control interface includes pipeline mode parameters, and wherein the decoder integration layer is configured to control the stages of an enhancement layer to be executed on a CPU or GPU based on the value of the pipeline mode parameters.

11. The video decoder of claim 1, wherein the decoder integration layer is configured to backtrack to pass the output of the base decoding layer as the decoded reconstruction of the original input video signal, wherein no encoded enhancement signal is received.

12. The video decoder of claim 1, wherein the control interface includes a frame skipping instruction and wherein the decoder integration layer is configured to control the operation to decode frames of the encoded enhanced signal and / or not decode frames of the encoded video signal in response to receiving the frame skipping instruction.

13. The video decoder of claim 1, wherein one or more decoder plugins provide a basic control interface to the basic decoder layer to call functions of the corresponding basic decoder.

14. The video decoder of claim 1, wherein the control interface includes a predetermined set of decoding options, and wherein the decoder integration layer is configured to retrieve a configuration data structure including a set of decoding settings corresponding to the predetermined set of decoding options.

15. The video decoder of claim 1, wherein the decoder integration layer is configured to receive, via the control interface, an indication of a mode in which the decoder integration layer should control the operation of the one or more decoder plugins and the enhancement decoder, wherein in synchronous mode, the decoder integration layer is configured to block calls to the decoding function until decoding is complete; and in asynchronous mode, the decoder integration layer is configured to return after calling the decoding function and to call back when decoding is complete.

16. The video decoder of claim 1, wherein the control interface includes a set of functions for indicating corresponding operational phases of the decoder integration layer, the set of functions including one or more of the following: A function is created in response to which an instance of the decoder integration layer is created; A destruction function that, in response, destroys the instance of the decoder integration layer; A decoding function, wherein the decoder integration layer responds to its control of the operation of one or more decoder plugins and the enhancement decoder to generate a decoded reconstruction of the original input video signal using the one or more layers of the decoded video signal from the base coding layer and residual data from the enhancement coding layer; A feed input function that passes an input buffer, including the encoded video signal and the encoded enhancement signal, to the video decoder; as well as A callback function that the decoder integration layer will respond to when the decoded reconstruction of the original input video signal is generated.

17. A method for generating a decoded and reconstructed original input video signal using a video decoder according to any one of claims 1 to 16, the method comprising: Initialize an instance of the decoder integration layer; The input, comprising an encoded video signal and an associated encoded enhancement signal, is fed to the video decoder. The video decoder includes one or more decoder plugins, each decoder plugin providing a wrapper for one or more corresponding base decoders to implement a base decoding layer for decoding the encoded video signal. Each decoder plugin provides a basic control interface for the corresponding base decoding layer to call functions of each corresponding base decoder. Each decoder plugin provides an application programming interface to control operation. Each wrapper implements an interface for exchanging data with each corresponding base decoder. The decoder integration layer is instructed to generate the decoded reconstruction; as well as Destroy the instance of the decoder integration layer.

18. A computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform the steps of claim 17.

19. A video decoding system, comprising: The video decoder according to any one of claims 1 to 16; One or more basic decoders; as well as A client, via the control interface, provides one or more calls to the video decoder to instruct the use of the video decoder to generate a decoded reconstruction of the original input video signal.

20. A decoder integration system for controlling the operation of one or more decoder plug-ins and enhancement decoders to generate a decoded reconstruction of the original input video signal using one or more layers of decoded video signal from a base coding layer and residual data from an enhancement coding layer, wherein the decoder integration system provides a control interface for the video decoder. Each of the one or more decoder plugins provides a wrapper for one or more corresponding base decoders to implement a base decoding layer for decoding the encoded video signal; each decoder plugin provides a basic control interface for the corresponding base decoding layer to call functions of each corresponding base decoder; each decoder plugin provides an application programming interface for controlling operation; and each wrapper implements an interface for exchanging data with each corresponding base decoder. The enhancement decoder implements an enhancement decoding layer, and the enhancement decoder is configured to: Receive the encoded enhanced signal; and The encoded enhanced signal is decoded to obtain one or more layers of residual data, the one or more layers of residual data being generated based on a comparison of data derived from the decoded video signal and data derived from the original input video signal.

Images