Encoder integrated for layered video coding

Abstract

A video encoder is provided having one or more encoder plugins providing wrappers for one or more respective base codecs to implement a base encoding layer, each wrapper implementing an interface for data exchange with a corresponding base codec, the base encoding layer configured to encode and decode video signals, an enhancement encoder to implement an enhancement encoding layer, the enhancement encoder configured to receive a decoded video signal from one of the one or more encoder plugins, the decoded video signal comprising a decoded version of an encoded video signal produced by the one of the one or more encoder plugins, and to produce an encoded enhancement signal for the encoded video signal, the encoded enhancement signal comprising one or more layers of residual data produced based on a comparison of data derived from the decoded video signal and data derived from an input video signal, and an encoder integration layer to control operation of the one or more encoder plugins and the enhancement encoder to produce an encoding of the input video signal using the base encoding layer and the enhancement encoding layer, wherein the encoder integration layer provides a control interface for the video encoder. A system, method, and computer readable medium are also provided.

Description

Technical Field

[0001] This invention relates to a method for encoding signals, such as video signals, using a layered coding format, as well as an encoder and encoding system. More particularly, this invention relates to a video encoder for integrating a layered encoder, preferably an LCEVC encoder, into an application or software stack. Background Technology

[0002] Hybrid backward-compatible coding techniques have previously been proposed in, for example, WO 2014 / 170819 and WO 2018 / 046940 (the contents of which are incorporated herein by reference). Other examples of layer-based coding formats include ISO / IEC MPEG-5 Part 2 LCEVC (hereinafter referred to as "LCEVC"). LCEVC is described in WO / 2020 / 188273A1 and related standard specification documents, the latter being the ISO / IEC DIS 23094-2 Low Complexity Enhanced Video Coding Draft, published at the MPEG 129 meeting held in Brussels from Monday, January 13, 2020 to Friday, January 17, 2020, both of which are incorporated herein by reference in their entirety.

[0003] In these encoding formats, the signal is broken down into multiple data "tiers" (also called "hierarchical layers"), each corresponding to a "quality level," ranging from the highest to the lowest sampling rate of the original signal. The lowest tier typically represents a low-quality reproduction of the original signal, while the other tiers contain information about corrections applied to the reconstructed reproduction to produce the final output.

[0004] LCEVC employs this multi-layered approach, where any underlying codec (e.g., Advanced Video Coding-AVC, also known as H.264, or High Efficiency Video Coding-HEVC, also known as H.265) can be enhanced via additional low-bit-rate streams. LCEVC is defined by two component streams: the underlying stream, which is typically decoded by a hardware decoder; and the enhancement stream, which consists of one or more enhancement layers suitable for software processing implementations with sustainable power consumption. These enhancements provide improved compression efficiency for existing codecs and reduce encoding and decoding complexity.

[0005] Because LCEVC and similar encoding formats fully utilize existing decoders and are inherently backward compatible, they require efficient and effective integration with existing video coding implementations without a complete redesign. Examples of known coding implementations include the software tool FFmpeg.

[0006] Furthermore, LCEVC is not limited to known codecs and can theoretically make full use of codecs yet to be developed. Therefore, any LCEVC implementation should be able to integrate with any known or yet to be developed codecs implemented in hardware or software without introducing coding complexity.

[0007] It should be noted that in some implementations of these encoding formats, both encoding and decoding operations can be performed by the underlying codec during video encoding. That is, the enhanced stream can be formed by comparing a version of the input signal with a decoded version of the encoded underlying stream. Therefore, in these implementations, it is necessary to efficiently and effectively instruct both encoding and decoding operations using the underlying codec at the encoder. Summary of the Invention

[0008] Various aspects and variations of the invention are set forth in the appended claims. Certain unclaimed aspects are further set forth in the following detailed description.

[0009] According to one aspect, a video encoder is provided, comprising: one or more encoder plugins that provide wrappers for one or more corresponding base codecs to implement a base coding layer, each wrapper implementing an interface for data exchange with the corresponding base codec, the base coding layer being configured to encode and decode a video signal; an enhancement encoder for implementing the enhancement coding layer, the enhancement encoder being configured to: receive a decoded video signal from one of the one or more encoder plugins, the decoded video signal including a decoded version of an encoded video signal generated by said one of the one or more encoder plugins, and generate an encoded enhancement signal for the encoded video signal, the encoded enhancement signal including one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal; and an encoder integration layer for controlling the operation of the one or more encoder plugins and the enhancement encoder to generate encoding of the input video signal using the base coding layer and the enhancement coding layer, wherein the encoder integration layer provides a control interface for the video encoder.

[0010] Preferably, the enhanced encoder is an LCEVC encoder, such that the encoder integration layer, one or more plugins, and the enhanced encoder together provide an LCEVC encoding software stack. The LCEVC encoding software stack can be implemented in one or more LCEVC encoder libraries and thus provides an optimized software library for encoding MPEG-5 enhanced streams.

[0011] Video encoders of this type are easy to deploy, meaning they can be supported by a wide range of encoding and player vendors or implemented to facilitate system updates. Various aspects of the invention provide the ability to deploy LCEVC at any level of a software stack with a range of implementation options, from low-level APIs to commonly used open-source encoders and players.

[0012] The encoder integration layer can be configured to: receive ownership of frames of the input video signal; indicate encoding of the frames via one or more encoding plugins and enhancement encoders; and provide a pointer to the location of the output data upon request via a control interface. In this way, the encoder integration layer allows the application layer to easily and simply retrieve the encoded data without interfering with the encoding process. In a preferred embodiment, along with the pointer, the encoder integration layer can provide a success indication via the control interface, allowing the user or application layer to identify that the data has been successfully processed and can be retrieved without complications. Additionally, the pointer can be provided within a data structure that further includes auxiliary information associated with the encoding of the input video signal. In some embodiments, only the data structure may be read. Advantageously, the data structure allows the application to avoid having to create it manually. The auxiliary information may contain all information about the encoded video frame, such as frame type, data length, quantization parameters used, timestamp information, configuration flags, or user data information.

[0013] The encoder integration layer may optionally be further configured to create data structures from frames of the input video signal for the type of frame to be encoded, the data structures including a set of planes that collectively constitute the frame; and to fill the planes with frame data from the frames of the input video signal. The planes of each frame may be, for example, YUV and are typically associated with the type of frame to be encoded. Therefore, the encoder integration layer provides an internal input mode where the application layer only needs to provide frames to the layer to perform encoding.

[0014] Alternatively, the encoder integration layer may be further configured to identify that a frame has been externally allocated and to retrieve a data structure including a set of planes that collectively constitute the input video signal and a set of parameters necessary for encoding the frame. Therefore, the application layer may also provide input to be externally allocated, allowing the user to specify frame padding. The data structure may lack parameters necessary for encoding the frame, and the encoder integration may fill such fields. The encoder integration layer may identify that a frame has been externally allocated during initialization. Examples of parameters necessary for encoding the frame include the number of planes, frame type, and timestamp information.

[0015] In some embodiments, the encoder integration layer may be configured to acquire a set of encoding attributes, and after the encoder integration layer is initialized, the initialization is confirmed to be successful via a control interface based on the encoding attributes. The encoding attributes may include, for example, resolution, frame rate, and bit rate, and may be included in an object sent to the control interface as part of the encoder implementation layer's initialization, thus preparing the encoder integration layer to receive frames.

[0016] The encoder integration layer can be configured to receive another set of encoding attributes and instruct the base encoder layer to be reconfigured using one or more encoder plugins. Reconfiguration can occur during video processing. In this way, encoding attributes, such as target bit rate or frame rate, can be dynamically reconfigured, with the base encoder also supporting dynamic reconfiguration.

[0017] In some embodiments, the control interface may provide a configuration system using a combination of read-only and writable attributes. This interface allows attributes accessible via application programming interface (API) calls to be efficiently transferred to the encoder integration layer.

[0018] In the first example of the configuration interface, the encoder integration layer is configured to: obtain a set of configuration default values; retrieve a configuration data structure including one or more configuration attributes; and override the configuration default values ​​in the set of configuration default values ​​corresponding to said one or more configuration attributes. That is, configuration default values ​​can be overridden using the corresponding configuration attributes. The configuration data structure can be a string or a JavaScript Object Markup (JSON) string transmitted via the control interface.

[0019] In the second example of the configuration interface, the control interface includes a set of predefined queries, where the encoder integration layer is configured to receive queries from the predefined query set and return the corresponding encoding or configuration attributes of the encoder integration layer. In this way, the encoding configuration attributes available for the base encoder and LCEVC enhancements are publicly available and can be queried via the API, i.e., the control interface. Queries can return arrays of all attribute groups, each containing an array of all attributes within that group. Alternatively, if the parameter name is known, then the parameter name can be queried directly.

[0020] In a third example of the configuration interface, the control interface includes a set of predefined metadata queries. The encoder integration layer is configured to receive queries for one or more metadata attributes, wherein the encoder integration layer is configured to retrieve metadata from one or more encoder plugins and return read-only metadata attributes corresponding to the one or more metadata attributes. This prevents the metadata from being externally set, but it can be accessed by the user or application layer via the control interface. The metadata may relate to a base code or enhanced encoder; for example, metadata attributes may contain global headers and shapes or the type of the underlying codec.

[0021] More preferably, the control interface may include a predefined set of metadata queries, wherein the encoder integration layer may be configured to receive queries for one or more metadata attributes, and the encoder integration layer may be configured to retrieve metadata from one or more encoder plugins and, in response to the query, return the location of a memory block storing the metadata. The block may be contiguous. In this way, a video encoder may have underlying encoder plugins to present metadata to the user without knowing its purpose. This method can be used to allow the encoder integration layer to report some internal values, or data that is not interpreted as attributes or cannot be expressed as attributes, to the user. For example, the H.264 global header may be accessed from memory by the user or the application layer.

[0022] Furthermore, the control interface can provide a set of predetermined optimization instructions, wherein the encoder integration layer can be configured to receive one or more of these predetermined instructions and optimize the operation of one or more encoder plug-ins and enhance the encoder according to predefined variations corresponding to the received instructions. The control interface can thus provide the user with a mechanism to 'tune' a set of predetermined parameters for encoding, for example, according to encoding objectives. In some other examples, encoding can be optimized for peak signal-to-noise ratio (PSNR) to achieve a target without the control interface specifying particular adjustments to those parameters.

[0023] According to another aspect, a method for encoding a video signal can be provided. The method may utilize a video encoder according to any of the above aspects. Preferably, the method provides steps for encoding a stream using an encoder integration layer as described above. The method may include: initializing the encoder integration layer; when a frame to be encoded of an input video signal is present: retrieving the input frame of the input video signal and transferring ownership of the frame to the video encoder via a control interface; waiting for the video encoder to encode the input frame and subsequently extracting output data; and releasing ownership of the output data back to the encoder integration layer and shutting down the encoder integration layer.

[0024] In some embodiments, the method for integrating an encoder integration layer according to any of the above aspects may include the following steps: creating an encoder integration layer; initializing the encoder integration layer; passing an input video signal to the encoder integration layer; extracting the encoded bitstream from the encoder integration layer; refreshing the encoder integration layer later; and destroying the encoder integration layer.

[0025] According to other aspects of the invention, the video encoder and method may be provided by a computer-readable medium including instructions that, when executed by a processor, cause the processor to perform the functionality of the video encoder or to carry out the steps of the method.

[0026] According to another aspect, a video encoding system may be provided, comprising: a video encoder according to any of the foregoing aspects; one or more base codecs; and an application layer that provides one or more calls to the video encoder via a control interface to instruct video encoding. The application layer may also be considered a functional layer and may be provided at any level of the software stack. The one or more base codecs may be associated with encoder plugins that provide wrappers for the base codecs as described in any of the foregoing aspects. Examples of the one or more base codecs include, for example, AVC, HEVC, VP9, ​​EVC, AV1, and may be implemented in software or hardware common in the art.

[0027] In some implementations, the application layer or functional layer can be configured to: initialize the encoder integration layer via the control interface of the first aspect of the invention; when there is an input frame of the input video signal to be encoded, retrieve the input frame of the input video signal and transfer ownership of the frame to the video encoder via the control interface; wait for the video encoder to encode the input frame and subsequently extract the output data; and release ownership of the output data back to the encoder integration layer and shut down the encoder integration layer. In this way, the encoder integration layer, together with a suitably configured functional layer, communicates with a simple yet powerful API.

[0028] According to another aspect of the invention, an encoder integration layer can be provided that controls the operation of one or more encoder plugins and an enhancement encoder to generate an encoding of an input video signal using a base coding layer and an enhancement coding layer. The encoder integration layer provides a control interface for a video encoder, wherein the one or more encoder plugins provide wrappers for one or more corresponding base codecs to implement the base coding layer, each wrapper implementing an interface for data exchange with the corresponding base codec, the base coding layer being configured to encode and decode the video signal; and the enhancement encoder implements an enhancement coding layer, the enhancement encoder being configured to: receive a decoded video signal from one of the one or more encoder plugins, the decoded video signal comprising a decoded version of the encoded video signal generated by said one of the one or more encoder plugins, and generate an encoded enhancement signal for the encoded video signal, the encoded enhancement signal comprising one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from the input video signal.

[0029] The encoder integration layer can also provide a control interface to one or more plug-ins, allowing additional wrappers for the base codec to be integrated with the encoder integration layer.

[0030] According to another illustrative aspect, an encoder is provided for encoding input data, wherein the input data is encoded into multiple data layers in a layered structure, wherein the multiple layers include base layer data and at least one enhancement layer data, wherein the at least one enhancement layer data is available to enhance the reproduction of the base layer data at a first quality level to an enhanced reproduction at a second quality level, the second quality level being higher than the first quality level. The encoder is characterized by comprising a basic functional arrangement (e.g., a basic codec or encoder / decoder) for generating the base layer data and an enhancement layer arrangement (e.g., an enhancement encoder) for generating at least one enhancement layer data. The encoder further comprises a plug-in system (e.g., one or more encoder plug-ins) implemented in software to intersect between the basic functional arrangement and the enhancement layer arrangement, and an application layer arrangement for executing one or more software applications executable on computing hardware for controlling the operation of the encoder. The encoder further comprises an orchestration unit (e.g., an encoder integration layer) for adapting or selecting one or more plug-ins of the plug-in system for use when transferring data between the enhancement layer arrangement and the basic functional arrangement; and wherein the orchestration unit, in operation, reconfigures the decoder via the plug-in system to at least adapt to changes in the operating characteristics of the basic functional arrangement and the enhancement layer arrangement.

[0031] Optionally, in the encoder, the orchestration unit (e.g., encoder integration layer) monitors changes in the operational characteristics of the basic function layout and enhancement layer layout while in operation, and reconfigures the plug-in system (e.g., one or more encoder plug-ins) according to the changes in operational characteristics.

[0032] Optionally, in the encoder, the orchestration unit (e.g., encoder integration layer) is arranged in operation to apply test data to the basic function arrangement and enhancement layer arrangement to determine its operational characteristics, and the selection or adaptation of a plug-in system (e.g., one or more encoder plug-ins) is implemented based on the operational characteristics. More optionally, in the encoder, the selection or adaptation is implemented using at least one of the following: machine learning, artificial intelligence (AI) algorithms. Optionally, for artificial intelligence (AI) algorithms, one or more recurrent neural networks (RNNs) are employed; a recurrent neural network (RNN) is a class of deep neural networks created by recursively applying the same given set of weights to a structured input to produce structured predictions or scalar predictions of a variable-sized input structure by traversing a given structure in topological order. For example, RNNs have been successful in learning sequences and tree structures in natural language processing. It should be understood that, in general, APIs have a predetermined structure; if the compiler can understand the API, then the RNN can also learn to adapt to the API.

[0033] Optionally, in the encoder, the application layer layout can be updated with enhancements that provide additional functionality to be provided by the decoder.

[0034] Optionally, in the encoder, the basic functional layout implements a base layer decoder that conforms to industry-recognized coding standards (e.g., AVC, H264, H265, MPEG2, MPEG4, MPEG5, LCEVC, VC-6, etc.).

[0035] According to another illustrative aspect, a method is provided for controlling the operation of an encoder for encoding input data (i.e., a method for controlling the operation of an encoder for encoding input data), wherein the input data is encoded into a plurality of data layers in a layered structure, wherein the plurality of layers include base layer data and at least one enhancement layer data, wherein the at least one enhancement layer data is available for enhancing the reproduction of the base layer data at a first quality level to an enhanced reproduction at a second quality level, the second quality level being higher than the first quality level, characterized in that the method further comprises: (a) arranging the encoder to include a basic functional arrangement (e.g., a base encoder or encoder / decoder) for generating base layer data and for generating at least one (a) an enhancement layer arrangement for enhancement layer data (e.g., an enhancement encoder); (b) an encoder arrangement to include a plug-in system (e.g., one or more encoder plug-ins) implemented using software to interface between the basic functional arrangement and the enhancement layer arrangement, and an application layer arrangement for executing one or more software applications executable on computing hardware for controlling the operation of the encoder; (c) an encoder arrangement to include an orchestration unit (e.g., an encoder integration layer) for adapting or selecting one or more plug-ins of the plug-in system for use when transferring data between the enhancement layer arrangement and the basic functional arrangement; and (d) an orchestration unit to reconfigure the decoder via the plug-in system during operation to at least accommodate changes in the operating characteristics of the basic functional arrangement and the enhancement layer arrangement. Attached Figure Description

[0036] Examples of systems and methods according to the present invention will now be described with reference to the accompanying drawings, in which:

[0037] Figure 1 A known high-level diagram illustrating a multi-layer coding process;

[0038] Figure 2 A schematic diagram illustrating a software stack according to an example of this disclosure;

[0039] Figure 3 A flowchart illustrating the functionality of the encoder integration layer; and,

[0040] Figure 4 A flowchart illustrating the use of the encoder integration layer. Detailed Implementation

[0041] This disclosure describes implementations for integrating hybrid backward-compatible coding techniques with existing encoders and decoders, optionally via software updates. In a non-limiting example, this disclosure relates to implementations and integration of MPEG-5 Part 2 Low Complexity Enhanced Video Coding (LCEVC). LCEVC is a hybrid backward-compatible coding technique that is a flexible, adaptable, efficient, and computationally inexpensive coding format that combines different video coding formats, underlying codecs (i.e., encoder-decoder pairs) (e.g., AVC / H.264, HEVC / H.265, or any other current or future codecs, and non-standard algorithms such as VP9, ​​AV1, etc.) with encoded data at one or more enhancement layers.

[0042] Example hybrid backward-compatible coding techniques use a downsampled source signal encoded with a base codec to form the base stream. Enhanced streams are formed using, for example, an encoded set of residuals from the base stream, either by increasing resolution or by improving frame rate correction. Multiple levels of enhanced data can exist in a hierarchical structure. In some arrangements, the base stream can be decoded by a hardware decoder, while the enhanced streams can be adapted for processing using a software implementation. Thus, the stream is viewed as a base stream and one or more enhanced streams, with two enhanced streams typically present. Notably, the base stream is typically decoded by a hardware decoder, while the enhanced streams are adapted for software processing implementations with appropriate power consumption.

[0043] Compared to the block-based approach used in the MPEG algorithm family, this method encodes video frames in a layered manner. Layered frame encoding involves generating residuals for the entire frame, and then generating residuals for the reduced or extracted frames, etc.

[0044] For context purposes only, since the detailed structure of LCEVC is known and stated in the approved draft standard specification, Figure 1 This section explains how LCEVC operates on the encoding side, assuming H.264 as the underlying codec. Those skilled in the art will understand how the examples described herein are also based on reference [reference / codec]. Figure 1 The general description of LCEVC presented applies to other multi-layer coding schemes (e.g., those using a base layer and enhancement layers). [Go to...] Figure 1 The scaled-down input is H.264 encoded at a lower resolution (typically one-quarter of the desired output resolution). The output of the H.264 encoded input, referred to in this paper as the base stream, can be supplemented with additional LCEVC data. This LCEVC data can be added in the Supplemental Enhancement Information (SEI) of the H.264 Network Abstraction Layer (NAL) or in an additional data stream identified using a Packet Identifier (PID). Figure 1In this process, the uncompressed full-resolution video is fed to the scaler 11 before being passed to the H.264 encoder 10. The LCEVC encoder 13 generates an enhancement stream from the two inputs: the H.264 (i.e., the base layer) encoding and the original uncompressed full-resolution video, thereby effectively filling the quality gap between the two inputs. The enhancement stream (e.g., encapsulated in an SEI message) and the base stream (e.g., constructed using VCL NAL as a non-enhanced stream) are then combined into an output encoded bitstream.

[0045] To fully leverage the benefits of LCEVC, it is advantageous to combine the LCEVC encoder with the underlying codec used for the supported base codecs. Therefore, this paper presents an encoder implementation and software stack to achieve this. The encoder implementation and software stack provide an optimized software library for encoding and decoding MPEG-5 LCEVC enhanced streams, offering a simple yet powerful API. This allows developers the flexibility to deploy LCEVC at any level of the software stack, offering a range of options from low-level APIs to implementations in commonly used open-source encoders and players.

[0046] Figure 2 The proposed example software stack 20 is conceptually illustrated. At the center of the stack is an encoder integration layer 21, which controls the operation of one or more encoder plug-ins 22 and enhancement encoders 23 to produce encodings of the input video signal using the base coding layer and enhancement coding layers. The encoder integration layer 21 provides a control interface for the video encoders. That is, the encoder integration layer 21 provides an API, which can be called from the application layer or functional layer 24, to indicate encoding query attributes or metadata, and provides a configuration system. The encoder integration layer 21 combines the base encoder and enhancements to orchestrate combined behavior. Furthermore, the layer is architected to support any base codec implementation by making it a generic and independent plug-in system. In other words, the encoder integration layer 21 integrates the user's expected functionality with the LCEVC and base coding layers.

[0047] In a preferred embodiment, the enhancement encoder 23 is an LCEVC encoder. The terms LCEVC encoder and enhancement encoder will be used interchangeably throughout. The LCEVC encoder 23 implements an enhancement coding layer and receives a decoded video signal from one of the encoder plugins, the decoded video signal comprising a decoded version of an encoded video signal controlled by the one or more encoder plugins, and then generates an encoded enhancement signal for the encoded video signal. The encoded enhancement signal includes one or more layers of residual data generated based on a comparison of data derived from the decoded video signal and data derived from the input video signal.

[0048] In the examples described herein, residuals can be viewed as errors or differences at a specific quality level or resolution. Further details are provided, for example, in the description of LCEVC cited herein.

[0049] In addition to controlling the operation of the base coding layer and the enhancement coding layer, the encoder integration layer 21 also provides a configuration system. Configuration attributes can be set by the functional layers and are supported by a comprehensive built-in system that optimizes parameter default values. The encoder integration layer 21 employs ideal settings for both the LCEVC encoder 23 and the base encoder 27 based on key parameters passed to it by the application, such as resolution, bit rate, and 'tuning'. Tuning will be described in more detail below.

[0050] One or more encoder plugins 22 provide wrappers for one or more corresponding base codecs 27 to implement a base coding layer. Each wrapper implements an interface for exchanging data with the corresponding base codec 27, which is configured to encode and decode video signals (i.e., by means of the base codec 27).

[0051] Functional layer 24 may be implemented in reference source code 25, and enhanced encoding software stack 20, including plug-in 22, enhanced encoder 23 and encoder integration layer 21, may be implemented as a collection of encoder libraries 26.

[0052] In this document, the functional layer 24 may be interchangeably referred to as the application layer or the user layer, as the functionality may be specifically encoded by the user depending on the required implementation. The functional layer 24 includes user-configurable operations for instructing the encoder integration layer 21, managing functionality, and providing calls to the API provided by the encoder integration layer 21 to, for example, query attributes or set configurations for enhanced coding.

[0053] As described above, functional layer 24 can be implemented at any level of the software stack. An example of an implementation of the functional layer is the video encoding tool FFmpeg. FFmpeg is a popular tool among video developers. In one example, the functional layer could be provided by FFmpeg functionality and a libav modification. In some cases, additional applications can be implemented above functional layer 24; for example, a client application could invoke video encoding functions provided by FFmpeg, which in turn invokes the control interface provided by the encoder integration layer 21 to provide enhanced encoding.

[0054] Figure 3A sample flowchart illustrating the steps for encoding a stream using an encoder integration layer is provided. In this example, the reference to "encoder" refers to an enhanced encoder implemented by software stack 20. In the example implementation, the steps are performed by a functional layer or an application layer. First, (step 30) the encoder integration layer, which forms the control interface for the encoder, is initialized. When a frame to be encoded exists (step 31), the following steps are performed: an input frame is obtained (step 32) and ownership of the input frame is transferred to the encoder, i.e., to the encoder integration layer (step 33); encoder contention is waited for (step 34) and then the encoded frame is extracted, for example, in the form of an encoded bitstream (step 35); and ownership of the output frame is released back to the encoder (step 36), for example, the memory allocated at the encoder for encoding the input frame and producing the encoded frame is released. Once no frames are available to the encoder, the encoder integration layer is shut down (step 37).

[0055] The input frame at step 32 may be obtained from an external source, such as being passed as data to the encoder integration layer, or it may be obtained from an internal source, such as being (directly) accessed from a referenced memory location. As described in more detail below, the input frame may be passed as a "picture" object. In this case, transferring ownership of the input frame may include transferring ownership of the picture object to a process executed as part of the encoder integration layer. Generally, "ownership" refers to control over the memory location where the frame data is stored (e.g., in random access memory RAM or cache memory for the processing unit, whether central or graphical). In one case, ownership of the picture object may first (temporarily) be transferred to an application in the functional layer or application layer to populate the picture object's data before executing step 33 (i.e., before ownership is transferred to the encoder integration layer for encoding). Although the encoder has ownership of the picture object, the application in the functional layer or application layer may not be able to manipulate the object. In the same way, releasing ownership may mean that the encoder relinquishes control of the picture object, making it possible for the application to manipulate it again, for example, to load it into the data of a subsequent frame.

[0056] Figure 4 This section describes example steps on how the encoder integration layer can be integrated into a software stack. Figure 4 The steps may include Figure 3 A specific implementation of a more general method.

[0057] At step 40, an instance of the encoder integration layer is first created. This can be done using a defined function call (e.g., an object instance creation (or "open") method in a programming language such as C / C++). The instance of the encoder integration layer can provide a handle to the encoder integration layer, which can be passed to other functions. The settings for opening and initializing the encoder integration layer can be defined as a data structure, such as a settings object. The settings may include, for example, the base encoder to be used, user data for the base encoder (e.g., non-default parameters), and logging parameters (e.g., file paths for log files). The user data may contain data to be passed to the base encoder via plugins. The settings object can be passed to the defined function call when the instance is created. In some implementations, the settings object can be initialized to a default state.

[0058] In some variations, an object instance creation function may be provided with a version of the encoder integration layer (i.e., the encoder API version) appended to the function name to ensure that a newer or older version of the encoder integration layer with interrupt changes is not misused. If the requested underlying encoder cannot be found and instantiated or opened, then an instance of the encoder integration layer may not be created (e.g., the object instance cannot be "opened"). The creation of an instance of the encoder integration layer creates a context that can be passed to other functions (i.e., a set of data that allows the execution of a process, including, for example, processor registers, reserved process memory, and control registers to manage tasks). Associated with the object instance creation function, a corresponding object instance destruction function (e.g., a "close" method) may also be defined. This can delete the instance and close or remove the context. As stated in step s45 below, the instance can be destroyed when the object instance destruction is invoked.

[0059] At step 41, the encoder integration layer is initialized (i.e., the instance of the encoder integration layer generated at step 40). Again, here, a settings object can be used to initialize various encoding parameters and attributes. For example, a settings object can be used or created to specify encoding attributes, such as resolution, frame rate, bit rate, transform type (e.g., a 2x2 or 4x4 Hadamard transform as defined within LCEVC), color format (e.g., YUV), and group of pictures (GOP) length (in frames). Based on the settings object, the encoder integration layer can determine the GOP configuration, such as a configuration where only the first frame is an I-frame (intra-frame) or a configuration where every frame is an I-frame. In some embodiments, encoding attributes may be in the form of a JSON object string including encoder options. The settings may also include flags indicating whether an externally allocated picture should be used, i.e., whether the plane of the input image should be allocated through the integration or encoder integration layer, as described elsewhere. If the color format is not as expected, then in one example, encoder initialization may convert the input to a suitable format, such as YUV.

[0060] Encoding properties can be set as part of a powerful configuration system provided by the encoder integration layer's interface. That is, in addition to the base frame information set during the encoder integration layer's opening or initialization, configuration properties can also be set via a JSON configuration string in the initialization settings data structure. Of course, it will be understood that JSON is merely one example of a data structure that can be used to pass configuration settings to the encoder integration layer using the control interface.

[0061] The configuration system can be supported by a comprehensive built-in system with optimized parameter default values. The encoder integration layer employs preferred settings for both the LCEVC encoder and the base encoder based on key parameters passed to it by the application, such as resolution, bit rate, and tuning. Tuning parameters will be discussed in more detail below.

[0062] Properties specified within an initialization settings data structure (such as a JSON string) can override any settings provided by default or available in a configuration database accessible by the encoder integration layer. Therefore, it is important that applications specify only the properties that the user has explicitly set within the JSON string. For example, in a command-line program, the JSON string should only contain options specified by the user on the command line; or, in a graphical user interface (GUI), only options that the user has manually selected should be included in the JSON string.

[0063] After successful initialization, the user or application layer can continue to encode image input (e.g., YUV input) using the instance of the encoder integration layer that was created and initialized (hereinafter referred to as the "encoder integration layer" for simplicity). The encoder integration layer can therefore be used as a control interface for encoding. At step 42, if the previous steps were successful, the encoder integration layer is now ready to receive frames.

[0064] In the example, two options are presented for passing video signals to the encoder integration layer: using an external input or data source (external allocation) and using an internal input or data source (internal allocation). These options can be signaled to the encoder integration layer during initialization settings, as described above. External inputs or data sources may include data objects generated outside the encoder integration layer and then passed to that layer. Internal inputs may include data retrieved using internal methods of the encoder integration layer (e.g., from a defined portion of memory).

[0065] If the encoder integration layer is configured not to use external input, i.e., using internal input or a data source, then pre-allocated frames can be extracted from the encoder integration layer (i.e., using internal methods). For example, a "GetPicture" method can be provided to obtain an image object with a pre-allocated plane. This plane can then be filled with frame data, for example, from a defined internal data source, such as memory or a file location. The encoder integration layer can additionally create an image pool from which data is extracted by the method. The size of the pool depends on the frame latency of the base encoder plus the frame latency of the encoder integration layer, where the frame latency is the number of frames that must be provided to the encoder before any output is produced.

[0066] Alternatively, the encoder integration layer's control interface also provides support for the functional layer to allocate images externally for encoding. For example, the encoder integration layer can create image objects and supply data from the functional layer (e.g., the functional layer can call an image object creation method and pass external data constituting the image for encoding). The encoder integration layer can then set a plane pointer to its allocated memory and set fields to indicate the number of bytes per line. Therefore, if the encoder integration layer is configured to accept externally allocated input, the functional layer creates the appropriate object and passes it to the encoder integration layer. In another implementation example, the encoder integration layer can also be configured to accept Android hardware buffers, such as when the encoder integration layer is configured in "GPU mode".

[0067] For example, an image object created by the functional layer in external input mode or retrieved using internal methods may contain all the information needed to encode a single video frame: such as the type of memory stored in the image plane (e.g., memory accessible from the CPU), pointers to the appropriate structure that holds the plane data (these planes may be interleaved in the case of RGB format; thus, the number of planes field may indicate the number of plane pointers rather than the number of channels - in the case of adjacent memory, each pointer may be set to the start address of each plane and the number of planes should be set as if the memory were not adjacent), the base type, the frame type, and the rendering timestamp.

[0068] To instruct the encoder integration layer to control the encoding of a frame, an image object is obtained for the type of frame to be encoded. As described above, for external input, the image object can be created or retrieved by a functional layer, which temporarily owns the image object after creation or retrieval. Subsequently, component data (i.e., YUV) can be loaded into pre-allocated memory of the planar components of the image object. Finally, encoding can be instructed to begin by calling an encoding method (e.g., Encode(PicObiect)) provided by the encoder integration layer and passing or referencing the image object. The encoding method can take both the image object and the encoding context as input, or the encoding context can be provided by the encoder integration layer. After this instruction begins encoding, the encoder regains ownership of the image object, and the functional layer's manipulation of the image object is no longer effective. This can be a blocking call for the duration of encoding a single frame.

[0069] At step 43, after encoding, the encoded bitstream is extracted from the encoder integration layer. The encoder integration layer may output encoded data after receiving one or more frames of input data. The encoded data can be extracted using an output function (e.g., GetOutput(...)). If output data is available, a success indication can be returned. If no data is available and no errors occurred during encoding, a completion notification can be returned. The output function of the encoder integration layer can provide a pointer to the encoded bitstream and grant ownership of the output data object to the calling program (e.g., the function layer). The encoder integration layer may, for example, produce more than one output per single input when generating interlaced encoding. The function layer can then utilize the output data, for example, by writing the output data directly to a file or copying the output data, if the output data must be buffered for output in a multiplexing system. After further processing of the encoded data, the function layer may release the output data object back to the encoder, making it no longer valid for the function layer to read from the object (e.g., during the encoding of subsequent images).

[0070] The output data can be provided as a data structure containing information for the encoded data frame. In the example implementation, these may not be created manually and can be processed as write-only data structures. In addition to pointers to data locations, the data structure may include or indicate various attributes of the encoded data, such as: frame type and / or field type (e.g., regardless of whether the frame that produced this output is progressive, interlaced, or field); base type and / or keyframe (e.g., the type of frame produced by the base encoder and whether it is marked as a keyframe); quantization parameters (e.g., the quantization parameter - QP of the frame, as reported by the base encoder); presentation timestamp (e.g., the presentation timestamp of the encoded frame, i.e., the input frame, or generated by the base encoder); decoding timestamp (e.g., the decoding timestamp of the encoded frame, as generated by the base encoder); configuration flags (e.g., indicating whether the data in this output is the encoder's configuration or the base encoder's configuration - e.g., if the base encoder is an H.264 encoder and configured to 1, then the output contains only SPS and PPS NAL units and the configuration output is presented before any frame data); data length; and user data (e.g., user data set on the picture that produced this frame).

[0071] Return to Figure 4 When an input data source, such as a file stream, has been exhausted, this can be signaled to the encoder integration layer to indicate that no more input data is available. For example, this can be done by passing a null value to the encoding function discussed above. After receiving a signal indicating that no further input is available, the encoder integration layer then completes processing any queued video frames and performs any actions that will be taken at the end of the stream. At step 44 of the diagram, to complete encoding, the encoding function is called using a null pointer for the picture object argument, and this serves as an indication for the encoder integration layer to refresh the entire pipeline. This is a blocking call while encoding the remaining pictures; subsequent calls may have no effect. Once encoding is complete, the encoder integration layer (i.e., the specific instance) can be destroyed at step 45. A shutdown function can be provided in the encoder integration layer's control interface to safely close and release the encoding context.

[0072] In the above methods, various functions associated with the encoder integration layer are described. These include instance creation functions (e.g., Open(...)), initialization functions (e.g., Init(...)), picture data retrieval functions (e.g., GetPicture(...) for internal data sources), encoding functions (e.g., Encode(...)), output retrieval functions (e.g., GetOutput(...)), output release functions (e.g., ReleaseOutput), and closing functions (e.g., Close(...)). These functions (which are considered non-restrictive and non-exhaustive) can be viewed as a control interface provided by the encoder integration layer. It should be noted that in this description, the terms "control interface," "callable function," and "API" are considered synonyms for mechanisms that allow control of the encoder integration layer from the functional layers. In some examples, the control interface provided by the encoder integration layer may also provide a mechanism for the functional layers to instruct dynamic reconfiguration during the encoding of the video signal. If the underlying encoder also supports dynamic reconfiguration, the encoder integration layer may instruct dynamic reconfiguration of the underlying encoder using plugins. This can be performed by passing settings to the reconfiguration functions provided by the control interface of the encoder integration layer. The encoder integration layer and plug-ins then have internal methods for controlling dynamic reconfiguration; for example, parameters can be passed to different interfaces of different underlying encoders using a single common plug-in interface available for the encoder integration layer. Settings can be provided in appropriate data structures.

[0073] As mentioned above Figure 2 As described in the context, software stack 20 provides one or a base encoder plugin 22. Based on the name of the base encoder specified in a data structure including settings for opening the encoder integration layer, the base encoder plugin implementing its corresponding wrapper can be dynamically loaded at runtime, as described above.

[0074] As described above, in some implementations, an interface may be provided for the encoder integration layer to interact with the plugins. This allows users to optionally implement their own underlying encoders via the plugin system without being embedded within the encoder integration layer, and provides the benefits of modularity. An example of an interface provided between two encoders is provided in WO 2017 / 089839, which is incorporated herein by reference.

[0075] As described above, the encoder integration layer can provide a robust configuration system as part of the control interface. For example, the encoder integration layer can provide one or more API commands to allow the functional layer to query certain properties. The configuration system can use a combination of read-only and writable properties accessible to the functional layer. In one implementation, all publicly available encoding configuration properties for the base encoder and LCEVC enhancements can be queried via the API.

[0076] In one example, attributes can be queried anonymously via a call to an attribute group. This returns an array of all attribute groups, each containing an array of all attributes within that group. If the user knows the attribute's name, the attribute can be queried directly via a specific call to the interface. Each attribute typically contains its name, description, expected type, and current value.

[0077] The interface also provides a mechanism for the functional layer (i.e., the user) to query the encoder's metadata. The metadata works in essentially the same way as attributes, except that it is only queried from the system and cannot be set externally. It can be processed as a read-only attribute.

[0078] In one particular implementation, metadata can be queried by name only.

[0079] The queried sample metadata includes: the basic technology of the underlying encoder plugin used (e.g., the libx264 plugin would output "H264" in this metadata); the width of the underlying encoded video; and the height of the underlying encoded video.

[0080] Advantageously, the interface also implements extensions to the attributes of adjacent blocks (e.g., so-called "blobs") that allow access to memory, into which the encoder integration layer can place metadata. This system also includes, or works with, a base encoder plugin, allowing the base encoder to output some data (e.g., configuration information) to one or more blocks of memory, the encoder integration layer being unaware of its purpose. This system allows the encoder integration layer to report some internal values, or data that is not interpreted as attributes or cannot be expressed as attributes, to the user. For example, an FFmpeg sample integration can leverage this feature to obtain H.264 global headers from the libx264 base encoder plugin for use in HTTP Real-Time Streaming (HLS).

[0081] In one implementation example, this extension is available when the 'Global Header' x264 attribute is set to 1, and allows access via the encoder integration layer to adjacent blocks of memory containing H.264 headers (PPS, SPS, etc.) from the base H.264 codec. This extension can be used when the output data structure is segmented, allowing the header to be injected at the beginning of each segment.

[0082] The configuration system also provides a mechanism for selecting a predetermined set of parameters to provide predefined optimizations. These parameters, mentioned above, are “tuning” parameters and can be instructed to the encoder integration layer as part of the initialization settings. In one case, each “tuning” (i.e., the set of parameters) can represent a different encoding goal or purpose. Depending on the selected tuning, the encoder can then implement the defined settings and parameters considered (e.g., via testing or experimentation) optimal for said goal. In the basic example, the settings can be selected as 'tuning' #1, and the encoder integration layer can select multiple parameters based on this optimization. When using the AVC base encoder, depending on the encoding goal, there can be a set of x264 'tunings', such as indicating different variations of LCEVC tuning. Examples include optimizations for visual quality, for Video Multi-Method Evaluation Fusion (VMAF) metrics, for VMAF NEG (no enhancement gain) metrics, for PSNR, for structural similarity (e.g., according to the Structural Similarity Index Measure -SSIM or the Multi-Scale Structural Similarity Index Measure -MS-SSIM), or for the visual quality of animation.

[0083] In some examples, the encoder, as described herein, can encode the enhanced stream in either Constant Bit Rate (CBR) mode or (uncapped or capped) Constant Rate Factor (pCRF) mode. For instance, these can be set as part of the setup for instantiating and / or initializing the encoder integration layer. Rate control can be applied depending on the mode: for example, "block" (default) or "rolling window". When in streaming "block" mode, the rate controller implemented by the encoder integration layer resets the bucket fill level at the beginning of each streaming block to avoid unnecessary effects from one block to the next (e.g., making a block slightly smaller than the target bit rate simply because the previous block was slightly larger, or vice versa). In "rolling window" mode, instead, the bucket fill level is never reset. "Block" mode is recommended for streaming based on Average Bit Rate (ABR) blocks, while "rolling window" mode is recommended for low-latency video and tests involving short self-similar sequences. "Block" mode can be used by default, so explicit specification of the corresponding settings is not required. To activate the "scrolling window" mode, commands or data items can be set within the settings properties. Similarly, the rate control window length can be changed. The pCRF mode can be applied in a manner similar to the comparative CRF modes in other codecs. This allows for custom processing within the encoder integration layer. Further details can be found in WO 2020 / 188230, which is incorporated herein by reference. The pCRF mode ensures that a certain quality factor is maintained throughout the encoded video, with an uncapped endpoint rate. Using a similar approach to comparatives in AVC, a lower pCRF value means less compression and higher quality, at the cost of a larger file size. The pCRF value can be defined as a floating-point fraction with a meaning similar to the CRF in AVC (e.g., a typical range of 20 to 36), thus controlling the overall quality of the base and enhanced encoding as controlled by the encoder integration layer. To activate pCRF, settings can be provided to the encoder integration layer, which in turn controls the plug-in and LCEVC encoder.

[0084] The examples presented in this paper provide an encoder integration layer that allows the coordination or orchestration of the behavior of a base encoder and an LCEVC encoder to encode a base stream and an LCEVC enhancement stream. A collection of plug-ins presents a common (i.e., a single specification) interface to the encoder integration layer, allowing the layer to control different base codecs. Each base codec can have different plug-ins, where plug-ins provide base codec-specific interfaces for controlling the base encoder and a common interface to the encoder integration layer, effectively “wrapping” the functionality of the base codec. This then allows the use of different plug-ins to support different base codecs, where plug-ins can be easily implemented as command mappings, and more complex control is performed by the encoder integration layer. Similarly, an LCEVC encoder may include an encoder configured to output an LCEVC enhancement stream that can be decoded by a standard-defined decoder. The encoder integration layer controls the operation of the base encoder via plug-ins and the LCEVC encoder. The encoder integration layer coordinates the exchange of picture data between the base encoder and the LCEVC encoder and controls the output of a single multiplexed stream for transmission or storage. The encoder integration layer provides an easy-to-use interface for both the base codec and LCEVC codec, which can be used by client applications within the functional layer. This allows client applications to create LCEVC codecs as easily as the base codec. The encoder integration layer appears to the client application as just "another encoder," and multi-layer processing is hidden from the client application.

[0085] At an encoder, for example, implemented in a streaming server or client device or an apparatus for encoding from a data repository, the methods and processes described herein may be embodied as code (e.g., software code) and / or data. The encoder may be implemented in hardware and / or software, as is well known in the field of data compression. For example, hardware acceleration using a specially programmed graphics processing unit (GPU) or a specially designed field-programmable gate array (FPGA) may provide some efficiency. This could be the case for an aggregate of underlying codecs and / or an LCEVC encoder. For integrity, the code and data may be stored on one or more computer-readable media, which may contain any means or medium capable of storing code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on the computer-readable medium, the computer system executes methods and processes embodied as data structures and code stored within the computer-readable storage medium. In some embodiments, one or more steps of the methods and processes described herein may be executed by a processor (e.g., a processor of a computer system or a data storage system).

[0086] Generally, any of the functionalities described in this text or illustrated in the diagrams may be implemented using software, firmware (e.g., a fixed logic circuit system), programmable or non-programmable hardware, or a combination of these embodiments. Generally, as used herein, the terms "component" or "function" refer to software, firmware, hardware, or a combination of these. For example, in the case of a software embodiment, the terms "component" or "function" may refer to program code that performs a specified task when executed on one or more processing devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

Claims

1. A video encoder, comprising: One or more encoder plugins that provide wrappers for one or more corresponding underlying codecs to implement an underlying coding layer, each wrapper implementing an interface for exchanging data with the corresponding underlying codec, the underlying coding layer being configured to encode and decode video signals; An enhanced encoder, used to implement an enhanced coding layer, is configured to: Receive a decoded video signal from one of the one or more encoder plugins, the decoded video signal comprising a decoded version of the encoded video signal generated by one of the one or more encoder plugins, and A coded enhancement signal is generated for the coded video signal, the coded enhancement signal including one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from the input video signal; as well as An encoder integration layer is used to control the operation of the one or more encoder plug-ins and the enhanced encoder to generate the encoding of the input video signal using the base coding layer and the enhanced coding layer. The encoder integration layer provides a control interface for the video encoder.

2. The video encoder of claim 1, wherein the encoder integration layer is configured to: receive ownership of frames of the input video signal; indicate encoding of the frames by the one or more encoding plugins and the enhancement encoder; and provide a pointer to the location of output data upon request via the control interface.

3. The video encoder of claim 2, wherein the pointer is provided within a data structure, the data structure further including auxiliary information associated with the encoding of the input video signal.

4. The video encoder of any one of claims 1 to 3, wherein the encoder integration layer is further configured to create a data structure from the frames of the input video signal for the type of the frame to be encoded, the data structure comprising a set of planes that collectively constitute the frame; and to fill the planes with frame data from the frames of the input video signal.

5. The video encoder according to any one of claims 1 to 3, wherein the encoder integration layer is further configured to identify that a frame is externally allocated and to retrieve a data structure, the data structure comprising a plane set of frames that together constitute the input video signal and a set of parameters necessary for encoding the frames.

6. The video encoder according to any one of claims 1 to 3, wherein the encoder integration layer is configured to obtain a set of encoding attributes, and after initializing the encoder integration layer, successful initialization is confirmed via the control interface based on the encoding attributes.

7. The video encoder of claim 6, wherein the encoder integration layer is configured to receive another set of encoding attributes and instruct the base encoding layer to be reconfigured using the one or more encoder plugins.

8. The video encoder according to any one of claims 1 to 3, wherein the encoder integration layer is configured to: obtain a set of configuration default values; retrieve a configuration data structure including one or more configuration attributes; and overwrite the configuration default values ​​in the set corresponding to the configuration default values ​​of the one or more configuration attributes.

9. The video encoder according to any one of claims 1 to 3, wherein the control interface includes a set of predefined queries, wherein the encoder integration layer is configured to receive queries from the set of predefined queries and return corresponding encoding or configuration attributes of the encoder integration layer.

10. The video encoder according to any one of claims 1 to 3, wherein the control interface includes a set of predefined metadata queries, the encoder integration layer is configured to receive queries for one or more metadata attributes, wherein the encoder integration layer is configured to retrieve metadata from the one or more encoder plugins and return read-only metadata attributes corresponding to the one or more metadata attributes.

11. The video encoder of any one of claims 1 to 3, wherein the control interface includes a set of predefined metadata queries, and wherein the encoder integration layer is configured to receive queries for one or more metadata attributes, and the encoder integration layer is configured to retrieve metadata from the one or more encoder plugins and, in response to the query, return the location of a memory block storing the metadata.

12. The video encoder according to any one of claims 1 to 3, wherein the control interface provides a set of predetermined optimization instructions, and wherein the encoder integration layer is configured to receive one or more of the predetermined optimization instructions and optimize the operation of the one or more encoder plug-ins and the enhanced encoder according to a predefined variant corresponding to the received instructions.

13. A method for encoding a video signal using a video encoder according to any one of claims 1 to 12, the method comprising: Initialize the encoder integration layer; When there is an input video signal to be encoded: Retrieve the input frames of the input video signal and transfer ownership of the frames to the video encoder via the control interface; Wait for the video encoder to encode the input frame and then extract the output data; as well as Ownership of the output data is released back to the encoder integration layer, and The encoder integration layer is turned off.

14. A computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform the method according to claim 13.

15. A video encoding system, comprising: The video encoder according to any one of claims 1 to 12; One or more basic codecs; as well as The application layer provides one or more calls to the video encoder via the control interface to instruct video encoding.

16. The video encoding system of claim 15, wherein the application layer is configured to be connected via the control interface: Initialize the encoder integration layer; When there is an input video signal to be encoded: Retrieve the input frames of the input video signal and transfer ownership of the frames to the video encoder via the control interface; Waiting for the video encoder to encode the input frame and then extract the output data; and Ownership of the output data is released back to the encoder integration layer, and The encoder integration layer is turned off.

17. An encoder integration system for controlling the operation of one or more encoder plug-ins and enhancement encoders to generate encodings of an input video signal using a base coding layer and enhancement coding layers. The encoder integration system described herein provides a control interface for the video encoder. The one or more encoder plugins provided wrappers for one or more corresponding underlying codecs to implement the underlying coding layer, each wrapper implementing an interface for data exchange with the corresponding underlying codec, the underlying coding layer being configured to encode and decode video signals; and The enhanced encoder implements an enhanced coding layer, and the enhanced encoder is configured to: Receive a decoded video signal from one of the one or more encoder plugins, the decoded video signal comprising a decoded version of the encoded video signal generated by one of the one or more encoder plugins, and A encoded enhancement signal is generated for the encoded video signal, the encoded enhancement signal including one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from the input video signal.

Images