A video encoding module for hierarchical video coding

Abstract

A video encoding module 30 comprising: encoder 32a and decoder 32b plug-ins providing a wrapper for respective first and second base codecs together implementing a base coding layer 37; an enhancement

Description

BACKGROUND A hybrid backward-compatible coding technology has been previously proposed, for example in WO 2014 / 170819 and WO 2018 / 046940, the contents of which are incorporated herein by reference. Further examples of tier-based coding formats include ISO / IEC MPEG-5 Part 2 LCEVC (hereafter “LCEVC”). LCEVC has been described in WO 2020 / 188273A1, and the associated standard specification documents including the Text of ISO / IEC 23094-2 Ed 1 Low Complexity Enhancement Video Coding published in November 2021, both documents being incorporated by reference herein in their entirety. In these coding formats a signal is decomposed in multiple “echelons” (also known as “hierarchical tiers”) of data, each corresponding to a “Level of Quality”, from the highest echelon at the sampling rate of the original signal to a lowest echelon. The lowest echelon is typically a low quality rendition of the original signal and other echelons contain information on correction to apply to a reconstructed rendition in order to produce the final output. LCEVC adopts this multi-layer approach where any base codec (for example H.264 or HEVC) can be enhanced via an additional low bitrate stream. LCEVC is defined by two component streams, a base stream typically decodable by a hardware decoder and an enhancement stream consisting of one or more enhancement layers suitable for software processing implementation with sustainable power consumption. The enhancement provides improved compression efficiency to existing codecs, and reduces encoding and decoding complexity. Since LCEVC and similar coding formats leverage existing decoders and are inherently backwards-compatible, there exists a need for efficient and effective integration with existing video encoding implementations without complete redesign. Examples of known encoding implementations include FFmpeg. Moreover, LCEVC is not limited to known codecs and is theoretically capable of leveraging yet-to-be-developed codecs. As such any LCEVC implementation should be capable of integration with any hitherto known or yet-to-be-developed codec, implemented in hardware or software, without introducing coding complexity. It should be noted that some realisations of these coding formats require both an encoding and decoding operation to be performed by the base codec during video encoding. That is, the enhancement stream is created by comparing a version of the input signal and a decoded version of the encoded base stream. Thus, in these realisations there exists a need to efficiently and effectively instruct both an encoding and decoding operation using the base codec, at the encoder device. Another issue being faced by developers is the burden of maintaining the growing collection of plug-ins. Many plug-ins share common functionality, yet come with their own maintenance workload. SUMMARY OF INVENTION The present invention relates to methods for encoding signals, such as video signals, using hierarchical coding formats, as well and encoders and encoding systems. The present invention specifically relates to a video encoder for integrating a hierarchical encoder, preferably an LCEVC encoder into an application or software stack, where the video encoder includes separate plug-ins for encoding a frame and generating a decoded version of the encoded frame. According to a first aspect of the invention there is provided a video encoding module comprising: an encoder plug in providing a wrapper for a first base codec to implement an interface for data exchange with the first base codec to generate an encoded version of an input video signal; a decoder plug in providing a wrapper for a second base codec to implement an interface for data exchange with the second base codec to generate a decoded version of the encoded video signal generated by the encoder plug in, wherein the encoder plug-in and the decoder plug-in together implement a base coding layer configured to encode and decode a video signal; and, an encoder integration layer to control operation of the encoder plug-in and the decoder plug-in and to control an enhancement encoder, wherein the encoder integration layer is configured to: instruct the encoder plugin to generate the encoded version of the input video signal; instruct the decoder plug-in to generate the decoded version of the encoded video signal generated by the encoder plug-in; and, instruct the enhancement encoder to generate an encoded enhancement signal for the encoded video signal, wherein the enhancement encoder implements the enhancement encoding layer, the enhancement encoder being configured to: receive the decoded video signal from the decoder plug-in; and, generate the encoded enhancement signal for the encoded video signal, the encoded enhancement signal comprising one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal. The video encoding module provided in this aspect includes an encoder plug-in and a decoder plug-in which can be separately and independently controlled by the encoder integration layer to provide encoding and decoding functionality even if the base codec does not include a decoding API. This provides more composable implementations of the video encoder which will increase rapid prototyping of new base encoder technologies. What we intend to mean by this is that because the decoder plug-in is provided separately to the encoder plug-in, the encoder integration layer is not limited using the decoding API of a base codec, which may not always exist. Further, the need to develop a new plug-in for every combination of base encoder and base decoder is done away with. The video encoder is therefore more customisable, and can be integrated with a range of base codecs, including those that do not provide a decoding API. In other words, the encoder integration layer generates an encoding of the input video signal using the base coding layer and an enhancement encoding layer. In some examples, the second base codec is different to the first base codec. In other examples, the first base codec is of the same type as the second base codec or alternatively the first base codec is the second base codec, for example, using the same software library. Since encoding and decoding functionalities are provided by separate plug-ins, this allows the video encoder to interface with different base codecs to perform different parts of the encoding process. This was only possible previously if a specific plug-in was designed to interact with the combination of the first base codec and second base codec. However, providing a separate encoder plug-in and decoder plug-in means that a specific plug-in for interacting with a specific combination of different base codecs does not need to be developed. For example, the first base codec may encode a frame, and the second base codec may decode and reconstruct the frame. Using first and second base codecs can potentially spread the processing load without compromising operating efficiency or the quality of the output (i.e. the encoding of the input video signal using the base layer and the enhancement layer). Further, this example demonstrates how providing separate encoder and decoder plug-ins increases the composability of the video encoding module, since different base codecs may now be utilised to perform different parts of the encoding process, which was not previously practical. In some examples, the first base codec and second base codec are each configured to perform respective encoding and decoding operations according to a first coding scheme. That is, the first base codec and second base codec perform respective encoding and decoding operations according to the same coding scheme. For example, the first base codec may be a hardware encoder and the second base codec may be a GPU configured to perform decoding operations. Alternatively, the first base codec may be a GPU configured to perform encoding operations and the second base codec may be a hardware decoder. Other configurations of the first and second base codec are of course possible. For example the first base codec may be a hardware encoder and the second base codec may be a hardware decoder, i.e. a different hardware decoder. Equally, the first base codec may be a GPU configured to perform encoding operations and the second base codec may be a GPU configured to perform decoding operations. In further combinable examples a CPU configured to perform encoding operations or a CPU configured to perform decoding operations may be used in combination with a hardware encoder / decoder or GPU configured as discussed above. That is, any conceivable mix of a hardware encoder, hardware decoder, GPU configured to perform encoding, GPU configured to perform decoding, CPU configured to perform encoding, and a CPU configured to perform decoding may be used. This further demonstrates the composability of the video encoding module. It has effectively become possible to mix and match software and hardware encoders and decoders. This allows for the video encoder to work in a greater range of scenarios, for example in platforms with discrete hardware blocks for encoding and decoding in different APIs. In some advantageous examples, the encoder integration layer comprises one or more callback functions, and the encoder integration layer is configured to pass one or more pointers to a respective callback function to the encoder plug-in. Providing callback functions in the encoder integration layer reduces the maintenance burden on developers, especially as the number of plug-ins increases. This is because common functionality shared between various plug-ins, for example as implemented by different vendors, is promoted from the plug-in to the encoder integration layer. This reduces the burden on plug-in authors, who otherwise would have to include similar functionality in each plug-in that they develop. In some further, optional, implementations the one or more pointers may point to an OutputReady callback function, the OutputReady callback function being configured to be called by the encoder plug-in or the decoder plug-in when an encode operation or a decode operation is completed to indicate to the encoder integration layer that a frame of the video signal has been encoded or decoded. The OutputReady callback reduces latency which is inherent to the encoding pipeline and allows the encoder integration layer to operate with increased efficiency with a range of base codecs despite different codecs operating in different ways. For example, some codecs are blocking, some are asynchronous polling interfaces and some have callback mechanisms. However, the OutputReady callback allows the encoder plug-in to call into the encoder integration layer to notify the encoder integration layer that is has completed encoding a frame. That is, the plug-in can, at any time, call into the encoder integration layer to notify that an output is available. In other words, the encoder plug-in or the decoder plug-in may be configured to call a callback function in the encoder integration layer in response to the encoder plug-in or the decoder plug-in completing a respective encoding operation or decoding operation. The respective encoding operation or decoding operation may be performed in the base coding layer. In additional or alternative examples, the one or more pointers may point to an Error callback function, the Error callback function being configured to be called by the encoder plug-in or the decoder plug-in when an error occurs in an encoding operation, to indicate to the encoder integration layer that an error has occurred in an encoding or decoding operation. Similarly to the Output Ready callback, the Error callback can be called at any time by the encoder plug-in or the decoder plug-in, thereby notifying the encoder integration layer of an error as soon as it occurs. This allows the encoder integration layer to handle the error in real-time. In some optional implementations, the encoder integration layer may be configured to instruct the encoder plug-in to generate the encoded version of the input video signal in response to a call made from an application layer to instruct video encoding, wherein the encoder integration layer provides a control interface for the video encoding module. According to a second aspect there is provided a method of encoding a video signal, the method comprising: instructing, by an application layer, an encoder integration layer to encode an input video signal comprising a frame; controlling, by the encoder integration layer, an encoder plug-in providing a wrapper for a first base codec to generate an encoded version of the frame according to the first base codec; instructing, by the encoder plug-in, the first base codec to generate the encoded version of the frame; passing, by the encoder plug in, the encoded version of the frame to the encoder integration layer; controlling, by the encoder integration layer, a decoder plug-in providing a wrapper for a second base codec to generate a decoded version of the encoded version of the frame according to a second base codec; instructing, by the decoder plug-in, the second base codec to generate the decoded frame; passing, by the decoder plug in, the decoded version of the frame to the encoder integration layer; controlling, by the encoder integration layer, an enhancement encoder to generate an encoding of the input video signal comprising an encoded enhancement signal comprising one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal. In some implementations, the second base codec may be different to the first base codec. Further, the first base codec and second base codec may each be configured to perform respective encoding and decoding operations according to a first coding scheme. In some examples, the first base codec is a hardware encoder and the second base codec is a GPU configured to perform decoding operations. Optionally, the first base codec may be a GPU configured to perform encoding operations and the second base codec is a hardware decoder. In some advantageous examples, the encoder integration layer comprises one or more callback functions, and the method further comprises: passing, by the encoder integration layer, one or more pointers to a respective callback functions to the encoder plug in and / or the decoder plug-in. For example, the one or more callback function pointers points to an OutputReady callback function, and the method may further comprise: calling, by the encoder plug in, the Output Ready callback when an encoding operation is completed to indicate to the encoder integration layer that a frame of the video signal has been encoded; and / or calling, by the decoder plug in, the Output Ready callback when an decoding operation is completed to indicate to the encoder integration layer that a frame of the video signal has been decoded. Additionally or alternatively, the one or more pointers may point to an Error callback function, and the method may comprise: calling, by the encoder plug-in, the Error callback function when an error occurs in an encoding operation, to indicate to the encoder integration layer that an error has occurred in an encoding operation; and / or calling, by the decoder plug-in, the Error callback function when an error occurs in an decoding operation, to indicate to the encoder integration layer that an error has occurred in a decoding operation. According to a third aspect there is provided a computer readable medium comprising instructions which when executed by a processor, cause the processor to carry out the steps of any of the above discussed aspects. According to a fourth aspect there is provided a video encoding system comprising: a video encoding module according to the first aspect or any example implementation of the first aspect; a first base codec and a second base codec; and an application layer configured to provide one or more calls to the video encoder via a control interface provided by the encoder integration layer to instruct video encoding. According to a fifth aspect there is provided an encoder integration layer to control operation of: an encoder plug-in; a decoder plug-in; and, an enhancement encoder, wherein the encoder integration layer is configured to generate an encoding of an input video signal using a base coding layer and an enhancement encoding layer, wherein the encoder plug-in provides a wrapper for a first base codec to implement an interface for data exchange with the first base codec to generate an encoded version of an input video signal, wherein the decoder plug in provides a wrapper for a second base codec to implement an interface for data exchange with the second base codec to generate a decoded version of the encoded video signal generated by the encoder plug-in, wherein the encoder plug in and decoder plug in together implement the base coding layer, the base coding layer being configured to encode and decode a video signal; and, the encoder integration layer controls the enhancement encoder to implement an enhancement encoding layer, the encoder integration layer being configured to: instruct the encoder plug-in to generate the encoded version of the input video signal; instruct the decoder plug-in to generate the decoded version of the encoded video signal generated by the encoder plug-in; and, instruct the enhancement encoder to generate an encoded enhancement signal for the encoded video signal, the enhancement encoder being configured to: receive the decoded video signal from the decoder plug-in, and generate an encoded enhancement signal for the encoded video signal, the encoded enhancement signal comprising one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal. According to a sixth aspect there is provided an encoder plug-in and a decoder plug-in suitable for use with a video encoding module, wherein in use the encoder plug-in provides a wrapper for a first base codec to implement an interface for data exchange with the first base codec to generate an encoded version of an input video signal, the decoder plug-in provides a wrapper for a second base codec to implement an interface for data exchange with the second base codec to generate a decoded version of the encoded video signal generated by the encoder plug-in, wherein the encoder plug in and decoder plug-in together implement a base coding layer configured to encode and decode a video signal; the encoder plug in and decoder plug in configured to be controlled by an encoder integration layer to generate an encoding of the input video signal using the base coding layer and an enhancement encoding layer, the encoder plug-in configured to generate the encoded version of the input video signal in response to an instruction from the encoder integration layer; the decoder plug-in configured to generate the decoded version of the encoded video signal generated by the encoder plug-in in response to an instruction from the encoder integration layer; wherein an enhancement encoder controlled by the encoder integration layer implements an enhancement encoding layer, the enhancement encoder being configured to: receive the decoded video signal from the decoder plug-in, and generate an encoded enhancement signal for the encoded video signal, the encoded enhancement signal comprising one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal. A further aspect provides encoder plug-in and / or a decoder plug-in suitable for use with the video encoding module of any of the above aspects. A further example provides an encoder plug-in configured to produce a decoded version of encoded data. The encoder plug-in therefore provides a wrapper for data exchange with a base codec to generate an encoded version of an input video signal, and a decoded version of the encoded signal. The encoder plug-in is provided with pointers to write and / or store the recon data to. The pointers may be provided by the encoder integration layer, for example. The encoder plug-in is able to produce a decoded version of the encoded image. That is to say, in some examples, the encoder plug-in can produce recon data. The concept of recon data is discussed further below. In such situations, the encoder plug-in can handle encoding operations and decoding operations, so the decoder plug-in is not needed, and the encoder plug-in is provided with pointers to write and / or store the recon data to. The pointers may be provided by the encoder integration layer, for example. BRIEF DESCRIPTION OF DRAWINGS Example methods and systems in accordance with the invention will now be described with reference to the accompanying drawings, in which: Figure 1 shows a high-level schematic of a known multi-layer encoding process; Figure 2 shows a schematic of how a known video encoder interacts with a base codec; Figure 3 shows a software stack including a video encoding module according to examples of the present disclosure; Figure 4 shows a schematic of a video encoding module with an encoder plug-in and a decoder plug-in interacting with a base codec in an encoding process; Figure 5 shows a schematic of callback functions in an encoder integration layer being called by an encoder plug-in and a decoder plug-in; Figure 6 shows a schematic of a plug-in API encode sequence for a base codec with a function call-based interface, in which the plug-in utilises the Output Ready callback function; Figure 7 shows a schematic of a plug-in API encode sequence for a base codec with a callback interface, in which the plug-in utilises the Output Ready callback; and Figure 8 shows a flow diagram of an encoding method using a video encoder. DETAILED DESCRIPTION This disclosure describes an implementation for integration of a hybrid backwardcompatible coding technology with existing encoders and decoders, optionally via a software update. In a non-limiting example, the disclosure relates to an implementation and integration of MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC). LCEVC is a hybrid backward-compatible coding technology which is a flexible, adaptable, highly efficient and computationally inexpensive coding format which combines a different video coding format, a base codec (i.e. encoder-decoder), (e.g. AVC / H.264, HEVC / H.265, or any other present or future codec, as well as non-standard algorithms such as VP9, AV1 and others) with one or more enhancement levels of coded data. Example hybrid backward-compatible coding technologies use a down-sampled source signal encoded using a base codec to form a base stream. An enhancement stream is formed using an encoded set of residuals which correct or enhance the base stream for example by increasing resolution or by increasing frame rate. There may be multiple levels of enhancement data in a hierarchical structure. In certain arrangements, the base stream may be decoded by a hardware decoder while the enhancement stream may be suitable for being processed using a software implementation. Thus, streams are considered to be a base stream and one or more enhancement streams, where there are typically two enhancement streams. It is worth noting that typically the base stream may be decodable by a hardware decoder while the enhancement stream(s) may be suitable for software processing implementation with suitable power consumption. The video frame is encoded hierarchically as opposed to using block-based approaches as done in the MPEG family of algorithms. Hierarchically encoding a frame includes generating residuals for the full frame, and then a reduced or decimated frame and so on. For context purposes only, as the detailed structure of LCEVC is known and set out in the approved standards specification, Figure 1 illustrates how LCEVC operates on the encoding side, assuming H.264 as the base codec. Those skilled in the art will understand how the examples described herein are also applicable to other multi-layer coding schemes (e.g., those that use a base layer and an enhancement layer) based on the general description of LCEVC that is presented with reference to Figure 1. Turning to Figure 1, the H.264 encoding 10 is performed on a down-scaled input at a lower resolution, typically a quarter of the desired output resolution. The output of the H.264 encoding 10, referred to herein as a base stream, may be supplemented with additional LCEVC data. The LCEVC data may be added, either in Supplemental Enhancement Information (SEI) of the H.264 Network Abstraction Layer (NAL), or in an additional data stream as identified using a Packet Identifier (PID). In Figure 1, the uncompressed full resolution video is fed to a downscaler 11 before being passed to the H.264 encoding 10. The LCEVC encoder 13 generates the enhancement stream from two inputs: the H.264 (i.e. base layer) encoding, and the original uncompressed full resolution video, effectively filling the quality gap between the two. The enhancement stream (e.g. as encapsulated in the SEI messages) and the base stream (e.g. as constructed using VCL NAL as per a non-enhanced stream) are then combined into an output encoded bitstream. To fully leverage the benefits of LCEVC, it is beneficial to combine an LCEVC encoder with a base codec for supported base codecs. There is thus proposed herein an encoder implementation and software stack to do this. The encoder implementation and software stack may be provided as an optimised software library for encoding and decoding MPEG-5 LCEVC enhanced streams, providing a simple yet powerful API. This allows developers flexibility and the ability to deploy LCEVC at any level of a software stack with a range of options from low-level APIs to implementations in commonly used open-source encoders and players. Referring now to Figure 2, an example of an existing hierarchical encoding process 20 is shown. The process 20 includes steps labelled as 24 to 31 in Figure 2 and is performed by an encoder integration layer (EIL) 21, a plug-in 22 and a base decoding layer 23. It can be seen that EIL 21 interacts with a plug-in 22, and the plug-in 22 in turn interacts with a base coding layer 23, for example a H.264 codec. It would be understood that the base coding layer 23 comprises a base encoder and a base decoder to implement a base coding layer. The EIL is called to perform encoding of input data at step 24, for example an input video signal. For example, the EIL 21 may be called from an application layer or a functional layer via an API. At step 25, the EIL 21 controls operation of the plug-in 22 to generate an encoded version of the input signal and a decoded version of the encoded video signal. Specifically, the plug-in 22 uses the base coding layer 23 to encode the input data signal at step 26. Once encoding is complete, the base coding 23 returns Done to the plug-in 22 at step 27. Note that the input signal may be downscaled before being encoded. After the base layer 23 returns Done to the plug-in 22 at step 27, the plug-in 22 returns the encoded data to the encoder integration layer 21 at step 28. Note that the step of returning encoded data to the encoder integration layer is not the end of the process, since a decoded version of the encoded data must still be generated and provided to the encoder integration layer. Until the decoded version of the encoded data is received by the encoder integration layer 21, the encoder integration layer 21 cannot proceed with any subsequent processing. As such, the plug-in 22 then uses the base layer 23 to decode the encoded version of the input signal at step 29. The decoded version of the encoded signal may be referred to as reconstructed data, or recon data. The steps involved in decoding an encoded frame may be referred to as recon generation. It would be understood that recon generation may involve upscaling the decoded version of the encoded image. It is to be noted that upscaling does not happen in the plug-in or the encoder integration layer. Rather, the enhancement encoder handles upscaling tasks This is to allow for a meaningful comparison between the original signal and the reconstructed signal. We refer to these terms throughout. Once the decode has been completed at step 29, the base layer 23 returns output to the plug-in 22 at step 30. The plug-in 22 returns the decoded version of the encoded data signal, i.e. the recon data, back to the EIL 21 at step 31. At this point, ownership of the encoded data is released to the encoder integration layer 21. The encoder integration layer 21 may continue with subsequent processing steps. As described above, the encoding process utilises a plug-in 22 to use the base layer 23 to perform the encoding (step 26) and decoding (step 29). However, it has been observed that many base coder APIs do not provide decoding functionality. This means that for those base coders, recon data cannot be generated. A current workaround is to integrate decoding functionality into the same plug-in as the encoder, using a separate software or hardware decoder. This is suboptimal because it raises a barrier for developers seeking to implement an enhancement encoder such as LCEVC in their platform. Further, the plug-in 22 is designed to work with a specific combination of base encoder and base decoder, which increases the workload required for LCEVC to be integrated into new codecs, new codec implementations or new combinations of codecs. To address this issue, a video encoding module with an encoder plug-in and a decoder plug-in is provided, as conceptually shown in Figure 3. Figure 3 shows a software stack 30 comprising an encoder integration layer 31, an encoder plug-in 32a, a decoder plug-in 32b and an enhancement encoder 33, such as an LCEVC encoder. The encoder integration layer 31 controls operation of the encoder plug-in 32a, the decoder plug-in 32b and the enhancement encoder 33 to generate an encoding of an input video signal using a base coding layer and an enhancement layer. Further, the encoder integration layer 32 provides a control interface for the video encoding module. That is, the encoder integration layer 31 provides an API through which calls can be made from an application layer, or functional layer 34, to instruct the encoding, query properties or metadata of the encoding, and provide a configuration system. The encoder integration layer 31 integrates the base coder 37 and the enhancement encoder 33, orchestrating the integrated behaviour. Moreover, the encoder integration layer 31 is architected to support a base codec implementation through a plug-in system that makes it generic and independent. In other words, the encoder integration layer 31 integrates the user’s intended functionality with the enhancement encoding layer and base coding layer. The encoder plug-in 32a and decoder plug-in 32b each provide a wrapper for a base coder 37 to implement a base coding layer. Each wrapper implements an interface for data exchange with the base coder 37. The base coding layer is configured to encode and decode an input video signal. That is the encoder plug-in 32a implements a wrapper for a first base codec to implement an interface for data exchange with the first base codec to generate an encoded version of an input video signal, and the decoder plug-in 32b provides a wrapper for a second base codec to implement an interface for data exchange with the second base codec to generate a decoded version of the encoded video signal generated by the encoder plug-in. The encoder plug-in and the decoder plug-in together implement a base coding layer configured to encode and decode a video signal. Optionally, the first base codec and the second base codec may be configured to code according to the same coding scheme, i.e. the same coding standard such as AVC / H.264, HEVC / H.265, or any other present or future codec, as well as nonstandard algorithms such as VP9, AV1 and others. The first base codec and second base codec could, for example, be made by different developers or be constructed with different parameters. Compared to an existing implementation in which one plug-in is controlled to perform encoding of an input video signal and provide recon data, according to the disclosure, an encoder plug-in 32a is provided separately to a decoder plugin 32b. The encoder integration layer is therefore not limited to using a decoding and / or debug recon API of a base codec, which may not always exist. This allows for more composable implementations of the video encoder and will facilitate rapid prototyping of new base coder technologies. Specifically, splitting the functionality previously provided by one plug-in across two plug-ins means that different combinations of base encoders and base decoders can be used without creating a new plug-in for every combination of base encoder and base decoder. As a result, base coders can be efficiently and effectively interfaced with the encoder integration layer. Further, the encoder plug-in 32a and decoder plug-in 32b facilitate composable acceleration of the encode and recon generation portions of the encoding process. For example, in a system with discrete hardware blocks for encoding and decoding in different APIs, the encoder plug-in 32a and decoder plug-in 32b make it possible to utilise different hardware blocks without compromising either hardware block with the load of the other piece of work to be performed. It should be noted that some base encoders can produce recon data. In such a situation, pointers for image data and recon data may be passable to the encoder plug-in 32aplug-in. That is, in some examples, the encoder plug-in 32a may control a base encoder in the coding layer to encode an input signal, and produce recon data. In such examples, the encoder plug-in passes recon data to an enhancement encoder for further processing. In some examples, the encoder plug-in 32a and the decoder plug-in 32b may be backwards compatible from the encoder integration layer 31 to the encoder plug-in 32a and / or decoder plug-in 32b. In other words, the encoder integration layer 31 may be able to load and utilise any plug-in version up to the version that the encoder integration layer 31 supports. In preferred embodiments, the enhancement encoder 33 is an LCEVC encoder. The terms LCEVC encoder and enhancement encoder will be used throughout interchangeably. The LCEVC encoder 33 implements an enhancement encoding layer and receives a decoded video signal from the decoder plug-in 32b, the decoded video signal comprising a decoded version of an encoded video signal as generated by the encoder plug-in 32a, and then generates an encoded enhancement signal for the encoded video signal. The encoded enhancement signal comprises one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal. In the examples described herein, residuals may be considered to be errors or differences at a particular level of quality or resolution. Further details are provided, for example, in the descriptions of LCEVC referenced herein. In addition to controlling the operation of the base encoding layer and the enhancement encoding layer, the encoder integration layer 31 provides a configuration system. Configuration properties may be set by the functional layer 34 and are supported by a comprehensive built-in system of optimised parameter defaults. The encoder integration layer 31 is optionally operable to employ preferred settings for both the LCEVC encoder 33 and base coder 37, based upon key parameters passed to it by the application, e.g., resolution and bitrate. The functional layer 34 may be implemented in reference source code 35 and the enhancement encoding software stack 30, comprising the encoder plug-in 32a, the decoder plug-in 32b, the enhancement encoder 33 and the encoder integration layer 31, may be implemented as a set of encoder libraries 36. Herein the functional layer 34 may be referred to interchangeably as the application layer or the user layer, as the functionality may be specifically coded by a user depending on the implementation required. The functional layer 34 comprises operations configured by the user to instruct the encoder integration layer 31. The functional layer 34 also manages the functionality of the encoder integration layer 31 and provides calls to the API provided by the encoder integration layer 31, e.g. to query the properties or set the configurations of the enhancement coding. As stated above, the functional layer 34 may be implemented at any level of a software stack. An example of an implementation of the functional layer may be the video coding tool FFmpeg. FFmpeg is a popular tool amongst video developers. In an example, the functional layer may be provided by FFmpeg functionalities and libav modifications. In certain cases, additional applications may be implemented above the functional layer 34, e.g. client applications may call upon video coding functions provided by FFmpeg, which in turn calls upon the control interface provided by the encoder integration layer 31 to provide the enhancement encoding. Figure 4 shows an example encoding process 40 involving a separate encoder plug-in 43 and decoder plug-in 42. The encoder integration layer 41 interacts with the base coding layer 44 through the encoder plug-in 43 and decoder plug-in 42. It will be understood that the base coding layer includes one or more base codecs, or in other words may comprise one or more of a base encoder and a base decoder. In a first step the encoder integration layer 41 is provided with input data such as an input video signal and instructed to encode the input video signal (step 45). The input data may be provided by an application or some other client. The encoder integration layer 41 controls, that is, calls or instructs the encoder plug-in 43 to encode the input data by sending the encoder plug-in 43 an instruction to encode the input video signal (step 46). The input video signal may be downscaled. The encode instruction includes a parameter * which can be regarded as a handle or pointer to a picture. That is, * encapsulates all the state needed to encode a frame or a field of the picture. For example, * at least includes a pointer to the memory containing the raw video data. The encoder plug-in 43 instructs the base coding layer 44 to encode the data (step 47), and the base coding layer 44 then notifies the encoder plug-in 43 that it has completed an encode operation via some mechanism. This mechanism used by the base coding layer 44 to notify the encoder plug-in 44 that it has competed an encode operation depends on the particular codec used to implement the base coding layer 44. In some examples, the base coding layer 44 returns done to the encoder plug-in 43 (step 48). The encoder plug-in 44 passes some encoded data for the picture back to the encoder integration layer 41 using a callback function, for example an Add Data call back function (step 49). The Add Data function passes a parameter* and data to the encoder integration layer 41. The Add Data callback function may be called one or more times for a given picture. It is to be noted that Add Data callbacks need not occur in the same sequence or thread as the encode function (step 46). Calls to Add Data may be interleaved. The encoder plug-in 43 returns done to the encoder integration layer 41 (step 50). That is, the encoder plug-in 43 notifies the encoder integration layer 41 that base encoding of the input video signal is complete. The encoder plug-in may also make the encoded input video signal available to the encoder integration layer 41, for example by releasing ownership of the encoded input video signal passing the encoded input signal to the encoder integration layer 41 or passing a pointer to the relevant location in memory to the encoder integration layer 41. The encoder integration layer 41 queues the encoded version of the input image (step 51). By doing so, the encoder integration layer 41 indicates that it has completed base encoding. The encoded version of the image may be encapsulated and transmitted or stored temporarily while the rest of the process is completed. It is to be noted that if the base encoded data is encapsulated and transmitted, this achieves a reduction in latency between the start of the encoding process and the time at which the encoded base data becomes available. This is beneficial for example when the base encoding of the video signal can be sent without enhancement data. The enhancement data may be sent later. Still referring to Figure 4, in the next part of the process, the encoder integration layer 41 controls the decoder plug-in 42 to generate recon data. Recon data has been explained above. Specifically, the encoder integration layer 41 controls the decoder plug-in 42 and instructs the decoder plug-in 42 to decode the encoded version of the input signal previously generated by the encoder plug-in 43 (step 52). Pictures may be passed to the decode function in the elementary stream’s decode order. The decoder plug-in 42 gets the required data at step 53, i.e. the encoded version of the input video signal and uses the base coding layer 44 to decode the data (step 54). Step 53 may be done by using a Get Data function of the encoder integration layer 41. The base coding layer 44 returns Done to the decoder plug-in 42 (step 55). That is, the base coding layer 44 indicates to the decoder plug-in 42 that base decoding of the encoded version of the input video signal is complete. In other words, the base coding layer 44 indicates to the decoder plug-in 42 that recon generation is complete. The mechanism of doing so depends on the implementation of the base coding layer, and it would be understood that returning Done is one example of a mechanism. The decoder plug-in 42 calls a Get Recon encoder integration layer 41 to obtain a picture data buffer into which the decoder plug-in 42 will copy the obtained decoded picture (step 56). A suitable buffer for the recon data is provided to the decoder plug-in 42 (step 57). The decoder plug-in 42 returns done to the encoder integration layer at step 58. By this we intend to mean that the decoder plug-in 42 notifies or indicates to the EIL 41 that the base coding layer 44, for which the decoder plug-in 42 provides a wrapper, has completed processing. The EIL 41 may then be operable to use the decoded version of the encoded frame, i.e. the recon data, for further LCEVC processing. This may include passing the recon data to an upscaler for upscaling. It would be understood that the specific function names referred to in Figure 4, such as Add Data, Get Recon, Done, Encode and Decode, are dependent on the implementation of the base coding layer 44, the encoder plug-in 43, the decoder plug-in 42 and the encoder integration layer. Different function names may be used to provide the same functionality in different implementations. The process of Figure 4 highlights the achieved composability provided by a separate encoder plug-in 43 and decoder plug-in 42. The encoder plug-in 43 and decoder plug-in 42 interface with the base coding layer 44 separately, meaning that different plug-ins could be utilised depending on the specific base encoders and base decoders in the base coding layer 44. The video coder as a whole is therefore more customisable and can be integrated with a wider range of base coders, especially new combinations of known or yet-to-be developed base coders and base decoders. That is, the encoder plug-in may provide a wrapper for a first base codec, while the decoder plug-in may provide a wrapper for a second, different, base codec. The first base codec and second base codec may be some combination of a hardware coder and a GPU configured for coding or decoding operations. Optionally, although the first base codec and the second base codec may be different, the first base codec and the second base codec may be configured to code according to the same coding scheme, i.e. the same coding standard such as AVC / H.264, HEVC / H.265, or any other present or future codec, as well as nonstandard algorithms such as VP9, AV1 and others. The first base codec and second base codec could, for example, be made by different developers or be constructed with different parameters For example, the first base codec may be a hardware encoder and the second base codec may be a GPU configured to perform corresponding decoding operations. Alternatively, the first base codec may be a GPU configured to perform encoding operations and the second base codec may be a corresponding hardware decoder. Other configurations of the first and second base codec are of course possible. For example the first base codec may be a hardware encoder and the second base codec may be a corresponding hardware decoder. Equally, the first base codec may be a GPU configured to perform encoding operations and the second base codec may be a GPU configured to perform corresponding decoding operations. From the above, it should be understood that separate encoder and decoder plug-ins remove barriers which previously prevented combinations of the base encoders and decoders being used due to the lack of a suitable plug-in which is able to interface or provide a wrapper for implementing data exchange with both the base encoder and base decoder. Although the encoder and decoder plug-ins discussed above have different functions, it will be appreciated that they will have some features in common. To avoid duplicating code, which increases the maintenance burden for developers and also increases the effort required for new plug-ins to be developed, the encoder integration layer may comprise one or more callback functions which can be called by the encoder plug-in and decoder plug-in. This is shown in Figure 5. Figure 5 may illustrate a graphics card or GPU performing the functions of encoder plug-in 53, such as Nvidia NVENC. Similarly decoder plug-in 52 may also optionally be performed by an onboard decoder such as NVDEC. In Figure 5, an encoder integration Iayer51 is shown to sit on top of an encoder plug-in 53 and a decoder plug-in 52. The encoder integration layer comprises one or more callback functions, which are represented by boxes 54. The callback functions 54 may be specific to encoder plug-ins, specific to decoder plug-ins or else be common to both encoder plug-ins and decoder plug-ins. For example, in Figure 5, the left-hand group of callback functions 55 may only be relevant for an encoder plug-in, the centre group of callback functions 56 may be relevant for encoder plug-ins and decoder plug-ins, and the right-hand group 57 may be relevant to decoder plug-ins. In use, the encoder plug-in 53 or decoder plug-in 52 call a callback function in the encoder integration layer 51 . The plug-ins can therefore rely on the functionality provided by the encoder integration layer. The burden on plug-in developers to include common functionality in every plug-in, and subsequently maintain those plug-ins, is reduced, and plug-in design and operation is therefore streamlined. One example of a callback function in the common support layer is an Output Ready callback. As the name suggests, the Output Ready callback allows a plugin to notify an encoder integration layer that some output is available. For example, a frame of an input video signal may have been encoded. Equally, the Output Ready callback may indicate that a decoded version of an encoded frame, i.e. recon data, is ready for further processing. It is to be understood that Output Ready is a symbolic name for a function that indicates that some data is available. The Output Ready callback may be implemented in terms of an OnPictureEncodeComplete callback and OnPictureDecodeComplete. As the names suggest, OnPictureEncodeComplete may be called to allow an encoder plug-in to notify the encoder integration layer that data resulting from an encode operation is available, and OnPictureDecodeComplete may be called to allow a decoder plug-in to notify the encoder integration layer that data resulting from a decode operation is available. Providing the Output Ready callback function in the encoder integration layer 51 has the benefit that an encoder plug-in or a decoder plug-in is able to call the function in the encoder integration layer at any time when appropriate. However, the Output Ready function itself provides a further benefit in terms of a latency reduction between the inputting a video signal and obtaining an encoded version of the image. Specifically, in existing plug-in implementations an encoder integration layer would call an encode function in the plug-in and pass it data for encoding. However, the plug-in does not return the output until the base layer has completed processing the base encoding. For example, see step 31 in Figure 2, where the plug-in 22 waits until encoding and recon generation is complete before returning the output to the encoder integration layer 21. It will be noted that the encoded data is given to the encoder integration layer at step 28, however the encoder integration layer cannot use this data until the plug-in 22 has completed all processing. In other words, even once the base layer encoding is complete, the base encoded version of the image is not made available to the encoder integration layer until recon generation is complete. Instead, the Output Ready callback allows the plug-in to call into the encoder integration layer as needed when output becomes available. In other words, by allowing the plug-in to call the Output Ready callback, the plug-in can at any time, call into the encoder integration layer to notify the encoder integration layer that output is available. It is to be noted that the callback function could be provided as part of the plug-in itself. This still provides the advantage of latency reduction by allowing the plugin to call into the encoder integration layer to notify of available output at any time. Another example of a callback function is the Error callback function. The error callback function allows a plug-in to call into an encoder integration layer to notify that an error has occurred in an encoding or decoding process. The encoder integration layer can react in real-time in response to the indication from the Error callback function. The Error callback could be called in response to any kind of error encountered by the plug-in during an encoding or decoding operation. In an example, there may be three ways of responding to an Error notification. These are given purely as an example, and it would be understood that there are additional ways of responding to an Error notification. 1) Abort processing immediately; 2) set an error flag in the base plug-in asynchronous control loop and continue executing; or 3) set an error flag in the base plug-in asynchronous control loop and attempt to synchronise with control loop to prevent any further calls to the base plug-in. Providing one or more callback functions in an encoder integration layer also overcomes the need for a plug-in to be designed around the interface used by a specific base codec. Broadly speaking, there are three types of interfaces utilised by various base codecs. These are: 1) Synchronous function-call interface, in which the encode function blocks until output is available; 2) Asynchronous function-call interface, in which a user typically calls both an encode function and a get_output function; and 3) Asynchronous callback interface. Providing an encoder integration layer which passes pointers to certain callback functions to a plug-in allows one plug-in API to support the above three interfaces with minimal friction. An asynchronous callback interface can be supported with a callback based plug-in, however for the synchronous function-call and asynchronous function-call interfaces, the plug-in must be able to call a callback function repeatedly, since it would be suboptimal to create a thread in the plug-in to support each interface separately. This can be achieved by introducing a special return code from the encode / decode entry point which signifies to the encoder integration layer that it may call the AGAIN function. The AGAIN function is discussed further below.. Examples of call sequences implementing the Output Ready callback function are shown in Figures 6 and 7. Figure 6 shows an example encode call sequence for asynchronous function-call- interface base codecs. At initialisation time, the encoder integration layer 61 will pass a table of function pointers pointing to callback functions in the encoder integration layer into the plug-in 62. Ina filling phase 63 of the plug-in pipeline, the encoder integration layer 61 instructs the plug-in 62 to encode two images, referred to as picO and pic1. Note that initially no output is returned for picO and pic1. This is because the plug-in has a two-picture delay. By this we intend to mean that in operation, the output for the first picture inputted passed to the plug-in, picO, is not ready until two further pictures, i.e.- pic1 and pic2 have been passed to the plug-in. Conceptually, it can be said that the pictures are pushed through the plug-in in such a way that passing images into the plug-in drives processing operations in the plug-in. Other plug-ins may have a different amount of delay. During a subsequent regular operation phase 64, encoding of a third image pic2 is instructed and the plug-in calls Output Ready 65 to notify the encoder integration layer that picO has been encoded. The encoder integration layer 61 enqueues picO onto an internal queue for further processing (namely, the decoding and enhancement coding phases). The regular operation phase may continue until there are no more input images to pass to the plug-in 62. At such a time, a flush and termination phase 65 of the process begins. In the flush and termination phase the encoder integration layer 61 passes nullptr to the plug-in 62 to signify the end of the input. Output Ready is called for any images still being processed. In this case, the plug-in 62 notifies the encoder integration layer 61 that pic1 is ready, and pic1 is enqueued by the encoder integration layer 61. The plug-in 62 then passes an AGAIN code to the encoder integration layer 61, which instructs the encoder integration layer 61 to enter a flush sequence. An AGAIN code signifies to the encoder integration layer 61 that the plug-in 62 is requested input of another nullptr so that the plug-in 62 can flush any pictures remaining in the plug-in 62. The AGAIN code causes the encoder integration layer 61 to repeatedly call EncodePicture(nullptr) until the plug-in 62 indicates to the encoder integration layer 61 that processing of any pictures remaining in the plug-in 62 is done. The plug-in 62 notifies the encoder integration layer 61 that output is available for pic2, and the encoder integration layer 61 enqueues pic2. No more images remain in the plug-in 62, and the encoder integration layer is aware that all images have been returned to it. The plug-in 62 may optionally return Done() to the encoder integration layer 61. Done() may be an example of a callback which the plug-in 62 may call to notify the encoder integration layer 61 that the plug-in 62 has completed processing, i.e. no pictures remain in the plug-in 62. Although the encoder integration layer 61 is typically already aware that all pictures pushed into the plug-in 62 has been returned the Done callback may be useful for the purpose of error detection. For example, if 100 images are passed to the plug-in, and the plug-in returns 90 images to the encoder integration layer but also calls Done, the encoder integration layer can be made aware that an error has occurred (10 images are missing) and take appropriate steps. After the plug-in 62 has completed processing, the plug-in 62 then returns OK to the encoder integration layer 61, at which point the encoder integration layer 61 closes or destroys the plug-in 62. The OK code is an example of how the plug-in 62 signifies to the encoder integration layer 61 that the plug-in 62 has finished processing all pictures and that the encoder integration layer 61 may close or destroy the plug-in 62. Figure 7 shows an encode sequence 70 for a callback interface base codec. Again, at initialisation time, the encoder integration layer 71 will pass a table of function pointers pointing to callback functions in the encoder integration layer into the plug-in 72. Unlike in Figure 6, the encode sequence for a callback interface-based codec may not be segmented into phases. However, the example process is still largely similar, in that the encoder integration layer 71 calls EncodePicture with images picO and pic1, and calls EncodePicture(nullptr) when no images remain to be encoded. The Output Ready callback is called when output is available and it will be noted that the Output Ready callback may be called from another thread in the plug-in 72. When EncodePicture(nullptr) is called, since the base encoder has a callback interface, the encoder integration layer 71 does not need to call the flush sequence, and the plug-in can simply return OK. That is to say, once EncodePicture(nullptr) has been called, EncodePicture is not called again. After the plug-in 72 returns OK, signifying that the plug-in 62 has completed processing, the encoder integration layer 71 closes or destroys the plug-in 72. It is to be noted that an encoder integration layer can ignore any callbacks received after Close() has completed, though this is not expected to occur. Figure 8 provides a flow diagram showing a method 800 of encoding a video signal using a video encoder as discussed above. At step 801, the encoder integration layer is instructed to encode an input video signal comprising a frame. Then at step 802, the encoder integration layer controls the encoder plug-in to generate an encoded version of the frame. Subsequently, at step 803, the encoder plug-in instructs the base codec to generate the encoded version of the frame. At step 804, the encoder plug-in passes the encoded version of the frame to the encoder integration layer. Step 805 comprises controlling, by the encoder integration layer, the decoder plugin to generate a decoded version of the encoded version of the frame. Then in step 806 the decoder plug-in instructs the base codec to generate the decoded frame. In step 807 the decoder plug-in passes the decoded version of the frame to the encoder integration layer. Examples presented herein provide a video encoder comprising an encoder plug-in and a decoder plug-in each providing a respective wrapper for a base codec to implement a base coding layer, each wrapper implementing an interface for data exchange with the base codec, the base coding layer being configured to encode and decode a video signal. Providing a separate encoder plug-in and decoder plug-in provides increased composability of the base coding layer. This means that a base coding layer can be implemented with combination of different base encoders and decoders. Whereas in existing implementations a plug-in had to be designed for every combination, the split plug-in arrangement of this disclosure allows for greater freedom in mixing and matching base coders. Base coder developers therefore face fewer challenges in interfacing newly developed or updated base coders with an enhancement encoder. Also discussed was the concept of an encoder integration layer comprising one or more callback functions which the encoder plug-in and decoder plug-in can call. An example callback function is the Output Ready callback, which allows a plug-in to call the encoder integration layer at any time to notify of an available output. This reduces latency in the encoding pipeline. This simplifies plug-in development and again decreases the challenge faced by plug-in developers seeking to integrate an enhancement encoder. At the encoder, for example implemented in a streaming server or client device or device encoding from a data store, methods and processes described herein can be embodied as code (e.g., software code) and / or data. The encoder may be implemented in hardware and / or software as is well-known in the art of data compression. For example, hardware acceleration using a specifically programmed Graphical Processing Unit (GPU) or a specifically designed Field Programmable Gate Array (FPGA) may provide certain efficiencies. This may be the case for implementation of a set of base codecs and / or the LCEVC encoder. For completeness, code and data can be stored on one or more computer-readable media, which may include any device or medium that can store code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on a computer-readable medium, the computer system performs the methods and processes embodied as data structures and code stored within the computer-readable storage medium. In certain embodiments, one or more of the steps of the methods and processes described herein can be performed by a processor (e.g., a processor of a computer system or data storage system). Generally, any of the functionality described in this text or illustrated in the figures can be implemented using software, firmware (e.g., fixed logic circuitry), programmable or nonprogrammable hardware, or a combination of these implementations. The terms “component” or “function” as used herein generally represents software, firmware, hardware or a combination of these. For instance, in the case of a software implementation, the terms “component” or “function” may refer to program code that performs specified tasks when executed on a processing device or devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

Claims

1. A video encoding module comprising:an encoder plug-in providing a wrapper for a first base codec to implement an interface for data exchange with the first base codec to generate an encoded version of an input video signal;a decoder plug-in providing a wrapper for a second base codec to implement an interface for data exchange with the second base codec to generate a decoded version of the encoded video signal generated by the encoder plug-in, wherein the encoder plug-in and the decoder plug-in together implement a base coding layer configured to encode and decode a video signal; and,an encoder integration layer, to control operation of the encoder plug-in and the decoder plug-in and to control an enhancement encoder, wherein the encoder integration layer is configured to:instruct the encoder plug-in to generate the encoded version of the input video signal;instruct the decoder plug-in to generate the decoded version of the encoded video signal generated by the encoder plug-in; and,instruct the enhancement encoder to generate an encoded enhancement signal for the encoded video signal,wherein the enhancement encoder implements the enhancement encoding layer, the enhancement encoder being configured to:receive the decoded video signal from the decoder plug-in; and, generate the encoded enhancement signal for the encoded videosignal, the encoded enhancement signal comprising one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal.

2. The video encoding module of claim 1, wherein the second base codec is different to the first base codec.

3. The video encoding module of claim 2, wherein the first base codec and second base codec are each configured to perform respective encoding and decoding operations according to a first coding scheme.

4. The video encoding module of claim 2 or 3 wherein the first base codec is a hardware encoder and the second base codec is a GPU configured to perform decoding operations.

5. The video encoding module of claim 2 or 3, wherein the first base codec is a GPU configured to perform encoding operations and the second base codec is a hardware decoder.

6. The video encoding module of any preceding claim, wherein the encoder integration layer comprises one or more callback functions, and the encoder integration layer is configured to pass one or more pointers to respective callback functions to the encoder plug-in and / or the decoder plug-in.

7. The video encoding module of claim 6, wherein the one or more pointers points to an OutputReady callback function, the OutputReady callback function being configured to be called by the encoder plug-in or the decoder plug-in when an encoding operation or a decoding operation is completed to indicate to the encoder integration layer that a frame of the video signal has been encoded or decoded.

8. The video encoding module of any of claims 6 or 7, wherein the one or more pointers points to an Error callback function, the Error callback function being configured to be called by the encoder plug-in or the decoder plug-in when an error occurs in an encoding operation or a decoding operation, to indicate to the encoder integration layer that an error has occurred in an encoding operation or in a decoding operation.

9. The video encoding module of any preceding claim wherein the encoder integration layer is configured to instruct the encoder plug-in to generate the encoded version of the input video signal in response to a call made from anapplication layer to instruct video encoding, wherein the encoder integration layer provides a control interface for the video encoding module.

10. A method of encoding a video signal using a video encoding module the method comprising:instructing, by an application layer, an encoder integration layer to encode an input video signal comprising a frame;controlling, by the encoder integration layer, an encoder plug-in providing a wrapper for a first base codec to generate an encoded version of the frame according to the first base codec;instructing, by the encoder plug-in, the first base codec to generate the encoded version of the frame;passing, by the encoder plug-in, the encoded version of the frame to the encoder integration layer;controlling, by the encoder integration layer, a decoder plug-in providing a wrapperfor a second base codec to generate a decoded version of the encoded version of the frame according to a second base codec;instructing, by the decoder plug-in, the second base codec to generate the decoded frame;passing, by the decoder plug-in, the decoded version of the frame to the encoder integration layer;controlling, by the encoder integration layer, an enhancement encoder to generate an encoding of the input video signal comprising an encoded enhancement signal comprising one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal.

11. The method of claim 10, wherein the second base codec is different to the first base codec.

12. The method of claim 11, wherein the first base codec and second base codec are each configured to perform respective encoding and decoding operations according to a first coding scheme.

13. The method according to claim 11 or 12, wherein the first base codec is a hardware encoder and the second base codec is a GPU configured to perform decoding operations.

14. The method of claim 11 or 12, wherein the first base codec is a GPU configured to perform encoding operations and the second base codec is a hardware decoder.

15. The method according to any of claims 10 to 14, wherein the encoder integration layer comprises one or more callback functions, and the method further comprises:passing, by the encoder integration layer, one or more pointers to respective callback functions to the encoder plug-in and / or the decoder plug-in.

16. The method according to claim 15, wherein the one or more pointers points to an OutputReady callback function, the method further comprising:calling, by the encoder plug-in, the Output Ready callback when an encoding operation is completed to indicate to the encoder integration layer that a frame of the video signal has been encoded; and / orcalling, by the decoder plug-in, the Output Ready callback when a decoding operation is completed to indicate to the encoder integration layer that a frame of the video signal has been decoded.

17. The method according to claim 15 or 16, wherein the one or more pointers points to an Error callback function, the method further comprising:calling, by the encoder plug-in, the Error callback function when an error occurs in an encoding operation, to indicate to the encoder integration layer that an error has occurred in an encoding operation; and / orcalling, by the decoder plug-in, the Error callback function when an error occurs in a decoding operation, to indicate to the encoder integration layer that an error has occurred in a decoding operation.

18. A computer readable medium comprising instructions which when executed by a processor, cause the processor to carry out the steps of any of claims 10 to 17.

19. A video encoding system comprising:a video encoding module according to any of claims 1 to 9;a first base codec and a second base codec; andan application layer configured to provide one or more calls to the video encoding module via a control interface provided by the encoder integration layer to instruct video encoding.

20. An encoder integration layer to control operation of:an encoder plug-in;a decoder plug-in; and,an enhancement encoder,wherein the encoder integration layer is configured to generate an encoding of an input video signal using a base coding layer and an enhancement encoding layer,wherein the encoder plug-in provides a wrapper for a first base codec to implement an interface for data exchange with the first base codec to generate an encoded version of an input video signal,wherein the decoder plug-in provides a wrapper for a second base codec to implement an interface for data exchange with the second base codec to generate a decoded version of the encoded video signal generated by the encoder plug-in, wherein the encoder plug-in and decoder plug-in together implement the base coding layer, the base coding layer being configured to encode and decode a video signal; and,the encoder integration layer controls the enhancement encoder to implement an enhancement encoding layer,the encoder integration layer being configured to:instruct the encoder plug-in to generate the encoded version of the input video signal;instruct the decoder plug-in to generate the decoded version of the encoded video signal generated by the encoder plug-in; and,instruct the enhancement encoder to generate an encoded enhancement signal for the encoded video signal, the enhancement encoder being configured to:receive the decoded video signal from the decoder plug-in, and generate an encoded enhancement signal for the encoded video signal, the encoded enhancement signal comprising one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal.

21. The encoder integration layer of claim 20, wherein the second base codec is different to the first base codec.

22. The encoder integration layer of claim 21, wherein the first base codec and second base codec are each configured to perform respective encoding and decoding operations according to a first coding scheme.

23. The encoder integration layer of claim 21 or 22, wherein the first base codec is a hardware encoder and the second base codec is a GPU configured to perform decoding operations.

24. The encoder integration layer of claim 21 or 22, wherein the first base codec is a GPU configured to perform encoding operations and the second base codec is a hardware decoder.

25. The encoder integration layer of any of claims 20 to 24, wherein the encoder integration layer comprises one or more callback functions, and the encoder integration layer is configured to pass one or more pointers to respective callback functions to the encoder plug-in and / or the decoder plug-in.

26. The encoder integration layer of claim 25, wherein the one or more pointers points to an OutputReady callback function, the OutputReady callback function being configured to be called by the encoder plug-in and / or the decoder plug-in when an encoding operation or decoding is completed to indicate to the encoder integration layer that a frame of the video signal has been encoded or decoded.

27. The encoder integration layer of any of claims 25 or 26, wherein the one or more callback function pointers points to an Error callback function, the Error callback function being configured to be called by the encoder plug-in and / or the decoder plug-in when an error occurs in an encoding operation or in a decoding operation, to indicate to the encoder integration layer that an error has occurred in an encoding operation or in a decoding operation.

28. An encoder plug-in and a decoder plug-in suitable for use with a video encoding module, wherein in use the encoder plug-in provides a wrapper for a first base codec to implement an interface for data exchange with the first base codec to generate an encoded version of an input video signal,the decoder plug-in provides a wrapper for a second base codec to implement an interface for data exchange with the second base codec to generate a decoded version of the encoded video signal generated by the encoder plug-in, wherein the encoder plug-in and decoder plug-in together implement a base coding layer configured to encode and decode a video signal;the encoder plug-in and decoder plug-in configured to be controlled by an encoder integration layer to generate an encoding of the input video signal using the base coding layer and an enhancement encoding layer,the encoder plug-in configured to generate the encoded version of the input video signal in response to an instruction from the encoder integration layer;the decoder plug-in configured to generate the decoded version of the encoded video signal generated by the encoder plug-in in response to an instruction from the encoder integration layer;wherein an enhancement encoder controlled by the encoder integration layer implements an enhancement encoding layer, the enhancement encoder being configured to:receive the decoded video signal from the decoder plug-in, and generate an encoded enhancement signal for the encoded video signal, the encoded enhancement signal comprising one or more layers of residual data, the residual data being generated based on a comparison of data derived from the decoded video signal and data derived from an input video signal.

29. An encoder plug-in and / or a decoder plug-in suitable for use with the video encoding module of any of claims 1 to 9.

Images