Method and module for a video pipeline

Abstract

A method for use in a video pipeline, the method comprising: obtaining a base decoded video signal from a base decoding layer; obtaining one or more layers of positive residual data from an enhancement decoding layer, the one or more layers of residual data being generated based on a comparison of data derived from the base decoded video signal and data derived from an original input video signal, wherein the base decoded video signal comprises values greater than the original input video signal and values lower than the original input video signal and the positive residual data comprises only values greater than or equal to zero; and calculating a weighted sum of the base decoded video signal and the one or more layers of positive residual data; and converting the weighted sum to an enhanced decoded video signal by applying an excess positivity correction.

Description

[0001] METHOD AND MODULE FOR A VIDEO PIPELINE

[0002] FIELD OF THE INVENTION

[0003] The application relates to decoders, and particularly enhancement-type decoders. Additionally, the application relates to video pipelines. A video pipeline may comprise one or more of: one or more decoders (for example, one or more of a base decoder and an enhancement decoder), a video blender, a secure memory, an unsecured memory, and a secure and / or unsecure interface for connecting to a display.

[0004] BACKGROUND

[0005] In an enhancement-type coding algorithm, such as MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC), one or more layers of residual data can be used to improve the performance of a base coding algorithm.

[0006] At an enhancement-type encoder, the residual data is calculated based on a comparison of a base decoded video signal and an original input video signal. In general the base decoded video signal is the result of encoding and then decoding the video signal, according to a base codec. Each image element (such as a pixel) of the original input video signal can have a value that is lower than the corresponding image element of the base decoded video signal or a value that is higher than the corresponding image element of the original input video. In general, the differences will comprise a mixture of higher values (positive values of the residual data) for some image elements and lower values (negative values of the residual data) for other image elements.

[0007] At an enhancement-type decoder, the residual data is used to recover the original input video signal based on the base decoded video signal. This typically involves adding the positive values of the residual data to the base decoded video signal and subtracting the negative values of the residual data from the base decoded video signal. SUMMARY OF INVENTION

[0008] It is desirable to implement an enhancement-type decoder as far as possible using existing video pipeline hardware.

[0009] Video pipelines, and especially secure video pipelines such as found in devices such as televisions and set-top boxes, often comprise fixed decoder hardware for one or more base codecs. Video pipelines additionally often comprise a blend unit used to blend two images, for example to display a configuration menu over a displayed video. Video pipelines further often comprise numerical mapping functions, for example in the form of colour corrections or lookup tables.

[0010] The present invention provides an enhancement-type decoding method which can take advantage of such features and efficiently recover an enhanced decoded video signal by applying residuals.

[0011] Additionally, by utilising existing hardware blocks in a new way, a decoder can decode video at a higher quality than was possible before. For example, if a set top box has a chip that could previously decode HD in real time, it could be possible to use the chip to decode UHD video in real time without any upgrading of hardware blocks and simply by using the hardware blocks in the new way.

[0012] According to a first aspect, there is described a method for use in a video pipeline, the method comprising: obtaining a base decoded video signal from a base decoding layer; obtaining one or more layers of positive residual data from an enhancement decoding layer, the one or more layers of residual data being generated based on a comparison of data derived from the base decoded video signal and data derived from an original input video signal, wherein the base decoded video signal comprises values greater than the original input video signal and values lower than the original input video signal and the positive residual data comprises only values greater than or equal to zero; and calculating a weighted sum of the base decoded video signal and the one or more layers of positive residual data; and converting the weighted sum to an enhanced decoded video signal by applying an excess positivity correction. Using the above method, much of the method can be performed within the video output path of a secure decoder, and the method can be performed more efficiently, enabling higher quality decoded video for the same processing and memory resources.

[0013] An alternative aspect is similar to the first aspect, except the excess positivity correction is applied to the one or more layers of positive residual data rather than the weighted sum, and the resulting corrected residual data is used when calculating the weighted sum. In other words, there is described a method for use in a video pipeline, the method comprising: obtaining a base decoded video signal from a base decoding layer; obtaining one or more layers of positive residual data from an enhancement decoding layer, the one or more layers of residual data being generated based on a comparison of data derived from the base decoded video signal and data derived from an original input video signal, wherein the base decoded video signal comprises values greater than the original input video signal and values lower than the original input video signal and the positive residual data comprises only values greater than or equal to zero; converting the one or more layers of positive residual data to one or more layers of corrected residual data by applying an excess positivity correction; and calculating a weighted sum of the base decoded video signal and the one or more layers of corrected residual data.

[0014] Optionally in the first aspect or the alternative aspect, the method comprises calculating the weighted sum using a blend unit.

[0015] Optionally in the first aspect or the alternative aspect, obtaining the one or more layers of positive residual data comprises: obtaining one or more layers of residual data from the enhancement decoding layer, the one or more layers of residual data being generated based on a comparison of data derived from a decoded video signal and data derived from an original input video signal; and processing the one or more layers of residual data to generate the one or more layers of positive residual data. Optionally in the first aspect or the alternative aspect, processing the one or more layers of residual data to generate the one or more layers of positive residual data comprises clipping the one or more layers of positive residual data according to a maximum residual value and / or a minimum residual value.

[0016] Optionally in the first aspect or the alternative aspect, a weight associated with each of the base decoded video signal and the one or more layers of residual data is greater than or equal to zero.

[0017] Optionally in the first aspect or the alternative aspect, the excess positivity correction comprises a predetermined gain and / or a predetermined offset.

[0018] Optionally in the first aspect or the alternative aspect, the excess positivity correction comprises using a lookup table to convert from a weighted sum value to an enhanced decoded video signal value.

[0019] Optionally in the first aspect or the alternative aspect, the excess positivity correction comprises applying a colour correction algorithm.

[0020] Optionally in the first aspect or the alternative aspect, the excess positivity correction further comprises clipping the enhanced decoded video signal according to a maximum decoded value and / or a minimum decoded value.

[0021] According to a second aspect, there is described a module for use in a video pipeline, wherein the module is configured to perform a method according to the first aspect or the alternative aspect.

[0022] According to further aspects, there is described a bitstream comprising data produced by a method according to the first aspect or data produced at an intermediate step of a method according to the first aspect.

[0023] In one aspect, there is described a bitstream comprising a weighted sum of a base decoded video signal and one or more layers of positive residual data, the one or more layers of positive residual data being generated based on a comparison of data derived from the base decoded video signal and data derived from an original input video signal, wherein the base decoded video signal comprises values greater than the original input video signal and values lower than the original input video signal and the positive residual data comprises only values greater than or equal to zero.

[0024] In another aspect, there is described a bitstream comprising positive residual data, the positive residual data being generated based on a comparison of data derived from a base decoded video signal and data derived from an original input video signal, wherein the base decoded video signal comprises values greater than the original input video signal and values lower than the original input video signal and the positive residual data comprises only values greater than or equal to zero.

[0025] In a further aspect there is described a computer-readable storage device storing the bitstream of one of the preceding aspects.

[0026] BRIEF DESCRIPTION OF DRAWINGS

[0027] Examples of systems and methods in accordance with the invention will now be described with reference to the accompanying drawings, in which:

[0028] Figure 1 shows a known, high-level schematic of an LCEVC decoding process;

[0029] Figures 2a and 2b respectively show a schematic of a comparative base decoder and a schematic of a decoder integration layer in a video pipeline;

[0030] Figure 3 illustrates a known, high-level schematic of a video decoder chipset;

[0031] Figure 4 illustrates a schematic of a video pipeline according to examples of the present disclosure;

[0032] Figure 5 illustrates a schematic block diagram of a video decoding method according to examples of the present disclosure;

[0033] Figure 6A illustrates a flow diagram of a method of generating positive residuals according to examples of the present disclosure; Figure 6B illustrates a flow diagram of a method of generating an intermediate video signal according to examples of the present disclosure;

[0034] Figure 6C illustrates a flow diagram of a method of reconstructing an original input video signal according to examples of the present disclosure;

[0035] Figure 7 illustrates a high-level schematic of a video decoder chipset according to examples of the present disclosure; and

[0036] Figure 8 illustrates a block diagram of integration of an enhancement decoder according to examples of the present disclosure.

[0037] DETAILED DESCRIPTION

[0038] A hybrid backward-compatible coding technology has been previously proposed, for example in WO 2013 / 171173, WO 2014 / 170819, WO 2019 / 141987, and WO 2018 / 046940, the contents of which are incorporated herein by reference. Further examples of tier-based coding formats include ISO / IEC MPEG-5 Part 2 LCEVC (hereafter ‘LCEVC’). LCEVC has been described in WO 2020 / 188273A1 , GB 2018723.3, WO 2020 / 188242, and the associated standard specification documents including the Draft Text of ISO / IEC DIS 23094-2 Low Complexity Enhancement Video Coding published at MPEG 129 meeting in Brussels, held Monday, 13 January 2020 to Friday, 17 January 2020, all of these documents being incorporated by reference herein in their entirety.

[0039] In these coding formats a signal is decomposed in multiple “echelons” (also known as “hierarchical tiers”) of data, each corresponding to a “Level of Quality”, from the highest echelon at the sampling rate of the original signal to a lowest echelon. The lowest echelon is typically a low quality rendition of the original signal and other echelons contain information on correction to apply to a reconstructed rendition in order to produce the final output.

[0040] LCEVC adopts this multi-layer approach where any base codec (for example Advanced Video Coding - AVC, also known as H.264, or High Efficiency Video Coding - HEVC, also known as H.265) can be enhanced via an additional low bitrate stream. LCEVC is defined by two component streams, a base stream typically decodable by a hardware decoder and an enhancement stream consisting of one or more enhancement layers suitable for software processing implementation with sustainable power consumption.

[0041] In the specific LCEVC example of these tiered formats, the process works by encoding a lower resolution version of a source image using any existing codec (the base codec) and the difference between the reconstructed lower resolution image and the source using a different compression method (the enhancement).

[0042] The remaining details that make up the difference with the source are efficiently and rapidly compressed with LCEVC, which uses specific tools designed to compress residual data. The LCEVC enhancement compresses residual information on at least two layers, one at the resolution of the base to correct artefacts caused by the base encoding process and one at the source resolution that adds details to reconstruct the output frames. Between the two reconstructions the picture is upscaled using either a normative up-sampler or a custom one specified by the encoder in the bitstream. In addition, LCEVC also performs some non-linear operations called residual prediction, which further improve the reconstruction process preceding residual addition, collectively producing a low-complexity smart content-adaptive (i.e., encoder driven) upscaling.

[0043] Since LCEVC and similar coding formats leverage existing decoders and are inherently backwards-compatible, there exists a need for efficient and effective integration with existing video coding implementations without complete redesign. Examples of known video coding implementations include the software tool FFmpeg, which is used by the simple media player FFplay.

[0044] Moreover, LCEVC is not limited to known codecs and is theoretically capable of leveraging yet-to-be-developed codecs. As such any LCEVC implementation should be capable of integration with any hitherto known or yet-to-be-developed codec, implemented in hardware or software, without introducing coding complexity. LCEVC is an enhancement codec, meaning that it does not just upsample well: it will also encode the residual information necessary for true fidelity to the source and compress it (transforming, quantizing and coding it). LCEVC can also produce mathematically lossless reconstructions, meaning all of the information can be encoded and transmitted and the image perfectly reconstructed. Creator’s intent, small text, logos, ads and unpredictable high-resolution details are preserved with LCEVC.

[0045] As an example:-

[0046] LCEVC can deliver 2160p 10-bit HDR video over an 8-bit AVC base encoder.

[0047] When using an HEVC base encoder for a 2160p stream, LCEVC can deliver the same quality at typically 33% less of the original bitrate i.e., lower a typical bitrate of 20 Mbit / s (HEVC only) to 15 Mbit / s or lower (LCEVC on HEVC).

[0048] The many unique benefits of LCEVC can be summarised as follows. LCEVC ... rapidly enhances the quality and cost efficiency of all codec workflows, reduces processing power requirements for serving a given resolution, is deployable via software, resulting in much lower power consumption, simplifies the transition from older generation to newer generation codecs. improves engagement by increasing visual quality at a given bitrate, is retrofittable and backward compatible. is immediately deployable at scale via software update, has low battery consumption on user devices. reduces new codecs complexity and makes them readily deployable.

[0049] With a view to all of the above, LCEVC allows for some interesting and highly economic ways to utilise legacy devices / platforms for higher resolutions and frame rates without the need to swap the entire hardware, ignoring customers with legacy devices, or creating duplicate services for new devices. That way the introduction of higher quality video services on legacy platforms at the same time generates demand for devices with even better coding performance. In addition, LCEVC not only eliminates the need to upgrade the platform, but it also allows for delivery of higher resolution content over existing delivery networks that might have limited bandwidth capability.

[0050] The approach of LCEVC being a codec agnostic enhancer based on a software- driven implementation, which leverages available hardware acceleration, also shows in the wider variety of implementation options on the decoding side. While existing decoders are typically implemented in hardware at the bottom of the stack, LCEVC basically allows for implementation on a variety of levels i.e., from Scripting and Application to the OS and Driver level and all the way to the SoC and ASIC. In other words, there is more than one solution to implement LCEVC on the decoder side. Generally speaking, the lower in the stack the implementation takes place, the more device specific the approach becomes. Except for an implementation on ASIC level, no new hardware is needed.

[0051] Challenges exist when attempting to integrate LCEVC decoding into video decoder pipeline chipsets without re-designing those chipsets. It is desirable, at least in the short term, to implement LCEVC in a simple manner using existing architectures and designs. There are particular implementation challenges in relation to secure decoding of protected (e.g. premium) content.

[0052] In general, one place to perform operations for the LCEVC reconstruction stage, i.e. the combination of the residuals of the decoded enhancement and the base decoded video, is in the video output path. This is because the video output path is the most secure but also because such use is memory efficient, involving direct operations being performed on secure memory.

[0053] However, such an implementation in the video output path involves dealing with inherent hardware limitations. These hardware limitations include for example low memory bandwidth and limitations on the type of operations that can be performed. Elements of the video output path such as the video shifter (alternatively referred to as the graphics feeder) are specifically designed for, and excel, at functions such as overlay and colour space conversion but are limited for wider use. Different blocks of the video output path have different limitations and trade-offs and different blocks from different manufacturers have different functionalities. For example, a hardware upscaler designed for that specific use might have different trade-offs to a video shifter. Identifying how to implement LCEVC reconstruction within the video output path involves compromises. These challenges are exacerbated when dealing with operations at UHD resolutions.

[0054] Alternative implementation of LCEVC reconstruction into the decoder CPU may be insecure as the CPU is not a protected pipeline, while implementations of LCEVC into the video output path are potentially limited by those inherent hardware limitations of the blocks of the path. Implementations thus have the potential to be inefficient.

[0055] Innovations are sought which address the limitations of video decoder chipsets and facilitate the introduction and implementation of enhancement decoders, such as LCEVC, into the wider video decoder ecosystems.

[0056] This disclosure describes an implementation for integration of a hybrid backward-compatible coding technology with existing decoders, optionally via a software update. In a non-limiting example, the disclosure relates to an implementation and integration of MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC). LCEVC is a hybrid backward-compatible coding technology which is a flexible, adaptable, highly efficient and computationally inexpensive coding format combining a different video coding format, a base codec (i.e. an encoder-decoder pair such as AVC / H.264, HEVC / H.265, or any other present or future codec, as well as non-standard algorithms such as VP9, AV1 and others) with one or more enhancement levels of coded data.

[0057] Example hybrid backward-compatible coding technologies use a down-sampled source signal encoded using a base codec to form a base stream. An enhancement stream is formed using an encoded set of residuals which correct or enhance the base stream for example by increasing resolution or by increasing frame rate. There may be multiple levels of enhancement data in a hierarchical structure. In certain arrangements, the base stream may be decoded by a hardware decoder while the enhancement stream may be suitable for being processed using a software implementation. Thus, streams are considered to be a base stream and one or more enhancement streams, where there are typically two enhancement streams possible but often one enhancement stream used. It is worth noting that typically the base stream may be decodable by a hardware decoder while the enhancement stream(s) may be suitable for software processing implementation with suitable power consumption. Streams can also be considered as layers.

[0058] The video frame is encoded hierarchically as opposed to using block-based approaches as done in the MPEG family of algorithms. Hierarchically encoding a frame includes generating residuals for the full frame, and then a reduced or decimated frame and so on. In the examples described herein, residuals may be considered to be errors or differences at a particular level of quality or resolution.

[0059] For context purposes only, as the detailed structure of LCEVC is known and set out in the approved draft standards specification, Figure 1 illustrates, in a logical flow, how LCEVC operates on the decoding side assuming H.264 as the base codec. Those skilled in the art will understand how the examples described herein are also applicable to other multi-layer coding schemes (e.g., those that use a base layer and an enhancement layer) based on the general description of LCEVC that is presented with reference to Figure 1. Turning to Figure 1 , the LCEVC decoder 10 works at individual video frame level. It takes as an input a decoded low-resolution picture from a base (H.264 or other) video decoder 11 and the LCEVC enhancement data to produce a decoded full-resolution picture ready for rendering on the display view. The LCEVC enhancement data is typically received either in Supplemental Enhancement Information (SEI) of the H.264 Network Abstraction Layer (NAL), or in an additional data Packet Identifier (PID) and is separated from the base encoded video by a demultiplexer 12. Hence, the base video decoder 11 receives a demultiplexed encoded base stream and the LCEVC decoder 10 receives a demultiplexed encoded enhancement stream, which is decoded by the LCEVC decoder 10 to generate a set of residuals for combination with the decoded low-resolution picture from the base video decoder 11 .

[0060] LCEVC can be rapidly implemented in existing decoders with a software update and is inherently backwards-compatible since devices that have not yet been updated to decode LCEVC are able to play the video using the underlying base codec, which further simplifies deployment.

[0061] In this context, there is proposed herein a decoder implementation to integrate decoding and rendering with existing systems and devices that perform base decoding. The integration is easy to deploy. It also enables the support of a broad range of encoding and player vendors, and can be updated easily to support future systems. Embodiments of the invention specifically relate to how to implement LCEVC in such a way as to provide for decoding of protected content in a secure manner.

[0062] The proposed decoder implementation may be provided through an optimised software library for decoding MPEG-5 LCEVC enhanced streams, providing a simple yet powerful control interface or API. This allows developers flexibility and the ability to deploy LCEVC at any level of a software stack, e.g. from low-level command-line tools to integrations with commonly used open-source encoders and players. In particular, embodiments of the present invention generally relate to a driver-level implementations and a System on a chip (SoC) level implementation.

[0063] The terms LCEVC and enhancement may be used herein interchangeably, for example, the enhancement layer may comprise one or more enhancement streams, that is, the residuals data of the LCEVC enhancement data.

[0064] Figure 2a illustrates an unmodified video pipeline 20. In this conceptual pipeline, obtained or received Network Abstraction Layer (NAL) units are input to a base decoder 22. The base decoder 22 may, for example, be a low-level media codec accessed using a mechanism such as MediaCodec (e.g. as found in the Android (RTM) operating system), VTDecompression Session (e.g. as found in the iOS (RTM) operating system) or Media Foundation Transforms (MFT - e.g. as found in the Windows (RTM) family of operating systems), depending on the operating system. The output of the pipeline is a surface 23 representing the decoded original video signal (e.g. a frame of such a video signal, where sequential display of success frames renders the video).

[0065] Figure 2b illustrates a proposed video pipeline using an LCEVC decoder integration layer, conceptually. Like the comparative video decoder pipeline of Figure 2a, NAL units 24 are obtained or received and are processed by an LCEVC decoder 25 to provide a surface 28 of reconstructed video data. Through the use of the LCEVC decoder 25, the surface 28 may be higher quality than the comparative surface 23 in Figure 2a or the surface 28 may be at the same quality as the comparative surface 23 but require fewer processing and / or network resources.

[0066] In Figure 2b, the LCEVC decoder 25 is implemented in conjunction with a base decoder 26. The base decoder 26 may be provided by a variety of mechanisms, including by an operating system function as discussed above (e.g. may use a MediaCodec, VTDecompression Session or MFT interface or command). The base decoder 26 may be hardware accelerated, e.g. using dedicated processing chips to implement operations for a particular codec. The base decoder 26 may be the same base decoder that is shown as 22 in Figure 2a and that is used for other non-LCEVC video decoding, e.g. may comprise a pre-existing base decoder.

[0067] In Figure 2b, the LCEVC decoder 25 is implemented using a decoder integration layer (DIL) 27. The decoder integration layer 27 acts to provide a control interface for the LCEVC decoder 25, such that a client application may use the LCEVC decoder 25 in a similar manner to the base decoder 22 shown in Figure 2a, e.g. as a complete solution from buffer to output. The decoder integration layer 27 functions to control operation of a decoder plug-in (DPI) 27a and an enhancement decoder 27b to generate a decoded reconstruction of an original input video signal. In certain variations, as shown in Figure 2b, the decoder integration layer may also control GPU functions 27c such as GPU shaders to reconstruct the original input video signal from the decoded base stream and the decoded enhancement stream.

[0068] NAL units 24 comprising the encoded video signal together with associated enhancement data may be provided in one or more input buffers. The input buffers may be fed (or made available) to the base decoder 26 and to the decoder integration layer 27, in particular the enhancement decoder that is controlled by the decoder integration layer 27. In certain examples, the encoded video signal may comprise an encoded base stream and be received separately from an encoded enhancement stream comprising the enhancement data; in other preferred examples, the encoded video signal comprising the encoded base stream may be received together with the encoded enhancement stream, e.g. as a single multiplexed encoded video stream. In the latter case, the same buffers may be fed (or made available) to both the base decoder 26 and to the decoder integration layer 27. In this case, the base decoder 26 may retrieve the encoded video signal comprising the encoded base stream and ignore any enhancement data in the NAL units. For example, the enhancement data may be carried in SEI messages for a base stream of video data, which may be ignored by the base decoder 26 if it is not adapted to process custom SEI message data. In this case, the base decoder 26 may operate as per the base decoder 22 in Figure 2a, although in certain cases, the base video stream may be at a lower resolution that comparative cases.

[0069] On receipt of the encoded video signal comprising the encoded base stream, the base decoder 26 is configured to decode and output the encoded video signal as one or more base decoded frames. This output may then be received or accessed by the decoder integration layer 27 for enhancement. In one set of examples, the base decoded frames are passed as inputs to the decoder integration layer 27 in presentation order.

[0070] The decoder integration layer 27 extracts the LCEVC enhancement data from the input buffers and decodes the enhancement data. Decoding of the enhancement data is performed by the enhancement decoder 27b, which receives the enhancement data from the input buffers as an encoded enhancement signal and extracts residual data by applying an enhancement decoding pipeline to one or more streams of encoded residual data. For example, the enhancement decoder 27b may implement an LCEVC standard decoder as set out in the LCEVC specification.

[0071] A decoder plug-in is provided at the decoder integration layer to control the functions of the base decoder. In certain cases, the decoder plug-in 27a may handle receipt and / or access of the base decoded video frames and apply the LCEVC enhancement to these frames, preferably during playback. In other cases, the decoder plug-in may arrange for the output of the base decoder 26 to be accessible to the decoder integration layer 27, which is then arranged to control addition of a residual output from the enhancement decoder to generate the output surface 28. Once integrated in a decoding device, the LCEVC decoder 25 enables decoding and playback of video encoded with LCEVC enhancement. Rendering of a decoded, reconstructed video signal may be supported by one or more GPU functions 27c such as GPU shaders that are controlled by the decoder integration layer 27.

[0072] In general, the decoder integration layer 27 controls operation of the one or more decoder plug-ins and the enhancement decoder to generate a decoded reconstruction of the original input video signal 28 using a decoded video signal from the base encoding layer (i.e. as implemented by the base decoder 26) and the one or more layers of residual data from the enhancement encoding layer (i.e. as implemented by the enhancement decoder). The decoder integration layer 27 provides a control interface, e.g. to applications within a client device, for the video decoder 25.

[0073] Depending on configuration, the decoder integration layer may output the surface 28 of decoded data in different ways. For example, as a buffer, as an offscreen texture or as an on-screen surface. Which output format to use may be set in configuration settings that are provided upon creation of an instance of the decoding integration layer 27, as further explained below. In certain implementations, where no enhancement data is found in the input buffers, e.g. where the NAL units 24 do not contain enhancement data, the decoder integration layer 27 may fall back to passing through the video signal at the lower resolution to the output, that is, the output of the base decoding layer as implemented by the base decoder 26. In this case, the LCEVC decoder 25 may operate as per the video decoder pipeline 20 in Figure 2a.

[0074] The decoder integration layer 27 can be used for both application integration and operating system integration, e.g. for use by both client applications and operating systems. The decoder integration layer 27 may be used to control operating system functions, such as function calls to hardware accelerated base codecs, without the need for a client application to have knowledge of these functions. In certain cases, a plurality of decoder plug-ins may be provided, where each decoder plug-in provides a wrapper for a different base codec. It is also possible for a common base codec to have multiple decoder plug-ins. This may be the case where there are different implementations of a base codec, such as a GPU accelerated version, a native hardware accelerated version and an open-source software version.

[0075] When viewing the schematic diagram of Figure 2b, the decoder plug-ins may be considered integrated with the base decoder 26 or alternatively a wrapper around that base decoder 26. Effectively Figure 2b can be thought of as a stacked visualisation. The decoder integration layer 27 in Figure 2b, conceptually includes functionality to extract the enhancement data from the NAL units 27b, functionality 27a to communicate with the decoder plug-ins and apply enhancement decoded data to base decoded data and one or more GPU functions 27c.

[0076] The set of decoder plug-ins are configured to present a common interface (i.e. a common set of commands) to the decoder integration layer 27, such that the decoder integration layer 27 may operate without knowledge of the specific commands or functionality of each base decoder. The plug-ins thus allow for base codec specific commands, such as MediaCodec, VTDecompression Session or MFT, to be mapped to a set of plug-in commands that are accessible by the decoder integration layer 27 (e.g. multiple different decoding function calls may be mapped to a single common plug-in “Decode(...)” function).

[0077] Since the decoder integration layer 27 effectively comprises a ‘residuals engine’, i.e. a library that from the LCEVC encoded NAL units produces a set of correction planes at different levels of quality, the layer can behave as a complete decoder (i.e. the same as decoder 22) through control of the base decoder.

[0078] For simplicity, we will refer to the instructing entity here as the client but it will be understood that the client may be considered to be any application layer or functional layer and that the decoder integration layer 27 may be integrated simply and easily into a software solution. The terms client, application layer and user may be used herein interchangeably.

[0079] In an application integration, the decoder integration layer 27 may be configured to render directly to an on-screen surface, provided by a client, of arbitrary size (generally different from the content resolution). For example, even though a base decoded video may be Standard Definition (SD), the decoder integration layer 27, using the enhancement data, may render surfaces at High Definition (HD), Ultra High Definition (UHD) or a custom resolution. Further details of out- of-standard methods of upscaling and post-processing that may be applied to a LCEVC decoded video stream are found in PCT / GB2020 / 052420, the contents of which are incorporated herein by reference. Example application integrations include, for example, use of the LCEVC decoder 25 by ExoPlayer, an application level media player for Android, or VLCKit, an objective C wrapper for the libVLC media framework. In these cases, VLCKit and / or ExoPlayer may be configured to decode LCEVC video streams by using the LCEVC decoder 25 “under the hood”, where computer program code for VLCKit and / or ExoPlayer functions is configured to use and call commands provided by the decoder integration layer 27, i.e. the control interface of the LCEVC decoder 25. A VLCKit integration may be used to provide LCEVC rendering on iOS devices and an ExoPlayer integration may be used to provide LCEVC rendering on Android devices. In an operating system integration, the decoder integration layer 27 may be configured to decode to a buffer or draw on an off-screen texture of the same size of the content final resolution. In this case, the decoder integration layer 27 may be configured such that it does not handle the final render to a display, such as a display device. In these cases, the final rendering may be handled by the operating system, and as such the operating system may use the control interface provided by the decoder integration layer 27 to provide LCEVC decoding as part of an operating system call. In these cases, the operating system may implement additional operations around the LCEVC decoding, such as YUV to RGB conversion, and / or resizing to the destination surface prior to the final rendering on a display device. Examples of operating system integration include integration with (or behind) MFT decoder for Microsoft Windows (RTM) operating systems or with (or behind) Open Media Acceleration (OpenMAX - OMX) decoder, OMX being a C-language based set of programming interfaces (e.g. at the kernel level) for low power and embedded systems, including smartphones, digital media players, games consoles and set-top boxes.

[0080] These modes of integration may be set by a client device or application.

[0081] The configuration of Figure 2b, and the use of a decoder integration layer, allows LCEVC decoding and rendering to be integrated with many different types of existing legacy (i.e. base) decoder implementations. For example, the configuration of Figure 2b may be seen as a retrofit for the configuration of Figure 2a as may be found on computing devices. Further examples of integrations include the LCEVC decoding libraries being made available within common video coding tools such as FFmpeg and FFplay. For example, FFmpeg is often used as an underlying video coding tool within client applications. By configuring the decoder integration layer as a plug-in or patch for FFmpeg, an LCEVC-enabled FFmpeg decoder may be provided, such that client applications may use the known functionalities of FFmpeg and FFplay to decode LCEVC (i.e. enhanced) video streams. For example an LCEVC-enabled FFmpeg decoder may provide video decoding operations, such as: playback, decoding to YUV and running metrics (e.g. peak signal-to-noise ratio - PSNR or Video Multimethod Assessment Fusion - VMAF - metrics) without having to first decode to YUV. This may be possible by the plug-in or patch computer program code for FFmpeg calling functions provided by the decoder integration layer.

[0082] As described above, to integrate an LCEVC decoder such as 25 into a client, i.e. an application or operating system, a decoder integration layer such as 27 provides a control interface, or API, to receive instructions and configurations and exchange information.

[0083] Figure 3 illustrates a computing system 100a comprising a conventional video shifter 131 a. The computing system 100a is configured to decode a video signal, where the video signal is encoded using a single codec, for example VVC, AVC or HEVC. In other words, the computing system 100a is not configured to decode a video signal encoded using a tier-based codec such as LCEVC. The computing system 100a further comprises a receiving module 103a, a video decoding module 117a, an output module 131 a, an unsecure memory 109a, a secure memory 110a, and a CPU or GPU 113a. The computing system 100a is in connection with a protected display (not illustrated).

[0084] The receiving module 103a is configured to receive an encrypted stream 101 a, separate the encrypted stream, and output decrypted secure content 107a (e.g. decrypted encoded video signal, encoded using a single codec) to secure memory 110a. The receiving module 103a is configured to output unprotected content 105a, such as audio or subtitles, to the unsecure memory 109a. The unprotected content may be processed 111a by the CPU or GPU 113a. The (processed) unprotected content is output 115a to the video shifter 131 a.

[0085] The video decoder 117a is configured to receive 119a the decrypted secure content (e.g. decrypted encoded video signal) and decode the decrypted secure content. The decoded decrypted secure content is sent 121 a to the secure memory 110a and subsequently stored in the secure memory 110a. The decoded decrypted secure content is output 125a, from the secure memory, to the video shifter 131 a. In other words, the video shifter 131a: reads the decoded decrypted secure content 125a from the secure memory; reads 115a the unsecure content, for example, subtitles from the unsecure memory 109a; combines the decoded decrypted secure content and the subtitles; and outputs the combined data 133a to a protected display.

[0086] The various components (i.e. the modules and the memory memory) are connected via a number of channels. The channels, also referred to as pipes, are communication channels that allow data to flow between the two components at each end of the channel. In general, channels connected to the secure memory 110c are secured channels. Channels connected to the unsecure memory 109c are unsecure channels.

[0087] PCT / GB2022 / 051238, herein incorporated by reference in its entirety, discusses various examples of implementing LCEVC reconstruction on video decoders used in, for example, set-top boxes. The security relevant part of the tier-based (e.g. LCEVC) decoder implementation lies in the processing steps where the decoded enhancement layer is combined with the decoded (and upscaled) base layer to create the final output sequence. Depending on what level of the stack the tier based (e.g. LCEVC) decoder is being implemented, different approaches exist to establish a secure and ECP compliant content workflow.

[0088] With the base decoder utilising the Secure Memory, PCT / GB2022 / 051238 discuss how to combine the output from the base decoder in Secure Memory and the LCEVC decoder output in General Purpose Memory to assemble the enhanced output sequence. Two similar approaches are proposed: to provide a secure decoder when LCEVC is implemented at a driver level implementation; or to provide a secure decoder when LCEVC is implemented at a System on a Chip (SoC) level. Which approach of the two is utilised may depend on the capabilities of the chipset used in the respective decoding device.

[0089] Implementing LCEVC (or other tier-based codecs) on a device driver level utilises hardware blocks or GPU. In general, once the base layer and the (e.g. LCEVC) enhancement layer have been separated, most of the decoding of the (e.g. LCEVC) enhancement layer can take place in the CPU and hence in General Purpose (unsecure) Memory. PCT / GB2022 / 051238 proposes that a module (e.g. a secure hardware block or GPU) is used to up-sample the output of the base encoder using Secure Memory, combines the upsampled output with predicted residuals and applies the decoded enhancement layer (e.g. LCEVC residual map) coming from General Purpose (unsecure) Memory. Afterwards, the output sequence (e.g. an output plane) can be sent to a protected display via an output module (e.g. a Video Shifter), which is part of an output video path in the decoder (i.e. in the chipset).

[0090] In short, the LCEVC reconstruction stage, i.e. the steps of upsampling the base decoded video signal and combining that base decoded video signal with the one or more residual layers to create the reconstructed video, can be performed on aspects of the computing system which have access to secure memory. Examples include the video output path, such as the video shifter, a hardware block such as a hardware upscaler, or GPU of the computing system. The video shifter may also be referred to as a graphics feeder.

[0091] When a module implementing the LCEVC reconstruction stage is a hardware block, the hardware block can be used to process the data very efficiently (for example by maximising page efficiency Double Data Rate, DDR, memory).

[0092] However, not all devices have hardware blocks, moreover, not all of these blocks can read secure memory. In such cases, it may be preferable to have the module’s functionality in a GPU module (which many relevant devices have), this provides a flexible approach and can be implemented on many different devices (including phones). By writing the functionality of the module as a layer running on the GPU (e.g. using open GLES), implementations can function on a variety of different GPUs (and hence different devices), this provides a single solution to the problem (i.e. of providing secure video) that can be implemented on many devices. In this sense). This is generally in contrast with, a SoC level implementation that generally uses a device (video shifter) architecture specific implementation and therefore use a unique solution for each video shifter to, for example, call the correct functions and connecting them up. While examples described herein are provided in the context of a secure memory, for example the implementations described in PCT / GB2022 / 051238, it will be understood that the principles proposed herein are not limited as such but are provided merely for context. The benefits of the invention can be realised in video decoder implementations that do not require a protected pipeline and the protected pipeline is provided merely for additional explanation.

[0093] When integrating LCEVC into existing video decoder architectures, it may be an objective to do so in the most simple and efficient manner. While it is contemplated that LCEVC can be retrofit to existing set-top boxes, it is also advantageous to integrate LCEVC into new chipsets. It might be desirable to integrate LCEVC without significant changes to the architecture so that chipset manufacturers do not need to change design but can simply rollout LCEVC decoding quickly and easily. The ease of integration is one of the many known advantages of LCEVC. However, to implement LCEVC in this way on existing chipset designs introduces challenges.

[0094] Handling secure content is one such example, as identified above. Another example of these integration challenges is the inherent hardware limitations of the existing video decoder architectures.

[0095] In general, it may be that the most appropriate place to perform the operations of the LCEVC reconstruction stage is in the video output path of the video decoder chipset. This addresses security needs by keeping the video in the protected pipeline but it is also the most memory efficient.

[0096] Examples of hardware limitations include resources issues to handle UHD, the inability to handle ‘signed’ values, i.e. a hardware block might only handle positive values, and / or the inability to perform a subtract operation. These three example limitations introduce compromises due to the nature of the LCEVC reconstruction stage. That is, the one or more residual layers output by the enhancement decoder typically comprise ‘signed’ values (i.e. positive or negative) and the base decoded video signal must be upsampled and combined with these signed values to reconstruct the original video. In more specific examples of hardware limitations, in certain chipsets, when video is overlayed it is only possible to perform add (because the block is performing blending). Typically with blending hardware it is possible to put the hardware in additive mode but not a signed additive mode. LCEVC relies on additions and subtractions.

[0097] In an alternative specific example of a hardware limitation, a set-top box might have limited memory bandwidth. The addition and subtraction of UHD to UHD is 4x HD. For UHD, the base video is an HD image. Thus one might be able to use a hardware block to perform addition and subtraction of HD values but when these are attempted for UHD, the hardware block either breaks or it is very inefficient (caused by a lack of memory bandwidth).

[0098] So, in an example, a hardware block such as a hardware upscaler or other similar component might be able to perform subtraction but a video shifter cannot and a video shifter might not be able to handle signed values.

[0099] Typically, processors of the video pipeline are also unable to perform the necessary operations at the UHD resolution but may be able to perform certain operations of the input is in a certain form.

[0100] An overview of the present invention is illustrated in Figure 4. The invention sets out to realise an implementation in which the video output path (‘video pipeline’) is used for as many operations as possible and a hardware block, CPU or GPU for any remaining operations. Guiding principles for the implementation are primarily simplicity and, secondarily, security, i.e. the ability to decode secure content.

[0101] As shown in Fig. 4, a base decoder 401 is configured to receive a base encoded signal corresponding to a lower-quality video, and an enhancement decoder 402 is configured to receive an enhancement signal which may be used to enhance the lower-quality video and obtain a higher-quality video. The base encoded signal and the enhancement signal may be combined together in a transmitted and received signal. The enhancement decoder 402, such as for example an LCEVC decoder, comprises a residual generator 403. The residual generator is part of the enhancement operations and generates one or more layers of residual data R, by decoding the enhancement signal. Each layer may correspond to different level of quality (e.g. a different resolution). The residual data is a set of signed values (i.e. positive and negative) which generally correspond to the difference between a decoded version of an input video, decoded using the base codec, and the original input video signal.

[0102] A module 404 is proposed herein which processes the one or more or more layers of residual data to generate one or more layers of positive residual data R+. The positive residual data is a modified form of the residual data using only positive values. Negative parts of a layer obtained by the residual generator 403 are still included within the positive residual data but are modified to have values greater than or equal to zero. In other words, the ‘positive residuals’ could equally be referred to as the ‘modified residuals’ and have a similar meaning.

[0103] In some embodiments, the positive residual generator 404 generates the positive residual data by applying a fixed offset to an entire layer of residual data obtained by the residual generator 403. For example, if a layer of the enhancement signal comprises residuals with values between -X (e.g. -128) and +Y (e.g. 127), the positive residual generator 404 may increase all residual values by X, to give a range from 0 to (X + Y) (e.g. from 0 to 255).

[0104] Alternatively, the positive residual generator 404 may apply a dynamic offset to the residuals obtained by the residual generator 403. For example, the positive residual generator 404 may monitor a distribution of values of the residuals obtained by the residual generator 403, and identify an offset which would be sufficient to ensure that at least a certain proportion (e.g. 90% or 100%) of modified residuals are greater than or equal to zero. This has the advantage that the positive residual generator 404 can handle a variable bit depth for the enhancement layer. The positive residual generator 404 may apply a weighting factor before or after applying an offset. For example, a 50% weighting may be applied to reduce the bit depth of residuals before the enhancement layer is combined with the base decoded layer, for example to prevent integer overflow. The weighting factor may be between 0 and 1 . Alternatively, the weighting factor may be greater than 1 , for example in a case where the received enhancement layer comprises down- sampled or otherwise compressed residual values.

[0105] The positive residual generator 404 may apply clipping, which may be applied before or after applying an offset and / or applying a weighting factor. For example, the positive residual generator may be configured with an allowed positive residual value range, and may be configured to replace any individual residual value obtained by the residual generator 403 falling outside the allowed range with the maximum or minimum residual value of the allowed range. In other words, the allowed positive residual value range may be a smaller range than a residual value range of residuals that can be obtained by the residual generator 403.

[0106] The positive residual generator 404 is illustrated in Figure 4 as a module within the enhancement decoder 402. It should be understood that this module may be separate from the enhancement decoder and may receive the residuals generated by the enhancement decoding process, or this module may be integrated within the enhancement decoder itself.

[0107] The base decoded video Io is fed from the base decoder 401 to an upsampler 405. The upsampler 405 matches a resolution associated with the upsampled base decoded video louio a resolution associated with a layer (or each layer) of the positive residual data FF. (Of course, if the resolution of the base decoded video already matches the resolution of the layer of positive residual data, then the upsampler 405 may be omitted.) The upsampler 405 may also apply a weighting factor to be used for combining the base decoded video with the positive residual data. For example, a 50% weighting may be applied to reduce the bit depth of the base decoded video before the enhancement layer is combined with the base decoded layer. In one case, the weighting is applied to prevent integer overflow. In other words, the weighting ensures that the sum of each image element (e.g. pixel) of the base decoded video and a corresponding image element of the enhancement layer is not greater than a maximum supported value for image elements of the base decoded video.

[0108] The weighting factor for the base decoded video may be between 0 and 1. Alternatively, the weighting factor may be greater than 1 .

[0109] The upsampled base decoded video lougenerated by the upsampler 405 is combined 406 with the positive residual data R+generated by the positive residual generator 404 to produce an intermediate video signal h+. The combination may be a simple sum of residual values with data values of the base decoded video. In this case, the combination may be implemented using an adder 406. Alternatively, the combination may be a weighted sum. As previously mentioned, weights may be applied in advance by the positive residual generator 404 and / or the upsampler 405. Alternatively, weights may be applied at the stage of combination 406. In the above description, weighting is used to prevent overflow, but the weighting may have other purposes such as supporting customised ranges or specific compression of the residual values in the enhancement layer. Furthermore, in some cases, weighting may be unnecessary (i.e. weights of 1 may be used for the residual values and the data values of the base decoded video).

[0110] Some hardware components, such as a blend unit, are designed to combine values as required at stage 406, but are only capable of (or optimised for) combining positive values. Hence, by generating positive residual data before combining the residual data with the base decoded video, it becomes possible to implement the combining with a greater range of hardware. This is particularly advantageous when there are constraints on the available hardware, such as in the video output path of a secure video decoder. As one particular example, the hardware constraint may be that the enhancement decoding must be backwards compatible. One example of backwards compatible enhancement decoding is LCEVC. The advantage of backwards compatibility is that existing video pipelines which support a base decoder can be configured to also support enhancement decoding, without replacing any hardware.

[0111] The intermediate video signal h+produced by the combining 406 is expected to be different from the original video signal which was encoded and transmitted, because the residuals were modified before the combining. Accordingly, a further module 407 is provided to effectively reverse the effects of the positive residual generator 404, and output enhanced decoded video signal data h. This is described herein as an “excess positivity correction”.

[0112] As previously mentioned, some hardware (particularly in the context of a secure video decoder) is not capable of performing the full range of mathematical operations. For example, in some embodiments it is again necessary to avoid subtraction when implementing the module 407. For example, the module 407 may comprise a lookup table for converting values of the intermediate video signal to values of the enhanced decoded video signal. The conversion performed using a lookup table may, for example, be equivalent to applying a predetermined negative offset that is equal and opposite to an offset applied by the positive residual generator 404.

[0113] Video output paths of existing decoders often already incorporate a lookup table function, for example in order to perform colour correction. In such examples, a lookup table for “excess positivity correction” may be combined with an existing lookup table, to produce a combined lookup table which can be installed in the video output path instead of the colour correction lookup table, without increasing processing or memory requirements for the video output path. In decoders which are capable of receiving software updates, this can even be installed to an existing decoder.

[0114] Similarly, if colour correction is performed by any other algorithm, a combined algorithm can be calculated which performs both excess positivity correction and colour correction, and the combined algorithm can be installed in the decoder instead of the colour correction algorithm. Furthermore, in embodiments where hardware components do support subtraction, the module 407 may apply a negative offset directly to the intermediate video signal, without using a lookup table. For example, while a blend unit used for the combination 406 may not support subtraction, another hardware module that is available downstream of a blend unit may support subtraction.

[0115] Additionally or alternatively, the excess positivity correction performed by module 407 may comprise applying a predetermined gain to the intermediate video signal. For example, each value of the intermediate video signal may be multiplied by a weight that depends upon the weights applied in advance by the positive residual generator 404, the upsampler 405, and / or at the stage of combination 406.

[0116] The module 407 may also apply clipping, which may be applied before or after applying an offset and / or applying a weighting factor. For example, the module 407 may be configured with an allowed enhanced decoded value range, and may be configured to replace any enhanced decoded value falling outside the allowed range with the maximum or minimum enhanced decoded value of the allowed range.

[0117] As further shown in Figure 4, a secure decoder may typically have a secured portion. The secure portion has protection against actions such as unauthorised copying of a video signal, and the decoder is designed to prevent a decoded signal from passing from the secured portion to any unsecured (or ‘clear’) portion of the decoder. The unsecured portion of the decoder may implement additional functions which do not require security. For example, in the case of a video that contains subtitles, the decoder may handle a video signal within the secure portion, while subtitles are initially processed in the unsecured portion before being securely combined with the video signal (for example using a blend unit). The hardware in the secure portion is typically tightly constrained, and therefore the unsecured portion can be used more flexibly for some functions. Enhanced video decoding may comprise decoding a base layer in the secure portion, decoding a residual layer in the unsecured portion, and producing a combined video signal in the secure portion.

[0118] The secure portion may advantageously comprise the base video decoder 401 and a video output path 408. The video output path 408 may include one or more hardware blocks such as blend units, CPUs. GPUs and so on, and the output path 408 may implement an upsampler 405, a combining module 406 and an excess positivity correction module 407.

[0119] The unsecured portion may be configured to implement an enhancement decoder 402, comprising a residual generator 403 and a positive residual generator 404, using dedicated hardware and / or using software configurations.

[0120] Figure 5 illustrates a schematic block diagram of a video decoding method according to examples of the present disclosure. This is used to provide a worked example of a method according to the present disclosure, summarised in the following table:

[0121] In the example of Fig. 5, prior to encoding and transmission, an enhanced video signal comprised a sequence of values. The meaning of each value in the sequence may depend upon how the video signal is described. For example, each value may relate to a pixel of a frame of the video signal. Alternatively, each value may relate to a property of a vector graphic element. Different values in the video signal may have different meanings - for example the signal may comprise some values which relate to multiple frames and other values which define part of the content of a single frame. The meaning of a “value” in a video signal is not constrained here. One of these values, which is used for the worked example, is said to be “14”.

[0122] After the enhanced video signal is received at its destination, and a base layer of the enhanced video signal is decoded, the decoded base layer 5A has an example value “64”.

[0123] The base video signal value 5A is converted to a weighted base value 5B. In this example, the base weight is 50%. Therefore, the base value of “64” becomes a weighted base value of “32”. This weight may be applied by, for example, an upsampler 405 or a combiner 406.

[0124] After an enhancement layer of the enhanced video signal is decoded, the decoded enhancement layer comprises a corresponding residual value 5C has a value “-50”. The decoded enhancement layer may be output by the residual generator 403 in the configuration of Figure 4.

[0125] The residual value 5C is converted to a positive residual value 5D. In this example, the conversion comprises applying a fixed offset of 128 to each residual value. Therefore, the residual value of “-50” becomes a positive residual value of “78”. This may be performed by a positive residual generator 404 as discussed above.

[0126] The positive residual value 5D is converted to a weighted residual value 5E. In this example, the residual weight is 50%. Therefore, the positive residual value of “78” becomes a weighted residual value of “39”. This weight may be applied by, for example, the positive residual generator 404 or combiner 406 as discussed above.

[0127] The weighted base value 5B is combined with the weighted residual value 5E to give an intermediate video signal value 5F. In this example, the combination is a simple sum. Typically, the weights used for the simple sum add up to 100%. The weighted base value of “32” and the weighted residual value of “39” combine to give an intermediate video signal value of “71 ”. This may be performed by a combiner 406 as discussed above (e.g. a blend unit). For example, a blend unit may be configured to simultaneously implement the weighting and combining stages 5B, 5E, 5F.

[0128] A correction 5G is applied to the intermediate video signal value 5F to give an enhanced decoded video signal 5H. The enhanced decoded video signal may then be output to a display for viewing. In this example, the correction 5G comprises a gain of 2 (corresponding inversely to the previously mentioned weights) followed by an offset of -128 (matching the previously mentioned offset for the positive residual value 5D). Therefore, the intermediate video signal value “71 ” becomes “142” and then “14” as the value of the enhanced decoded video signal. The value “14” is the same as the enhanced video signal prior to encoding and transmission.

[0129] Of course the values discussed above with respect to Figure 5 are merely examples, and different weightings and offsets may be applied in other embodiments.

[0130] Figures 6A, 6B and 6C each represent flow diagrams of three example stages of the concepts proposed. As noted, each stage may be performed by the same or different modules of a video pipeline. For convenience we will refer to these as generation, combination and correction.

[0131] In the generation stage of Figure 6A, the module receives one or more layers of residual data (step 601) and then processes the residual data to generate one or more layers of positive residuals (step 602). The positive residual data comprises only values greater than or equal to zero. The generation stage may be implemented in an unsecured portion of a video pipeline, or alternatively may be implemented in a secure portion of the video pipeline.

[0132] At the combination stage, illustrated by the flow chart of Figure 6B, the base decoded video signal is received (step 611 ), for example from a standardised base decoder. By base decoder, here we mean a decoder implementing a base codec (for example Advanced Video Coding - AVC, also known as H.264, or High Efficiency Video Coding - HEVC, also known as H.265). If necessary to match the resolution of the residual data, the base decoded video signal is upsampled or upscaled (step 612). The terms upsampling and upscaling are used interchangeably herein. The positive residuals are received (step 613) and combined with the upscaled base decoded video signal (step 614). After combination, the combination stage may generate or output an intermediate video signal (step 615) from the combination of the positive residuals and the (upsampled) base decoded video signal.

[0133] Figure 6C illustrates the step of modifying the intermediate decoded video to compensate for the adjustment of the original residuals to convert them into only positive values. The correction stage first receives the intermediate video signal (step 621 ) after combination. An excess positivity correction (step 622) is then applied, which may comprise using a lookup table, a gain, an offset, clipping, etc. as discussed above. The correction stage outputs or generates a reconstructed video signal (step 623). A final step may comprise storing the output plane and outputting the output plane to an output module for sending to a display.

[0134] Figure 7 illustrates a high-level schematic of a video decoder chipset according to examples of the present disclosure.

[0135] Figure 7 illustrates the principles of the disclosure being implemented in a video decoding computer system 100b comprising normal general purpose memory and secure memory. The computing system comprises a receiving module 103b, a base decoding module 117b, an output module 706b, an enhancement layer decoding module 113b, an unsecure memory 109b, and a secure memory 110b. The computing system is in connection with a protected display (not illustrated).

[0136] The various components (i.e. the modules and the memory memory) are connected via a number of channels. The channels, also referred to as pipes, are communication channels that allow data to flow between the two components at each end of the channel. In general, channels connected to the secure memory 110c are secured channels. Channels connected to the unsecure memory 109c are unsecure channels. For ease of display, the channels are not explicit illustrated in the figures, rather, the data flow between various modules is shown.

[0137] The receiving module 103b is configured to receive, as a single stream, the video signal 101 b. The video signal comprises the encrypted encoded rendition of the base layer 107b and the encoded rendition of the enhancement layer 105b. The receiving module 103b is configured to separate the video signal into: the encrypted encoded rendition of the base layer and the encoded rendition of the enhancement layer. The receiving module 103b is configured to decrypt the encrypted encoded rendition of the base layer. The receiving module 103b is configured to output the encoded rendition of the enhancement layer 105b to the unsecure memory 109b. The receiving module 103b is configured to output the decrypted encoded rendition of the base layer 107b to the secure memory 110b.

[0138] The received encoded rendition of the enhancement layer may be received by the receiving module 103b as an encrypted version of the encoded rendition of the enhancement layer. In such an example, the receiving module 103b is configured to, before outputting the encoded rendition of the enhancement layer, decrypt the encrypted version of the encoded rendition of the enhancement layer to obtain the encoded rendition of the enhancement layer 105b.

[0139] The unsecure memory 109b is configured to receive, from the receiving module 103b (via an unsecured channel), and store an encoded rendition of the enhancement layer 105b of the video signal. The unsecure memory 109b is configured to output the encoded rendition of the enhancement layer to the enhancement decoding module 113b configured to generate the decoded rendition of the enhancement layer by decoding the encoded rendition of the enhancement layer. The decoded rendition of the residual layer has a first resolution. The unsecure memory 109b is configured to receive, from the unsecure decoding module 113b, and store the decoded rendition of the enhancement layer. The unsecure memory 109b is configured to output the decoded rendition of the enhancement layer to the enhancement decoding module 113b configured to generate the positive residual layer at the first resolution. The unsecure memory 109b is configured to receive, from the unsecure decoding module 113b, and store the positive residual layer.

[0140] The unsecure memory 109b may further be configured to output the decoded rendition of the enhancement layer to the enhancement decoding module 113b configured to identify a correction requirement. The correction requirement provides a way for the secure portion of the decoder to implement a correction which flexibly matches the processing of the enhancement layer in the unsecure decoding module 113b. For example, the correction requirement may describe processing to generate the positive residual layer (e.g. an offset parameter or a weight). The unsecure memory 109b is configured to receive, from the unsecure decoding module 113b, and store the correction requirement.

[0141] The generation of the decoded rendition of the enhancement layer, the generation of the positive residual layer and the identification of a correction requirement may be performed in multiple stages, 702b, 704b, 706b, or a single stage, 113b. In the single stage, the unsecured memory 109b outputs the encoded rendition of the enhancement layer 105b and stores the positive residual map and the correction requirement.

[0142] The secure memory 110b is configured to receive, from the receiving module 103b, a decrypted encoded rendition of the base layer 107b of the video signal. The secure memory 110b is configured to output 119b the decrypted encoded rendition of the base layer to the base decoding module 117b. The secure memory 110b is configured to receive, from the base decoding module 117b, the decrypted decoded rendition of the base layer 121 b of the video signal generated by the base decoding module 117b. The secure memory 110b is configured to store the decrypted decoded rendition of the base layer 121b.

[0143] The output module 708b has access to the secure memory 110b and to the unsecure memory 109b. The output module 708b is configured to read, from the secure memory 110b (via a secured channel), a decrypted decoded rendition of a base layer of a video signal. The decrypted decoded rendition of the base layer has a second resolution. In this illustrated embodiment, the second resolution is lower than the first resolution, (however, this is not essential, the second resolution may be the same as the first resolution, in which case, upsampling may not be performed on the decrypted decoded rendition of the base layer). The output module 708b is configured to generate an upsampled decrypted decoded rendition of the base layer of the video signal by upsampling the decrypted decoded rendition of the base layer such that the upsampled decrypted decoded rendition of the base layer has the first resolution.

[0144] The output module 708b is configured to read, from the unsecure memory 109b (e.g. via an unsecured channel), a decoded rendition of a positive residual layer 712b of the video signal, labelled in Figure 7 as the LCEVC positive residual map. The output module 708b is configured to apply the decoded rendition of the positive residual layer 712b to the upsampled decrypted decoded rendition of the base layer to generate an intermediate video signal.

[0145] The output module 708b is further configured to convert the intermediate video signal to an enhanced decoded video signal by applying an excess positivity correction as discussed above. The output module 708b may further be configured to read, from the unsecure memory 109b (e.g. via an unsecured channel), a correction requirement 714b, labelled in Figure 7 as the LCEVC correction requirement. In embodiments which do not use the correction requirement, or when the correction requirement is unavailable, the output module 708b uses static parameters for the correction. On the other hand, when a correction requirement is available, the output module 708b may use dynamic parameters to control how to correct the intermediate video signal.

[0146] The output module 708b is configured to output the output plane 133b, via a secured channel, to a protected display (not illustrated). The output module may comprise a video shifter. The output module may further comprise a subtraction module, which may for example be a hardware scaling and compositing block as typically found within a Video decoder SoC, or a GPU operating in the secure memory. The upsampling, the generation of the intermediate video signal, and the generation of the enhanced decoded video signal may be performed in multiple stages or a single stage, 708b. Different stages may be performed by different hardware blocks such as a blend unit, a hardware scaling and compositing block, a hardware 2D processor, or a GPU.

[0147] Predicted residuals, e.g. using a predicted average based on lower resolution data, as described in WO 2013 / 171173 (which is incorporated by reference) and as may be applied (such as in section 8.7.5 of LCEVC standard) as part of a modified upsampling procedure as described in WO / 2020 / 188242 (incorporated by reference) may be processed by the output module 131 b. WO / 2020 / 188242 is particularly directed to section 8.7.5 of LCEVC, as the predicted averages are applied via what is referred to as "modified upsampling". In general, WO 2013 / 171173 describes the predicted average being computed / reconstructed at a pre-inverse-transformation stage (i.e. in transformed coefficient space) but the modified upsampling in WO 2020 / 188242 moves the application of the predicted average modifier outside of the pre- inverse-transformation stage and applies it during upsampling (in a post-inverse transformation or reconstructed image space), this is possible as the transforms are (e.g. simple) linear operations so the application of them can be moved within the processing pipeline. Therefore, the output module 708b may be configured to: generate the predicted residuals (in line with the methods described in WO 2020 / 188242); and apply the predicted residuals (generated by the modified upsampling) to the upsampled decrypted decoded rendition of the base layer (in addition to applying the decoded rendition of the enhancement layer 115b) to generate the output plane. In general, the output module 708b generates the predicted residuals by determining a difference between: an average of a 2 by 2 block of the upsampled decrypted decoded rendition of the base layer; and a value of a corresponding pixel of the (i.e. not upsampled) decrypted decoded rendition of the base layer. Figure 8 illustrates a block diagram of an enhancement decoder incorporating the steps of the separation and subtraction stages described elsewhere in this disclosure, as well as the broad general steps of an enhancement decoder. As described elsewhere, the residuals may be generated in separated form as illustrated here, rather than separated from a set of residuals created by an enhancement decoder.

[0148] The encoded base stream and one or more enhancement streams are received at the decoder 800.

[0149] The encoded base stream is decoded at base decoder 880 in order to produce a base reconstruction of the input signal received at encoder. This base reconstruction may be used in practice to provide a viewable rendition of the signal at the lower quality level. However, this base reconstruction signal also provides a base for a higher quality rendition of the input signal.

[0150] Figure 8 illustrates both sub layer 1 reconstruction and sub layer 2 reconstruction. In the illustrated enhancement decoder, the reconstruction of sub layer 1 is optional.

[0151] At sub layer 1 , in order to reconstruct the level 1 video signal, the decoded base stream is provided to a processing block. The processing block also receives an encoded level 1 stream and reverses any encoding, quantization and transforming that has been applied by the encoder. The processing block comprises an entropy decoding process 810-1 , an inverse quantization process 820-1 , and an inverse transform process 830-1. Optionally, only one or more of these steps may be performed depending on the operations carried out at corresponding block at the encoder. By performing these corresponding steps, a decoded level 1 stream comprising the first set of residuals is made available at the decoder 800.

[0152] The first set of residuals is processed 840-1 to generate a layer of positive residual data, for example according to the techniques described above with reference to Figure 4 or Figure 5. The first set of positive residuals is combined with the decoded base stream from base decoder 880 (i.e. a summing operation 860-1 is performed on a decoded base stream and the decoded first set of residuals to generate an intermediate sub layer 1 video signal).

[0153] An excess positivity correction 870-1 is applied to the intermediate sub layer 1 video signal to reconstruct a downsampled version of the input video.

[0154] Additionally, and optionally in parallel, the encoded level 2 stream is processed in order to produce a decoded further set of residuals. Similarly to above level 1 processing block, the level 2 processing block comprises an entropy decoding process 810-2, an inverse quantization process 820-2 and an inverse transform process 830-2. These operations will correspond to those performed at block in the encoder, and one or more of these steps may be omitted as necessary.

[0155] The second set of residuals is processed 840-2 to generate a layer of positive residual data, for example according to the techniques described above with reference to Figure 4 or Figure 5.

[0156] The decoded base stream is upsampled at upsampler 850-2 and summed with the positive residuals at the higher resolution at operation 860-2 in order to create an intermediate sub layer 2 video signal.

[0157] An excess positivity correction 870-2 is applied to the intermediate sub layer 2 video signal to reconstruct a level 2 version of the input video.

[0158] As noted above, the enhancement stream may comprise two streams, namely the encoded level 1 stream (a first level of enhancement) and the encoded level 2 stream (a second level of enhancement). The encoded level 1 stream provides a set of correction data which can be combined with a decoded version of the base stream to generate a corrected picture.

[0159] An architecture for implementing the above concepts may comprise two main components. A first component may be a user space application. Its purpose may be to parse the input transport stream (e.g. MPEG2), extract the base video and LCEVC stream (e.g. SEI NALU and dual track multiplexing). The function of the application is to: configure the hardware base video decoders and pass the base video for decoding; decode the LCEVC stream using the DPI to create a positive residual plane; and the base video decode and the residuals are sent to the display driver.

[0160] A second component of the architecture may be a display driver. Its purpose is that modified video device drivers perform upscaling and composition using the a Blender and a set of hardware compositors. The Blender may be used to compose multiple video planes into a single output. The function of the display driver is that: the base video decode is upscaled, then composed using the Blender (through addition with a pre-computed alpha) with the full resolution positive residuals, a correction is applied using colour management, and a randomly generated dither mask placed on an On-Screen Display (OSD) plane; and, the output of the Blender is sent to the Display.

[0161] In an implementation, the base and enhanced video will be held in hardware protected buffers throughout this process (i.e. a secure video path).

[0162] The implementation varies slightly across different SoC variants. Some variants of the SoC have more features allowing extra capabilities such as negative residuals at enhanced resolution, a second upscale, colour management or image sharpening. Fundamentally the architecture remains the same, i.e.: the blender is used for positive residuals; and a correction is applied at the postblend stage (for example using colour management).

[0163] A desirable method for enhancing base video enhancing the base video with LCEVC is: perform a x2 upscale of the base video using specified scaler coefficients (kernel); add Predictive Averages, i.e.the difference between a pixel value in the base video and the average of 4 pixels in the corresponding 2x2 upscaled block; apply a plane of signed offsets to the result; and, dither the output by adding a plane of signed random values. Preferably these steps are performed in hardware. For dithering, one implementation is to use blending to add a plane of positive random values — scaled so that they don’t exceed the specified dither strength.

[0164] Dithering may be applied at a lower resolution, which is then combined with the video signal to produce the final output. This approach leads to surprisingly good visual quality. In other words, the dithering is applied at a separate plane and at lower resolution than the output resolution. Moreover, the dithering may be applied to each of the YUV planes, whereas typically dithering may be applied to only one.

[0165] A dither plane, such as a 960x540 dither plane, may be scaled and applied at a post-blend stage to a scaled version of the LCEVC enhancement output itself scaled for display resolution. The video may then be output for display.

[0166] Put another way, the LCEVC enhancement output, i.e. the output of the preblend and the enhanced video data, may be scaled to a display resolution such as a 4:4:4 display resolution in which Luma and Chroma have the same spatial resolution (other display resolutions such as 4:2:2 or 4:2:0 are of course contemplated). A dither plane may also be scaled to a display resolution, in this example 4:2:2. The dither plane and the scaled enhanced video signal are then combined at a post-blend stage to generate the video for display.

[0167] As noted above, applying dithering in this way, i.e. the enhanced video is output and then dithering is applied at a post-blend stage yields surprisingly good display quality. Moreover, arranging the video display path in this way allows for display to be in any resolution.

[0168] The dither plane is input, i.e. applied, at a lower resolution before scaling.

[0169] In another example, the dither plane may combined with the base decoded video signal at a Pre-Blend stage before the positive residuals are later combined. In another example, the dither plane may be combined with the base decoded video simultaneously with the positive residuals. This may be performed in a blend unit, i.e. at the Blend stage.

[0170] Positive dithering may be applied at lower resolution, upscaled, then added to the final output. In other words, the dithering may be applied at a separate plane and at lower resolution than the output resolution. Moreover, the dithering may be applied to each of the YUV planes, whereas often dithering will be applied to just one.

[0171] Generally, any of the functionality described in this text or illustrated in the figures can be implemented using software, firmware (e.g., fixed logic circuitry), programmable or nonprogrammable hardware, or a combination of these implementations. The terms “component” or “function” as used herein generally represents software, firmware, hardware or a combination of these. For instance, in the case of a software implementation, the terms “component” or “function” may refer to program code that performs specified tasks when executed on a processing device or devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

Claims

CLAIMS1 . A method for use in a video pipeline, the method comprising: obtaining a base decoded video signal from a base decoding layer; obtaining one or more layers of positive residual data from an enhancement decoding layer, the one or more layers of positive residual data being generated based on a comparison of data derived from the base decoded video signal and data derived from an original input video signal, wherein the base decoded video signal comprises values greater than the original input video signal and values lower than the original input video signal and the positive residual data comprises only values greater than or equal to zero; and calculating a weighted sum of the base decoded video signal and the one or more layers of positive residual data; and converting the weighted sum to an enhanced decoded video signal by applying an excess positivity correction.

2. A method according to claim 1 , comprising calculating the weighted sum using a blend unit.

3. A method according to any preceding claim, wherein obtaining the one or more layers of positive residual data comprises: obtaining one or more layers of residual data from the enhancement decoding layer, the one or more layers of residual data being generated based on a comparison of data derived from a decoded video signal and data derived from an original input video signal; and processing the one or more layers of residual data to generate the one or more layers of positive residual data.

4. A method according to claim 3, wherein processing the one or more layers of residual data to generate the one or more layers of positive residual data comprises clipping the one or more layers of positive residual data according to a maximum residual value and / or a minimum residual value.

5. A method according to any preceding claim, wherein a weight associated with each of the base decoded video signal and the one or more layers of residual data is greater than or equal to zero.

6. A method according to any preceding claim, wherein the excess positivity correction comprises a predetermined gain and / or a predetermined offset.

7. A method according to any preceding claim, wherein the excess positivity correction comprises using a lookup table to convert from a weighted sum value to an enhanced decoded video signal value.

8. A method according to any preceding claim, wherein the excess positivity correction comprises applying a colour correction algorithm.

9. A method according to claim 8, wherein the excess positive correction comprises applying a combined algorithm which performs both excess positivity correction and colour correction.

10. A method according to any of claims 6 to 9, wherein the excess positivity correction further comprises clipping the enhanced decoded video signal according to a maximum decoded value and / or a minimum decoded value.

11. A module for use in a video pipeline, configured to: obtain a base decoded video signal from a base decoding layer; obtain one or more layers of positive residual data from an enhancement decoding layer, the one or more layers of positive residual data being generated based on a comparison of data derived from the base decoded video signal and data derived from an original input video signal, wherein the base decoded video signal comprises values greater than the original input video signal and values lower than the original input video signal and the positive residual data comprises only values greater than or equal to zero; and calculate a weighted sum of the base decoded video signal and the one or more layers of positive residual data; andconvert the weighted sum to an enhanced decoded video signal by applying an excess positivity correction.

12. A module according to claim 11 , comprising a blend unit, wherein the weighted sum is calculated by the blend unit.

13. A module according to claim 11 or claim 12, wherein obtaining the one or more layers of positive residual data comprises: obtaining one or more layers of residual data from the enhancement decoding layer, the one or more layers of residual data being generated based on a comparison of data derived from a decoded video signal and data derived from an original input video signal; and processing the one or more layers of residual data to generate the one or more layers of positive residual data.

14. A module according to claim 13, wherein processing the one or more layers of residual data to generate the one or more layers of positive residual data comprises clipping the one or more layers of positive residual data according to a maximum residual value and / or a minimum residual value.

15. A module according to any of claims 11 to 14, wherein a weight associated with each of the base decoded video signal and the one or more layers of residual data is greater than or equal to zero.

16. A module according to any of claims 11 to 15, wherein the excess positivity correction comprises a predetermined gain and / or a predetermined offset.

17. A module according to any of claims 11 to 16, wherein the excess positivity correction comprises using a lookup table to convert from a weighted sum value to an enhanced decoded video signal value.

18. A module according to any of claims 11 to 17, wherein the excess positivity correction comprises applying a colour correction algorithm.

19. A module according to any of claims 16 to 18, wherein the excess positivity correction further comprises clipping the enhanced decoded video signal according to a maximum decoded value and / or a minimum decoded value.

20. A bitstream comprising a weighted sum of a base decoded video signal and one or more layers of positive residual data, the one or more layers of positive residual data being generated based on a comparison of data derived from the base decoded video signal and data derived from an original input video signal, wherein the base decoded video signal comprises values greater than the original input video signal and values lower than the original input video signal and the positive residual data comprises only values greater than or equal to zero.

21. A bitstream comprising positive residual data, the positive residual data being generated based on a comparison of data derived from a base decoded video signal and data derived from an original input video signal, wherein the base decoded video signal comprises values greater than the original input video signal and values lower than the original input video signal and the positive residual data comprises only values greater than or equal to zero.

22. A computer-readable storage device storing a bitstream according to claim 20 or claim 21 .

Images