Video encoding pre-analysis

Abstract

There is disclosed a method of selecting an encoder configuration for encoding video data. The method comprising receiving video data to be encoded; receiving a quality factor to be used for encoding the video data; in dependence on one or more attributes of the video data and the quality factor, determining an encoder configuration. The encoder configuration can then be used to encode the video to an acceptably good standard. The encoder configuration is usefully content dependent.

Description

[0001] Video Encoding Pre-Analysis

[0002] Technical Field

[0003] The following disclosure relates to video encoding. In particular, the disclosure relates to efficient encoding which improves compressibility while maintaining perceived quality of reconstructed video, by performing pre-analysis before encoding. The following disclosure is particularly applicable to pre-analysis before LCEVC (Low Complexity Enhancement Video Coding) encoding, although the described techniques can be used as pre-analysis before using other standardized encoding techniques, in particular but not exclusively scalable or hierarchical encoding techniques.

[0004] Background

[0005] When encoding and decoding video, it is always necessary to strike a balance between improving compression of the encoded video and improving the fidelity of the decoded video to the original video.

[0006] In a simple case, encoder parameters such as a desired bit rate are chosen when configuring a codec, and the chosen encoder parameters are applied for encoding an entire video.

[0007] However, it is desirable to configure an encoder more intelligently, to prioritize compression in parts of a video which contain less information and to prioritize fidelity in parts of a video which contain more information, or to ensure that the encoded video meets a set of requirements (e.g. a maximum bit rate) regardless of variations in the information density or other properties of the raw video.

[0008] The encoding techniques in the following specification are particularly suited to be used with existing Low Complexity Enhancement Video Coding (LCEVC) techniques.

[0009] A standard specification for LCEVC is provided in the Text of ISO / IEC 23094-2 Ed 1 Low Complexity Enhancement Video Coding published in November 2021, and many possible implementation details of LCEVC are described in patent publications WO 2020 / 188273 and WO 2020 / 188229. Each of these earlier documents is incorporated here by reference.

[0010] Broadly speaking, LCEVC enhances the reproduction fidelity of a decoded video after encoding and decoding using an existing codec. This is achieved by combining a base layer with an enhancement layer, where the base layer contains the video encoded using the existing codec, and the enhancement layer indicates a residual difference between the original video and an expected decoded video produced by decoding the base layer using the existing codec. The enhancement layer can be combined with the decoded base layer to more accurately reproduce the original video.

[0011] In the context of LCEVC, the enhancement layer can also be encoded, for example by down-sampling the enhancement layer to a lower resolution or quantizing values of the enhancement layer or transforming the enhancement layer. The choice of downsampler and upsampler in LCEVC is configurable, as is the option of adding pre-processing filtering. As a result, even within the standardized techniques of LCEVC there are a number of encoder parameters which can be adapted to provide an improved compression-fidelity balance.

[0012] Summary

[0013] According to a first aspect of the invention, there is provided a method of selecting an encoder configuration for encoding video data. The method comprising: receiving video data to be encoded; receiving a quality factor to be used for encoding the video data; in dependence on one or more attributes of the video data and the quality factor, determining an encoder configuration.

[0014] Preferably, wherein the method comprises encoding the video data with an encoder configured according to the encoder configuration.

[0015] Preferably, wherein the method comprises generating a concatenated bitstream where different encoding configurations are signalled for each concatenated part in dependence on the corresponding encoding configuration used when encoding that concatenated part.

[0016] Preferably, wherein the quality factor is one of the following: constant rate factor, CRF, a modified CRF for hierarchical coding, and a video resolution.

[0017] Preferably, wherein the determining comprises predicting and selecting an encoder configuration in dependence on the one or more attributes and the quality factor.

[0018] Preferably, wherein the determining comprises using a heuristic-based set of rules, discovered via data analysis on collected datasets of source video, and a machine learning process to determine which encoder configuration to predict and select in dependence on the one or more attributes and the quality factor.

[0019] Preferably, wherein the determining comprises using a cascading classifier model cascading to settle at a discrete set of configurations.

[0020] Preferably, wherein the method comprises analysing the video data to determine the one or more attributes.

[0021] Preferably, wherein the attributes are in dependence on pixel data of the video data. Preferably, wherein the attributes are in dependence on statistical data derived from the pixel data.

[0022] Preferably, wherein the statistical data are derived from (a) features of the video data, and wherein each feature of the features is a 4x4 block surface, and wherein each feature has at least one of the following surfaces: contrast, texture, type analysis, mean, and histogram.

[0023] Preferably, wherein the attributes comprise a single number for each feature surface for each frame of the video data.

[0024] Preferably, wherein the statistical data are derived from (b) the video data and depend on the encoder and rate control method used for encoding the video data, and the method comprises simulating encoding and / or rate control on the source video to obtain the statistical data.

[0025] Preferably, wherein residual information generated by the simulating is summarised using one or more of the following: minimum / maximum values, and standard deviation.

[0026] Preferably, wherein the video data is a component of a larger video.

[0027] Preferably, wherein the component is less than 10 frames, is less than 9 frames, less than 8 frames, is less than 7 frames, is less than 6 frames, is less than 5 frames, is less than 4 frames, is less than 3 frames, is 2 frames or is 1 frame; is more than one frame but less than the aforementioned number of frames.

[0028] Preferably, wherein the component is a scene or is part of a scene.

[0029] Preferably, wherein the method comprises using a scene detector to determine the scene.

[0030] Preferably, wherein the component is one of: an I-frame in a group of pictures; and a plurality of I-frames from adjacent groups of pictures.

[0031] Preferably, wherein the encoding parameters comprise one or more of the following: scaling mode; transform on / off; transform type; upsampler; downsampler; upsampler / downsampler pair; and M-filter.

[0032] Preferably, wherein the encoding is a hierarchical encoding including a base layer and at least one enhancement layer.

[0033] Preferably, wherein there are two enhancement layers.

[0034] Preferably, wherein an enhancement layer is at the same resolution as the base layer resolution. Preferably, wherein an enhancement layer is at a higher resolution than the base layer resolution.

[0035] According to a second aspect of the invention, there is provided a system or an apparatus for video encoding configured to perform the method of any previous method statement.

[0036] According to a third aspect of the invention, there is provided a computer-readable medium comprising instructions which when executed cause a processor to perform the method of any previous method statement.

[0037] According to a fourth aspect of the invention, there is provided a computer program comprising instructions which when executed cause a processor to perform the method of any previous method statement.

[0038] According to a fifth aspect of the invention, there is provided a signal comprising the computer program of the previous computer program statement.

[0039] Brief Description of the Drawings

[0040] The invention shall now be described, by way of example only, with reference to the accompanying drawings in which:

[0041] Figure 1 is a high-level schematic of an encoding process;

[0042] Figure 2 is a high-level schematic of a decoding process;

[0043] Figure 3 is a high-level schematic of an encoding process and specific encoding steps;

[0044] Figure 4 is a high-level schematic of a decoding process and specific decoding steps;

[0045] Figure 5 is a high-level schematic of an encoding process and residual processing;

[0046] Figure 6 is a high-level schematic of a further decoding process;

[0047] Figure 7 is a high-level schematic of an encoding process and residual mode control;

[0048] Figure 8 is classification and residuals weighting;

[0049] Figure 9 illustrates a decoding process which uses temporal prediction;

[0050] Figure 10 is a block diagram of a pre-encoding analysis block according to an aspect of the invention and an encoder; Figure 11 shows an example cascading classifier model used in the specific embodiment described with reference to Figure 10;

[0051] Figure 12 illustrates a flow chart of a method according to one aspect of the invention.

[0052] Detailed Description

[0053] Before discussing the encoder parameter pre-analysis techniques of the present invention, the following provides a discussion of a coding technology with which the determined encoder parameter may be used. This also explains much of the terminology used in the encoder parameter pre-analysis techniques. Such a coding technology has been previously described in, for example, WO 2020 / 188229.

[0054] The pre-analysis techniques discussed herein may be used with a flexible, adaptable, highly efficient and computationally inexpensive coding technology and format which combines a video coding format, a base codec, (e.g. AVC, HEVC, or any other present or future codec) with an enhancement level of coded data, encoded using a different technique. The technology uses a down-sampled source signal encoded using a base codec to form a base stream. An enhancement stream is formed using an encoded set of residuals which correct or enhance the base stream for example by increasing resolution or by increasing frame rate. There may be multiple levels of enhancement data in a hierarchical structure. In certain arrangements, the base stream may be decoded by a hardware decoder while the enhancement stream may be suitable for a software implementation.

[0055] It is important that any optimisation used in the coding technology is tailored to the specific requirements or constraints of the enhancement stream and is of low complexity. Such requirements or constraints include: the potential reduction in computational capability resulting from the need for software decoding of the enhancement stream; the need for combination of a decoded set of residuals with a decoded frame; the likely structure of the residual data, i.e. the relatively high proportion of zero values with highly variable data values over a large range; the nuances of a quantized block of coefficients; and, the structure of the enhancement stream being a set of discrete residual frames separated into various components. Note that the constraints placed on the enhancement stream mean that a simple and fast entropy coding operation is essential to enable the enhancement stream to effectively correct or enhance individual frames of the base decoded video. Note that in some scenarios the base stream is also being decoded substantially simultaneously before combination, putting a strain on resources. In one case, the methods described herein may be applied to so-called planes of data that reflect different colour components of a video signal. For example, the methods described herein may be applied to different planes of YUV or RGB data reflecting different colour channels. Different colour channels may be processed in parallel. Hence, references to sets of residuals as described herein may comprise multiple sets of residuals, where each colour component has a different set of residuals that form part of a combined enhancement stream. The components of each stream may be collated in any logical order, for example, each plane at the same level may be grouped and sent together or, alternatively, the sets of residuals for different levels in each plane may be sent together.

[0056] This present document preferably fulfils the requirements of the following ISO / IEC documents: "Call for Proposals for Low Complexity Video Coding Enhancements" ISO / IEC JTC1 / SC29 / WG11 N17944, Macao, CN, Oct. 2018 and "Requirements for Low Complexity Video Coding Enhancements" ISO / IEC JTC1 / SC29 / WG11 N 18098, Macao, CN, Oct. 2018 (which are incorporated by reference herein). Moreover, approaches described herein may be incorporated into products as supplied by V-Nova International Ltd.

[0057] The general structure of an encoding scheme in which the presently described techniques can be applied, uses a down-sampled source signal encoded with a base codec, adds a first level of correction data to the decoded output of the base codec to generate a corrected picture, and then adds a further level of enhancement data to an up-sampled version of the corrected picture. Thus, the streams are considered to be a base stream and an enhancement stream. This structure creates a plurality of degrees of freedom that allow great flexibility and adaptability to many situations, thus making the coding format suitable for many use cases including Over-The-Top (OTT) transmission, live streaming, live Ultra High Definition (UHD) broadcast, and so on. Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders and, where considered suitable, also usable as a lower resolution output. In certain cases, a base codec may be used to create a base stream. The base codec may comprise an independent codec that is controlled in a modular or "black box" manner. The methods described herein may be implemented by way of computer program code that is executed by a processor and makes function calls upon hardware and / or software implemented base codecs.

[0058] In general, the term "residuals" as used herein refers to a difference between a value of a reference array or reference frame and an actual array or frame of data. The array may be a one or two-dimensional array that represents a coding unit. For example, a coding unit may be a 2x2 or 4x4 set of residual values that correspond to similar sized areas of an input video frame. It should be noted that this generalised example is agnostic as to the encoding operations performed and the nature of the input signal. Reference to "residual data" as used herein refers to data derived from a set of residuals, e.g. a set of residuals themselves or an output of a set of data processing operations that are performed on the set of residuals. Throughout the present description, generally a set of residuals includes a plurality of residuals or residual elements, each residual or residual element corresponding to a signal element, that is, an element of the signal or original data. The signal may be an image or video. In these examples, the set of residuals corresponds to an image or frame of the video, with each residual being associated with a pixel of the signal, the pixel being the signal element. Examples disclosed herein describe how these residuals may be modified (i.e. processed) to impact the encoding pipeline or the eventually decoded image while reducing overall data size. Residuals or sets may be processed on a per residual element (or residual) basis, or processed on a group basis such as per tile or per coding unit where a tile or coding unit is a neighbouring subset of the set of residuals. In one case, a tile may comprise a group of smaller coding units. A tile may comprise a 16x16 set of picture elements or residuals (e.g. an 8 by 8 set of 2x2 coding units or a 4 by 4 set of 4x4 coding units). Note that the processing may be performed on each frame of a video or on only a set number of frames in a sequence.

[0059] In general, each or both enhancement streams may be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Layer Units (NALUs). The NALUs are meant to encapsulate the enhancement bitstream in order to apply the enhancement to the correct base reconstructed frame. The NALU may for example contain a reference index to the NALU containing the base decoder reconstructed frame bitstream to which the enhancement has to be applied. In this way, the enhancement can be synchronised to the base stream and the frames of each bitstream combined to produce the decoded output video (i.e. the residuals of each frame of enhancement level are combined with the frame of the base decoded stream). A group of pictures may represent multiple NALUs.

[0060] Returning to the initial process described above, where a base stream is provided along with two levels (or sub-levels) of enhancement within an enhancement stream, an example of a generalised encoding process is depicted in the block diagram of Figure 1. An input full resolution video 100 is processed to generate various encoded streams 101, 102, 103. A first encoded stream (encoded base stream) is produced by feeding a base codec (e.g., AVC, HEVC, or any other codec) with a down-sampled version of the input video. The encoded base stream may be referred to as the base layer or base level. A second encoded stream (encoded level 1 stream) is produced by processing the residuals obtained by taking the difference between a reconstructed base codec video and the down- sampled version of the input video. A third encoded stream (encoded level 2 stream) is produced by processing the residuals obtained by taking the difference between an up- sampled version of a corrected version of the reconstructed base coded video and the input video. In certain cases, the components of Figure 1 may provide a general low complexity encoder. In certain cases, the enhancement streams may be generated by encoding processes that form part of the low complexity encoder and the low complexity encoder may be configured to control an independent base encoder and decoder (e.g. as packaged as a base codec). In other cases, the base encoder and decoder may be supplied as part of the low complexity encoder. In one case, the low complexity encoder of Figure 1 may be seen as a form of wrapper for the base codec, where the functionality of the base codec may be hidden from an entity implementing the low complexity encoder.

[0061] A down-sampling operation illustrated by downsampling component 105 may be applied to the input video to produce a down-sampled video to be encoded by a base encoder 113 of a base codec. The down-sampling can be done either in both vertical and horizontal directions, or alternatively only in the horizontal direction. The base encoder 113 and a base decoder 114 may be implemented by a base codec (e.g. as different functions of a common codec). The base codec, and / or one or more of the base encoder 113 and the base decoder 114 may comprise suitably configured electronic circuitry (e.g. a hardware encoder / decoder) and / or computer program code that is executed by a processor.

[0062] Each enhancement stream encoding process may not necessarily include an up-sampling step. In Figure 1 for example, the first enhancement stream is conceptually a correction stream while the second enhancement stream is up-sampled to provide a level of enhancement.

[0063] Looking at the process of generating the enhancement streams in more detail, to generate the encoded Level 1 stream, the encoded base stream is decoded by the base decoder 114 (i.e. a decoding operation is applied to the encoded base stream to generate a decoded base stream). Decoding may be performed by a decoding function or mode of a base codec. The difference between the decoded base stream and the down-sampled input video is then created at a level 1 comparator 110 (i.e. a subtraction operation is applied to the down-sampled input video and the decoded base stream to generate a first set of residuals). The output of the comparator 110 may be referred to as a first set of residuals, e.g. a surface or frame of residual data, where a residual value is determined for each picture element at the resolution of the base encoder 113, the base decoder 114 and the output of the downsampling block 105. The difference is then encoded by a first encoder 115 (i.e. a level 1 encoder) to generate the encoded Level 1 stream 102 (i.e. an encoding operation is applied to the first set of residuals to generate a first enhancement stream).

[0064] As noted above, the enhancement stream may comprise a first level of enhancement 102 and a second level of enhancement 103. The first level of enhancement 102 may be considered to be a corrected stream, e.g. a stream that provides a level of correction to the base encoded / decoded video signal at a lower resolution than the input video 100. The second level of enhancement 103 may be considered to be a further level of enhancement that converts the corrected stream to the original input video 100, e.g. that applies a level of enhancement or correction to a signal that is reconstructed from the corrected stream.

[0065] In the example of Figure 1, the second level of enhancement 103 is created by encoding a further set of residuals. The further set of residuals are generated by a level 2 comparator 119. The level 2 comparator 119 determines a difference between an up-sampled version of a decoded level 1 stream, e.g. the output of an upsampling component 117, and the input video 100. The input to the upsampling component 117 is generated by applying a first decoder (i.e. a level 1 decoder) to the output of the first encoder 115. This generates a decoded set of level 1 residuals. These are then combined with the output of the base decoder 114 at summation component 120. This effectively applies the level 1 residuals to the output of the base decoder 114. It allows for losses in the level 1 encoding and decoding process to be corrected by the level 2 residuals. The output of summation component 120 may be seen as a simulated signal that represents an output of applying level 1 processing to the encoded base stream 101 and the encoded level 1 stream 102 at a decoder.

[0066] As noted, an up-sampled stream is compared to the input video which creates a further set of residuals (i.e. a difference operation is applied to the up-sampled re-created stream to generate a further set of residuals). The further set of residuals are then encoded by a second encoder 121 (i.e. a level 2 encoder) as the encoded Level 2 enhancement stream (i.e. an encoding operation is then applied to the further set of residuals to generate an encoded further enhancement stream).

[0067] Thus, as illustrated in Figure 1 and described above, the output of the encoding process is a base stream 101 and one or more enhancement streams 102, 103 which preferably comprise a first level of enhancement and a further level of enhancement. The three streams 101, 102 and 103 may be combined, with or without additional information such as control headers, to generate a combined stream for the video encoding framework that represents the input video 100. It should be noted that the components shown in Figure 1 may operate on blocks or coding units of data, e.g. corresponding to 2x2 or 4x4 portions of a frame at a particular level of resolution. The components operate without any interblock dependencies, hence they may be applied in parallel to multiple blocks or coding units within a frame. This differs from comparative video encoding schemes wherein there are dependencies between blocks (e.g. either spatial dependencies or temporal dependencies). The dependencies of comparative video encoding schemes limit the level of parallelism and require a much higher complexity.

[0068] Figure 1 illustrates a residual mode selection block 140. If residual mode (RM) has been selected, residuals are processed (i.e. modified and / or ranked and selected) in order to determine which residuals should be transformed and encoded, i.e. which residuals are to be processed by the first and / or second encoders 115 and 121. Preferably this processing is performed prior to entropy encoding. Residual Mode selection 140 is an optional step that may configure or activate processing or modification of residuals i.e. residual processing is performed according to a selected mode. For example, the "residual mode (RM)" may correspond to a residual pre-processing mode, wherein residuals for enhancement layers are pre-processed prior to encoding. It should be noted that the residual pre-processing may be used independently from the encoder parameter preprocessing discussed below. This mode may be turned on and off depending on requirements. For example, the residual mode may be configured via one or more control headers or fields, and the residual mode is an example of the encoder parameter which can be determined using the encoder parameter pre-processing. In alternative embodiments, the residuals may always be modified (i.e. pre-processed) and so selection of a mode is not required. In this case, residual pre-processing may be hard-coded. Examples of residuals processing will be described in detail below. The residual mode, if selected, may act to filter residuals within one or more of the level 1 and level 2 encoding operations, preferably at a stage prior to the encoding sub-components.

[0069] A corresponding generalised decoding process is depicted in the block diagram of Figure 2. Figure 2 may be said to show a low complexity decoder that corresponds to the low complexity encoder of Figure 1. The low complexity decoder receives the three streams 101, 102, 103 generated by the low complexity encoder together with headers 204 containing further decoding information. The headers 204 may include the encoder parameter determined using the encoder parameter pre-analysis described below. The encoded base stream 101 is decoded by a base decoder 210 corresponding to the base codec used in the low complexity encoder. The encoded level 1 stream 102 is received by a first decoder 211 (i.e. a level 1 decoder), which decodes a first set of residuals as encoded by the first encoder 115 of Figure 1. At a first summation component 212, the output of the base decoder 210 is combined with the decoded residuals obtained from the first decoder 211. The combined video, which may be said to be a level 1 reconstructed video signal, is up-sampled by upsampling component 213. The encoded level 2 stream 103 is received by a second decoder 214 (i.e. a level 2 decoder). The second decoder 214 decodes a second set of residuals as encoded by the second encoder 121 of Figure 1. Although the headers 204 are shown in Figure 2 as being used by the second decoder 214, they may also be used by the first decoder 211 as well as the base decoder 210. The output of the second decoder 214 is a second set of decoded residuals. These may be at a higher resolution to the first set of residuals and the input to the upsampling component 213. At a second summation component 215, the second set of residuals from the second decoder 214 are combined with the output of the upsampling component 213, i.e. an upsampled reconstructed level 1 signal, to reconstruct decoded video 250.

[0070] As per the low complexity encoder, the low complexity decoder of Figure 2 may operate in parallel on different blocks or coding units of a given frame of the video signal. Additionally, decoding by two or more of the base decoder 210, the first decoder 211 and the second decoder 214 may be performed in parallel. This is possible as there are no inter-block dependencies.

[0071] In the decoding process, the decoder may parse the headers 204 (which may contain global configuration information, picture or frame configuration information, and data block configuration information) and configure the low complexity decoder based on those headers. In order to re-create the input video, the low complexity decoder may decode each of the base stream, the first enhancement stream and the further or second enhancement stream. The frames of the stream may be synchronised and then combined to derive the decoded video 250. The decoded video 250 may be a lossy or lossless reconstruction of the original input video 100 depending on the configuration of the low complexity encoder and decoder. In many cases, the decoded video 250 may be a lossy reconstruction of the original input video 100 where the losses have a reduced or minimal effect on the perception of the decoded video 250.

[0072] In each of Figures 1 and 2, the level 2 and level 1 encoding operations may include the steps of transformation, quantization and entropy encoding (e.g. in that order). Similarly, at the decoding stage, the residuals may be passed through an entropy decoder, a dequantizer and an inverse transform module (e.g. in that order). Any suitable encoding and corresponding decoding operation may be used. Preferably however, the level 2 and level 1 encoding steps may be performed in software (e.g. as executed by one or more central or graphical processing units in an encoding device). The transform as described herein may use a directional decomposition transform such as a Hadamard-based transform. Both may comprise a small kernel or matrix that is applied to flattened coding units of residuals (i.e. 2x2 or 4x4 blocks of residuals). More details on the transform can be found for example in patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. The encoder may select between different transforms to be used, for example between a size of kernel to be applied.

[0073] The transform may transform the residual information to four surfaces. For example, the transform may produce the following components: average, vertical, horizontal and diagonal.

[0074] In summary, the methods and apparatuses herein are based on an overall approach which is built over an existing encoding and / or decoding algorithm (such as MPEG standards such as AVC / H.264, HEVC / H.265, etc. as well as non-standard algorithm such as VP9, AVI, and others) which works as a baseline for an enhancement layer which works accordingly to a different encoding and / or decoding approach. The idea behind the overall approach of the examples is to hierarchically encode / decode the video frame as opposed to the use block-based approaches as used in the MPEG family of algorithms. Hierarchically encoding a frame includes generating residuals for the full frame, and then a decimated frame and so on.

[0075] The video compression residual data for the full-sized video frame may be referred to as LoQ-2 (e.g. 1920 x 1080 for an HD video frame), while that of the decimated frame may be referred to as LoQ-x, where x denotes a number corresponding to a hierarchical decimation. In the described examples of Figures 1 and 2, the variable x may have values of 1 and 2 represent the first and second enhancement streams. Hence there are 2 hierarchical levels for which compression residuals will be generated. Other naming schemes for the levels may also be applied without any change in functionality (e.g. the level 1 and level 2 enhancement streams described herein may alternatively be referred to as level 1 and level 2 streams - representing a count down from the highest resolution).

[0076] A more detailed encoding process is depicted in the block diagram of Figure 3. The encoding process is split into two halves as shown by the dashed line. Below the dashed line is the base level of an encoder 300, which may usefully be implemented in hardware or software. Above the dashed line is the enhancement level, which may usefully be implemented in software. The encoder 300 may comprise only the enhancement level processes, or a combination of the base level processes and enhancement level processes as needed. The encoder 300 may usefully be implemented in software, especially at the enhancement level. This arrangement allows, for example, a legacy hardware encoder that provides the base level to be upgraded using a firmware (e.g. software) update, where the firmware is configured to provide the enhancement level. In newer devices, both the base level and the enhancement level may be provided in hardware and / or a combination of hardware and software.

[0077] The encoder topology at a general level is as follows. The encoder 300 comprises an input I for receiving an input signal 30. The input signal 30 may comprise an input video signal, where the encoder is applied on a frame-by-frame basis. The input I is connected to a down-sampler 305D and processing block 300-2. The down-sampler 305D may correspond to the downsampling component 105 of Figure 1 and the processing block 300-2 may correspond to the second encoder 121 of Figure 1, The down-sampler 305D outputs to a base codec 320 at the base level of the encoder 300. The base codec 320 may implement the base encoder 113 and the base decoder 114 of Figure 1. The down-sampler 305D also outputs to processing block 300-1. The processing block 300-1 may correspond to the first encoder 115 of Figure 1. Processing block 300-1 passes an output to an up-sampler 305U, which in turn outputs to the processing block 300-2. The upsampler 305U may correspond to the upsampling component 117 of Figure 1. Each of the processing blocks 300-2 and 300-1 comprise one or more of the following modules: a transform block 310, a quantization block 320, an entropy encoding block 330 and a residual processing block 350. The residual block 350 may occur prior to the transform block 310 and / or control residual processing in the processing blocks 300. The order of processing may be as set out in the Figures.

[0078] The input signal 30, such as in this example a full (or highest) resolution video, is processed by the encoder 300 to generate various encoded streams. A base encoded stream is produced by feeding the base codec 320 (e.g., AVC, HEVC, or any other codec) at the base level with a down-sampled version of the input video 30, using the downsampler 305D. The base encoded stream may comprise the output of a base encoder of the base codec 320. A first encoded stream (an encoded level 1 stream) is created by reconstructing the encoded base stream to create a base reconstruction, and then taking the difference between the base reconstruction and the down-sampled version of the input video 30. Reconstructing the encoded base stream may comprise receiving a decoded base stream from the base codec (i.e. the input to processing block 300-1 comprises a base decoded stream as shown in Figure 1). The difference signal is then processed at block 300-1 to create the encoded level 1 stream. Block 300-1 comprises a transform block 310- 1, a quantization block 320-1 and an entropy encoding block 330-1. A second encoded stream (an encoded level 2 stream) is created by up-sampling a corrected version of the base reconstruction, using the up-sampler 305U, and taking the difference between the corrected version of the base reconstruction and the input signal 30. This difference signal is then processed at block 300-2 to create the encoded level 2 stream. Block 300-2 comprises a transform block 310-2, a quantization block 320-2, an entropy encoding block 330-2 and a residual processing block 350-2. As per processing block 300-1, the blocks may be performed in the order shown in the Figures (e.g. residual processing followed by transformation followed by quantization followed by entropy encoding).

[0079] Any known quantization scheme may be useful to create the residual signals into quanta, so that certain variables can assume only certain discrete magnitudes. In one case quantizing comprises actioning a division by a pre-determined step-width. This may be applied at both levels (1 and 2). For example, quantizing at block 320 may comprise dividing transformed residual values by a step-width. The step-width may be predetermined, e.g. selected based on a desired level of quantization. In one case, division by a step-width may be converted to a multiplication by an inverse step-width, which may be more efficiently implemented in hardware. In this case, de-quantizing, such as at block 320, may comprise multiplying by the step-width. Entropy encoding as described herein may comprise run length encoding (RLE), then processing the encoded output is processed using a Huffman encoder. In certain cases, only one of these schemes may be used when entropy encoding is desirable.

[0080] The encoded base stream may be referred to as the base level stream.

[0081] Figure 3 illustrates the residual processing blocks 350-2, 350-1 which are located prior to transformation block 310. Although residual processing is shown prior to transformation, optionally, the processing step may be arranged elsewhere, for example, later in the encoding process; however, when being located before the transformation step, residual processing may have the biggest impact throughout the encoding pipeline as efficiencies are propagated through the pipeline. For example, if residual values are filtered at an early stage (e.g. by setting to 0), then this reduces an amount of computation that needs to be performed at subsequent stages within the processing blocks 300. The residual processing block 350 may be activated or configured by residual mode selection block 140 (not shown in Figure 3, shown in Figure 1). For example, if a residual mode is selected (e.g. turned on), then the residual processing block 350 may be activated. The residual mode may be selected independently for the first and second enhancement streams (e.g. residual processing blocks 350-2 and 350-1 may be activated and applied separately where one may be off while another is on).

[0082] The residual processing block is configured to modify a set of residuals. Certain specific functionality of the residual processing block 310 is described in detail below however, conceptually, the residual processing block 310 functions to modify the residuals. This may be seen as a form of filtering or pre-processing. In certain examples, the residuals may be ranked or given a priority as part of the filtering or pre-processing, whereby those with a higher rank or priority are passed for further processing while those with a lower rank or priority are not passed for further processing (e.g. are set to 0 or a corresponding low value). In effect, the residual processing block is configured to 'kill' one or more residuals prior to transformation such that transformation operates on a subset of the residuals.

[0083] The residual processing block 310 may be the same in the L2 and LI pathways or may be configured differently (or not included in a particular pathway) so as to reflect the different nature of those streams.

[0084] Certain examples may implement different residual processing modes. A residual mode selection block 140 may indicate whether or not residuals are to be processed and also, in certain embodiments, the type of processing performed. In general, an encoder (such as the low complexity encoder of Figure 1 or the encoder 300 of Figure 3) may comprise a residual mode control component 140 that selects and implements a residual mode and residual mode implementation components that implements processing for a selected residual mode in relation to the one or more enhancement streams. In other cases, only residual processing blocks 350 may be provided within each level of enhancement encoding without higher control functionality (e.g. within higher level control component such as control component 140). In this latter case, the functionality of the residual mode control component 140 may be seen to be incorporated into the first and / or second encoders 115 and 121 of Figure 1.

[0085] Examples of residual modes that may be implemented include, but are not limited to a mode where no residual processing is performed, a binary mode whereby certain residuals are multiplied by 0 or 1, a weighting mode whereby residuals are multiplied by a weighting factor, a control mode whereby certain blocks or coding units are not to be processed (e.g. equivalent to setting all residual values in a 2x2 or 4x4 coding unit to 0), a ranking or priority mode whereby residuals are ranked or given a priority within a list and selected for further processing based on the rank or priority, a scoring mode whereby residuals are given a score that is used to configure residual encoding and a categorization mode whereby residuals and / or picture elements are categorised and corresponding residuals are modified or filtered based on the categorization.

[0086] As indicated herein, once the residuals have been computed (e.g. by comparators 110 and / or 119 in Figure 1), the residuals may be processed to decide how the residuals are to be encoded and transmitted. As described earlier, residuals are computed by comparing an original form of an image signal with a reconstructed form of an image signal. For example, in one case, residuals for an L-2 enhancement stream are determined by subtracting an output of the upsampling from an original form of an image signal (e.g. the input video as indicated in the Figures). The input to the upsampling may be said to be a reconstruction of a signal following a simulated decoding. In another case, residuals for an L-l enhancement stream are determined by subtracting an image stream output by the base decoder from a downsampled form of the original image signal (e.g. the output of the downsampling).

[0087] In one residual mode, a decision may be made as to whether to encode and transmit a given set of residuals. For example, in one residual mode, certain residuals (and / or residual blocks - such as the 2x2 or 4x4 blocks described herein) may be selectively forwarded along the L-2 or L-l enhancement processing pipelines by the ranking components and / or the selection components. Put another way, different residual modes may have different residual processing in the L-2 and L-l encoding components in Figure 1. For example, in one residual mode, certain residuals may not be forwarded for further L-2 or L-l encoding, e.g. may not be transformed, quantized and entropy encoded. In one case, certain residuals may not be forwarded by setting the residual value to 0 and / or by setting a particular control flag relating to the residual or a group that includes the residual. Control flags will be discussed in more detail below.

[0088] In one residual mode, a binary weight of 0 or 1 may be applied to residuals, e.g. by the components discussed above. This may correspond to a mode where selective residual processing is "on". In this mode, a weight of 0 may correspond to "ignoring" certain residuals, e.g. not forwarding them for further processing in an enhancement pipeline. In another residual mode, there may be no weighting (or the weight may be set to 1 for all residuals); this may correspond to a mode where selective residual processing is "off". In yet another residual mode, a normalised weight of 0 to 1 may be applied to a residual or group of residuals. This may indicate an importance or "usefulness" weight for reconstructing a video signal at the decoder, e.g. where 1 indicates that the residual has a normal use and values below 1 reduce the importance of the residual. In other cases, the normalised weight may be in another range, e.g. a range of 0 to 2 may give prominence to certain residuals that have a weight greater than 1.

[0089] In the residual modes described above, the residual and / or group of residuals may be multiplied by an assigned weight, where the weight may be assigned following a categorization process applied to a set of corresponding elements and / or groups of elements. For example, in one case, each element or group of elements may be assigned a class represented by an integer value selected from a predefined set or range of integers (e.g. 10 classes from 0 to 9). Each class may then have a corresponding weight value (e.g. 0 for class 0, 0.1 for class 1 or some other non-linear mapping). The relationship between class and weight value may be determined by analysis and / or experimentation, e.g. based on picture quality measurements at a decoder and / or within the encoder. The weight may then be used to multiply a corresponding residual and / or group of residuals, e.g. a residual and / or group of residuals that correspond to the element and / or group of elements. In one case, this correspondence may be spatial, e.g. a residual is computed based on a particular input element value and the categorisation is applied to the particular input element value to determine the weight for the residual. In other words, the categorization may be performed over the elements and / or group of elements of the input image, where the input image may be a frame of a video signal, but then the weights determined from this categorization are used to weight co-located residuals and / or group of residuals rather than the elements and / or group of elements. In this way, the characterization may be performed as a separate process from the encoding process, and therefore it can be computed in parallel to the encoding of the residuals process.

[0090] It was described above how certain residuals may not be forwarded by setting the residual value to 0 and / or by setting a particular control flag relating to the residual or a group that includes the residual. In the latter case, a set of flags or binary identifiers may be used, each corresponding to an element or group of elements of the residuals. Each residual may be compared to the set of flags and prevented from being transformed based on the flags. In this way the residuals processing may be non-destructive. Alternatively the residuals may be deleted based on the flags. The set of flags is further advantageous as it may be used repeatedly for residuals or groups of residuals without having to process each set or residual independently and can be used as a reference. For example, each frame may have a binary bitmap that acts a mask to indicate whether a residual is to be processed and encoded. In this case, only residuals that have a corresponding mask value of 1 may be encoded and residuals that have a corresponding mask value of 0 may be collectively set to 0.

[0091] In a ranking and filtering mode, the set of residuals may be assigned a priority or rank, which is then compared to a threshold to determine which residuals should be de-selected or 'killed'. The threshold may be predetermined or may be variable according to a desired picture quality, transmission rate or computing efficiency. For example, the priority or rank may be a value within a given range of values e.g. floating point values between 0 to 1 or integer values between 0 and 255. The higher end of the range (e.g. 1 or 255) may indicate a highest rank or priority. In this case, a threshold may be set as a value within the range. In a comparison, residuals with corresponding rank or priority values below the threshold may be de-selected (e.g. set to 0). A decoder 400 that performs a decoding process corresponding to the encoder of Figure 3 is depicted in the block diagram of Figure 4. The decoding process is split into two halves as shown by the dashed line. Below the dashed line is the base level of the decoder 400, which may usefully be implemented in hardware. Above the dashed line is the enhancement level, which may usefully be implemented in software. The decoder 400 may comprise only the enhancement level processes, or a combination of the base level processes and enhancement level processes as needed. The decoder 400 may usefully be implemented in software, especially at the enhancement level, and may suitably sit over legacy decoding technology, particularly legacy hardware technology. By legacy technology, it is meant older technology previously developed and sold which is already in the marketplace, and which would be inconvenient and / or expensive to replace, and which may still serve a purpose for decoding signals. In other cases, the base level may comprise any existing and / or future video encoding tool or technology.

[0092] The decoder topology at a general level is as follows. The decoder 400 comprises an input (not shown) for receiving one or more input signals comprising the encoded base stream, the encoded level 1 stream, and the encoded level 2 stream together with optional headers containing further decoding information. The decoder 400 comprises a base decoder 420 at the base level, and processing blocks 400-1 and 400-2 at the enhancement level. An up-sampler 405U is also provided between the processing blocks 400-1 and 400-2 to provide processing block 400-2 with an up-sampled version of a signal output by processing block 400-1. The base decoder 420 may correspond to the base decoder 210 of Figure 2, the processing block 400-1 may correspond to the first decoder 211 of Figure 2, the processing block 400-2 may correspond to the second decoder 214 of Figure 2 and the upsampler 405U may correspond to the upsampler 213 of Figure 2.

[0093] The decoder 400 receives the one or more input signals and directs the three streams generated by the encoder 300. The encoded base stream is directed to and decoded by the base decoder 420, which corresponds to the base codec 420 used in the encoder 300, and which acts to reverse the encoding process at the base level. The encoded level 1 stream is processed by block 400-1 of decoder 400 to recreate the first set of residuals created by encoder 300. Block 400-1 corresponds to the processing block 300-1 in encoder 300, and at a basic level acts to reverse or substantially reverse the processing of block 300-1. The output of the base decoder 420 is combined with the first set of residuals obtained from the encoded level 1 stream. The combined signal is up-sampled by upsampler 405U. The encoded level 2 stream is processed by block 400-2 to recreate the further residuals created by the encoder 300. Block 400-2 corresponds to the processing block 300-2 of the encoder 300, and at a basic level acts to reverse or substantially reverse the processing of block 300-2. The up-sampled signal from up-sampler 405U is combined with the further residuals obtained from the encoded level 2 stream to create a level 2 reconstruction of the input signal 30. The output of the processing block 400-2 may be seen as decoded video similar to the decoded video 250 of Figure 2.

[0094] As noted above, the enhancement stream may comprise two streams, namely the encoded level 1 stream (a first level of enhancement) and the encoded level 2 stream (a second level of enhancement). The encoded level 1 stream provides a set of correction data which can be combined with a decoded version of the base stream to generate a corrected picture.

[0095] Figure 5 shows the encoder 300 of Figure 1 in more detail. The encoded base stream is created directly by the base encoder 320E, and may be quantized and entropy encoded as necessary. In certain cases, these latter processes may be performed as part of the encoding by the base encoder 320E. To generate the encoded level 1 stream, the encoded base stream is decoded at the encoder 300 (i.e. a decoding operation is applied at base decoding block 320D to the encoded base stream). The base decoding block 320D is shown as part of the base level of the encoder 300 and is shown separate from the corresponding base encoding block 320E. For example, the base decoder 320D may be a decoding component that complements an encoding component in the form of the base encoder 320E with a base codec. In other examples, the base decoding block 320D may instead be part of the enhancement level and in particular may be part of processing block 300- 1.

[0096] Returning to Figure 5, a difference between the decoded base stream output from the base decoding block 320D and the down-sampled input video is created (i.e. a subtraction operation 310-S is applied to the down-sampled input video and the decoded base stream to generate a first set of residuals). Here the term residuals is used in the same manner as that known in the art; that is, residuals represent the error or differences between a reference signal or frame and a desired signal or frame. Here the reference signal or frame is the decoded base stream and the desired signal or frame is the down-sampled input video. Thus the residuals used in the first enhancement level can be considered as a correction signal as they are able to 'correct' a future decoded base stream to be the or a closer approximation of the down-sampled input video that was used in the base encoding operation. This is useful as this can correct for quirks or other peculiarities of the base codec. These include, amongst others, motion compensation algorithms applied by the base codec, quantization and entropy encoding applied by the base codec, and block adjustments applied by the base codec. The components of block 300-1 in Figure 3 are shown in more detail in Figure 5. In particular, the first set of residuals are transformed, quantized and entropy encoded to produce the encoded level 1 stream. In Figure 5, a transform operation 310-1 is applied to the first set of residuals; a quantization operation 320-1 is applied to the transformed set of residuals to generate a set of quantized residuals; and, an entropy encoding operation 330-1 is applied to the quantized set of residuals to generate the encoded level 1 stream at the first level of enhancement. However, it should be noted that in other examples only the quantization step 320-1 may be performed, or only the transform step 310-1. Entropy encoding may not be used, or may optionally be used in addition to one or both of the transform step 110-1 and quantization step 320-1. The entropy encoding operation can be any suitable type of entropy encoding, such as a Huffmann encoding operation or a run-length encoding (RLE) operation, or a combination of both a Huffmann encoding operation and a RLE operation. A residuals processing operation 350-2, 350-1 may be provided in certain embodiments prior to either transform operation 310-2, 310- 1 or both. The residual processing operation 350 applies residual pre-processing as described herein, e.g. filtering the residuals received by the block so as to only pass a subset of the received residuals onto the transform operation 310 (or in other words to set certain residual values to zero such that the original values are not processed within the subsequent operations of the pipeline).

[0097] As noted above, the enhancement stream may comprise the encoded level 1 stream (the first level of enhancement) and the encoded level 2 stream (the second level of enhancement). The first level of enhancement may be considered to enable a corrected video at a base level, that is, for example to correct for encoder and / or decoder artefacts. The second level of enhancement may be considered to be a further level of enhancement that is usable to convert the corrected video to the original input video or a close approximation thereto (e.g. to add detail or sharpness). For example, the second level of enhancement may add fine detail that is lost during the downsampling and / or help correct from errors that are introduced by one or more of the transform operation 310-1 and the quantization operation 320-1.

[0098] Referring to Figure 3 and Figure 5, to generate the encoded level 2 stream, a further level of enhancement information is created by producing and encoding a further set of residuals at block 300-2. The further set of residuals are the difference between an up-sampled version (via up-sampler 305U) of a corrected version of the decoded base stream (the reference signal or frame), and the input signal 30 (the desired signal or frame).

[0099] To achieve a reconstruction of the corrected version of the decoded base stream as would be generated at the decoder 400, at least some of the processing steps of block 300-1 are reversed to mimic the processes of the decoder 200, and to account for at least some losses and quirks of the transform and quantization processes. To this end, block 300-1 comprises an inverse quantize block 320-li and an inverse transform block 310-li. The quantized first set of residuals are inversely quantized at inverse quantize block 320-li and are inversely transformed at inverse transform block 310-li in the encoder 100 to regenerate a decoder-side version of the first set of residuals.

[0100] The decoded base stream from decoder 320D is combined with this improved decoderside version of the first set of residuals (i.e. a summing operation 310-C is performed on the decoded base stream and the decoder-side version of the first set of residuals). Summing operation 310-C generates a reconstruction of the down-sampled version of the input video as would be generated in all likelihood at the decoder — i.e. a reconstructed base codec video). As illustrated in Figure 3 and Figure 5, the reconstructed base codec video is then up-sampled by up-sampler 305U.

[0101] The up-sampled signal (i.e. reference signal or frame) is then compared to the input signal 30 (i.e. desired signal or frame) to create a second set of residuals (i.e. a difference operation 300-S is applied to the up-sampled re-created stream to generate a further set of residuals). The second set of residuals are then processed at block 300-2 to become the encoded level 2 stream (i.e. an encoding operation is then applied to the further or second set of residuals to generate the encoded further or second enhancement stream).

[0102] In particular, the second set of residuals are transformed (i.e. a transform operation 310- 2 is performed on the further set of residuals to generate a further transformed set of residuals). The transformed residuals are then quantized and entropy encoded in the manner described above in relation to the first set of residuals (i.e. a quantization operation 320-2 is applied to the transformed set of residuals to generate a further set of quantized residuals; and, an entropy encoding operation 320-2 is applied to the quantized further set of residuals to generate the encoded level 2 stream containing the further level of enhancement information). However, only the quantization step 20-1 may be performed, or only the transform and quantization step. Entropy encoding may optionally be used in addition. Preferably, the entropy encoding operation may be a Huffmann encoding operation or a run-length encoding (RLE) operation, or both. Similar to block 300-1, the residual processing operation 350-2 acts to pre-process, i.e. filter, residuals prior to the encoding operations of this block.

[0103] Thus, as illustrated in Figures 3 and 5 and described above, the output of the encoding process is a base stream at a base level, and one or more enhancement streams at an enhancement level which preferably comprises a first level of enhancement and a further level of enhancement. As discussed with reference to previous examples, the operations of Figure 5 may be applied in parallel to coding units or blocks of a colour component of a frame as there are no inter-block dependencies. The encoding of each colour component within a set of colour components may also be performed in parallel (e.g. such that the operations of Figure 5 are duplicated according to (number of frames) * (number of colour components) * (number of coding units per frame)). It should also be noted that different colour components may have a different number of coding units per frame, e.g. a luma (e.g. Y) component may be processed at a higher resolution than a set of chroma (e.g. U or V) components as human vision may detect lightness changes more than colour changes.

[0104] The encoded base stream and one or more enhancement streams are received at the decoder 400. Figure 6 shows the decoder of Figure 4 in more detail.

[0105] The encoded base stream is decoded at base decoder 420 in order to produce a base reconstruction of the input signal 30 received at encoder 300. This base reconstruction may be used in practice to provide a viewable rendition of the signal 30 at the lower quality level. However, the primary purpose of this base reconstruction signal is to provide a base for a higher quality rendition of the input signal 30. To this end, the decoded base stream is provided to processing block 400-1. Processing block 400-1 also receives encoded level 1 stream and reverses any encoding, quantization and transforming that has been applied by the encoder 300. Block 400-1 comprises an entropy decoding process 430-1, an inverse quantization process 420-1, and an inverse transform process 410-1. Optionally, only one or more of these steps may be performed depending on the operations carried out at corresponding block 300-1 at the encoder. By performing these corresponding steps, a decoded level 1 stream comprising the first set of residuals is made available at the decoder 400. The first set of residuals is combined with the decoded base stream from base decoder 420 (i.e. a summing operation 410-C is performed on a decoded base stream and the decoded first set of residuals to generate a reconstruction of the down-sampled version of the input video — i.e. the reconstructed base codec video). As illustrated in Figure 4 and Figure 6, the reconstructed base codec video is then up-sampled by up- sampler 405U.

[0106] Additionally, and optionally in parallel, the encoded level 2 stream is processed at block 400-2 of Figure 2 in order to produce a decoded further set of residuals. Similarly to processing block 300-2, processing block 400-2 comprises an entropy decoding process 430-2, an inverse quantization process 420-2 and an inverse transform process 410-2. Of course, these operations will correspond to those performed at block 300-2 in encoder 300, and one or more of these steps may be omitted as necessary. Block 400-2 produces a decoded level 2 stream comprising the further set of residuals and these are summed at operation 400-C with the output from the up-sampler 405U in order to create a level 2 reconstruction of the input signal 30. The level 2 reconstruction may be viewed as an output decoded video such as 250 in Figure 2. In certain examples, it may also be possible to obtain and view the reconstructed video that is passed to the upsampler 405U - this will have a first level of enhancement but may be at a lower resolution than the level 2 reconstruction.

[0107] Thus, as illustrated and described above, the output of the decoding process is an (optional) base reconstruction, and an original signal reconstruction at a higher level. This example is particularly well-suited to creating encoded and decoded video at different frame resolutions. For example, the input signal 30 may be an HD video signal comprising frames at 1920 x 1080 resolution. In certain cases, the base reconstruction and the level 2 reconstruction may both be used by a display device. For example, in cases of network traffic, the level 2 stream may be disrupted more than the level 1 and base streams (as it may contain up to 4x the amount of data where downsampling reduces the dimensionality in each direction by 2). In this case, when traffic occurs the display device may revert to displaying the base reconstruction while the level 2 stream is disrupted (e.g. while a level 2 reconstruction is unavailable), and then return to displaying the level 2 reconstruction when network conditions improve. A similar approach may be applied when a decoding device suffers from resource constraints, e.g. a set-top box performing a systems update may have an operation base decoder 220 to output the base reconstruction but may not have processing capacity to compute the level 2 reconstruction.

[0108] The encoding arrangement also enables video distributors to distribute video to a set of heterogeneous devices; those with just a base decoder 220 view the base reconstruction, whereas those with the enhancement level may view a higher-quality level 2 reconstruction. In comparative cases, two full video streams at separate resolutions were required to service both sets of devices. As the level 2 and level 1 enhancement streams encode residual data, the level 2 and level 1 enhancement streams may be more efficiently encoded, e.g. distributions of residual data typically have much of their mass around 0 (i.e. where there is no difference) and typically take on a small range of values about 0. This may be particularly the case following quantization. In contrast, full video streams at different resolutions will have different distributions with a non-zero mean or median that require a higher bit rate for transmission to the decoder.

[0109] As is seen by the examples of Figures 4 and 6, the residual modes may be applied at the encoder and the decoder may not require any additional residual processing. However, when residual processing is applied at the encoder, the level 1 and / or level 2 enhancement streams that are received at the decoder may differ from a comparative case wherein residual processing is not applied at the encoder. For example, when residual processing is applied, e.g. as per any of the examples described herein, the level 1 and / or level 2 enhancement streams will typically contain a greater number of 0 values that may be more efficiently compressed by the entropy encoding stages.

[0110] Figure 7 illustrates an implementation example of the encoding process described above and illustrated. As is clearly identifiable, the encoding and decoding steps of the stream and expanded in detail.

[0111] In general, the steps include a residuals filtering mode step, a transform step, a quantization step and an entropy encoding step. The encoding process identifies if the residuals filtering mode is selected. The residual filtering mode may comprise a form of residual ranking. At a lowest level the ranking may be binary, e.g. residuals are ranked as either 0 or 1, if residuals are ranked at 0 they may not be selected for further processing; only residuals ranked 1 may be passed for further processing. In other cases, the ranking may be based on a greater number of levels. If residuals mode is selected the residuals filtering step may be performed (e.g. a residuals ranking operation may be performed on the first step of residuals to generate a ranked set of residuals). The ranked set of residuals may be filtered so that not all residuals are encoded into the first enhancement stream (or correction stream). In certain cases, the steps of ranking and filtering may be combined into a single step, i.e. some residual values are filtered out whereas other residuals values are passed for encoding.

[0112] In the example of Figure 7, if a residual mode is applied such that residual values are processed prior to encoding within one or more enhancement levels, the result of the residual processing (e.g. a modified set of residuals) is then transformed, quantized and entropy encoded to produce the encoded level 1 or level 2 streams. If a residual mode is not selected, then residual values may be passed through the residual processing component for transformation, quantization and entropy encoding.

[0113] As noted above, generally it is preferred to 'kill' residuals rather than transformed coefficients. This is because processing the residuals at an early stage, e.g. by filtering the residuals based on a rank or other categorisation, means that values may be set to 0 to simplify the computations in the later more computationally expensive stages. Moreover, in certain cases, a residual mode may be set at a block or tile level. In this case, residual pro-processing (i.e. a residual mode) may be selected for all residual values corresponding to a particular coding unit or for a particular group of coding units. As there is no interblock dependency, it does not matter if certain residual values are pre-processed whereas other residual values are not pre-processed. Being able to select a residual mode at a block or tile level enhances the flexibility of the proposed encoding scheme.

[0114] Figure 7 illustrates a residual mode control block 360-1 that may be used to apply a residual mode at one or more later stages in an encoding pipeline. Here residual mode control is shown only in the LI pathway but it may also be configured in the L2 pathway. The residual mode control block 360-1 is positioned preferably between the quantization 320-1 and entropy coding 330-1 blocks. In this case, residual values may be categorised, ranked and / or assigned a score at a residual mode selection block 350-1, yet the modification of the residual values may occur later than the residual mode selection block 350-1. Although not shown in the Figure, the residual mode control block 360-1 may control one or more of the transform operation 310-1 and the quantize operation 320-1. In one case, the residual mode selection block 350-1 may set control flags for residual elements (e.g. as described above) and these control flags may be used by the residual mode control block 360-1 to control one or more of the transform operation 310-1 and the quantize operation 320-1, or a further operation following the quantize operation 320- 1. In one case, all residual values may be processed by the transform operation 310-1 and the quantize operation 320-1 yet filtered, weighted and / or set to zero via the residual mode control block 360-1. In another case, the quantize operation 320-1 may be configured to apply a coarser level of quantization based on a rank or priority of a residual (including binary ranks and priorities), such that the quantization operation 320-1 effectively sets a greater proportion of residual values to zero as compared to a case wherein a residual mode is not activated.

[0115] The residual mode control block 360-1 optionally also provides a degree of feedback and analyses the residuals after the effect of the processing to determine if the processing is having an appropriate effect or if it should be adjusted.

[0116] Figure 8 shows an example 800 of a residual mode being applied. The example 800 relates to an example whereby classification (i.e. categorisation) and weighting is applied. Concepts described with reference to the present example may also be applied in part to other residual modes. This example relates to a L-2 stream but a similar set of components may be provided for a L-l stream. The example is described with reference to a 2x2 coding unit but other coding units and / or pixel groupings may be used. A set of input image elements 801 (shown as pixel values ig - e.g. these may be a 16-bit or 8-bit integer representing a particular colour component, such as one of YUV or RGB, where i indicates an image row and j indicates an image column) are classified via a classification process 802 to generate a set of class indications 803 (e.g. in an integer range of 0 to 4 representing 5 classes). The class may indicate a level of contrast and / or texture. In other examples, the "class" may comprise a range for a metric, such as a contrast and / or texture metric for a grouping of pixels or residuals.

[0117] In Figure 8, the class indications 803 are then used by a weight mapping component 804 to retrieve a set of weights 805 associated with the class indications 803. In this simple example, the weights are a set of values between 0 and 1. Each class may have an associated weight that may be retrieved from a look-up table. In other cases, each weight may be a function of a class or metric value (e.g. as an example the weights in Figure 8 are l / 10thof the class value but the relationship between class value and weight may be any relationship as set by a lookup table).

[0118] In parallel in Figure 8, a set of reconstructed upsampled elements 806 (shown as elements Uij) are subtracted, by a subtraction component 807, from the input image elements 801 to generate an initial set of residuals 808 (shown as elements pj). As is shown in the Figure, each coding unit or block of residual values may be associated with a corresponding coding unit or block of picture elements and / or reconstructed picture elements at a particular resolution (for level 1 residuals, a similar process may apply but the picture elements may correspond to downsampled pixels). The residuals 808 and the set of weights 805 are then input to a weight multiplication component 809 that multiplies the residuals 808 by the set of weights 805 to output a set of modified residuals 810 (shown as r'ij). As may be seen, a weight of 0 may act to set a subset of the residuals to 0 (see 812). As such in the example of Figure 8, the original residual value rio is not passed on to further processing, instead it is set to 0. Residuals that have a non-zero weight applied (such as 811) are passed on for further processing but have been modified. In a simple case with binary weights (e.g. two classes), then a weight of 1 may indicate that the residual value is to be processed without modification. Non-zero weights may modify residuals in a manner that modifies how they are encoded. For example, the classification at block 802 may comprise an image classification, whereby residuals are modified based on the image classification of particular pixels. In another case, the classification at block 802 may comprise assigning the image values 801 to a particular grouping based on one or more of luma and contrast. In other examples, the classification at block 802 may select a single class and weight for the coding unit of four elements.

[0119] In certain cases, the characterization may be performed at a location remote from the encoder and communicated to the encoder. For example, a pre-recorded movie or television show may be processed once (e.g. by applying classification 802 and weight mapping 804) to determine a set of weights 805 for a set of residuals or group of residuals. These weights may be communicated over a network to the encoder, e.g. they may comprise the residual masks described with reference to Figures 9A to 9C, as will be described in more detail below. Alternatively, the classification 802, or both the classification 802 and the weight mapping 804, may instead be performed as part of the encoder parameter pre-processing described below.

[0120] In one case, instead of, or as well as weighting the residuals, the residuals may be compared against one or more thresholds derived from the categorization process. For example, the categorisation process may determine a set of classes that have an associated set of weights and thresholds, or just an associated set of thresholds. In this case, the residuals are compared with the determined thresholds and residuals that fall below a certain one or more thresholds are discarded and not encoded. For example, additional threshold processing may be applied to the modified residuals from Figure 8 and / or the weight mapping 804 and weight multiplication 809 stages may be replaced with threshold mapping and threshold application stages. In general, in both cases for this example, residuals are modified for further processing based on a categorisation process, where the categorisation process may be applied to corresponding image elements.

[0121] Note that illustrated in Figure 8, for one particular implementation, a local classification step may be optional (e.g. as indicated by the dotted line). In this case, one or more of the class indications 803 and the set of weights 805 may be obtained by a local process (e.g. from a remote location and / or from a stored file, and may be obtained from an encoder parameter pre-processing technique described below).

[0122] The above described methods of residual mode processing may be applied at the encoder but not applied at the decoder. This thus represents a form of asymmetrical encoding that may take into account increased resources at the encoder to improve communication. For example, residuals may be weighted to reduce a size of data transmitted between the encoder and decoder, allowing increases of quality for constrained bit rates (e.g. where the residuals that are discarded have a reduced detectability at the decoder). Residual weighting may have a complex effect on transformation and quantization. Hence, residual weights may be applied so as to control the transformation and quantization operations, e.g. to optimise a bit-stream given a particular available bandwidth.

[0123] In addition to the above described encoding and decoding technologies, in certain examples described herein, information from two of more frames of video that relate to different time samples may be used, as described in WO 2020 / 188273. This may be described as a temporal mode, e.g. as it relates to information from different times. Not all embodiments may make use of temporal aspects.

[0124] A step of encoding one or more sets of residuals may utilise a temporal buffer that is arranged to store information relating to a previous frame of video. In one case, a step of encoding a set of residuals may comprise deriving a set of temporal coefficients from the temporal buffer and using the retrieved set of temporal coefficients to modify a current set of coefficients. "Coefficients", in these examples, may comprise transformed residuals, e.g. as defined with reference to one or more coding units of a frame of a video stream - approaches may be applied to both residuals and coefficients. In certain cases, the modifying may comprise subtracting the set of temporal coefficients from the current set of coefficients. This approach may be applied to multiple sets of coefficients, e.g. those relating to a level 1 stream and those relating to a level 2 stream. The modification of a current set of coefficients may be performed selectively, e.g. with reference to a coding unit within a frame of video data.

[0125] Referring to Figure 5, temporal prediction may be applied between the transformation 310-1, 310-2 and quantization 320-1, 320-2 steps of encoding the level 1 and / or level 2 stream. Referring to Figure 6, temporal prediction may be applied after the inverse transformation 410-1, 410-2 steps of decoding the level 1 and / or level 2 stream.

[0126] In certain examples, there may be at least two temporal modes.

[0127] • A first temporal mode that does not use the temporal buffer or that uses the temporal buffer with all zero values. The first temporal mode may be seen as an intra-frame mode as it only uses information from within a current frame. In the first temporal mode, following any applied ranking and transformation, coefficients may be quantized without modification based on information from one or more previous frames.

[0128] • A second temporal mode that makes use of the temporal buffer, e.g. that uses a temporal buffer with possible non-zero values. The second temporal mode may be seen as an inter-frame mode as it uses information from outside a current frame, e.g. from multiple frames. In the second temporal mode, following any applied residual prioritization and transformation, previous frame dequantized coefficients may be subtracted from the coefficients to be quantized.

[0129] Temporal processing may be selectively applied at the encoder and / or the decoder based on an indicated temporal mode. A temporal mode may be signalled for one or more of the two enhancement streams (e.g. at level 2 and / or at level 1). The temporal mode may be signalled independently for each level of enhancement. Each level of enhancement may use a different temporal buffer.

[0130] Additionally, a temporal refresh parameter may signal when a temporal buffer is to be refreshed, e.g. where a first set of values stored in the temporal buffer are to be replaced with a second set of values. Temporal refresh may be applied at one or more of the encoder and the decoder. For example, in the encoder, a temporal buffer may store dequantized coefficients for a previous frame that are loaded when a temporal refresh flag is set (e.g. is equal to 1 indicating "refresh"). In this case, the dequantized coefficients are stored in the temporal buffer and used for temporal prediction for future frames (e.g. for subtraction) while the temporal refresh flag for a frame is unset (e.g. is equal to 0 indicating "no refresh"). In this case, when a frame is received that has an associated temporal refresh flag set to 1, the contents of the temporal buffer are replaced. This may be performed on a per frame basis and / or applied for portions of a frame such as tiles or coding units.

[0131] Figure 9 shows a graphical representation 900 of the decoding process described in certain examples herein, making use of a temporal buffer. The various stages in the decoding process are shown from left to right in Figure 9. The example of Figure 9 shows how an additional up-sampling operation may be applied following the decoding of the base picture.

[0132] At the far left of Figure 9, a decoded base picture 902 is shown. This may comprise the output of the base decoder as described in examples herein. In the present example, a selectable up-sampling (i.e. up-scaling) is performed on a lower resolution decoded base picture 902. The lower resolution decoded base picture 902 may be considered as a level 0 or layer 0 signal. Up-sampling of a decoded base picture may be applied based on a signalled scaling factor.

[0133] Figure 9 shows a first up-sampling operation to generate a preliminary intermediate picture 904. This may be considered to be at a spatial resolution associated with the level 1 enhancement (e.g. a level 1 signal). In Figure 9, the preliminary intermediate picture 904 is added 906 to a first layer of decoded residuals 908 (e.g. as resulting from enhancement level 1) to generate a combined intermediate picture 910. The combined intermediate picture 910 may then be up-sampled during a second upsampling operation to generate a preliminary output picture 912. The second up-sampling operation may be selectively applied (e.g. may be omitted or only performed in one dimension rather than two) depending on a signalled scaling factor. The preliminary output picture 912 may be considered to be at a level 2 spatial resolution.

[0134] At stage 914, the preliminary output picture 912 is added to a second layer of decoded residuals 916 (e.g. as resulting from enhancement level 2). The second layer of decoded residuals 916 are shown with an added 918 contribution from information stored in a temporal buffer 920. The information 920 may reduce the amount of information needed to reconstruct the second layer of residuals 916. This may be of benefit as there is more data at the second level (level 2) due to the increased spatial resolution (e.g. as compared to the first level - level 1 - resolution). In Figure 9, the output of the last addition is a final combined output picture 922. This may be viewed as a monochrome video, and / or the process may be repeated for a plurality of colour components or planes to generate a colour video output.

[0135] Further to the above described encoding and decoding techniques, in certain implementations, one or more parameters of the quantization operation may be controlled to control a bit rate of one or more of the encoded streams, as described in WO 2020 / 188273. Furthermore, different encoded streams (e.g. the encoded level 1 stream, and the encoded level 2 stream) may each be controlled to have a different bit rate.

[0136] In certain cases, the quantization parameters may be set based on an analysis of one or more of the base encoding and the enhancement stream encoding. Quantization parameters may be chosen to provide a desired quality level, or to maximise a quality level within a set of pre-defined bit-rate constraints.

[0137] For example, an encoder may comprise an output buffer configured to receive bits at variable bit rates and to output bits at a constant rate. Quantization parameter(s) may be controlled by reading the status of the buffer and ensuring that the buffer does not overflow or become empty, such that data are always available to be read at its output.

[0138] Such an output buffer may be arranged at an output of the encoded level 1 stream, the encoded level 2 stream or a combination of streams. Furthermore, each stream may have an individual buffer for output rate control.

[0139] In one case, the quantization parameters values are inversely related to the amount of data in the buffer. For example, in order to reduce the amount of residual data that is encoded, low values of a quantization parameter may correspond to larger quantization step-width values that result in fewer quantization bins for a given range of residual values. Conversely, high values of the quantization parameter may correspond to smaller quantization step-width values which increase the amount of encoded residual data but may also increase the fidelity of the decoded video.

[0140] Quantization parameters may in some cases include:

[0141] • a dead zone parameter such that residuals having a value in a certain range are set to zero (and effectively discarded), • a bin folding parameter such that residuals having a greater than maximum value are set to the maximum value,

[0142] • a quantization offset parameter (typically used with a dead zone parameter) such that the quantized residual values associated with quantization bins are offset from the corresponding residual value input for quantization. This can reduce the number of bits required to represent a quantized residual value, and / or

[0143] • a quantization matrix parameter. After transformation, the residual for a 2x2 or 4x4 coding unit may include coefficients such as a horizontal, vertical and diagonal directional decomposition. These may more heavily influence perception of a decoded signal than other values in a transformed residual. Accordingly, these more influential values may be quantized with smaller quantization bins, and less influential values may be quantized with larger quantization bins. The different quantization parameters for different values may be provided as a quantization matrix parameter.

[0144] Additionally, a bit rate controller of an encoder may be configured to identify "filler" bits of the encoded base stream and to discard or replace such "filler".

[0145] Having described various implementations of an encoder and decoder, the following description provides examples of a pre-analysis module which can be used with an encoder, such as an encoder as described above. Furthermore, the pre-analysis module may generate header data which can be incorporated as a stream with encoded data generated by the encoder, wherein the header data is configured to be used by a corresponding decoder.

[0146] The encoder may receive one or more of encoder parameters and residual masks. Encoder parameters may comprise values for one or more parameters that control the encoder 1000. In one case, encoder parameters may include parameters for one or more of the base encoder, the processing components for the level 1 stream and the processing components for the level 2 stream. The encoder parameters or the residual masks may be applied at a per-residual or per-residual-group (e.g. coding unit or block) level, or a perframe level or per-frame-group level. The encoder parameters may be used to configure one or more of a stream resolution, quantization, sequence processing, temporal prediction, bitrates and codec for each stream. Residual masks may comprise a weighting, e.g. from 0 to 1, to apply to sets of residuals, e.g. to apply to 2x2 or 4x4 groupings (i.e. blocks) of residuals. The residual masks may be similar to one of the class indications 803 and the set of weights 805 in Figure 8. A residual mask may be supplied as a surface for each frame of video (whereby there may multiple surfaces for different colour components). If the mask is applied at the group level, any received surface may be at a reduce resolution (e.g. for a 2x2 coding block the mask may comprise a video at halfresolution containing residual weight values). The residual masks may indicate a priority for delivery of the blocks to the decoder and / or for encoding. In another case, the residual masks may comprise a weighting that control processing of the blocks, e.g. certain blocks may be visually enhanced or weighted. Weighting may be set based on a class (e.g. a label or numeric value) applied to one or more blocks of residuals. In certain cases, the residual masks may be binary masks (e.g. binary bitmaps) indicating whether to encode the residual values.

[0147] In certain cases, the encoder may be adapted to perform encodings at a plurality of bitrates. In this case, the encoder parameters may be supplied for each of the plurality of bitrates. In certain cases, configuration data may be provided as one or more of global configuration data, per frame data and per block data. In examples, residual masks and temporal signalling may be provided on a per frame basis. For example, the plurality of bitrates may be set based on an available capacity of a communications channel, e.g. a measured bandwidth, and / or a desired use, e.g. use 2 Mbps of a 10 Mbps downlink channel.

[0148] Additionally, the encoder may feedback data to the pre-analysis module. The data may include one or more of a base codec type, a set of required bitrates and sequence information. The base codec type may indicate a type of base encoder that is used for a current set of processing. In certain cases, different base encoders may be available. In one case, the base encoder may be selected based on a received base codec type parameter; in another case, a base codec type may be selected based on local processing within the encoder. The set of bitrates that are required may indicate one or more bitrates that are to be used to encode one or more of the base stream and the two enhancement streams. Different streams may use different (or respective) bit rates. The enhancement streams may use additional bandwidth if available; e.g. if bandwidth is not available then bandwidth may be used by the encoded base and level 1 streams to provide a first level of quality at a given bitrate; the encoded level 2 stream may then use a second bit rate to provide further improvements. This approach may also be applied differentially to the base and level 2 streams in place of the base and level 1 streams. The residual processing described herein may be used together with bit rate parameters to control a bit rate of one or more of the enhancement streams.

[0149] In one case, the encoder parameters received may indicate one or more of residual modes to be applied by the encoder. Again, a residual mode may be set at a per frame, per tile, and / or per block or coding unit level. The encoder parameters may indicate modes for each stream separately or indicate a common mode for both enhancement streams. The residual mode parameters may be received by the residual mode selection components described herein. In certain cases, the residual mode selection components may be omitted and the residual mode parameters may be received by other components of the encoder directly, e.g. the components of examples herein may receive the residual mode parameters from a cloud interface of the encoder. In certain cases, each residual mode may be indicated by an integer value. The residual mode may indicate what form of residual (pre-) processing is to be applied.

[0150] The pre-analysis module may be local to the encoder or remote and may be configured to operate at the same time as the encoder. For example, the pre-analysis module and the encoder may be separate software or hardware modules of a single device. Alternatively, the pre-analysis module may be configured to operate asynchronously from the encoder.

[0151] Introduction to the Invention

[0152] In the examples below, a particular combination of a blurring filter and a sharpening filter prior to a downsampling operation during encoding, as has been described in published patent application WO2022 / 112774 Al to V-Nova International Limited, is referred to as a M-filter as shorthand. M-filter functionality during encoding may be switchable to on or off.

[0153] In the examples below, depending on the encoding scheme used, given down-sampler and up-sampler pairs at different stages may be enabled or disabled in different combinations. In one case, a direction of scaling can be set, e.g. as per the horizontal-only down sampling / up-sampling which is known as scaling mode ID. Scaling mode 2D downsamples and up-samples in both horizontal and vertical directions. Scaling mode OD does not down-sample or up-sample.

[0154] In video encoding there is concept of per-title, per-content and per-shot encoding, generally they are different strategies to optimize the encoding process for specific scenarios.

[0155] The impact of different encoding configurations on different contents has been explored by the inventors and the results show that an optimal selection of certain coding parameters in a scalable or hierarchical coding scheme (in LCEVC: examples are scaling mode (2D, ID, OD), transform on / off, transform type (e.g. 2x2 or 4x4), up / down-sampler pair and M-filter) could lead to a circa 10% BD-Rate-VMAF-NEG average gain across the whole quality spectrum (1080p, LCEVC x264 example). The concept involves initially analysing a few frames from the source video to be encoded. Subsequently, the results of this analysis are leveraged to determine or calculate optional encoding configurations using a machine learning model, which optional configurations are then employed to initiate the encoding process within the LCEVC Encoder SDK, or another suitable scalable or hierarchical encoder. In other words, the inventors have implemented a content-adaptive configuration selector for video encoding.

[0156] This disclosure is directed to an encoding model using a number of frames to capture a portion of the theoretical achievable benefit using existing pre-analysis and rate control statistics. The following description uses the term per-shot encoding to describe the concept of capturing and analysing a few frames of video (of a particular scene, or even the whole video) to generate pre-analysis and rate control statistics to be used by a configuration selector for selecting encoding parameters. The term "per-shot encoding" generally refers to splitting a long video (e.g. a movie) into shorter clips (e.g. scenes) and encoding / optimising each of them independently. A trade off exists between the time required to analyse the captured frames and the benefit of improved encoding. The inventive concept may be applied to embodiments that use more or fewer than a few frames of video. In one example, only the I-frames of a clip or scene are used for preanalysis. Alternatively, another example may run a pre-analysis on all frames in a scene, or on a few frames of a scene particularly but not exclusively the first few frames in the scene. In extreme cases, the pre-analysis may be performed on the full title or full content. Optionally, a scene detector is used to perform scene detection. While the disclosure refers to the use of the configuration selector for per-shot encoding, the use of the configuration selector is not intended to be limited to per-shot encoding.

[0157] Examples of existing rate control techniques can be found in patent publication W02020 / 188230 Al to V-Nova International Limited and pre-analysis statistics can be found in patent publication WO2023 / 187208 Al to V-Nova International Limited, the teachings of which are both incorporated by reference.

[0158] When encoding data, for example video data, it is known to control the number of bits required to encode a portion of the data. In the case of video data, this may be the number of bits to encode a frame of video data. The control of the number of bits required is known as rate control. It is known to set the bit rate at a constant, or variable value.

[0159] The most common form of rate control is known as "Constant Bit Rate", or CBR, encoding whereby a target bit rate, e.g. in kilobytes or megabytes per second for an encoded video stream, is supplied as an input parameter for an encoding process. The encoding process then aims to achieve the target bit rate over a set of encoded frames. For the encoding, an average bit rate may be constrained to be within a particular tolerance range of the target bit rate.

[0160] "Variable Bit Rate", or VBR, encoding is a variation of CBR encoding. In this case, a bit rate is allowed to vary during encoding. For example, the bit rate may be allowed to vary within a defined range supplied as an input parameter based on the complexity of different scenes, with more complex scenes having a bit rate towards the maximum of the defined range and with less complex scenes having a bit rate towards the minimum of the defined range.

[0161] Another known form of rate control uses a "Constant Rate Factor", or CRF. In this case, the data rate is adjusted to achieve, or maintain, a desired visual quality of the encoding. For encoding, the encoder chooses the bit rate to meet the desired quality and the bit rate may increase or decrease depending on the complexity of the scene to be encoded. For example, a more complex scene will require more data to encode a given level of quality than a less complex scene at the same level of quality. Thus, CRF encoding aims to maintain a constant level of visual quality when encoding, compared to maintaining a constant bitrate as is found in constant bitrate encoding.

[0162] A variation of CRF encoding is capped CRF encoding. In this case, a CRF is used as above but a further maximum bit rate constraint is provided. For example, a user may supply a maximum bit rate as an input and an encoder encodes the video in a CRF mode while attempting not to exceed the maximum bit rate.

[0163] Therefore, in video encoding, the bit rate may increase or decrease depending on the complexity of the scene to be encoded. A more complex scene will require more data to encode a given level of quality than a less complex scene at the same level of quality. CRF for example will maintain a constant level of quality when encoding, compared to maintaining a constant bitrate.

[0164] A modified version of CRF which is useful in scalable or hierarchical coding schemes of the type described herein exists, in which a single encoding quality factor for a video to be encoded (e.g., a CRF for the video) may be passed to an enhancement encoder and converted into quality factors for a base layer and an enhancement layer. The quality factor for the base layer may be determined for the video and per-frame quality factors may be determined for the enhancement layer based on encoding parameters received from a base encoder. These per-frame quality factors may then be used to output enhancement encoding parameters. The enhancement encoder is thus able to adapt the enhancement encoding based on properties of the base encoding to achieve a desired quality level. Using a single encoding quality factor for both layers allows the enhancement encoder to emulate the visual quality range of existing single-layer encoders such as H.264 or H.265 encoders. In the hierarchical examples given for an understanding of the invention, the modified CRF is used.

[0165] Generate data and evaluate configurations

[0166] As a first step, data was collected to determine the optimal configurations for an example hierarchical and scalable codec, in this instance an LCEVC encoder. The configurations chosen were scaling mode (2D, ID, OD), transform (DD / DDS), M-Filter and upsampling / dowsampling pairs yielding a total of 81 potential configuration setting permutations (config IDs).

[0167] Then, a determination was performed to find the best config IDs for each source video resolution (e.g., 540p, 1080p) and / or target quality band using a modified CRF measure (e.g., for three quality bands: high: modified CRF 14-23, medium: modified CRF 21-30, low: modified CRF>28).

[0168] The best or most valuable configurations were chosen for each resolution and each quality band though an evaluation process, pruning the possible configurations from 81 to circa 10-20. Video multi-method assessment fusion (VMAF) with or without no enhancement gain (NEG) can be used to evaluate the outcomes. VMAF, with or without NEG, can be used to calculate a "Bjontegaard delta rate", or BD-rate for making the evaluation, or the BD-rate may be calculated by another method in order to make the evaluation. The BD- rate allows the measurement of the bitrate reduction offered by a codec or codec feature while maintaining the same quality as measured by objective metrics. Additionally, a structural similarity index measure (SSIM) and peak signal-to-noise ratio PSNR can be used for evaluating configurations. Once the evaluation has been performed, the best or most valuable configurations are determined.

[0169] Content-adaptive configuration selector

[0170] Following on, the rate control and pre-analysis parameters expected to correlate with the configuration choices were identified (e.g., % edges, motion modifier, % tiles with residuals, complexity indexes). Data was analysed correlations determined to predict the best configurations and the optimal number of frames to use for an optimised or best or acceptable speed / quality trade-off. The outcome was a description of a prediction algorithm for implementation (possible models: decision trees, simple linear / non-linear models).

[0171] Implementation Enabling a configuration selector is computationally expensive and time-consuming, so implementation was considered carefully. For example an option to disable the configuration selector in low latency scenarios is useful and is implemented on one specific embodiment (e.g. low latency is an option of lcevc_tune which can then disable the configuration selector (e.g. per-shot encoding optimisation)).

[0172] The number of frames of source video for pre-analysis may be user configurable or preset, at for example, any one of 1 to 20 frames, and possibly 10 frames. In some examples, the number of frames is less than 10 frames, is less than 9 frames, less than 8 frames, is less than 7 frames, is less than 6 frames, is less than 5 frames, is less than 4 frames, is less than 3 frames, is 2 frames or is 1 frame. In some examples, the number of frames is more than one frame but less than the aforementioned numbers of frames.

[0173] In one particular embodiment, the video decoder should be able to decode concatenated streams where different encoding configurations have been used in each concatenated part. In an example LCEVC implementation, the LCEVC specification allows for configuration changes within a video.

[0174] Exemplary embodiment

[0175] In order to achieve the aforesaid optimised configuration settings, an exemplary embodiment is described with reference to Figure 10 which shows a block diagram of a source video 1000 which is to be encoded, a pre-encoding analysis block 1010 according to an aspect of the invention and video encoder 1040. The pre-encoding analysis block 1010 comprises a video analysis module 1020 and a configuration selector 1030.

[0176] The video analysis module 1020 is configured to receive the source video 1000 as a sample video, possibly comprising a predetermined number or selected number of frames of the source video 1000 to be encoded 1000, as explained in more detail above, as an input. The sample video may be one or more frames of the video to be encoded 1000, or may be one or more frames from a shot within the video (e.g. renewed when a scene change is detected), or may be the whole source video 1000 especially if the source video 1000 is particularly short, or may be a group of pictures (GOP), or may be the I-frames from a plurality of GOPs. One use of the disclosed content-adaptive configuration selector could be to analyse the I-frame of a GOP and then choose the optimal configuration for the GOP frames (possibly including the I-frame too). Doing the selection on each GOP based on the I-frame (even without analysing the subsequent GOP frames) has particular but not exclusive use for real-time applications. The video analysis module 1020 is configured to receive a default configuration 1022 for the video encoder 1040 as an input, wherein the default configuration 1022 is to be used for deriving rate control and pre-analysis parameters based on the sample video. The video analysis module 1020 is configured to receive a quality parameter or factor 1024 as an input, the quality parameter or factor 1024 to be used for deriving rate control and preanalysis parameters based on the sample video. The quality parameter or factor may comprise one or more of the following: CRF, modified CRF and video resolution. As mentioned above, the modified CRF is a special type of CRF value that creates an encoded video having broadly constant quality and which is used in scalable or hierarchical coding schemes, such as MPEG-5 LCEVC type compatible coding schemes. The modified CRF is used in a content adaptive rate control algorithm. The video resolution is the video resolution that is to be produced by the video encoder at the highest level, and is typically the resolution of the source video 1000. The modified CRF value for the encoding and a desired bitrate are useful for video analysis module to compile most useful statistics.

[0177] The video analysis module 1020 is configured to extract video statistics 1028 from the sample video before the source video is encoded by performing some pre-encoding analysis using the sample video and for some video statistics also the inputs.

[0178] A first type of video statistics are derived from the source video or a sample thereof and are calculated from the pixel data.

[0179] As a first stage of pre-analysis, a perception metric generator generates a detail perception metric based on one or more frames of the input video 100.

[0180] The detail perception metric is a metric for how noticeable the details shown in the one or more frames are expected to be, and / or how noticeable changes in those details are expected to be. The detail perception metric may be generated for individual coding units or blocks of a frame, or for a whole frame. Additionally, the detail perception metric may be generated for different planes of data, such as different colour components of a video signal. Additionally or alternatively, the detail perception metric may be calculated by comparing two or more frames.

[0181] In some examples, the detail perception metric may comprise an edge detection metric. A user may be more likely to notice loss of detail in the edge of an object depicted in a frame, when compared to loss of detail in the bulk of the object.

[0182] The edge detection metric may be implemented using a transform. The transform may be similar to the method using for transforming residuals in elements 310-1 and 310-2 of Figure 3, but applied to frames (or coding units or blocks thereof). The transform as described herein may use a directional decomposition transform such as a Hadamard- based transform. The transform may comprise a small kernel or matrix that is applied to flattened coding units of the frame (i.e. 2x2 or 4x4 blocks of pixels). The pre-analysis module may select between different transforms to be used, for example between a size of kernel to be applied.

[0183] The transform may transform the pixel information to surfaces. For example, the transform may produce the following components: vertical tilt ("V", corresponding to the vertical difference of the pixels, as the sum of the pixels on top minus sum of the pixels on the bottom), horizontal tilt ("H", corresponding to the horizontal difference of the pixels, as the sum of the pixels on the left minus the sum of the pixels on the right) and diagonal tilt ("D”, corresponding to the remaining differences, not explained by a simple directional transition in the blocks of pixels). For reference, an example of directional decomposition is shown in PCT / EP2013 / 059847 Figures 3A and 3B.

[0184] The edge detection metric may alternatively comprise a binary choice, or a selection from a discrete set of options, such as: no edges, few edges, many edges. Such a selection may be based on comparing one or more elements of the directional decomposition transform to one or more respective thresholds. The thresholds may in turn depend on high-level parameters for encoding, such as a required bit rate. For example, when a low bitrate is required, the threshold for determining that there are edges may be relatively high, so that most of the input video is encoded more compactly.

[0185] Furthermore, the edge detection metric may comprise a text detection metric. Text features are commonly defined by edges, and the user is particularly likely to notice loss of detail in text depicted in a frame.

[0186] In some examples, the detail perception metric may comprise a motion metric based on comparing two or more frames. A user may be more likely to notice loss of detail in directional motion, when compared to loss of detail in other types of motion. Furthermore, when a frame or portion of a frame is static, it may be easier for viewers to spot tiny details, and therefore it may be important to preserve residual information, e.g. a priority of certain static residual elements may be higher than a comparative set of transient residual elements. Also sources of noise in an original video recording at higher resolutions (e.g. an L-2 enhancement stream) may lead to many small yet transient residual values (e.g. normally distributed values of -2 or -1 or 1 or 2) - these may be given a lower priority and / or set to 0 prior to residual processing in the enhancement level encoders. The motion metric may comprise a sum of absolute differences (SAD) between a pair of frames. The motion metric may be evaluated in this manner per frame pair, per block pair or per coding unit pair.

[0187] For example, a motion metric for motion between a frame m and a frame n may be based on Jo = Sum(abs(Ix,y,n - Ix,y,m)), where Ix,y,n is a value for coding unit (x,y) of frame n, and Ix,y,m is a value for coding unit (x,y) of frame m.

[0188] Furthermore, when the motion metric is based on comparing more than two frames, the motion metric may comprise a weighted sum of SAD values. For example, a detail perception metric for a frame n may be calculated by comparing frame n to each of preceding frames k and m, and the motion metric may be based on: o = Sum(abs(Ix,y,n - Ix,y,m)) + Sum(abs(Ix,y,n - Ix,v,k)), Or

[0189] Jo=Wm‘Sum(abS(Ix,y,n—Ix,y,m)) + Wk‘Sum(abS(Ix,y,n—Ix,y,k)), where wmand Wk are weighting factors.

[0190] The motion metric may alternatively comprise a binary choice, or a selection from a discrete set of options, such as: no motion, low motion, high motion. Such a selection may be based on comparing the sum of absolute differences to one or more thresholds. The thresholds may in turn depend on high-level parameters for encoding, such as a required bit rate. For example, when a low bitrate is required, the threshold for determining that there is motion may be relatively high, so that most of the input video is encoded more compactly.

[0191] The first frame and second frame used to generate the motion metric may be consecutive frames of the input video 100, or the motion metric may be generated at a reduced frequency (e.g. comparing motion between two frames separated by N>1 intermediate frames of the input video, comparing motion between randomly sampled frames of the input video, etc.) depending on contextual requirements. The frequency of generating the motion metric may depend upon the motion metric (for example, decreasing motion metric generation frequency after generating the detail perception metric for a series of frames exhibiting low motion).

[0192] The number of times the motion metric is calculated may be reduced by reusing the same calculation for forward and backward motion. In other words, when a motion metric is calculated by comparing frames m and n, this motion metric may be used when generating a detail perception metric for frame m and when generating a detail perception metric for frame n. For example, adjacent frames may be paired up, with the motion metric calculated once for each pair of frames (i.e. a motion metric is calculated for frames 1 and 2, for frames 3 and 4, for frames 5 and 6, etc.).

[0193] The detail perception metric may comprise a combination of metrics. For example, the detail perception metric may comprise an edge detection metric based on a second frame and a motion metric based on a difference between first and second frames.

[0194] More specifically in some implementations, the statistics are derived from features of the source video or a sample thereof wherein each feature is a 4x4 block surface. A non- exhaustive list of video statistics include histogram-based statistics and motion-based statistics. The video statistics can reveal properties of the source video, such as a low or a high proportion of edges, a low or a high amount of motion, a low or a high amount of camera movement, etc. The first type of video statistics are derived in this specific embodiment independently from any encoding parameters (except for the choice of downsampler if the statistics are computed on a downsampled version of the source video, e.g. LCEVC LI). Each feature may have one or more of the following surfaces: contrast (higher block value -> higher contrast in block); texture (similar to an edge filter); type analysis (based on texture and contrast); mean (mean pixel value, useful for postprocessing normalisation); histogram.

[0195] The video analysis module 1020 is configured to create post-processed / summarised versions of the features. In a particular implementation, each feature surface is summarised into a single number for each frame.

[0196] In this described example embodiment, for continuously pre-analysed features (histogram, contrast, mean, texture), the mean and the variance are calculated, e.g. histogram_variance; histogram_mean; contrast_variance; contrast_mean; mean_variance; mean_mean; texture_variance; and texture_mean.

[0197] In this described example embodiment, for discrete pre-analysed features (only the type analysis), a histogram is calculated, and for each type, a new feature is created: green_percent; blue_percent; red_percent; and pink_percent.

[0198] A second type of video statistics are derived from the source video or a sample thereof and depend on the encoder and rate control method used. The encoding process and / or rate control is performed or simulated on the source video or sample. For example, a quantisation parameter may be chosen depending on a modified CRF and target bitrate, and transforms may or may not be used in the encoding process depending on the default configuration for the encoder. Residual information generated by the encoder or encoder simulation is then summarised using at least one or more of the following: minimum / maximum values, and standard deviation. Rate control information is obtained.

[0199] The video analysis module 1020 is configured to output the video statistics 1028 to the configuration selector 1030.

[0200] Configuration Selector

[0201] The configuration selector 1030 is configured to receive the video statistics 1028 from the video analysis module 1020 as an input. The configuration selector 1030 is configured to receive the default configuration 1022 for the encoder 1010 as an input. The configuration selector 1030 is configured to receive the quality parameter or factor 1024 as an input. The quality parameter or factor 1024 may comprise one or more of the following: CRF, modified CRF and video resolution, as necessary.

[0202] The configuration selector 1030 is configured to predict and select an encoder configuration 1038 based on the video statistics 1028 received from the video analyser 1020 whereby the encoder configuration provides an optimal, near optimal, efficient or satisfactory encoding of the source video data at the desired quality parameter or factor 1024.

[0203] The configuration selector 1030 is configured to use a heuristic-based set of rules, discovered via data analysis on collected datasets of source video, as mentioned above. The configuration selector 1030 uses a machine learning process to determine which encoder configuration to predict and select. In this specific example, the configuration selector 1030 uses a cascading classifier model as would be known to those skilled in the art of machine learning. The cascading classifier model cascades to settle at a discrete set of configurations (config IDs) rather than running through every possible parameter.

[0204] The encoder configuration 1038, when determined, is output to the video encoder 1040.

[0205] The video encoder 1040 is configured to encode the source video 1000 using the encoder configuration 1038 received from the configuration selector 1030. The video encoder also receives as inputs the quality metric 1024.

[0206] The video encoder 1040 is configured to output corresponding encoded video 1050. The encoded video 1050 is encoded to the quality metric 1024 (e.g. the CRF, modified CRF and / or resolution) at an efficient bitrate because of the pre-encoding analysis block 1010. Figure 11 shows an example cascading classifier model used in the specific embodiment described with reference to Figure 10 to determine the encoder configuration from the video statistics 1028.

[0207] Each circle in Figure 11 is a classifier and corresponds to a multiclass classifier (e.g. a decision tree or a neural network), possibly choosing the value of one configuration parameter (e.g. scaling) or of another user-defined property (e.g. static / high-motion video).

[0208] Each square corresponds to a specific configuration selected by the configuration selector 1030.

[0209] Classifier 1110 is a "corner"-case video and is defined as a video where a default codec configuration is predicted to underperform compared to other alternative configurations. If false, then the selector chooses the default codec configuration 81. If true, then the selector moves to the next classifier 1120.

[0210] Classifier 1120 is a "0D / 2D" classifier and predicts the scaling configuration to be used. If 0D scaling is predicted then the selector chooses the codec configuration 79. If 2D is predicted, then the selector moves to the next classifier 1130.

[0211] Classifier 1130 is a "M-Filter On / Off" classifier and predicts whether M-Filtering should be used during encoding. If M-Filter Off is predicted then the selector chooses the codec configuration 7. If M-Filter On is predicted, then the selector moves to the next classifier 1140.

[0212] Classifier 1140 is a transform type or "2x2 vs 4x4" classifier and predicts whether a 2x2 or a 4x4 transform should be used during encoding. If 4x4 is predicted then the selector chooses the codec configuration 21. If 2x2 is predicted, then the selector chooses the codec configuration 33.

[0213] Table 1 below shows the configuration choices for each codec configuration shown in Figure 11.

[0214] Table 1 : Codec configurations corresponding to the selections of the selector of Figure 11

[0215] The number of circle classifiers is as large as needed. For instance, in the example of Figure 11, there is no dedicated classifier for downsampler and upsampler because the other classifiers imply the downsampler / upsampler values. However, this is an optimisation and a design choice and a skilled person could design a different cascade classifier which includes circle classifiers for downsampler or upsampler.

[0216] Figure 12 illustrates a flow chart of a method according to one aspect of the invention. The method begins at step 1210: receiving video data to be encoded. The video data, as described above, may be a whole video, or a part thereof, such as a sample or component part. The sample or component part may be a scene within the whole video, or may be selected frames from the whole video or sample or component part. Attributes of the video are then analysed as already mentioned and described in this disclosure in relation to Figure 10 and is not repeated here. The method continues, at step 1220: receiving a quality factor. The quality factor or parameter is described in this disclosure in relation to Figure 10 and is not repeated here. The method continues, at step 1230: determine an encoder configuration. The determining is in dependence on one or more attributes of the video data and the quality factor, as has already been described with reference to Figures 10 and 11 and is not repeated here. The method is embodied as a computer program in some embodiments, and as a signal in other embodiments, and may be found on a computer readable medium as instructions that when executed by a processor, cause the method to be performed.

[0217] The above embodiments are to be understood as illustrative examples. Further embodiments are envisaged. It is to be understood that any feature described in relation to any one embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.

Claims

Claims1. A method of selecting an encoder configuration for encoding video data, the method comprising: receiving video data to be encoded; receiving a quality factor to be used for encoding the video data; in dependence on one or more attributes of the video data and the quality factor, determining an encoder configuration.

2. The method of claim 1, wherein the method comprises encoding the video data with an encoder configured according to the encoder configuration.

3. The method of claim 2, wherein the method comprises generating a concatenated bitstream where different encoding configurations are signalled for each concatenated part in dependence on the corresponding encoding configuration used when encoding that concatenated part.

4. The method of any preceding claim, wherein the quality factor is one of the following: constant rate factor, CRF, a modified CRF for hierarchical coding, and a video resolution.

5. The method of any preceding claim, wherein the determining comprises predicting and selecting an encoder configuration in dependence on the one or more attributes and the quality factor.

6. The method of claim 5, wherein the determining comprises using a heuristic-based set of rules, discovered via data analysis on collected datasets of source video, and a machine learning process to determine which encoder configuration to predict and select in dependence on the one or more attributes and the quality factor.

7. The method of claim 6, wherein the determining comprises using a cascading classifier model cascading to settle at a discrete set of configurations.

8. The method of any preceding claim, wherein the method comprises analysing the video data to determine the one or more attributes.

9. The method of any preceding claim, wherein the attributes are in dependence on pixel data of the video data.

10. The method of claim 9, wherein the attributes are in dependence on statistical data derived from the pixel data.

11. The method of claim 10, wherein the statistical data are derived from (a) features of the video data, and wherein each feature of the features is a 4x4 block surface, and wherein each feature has at least one of the following surfaces: contrast, texture, type analysis, mean, and histogram.

12. The method of claim 11, wherein the attributes comprise a single number for each feature surface for each frame of the video data.

13. The method of claim 10, wherein the statistical data are derived from (b) the video data and depend on the encoder and rate control method used for encoding the video data, and the method comprises simulating encoding and / or rate control on the source video to obtain the statistical data.

14. The method of claim 13, wherein residual information generated by the simulating is summarised using one or more of the following: minimum / maximum values, and standard deviation.

15. The method of any preceding claim, wherein the video data is a component of a larger video.

16. The method of claim 15, wherein the component is less than 10 frames, is less than 9 frames, less than 8 frames, is less than 7 frames, is less than 6 frames, is less than 5 frames, is less than 4 frames, is less than 3 frames, is 2 frames or is 1 frame; is more than one frame but less than the aforementioned number of frames.

17. The method of claim 15 or claim 16, wherein the component is a scene or is part of a scene.

18. The method of claim 17, wherein the method comprises using a scene detector to determine the scene.

19. The method of any of claims 15 to 18, wherein the component is one of: an I-frame in a group of pictures; and a plurality of I-frames from adjacent groups of pictures.

20. The method of any preceding claim, wherein the encoding parameters comprise one or more of the following: scaling mode; transform on / off; transform type; upsampler; downsampler; upsampler / downsampler pair; and M-filter.

21. The method of any preceding claim, wherein the encoding is a hierarchical encoding including a base layer and at least one enhancement layer.

22. The method of claim 21, wherein there are two enhancement layers.

23. The method of claim 21 or claim 22, wherein an enhancement layer is at the same resolution as the base layer resolution.

24. The method of any of claims 21 to 23, wherein an enhancement layer is at a higher resolution than the base layer resolution.

25. A system or an apparatus for video encoding configured to perform the method of any previous claim.

26. A computer-readable medium comprising instructions which when executed cause a processor to perform the method of any of claims 1 to 24.

27. A computer program comprising instructions which when executed cause a processor to perform the method of any of claims 1 to 24.

28. A signal comprising the computer program of claim 27.

Images