Processing of residuals in video coding

Abstract

A method of encoding an input signal, the method comprising: receiving an input signal; generating a set of residuals based on a difference between the input signal and a reconstructed signal; applying a transform operation to the set of residuals to generate a set of transform coefficients; applying a quantization operation to the set of transform coefficients to generate a set of quantized values; and applying an encoding operation to the set of quantized values to generate an encoded stream; wherein applying the quantization operation comprises modifying a quantized value from a first quantized value to a second quantized value in order to adjust a quantization bin of a transform coefficient, such that the transform coefficient is quantized into second quantization bin instead of a first quantization bin.

Description

[0001] PROCESSING OF RESIDUALS IN VIDEO CODING

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to methods for processing image and video data signals.

[0004] BACKGROUND

[0005] Briefly, hybrid backward-compatible coding technologies parse a data stream into first portions of encoded data and second portions of encoded data, implement a first decoder to decode the first portions of encoded data into a first rendition of a signal, implement a second decoder to decode the second portions of encoded data into reconstruction data, the reconstruction data specifying how to modify the first rendition of the signal, and apply the reconstruction data to the first rendition of the signal to produce a second rendition of the signal.

[0006] A set of residual elements is useable to reconstruct a rendition of a first time sample of a signal. A set of spatio-temporal correlation elements associated with the first time sample is generated. The set of spatio-temporal correlation elements is indicative of an extent of spatial correlation between a plurality of residual elements and an extent of temporal correlation between first reference data based on the rendition and second reference data based on a rendition of a second time sample of the signal. The set of spatio-temporal correlation elements is used to generate output data. The set of residuals are encoded to reduce overall data size.

[0007] Encoding applications have typically employed a quantization operation in which each of one or more ranges of data values is compressed into a single value representing a bin. In other words, quantization is the procedure of constraining values of signal properties (e.g., luminance of the pixels of a picture) from a substantially continuous set of values (such as the real numbers, or a high-bit- depth set of digital values, “Original Set”) to a relatively small discrete set (such as a finite and low-bit-depth set of integer values, “Quantized Set”). Quantization allows the number of different values in a set of video data to be reduced, thereby rending that data more compressible. In this way, quantization schemes have been useful in some video for changing signals into quanta, so that certain variables can assume only certain discrete magnitudes. Generally a video codec divides visual data, in the form of a video frame, into discrete blocks, typically of a predetermined size or number of pixels. A transform is then applied to the blocks so as to express the visual data in terms of sums of frequency components. The quantization operation is applied in the domain of transformed coefficients, invertible with respect to display settings coordinates (e.g., luminance and chrominance values, RGB values, etc.). The transformed data can be premultiplied by a quantization scale code, and then subjected to division element- wise by the quantization matrix, with the output elements of the division of each transformed, pre-multiplied element by the matrix element, then being rounded. The treatment of different transformed elements with divisors, namely different elements of a quantization matrix, is typically used to allow for those frequency elements that have a greater impact upon visual appearance of the video to a viewer to be effectively allotted more data, or resolution, than less perceptible components.

[0008] Codecs (i.e. coding and decoding algorithm / processes) operating based on a multi-layer non block-based approach have been developed as an international standard by MPEG / ISO as MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC). This approach is completely agnostic of the codec used to encode the lower layer because the upper layer is decodable without any information about the lower layer. This approach uses an encoding and decoding process which processes the picture without using any inter-block prediction. Instead the picture is processed by transforming an NxM block of picture elements (e.g., 2x2 or 4x4) and processing the blocks independently from each other, resulting in efficient processing as well as in no-dependency from neighbouring blocks. The processing of the picture can therefore be parallelised.

[0009] The encoding techniques in the following specification are particularly suited to be used with LCEVC techniques. A standard specification for LCEVC is provided in the Text of ISO / IEC 23094-2 Ed 1 Low Complexity Enhancement Video Coding published in November 2021. LCEVC enhances the reproduction fidelity of a decoded video after encoding and decoding using an existing codec. This is achieved by combining a base layer with an enhancement layer, where the base layer contains the video encoded using the existing codec, and the enhancement layer indicates a residual difference between the original video and a predicted decoded video produced by decoding the base layer using the existing codec. The enhancement layer can be combined with the decoded base layer to more accurately reproduce the original video. Throughout the description it should be understood that references to video signal also apply to other types of data mutatis mutandis.

[0010] It remains an objective to effectively and efficiently integrate enhancement coding into existing ecosystems. Examples according to the present disclosure provide an integration of enhancement coding (i.e. LCEVC) in current TV specifications to enhance an available interlaced 1080i SDR stream to a progressive 1080p HDR stream to improve current TV use cases, particularly within TV hardware legacy ecosystems. One of the key advantages of enabling a conversion from an SDR interlaced video to an HDR progressive video lies in the ability to move the production of the input video from interlaced to progressive, thus simplifying the production of videos for those services that currently needs to produce both an interlaced and a progressive sequence due to the fact that many legacy systems still support interlace whereas more modem ones are progressive-based. In this sense, this solution simplifies the technical workflow for the input video, and uniform it towards a single format (typically HDR progressive 4K or higher). It should be noted that solutions presented herein are also suitable for enhancing a lower resolution interlaced video (e.g., 576i or other).

[0011] In a specific current use case, in the majority of Brazil the TV 2.0 standard is used for broadcasting as of today. While TV 3.0 will replace this in the near future, it remains an aim to enable the distribution of an enhancement layer on top of a TV 2.0 base to deliver a 10 bit HDR stream to supported receivers. TV 2.0 currently uses AVC 1080i30 SDR (BT.709 colour space) and will likely be required for backwards compatibility. TV 2.5 enables a TV 2.0 base to be enhanced with LCEVC for playback as 1080p60 HDR with BT.2020 colour space. Various types of interlacers and deinterlacers can be done to fit the TV 2.5 requirements, but it has been found that implement top field interlacer and BWDIF deinterlacer is best to implement in the TV 2.5 pipeline.

[0012] In order to implement the TV 2.5 pipeline there are 2 test cases. Firstly, in-loop application of LCEVC considers enhancement being applied between the original HDR sequence and reconstructed tone mapped HDR sequence. In this case tone mapping is done with an LUT. Secondly, out-of-loop application of LCEVC considers enhancement being applied between the tone mapped 10 bit SDR sequence and reconstructed 10 bit SDR sequence. In this case tone mapping is done with SL-HDR1 which is Interdigital solution already implemented in the TV 2.5 standard.

[0013] It has been found that LCEVC is able to better correct impairments between the source and HDR construction, as well as achieving better HDR reconstruction at low enhancement bitrate when applied on the in-loop. In the out-of-loop, LCEVC is not able to correct the HDR reconstruction artefacts, and it only fixes encoding, deinterlacing and shifting from 10 to 8 bits. It was also found that the LCEVC residuals are mostly being applied to fix artefacts of deinterlacing, rather than any issues brought by encoding or tone mapping.

[0014] Given that most of the noticeable visual quality artefacts are due to deinterlacing, it would be beneficial to provide improved LCEVC algorithms and methods which reduce the presence of artefacts in deinterlaced content.

[0015] SUMMARY OF INVENTION

[0016] According to an aspect of the present invention there is provided a method of encoding an input signal, the method comprising: receiving an input signal; generating a set of residuals based on a difference between the input signal and a reconstructed signal; applying a transform operation to the set of residuals to generate a set of transform coefficients; applying a quantization operation to the set of transform coefficients to generate a set of quantized values; and applying an encoding operation to the set of quantized values to generate an encoded stream; wherein applying the quantization operation comprises modifying a quantized value from a first quantized value to a second quantized value in order to adjust a quantization bin of a transform coefficient, such that the transform coefficient is quantized into second quantization bin instead of a first quantization bin.

[0017] A transform coefficient refers to a value that is produced when a transformation is applied to a residual or data derived from a residual. It may be a scalar quantity, that is considered to be in a transformed domain. In one case, an M by N coding unit may be flattened into an M*N one-dimensional array. In this case, a transformation may comprise a multiplication of the one-dimensional array with an M by N transformation matrix. In this case, an output may comprise another (flattened) M*N one-dimensional array. In this output, each element may relate to a different “coefficient”, e.g. for a 2x2 coding unit there may be 4 different types of coefficient. As such, the term “coefficient” may also be associated with a particular index in an inverse transform part of the decoding process, e.g. a particular index in the aforementioned one-dimensional array that represented transformed residuals.

[0018] The above method describes the process of residual promotion in which the quantization bin of a transform coefficient is adjusted from a first quantized value to a second quantized value, where a quantized value corresponds to a quantization bin. Preferably the first quantized value is zero and so the first quantization bin corresponds to a deadzone. In accordance with the residual promotion process, the transform coefficient is effectively rescued from the first quantization bin and moved to the second quantization bin. The move from one bin to another higher bin is referred to as promotion.

[0019] As discussed above, those most noticeable visual quality artefacts are due to deinterlacing and so it is an aim of the invention to try and fix these issues. In some examples, the quantization of all residuals in LCEVC is typically constant per frame, and so the idea of residual promotion is to make sure that residuals which are not passing the quantization process, but are important because they improve the visual quality of the video, are saved by changing the quantization bin they will be quantized to, and increasing the probability of the residuals surviving the quantization process.

[0020] Preferably, applying the quantization operation comprises: determining whether a transform coefficient would be quantized to zero; and in response to a positive determination, modifying the quantized value. Not all coefficients that fall within a deadzone, and are therefore quantized to zero, are considered important because they do not noticeably affect the visual quality of the video. Thus, it is preferable to initially determine whether or not a residual is visually relevant, and if so, the residual promotion process can be carried out. If it is decided that the residual is not visually relevant then the residual promotion process does not need to be carried out. Residual promotion can therefore be thought of as comprising a decision phase, in which is it determined whether or not the residual is to be promoted, and a promotion phase, in which the residual is promoted through adjustment of the quantized value of the transform coefficient of said residual. In this context, importance can be in terms of important for improving visual quality, visual perception, improving noise ratio, or any other quality which would improve the overall viewing experience. Throughout this disclosure, importance, and whether or not a residual or coefficient is important, will generally be understood to mean visually relevant i.e. an important residual or coefficient is a visually relevant residual or coefficient.

[0021] The decision phase may comprise determining that the transform coefficient would be quantized to a deadzone, wherein the deadzone is a quantization bin having a quantization value of zero. In this way, the residual promotion process only needs to continue if it is determined that the residual would not form part of the final encoded stream. If the residual is already part of the final encoded stream then the residual process is not applied.

[0022] In response to determining that a transform coefficient would be quantized to zero, the method may comprise determining whether the transform coefficient is to be promoted. A transform coefficient may be determined to be promoted if said transform coefficient should be quantized into a second quantitation bin instead of a first quantitation bin, because said transform coefficient has been determined to provide an improvement in some aspect of the overall visual quality of the video such that it is considered to be important. In particular, the transform coefficient may be determined to be promoted if said transform coefficient should be quantized into a second quantitation bin instead of a deadzone. In response to a positive determination that the transform coefficient is to be promoted, the quantized value of said transform coefficient may be modified. In this way, only important transform coefficients, i.e. that ones that have been identified as having the potential to provide an improvement in the visual quality of a video, are rescued from the deadzone and promoted rather than promoting all transform coefficients which are in the deadzone. This ensures that only residuals in which there is an expectation of providing a noticeable effect on improving the visual quality of the video are promoted, and the residuals that will not visually improve the video do not need to be promoted.

[0023] In some examples, the modifying the quantized value comprises calculating a modification factor, wherein the modification factor is used to adjust the quantization bin. This provides a simple and effective means of determining by how much the quantitation bin should be adjusted.

[0024] The step of calculating the modification factor may comprise calculating a modified transform coefficient. The modified transform coefficient may be calculated as the absolute value of the transform coefficient minus an offset value.

[0025] Calculating the modification factor may comprise determining whether the modified transform coefficient is greater than zero. This may ensure that the residual promotion operation is only applied to transform coefficients that are close to a boundary between two quantization bins. It is important to only promote residuals that are close to the boundary between two bins to avoid promoting residuals that essentially just represent noise and should remain in the deadzone. In addition, calculating the modification factor may comprises determining whether the modified transform coefficient is less than a threshold value, wherein the threshold value is a fraction of a width of a quantization bin. This may ensure that, if residual promotion is to be performed, the transform coefficient is only promoted to the next bin rather than several bins higher than its current bin. In other words, promotion only moves the transform coefficient by one bin not by more than one bin.

[0026] The threshold value may be a fraction of a width of a quantization bin, and the quantization bin may be a deadzone. The residual promotion operation is preferably promoting residuals and transform coefficients out of the deadzone into the next bin. In this context, the next bin is a bin that is a direct neighbour of, or adjacent to, the deadzone.

[0027] The modification factor may be calculated to be zero if the modified transform coefficient is less than or equal to zero. In this way, the quantized value of the transform coefficient is not modified and so the residual promotion operation is not applied to the transform coefficient. This prevents transform coefficients that are far away from a boundary between quantization bins (in particular, that are far away from the boundary between the deadzone and the next bin) from being promoted. This helps ensure that residuals that would not have a significant effect on the visual quality of the video do not get promoted and remain in the deadzone. This reduces the amount of data to be transmitted over a network and reduces the bit rate.

[0028] Modifying the quantized value of the transform coefficient, such that the transform coefficient is quantized into a different bin that it would otherwise have been quantized into, means that the final video output is changed compared to the original input video. Generally, changing the final video is preferable to not changing the final video at all, because there are improvements in the visual perception of the final image as a result of the modified quantization. However, some transform coefficients have their quantized value modified when it should not have been modified (e.g. when the residual promotion is applied at a block or tile level rather than at an individual coefficient level) and so these inadvertently affected transform coefficients also end up significantly changing the final image. To mitigate the effect that these inadvertently affected transform coefficients have on the final image, it is preferable to only promote transform coefficient(s) that are close to the boundary between the deadzone and the next bin. Promoting only residual coefficients that are just below the higher bin rather than all the residual coefficients that are in the deadzone avoids promoting residual coefficients are noise which is the type of signal that normally stays in the deadzone. In addition, promoting all residual coefficients in the deadzone, rather than only visually important residual next to the bin boundary, can cause the bitrate to significantly increase.

[0029] The modification factor may be calculated to be zero if the modified transform coefficient is greater than the threshold value. This may prevent the residual promotion operation promoting the transform coefficient by more than one bin.

[0030] Preferably, the second quantization bin is adjacent to the first quantization bin. In this way, the residual is promoted to the next bin, rather than several bins higher. All that is needed during residual promotion is to rescue residuals and transform coefficients from the deadzone. If they are promoted to bins that are several bins higher than the deadzone, then the residual has been given higher priority than it should be and becomes more significant in the final image that it should be.

[0031] Preferably, the quantization bin is adjusted from the first quantization bin to the second quantization bin when the modification factor is calculated to be non-zero. Preferably, the modification factor is one. In this way, residual promotion is only applied if the two conditions which are used to calculate the modification factor are satisfied. If satisfied, the quantization bin is adjusted by 1 . The two conditions are that the transform coefficient is close to a boundary between the deadzone and the next bin, and that the transform coefficient would only be promoted by one bin value to the next bin. This effectively limits the number of residuals which could be promoted so that the residual promotion operation is only applied to residuals which will improve the visual quality of the final video. This helps keep the bit rate low.

[0032] If the transform coefficient is positive, the transform coefficient is promoted in the positive direction i.e. from N-bin to (N+1)-bin. If the transform coefficient is negative, the transform coefficient is promoted in the negative direction i.e. from N-bin to (N-1 )-bin. In this latter case, the modification factor is -1. The transform coefficient is only promoted by one bin, in either the positive or negative direction, wherein the direction of promotion depends on the sign of the transform coefficient (i.e. whether the transform coefficient is positive or negative). When the transform coefficient is negative, the same checks and conditions are applied as those that are applicable when the transform coefficient is positive. In other words, the negative transform coefficient must still be visually relevant and is still promoted in the negative direction by one bin value if the transform coefficient lies close to a bin boundary in the negative direction. Effectively, the residual promotion operation is performed on the absolute value, and the sign of the residual (positive or negative) is included when considering the sign of the modification factor.

[0033] As has been mentioned, not all transform coefficients that fall within a deadzone, and are therefore quantized to zero, are considered important because they do not noticeable affect the visual quality of the video. Thus, it is preferable to initially determine whether or not a residual is important, and if so, the residual promotion process can be carried out. As such, the step of determining whether the transform coefficient is to be promoted may comprise determining that a residual is important and should form part of the encoded stream and identifying the corresponding transform coefficient. In this way, only residuals that would not form part of the encoded stream, but that would provide an improvement in the visual quality of the final video, are promoted. Residuals which would not form part of the encoded stream and are not needed (because they will not improve the visual quality of the video) are not promoted.

[0034] In some implementations, the step of applying the quantization operation may comprise modifying all the quantized values of all the transform coefficients of a block comprising the identified transform coefficient. In some implementations, the step of applying the quantization operation may comprise modifying all the quantized values of all the transform coefficients of a tile comprising the identified transform coefficient. Here, a tile is a group of blocks, preferably an nxm grid of blocks. In these situations, there is a trade off between improving the final image in some places (as a result of promoting important residuals which positively contribute towards improving the visual quality of the image) and making the image worse in other places (as a result of promoting residuals that form part of the block or tile such that the image is changed but do not actually change the image positively because they do not contribute towards improving the visual quality of the image).

[0035] The residual may be determined to be important if a frame of the input signal is an intra frame and a block comprising the residual is static. In this context, the frame is a frame in the input sequence. The residual may be determined to be important if a frame of the input signal in an inter frame and a temporal type of a block comprising the residual is intra. In some examples, the residual may be determined to be important if a block comprising the residual belongs to an edgetype block, preferably wherein at least one neighbouring block of said block is also an edge-type block.

[0036] According to another aspect there is provided an encoder configured to perform the method as described above. The method is performed in the transform domain compared to the residual domain.

[0037] BRIEF DESCRIPTION OF DRAWINGS

[0038] The present invention will be described by way of example only with reference to the accompanying drawings in which:

[0039] Figure 1 shows a high-level schematic of an encoding process;

[0040] Figure 2 shows a high-level schematic of a decoding process; Figure 3 shows a high-level schematic of an encoding process and specific encoding steps;

[0041] Figure 4 shows how the quantization in an encoding process may be performed; and

[0042] Figure 5 shows how the quantization in an encoding process may be performed.

[0043] DETAILED DESCRIPTION

[0044] The coding technology discussed herein is a flexible, adaptable, highly efficient and computationally inexpensive coding format which combines a video coding format, a base codec, (e.g. AVC, HEVC, or any other present or future codec) with an enhancement level of coded data, encoded using a different technique.

[0045] The general structure of the proposed encoding scheme in which the presently described techniques can be applied, uses a down-sampled source signal encoded with a base codec to form a base stream. An enhancement stream is formed using an encoded set of residuals which correct or enhance the base stream for example by increasing resolution or by increasing frame rate. There may be multiple levels of enhancement data in a hierarchical structure. A first level of correction data can be added to the decoded output of the base codec to generate a corrected picture, and then a further level of enhancement data can be added to an up-sampled version of the corrected picture.

[0046] This structure creates a plurality of degrees of freedom that allow great flexibility and adaptability to many situations, thus making the coding format suitable for many use cases including Over-The-Top (OTT) transmission, live streaming, live Ultra High Definition (UHD) broadcast, and so on. Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders and, where considered suitable, also usable as a lower resolution output.

[0047] In general, the term “residuals” refers to a difference between a value of a reference array or reference frame and an actual array or frame of data. The array may be a one or two-dimensional array that represents a coding unit or block. For example, a coding unit (block) may be a 2x2 or 4x4 set of residual values that correspond to similar sized areas of an input video frame. It should be noted that this generalised example is agnostic as to the encoding operations performed and the nature of the input signal. Reference to “residual data” as used herein refers to data derived from a set of residuals, e.g. a set of residuals themselves or an output of a set of data processing operations that are performed on the set of residuals.

[0048] Generally a set of residuals includes a plurality of residuals or residual elements, each residual or residual element corresponding to a signal element, that is, an element of the signal or original data. The signal may be an image or video. The set of residuals corresponds to an image or frame of the video, with each residual being associated with a pixel of the signal, the pixel being the signal element. The following description will describe how these residuals may be modified (i.e. processed) to impact the encoding pipeline or the eventually decoded image while reducing overall data size. Residuals or sets may be processed on a per residual element (or residual) basis, or processed on a group basis such as per tile or per coding unit where a tile or coding unit is a neighbouring subset of the set of residuals. In one case, a tile may comprise a group of smaller coding units. Note that the processing may be performed on each frame of a video or on only a set number of frames in a sequence.

[0049] In general, enhancement streams may be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Layer Units (NALUs). The NALUs are meant to encapsulate the enhancement bitstream in order to apply the enhancement to the correct base reconstructed frame. The NALU may for example contain a reference index to the NALU containing the base decoder reconstructed frame bitstream to which the enhancement has to be applied. In this way, the enhancement can be synchronised to the base stream and the frames of each bitstream combined to produce the decoded output video (i.e. the residuals of each frame of enhancement level are combined with the frame of the base decoded stream). A group of pictures may represent multiple NALUs. An example of a generalised encoding process is depicted in the block diagram of Figure 1 , where a base stream is provided along with two levels (or sub-levels) of enhancement within an enhancement stream. An input full resolution video 100 is processed to generate various encoded streams 101 , 102, 103. A first encoded stream (encoded base stream 101) is produced by feeding a base codec with a down-sampled version of the input video 100. The encoded base stream 101 may be referred to as the base layer or base level. A down-sampling operation illustrated by downsampling component 105 may be applied to the input video to produce a down-sampled video to be encoded by a base encoder 113 of a base codec. The down-sampling can be done either in both vertical and horizontal directions, or alternatively only in the horizontal direction. The base encoder 113 and a base decoder 114 may be implemented by a base codec (e.g. as different functions of a common codec).

[0050] A second encoded stream (encoded level 1 stream 102) is generally produced by processing the residuals obtained by taking the difference 110 between a reconstructed base codec video and the down-sampled version of the input video. The encoded base stream is decoded by the base decoder 114 (i.e. a decoding operation is applied to the encoded base stream to generate a decoded base stream). Decoding may be performed by a decoding function or mode of a base codec. The difference between the decoded base stream and the down-sampled input video is then created at a level 1 comparator 110 (i.e. a subtraction operation is applied to the down-sampled input video and the decoded base stream to generate a first set of residuals). The output of the comparator 110 may be referred to as a first set of residuals, e.g. a surface or frame of residual data, where a residual value is determined for each picture element at the resolution of the base encoder 113, the base decoder 114 and the output of the downsampling block 105. The difference is then encoded by a first encoder 115 (i.e. a level 1 encoder) to generate the encoded level 1 stream 102 (i.e. an encoding operation is applied to the first set of residuals to generate a first enhancement stream).

[0051] A third encoded stream (encoded level 2 stream 103) is generally produced by processing the residuals obtained by taking the difference 119 between an up- sampled version of a corrected version of the reconstructed base coded video and the input video. In particular, the second level of enhancement 103 is created by encoding a further set of residuals. The further set of residuals are generated by a level 2 comparator 119. The level 2 comparator 119 determines a difference between an up-sampled version of a decoded level 1 stream, e.g. the output of an upsampling component 117, and the input video 100. The input to the upsampling component 117 is generated by applying a first decoder (i.e. a level 1 decoder) to the output of the first encoder 115. This generates a decoded set of level 1 residuals. These are then combined with the output of the base decoder 114 at summation component 120. This effectively applies the level 1 residuals to the output of the base decoder 114. It allows for losses in the level 1 encoding and decoding process to be corrected by the level 2 residuals. The output of summation component 120 may be seen as a simulated signal that represents an output of applying level 1 processing to the encoded base stream 101 and the encoded level 1 stream 102 at a decoder. An up-sampled stream is compared to the input video which creates a further set of residuals (i.e. a difference operation is applied to the up-sampled re-created stream to generate a further set of residuals). The further set of residuals are then encoded by a second encoder 121 (i.e. a level 2 encoder) as the encoded level 2 enhancement stream (i.e. an encoding operation is then applied to the further set of residuals to generate an encoded further enhancement stream).

[0052] The first level of enhancement 102 is a corrected stream, e.g. a stream that provides a level of correction to the base encoded / decoded video signal at a lower resolution than the input video 100. The second level of enhancement 103 is a further level of enhancement that converts the corrected stream to the original input video 100, e.g. that applies a level of enhancement or correction to a signal that is reconstructed from the corrected stream. An enhancement stream encoding process may not necessarily include an up-sampling step

[0053] The output of the encoding process of Figure 1 is a base stream 101 and one or more enhancement streams 102, 103 which preferably comprise a first level of enhancement and a further level of enhancement. The three streams 101 , 102 and 103 may be combined, with or without additional information such as control headers, to generate a combined stream for the video encoding framework that represents the input video 100.

[0054] The components of Figure 1 may provide a general low complexity encoder. The enhancement streams may be generated by encoding processes that form part of the low complexity encoder and the low complexity encoder may be configured to control an independent base encoder and decoder (e.g. as packaged as a base codec). In other cases, the base encoder and decoder may be supplied as part of the low complexity encoder. In one case, the low complexity encoder of Figure 1 may be seen as a form of wrapper for the base codec, where the functionality of the base codec may be hidden from an entity implementing the low complexity encoder. The components shown in Figure 1 may operate on blocks or coding units of data, e.g. corresponding to 2x2 or 4x4 portions of a frame at a particular level of resolution. As mentioned, the described processes operate without any inter-block dependencies, hence they may be applied in parallel to multiple blocks or coding units within a frame.

[0055] A corresponding generalised decoding process is depicted in the block diagram of Figure 2, which shows a low complexity decoder that corresponds to the low complexity encoder of Figure 1 . The low complexity decoder receives the three streams 101 , 102, 103 generated by the low complexity encoder together further decoding information for example which may be contained within headers 204 or within data configuration blocks of NAL units. The encoded base stream 101 is decoded by a base decoder 210 corresponding to the base codec used in the low complexity encoder. The encoded level 1 stream 102 is received by a first decoder 211 (i.e. a level 1 decoder), which decodes a first set of residuals as encoded by the first encoder 115 of Figure 1. At a first summation component 212, the output of the base decoder 210 is combined with the decoded residuals obtained from the first decoder 211. The combined video, which may be said to be a level 1 reconstructed video signal, is up-sampled by upsampling component 213. The encoded level 2 stream 103 is received by a second decoder 214 (i.e. a level 2 decoder). The second decoder 214 decodes a second set of residuals as encoded by the second encoder 121 of Figure 1. Although the headers 204 are shown in Figure 2 as being used by the second decoder 214, they may also be used by the first decoder 211 as well as the base decoder 210. The output of the second decoder 214 is a second set of decoded residuals. These may be at a higher resolution to the first set of residuals and the input to the upsampling component 213. At a second summation component 215, the second set of residuals from the second decoder 214 are combined with the output of the upsampling component 213, i.e. an upsampled reconstructed level 1 signal, to reconstruct decoded video 250.

[0056] As per the low complexity encoder, the low complexity decoder of Figure 2 may operate in parallel on different blocks or coding units of a given frame of the video signal. Additionally, decoding by two or more of the base decoder 210, the first decoder 211 and the second decoder 214 may be performed in parallel because there are no inter-block dependencies.

[0057] In the decoding process, the decoder may parse the headers 204 (which may contain global configuration information, picture or frame configuration information, and data block configuration information) and configure the low complexity decoder based on those headers. In order to re-create the input video, the low complexity decoder may decode each of the base stream, the first enhancement stream and the further or second enhancement stream. The frames of the stream may be synchronised and then combined to derive the decoded video 250. The decoded video 250 may be a lossy or lossless reconstruction of the original input video 100 depending on the configuration of the low complexity encoder and decoder. In many cases, the decoded video 250 may be a lossy reconstruction of the original input video 100 where the losses have a reduced or minimal effect on the perception of the decoded video 250.

[0058] In each of Figures 1 and 2, the level 2 and level 1 encoding operations may include the steps of transformation, quantization and entropy encoding. Similarly, at the decoding stage, the residuals may be passed through an entropy decoder, a dequantizer and an inverse transform module. A more detailed encoding process is depicted in the block diagram of Figure 3, which illustrates these processes. Looking at Figure 3, the encoding process is split into two halves as shown by the dashed line. Below the dashed line is the base level of an encoder 300, which may usefully be implemented in hardware or software. Above the dashed line is the enhancement level, which may usefully be implemented in software. The encoder 300 may comprise only the enhancement level processes, or a combination of the base level processes and enhancement level processes as needed. This arrangement allows a legacy hardware encoder that provides the base level to be upgraded using a firmware (e.g. software) update, where the firmware is configured to provide the enhancement level. In newer devices, both the base level and the enhancement level may be provided in hardware and / or a combination of hardware and software.

[0059] The encoder topology at a general level is as follows. The encoder 300 comprises an input I for receiving an input signal 30. The input signal 30 may comprise an input video signal, where the encoder is applied on a frame-by-frame basis. The input signal 30, such as in this example a full (or highest) resolution video, is processed by the encoder 300 to generate various encoded streams. A base encoded stream is produced by substantially the same process as described with reference to Figure 1. First and second encoded streams (encoded level 1 and level 2 streams) are generated by substantially the same process as described with reference to Figure 1 .

[0060] With particular reference to Figure 3, the input I is connected to a down-sampler 305D and processing block 300-2. The down-sampler 305D may correspond to the downsampling component 105 of Figure 1 and the processing block 300-2 may correspond to the second encoder 121 of Figure 1 , The down-sampler 305D outputs to a base codec 320 at the base level of the encoder 300. The base codec 320 may implement the base encoder 113 and the base decoder 114 of Figure 1. The down-sampler 305D also outputs to processing block 300-1 . The processing block 300-1 may correspond to the first encoder 115 of Figure 1 . Processing block 300-1 passes an output to an up-sampler 305U, which in turn outputs to the processing block 300-2. The upsampler 305U may correspond to the upsampling component 117 of Figure 1. Each of the processing blocks 300-2 and 300-1 comprise one or more of the following modules: a transform block 310, a quantization block 320, an entropy encoding block 330 and a residual promotion block 350. The residual promotion block 350 occurs after the transform block 310. The order of processing may be as set out in the Figures. Additionally, the operations of Figure 3 may be applied in parallel to coding units or blocks of a frame.

[0061] As before with reference to Figure 1 , the first level of enhancement (corresponding to the first encoded stream) provides a corrected video at a base level, for example to correct for encoder and / or decoder artefacts. The second level of enhancement (corresponding to the second encoded stream) is a further level of enhancement that converts the corrected video to the original input video or a close approximation thereto (e.g. to add detail or sharpness). For example, the second level of enhancement may add fine detail that is lost during the downsampling and / or help correct from errors that are introduced by one or more of the transform operation 310-1 and the quantization operation 320-1.

[0062] Further details regarding how the first and second encoded steams are produced in Figure 3 will now be discussed.

[0063] The first encoded stream is produced by block 300-1 by processing a first set of residuals obtained by taking the difference between the base reconstruction signal and the down-sampled version of the input signal. As mentioned previously, block 300-1 comprises a transform block 310-1 , a quantization block 320-1 and an entropy encoding block 330-1 .

[0064] Processing the first set of residuals comprises transforming, quantizing and entropy encoding the set of residuals to produce the encoded level 1 stream. A transform operation 310-1 is applied to the first set of residuals. The transform may transform the residual information to four surfaces. For example, the transform may produce the following components: average, vertical, horizontal and diagonal. The components that are output by the transform may be taken as the coefficients to be quantized. A quantization operation 320-1 is applied to the transformed set of residuals to generate a set of quantized residuals, and an entropy encoding operation 330-1 is applied to the quantized set of residuals to generate the encoded level 1 stream at the first level of enhancement. Entropy encoding may not be used.

[0065] The second encoded steam is produced by block 300-2 by processing a second set of residuals obtained by taking the difference between the up-sampled version of a corrected version of the decoded base stream (the reference signal or frame), and the input signal 30 (the desired signal or frame). Block 300-2 comprises a transform block 310-2, a quantization block 320-2, an entropy encoding block 330- 2 and a residual processing block 350-2.

[0066] Processing the second set of residuals is substantially the same process as processing the first set of residuals, described above. Processing the second set of residuals comprises performing a transform operation 310-2 on the second set of residuals to generate a further transformed set of residuals). The transformed residuals are then quantized and optionally entropy encoded in the manner described above in relation to the first set of residuals (i.e. a quantization operation 320-2 is applied to the transformed set of residuals to generate a further set of quantized residuals, and an entropy encoding operation 320-2 is be applied to the quantized further set of residuals to generate the encoded level 2 stream containing the further level of enhancement information). Similar to block 300-1 , a residual promotion operation 350-2 is performed before the quantization operation and acts to process residuals prior to the quantization and entropy encoding operations of this block. Optionally, the residual promotion operation could take place as part of the quantization operation but is schematically shown as separate for ease of illustration.

[0067] As we have seen, after applying a transform to a set of residuals to create a first set of coefficients, the quantization operation is applied to the first set of coefficients to create a first set of quantized coefficients. The encoding operation is applied to the first set of quantized coefficients. As has already been mentioned, the quantization operation turns the residuals into quanta, so that certain variables can assume only certain discrete magnitudes. Effectively, quantization maps the transformed residuals (coefficients), which form a large, often continuous set, to a smaller set with a finite number of elements, called quantization bins. Each bin has a certain size known as a step-width.

[0068] Figure 4 provides an example of how quantization of residuals and / or coefficients (transformed residuals) may be performed based on bins having a defined step width. The x-axis of Figure 4 represents residual or coefficient values. In this example a number of bins are defined with a step-width of 5. The step-width may be understood as the quantization step size, as shown in the drawings. The size of the step-width may be selectable, e.g. based on a parameter value. In certain cases, the size of the step-width may be set dynamically, e.g. based on the rate control examples described above. In Figure 4, the step-width results in bins corresponding to residual values in the ranges of 0-4, 5-9, 10-14, 15-19 (i.e. 0 to 4 including both 0 and 4). Bin widths may be configured to include or exclude end points as required. In this example, quantization is performed by replacing all coefficient values that fall into the bin with an integer value (e.g. residual values of between 0 and 4 inclusive have a quantized value of 1). In Figure 4, quantization may be performed by dividing by the step-width (e.g. 5), taking the floor of the result (i.e. the nearest integer less than a decimal for positive values) and then adding one. Negative values may be treated in a similar way, e.g. by working on absolute values then converting to negative values following calculation (e.g. abs(- 9) = 9, 9 / 5=1.8, floor(1.8) = 1 , 1 + 1 =2, 2*-1 = -2). Figure 4 shows a case of linear quantization where all bins have a common step-width. It should be noted that various different implementations based on this approach may be enacted, for example, a first bin may have a quantized value of 0 instead of 1 , or may comprise values from 1 to 5 inclusive. Section 8.5 of a standard specification for LCEVC provided in the Text of ISO / IEC 23094-2 Ed 1 Low Complexity Enhancement Video Coding published in November 2021 provides further details regarding a decoding process for dequantization.

[0069] Figure 5 shows how a so-called “deadzone” (DZ) may be implemented. A deadzone is an area of a spectrum in which no values are quantized. This deadzone may correspond to a distance from the threshold or may be a multiplier (e.g. of a step-width). The deadzone may be a region around the zero output value of a quantizer, that is a band containing a zero signal and having a size that may be the same as or different from the step-width. Thus, for this band of inputs that are close to zero, the signal may effectively be attenuated so that low-level signals, which may typically correspond to noise in visual data, are not allocated data unnecessarily. In Figure 5, residuals or coefficients with a value within a pre-defined range are set to 0. In Figure 5 the pre-defined range is a range around a value of 0. In Figure 5, values that are less than 6 and greater than -6 are set to 0. The deadzone may be set as a fixed range (e.g. -6 to 6) or may be set based on the step-width. In one case, the deadzone may be set as a predefined multiple of the step-width, e.g. as a linear function of a step-width value. In other case, the deadzone may be set as a non-linear function of a step-width value. In some cases, the deadzone is set based on a dynamic step-width, e.g. may be adaptive, and so the deadzone may change as the step-width changes. In some cases, the multiplier may also be adaptive, e.g. based on operating conditions such as available bit rates.

[0070] Generally, having a deadzone may help reduce an amount of data to be transmitted over a network, e.g. help reduce a bit rate. When using a deadzone, residual or coefficient values that fall into the deadzone are effectively ignored. The deadzone need only be enacted at the encoder, the decoder simply receives a quantized value of 0 for any residual or coefficient that falls within the deadzone. Residual or coefficient values that fall within the deadzone therefore do not get encoded and do not form part of an encoded stream.

[0071] The sets of residuals as described herein may be seen as sparse data, e.g. in many cases there is no difference for a given pixel or area and the resultant residual value is zero. Residuals may be treated as a two-dimensional image in themselves, e.g. a delta image of differences. The sparsity of the data may be seen to relate features like “dots”, small “lines”, “edges”, “comers”, etc. that are visible in the residual images. These features are typically not fully correlated (e.g. in space and / or in time). They have characteristics that differ from the characteristics of the image data they are derived from (e.g. pixel characteristics of the original video signal). The temporal characteristics of residuals, as well as spatial characteristics, are important to consider. For example, in residual images details like “edges” and “dots” that may be observed in residual “images” show little temporal correlation. This is because “edges” in residual images often don’t translate or rotate like edges as perceived in a normal video stream. For example, within residual images, “edges” may actually change shape over time, e.g. a head turning may be captured within multiple residual image “edges” but may not move in a standard manner (as the “edge” reflects complex differences that depend on factors such as lighting, scale factors, encoding factors etc.). These temporal aspects of residual images, e.g. residual “video” comprising sequential residual “frames” or “pictures” typically differ from the temporal aspects of conventional images, e.g. normal video frames (e.g. in the Y, U or V planes). These temporal characteristics of residuals can be used as a basis for residual processing such that it may be possible to discard residual information that has little effect on a perception of a decoded video signal as well as enhance residual information that has a greater effect of a perception of a decoded video signal.

[0072] The process of enhancing residual information that has a greater effect of a perception of a decoded video signal is called residual promotion. As was briefly introduced above, the processing blocks 300-1 and 300-2 comprise residual promotion operations. Briefly, the residual promotion operation effectively rescues residuals that would not otherwise be encoded into the first enhancement stream (or correction stream). In this way, certain residuals are passed for encoding that would otherwise have not been encoded. In other words, residual information that would have been quantized to 0 (i.e. residuals or coefficients that fall within the deadzone) can be enhanced, through the residual promotion operation, such that they survive quantization (i.e. have a quantization value greater than 0) and form part of the encoded stream. The residual information is enhanced by modifying the transform coefficient so that the residual passes through quantization. In other words, residual promotion can be thought of as rescuing or promoting residual or coefficient values that fall into the deadzone so that they are no longer ignored and instead form part of the data to be sent. Specific functionality of the residual promotion blocks 350-1 , 350-2 is described in detail below however, conceptually, these blocks 350-1 , 350-2 function to modify the residuals, specifically the transformed residuals i.e. the transform coefficients. In effect, the residual processing blocks are configured to ‘promote’ one or more transformed residuals prior to quantization such that quantization operates on a modified form of the transform coefficients.

[0073] Generally, residual promotion involves, after applying a transform operation to the residuals, applying the quantization operation to the set of transform coefficients to generate a set of quantized values, wherein applying the quantization operation comprises modifying a quantized value from a first quantized value to a second quantized value in order to adjust a quantization bin of a transform coefficient, such that the corresponding transform coefficient is quantized into second quantization bin instead of a first quantization bin. This means that after the quantization operation is carried out, the resulting quantized value is adjusted from a first quantized value (i.e. the quantized value the transformed coefficient would have originally had) to a second quantized value (i.e. the quantized value the transform coefficient has after modification), wherein a quantization value corresponds to a quantization bin.

[0074] The residual promotion operation is a way of allowing values in a first quantization bin (referred to herein as an N-bin) that are near to an adjacent quantization bin (referred to herein as an (N+1 )-bin) to be promoted into the (N+1 )-bin. Although this can be thought of as effectively just shrinking the overall width of the N-bin and expanding the (N+1 )-bin, the actual sizes of the bin widths themselves do not change. Instead, more coefficient values are put into the (N+1 )-bin, from the N- bin. Thus, the bins themselves are unaffected by this function and so the actual step-width of the bins does not change. Since the bin sizes do not change at all, the decoder is entirely unaware of this functionality.

[0075] Since the decoder does not know about the residual promotion operation, the decoding process remains unchanged. Thus the decoding process is substantially the same as described in relation to Figure 2. Since some of the transform coefficients were quantized to different bins that they otherwise would have, the final reconstructed image will have some pixels that have been changed compared to the original. For example, if the residuals are implemented in the luna plane, and some of the corresponding transform coefficients have been promoted, the final reconstructed image will have one or more pixels that are lighter or darker than in the original.

[0076] The main advantage of residual promotion is that it allows stronger than normal coefficients to be signalled if desired. The N-bin corresponds to the deadzone (having a quantized value of zero), and so the coefficients that would have been quantized to the deadzone would instead be promoted to the (N+ 1 )-bin (having a quantized value of 1) and as such would now form part of the encoded signal to be transmitted. Details of the residual promotion operation will now be described.

[0077] The residual promotion operation comprises receiving a value which may be referred to as PromotionFactor, which can be set by a user. Based on this parameter two values, Promotionwidth and Promotionoffset are calculated, as follows:

[0078] Promotionwidth = abs(Deadzone) * PromotionFactor (Eqn 1)

[0079] PromotionOffset = step-width + abs(Deadzone) - Promotionwidth (Eqn 2)

[0080] Promotionwidth effectively corresponds to a fraction of the size of the deadzone. When PromotionFactor is equal to 1 , the fraction is 100% of the size of the deadzone, and so Promotionwidth is equal to the size of the deadzone.

[0081] PromotionOffset effectively corresponds to step-width plus the difference between the original width of the deadzone (N-bin) and the fractional width of the deadzone (Promotionwidth).

[0082] Without residual promotion the quantized value of the transform coefficient is calculated as: valueSign = coefficient > 0 ? 1 : -1 (Eqn 3)

[0083] QuantizedValue = valueSign * (a bs (coefficient) + Deadzone) / step-width (Eqn 4) With residual promotion, the quantized value is instead calculated using:

[0084] QuantizedValue = valueSign * [(abs(coefficient) + Deadzone) I step-width] + Promo (Eqn 5)

[0085] With reference to Equation 3, the notation “X = Y ? 1 : -1” should be read as: If the condition Y evaluates to true then X takes the value 1 , but if the condition Y evaluates to false then X takes the value 0. Thus, in Equation 3, if the value of the coefficient is greater than 0 then the valuSign is 1 , and if the value of the coefficient is less than 0 then the valuSign is -1 .

[0086] With reference to Equation 5, Promo is effectively a modification factor that is used to adjust the quantized value, so that the transform coefficient is quantized into a second quantization bin rather than a first quantization bin, and the coefficient is quantized into a different bin (second quantization bin) that it otherwise would have been (first quantization bin).

[0087] In order to calculate Promo, it is first necessary to calculate a modified transform coefficient value, which comprises modifying the transform coefficient by the PromotionOffset:

[0088] Promotionvalue = abs(coefficient) - PromotionOffset (Eqn 6)

[0089] In this way, the transform coefficient has been modified from a first coefficient value (its initial value) to a second coefficient value (Promotionvalue).

[0090] Promo is then calculated as follows:

[0091] Promo = ((Promotionvalue > 0) && (Promotionvalue < Promotionwidth)) ? 1 : 0

[0092] As before, in the above equation the notation “X = Y ? 1 : 0” should be read as: If the condition Y evaluates to true then X takes the value 1 , but if the condition Y evaluates to false then X takes the value 0.

[0093] In other words, with reference to the above equation, if Promotionvalue is greater than 0 and less than Promotionwidth, Promo has a value of 1 . The modified value of the transform coefficient (the second coefficient value) must be both greater than 0 and less than the fractional width of the deadzone, for Promo to be 1 . In all other cases Promo has a value of 0.

[0094] Looking back at the formula for calculating QuantizedValue, it can be seen that when Promo is equal to 1 , the QuantizedValue computed during residual promotion is different to the QuantizedValue computed without residual promotion. Thus, the QuantizedValue, and therefore the quantized bin, is different when the modification factor (Promo) is non-zero and so Promo has the effect of adjusting the quantization bin of the transform coefficient.

[0095] As can be seen, residual promotion can be used to modify the quantization value of a transform coefficient such that the transform coefficient survives quantization when it would otherwise have been quantized to zero.

[0096] The residual promotion operation can be thought of as comprising two parts: a promotion part and a determination part. The promotion part is given by Equation 5 above, and involves modifying the quantized value of the transform coefficient by the value Promo. The determination part comprises the two conditional determinations in the Promo calculation, which effectively determines whether the transform coefficient is promotable. The first condition that is evaluated during preanalysis (i.e. Promotionvalue > 0) effectively ensures that only transform coefficients that are close to the boundary between the N-bin and the (N+1) bin are promoted. The second condition that is evaluated during determination (i.e. Promotionvalue < Promotionwidth) effectively ensures that transform coefficients are only promoted from the deadzone to the next bin (the (N+ 1 )-bin) rather than being promoted several bins higher (e.g. (N+2)-bin or higher). That is, the transform coefficient is only promoted by 1 to the next bin and is not promoted by more than 1 bin.

[0097] The purpose of Equation 3, to calculate valueSign is to ensure that the promotion is applied in the correct direction. In other words, positive coefficients should be promoted in the positive direction from N-bin to (N+1) bin and negative coefficients should be promoted in the negative direction from N-bin to (N-1 )-bin. Not all residual or coefficient values that would have been ignored (e.g. because they fall into the deadzone) need to be recovered. Generally, the aim of residual promotion is to prioritise (through promotion) slow moving objects and edges where, visually-wise, the visual improvement that LCEVC can bring is higher. Thus, only residual information that has a greater effect on how a decoded video signal is perceived needs to be promoted. As such, the residual promotion operation does not need to be applied to all residuals or coefficient values that fall into the deadzone. Instead, the residual or transform coefficients are only promoted if they are considered visually important. Here, visually important residuals or coefficients are those that provide noticeable improvements in the perception of a decoded video signal.

[0098] The term “promotion" means that the transform coefficients have to survive the quantization stage regardless of what their magnitude is, and before the promotion is applied, it must be decided which transform blocks to apply the promotion to.

[0099] A step of identifying the residuals or transform coefficients that need to be promoted is preferably carried out before the residual promotion operation is applied. Briefly, the identifying step comprises determining that a residual is important and should form part of the encoded stream and identifying the corresponding transform coefficient to be modified. Several different approaches can be taken to make this decision.

[0100] In some implementations, to identify the appropriate residuals to promote, the set of residuals may be analysed and characteristics or patterns may be identified. Alternatively, the process may analyse the original input signal corresponding to that set of residuals. Further, the process may predict an effect on a reconstructed image by that set of residuals (or the set as modified). The prediction may include reconstructing the image by combining the residuals with the signal from a lower level, analysing the reconstructed signal and processing the residuals accordingly or iteratively. Since the magnitude of the transform coefficient is known, as well as the nature of the associated transform block, the promoting decision can preferably be a combination of these conditions, some of which are discussed below.

[0101] In a first decision process, a decision can be taken to favour a transform block in an intra frame in order to improve the visual quality of the intra reference frames. In this situation, if the frame is Intra and the transform block is classified as static, the transform coefficients in that block are promoted using the residual promotion operation.

[0102] In a second decision process, a decision can be taken to favour a moving transform block in an inter frame. For example, if a temporal type for a current transform block is intra and the frame is inter, the transform coefficients in that block are promoted.

[0103] In a third decision process, the edges in the original image can be favoured. In this case, pre-analysis can be carried out on the residuals to extract various features of the residuals. During the pre-analysis, four block types may be generated (each generally consisting of 8x8 pixels): CoarseTexture, SmoothTexture, Plain, and Edge. If a transfer block belongs to an Edge type, the transform coefficients in that block are promoted. In some situations, the feature extraction can generate several false positive edges and this generally happens if the classification block size is too big (the area is too large and it can go beyond the edge) or if it is too small (noise is classified as edge even in plain area). A combination of analysis of the resulting type at a different block size can help to reduce the false positives. For example, if block is classified as an Edge type but the none of the neighbouring blocks are Edge type, then the block in question should not be classified as an Edge.

[0104] The different promotion decisions can be summarised as follows:

[0105] 1 . Favour a static transform block: a. If the frame is intra and the transform block is static: promote the residuals in the block. 2. Favour a moving transform block: a. If the transform block is intra and the frame is inter: promote the residuals in the block.

[0106] 3. Favour edges: a. If a transform block is an Edge type block, and preferably at least one neighbouring block is also an Edge type: promote the residuals in the block.

[0107] In some examples, the importance of a residual can be determined using a perception metric. A perception metric for one or more residuals can be compared to a set of ranges, which may indicate how important the residual is. The perception metric in one case may be determined per block or for a tile (an nxm unit of blocks). In some examples, the perception metric may be determined based on at least luma (e.g. Y) picture elements. The perception metric may be determined for picture elements that correspond to a given set of residuals (e.g. where the picture elements and residuals relate to the same spatial indexes in an image matrix). The perception metric may be based on one or more of texture and contrast. The perception metric may be used to indicate which action should be taken for that residual e.g. promote or not promote. The perception metric can be used as a form of ranking, with important residuals have a corresponding perception metric at one end of the ranking (e.g. having high perception metric values) and less important residuals have a corresponding perception metric at the other end of the ranking (e.g. having low perception metric values). In this way, important residuals can be easily identified.

[0108] In some implementations, a residual mode can be used to indicate how the residuals should be processed. A residual mode can include a mode corresponding to “residual promotion” and a mode corresponding to “no residual promotion”. The residual mode can be selected using a residual mode selection block, which can be implemented as part of the encoder. If a residual mode indicates that residual values are promoted prior to encoding within one or more enhancement levels, the output of the transformation process (i.e. the transformed coefficients) are modified in accordance with the residual promotion operation described above and the result of the residual promotion (i.e a modified set of transformed residuals) is then quantized and optionally entropy encoded to produce the encoded level 1 or level 2 streams. If a residual mode is selected which indicates that no residual promotion should be done, then residual values may be passed through the transformation component, without modification, for quantization and entropy encoding.

[0109] The residual promotion block 350 may be activated or configured by the residual mode selection block. For example, if a residual mode is selected (e.g. turned on), then the residual processing block 350 may be activated. The residual mode may be selected independently for the first and second enhancement streams (e.g. residual processing blocks 350-2 and 350-1 may be activated and applied separately where one may be off while another is on).

[0110] Examples of residual modes that may be implemented include, but are not limited to a mode where no residual processing is performed, a promotion mode whereby certain transformed residuals are modified by a factor, a control mode whereby certain blocks are not to be processed (e.g. equivalent to setting all residual values in a 2x2 or 4x4 coding unit to 0), a ranking or priority mode whereby residuals are ranked or given a priority within a list to identify residuals that are important and that should be selected for promotion based on the rank or priority, and a categorization mode whereby residuals and / or picture elements are categorised and corresponding residuals are modified (e.g. promoted) based on the categorization. The residual mode may be selected at a block or tile level. In this case, residual promotion may be selected for all residual values corresponding to a particular coding unit or for a particular group of coding units. Being able to select a residual mode at a block or tile level enhances the flexibility of the encoding scheme.

[0111] The residual promotion operation and the calculations carried out as part of this operation are done at the coefficient level in the transform domain. In a 2x2 block, it may be the case that only the A component of the AHVD component has a coefficient value that should be promoted. Deciding whether or not to promote a 2x2 transform block depends on the results of the analysis of the source image. Generally, when using an interlaced input, residual promotion should be used on residual coefficients belonging to lines or edges because it is these areas where the effect of the interlacing operation is more evident.

[0112] The above described methods of residual processing are applied at the encoder but not at the decoder. This therefore represents a form of asymmetrical encoding that may take into account increased resources at the encoder to improve communication.

[0113] At both the encoder and decoder, for example implemented in a streaming server or client device or client device decoding from a data store, methods and processes described herein can be embodied as code (e.g., software code) and / or data. The encoder and decoder may be implemented in hardware or software as is well-known in the art of data compression. For example, hardware acceleration using a specifically programmed Graphical Processing Unit (GPU) ora specifically designed Field Programmable Gate Array (FPGA) may provide certain efficiencies. For completeness, such code and data can be stored on one or more computer-readable media, which may include any device or medium that can store code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on a computer-readable medium, the computer system performs the methods and processes embodied as data structures and code stored within the computer-readable storage medium. In certain embodiments, one or more of the steps of the methods and processes described herein can be performed by a processor (e.g., a processor of a computer system or data storage system).

[0114] Generally, any of the functionality described in this text or illustrated in the figures can be implemented using software, firmware (e.g., fixed logic circuitry), programmable or nonprogrammable hardware, or a combination of these implementations. The terms “component” or “function” as used herein generally represents software, firmware, hardware or a combination of these. For instance, in the case of a software implementation, the terms “component” or “function” may refer to program code that performs specified tasks when executed on a processing device or devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

Claims

CLAIMS1 . A method of encoding an input signal, the method comprising: receiving an input signal; generating a set of residuals based on a difference between the input signal and a reconstructed signal; applying a transform operation to the set of residuals to generate a set of transform coefficients; applying a quantization operation to the set of transform coefficients to generate a set of quantized values; and applying an encoding operation to the set of quantized values to generate an encoded stream; wherein applying the quantization operation comprises modifying a quantized value from a first quantized value to a second quantized value in order to adjust a quantization bin of a transform coefficient, such that the transform coefficient is quantized into second quantization bin instead of a first quantization bin.

2. The method of claim 1 wherein applying the quantization operation comprises: determining whether a transform coefficient would be quantized to zero; and in response to a positive determination, modifying the quantized value.

3. The method of claim 2 further comprising: in response to a positive determination: determining whether the transform coefficient is to be promoted, wherein a transform coefficient is determined to be promoted if said transform coefficient should be quantized into a second quantitation bin instead of a first quantitation bin; in response to a positive determination that the transform coefficient is to be promoted, modifying the quantized value of said transform coefficient.

4. The method of claim 2 or 3 wherein the modifying comprises: calculating a modification factor, wherein the modification factor is used to adjust the quantization bin.

5. The method of claim 4 wherein calculating the modification factor comprises calculating a modified transform coefficient.

6. The method of claim 5 wherein calculating the modification factor comprises determining whether the modified transform coefficient is greater than zero.

7. The method of claim 5 or claim 6 wherein calculating the modification factor comprises determining whether the modified transform coefficient is less than a threshold value, wherein the threshold value is a fraction of a width of a quantization bin.

8. The method of claim 7 wherein the threshold value is a fraction of a width of a quantization bin, and the quantization bin is a deadzone.

9. The method of any of claims 6 to 8 wherein the modification factor is calculated to be zero if the modified transform coefficient is less than or equal to zero.

10. The method of any of claims 7 to 9 wherein the modification factor is calculated to be zero if the modified transform coefficient is greater than the threshold value.

11. The method of any of claims 4 to 10 wherein the quantization bin is adjusted from the first quantization bin to the second quantization bin when the modification factor is calculated to be non-zero.

12. The method of claim 3 wherein the determining comprises determining that a residual is important and should form part of the encoded stream and identifying the corresponding transform coefficient.

13. The method of claim 12 wherein applying the quantization operation comprises modifying all the quantized values of all the transform coefficients of a block comprising the identified transform coefficient.

14. The method of claim 12 or claim 13 wherein applying the quantization operation comprises modifying all the quantized values of all the transform coefficients of a tile comprising the identified transform coefficient.

15. The method of any of claims 12 to 14 wherein the residual is determined to be important if a frame of the input signal is an intra frame and a block comprising the residual is static.

16. The method of any of claims 12 to 14 wherein the residual is determined to be important if a frame of the input signal in an inter frame and a temporal type of a block comprising the residual is intra.

17. The method of any of claims 12 to 14 wherein the residual is determined to be important if a block comprising the residual belongs to an edge-type block, preferably wherein at least one neighbouring block of said block is also an edgetype block.

18. The method of any preceding claim wherein the second quantization bin is adjacent to the first quantization bin.

19. An encoder configured to perform the method of any of claims 1 to 18.

Images