Residual Filtering in Signal Enhancement Coding

By using a filter or a mask to filter the decoded residuals during encoding and decoding, the problem that image quality is affected by block effects and artifacts in the prior art is solved, and high-quality image reconstruction is achieved.

CN113994689BActive Publication Date: 2024-12-06V NOVA INT LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080036648.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-04-15
Filing Date
2020-02-28
Publication Date
2024-12-06
Estimated Expiration
2040-02-28

AI Technical Summary

Technical Problem

Existing hybrid backward compatible encoding technologies are difficult to effectively reduce block effects and artifacts when improving image quality, especially in low-resolution applications, resulting in poor picture quality.

Method used

Decoded residuals are filtered by applying a filter or mask during encoding and decoding, especially weighting at block boundaries, to reduce block effects and improve picture quality.

Benefits of technology

It effectively reduces block effects and illusions, improves picture quality, ensures high-quality output of image reconstruction signals, and reduces the impact of the blocking effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113994689B_ABST
    Figure CN113994689B_ABST
Patent Text Reader

Abstract

According to an aspect of the present invention, a method for encoding an input signal is provided, the method comprising: generating a base coded signal by feeding a downsampled version of the input signal to an encoder; generating a first residual signal by the following operations: obtaining a decoded version of the base coded signal; and using the difference between the decoded version of the base coded signal and the downsampled version of the input signal to generate a first residual signal; encoding the first residual signal to generate a first coded residual signal; generating a second residual signal by the following operations: decoding the first coded residual signal to generate a first decoded residual signal; A first decoded residual signal is filtered to produce a filtered version of the first decoded residual signal; the decoded version of the base coded signal is corrected using the filtered version of the first decoded residual signal to produce a corrected decoded version; the corrected decoded version is upsampled; and the difference between the corrected decoded signal and the input signal is used to produce the second residual signal; and the second residual signal is encoded to produce a second encoded residual signal, wherein the base coded signal, the first encoded residual signal, and the second encoded residual signal include an encoding of the input signal. An encoder, a decoder, and a computer-readable medium may also be provided.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Hybrid backward-compatible encoding techniques have been previously proposed in, for example, WO 2014 / 170819 and WO 2018 / 046940, the contents of which are incorporated herein by reference.

[0002] The encoding technique is a flexible, adaptable, efficient, and computationally inexpensive encoding format that combines a video coding format, a base codec (eg, AVC, HEVC, or any other current or future codec) with encoded data of enhancement layers encoded using different techniques.

[0003] The technique uses a downsampled source signal that is encoded using a base codec to form a base stream. An enhancement stream is formed from a set of encoded residuals that correct or enhance the base stream, for example by increasing the resolution or by increasing the frame rate. There may be multiple levels of enhancement data in the hierarchical structure. It is noteworthy that the base stream is generally expected to be decodable by a hardware decoder, while the enhancement stream is expected to be suitable for software processing implementation with suitable power consumption.

[0004] It is desirable to apply hybrid backward-compatible coding techniques in a way that produces a high quality reconstructed signal with minimal image artifacts. Summary of the invention

[0005] According to aspects of the present invention, there is provided a method for encoding and decoding a signal. The method comprises filtering the decoded residual by applying a filter or mask to improve picture quality and reduce blockiness. Preferably, the filter or mask is a deblocking filter, the deblocking filter comprising weighting the boundaries of the blocks of the inverse transformed residual on a specific path.

[0006] There is provided a method, a computer program, a computer readable medium and an encoder as set out in the accompanying claims.

[0007] According to a first aspect of the present invention, a method for encoding an input signal is provided, the method comprising: generating a base coded signal by feeding a downsampled version of the input signal to an encoder; generating a first residual signal by the following operations: obtaining a decoded version of the base coded signal; and using the difference between the decoded version of the base coded signal and the downsampled version of the input signal to generate a first residual signal; encoding the first residual signal to generate a first coded residual signal; generating a second residual signal by the following operations: decoding the first coded residual signal to generate a first decoded residual signal; The invention relates to a method for filtering a first decoded residual signal to produce a filtered version of the first decoded residual signal; correcting the decoded version of the base coded signal using the filtered version of the first decoded residual signal to produce a corrected decoded version; upsampling the corrected decoded version; and using a difference between the corrected decoded signal and the input signal to produce the second residual signal; and encoding the second residual signal to produce a second encoded residual signal, wherein the base coded signal, the first encoded residual signal, and the second encoded residual signal comprise an encoding of the input signal.

[0008] Filtering is applied to improve the picture quality of the residual and reduce occlusion effects. These may sometimes occur when a block-based transform is applied at a low resolution. It should be noted that the filter is applied to the decoded version of the encoded residual set rather than the encoded or reconstructed picture signal. Since the residual is used later in the architecture, any artifacts, blocking effects or visible block structure may have an aggravating effect on the overall picture quality because the residual has a conversion effect on the base signal, and any changes caused by the combination of the residual and the base signal propagate through the architecture and are unlikely to be compensated in the output signal. Preferably, the filter operates on each block of the transformed (or inverse transformed) residual by applying a mask to each block.

[0009] The step of generating a first residual signal may include quantizing the first residual signal to generate a first quantized residual signal, and the step of decoding the first encoded residual signal includes dequantizing the first quantized residual signal. Thus, the mask may operate on quantized information to improve the quality of the information.

[0010] In an embodiment, the first residual signal may include a plurality of N×N data blocks, and wherein encoding the first residual signal may include transforming the first residual signal on a block-by-block basis, wherein decoding the first encoded residual signal may include performing an inverse transform on N×N data blocks of a signal derived from the first encoded residual signal to produce the first decoded residual signal, and wherein filtering the first decoded residual signal may include applying an N×N mask to each data block of the first decoded residual signal. Thus, effects introduced by the transform may be mitigated by the mask. Typically, filters are applied to reduce occlusion effects in a low-resolution signal that become apparent by a subsequent upsampling step that propagates artifacts through the pipeline and into a high-resolution signal.

[0011] Optionally, the step of transforming may comprise applying a directional decomposition transform to the first residual signal to produce a directional component corresponding to the first residual signal. The filter may thus compensate for the effects produced by the directional decomposition transform, specifically at the edges of blocks. N may be 4 and the transform kernel used for the transform may be a 4×4 kernel.

[0012] The mask may be a weighted mask and each coefficient of the mask may be a weight between 0 and 1. Thus, the coefficients may be variable in order to adaptively adjust the boundary effects of the residual.

[0013] Preferably, the mask comprises a set of coefficients around the edge of the mask. More preferably, the mask comprises first coefficients for the sides of the mask and second coefficients for the corners of the mask. Thus, the coefficients may be defined separately for the sides and corners in order to adjust the residuals differently and compensate for blocking artifacts differently while reducing complexity.

[0014] In a specific embodiment, N may be 4 and the mask may be:

[0015]

[0016] where α and β are variable coefficients and where 0≤α≤1 and 0≤β≤1. In this way, the center of the block is not affected, but the effect of the residual at the border of the block is reduced.

[0017] Filtering the first decoded residual signal may include applying an in-loop filter.

[0018] The method may further comprise: signalling coefficients of the mask in configuration metadata accompanying the first or second residual signal or both.The encoder may thus be able to signal the coefficients used to the decoder so that any coefficients applied during the encoding process may be similarly applied at the decoding side.

[0019] An encoder configured to implement the method of any of the above aspects or embodiments may also be provided.

[0020] According to another aspect of the present invention, a method for decoding an input signal from a plurality of encoded signals is provided, the method comprising: receiving a base encoded signal and indicating decoding of the base encoded signal to produce a base decoded signal; receiving a first encoded residual signal and decoding the first encoded residual signal to produce a first decoded residual signal; filtering the first decoded residual signal to produce a filtered version of the first decoded residual signal; correcting the base decoded signal using the filtered version of the first decoded residual signal to produce a corrected version of the base decoded signal; upsampling the corrected version of the base decoded signal to produce an upsampled signal; receiving a second encoded residual signal and decoding the second encoded residual signal to produce a second decoded residual signal; and combining the upsampled signal with the second decoded residual signal to produce a reconstructed version of the input signal. Picture quality can therefore be improved by applying a filter on the level 1 path of the architecture. Thus, a mask is applied to the first residual signal, which is subsequently used to generate an output from the second residual signal, such that effects introduced during the process of decoding the residual signal are not exacerbated.

[0021] Preferably, the step of decoding the first encoded residual signal may comprise applying an inverse quantization operation to the first encoded residual signal. More preferably, the step of decoding the first encoded residual signal comprises applying an inverse transform, the inverse transform being applied on a block-by-block basis, and wherein filtering the first decoded residual signal comprises applying an N×N mask to each block of the inverse transformed signal, wherein N×N is the size of a transform kernel used in the inverse transform. Thus, the mask may be able to compensate for effects introduced during a quantization or transform process.

[0022] The step of inverse transforming may include applying an inverse directional decomposition transform to the directional component of the first encoded residual signal to obtain residual values ​​corresponding to the first residual signal. N may be 4 and the transform kernel may be a 4×4 kernel. The mask may be a weighted mask and each coefficient of the mask may be a weight between 0 and 1. The mask may include a set of coefficients around the edge of the mask. The mask may include first coefficients for the sides of the mask and second coefficients for the corners of the mask. The mask may be:

[0023]

[0024] Where α and β are variable coefficients and where 0≤α≤1 and 0≤β≤1.

[0025] The method may further comprise: retrieving coefficients of the mask from configuration metadata.In this way, appropriate coefficients may be set depending on the effect that needs to be compensated and may be variable.

[0026] The method may further comprise retrieving the coefficients from a stream header.Thus, the coefficients may be efficiently exchanged between an encoder and a decoder and be variable according to a specific use, such that customized adjustments are achieved in an efficient manner.

[0027] The method may further comprise identifying a transform applied to the first residual signal, and applying the mask only if the transform is a transform of a predetermined size.Thus, steps may be applied only when they will have a suitable effect, thus reducing process complexity.

[0028] A decoder configured to implement the method of any of the above aspects or embodiments may also be provided.

[0029] According to a further aspect of the present invention, a computer readable medium may be provided, which, when executed by a processor, causes the processor to perform any one of the methods of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0031] Figure 1 is a block diagram illustrating the encoding process;

[0032] Figure 2 is shown corresponding to Figure 1 Block diagram of the decoding process of the encoding process;

[0033] Figure 3 is shown in more detail Figure 1 Block diagram of the encoding process;

[0034] Figure 4 is shown in more detail Figure 2 Block diagram of the decoding process;

[0035] Figure 5 is a flow chart showing an encoding method; and

[0036] Figure 6 is a flowchart illustrating an encoding method. DETAILED DESCRIPTION

[0037] The present disclosure describes a hybrid backward compatible coding technique. This technique is a flexible, adaptable, efficient and computationally inexpensive coding format that combines coded data of different video coding formats, base codecs (i.e., encoder-decoders) (e.g., AVC / H.264, HEVC / H.265, or any other current or future codecs, as well as non-standard algorithms such as VP9, ​​AV1, etc.) with at least two enhancement layers.

[0038] The general structure of the coding scheme uses a downsampled source signal encoded with a base codec, adds a first level of correction or enhancement data to the decoded output of the base codec to produce a corrected picture, and then adds another level of correction or enhancement data to an upsampled version of the corrected picture.

[0039] Thus, the streams are considered to be a base stream and one or more enhancement streams, where there are typically two enhancement streams. It is worth noting that typically the base stream can be decoded by a hardware decoder, while the enhancement streams can be suitable for software processing implementation with suitable power consumption.

[0040] This structure creates multiple degrees of freedom, which allows great flexibility and adaptability in many scenarios, making the encoding format suitable for many use cases, including over-the-top (OTT) delivery, live streaming, live ultra-high-definition (UHD) broadcasting, etc. It also provides low-complexity video encoding.

[0041] Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders and also usable as a lower resolution output if deemed appropriate.

[0042] The codec format uses a minimum number of relatively simple coding tools that, when combined synergistically, can provide visual quality improvements over full-resolution pictures encoded with the base codec, while allowing flexibility in the ways in which they can be used.

[0043] The method and apparatus are based on an overall algorithm that is built over existing encoding and / or decoding algorithms (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, and non-standard algorithms such as VP9, ​​AV1, etc.) that are used as a baseline for enhancement layers. The enhancement layers operate according to different encoding and / or decoding algorithms. The idea behind the overall algorithm is to encode / decode video frames in a hierarchical manner, as opposed to using a block-based approach as done in the MPEG family of algorithms. Encoding frames in a hierarchical manner includes generating residuals for full frames, and then generating residuals for reduced or decimated frames, etc.

[0044] In general, the term "residual" as used herein refers to the difference between the values ​​of a reference array or reference frame and the actual array or frame of data. The array may be a one-dimensional or two-dimensional array representing a coding unit. For example, a coding unit may be a 2×2 or 4×4 set of residual values ​​corresponding to a similarly sized area of ​​an input video frame. It should be noted that this generalized example is agnostic to the encoding operation performed and the nature of the input signal. Reference to "residual data" as used herein refers to data derived from a residual set, such as the residual set itself or the output of a data set processing an operation performed on the residual set. Throughout this specification, in general, a residual set includes a plurality of residuals or residual elements, each of which corresponds to a signal element, i.e., an element of a signal or original data. The signal may be an image or video. In these examples, the residual set corresponds to an image or frame of a video, wherein each residual is associated with a pixel of the signal, which is a signal element.

[0045] Figure 1 The encoding process is depicted in a block diagram of . As shown by the dotted line, the encoding process is split into two halves. Below the dotted line is the base layer of the encoder 100, which can be usefully implemented in hardware. Above the dotted line is the enhancement layer, which can be usefully implemented in software. The encoder 100 can include only the enhancement layer process, or a combination of the base layer process and the enhancement layer process as desired. The encoder 100 can be usefully implemented in software, especially at the enhancement layer. This arrangement allows, for example, an old hardware encoder that provides a base layer to be upgraded using a firmware (e.g., software) update, where the firmware is configured to provide the enhancement layer. In newer devices, both the base layer and the enhancement layer can be provided in hardware and / or a combination of hardware and software.

[0046] As noted, a base stream may be provided with two enhancement levels (sub-levels) within the enhancement stream. The signal is processed to produce various encoded streams. The base encoded stream is produced by feeding a downsampled version of the input signal to a base codec (e.g., AVC, HEVC, or any other codec). The encoded base stream may be referred to as a base layer or base layer. A first encoded stream (encoded level 1 stream) is produced by processing the residual obtained by taking the difference between the reconstructed base codec video and the downsampled version of the input video. A second encoded stream (encoded level 2 stream) is produced by processing the residual obtained by taking the difference between the upsampled version of the corrected version of the reconstructed base encoded video and the input video. In some cases, Figure 1The components of may provide a general low-complexity encoder. In some cases, the enhancement stream may be generated by an encoding process that forms part of the low-complexity encoder, and the low-complexity encoder may be configured to control an independent base encoder and decoder (e.g., encapsulated as a base codec). In other cases, the base encoder and decoder may be provided as part of the low-complexity encoder. In one case, Figure 1 The low-complexity encoder may be viewed as a form of wrapper for a base codec, where the functionality of the base codec may be hidden from the entity implementing the low-complexity encoder.

[0047] The encoder topology under the general hierarchy is as follows. The encoder 100 includes an input I for receiving an input signal 10. The input I is connected to a downsampler 105D to generate a downsampled signal to be encoded by a base encoder 120E of a base codec. Downsampling can be performed in both vertical and horizontal directions, or alternatively only in the horizontal direction. The downsampler 105D outputs to the base encoder 120E at the base level of the encoder 100. The base encoder 120E outputs a base coded stream. Therefore, a base coded stream is generated by feeding a downsampled version of the input video 10 to a base codec 120 (e.g., AVC, HEVC or any other codec) at the base level using the downsampler 105D.

[0048] As mentioned above, an input signal 10, such as full (or highest) resolution video in this example, is processed by encoder 100 to produce various encoded streams.

[0049] The first encoded stream (encoded level 1 stream) is produced by reconstructing the encoded base stream to produce a base reconstruction and then taking the difference between the base reconstruction and a downsampled version of the input video 10. The second encoded stream (encoded level 2 stream) is produced by upsampling a corrected version of the base reconstruction using an upsampler 105U and taking the difference between the corrected version of the base reconstruction and the input signal 10.

[0050] To produce the encoded level 1 stream, the encoded base stream is decoded by the base decoder 120D (i.e., a decoding operation is applied to the encoded base stream to produce a decoded base stream). The decoding may be performed by a decoding function or mode of the base codec. The difference between the decoded base stream and the downsampled input video is then generated at the level 1 comparator 110-S (i.e., a subtraction operation is applied to the downsampled input video and the decoded base stream). The difference is then encoded by the first residual encoder 100-1 (i.e., the level 1 encoder) to produce the encoded level 1 stream (i.e., an encoding operation is applied to the first residual set to produce the first enhancement stream).

[0051] The base encoder 120E and the base decoder 120D may be implemented by a base codec (e.g., as different functions of a common codec). The base codec and / or one or more of the base encoder 120E and the base decoder 120D may include appropriately configured electronic circuitry (e.g., a hardware encoder / decoder) and / or computer program code executed by a processor.

[0052] As described above, the enhancement stream may include a first enhancement level and a second enhancement level. The first enhancement level may be considered a corrected stream, e.g., a stream that provides a correction level to a base encoded / decoded video signal at a lower resolution than the input video 100. The second enhancement level may be considered to convert the corrected stream to another enhancement level of the original input video 100, e.g., it applies an enhancement or correction level to a signal reconstructed from the corrected stream.

[0053] exist Figure 1 In the example of , a second enhancement level is generated by encoding another set of residuals. The other set of residuals is generated by a 2-level comparator 100-S. The 2-level comparator 100-S determines the difference between the upsampled version of the decoded level 1 stream, such as the output of the upsampling component 105U, and the input signal 100. The input to the upsampling component 105U is generated by applying the first decoder 100-1i (i.e., the level 1 decoder) to the output of the first encoder 100-1. This generates a decoded level 1 residual set. These are then combined with the output of the base decoder 120D at the summing component 110-C. This effectively applies the level 1 residuals to the output of the base decoder 120D and allows the losses in the level 1 encoding and decoding process to be corrected by the level 2 residuals. The output of the summing component 110-C can be regarded as an analog signal representing the output of applying the level 1 processing to the encoded base stream at the decoder and the encoded level 1 stream.

[0054] As mentioned, the upsampled stream is compared to the input signal, which forms another set of residuals (i.e., a difference operation is applied to the upsampled regenerated stream to produce another set of residuals). The other set of residuals is then encoded by the second encoder 100-2 (i.e., the level 2 encoder) as an encoded level 2 enhancement stream (i.e., the encoding operation is then applied to the other set of residuals to produce an encoded another enhancement stream).

[0055] Therefore, if Figure 1 As shown in and described above, the output of the encoding process is a base stream and one or more enhancement streams, preferably including a first enhancement level and another enhancement level. The three streams can be combined with or without additional information such as a control header to produce a combined stream for representing the video coding architecture of the input video. It should be noted that Figure 1The components shown in may operate on blocks or coding units of data, which correspond, for example, to 2×2 or 4×4 portions of a frame at a particular resolution level. The components operate without any inter-block dependencies; thus, they may be applied to multiple blocks or coding units within a frame in parallel. This is different from contrasting video coding schemes, in which there are dependencies (e.g., spatial or temporal dependencies) between blocks. The dependencies of contrasting video coding schemes limit the level of parallelism and require much higher complexity.

[0056] According to certain embodiments of the present invention, deblocking filter block 140-1 may apply a filter to the corrected version before correcting the base reconstruction.

[0057] Deblocking filter block 140-1 or module can be operated on each block of transformed residual by applying a mask to which weights can be specified. More details will be provided below. In summary, embodiments of the present invention teach that filters and / or masks are applied to decoded residual. Deblocking includes weighting the boundaries of the blocks of transformed residual to improve picture quality and / or reduce closure.

[0058] exist Figure 2 The corresponding decoding process is depicted in the block diagram of . As shown by the dotted line, the decoding process is split into two halves. Below the dotted line is the base level of the decoder 200, which can be usefully implemented in hardware. Above the dotted line is the enhancement level, which can be usefully implemented in software. The decoder 200 can include only the enhancement level process, or a combination of the base level process and the enhancement level process as desired. The decoder 200 can be usefully implemented in software, especially at the enhancement level, and can be suitably superior to legacy decoding techniques, especially legacy hardware techniques. Legacy technology refers to an earlier technology that was previously developed and sold, which has appeared in the market and is inconvenient and / or expensive to replace and can still be used for the purpose of decoding a signal.

[0059] The decoder topology at a general level is as follows. The decoder 200 includes an input (not shown) for receiving one or more input signals including an encoded base stream, an encoded level 1 stream, and an encoded level 2 stream together with an optional header containing additional decoding information. The decoder 200 includes a base decoder 220 at a base level.

[0060] The decoder 200 receives the one or more input signals and directs the three streams generated by the encoder 100. The encoded base streams are directed to and decoded by a base decoder 213, which corresponds to the base codec 120 used in the encoder 100 and is used to reverse the encoding process at the base level.

[0061] The encoded level 1 stream is received by the first decoder 200-1 (ie, level 1 decoder), which decodes the encoded level 1 stream as received by Figure 1 The first residual set encoded by the first encoder 100-1 is decoded. At the first summing component 210-C, the output of the base decoder 220 is combined with the decoded residual obtained from the first decoder 200-1. The combined video, which may be referred to as a level 1 reconstructed video signal, is upsampled by an upsampling component 205U. The encoded level 2 stream is received by the second decoder 200-2 (i.e., a level 2 decoder). The second decoder 200-2 performs an upsampling operation on the level 2 stream as generated by the first decoder 100-1. Figure 1 The second residual set encoded by the second encoder 100-2 is decoded. The output of the second decoder 200-2 is a second decoded residual set. These may have a higher resolution to the first residual set and the input to the upsampling component 205U. At the second summing component 200-C, the second residual set from the second decoder 200-2 is combined with the output of the upsampling component 205U, i.e., the upsampled reconstructed level 1 signal to reconstruct the original signal.

[0062] On the decoding side, a deblocking filter module 240-1 may be provided according to an embodiment of the present invention. In a manner similar to the encoding side, the deblocking filter module 240-1 may be arranged before the upsampling module 205U and before the first summing component 210-C. As described above, the deblocking filter block 240-1 or module may operate on each block of the transformed residual by applying a mask to which weights may be specified. More details will be provided below.

[0063] According to the low complexity encoder, Figure 2 The low complexity decoder of the video signal can operate in parallel on different blocks or coding units of a given frame of the video signal. In addition, the decoding performed by two or more of the base decoder 210, the first decoder 211, and the second decoder 214 can be performed in parallel (e.g., different areas of the output video can be reconstructed in parallel). This is possible because there are no inter-block dependencies.

[0064] In the decoding process, the decoder may parse a set of headers (not shown, which may contain global configuration information, picture or frame configuration information, and data block configuration information) and configure the low-complexity decoder based on those headers. In the case where the input signal is a video signal, in order to regenerate the input video, the low-complexity decoder may decode each of the base stream, the first enhancement stream, and the other or second enhancement stream. The frames of the streams may be synchronized and then combined to derive the decoded video. Depending on the configuration of the low-complexity encoder and decoder, the decoded video may be a lossy or lossless reconstruction of the original input video. In many cases, the decoded video may be a lossy reconstruction of the original input video, where the loss has a reduced or minimal impact on the perception of the decoded video.

[0065] exist Figure 1 and 2 In each of the 2-level and 1-level encoding operations, the 2-level and 1-level encoding operations may include the steps of transform, quantization, and entropy encoding (e.g., in the order described). Similarly, in the decoding stage, the residual may be passed through an entropy decoder, a dequantizer, and an inverse transform module (e.g., in the order described). Any suitable encoding and corresponding decoding operations may be used. However, preferably, the 2-level and 1-level encoding steps may be performed in software (e.g., as performed by one or more central or graphics processing units in the encoding device).

[0066] Transforms as described herein may use directional decomposition transforms, such as Hadamard-based transforms. Both may include a small kernel or matrix applied to a flat coding unit of the residual (i.e., a 2×2 or 4×4 residual block). More details on the transform may be found, for example, in patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. The encoder may choose between different transforms to be used, for example, choosing between the sizes of the kernels to be applied.

[0067] The transform may transform the residual information to four surfaces. For example, the transform may produce the following components: average, vertical, horizontal, and diagonal.

[0068] Any known quantization scheme may be used to generate the residual signal into quanta so that certain variables may take only certain discrete values. In one case, quantization includes performing a division by a predetermined step width. This may be applied at two levels (0 and 1). For example, quantization may include dividing the transformed residual value by the step width. The step width may be predetermined, such as selected based on the desired quantization level. In one case, dividing by the step width may be converted to multiplying by the inverse step width, which may be more efficiently implemented in hardware. In this case, dequantization may include multiplying by the step width.

[0069] The video compression residual data of the full-size video frame may be referred to as LoQ-2 (e.g., 1920×1080 for an HD video frame), while the video compression residual data of the decimated frame may be referred to as LoQ-x, where x represents the number of corresponding hierarchical decimations. Figure 1 and 2 In the described example of , the variable x may have values ​​1 and 2 representing the first and second enhancement streams. Thus, there are 2 hierarchical levels that will produce a compressed residual. Other naming schemes for the levels may also be applied without any functional changes (e.g., the level 1 and level 2 enhancement streams described herein may alternatively be referred to as level 1 and level 2 streams, counting down from the highest resolution).

[0070] Figure 3 Show in more detail Figure 1 1. The encoded base stream is directly generated by the base encoder 120E and may be quantized and entropy encoded as needed. In some cases, these latter processes may be performed as part of the encoding performed by the base encoder 120E. In order to produce the encoded level 1 stream, the encoded base stream is decoded at the encoder 100 (i.e., the decoding operation is applied to the encoded base stream at the base decoding block 120D). The base decoding block 120D is shown as part of the base layer of the encoder 100 and is shown separated from the corresponding base encoding block 120E. For example, the base decoder 120D may be a decoding component that supplements the encoding component in the form of the base encoder 120E with a base codec. In other examples, the base decoding block 120D may actually be part of the enhancement layer, and more specifically may be part of the processing block 100-1.

[0071] Return to Figure 3 , the difference between the decoded base stream output from the base decoding block 120D and the downsampled input video is produced (i.e., the subtraction operation 110-S is applied to the downsampled input video and the decoded base stream to produce a first set of residuals). Here, the term "residual" is used in the same manner as is known in the art; that is, the residual represents the error or difference between a reference signal or frame and a desired signal or frame. Here, the reference signal or frame is the decoded base stream, and the desired signal or frame is the downsampled input video. Therefore, the residuals used in the first enhancement layer can be regarded as correction signals because they are able to 'correct' future decoded base streams to a closer approximation of the downsampled input video used in the base encoding operation. This is useful because this can correct for quirks or other characteristics of the base codec. These include, among other things, the motion compensation algorithm applied by the base codec, the quantization and entropy coding applied by the base codec, and the block adjustments applied by the base codec.

[0072] The first residual set is Figure 1 The processing is performed at block 100-1 in . Figure 3 The components of this block are shown in more detail in . Specifically, the first set of residuals is transformed, quantized, and entropy encoded to produce an encoded level 1 stream. Figure 3, a transform operation 110-1 is applied to a first residual set; a quantization operation 120-1 is applied to the transformed residual set to produce a quantized residual set; and an entropy encoding operation 130-1 is applied to the quantized residual set to produce an encoded level-1 stream at a first enhancement layer level. However, it should be noted that in other examples, only the quantization step 120-1, or only the transform step 110-1 may be performed. Entropy encoding may not be used, or may optionally be used as a supplement to one or both of the transform step 110-1 and the quantization step 120-1. The entropy encoding operation may be any suitable type of entropy encoding, such as a Huffmann encoding operation or a run-length encoding (RLE) operation, or a combination of both Huffmann encoding operations and RLE operations.

[0073] As described above, the enhancement stream may include an encoded level 1 stream (first enhancement layer) and an encoded level 2 stream (second enhancement layer). The first enhancement layer may be considered to implement the corrected video at the base layer, that is, for example, correcting encoder quirks. The second enhancement layer may be considered to be another enhancement layer that is used to convert the corrected video to the original input video or a close approximation thereof. For example, the second enhancement layer may add fine details lost during downsampling and / or help correct errors introduced by one or more of the transform operation 110-1 and the quantization operation 120-1.

[0074] See also Figure 1 and Figure 3 Both, to generate the encoded level 2 stream, another level of enhancement information is generated at block 100-2 by generating and encoding another set of residuals. The other set of residuals is the difference between an upsampled version (via upsampler 105U) of the corrected version of the decoded base stream (reference signal or frame) and the input signal 10 (desired signal or frame).

[0075] In order to achieve reconstruction of a corrected version of the decoded base stream as would be produced at the decoder 200, at least some of the processing steps of block 100-1 are undone to emulate the process of the decoder 200 and to account for at least some of the losses and quirks of the transform and quantization processes. To this end, block 100-1 includes an inverse quantization block 120-li and an inverse transform block 110-li. The quantized first residual set is inverse quantized at the inverse quantization block 120-li in the encoder 100 and inverse transformed at the inverse transform block 110-li to regenerate a decoder-side version of the first residual set.

[0076] Figure 3A deblocking filter block 140-1 is shown which may be provided in accordance with certain embodiments of the present invention. The deblocking filter step is optional depending on the transform applied and comprises applying a weighted mask to each block of the de-transformed first residual set. That is, the inverse transformed residual represents a set of blocks, wherein the size of the set of blocks depends on the transform used. Preferably, a mask of similar size is applied to each block. In summary, after the inverse transform, the output residual is filtered (or masked) by a deblocking filter module to further improve the decoder-side version of the re-generation of the first residual set prior to the summing operation 110-C. Optionally, it may be the case that the transform is such that a deblocking filter does not provide sufficient improvement. For example, deblocking may be provided for 4×4 transform blocks rather than 2×2.

[0077] The decoded base stream from decoder 120D is combined with this improved decoder-side version of the first residual set (i.e., a summation operation 110-C is performed on the decoded base stream and the decoder-side version of the first residual set). The summation operation 110-C produces a reconstruction of a downsampled version of the input video as it would likely be produced at the decoder, i.e., the reconstructed base codec video). Figure 1 and Figure 3 As shown in , the reconstructed base codec video is then upsampled by upsampler 105U.

[0078] The upsampled signal (i.e., reference signal or frame) is then compared to the input signal 10 (i.e., desired signal or frame) to produce another set of residuals (i.e., difference operation 100-S is applied to the upsampled regenerated stream to produce another set of residuals). The other set of residuals is then processed at block 100-2 to become an encoded level 2 stream (i.e., encoding operations are then applied to the other set of residuals to produce another encoded enhancement stream).

[0079] Specifically, the other residual set is transformed (i.e., the transform operation 110-2 is performed on the other residual set to produce another transformed residual set). The transformed residual is then quantized and entropy encoded in the manner described above with respect to the first residual set (i.e., the quantization operation 120-2 is applied to the transformed residual set to produce another quantized residual set; and the entropy encoding operation 120-2 is applied to the quantized other residual set to produce an encoded level 2 stream containing another level of enhancement information). However, only the quantization step 120-1, or only the transformation and quantization steps may be performed. Entropy encoding may optionally be used as a supplement. Preferably, the entropy encoding operation may be a Huffman encoding operation or a run length encoding (RLE) operation or both.

[0080] Therefore, if Figure 1 and 3As shown in and described above, the output of the encoding process is a base stream at a base layer level and one or more enhancement streams at enhancement layers, which preferably include a first enhancement layer and a further enhancement layer.

[0081] like Figure 3 As can be seen in FIG. 1 , the base codec can be divided into two components, namely a base encoder 120E and a base decoder 120D. It should be noted that the encoding and decoding parts can be in a single codec module, either in hardware or software, or they can be separate modules or components. If desired, the base decoder 120D can be present at the enhancement level in the encoder 100.

[0082] An encoded base stream and one or more enhancement streams are received at the decoder 200 . Figure 4 Show in more detail Figure 2 The decoder.

[0083] The encoded base stream is decoded at the base decoder 220 to produce a base reconstruction of the input signal 10 received at the encoder 100. This base reconstruction can be used in practice to provide a visual reproduction of the signal 10 at a lower quality level. However, the main purpose of this base reconstruction signal is to provide a basis for a higher quality reproduction of the input signal 10. To this end, the decoded base stream is provided to the processing block 200-1. The processing block 200-1 also receives the encoded level 1 stream and reverses any encoding, quantization and transformation applied by the encoder 100. Block 200-1 includes an entropy decoding process 230-1, an inverse quantization process 220-1 and an inverse transformation process 210-1. Optionally, only one or more of these steps may be performed depending on the operations implemented at the corresponding block 100-1 at the encoder. By performing these corresponding steps, the decoded level 1 stream including the first residual set is made available at the decoder 200. The first residual set is combined with the decoded base stream from the base decoder 220 (i.e., a summation operation 210-C is performed on the decoded base stream and the decoded first residual set to produce a reconstruction of a downsampled version of the input video, i.e., the reconstructed base codec video). Figure 2 and Figure 4 As shown in , the reconstructed base codec video is then upsampled by upsampler 205U.

[0084] Prior to the upsampler 205U and in this embodiment prior to the summing operation 210-C, the deblocking filter block 240-1 applies a mask to the inverse transformed residuals to compensate for effects or artifacts that may occur as part of the quantization and transform process. The deblocking filter step is optional depending on the transform applied, and includes applying a weighted mask to each block of the inverse transformed first residual set. That is, the inverse transformed residuals represent a set of blocks, where the size of the set of blocks depends on the transform used. Preferably, a mask of similar size is applied to each block. In summary, after the inverse transform, the output residuals are filtered (or masked) by the deblocking filter module to further improve the regenerated decoder-side version of the first residual set prior to the summing operation 210-C.

[0085] Whether a deblocking filter should be applied may be signaled by the encoder to the decoder. For example, a deblocking_signalled parameter may be included in the stream header and specify whether a deblocking filter should be applied. This flag indicates to the decoder that a deblocking filter should be used. Optionally, this flag indicates that a deblocking filter is used in the level 1 return path, e.g. Figure 3 In an example, the flag may be set in the configuration data of the payload, such as included in a header corresponding to the particular payload.

[0086] Signaling instances include: deblocking_signalled, which specifies whether the deblocking filter should be applied; deblocking_first_coefficient, which specifies the value of the first coefficient in the deblocking mask, i.e., the 4x4 block corner residual weights; and deblocking_second_coefficient, which specifies the value of the second coefficient in the deblocking mask, i.e., the 4x4 block side residual weights.

[0087] In some embodiments, the deblocking filter may be referred to as an in-loop filter. Conceptually, the filter is referred to as an in-loop filter because the processing affects the reference data so that the manipulated reference data may propagate the effect for upcoming picture encoding. The deblocking filter operates on transform block edges to reduce the amount of visible block structure, which is caused by the block-based nature of the encoding scheme.

[0088] As mentioned elsewhere herein, the transform type may be variable. In some embodiments, the transform may be a 2×2 or 4×4 transform using a correspondingly sized kernel. At the decoder, an operation may identify the transform type used and therefore identify that deblocking is not necessary or appropriate and may not be performed. For example, a 2×2 transform may not produce or have the same block edges as a 4×4 transform and may not be necessary.

[0089] The deblocking filter may involve applying a mask having a set of weights to each block of the transformed residual. The weights may be arranged around a marked boundary or edge. The weights may be predetermined or may be variable. In a preferred example, the weights may be signaled by the encoder to the decoder, preferably in the form of configuration metadata such as in a stream header.

[0090] In a specific example, the mask may include two different coefficients. The first coefficient may be positioned to be applied at the sides of the mask, and the second coefficient may be positioned to be applied at the corners of the mask. In a specific example, where the filter operates on each of a set of 4×4 blocks of transformed residuals by applying the mask, the weights may be signaled as follows:

[0091]

[0092] Where 0≤α≤1 and 0≤β≤1, and where α is the deblocking_first_coefficient and specifies the value of the first coefficient in the deblocking mask, i.e., the 4×4 block corner residual weights, and where β is the deblocking_second_coefficient and specifies the value of the second coefficient in the deblocking mask, i.e., the 4×4 block side residual weights.

[0093] Thus, in summary, an in-loop filter, namely the L-1 residual filter, is applied to the L-1 residual surface block, which is then added to the base reconstructed picture. In a specific decoding process implementation, the input to this process is: the sample position (xTb0, yTb0), which specifies the top left sample of the current transform block relative to the top left sample of the current picture; and an array of size 4×4 resL1Residuals, which specifies the level 1 residual. The output of this specific decoding process implementation is a 4×4 array of residuals resL1FilteredResiduals with elements resL1FilteredResiduals[x][y]. The in-loop filter L-1 residual filter is applied in this specific decoding process implementation as specified by the following ordered steps:

[0094] 1. Export the variables deblockEnabled, α, and β as follows:

[0095] deblockEnabled=deblocking_signalled

[0096] if(deblockEnabled)

[0097] α=deblocking_first_coefficient

[0098] β = deblocking_second_coefficient

[0099] else

[0100] α = 1

[0101] β = 1

[0102] 2. If deblockEnabled is true, then apply the following steps:

[0103] resL1FilteredResiduals[0][0] = (resL1Residuals[0][0] * α) >> 16

[0104] resL1FilteredResiduals[0][3] = (resL1Residuals[0][3] * α) >> 16

[0105] resL1FilteredResiduals[3][0] = (resL1Residuals[3][0] * α) >> 16

[0106] resL1FilteredResiduals[3][3] = (resL1Residuals[3][3] * α) >> 16

[0107] resL1FilteredResiduals[0][1] = (resL1Residuals[0][1] * β) >> 16

[0108] resL1FilteredResiduals[0][2] = (resL1Residuals[0][2] * β) >> 16

[0109] resL1FilteredResiduals[1][0] = (resL1Residuals[1][0] * β) >> 16

[0110] resL1FilteredResiduals[2][0] = (resL1Residuals[2][0] * β) >> 16

[0111] resL1FilteredResiduals[1][3] = (resL1Residuals[1][3] * β) >> 16

[0112] resL1FilteredResiduals[2][3]=(resL1Residuals[2][3]*β)>>16

[0113] resL1FilteredResiduals[3][1]=(resL1Residuals[3][1]*β)>>16

[0114] resL1FilteredResiduals[3][2]=(resL1Residuals[3][2]*β)>>16

[0115] resL1FilteredResiduals[1][1]=resL1Residuals[1][1]

[0116] resL1FilteredResiduals[1][2]=resL1Residuals[1][2]

[0117] resL1FilteredResiduals[2][1]=resL1Residuals[2][1]

[0118] resL1FilteredResiduals[2][2]=resL1Residuals[2][2]

[0119] otherwise:

[0120] The filtered residual is set at the decoded residual (eg, resL1FilteredResiduals[i][j]=resL1Residuals[i][j]).

[0121] Additionally, and optionally in parallel, Figure 2 The encoded level 2 stream is processed at block 200-2 of the encoder 100 to produce another set of decoded residuals. Similar to processing block 100-2, processing block 200-2 includes an entropy decoding process 230-2, an inverse quantization process 220-2, and an inverse transform process 210-2. Of course, these operations will correspond to those performed at block 100-2 in encoder 100, and one or more of these steps may be omitted as desired. Block 200-2 produces a decoded level 2 stream including another set of residuals and these are summed at operation 200-C with the output from upsampler 205U to produce a level 2 reconstruction of input signal 10.

[0122] Therefore, if Figures 1 to 4As shown and described above, the output of the decoding process is a base reconstruction, and a reconstruction of the original signal at a higher level. This embodiment is particularly well suited for generating encoded and decoded video at different frame resolutions. For example, the input signal 10 may be an HD video signal comprising frames at a resolution of 1920×1080. In some cases, both the base reconstruction and the level 2 reconstruction may be used by the display device. For example, in the case of network traffic, the level 2 stream may be more severely interrupted than the level 1 and base streams (because they may contain up to 4× the amount of data, where downsampling reduces the dimensions in each direction by 2). In this case, when traffic occurs, the display device may resume displaying the base reconstruction while the level 2 stream is interrupted (e.g., while the level 2 reconstruction is unavailable), and then resume displaying the level 2 reconstruction when network conditions improve.

[0123] A similar approach may be applied when the decoding device suffers from resource constraints, for example a set-top box performing a system update may have an operating base decoder 220 to output a base reconstruction, but may not have the processing capacity to compute a level 2 reconstruction.

[0124] Figure 1 and 3 The encoding arrangement also enables the video distributor to distribute the video to a heterogeneous set of devices; those devices with only the base decoder 220 view the base reconstruction, while those devices with the enhancement layer can view the higher quality Level 2 reconstruction. In a comparative case, two complete video streams at separate resolutions are required to serve the two sets of devices. Since the Level 2 and Level 1 enhancement streams encode residual data, the Level 2 and Level 1 enhancement streams can be encoded more efficiently, for example, the distribution of residual data is typically mostly around 0 in quality (i.e., there is no difference) and typically takes a small range of values ​​about 0. This may be the case especially after quantization. In contrast, the complete video streams at different resolutions will have different distributions of non-zero means or medians, which require higher bit rates for transmission to the decoder.

[0125] In summary, the quantized (or controlled) residual set is inverse quantized and inverse transformed, and then a deblocking filter is applied to produce a decoded first residual set (i.e., an inverse quantization operation is applied to the quantized first residual set to produce a dequantized first residual set; an inverse transform operation is applied to the dequantized first residual set to produce a detransformed first residual set; and a deblocking filter operation is applied to the detransformed first residual set to produce a decoded first residual set). The deblocking filter step is optional depending on the transform applied, and includes applying a weighted mask to each block of the detransformed first residual set.

[0126] Figure 5 is a flow chart showing a basic encoding method. The method is as follows and Figure 1 The process block is described in the context of:

[0127] Step 510: Receive a base encoded signal (output of base encoder 120E) generated from a downsampled version of the input signal (input signal 10) (output of downsampler 105D).

[0128] Step 520: Receive a decoded version of the base encoded signal (output of base decoder 120D).

[0129] Step 530: Compare the down-sampled version with the decoded version to generate a first residual signal (output of the comparator 110-S).

[0130] Step 540: Encode the first residual signal to generate a first encoded residual signal (the output of the residual encoder 100-1).

[0131] Step 550: Decode the first encoded residual signal to generate a first decoded residual signal (the output of the residual decoder 100-1i).

[0132] Step 555: Filter the first decoded residual signal (output of filter 140-1).

[0133] Step 560: Combine the decoded base encoded signal with the dequantized first residual signal (output of summation block 110-C).

[0134] Step 570: Upgrade the combined signal (output of upsampler 105U).

[0135] Step 580: Compare the input signal with the upgraded signal to generate a second residual signal (output of the comparator 100-S).

[0136] Of course, the method may include Figures 1 to 4 Specifically, the method may further comprise transforming and inversely transforming the first residual signal.

[0137] In the examples described herein, the residuals may be viewed as errors or differences at a particular quality level or resolution. In the described examples, there are two quality levels or resolutions and therefore two residual sets (level 1 and level 2). Each residual set described herein simulates a different form of error or difference. For example, level 1 residuals typically correct for characteristics of the base encoder, such as correcting for artifacts introduced by the base encoder as part of the encoding process. In contrast, for example, level 2 residuals typically correct for complex effects introduced by transitions in quality levels and differences introduced by level 1 corrections (e.g., artifacts produced by the level 1 encoding pipeline at a wider spatial scale, such as an area of ​​4 or 16 pixels). This means that it is not obvious that operations performed on one residual set will necessarily provide the same effect to another residual set, for example, each residual set may have a different statistical pattern and correlation set.

[0138] In the examples described herein, the residual is encoded by an encoding pipeline. This may include transform, quantization, and entropy encoding operations. It may also include residual scaling, weighting and filtering, and temporal processing. These pipelines are shown in Figure 1 and 3 A and 3B. The residual is then transmitted to the decoder, for example as first and second enhancement streams, which may be combined with the base stream as a hybrid stream (or transmitted separately). In one case, a bit rate is set for a hybrid data stream comprising the base stream and the two enhancement streams, and then different adaptive bit rates are applied to the individual streams based on the data being processed to meet the set bit rate (e.g., high quality video perceived with low-level artifacts can be constructed by adaptively assigning bit rates to different individual streams (even at a frame-by-frame level) so that constrained data can be used by the individual streams that are most perceptually affected, which may change as the image data changes).

[0139] The residual set as described herein can be considered as sparse data, e.g., in many cases there is no difference for a given pixel or region, and the resulting residual value is zero. When looking at the distribution of the residuals, a lot of probability mass is assigned to small residual values ​​that are close to zero, e.g., certain video values ​​of -2, -1, 0, 1, 2, etc. occur most frequently. In some cases, the distribution of residual values ​​is symmetric or approximately symmetric about 0. In some test video cases, it was found that the distribution of residual values ​​(e.g., symmetrically or approximately symmetric) is shaped like a logarithmic or exponential distribution about 0. The exact distribution of residual values ​​may depend on the content of the input video stream.

[0140] The residual can itself be processed as a two-dimensional image, such as a delta image of differences. In this way, it can be seen that the sparsity of the data is related to features such as "dots", small "lines", "edges", "corners", etc. that are visible in the residual image. It has been found that these features are usually not completely correlated (e.g., spatially and / or temporally). The features have characteristics that are different from the characteristics of the image data from which they are derived (e.g., pixel characteristics of the original video signal).

[0141] Because the characteristics of the residual are different from the characteristics of the image data from which the residual is derived, it is generally not possible to apply standard encoding methods, such as those found in the traditional Moving Picture Experts Group (MPEG) encoding and decoding standards. For example, many comparison schemes use larger transforms (e.g., transforms of larger pixel areas in normal video frames). Due to the characteristics of the residual, such as described above, it would be very inefficient to use these relatively large transforms on the residual image. For example, it would be very difficult to encode a small point in the residual image using a large block designed for an area of ​​the normal image.

[0142] Certain examples described herein address these issues by using smaller and simpler transform kernels instead (e.g., 2×2 or 4×4 kernels—directional decomposition and directional decomposition squared—as presented herein. This moves in a different direction than comparative video coding methods. Applying these new methods to residual blocks yields compression efficiencies. For example, certain transforms produce uncorrelated coefficients (e.g., in space) that can be efficiently compressed. While correlations between coefficients can be exploited, such as for lines in the residual image, these can produce coding complexity that makes implementation difficult on legacy and low-resource devices, and these often produce other complex artifacts that need to be corrected.

[0143] Certain examples described herein also consider temporal characteristics of the residual, such as well as spatial characteristics. For example, in a residual image, details such as "edges" and "points" that may be observed in the residual "image" show very little temporal correlation. This is because "edges" in a residual image do not typically translate or rotate like edges perceived in a normal video stream. For example, within a residual image, an "edge" may actually change shape over time, such as a head turn that may be captured within multiple residual image "edges," but the head turn may not move in a standard manner (because the "edge" reflects complex differences that depend on factors such as lighting, scaling factors, coding factors, etc.). These temporal aspects of a residual image, such as a residual "video" comprising sequential residual "frames" or "pictures," are typically different from the temporal aspects of a conventional image, such as a normal video frame (e.g., in the Y, U, or V plane). Therefore, it is not obvious how to apply conventional coding methods to residual images; in fact, it has been found that motion compensation methods from comparative video coding schemes and standards cannot encode residual data (e.g., in a useful manner).

[0144] In addition, many comparative video coding methods attempt to provide temporal prediction and motion compensation as the default for conventional video data. These "in-machine" methods may not only fail when applied to sequential residual pictures, they may take up unnecessary processing resources (e.g., these resources may be used while actually damaging the video coding). They may also generate unnecessary bits that take up the assigned bit rate. How to solve these problems is not obvious from conventional methods.

[0145] FIG7 is a flow chart showing a basic decoding method. The method is as follows and Figure 2 The process block is described in the context of:

[0146] Step 610: Receive a base encoded signal and instruct decoding to generate a base decoded signal (output of the base decoder 220).

[0147] Step 620: Receive the first encoded residual signal and decode (output of the basic decoder 200-1).

[0148] Step 630: Filter the decoded first residual signal (output of filter 240-1).

[0149] Step 640: Correct the decoded base encoded signal (output of summation block 210-C) with the filtered decoded first encoded residual signal.

[0150] Step 650: Upsample the corrected signal (output of upsampler 205U).

[0151] Step 660: Receive the second encoded residual signal and decode it (output of the residual decoder 200-2).

[0152] Step 670: Combine the upsampled signal with the decoded second residual signal (output of summation block 200-C).

[0153] Of course, the method may include Figures 1 to 4 Specifically, the method may include inverse transforming the first encoded residual signal.

[0154] The methods and processes described herein may be embodied as code (e.g., software code) and / or data at both encoders and decoders implemented in, for example, a streaming media server or client device or a client device decoded from a data storage device. The encoders and decoders may be implemented in hardware or software as is well known in the art to which data compression belongs. For example, hardware acceleration using a specially programmed graphics processing unit (GPU) or a specially designed field programmable gate array (FPGA) may provide certain efficiencies. For completeness, such codes and data may be stored on one or more computer-readable media, which may include any device or medium that can store code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on a computer-readable medium, the computer system executes the methods and processes embodied as data structures and codes stored in a computer-readable storage medium. In certain embodiments, one or more of the steps of the methods and processes described herein may be executed by a processor (e.g., a processor of a computer system or a data storage system).

[0155] In general, any of the functionality described in this text or shown in the figures may be implemented using software, firmware (e.g., fixed logic circuitry), programmable or non-programmable hardware, or a combination of these embodiments. In general, the terms "component" or "function" as used herein refer to software, firmware, hardware, or a combination of these. For example, in the case of a software embodiment, the term "component" or "function" may refer to program code that performs a specified task when executed on one or more processing devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

Claims

1. A method for encoding an input signal, the method comprising: generating a base encoded signal by feeding a downsampled version of the input signal to an encoder; The first residual signal is generated by the following operation: obtaining a decoded version of the base coded signal; as well as generating a first residual signal using a difference between the decoded version of the base encoded signal and the downsampled version of the input signal; encoding the first residual signal to generate a first encoded residual signal; The second residual signal is generated by the following operation: decoding the first encoded residual signal to produce a first decoded residual signal; filtering the first decoded residual signal to produce a filtered version of the first decoded residual signal; correcting the decoded version of the base coded signal using the filtered version of the first decoded residual signal to produce a corrected decoded version; upsampling the corrected decoded version; as well as generating the second residual signal using a difference between the corrected decoded signal and the input signal; as well as encoding the second residual signal to generate a second encoded residual signal, wherein the base coded signal, the first coded residual signal and the second coded residual signal comprise a coding of the input signal; The first residual signal comprises a plurality of N×N data blocks, and wherein encoding the first residual signal comprises: transforming the first residual signal on a block-by-block basis, The decoding of the first encoded residual signal comprises: performing an inverse transform on an N×N data block of a signal derived from the first encoded residual signal to produce the first decoded residual signal, and The filtering of the first decoded residual signal comprises: applying an N×N mask to each data block of the first decoded residual signal; The mask is a weighted mask, and each coefficient of the mask is a weight between 0 and 1.

2. The method according to claim 1, wherein the step of encoding the first residual signal comprises: The first residual signal is quantized to produce a first quantized residual signal, and the step of decoding the first encoded residual signal includes dequantizing the first quantized residual signal.

3. The method according to claim 1 or 2, wherein the step of transforming comprises applying a directional decomposition transform to the first residual signal to produce a directional component corresponding to the first residual signal. The method according to claim 1 , wherein N is 4 and the transform kernel used for the transform is a 4×4 kernel. The method of claim 1 , wherein the mask comprises a set of coefficients surrounding an edge of the mask.

6. The method of claim 1 or 2, wherein the mask comprises first coefficients for sides of the mask and second coefficients for corners of the mask.

7. The method according to claim 1 or 2, wherein N is 4 and the mask is: Where α and β are variable coefficients and where 0≤α≤1 and 0≤β≤1.

8. The method of claim 1 or 2, wherein filtering the first decoded residual signal comprises applying an in-loop filter.

9. The method according to claim 1 or 2, further comprising: The coefficients of the mask are signalled in configuration metadata accompanying the first or second residual signal or both.

10. A method of decoding an input signal from a plurality of encoded signals, the method comprising: receiving a base coded signal and instructing decoding of the base coded signal to generate a base decoded signal; receiving a first encoded residual signal and decoding the first encoded residual signal to generate a first decoded residual signal; filtering the first decoded residual signal to produce a filtered version of the first decoded residual signal; correcting the base decoded signal using the filtered version of the first decoded residual signal to produce a corrected version of the base decoded signal; upsampling the corrected version of the base decoded signal to produce an upsampled signal; receiving a second encoded residual signal and decoding the second encoded residual signal to generate a second decoded residual signal; as well as combining the upsampled signal and the second decoded residual signal to generate a reconstructed version of the input signal; wherein the step of decoding the first encoded residual signal comprises applying an inverse transform, the inverse transform being applied on a block-by-block basis, and wherein filtering the first decoded residual signal comprises applying an N×N mask to each block of the inverse transformed signal, wherein N×N is the size of a transform kernel used in the inverse transform; Wherein the mask is a weighted mask, and each coefficient of the mask is a weight between 0 and 1.

11. The method of claim 10, wherein the step of decoding the first encoded residual signal comprises applying an inverse quantization operation to the first encoded residual signal.

12. The method according to claim 10 or 11, wherein the step of inverse transforming comprises applying an inverse directional decomposition transform to a directional component of the first encoded residual signal to obtain residual values ​​corresponding to the first residual signal.

13. The method of claim 10 or 11, wherein N is 4 and the transform kernel is a 4x4 kernel.

14. The method of claim 10 or 11, wherein the mask comprises a set of coefficients surrounding an edge of the mask.

15. The method of claim 10 or 11, wherein the mask comprises first coefficients for sides of the mask and second coefficients for corners of the mask.

16. The method according to claim 10 or 11, wherein the mask is: Where α and β are variable coefficients and where 0≤α≤1 and 0≤β≤1.

17. The method according to claim 10 or 11, further comprising: The coefficients of the mask are retrieved from the configuration metadata.

18. The method according to claim 17, further comprising: The coefficients are retrieved from the stream header.

19. A method according to claim 10 or 11, identifying a transform applied to the first residual signal and applying the mask only if the transform is a transform of a predetermined size.

20. An apparatus configured to perform a method according to any one of claims 1 to 9 or any one of claims 10 to 19.

21. A computer readable medium comprising instructions which, when executed by a processor, cause the processor to carry out the method according to any one of claims 1 to 20.