Modified upsampling for video coding techniques

Abstract

A first-layer coded stream is received and decoded to derive a first residual set. A first output video is received comprising the output of a base decoder applied to a base-layer coded stream. The first residual set is combined with the first output video to generate a second output video. The second output video is upsampled to generate an upsampled second output video. The upsampling comprises adding values ​​derived from elements in the first residual set of blocks in the upsampled second output video from which the upsampled second output video is derived to the blocks in the upsampled second output video. A second-layer coded stream is received and decoded to derive a second residual set. The second residual set is combined with the upsampled second output video to generate a reconstructed output video.

Description

Technical Field

[0001] The present invention relates to methods, apparatus, computer programs and computer-readable media for use in video coding techniques. Background Art

[0002] Compression and decompression of signals is a consideration in many known systems. Many types of signals, such as video, can be compressed and encoded for transmission, for example, over a data communications network. When decoding such signals, it may be desirable to increase the quality level of the signal and / or recover as much information as possible from the original signal.

[0003] Some known systems employ scalable coding techniques. Scalable coding involves encoding a signal along with information to allow reconstruction of the signal at one or more different quality levels, depending on, for example, the capabilities of the decoder and the available bandwidth.

[0004] There are several considerations regarding the reconstruction of signals in scalable coding systems. One such consideration is the ability of the encoder and / or decoder to process information efficiently. The efficiency with which the encoder and / or decoder process information can be a factor in the performance level of the encoder and / or decoder. Summary of the Invention

[0005] Various aspects of the invention are set out in the accompanying claims.

[0006] Further features and advantages will become apparent from the following description made with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram illustrating an encoding process according to examples herein;

[0008] Figure 2 is a schematic diagram illustrating a decoding process according to examples herein;

[0009] Figure 3 is a schematic diagram illustrating an encoding process according to examples herein; and

[0010] Figure 4 is a schematic diagram illustrating a decoding process according to examples herein. DETAILED DESCRIPTION

[0011] A hybrid backward-compatible coding technique is described herein. The examples described herein provide a flexible, adaptable, efficient, and computationally inexpensive coding format that combines coded data of different video coding formats, a base codec (e.g., AVC, HEVC, or any other current or future codec) with at least two enhancement layers.

[0012] The general structure of the coding scheme uses a downsampled source signal encoded with a base codec, adds a first level of correction data to the decoded output of the base codec to generate a corrected picture, and then adds another enhancement level of data to an upsampled version of the corrected picture.

[0013] Thus, the streams are considered to be a base stream and an enhancement stream.It is worth noting that the base stream is generally expected to be decodable by a hardware decoder, while the enhancement stream is expected to be suitable for software processing implementation with suitable power consumption.

[0014] This structure creates multiple degrees of freedom, allowing great flexibility and adaptability to many situations, making the decoding format suitable for many use cases, including over-the-top OTT delivery, live streaming, live UHD broadcasting, etc.

[0015] Although the decoded output of the base codec is not intended for viewing, it is fully decoded video at a lower resolution, making the output compatible with existing decoders and usable as lower resolution output if deemed appropriate.

[0016] The codec format uses a minimal number of simple coding tools. When combined synergistically, they can provide visual quality improvements compared to full-resolution pictures encoded with the base codec, while making the way they can be used flexible.

[0017] Figure 1 The encoding process is depicted in a block diagram of FIG. Input full-resolution video 102 is processed to generate various encoded streams. A first encoded stream (encoded base stream 110) is generated by feeding a downsampled version of the input video produced by downsampling 104 the input video 102 to a base encoder 106 (e.g., AVC, HEVC, or any other codec). A second encoded stream (encoded layer 1 stream 116) is generated by applying an encoding operation 114 to a residual obtained by taking the difference 112 between the reconstructed base codec video and the downsampled version of the input video. The reconstructed base codec video is obtained by decoding the output of the base encoder 106 with a base decoder 108. A third encoded stream (encoded layer 2 stream 128) is generated by processing 126 the residual obtained by taking the difference 124 between the input video 102 and an upsampled version of the corrected version of the reconstructed base decoded video. A corrected version of the reconstructed base codec video is obtained by summing 120 the reconstructed base codec video with the residual obtained by applying the decoding operation 118 to the encoded level 1 stream 116 .

[0018] Figure 2The decoding process is depicted in a block diagram of FIG. The decoder receives the three streams generated by the encoder (encoded base stream 110, encoded layer 1 stream 116, and encoded layer 2 stream 128) along with a header 130 containing additional decoding information. Encoded base stream 110 is decoded by a base decoder 208, which corresponds to base decoder 108 used in the encoder, and its output is combined 136 with the decoded residual obtained by decoding 132 encoded layer 1 stream 116. The combined video is upsampled 138 and further combined 140 with the decoded residual obtained by applying decoding operation 134 to encoded layer 2 stream 128.

[0019] Figure 3 Show the above brief description and Figure 1 As can be clearly seen, the encoding and decoding steps of the stream are detailed in more detail to provide examples of how the steps can be performed.

[0020] The base stream 110 is generally created by the process described above. That is, the input video 102 is downsampled 104 (i.e., the downsampling operation 104 is applied to the input video 102 to generate a downsampled input video). The downsampled video obtained by downsampling 104 the input video 102 is then encoded using a first base encoder 106 (i.e., the encoding operation is applied to the downsampled input video to generate the encoded base stream 110 using the first or base encoder 106). Preferably, the first or base encoder 106 is a codec suitable for hardware decoding. The encoded base stream 110 may be referred to as a base layer or base level.

[0021] As mentioned above, the enhancement stream may comprise two streams. The first enhancement level provides a set of correction data that can be combined with a decoded version of the base stream to generate a corrected picture. This first enhancement stream is Figure 1 and 3 Shown in FIG. 1 is an encoded layer 1 stream 116 .

[0022] To generate the coded layer 1 stream 116, the coded base stream 110 is decoded using the base decoder 108 (i.e., a decoding operation is applied to the coded base stream 110 to generate the decoded base stream). A difference 112 is then created between the decoded base stream and the downsampled input video obtained by downsampling 104 the input video 102 (i.e., a subtraction operation 112 is applied to the downsampled input video and the decoded base stream to generate a first residual set). Here, the term "residual" is used in the same manner as is known in the art, i.e., the error between a reference frame and a desired frame. Here, the reference frame is the decoded base stream, and the desired frame is the downsampled input video. Thus, the residual used in the first enhancement layer can be considered as corrected video because it 'corrects' the decoded base stream to the downsampled input video used in the underlying encoding operation.

[0023] The difference 112 is then encoded to generate the encoded level 1 stream 116 (ie, the encoding operation 114 is applied to the first residual set to generate the first enhancement stream 116).

[0024] exist Figure 3 In an example embodiment of , the encoding operation includes several steps, each of which is optional and preferred and provides specific benefits.

[0025] Figure 3 , the steps include a transform step 144 , a quantization step 146 and an entropy coding step 148 .

[0026] Although not shown in the figure, in some examples, the encoding process 114 identifies whether a residual scalability mode is selected. If the residual mode is selected, a residual scalability step may be performed (i.e., a residual scalability operation may be performed on the first step of the residual to generate a scalable set of residuals). The scalable set of residuals may be filtered so that not all of the residuals are encoded into the first enhancement stream 116 (or correction stream).

[0027] The first residual set or the scaled or filtered first residual set is then transformed 144, quantized 146, and entropy encoded 148 to produce the encoded layer 1 stream 116 (i.e., depending on whether the scaled mode is selected, the transform operation 144 is applied to the first residual set or the filtered first residual set to generate a transformed residual set; the quantization operation 146 is applied to the transformed residual set to generate a quantized residual set; and the entropy encoding operation 148 is applied to the quantized residual set to generate the first enhancement layer stream 116). Preferably, the entropy encoding operation 148 can be a Huffman encoding operation or a run-length encoding operation or both. Optionally, a control operation (not shown) can be applied to the quantized residual set to correct for the effects of the scaled operation.

[0028] As described above, the enhancement stream may include a first enhancement level 116 and a second enhancement level 128. The first enhancement level 116 may be considered a corrected stream. The second enhancement level 128 may be considered another enhancement level that converts the corrected stream into the original input video.

[0029] Another enhancement layer 128 is created by encoding 126 another set of residuals as the difference 124 between an upsampled version of the decoded layer 1 stream and the input video 102, as Figure 1 shown.

[0030] Figure 3 , the quantized (or manipulated) residual set is inverse quantized 150 and inverse transformed 152, and then a deblocking filter (not shown) is optionally applied to generate a decoded first residual set (i.e., the inverse quantization operation 150 is applied to the quantized first residual set to generate a dequantized first residual set; the inverse transform operation 152 is applied to the dequantized first residual set to generate a detransformed first residual set; and a deblocking filter operation is optionally applied to the detransformed first residual set to generate a decoded first residual set). The deblocking filter step is optional depending on the transform 144 applied and includes applying a weighted mask to each block of the detransformed 152 first residual set.

[0031] The decoded base stream is combined 120 with the decoded first residual set (ie, a summation operation 120 is performed on the decoded base stream and the decoded first residual set to generate a recreated first stream). Figure 1 and 3 As shown in , the combination is then upsampled 122 (ie, an upsampling operation 122 is applied to the recreated first stream to generate an upsampled recreated stream).

[0032] The upsampled stream is then compared to the input video 102 creating another residual set (i.e., the difference operation 124 is applied to the upsampled recreated stream to generate another residual set). The other residual set is then encoded as an encoded level 2 enhancement stream 128 (i.e., the encoding operation 126 is then applied to the other residual set to generate the encoded other enhancement stream 128).

[0033] As with the encoded level 1 stream 116, the encoding applied to the level 2 residual may include several steps. Figure 3 The steps are shown as transform 154 , quantization 156 , and entropy encoding 158 .

[0034] Although not shown in the figure, in some examples, the encoding process identifies whether residual scalability mode is selected. If residual scalability mode is selected, a residual scalability step may be performed (i.e., a residual scalability operation may be performed on another set of residuals to generate another set of scalable residuals). The another set of scalable residuals may be filtered so that not all of the residuals are encoded into the other enhancement stream 128.

[0035] The other set of residuals or the other set of tiered residuals is then transformed 154 (i.e., the transform operation 154 is performed on the other set of tiered residuals to generate another set of transformed residuals). As shown, the transform operation 154 may utilize prediction coefficients or prediction means derived from the recreated first stream prior to upsampling 122. Additional information is as follows.

[0036] Although not shown in the figure, in some examples, another transformed 154 residual set may undergo a temporal prediction step. Note that the temporal prediction step may also be applied to the first residual set in the series, thereby creating the level 1 enhancement stream 116. When temporal prediction is selected, the encoder may further modify the coefficients by subtracting the corresponding coefficients derived from the temporal buffer. In general, the encoded coefficients correspond to the difference between the frame and another frame of the stream. The other frame may be an earlier or later frame (or block within a frame) in the stream. Thus, instead of encoding 126 the residual between the upsampled recreated stream and the input video 102, the encoding process 126 may encode the difference between the transformed frame in the stream and the transformed residual of the frame. Thus, entropy may be reduced. When temporal prediction is active, each transformed coefficient may be:

[0037] Δ=F 当前 -F 缓冲器

[0038] Where the buffer may store the previous frame and the difference may be sent to the buffer instead of the tile. Thus, in effect, the residual is relative to the buffer instead of the frame.

[0039] The transformed residuals (temporally predicted or otherwise) are then quantized 156 and entropy encoded 158 in the manner described elsewhere (i.e., the quantization operation 156 is applied to the transformed 154 set of residuals to generate another set of quantized residuals; and the entropy encoding operation 158 is applied to the quantized another set of residuals to generate another enhancement layer stream).

[0040] Therefore, if Figure 3 As shown in and described above, the output of the encoding process is a base stream 110 and one or more enhancement streams, which preferably include a first enhancement layer level 116 and a further enhancement layer level 128.

[0041] Figure 4 Show the above brief description and Figure 2As can be clearly seen, the steps are detailed in more detail to provide examples of how the steps can be performed. Figure 4 .

[0042] During the decoding process, the decoder may parse headers 130 (e.g., containing global configuration data, picture configuration data, and other data blocks) and configure the decoder based on those headers 130. To recreate the input video 102, the decoder may decode each of the base stream 110, the first enhancement stream 116, and the further enhancement stream 128. The frames of the streams may be synchronized and then combined to derive the decoded video 142.

[0043] In each decoding process, the enhancement stream may undergo entropy decoding 160, 172, inverse quantization 162, 174, and inverse transformation 164, 176 steps to recreate the residual set.

[0044] Figure 4 The decoding process of comprises retrieving an array of entropy-decoded quantized coefficients representing the first enhancement layer and outputting an array of L-1 residuals. Figure 4 The decoding process further includes retrieving an array of samples of the output of the base decoder 208 . Figure 4 The decoding process further includes applying a dequantization process 162 to the array of entropy decoded quantized coefficients to derive a set of dequantized coefficients, applying a transform process 164 to the set of dequantized coefficients, and optionally applying a filtering process ( Figure 4 (not shown) to output an array representing the L-1 residual of the first enhancement level. Figure 4 The decoding process of then further comprises recreating 136 the picture by combining 136 the array of L−1 residuals with the array of samples of the output of the base decoder 208 . Figure 4 The decoding process includes applying a transform process 164 from a set of predetermined transform processes according to the signaled parameters. For example, the transform process 164 can be applied to a 2×2 coding unit or a 4×4 coding unit. A coding unit can be referred to herein as a block of elements in an array, in this case an array of L-1 residuals.

[0045] Figure 4 The decoding process of comprises retrieving an array of entropy-decoded quantized coefficients representing the first enhancement layer and outputting an array of residuals. Figure 4 In the decoding process shown in , the other enhancement level is the second enhancement level, and the array of residual outputs is an array of L-2 residuals. Figure 4 The method further comprises retrieving an array of L-1 residuals of the first enhancement layer corresponding to an array of entropy-decoded quantized coefficients representing another enhancement layer. Figure 4The method further comprises applying an upsampling process 138 to the array of residuals of the first enhancement level. Figure 4 In , the upsampling process 138 is applied to the combination of the array of L-1 residuals of the first enhancement level and the corresponding array of samples of the output of the base decoder 208. Figure 4 Not shown in , but in some examples, the method includes applying a temporal prediction process to an array of entropy-decoded quantized coefficients representing another enhancement layer to derive an array of temporally predicted samples. Figure 4 The method further includes applying a dequantization process 174 to the array of entropy-decoded quantized coefficients to derive a set of dequantized coefficients, applying a transform process 176 to the set of dequantized coefficients to derive a set of transformed coefficients. In some examples, the array of temporally predicted samples may then be combined with the set of transformed coefficients to derive an array of L-2 residuals for another enhancement layer. Figure 4 The method then further includes recreating 140 the picture from the array of L-2 residuals, for example, by adding 140 the array of L-2 residuals to the array produced by the upsampling process 138. The method may include applying a transform process 176 from a set of predetermined transform processes according to the signaled parameters. For example, the transform process 176 may be applied to a 2x2 block of elements in the array of L-2 residuals or a 4x4 block of elements in the array of L-2 residuals.

[0046] Figure 4 The decoding process of includes performing a modified upsampling process 138, as further described below. Figure 4 The step of adding the modifier 168 defined below may be performed as part of the transform process 176. Alternatively, because the transform process 176 involves a linear transform, the step of adding the modifier 168 may be performed as part of the modified upsampling process 138 described below. The step of adding the modifier 168 thus results in a modification of the residual. The modification may be performed based on the position of the residual in the frame. The modification may be a predetermined value.

[0047] Each enhancement stream, or both, is encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Layer Units (NALUs). A NALU is intended to encapsulate the enhancement bitstream so that the enhancement is applied to the correct base reconstructed frame. For example, a NALU may contain a reference index to a NALU containing the base decoder reconstructed frame bitstream to which the enhancement must be applied. In this way, the enhancement can be synchronized to the base stream, and each frame of the bitstream is combined to produce the decoded output video (i.e., the residual of each frame of the enhancement layer is combined with a frame of the base decoded stream). A group of pictures can represent multiple NALUs.

[0048] It was mentioned above how a set of tools can be applied to each of the enhancement streams (or input video 102) throughout the process. The following provides an overview of each of the tools and their use in Figures 1 to 4 The functionality within the overall process shown in .

[0049] A downsampling process 104 is applied to the input video 102 to produce a downsampled video to be encoded by an underlying encoder 106. The downsampling 104 may be performed in both the vertical and horizontal directions, or only in the horizontal direction.

[0050] The input to the L-1 encoding operation 114 includes the L-1 residual obtained by taking the difference 112 between the decoded output of the base decoder 108 and the downsampled video obtained by downsampling 104 the input video 102. The L-1 residual is then transformed 144, quantized 146, and encoded 148, as further described below.

[0051] There are two types of transforms that can be used in the transform process 144. Both utilize a small kernel that is applied directly to the residual that remains after the stage of applying predictive averaging.

[0052] The first transform has a 2×2 kernel applied to a 2×2 block of the residual. The resulting coefficients are as follows:

[0053]

[0054] The second transform has a 4×4 kernel applied to a 4×4 block of the residual. The resulting coefficients are as follows:

[0055]

[0056] If downsampling and / or upsampling is performed only in the horizontal direction (eg certain elements are set to 0), a suitably adapted transform may also be applied.

[0057] The coefficients are then quantized using a linear quantizer 146. The linear quantizer may use a variable size dead zone. The linear quantizer may use a dead zone of varying sizes with respect to the quantization step and the non-centered dequantization offset.

[0058] The quantized coefficients are encoded using an entropy coder 148. There are two schemes for entropy coding 148. In the first scheme, the quantized coefficients are encoded using a run-length encoder (RLE). In the second scheme, the quantized coefficients are first encoded using RLE, and then the encoded output is processed using a Huffman encoder.

[0059] If residual mode (RM) has been selected, the L-1 residuals are further ranked and selected in order to determine which residuals should be transformed 144 and encoded. Preferably, this is preformed before entropy encoding 148.

[0060] If the temporal selective mode is selected, the encoder will further modify the coefficients by subtracting the corresponding coefficients derived from the temporal buffer, ie the temporal prediction described below.

[0061] The input to the L-1 decoding operation 118 comprises the L-1 encoded residual, which has passed through the entropy decoder, dequantizer, and inverse transform modules. The operations performed by these modules are the inverse operations performed by the modules described above.

[0062] If the temporal selective mode has been selected, the residual may be predicted in part from the co-located residual from the temporal buffer.

[0063] If a 4x4 transform is used, the decoded residual can be fed into a deblocking filter module. The deblocking filter operates on each block of the transformed residual by applying a mask whose weights can be specified. The general structure of the mask is as follows:

[0064]

[0065] Where 0≤α≤1 and 0≤β≤1.

[0066] The output from combining 120 the decoded (and deblocked, if applicable) L-1 residual and the base decoded video is upsampled 122 to generate upsampled reconstructed video. Upsampling may be optional and signaled in the byte stream.

[0067] The input to the L-2 encoding operation 126 includes the L-2 residual obtained by taking the difference 124 between the upsampled reconstructed video and the input video 102. The L-2 residual is then transformed 154, quantized 156, and encoded 158, as further described below. The transformation 154, quantization 156, and encoding 158 are performed in the same manner as described with respect to the L-1 encoding 114. If RM is selected, the L-2 residuals are further ranked and selected to determine which residuals should be transformed and encoded. The L-2 encoding operation 126 may further include two additional processes as described below.

[0068] If the prediction coefficient mode is selected, the encoder will further modify the transformed coefficient C00 (e.g., the "average" or "A" coefficient for a 2x2 transform). If a 2x2 transform is used, C00 will be modified by subtracting the value of the upsampled residual of the transformed block from which the prediction residual is obtained. If a 4x4 transform is used, C00 will be modified by subtracting the average of the four upsampled residuals of the transformed block from which the prediction residual is obtained.

[0069] If the temporal selective mode is selected, the encoder will further modify the coefficients by subtracting the corresponding coefficients derived from the temporal buffer, as described above.

[0070] The input to the L-2 decoding operation 134 includes the encoded L-2 residual. The decoding process of the L-2 residual passes through the entropy decoder 172, the dequantizer 174, and the inverse transform module 176. The operations performed by these modules are the inverse operations performed by the modules described above. If the temporal selection mode is selected, the residual can be partially predicted from the co-located residual from the temporal buffer.

[0071] The type of transform performed by inverse transform module 176 may be indicated by a transform type parameter received by the decoder. For example, a transform type parameter value of 0 may be used to indicate that a 2x2 directional decomposition transform is to be performed, and a transform type parameter value of 1 may be used to indicate that a 4x4 directional decomposition transform is to be performed.

[0072] In one example, a parameter is received at the decoder indicating that the upsampling process 138 is to be performed only in the horizontal direction. That is, the upsampling 138 of a single element outputs a 2×1 block of elements. This parameter will be described in more detail further below. In this case, the transform operation 176 applied to the 2×2 block in the array of dequantized coefficients is as follows:

[0073]

[0074] When upsampling 138 is performed only in the horizontal direction, the transform operation 176 applied to the 4x4 block in the array of dequantized coefficients is as follows:

[0075]

[0076] The modified upsampling process 138 comprises two steps, the second of which depends on signaling received by the decoder. In the first step, the combination 136 of the decoded (and deblocked, if applicable) L-1 residual and the base decoded video 208 (L-1 reconstructed video) is upsampled 166 to generate upsampled reconstructed video. If the prediction coefficient mode is selected, the second step is performed. Specifically, the values ​​168 of the elements of the L-1 reconstruction values ​​of the 2×2 block in the upsampled reconstructed video from which the values ​​are derived are added 170 to the 2×2 block in the upsampled reconstructed video.

[0077] Figure 3 and 4 The encoding and decoding scheme uses the tools described above and the following tools.

[0078] Based on the selected prediction residual mode, the encoder will further modify the upsampled reconstructed video by applying the prediction residual (PR). The prediction residual is based on the upsampled reconstructed video and on the reconstructed lower resolution video before upsampling. The prediction residual can also be called a prediction mean or modifier, as described in more detail below.

[0079] If the time selective mode is selected, the encoder will further modify the coefficients by adding the corresponding coefficients derived from the time buffer.

[0080] Throughout the above, the term bitstream may be replaced by stream or bytestream or NALUstream as needed.

[0081] Figure 4 Show Figure 2 An example of a modified upsampling process 138 used in the decoding process is shown in FIG. Figure 4 The modified upsampling process 138 is described in more detail in to provide an example of how this process may be performed.

[0082] An example of the modified upsampling process 138 may be summarized using the following pseudocode:

[0083]

[0084] Where: nCurrS = the size of the (nCurrS)X(nCurrS) array of elements in the L-1 reconstructed video; xSrc, ySrc are index values ​​for specifying a given element in the array of elements in the L-1 reconstructed video; xDst, yDst are index values ​​for specifying a given element in the array of elements in the upsampled reconstructed video or the array of elements in the modified upsampled reconstructed video; resFilteredSamples is the array of elements in the L-1 reconstructed video; resUpsampledSamples is the array of elements in the upsampled reconstructed video; and resModifiedUpsampledSamples is the modified array of elements in the upsampled reconstructed video.

[0085] In this example, the input to the modified upsampling process 138 is a combination of the decoded (and deblocked, if applicable) L-1 residual and the base decoded video. This combination is represented by the variable resFilteredSamples in the pseudo code above. This combination can be referred to as the L-1 reconstructed video.

[0086] The L-1 reconstructed video is upsampled 166 to generate an upsampled reconstructed video. In this example, the upsampling 166 is performed in both the vertical and horizontal directions so that a single value in the L-1 reconstructed video is upsampled 166 to generate a 2×2 block of elements in the upsampled reconstructed video. Thus, in this example, the size of the array has increased from 1×1 in the L-1 reconstructed video to 2×2 in the upsampled reconstructed video due to the upsampling 166, thus increasing along both dimensions of the array (vertical and horizontal). The 2×2 block of elements in the upsampled reconstructed video is represented by the variable resUpsampledSamples in the pseudocode above.

[0087] The modifier 168 is calculated using the values ​​in the L-1 reconstructed video and the 2×2 block of elements in the upsampled reconstructed video. The modifier 168 may also be referred to as a predicted average. In this example, the modifier 168 represents the difference between the values ​​in the L-1 reconstructed video and the average of the values ​​in the 2×2 block in the upsampled reconstructed video.

[0088] The modifier 168 is then added 170 to each of the elements in the 2x2 block in the upsampled reconstructed video, thereby generating a modified 2x2 block of elements in the upsampled reconstructed video. The modified 2x2 block of elements in the upsampled reconstructed video is represented in the pseudo code above by the variable resModifiedUpsampledSamples.

[0089] Adding modifier 168 may be performed as part of transform process 176. Adding modifier 168 as part of modified upsampling process 138 may reduce the complexity of the decoding process relative to adding modifier 168 as part of transform process 176. For example, this approach avoids the need for transform process 176 to additionally involve transforming modifiers and dequantizing coefficients obtained from encoded layer-2 stream 128. Adding modifier 168 as part of modified upsampling process 138, relative to adding modifier 168 as part of transform process 176, is possible because transform process 176 involves a linear transform. Thus, the decoded video 142 resulting from adding modifier 168 as part of the transform process is identical to the decoded video 142 resulting from adding modifier 168 as part of the modified upsampling process, but is generated more efficiently.

[0090] The modified 2×2 block of the element in the upsampled reconstructed video output from the modified upsampling process 138 is then added 140 to the corresponding 2×2 block of the L-2 residual to generate the 2×2 block of the element in the decoded video 142 .

[0091] The addition of modifier 168 may be performed based on receiving parameters indicating that modifier 168 is to be added to the 2×2 block in the upsampled reconstructed video to generate a modified 2×2 block. In this example, if the parameters are not received, modifier 168 is not added, and the 2×2 block in the upsampled reconstructed video generated from upsampling process 166 is added 140 to the corresponding 2×2 block of the L-2 residual to generate the 2×2 block of the element of decoded video 142. Alternatively, the parameters may be received, but the parameters may indicate that modifier 168 is not to be added. The value of the parameter may have a bit length of one bit. The parameters may be received once for a group of pictures associated with the encoded stream discussed above, which is a set of consecutive pictures within the coded video stream. The modified upsampling process 138 may include retrieving an indicator from a header accompanying the data. The indicator may be retrieved from a predetermined location in the header and may indicate one or more actions. The indicator may indicate whether the modified upsampling process 138 is performed.

[0092] In another example, the modified upsampling process 138 can be summarized by the following pseudo-code using the same variables as defined in the pseudo-code above:

[0093]

[0094] In this other example, upsampling 166 is performed only in the horizontal direction, so that a single value in the L-1 reconstructed video is upsampled 166 to generate a 2×1 block of elements in the upsampled reconstructed video. As such, the size of the array has increased from 1×1 in the L-1 reconstructed video to 2×1 in the upsampled reconstructed video due to upsampling 166, and thus has only increased along one direction of the array (horizontally). In this case, the modifier 168 represents the difference between the value in the L-1 reconstruction and the average of the values ​​in the 2×1 block of elements in the upsampled reconstructed video. Adding 170 the modifier 168 to each of the elements in the 2×1 block in the upsampled reconstruction generates a modified 2×1 block of elements in the upsampled reconstructed video.

[0095] The direction in which upsampling 166 is performed may be determined by a received parameter indicating the direction in which upsampling 166 is to be performed. For example, a value of 1 for the parameter may be used to indicate upsampling 166 in the horizontal direction only, while a value of 0 for the parameter may be used to indicate upsampling 166 in both the vertical and horizontal directions.

[0096] In the decoding process described herein, generating decoded video 142 may be performed block by block. In this manner, generation of a block of an element in a frame of decoded video 142 may be performed without using another block of the element in the same frame that was previously generated for decoded video 142. For this reason, modified upsampling process 138 may be performed in parallel for all blocks of the element in the frame, as opposed to sequentially performing modified upsampling process 138 for each block of the element in the frame.

[0097] Other examples are described below.

[0098] Some other examples relate to a method for encoding an input video into multiple encoded streams such that the encoded streams can be combined to reconstruct the input video, the method comprising: receiving a full-resolution input video; downsampling the full-resolution input video to create a downsampled video; encoding the downsampled video using a first codec to create a base encoded stream; reconstructing a video from the encoded video to generate a reconstructed video; comparing the reconstructed video to the input video; and creating one or more additional encoded streams based on the comparison. The input video may be a downsampled video compared to the reconstructed video.

[0099] According to an example method, comparing the reconstructed video to the input video includes comparing the reconstructed video to the downsampled video to create a first residual set, and wherein creating the one or more additional encoded streams includes encoding the first residual set to create a first-level encoded stream.

[0100] The input video may be a full resolution input video and the reconstructed video may be upsampled compared to the reconstructed video.

[0101] According to an example method, comparing the reconstructed video to the input video includes: upsampling the reconstructed video to generate an upsampled reconstructed video; and comparing the upsampled reconstructed video to the full-resolution input video to create a second set of residuals, and wherein creating the one or more additional encoded streams includes encoding the second differences to create a second-level encoded stream.

[0102] Accordingly, in an example, the method may generate a base coded stream, a first-level coded stream, and a second-level coded stream according to the example method defined above. Each of the first-level coded stream and the second-level coded stream may contain enhancement data used by the decoder to enhance the coded base stream.

[0103] According to an example method, encoding the first set of residuals includes applying a transform to the set of residuals to create a set of coefficients; applying a quantization operation to the coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0104] According to an example method, encoding the second set of residuals includes applying a transform to the second set of residuals to create a set of coefficients; applying a quantization operation to the coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0105] The transform used to encode the first and / or second residual sets may be, for example, a discrete cosine transform or a wavelet transform. In an alternative example, the transform may be a small transform that decomposes the block of elements into directional components (e.g., using a 2×2 kernel or a 4×4 kernel). For example, the 2×2 kernel may be a Hadamard transform. For more details on the transform, see, for example, patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. In another example, the encoder may choose between different transforms to be used, for example, between a 2×2 kernel and a 4×4 kernel. This allows for greater flexibility in the way the residuals are encoded. The choice of transform may be based on an analysis of the data to be transformed.

[0106] The first and second residual sets may have different transforms applied to them, and the selection may be predetermined or selected during the process.The transform used may be signaled in the header.

[0107] The quantization used to encode the first and / or second residual sets may be, for example, linear quantization. The linear quantizer may use a variable-sized dead zone. The encoding operation may be, for example, an entropy encoder and may include run-length encoding and / or Huffman encoding.

[0108] The residual may be the difference between two videos or frames.

[0109] The step of encoding the first set of residuals may include ranking the first set of residuals based on a pre-analysis of the first set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0110] In one example, the method includes analyzing a first set of residuals and, based on the analysis, performing or not performing the steps of: ranking the first set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0111] In one example, the method comprises analyzing a first set of residuals, and: ranking the first set of residuals; and selecting a subset of residuals to be transformed and encoded, such that the ranking and / or selection steps are performed differentially based on the analysis.

[0112] According to an example method, the step of applying a transform is performed on a selected subset of residuals of the first set of residuals.

[0113] The step of encoding the second set of residuals may include ranking the second set of residuals based on a pre-analysis of the second set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0114] In one example, the method includes analyzing the second set of residuals and, based on the analysis, performing or not performing the steps of: ranking the second set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0115] In one example, the method comprises analyzing a second set of residuals, and: ranking the second set of residuals; and selecting a subset of the residuals to be transformed and encoded, such that the ranking and / or selecting steps are performed differentially based on the analysis.

[0116] According to an example method, the step of applying the transform is performed on a selected subset of the residuals of the second set of residuals.

[0117] The encoded stream may be accompanied by one or more headers containing parameters indicating aspects of the encoding process to facilitate decoding. For example, the header may include the codec used, the transform applied, the quantization applied, and / or other decoding parameters.

[0118] In some examples, the step of quantizing may include adapting the quantization based on an analysis of the coefficients and / or data to be transformed (eg, residual data). In some examples, the distribution used in the step of quantizing may be adapted.

[0119] The step of encoding the first residual set may include: deriving a set of temporal coefficients from the temporal buffer; and subtracting the set of temporal coefficients from the set of coefficients.

[0120] The step of encoding the second set of residuals may include: deriving a set of temporal coefficients from the temporal buffer; and subtracting the set of temporal coefficients from the set of coefficients.

[0121] The above describes how the steps of grading and selection can be applied to the residual data, the step of subtracting the temporal coefficients can be performed, and the quantization can also be adapted. Each of these steps can be predetermined and selectively applied, or can be applied based on analysis of the input video, downsampled video, reconstructed video, upsampled video, or any combination thereof to improve the overall performance of the encoder. The steps can be selectively applied based on a predetermined set of rules or deterministically applied based on analysis or feedback of performance.

[0122] According to an example method, the first codec is a hardware-based codec, preferably the first codec is AVC, HEVC, AV1, VP8 or VP9.

[0123] The example method further includes sending the base encoded stream.

[0124] The example method further includes sending the first-level encoded stream.

[0125] The example method further includes sending the second-level encoded stream.

[0126] Some other examples relate to a method of decoding multiple encoded streams into a reconstructed output video, the method comprising: receiving a first base encoded stream; decoding the first base encoded stream according to a first codec to generate a first output video; receiving one or more additional encoded streams; decoding the one or more additional encoded streams to generate a residual set; and combining the residual set with the first video to generate a decoded video.

[0127] In one example, the method includes retrieving a plurality of decoding parameters from the header. The decoding parameters may indicate which procedural steps are included in the encoding process.

[0128] In an example, decoding the one or more additional encoded streams to generate a set of residuals includes applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to generate the set of residuals.

[0129] In an example, decoding the one or more further encoded streams to generate a set of residuals includes predicting a subset of the residuals based on a co-located residual from a temporal buffer.

[0130] In one example, the method may include receiving a first-level coded stream and receiving a second-level coded stream. In this example, decoding the one or more additional coded streams to generate a residual set includes decoding the first-level coded stream to derive a first residual set; wherein combining the residual set with the first video to generate a decoded video includes combining the first residual set with the first output video to generate a second output video; upsampling the second output video to generate an upsampled second output video; decoding the second-level coded stream to derive a second residual set; and combining the second residual set with the second output video to generate a reconstructed output video.

[0131] In one example, upsampling the second output video to generate the upsampled second output video includes adding values ​​of elements in a first residual set of blocks in the upsampled second output video from which the blocks are derived to corresponding blocks in the upsampled second output video. The blocks may be 2×2 blocks. This adding may be selectively performed based on a predetermined value or signal included in a header.

[0132] In one example, decoding the first-level coded stream to derive the first residual set includes: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to generate the first residual set.

[0133] In this example, decoding the first-level coded stream to derive the first residual set includes applying a deblocking filter configured to apply a mask to the residual block. The mask may be weighted according to a set of predefined weights.

[0134] In one example, decoding the second-level coded stream to derive the second residual set includes: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to generate the second residual set.

[0135] The inverse transform operation may be the inverse of the operation as defined above, or may be a substantially mirrored operation. That is, a 2x2 block or 4x4 block transform may be selectively applied. The transform may be detected by a decoding method or signaled in a header.

[0136] If a 2x2 transform is used, the coefficients may be modified by adding the value of the residual from which the transformed residual block was predicted. If a 4x4 transform is used, the coefficients will be modified by adding the average of the four residuals.

[0137] The method may further include displaying or outputting the reconstructed output.In an example, the method described above further includes retrieving an indicator from a header accompanying the data.

[0138] The indicator can be retrieved from a predetermined location in the header and can indicate one or more actions. The indicator can indicate whether the steps of adding residuals and / or prediction residuals specified in the above statements are performed. The indicator can indicate whether the decoder should perform specific operations, or be configured to perform specific operations, in order to decode the bitstream. The indicator can also indicate whether such steps have already been performed at the encoder stage.

[0139] Some other examples relate to a device for encoding a data set into an encoded data set including a header and a payload. The device is configured to encode an input video according to the above steps. The device may include a processor configured to implement the method of any of the above aspects.

[0140] Some other examples relate to a device for decoding a data set including a header and a payload into a reconstructed video. The device is configured to decode the output video according to the above steps. The device may include a processor configured to implement the method of any of the above aspects.

[0141] Encoders and decoders are also available.

[0142] Some other examples relate to a computer-readable medium that, when executed by a processor, causes the processor to perform any of the methods of the above aspects.

[0143] The above embodiments should be understood as illustrative examples. Other examples are contemplated. It should be understood that any feature described with respect to any one example may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other example, or in any combination of any other example. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the appended claims.

Claims

1. A method for decoding a plurality of encoded streams into a reconstructed output video, the method comprising: receiving a first-level coded stream; decoding the first-level coded stream to derive a first residual set; receiving a first output video comprising an output of a base decoder applied to a base layer coded stream; combining the first set of residuals with the first output video to generate a second output video; performing modified upsampling on the second output video to generate a modified upsampled second output video, the modified upsampling comprising: upsampling the second output video to generate an upsampled second output video; and adding a modifier to each block in the upsampled second output video that represents a difference between a value of an element in the second output video and an average of values ​​in blocks in the upsampled second output video, wherein the blocks in the upsampled second output video are derived from the elements in the second output video, and wherein the values ​​of the elements in the second output video represent a combination of an element in the first residual set and a corresponding element in the first output video, wherein if the upsampling is performed in both horizontal and vertical directions, the blocks in the upsampled second output video are 2×2 element blocks, and If the upsampling is performed only in the horizontal direction, the blocks in the upsampled second output video are 2×1 element blocks; receiving a second-level coded stream; decoding the second-level coded stream to derive a second residual set; and The second set of residuals is combined with the upsampled second output video to generate the reconstructed output video. 2 . The method of claim 1 , wherein adding the modifier is performed based on receiving a parameter indicating that the modifier is to be added to the block in the upsampled second output video. The method according to claim 2 , wherein the bit length of the value of the parameter is one bit. 4 . The method of claim 2 , wherein the parameters are received once for a group of pictures associated with the plurality of encoded streams.

5. The method of any one of claims 1 to 3, wherein a first block of elements from a frame in the reconstructed output video corresponding to the block in the upsampled second output video is generated without using a second block of elements from the frame in the reconstructed output video.

6. The method according to any one of claims 1 to 3, wherein decoding the first-level coded stream to derive the first residual set comprises: applying a first entropy decoding operation; applying a first dequantization operation; as well as, A first transform operation is applied to generate the first set of residuals.

7. The method according to any one of claims 1 to 3, wherein decoding the first level coded stream to derive the first residual set comprises applying a deblocking filter configured to apply a mask to a residual block, the mask being weighted according to a set of predefined weights.

8. The method according to any one of claims 1 to 3, wherein decoding the second-level coded stream to derive the second residual set comprises: applying a second entropy decoding operation; applying a second dequantization operation to generate a set of dequantized transform coefficients; as well as, A second transform operation is applied to the set of dequantized transform coefficients to generate the second set of residuals.

9. The method of claim 8, comprising receiving a transform type parameter indicating a type of the second transform operation to be applied.

10. The method of claim 8, wherein the second transform operation is a 2x2 directional decomposition transform, the set of dequantized transform coefficients represents a 2x2 block of elements from the set of dequantized transform coefficients, and the second set of residuals represents a 2x2 block of elements from the second set of residuals.

11. The method of claim 8, wherein the second transform operation uses a 4x4 directional decomposition transform, the set of dequantized transform coefficients represents a 4x4 block of elements from the set of dequantized transform coefficients, and the second set of residuals represents a 4x4 block of elements from the second set of residuals.

12. The method of claim 11, wherein the value derived from the element in the first set of residuals is based on an average of four residuals of a 4x4 block from which elements from the reconstructed output video are derived.

13. The method of any one of claims 1 to 3, wherein receiving the first output video comprises: receiving the base layer coded stream; The base layer encoded stream is decoded according to the base decoder to generate the first output video.

14. A decoding device, configured to execute the method according to any one of claims 1 to 13.

15. A computer-readable medium comprising instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 13.

Images