Method of decoding multiple encoded streams into reconstructed output video

Abstract

The invention relates to a method of decoding a plurality of encoded streams into a reconstructed output video, comprising: receiving and decoding a first level encoded stream to derive a first set of residuals; receiving a first output video; combining the first residual set with the first output video to generate a second output video; upsampling the second output video to generate an upsampled second output video; receiving a second level coded stream and decoding to derive a second residual set; and combining the second set of residuals with the up-sampled second output video to generate a reconstructed output video.

Description

[0001] This application is a divisional application of Chinese invention patent application CN202080037119.X, filed on March 10, 2020, entitled "Modified Upsampling for Video Decoding Technology". Technical Field

[0002] This invention relates to methods, apparatus, computer programs, and computer-readable media for use in video decoding technology. Background Technology

[0003] Signal compression and decompression are considerations in many known systems. For example, many types of signals, such as video, can be compressed and encoded for transmission, for example, via data communication networks. When decoding this signal, it may be necessary to increase the signal's quality level and / or recover as much information as possible from the original signal.

[0004] Some known systems employ scalable coding techniques. Scalable coding involves encoding a signal along with its information to allow the signal to be reconstructed at one or more different quality levels, depending, for example, on the capabilities of the decoder and the available bandwidth.

[0005] Several factors need to be considered regarding signal reconstruction in scalable coding systems. One such factor is the ability of the encoder and / or decoder to efficiently process information. The efficiency with which the encoder and / or decoder process information can be a factor in the performance level of the encoder and / or decoder. Summary of the Invention

[0006] This invention relates to a method for decoding multiple encoded streams into a reconstructed output video, the method comprising: receiving a first-level encoded stream; decoding the first-level encoded stream to derive a first residual set; receiving a first output video, the first output video including the output of a base decoder applied to a base-level encoded stream; combining the first residual set with the first output video to generate a second output video; upsampling the second output video to generate an upsampled second output video, the upsampling comprising adding values ​​derived from elements in the first residual set from which blocks in the upsampled second output video are derived to the blocks in the upsampled second output video, wherein the addition of the values ​​derived from the elements in the first residual set is based on received parameters indicating that the values ​​derived from the elements in the first residual set will be added to the blocks in the upsampled second output video; receiving a second-level encoded stream; decoding the second-level encoded stream to derive a second residual set; and combining the second residual set with the upsampled second output video to generate the reconstructed output video.

[0007] The present invention also relates to corresponding decoding devices, methods for encoding multiple streams to provide reconstructed output video, encoding and decoding devices, computer programs, and computer-readable media.

[0008] Further features and advantages will become apparent from the following description made with reference to the accompanying drawings. Attached Figure Description

[0009] Figure 1 This is a schematic diagram illustrating the encoding process based on the examples in this article;

[0010] Figure 2 This is a schematic diagram illustrating the decoding process based on the examples in this article;

[0011] Figure 3 This is a schematic diagram illustrating the encoding process based on the examples in this article; and

[0012] Figure 4 This is a schematic diagram illustrating the decoding process based on the examples in this article. Detailed Implementation

[0013] This paper describes a hybrid backward-compatible decoding technique. The examples described herein provide a flexible, adaptable, efficient, and computationally inexpensive decoding format that combines different video decoding formats, underlying codecs (e.g., AVC, HEVC, or any other current or future codecs), and decoded data with at least two enhancement layers.

[0014] The general structure of the encoding scheme uses a downsampled source signal encoded by a base codec, adds first-level correction data to the decoded output of the base codec to generate a corrected image, and then adds another enhancement level of data to the upsampled form of the corrected image.

[0015] Therefore, the stream is considered as a basic stream and an enhanced stream. It is worth noting that the basic stream is generally expected to be decoded by a hardware decoder, while the enhanced stream is expected to be suitable for software processing implementations with appropriate power consumption.

[0016] This structure creates multiple degrees of freedom, allowing for great flexibility and adaptability to many situations, thus making the decoding format suitable for many use cases, including over-the-top OTT transmission, live streaming, live UHD broadcasting, etc.

[0017] Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, thus making the output compatible with existing decoders and usable as a lower resolution output where appropriate.

[0018] The codec formats use a minimal number of simple decoding tools. When combined in combination, they offer improved visual quality compared to full-resolution images encoded with the underlying codec, while also making them more flexible in how they can be used.

[0019] Figure 1 The block diagram depicts the encoding process. The input full-resolution video 102 is processed to generate various encoded streams. A first encoded stream (encoded base stream 110) is generated by feeding a downsampled form of the input video, obtained through downsampling 104 of the input video 102, to a base encoder 106 (e.g., AVC, HEVC, or any other codec). A second encoded stream (encoded level 1 stream 116) is generated by applying encoding operation 114 to the residue obtained by taking the difference 112 between the reconstructed base codec video and the downsampled form of the input video. The reconstructed base codec video is obtained by decoding the output of the base encoder 106 with the base decoder 108. A third encoded stream (encoded level 2 stream 128) is generated by processing the residue obtained by 126 through the difference 124 between the upsampled form of the reconstructed base decoded video and the input video 102. The corrected form of the reconstructed base codec video is obtained by summing the remnants 120 obtained by applying decoding operation 118 to the encoded layer 1 stream 116.

[0020] Figure 2 The block diagram depicts the decoding process. The decoder receives three streams generated by the encoder (encoded base stream 110, encoded layer 1 stream 116, and encoded layer 2 stream 128) along with a header 130 containing additional decoding information. Encoded base stream 110 is decoded by base decoder 208 corresponding to base decoder 108 used in the encoder, and its output is combined with the decoded residual obtained by decoding layer 1 stream 116 through decoding 132, resulting in a combined video 136. The combined video is upsampled 138 and further combined with the decoded residual obtained by applying decoding operation 134 to encoded layer 2 stream 128, resulting in a combined video 140.

[0021] Figure 3 The above brief description and Figure 1 The following is an example implementation of the encoding process. As can be clearly seen, the encoding and decoding steps of the stream are described in more detail to provide examples of how the steps can be performed.

[0022] The base stream 110 is generally created by the process described above. That is, the input video 102 is downsampled 104 (i.e., the downsampling operation 104 is applied to the input video 102 to generate a downsampled input video). The downsampled video obtained by downsampling the input video 102 104 is then encoded using a first base encoder 106 (i.e., the encoding operation is applied to the downsampled input video to generate the encoded base stream 110 using the first or base encoder 106). Preferably, the first or base encoder 106 is a codec suitable for hardware decoding. The encoded base stream 110 may be referred to as a base layer or base level.

[0023] As described above, the enhancement stream may include two streams. The first enhancement layer provides a set of correction data, which can be combined with the decoded format of the base stream to generate a corrected image. This first enhancement stream... Figure 1 and 3 The diagram shows stream 116 after encoding level 1.

[0024] To generate the encoded layer 1 stream 116, the encoded base stream 110 is decoded using the base decoder 108 (i.e., the decoding operation is applied to the encoded base stream 110 to generate the decoded base stream). Then, a difference 112 is created between the decoded base stream and the downsampled input video obtained by downsampling the input video 102 (i.e., the subtraction operation 112 is applied to the downsampled input video and the decoded base stream to generate a first residual set). Here, the term "residual" is used in the same manner as known in this art, i.e., the error between a reference frame and a desired frame. Here, the reference frame is the decoded base stream, and the desired frame is the downsampled input video. Therefore, the residual used in the first enhancement layer can be considered as corrected video because it 'corrects' the decoded base stream to the downsampled input video used in the base coding operation.

[0025] The difference 112 is then encoded to generate an encoded level 1 stream 116 (i.e., the encoding operation 114 is applied to the first residual set to generate the first enhanced stream 116).

[0026] exist Figure 3 In some example implementations, the encoding operation includes several steps, each of which is optional and preferred and provides specific benefits.

[0027] Figure 3 The steps include a transformation step 144, a quantization step 146, and an entropy coding step 148.

[0028] Although not illustrated, in some instances, the encoding process 114 identifies whether a residual grading mode is selected. If a residual mode is selected, a residual grading step can be performed (i.e., a residual grading operation can be performed on the first step of the residual to generate a graded set of residuals). The graded set of residuals can be filtered so that not all residuals are encoded into the first enhancement stream 116 (or correction stream).

[0029] Next, the first residual set, or the graded or filtered first residual set, is subjected to transform 144, quantization 146, and entropy coding 148 to produce a coded hierarchical stream 116 (i.e., depending on whether the graded mode is selected, transform operation 144 is applied to the first residual set or the filtered first residual set to generate a transformed residual set; quantization operation 146 is applied to the transformed residual set to generate a quantized residual set; and entropy coding operation 148 is applied to the quantized residual set to generate a first enhanced hierarchical stream 116). Preferably, entropy coding operation 148 can be a Huffman coding operation or a run-length coding operation or both. Optionally, control operations (not shown in the figure) can be applied to the quantized residual set to correct for the effects of the graded operation.

[0030] As described above, the enhanced stream may include a first enhancement level 116 and a second enhancement level 128. The first enhancement level 116 can be considered as a corrected stream. The second enhancement level 128 can be considered as another enhancement level that converts the corrected stream into the original input video.

[0031] Another enhancement layer 128 is created by encoding 126 as another set of residuals between the upsampled format of the decoded layer 1 stream and the input video 102, such as... Figure 1 As shown.

[0032] Figure 3 In this process, the quantized (or controlled) residual set undergoes inverse quantization 150 and inverse transform 152, and then optionally a deblocking filter (not shown in the figure) is applied to generate the decoded first residual set (i.e., inverse quantization operation 150 is applied to the quantized first residual set to generate the dequantized first residual set; inverse transform operation 152 is applied to the dequantized first residual set to generate the detransformed first residual set; and optionally a deblocking filter operation is applied to the detransformed first residual set to generate the decoded first residual set). The deblocking filter step is optional depending on the applied transform 144 and includes applying a weighted mask to each block of the first residual set of detransformation 152.

[0033] The decoded base stream is combined with the decoded first residual set 120 (i.e., a summation operation 120 is performed on the decoded base stream and the decoded first residual set to generate a recreated first stream). Figure 1 and3 As shown, the combination is then upsampled 122 (i.e., the upsampling operation 122 is applied to the recreated first stream to generate the upsampled recreated stream).

[0034] The upsampled stream is then compared with the input video 102 that creates another set of residuals (i.e., the difference operation 124 is applied to the upsampled recreated stream to generate another set of residuals). The other set of residuals is then encoded into a level 2 enhanced stream 128 (i.e., the encoding operation 126 is then applied to the other set of residuals to generate another encoded enhanced stream 128).

[0035] As with the encoded Level 1 stream 116, the encoding applied to the Level 2 residual may include several steps. Figure 3 The steps are shown as Transformation 154, Quantization 156, and Entropy Encoding 158.

[0036] Although not illustrated in the figure, in some instances, the encoding process identifies whether a residual grading mode is selected. If a residual mode is selected, a residual grading step can be performed (i.e., a residual grading operation can be performed on another set of residuals to generate another graded set of residuals). The other graded set of residuals can be filtered so that not all residuals are encoded into another enhancement stream 128.

[0037] Subsequently, transformation 154 is performed on another set of residuals or another graded set of residuals (i.e., transformation operation 154 is performed on another graded set of residuals to generate another transformed set of residuals). As shown, prior to upsampling 122, transformation operation 154 may utilize the prediction coefficients or prediction averages derived from the recreated first stream. Further information is as follows.

[0038] Although not illustrated, in some instances, another set of residuals after transformation 154 may undergo a time prediction step. Note that the time prediction step can also be applied to the first set of residuals in the series, thereby creating a level 1 enhanced stream 116. When time prediction is selected, the encoder can further modify the coefficients by subtracting the corresponding coefficients derived from the time buffer. In summary, the encoded coefficients correspond to the difference between the frame in the stream and another frame. The other frame can be an earlier or later frame in the stream (or a block within a frame). Therefore, instead of encoding 126 the residuals between the upsampled recreated stream and the input video 102, the encoding process 126 can encode the difference between the transformed frames in the stream and the transformed residuals of the frames. Thus, entropy can be reduced. When time prediction is active, each transformed coefficient can be:

[0039]

[0040] The buffer stores previous frames and sends differences to the buffer instead of splicing. Therefore, the residual is actually relative to the buffer, not the frame.

[0041] The transformed residuals (predicted in time or otherwise) are then quantized 156 and entropy encoded 158 in a manner described elsewhere (i.e., quantization operation 156 is applied to the set of residuals after transformation 154 to generate another set of quantized residuals; and entropy encoding operation 158 is applied to the other set of quantized residuals to generate another enhanced hierarchical flow).

[0042] Therefore, as Figure 3 As shown and described above, the output of the encoding process is a base stream 110 and one or more enhancement streams, wherein the enhancement streams preferably include a first enhancement layer 116 and another enhancement layer 128.

[0043] Figure 4 The above brief description is shown and Figure 2 An example implementation of the decoding process is shown below. As can be clearly seen, the steps are described in more detail to provide an example of how they can be performed. Further details are described below. Figure 4 .

[0044] During decoding, the decoder can parse header 130 (e.g., containing global configuration data, image configuration data, and other data blocks) and configure the decoder based on those headers 130. To recreate the input video 102, the decoder can decode each of the base stream 110, the first enhancement stream 116, and another enhancement stream 128. Frames from the streams can be synchronized and then combined to derive the decoded video 142.

[0045] In each decoding process, the enhanced stream can undergo entropy decoding 160, 172, inverse quantization 162, 174 and inverse transform 164, 176 steps to recreate the residual set.

[0046] Figure 4 The decoding process includes retrieving an array of quantized coefficients representing the entropy decoding of the first enhancement level and outputting an array of L-1 residuals. Figure 4 The decoding process further includes retrieving an array of samples from the output of the base decoder 208. Figure 4 The decoding process further includes applying dequantization process 162 to the array of quantized coefficients of entropy decoding to derive a set of dequantized coefficients, applying transformation process 164 to the set of dequantized coefficients, and optionally applying a filtering process. Figure 4 (Not shown in the figure) An array representing the L-1 residuals of the first enhancement level as output. Figure 4The decoding process then further includes recreating the 136-image by combining the array of L-1 residuals with the array of samples of the output of the base decoder 208. Figure 4 The decoding process includes a transformation process 164 applying a set of predetermined transformation processes based on the transmitted parameters. For example, transformation process 164 can be applied to a 2×2 decoding unit or a 4×4 decoding unit. A decoding unit may be referred to herein as a block of elements in an array, in this case an array of L⁻¹ residuals.

[0047] Figure 4 The decoding process includes retrieving an array of quantized coefficients representing the entropy decoding of the first enhancement level and outputting an array of residuals. Figure 4 In the decoding process shown, another enhancement level is the second enhancement level, and the array of residual outputs is an array of L-2 residuals. Figure 4 The method further includes retrieving an array of L-1 residuals of a first enhancement level corresponding to an array of quantized coefficients representing entropy decoding of another enhancement level. Figure 4 The method further includes applying the upsampling process 138 to an array of residuals from the first enhancement level. Figure 4 In this process, the upsampling process 138 is applied to the combination of the array of L-1 residuals of the first enhancement layer and the corresponding array of samples from the output of the basic decoder 208. Although Figure 4 Not illustrated, but in some instances, the method includes applying a time prediction process to an array of quantized coefficients representing entropy decoding of another enhancement level to derive an array of time-predicted samples. Figure 4 The method further includes applying a dequantization process 174 to the array of entropy-decoded quantized coefficients to derive a set of dequantized coefficients, and applying a transformation process 176 to the set of dequantized coefficients to derive a set of transformed coefficients. In some instances, the array of time-predicted samples can then be combined with the set of transformed coefficients to derive an array of L⁻² residuals for another enhancement layer. Figure 4 The method further includes, for example, recreating the image 140 from the L-2 residual array by adding the L-2 residual array 140 to the array generated by the oversampling process 138. The method may include applying a transformation process 176 from a predetermined set of transformation processes according to transmitted parameters. For example, transformation process 176 may be applied to a 2x2 block or a 4x4 block of features in the L-2 residual array.

[0048] Figure 4 The decoding process includes performing a modified upsampling process 138, as further described below. Figure 4As described below, the step of adding the modifier 168, as defined below, can be performed as part of the transformation process 176. Alternatively, because the transformation process 176 involves a linear transformation, the step of adding the modifier 168 can be performed as part of a modified upsampling process 138 as described below. The step of adding the modifier 168 thus results in a modification of the residual. The modification can be performed based on the position of the residual in the frame. The modification can be a predetermined value.

[0049] Each or two enhancement streams can be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Units (NALUs). NALUs are intended to encapsulate enhancement bitstreams so that enhancements are applied to the correct underlying reconstructed frames. A NALU may, for example, contain a reference index to the NALU containing the underlying decoded reconstructed frame bitstream to which the enhancements must be applied. In this way, enhancements can be synchronized to the underlying stream, and frames from each bitstream are combined to produce the decoded output video (i.e., the residual of each frame from the enhancement layer combined with frames from the underlying decoded stream). A group of images can represent multiple NALUs.

[0050] The above section mentioned how a set of tools can be applied to enhance each element of the stream (or input video 102) throughout the process. The following section provides an overview of each of these tools and their applications, such as... Figures 1 to 4 The functionality within the overall process shown.

[0051] The downsampling process 104 is applied to the input video 102 to produce a downsampled video to be encoded by the base encoder 106. Downsampling 104 can be performed in both the vertical and horizontal directions, or only in the horizontal direction.

[0052] The input to the L-1 encoding operation 114 includes an L-1 residual obtained by taking the difference 112 between the decoded output of the base decoder 108 and the downsampled video, which is obtained by downsampling the input video 102 104. The L-1 residual is then transformed 144, quantized 146, and encoded 148, as further described below.

[0053] There are two types of transformations that can be used in transformation process 144. Both utilize a small kernel, which is applied directly to the residuals retained after the stage of applying the predictive average.

[0054] The first transformation has a 2×2 kernel applied to the residual in a 2×2 block. The resulting coefficients are as follows:

[0055]

[0056] The second transformation has a 4×4 kernel applied to the residual in 4×4 blocks. The resulting coefficients are as follows:

[0057]

[0058] If downsampling and / or upsampling are performed only in the horizontal direction (e.g., some features are set to 0), then a suitable adaptation transformation can also be applied.

[0059] The coefficients are then quantized using a linear quantizer (146). The linear quantizer can use a variable-size dead zone. The linear quantizer can use a dead zone of a different size than the quantization step and the non-centered dequantization offset.

[0060] The quantized coefficients are encoded using an entropy decoder 148. There are two schemes for the entropy decoder 148. In the first scheme, a run-length encoder (RLE) is used to encode the quantized coefficients. In the second scheme, the RLE is first used to encode the quantized coefficients, followed by a Huffman encoder processing the encoded output.

[0061] If a residual pattern (RM) has been selected, the residuals are further graded and L-1 residuals are selected to determine which residuals should be transformed 144 and encoded. Preferably, this is pre-formed before entropy encoding 148.

[0062] If the time selection mode is selected, the encoder will further modify the coefficients by subtracting the corresponding coefficients derived from the time buffer, i.e., the time prediction described below.

[0063] The input to L-1 decoding operation 118 includes the L-1 encoded residual, which passes through an entropy decoder, a dequantizer, and an inverse transform module. The operation performed by these modules is the inverse operation performed by the modules described above.

[0064] If the time selection mode has been selected, residuals can be partially predicted from the residuals of co-location from the time buffer.

[0065] If a 4×4 transform is used, the decoded residual can be fed into the deblocking filter module. The deblocking filter operates on each block of the transformed residual by applying a mask with specified weights. The general structure of the mask is as follows:

[0066]

[0067] in .

[0068] The output from the decoded (and deblocked, if applicable) L-1 residual and the underlying decoded video from Combination 120 is upsampled 122 to generate an upsampled reconstructed video. Upsampling may be optional and is transmitted in the byte stream.

[0069] The input to L-2 encoding operation 126 includes the L-2 residual obtained by the difference 124 between the upsampled reconstructed video and the input video 102. The L-2 residual is then transformed 154, quantized 156, and encoded 158, as further described below. Transformation 154, quantization 156, and encoding 158 are performed in the same manner as described with respect to L-1 encoding 114. If RM has been selected, the L-2 residuals are further graded and selected to determine which residuals should be transformed and encoded. L-2 encoding operation 126 may further include two additional processes as described below.

[0070] If the prediction coefficient mode is selected, the encoder will further modify the transformed coefficient C00 (e.g., the "average" or "A" coefficient for a 2×2 transform). If a 2×2 transform is used, C00 is modified by subtracting the value of the oversampled residuals from the transformed block whose prediction residuals are used. If a 4×4 transform is used, C00 is modified by subtracting the average of the four oversampled residuals from the transformed block whose prediction residuals are used.

[0071] If the time selection mode is selected, the encoder will further modify the coefficients by subtracting the corresponding coefficients derived from the time buffer, as described above.

[0072] The input to L-2 decoding operation 134 includes the encoded L-2 residual. The decoding process of the L-2 residual is performed by an entropy decoder 172, a dequantizer 174, and an inverse transform module 176. The operation performed by these modules is the inverse operation performed by the modules described above. If a time selection mode is selected, the residual can be partially predicted from the co-located residual from the time buffer.

[0073] The type of transformation performed by the inverse transform module 176 can be indicated by the transform type parameter received by the decoder. For example, a value of 0 for the transform type parameter can be used to indicate that a 2×2 oriented decomposition transformation will be performed, and a value of 1 for the transform type parameter can be used to indicate that a 4×4 oriented decomposition transformation will be performed.

[0074] In one example, the decoder receives a parameter indicating that the upsampling process 138 will be performed only in the horizontal direction. That is, the upsampling 138 of a single feature outputs a 2×1 block of features. This parameter will be described in further detail below. In this case, the transformation operation 176 applied to the 2×2 block of the array of dequantized coefficients is as follows:

[0075]

[0076] When upsampling 138 is performed only in the horizontal direction, the transformation operation 176 applied to the 4×4 block of the array of dequantized coefficients is as follows:

[0077]

[0078] The modified upsampling process 138 comprises two steps, the second step depending on the signaling received by the decoder. In the first step, the combination 136 of the decoded (and deblocked, if applicable) L-1 residual and the base decoded video 208 (L-1 reconstructed video) is upsampled 166 to generate an upsampled reconstructed video. If a prediction coefficient mode has been selected, the second step is performed. Specifically, the values ​​168 of the elements in the L-1 reconstructed values ​​of the 2×2 block from which the upsampled reconstructed video is derived are added 170 to the 2×2 block in the upsampled reconstructed video.

[0079] Figure 3 and 4 The encoding and decoding schemes use the tools described above as well as the following tools.

[0080] Based on the selected prediction residual pattern, the encoder further modifies the upsampled reconstructed video by applying prediction residuals (PR). The prediction residuals are based on the upsampled reconstructed video, and also on the reconstructed lower-resolution video before upsampling. Prediction residuals may also be referred to as prediction averages or modifiers, as described in more detail below.

[0081] If the time selection mode is selected, the encoder will further modify the coefficients by adding the corresponding coefficients derived from the time buffer.

[0082] Throughout the above, the term bitstream can be replaced by stream, byte stream, or NALU stream as needed.

[0083] Figure 4 Show Figure 2 An example of the modified upsampling process 138 used in the decoding process is shown. Figure 4 The modified upsampling process 138 is described in more detail to provide examples of how this process can be performed.

[0084] An example of the modified upsampling process 138 can be summarized using the following pseudocode:

[0085]

[0086] Where: nCurrS = size of the (nCurrS) x (nCurrS) array of features in the L-1 reconstructed video; xSrc, ySrc are the index values ​​of a given feature in the array of features in the L-1 reconstructed video; xDst, yDst are the index values ​​of a given feature in the array of features in the upsampled reconstructed video or the array of features in the modified upsampled reconstructed video; resFilteredSamples is the array of features in the L-1 reconstructed video; resUpsampledSamples is the array of features in the upsampled reconstructed video; and resModifiedUpsampledSamples is the modified array of features in the upsampled reconstructed video.

[0087] In this example, the input to the modified upsampling process 138 is a combination of the decoded (and unblocked, if applicable) L-1 residual and the underlying decoded video. This combination is represented by the variable resFilteredSamples in the pseudocode above. This combination can be referred to as the L-1 reconstructed video.

[0088] The L-1 reconstructed video is upsampled by 166 to generate an upsampled reconstructed video. In this example, upsampling by 166 is performed in both the vertical and horizontal directions, such that a single value in the L-1 reconstructed video is upsampled by 166 to generate a 2×2 block of features in the upsampled reconstructed video. Thus, in this example, the size of the array has increased from 1×1 in the L-1 reconstructed video to 2×2 in the upsampled reconstructed video due to upsampling by 166, and therefore increases along both dimensions of the array (vertical and horizontal). The 2×2 block of features in the upsampled reconstructed video is represented by the variable resUpsampledSamples in the pseudocode above.

[0089] Modifier 168 is calculated using the values ​​in the L-1 reconstructed video and a 2×2 block of features in the upsampled reconstructed video. Modifier 168 can also be referred to as the predicted mean. In this example, modifier 168 represents the difference between the values ​​in the L-1 reconstructed video and the mean of the values ​​in the 2×2 block of the upsampled reconstructed video.

[0090] Modifier 168 then adds 170 to each feature in the 2×2 block of the upsampled reconstructed video, thereby generating a modified 2×2 block of the features in the upsampled reconstructed video. The modified 2×2 block of the features in the upsampled reconstructed video is represented by the variable resModifiedUpsampledSamples in the pseudocode above.

[0091] Adding modifier 168 can be performed as part of the transform process 176. Adding modifier 168 as part of the modified upsampling process 138, rather than as part of transform process 176, reduces the complexity of the decoding process. For example, this method avoids the need for transform process 176 to additionally involve transforming the modifiers obtained from the encoded level 2 stream 128 and the dequantization coefficients. Adding modifier 168 as part of the modified upsampling process 138 is possible compared to adding modifier 168 as part of transform process 176 because transform process 176 involves linear transformation. Therefore, the decoded video 142 generated by adding modifier 168 as part of the transform process is the same as the decoded video 142 generated by adding modifier 168 as part of the modified upsampling process, but generated more efficiently.

[0092] Subsequently, 140 is added to the corresponding 2×2 block of the L-2 residual to the modified 2×2 block of the features in the upsampled reconstructed video output from the modified upsampling process 138 to generate a 2×2 block of the features in the decoded video 142.

[0093] Modifier 168 can be added based on received parameters indicating that modifier 168 will be added to a 2×2 block in the upsampled reconstructed video to generate a modified 2×2 block. In this example, if the parameters are not received, modifier 168 will not be added, and the 2×2 block in the upsampled reconstructed video generated from upsampling process 166 will be added to the corresponding 2×2 block of the L-2 residual by 140 to generate a 2×2 block of the elements of decoded video 142. Alternatively, the parameters can be received, but the parameters may indicate that modifier 168 will not be added. The value of the parameters may have a bit length of one bit. The parameters may be received once for a group of pictures associated with the encoded stream discussed above, the group of pictures being a collection of consecutive pictures within the decoded video stream. Modified upsampling process 138 may include retrieving an indicator from a header accompanying the data. The indicator may be retrieved from a predetermined position in the header, and the indicator may indicate one or more actions. The indicator can indicate whether the modified upsampling procedure 138 is performed.

[0094] In another instance, the modified upsampling process 138 can be summarized using the same variables as those defined in the pseudocode above:

[0095]

[0096] In this other instance, upsampling 166 is performed only in the horizontal direction, such that a single value in the L-1 reconstructed video is upsampled 166 times to generate a 2×1 block of features in the upsampled reconstructed video. Thus, the array size has increased from 1×1 in the L-1 reconstructed video to 2×1 in the upsampled reconstructed video due to upsampling 166, and therefore only along one direction (horizontally) of the array. In this case, modifier 168 represents the difference between the values ​​in the L-1 reconstruction and the average of the values ​​in the 2×1 block of features in the upsampled reconstructed video. Adding modifier 168 to each feature in the 2×1 block of the upsampled reconstructed video generates a modified 2×1 block of features in the upsampled reconstructed video.

[0097] The direction of upsampling 166 can be determined by the received parameter indicating the direction in which upsampling 166 will be performed. For example, a value of 1 for the parameter can be used to indicate that upsampling 166 is performed only in the horizontal direction, while a value of 0 for the parameter can be used to indicate that upsampling 166 is performed in both the vertical and horizontal directions.

[0098] In the decoding process described herein, the generation of decoded video 142 can be performed block by block. In this way, the generation of blocks of elements in frames of decoded video 142 can be performed without using another block of elements in the same frame of previously generated decoded video 142. For this reason, the modified upsampling process 138 can be performed in parallel for all blocks of elements in the frame, which is compared to performing the modified upsampling process 138 sequentially for each block of elements in the frame.

[0099] Other examples are described below.

[0100] Some other examples relate to a method for encoding an input video into multiple encoded streams such that the encoded streams can be combined to reconstruct the input video, the method comprising: receiving a full-resolution input video; downsampling the full-resolution input video to create a downsampled video; encoding the downsampled video using a first codec to create a base encoded stream; reconstructing the video from the encoded video to generate a reconstructed video; comparing the reconstructed video with the input video; and creating one or more additional encoded streams based on the comparison. The input video may be the downsampled video, in contrast to the reconstructed video.

[0101] According to one example method, comparing a reconstructed video with an input video includes: comparing the reconstructed video with a downsampled video to create a first residual set, and wherein creating the one or more additional encoded streams includes encoding the first residual set to create a first-level encoded stream.

[0102] Compared to reconstructed video, input video can be full-resolution input video and reconstructed video can be upsampled.

[0103] According to one example method, comparing a reconstructed video with an input video includes: upsampling the reconstructed video to generate an upsampled reconstructed video; and comparing the upsampled reconstructed video with a full-resolution input video to create a second residual set, wherein creating the one or more additional coded streams includes encoding the second difference to create a second-level coded stream.

[0104] Accordingly, in the example, the method can generate a base encoded stream, a first-level encoded stream, and a second-level encoded stream according to the example method defined above. Each of the first-level encoded stream and the second-level encoded stream may contain augmented data used by the decoder to enhance the encoded base stream.

[0105] According to one example method, the steps of encoding a first set of residuals include: applying a transformation to the set of residuals to create a set of coefficients; applying a quantization operation to the coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0106] According to one example method, the steps of encoding a second set of residuals include: applying a transformation to the second set of residuals to create a set of coefficients; applying a quantization operation to the coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0107] The transform used to encode the first and / or second set of residuals can be, for example, a discrete cosine transform or a wavelet transform. In alternative instances, the transform can be a small transform that decomposes the blocks of features into directional components (e.g., using a 2×2 kernel or a 4×4 kernel). For example, a 2×2 kernel could be an Adama transform. Further details regarding the transform can be found, for example, in patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. In another instance, the encoder can choose between different transforms to be used, such as between a 2×2 kernel and a 4×4 kernel. This allows for greater flexibility in how the residuals are encoded. The choice of transform can be based on analysis of the data to be transformed.

[0108] The first and second residual sets may have different transformations applied to them, and the selection can be predetermined or chosen during the process. The transformations used can be communicated in the header.

[0109] The quantization used to encode the first and / or second set of residuals can be, for example, linear quantization. A linear quantizer may use a variable-size dead zone. The encoding operation can be, for example, an entropy encoder, and may include run-length encoding and / or Huffman coding.

[0110] The residual can be the difference between two videos or frames.

[0111] The steps of encoding the first set of residuals may include: classifying the first set of residuals based on a preliminary analysis of the first set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0112] In one instance, the method includes analyzing a first set of residuals and, based on the analysis, performing or not performing the following steps: classifying the first set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0113] In one instance, the method includes analyzing a first set of residuals, and: classifying the first set of residuals; and selecting a subset of the residuals to be transformed and encoded, such that the classification and / or selection steps are performed differentially based on the analysis.

[0114] According to an instance method, the step of applying the transformation is performed on a selected subset of the residuals of the first residual set.

[0115] The steps of encoding the second residual set may include: classifying the second residual set based on a preliminary analysis of the second residual set; and selecting a subset of the residuals to be transformed and encoded.

[0116] In one instance, the method includes analyzing a second set of residuals and, based on the analysis, performing or not performing the following steps: classifying the second set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0117] In one instance, the method includes analyzing a second set of residuals, and: classifying the second set of residuals; and selecting a subset of the residuals to be transformed and encoded, such that the classification and / or selection steps are performed differentially based on the analysis.

[0118] According to an example method, the step of applying the transformation is performed on a selected subset of the residuals in the second residual set.

[0119] The encoded stream may be accompanied by one or more headers, which contain parameters indicating aspects of the encoding process to facilitate decoding. For example, the headers may include the codec used, the transforms applied, the quantization applied, and / or other decoding parameters.

[0120] In some instances, the quantization step may include analytical adaptation quantization based on the coefficients to be transformed and / or data (e.g., residual data). In some instances, the distribution used in the quantization step may be adapted.

[0121] The steps of encoding the first residual set may include: deriving the set of time coefficients from the time buffer; and subtracting the set of time coefficients from the set of coefficients.

[0122] The steps of encoding the second residual set may include: deriving the set of time coefficients from the time buffer; and subtracting the set of time coefficients from the set of coefficients.

[0123] The preceding text describes how the grading and selection steps can be applied to residual data, including subtracting time coefficients and adjusting quantization. Each of these steps can be predetermined and applied selectively, or applied based on analysis of the input video, downsampled video, reconstructed video, upsampled video, or any combination thereof to improve the overall performance of the encoder. These steps can be applied selectively based on a predetermined set of rules or deterministically based on performance analysis or feedback.

[0124] According to one example method, the first codec is a hardware-based codec, preferably AVC, HEVC, AV1, VP8, or VP9.

[0125] The instance method further includes sending the underlying encoded stream.

[0126] The instance method further includes sending the first-level encoded stream.

[0127] The instance method further includes sending a second-level encoded stream.

[0128] Some other examples relate to a method for decoding multiple encoded streams into a reconstructed output video, the method comprising: receiving a first base encoded stream; decoding the first base encoded stream according to a first codec to generate a first output video; receiving one or more additional encoded streams; decoding the one or more additional encoded streams to generate a residual set; and combining the residual set with the first video to generate a decoded video.

[0129] In one example, the method includes retrieving multiple decoding parameters from a header. These decoding parameters can indicate which program steps are included in the encoding process.

[0130] In one instance, the step of decoding the one or more additional encoded streams to generate a residual set includes: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to generate the residual set.

[0131] In one instance, the step of decoding the one or more additional encoded streams to generate a set of residuals includes: predicting a subset of residuals based on the residuals from co-location from the time buffer.

[0132] In one example, the method may include receiving a first-level encoded stream and receiving a second-level encoded stream. In this example, the step of decoding the one or more additional encoded streams to generate a residual set includes: decoding the first-level encoded stream to derive a first residual set; wherein the step of combining the residual set with a first video to generate a decoded video includes: combining the first residual set with a first output video to generate a second output video; upsampling the second output video to generate an upsampled second output video; decoding the second-level encoded stream to derive a second residual set; and combining the second residual set with the second output video to generate a reconstructed output video.

[0133] In one example, the step of upsampling a second output video to generate an upsampled second output video includes adding values ​​of elements from a first residual set of blocks from which the upsampled second output video is derived to the corresponding blocks in the upsampled second output video. The blocks may be 2×2 blocks. This adding step may be performed selectively based on predetermined values ​​or signals contained in the header.

[0134] In one instance, the steps of decoding the first-level encoded stream to derive the first residual set include: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to generate the first residual set.

[0135] In this example, the step of decoding the first-level encoded stream to derive the first set of residuals includes applying a deblocking filter configured to apply a mask to the residual blocks. The mask may be weighted according to a set of predefined weights.

[0136] In one instance, the steps of decoding the second-level encoded stream to derive the second residual set include: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to generate the second residual set.

[0137] The inverse transform operation can be the inverse of the operation defined above, or it can be a largely mirroring operation. That is, a 2×2 block or 4×4 block transform can be selectively applied. The transform can be detected by decoding methods or transmitted in the header.

[0138] If a 2×2 transformation is used, the coefficients can be modified by adding the values ​​of the residuals from the predicted transformed residual block. If a 4×4 transformation is used, the coefficients will be modified by adding the average of the four residuals.

[0139] The method may further include displaying or outputting the reconstructed output. In one example, the method described above further includes retrieving an indicator from the header accompanying the data.

[0140] Indicators can be retrieved from predetermined locations in the header, and these indicators can indicate one or more actions. Indicators can indicate whether the steps specified above for adding residuals and / or predicting residuals should be performed. Indicators can indicate whether the decoder should perform a specific operation, or is configured to perform a specific operation, in order to decode the bitstream. Indicators can indicate whether these steps have already been performed during the encoder stage.

[0141] Other examples relate to a device for encoding a dataset into an encoded dataset including a header and a payload. The device is configured to encode input video according to the steps described above. The device may include a processor configured to perform the methods of any of the foregoing aspects.

[0142] Other examples relate to an apparatus for decoding a dataset, including a header and a payload, into a reconstructed video. The apparatus is configured to decode the output video according to the steps described above. The apparatus may include a processor configured to perform the methods of any of the foregoing aspects.

[0143] Encoders and decoders are also available.

[0144] Other examples involve computer-readable media that, when executed by a processor, cause the processor to perform any of the methods described above.

[0145] The above embodiments should be understood as illustrative examples. Other examples are contemplated. It should be understood that any feature described with respect to any example may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other example, or in any combination of any other example. Furthermore, equivalents and modifications not described above may also be adopted without departing from the scope of the appended claims.

Claims

1. A method for decoding multiple encoded streams into a reconstructed output video, the method comprising: Receive the first-level encoded stream; Decode the first-level encoded stream to derive the first residual set; Receive a first output video, the first output video including the output of a base decoder applied to a base-level encoded stream; The first residual set is combined with the first output video to generate the second output video; Upsampling is performed on the second output video to generate an upsampled second output video, wherein the upsampling includes: Values ​​derived from elements in the first residual set from which the block of the upsampled second output video is derived are added to the block of the upsampled second output video, wherein the addition of the values ​​derived from the elements in the first residual set is based on received parameters indicating that the values ​​derived from the elements in the first residual set are to be added to the block of the upsampled second output video. Receive the second-level encoded stream; Decode the second-level encoded stream to derive the second residual set; and The second residual set is combined with the upsampled second output video to generate the reconstructed output video.

2. The method of claim 1, wherein the value derived from the element in the first residual set is based on a modifier representing the difference between the value from the second output video and the average value in the block of the upsampled second output video, wherein the value from the second output video represents a combination of the element in the first residual set and the corresponding element in the first output video, and adding the value derived from the element in the first residual set to the block of the upsampled second output video includes adding the modifier to the block of the upsampled second output video.

3. The method according to claim 1, wherein the bit length of the value of the parameter is one bit.

4. The method of claim 1, wherein the parameters are received once for a group of images associated with the plurality of encoded streams.

5. The method of claim 1, wherein a first block of elements from the frames in the reconstructed output video corresponding to the block in the second output video of the upsampled video is generated, without using a second block of elements from the frames in the reconstructed output video.

6. The method of claim 1, wherein decoding the first level encoded stream to derive the first residual set comprises: Apply first entropy decoding operation; Apply the first solution quantization operation; as well as The first transformation operation is applied to generate the first residual set.

7. The method of claim 1, wherein decoding the first level encoded stream to derive the first residual set includes applying a deblocking filter configured to apply a mask to the residual blocks, the mask being weighted according to a predefined set of weights.

8. The method of claim 1, wherein decoding the second-level encoded stream to derive the second residual set comprises: Apply second entropy decoding operation; A second dequantization operation is applied to generate a set of dequantized transform coefficients; as well as The second transformation operation is applied to the set of dequantized transformation coefficients to generate the second residual set.

9. The method of claim 8, further comprising receiving a transformation type parameter indicating the type of the second transformation operation to be applied.

10. The method according to claim 8 or 9, wherein the second transformation operation is a 2×2 directional decomposition transformation, the set of dequantized transformation coefficients represents a 2×2 block of elements from the set of dequantized transformation coefficients, and the second residual set represents a 2×2 block of elements from the second residual set.

11. The method according to claim 8 or 9, wherein the second transformation operation uses a 4×4 oriented decomposition transformation, the set of dequantized transformation coefficients represents a 4×4 block of elements from the set of dequantized transformation coefficients, and the second residual set represents a 4×4 block of elements from the second residual set.

12. The method of claim 11, wherein the value derived from the element in the first residual set is based on the average of four residuals derived from a 4×4 block of elements from the reconstructed output video.

13. The method of claim 1, wherein the block in the upsampled second output video is a 2×2 block of elements from the upsampled second output video.

14. The method of claim 1, wherein receiving the first output video comprises: Receive the base-level encoded stream; The base-level encoded stream is decoded according to the base decoder to generate the first output video.

15. A decoding device, comprising: The first decoder is configured to receive the first-level encoded stream; And decode the first level encoded stream to derive the first residual set; A first combiner is configured to: receive a first output video, the first output video including the output of a base decoder applied to a base-level coded stream; and combine the first residual set with the first output video to generate a second output video. An upsampler configured to upsample the second output video to generate an upsampled second output video, the upsampling comprising: Values ​​derived from elements in the first residual set from which the block of the upsampled second output video is derived are added to the block of the upsampled second output video, wherein the upsampler is configured to add the values ​​derived from the elements in the first residual set based on received parameters indicating that the values ​​derived from the elements in the first residual set are to be added to the block of the upsampled second output video; A second decoder is configured to: receive a second-level encoded stream; and decode the second-level encoded stream to derive a second residual set; and A second combiner is configured to combine the second residual set with the upsampled second output video to generate the reconstructed output video.

16. The decoding apparatus of claim 15, wherein the value derived from the element in the first residual set is based on a modifier representing the difference between the value from the second output video and the average value in the block of the upsampled second output video, wherein the value from the second output video is derived from a combination of the element in the first residual set and the corresponding element in the first output video, and adding the value derived from the element in the first residual set to the block of the upsampled second output video comprises adding the modifier to the block of the upsampled second output video.

17. A computer program comprising instructions that, when executed, cause a device to perform the method according to claim 1.

18. A computer-readable medium comprising the computer program of claim 17.

19. A bit stream comprising one or more of the following: The first level is encoded into a stream, which includes a first set of residuals; and The second-level flow includes the second set of residuals; in, The first residual set is configured to be combined with a first output video to generate a second output video, the first output video including the output of a base decoder applied to a base-level coded stream; The second residual set is configured to be combined with the upsampled second output video after upsampling of the second output video to generate a reconstructed output video, the upsampling including: Values ​​derived from elements in the first residual set from which the block in the upsampled second output video is derived are added to the block in the upsampled second output video, wherein the addition of the values ​​derived from the elements in the first residual set is based on received parameters indicating that the values ​​derived from the elements in the first residual set are to be added to the block in the upsampled second output video.

20. A method for encoding multiple streams to provide a reconstructed output video, the method comprising: Receive a first output video, the first output video including the output of a base decoder applied to a base-level encoded stream; A first-level encoded stream is generated, the first-level encoded stream including a first residual set, wherein the first residual set is configured to be combined with the first output video to generate a second output video; An upsampled second output video is generated by upsampling the second output video, wherein the upsampling includes: Values ​​derived from elements in the first residual set from which the block in the upsampled second output video is derived are added to the block in the upsampled second output video, wherein the addition of the values ​​derived from the elements in the first residual set is based on received parameters indicating that the values ​​derived from the elements in the first residual set will be added to the block in the upsampled second output video; and A second-level encoded stream is generated, the second-level encoded stream including a second residual set, wherein the second residual set is configured to be combined with the upsampled second output video to generate the reconstructed output video.

21. An encoding / decoding device, comprising: A first encoder is configured to receive a first output video, the first output video including the output of a base decoder applied to a base-level encoded stream; And generate a first-level encoded stream, the first-level encoded stream including a first residual set, the first residual set being configured to be combined with the first output video to generate a second output video; First combiner; An upsampler configured to upsample the second output video to generate an upsampled second output video, the upsampling comprising: Values ​​derived from elements in the first residual set from which the upsampled second output video is derived are added to the block in the upsampled second output video, wherein the upsampler is configured to add the values ​​derived from the elements in the first residual set based on received parameters indicating that the values ​​derived from the elements in the first residual set are to be added to the block in the upsampled second output video; and A second encoder is configured to receive a second-level encoded stream, the second-level encoded stream including a second set of residuals, wherein the second set of residuals is configured to be combined with the upsampled second output video to generate a reconstructed output video.

Images