Temporal Processing for Video Decoding

Abstract

A decoder (400) is configured to: receive a first output video and one or more additional encoded streams (416, 428); decode corresponding frames of the one or more additional encoded streams to derive corresponding residual sets; and combine the residual sets with the first output video (470) to generate a reconstructed output video (448). Each frame is divided into a plurality of tiles, and each tile is divided into a plurality of blocks. To decode the corresponding frame, the decoder is configured to: obtain (440, 446) a preliminary residual set from the one or more additional encoded streams; derive a set of temporal predictions using a temporal buffer (432); and combine the set of temporal predictions with the preliminary residual set (468). The decoder is configured to implement zeroing of values ​​of the set of temporal predictions at a frame level, a tile level, and a block level.

Description

Technical Field

[0001] The present invention relates to methods, apparatus, computer programs and computer-readable media for use in video coding techniques. Background Art

[0002] Compression and decompression of signals is a consideration in many known systems. Many types of signals, such as video, may be compressed and encoded for transmission, such as via a data communications network. When decoding such a signal, it may be desirable to increase the quality level of the signal and / or to recover as much information as possible contained in the original signal.

[0003] Some known systems employ scalable coding techniques. Scalable coding involves encoding a signal together with information to allow reconstruction of the signal at one or more different quality levels, depending on, for example, the capabilities of the decoder and the available bandwidth.

[0004] There are several considerations regarding the reconstruction of signals in a scalable coding system. One such consideration is the ability of an encoder and / or decoder to efficiently process information. The efficiency with which an encoder and / or decoder processes information can be a factor in the performance level of an encoder and / or decoder. Summary of the invention

[0005] Various aspects of the invention are set out in the accompanying claims.

[0006] Further features and advantages will appear from the following description, given by way of example only, made with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram illustrating an encoding process according to examples herein;

[0008] Figure 2 is a schematic diagram illustrating a decoding process according to examples herein;

[0009] Figure 3A and 3B are schematic diagrams each illustrating an encoding process according to an example herein; and

[0010] Figure 4A and 4B are schematic diagrams each illustrating a decoding process according to examples herein. DETAILED DESCRIPTION

[0011] This paper describes a hybrid backward-compatible decoding technique.

[0012] The examples described herein provide a flexible, adaptable, efficient, and computationally inexpensive coding format that combines different video coding formats, base codecs (eg, AVC, HEVC, or any other current or future codec) with coded data of at least two enhancement layers.

[0013] The general structure of the encoding scheme uses a downsampled source signal encoded with a base codec, adds a first level of correction data to the decoded output of the base codec to generate a corrected picture, and then adds another level of enhancement data to an upsampled version of the corrected picture.

[0014] Thus, the streams are considered to be a base stream and an enhancement stream.It is noteworthy that the base stream is generally expected to be decodable by a hardware decoder, while the enhancement stream is expected to be suitable for software processing implementation with suitable power consumption.

[0015] This structure creates multiple degrees of freedom, allowing great flexibility and adaptability to many situations, making the decoding format suitable for many use cases, including over-the-top (OTT) transmission, live streaming, live ultra-high-definition (UHD) broadcasting, etc.

[0016] Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders and also usable as a lower resolution output if deemed appropriate.

[0017] The codec formats use a minimal number of simple coding tools. When combined synergistically, they can provide visual quality improvements over full-resolution pictures encoded with the base codec, while making the way they can be used flexible.

[0018] Figure 1 A first example encoder 100 is shown. The components shown may also be implemented as steps of a corresponding encoding process.

[0019] In encoder 100, input full resolution video 102 is processed to generate various encoded streams. A first encoded stream (encoded base stream 110) is produced by feeding a downsampled version of the input video produced by downsampling 104 the input video 102 to a base encoder 106 (e.g., AVC, HEVC, or any other codec). A second encoded stream (encoded layer 1 stream 116) is produced by applying an encoding operation 114 to a residual obtained by taking the difference 112 between the reconstructed base codec video and the downsampled version of the input video. The reconstructed base codec video is obtained by decoding the output of the base encoder 106 with a base decoder 108. A third encoded stream (encoded layer 2 stream 128) is produced by processing 126 the residual obtained by taking the difference 124 between the upsampled version of the corrected version of the reconstructed base decoded video and the input video 102. A corrected version of the reconstructed base codec video is obtained by combining 120 the reconstructed base codec video with the residual obtained by applying the decoding operation 118 to the encoded level 1 stream 116 .

[0020] The layer 1 encoding operation 114 operates with an optional layer 1 temporal buffer 130, which can be used to apply temporal processing as described further below. The layer 2 encoding operation 126 also operates with an optional layer 2 temporal buffer 132, which can be used to apply temporal processing as described further below. The layer 1 temporal buffer 130 and the layer 2 temporal buffer 132 can operate under the control of a temporal selection component 134. The temporal selection component 134 can receive one or more of the input video 102 and the output of the downsampler 104 to select a temporal mode. This is explained in more detail in a later example.

[0021] Figure 2 A first example decoder 200 is shown. The components shown may also be implemented as steps of a corresponding decoding process. The decoder receives a signal transmitted by, for example, Figure 1 The three streams (encoded base stream 210, encoded layer 1 stream 216, and encoded layer 2 stream 228) generated by an encoder such as encoder 100 of FIG. 1 are combined 238 with a decoded residual obtained by decoding 240 the encoded layer 1 stream 216 and further combined 244 with a decoded residual obtained by applying a decoding operation 246 to the encoded layer 2 stream 228, together with a header 236 containing further decoding information. The encoded base stream 210 is decoded by a base decoder 208 corresponding to the base decoder used in the encoder, and its output is combined 238 with a decoded residual obtained by decoding 240 the encoded layer 1 stream 216. The combined video is upsampled 242 and further combined 244 with a decoded residual obtained by applying a decoding operation 246 to the encoded layer 2 stream 228.

[0022] Figure 3A and 3B Different variations of the second example encoder 300, 380 are shown. The second example encoder 300, 380 may include Figure 1 Implementation of a first example encoder 100. Figure 3A and 3B In the example of FIG. 1 , the encoding steps of the stream are described in more detail to provide an example of how the steps can be performed. Figure 3A A first variation is shown with temporal prediction provided only in the second level of the enhancement process (ie with respect to level 2 coding). Figure 3B A second variation is shown with temporal prediction performed in two enhancement levels (ie, levels 1 and 2).

[0023] The basic stream 310 is generally obtained by referring to Figure 1 The process explained above creates a video stream 310. That is, the input video 302 is downsampled 304 (i.e., the downsampling operation 304 is applied to the input video 302 to generate a downsampled input video). The downsampled video obtained by downsampling 304 the input video 302 is then encoded using a first base encoder 306 (i.e., the encoding operation is applied to the downsampled input video to generate an encoded base stream 310 using the first or base encoder 306). Preferably, the first or base encoder 306 is a codec suitable for hardware decoding. The encoded base stream 310 may be referred to as a base layer or base level.

[0024] As described above, the enhancement stream may include two streams. The first level of enhancement (described herein as "level 1") provides a set of correction data that can be combined with a decoded version of the base stream to generate a corrected picture. This first enhancement stream is Figure 1 3 as the encoded layer 1 stream 316. The enhancement stream may be generated by an enhancement encoder. The enhancement encoder may be different from the base encoder 306 used to generate the encoded base stream 310.

[0025] To generate the encoded level 1 stream 316, the encoded base stream 310 is decoded using the base decoder 308 (i.e., a decoding operation is applied to the encoded base stream 310 to generate a decoded base stream). A difference 312 is then created between the decoded base stream and the downsampled input video obtained by downsampling 304 the input video 302 (i.e., a subtraction operation 312 is applied to the downsampled input video and the decoded base stream to generate a first set of residuals). Here, the term residual is used in the same manner as is known in the art, i.e., the error between a reference frame and a desired frame. Here, the reference frame is the decoded base stream, and the desired frame is the downsampled input video. Thus, the residual used in the first enhancement layer may be considered as corrected video, since it 'corrects' the decoded base stream to the downsampled input video used in the underlying encoding operation.

[0026] The difference 312 is then encoded to generate an encoded level 1 stream 316 (ie, the encoding operation is applied to the first residual set to generate the first enhancement stream 316).

[0027] exist Figure 3A and 3B In an example embodiment of , the encoding operation includes several steps, each of which is optional and preferred and provides specific benefits.

[0028] In FIG. 3 , the steps include a transform step 336 , a quantization step 338 , and an entropy encoding step 340 .

[0029] Although not shown in the figure, in some examples, the encoding process identifies whether the residual scalability mode is selected. If the residual mode is selected, the residual scalability step may be performed (i.e., the residual scalability operation may be performed on the first step of the residual to generate a scalable residual set). The scalable residual set may be filtered so that not all residuals are encoded into the first enhancement stream 316 (or correction stream).

[0030] The first residual set or the hierarchical or filtered first residual set is then transformed 336, quantized 338, and entropy encoded 340 to produce the encoded level 1 stream 316 (i.e., depending on whether the hierarchical mode is selected, the transform operation 336 is applied to the first residual set or the filtered first residual set to generate a transformed residual set; the quantization operation 338 is applied to the transformed residual set to generate a quantized residual set; and the entropy encoding operation 340 is applied to the quantized residual set to generate the first level enhancement stream 316). Preferably, the entropy encoding operation 340 can be a Huffman encoding operation or a run-length encoding operation or both. Optionally, a control operation (not shown in the figure) can be applied to the quantized residual set to correct the effects of the hierarchical operation.

[0031] As described above, the enhancement stream may include a first level of enhancement 316 and a second level of enhancement 328. The first level of enhancement 316 may be considered a corrected stream. The second level of enhancement 328 may be considered another level of enhancement that converts the corrected stream into the original input video.

[0032] Another level of enhancement 328 is created by encoding another set of residuals, which are the differences 324 between the upsampled version of the decoded level 1 stream and the input video 302 .

[0033] In FIG. 3 , the quantized (or controlled) residual set is inverse quantized 342 and inverse transformed 344, and then a deblocking filter (not shown) is optionally applied to generate a decoded first residual set (i.e., the inverse quantization operation 342 is applied to the quantized first residual set to generate a dequantized first residual set; the inverse transform operation 344 is applied to the dequantized first residual set to generate a detransformed first residual set; and a deblocking filtering operation is optionally applied to the detransformed first residual set to generate a decoded first residual set). The deblocking filtering step is optional depending on the transform 336 applied, and includes applying a weighted mask to each block of the detransformed 344 first residual set.

[0034] The decoded base stream is combined 320 with the decoded first residual set (ie, a summation operation is performed 320 on the decoded base stream and the decoded first residual set to generate a recreated first stream). Figure 3A and 3B As shown in , the combination is then upsampled 322 (ie, an upsampling operation 322 is applied to the recreated first stream to generate an upsampled recreated stream).

[0035] The upsampled stream is then compared to the input video 302 creating another residual set (i.e., a difference operation 324 is applied to the upsampled recreated stream to generate another residual set). The other residual set is then encoded as an encoded level 2 enhancement stream 328 (i.e., the encoding operation is then applied to the other residual set to generate the encoded another enhancement stream 328).

[0036] As with the encoded level 1 stream 316, the encoding applied to the level 2 residual may include several steps. Figure 3A The steps are shown as temporal prediction (described further below), transform 348 , quantization 350 , and entropy encoding 352 .

[0037] Although not shown in the figure, in some examples, the encoding process identifies whether the residual scalability mode is selected. If the residual mode is selected, the residual scalability step may be performed (i.e., the residual scalability operation may be performed on another residual set to generate another scalable residual set). The other scalable residual set may be filtered so that not all residuals are encoded into another enhancement stream 328.

[0038] The other set of residuals or the other set of tiered residuals is then transformed 348 (i.e., the transform operation 348 is performed on the other set of tiered residuals to generate another set of transformed residuals). As shown, the transform operation 348 may utilize prediction coefficients or prediction means derived from the recreated first stream prior to upsampling 322. Additional information is as follows.

[0039] Figure 3A A variation of a second example encoder 300 is shown in which temporal prediction is performed as part of a level 2 encoding process. Temporal prediction is performed using a temporal selection component 334 and a level 2 temporal buffer 332. The temporal selection component 334 can determine a temporal processing mode as described in more detail below, and control the use of the level 2 temporal buffer 332 accordingly. For example, if temporal processing is not to be performed, the temporal selection component 334 can indicate that the contents of the level 2 temporal buffer 332 are to be set to zero. Figure 3B A variation of a second example encoder 380 is shown in which temporal prediction is performed as part of both the level 1 and level 2 encoding processes. Figure 3B , a level 1 time buffer 330 is provided in addition to a level 2 time buffer 332. Although not shown, further variations are possible in which time processing is performed at level 1 but not at level 2.

[0040] When choosing time forecast, Figure 3A The second example encoder 300, 380 of 3B or 3B can further modify the coefficients (i.e., the transformed residual output by the transform component) by subtracting the corresponding set of coefficients derived from the appropriate time buffer. The corresponding set of coefficients may include a set of coefficients of the same spatial region (e.g., the same decoding unit located within the frame) derived from the previous frame (e.g., coefficients for the same region of the previous frame). These coefficients may be derived or otherwise obtained from the time buffer. The coefficients obtained from the time buffer may be referred to as time coefficients in this article. Subtraction may be applied by subtraction components such as the third subtraction components 354 and 356 (for the corresponding levels 2 and 1). This time prediction step will be further described with respect to later examples. In general, when time prediction is applied, the encoded coefficients correspond to the difference between the frame and another frame of the stream. The other frame may be an earlier or later frame (or block in a frame) in the stream. Therefore, instead of encoding the residual between the upsampled recreated stream and the input video, the encoding process may encode the difference between the transformed frame in the stream and the transformed residual of the frame. Therefore, entropy can be reduced. Temporal prediction may be selectively applied for groups of coding units (referred to herein as "tiles") based on control information, and application of temporal prediction at the decoder may be applied by sending additional control information along with the encoded stream (eg, within a header).

[0041] like Figure 3A and 3B As shown in , when temporal prediction is active, each transformed coefficient may be:

[0042] Δ=F 当前 -F 缓冲器

[0043] Wherein the temporal buffer may store data associated with a previous frame. Temporal prediction may be performed for one color plane or for multiple color planes. In general, the subtraction may be applied for a video "frame" as an element-by-element subtraction, where the elements of the frame represent transformed coefficients, where the transform is applied relative to a particular n by n coding unit size (e.g., 2×2 or 4×4). The differences resulting from the temporal prediction (e.g., the deltas described above) may be stored in a buffer for use in subsequent frames. Thus, in effect, the residuals resulting from the temporal prediction are coefficient residuals relative to the buffer. Although Figure 3A and 3B Temporal prediction is shown to be performed after the transform operation, but it can also be performed after the quantization operation. This can avoid the need to apply the level 2 inverse quantization component 358 and / or the level 1 inverse quantization component 360. Figure 3A and 3B As shown in and described above, the output of the second example encoder 300, 380 after performing the encoding process is an encoded base stream 310 and one or more enhancement streams, which preferably include an encoded layer 1 stream 316 for enhancement of the first layer and an encoded layer 2 stream 328 for enhancement of another or second layer.

[0044] Figure 4A and 4B The corresponding changes of the second example decoders 400 and 480 are shown. The changes of the second example decoders 400 and 480 can be implemented to correspond to Figure 2 A first example decoder 200 of FIG. As can be clearly seen, the decoding steps and components are detailed in more detail to provide an example of how decoding can be performed. Figure 3A and 3B , Figure 4A shows a change in which temporal prediction is used only for the second level (ie, level 2), and Figure 4B A variation is shown in which temporal prediction is used for two layers, ie, layers 1 and 2. As previously mentioned, further variations are envisaged (eg, layer 1, but not layer 2), where the form of the configuration may be controlled using signaling information.

[0045] like Figure 4B , during the decoding process, the decoder 480 may parse the headers 436 (e.g., containing global configuration data, picture configuration data, and other data blocks) and configure the decoder based on those headers 436. To recreate the input video, the decoders 400, 480 may decode each of the base stream 410, the first enhancement stream 416, and the further enhancement stream 428. The frames of the streams may be synchronized and then combined to derive the decoded video 448.

[0046] In each decoding process, the enhancement stream may go through the steps of entropy decoding 450, 452, inverse quantization 454, 456, and inverse transformation 458, 460 to recreate the residual set.

[0047] Figure 4A and 4B The decoding process of includes retrieving an array of entropy decoded quantized coefficients representing the enhancement of the first level, and outputting an array of L-1 residuals. In this case, the entropy decoded quantized coefficients are obtained by applying the entropy decoding 450 operation to the encoded L-1 stream 416. Figure 4A and 4B The decoding process further includes retrieving an array of samples of the output of the base decoder 408 . Figure 4A and 4B The decoding process further includes applying a dequantization process 454 to the array of entropy decoded quantized coefficients to derive a set of dequantized coefficients, applying a transform process 458 to the set of dequantized coefficients, and optionally applying a filtering process ( Figure 4A and 4B 4 (not shown in the figure) to output an array of L-1 residuals representing the enhancement of the first level, which may be referred to as a preliminary residual set. In this case, a dequantization process 454 is applied to the entropy decoded quantized coefficients of the corresponding block of the frame of the encoded level 1 stream 416, and a transform process 458 (which may be referred to as an inverse transform operation) is applied to the output of the dequantization process 454 of the corresponding block of the frame. Figure 4A and 4B The decoding process of then further includes recreating the picture by combining 462 the array of L−1 residuals with the array of samples of the output of the base decoder 408 . Figure 4A and 4B The decoding process of includes applying a transform process 458 from a set of predetermined transform processes according to the signaled parameters. For example, the transform process 458 can be applied to a 2×2 coding unit or a 4×4 coding unit. A coding unit may be referred to herein as a block of elements in an array, in this case an array of L-1 residuals.

[0048] Figure 4A and 4B The decoding process of comprises retrieving an array of entropy-decoded quantized coefficients representing the enhancement of another level, and outputting an array of residuals. Figure 4A and 4B In the decoding process shown in , the other level of enhancement is the second level of enhancement, and the array of residual output is an array of L-2 residuals. Figure 4A and 4B The method further comprises retrieving an array of enhanced L-1 residuals of the first level corresponding to an array of quantized coefficients representing enhanced entropy decoding of another level. Figure 4A and4B The method further includes applying an upsampling process 464 to the array of enhanced residuals of the first level. Figure 4A and 4B In , an upsampling process 464 is applied to the combination of the array of enhanced L-1 residuals of the first level and the corresponding array of samples of the output of the base decoder 408 .

[0049] exist Figure 4A and 4B In the embodiment of the present invention, the upsampling process 464 is a modified upsampling process in which a modifier is added to the residual. The step of adding the modifier can be performed as part of the transform process 460. Alternatively, because the transform process 460 involves a linear transform, the step of adding the modifier can be performed as part of the modified upsampling process 464, such as Figure 4A and 4B The step of adding the modifier thus results in a modification of the residual. The modification may be performed based on the position of the residual in the frame. The modification may be a predetermined value.

[0050] exist Figure 4A In , temporal prediction is applied during layer 2 decoding. Figure 4A In the example of , the temporal prediction is controlled by the temporal prediction component 466. In this variation, the control information for the temporal prediction is extracted from the coded level 2 stream 428, as indicated by the arrow from the stream to the temporal prediction component 466. Figure 4B In other implementations shown in , the control information for temporal prediction may be sent separately from the encoded layer 2 stream 428, for example in a header 436. A temporal prediction component 466 controls the use of the layer 2 temporal buffer 432, for example it may determine the temporal mode and control temporal refreshing, as described with reference to later examples. The contents of the temporal buffer 432 may be updated based on data of a previous frame of residuals. When the temporal buffer 432 is applied, the contents of the buffer are added 468 to the second residual set. In Figure 4A In the example, the contents of the time buffer 432 are added 468 to the level 2 decoding component 446 (which is in Figure 4A460). The temporally corrected second residual set is then combined 470 with the output of upsampling 464 to generate decoded video 448. Decoded video 448 is at a level 2 spatial resolution, which may be higher than level 1 spatial resolution. The second residual set applies the corrections to the (viewable) upsampled reconstructed video, where the corrections add back in detail and improve the sharpness of lines and features.

[0051] The transform processes 458, 460 may be selected from a set of predetermined transform processes according to signaled parameters.For example, the transform process 460 may be applied to a 2x2 block of elements in an array of L-2 residuals or a 4x4 block of elements in an array of L-2 residuals.

[0052] Figure 4B A variation of a second example decoder 480 is shown. In this case, temporal prediction control data is received by a temporal prediction component 466 from header 436. Temporal prediction component 466 controls both level 1 and level 2 temporal prediction, but in other examples, separate control components may be provided for both levels as desired. Figure 4B 468 to the output of the level 2 decoding component 446 may be fed back to be stored in the level 2 temporal buffer 432 for the next frame (for clarity, from Figure 4A The feedback is omitted). Also shown is a level 1 temporal buffer 430 that operates in a similar manner to the level 2 temporal buffer 432 described above, and the feedback loop for the buffer is shown in this figure. The contents of the level 1 temporal buffer 430 are added to the level 1 residual processing pipeline via a summation 472. Again, the location of this summation 472 may vary along the level 1 residual processing pipeline depending on where temporal prediction is applied (e.g., if temporal prediction is applied in transformed coefficient space, it may be located before the level 1 inverse transform component 458).

[0053] Figure 4B Two ways are shown in which temporal control information can be signaled to the decoder. The first way is via header 436, as described above. The second way, which can be used as an alternative or additional signaling path, is via data encoded within the residual itself. Figure 4B A case is shown where data 474 may be encoded into the HH transform coefficients and thus may be extracted after entropy decoding 452. This data 474 may be extracted from the level 2 residual processing pipeline and passed to the temporal prediction component 466.

[0054] Each enhancement stream or both enhancement streams may be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Layer Units (NALUs). A NALU is intended to encapsulate an enhancement bitstream in order to apply the enhancement to the correct base reconstructed frame. A NALU may, for example, contain a reference index to a NALU containing the base decoder reconstructed frame bitstream to which the enhancement must be applied. In this way, the enhancement may be synchronized to the base stream and the frames of each bitstream combined to produce the decoded output video (i.e., the residual of each frame of the enhancement level is combined with a frame of the base decoded stream). A group of pictures may represent multiple NALUs.

[0055] Each frame may consist of three different planes representing different color components, for example, each component of a three-channel YUV video may have a different plane. Each plane may thus have enhanced residual data for a given level, for example, a Y plane may have a set of level 1 residual data and a set of level 2 residual data. In some cases, such as for a monochrome signal, there may be only one plane; in this case, the terms frame and plane may be used interchangeably. The level 1 residual data and the level 2 residual data may be partitioned as follows. The residual data is divided into blocks, the size of which depends on the size of the transform used. If a 2×2 directional decomposition transform is used, the blocks are, for example, 2×2 blocks of elements, or if a 4×4 directional decomposition transform is used, the blocks are 4×4 blocks of elements. A patch is a group of blocks covering a region of a frame (e.g., an M by N region, which may be a square region). A patch is, for example, a 32×32 patch of elements. In this way, each frame in the encoded stream may be divided into multiple patches, and each patch of the multiple patches may be divided into multiple blocks. For color video, each frame may be partitioned into multiple planes, where each plane is divided into multiple tiles, and each tile of the multiple tiles is divided into multiple blocks.

[0056] It is mentioned above how a collection of processing components or tools can be applied to each of the enhancement streams (or input videos 102, 302) throughout the process. An overview of each of the tools and their use in the following is provided. Figure 1 to the functionality within the overall process shown in 4.

[0057] The downsampling process 104, 304 is applied to the input video 102, 302 to produce a downsampled video to be encoded by the base encoder 106, 306. The downsampling 104, 304 may be done in both the vertical and horizontal directions, or alternatively only in the horizontal direction.

[0058] The input to the L-1 encoding operation 114 includes an L-1 residual obtained by taking the difference 112, 312 between the decoded output of the base decoder 108, 308 and the downsampled video obtained by downsampling 104, 304 the input video 102, 302. The L-1 residual is then transformed 336, quantized 338, and encoded 340, as further described below. The transform 336 outputs transform coefficients (i.e., the transformed L-1 residual).

[0059] There are two types of transforms that can be used in the transform process 336. Both utilize a small kernel that is applied directly to the residuals that remain after the stage of applying predictive averaging.

[0060] The first transform has a 2×2 kernel applied to a 2×2 block of the residual. The resulting coefficients are as follows:

[0061]

[0062] The second transform has a 4×4 kernel applied to a 4×4 block of the residual. The resulting coefficients are as follows:

[0063]

[0064] If downsampling and / or upsampling is performed only in the horizontal direction (e.g., certain elements are set to 0), a suitably adapted transform may also be applied. If a Hadamard transform is used, such as shown in the example matrices above, the decoding or inverse transform may use the same matrix, such as the Hadamard matrix is ​​its own inverse. In this case, for example, an (MxN)x1 array of residuals R for an MxN block may be derived at the decoder from a corresponding (MxN)x1 array of coefficients C using R=H*C, where H is equal to one of the Hadamard matrices shown above.

[0065] The coefficients are then quantized using a linear quantizer 338. The linear quantizer may use variable sized dead zones. The linear quantizer may use dead zones of different sizes than the quantization step and the non-centered dequantization offset.

[0066] The quantized coefficients are encoded using an entropy coder 340. There are two schemes for entropy encoding 340. In the first scheme, the quantized coefficients are encoded using a run length encoder (RLE). In the second scheme, the quantized coefficients are first encoded using RLE, and then the encoded output is processed using a Huffman encoder. This can advantageously take advantage of RLE encoding the long stream of zeros that are often found with the transformed residual, and then further advantageously use the Huffman encoder to encode the different frequencies of the quantized values ​​(e.g., which often have a reduced number because the values ​​increase due to the distribution of the residual and its linear transform values).

[0067] If residual mode (RM) has been selected, the L-1 residuals are further ranked and selected in order to determine which residuals should be transformed 336 and encoded. Preferably, this is preformed prior to entropy encoding 340.

[0068] If the temporal selective mode is selected for L-1 encoding, the encoder will further modify the coefficients by subtracting the corresponding coefficients derived from the level 1 temporal buffer 130, 330, ie the temporal prediction described below.

[0069] The input to the L-1 decoding operation 118 includes the L-1 encoded residual, which passes through the entropy decoder 450, the dequantizer 454, and the inverse transform module 458. The operations performed by these modules are the inverse operations performed by the modules described above.

[0070] If the temporal selection mode has been selected for L-1 encoding, the residual may be partially predicted from the co-located residual from the level 1 temporal buffer 130, 330. The co-located residual may be referred to herein as temporal prediction.

[0071] If a 4x4 transform is used, the decoded residual can be fed to a deblocking filter module. The deblocking filter operates on each block of the transformed residual by applying a mask whose weights can be specified. The general structure of the mask is as follows:

[0072]

[0073] Where 0≤α≤1 and 0≤β≤1.

[0074] The output from combining 120, 320 the decoded (and deblocked, if applicable) L-1 residual and base decoded video is upsampled 122, 322 in order to generate upsampled reconstructed video. Upsampling may be optional and signaled in the byte stream.

[0075] The input to the L-2 encoding operation 126 includes the L-2 residual obtained by taking the difference 124, 324 between the upsampled reconstructed video and the input video 102, 302. The L-2 residual is then transformed 348, quantized 350, and encoded 352, as further described below. The transformation 348, quantization 350, and encoding 352 are performed in the same manner as described with respect to the L-1 encoding 114. As explained with reference to the L-1 encoding 114, the transformation 348 outputs the transform coefficients (i.e., the transformed L-2 residual). If RM has been selected, the L-2 residuals are further graded and selected in order to determine which residuals should be transformed and encoded. The L-2 encoding operation 126 may further include two additional processes as described below.

[0076] If the prediction coefficient mode is selected, the encoder will further modify the transformed coefficients C00 (e.g., the "average" or "A" coefficient for a 2x2 transform). If a 2x2 transform is used, C00 will be modified by subtracting the value of the upsampled residual of the transformed block from which the prediction residual is obtained. If a 4x4 transform is used, C00 will be modified by subtracting the average of the four upsampled residuals of the transformed block from which the prediction residual is obtained.

[0077] If the temporal selection mode is selected for L-2 encoding, the encoder will further modify the coefficients by subtracting the corresponding coefficients derived from the level 2 temporal buffer 132, 332, as described above.

[0078] The input to the L-2 decoding operation 246, 446 includes the encoded L-2 residual. The decoding process of the L-2 residual passes through the entropy decoder 452, the dequantizer 456, and the inverse transform module 460. The operations performed by these modules are the inverse operations performed by the modules described above. If the temporal selection mode has been selected for L-2 encoding, the residual can be partially predicted from the co-located residual from the level 2 temporal buffer 132, 332. The co-located residual may be referred to as temporal prediction herein.

[0079] The modified upsampling process 242, 464 comprises two steps, the second step depending on the signaling received by the decoder. In the first step, the combination 238, 462 of the decoded (and deblocked, if applicable) L-1 residual and the base decoded video 208, 408 (L-1 reconstructed video) is upsampled to generate an upsampled reconstructed video. If the prediction coefficient mode has been selected, the second step is implemented. Specifically, the values ​​of the elements in the L-1 reconstruction values ​​of the 2×2 block in the upsampled reconstructed video from which they are derived are added to the 2×2 block in the upsampled reconstructed video.

[0080] Throughout the above, the term bitstream may be replaced by stream or byte stream or NALU stream as required.

[0081] refer to Figure 4A and 4B , the following examples relate to a temporal prediction process applied during layer 2 decoding. However, it should be appreciated that the following temporal prediction process may additionally or alternatively be applied during layer 1 decoding.

[0082] In this example, the decoder 400, 480 is configured to receive a temporal_enabled parameter that specifies whether temporal prediction should be used when decoding a picture. The temporal_enabled parameter may be referred to herein as a first parameter having a first value indicating that temporal processing is enabled. In this case, the temporal_enabled parameter indicates whether the decoder 400, 480 will refresh the value of the temporal prediction derived from the contents of the temporal buffer 432. The value of the temporal_enabled parameter may have a bit length of one bit. In this example, a value of 1 specifies that temporal prediction will be used when decoding a picture, and a value of 0 specifies that temporal prediction will not be used when decoding a picture. The temporal_enabled parameter may be received once for a group of pictures associated with the encoded stream discussed above, the picture group being a collection of consecutive pictures within a coded video stream.

[0083] In this example, the decoder 400, 480 is configured to receive a temporal_refresh_bit parameter that specifies whether the temporal buffer 432 should be refreshed for a frame. If a frame includes multiple planes, refresh may be applied for all planes in the frame (i.e., for the frame including the planes). Refreshing the temporal buffer 432 may include setting the value from the temporal buffer 432 to zero. In this way, when the contents of the temporal buffer 432 are added 468 to the second residual set, the second residual set will be unchanged in the same manner as if the temporal buffer 432 was not applied. The value of the temporal_refresh_bit parameter may have a bit length of one bit. In this example, a value of 1 specifies that the temporal buffer 432 will be refreshed for a frame, and a value of 0 indicates that the temporal buffer 432 will not be refreshed for a frame. The temporal_refresh_bit parameter may be received once for each picture in the decoded video stream. The temporal_enabled and temporal_refresh_bit parameters may be included in temporal control information signaled to the decoder, for example, via the header 436 as described above.

[0084] In this example, if the variable temporal_enabled is equal to 1 and temporal_refresh_bit is equal to 0, then the temporal prediction process is called as specified below.

[0085] The input to the called time prediction process is:

[0086] • Position (xTbP, yTbP) which specifies the top left sample of the current luma or chroma transform block relative to the top left luma or chroma sample of the current picture. (xTbP, yTbP) can be relative to either luma or chroma plane, depending on which plane the transform coefficient belongs to.

[0087] • Parameter nTbS, which specifies the size of the current transform block. For example, nTbS is equal to 2 when a 2x2 directional decomposition transform is to be used in the transform process described above, and nTbS is equal to 4 when a 4x4 directional decomposition transform process is to be used.

[0088] · A parameter temporal_tile_intra_signalling_enabled, which specifies whether temporal tile prediction should be used when decoding tiles of an element. The parameter temporal_tile_intra_signalling_enabled may be referred to herein as a third parameter having a third value. The value of the temporal_tile_intra_signalling_enabled parameter may have a bit length of one bit. In this example, if the value of the temporal_tile_intra_signalling_enabled parameter is equal to 1, the tile temporal prediction process will be enabled. The temporal_tile_intra_signalling_enabled parameter may be received once for a group of pictures associated with the encoded stream discussed above. The temporal_tile_intra_signalling_enabled parameter may be included in temporal control information signaled to a decoder, for example, via the header 436. This parameter may indicate whether temporal signaling involving tiles may be provided, wherein the signaling is encapsulated within the coefficient values ​​(i.e., "within" or within the data) as described below. If this parameter is set to 1, the first block in a tile may carry the temporal refresh signaling for the tile (eg, via coefficient values ​​within the first block).

[0089] • An array TransCoeffQ of size (nTbS)x(nTbS) containing a block of entropy decoded quantized coefficients with elements TransCoeffQ[x][y].

[0090] The output to this process is an (nTbS)x(nTbS) array of modified TransCoeffQ coefficients and an (nTbS)x(nTbS) array tempPredSamples with elements tempPredSamples[x][y]. The array of tempPredSamples represents the set of temporal predictions derived using the temporal buffer 432.

[0091] In this example, the following sequential steps apply:

[0092] 1. If the parameter temporal_tile_intra_signalling_enabled is equal to 1, the temporal tile prediction process will be used when decoding tiles of elements. In the temporal tile prediction process, if xTbP>>5 is equal to 0, yTbP>>5 is equal to 0 (where "x>>y" is an arithmetic right shift of the two's complement integer representation of the x times y binary number) and TransCoeffQ[nTbs-1][0]&0x1 is equal to 1 (where "&" represents the bitwise AND operator), the tile temporal refresh process is called with the position (xTbP, yTbP) as its input. The right shift allows elements of consecutive tiles to be skipped, for example a 32×32 tile has a length and width of 2. 5, in this case, the 5-bit shift equal to zero indicates that the current position is a multiple of 32 and therefore relates to the first block of the tile. Next, in this case, the first block in the tile is used to signal the tile temporal refresh. The output of the tile temporal refresh process is that the contents of the temporalBuffer of the tile at position (xTbP, yTbP) are set to zero. In this example, temporalBuffer represents the temporal buffer 432 storing data associated with the second frame, which is not the current frame from which the value of the temporal prediction can be derived. As such, the second frame can be an earlier or later frame in the stream. TransCoeffQ[nTbS-1][0]&0x1 is a parameter of the current transform block with a certain value, which indicates whether the value of the set of temporal predictions derived from the temporal buffer 432 of the corresponding block will be refreshed. In the current case, the parameter is "carried" within the transform coefficient value. An "&" (i.e., logical "and") operation with a bit value of 1 (e.g., 0x1) simply converts any non-zero value to an output bit value of 1 (and any zero value remains zero). In general, the TransCoeffQ[nTbS-1][0]&0x1 parameter may be referred to herein as a second parameter having a second value that provides temporal signaling, which in this case indicates whether the values ​​in the temporal buffer 432 associated with a given tile are to be refreshed. In this case, the parameter is used to indicate a tile refresh; in the example below, it is used to indicate a block refresh. Refreshing may involve setting the values ​​of the corresponding tile or block to zero. In this example, the value of the TransCoeffQ[nTbS-1][0]&0x1 parameter has a bit length of one bit, and a value of 1 indicates that the values ​​in the temporal buffer 432 are to be refreshed. In the temporal tile prediction process, a tile temporal refresh process is performed for the tile of the temporal buffer 432 in response to receiving the parameter TransCoeffQ[nTbS-1][0]&0x1 for the block within the tile. In other words, the decoder is configured to refresh the temporal buffer 432 of the tile in response to receiving the parameter TransCoeffQ[nTbS-1][0]&0x1 (second parameter) for the specified block within the tile. In this way, the refresh of a tile can be signaled by a parameter in a single block within the tile, rather than having to signal refresh individually for every other block within the tile. In this example, the decoder is therefore configured to refresh the value of the temporal prediction of the tile in response to receiving both the second parameter (TransCoeffQ[nTbS-1][0]&0x1), whose second value indicates that the decoder is to refresh the value of the temporal prediction of the block in case 2) below, and the third parameter (temporal_tile_intra_signalling_enabled), whose second value indicates that the decoder is to refresh the value of the temporal prediction of the block, and the third parameter indicating that receiving the specified block with a second value indicates that the value in the temporal buffer 432 associated with the tile is to be refreshed.The parameter TransCoeffQ[nTbS-1][0]&0x1 is an example of temporal control information that may be signaled to the decoder via data encoded within the encoded residual itself, so that this temporal control information may be extracted after entropy decoding 452. For example, this parameter (the second parameter) may be obtained from one of the coefficient sets of the block.

[0093] 2. In the case where no temporal tile prediction process occurs, if TransCoeffQ[nTbs-1][0]&0x1 is equal to 0, then tempPredSamples[x][y]=

[0094] temporalBuffer[xTbP+x][yTbP+y], where x and y are in the range [0, nTbS-1]. Otherwise, tempPredSamples[x][y] are all set to 0. In this way, for each corresponding block, the value of the temporal prediction of the corresponding block derived from the temporal buffer 432 is refreshed in response to obtaining the second parameter (TransCoeffQ[nTbS-1][0]&0x1 parameter) from the data of the one or more coded streams (in this case, the L-2 coded stream) from the corresponding block. In response to determining that the second parameter of the corresponding block has a second value that provides the temporal signaling of the corresponding block, the value of the temporal coefficient of the corresponding block is refreshed. In this case, refreshing involves setting the value of the temporal prediction of the corresponding block to zero. In this case, refreshing of the set of values ​​of the temporal prediction of each block is indicated individually for each block. Otherwise, if TransCoeffQ[nTbS-1][0]&0x1 is equal to 0, the temporal buffer 432 is not refreshed, and the value of the set of temporal predictions of the corresponding block is derived from the temporal buffer 432 of the corresponding block.

[0095] 3. The value of TransCoeffQ[nTbs-1][0] is set to TransCoeffQ[nTbs-1][0]>>1. If TransCoeffQ[nTbs-1][0] has a binary value of 0 or 1, this operation sets the transform coefficient value to 0 for subsequent operations (e.g., a later inverse transform). This means that a value of 1 used for signaling does not translate into image artifacts in the block (a residual value of 0 would be just as unchanged).

[0096] The array tempPredSamples of size (nTbS) x (nTbS) is added to the (nTbS) x (nTbS) array resSamples, and the resSamples array is stored to temporalBuffer at position (xTbP, yTbP). In this example, resSamples represents the second set of residuals, and tempPredSamples represents a set of temporal predictions, which may be derived from the contents of the temporal buffer 432, e.g., depending on temporal signaling received at the decoder, as described above. In this way, if temporal processing is enabled and this is signaled to be applied at the block level, the contents of the temporal buffer 432 are added 468 to the second set of residuals to generate a temporally corrected second set of residuals, which is then stored to the temporal buffer 432.

[0097] In the examples herein, a set of temporal predictions derived from a temporal buffer is combined with a set of residuals for each of a plurality of blocks to perform temporal prediction. This, for example, involves obtaining respective prepared residual elements of a set of prepared residuals for respective elements of a block, and obtaining respective temporal predictions associated with the elements, wherein the temporal predictions may be set to zero in certain circumstances, or derived from the contents of the temporal buffer, which in turn may be zeroed by specific temporal signaling received at the decoder. The respective prepared residual elements may then be combined with the temporal predictions for the blocks to output a set of reconstructed residual values ​​(e.g., layer 2 residual values) for combination with an upsampled reconstructed signal from a lower layer (e.g., layer 1 and base layer).

[0098] In the examples described herein, the decoders 400, 480 are configured to provide zeroing of values ​​of a set of temporal predictions at three levels: at the frame level; at the tile level; and at the block level. For example, the temporal_refresh_bit described above may be used to indicate a refresh of the temporal buffer 432 at the frame level (e.g., zeroing of temporal buffer values), where all values ​​in the temporal buffer, e.g., relating to a complete frame, are set to zero. This may then provide zeroing of temporal prediction values, where these values ​​are later derived from the temporal buffer 432. At the block level, temporal signaling indicating whether the temporal prediction values ​​will be derived from the temporal buffer 432 or set to zero may be carried within the transform coefficient values ​​(e.g., the values ​​of coefficients such as HH of a 4×4 transform prior to inverse transform). At the tile level, parameters such as temporal_tile_intra_signalling_enabled may be used to indicate that block-level signaling for a block in a tile will be used to indicate a refresh of the temporal buffer 432 for the tile, i.e., block-level signaling for a block in the tile becomes tile-level signaling. When the time buffer 432 is flushed, the value in the time buffer is set to zero, which thus means that for subsequent blocks in the same tile, the value from the time buffer applied will be zero, regardless of block level signaling.

[0099] This approach provides flexibility with respect to the refreshing of the temporal buffer 432. For example, the temporal buffer 432 may be refreshed at the frame level for a first frame and at the tile level for at least one tile of a second frame, and the temporal prediction may be zeroed at the block level for at least one block of a third frame. At the block level, the zeroing of the temporal prediction may be considered an equivalent operation to zeroing the temporal buffer 432, both operations resulting in an element to which a zero value is applied at the addition 468. For example, the decoder 400, 480 may be configured to refresh the value of the temporal buffer 432 for a first tile of a frame, and apply zero to the temporal prediction for a first block of a second tile of the same frame, while applying a non-zero value to the temporal prediction for a second block of the second tile.

[0100] In the decoding process described herein, the generation of the decoded video may be performed block by block. In this way, the generation of a block of an element in a frame of the decoded video may be performed without using another block of the element in the same frame of the decoded video that was previously generated. For this reason, the temporal prediction process may be performed in parallel for all blocks of the element in the frame, as opposed to performing the temporal prediction process sequentially for each block of the element in the frame.

[0101] Other examples are described below.

[0102] Some other examples relate to a method of encoding an input video into multiple encoded streams so that the encoded streams can be combined to reconstruct the input video, the method comprising: receiving a full-resolution input video; downsampling the full-resolution input video to create a downsampled video; encoding the downsampled video using a first codec to create a base encoded stream; reconstructing the video from the encoded video to generate a reconstructed video; comparing the reconstructed video to the input video; and creating one or more additional encoded streams based on the comparison. The input video may be a downsampled video compared to the reconstructed video.

[0103] According to an example method, comparing the reconstructed video to the input video includes comparing the reconstructed video to the downsampled video to create a first residual set, and wherein creating the one or more further encoded streams includes encoding the first residual set to create a first level encoded stream.

[0104] The input video may be a full resolution input video and the reconstructed video may be upsampled compared to the reconstructed video.

[0105] According to an example method, comparing the reconstructed video with the input video includes: upsampling the reconstructed video to generate an upsampled reconstructed video; and comparing the upsampled reconstructed video with the full-resolution input video to create a second residual set, and wherein creating the one or more additional encoded streams includes encoding the second difference to create a second level encoded stream.

[0106] Accordingly, in an example, the method may generate a base coded stream, a first-level coded stream, and a second-level coded stream according to the example method defined above. Each of the first-level coded stream and the second-level coded stream may contain enhancement data used by a decoder to enhance the coded base stream.

[0107] According to an example method, the step of encoding the first set of residuals includes applying a transform to the set of residuals to create a set of coefficients; applying a quantization operation to the coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0108] According to an example method, encoding the second set of residuals includes applying a transform to the second set of residuals to create a set of coefficients; applying a quantization operation to the coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0109] The transform used to encode the first and / or second residual sets may be, for example, a discrete cosine transform or a wavelet transform. In an alternative example, the transform may be a small transform that decomposes a block of elements into directional components (e.g., using a 2×2 kernel or a 4×4 kernel). For example, the 2×2 kernel may be a Hadamard transform. For more details about the transform, see, for example, patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. In another example, the encoder may choose between different transforms to be used, such as between a 2×2 kernel and a 4×4 kernel. This enables greater flexibility in the way the residuals are encoded. The choice of transform may be based on an analysis of the data to be transformed.

[0110] The first set of residuals and the second set of residuals may have different transforms applied to them, and the selection may be predetermined or selected during the process.The transform used may be signaled in the header.

[0111] The quantization used to encode the first and / or second residual set may be, for example, linear quantization. A linear quantizer may use a variable size dead zone. The encoding operation may be, for example, an entropy encoder and may include run length encoding and / or Huffman encoding.

[0112] The residual may be the difference between two videos or frames.

[0113] The step of encoding the first set of residuals may include: ranking the first set of residuals based on a pre-analysis of the first set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0114] In one example, the method includes analyzing a first set of residuals, and based on the analysis, performing or not performing the following steps: ranking the first set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0115] In an example, the method comprises analyzing a first set of residuals, and: ranking the first set of residuals; and selecting a subset of residuals to be transformed and encoded, such that the steps of ranking and / or selecting are performed differentially based on the analysis.

[0116] According to an example method, the step of applying the transform is performed on a selected subset of residuals of the first set of residuals.

[0117] The step of encoding the second set of residuals may include: ranking the second set of residuals based on a pre-analysis of the second set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0118] In one example, the method includes analyzing the second set of residuals, and based on the analysis, performing or not performing the following steps: ranking the second set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0119] In an example, the method comprises analyzing a second set of residuals, and: ranking the second set of residuals; and selecting a subset of residuals to be transformed and encoded, such that the steps of ranking and / or selecting are performed differentially based on the analysis.

[0120] According to an example method, the step of applying the transform is performed on a selected subset of the residuals of the second set of residuals.

[0121] The encoded stream may be accompanied by one or more headers, which contain parameters indicating aspects of the encoding process to facilitate decoding. For example, the header may include the codec used, the transform applied, the quantization applied, and / or other decoding parameters.

[0122] In some examples, the step of quantizing may include adapting the quantization based on an analysis of the coefficients and / or data to be transformed (eg, residual data).In some examples, the distribution used in the step of quantizing may be adapted.

[0123] The step of encoding the first set of residuals may include: deriving a set of temporal coefficients from the temporal buffer; and subtracting the set of temporal coefficients from the set of coefficients.

[0124] The step of encoding the second set of residuals may include: deriving a set of temporal coefficients from the temporal buffer; and subtracting the set of temporal coefficients from the set of coefficients.

[0125] The above describes how the steps of grading and selection can be applied to the residual data, the step of subtracting the temporal coefficients can be performed, and the quantization can also be adapted. Each of these steps can be predetermined and selectively applied, or can be applied based on analysis of the input video, downsampled video, reconstructed video, upsampled video, or any combination of the above to improve the overall performance of the encoder. The steps can be selectively applied based on a predetermined set of rules or deterministically applied based on analysis or feedback of performance.

[0126] According to an example method, the first codec is a hardware-based codec, preferably the first codec is AVC, HEVC, AV1, VP8 or VP9.

[0127] The example method further includes sending the base encoded stream.

[0128] The example method further includes sending the first-level encoded stream.

[0129] The example method further includes sending the second-level encoded stream.

[0130] Some other examples relate to a method of decoding multiple encoded streams into a reconstructed output video, the method comprising: receiving a first base encoded stream; decoding the first base encoded stream according to a first codec to generate a first output video; receiving one or more additional encoded streams; decoding the one or more additional encoded streams to generate a residual set; and combining the residual set with the first video to generate a decoded video.

[0131] In one example, the method includes retrieving a plurality of decoding parameters from the header. The decoding parameters may indicate which procedural steps are included in the encoding process.

[0132] In an example, decoding the one or more additional encoded streams to generate a residual set includes: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to generate the residual set.

[0133] In an example, decoding the one or more further encoded streams to generate a set of residuals includes predicting a subset of the residuals based on a co-located residual from a temporal buffer.

[0134] In one example, the method may include receiving a first-level coded stream and receiving a second-level coded stream. In this example, the step of decoding the one or more additional coded streams to generate a residual set includes: decoding the first-level coded stream to derive a first residual set; wherein the step of combining the residual set with the first video to generate a decoded video includes: combining the first residual set with the first output video to generate a second output video; upsampling the second output video to generate an upsampled second output video; decoding the second-level coded stream to derive a second residual set; and combining the second residual set with the second output video to generate a reconstructed output video.

[0135] In one example, the step of upsampling the second output video to generate the upsampled second output video includes: adding the values ​​of the elements in the first residual set of the blocks in the upsampled second output video from which the upsampled second output video is derived to the corresponding blocks in the upsampled second output video. The blocks may be 2×2 blocks. This adding step may be selectively performed based on a predetermined value or signal included in the header.

[0136] In one example, the step of decoding the first-level coded stream to derive the first residual set includes: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to generate the first residual set.

[0137] In this example, the step of decoding the first level coded stream to derive the first residual set comprises applying a deblocking filter configured to apply a mask to the residual block. The mask may be weighted according to a set of predefined weights.

[0138] In one example, the step of decoding the second-level coded stream to derive the second residual set includes: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to generate the second residual set.

[0139] The inverse transform operation may be the inverse of the operation as defined above, or may be a substantially mirrored operation. That is, a 2x2 block or 4x4 block transform may be selectively applied. The transform may be detected by a decoding method or signaled in a header.

[0140] If a 2x2 transform is used, the coefficients may be modified by adding the value of the residual from which the transformed residual block was predicted. If a 4x4 transform is used, the coefficients will be modified by adding the average of the four residuals.

[0141] The method may further include displaying or outputting the reconstructed output.

[0142] In one example, a method of decoding a plurality of coded streams into a reconstructed output video includes: receiving a coded enhancement stream; decoding the coded enhancement stream to derive a residual set, the decoding including: obtaining a set of preliminary residual sets from the coded enhancement stream; deriving a set of temporal predictions using a temporal buffer; and adding the set of temporal predictions to the set of preliminary residuals to output a processed residual set (e.g., a layer 2 residual). In this example, the method of decoding the plurality of coded streams into a reconstructed output video also includes: receiving a first output video, the first output video comprising an output of a base decoder applied to a base layer coded stream; receiving another coded enhancement stream for a lower layer; decoding the other coded enhancement stream to derive another residual set, combining the other residual set with the first output video to generate a second output video; upsampling the second output video to generate an upsampled second output video; and combining the processed residual set with the upsampled second output video to generate a reconstructed output video.

[0143] In one example, decoding the encoded enhancement stream to derive the prepared residual set includes: applying an entropy decoding operation; applying a dequantization operation; and applying a transform operation to generate the prepared residual set.

[0144] In one example, decoding another encoded enhancement stream to derive another residual set includes: obtaining another prepared residual set from another encoded enhancement stream; deriving another temporal prediction set using a second temporal buffer; and adding the other temporal prediction set to the other prepared residual set to output another (level 1) residual set.

[0145] In an example, the decoding of another encoded enhancement stream (i.e., layer 1 stream) to derive another residual set (i.e., layer 1 residual) includes: applying an entropy decoding operation; applying a dequantization operation; applying a transform operation to generate another residual set. The temporal processing as described in these examples may be similarly and selectively applied to each of the enhancement levels described herein.

[0146] Specific examples described herein relate to temporal processing associated with one or more encoded streams.A decoder may be configured to apply temporal processing using a temporal buffer.

[0147] The decoder may be configured to decode a plurality of coded streams into reconstructed output videos. The decoder may be configured to receive a first output video comprising an output of a base decoder applied to a base layer coded stream. The decoder may be configured to receive one or more additional coded streams. The decoder may be configured to decode corresponding frames of the one or more additional coded streams to derive corresponding residual sets.

[0148] In some cases, each of the corresponding frames may be divided into a plurality of tiles. Each tile may be divided into a plurality of blocks.

[0149] In other cases, each frame of the corresponding frame can be divided into a plurality of planes. Each plane can be divided into a plurality of tiles. Each tile of the plurality of tiles can be divided into a plurality of blocks.

[0150] The decoder may also be configured to combine the set of residuals with the first output video to generate a reconstructed output video. To decode the corresponding frame, the decoder may be configured to obtain a set of preliminary residuals from the one or more additional coded streams for each of the plurality of blocks. To decode the corresponding frame, the decoder may be configured to derive a set of temporal predictions using the temporal buffer. To decode the corresponding frame, the decoder may be configured to combine the set of temporal predictions with the preliminary residual set to output data to be combined with the first output video.

[0151] The decoder may be configured to implement selective zeroing of values ​​of the set of temporal predictions. The decoder may be configured to implement zeroing of values ​​of the set of temporal predictions at a frame level for at least one of the corresponding frames. The decoder may be configured to implement zeroing of values ​​of the set of temporal predictions at a tile level for at least one of the plurality of tiles. The decoder may be configured to implement zeroing of values ​​at a block level for at least one of the plurality of blocks.

[0152] The decoder may be configured to obtain respective residual elements for a block of the plurality of blocks.The decoder may be configured to derive, for each of the respective residual elements, a respective temporal prediction of the set of temporal predictions from the temporal buffer.

[0153] The decoder may be configured to reset the value of the set of temporal predictions to zero by refreshing at least a portion of the temporal buffer. Some other examples relate to a device for encoding a data set into a set of encoded data including a header and a payload. The device is configured to encode an input video according to the above steps. The device may include a processor configured to implement the method of any one of the above aspects.

[0154] Some other examples relate to a device for decoding a data set into a reconstructed video from a data set including a header and a payload. The device is configured to decode the output video according to the above steps. The device may include a processor configured to implement the method of any one of the above aspects.

[0155] Encoders and decoders are also available.

[0156] Some other examples relate to a computer-readable medium which, when executed by a processor, causes the processor to perform any of the methods of the above aspects.

[0157] The above embodiments are to be understood as illustrative examples. Additional embodiments are contemplated. It is to be understood that any feature described with respect to any one embodiment may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other embodiment, or in combination with any combination of any other embodiment. In addition, equivalents and modifications not described above may also be employed within the scope of the appended claims.

Claims

1. A decoder configured to decode a plurality of encoded streams into a reconstructed output video, the decoder being configured to: receiving an encoded elementary stream, wherein the encoded elementary stream is configured to be decoded by an elementary decoder to produce a first output video; receiving one or more encoded streams; decoding respective frames of the one or more encoded streams to derive respective residual sets, each of the respective frames being divided into a plurality of tiles, and each of the plurality of tiles being divided into a plurality of blocks; and combining the residual set with the first output video to generate the reconstructed output video, Wherein, in order to decode the corresponding frame, the decoder is configured to: obtaining, for each of the plurality of blocks, a set of preliminary residuals from the one or more encoded streams; deriving a set of temporal predictions using the temporal buffer; and combining the set of temporal predictions with the set of prepared residuals to output data to be combined with the first output video, The decoder is configured to realize zeroing of the value of the set of temporal predictions, which is realized by the following specific configuration: obtaining, for each respective block of the plurality of blocks, a second parameter at a block level from data from the one or more encoded streams, wherein the second parameter has a second value that provides temporal signaling for the respective block and instructs the decoder to refresh a value of a temporal prediction for the block; obtaining a third parameter at a tile level, the third parameter having a third value indicating that obtaining a second parameter having a second value for a particular block means refreshing a value of the temporal buffer associated with the tile; and The value of the temporal buffer associated with a corresponding tile of the plurality of tiles is refreshed by setting the value of the temporal prediction for the tile to zero.

2. The decoder of claim 1, wherein the decoder is configured to generate a first block of elements from a frame in the reconstructed output video without using a second block of elements from the frame in the reconstructed output video.

3. The decoder according to claim 1 or 2, wherein the decoder is configured to: Obtaining corresponding residual elements of blocks in the plurality of blocks; and A respective temporal prediction of the set of temporal predictions is derived from the temporal buffer for each of the respective residual elements.

4. The decoder according to claim 1 or 2, wherein the decoder is further configured to apply an entropy decoding operation to the one or more encoded streams, and obtain the prepared residual set by applying a dequantization operation to the corresponding block and applying an inverse transform operation to the corresponding block.

5. The decoder of claim 1 or 2, wherein the decoder is configured to zero the value of the temporal prediction in response to receiving a first parameter having a first value indicating that temporal processing is enabled. The decoder of claim 5 , wherein the bit length of the first value of the first parameter is one bit.

7. The decoder of claim 5, wherein the first parameter is received once for a group of pictures associated with the plurality of encoded streams.

8. The decoder according to claim 1 or 2, wherein the bit length of the third value of the third parameter is one bit.

9. The decoder of claim 1 or 2, wherein the third parameter is received once for a group of pictures associated with the plurality of encoded streams.

10. A decoder according to claim 1 or 2, wherein the decoder is configured to refresh the value of the temporal buffer for a first tile of a frame in the corresponding frame, and to reset the value of the set of temporal predictions to zero for a first block of a second tile of the frame, without resetting the value of the set of temporal predictions to zero for a second block of the second tile.

11. A decoder according to claim 1 or 2, wherein the decoder includes an enhancement decoder, which is configured to receive residual data generated by comparing data derived from an input video with data derived from a base layer encoded stream, wherein the one or more encoded streams include encoded residual data, and the encoded residual data can be decoded to reconstruct one or more additional residual data sets for supplying for the first output video.

12. The decoder of claim 11, wherein the enhancement decoder is different from the base decoder.

13. A decoder according to claim 1 or 2, wherein a set of transform coefficients of a residual set used to decode a first frame of the one or more encoded streams represents a difference between the set of transform coefficients of the first frame and another set of transform coefficients of a second frame of the one or more encoded streams that is different from the first frame.

14. A method of decoding a plurality of encoded streams into a reconstructed output video, the method comprising: Receiving an encoded elementary stream for decoding by an elementary decoder and generating a first output video; receiving one or more encoded streams; decoding corresponding frames of the one or more encoded streams to derive corresponding residual sets, each of the corresponding frames being divided into a plurality of tiles, and each of the plurality of tiles being divided into a plurality of blocks; as well as combining the residual set with the first output video to generate the reconstructed output video, The decoding further comprises: obtaining, for each of the plurality of blocks, a set of preliminary residuals from the one or more encoded streams; deriving a set of temporal predictions using the temporal buffer; and combining the set of temporal predictions with the set of prepared residuals, wherein the decoding comprises implementing zeroing of values ​​of the set of temporal predictions, wherein implementing zeroing of values ​​of the set of temporal predictions comprises: obtaining, for each respective block of the plurality of blocks, a second parameter at a block level from data from the one or more encoded streams, wherein the second parameter has a second value that provides temporal signaling for the respective block and instructs the decoder to refresh a value of a temporal prediction for the block; obtaining a third parameter at a tile level, the third parameter having a third value indicating that obtaining a second parameter having a second value for a particular block means refreshing a value of the temporal buffer associated with the tile; and The value of the temporal buffer associated with a corresponding tile of the plurality of tiles is refreshed by setting the value of the temporal prediction for the tile to zero.

15. The method of claim 14, comprising generating a first block of elements from a frame in the reconstructed output video without using a second block of elements from the frame in the reconstructed output video.

Images