Time signaling for video coding technology

Abstract

An encoder (300) is configured to receive an input video (302) comprising corresponding frames, each frame being divided into a plurality of tiles and each tile being divided into a plurality of blocks. The encoder is configured to generate a base encoded stream (310) using a base encoder (306), determine (334) a time pattern for generating one or more additional encoded enhanced streams (328) using an enhanced encoder, and generate the one or more additional encoded enhanced streams (328) according to the determined time pattern. The time pattern is either a first time pattern that does not apply non-zero values ​​from a time buffer or a second time pattern that applies non-zero values ​​from the time buffer (332). Generating the one or more additional encoded enhanced streams includes applying a transform (348) to each of a series of blocks. The time pattern is determined for one or more frames, tiles, or blocks.

Description

TECHNICAL FIELD

[0001] The present invention relates to methods, devices, computer programs and computer readable media for use in video coding techniques. BACKGROUND

[0002] Compression and decompression of signals is a consideration in many known systems. Many types of signals, such as video, can be compressed and encoded for transmission, for example via a data communication network. When decoding such a signal, it can be desirable to increase the quality level of the signal and / or to recover as much information as possible that was contained in the original signal.

[0003] Some known systems employ scalable coding techniques. Scalable coding involves encoding a signal together with information to allow the signal to be reconstructed at one or more different quality levels, for example depending on the capabilities of the decoder and the available bandwidth.

[0004] There are several considerations regarding the reconstruction of a signal in a scalable coding system. One such consideration is the ability of the encoder and / or decoder to efficiently process the information. The efficiency with which the encoder and / or decoder processes the information can be a factor in the performance level of the encoder and / or decoder. SUMMARY

[0005] Various aspects of the invention are set out in the appended claims.

[0006] Further features and advantages will become apparent from the following description, with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a schematic diagram illustrating an encoding process according to examples herein;

[0008] Figure 2 is a schematic diagram illustrating a decoding process according to examples herein;

[0009] Figure 3A and 3B is a schematic diagram illustrating an encoding process according to examples herein;

[0010] Figure 4A and 4B is a schematic diagram illustrating a decoding process according to examples herein;

[0011] Figures 5A to 5C Example operations in an encoder for two respective temporal modes are shown;

[0012] Figures 6A to 6E is a schematic diagram illustrating various features of a temporal prediction process according to examples herein; and

[0013] Figure 7A and 7B are two halves of a flowchart showing a temporal processing method according to an example. DETAILED DESCRIPTION

[0014] A hybrid backward-compatible encoding technique is described herein.

[0015] The examples described herein provide a flexible, adaptable, efficient and computationally light encoding format that combines different video encoding formats, base codecs (e.g. AVC, HEVC, or any other current or future codec) with at least two levels of enhancement data.

[0016] The general structure of the encoding scheme uses a down-sampled source signal encoded in a base codec, adds a first level of correction data to the decoded output of the base codec to produce a corrected picture, and then adds another level of enhancement data to an up-sampled version of the corrected picture.

[0017] The streams are thus considered as a base stream and an enhancement stream. Notably, it is generally expected that the base stream is decoded by a hardware decoder, while the enhancement stream is adapted to a software processing implementation with suitable power consumption.

[0018] This structure forms multiple degrees of freedom, which allow great flexibility and adaptability to many scenarios, making the encoding format suitable for many use cases, including over-the-top (OTT) transmission, live streaming, live ultra-high-definition (UHD) broadcast, etc.

[0019] Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at lower resolution, making the output compatible with existing decoders, and can also be used as a lower resolution output if deemed appropriate.

[0020] The codec format uses a minimal number of simple encoding tools. When combined synergistically, they can provide visual quality improvement compared to a full resolution picture encoded in the base codec, while making it more flexible the way it can be used.

[0021] Figure 1 A first example encoder 100 is shown. The shown components can also be implemented as steps of a corresponding encoding process.

[0022] In encoder 100, input full-resolution video 102 is processed to generate various encoded streams. Input video 102 includes corresponding frames, each of which is divided into multiple tiles, and each of the multiple tiles is divided into multiple blocks. A first encoded stream (encoded base stream 110) is generated by feeding a downsampled form of the input video, obtained by downsampling 104 of input video 102, to a base encoder 106 (e.g., AVC, HEVC, or any other codec). A second encoded stream (encoded level 1 stream 116) is generated by applying encoding operation 114 to a residual obtained by obtaining the difference 112 between the reconstructed base codec video and the downsampled form of the input video. The reconstructed base codec video is obtained by decoding the output of base encoder 106 with base decoder 108. A third encoded stream (encoded level 2 stream 128) is generated by processing 126 to obtain a residual obtained by obtaining the difference 124 between the upsampled form of the corrected form of the reconstructed base encoded video and the input video 102. The corrected form of the reconstructed base codec video is obtained by combining the reconstructed base codec video with the residual obtained by applying decoding operation 118 to the encoded Level 1 stream 116 120.

[0023] Level 1 encoding operation 114 is operated with an optional Level 1 time buffer 130, which can be used to apply time processing as further described below. Level 2 encoding operation 126 is also operated with an optional Level 2 time buffer 132, which can be used to apply time processing as further described below. Level 1 time buffer 130 and Level 2 time buffer 132 are operated under the control of a time selection component 134. The time selection component 134 can receive one or more of the input video 102 and the output of downsampling 104 to select a time mode. This is explained in more detail in a later example.

[0024] Figure 2 The first instance decoder 200 is shown. The components shown can also be implemented as steps in a corresponding decoding process. The decoder receives data from, for example... Figure 1 The encoder, such as encoder 100, produces three streams (encoded base stream 210, encoded Level 1 stream 216, and encoded Level 2 stream 228) along with a header 236 containing further decoding information. Encoded base stream 210 is decoded by a base decoder 208 corresponding to the base decoder used in the encoder, and its output is combined 238 with the decoded residual obtained by decoding 240 of encoded Level 1 stream 216. The combined video is upsampled 242 and further combined 244 with the decoded residual obtained by applying decoding operation 246 to encoded Level 2 stream 228.

[0025] Figure 3A and 3B A second example encoder 300, 380 is shown in different variations. The second example encoder 300, 380 can include Figure 1 embodiments of the first example encoder 100. In Figure 3A and 3B In examples of the second example encoder 300, 380, the encoding steps of the stream are expanded in more detail to provide examples of how the steps can be performed. Figure 3A A first variation with temporal prediction in the enhancement process set only at the second level, i.e. relative to 2-level encoding, is shown. Figure 3B A second variation with temporal prediction performed in two enhancement levels, i.e. level 1 and level 2, is shown.

[0026] The base stream 310 is generated generally by processes as explained with reference to Figure 1 That is, the input video 302 is down-sampled 304 (i.e. a down-sampling operation 304 is applied to the input video 302 to produce a down-sampled input video). The down-sampled video obtained by down-sampling 304 the input video 302 is then encoded using a first base encoder 306 (i.e. an encoding operation is applied to the down-sampled input video to produce an encoded base stream 310 using the first or base encoder 306). Preferably, the first or base encoder 306 is a codec suitable for hardware decoding. The encoded base stream 310 can be referred to as a base layer or base level.

[0027] As mentioned above, the enhancement stream can include two streams. A first enhancement level (described herein as "level 1") provides a set of correction data that can be combined with a decoded version of the base stream to produce a corrected picture. This first enhancement stream is shown in Figure 1 and 3 as an encoded level 1 stream 316. The enhancement stream can be produced by an enhancement encoder. The enhancement encoder can be different to the base encoder 306 used to produce the encoded base stream 310.

[0028] To produce the encoded level 1 stream 316, the encoded base stream 310 is decoded using a base decoder 308 (i.e. a decoding operation is applied to the encoded base stream 310 to produce a decoded base stream). A difference 312 between the decoded base stream and the down-sampled input video obtained by down-sampling 304 the input video 302 is then produced (i.e. a subtraction operation 312 is applied to the down-sampled input video and the decoded base stream to produce a first set of residuals). Here, the term residuals is used in the same way as it is known in the art, i.e. an error between a reference frame and a desired frame. Here, the reference frame is the decoded base stream and the desired frame is the down-sampled input video. Thus, the residuals used in the first enhancement level can be considered as a corrected video, as it 'corrects' the decoded base stream to the down-sampled input video used in the base encoding operation.

[0029] The difference 312 is then encoded to produce an encoded Level 1 stream 316 (i.e., the encoding operation is applied to the first residual set to produce the first enhanced stream 316).

[0030] exist Figure 3A and 3B In some example implementations, the encoding operation includes several steps, each of which is optional and preferred and provides a specific benefit.

[0031] In Figure 3, the steps include transformation step 336, quantization step 338, and entropy coding step 340.

[0032] Although not shown in the figure, in some instances, the encoding process identifies whether a residual grading mode is selected. If the residual mode is selected, a residual grading step can be performed (i.e., a residual grading operation can be performed on the first residual step to produce a graded residual set). The graded residual set can be filtered so that not all residuals are encoded into the first enhancement stream 316 (or correction stream).

[0033] Next, the first residual set, or the graded or filtered first residual set, is subjected to transform 336, quantization 338, and entropy encoding 340 to produce an encoded first-level stream 316 (i.e., depending on whether the graded mode is selected, transform operation 336 is applied to the first residual set or the filtered first residual set to produce a transformed residual set; quantization operation 338 is applied to the transformed residual set to produce a quantized residual set; and entropy encoding operation 340 is applied to the quantized residual set to produce a first-level enhanced stream 316). Preferably, entropy encoding operation 340 may be a Huffman coding operation or a run-length coding operation or both. Optionally, control operations (not shown in the figure) may be applied to the quantized residual set to correct for the effects of the graded operation.

[0034] As described above, the enhanced stream may include a first enhancement level 316 and a second enhancement level 328. The first enhancement level 316 can be considered as a corrected stream. The second enhancement level 328 can be considered as another enhancement level that converts the corrected stream into the original input video.

[0035] Another enhancement level 328 is generated by encoding another set of residuals, the residuals being the difference 324 between the upsampled form of the decoded Level 1 stream and the input video 302.

[0036] In Figure 3, the quantized (or controlled) residual set is inverse-quantized 342 and inverse-transformed 344, and then a deblocking filter (not shown) is optionally applied to produce a decoded first residual set (i.e., inverse-quantization operation 342 is applied to the quantized first residual set to produce a dequantized first residual set; inverse-transformation operation 344 is applied to the dequantized first residual set to produce a detransformed first residual set; and deblocking filter operation is optionally applied to the detransformed first residual set to produce a decoded first residual set). The deblocking filter step is optional, depending on the applied transform 336, and includes applying a weighted mask to each block of the detransformed first residual set.

[0037] The decoded base stream is combined with the decoded first residual set 320 (i.e., a summation operation 320 is performed on the decoded base stream and the decoded first residual set to produce a regenerated first stream). For example... Figure 3A and 3B As shown, the combination is then upsampled 322 (i.e., the upsampling operation 322 is applied to the regenerated first stream to produce an upsampled regenerated stream).

[0038] The upsampled stream is then compared with the input video 302, which forms another set of residuals (i.e., difference operation 324 is applied to the regenerated upsampled stream to produce another set of residuals). This other set of residuals is then encoded into an encoded 2-level enhanced stream 328 (i.e., encoding operation is then applied to the other set of residuals to produce another encoded enhanced stream 328).

[0039] As with the encoded Level 1 stream 316, the encoding applied to Level 2 residuals may include several steps. Figure 3A The steps are shown as time prediction (described further below), transformation 348, quantization 350, and entropy coding 352.

[0040] Although not shown in the figure, in some instances, the encoding process identifies whether a residual grading mode is selected. If the residual mode is selected, a residual grading step can be performed (i.e., a residual grading operation can be performed on another set of residuals to produce another graded set of residuals). The other graded set of residuals can be filtered so that not all residuals are encoded into another enhancement stream 328.

[0041] Another set of residuals, or another set of graded residuals, is then transformed by 348 (i.e., transformation operation 348 is performed on another set of graded residuals to produce another set of transformed residuals). As shown, prior to upsampling 322, transformation operation 348 may utilize the prediction coefficients or prediction averages derived from the regenerated first stream. Further information is as follows.

[0042] Figure 3AThe second example encoder 300 is shown, where timing prediction is performed as part of the Level 2 encoding process. Timing prediction is performed using a timing selection component 334 and a Level 2 timing buffer 332. The timing selection component 334, as described in more detail below, determines the timing processing mode and accordingly controls the use of the Level 2 timing buffer 332. For example, if no timing processing is to be performed, the timing selection component 334 may instruct the contents of the Level 2 timing buffer 332 to be set to 0. Figure 3B The second example encoder 380 is shown, where time prediction is performed as part of both the Level 1 and Level 2 encoding processes. Figure 3B In addition to the level 2 time buffer 332, a level 1 time buffer 330 is also provided. Although not shown, other variations in which time processing is performed at level 1 instead of level 2 are also possible.

[0043] When time forecasting is selected Figure 3A The second instance encoders 300 and 380 of 3B can further modify the coefficients (i.e., the transformed residuals output by the transform components) by subtracting a corresponding set of coefficients derived from an appropriate time buffer. The corresponding set of coefficients may include a set of coefficients for the same spatial region (e.g., the same coding unit located within the frame) derived from a previous frame (e.g., coefficients for the same region of the previous frame). These coefficients can be derived or otherwise obtained from the time buffer. The coefficients obtained from the time buffer may be referred to herein as time coefficients. Subtraction can be applied by, for example, third subtraction components 354 and 356 (for corresponding levels 2 and 1). This time prediction step will be described further with respect to later examples. In summary, when time prediction is applied, the encoded coefficients correspond to the difference between a frame and another frame in the stream. The other frame may be an earlier or later frame in the stream (or a block within a frame). Therefore, instead of encoding the residual between the upsampled regenerated stream and the input video, the encoding process can encode the difference between the transformed frame in the stream and the transformed residual of said frame. Thus, entropy can be reduced. Timing predictions can be selectively applied to groups of coding units (referred to herein as “tiles”) based on control information, and the application of timing predictions at the decoder can be achieved by sending additional control information along with the encoded stream (e.g., within the header or as another surface as described in the example later).

[0044] like Figure 3A and 3B As shown in the figure, when the time prediction is active, each transformation coefficient can be:

[0045] Δ=F 当前 -F 缓冲器

[0046] The time buffer stores data associated with previous frames. Temporal prediction can be performed on a single color plane or multiple color planes. Generally, subtraction can be applied to video "frames" as element-wise subtraction, where the elements of the frame represent transformed coefficients, with the transform applied relative to a specific n-by-n coding unit size (e.g., 2×2 or 4×4). The difference generated by the temporal prediction (e.g., the difference mentioned above) can be stored in the buffer for use in subsequent frames. Therefore, in practice, the residual generated by the temporal prediction is the coefficient residual relative to the buffer. Although... Figure 3A and 3B The timing prediction is performed after the transformation operation, but it can also be performed after the quantization operation. This avoids the need to apply a level 2 inverse quantization component 358 and / or a level 1 inverse quantization component 360. Therefore, as Figure 3A and 3B As shown and described above, after the encoding process is performed, the output of the second instance encoders 300, 380 is an encoded base stream 310 and one or more enhancement streams, the one or more enhancement streams preferably including an encoded Level 1 stream 316 for a first enhancement level and an encoded Level 2 stream 328 for another or a second enhancement level.

[0047] Figure 4A and 4B The corresponding changes to the second instance decoders 400 and 480 are shown. These changes to the second instance decoders 400 and 480 can be implemented to correspond to... Figure 2 The first instance decoder 200. As can be clearly seen, the decoding steps and components are expanded in more detail to provide an example of how decoding can be performed. (As...) Figure 3A and 3B , Figure 4A This shows that time prediction is only used for changes at the second level (i.e., level 2), and Figure 4B The diagram illustrates the use of time prediction for changes at two levels (i.e., Level 1 and Level 2). As previously mentioned, consider alternative changes (e.g., Level 1 but not Level 2) where signaling information can be used to control the configuration.

[0048] like Figure 4B As shown in the example, during decoding, decoder 480 can parse header 436 (e.g., containing global configuration data, image configuration data, and other data blocks) and configure the decoder based on those headers 436. To regenerate the input video, decoders 400 and 480 can decode each of the base stream 410, the first enhanced stream 416, and another enhanced stream 428. Frames from the streams can be synchronized and then combined to derive the decoded video 448.

[0049] In each decoding process, the enhanced stream can undergo entropy decoding 450, 452, dequantization 454, 456 and inverse transformation 458, 460 to regenerate the residual set.

[0050] Figure 4A and 4B The decoding process includes retrieving an array of entropy-decoded quantized coefficients representing the first enhancement level, and an array of output L-1 residuals. In this case, the entropy-decoded quantized coefficients are obtained by applying the entropy decoding 450 operation to the encoded L-1 stream 416. Figure 4A and 4B The decoding process further includes retrieving an array of samples of the output of the base decoder 408. Figure 4A and 4B The decoding process further includes applying a dequantization process 454 to the array of entropy-decoded quantized coefficients to derive a set of dequantized coefficients, applying a transform process 458 to the set of dequantized coefficients, and optionally applying a filtering process. Figure 4A and 4B (Not shown in the text) to output an array representing the L-1 residuals of the first enhancement level, which may be referred to as the pre-residual set. In this case, the dequantization process 454 is applied to the entropy-decoded quantized coefficients of the corresponding block of the frame encoded with Level 1 stream 416, and the transform process 458 (which may be referred to as the inverse transform operation) is applied to the output of the dequantization process 454 of the corresponding block of the frame. Figure 4A and 4B The decoding process then further includes regenerating the image by combining the array of L-1 residuals with the array of samples output from the base decoder 408 462. Figure 4A and 4B The decoding process includes a transformation process 458 that applies a set of predetermined transformation processes based on the transmitted parameters. For example, transformation process 458 may be applied to a 2×2 coding unit or a 4×4 coding unit. A coding unit may be referred to herein as an element block in an array, in this case as an array of L-1 residuals.

[0051] Figure 4A and 4B The decoding process includes retrieving an array of quantized coefficients representing another enhancement level after entropy decoding, and an array of output residuals. Figure 4A and 4B In the decoding process shown, another enhancement level is the second enhancement level, and the array of residual outputs is an array of L-2 residuals. Figure 4A and 4B The method further includes retrieving an array of L-1 residuals of a first enhancement level corresponding to an array of entropy-decoded quantized coefficients representing another enhancement level. Figure 4A and4B The method further includes applying an upsampling process 464 to an array of residuals from the first enhancement level. Figure 4A and 4B In this process, the upsampling process 464 is applied to the combination of the array of L-1 residuals of the first enhancement layer and the corresponding array of samples of the output of the basic decoder 408.

[0052] exist Figure 4A and 4B In this process, upsampling procedure 464 is a modified upsampling procedure in which modifiers are added to the residuals. The step of adding modifiers can be performed as part of transform procedure 460. Alternatively, because transform procedure 460 involves a linear transformation, the step of adding modifiers can be performed as part of modified upsampling procedure 464, such as... Figure 4A and 4B As shown in the diagram. The step of adding a modifier thus causes a modification to the residual. This modification can be performed based on the position of the residual in the frame. The modification can be a predetermined value.

[0053] exist Figure 4A In this process, time prediction is applied during Level 2 decoding. Figure 4B In this example, timing prediction is controlled by timing prediction component 466. In this variation, control information for timing prediction is extracted from the encoded Level 2 stream 428, as indicated by the arrows from the stream to the timing prediction component 466. For example... Figure 4A In other embodiments shown, control information for time prediction may be sent separately, for example, in header 436, from the encoded Level 2 stream 428. The time prediction component 466 controls the use of the Level 2 time buffer 432, for example, it may determine the time pattern and control time refresh, as described with reference to a later example. The contents of the time buffer 432 may be updated based on data from previous frames of residuals. When the time buffer 432 is applied, the contents of the buffer are added 468 to the second residual set. Figure 4A In the middle, the contents of time buffer 432 are added to 468 and then to level 2 decoding component 446 (which is in Figure 4BThe output of entropy decoding 452, inverse quantization 456, and inverse transform 460 is implemented. In other instances, the contents of the time buffer can represent any set of intermediate decoded data, and therefore, addition 468 can be appropriately shifted to apply the contents of the time buffer at the appropriate stage (e.g., if the time buffer is applied during the dequantization coefficient stage, then addition 468 can be placed before inverse transform 460). The second set of time-corrected residuals is then combined 470 with the output of upsampling 464 to produce decoded video 448. Decoded video 448 is at level 2 spatial resolution, which can be higher than level 1 spatial resolution. The second set of residuals applies corrections to the (inspectable) upsampled reconstructed video, where the corrections are added back in detail and improve the sharpness of lines and features.

[0054] Transformation processes 458 and 460 can be selected from a predetermined set of transformation processes based on the transmission parameters. For example, transformation process 460 can be applied to a 2×2 block or a 4×4 block of elements in an L-2 residual array.

[0055] Figure 4B The second instance decoder 480 is shown as a variation. In this case, time prediction control data is received from header 436 by time prediction component 466. Time prediction component 466 controls both level 1 and level 2 time prediction, but in other instances, separate control components may be provided for the two levels as needed. Figure 4A This demonstrates how the reconstructed second residual set of the output of adding 468 to the level 2 decoding component 446 can be fed back to be stored in the level 2 time buffer 432 for the next frame (for clarity, from...). Figure 4B (The feedback is omitted here). A Level 1 time buffer 430, operating in a similar manner to the Level 2 time buffer 432 described above, is also shown, and the feedback loop for the buffer is shown in this figure. The contents of the Level 1 time buffer 430 are added to the Level 1 residual processing pipeline via a summation 472. Again, the location of this summation 472 may vary along the Level 1 residual processing pipeline depending on where the time prediction is applied (e.g., if the time prediction is applied in the transform coefficient space, then the summation may be located before the Level 1 inverse transform component 458).

[0056] Figure 4B This demonstrates two ways in which timing control information can be relayed to the decoder. The first method is via header 436, as described above. The second method, which can be used as an alternative or additional relay path, is via data encoded within the residual itself. Figure 1 This demonstrates a case where data 474 can be encoded into the HH transform coefficients and therefore extracted after entropy decoding 452. This data 474 can be extracted from the Level 2 residual processing pipeline and passed to the time prediction component 466.

[0057] Each or two enhancement streams can be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Units (NALUs). NALUs are intended to encapsulate enhancement bitstreams so that enhancements are applied to the correct underlying reconstructed frames. A NALU may, for example, contain a reference index containing the underlying decoded reconstructed frame bitstream to which the enhancements must be applied. In this way, enhancements can be synchronized to the combined underlying stream and frames of each bitstream to produce the decoded output video (i.e., the residual of each frame from the enhancement layer is combined with frames from the underlying decoded stream). A group of images can represent multiple NALUs.

[0058] Each frame can consist of three distinct planes representing different color components; for example, each component of a three-channel YUV video can have a different plane. Each plane can then have residual data with respect to a given enhancement level; for example, the Y plane can have a set of level 1 residual data and a set of level 2 residual data. In some cases, such as for monochrome signals, only one plane may exist; in this case, the terms frame and plane are used interchangeably. The level 1 and level 2 residual data can be partitioned as follows. The residual data is divided into blocks, the size of which depends on the size of the transform used. If a 2×2 directional decomposition transform is used, then the blocks are, for example, 2×2 blocks of elements, or if a 4×4 directional decomposition transform is used, then the blocks are 4×4 blocks of elements. A tile is a group of blocks that cover an area of ​​the frame (e.g., an M×N area, which can be a square area). A tile is, for example, a 32×32 tile of elements. Thus, each frame in the encoded stream can be divided into multiple tiles, and each of these multiple tiles can be divided into multiple blocks. For color video, each frame can be divided into multiple planes, where each plane is divided into multiple tiles, and each tile of the multiple tiles is divided into multiple blocks.

[0059] The preceding text discussed how a collection of processing components or tools can be applied to enhance each element in the stream (or input video 102, 302) throughout the process. The following text provides an overview of each of these tools and their application, such as... Figures 1 to 4B The functionality within the overall process shown in Figure 4.

[0060] Downsampling processes 104 and 304 are applied to the input videos 102 and 302 to produce downsampled videos to be encoded by the base encoders 106 and 306. Downsampling 104 and 304 can be performed in both the vertical and horizontal directions, or alternatively only in the horizontal direction.

[0061] The input to L-1 encoding operation 114 includes an L-1 residual, which is obtained by taking the differences 112 and 312 between the decoded outputs of the base decoders 108 and 308 and the downsampled videos obtained by downsampling the input videos 102 and 302 by 104 and 304. The L-1 residual is then subjected to transform 336, quantization 338, and encoding 340, as further described below. Transform 336 outputs transform coefficients (i.e., the transformed L-1 residual).

[0062] There are two types of transformations that can be used in transformation process 336. Both utilize a small kernel, which is applied directly to the residuals retained after the stage of applying the predicted average.

[0063] The first transformation has a 2×2 kernel applied to the residual in a 2×2 block. The resulting coefficients are as follows:

[0064]

[0065] The second transformation has a 4×4 kernel applied to the residual in 4×4 blocks. The resulting coefficients are as follows:

[0066]

[0067] If downsampling and / or upsampling are performed only in the horizontal direction (e.g., some elements are set to 0), then a suitably adapted transform can also be applied. If a Hadamard transform is used, such as that shown in the example matrix above, then the decoding or inverse transform can use the same matrix, for example, the Hadamard matrix is ​​its own inverse. In this case, for example, at the decoder, an (M×N)×1 array of residuals R for M×N blocks can be derived from the corresponding (M×N)×1 array of coefficients C using R = H*C, where H is equal to one of the Hadamard matrices shown above.

[0068] The coefficients are then quantized using a linear quantizer (338). The linear quantizer can use a variable-size dead zone. The linear quantizer can use a dead zone of a different size than the quantization step and the non-centered dequantization offset.

[0069] The quantized coefficients are encoded using an entropy encoder 340. Two schemes for the entropy encoding 340 exist. In the first scheme, a run-length encoder (RLE) is used to encode the quantized coefficients. In the second scheme, the quantized coefficients are first encoded using an RLE, followed by processing the encoded output using a Huffman encoder. This leverages the RLE's advantage in encoding the long stream of zeros typically found along with the transform residuals, and then further utilizes the Huffman encoder to advantageously encode different frequencies of the quantized values ​​(e.g., which often have a decreasing number because these values ​​increase due to the distribution of the residuals and their linear transform values).

[0070] If the residual pattern (RM) has been selected, then the residuals are further graded and L-1 residuals are selected to determine which residuals should be transformed 336 and encoded. Preferably, this is pre-formed before entropy encoding 340.

[0071] If a time selection mode is selected for L-1 encoding, the encoder will further modify the coefficients by subtracting the corresponding coefficients derived from the level 1 time buffers 130 and 330, i.e., the time predictions described below.

[0072] The input to the L-1 decoding operation 118 includes an L-1 encoded residual, which is passed through an entropy decoder 450, a dequantizer 454, and an inverse transform module 458. The operation performed by these modules is the inverse operation performed by the modules described above.

[0073] If a time selection mode has been selected for L-1 encoding, then the residuals can be partially predicted from the co-localization residuals from the Level 1 time buffers 130 and 330. The co-localization residuals are referred to as time predictions in this paper.

[0074] If a 4×4 transform is used, the decoded residual can be fed into the deblocking filter module. The deblocking filter operates on each block of the transformed residual by applying a mask with specified weights. The general structure of the mask is as follows:

[0075]

[0076] Where 0 ≤ α ≤ 1 and 0 ≤ β ≤ 1.

[0077] The output from the decoded (and deblocked, if applicable) L-1 residual and the underlying decoded video from combinations 120 and 320 is upsampled 122 and 322 to produce an upsampled reconstructed video. Upsampling is optional and is signaled in the byte stream.

[0078] The input to L-2 encoding operation 126 includes L-2 residuals obtained by taking the differences 124 and 324 between the upsampled reconstructed video and the input videos 102 and 302. The L-2 residuals are then transformed 348, quantized 350, and encoded 352, as further described below. Transformation 348, quantization 350, and encoding 352 are performed in the same manner as described with respect to L-1 encoding 114. As explained with reference to L-1 encoding 114, transform 348 outputs transform coefficients (i.e., the transformed L-2 residuals). If RM has been selected, then the L-2 residuals are further graded and selected to determine which residuals should be transformed and encoded. L-2 encoding operation 126 may further include two additional processes as described below.

[0079] If the predicted coefficients mode is selected, the encoder will further modify the transformed coefficients C00 (e.g., the "average" or "A" coefficients used for 2×2 transforms). If a 2×2 transform is used, C00 is modified by subtracting the value of the upsampled residual on which the transformed block of the predicted residual is based. If a 4×4 transform is used, C00 is modified by subtracting the average of the four upsampled residuals on which the transformed block of the predicted residual is based.

[0080] If a time selection mode is selected for L-2 encoding, the encoder will further modify the coefficients by subtracting the corresponding coefficients derived from the level 2 time buffers 132 and 332, as described above.

[0081] The inputs to L-2 decoding operations 246 and 446 include the encoded L-2 residuals. The decoding process of the L-2 residuals involves an entropy decoder 452, a dequantizer 456, and an inverse transform module 460. The operations performed by these modules are the inverse operations performed by the modules described above. If a time selection mode has been selected for L-2 encoding, the residuals can be partially predicted from the co-localized residuals from the two-stage time buffers 132 and 332. The co-localized residuals may be referred to herein as time predictions.

[0082] The modified upsampling processes 242 and 464 comprise two steps, the second of which depends on the signaling received by the decoder. In the first step, the combination 238 and 462 of the decoded (and deblocked, if applicable) L-1 residual and the base decoded video 208 and 408 (L-1 reconstructed video) is upsampled to produce an upsampled reconstructed video. If a prediction coefficient mode has been selected, then the second step is performed. Specifically, the values ​​of elements in the L-1 reconstructed values ​​of the resulting 2×2 block in the upsampled reconstructed video are added to the 2×2 block in the upsampled reconstructed video.

[0083] Throughout the above, the term bitstream can be replaced by stream, byte stream, or NALU stream as needed.

[0084] The following describes some changes and implementation details of time prediction, including some aspects of time signaling.

[0085] In some instances described herein, information from two or more video frames associated with samples from different times can be used. This can be described as a temporal pattern, for example, because it involves information from different times. As described herein, the step of encoding one or more sets of residuals can utilize a time buffer arranged to store information associated with the previous video frame. In one case, the step of encoding the set of residuals may include deriving a set of temporal coefficients from the time buffer and modifying the current set of coefficients using the retrieved set of temporal coefficients. In these instances, "coefficients" may include transformed residuals, such as those defined by one or more coding units of a frame of the reference video stream, and methods may be applied to both residuals and coefficients. In some cases, asymmetric methods at the encoder and decoder may be used. For example, as... Figure 4A As shown, encoding can be time-processed based on coefficients (e.g., transformed residuals), while decoding can be based on residuals (e.g., Figure 3A Temporal processing is performed using the level 2 residuals. In some cases, modifications at the encoder may include subtracting the temporal coefficient set from the current coefficient set. This method can be applied to multiple coefficient sets, such as those involving level 2 streams and those involving level 1 streams. Modifications to the current coefficient set can be selectively performed, for example, by referring to coding units within a video data frame.

[0086] Timing can be applied at both the encoding and decoding stages. The use of a time buffer is demonstrated in... Figure 4A and 3B Encoders 300 and 380 are shown in Figure 3A and 4B In decoders 400 and 480. As described herein, the current coefficient set may undergo one or more of a grading and a transform before being modified at the encoder. In one case, the dequantized transformed coefficients dqC from the previously encoded (n-1) frames at the corresponding location (e.g., the same location or a mapped location) are transformed. x，y，n-1 The coefficients C used to predict the frame to be encoded as (n) x，y，n If a 4×4 transform is used, then x and y can be in the range [0, 3]; if a 2×2 transform is used, then x and y can be in the range [0, 1]. Quantization coefficients can be generated by an inverse quantization block OR operation. For example, in Figure 5A and 3B In this process, dequantized coefficients are generated using inverse quantization components 358 and 360. Using dequantized coefficients, such as coefficients from a previous frame that have been quantized and subsequently dequantized, allows for some correction of features introduced by (lossy) quantization. In other instances, unquantized coefficients can be acquired and buffered before quantization. Either method can be used.

[0087] In some instances, at least two time patterns may exist:

[0088] • First-time mode without using a time buffer or using a time buffer with all zero values. First-time mode can be considered an intra-frame mode because it only uses information from the current frame. In first-time mode, after any applied grading and transformation, coefficients can be quantized without modification based on information from one or more previous frames.

[0089] • A second time mode utilizes a time buffer, for example, a time buffer with potentially non-zero values. The second time mode can be considered an inter-frame mode because it uses information from outside the current frame, such as from multiple frames. In the second time mode, after any applied grading and transformation, the dequantized coefficients from the previous frame—C—can be subtracted from the coefficients to be quantized. x，y，n，帧间 =C x，y，n -dqC x，y，n-1 .

[0090] In one scenario, the first time mode can be applied by subtracting from the set of zero-time coefficients. In another scenario, subtraction can be performed selectively based on time signaling data. Figure 5A and 5B This demonstrates example operations in encoders used for two corresponding time modes. Figure 5B The first instance 500 in the model displays the set of coefficients by the encoding component 502 in the first time mode C. x，y，n，帧内 These are generated in the process. They are then passed for quantization. Figure 3A The second instance 504 in the diagram shows the coefficient set generated by the encoding component 506 in the second time mode C. x，y，n，帧间 The value is generated by subtraction 508 as described above and then passed for quantization. Subsequently, according to... Figure 5C and 3B Encode the quantized coefficients for both cases. Note that in other instances, the timing pattern may be applied after quantization or at another point in the encoding pipeline.

[0091] It can transmit each of two time modes. Time signaling can be provided between the encoder and decoder. The two time modes can be selectable within the video stream; for example, different modes can be applied to different parts of the video stream. Time modes can also, or alternatively, be signaled for the entire video stream. Time signaling can form, for example, parts of metadata transmitted from the encoder to the decoder. Time signaling can be encoded.

[0092] In one scenario, a global configuration variable can be defined for the video stream, such as for multiple frames within the video stream. For example, this could include a `temporal_enabled` flag, where a value of 0 indicates a first time mode and a value of 1 indicates a second time mode. In other scenarios, and / or instead of a global configuration value, a flag indicating the time mode can be assigned to each frame or "picture" within the video stream. If the `temporal_enabled` flag is used as a global configuration variable, this can be set by the encoder and passed to the decoder. Flag values ​​can be selected to reduce the amount of data that needs to be transmitted in the bitstream (e.g., a run-length encoding, described in more detail below, can be used to compress a value of 0).

[0093] In some cases, one or more portions of a frame in a video stream may be assigned variables indicating the temporal pattern of said portion. For example, said portion may include coding units or blocks, such as 2×2 or 4×4 regions transformed by a 2×2 or 4×4 transform matrix. In some cases, each coding unit may be assigned variables indicating the temporal pattern. For example, a value of 1 may indicate a first temporal pattern (e.g., the unit is an "intra-frame" unit), and a value of 0 may indicate a second temporal pattern (e.g., the unit is an "inter-frame" unit). The variables associated with each portion can be signaled between the encoder and decoder. 0 and 1 values ​​are provided only as examples; in some cases, these values ​​may be assigned to different patterns, for example, to reduce signaling costs. In one case, this can be done by setting one of the transformed coefficients as the variable value, for example, by setting the H coefficient for a 2×2 coding unit or the HH coefficient for a 4×4 coding unit as the variable value (e.g., 0 or 1). In another case, each coding unit may include metadata and / or sideband signaling indicating the temporal pattern. Figure 5C Example 510 illustrates the former case. In this example 510, there are four coefficients 512 generated by a 2×2 transform. These four coefficients 512 can be generated by transforming the residual into a 2×2 coding unit (e.g., for a given plane). When using a Hadamard transform, the four coefficients can be referred to as A, H, V, and D components 514, which represent the average, horizontal, vertical, and diagonal aspects within the coding unit, respectively. Figure 3A In example 510, the H component is used for the transmission time pattern, as shown in 516.

[0094] Temporal processing can be selectively applied at the encoder and / or decoder based on the indicated temporal pattern. For example, run-length encoding can be used to encode the metadata and / or temporal signaling within the sideband channels of a portion of the frame used to enhance the stream, thereby reducing the size of the data to be transmitted to the decoder. In this case, the temporal signaling can be structured into a temporal surface, where the surface has a size of coding_units_width × coding_units_height, for example, a bitmap or other image having a size equal to the image width and height divided by the coding unit size (i.e., the number of coding units per image dimension). The temporal surface can be viewed as similar to a coded coefficient surface, for example, all A components of a plane for a video frame can be encoded as an "A" surface (e.g., for other coefficients). Run-length encoding can be advantageous for small portions, such as coding units and / or tiles, where several temporal patterns exist (e.g., because this metadata may include a stream of '0' and '1' sequences with repeating values).

[0095] In some cases, the cost of each time-mode used for at least a portion of the video can be estimated. This can be performed at the encoder or in different devices. In some cases, time-modes with lower transmission costs are selected. In the encoder, this can be achieved by... Figure 3B and 3B The time selection block 334 and / or time mode selection blocks 362, 364 shown are executed. The decoder can then decode the signaling and apply the selected time mode, such as that indicated by the encoder. The cost function can be based on the data to be sent to the decoder, including the encoded residual data and signaling data.

[0096] Cost accounting can be performed on a per-frame basis and / or a per-part basis, such as per tile and / or per coding unit. In the latter case, the results of the cost accounting assessment can be used to set the temporal pattern variables for the coding unit prior to quantization and coding.

[0097] In some cases, a graph indicating the initial temporal pattern of video frames or a partial set of frames can be provided. This graph can be used by the encoder. In one case, the temporal_type variable can be obtained by encoding it for cost estimation, as described in more detail below.

[0098] In one scenario, the cost of selecting a time pattern can be controllable, for example, by setting parameters in a configuration file. In another scenario, the cost of selecting a time pattern can be based on the difference between the input frame and one or more sets of residuals (e.g., reconstructed). In yet another scenario, the cost function can be based on the difference between the input frame and the reconstructed frame. The cost of each time pattern can be evaluated, and the pattern with the lowest cost can be selected. The cost can be calculated based on the sum of absolute differences (SAD). The cost can be evaluated in this manner per frame and / or per coding unit.

[0099] For example, the first cost function can be based on J o =Total(abs(I) x，y，n -R x，y，v，o )), where I x，y，n For input values ​​(e.g., from input video 102), R x，y，v，o Let o be the reconstructed residual and o be an intra-frame or inter-frame frame (i.e., indicating the first or second time mode). The cost function can be evaluated using the reconstructed residual from each time mode, and the results of the cost function can then be compared for each time mode. The second cost function can be based on additional terms applied for penalizing non-zero quantized coefficients and / or on the values ​​of one or more orientation components where these quantized coefficients are used for signaling (e.g., after transformation). In the second case, the second cost function can be based on Jo = Sum(abs(I x，y，n -R x，y，v，o ))+step_widthAA*total((qC x，y，n，o ! = 0) + ((o == within the frame) & (qC 0，3，n，帧内 ==0), where the step width is a configurable weight or multiplier that can be adjusted empirically, qC x，y，n，o It is a quantized coefficient and qC 0，3，n，帧内 These are coefficients related to the H (for 2×2 transformation) or HH (for 4×4 transformation) elements. In other cases, when sideband signaling is in use, the cost of setting these bits to 1 can be incorporated into the second cost function. For the first time mode (e.g., intra-frame mode), it can be determined according to R... x，y，n，帧内 =Transformation(dqC) x，y，n，帧内 Reconstruct the residual, where "dq" indicates dequantization. For the second temporal mode (e.g., inter-frame mode), it can be based on R. x，y，n，帧间 =Transformation(dqC) x，y，n，帧间 +dqC x，y，n-1Reconstructing the residuals. In both cases, "transformation" can indicate the inverse transform of the coefficients. If the transform matrix is ​​an inverse matrix, then a common or shared matrix can be used for both the forward and inverse transforms. As previously mentioned, the time pattern used can be indicated in signaling information, such as metadata and / or setting parameter values. This signaling information can be referred to as time pattern signaling data. The encoder can be configured to generate time pattern signaling data that indicates the time pattern for one or more encoded enhancement streams used by the decoder. The encoder can compress the time pattern signaling data, for example, by encoding the time pattern signaling data, such as using run-length encoding discussed further below.

[0100] The cost function may include the cost of transmitting time-mode signaling data, as described in the second cost function above. In this way, the cost of transmitting time-mode signaling data for time-mode can make one value of the time-mode signaling data disadvantageous compared to other values ​​of the time-mode signaling data.

[0101] In one scenario, cost can be evaluated at the encoder. For example, the time selection block can be used to evaluate cost. In other scenarios, cost can be evaluated by a separate entity (e.g., a remote server during video stream preprocessing) and the time pattern communicated to the encoder and / or decoder. In either case, the encoder can be configured to determine the time pattern based on a cost function.

[0102] If the second time mode is selected (e.g., inter-frame processing), then the modified quantization coefficients (e.g., by...) are then... Figure 3B The subtraction blocks 354 and 356 (outputs) between transforms 348 and 336 and quantizations 350 and 336 are sent for entropy coding 352 and 340. The dequantized values ​​of these coefficients, obtained by dequantizing them 358 and 360, can then be retained for timing prediction in the next frame, e.g., frame n+1. Although Figure 3B Two separate inverse quantization operations 342 and 360 are shown for the first-level stream, but it should be noted that in some cases these may include a single common inverse quantization operation.

[0103] Time mode selection and time prediction can be applied to Figure 6A This includes one or more of the Level 2 and Level 1 streams shown (e.g., applied to one or two sets of residuals). In some cases, the time pattern can be configured and / or signaled separately for each stream.

[0104] As described later, in some instances, a second time mode may utilize a time refresh parameter. This parameter can be signaled when a time buffer is about to be refreshed, for example, when a first set of values ​​stored in the time buffer is about to be replaced with a second set of values. Time refresh can be applied at one or more points in the encoder and decoder. If the decoder uses a time buffer that stores residuals rather than coefficient values, then time refresh can be applied to said buffer.

[0105] In the encoder, a time buffer may store the dequantized coefficients of the previous frame loaded when the time refresh flag is set (e.g., equal to 1 indicating "refresh"). The time buffer may be any of time buffers 130, 132, 230, 232, 330, 332, 430, and 432. In this case, the dequantized coefficients are stored in the time buffer and used for timing prediction of future frames (e.g., for subtraction), while the frame's time refresh flag is not set (e.g., equal to 0 indicating "no refresh"). In this case, when a frame with an associated time refresh flag set to 1 is received, the contents of the time buffer are replaced. This can be performed on a per-frame basis and / or applied to portions of a frame, such as tiles or coding units. As explained above, the reference to "frame" herein also applies to the plane of a frame in color video (e.g., where the plane and frame may be the same for monochrome video).

[0106] The time refresh parameter can be used to represent a set of frames in a slowly changing or relatively static scene; for example, the first snapshot of the frame set can be used for subsequent frames in the scene. When the scene changes again, the first frame in the next scene's frame set can indicate that a time refresh is needed again. This can help speed up time prediction operations.

[0107] The time refresh operation of the time buffer can be achieved by setting all values ​​of the time buffer to zero.

[0108] The encoder can signal the time refresh parameter to the decoder, for example, as a binary temporal_refresh_bit, where 1 indicates that the decoder will refresh the time buffer used for a specific encoded stream (e.g., level 0 or level 1).

[0109] As described herein, in some instances, data can be grouped into tiles, such as 32×32 tiles of an image. In this case, time refresh operations, such as those described above, can be performed on a tile-by-tile basis for each frame, where coefficients are stored in a time buffer and are addressable by tiles. Mechanisms for tiling time refresh can be applied asymmetrically at the encoder and decoder.

[0110] In one scenario, timing processing can be performed at the encoder to determine the timing refresh logic based on each frame or each block / coding unit. In other cases, the signaling used for timing refresh at the decoder can be adapted to store the number of bits transmitted from the encoder to the decoder.

[0111] Figure 6A Example 600 shows time processing that can be performed at the encoder. Figure 3A The example encoder's time processing subunit 602 is shown. This encoder can be based on... Figure 6A The encoder 300 or 380 may be a 3B encoder. The timing processing subunit 602 receives a set of residuals indicated as R. These may be level 2 or level 1 residuals as described herein. They may include a set of graded and filtered residuals or a set of ungraded and unfiltered residuals. The timing processing subunit 602 outputs a set of quantized coefficients indicated as qC, which may then be entropy encoded. In this example, the timing processing subunit 602 also outputs timing signaling data indicated as TS for transmission to the decoder. The timing signaling data TS may be encoded together with or separately from the quantized coefficients. The timing signaling data TS may be provided as header data and / or as part of a sideband signaling channel. The timing signaling data may include data indicating the timing pattern of each coding unit (e.g., a variable TransTempSig for each coding unit). Alternatively or concurrently, the timing signaling data may include data indicating the timing pattern of each tile (e.g., a variable TileTempSig for each tile). In this way, the time refresh of the tiled surface can be transmitted to the decoder via a portion of the sideband signaling channel that is separately encoded from the quantization coefficients.

[0112] exist Figure 3A In instance 600, the residual (R) is received by transformation component 604. This may correspond to transformation components in other instances, for example, arranged to perform... Figure 6A and 3B The transformation component 604 is one of the components of transformation steps 336 and 348. Transformation component 604 outputs transformation coefficients (i.e., transformed residuals) as described herein. The temporal processing subunit 602 also includes a central temporal processor 606. This also receives metadata in the form of a tile-based temporal refresh parameter (temporal_refresh_per_tile) and an estimate of the initial temporal mode (initial_temporal_mode). The temporal mode estimate can be provided per frame of coding units, and the tile-based temporal refresh parameter can be provided per tile. For example, if a 2×2 transformation is used, then the coding unit involves a 2×2 region, and there are 16×16 such regions in a 32×32 tile, and therefore 256 coding units. The metadata can be generated by another subunit of the encoder, for example in a preprocessing operation, and / or can be supplied to the encoder, for example, via a web application programming interface.

[0113] exist Figure 3A In this example, time processor 606 receives metadata and is configured to determine the timing mode for each coding unit and the value of the timing refresh bit for the entire frame or picture. Time processor 606 controls the application of time buffer 608. Time buffer 608 may correspond to the time buffer of the previous example mentioned above. Time buffer 608 receives dequantized or dequantized coefficients from inverse quantization component 610, which may correspond to... Figure 3A and 3B One of the inverse quantization components 358 and 360. The inverse quantization component 610 is then communicatively coupled to the output of the quantization component 612, which may correspond to... Figure 3A and 3B One of the quantization components 338 and 350. The time processor 606 can implement certain functions of the time mode selection components 362 and 364, such as... Figure 6A and 3B As shown in [the document]. Although Figure 6A This illustration shows a coupling between quantization component 612, inverse quantization component 610, and time buffer 608. However, in other instances, time buffer 608 may receive the output of time processor 606 before quantization, and therefore inverse quantization component 610 may be omitted. Figure 6B The document also demonstrates a time signaling component 614 that generates time signaling TS based on the operation of time processor 606.

[0114] Figure 6B The corresponding instance decoder 616 is shown, where the decoder receives the temporal_refresh bit for each frame and the temporal_mode bit for each coding unit. As discussed above, in some cases, the temporal mode for each coding unit can be set within the coded coefficients, for example, by replacing the H or HH values ​​within the coefficients. In other instances, the temporal mode for each coding unit can be transmitted via additional signaling information, such as via sidebands and / or as part of the frame metadata. This can be described as using a time surface that provides time signaling.

[0115] exist Figure 4A In the instance decoder 616, a time processing subunit 618 is provided at the decoder. This may implement at least a portion of a level 1 or level 2 decoding component. The time processing subunit 618 includes an inverse quantization component 620, an inverse transform component 622, a time processor 624, and a time buffer 626. The inverse quantization component 620 and the inverse transform component 622 may include Figure 6B and 4BThe embodiments of inverse quantization components 454, 456 and inverse transform components 458, 460 are shown. Time processor 624 may correspond to the functionality applied by time prediction component 466 and summation component 468 or by time prediction component 466 and summation component 472. Time buffer 626 may correspond to one of time buffers 430, 432. Figure 4B In addition, there is a time signaling component 628 that receives data 630, which in this example is indicated in a set of headers H of the bit stream. These headers H may correspond to Figure 6A The header is 436. It should be noted that timing subunits 602 and 618 may, in some cases, be implemented by corresponding encoders and decoders different from those in other instances herein.

[0116] In some cases, when time mode is enabled, such as when enabled by the global temporal_enabled bit, Figure 6A The timing processor 606 is configured to use the tile-based estimation of the temporal_refresh_per_tile and the initial_temporal_mode to determine the value of the temporal mode for each coding unit and the value of the temporal refresh bits for the entire frame to improve the communication efficiency between the encoder and decoder.

[0117] In one scenario, the time processor 606 may determine the cost based on an estimate of the initial_temporal_mode and use these costs to set the values ​​transmitted to the decoder.

[0118] In one scenario, the time processor 606 may initially determine whether a per-frame refresh should be performed and signaled based on the percentage of different estimated time modes across the set of coding units spanning a frame, for example, when the coding units have an initial estimate of a time mode. For instance, firstly, all coding units of two estimated time modes (e.g., elements associated with a 2×2 or 4×4 transform) may be ignored if their sum of absolute differences is zero (e.g., where no residual exists). The refresh bits of the frame may then be estimated based on the proportion (e.g., percentage) of non-zero coding units. In some instances, the refresh operation of the time buffer contents may be set based on the percentage of coding units initially estimated to involve a first time mode. For example, if more than 60% of the coding units are estimated to involve the first time mode without setting temporal_refresh_per_tile, or if more than 75% of the coding units are considered to involve the first time mode with temporal_refresh_per_tile set, then the time buffer 608 may be refreshed for the entire frame and signaled appropriately for the decoder (e.g., by zeroing the values ​​within the buffer). In these cases, even if time processing is enabled (e.g., via temporal_enabled signaling), any subtraction is performed relative to the zeroed value within time buffer 608, thus suppressing time prediction at the decoder, similar to the first time mode. This can be used to revert to the first time mode based on changes within the video stream (e.g., if it is a live stream), even if a second time mode with time prediction is signaled. This improves viewing quality.

[0119] Similarly, in some cases, even if the second time mode is selected for the encoding unit and signaled to the decoder, if the frame encoded by the underlying encoder is set to I or an intra-frame (e.g., by setting the frame's temporal_refresh_bit), then the time buffer 608 is refreshed as described above (e.g., implementing processing similar to the first time mode). This helps ensure adherence to, for example, the picture group (GoP) boundaries of the encoded underlying stream when time processing is enabled.

[0120] Whether a time-based refresh is performed, for example, for tiles, may depend on whether the noise sequence is present alongside isolated static edges. The exact form of the cost function may depend on the implementation scheme.

[0121] Return to Figure 3AThe processing performed by the time processing subunit 602, after a decision on refreshing the entire frame, may involve tile-based processing based on the temporal_refresh_per_tile bit value in the second stage. This can be performed per tile for a given set of tiles in the frame. If temporal_refresh_per_tile is used, and if the flag temporal_refresh_per_tile is set in the metadata received by the time processor 608, then the following processing can be performed.

[0122] In the first sub-stage, it can be checked whether the time buffer for a given tile is empty. If so, all time signals in the tile are zero, and the coding units in this tile are encoded in a second time mode (e.g., inter-frame coding). For example, the time mode for said unit is set to the second mode, with further timing processing performed at the encoder regarding this mode, and the time mode is signaled to the decoder (e.g., by setting coefficient values ​​or via sideband signaling). This allows for efficient encoding of the tile according to the first time mode (e.g., intra-frame coding) when the time buffer is empty. If the second time mode (e.g., inter-frame mode) is set via a 0 value in the time mode bits, this method can reduce the number of bits that need to be sent to the decoder when the time buffer will be empty.

[0123] If the flag `temporal_refresh_per_tile` is not set for a given tile, then the first coding unit in the tile can be encoded according to a second time mode (e.g., as an inter-frame unit), and the time signaling for this tile is not set. In this case, the cost accounting operation as described above is performed for the other coding units within the tile (e.g., the first or second time mode can be determined based on the sum of absolute differences (SAD) metric). In this case, for the other coding units, the initial estimated time mode information is recalculated based on the current (e.g., on-site) coding conditions. All other coding units in the tile can undergo the above procedures and cost accounting steps. Encoding the first coding unit in the tile into the second time mode can be used to indicate the initial time processing at the decoder (e.g., indicating the initial refresh of the tile), wherein time processing for other coding units is performed at the decoder based on the confirmed value of the `temporal_mode` bit set for the coding unit.

[0124] If the flag `temporal_refresh_per_tile` is set for a given tile and the time buffer for the tile is not empty, then the time processor can schedule a time refresh for the tile, wherein time signaling is then set to indicate this at the decoder. This can be done by setting the time mode value of the first coding unit to 1 and the time mode values ​​of all other coding units to 0. This content of 1 in the first coding unit and 0 in the other coding units indicates to the decoder that a refresh operation will be performed relative to the tile, but the amount of information to be transmitted across the tile will be reduced. In this case, the time processor effectively ignores the time mode value and encodes all coding units according to the first time mode (e.g., encoded as intra-coding units without time prediction).

[0125] Therefore, in these instances, when temporal_refresh_per_tile is set as part of the encoder metadata, the first encoding unit can be used to instruct the decoder to clean (i.e., clear) its corresponding time buffer at the location of the tile, and the encoder logic can apply time processing as the appropriate time pattern.

[0126] The above methods allow for temporal prediction to be performed on a per-tile basis, based on coding units within a tile. The configuration of a given tile can be set for a single coding unit within the tile. These methods can be applied to one or more of two-level and one-level streams, for example, to one or more residual sets.

[0127] In some cases, the global parameter temporal_tile_intra_signalling can be set for the video stream to indicate that the tile refresh logic described above will be used at the decoder.

[0128] In some instances, initial_temporal_mode data can be provided for multiple frames (e.g., for the current frame and the next frame). In these instances, the initial_temporal_mode estimate for the next frame, such as frame n+1, can also be used to remove quantized values ​​that are not important for reducing the bit rate. The estimated temporal mode information can be used to control comparisons with one or more thresholds to indicate the removal of quantized values ​​(e.g., in...). Figure 6C (Or at one of the quantization steps 350 and 338 in 3B, or at one of the time mode selection steps 362 and 364).

[0129] In some cases, if the estimated initial_temporal_mode of the coding unit at the same location in the next frame is related to the first temporal mode (e.g., intra-frame mode), then it can be assumed that the residual to be encoded in the current coding unit will disappear in the next frame, and therefore residuals less than or equal to a given threshold can be removed. As an example, in the test case, this threshold can be set to 2, meaning that all quantized values ​​less than + / -3 will be removed from the coding unit.

[0130] Figure 6C Example 632 demonstrates how time signaling information can be provided for frames (or planes) with residual 634. Figure 6C The left-hand side shows how the residual frame can be divided into several tiles 636. Figure 6C The right-hand side shows how time signaling information can be assigned to each tile. For example, circle 638 indicates the first tile 640. In frame 634, tiles form raster-like rows across frame 634. The right-hand side shows the first tile 640 in more detail.

[0131] Figure 6C The circle 638 on the right-hand side illustrates how each tile 640 comprises several units 642, which may be referred to as coding units. A unit may include one or more residuals. In one case, a unit may relate to a residual block associated with a transform operation, such as a 2×2 block as described herein, which may relate to a Directed Decomposition Transformation (DD, described in more detail below), or a 4×4 block as described herein, which may relate to a Directed Decomposition Square (DDS). Figure 6D In this context, each tile 640 has a temporal_refresh_per_tile flag (displayed as "TR"), and each unit 642 within a tile has a temporal_type flag 644 (displayed as "TT"). This information can be obtained by the encoder and used to apply the time coding as described above.

[0132] As described above, in one case, timing signaling can be provided "in the stream," for example, as part of the enhancement stream. This can be done by replacing specific coefficients after the transform, for example, by embedding the timing signaling within the transform coefficients. In one case, level coefficients (e.g., H in a 2×2 directional decomposition transform or HH in a 4×4 directional decomposition square transform) can be used to signal the timing pattern for a specific coding unit. Level coefficients can be used because this minimizes the effect on the reconstructed signal. In some cases, such as based on data carried by other coefficients in the coding block, the effect of the level coefficients can be reconstructed via an inverse transform at the decoder.

[0133] In another scenario, metadata can be used to perform timing signaling. The metadata used here may be in the form of sideband signaling, for example, that does not form part of the base stream or enhancement stream. In one case, metadata is transmitted in a separate stream received by the decoder (e.g., via an encoder or remote server). This separate stream may be encoded, for example, using run-length encoding with entropy encoding, and may be referred to as a time-surface encoded stream. This stream may be combined with other streams output by the encoder (e.g., the base stream and enhancement streams) to form a combined bitstream transmitted to the decoder.

[0134] While "in-stream" time signaling offers some advantages for compression, sending the frame's time data as a separate information block (e.g., metadata) allows for different and potentially more efficient entropy encoding for this information. It also allows for time control and processing, such as those described above, to be performed without requiring the received augmented stream data. This allows for the preparation of time buffers and in-loop time decoding of simple augmentation processes.

[0135] In the second time mode (e.g., when time processing is enabled), three levels of time signaling can exist:

[0136] At the first level, per-frame timing signals may exist. These may include per-frame time refresh signals. The per-frame time refresh signals may be based on frame-based timing parameters. This may be per-frame refresh bits. If configured this way, the entire frame can be encoded without timing prediction. Signals at this level can be used to encode frames and can be transmitted to the decoder.

[0137] At the second level, per-tile time signals may exist. For example, these can be set for m multiplied by n tiles, where m and n can be 32. The per-tile time signals may include per-tile time refresh signals. The per-tile time refresh signals can be tile-based time parameters. This can be a per-tile refresh bit. If a time refresh signal is set for a tile, the entire tile is encoded without time information. This time signaling level can be used to encode frames. In one case, it may not be explicitly signaled to the decoder; in this case, the tile refresh signal can be indicated by the first time signal at the third level, as described below. In another case, the per-tile time refresh signal can be explicitly signaled to the decoder.

[0138] At the third level, per-block or per-encoding-unit time signals may exist. These may include time pattern signals for the blocks. This can be signaled to the decoder. If the per-block time refresh signal is set to 1, and the entire block is encoded without time information (e.g., according to a first time pattern), then this can be signaled to the decoder for the first block using a bit of the per-block time signal that can be set to 1. If the per-block time refresh signal is set to 0, then the first transform block (e.g., a 2×2 or 4×4 block) in the block can be encoded using time prediction (e.g., using a time buffer). In this case, the per-block time signal can be set to 0, indicating that time prediction has been used (e.g., encoded according to a second time pattern). If the time refresh signal for each tile is set to 0, then all other transform blocks within the tile can have a bit-time signal. This bit-time signal is set to 1 when the tile is encoded without time information, and is set to 0 when the transform coefficients from the previous frame at the same spatial location are first subtracted from the transform coefficients and then the difference is quantized and passed to the entropy encoder (i.e., when a second time mode and time buffer are to be used). The time mode signal for the block can be a block-based time parameter. The bit length of the value of the block-based time parameter can be one bit.

[0139] Figure 6D A representation 646 of a time signal for a 4×4 transform size (e.g., DDS transform) is shown. The data shown here (e.g., the displayed 1 and 0 values) may be referred to as a time surface. A 2×2 transform size can be transmitted in a corresponding manner. Figure 6C Displaying multiple tiles 652, 654 (e.g., similar to...) Figure 6D The elements of frame 648 are used. Time signals are organized using tiles 652 and 654. For 4×4 transforms and 32×32 tiles, there is an 8×8 time signal per tile (i.e., 32 / 4). For 2×2 transforms and 32×32 tiles, there is a 16×16 time signal per tile (i.e., 32 / 2). The set of time signals for frame 648 used for the residual is, for example, as... Figure 6D The term shown here can be referred to as a "time map," as an alternative term for a time surface. A time map, or surface, can be transmitted from the encoder to the decoder.

[0140] Figure 6DThis demonstrates how the timing signals of the first transform blocks 656 and 658 within blocks 652 and 654 can indicate whether the blocks will be processed in a first or second timing mode. The timing signals can be bits indicating the timing mode. If the bit is set to 1 for the first transform block, for example as shown for block 656, then this indicates that block 652 will be decoded according to the first timing mode, for example, without using a timing buffer. In this case, bits for other transform blocks may not be set. This reduces the amount of timing data transmitted to the decoder. For example, compared to the second block 654, where multiple values ​​(one per coding unit) exist, in the first block 652, only one bit value (1) is used for timing signaling. If the timing signaling bit of the first transform block is set to 0, for example as shown for block 658, then this indicates that... Figure 6D The instruction indicates that block 654 will be decoded according to a second timing pattern, for example, through timing prediction and use of a timing buffer. In this case, the timing signaling bits of the remaining transform blocks are set to 0 or 1, thereby providing a level of timing control at the (third) per-block level. The way in which the value indicating the timing pattern (e.g., 0 or 1) can cause a decrease in the signaling level between the encoder and decoder is assigned.

[0141] In some cases, as described above, time signaling at the third level can be effectively encoded when it is sent as metadata (e.g., sideband data).

[0142] In the cases described above, and for example as Figure 6D As shown, a time surface or map of a frame can be sent to a run-length encoder (e.g., where the frame is a "picture" of an encoded residual). Run-length encoding can be used to efficiently encode the time surface. Run-length encoding can be performed using the same run-length encoder used in one or more of the "entropy encoding" components in the first and second enhancement streams (or copies of this encoder process). In other cases, different run-length encoders can be used.

[0143] If run-length encoding is used, several operations can occur when the time surface is received by the run-length encoder. In one case, if the first time signal in the tile is 1, then the time signaling of the remaining tiles is skipped. This is illustrated by the arrow from the first transform block 656 with a value of 1. In this case, if the first time signal in the tile is 0, for example, as for... Figure 6D As shown in subsequent block 654, the time signaling bits of the block can then be scanned line by line (e.g., along the first row of transform blocks before moving to the next row of transform blocks, at each step of moving to the next column of transform blocks). Figure 6EIn this process, each tile has 8 rows and 8 columns. Therefore, for bit 0, the iteration is performed on the first 8 columns of the first row, and then the iteration is repeated on the same 8 columns of the second row, and so on, until all the time signals of the transform block for the particular tile are encoded.

[0144] In one scenario, the run-length encoder for the time signal can have two states representing bit values ​​of 0 and 1 (i.e., a second time mode and a first time mode). These can be used to encode runs of 1 and runs of 0. In another scenario, the run-length encoder can encode the run byte-by-byte, using 7 bits per byte to encode the run and using bit 7 to encode cases where more bits are needed to encode the run (set to 1) or when the context changes. By convention, the first symbol in the stream is always encoded as 0 or 1, so the decoder can initialize the state machine. A usable state machine 670 is shown in... Figure 7A middle.

[0145] In an example, the cost function used by the encoder to determine the time pattern (as described above) may have a cost that prevents encoding runs with the same value, for example, when encoding time pattern signaling data indicating the time pattern value as discussed above. In this way, the cost function can apply a penalty when a run of 1 changes to a run of 0, or vice versa. This explains the fact that stopping encoding runs during run-length encoding is entropy-significant. Evaluating the cost function may include rate distortion optimization (RDO) decisions that disadvantage changes in runs with the same value.

[0146] In one scenario, the information generated by the run-length encoder can be sent to another entropy encoder component. This could include a Huffman encoder. The Huffman encoder can write two Huffman codes to the metadata stream for each state and the Huffman-encoded value. Run-length encoding and entropy encoding can therefore utilize existing entropy encoding components and / or appropriately adapted copies of these components (e.g., as appropriately initialized threads). This simplifies encoding and decoding because the components can be reused with different configuration information. In one scenario, with two time pattern values ​​(e.g., 0 and 1), only run-length encoding may be available because there may be no benefit from additional Huffman encoding attributable to the finite number of symbols to be encoded.

[0147] Figure 6C and 7B These are two halves of a flowchart, 700 and 740, illustrating the time processing methods based on an instance. Time processing methods can be executed at the encoder. These time processing methods can implement some of the procedures described above. The processing methods can be applied to... Figure 6C The residual frames shown in the image.

[0148] At block 702, it is checked whether the current frame of the residual is an I-frame (i.e., an intra-coded frame). If the current frame of the residual is an I-frame, then the time buffer is refreshed at block 704, and the current frame of the residual is encoded as an inter-frame at block 706, where per-picture signaling is set to 1 at block 708. If it is determined at block 702 that the current frame of the residual is not an I-frame, then the first tile is selected, and a check is performed at block 710 to determine whether the temporal_refresh_per_tile flag is set (e.g., with a value of 1). This could be TR variable 644, such as... Figure 6C As shown on the right side. If the `temporal_refresh_per_tile` flag is set, then at the next block 712, the `temporal_type` flag of the cells within the current tile is analyzed. For example, for the first tile, these could be... Figure 7B The temporal_type flag of the cell is shown on the right side. At the next block 714, the percentage of the I or first time mode flag value (e.g., a '1' value) can be counted. If these are greater than 75%, then the time buffer is flushed at block 716 and inter-frame coding of the tiles is performed at block 718, where the time signal in each tile is set to 0 at block 720. If these are less than 75%, then the method proceeds to... Figure 7B (For example, via node A). If `temporal_refresh_per_tile` is not set (e.g., has a value of 0), then a similar process is performed, where the temporal-type flag of the analyzed cell is checked at block 722, and it is determined at block 724 whether more than 60% of the cells within the current tile have their `temporal_type` flag set to I or first-time mode (e.g., has a value of '1'). If so, then a similar process based on the previous 75% check is performed (e.g., blocks 716, 718, and 720 are executed). If less than 60% of the cells within the current tile have their `temporal_type` flag set to I or first-time mode, then the method proceeds again to... Figure 7B (For example, via node B).

[0149] Turning Figure 7B The second half of the flowchart, 740, and in Figure 1Starting with node A on the left-hand side, if less than 75% of the cells have I or the first time mode, then at block 726, check if the time buffer is empty. If the time buffer is empty, then at block 728, inter-frame coding is performed on the cells within the block, and at block 730, the time signal is set to 0 for the cells in the block. If the time buffer is not empty, then at block 732, intra-frame coding is performed on the cells within the block. In this case, the time signal for the first cell is set to 1, and at block 734, the time signal for all other cells in the block is set to 0.

[0150] Starting at node B on the right-hand side of Figure 13B, if less than 60% of the cells have an I or first time mode, then the first cell in the current tile is inter-coded at block 736, and the time signal of the first cell is set to 0 at block 738. Next, at block 740, it is checked whether the temporal_type of the co-located n+1 cell (i.e., the cell co-located in the next frame) is set to 1. If so, then at block 742, it is checked whether the residual value is less than 2. If the residual value is less than 2, then the residual is removed at block 744, for example, by setting the residual value to 0. If the residual value is not less than 2, or if the co-located cell is not set to 1, then at block 746, a determination is made regarding whether the next cell in the tile will be intra- or inter-coded based on the cost function. At block 748, the time signal of the next cell can be set according to the cost function classification. This can be repeated for the remaining cells in the tile. The method can be repeated for each tile in the frame, for example from the check of temporal_refresh_per_tile.

[0151] In this example, the encoder (or encoding process) can communicate with one or more remote devices. The encoder can be, for example... ​ The encoder shown in any of A, 3A and 3B.

[0152] The encoder can communicate across networks. In one case, the encoder can receive and / or transmit configuration data across networks. The encoder can receive one or more of encoder parameters, timing signaling, and residual masks. Timing signaling may include any of the timing signaling discussed herein.

[0153] In one scenario, encoder parameters received across the network can indicate one or more of the residual and time modes to be applied by the encoder. The encoder parameters can individually indicate the mode for each first-order stream, or indicate a common mode for both enhancement streams.

[0154] In one scenario, the encoder may have different configuration settings regarding remote or cloud configuration. In one mode, which may be a "default" mode, the encoder may be configured to make remote program calls across the network to retrieve initial configuration parameters to perform encoding as described herein. In another mode, which may be a "custom" mode, the encoder may retrieve local parameter values ​​indicating a specific user configuration, such as a specific set of tools used by the encoder and / or the configuration of those tools. In another scenario, the encoder may have different modes indicating which parameters to retrieve from a remote device and which parameters to retrieve from local storage.

[0155] In one scenario, timing signaling can instruct certain processing of a video data frame, such as as described above. Timing signaling can, for example, indicate the timing mode of a specific frame as described above (e.g., indicating mode 1 or 0 within or between frames). Timing signaling can be provided for one or both of the enhanced streams.

[0156] The encoder can send configuration data to and / or receive configuration data from a remote control server. The control server may include a server computing device implementing an application programming interface for receiving or sending data. For example, the control server may implement a RESTful interface, thereby transmitting data via (secure) Hypertext Transfer Protocol (HTTP) requests and responses. In another case, a side channel implemented using a specific communication protocol (e.g., at the transport or application layer) may be used for communication between the control server and the encoder via a network. The network may include one or more wired and / or wireless networks, including local area networks (LANs) and wide area networks (WANs). In one case, the network may include the Internet.

[0157] Using a cloud configuration as described in this article offers implementation advantages. For example, encoders can be remotely controlled, for instance, based on network-controlled systems and measurements. Encoders can also be upgraded to provide new functionality, for example, by upgrading firmware that provides additional data to the enhancement processing based on measurements or preprocessing supplied by one or more remote data sources or control servers. This provides a flexible way to upgrade and control legacy hardware.

[0158] The examples described in this paper consider the temporal and spatial characteristics of residuals. For example, in a residual image, details such as "edges" and "points" observable in the residual "image" exhibit minimal temporal correlation. This is because "edges" in a residual image typically do not translate or rotate as they would in a normal video stream. For instance, within a residual image, "edges" can actually change shape over time; for example, head turning can be captured within multiple residual image "edges," but this head turning may not move in a standard manner (because "edges" reflect complex differences depending on factors such as lighting, scaling factors, encoding factors, etc.). These temporal aspects of residual images, such as residual "video" comprising sequential residual "frames" or "pictures," typically differ from the temporal aspects of conventional images, such as normal video frames (e.g., in the Y, U, or V planes). Therefore, it is not obvious how conventional coding methods can be applied to residual images; in fact, it has been found that comparative video coding schemes and standard motion compensation methods are ineffective at encoding residual data (e.g., in a useful way).

[0159] Furthermore, many comparative video coding methods attempt to provide temporal prediction and motion compensation as defaults for regular video data. These "in-machine" methods may not only fail when applied to sequential residual images, but they can also consume unnecessary processing resources (e.g., resources that are used while actually corrupting video coding). They can also generate unnecessary bits that consume the assigned bit rate. How to solve these problems is not immediately obvious from conventional methods.

[0160] The examples described in this paper provide efficient ways to predict temporal features within residual images. Some examples use zero-motion vector prediction to efficiently predict temporal aspects and movement within the residual. It can be seen that these predictables are used for movement of relatively static features (e.g., applying a second temporal pattern, i.e., inter-frame prediction, to residual features that persist over time), and then a first temporal pattern (e.g., intra-frame prediction) is used for all other features. Therefore, some of the examples described in this paper do not attempt to waste resources and predict bit rates of temporarily irrelevant temporal features in the residual “video.”

[0161] In the encoding process described herein, the input video can be encoded into multiple encoded streams block by block. In this way, the first element block from a given frame in the input video can be encoded into multiple encoded streams without using the second element block from the same frame in the input video.

[0162] Other examples are described below. Other examples relate to a method for encoding an input video into multiple encoded streams such that the encoded streams can be combined to reconstruct the input video, the method comprising: receiving a full-resolution input video; downsampling the full-resolution input video to produce a downsampled video; encoding the downsampled video using a first codec to produce a base encoded stream; reconstructing the video from the encoded video to produce a reconstructed video; comparing the reconstructed video with the input video; and generating one or more additional encoded streams based on the comparison.

[0163] Compared to the reconstructed video, the input video can be a downsampled video.

[0164] According to the example method, comparing the reconstructed video with the input video includes: comparing the reconstructed video with the downsampled video to generate a first residual set, and generating one or more additional coded streams includes encoding the first residual set to generate a first-level coded stream.

[0165] Compared to reconstructed video, input video can be full-resolution input video and reconstructed video can be upsampled.

[0166] According to the example method, comparing the reconstructed video with the input video includes: upsampling the reconstructed video to produce an upsampled reconstructed video; and comparing the upsampled reconstructed video with the full-resolution input video to produce a second residual set, wherein producing one or more additional coded streams includes encoding the second difference to produce a second-level coded stream.

[0167] Therefore, in this example, the method can generate a base encoded stream, a first-level encoded stream, and a second-level encoded stream according to the example method defined above. Each of the first-level and second-level encoded streams may contain enhancement data used by the decoder to enhance the encoded base stream.

[0168] According to the example method, the steps of encoding the first residual set include: applying a transformation to the residual set to generate a coefficient set; applying a quantization operation to the coefficients to generate a quantized coefficient set; and applying an encoding operation to the quantized coefficients.

[0169] The transform can be, for example, a discrete cosine transform or a wavelet transform. In alternative instances, the transform can be a small transform that decomposes the block of elements into directional components (e.g., using a 2×2 kernel or a 4×4 kernel). For example, a 2×2 kernel could be a Hadamard transform. Further details regarding the transform can be found, for example, in patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. In another instance, the encoder can choose between different transforms to be used, such as between a 2×2 kernel and a 4×4 kernel. This allows for greater flexibility in how the residuals are encoded.

[0170] The selection can be based on the analysis of the data to be transformed.

[0171] The quantization can be, for example, linear quantization. A linear quantizer can use a variable-size dead zone. The encoding operation can be, for example, an entropy encoder, and can include run-length encoding and / or Huffman coding.

[0172] According to the example method, the steps of encoding the second residual set include: applying a transformation to the second residual set to generate a coefficient set; applying a quantization operation to the coefficients to generate a quantized coefficient set; and applying an encoding operation to the quantized coefficients.

[0173] The transform can be, for example, a discrete cosine transform or a wavelet transform. In alternative instances, the transform can be a small transform that decomposes the block of elements into directional components (e.g., using a 2×2 kernel or a 4×4 kernel). For example, a 2×2 kernel could be a Hadamard transform. Further details regarding the transform can be found, for example, in patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. In another instance, the encoder can choose between different transforms to be used, such as between a 2×2 kernel and a 4×4 kernel. This allows for greater flexibility in how the residuals are encoded.

[0174] The selection can be based on the analysis of the data to be transformed.

[0175] The first and second residual sets may have different transformations applied to them, and the selection may be predetermined or chosen during the process. The transformations used may be indicated in the header.

[0176] The quantization can be, for example, linear quantization. A linear quantizer can use a variable-size dead zone. The encoding operation can be, for example, an entropy encoder, and can include run-length encoding and / or Huffman coding.

[0177] The residual can be the difference between two videos or frames.

[0178] The steps of encoding the first set of residuals may include: classifying the first set of residuals based on a prior analysis of the first set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0179] In an example, the method includes analyzing a first set of residuals and, based on the analysis, performing or not performing the following steps: classifying the first set of residuals; and selecting a subset of the residuals to be transformed and encoded.

[0180] In an example, the method includes analyzing a first set of residuals and classifying the first set of residuals, and selecting a subset of the residuals to be transformed and encoded, such that the classification and / or selection steps are performed differentially based on the analysis.

[0181] Based on the instance method, the steps of applying the transformation are performed on the selected subset of residuals.

[0182] The steps of encoding the second residual set may include: classifying the second residual set based on a prior analysis of the second residual set; and selecting a subset of the residuals to be transformed and encoded.

[0183] In an example, the method includes analyzing a second set of residuals and, based on the analysis, performing or not performing the following steps: classifying the second set of residuals; and / or selecting a subset of the residuals to be transformed and encoded.

[0184] In an example, the method includes analyzing a second set of residuals and classifying the second set of residuals, and selecting a subset of the residuals to be transformed and encoded, such that the classification and / or selection steps are performed differentially based on the analysis.

[0185] Based on the instance method, the steps of applying the transformation are performed on the selected subset of residuals.

[0186] The encoded stream may be accompanied by one or more headers, which contain parameters indicating aspects of the encoding process to facilitate decoding. For example, the headers may include the codec used, the transforms applied, the quantization applied, and / or other decoding parameters.

[0187] In some instances, the quantization step may include adapting the quantization based on analysis of the coefficients to be transformed and / or data, such as residual data. In some instances, the distribution used in the quantization step may be adapted.

[0188] The step of encoding the first residual set may include deriving the time coefficient set from the time buffer and subtracting the time coefficient set from the coefficient set.

[0189] The steps of encoding the second residual set may include deriving the time coefficient set from the time buffer and subtracting the time coefficient set from the coefficient set.

[0190] The preceding text describes how the grading and selection steps can be applied to the residual data, how the subtraction of time coefficients can be performed, and how quantization can be adjusted. Each of these steps can be applied pre-definedly and selectively, or based on analysis of the input video, downsampled video, reconstructed video, upsampled video, or any combination thereof, to improve the overall performance of the encoder. These steps can be applied selectively based on a predetermined set of rules or deterministically based on performance analysis or feedback.

[0191] According to the example method, the first codec is a hardware-based codec, preferably AVC, HEVC, AV1, VP8 or VP9.

[0192] According to other examples, a method is provided for encoding an input video into multiple encoded streams such that the encoded streams can be combined to reconstruct the input video. The method includes: receiving a full-resolution input video; generating a base encoded stream at a resolution lower than that of the full-resolution input video; determining a time pattern for one or more additional encoded streams used to reconstruct the full-resolution input video together with the base encoded stream; and generating one or more additional encoded streams by selectively applying a time buffer based on the time pattern.

[0193] The method may include determining a time pattern as either a first time pattern without using a time buffer or a second time pattern using a time buffer. The time buffer can be used to apply time prediction.

[0194] The method may include: obtaining time pattern metadata of multiple coding units at an encoder; determining a time pattern based on the obtained time pattern metadata for encoding the multiple coding units; and generating time pattern signaling data of the multiple coding units based on the determined time pattern and the obtained time pattern metadata.

[0195] Temporal prediction can be applied at the encoder by subtracting the dequantized set of transformed coefficients stored in the temporal buffer from the current set of transformed coefficients. The current set of transformed coefficients can be associated with the current frame in the full-resolution input video, and the dequantized set of transformed coefficients can be associated with the previous frame in the full-resolution input video.

[0196] In some instances, determining the temporal pattern may include estimating a cost function. The cost function may include a function of the full-resolution input video and one or more additional coded streams. The cost function can be evaluated by encoding one or more additional coded streams using two temporal patterns and comparing one or more metrics determined for each temporal pattern. The cost function can also be evaluated for one or more portions of a frame, such as one or more coding units.

[0197] In some instances, determining the time pattern may include setting time refresh parameters for the frame. Time refresh parameters can be used to refresh the signaling time buffer, such as zeroing out one or more values ​​within the buffer. In some instances, time signaling may be used at the encoder to indicate per-tile time refreshes.

[0198] In some instances, in a second time mode that uses a time buffer, the time refresh parameter can be configured to temporarily implement the processing associated with the first time mode.

[0199] In some instances, the encoder, as stated in any of these statements, may receive configuration parameters via a network, for example, from a remote server device. In some instances, the encoder may additionally or alternatively transmit configuration parameters to the remote server device. The configuration parameters may configure the operation of the encoder as described in any of these statements.

[0200] Other examples relate to a method for encoding an input video, the method comprising: receiving the input video at a first resolution; generating one or more residuals at one or more corresponding resolutions based on the difference between the input video and one or more reconstructed videos; modifying the one or more residuals based on a selected residual mode; and generating one or more encoded streams from the one or more modified residuals.

[0201] The method may include: downsampling an input video to generate a downsampled video at a second resolution; encoding the downsampled video using a first codec to generate a base encoded stream; reconstructing a video from the encoded video to generate a reconstructed video; comparing the reconstructed video with the input video; and generating one or more additional encoded streams based on the comparison.

[0202] A set of residuals can be at a first resolution, and a set of residuals can be at a second resolution.

[0203] Modifying one or more residuals may include: receiving a set of residual weights; and applying the set of residual weights to a set of residuals to produce modified residuals.

[0204] The method may further include limiting the modified residuals using a set of thresholds. In some instances, one or more of a residual weight set and a threshold set are determined based on the classification of the input video. In some instances, the residual weight set includes a residual mask received from a remote location. In some instances, one or more of the residual weight set and the threshold set are applied to a group of residuals.

[0205] Other examples relate to a method for encoding an input video, the method comprising: receiving the input video at a first resolution; obtaining a desired bit rate of one or more mixed video streams; generating one or more residuals at one or more corresponding resolutions based on the difference between the input video and one or more reconstructed videos; determining quantization parameters for the one or more residuals based on the desired bit rate; quantizing the one or more residuals based on the quantization parameters; and generating one or more encoded streams from the one or more quantized residuals.

[0206] The method may include: downsampling an input video to generate a downsampled video at a second resolution; encoding the downsampled video using a first codec to generate a base encoded stream; reconstructing a video from the encoded video to generate a reconstructed video; comparing the reconstructed video with the input video; and generating one or more additional encoded streams based on the comparison.

[0207] A set of residuals can be at a first resolution, and a set of residuals can be at a second resolution.

[0208] Determining quantization parameters may include: receiving the state of a buffer that receives one or more encoded streams and a base encoded stream; and using the state to determine the quantization parameters.

[0209] Determining quantization parameters may include: receiving the state of the underlying coding stream of the underlying coding layer; and using the state to determine the quantization parameters.

[0210] Quantization parameters can be determined for each frame, residual, and / or residual group. In one case, the quantization parameters of a frame are determined based on the target data size of the frame and the current data size of the frame using a previous set of quantization parameters. In another case, the quantization parameters are based on a previous set of quantization parameters.

[0211] In one instance, the method includes limiting the determined quantization parameters based on the current state of the encoder.

[0212] In one case, the quantization parameters are used to determine the step size for quantization. In another case, the quantization parameters include the Q value, where the step size for quantization is an inverse function of the Q value.

[0213] The instance method further includes: sending the basic encoded stream. The instance method further includes: sending the first-level encoded stream. The instance method further includes: sending the second-level encoded stream.

[0214] According to another example, a decoding method is provided. Some examples relate to a method for decoding multiple encoded streams into a reconstructed output video, the method comprising: receiving a first base encoded stream; decoding the first base encoded stream according to a first codec to produce a first output video; receiving one or more additional encoded streams; decoding the one or more additional encoded streams to produce a residual set; and combining the residual set with the first video to produce a decoded video.

[0215] In this example, the method includes retrieving multiple decoding parameters from the header. These decoding parameters indicate which program steps are included in the encoding process.

[0216] In this example, the steps of decoding one or more additional encoded streams to produce a residual set include: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to produce the residual set.

[0217] In an example, the step of decoding one or more additional encoded streams to produce a set of residuals includes: predicting a subset of residuals based on the residuals from co-location from a time buffer.

[0218] In one example, the method may include receiving a first-level encoded stream and receiving a second-level encoded stream. In this example, the step of decoding one or more additional encoded streams to produce a residual set includes: decoding the first-level encoded stream to derive a first residual set; wherein the step of combining the residual set with a first video to produce a decoded video includes: combining the first residual set with a first output video to produce a second output video; upsampling the second output video to produce an upsampled second output video; decoding the second-level encoded stream to derive a second residual set; and combining the second residual set with the second output video to produce a reconstructed output video.

[0219] In this example, the step of upsampling the second output video to produce an upsampled second output video includes adding the values ​​of elements from a first residual set of blocks in the derived upsampled second output video to the corresponding blocks in the upsampled second output video. The blocks may be 2×2 blocks. This addition step may be performed selectively based on predetermined values ​​or signals contained in the header.

[0220] In the example, the steps of decoding the first-level encoded stream to derive the first residual set include: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to produce the first residual set.

[0221] In this example, the step of decoding the first-level coded stream to derive the first set of residuals includes applying a deblocking filter configured to apply a mask to the residual blocks. The mask may be weighted according to a predefined set of weights.

[0222] In the example, the steps of decoding the second-level encoded stream to derive the second residual set include: applying an entropy decoding operation; applying a dequantization operation; and applying an inverse transform operation to produce the second residual set.

[0223] The inverse transform operation can be the inverse of the operation defined above, or it can be a largely mirror operation. That is, a 2×2 block or a 4×4 block transform can be selectively applied. The transform can be detected by a decoding method or transmitted in the header.

[0224] If a 2×2 transformation is used, the coefficients can be modified by adding the value of the residual on which the transformed block of the predicted residual is based. If a 4×4 transformation is used, the coefficients will be modified by adding the average of the four residuals.

[0225] According to other examples, a method is provided for decoding one or more encoded streams into a reconstructed output video, the method comprising: receiving a first base encoded stream; decoding the first base encoded stream according to a first codec to produce a first output video; receiving one or more additional encoded streams; receiving data indicating a time pattern for one or more portions of the one or more additional encoded streams; decoding the data indicating the time pattern and configuring one or more corresponding time buffers for the one or more additional encoded streams; decoding the one or more additional encoded streams to produce a residual set, including selectively applying data from the one or more time buffers to the decoded one or more additional encoded streams; and combining the residual set with the first video to produce the decoded video.

[0226] It can be applied in a manner corresponding to the decoding method, as can be applied to variations in the encoding method.

[0227] In one instance, the method further includes: receiving time signaling indicating a time refresh of the frame; and zeroing the values ​​in the time buffers before selectively applying data relating to the decoded data of the frame from one or more time buffers.

[0228] In one instance, selectively applying data from one or more time buffers to one or more additional encoded streams after decoding includes subtracting data from one or more time buffers in response to an indication of a second time mode, and not subtracting data from one or more time buffers in response to an indication of a first time mode.

[0229] In one instance, the data indicating the time pattern of one or more portions of one or more additional encoded streams includes bits for each encoded unit.

[0230] The method may further include displaying or outputting the reconstructed output.

[0231] Some of the examples described in this article involve time processing associated with one or more encoded streams. The encoder can be configured to apply time processing using a time buffer.

[0232] The encoder can be configured to encode input video into multiple encoded streams, for example, such that the encoded streams can be combined to reconstruct the input video. The encoder can be configured to receive input video including corresponding frames.

[0233] In some cases, each frame of a given frame can be divided into multiple tiles. Each tile can be divided into multiple blocks.

[0234] In other cases, each frame of the corresponding frame can be divided into multiple planes. Each plane can be divided into multiple tiles. Each tile of the multiple tiles can be divided into multiple blocks.

[0235] The encoder can also be configured to generate a base encoded stream using a base encoder. The encoder can be configured to determine a timing pattern for reconstructing one or more additional encoded augmented streams of input video together with the base stream. One or more additional encoded augmented streams can be generated using an augmentation encoder. The augmentation encoder may differ from the base encoder. The timing pattern may be one of a first timing pattern and a second timing pattern, wherein the first timing pattern does not apply non-zero values ​​from the timing buffer used to generate one or more additional encoded augmented streams, and the second timing pattern applies non-zero values ​​from the timing buffer used to generate one or more additional encoded augmented streams. Applying non-zero values ​​from the timing buffer in the second timing pattern may include deriving a set of non-zero timing coefficients from the timing buffer and using said set of non-zero timing coefficients to modify the current set of coefficients used to generate one or more additional encoded streams.

[0236] An encoder can be configured to generate one or more additional encoded enhanced streams based on data derived from a base encoded stream and the input video, according to a determined temporal pattern. Generating the one or more additional encoded enhanced streams may include applying transforms to each of a series of blocks. The encoder can be configured for one or more determined temporal patterns of frames, tiles, or blocks of the input video.

[0237] The encoder can be configured to determine the time pattern based on a cost function. The cost function may include the cost of transmitting time pattern signaling data for the time pattern. The cost of transmitting time pattern signaling data for the time pattern can make one value of the time pattern signaling data disadvantageous compared to other values ​​of the time pattern signaling data.

[0238] The cost function may include a function of the input video and at least one of one other encoded augmented streams. An encoder may be configured to evaluate the cost function of frames of the input video. An encoder may be configured to evaluate the cost function of tiles of the input video. An encoder may be configured to evaluate the cost function of blocks of the input video.

[0239] To evaluate the cost function, the encoder can be configured to encode one or more additional encoded augmented streams using each of a first time mode and a second time mode. To evaluate the cost function, the encoder can also be configured to compare one or more metrics determined for each of the first and second time modes.

[0240] To determine the time pattern, the encoder can be configured to obtain time pattern metadata for a set of blocks, comprising multiple blocks. The encoder can also be configured to determine the time pattern to be used for encoding the set of blocks based on the time pattern metadata.

[0241] The encoder can be configured to encode time pattern signaling data indicating the time pattern for the one or more additional encoded streams separately from the one or more additional encoded streams. The encoder can use run-length encoding to encode the time signaling data.

[0242] The encoder can be configured to determine whether to refresh the time buffer for a given frame of the input video based on at least one of: a first scale of a block of the given frame, a first time pattern to be used for reconstructing the given frame at the first scale; and a second scale of a block of the given frame, a second time pattern to be used for reconstructing the given frame at the second scale. Refreshing the time buffer for a given frame may include setting a value within the time buffer to zero.

[0243] The encoder can be configured to determine the timing pattern of the second frame of the input video after the first frame. The encoder can also be configured to omit the quantized values ​​of the transformed blocks of the first frame from one or more additional encoded augmented streams based on the timing pattern determined for the second frame.

[0244] The encoder can be configured to generate time-mode signaling data that indicates the time mode for one or more additional encoded enhancement streams used by the decoder. The time-mode signaling data can be compressed.

[0245] Some of the examples described in this article involve time processing associated with one or more encoded streams. The decoder can be configured to apply time processing using a time buffer.

[0246] The decoder can be configured to receive multiple encoded streams. These multiple encoded streams may have been generated according to the examples above. The decoder can be configured to decode the multiple encoded streams into a reconstructed output video. The decoder can be configured to receive a first output video, which includes the output of a base decoder applied to the base-level encoded streams. The decoder can be configured to receive one or more additional encoded streams. The decoder can be configured to decode corresponding frames of the one or more additional encoded streams to derive corresponding residual sets.

[0247] In some cases, each frame of a given frame can be divided into multiple tiles. Each tile can be divided into multiple blocks.

[0248] In other cases, each frame of the corresponding frame can be divided into multiple planes. Each plane can be divided into multiple tiles. Each tile of the multiple tiles can be divided into multiple blocks.

[0249] The decoder can also be configured to combine the residual set with the first output video to produce a reconstructed output video. To decode a corresponding frame, the decoder can be configured to obtain a pre-residual set from one or more additional coded streams for each of multiple blocks. To decode a corresponding frame, the decoder can be configured to derive a temporal prediction set using a temporal buffer. To decode a corresponding frame, the decoder can be configured to combine the temporal prediction set with the pre-residual set to output data that is then combined with the first output video.

[0250] The decoder can be configured to decode one or more additional encoded streams according to a timing pattern. The timing pattern can be one of a first timing pattern and a second timing pattern, wherein the first timing pattern does not apply non-zero values ​​from the timing buffers used to generate the one or more additional encoded streams, and the second timing pattern applies non-zero values ​​from the timing buffers used to generate the one or more additional encoded streams. The timing pattern can be determined at the encoder and signaled to the decoder according to the above example.

[0251] The decoder can be configured to receive time pattern signaling data indicating the time pattern for each coded unit, i.e., each block. The time pattern signaling data received by the decoder may have been encoded separately from one or more other coded streams received by the decoder.

[0252] According to other examples, an apparatus may be provided for encoding a dataset into an encoded dataset including a header and a payload. The apparatus may be configured to encode an input video according to the steps described above. The apparatus may include a processor configured to perform any of the methods described above.

[0253] According to other examples, an apparatus may be provided for decoding a dataset, including a header and a payload, into a reconstructed video. The apparatus may be configured to decode the output video according to the steps described above. The apparatus may include a processor configured to perform the methods of any of the above aspects.

[0254] An encoder and a decoder may also be provided.

[0255] According to other examples, a computer-readable medium may be provided that, when executed by a processor, causes the processor to perform any of the methods described above. The above embodiments should be understood as exemplary examples. Other examples are contemplated. It should be understood that any feature described with respect to any example may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other example, or in any combination of any other example. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the appended claims.

Claims

1. An encoder configured to encode an input video into a plurality of encoded streams such that the encoded streams can be combined to reconstruct the input video, the encoder configured to: receive an input video comprising respective frames, each frame of the respective frames being divided into a plurality of tiles and each tile of the plurality of tiles being divided into a plurality of blocks; generate a base encoded stream using a base encoder; determine a temporal pattern for one or more further encoded enhancement streams used to reconstruct the input video with the base encoded stream, the one or more further encoded enhancement streams being generated using an enhancement encoder, wherein the temporal pattern is one of a first temporal pattern and a second temporal pattern, the first temporal pattern indicating a non-zero value from a temporal buffer not applied when generating the one or more further encoded enhancement streams, the second temporal pattern indicating a non-zero value from the temporal buffer applied when generating the one or more further encoded enhancement streams; and generate the one or more further encoded enhancement streams based on data derived from the base encoded stream and the input video in accordance with the determined temporal pattern, wherein generating the one or more further encoded enhancement streams comprises applying a transform to each of a chain of blocks of the plurality of blocks, and wherein the temporal pattern is determined for one or more of a frame, tile or block of the input video.

2. The encoder of claim 1, wherein the encoder is configured to determine the temporal pattern based on a cost function.

3. The encoder of claim 2, wherein the cost function comprises a cost of transmitting temporal pattern signaling data for the temporal pattern.

4. The encoder of claim 3, wherein the cost of transmitting temporal pattern signaling data for the temporal pattern penalizes one value of the temporal pattern signaling data over other values of the temporal pattern signaling data.

5. The encoder of any of claims 2 to 4, wherein the cost function comprises a function of the input video and at least one of the one or more further encoded enhancement streams.

6. The encoder of any of claims 2 to 4, wherein the encoder is configured to evaluate the cost function for a frame of the input video, a tile of the input video and a block of the input video.

7. The encoder of any of claims 2 to 4, wherein the cost function is based on a difference between a frame of the input video and a corresponding frame of a reconstructed version of the input video.

8. The encoder of any of claims 2 to 4, wherein the encoder is configured to evaluate the cost function by: encoding the one or more further encoded enhancement streams using each of the first temporal pattern and the second temporal pattern; and comparing one or more metrics determined for each of the first temporal pattern and the second temporal pattern.

9. The encoder of any of claims 2-4, wherein the cost function is based on a sum of absolute differences calculation.

10. The encoder of any of claims 1-4, wherein the encoder is configured to encode a first block element from a given frame in the input video to the plurality of encoded streams without using a second block element from the given frame in the input video.

11. The encoder of any of claims 1-4, wherein the enhancement encoder is different from the base encoder.

12. The encoder of any of claims 1-4, wherein the enhancement encoder is configured to receive residual data generated by a comparison of data derived from the input video and data derived from the base encoded stream, wherein the one or more additional encoded enhancement streams comprise encoded residual data decodable to reconstruct one or more additional sets of residual data for application to a decoded version of the base encoded stream.

13. The encoder of any of claims 1-4, wherein determining the temporal pattern comprises: obtaining temporal pattern metadata for a set of blocks of the plurality of blocks; and determining the temporal pattern based on the temporal pattern metadata for encoding the set of blocks.

14. The encoder of claim 13, wherein the encoder is configured to generate temporal pattern signaling data for the set of blocks based on the temporal pattern and the temporal pattern metadata.

15. The encoder of any of claims 1-4, wherein the encoder is configured to encode temporal pattern signaling data indicating the temporal pattern for the one or more additional encoded streams separately from the one or more additional encoded streams.

16. The encoder of claim 15, wherein the encoder is configured to encode the temporal pattern signaling data using run-length encoding.

17. The encoder of claim 16, wherein the run-length encoding is performed using the same run-length encoding process as used by the enhancement encoder to encode the one or more additional encoded enhancement streams.

18. The encoder of claim 16 or claim 17, wherein the encoder is configured to: encode temporal pattern signaling data indicating the temporal pattern of a first block within a tile using the run-length encoding, the temporal pattern of the first block being the second temporal pattern; and skip the run-length encoding of the temporal pattern signaling data for remaining blocks within the tile.

19. The encoder of any of claims 1-4, wherein applying non-zero values from the temporal buffer in the second temporal pattern comprises deriving a set of non-zero temporal coefficients from the temporal buffer and using the set of non-zero temporal coefficients to modify a current set of coefficients used to generate the one or more additional encoded enhancement streams. ​ 20. The encoder of any of claims 1-4, wherein the encoder is configured to assign a respective value to at least one of: a frame-based temporal parameter for a frame of the input video; a tile-based temporal parameter for a tile of the input video; and a block-based temporal parameter for a block of the input video, wherein the value assigned to the frame-based temporal parameter indicates the temporal mode for the frame of the input video, the value assigned to the tile-based temporal parameter indicates the temporal mode for the tile of the input video, and the value assigned to the block-based temporal parameter indicates the temporal mode for the block of the input video.

21. The encoder of claim 20, wherein at least one of: a bit length of the value of the frame-based temporal parameter is one bit; a bit length of the value of the tile-based temporal parameter is one bit; and a bit length of the value of the block-based temporal parameter is one bit.

22. The encoder of claim 20, wherein the encoder is configured to assign the value of the block-based temporal parameter for the block of the input video to further indicate the temporal mode for the tile including the block.

23. The encoder of any of claims 1-4, wherein the encoder is configured to determine whether to flush the temporal buffer for a given frame of the input video based on at least one of: a first proportion of blocks of the given frame for which the first temporal mode is to be used to reconstruct the given frame; and a second proportion of the blocks of the given frame for which the second temporal mode is to be used to reconstruct the given frame.

24. The encoder of claim 23, wherein the flushing of the temporal buffer for the given frame comprises setting values within the temporal buffer to zero.

25. The encoder of any of claims 1-4, wherein the encoder is configured to: determine the temporal mode for a second frame of the input video after a first frame; and omit quantized values of transformed blocks of the first frame from the one or more additional encoded enhancement streams based on the temporal mode determined for the second frame.

26. The encoder of claim 25, wherein the encoder is configured to use the temporal mode determined for the second frame to control a comparison between the quantized values and one or more thresholds to determine whether to omit the quantized values.

27. The encoder of any of claims 1-4, wherein the encoder is configured to generate temporal mode signaling data indicating the temporal mode of the one or more additional encoded enhancement streams for a decoder.

28. The encoder of claim 27, wherein the temporal mode signaling data is compressed.

29. A method of encoding an input video into a plurality of encoded streams such that the encoded streams can be combined to reconstruct the input video, the method comprising: receiving an input video comprising respective frames, each frame of the respective frames being divided into a plurality of tiles and each tile of the plurality of tiles being divided into a plurality of blocks; generating a base encoded stream using a base encoder; determining a temporal pattern for one or more further encoded enhancement streams used to reconstruct the input video with the base encoded stream, the one or more further encoded enhancement streams being generated using an enhancement encoder, wherein the temporal pattern is one of a first temporal pattern and a second temporal pattern, the first temporal pattern indicating a non-zero value from a temporal buffer not applied when generating the one or more further encoded enhancement streams, the second temporal pattern indicating a non-zero value from the temporal buffer applied when generating the one or more further encoded enhancement streams; and generating the one or more further encoded enhancement streams based on data derived from the base encoded stream and the input video in accordance with the determined temporal pattern, wherein generating the one or more further encoded enhancement streams comprises applying a transform to each of a run of blocks of the plurality of blocks, and wherein the temporal pattern is determined for one or more of a frame, tile or block of the input video.

30. The method of claim 29, comprising determining the temporal pattern based on a cost function.

31. The method of claim 30, wherein the cost function comprises a cost of transmitting temporal pattern signalling data for the temporal pattern.

32. The method of claim 31, wherein the cost of transmitting temporal pattern signalling data for the temporal pattern penalises one value of the temporal pattern signalling data over other values of the temporal pattern signalling data.

33. The method of any of claims 30 to 32, wherein the cost function comprises a function of the input video and at least one of the one or more further encoded enhancement streams.

34. The method of any of claims 30 to 32, comprising evaluating the cost function for a frame of the input video, a tile of the input video and a block of the input video.

35. The method of any of claims 29 to 32, comprising encoding temporal pattern signalling data indicating the temporal pattern for the one or more further encoded streams separately from the one or more further encoded streams using run length encoding, wherein the run length encoding is performed using the same run length encoding process as used by the enhancement encoder to encode the one or more further encoded enhancement streams.

36. The method of any of claims 29-32, comprising determining whether to flush the temporal buffer for a given frame of the input video based on at least one of: a first proportion of blocks of the given frame for which the first temporal mode is to be used to reconstruct the given frame; and a second proportion of the blocks of the given frame for which the second temporal mode is to be used to reconstruct the given frame, wherein the flushing of the temporal buffer for the given frame comprises setting values within the temporal buffer to zero.

37. The method of any of claims 29-32, comprising: determining the temporal mode for a second frame of the input video after the first frame; controlling a comparison between quantized values of transformed blocks of the first frame from the one or more further encoded enhancement streams and one or more thresholds using the temporal mode determined for the second frame to determine whether the quantized values are to be omitted; and omitting the quantized values of transformed blocks of the first frame from the one or more further encoded enhancement streams based on the temporal mode determined for the second frame.

38. The method of any of claims 29-32, comprising: generating temporal mode signaling data indicative of the temporal mode of the one or more further encoded streams for a decoder.

39. The method of claim 38, wherein the temporal mode signaling data is compressed.

40. A non-transitory computer readable medium having stored thereon computer- executable instructions, the computer-executable instructions being executable by a processor and, when executed, cause the processor to perform the method of any of claims 29-39.

Images