Encoding and decoding a sequence of pictures

Abstract

An apparatus for decoding a sequence of pictures from a data stream is configured to decode a picture of the sequence by deriving a residual transform signal of the picture from the data stream, combining the residual transform signal with a buffered transform signal of a previous picture of the sequence to obtain a transform signal of the picture, the transform signal representing the picture in terms of spectral components, and subjecting the transform signal to a spectral-spatial transformation, the buffered transform signal including a selection from the spectral components representing the previous picture.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to an apparatus for decoding a sequence of pictures and an apparatus for encoding a sequence of pictures. Further embodiments relate to a method for decoding a sequence of pictures and a method for encoding a sequence of pictures. Further embodiments relate to a computer program for performing the same methods. Further embodiments relate to a data stream obtained by a method for encoding a sequence of pictures. Some embodiments relate to partial-temporal prediction for image sequence coding. [Background technology]

[0002] When transmitting a sequence of images, such as a sequence of images captured by a camera or a sequence of screen captures of a computer desktop, only limited bandwidth is available to transfer this information. To address this issue, image data is lossily compressed, for example, by first incorporating a transform that exploits spatial and / or temporal correlations in the source data, and then quantizing and encoding the resulting coefficients of the decorrelated transform. To enable temporal prediction, both the encoder and decoder must maintain data for past frames in units referred to as "frame buffers." In FPGA implementations, this creates the additional burden of having to simultaneously transfer large amounts of data from both the data source, such as a camera or computer desktop, and the frame buffer. Because bandwidth between the source, frame buffer, and processing device is also a limiting factor, maintaining a full-sized frame buffer can be limiting or even impossible for applications. Summary of the Invention [Problem to be solved by the invention]

[0003] Therefore, a video coding concept that provides an improved trade-off between a good rate-distortion relationship for coded video data (the ratio between the rate, e.g., the size of the data at which the sequence is coded, and the distortion, e.g., the loss of information or quality) and low buffer requirements (e.g., a low-sized buffer needed to encode or decode a sequence of pictures) would be desirable. [Means for solving the problem]

[0004] According to one embodiment, an apparatus for decoding a sequence of pictures from a data stream is configured to decode a picture of the sequence (e.g., the currently decoded picture) by deriving a residual transform signal of the picture from the data stream, combining the residual transform signal with a buffered transform signal of a previous picture of the sequence (e.g., the picture (directly) previous to the currently decoded picture in the sequence) to obtain a transform signal of the picture, where the transform signal represents the picture in terms of spectral components (e.g., the transform signal is independent of the previous picture), and subjecting the transform signal to a spectral-spatial transform (e.g., an inverse frequency transform). According to this embodiment, the buffered transform signal includes a selection from the spectral components representing the previous picture.

[0005] According to one embodiment, an apparatus for encoding a sequence of pictures into a data stream is configured to encode a picture of a sequence (e.g., the currently coded picture) by subjecting a spatial representation of the picture to a spatio-spectral transform (e.g., a wavelet transform or a frequency transform) to obtain a transform signal, the transform signal representing the picture in terms of spectral components (e.g., the transform signal is independent of previous pictures), deriving a residual transform signal based on the buffered transform signal and the transform signal of a previous picture of the sequence (e.g., the picture (directly) preceding the currently decoded picture in the sequence), and encoding the residual transform signal of the picture into a data stream. According to this embodiment, the buffered transform signal includes a selection from the spectral components representing the previous picture.

[0006] Embodiments of the present invention rely on the idea of ​​enhancing the rate-distortion relationship for encoding a transform signal representing a picture to be coded with spectral components by using a buffered transform signal obtained from a previous picture to obtain a residual transform signal, which is then coded into a data stream. Correlation between subsequent pictures in a sequence of pictures may thus be considered, in examples, as temporal correlation and may be exploited so that the residual transform signal may have a lower data rate than the transform signal. Buffering a selection from the spectral components of a previous picture makes it possible to control the buffer size required to buffer the buffered control signal within the target application. For example, the selection need not necessarily include all spectral components, but may include a subset of the spectral components of the picture, thereby reducing the data rate of the buffered transform signal. In contrast to temporal prediction performed on an untransformed picture, i.e., in the spatial domain, prior to spatial-spectral transformation, examples of the concepts disclosed herein may provide a computationally efficient temporal prediction method without buffering the entire picture. In particular, buffering a selection of spectral components of a previous picture may make it possible to achieve a relatively high reduction in the data rate of the residual transform signal (i.e., a relatively high enhancement of the rate-distortion relationship of the coded picture) with a relatively moderate buffer size. These advantages apply equally to the encoder and decoder sides, such that at the decoder side, the spectral components of the previous picture that were used to derive the residual transform signal of the current picture at the encoder side are buffered to determine the transform signal of the current picture based on the residual transform signal.

[0007] According to one embodiment, a method for decoding a sequence of pictures from a data stream comprises the steps of: deriving a residual transform signal of a picture from the data stream; combining the residual transform signal with a buffered transform signal of a previous picture of the sequence to obtain a transform signal of the picture, the transform signal representing the picture in terms of spectral components; and subjecting the transform signal to a spectral-spatial transformation.

[0008] According to one embodiment, a method for encoding a sequence of pictures into a data stream comprises the steps of encoding a sequence of pictures by subjecting a spatial representation of the picture to a spatio-spectral transform to obtain a transform signal, the transform signal representing the picture in terms of spectral components, deriving a residual transform signal based on the buffered transform signal and the transform signal of a previous picture of the sequence, and encoding the residual transform signal of the picture into a data stream, wherein the buffered transform signal includes a selection from the spectral components representing the previous picture.

[0009] According to one embodiment, a non-transitory digital storage medium has a computer program stored thereon which, when executed by a computer, performs a method for encoding a sequence of pictures into a data stream, the method comprising the steps of encoding a sequence of pictures by: subjecting a spatial representation of the picture to a spatio-spectral transform to obtain a transform signal, the transform signal representing the picture in terms of spectral components; deriving a residual transform signal based on the buffered transform signal and the transform signal of a previous picture of the sequence; and coding the residual transform signal of the picture into a data stream, the buffered transform signal including a selection from the spectral components representing the previous picture.

[0010] According to one embodiment, a non-transitory digital storage medium has a computer program stored thereon which, when executed by a computer, performs a method for decoding a sequence of pictures from a data stream, the method comprising the steps of: deriving a residual transform signal of a picture from the data stream; and decoding a picture of the sequence by combining the residual transform signal with a buffered transform signal of a previous picture of the sequence to obtain a transform signal of the picture, the transform signal representing the picture in terms of spectral components; and subjecting the transform signal to a spectral-spatial transformation, the buffered transform signal including a selection from the spectral components representing the previous picture.

[0011] According to one embodiment, a data stream has a sequence of pictures encoded therein, the sequence of pictures being encoded into a data stream by a method for encoding a sequence of pictures into a data stream, the method comprising the steps of encoding a picture of the sequence by subjecting a spatial representation of the picture to a spatio-spectral transformation to obtain a transform signal, the transform signal representing the picture in terms of spectral components, deriving a residual transform signal based on the buffered transform signal and the transform signal of a previous picture of the sequence, and coding the residual transform signal of the picture into the data stream, the buffered transform signal including a selection from the spectral components representing the previous picture.

[0012] Embodiments of the present invention are described in detail below with reference to the accompanying drawings. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 2 illustrates an encoder according to one embodiment. [Figure 2] FIG. 2 illustrates a decoder according to one embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of a spatial-spectral transformation. [Figure 4] FIG. 2 illustrates an example of segmenting a picture into spectral components. [Figure 5] FIG. 10 illustrates an example of a rearranged representation of a transformed picture. [Figure 6] FIG. 2 illustrates an encoder according to another embodiment. [Figure 7] FIG. 4 illustrates a decoder according to another embodiment. [Figure 8] FIG. 2 illustrates an encoder according to another embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a refresh sequence according to an embodiment. [Figure 10] FIG. 10 is a diagram illustrating another example of an encoding scheme. DETAILED DESCRIPTION OF THE INVENTION

[0014] Although the embodiments are described in detail below, it should be understood that the embodiments provide many applicable concepts that can be implemented in a wide variety of coding concepts. The specific embodiments described are merely illustrative of particular ways to implement and use the concepts and do not limit the scope of the embodiments. In the following description, numerous details are set forth to provide a more thorough explanation of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that other embodiments can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the examples described herein. Furthermore, features of different embodiments described herein can be combined with each other unless specifically stated otherwise.

[0015] In the following description of the embodiments, the same or similar elements or elements having the same functions are given the same reference numerals or are identified by the same names, and repeated descriptions of elements given the same reference numerals or are identified by the same names are generally omitted. Therefore, the descriptions provided for elements having the same or similar reference numerals or identified by the same names can be interchangeable or applied to each other in different embodiments.

[0016] FIG. 1 illustrates an apparatus 10 for encoding or coding a sequence of pictures 15 into a data stream 14 according to one embodiment. The apparatus 10 may also be referred to as an encoder 10. The pictures of the sequence 15 may have an order 16, e.g., a temporal order or a picture order. The sequence 15 includes a picture 12, e.g., a currently coded picture, and a previous picture 12* that precedes the picture 12 in the order 16 of the sequence 15. For example, the previous picture 12* is a picture that was previously coded relative to the current picture 12, e.g., immediately before the current picture 12. FIG. 2 illustrates a corresponding decoder 11, i.e., an apparatus 11 for decoding a sequence of pictures from a data stream 14, according to one embodiment. In other words, the decoder 11 is for reconstructing a picture 12′ from the data stream; the apostrophe is used to indicate that the picture 12′ reconstructed by the decoder 11 may deviate from the picture 12 originally coded by the apparatus 10, e.g., with respect to coding loss introduced by quantization. The same may be true for further signals introduced below. In the following description, reference is made to both Figures 1 and 2.

[0017] The encoder 10 comprises a transform module 20 configured to subject the picture 12, or more precisely, the spatial representation of the picture 12, to a spatio-spectral transform, which may comprise, for example, one or more two-dimensional arrays containing sample values ​​of one or more components of the picture 12, to obtain a transformed signal 22 of the picture 12. It should be noted that, depending on the applied spatio-spectral transform, the transformed signal may comprise spatial information about the picture in addition to spectral information. This may be particularly true in instances where a discrete wavelet transform (DWT) is used, as will be described in more detail with respect to, for example, FIGS. 3 to 5. Further examples of possible transforms are the discrete sine transform and the discrete cosine transform. The transformed signal 22 provided by the transform module 20 represents the picture 12 in terms of spectral components. That is, for example, the transformed signal 22 comprises transform coefficients resulting from the spectral components of the picture. For example, the spectral components may be associated with a frequency and, optionally, with the transform direction used when the spectral components were obtained.

[0018] The encoder 10 further comprises a residual former 30 configured to derive a residual transform signal 32 based on the transform signal 22 of the previous picture 12* and a buffered transform signal 52. The buffered transform signal 52 includes a selection from spectral components representing the previous picture 12*. For example, the encoder 10 may obtain the buffered transform signal 52 from a further transform signal representing the previous picture 12*. The encoder 10 further comprises an encoding module 70 configured to encode the residual transform signal 32 into the data stream 14. For example, the encoding module may quantize the residual transform signal 22 and encode the quantized signal using entropy coding.

[0019] The decoder 11 is for decoding or reconstructing a sequence of pictures 15, for example, in the same order 16 as they were encoded by the encoder 10. The decoder 11 comprises a decoding module 71 configured to derive or decode a residual transform signal 32' of a picture 12' from the data stream 14. As already mentioned, the residual transform signal 32' may differ from the residual transform signal 32 encoded by the encoder 10 in terms of coding loss. In other words, the residual transform signal 32' may correspond to the residual transform signal 32 despite the coding loss. Despite the coding loss, the description provided herein for the residual transform signal 32 may equally apply to the residual transform signal 32', and vice versa. In the following description, apostrophes may be omitted, i.e., for example, the reference numerals 12, 22, and 32 refer to signals on both the encoder and decoder sides.

[0020] The decoder 11 further comprises a residual synthesizer 31 configured to synthesize the residual transform signal 32′ with a buffered transform signal 52 of a previous picture 12*, e.g., decoded before the picture 12*. The buffered transform signal 52 may correspond to the buffered transform signal 52 used by the residual former 30 to derive the residual transform signal 32. For example, the decoder 11 may derive the buffered transform signal 52 from a further transform signal 22′* representing the previous picture 12*. For example, the decoder 11 may comprise a buffer module 51 configured to obtain the buffered transform signal 52 using the further transform signal 22′* representing the previous picture 12*. For example, the decoder 11 may derive the further transform signal 22′* of the previous picture as described for the transform signal 22′ of the current picture. For example, the residual synthesizer 31 may perform the inverse of the operations performed by the residual former 30. For example, residual former 30 may subtract buffered transform signal 52 from transform signal 22, and residual combiner 31 may add buffered transform signal 52 to residual transform signal 32. In other words, transform signal 22' may correspond to transform signal 22 despite any coding loss contained in residual transform signal 22' compared to residual transform signal 22. Despite the coding loss, the descriptions provided herein for transform signal 22 are equally applicable to transform signal 22', and vice versa.

[0021] The decoder 11 further comprises an inverse transform module 21 configured to subject the transformed signal to a spectral-spatial transform, e.g., the spectral-spatial transform is the inverse of the spatial-spectral transform applied to the picture 12 by the transform module 20. For example, the spectral-spatial transform results in a spatial representation of the picture 12′, e.g., as described with respect to the encoder 10.

[0022] The use of the buffered transform signal 52 to obtain the residual transform signal 32 can enhance the rate-distortion relationship as follows: The residual former 30 can use, for example, one of the spectral components contained in the buffered transform signal to determine a residual representing a deviation between the transform signal 22 and the buffered transform signal 52 in the coefficients of the spectral component. In other words, the residual former 30 can determine a residual between the transform signal 22 and the buffered transform signal 52 for a spectral component contained in the buffered transform signal. Thus, coefficients of the transform signal 22 that belong to one of the spectral components contained in the buffered transform signal 52 and that deviate little from their respective corresponding coefficients of the buffered transform signal 52 can be represented in the residual transform signal 32 by particularly small residual transform coefficients, which allows a good rate-distortion relationship in the encoding step 70.

[0023] According to one embodiment, the transformed signal 22, 22' is independent of previous pictures in the sequence of pictures. For example, the transformed signal 22 is obtained without prediction, e.g., without temporal prediction. Similarly, the spectral-spatial transform 21 may result in a spatial representation of the picture 12' that is a representation independent of previous pictures. Avoiding temporal prediction of pictures in the position domain (i.e., spatial representation) may enable small buffer sizes in the encoder 10 and the decoder 11.

[0024] For example, in existing methods of video compression, for prediction in the spatial domain, the encoder needs to include inverse quantization and inverse spatial decorrelation steps in order to model the decoder to follow a "closed-loop" design, as described, for example, with respect to FIG. 10. Compared to these methods, deriving a transform signal independently of the previous picture reduces the complexity of the encoder, since both transform and quantization do not have to be performed in both directions. For example, in the framework of JPEG XS, such simplicity can be beneficial.

[0025] For example, embodiments of the present invention may be applied within the framework of JPEG XS, a lightweight still image compression codec specifically designed for low-complexity and low-latency applications, i.e., applications where only limited processing power is available, only limited memory is available, and image data is encoded and decoded with only minimal latency. Thus, embodiments of the present invention may be implemented in accordance with JPEG XS, e.g., inverse transform module 21, residual combiner 31, and encoding module 71.

[0026] According to one embodiment, the transform signal 22 comprises a plurality of transform coefficients, each associated with a spectral component of the picture.

[0027] According to one embodiment, the transform 20 comprises a DWT and the inverse transform 21 comprises an inverse DWT. In this case, each spectral component may be represented by a number of transform coefficients, each associated with a spatial coordinate. That is, the transform signal 22 may contain both spectral and spatial information about the picture 12.

[0028] 3 illustrates an example of a spatial-spectral transform 20 according to one embodiment. According to this embodiment, the transform module 20 performs multiple or a sequence of DWTs on the picture 12 to obtain a transformed picture 24. For example, each DWT is performed using a respective wavelet for a respective transform direction (e.g., horizontal or vertical). The transformed picture 24 includes multiple transformed representations 25 of the picture 12. Each of the transformed representations 25 is obtained by applying a respective number of one or more DWTs to the picture 12 or to a previous transformed representation 25, if performed sequentially.

[0029] Each transformed representation 25 includes multiple transform coefficients. For example, each transformed representation 25 may include a 2D array of transform coefficients, where the coordinates of the transform coefficients within the transformed representation represent spatial information about the picture 12. The transformed representations 25, represented in FIG. 3 by transformed representations 251, 252, and 253, may have different sizes, i.e., different numbers of transform coefficients, depending on the number of subsequently applied DWTs and / or the wavelets and / or transform directions used. In the illustrative example of FIG. 3, transformed representation 252 may have been derived from transformed representation 251 by horizontal and vertical DWTs, and transformed representation 253 may have been derived from one of transformed representations 252 by a horizontal DWT.

[0030] Each of the transformed representations 25 may be associated with a respective spectral component, for example, depending on the wavelet used when the transformed representation was obtained. For example, each spectral component may be characterized by or associated with the frequency and / or transform direction of the respective wavelet. In other words, each of the transformed representations 25 is associated with a spectral component of the picture 12. Thus, each spectral component may include multiple transform coefficients, i.e., the transform coefficients of its associated transformed representation 25. The size of the transformed representation 25 may depend on the frequency associated with the transformed representation 25. More precisely, a transformed representation 25 representing a low-frequency spectral component may be represented by a smaller number of transform coefficients as a transformed representation 25 representing a high-frequency spectral component.

[0031] For example, transform 20 and inverse transform 21 may be implemented according to JPEG XS, which applies the discrete wavelet transform ("DWT") as a spatial decorrelation transform. This transform converts image data into multiple bands (e.g., the spectral components mentioned above), each corresponding to image structure at a particular scale, such that the largest bands contain information about the finest image structure and the smallest bands contain information about the coarsest image structure. Figure 4 illustrates such a division into wavelet bands. The shaded areas contain the three largest bands and contain the finest image structure.

[0032] In other words, FIG. 4 shows the example transformed picture 24 of FIG. 3 segmented into a transformed representation 251 associated with the highest frequency spectral component of the transformed picture 24.

[0033] 1 and 2 includes all transform coefficients of the transformed picture 24 except for the transform coefficients of the highest frequency spectral components of the transformed picture 24, i.e., the transform coefficients of the transformed representation 251. Dropping the highest frequency components may significantly reduce the required buffer size compared to other spectral coefficients, but may have a relatively small downside in the rate-distortion relationship.

[0034] Regarding the (inverse) transformations performed by transform module 20 and transform module 21, respectively, it should be noted that in alternative embodiments to those described with reference to Figures 3 and 4, an (inverse) discrete cosine transform (DCT) or an (inverse) discrete sine transform can be used. In this case, for example, the transformation by transform module 20 can result in a transformed picture including a plurality of transform coefficients, each associated with a respective frequency and transformation direction. In this case, each of the transform coefficients can represent a different combination of frequency and transformation direction, and as a result, each transform coefficient can represent an individual spectral component.

[0035] 1 and 2, the residual combiner 31 is, according to one embodiment, configured to combine the residual transform signal 32 with the buffered transform signal 52 by combining corresponding spectral components of the buffered transform signal 52 and the residual transform signal 32. For example, corresponding spectral components are those resulting from equivalent spatial-spectral transformations using the same frequency filters, associated with equivalent spatial coordinates, or associated with equivalent spectral components and equivalent parts of the picture.

[0036] According to one embodiment, the transform signal 22 includes a plurality of transform coefficients, e.g., as described with respect to Figures 3 and 4, and the residual transform signal includes, for each transform coefficient of the transform signal 22, a corresponding residual transform coefficient. According to this embodiment, the residual combiner 31 is configured to combine the mutually corresponding transform coefficients of the residual transform signal 22 and the buffered transform signal 52. The buffered transform signal may thus include a plurality of buffered transform coefficients associated with respective spectral components and, optionally, spatial coordinates. Similarly, the residual former 30 is configured, according to this embodiment, to derive the residual transform signal of the picture by predicting or forming spectral components of the transform signal using corresponding spectral components of the buffered transform signal.

[0037] According to one embodiment, the buffered transform signal 52 comprises, for each transform coefficient of the transform signal 22, a corresponding buffered transform coefficient, and the buffered transform coefficients associated with spectral components not included in the selection of spectral components may be set to a predetermined value, for example 0. In an alternative embodiment, the buffered transform signal 52 comprises only buffered transform coefficients related to spectral components included in the selection of spectral components. According to this alternative, the residual synthesizer 31 determines the transform coefficients corresponding to the buffered transform coefficients associated with spectral components not included in the selection of spectral components by taking over the values ​​of the mutually corresponding residual transform coefficients of the residual transform signal 32.

[0038] According to one embodiment, residual former 30 determines the residual transform coefficients of residual transform signal 32 by subtracting corresponding buffered transform coefficients of buffered transform signal 52 from corresponding transform coefficients of transform signal 22. According to this embodiment, residual combiner 31 determines the transform coefficients of transform signal 22 by adding corresponding buffered transform coefficients of buffered transform signal 52 to corresponding residual transform coefficients of residual transform signal 32.

[0039] 1 and 2, the encoder 10 and the decoder 11 may derive the buffered transformed signal 52 from a further transformed signal 22'* representing a previous picture 12*. The encoder 10 and the decoder 11 may, for example, derive the further transformed signal 22* in the same way that they derive the transformed signal 22 of the current picture, except that for the further transformed signal, the previous picture 12* is used as input. In other words, the decoder 11 may decode the previous picture 12* to obtain the further transformed signal 22*.

[0040] According to one embodiment, the decoder 11 decodes a previous picture by decoding, via the decoding module 71, a further residual transform signal of the previous picture from the data stream 14. The further residual transform signal is input to a residual synthesizer 31, which combines the further residual transform signal with a further buffered transform signal of an even earlier picture in the sequence to obtain a further transform signal. The further transform signal represents the previous picture in spectral content, similar to the transform signal of the current picture. The further transform signal is input to an inverse transformer 21, which subjects the further transform signal to a spectral-spatial transformation to obtain a spatial representation of the previous picture 12*.

[0041] Thus, the derivation process for the buffered transformed signal 52 can be exemplified as a loop in which a further transformed signal of a previous picture, which may be of the same type as the transformed signal of the current picture, is fed back to derive the buffered transformed signal 52 for the derivation of the current picture. In the diagram of Figure 2, reference numeral 19 indicates the position of the loop, and the use of the reference numeral switches between the derivation of the current picture and the derivation of the previous picture.

[0042] However, it should be noted that the previous picture 12* is not necessarily decoded using the further buffered transformed signal; rather, the previous picture 12* may be decoded independently of further pictures, or independently of any picture in the sequence before the previous picture, for example, for a particular picture, such as the first picture of a sequence, or a refresh picture, or an intra-coded picture. In this case, the synthesis step performed by the residual synthesizer 31 may be skipped. Similarly, for such pictures, the encoder 10 may skip the prediction step performed by the residual former 30.

[0043] According to one embodiment, the buffer module 51 derives the buffered transformed signal 52 from the further transformed signal by selecting a subset of the spectral components of the further transformed signal for the buffered transformed signal 52. In this example embodiment, the subset may be a proper subset of all spectral components of the further transformed signal, i.e., does not include all spectral components of the further transformed signal. According to this embodiment, the buffer module 51 sets the spectral components of the buffered transformed signal that are not included in the selection to a predetermined value. For example, the buffer module 51 sets all coefficients of the buffered transformed signal 52, e.g., the buffered transform coefficients mentioned above that are associated with the spectral components that are not included in the selection, to a predetermined value. For example, the predetermined value is 0, particularly when the residual synthesizer performs summation of transform coefficients and corresponding buffered transform coefficients.

[0044] For example, the buffer module 51 may buffer (e.g., store) spectral components of the further transformed signal that are included in the selection, e.g., exclusively buffer these spectral components, while discarding (or not buffering) spectral components that are not included in the selection.

[0045] According to one embodiment, the selection of spectral components of the buffered transformed signal 52 includes at least one spectral component associated with a non-zero frequency. Including non-zero frequency components makes it possible to achieve a good rate-distortion relationship, but the required buffer size may still be moderate, especially when selecting non-zero frequency components as low frequency components, as explained in relation to Figure 3. According to an alternative embodiment, the selection of spectral components of the buffered transformed signal 52 includes exclusively spectral components associated with a zero frequency or a constant level, or exclusively includes one or more spectral components associated with the lowest frequencies of the frequencies to which the spectral components of the picture are associated.

[0046] According to an embodiment, the pictures 12, 12* are coded into the data stream 14 in units of portions (e.g., called precincts), each portion being associated with a spatial region of the picture. To this end, the transform signal and / or the residual transform signal may be subdivided into portions, each of which may be associated with a spatial region within the picture.

[0047] FIG. 3 further illustrates an example of portion 26 of transformed signal 22. The portion 26 of the transformed signal includes transform coefficients within each block or region of one or more transformed representations 25 resulting from transform 20. For example, in FIG. 3, portion 26 includes transform coefficients within respective regions 261 of transformed representation 251, 262 of transformed representation 252, and 263 of transformed representation 253. That is, in FIG. 3, portion 26 is represented by transform coefficients within regions 261, 262, and 263. The transform coefficients within portion 26 relate to (e.g., represent information about) region 16 within picture 12, or more precisely, within a spatial representation (representation in the location domain) of picture 12. As illustrated in FIG. 3, portion 26 may be distributed across all transformed representations 25 within transformed picture 24, or may include sub-portions within each of the transformed representations 25.

[0048] For example, portion 26 may include all transform coefficients within each block of transformed representation 25, with each block of the transformed representation scaled in position and size according to the relative scale of the transformed representation.

[0049] For example, portion 26 may include transform coefficients associated with region 16 of picture 12. Note that the boundaries of portion 26 in the transform domain may be sharp, i.e., each transform coefficient may belong to exactly one portion of the transform signal. Therefore, the boundaries of region 16 in picture 12 may be blurred, the degree of blurring depending on the size of the wavelet template used in the DWT. Thus, one sample of picture 12 may belong to multiple regions 16, and this is especially true for samples located at the boundaries of region 16. Thus, region 16 may overlap with adjacent regions.

[0050] According to one embodiment, portion 26 and region 16 extend across the entire horizontal size of picture 12 .

[0051] According to an embodiment in which the transform 20 comprises a DWT, the encoder 10 rearranges the transform signal 22 into a rearranged representation, e.g., as described below with respect to Fig. 5, and the encoder 10 performs residual formation 30 using the rearranged representation of the transformed picture 24 signaled by the transform signal 22. The decoder 11 can thus inversely rearrange the transform signal 22' obtained by the residual synthesizer 31 and perform the inverse transform 31 on the inversely rearranged transform signal, which can correspond to the transform signal 22 at the encoder side before rearrangement, despite the coding loss.

[0052] Figure 5 shows an example of a rearranged representation 24' of a transformed picture 24 that may result from rearrangement of the transformed picture 24. The rearranged representation includes or is subdivided into a plurality of portions 26, of which reference numeral 26* is used to identify portion 26 of Figure 3. As shown in Figure 5, in rearranged representation 24', regions 261, 262, 263 that belong to different spectral components but that belong to portion 26* are rearranged or grouped together to form one block in the rearranged representation.

[0053] For example, one of the portions 26, or generally, a portion of the transformed signal carrying information about a region 16 of the picture, may be called a precinct. Note that, as shown in Figures 3 and 5, one precinct does not necessarily cover the entire width of the (transformed) picture. For example, the rearranged representation may be subdivided into slices, each of which covers the entire width of the rearranged representation 24' (and thus carries information for a slice of picture 12, which is a slice that covers the entire width of the picture). A slice may optionally be subdivided into one or more precincts. In the examples of Figures 3 and 5, each slice has one precinct, which therefore covers the entire width.

[0054] Depending on the embodiment, a slice may consist of eight or four precincts. For example, a slice may be 16 lines high and have one vertical decomposition level corresponding to eight precincts, each precinct having four precincts from two levels.

[0055] It should be noted that block-based coding is not limited to embodiments in which a DWT transform is used: for example, when using a DCT or DST, the picture 12 may be subdivided into blocks, and each block may undergo a DCT or DST transform to obtain respective transform blocks that form portions of the transformed signal, each portion being associated with a spatial region of the picture, i.e., a block into which the picture is subdivided.

[0056] Further, it should be noted that for each portion of the transform signal 22, there may be a corresponding portion of the residual transform signal 32 and a corresponding portion of the buffered transform signal 52. However, the buffered transform signal 52 may not necessarily include spectral components or buffered transform coefficients for all portions of the transform signal, as in the example, and only a subset of the portions may be buffered. For portions not included in the buffered transform signal, the transform coefficients may be set to a predetermined value, e.g., 0, or may be treated by the residual synthesizer 31 as having a predetermined value.

[0057] As mentioned above, embodiments of the present invention can be implemented in accordance with JPEG XS. JPEG XS groups transformed image data within wavelet bands, e.g., transformed signal 22, into so-called slices, each slice describing a spatial region of the original image, e.g., region 16, extending across the full image width but only across a subsection of the image's vertical extent, as shown, for example, in FIGS. 3 and 5. The example of FIG. 5 may have such an arrangement, with the dotted region in this figure consisting of one slice. Note that each band within a slice (e.g., one of the shaded regions) now describes image structure, e.g., spectral content, at a scale corresponding to the wavelet band, and each band describes structure within a portion of the image, e.g., extending through the full image width but only a subset of its vertical extent.

[0058] Continuing with the description of Figures 1 and 2, according to one embodiment, the residual transform signal 32 of Figures 1 and 2 includes, for each of the portions 26, a plurality of spectral components, e.g., as described with respect to Figure 3. According to this embodiment, the buffer module 31 derives the buffered transform signal 52 from the further transform signal by selecting and buffering a subset, e.g., an appropriate subset, of the spectral components of one or more portions 26, e.g., a slice, of a previous picture. According to this embodiment, the buffer module obtains the transform signal 22 by combining a spectral component of one of the portions 26 of the buffered transform signal 52 with a corresponding spectral component of the corresponding portion 26 of the residual transform signal.

[0059] Therefore, the buffering of the further transformed signal can be performed on a portion-by-portion basis, and consequently the selection of the spectral components to be buffered can be performed on a portion-by-portion basis, i.e., the encoder 10 can determine the selection of spectral components buffered from the previous picture separately for each portion of the transformed signal 22, or for each group of portions, e.g., for each of multiple slices of a picture.

[0060] Therefore, for example, the available buffers are not necessarily distributed evenly among the portions. For example, for a first portion, no or few spectral components can be buffered, while for a second portion, a larger number of spectral components can be buffered. For example, the selection may be performed for each portion based on the converted signal or the further converted signal, allowing for an efficient use of the available buffers, i.e., finding a good rate-distortion relationship taking into account the available buffers. The selection of portions may also be performed according to a refresh sequence, for example as described with reference to FIG. 8.

[0061] According to one embodiment, the decoder 11 derives a selection of spectral components to be buffered for a commonly buffered transform signal for all parts of a picture. For example, the same spectral components are selected for buffering for all parts 26 of a picture. According to an alternative embodiment, a picture is coded into the data stream in slices, each slice comprising one or more of the parts. For example, one slice extends across the entire horizontal size of the picture 12. According to this embodiment, the device derives a selection of spectral components to be buffered for a commonly buffered transform signal for all parts of one slice of a picture.

[0062] According to one embodiment, decoder 11 derives the selection of spectral components to be buffered for the buffered transformed signal from syntax elements signaled in the data stream, e.g., syntax element signaling control parameters 92 described with respect to Figure 8. For example, data stream 14 includes dedicated syntax elements for signaling the selection of spectral components. Encoder 10 may be configured to encode the syntax elements into data stream 14.

[0063] In an example, the selection of spectral components to be buffered may be performed portion-by-portion by determining, for each line of the transformed representation 25, e.g., for each line of a band resulting from the DWT, whether to buffer the further transformed signal for the respective line. Instead of determining line-by-line, the transformed representation may be subdivided into portions, e.g., bands, comprising one or more lines of the transformed representation (e.g., extending across the entire width of the transformed representation), and the determination may be made portion-by-portion of the transformed representation. Since each portion refers to a spatial region of the picture and a spectral component associated with the respective transformed representation, it may be determined for each spectral component which region of the picture should be buffered. It should be noted that, with regard to granularity, the height of the portions of the transformed representation may be the same or different for each of the transformed representations, e.g., bands, e.g., by several lines.

[0064] Thus, the decoder can derive the buffered transform signal 52 from the further transform signal 22'* by selecting and buffering one or more of the spectral components of the further transform signal, e.g., by determining, for each of the spectral components, whether to buffer the respective portion. For example, the portions may be selected in the manner described above for determining, for each of the spectral components of the transform representation, whether to buffer the respective portion of the further transform signal. As described above, each of the multiple portions may include one or more (e.g., entire) lines or rows of the transform representation. Thus, each portion may include or consist of coefficients derived from the spatial domain of the input image. The spatial domain buffering decision for each portion may be performed for all spectral components or for a subset of the spectral components, e.g., the subset of spectral components selected to be buffered. In this example, the device may set the portions of the buffered transform signal 52 that are not included in the selection to be buffered to a predetermined value. When obtaining the transform signal 22 for a picture, the decoder may combine the selected and buffered portions of the spectral components of the buffered transform signal 52 with the corresponding portions of the corresponding spectral components of the residual transform signal 32'.

[0065] FIG. 6 illustrates another example of an encoder 10 according to an embodiment. According to this embodiment, the encoder 10 comprises a residual synthesizer 31 configured to synthesize a further residual transform signal of a previous picture, denoted with reference sign 32′* in FIG. 6 , with a further buffered transform signal, denoted with reference sign 52*, to obtain a reconstructed transform signal 22′* of the previous picture. The reconstructed transform signal 22′* may correspond to the further transform signal of the previous picture despite the coding loss. The reconstructed transform signal 22′* may correspond to the further transform signal 22′* described above with respect to the decoder, but may differ from the further transform signal of the previous picture, as provided by the transform module 20, with respect to the coding loss. The residual synthesizer 31 of the encoder 10 may correspond to the residual synthesizer 31 of the decoder 11, i.e., it performs the same operations on its input signal as described with respect to the decoder 11. According to this embodiment, the encoder 10 further comprises a buffer module 51 which receives the reconstructed transformed signal 22'* and provides a buffered transformed signal 52, for example as described with respect to the buffer module 51 of the decoder 11.

[0066] 6, encoder 10 may encode a previous picture 12* by subjecting the further spatial representation of the previous picture 12* to a spatial-spectral transform performed by transform module 20, providing a further transform signal. Residual former 30 may derive a further residual transform signal based on the further buffered transform signal 52* and the further transform signal of an even earlier picture in the sequence, and encoder 10 may derive buffered transform signal 52 from the further residual transform signal.

[0067] According to one embodiment, the encoder 10 comprises a quantizer 60 configured to quantize the residual transform signal 32 of the picture 12, thereby obtaining a quantized signal 62. Similarly, the quantizer 60 may quantize a further residual transform signal of a previous picture 12*, thereby providing a further quantized signal 62*. The quantization step size or quantization model that the quantizer 60 uses for quantization may be described by a quantization parameter. According to this embodiment, the encoder 10 comprises an inverse quantizer 61. The inverse quantizer 61 scales or inverse quantizes the further quantized signal 62* to obtain a dequantized further residual signal 32′*, which is provided to the residual combiner 31. Because the dequantized further residual signal 32′* may correspond to the further residual transform signal of the previous picture 12*, as provided by the residual former 30, despite the coding loss introduced by quantization and inverse quantization, the dequantized further residual transform signal 32′* may also be referred to as the further residual transform signal 32′*, as was done above, for example, in the description of the residual synthesizer 31 of the encoder 10. In other words, in an embodiment including a quantizer 60 and an inverse quantizer 61, the further residual transform signal 32* provided to the residual synthesizer 31 is provided by the inverse quantizer 61 and may include quantization loss.

[0068] According to one embodiment, the encoder 10 comprises an encoding module 80 configured to encode the quantized signal 62 into the data stream 14. For example, the encoding module 80 may encode the quantized signal 62 using entropy coding, e.g., using a bounded code such as JPEG XS. Forming a residual of at least a portion of the transform coefficients by the residual former 30 and quantizing the residual transform signal 62 may result in a large number of quantization levels having a value of zero in the quantized signal 62, so that the entropy coding may result in a high compression ratio.

[0069] In other words, the wavelet transformed data, e.g., the residual transform signal obtained from the transformed signal 22, may be quantized to a limited precision to remove extraneous image content and create many zero coefficients. The quantized data may then be entropy coded. By quantizing many of the coefficients to zero, entropy coding can efficiently represent the quantized data using fewer bits than the original image, allowing for compact storage and transmission.

[0070] Figure 7 shows an example of a decoder 11 according to one embodiment that can follow the example of encoder 10 of Figure 6. In particular, decoder 11 of Figure 6 can, according to this embodiment, comprise an inverse quantizer 61 configured to inverse quantize quantized signal 62 to provide residual transform signal 32' that may deviate from residual transform signal 32 due to quantization losses.

[0071] According to one embodiment, the decoder 11 further comprises a decoding module 81 configured to derive or decode the quantized signal 62 from the data stream, for example using entropy decoding, for example using a bounded code such as JPEG XS.

[0072] For example, quantizer 60 and encoding module 80 may be part of encoding module 70 of Figure 1, and inverse quantizer 61 and decoding module 81 may be part of decoding module 71 of Figure 2. It should be noted that decoding module 71 may perform reverse operations of the operations of decoding module 71, which may be particularly applicable to the quantization and entropy coding steps.

[0073] FIG. 8 shows another embodiment of the encoder 10, which may optionally correspond to the embodiments of FIGS.

[0074] As mentioned above, embodiments of the present invention perform temporal prediction after spatial decorrelation but before quantization. In particular, a discrete wavelet transform ("DWT") can be applied as a first step to create a cascade of wavelet bands as seen in Figure 5, with temporal prediction applied within these bands, i.e., in the wavelet domain (or transform domain), and not in the spatial domain. As shown in Figure 8, decorrelation 20 is performed before temporal prediction 30; according to this embodiment, neither motion estimation nor motion prediction is performed, avoiding the complexity of such operations.

[0075] The inventive concept has the advantage that temporal prediction can be applied to only a portion of the bands, limiting the amount of frame buffer storage on the encoding and decoding sides. For example, the bands for which temporal prediction is used may be selected, controlled, or determined by a control parameter 92. FIG. 8 shows an exemplary implementation of the buffer module 51, according to which data in the frame buffer 56 can be replaced with a predetermined value 93, e.g., 0, depending on a control parameter 92 controlled, e.g., slice-by-slice, so that no prediction is performed for a subset of the bands in a given slice, e.g., the bands set to the predetermined value. For example, in this way, the buffered or stored data, i.e., the buffered transformed signal 52, requires less data storage. For example, the control parameter 92 may be determined by or in response to an optional quality control module 96, e.g., taking into account a desired rate-distortion relationship or available resources, as described above. Additionally or alternatively, the control parameter 92 may be determined by a gradual refresh mechanism, as described with reference to FIGS. 8 and 9.

[0076] Control parameters 92 may be included in the entropy-coded data, allowing the decoder to adjust its prediction mechanism to the same settings as the encoder. For example, if the three diagonal bands in Figure 4 were not stored in the frame buffer, the size of the frame buffer would be reduced to one-quarter of its original size, and the bandwidth between the frame buffer and the main processor would be reduced by a factor of four. Because the diagonal bands contain only the finest-scale high-frequency content, the impact on image quality remains minimal, and good compression is still obtained.

[0077] An additional advantage is that the encoder does not need to include an additional inverse transform to model the state of the decoder-side frame buffer: instead, it only needs to perform a low-complexity inverse quantization according to a "closed-loop" design.

[0078] In the coding schemes described with reference to FIGS. 1 to 8, the encoder 10 may transmit only the output of the temporal transform. Therefore, in some cases, the decoder 11 does not have access to the fully encoded sequence, for example, when switching to an already running connection. In such cases, the decoder is unable to reconstruct the image due to lack of information from past frames. However, typical applications require faithful reconstruction of the original data after a minimal delay after switching to the running stream (e.g., up to approximately 200 milliseconds). At the same time, it is also desirable to ensure minimal latency, i.e., the delay between the decoder receiving the image data and the decoder being able to reconstruct the same image data for the same timestamp and the same image region after switching to the running connection. This delay typically needs to be much lower than the start-up delay and may even be constrained, for example, to a fraction of a frame.

[0079] As a solution, an optional part of the present invention is the combination of the partial frame buffer (selective buffer of the previously transformed signal as described in connection with FIGS. 1 to 8) as described above with a mechanism, e.g., a so-called gradual refresh mechanism, that allows a decoder that switches to a connection without accessing the initial data to restore the complete image after some temporarily predicted frames. In particular, in combination with the portion-by-portion selection of spectral components, particularly the control parameters 92 described in FIG. 8, the selection by the control parameters can be adjusted slice by slice, so that prediction can be completely disabled for certain portions of the picture, as required, e.g., by the gradual refresh control. In such a partial refresh, none of the bands (e.g., bands of the slice to which the partial refresh refers), or even a subset thereof, receives a prediction from the frame buffer.

[0080] FIG. 9 illustrates an example of a subsequence 17 of a sequence of pictures 15 according to one embodiment. The subsequence 17 may also be referred to as a refresh sequence. The subsequence 17 includes multiple pictures of the sequence 15, an exemplary number of three pictures 121, 122, and 123 in FIG. 6. According to one embodiment, each picture of the subsequence 17 is encoded into the data stream 14 by the encoder 10 by, for each of the pictures 121, 122, and 123 of the subsequence 17, suppressing prediction of one or more spectral components of each of one or more portions of the picture using corresponding spectral components of the buffered transformed signal, and predicting one or more spectral components of each of one or more further portions of the picture using corresponding spectral components of the buffered transformed signal. The pictures are encoded such that the unbuffered portions of the entire pictures of the subsequence are combined to cover the entire picture area of ​​the picture. For example, as shown in FIG. 6, the portion 26 of the picture 121 is suppressed from predicting one or more spectral components of each of one or more portions of the picture using corresponding spectral components of the buffered transformed signal. 1 is coded without predicting the transformed signal 22 using the previous picture, and part 26 of the picture 122 2is coded without predicting the transformed signal 22 using the previous picture, and part 26 of picture 123 3 is coded without predicting the transformed signal 22 using the previous picture. 1 , 26 2 , 26 3 In other words, a sequence of pictures may be equally subdivided into slices, each of which may be coded without prediction from a previous picture, at least in the coding of one of the sequence of pictures.

[0081] Thus, for example, the last picture of a subsequence is coded independently of any pictures in sequence 15 that precede the first picture of the subsequence. Thus, after decoding an entire subsequence, decoder 11 can fully decode further pictures even if the decoder has no information about previous pictures in the subsequence.

[0082] According to the embodiment described in relation to Figure 6, the decoder 11 decodes the pictures of the subsequence 17 by, for each picture of the subsequence, suppressing the combining of one or more spectral components of each of one or more parts of the picture with corresponding spectral components of the buffered transformed signal, and combining one or more spectral components of each of one or more further parts of the picture with corresponding spectral components of the buffered transformed signal.

[0083] It should be noted that the pictures of the subsequences do not necessarily follow each other directly, but further pictures of sequence 15 may be alternated with pictures of subsequence 17 .

[0084] In other words, according to one embodiment, the encoder 10 includes a gradual refresh control module 90 as an optional feature, as shown in FIG. 8 . The gradual refresh control module 90 can signal control parameters 92 that determine, for a slice or for each slice, which bands are involved in prediction. For regions not involved in prediction, e.g., slices, the contents of the frame buffer 56 are replaced by a predetermined value, e.g., 0, so that the decoder can reconstruct all or part of the image data without relying on the contents of the frame buffer 56, e.g., without relying on previous pictures. A slice that does not include bands in the frame buffer and therefore does not undergo temporal prediction may be referred to below as an “I-slice,” and a slice that includes several temporally predicted bands is referred to as a “P-slice.” According to one embodiment, a portion of all slices are transmitted as I-slices, thereby allowing the decoder to refresh its frame buffer after a refresh interval of several frames, e.g., the subsequence 17 described in FIG. 8 , all slices are transmitted without temporal prediction, e.g., within at least one picture of the sequence.

[0085] As an example of such a scheme, consider an image consisting of N slices. In FIG. 8, N is illustratively 3. For frame number k, slice k mod N can be transmitted as an I-slice, where "mod" is the modulus operator. In this exemplary scheme, a full-frame refresh is reached after N subsequent frames. Other schemes for transmitting slice kp mod N as an I-slice can also be envisioned, for example, where p and N are relatively early. As a third example, the encoder can also select I-slices that benefit least from temporal prediction, i.e., I-slices that have changed too much from the previous frame to make temporal prediction worthwhile. Such a scheme can be combined with static I-slice allocation as given in the first or second examples.

[0086] Compared to conventional mechanisms that occasionally transmit I-frames (non-predicted frames) for decoder refresh, i.e., encoding entire frames without temporal prediction, the gradual refresh mechanism may cause little or no bitrate fluctuation. More precisely, during gradual refresh, the rates of subsequent frames in the coded refresh sequence may not deviate much from each other compared to encoding full-refresh pictures occasionally (e.g., every 10 P-frames). The rate of these I-frames is large, requiring a larger rate than P-frames and therefore a larger input buffer at the decoder. However, a larger input buffer at the decoder also increases latency to ensure a sufficient amount of data is buffered. Thus, gradual refresh may have the advantage of either a smaller buffer and therefore reduced delay, or increased quality of non-predicted frames.

[0087] In other words, one embodiment of the present invention described with reference to FIGS. 1-9 provides a coding method for a temporally related image sequence described above that exploits temporal relationships within the image sequence for further bandwidth reduction. The method includes a spatial decorrelation transform based on a wavelet filter, followed by a temporal decorrelation transform in the transform domain of the spatial decorrelation transform. That is, unlike common schemes, temporal prediction is performed in the filtered domain. To reduce processing complexity, temporal decorrelation may be applied to only a subset of the output of spatial decorrelation. To ensure that a decoder can reconstruct a complete image even when it has not received the first frame of a transmission, a gradual update procedure is described that transmits a window of the original image without temporal prediction. This gradual update window is moved over the frames to ensure that the full frame can be reconstructed after some time.

[0088] 10 illustrates an example of an alternative coding scheme, with reference to which the advantages of the present invention over some existing video encoding schemes are discussed below. Examples of typically used video codecs are H.261, H.264 (AVC), or H.265 (HEVC). In such schemes, data first undergoes temporal decorrelation, for example by creating difference data between a predicted frame and the original frame. The image is then divided into blocks, and a spatial decorrelation transform, such as a discrete cosine transform ("DCT") or a discrete sine transform ("DST"), is performed on these blocks. This prediction typically also includes estimating motion within the original data, and then predicting from the motion-compensated data alone. It should be noted that if this step is too complex, schemes such as H.261 can also function without motion estimation.

[0089] To allow the decoder to reconstruct the full frame after some time, even when switching to a live connection, such a design sometimes transmits a full frame without motion information. Hereinafter, such frames are denoted as "I" frames. Frames that include a prediction step from motion are called "P" frames. Additional prediction mechanisms may exist, but are not relevant to the invention described herein.

[0090] However, it is important to note that all of the above schemes perform temporal prediction before spatial decorrelation (i.e., a spatial decorrelation transform is performed over the temporally predicted data), the frame buffer is the same size as that of the original image, and I-frames contain data for a full refresh. However, some existing designs, such as video coding standards H.263+ and ISO / IEC 14496-2, allow additional spatial prediction of transform coefficients, i.e., provide additional optional prediction in the transform domain, but this relies on spatial correlation, i.e., different spatial regions of the same picture are used to predict the transform coefficients.

[0091] Temporal prediction typically follows a "closed-loop" design: the encoder contains a model of the decoder, allowing it to accurately predict what output the decoder will produce, and then performs its temporal prediction on this predicted output. Without such closed-loop temporal prediction, the output of the actual encoder and encoder-side frame buffer would slowly drift, which could cause visible image distortion after a period of time. Figure 10 provides an overview of the steps in a conventional video coding scheme.

[0092] In contrast, embodiments of the present invention may use temporal prediction in the transform domain rather than spatial prediction in the transform domain.

[0093] In conventional video codecs such as those described above, unpredicted frames ("I" frames) require a larger amount of bits even after conversion, or require higher quantization if only a maximum upper-bound bandwidth is available for transmitting the data, reducing their quality. This results either in transmission rate peaks requiring larger buffers at the encoder and decoder side to smooth such rate derivation, or in significant quality degradation for such unpredicted "I" frames.

[0094] According to embodiments of the present invention, gradual refresh can be easily implemented by modulating the control parameter defining the number of included bands to 0. In contrast to video coding standards such as H.264 and subsequent standards that already allow gradual refresh applied to all transformed coefficients as an entity, embodiments of the present invention allow modulation of the amount of temporal prediction, where no prediction as required for gradual refresh is only a special case. Thus, precise control of buffers and data rates can be achieved.

[0095] Other codecs, such as JPEG XS, currently do not include a mechanism for temporal prediction and therefore require too much bandwidth for some applications. Compared to the methods for temporal prediction in video coding standards H.261 and later, which require the encoder and decoder to include memory to represent a full frame and provide bandwidth between the actual processing units on the encoder / decoder and local storage ("RAM") to retrieve data from past frames to enable temporal prediction, the proposed scheme offers the advantage that the size of the local storage can be dynamically scaled by an external control parameter, allowing the system to adjust to the image complexity and system bandwidth. Note that in one extreme setting of this parameter, no data may be included at all, such that no temporal prediction is performed.

[0096] Although some aspects are described as features in the context of an apparatus, it is clear that such description may also be considered as a description of corresponding features of a method. Although some aspects are described as features in the context of a method, it is clear that such description may also be considered as a description of corresponding features with respect to the functionality of an apparatus. In particular, the block diagrams of Figures 1, 2, 6, 7, 8, and 9 used in describing the encoder and decoder may also be understood as block diagrams describing corresponding methods for encoding and decoding.

[0097] Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit, and in some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0098] The coded image signal of the present invention can be stored on a digital storage medium and can be transmitted over a transmission medium, such as a wireless or wired transmission medium, such as the Internet. In other words, further embodiments provide a video bitstream product, e.g., a digital storage medium having a video bitstream stored thereon, comprising a video bitstream according to any of the embodiments described herein.

[0099] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software, or at least partly in hardware, or at least partly in software. Implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to perform the respective methods. Thus, the digital storage medium may be computer-readable.

[0100] Some embodiments according to the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0101] Generally, embodiments of the present invention can be implemented as a computer program product having program code that operates to perform one of the methods when the computer program product is run on a computer, and the program code can be stored on, for example, a machine-readable carrier.

[0102] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0103] In other words, therefore, an embodiment of the inventive methods is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0104] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0105] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals can for example be arranged to be transmitted via a data communication connection, for example via the Internet.

[0106] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0107] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0108] Further embodiments according to the invention comprise an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0109] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods may be performed by any hardware apparatus.

[0110] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0111] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0112] While the present invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents that fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted to include all such alterations, permutations, and equivalents that fall within the true spirit and scope of the present invention.

[0113] In the foregoing Detailed Description, it can be seen that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the appended claims reflect, subject matter may lie in fewer than all features of a single disclosed example. Accordingly, the appended claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate example. While each claim may stand on its own as a separate example, it should be noted that while a dependent claim may refer to a specific combination with one or more other claims in the claim, other examples may also include a combination of a dependent claim with the subject matter of each of the other dependent claims, or a combination of each feature with other dependent or independent claims. Such combinations are suggested herein unless it is stated that a specific combination is not intended. Furthermore, including features of a claim in any other independent claim is also contemplated, even if that claim is not directly dependent on that independent claim.

[0114] The above-described embodiments are merely illustrative of the principles of the present disclosure. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented by way of description and explanation of the embodiments herein.

Claims

1. An apparatus (11) for decoding a sequence of pictures (15) from a data stream (14), said apparatus comprising: deriving (71) a residual transform signal (32') of said picture (12') from said data stream (14); combining (31) said residual transform signal (32') with a buffered transform signal (52) of a previous picture (12*) of said sequence to obtain a transform signal (22') of said picture, said transform signal (22') representing said picture in terms of spectral components; subjecting said transformed signal (22') to a spectral-spatial transformation (21); and adapted to decode a picture (12') of said sequence by Apparatus (11), wherein the buffered transformed signal (52) comprises a selection from spectral components of a further transformed signal representing the previous picture.

2. The apparatus of claim 1 , wherein the selection from spectral components is a subset of the spectral components representing the previous picture.

3. 2. The apparatus of claim 1, wherein the apparatus is configured to derive the buffered transformed signal from the previous picture by exclusively buffering spectral components included in the selection of spectral components representing the previous picture.

4. 2. The apparatus of claim 1, wherein the apparatus is configured to combine the residual transform signal (32′) with the buffered transform signal (52) by combining corresponding spectral components of the buffered transform signal (52) and the residual transform signal (32′).

5. 2. The apparatus of claim 1, wherein the selection of spectral components signaled in the buffered transformed signal includes at least one spectral component associated with a non-zero frequency.

6. The apparatus of claim 1 , wherein the spectral-spatial transform is an inverse discrete wavelet transform.

7. 2. The apparatus of claim 1, wherein the apparatus is for deriving the selection of spectral components to be buffered for the buffered transformed signal (52) from syntax elements signaled in the data stream.

8. The device, decoding the previous picture, decoding a further residual transform signal (32') of the previous picture from the data stream; - combining said further residual transform signal (32') with a further buffered transform signal (52*) of an even earlier picture of said sequence to obtain a further transform signal (22'), said further transform signal (22'*) representing said earlier picture in terms of spectral content; - subjecting said further transformed signal (22'*) to a spectral-spatial transformation in order to obtain a spatial representation of said previous picture; decoding the previous picture by deriving said buffered transformed signal (52) from said further transformed signal (22'*); The apparatus of claim 1 configured to:

9. The device, selecting, for said buffered transformed signal (52), a subset of the spectral components of said further transformed signal (22'*); setting spectral components of the buffered transformed signal (52) to predetermined values, the spectral components not being included in the selection; 9. The apparatus of claim 8, configured to derive the buffered transformed signal (52) from the further transformed signal (22'*) by:

10. The picture is coded into the data stream in portions, each portion being associated with a spatial domain of the picture, and the residual transform signal (32') comprises, for each of the portions, a plurality of spectral components, and the apparatus deriving the buffered transformed signal (52) from the further transformed signal (22'*) by selecting and buffering a subset of spectral components of one or more portions of the previous picture; obtaining the transform signal (22') of the picture by combining spectral components of one of the portions of the buffered transform signal (52) with corresponding spectral components of a corresponding portion of the residual transform signal (32'); The apparatus of claim 8 configured to:

11. the device is for deriving the selection of spectral components to be buffered for the buffered transformed signal (52) in common for all parts of the picture, or 11. The apparatus of claim 10, wherein the picture is coded into the data stream in slices, each of the slices including one or more of the portions, and the apparatus is for deriving the selection of spectral components to be buffered for the buffered transformed signal (52) in common for all portions of a slice of the picture.

12. The device, for each picture of a sub-sequence of said sequence, inhibiting combining one or more spectral components of each of one or more portions of said picture with corresponding spectral components of said buffered transformed signal (52); and combining one or more spectral components of each of one or more further portions of said picture with corresponding spectral components of said buffered transformed signal (52); and configured to decode the pictures of the sub-sequence of the sequence of pictures by The apparatus of claim 10 , wherein the unbuffered portions of all of the pictures of the subsequences combine to cover an entire picture area of ​​the picture.

13. The device, deriving the buffered transformed signal (52) from the further transformed signal (22'*) by selecting and buffering one or more of a plurality of portions of the spectral components of the further transformed signal; obtaining the transform signal (22') of the picture by combining the selected and buffered portions of the spectral components of the buffered transform signal (52) with corresponding portions of corresponding spectral components of the residual transform signal (32'); The apparatus of claim 8 configured to:

14. the transformed signal and the further transformed signal comprise, for each spectral component, a transformed representation of the picture, the transformed representation comprising a plurality of coefficients of a sample array; The apparatus of claim 13 , wherein each of the plurality of portions of the spectral component comprises one or more rows of the sample array of the transformed representation of the spectral component.

15. An apparatus (10) for encoding a sequence of pictures into a data stream, said apparatus (10) comprising: - subjecting the spatial representation of the picture to a spatial-spectral transform (20) to obtain a transformed signal (22), the transformed signal (22) representing the picture in terms of spectral components; deriving (30) a residual transform signal (32) based on the transform signal (22) and a buffered transform signal (52) of a previous picture of the sequence; encoding (70) the residual transform signal (32) of the picture into the data stream; and configured to encode the pictures of the sequence by An apparatus (10) in which the buffered transformed signal (52) includes a selection from spectral components of a further transformed signal representing the previous picture.

16. The apparatus of claim 15 , wherein the selection from spectral components is a subset of the spectral components representing the previous picture.

17. 16. The apparatus of claim 15, wherein the apparatus is configured to derive the buffered transformed signal from the previous picture by exclusively buffering spectral components included in the selection of spectral components representing the previous picture.

18. 16. The apparatus of claim 15, wherein the apparatus is configured to derive the residual transform signal (32) of the picture by predicting spectral components of the transform signal (22) using corresponding spectral components of the buffered transform signal (52).

19. 16. The apparatus of claim 15, wherein the selection of spectral components signaled in the buffered transformed signal (52) includes at least one spectral component associated with a non-zero frequency.

20. The apparatus of claim 15 , wherein the spatial-spectral transform is a discrete wavelet transform.

21. 16. The apparatus of claim 15, wherein the apparatus is for encoding syntax elements into the data stream that indicate the selection of buffered spectral components for the buffered transformed signal (52).

22. The device, encoding the previous picture, - subjecting a further spatial representation of said previous picture to said spatial-spectral transformation to obtain a further transformed signal, said further transformed signal representing said previous picture in spectral components; deriving a further residual transform signal (32*) based on a further buffered transform signal (52*) of an even earlier picture of the sequence and the further transform signal; encoding said further residual transformed signal (32*) into said data stream; encoding the previous picture by deriving the buffered transform signal (52) from the further residual transform signal (32*); 16. The apparatus of claim 15, configured to:

23. The device, combining said further residual transform signal (32*) with said further buffered transform signal (52) to obtain a reconstructed transform signal (22'*) of said previous picture; selecting, for the buffered transformed signal (52), a subset of the spectral components of the reconstructed transformed signal (22'*); setting spectral components of the buffered transformed signal (52) to predetermined values, the spectral components not being included in the selection; 23. The apparatus of claim 22, configured to derive the buffered transform signal (52) from the further residual transform signal (32*) by:

24. The device, encoding the residual transform signal (32) by quantizing the residual transform signal (32) and encoding the quantized residual transform signal (62) into the data stream; encoding the further residual transform signal (32*) by quantizing the further residual transform signal and encoding the quantized further residual transform signal into the data stream; deriving the buffered transform signal (52) by scaling the quantized further residual transform signal and combining the scaled further residual transform signal (32'*) with the further buffered transform signal (52*); 23. The apparatus of claim 22 configured to:

25. The apparatus is configured to code the picture into the data stream in portions, each of the portions being associated with a spatial domain of the picture, the residual transform signal (32) comprising a plurality of spectral components for each of the portions of the picture, and the apparatus: deriving the buffered transform signal (52) from the further residual transform signal (32'*) by selecting for the buffered transform signal (52) a subset of spectral components of one or more portions of the previous picture; deriving the residual transform signal (32) for the picture by predicting spectral components of one of the portions of the transform signal (22) using corresponding spectral components of a corresponding portion of the buffered transform signal (52); 23. The apparatus of claim 22 configured to:

26. the apparatus is for encoding into the data stream a syntax element indicating the selection of buffered spectral components for the buffered transformed signal (52) in common for all parts of the picture; 26. The apparatus of claim 25, wherein the apparatus is for encoding the picture into the data stream in slices, each of the slices including one or more of the portions, and the apparatus is for encoding into the data stream a syntax element indicating the selection of buffered spectral components for the buffered transformed signal (52) in common for all portions of one slice of the picture.

27. The apparatus is configured to encode a sub-sequence of pictures, the sub-sequence comprising: for each of the pictures of the subsequence, suppressing prediction of one or more spectral components of each of one or more portions of the picture using corresponding spectral components of the buffered transformed signal (52); and predicting one or more spectral components of each of one or more further portions of the picture using corresponding spectral components of the buffered transformed signal (52); a sub-sequence of said sequence of pictures according to 26. The apparatus of claim 25, wherein the unbuffered portions of all of the pictures of the subsequences combine to cover an entire picture area of ​​the picture.

28. The device, deriving the buffered transform signal (52) from the further residual transform signal (32'*) by selecting and buffering one or more of a plurality of portions of the spectral components of the further residual transform signal (32'*); deriving the residual transform signal (32) of the picture by using the selected and buffered portions of the spectral components of the further residual transform signal to predict corresponding portions of corresponding spectral components of the transform signal (22); 23. The apparatus of claim 22 configured to:

29. the residual transform signal and the further residual transform signal comprise, for each spectral component, a transform representation of the picture, the transform representation comprising a plurality of coefficients of a sample array; each of the plurality of portions of the spectral component comprises one or more rows of the sample array of the transformed representation of the spectral component; 29. The apparatus of claim 28.

30. 1. A method for decoding a sequence of pictures from a data stream, said method comprising: deriving (71) a residual transform signal (32') of said picture from said data stream; combining (31) said residual transform signal (32') with a buffered transform signal (52) of a previous picture of said sequence to obtain a transform signal (22') of said picture, said transform signal (22') representing said picture in terms of spectral components; subjecting (21) said transformed signal (22') to a spectral-spatial transformation; decoding the pictures of the sequence by A method wherein the buffered transformed signal (52) comprises a selection from spectral components of a further transformed signal representing the previous picture.

31. 1. A method for encoding a sequence of pictures into a data stream, said method comprising: - subjecting (20) the spatial representation of the picture to a spatial-spectral transformation to obtain a transformed signal (22), the transformed signal (22) representing the picture in terms of spectral components; deriving (30) a residual transform signal (32) based on the transform signal (22) and a buffered transform signal (52) of a previous picture of the sequence; encoding (70) the residual transform signal (32) of the picture into the data stream; encoding the pictures of the sequence by A method wherein the buffered transformed signal (52) comprises a selection from spectral components of a further transformed signal representing the previous picture.

32. 1. A non-transitory digital storage medium having stored thereon a computer program, which, when executed by a computer, provides a method for decoding a sequence of pictures from a data stream, the method comprising: deriving a residual transform signal (32') of said picture from said data stream; combining said residual transform signal (32') with a buffered transform signal (52) of a previous picture of said sequence to obtain a transform signal (22') of said picture, said transform signal (22') representing said picture in terms of spectral components; subjecting said transformed signal (22') to a spectral-spatial transformation; decoding the pictures of the sequence by A non-transitory digital storage medium carrying out a method, wherein the buffered transformed signal (52) comprises a selection from spectral components of a further transformed signal representing the previous picture.

33. 1. A non-transitory digital storage medium having stored thereon a computer program, which, when executed by a computer, provides a method for encoding a sequence of pictures into a data stream, the method comprising: - subjecting the spatial representation of the picture to a spatial-spectral transformation to obtain a transformed signal (22), the transformed signal (22) representing the picture in terms of spectral components; deriving a residual transform signal (32) based on the buffered transform signal (52) of a previous picture of the sequence and the transform signal (22); encoding the residual transform signal (32) of the picture into the data stream; encoding the pictures of the sequence by A non-transitory digital storage medium carrying out a method, wherein the buffered transformed signal (52) comprises a selection from spectral components of a further transformed signal representing the previous picture.

Images