Encoding and Decoding a Sequence of Pictures

Abstract

An apparatus for decoding a sequence of pictures from a data stream is configured to decode a picture of the sequence by deriving a residual transform signal of the picture from the data stream, combining the residual transform signal with a buffered transform signal of a previous picture of the sequence to obtain a transform signal of the picture, the transform signal representing the picture in terms of spectral components, and subjecting the transform signal to a spectral-spatial transformation, the buffered transform signal including a selection from the spectral components representing the previous picture.

Description

[Technical field]

[0001] Embodiments of the invention relate to an apparatus for decoding a sequence of pictures and an apparatus for encoding a sequence of pictures. Further embodiments relate to a method for decoding a sequence of pictures and a method for encoding a sequence of pictures. Further embodiments relate to a computer program for performing the same. Further embodiments relate to a data stream obtained by the method for encoding a sequence of pictures. Some embodiments relate to partial-temporal prediction for image sequence coding. [Background technology]

[0002] When transmitting a sequence of images, such as a sequence of images captured by a camera or a sequence of screen captures of a computer screen desktop, only limited bandwidth is available to transfer this information. To address this issue, image data is lossily compressed, for example, by first including a transform that exploits spatial and / or temporal correlations in the source data, and then quantizing and encoding the resulting coefficients of the decorrelated transform. To enable temporal prediction, both the encoder and the decoder need to hold past frames of data in units denoted as "frame buffers". In an FPGA implementation, this creates the additional burden of having to transfer large amounts of data from both the data source, e.g., a camera or computer desktop, and the frame buffer at the same time. Since the bandwidth between the source, frame buffer, and processing device is also a limiting factor, holding a full-sized frame buffer may be limiting or impossible for an application. Summary of the Invention [Problem to be solved by the invention]

[0003] It would therefore be desirable to have a video coding concept that provides an improved trade-off between a good rate-distortion relationship for coded video data (the ratio between the rate, e.g., the size of the data at which the sequence is coded, and the distortion, e.g., the loss of information or quality) and low buffer requirements (e.g., a low size buffer required to encode or decode a sequence of pictures). [Means for solving the problem]

[0004] According to one embodiment, an apparatus for decoding a sequence of pictures from a data stream is configured to decode a picture of the sequence (e.g. a currently decoded picture) by deriving a residual transform signal of the picture from the data stream, combining the residual transform signal with a buffered transform signal of a previous picture of the sequence (e.g. a picture (directly) previous to the currently decoded picture in the sequence) to obtain a transform signal of the picture, the transform signal representing the picture in terms of spectral components (e.g. the transform signal is independent of the previous picture), and subjecting the transform signal to a spectral-spatial transform (e.g. an inverse frequency transform). According to this embodiment, the buffered transform signal includes a selection from the spectral components representing the previous picture.

[0005] According to one embodiment, an apparatus for encoding a sequence of pictures into a data stream is configured to encode a picture of a sequence (e.g. the currently coded picture) by subjecting a spatial representation of the picture to a spatio-spectral transformation (e.g. a wavelet transform or a frequency transform) to obtain a transformed signal, the transformed signal representing the picture in terms of spectral components (e.g. the transformed signal is independent of the previous picture), deriving a residual transform signal based on the buffered transform signal and the transform signal of a previous picture of the sequence (e.g. the (direct) previous picture of the currently decoded picture in the sequence), and encoding the residual transform signal of the picture into the data stream. According to this embodiment, the buffered transform signal includes a selection from the spectral components representing the previous picture.

[0006] The embodiment of the present invention relies on the idea of ​​enhancing the rate-distortion relationship for coding a transform signal representing a picture to be coded with spectral components by using a buffered transform signal obtained from a previous picture to obtain a residual transform signal, which is then coded into a data stream. Correlation between subsequent pictures of a sequence of pictures may thus be considered as a temporal correlation in the example and may be exploited so that the residual transform signal may have a lower data rate than the transform signal. Buffering a selection from the spectral components of the previous picture makes it possible to control the buffer size required to buffer the buffered control signal within the provision of the target application. For example, the selection does not necessarily have to include all spectral components, but may include a subset of the spectral components of the picture, reducing the data rate of the buffered transform signal. In contrast to temporal prediction performed on the untransformed picture, i.e. in the spatial domain, before the spatial-spectral transformation, examples of the concepts disclosed herein may provide a computationally efficient method of temporal prediction without buffering the entire picture. In particular, buffering a selection of spectral components of the previous picture may make it possible to achieve a relatively high reduction in the data rate of the residual transform signal (i.e., a relatively high enhancement of the rate-distortion relationship of the coded picture) with a relatively moderate buffer size. These advantages apply equally to the encoder and decoder sides, such that at the decoder side, the spectral components of the previous picture that were used to derive the residual transform signal of the current picture at the encoder side are buffered to determine the transform signal of the current picture based on the residual transform signal.

[0007] According to one embodiment, a method for decoding a sequence of pictures from a data stream comprises the steps of: deriving a residual transform signal of a picture from the data stream, combining the residual transform signal with a buffered transform signal of a previous picture of the sequence to obtain a transform signal of the picture, the transform signal representing the picture in terms of spectral components, and decoding the picture of the sequence by subjecting the transform signal to a spectral-spatial transformation. The buffered transform signal includes a selection from the spectral components representing the previous picture.

[0008] According to one embodiment, a method for encoding a sequence of pictures into a data stream comprises the steps of encoding a picture of a sequence by subjecting a spatial representation of the picture to a spatio-spectral transformation to obtain a transformed signal, the transformed signal representing the picture in terms of spectral components, deriving a residual transform signal based on the buffered transform signal and the transform signal of a previous picture of the sequence, and encoding the residual transform signal of the picture into a data stream, the buffered transform signal including a selection from the spectral components representing the previous picture.

[0009] According to one embodiment, a non-transitory digital storage medium has a computer program stored thereon which, when executed by a computer, performs a method for encoding a sequence of pictures into a data stream, the method comprising the steps of encoding a picture of a sequence by: subjecting a spatial representation of the picture to a spatio-spectral transformation to obtain a transformed signal, the transformed signal representing the picture in terms of spectral components; deriving a residual transform signal based on the buffered transform signal and the transform signal of a previous picture of the sequence; and encoding the residual transform signal of the picture into the data stream, the buffered transform signal including a selection from the spectral components representing the previous picture.

[0010] According to one embodiment, a non-transitory digital storage medium has a computer program stored thereon which, when executed by a computer, performs a method for decoding a sequence of pictures from a data stream, the method comprising the steps of: deriving a residual transform signal of a picture from the data stream; combining the residual transform signal with a buffered transform signal of a previous picture of the sequence to obtain a transform signal of the picture, the transform signal representing the picture in terms of spectral components; and subjecting the transform signal to a spectral-spatial transformation, the buffered transform signal including a selection from the spectral components representing the previous picture.

[0011] According to one embodiment, a data stream has a sequence of pictures encoded therein, the sequence of pictures being encoded into the data stream by a method of encoding the sequence of pictures into a data stream, the method comprising the steps of encoding a picture of the sequence by subjecting a spatial representation of the picture to a spatial-spectral transformation to obtain a transformed signal, the transformed signal representing the picture in terms of spectral components, deriving a residual transform signal based on the buffered transform signal and the transform signal of a previous picture of the sequence, and encoding the residual transform signal of the picture into the data stream, the buffered transform signal including a selection from the spectral components representing the previous picture.

[0012] Embodiments of the present invention are described in detail below with reference to the accompanying drawings. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 illustrates an encoder according to one embodiment. [Diagram 2] FIG. 2 illustrates a decoder according to one embodiment. [Diagram 3] FIG. 1 illustrates an example of a spatial-spectral transformation. [Figure 4] FIG. 2 illustrates an example of segmenting a picture into spectral components; [Diagram 5] FIG. 13 illustrates an example of a rearrangement representation of a transformed picture. [Figure 6] FIG. 4 illustrates an encoder according to another embodiment. [Figure 7] FIG. 4 shows a decoder according to another embodiment. [Figure 8] FIG. 4 illustrates an encoder according to another embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a refresh sequence according to an embodiment. [Figure 10] FIG. 13 illustrates another example of an encoding scheme. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In the following, the embodiments are described in detail, but it should be understood that the embodiments provide many applicable concepts that can be implemented in a wide variety of coding concepts. The specific embodiments described are merely illustrative of specific ways to implement and use the concepts, and do not limit the scope of the embodiments. In the following description, a number of details are described to provide a more complete description of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that other embodiments can be implemented without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the examples described herein. Furthermore, features of different embodiments described herein can be combined with each other unless specifically stated otherwise.

[0015] In the following description of the embodiments, the same or similar elements or elements having the same functions are given the same reference numbers or are identified by the same names, and the repeated description of the elements given the same reference numbers or are identified by the same names is usually omitted. Therefore, the descriptions provided for the elements having the same or similar reference numbers or are identified by the same names can be mutually exchanged or applied to each other in different embodiments.

[0016] Fig. 1 shows an apparatus 10 for coding or encoding a sequence of pictures 15 into a data stream 14 according to an embodiment. The apparatus 10 may also be referred to as an encoder 10. The pictures of the sequence 15 may have an order 16, for example a temporal order or a picture order. The sequence 15 includes a picture 12, for example a picture currently being coded, and a previous picture 12* preceding the picture 12 in the order 16 of the sequence 15. For example, the previous picture 12* is a picture previously coded with respect to the current picture 12, for example immediately before the current picture 12. Fig. 2 shows a corresponding decoder 11, i.e. an apparatus 11 for decoding a sequence of pictures from a data stream 14, according to an embodiment. In other words, the decoder 11 is for reconstructing a picture 12' from the data stream, the apostrophe being used to indicate that the picture 12' reconstructed by the decoder 11 may deviate from the picture 12 originally coded by the apparatus 10, for example with respect to coding losses introduced by quantization. The same may be true for further signals introduced below. In the following description, reference is made to both Figures 1 and 2.

[0017] The encoder 10 comprises a transform module 20 configured to subject the picture 12, more precisely the spatial representation of the picture 12, to a spatial-spectral transform, which may for example comprise one or more two-dimensional arrays comprising sample values ​​of one or more components of the picture 12, to obtain a transformed signal 22 of the picture 12. It is noted that depending on the applied spatial-spectral transform, the transformed signal may comprise, in addition to the spectral information, also spatial information about the picture. This may be particularly true in examples where a discrete wavelet transform (DWT) is used, as for example described in more detail with respect to Figs. 3 to 5. Further examples of possible transforms are the discrete sine transform and the discrete cosine transform. The transformed signal 22 provided by the transform module 20 represents the picture 12 in terms of spectral components. That is to say, for example, the transformed signal 22 comprises transform coefficients attributable to the spectral components of the picture. For example, the spectral components may be associated with a frequency and, optionally, with the transform direction used when the spectral components were obtained.

[0018] The encoder 10 further comprises a residual former 30 configured to derive a residual transform signal 32 based on the transform signal 22 of the previous picture 12* and a buffered transform signal 52. The buffered transform signal 52 includes a selection from the spectral components representing the previous picture 12*. For example, the encoder 10 may obtain the buffered transform signal 52 from a further transform signal representing the previous picture 12*. The encoder 10 further comprises an encoding module 70 configured to encode the residual transform signal 32 into the data stream 14. For example, the encoding module may quantize the residual transform signal 22 and encode the quantized signal using entropy coding.

[0019] The decoder 11 is for decoding or reconstructing a sequence of pictures 15 in the same order 16 as they were encoded by the encoder 10. The decoder 11 comprises a decoding module 71 configured to derive or decode a residual transform signal 32' of a picture 12' from the data stream 14. As already mentioned, the residual transform signal 32' may differ from the residual transform signal 32 encoded by the encoder 10 in terms of coding losses. In other words, the residual transform signal 32' may correspond to the residual transform signal 32 despite the coding losses. The description provided herein for the residual transform signal 32 may equally apply to the residual transform signal 32' and vice versa, despite the coding losses. In the following description, apostrophes may be omitted, i.e. for example, the reference numbers 12, 22 and 32 refer to both the encoder-side and the decoder-side signals.

[0020] The decoder 11 further comprises a residual synthesizer 31 configured to synthesize the residual transform signal 32' with a buffered transform signal 52 of a previous picture 12*, for example decoded before the picture 12*. The buffered transform signal 52 may correspond to the buffered transform signal 52 used by the residual former 30 to derive the residual transform signal 32. For example, the decoder 11 may derive the buffered transform signal 52 from a further transform signal 22'* representing the previous picture 12*. For example, the decoder 11 may comprise a buffer module 51 configured to obtain the buffered transform signal 52 using the further transform signal 22'* representing the previous picture 12*. For example, the decoder 11 may derive the further transform signal 22'* of the previous picture as described for the transform signal 22' of the current picture. For example, the residual synthesizer 31 may perform the inverse of the operations performed by the residual former 30. For example, residual former 30 may subtract buffered transform signal 52 from transform signal 22, and residual combiner 31 may add buffered transform signal 52 to residual transform signal 32. In other words, transform signal 22' may correspond to transform signal 22 despite the coding loss contained in residual transform signal 22' compared to residual transform signal 22. Despite the coding loss, the descriptions provided herein for transform signal 22 may be equally applicable to transform signal 22', and vice versa.

[0021] The decoder 11 further comprises an inverse transform module 21 configured to subject the transformed signal to a spectral-spatial transform, for example the spectral-spatial transform being the inverse of the spatial-spectral transform applied to the picture 12 by the transform module 20. For example the spectral-spatial transform results in a spatial representation of the picture 12', for example as described with respect to the encoder 10.

[0022] The use of the buffered transform signal 52 to obtain the residual transform signal 32 can enhance the rate-distortion relationship as follows: the residual former 30 can use, for example, one of the spectral components contained in the buffered transform signal to determine a residual representative of the deviation between the transform signal 22 and the buffered transform signal 52 in the coefficients of the spectral component. In other words, the residual former 30 can determine, for a spectral component contained in the buffered transform signal, a residual between the transform signal 22 and the buffered transform signal 52. Thus, coefficients of the transform signal 22 that belong to one of the spectral components contained in the buffered transform signal 52 and that deviate little from their respective corresponding coefficients of the buffered transform signal 52 may be represented in the residual transform signal 32 by particularly small residual transform coefficients, which allows a good rate-distortion relationship in the encoding step 70.

[0023] According to one embodiment, the transformed signal 22, 22' is independent of previous pictures of the sequence of pictures. For example, the transformed signal 22 is obtained without prediction, e.g. without temporal prediction. Similarly, the spectral-spatial transform 21 may result in a spatial representation of the picture 12' that is a representation independent of previous pictures. Avoiding temporal prediction of pictures in the position domain (i.e. spatial representation) may allow small buffer sizes of the encoder 10 and the decoder 11.

[0024] For example, in existing schemes of video compression, for prediction in the spatial domain, the encoder needs to include inverse quantization and inverse spatial decorrelation steps to model the decoder to follow a "closed-loop" design, for example as described with respect to Fig. 10. Compared to these schemes, deriving the transform signal independently of the previous picture reduces the complexity of the encoder, since not both transform and quantization have to be performed in both directions. For example, in the framework of JPEG XS, such simplicity can be beneficial.

[0025] For example, embodiments of the present invention may be applied within the framework of JPEG XS, a lightweight still image compression codec specifically designed for low-complexity and low-latency applications, i.e., applications where only limited processing power is available, only limited memory is available, and image data is encoded and decoded with only minimal latency. Thus, embodiments of the present invention may be implemented in accordance with JPEG XS, e.g., inverse transform module 21, residual combiner 31, and encoding module 71.

[0026] According to one embodiment, the transform signal 22 comprises a number of transform coefficients, each associated with a spectral component of the picture.

[0027] According to one embodiment, the transform 20 comprises a DWT and the inverse transform 21 comprises an inverse DWT, where each of the spectral components may be represented by a number of transform coefficients, each associated with a spatial coordinate, i.e. the transform signal 22 may contain both spectral and spatial information regarding the picture 12.

[0028] 3 shows an example of a spatial-spectral transform 20 according to an embodiment. According to this embodiment, the transform module 20 performs multiple or a sequence of DWTs on the picture 12 to obtain a transformed picture 24. For example, each DWT is performed with a respective wavelet of a respective transform direction (e.g. horizontal or vertical). The transformed picture 24 comprises multiple transformed representations 25 of the picture 12. Each of the transformed representations 25 is obtained by a respective number of one or more DWTs applied to the picture 12 or to a previous transformed representation 25, if performed sequentially.

[0029] Each transformed representation 25 includes a number of transform coefficients. For example, each transformed representation 25 may include a 2D array of transform coefficients, where the coordinates of the transform coefficients within the transformed representation represent spatial information about the picture 12. 1 ,twenty five 2 ,twenty five 33 by , may have different sizes, i.e., different numbers of transform coefficients, depending on the number of subsequently applied DWTs and / or the wavelets and / or transform directions used. 2 is transformed by horizontal and vertical DWT. 1 may be derived from the transformation representation 25 3 is transformed by the horizontal DWT. 2 It may be derived from one of the following:

[0030] Each of the transformed representations 25 may be associated with a respective spectral component, for example depending on the wavelet used when the transformed representation was obtained. For example, each spectral component is characterized by or associated with the frequency and / or transformation direction of the respective wavelet. In other words, each of the transformed representations 25 is associated with a spectral component of the picture 12. Each spectral component may thus include a number of transform coefficients, i.e., the transform coefficients of its associated transformed representation 25. The size of the transformed representation 25 may depend on the frequency associated with the transformed representation 25. More precisely, a transformed representation 25 representing a low frequency spectral component may be represented by a smaller number of transform coefficients as a transformed representation 25 representing a high frequency spectral component.

[0031] For example, the transform 20 and inverse transform 21 may be implemented according to JPEG XS, which applies the discrete wavelet transform ("DWT") as a spatial decorrelation transform. This transform converts the image data into multiple bands (e.g., the spectral components mentioned above), each band corresponding to image structures at a particular scale, such that the largest bands contain information about the finest structures of the image and the smallest bands contain information about the coarsest image structures. Figure 4 shows such a division into wavelet bands. The shaded areas are the three largest bands and contain the finest image structures.

[0032] In other words, FIG. 4 shows an example of the transformed picture 24 of FIG. 3 being segmented into a transformed representation 25. 1 is associated with the highest frequency spectral component of the transformed picture 24 .

[0033] For example, the selection of the spectral components of the buffered transformed signal 52 of FIGS. 1 and 2 may be based on the transform coefficients of the highest frequency spectral components of the transformed picture 24, i.e., the transform representation 25. 1 Dropping the highest frequency components may result in a significant reduction in the required buffer size compared to other spectral coefficients, but the downside in the rate-distortion relationship may be relatively small.

[0034] With regard to the (inverse) transformations performed by the transformation modules 20 and 21, respectively, it should be noted that in alternative embodiments to those described with respect to Figures 3 and 4, an (inverse) discrete cosine transform (DCT) or an (inverse) discrete sine transform can be used. In this case, for example, the transformation by the transformation module 20 can result in a transformed picture comprising a number of transform coefficients, each associated with a respective frequency and transformation direction. In this case, each of the transform coefficients can represent a different combination of frequency and transformation direction, such that each transform coefficient can represent an individual spectral component.

[0035] 1 and 2, the residual combiner 31 is, according to one embodiment, configured to combine the residual transform signal 32 with the buffered transform signal 52 by combining mutually corresponding spectral components of the buffered transform signal 52 and the residual transform signal 32. For example, mutually corresponding spectral components are those resulting from equivalent spatial-spectral transformations using the same frequency filters, associated with equivalent spatial coordinates, or associated with equivalent spectral components and equivalent parts of the picture.

[0036] According to one embodiment, the transform signal 22 comprises a plurality of transform coefficients, for example as described with respect to figures 3 and 4, and the residual transform signal comprises, for each transform coefficient of the transform signal 22, a corresponding residual transform coefficient. According to this embodiment, the residual synthesizer 31 is configured to synthesize mutually corresponding transform coefficients of the residual transform signal 22 and the buffered transform signal 52. The buffered transform signal may thus comprise a plurality of buffered transform coefficients associated with respective spectral components and, optionally, spatial coordinates. Similarly, the residual former 30 is, according to this embodiment, configured to derive the residual transform signal of the picture by predicting or forming the spectral components of the transform signal using corresponding spectral components of the buffered transform signal.

[0037] According to one embodiment, the buffered transform signal 52 comprises for each transform coefficient of the transform signal 22 a corresponding buffered transform coefficient, the buffered transform coefficients associated with the spectral components not included in the selection of spectral components being set to a predefined value, for example 0. In an alternative embodiment, the buffered transform signal 52 comprises only buffered transform coefficients related to the spectral components included in the selection of spectral components. According to this alternative, the residual synthesizer 31 determines the transform coefficients corresponding to the buffered transform coefficients associated with the spectral components not included in the selection of spectral components by taking over the values ​​of the mutually corresponding residual transform coefficients of the residual transform signal 32.

[0038] According to one embodiment, the residual former 30 determines the residual transform coefficients of the residual transform signal 32 by subtracting corresponding buffered transform coefficients of the buffered transform signal 52 from corresponding transform coefficients of the transform signal 22. According to this embodiment, the residual combiner 31 determines the transform coefficients of the transform signal 22 by adding corresponding buffered transform coefficients of the buffered transform signal 52 to corresponding residual transform coefficients of the residual transform signal 32.

[0039] As already mentioned with respect to Figures 1 and 2, the encoder 10 and the decoder 11 may derive the buffered transformed signal 52 from a further transformed signal 22'* representing a previous picture 12*. The encoder 10 and the decoder 11 may for example derive the further transformed signal 22* in the same way as they derive the transformed signal 22 of the current picture, except that for the further transformed signal, the previous picture 12* is used as input. In other words, the decoder 11 may decode the previous picture 12* to obtain the further transformed signal 22*.

[0040] According to one embodiment, the decoder 11 decodes the previous picture by decoding, by means of a decoding module 71, a further residual transform signal of the previous picture from the data stream 14. The further residual transform signal is input to a residual synthesizer 31, which combines the further residual transform signal with a further buffered transform signal of a further earlier picture of the sequence to obtain a further transform signal. The further transform signal represents the previous picture in spectral content, similar to the transform signal of the current picture. The further transform signal is input to an inverse transformer 21, which subjects the further transform signal to a spectral-spatial transform to obtain a spatial representation of the previous picture 12*.

[0041] The derivation process for the buffered transformed signal 52 can therefore be illustrated as a loop in which a further transformed signal of a previous picture, which may be of the same manner as the transformed signal of the current picture, is fed back to derive the buffered transformed signal 52 for the derivation of the current picture. In the diagram of Figure 2, reference numeral 19 indicates the position of the loop, the use of which switches between the derivation of the current picture and the derivation of the previous picture.

[0042] However, it should be noted that the previous picture 12* is not necessarily decoded using the further buffered transformed signal, but rather the previous picture 12* may be decoded independently of further pictures or independently of any picture of the sequence before the previous picture, for example for a particular picture, such as the first picture of a sequence or a refresh picture or an intra-coded picture. In this case, the synthesis step performed by the residual synthesizer 31 may be skipped. Similarly, for such pictures, the encoder 10 may skip the prediction step performed by the residual former 30.

[0043] According to an embodiment, the buffer module 51 derives the buffered transformed signal 52 from the further transformed signal by selecting a subset of the spectral components of the further transformed signal for the buffered transformed signal 52. In this example embodiment, the subset may be a proper subset of all the spectral components of the further transformed signal, i.e. does not include all the spectral components of the further transformed signal. According to this embodiment, the buffer module 51 sets the spectral components of the buffered transformed signal, which are the spectral components not included in the selection, to a predetermined value. For example, the buffer module 51 sets all the coefficients of the buffered transformed signal 52, for example the above-mentioned buffered transform coefficients associated with the spectral components not included in the selection, to a predetermined value. For example, the predetermined value is 0, especially when the residual synthesizer performs a summation of the transform coefficients and the mutually corresponding buffered transform coefficients.

[0044] For example, the buffer module 51 may buffer (e.g. store) the spectral components of the further transformed signal based on which spectral components are included in the selection, e.g. exclusively buffering these spectral components while discarding (or not buffering) the spectral components that are not included in the selection.

[0045] According to one embodiment, the selection of the spectral components of the buffered transformed signal 52 includes at least one spectral component associated with a non-zero frequency. The inclusion of non-zero frequency components allows to achieve a good rate-distortion relationship, but the required buffer size may still be moderate, as explained with respect to figure 3, especially when selecting non-zero frequency components as low frequency components. According to an alternative embodiment, the selection of the spectral components of the buffered transformed signal 52 includes exclusively spectral components associated with a zero frequency or a constant level, or exclusively includes one or more spectral components associated with the lowest frequency of the frequencies to which the spectral components of the picture are associated.

[0046] According to an embodiment, the pictures 12, 12* are coded into the data stream 14 in units of portions (for example called precincts), each portion being associated with a spatial region of the picture. For this purpose, the transform signal and / or the residual transform signal may be subdivided into portions. Each of the portions may be associated with a spatial region within the picture.

[0047] 3 further illustrates an example of a portion 26 of the transformed signal 22. The portion 26 of the transformed signal includes transform coefficients within each block or region of one or more transformed representations 25 resulting from the transform 20. For example, in FIG. 1 Each of the areas 26 1 Of which, 25 conversion expressions 2 Each of the areas 26 2 25 Inner and Conversion Expressions 3 Each of the areas 26 3 That is, in FIG. 3, portion 26 includes transform coefficients in region 26 1 , 26 2 , 26 33, the transform coefficients in portion 26 relate to (e.g., represent information of) a region 16 in picture 12, or more precisely in a spatial representation (representation in the position domain) of picture 12. As shown in FIG. 3, portion 26 may be distributed in transformed picture 24 across all transformed representations 25, or may comprise sub-portions in each of the transformed representations 25.

[0048] For example, portion 26 may include all transform coefficients within a respective block of transformed representation 25, with each block of the transformed representation scaled in position and size according to the relative scale of the transformed representation.

[0049] For example, portion 26 may include transform coefficients associated with region 16 of picture 12. Note that the boundaries of portion 26 in the transform domain may be sharp, i.e. each of the transform coefficients may belong to exactly one portion of the transform signal. Thus, the boundaries of region 16 in picture 12 may be blurred, the degree of this blurring depending on the size of the wavelet template used in the DWT. Thus, one sample of picture 12 may belong to multiple regions 16, this is especially true for samples located at the borders of regions 16. Thus, region 16 may overlap with adjacent regions.

[0050] According to one embodiment, portion 26 and region 16 extend across the entire horizontal size of picture 12 .

[0051] According to an embodiment in which the transform 20 comprises a DWT, the encoder 10 rearranges the transformed signal 22 into a rearranged representation, for example as described below with respect to Fig. 5, and the encoder 10 performs a residual formation 30 using the rearranged representation of the transformed picture 24 signaled by the transform signal 22. The decoder 11 can thus inversely rearrange the transformed signal 22' obtained by a residual synthesizer 31 and perform an inverse transform 31 on the inversely rearranged transformed signal, which can correspond to the encoder-side transformed signal 22 before rearrangement despite the coding loss.

[0052] 5 shows an example of a rearranged representation 24' of the transformed picture 24 that may result from a rearrangement of the transformed picture 24. The rearranged representation includes or is sub-divided into a number of portions 26, of which reference sign 26* is used to identify portion 26 of FIG. 3. As shown in FIG. 5, in the rearranged representation 24', regions 26 belonging to different spectral components but belonging to portion 26* are 1 , 26 2 , 26 3 are rearranged or grouped to form a block in the rearranged representation.

[0053] For example, one of the portions 26, or generally a portion of the transformed signal carrying information about an area 16 of the picture, may be called a precinct. Note that one precinct does not necessarily cover the full width of the (transformed) picture, as shown in Figures 3 and 5. For example, the rearranged representation may be subdivided into slices, each of which covers the full width of the rearranged representation 24' (and thus carries information of a slice of picture 12, which is a slice covering the full width of the picture). One slice may optionally be subdivided into one or more precincts. In the example of Figures 3 and 5, each slice has one precinct, which therefore covers the full width.

[0054] Depending on the embodiment, a slice consists of 8 or 4 precincts. For example, a slice may be 16 lines high and have one vertical decomposition level corresponding to 8 precincts, each of which has 4 precincts from 2 levels.

[0055] It should be noted that block-based coding is not limited to embodiments in which a DWT transform is used: for example, when using a DCT or DST, the picture 12 may be subdivided into blocks, and each block may undergo a DCT or DST transform to obtain respective transformed blocks forming portions of the transformed signal, each portion being associated with a spatial region of the picture, i.e. a block into which the picture is subdivided.

[0056] It should further be noted that for each portion of the transform signal 22, there may be a corresponding portion of the residual transform signal 32 and a corresponding portion of the buffered transform signal 52. However, the buffered transform signal 52 may not necessarily include spectral components or buffered transform coefficients for all portions of the transform signal, as in the example, but only a subset of the portions may be buffered. For portions not included in the buffered transform signal, the transform coefficients may be set to a predetermined value, e.g. 0, or may be treated by the residual synthesizer 31 as having a predetermined value.

[0057] As mentioned above, embodiments of the present invention can be implemented according to JPEG XS. JPEG XS groups transformed image data in wavelet bands, e.g., transformed signal 22, into so-called slices, each slice describing a spatial region of the original image, e.g., region 16, that extends across the full image width but only over a subsection of the image's vertical extent, e.g., as shown in Fig. 3 and Fig. 5. The example of Fig. 5 may be such an arrangement, with the dotted region in this figure consisting of one slice. It should be noted that each band in a slice (e.g., one of the shaded regions) now describes image structure, e.g., spectral content, at a scale corresponding to the wavelet band, each band describing structure in a portion of the image, e.g., a portion of the image that extends through the full image width but only through a subset of its vertical extent.

[0058] Continuing with the description of figures 1 and 2, according to one embodiment, the residual transform signal 32 of figures 1 and 2 comprises a plurality of spectral components for each of the portions 26, for example as described with respect to figure 3. According to this embodiment, the buffer module 31 derives the buffered transform signal 52 from the further transform signal by selecting and buffering a subset, for example a suitable subset, of the spectral components of one or more portions, for example a slice, of a previous picture. According to this embodiment, the buffer module obtains the transform signal 22 by combining a spectral component of one of the portions 26 of the buffered transform signal 52 with a corresponding spectral component of the corresponding portion 26 of the residual transform signal.

[0059] Thus, the buffering of the further transformed signal can be performed on a portion-by-portion basis, such that the selection of the buffered spectral components can be performed on a portion-by-portion basis, i.e. the encoder 10 can determine the selection of the buffered spectral components from the previous picture separately for each portion of the transformed signal 22, or for each group of portions, e.g. for each of multiple slices of a picture.

[0060] So, for example, the available buffers are not necessarily evenly distributed among the parts: for example, for a first one of the parts, no or few spectral components can be buffered, whereas for a second one of the parts, a larger number of spectral components can be buffered. For example, the selection may be performed for each part based on the transformed signal or the further transformed signal, allowing an efficient use of the available buffers, i.e. finding a good rate-distortion relationship taking into account the available buffers. The selection of the parts may also be performed according to a refresh sequence, for example as described with respect to FIG. 8.

[0061] According to one embodiment, the decoder 11 derives the selection of the spectral components to be buffered for the commonly buffered transformed signal for all parts of the picture. For example, the same spectral components are selected for buffering for all parts 26 of the picture. According to an alternative embodiment, the picture is coded into the data stream slice by slice, each of the slices comprising one or more of the parts. For example, one slice extends over the entire horizontal size of the picture 12. According to this embodiment, the device derives the selection of the spectral components to be buffered for the commonly buffered transformed signal for all parts of one slice of the picture.

[0062] According to one embodiment, the decoder 11 derives the selection of the spectral components to be buffered for the buffered transformed signal from a syntax element signaled in the data stream, e.g., the syntax element signaling control parameter 92 described with respect to Fig. 8. For example, the data stream 14 includes a dedicated syntax element for signaling the selection of the spectral components. The encoder 10 may be configured to encode the syntax element into the data stream 14.

[0063] In an example, the selection of the spectral components to be buffered may be performed part-by-part by deciding for each line of the transformed representation 25, for example for each line of a band resulting from the DWT, whether to buffer the further transformed signal of the respective line. Instead of deciding line-by-line, the transformed representation may be subdivided into parts, for example bands, comprising one or more lines of the transformed representation (for example extending over the entire width of the transformed representation), and the decision may be made for each part of the transformed representation. Since each part refers to a spatial region of the picture and a spectral component associated with the respective transformed representation, it can be determined for each spectral component which region of the picture should be buffered. It is noted that, with regard to granularity, the height of the parts of the transformed representation may be the same or different for each of the transformed representations, for example bands, for example by several lines.

[0064] Thus, the decoder may derive the buffered transformed signal 52 from the further transformed signal 22'* by selecting and buffering one or more of the portions of the spectral components of the further transformed signal, e.g., by determining for each of the portions of the spectral components whether to buffer the respective portion. For example, the portions may be selected in the manner described above for determining for each of the portions of the transformed representation representing the spectral components whether to buffer the respective portion of the further transformed signal. As described above, each of the portions may include one or more (e.g., entire) lines or rows of the transformed representation. Thus, each portion may include or consist of coefficients derived from the spatial domain of the input image. The portion-wise decision regarding the buffering of the spatial domain may be performed for all spectral components or for a subset of the spectral components, e.g., the subset of the spectral components selected to be buffered. In this example, the device may set the portions of the buffered transformed signal 52 that are not included in the selection to be buffered to a predetermined value. In obtaining the transform signal 22 of the picture, the decoder may combine the selected and buffered portions of the spectral components of the buffered transformed signal 52 with the corresponding portions of the corresponding spectral components of the residual transform signal 32'.

[0065] Fig. 6 shows another example of an encoder 10 according to an embodiment. According to this embodiment, the encoder 10 comprises a residual synthesizer 31 configured to synthesize a further residual transform signal of the previous picture, indicated with reference 32'* in Fig. 6, with a further buffered transform signal, indicated with reference 52*, to obtain a reconstructed transform signal 22'* of the previous picture. The reconstructed transform signal 22'* may correspond to the further transform signal of the previous picture, despite the coding loss. The reconstructed transform signal 22'* may correspond to the further transform signal 22'* described above with respect to the decoder, but may differ from the further transform signal of the previous picture, as provided by the transform module 20, with respect to the coding loss. The residual synthesizer 31 of the encoder 10 may correspond to the residual synthesizer 31 of the decoder 11, i.e. it performs the same operations on its input signal as described with respect to the decoder 11. According to this embodiment, the encoder 10 further comprises a buffer module 51 which receives the reconstructed transformed signal 22 ′* and provides a buffered transformed signal 52 , eg as described with reference to the buffer module 51 of the decoder 11 .

[0066] 6, the encoder 10 may encode the previous picture 12* by subjecting the further spatial representation of the previous picture 12* to a spatio-spectral transformation performed by the transform module 20 to provide a further transformed signal. The residual former 30 may derive a further residual transform signal based on the further buffered transform signal 52* and the further transform signal of a further earlier picture in the sequence, and the encoder 10 may derive the buffered transform signal 52 from the further residual transform signal.

[0067] According to one embodiment, the encoder 10 comprises a quantizer 60 configured to quantize the residual transform signal 32 of the picture 12, thereby obtaining a quantized signal 62. Similarly, the quantizer 60 can quantize a further residual transform signal of the previous picture 12*, thereby providing a further quantized signal 62*. The quantization step size or the quantization model that the quantizer 60 uses for quantization may be described by a quantization parameter. According to this embodiment, the encoder 10 comprises an inverse quantizer 61. The inverse quantizer 61 scales or inverse quantizes the further quantized signal 62* so as to obtain a dequantized further residual signal 32'*, which is provided to the residual combiner 31. Since the dequantized further residual signal 32'* may correspond to the further residual transform signal of the previous picture 12*, as provided by the residual former 30, despite the coding losses introduced by quantization and inverse quantization, the dequantized further residual transform signal 32'* may also be referred to as the further residual transform signal 32'*, e.g., as done above in the description of the residual synthesizer 31 of the encoder 10. In other words, in an embodiment including a quantizer 60 and an inverse quantizer 61, the further residual transform signal 32* provided to the residual synthesizer 31 is provided by the inverse quantizer 61 and may include quantization losses.

[0068] According to one embodiment, the encoder 10 comprises an encoding module 80 configured to encode the quantized signal 62 into the data stream 14. For example, the encoding module 80 may encode the quantized signal 62 using entropy coding, for example using a bounded code such as JPEG XS. Forming a residual by the residual former 30 of at least a portion of the transform coefficients and quantizing the residual transform signal 62 may result in a large number of quantization levels having the value 0 in the quantized signal 62, so that the entropy coding may result in a high compression ratio.

[0069] In other words, the wavelet transformed data, e.g., the residual transform signal obtained from the transformed signal 22, may be quantized to a limited precision to remove extraneous image content and create many zero coefficients. The quantized data may then be entropy coded. By quantizing many of the coefficients to zero, entropy coding can efficiently represent the quantized data using fewer bits than the original image, allowing for compact storage and transmission.

[0070] Figure 7 shows an example of a decoder 11 according to an embodiment that may follow the example of the encoder 10 of Figure 6. In particular, the decoder 11 of Figure 6 may comprise, according to this embodiment, an inverse quantizer 61 configured to inverse quantize a quantized signal 62 so as to provide a residual transform signal 32' that may deviate from the residual transform signal 32 due to quantization losses.

[0071] According to one embodiment, the decoder 11 further comprises a decoding module 81 configured to derive or decode the quantized signal 62 from the data stream, for example using entropy decoding, for example using a bounded code such as JPEG XS.

[0072] For example, the quantizer 60 and the encoding module 80 may be part of the encoding module 70 of Figure 1, and the inverse quantizer 61 and the decoding module 81 may be part of the decoding module 71 of Figure 2. It should be noted that the decoding module 71 may perform reverse operations of the operations of the decoding module 71, which may be particularly applicable to the quantization and entropy coding steps.

[0073] FIG. 8 shows another embodiment of an encoder 10, which may optionally correspond to the embodiment of FIGS.

[0074] As mentioned above, an embodiment of the present invention performs temporal prediction after spatial decorrelation but before quantization. In particular, a discrete wavelet transform ("DWT") can be applied as a first step to create a cascade of wavelet bands as seen in Fig. 5, with temporal prediction being applied within these bands, i.e. in the wavelet domain (or transform domain), and not in the spatial domain. As shown in Fig. 8, decorrelation 20 is performed before temporal prediction 30, and according to this embodiment, neither motion estimation nor motion prediction is performed, avoiding the complexity of such operations.

[0075] The inventive concept has the advantage that the temporal prediction can be applied optionally only to a part of the bands, limiting the amount of storage of the frame buffer at the encoding and decoding side. For example, the bands for which the temporal prediction is used may be selected or controlled or determined by a control parameter 92. Fig. 8 shows an exemplary implementation of the buffer module 51, according to which, depending on a control parameter 92 controlled, for example per slice, data in the frame buffer 56 may be replaced with a predefined value 93, for example 0, so that no prediction is performed for a subset of the bands in a given slice, for example the bands set to the predefined value. For example, in this way the buffered or stored data, i.e. the buffered transformed signal 52, requires less data storage. For example, the control parameter 92 may be determined by or in response to an optional quality control module 96, for example taking into account a desired rate-distortion relationship or available resources as described above. Additionally or alternatively, the control parameter 92 may be determined by a gradual refresh mechanism, as described with reference to Figs. 8 and 9.

[0076] Control parameters 92 may be included in the entropy coded data, allowing the decoder to adjust its prediction mechanism to the same settings as the encoder. For example, if the three diagonal bands in FIG. 4 were not stored in the frame buffer, the size of the frame buffer would be reduced to 1 / 4 of its original size and the bandwidth between the frame buffer and the main processor would be reduced by a factor of 1 / 4. Because the diagonal bands contain only the finest scale high frequency content, the impact on image quality remains minimal and good compression is still obtained.

[0077] An additional advantage is that the encoder does not need to include an additional inverse transform to model the state of the decoder-side frame buffer: instead, it only needs to perform a low-complexity inverse quantization according to a "closed-loop" design.

[0078] In the coding scheme described with reference to figures 1 to 8, the encoder 10 may only transmit the output of the temporal transformation. In some cases, the decoder 11 therefore does not have access to the complete encoded sequence, for example when switching to an already running connection. In such a case, the decoder is unable to reconstruct the image due to lack of information from past frames. However, typical applications require faithful reconstruction of the original data after a minimal delay after switching to a running stream (for example, up to about 200 ms). At the same time, it is also desirable to ensure a minimal latency, i.e. to minimize the delay between the decoder receiving the image data and the decoder being able to reconstruct the same image data for the same timestamp and the same image area after switching to the running connection. This delay typically needs to be much lower than the start-up delay and may even be constrained, for example, to a fraction of a frame.

[0079] As a solution, an optional part of the invention is the combination of the partial frame buffer (selective buffer of the previously transformed signal described in connection with Figs. 1 to 8) mentioned above with a mechanism, for example called gradual refresh mechanism, that allows a decoder switching to a connection without access to the initial data to restore the complete image after some temporarily predicted frames. In particular, in combination with the selection per part of the spectral components, in particular the control parameters 92 described in Fig. 8, the selection by the control parameters for example can be adjusted per slice, so that prediction can be completely disabled for certain parts of the picture, as required for example by the gradual refresh control. In such a partial refresh, none of the bands (for example the bands of the slice to which the partial refresh refers), even a subset of them, receives a prediction from the frame buffer.

[0080] 9 shows an example of a subsequence 17 of a sequence 15 of pictures according to one embodiment. The subsequence 17 may be referred to as a refresh sequence. The subsequence 17 may include a number of pictures of the sequence 15, an exemplary number of three pictures 12 in FIG. 1 , 12 2 , 12 3 According to one embodiment, each picture of the subsequence 17 includes picture 12 of the subsequence 17. 1 , 12 2 , 12 3 6, picture 12 is encoded by encoder 10 into data stream 14 by suppressing prediction of one or more spectral components of each of the one or more portions of the picture using corresponding spectral components of the buffered transformed signal, and predicting one or more spectral components of each of the one or more further portions of the picture using corresponding spectral components of the buffered transformed signal. The pictures are encoded such that the unbuffered portions of entire pictures of the subsequence are combined to cover the entire picture area of ​​the picture. For example, as shown in FIG. 6, picture 12 is 1 Part 261 is coded without predicting the transform signal 22 using the previous picture, and picture 12 2 Part 26 2 is coded without predicting the transform signal 22 using the previous picture, and picture 12 3 Part 26 3 is coded without predicting the transformed signal 22 using the previous picture. 1 , 26 2 , 26 3 In other words, the pictures of the sequence may be equally subdivided into a number of slices, each of which may be coded without prediction from a previous picture, at least in the coding of one of the pictures of the sequence.

[0081] Thus, for example, the last picture of a subsequence is coded independently of any pictures of sequence 15 that precede the first picture of the subsequence, and therefore, after decoding an entire subsequence, decoder 11 can fully decode further pictures even if the decoder has no information about previous pictures of the subsequence.

[0082] According to the embodiment described in relation to FIG. 6, the decoder 11 decodes the pictures of the subsequence 17 by suppressing, for each picture of the subsequence, combining one or more spectral components of each of one or more parts of the picture with corresponding spectral components of the buffered transformed signal, and combining one or more spectral components of each of one or more further parts of the picture with corresponding spectral components of the buffered transformed signal.

[0083] It should be noted that the pictures of the subsequences do not necessarily follow each other directly, but further pictures of sequence 15 may be alternated with pictures of subsequence 17 .

[0084] In other words, according to one embodiment, the encoder 10 includes a gradual refresh control module 90 as shown in FIG. 8 as an optional feature. The gradual refresh control module 90 can signal a control parameter 92 that determines, for a slice or for each slice, which bands are involved in the prediction. For areas not involved in the prediction, e.g. slices, the decoder can reconstruct all or part of the image data without relying on the contents of the frame buffer 56, e.g. without relying on previous pictures, since the contents of the frame buffer 56 are replaced by a predefined value, e.g. 0. Slices that do not include bands in the frame buffer and therefore do not undergo temporal prediction may be denoted below as "I-slices", and slices that include some temporally predicted bands as "P-slices". According to one embodiment, a portion of all slices are transmitted as I-slices, so that after a refresh interval of several frames, e.g. the subsequence 17 described in FIG. 8, all slices are transmitted without temporal prediction, e.g. within at least one picture of the sequence, allowing the decoder to refresh its frame buffer.

[0085] As an example of such a scheme, consider an image consisting of N slices. In FIG. 8, N is illustratively 3. For frame number k, slice k mod N can be transmitted as an I-slice, where "mod" is the modulus operator. In this exemplary scheme, a full frame refresh is reached after N subsequent frames. Other schemes for transmitting slice kp mod N as an I-slice can also be envisaged, for example, where p and N are relatively early. As a third example, the encoder can also select those slices as I-slices that benefit least from temporal prediction, i.e., those slices that have changed too much with respect to the previous frame to make temporal prediction worthwhile. Such a scheme can be combined with static I-slice allocation as given in the first or second examples.

[0086] Compared to the conventional mechanism of occasionally sending an I-frame (non-predictive frame) for decoder refresh, i.e., encoding the entire frame without temporal prediction, the gradual refresh mechanism may cause no or little bitrate fluctuation. More precisely, during gradual refresh, the rates of subsequent frames of the coded refresh sequence may not deviate much from each other, compared to coding a full refresh picture every now and then (e.g., an "I-frame" every 10 "P-frames"). This rate of I-frames is large and requires a larger rate than P-frames, and therefore also a larger input buffer at the decoder. However, a larger input buffer at the decoder also increases the latency to ensure that a sufficient amount of data is buffered. Thus, gradual refresh may have the advantage of either a smaller buffer and therefore a reduced delay, or an increased quality of the non-predictive frames.

[0087] In other words, one embodiment of the present invention described with respect to Figs. 1-9 provides a method for coding a temporally related image sequence described, which exploits temporal relationships within the image sequence for further bandwidth reduction. The method includes a spatial decorrelation transform based on a wavelet filter followed by a temporal decorrelation transform in the transform domain of the spatial decorrelation transform. That is, unlike common schemes, the temporal prediction is performed in the filtered domain. To reduce processing complexity, the temporal decorrelation may be applied only to a subset of the output of the spatial decorrelation. To ensure that the decoder can reconstruct the complete image even when it has not received the first frame of the transmission, a gradual update procedure is described, which transmits a window of the original image without temporal prediction. This gradual update window is moved over the frames to ensure that the full frame can be reconstructed after some time.

[0088] Fig. 10 shows an example of an alternative coding scheme, in connection with which the advantages of the present invention over some existing video coding schemes are discussed below. Examples of video codecs typically used are H.261, H.264 (AVC) or H.265 (HEVC). In such schemes, the data first undergoes a temporal decorrelation, for example by creating difference data between the predicted frame and the original frame, then the image is divided into blocks and a spatial decorrelation transform, such as a discrete cosine transform ("DCT") or a discrete sine transform ("DST"), is performed on these blocks. Usually, this prediction also includes an estimation of the motion in the original data, which is then predicted only from the motion compensation data. It is noted that schemes such as H.261 can also work without estimating the motion, if this step is too complex.

[0089] In order to allow the decoder to reconstruct the full frame after some time, even when switching to a running connection, such a design sometimes transmits a full frame without motion information. In the following, such frames are denoted as "I" frames. Frames that include a prediction step from motion are called "P" frames. Additional prediction mechanisms may exist, but are not relevant to the invention described herein.

[0090] However, it is important to note that all of the above schemes perform temporal prediction before spatial decorrelation (i.e., a spatial decorrelation transform is performed over the temporally predicted data), the frame buffer is the same size as that of the original image, and the I-frame contains data for a full refresh. However, some existing designs, such as video coding standards H.263+ and ISO / IEC 14496-2, allow additional spatial prediction of the transform coefficients, i.e., provide additional optional prediction in the transform domain, but this relies on spatial correlation, i.e., different spatial regions of the same picture are used for the prediction of the transform coefficients.

[0091] Temporal prediction typically follows a "closed-loop" design, i.e., the encoder contains a model of the decoder, allowing it to accurately predict what output the decoder will produce, and performs its temporal prediction on this predicted output. Without such closed-loop temporal prediction, the output of the actual encoder and encoder-side frame buffer would slowly drift, which after a while could cause visible image distortion. Figure 10 provides an overview of the steps of a conventional video coding scheme.

[0092] In contrast, embodiments of the present invention may use temporal prediction in the transform domain rather than spatial prediction in the transform domain.

[0093] In conventional video codecs such as those mentioned above, non-predicted frames ("I" frames) require a larger amount of bits even after conversion, or require higher quantization, degrading their quality, if only a maximum upper limit bandwidth is available for transmitting the data. This results in either peaks in the transmission rate, which requires larger buffers at the encoder and decoder side to smooth such rate derivation, or a noticeable quality degradation for such non-predicted "I" frames.

[0094] According to an embodiment of the present invention, gradual refresh can be easily implemented by modulating the control parameter defining the number of bands involved to 0. In contrast to video coding standards such as H.264 and subsequent entities that already allow gradual refresh applied to all transformed coefficients, an embodiment of the present invention allows modulation of the amount of temporal prediction where no prediction as required for gradual refresh is only a special case. Thus, precise control of buffers and data rates can be achieved.

[0095] Other codecs, such as JPEG XS, currently do not include a mechanism for temporal prediction and therefore require bandwidth that is too high for some applications. Compared to the methods for temporal prediction in the video coding standard H.261 and later, where the encoder and decoder must include memory to represent a full frame and bandwidth must be provided between the actual processing units on the encoder / decoder and local storage ("RAM") to retrieve data from past frames to enable temporal prediction, the proposed scheme offers the advantage that the size of the local storage can be dynamically scaled by an external control parameter, allowing the system to be adjusted to the image complexity and system bandwidth. Note that in one example of an extreme setting of this parameter, no data may be included at all, such that no temporal prediction is performed.

[0096] Although some aspects are described as features in the context of an apparatus, it is clear that such description may also be considered as a description of the corresponding features of a method. Although some aspects are described as features in the context of a method, it is clear that such description may also be considered as a description of the corresponding features in terms of the functionality of the apparatus. In particular, the block diagrams of Figures 1, 2, 6, 7, 8, and 9 used in describing the encoder and decoder may also be understood as block diagrams describing the corresponding methods for encoding and decoding.

[0097] Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0098] The inventive encoded image signal can be stored on a digital storage medium and can be transmitted over a transmission medium, such as the Internet, a wireless transmission medium or a wired transmission medium, etc. In other words, further embodiments provide a video bitstream product, such as a digital storage medium having a video bitstream stored thereon, comprising a video bitstream according to any of the embodiments described herein.

[0099] Depending on the specific implementation requirements, embodiments of the present invention can be implemented in hardware or software, or at least partly in hardware, or at least partly in software. The implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to execute the respective method. The digital storage medium may therefore be computer readable.

[0100] Some embodiments according to the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0101] Generally, embodiments of the present invention can be implemented as a computer program product having program code that operates to perform one of the methods when the computer program product is run on a computer. The program code can, for example, be stored on a machine readable carrier.

[0102] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0103] In other words, therefore, an embodiment of the inventive methods is a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0104] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0105] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals can for example be arranged to be transferred via a data communication connection, for example via the Internet.

[0106] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0107] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0108] Further embodiments according to the invention comprise an apparatus or a system arranged to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0109] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods may be performed by any hardware apparatus.

[0110] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0111] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0112] Although the present invention has been described with respect to several embodiments, there are modifications, permutations, and equivalents that fall within the scope of the present invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. Therefore, it is intended that the following appended claims be interpreted to include all such modifications, permutations, and equivalents that fall within the true spirit and scope of the present invention.

[0113] In the above detailed description, it can be seen that various features have been grouped together in the examples for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the appended claims reflect, subject matter may reside in less than all features of a single disclosed example. Accordingly, the appended claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate example. Although each claim may stand on its own as a separate example, it should be noted that although a dependent claim may refer to a specific combination with one or more other claims in a claim, other examples may also include a combination of the dependent claim with the subject matter of each of the other dependent claims, or a combination of each feature with other dependent or independent claims. Such combinations are suggested herein unless it is stated that a specific combination is not intended. Furthermore, the inclusion of a feature of a claim in any other independent claim is also contemplated, even if that claim is not directly dependent on the independent claim.

[0114] The above-described embodiments are merely illustrative of the principles of the present disclosure. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented as descriptions and explanations of the embodiments herein.

Claims

1. An apparatus (11) for decoding a sequence of pictures (15) from a data stream (14), said apparatus comprising: deriving (71) a residual transform signal (32') of said picture (12') from said data stream; combining (31) said residual transform signal (32') with a buffered transform signal (52) of a previous picture (12*) of said sequence to obtain a transform signal (22') of said picture, said transform signal representing said picture in terms of spectral components; subjecting said transformed signal to a spectral-spatial transformation (21); and adapted to decode a picture (12') of said sequence by An apparatus (11), wherein the buffered transformed signal (52) includes a selection from spectral components representing the previous picture.

2. 2. The apparatus of claim 1, wherein the apparatus is configured to combine the residual transform signal (32') with the buffered transform signal (52) by combining mutually corresponding spectral components of the buffered transform signal (52) and the residual transform signal (32').

3. 3. The apparatus of claim 1 or 2, wherein the selection of spectral components signaled in the buffered transformed signal (52) includes at least one spectral component associated with a non-zero frequency.

4. The apparatus according to claim 1 , wherein the spectral-spatial transformation is an inverse discrete wavelet transform.

5. 5. The apparatus of claim 1, wherein the apparatus is for deriving the selection of spectral components to be buffered for the buffered transformed signal (52) from syntax elements signaled in the data stream.

6. The apparatus, decoding the previous picture, decoding a further residual transform signal (32') of the previous picture from the data stream; - combining said further residual transform signal (32') with a further buffered transform signal (52*) of an even earlier picture of said sequence to obtain a further transform signal (22'), said further transform signal (22'*) representing said earlier picture in terms of spectral content; - subjecting said further transformed signal (22'*) to a spectral-spatial transformation in order to obtain a spatial representation of said previous picture; decoding the previous picture by deriving said buffered transformed signal (52) from said further transformed signal (22'*); 6. The apparatus of claim 1 , configured to:

7. The apparatus, selecting, for said buffered transformed signal (52), a subset of the spectral components of said further transformed signal (22'*); setting a spectral component of the buffered transformed signal (52) to a predetermined value, the spectral component not being included in the selection; 7. The apparatus of claim 6, configured to derive the buffered transformed signal (52) from the further transformed signal (22'*) by:

8. The picture is coded into the data stream in portions, each of the portions being associated with a spatial domain of the picture, the residual transform signal (32') comprising, for each of the portions, a plurality of spectral components, the apparatus comprising: deriving the buffered transformed signal (52) from the further transformed signal (22'*) by selecting and buffering a subset of spectral components of one or more portions of the previous picture; obtaining the transform signal (22) of the picture by combining spectral components of one of the portions of the buffered transform signal (52) with corresponding spectral components of a corresponding portion of the residual transform signal (32'); 8. Apparatus according to claim 6 or 7, configured to:

9. the device is for deriving the selection of buffered spectral components for the buffered transformed signal (52) in common for all parts of the picture, or 9. The apparatus of claim 8, wherein the picture is coded into the data stream in slices, each of the slices including one or more of the portions, and the apparatus is for deriving the selection of spectral components to be buffered for the buffered transformed signal (52) in common for all portions of a slice of the picture.

10. The apparatus, for each of the pictures of the subsequence, inhibiting combining one or more spectral components of each of one or more portions of the picture with corresponding spectral components of the buffered transformed signal (52); and combining one or more spectral components of each of one or more further portions of the picture with corresponding spectral components of the buffered transformed signal (52); and configured to decode a sub-sequence of pictures of said sequence of pictures by 10. The apparatus of claim 8 or 9, wherein the unbuffered portions of all of the pictures of the subsequences in combination cover a full picture area of ​​the picture.

11. The apparatus, deriving the buffered transformed signal (52) from the further transformed signal (22'*) by selecting and buffering one or more of a plurality of portions of the spectral components of the further transformed signal; obtaining the transform signal (22) of the picture by combining the selected and buffered portions of the spectral components of the buffered transform signal (52) with corresponding portions of corresponding spectral components of the residual transform signal (32'); 11. The apparatus of claim 6, configured to:

12. the transformed signal and the further transformed signal comprise, for each spectral component, a transformed representation of the picture, the transformed representation comprising a plurality of coefficients of a sample array; The apparatus of claim 11 , wherein each of the plurality of portions of the spectral component comprises one or more rows of the sample array of the transformed representation of the spectral component.

13. An apparatus (10) for encoding a sequence of pictures into a data stream, said apparatus (10) comprising: - subjecting the spatial representation of the picture to a spatial-spectral transformation (20) to obtain a transformed signal (22), the transformed signal (22) representing the picture in terms of spectral components; deriving (30) a residual transform signal (32) based on a buffered transform signal (52) of a previous picture of the sequence and on the transform signal (22); encoding (70) the residual transform signal (32) of the picture into the data stream; and adapted to encode the pictures of the sequence by An apparatus (10) wherein the buffered transformed signal (52) includes a selection from spectral components representing the previous picture.

14. 14. The apparatus of claim 13, wherein the apparatus is configured to derive the residual transform signal (32) of the picture by predicting spectral components of the transform signal (22) using corresponding spectral components of the buffered transform signal (52).

15. 15. Apparatus according to claim 13 or 14, wherein the selection of spectral components signalled in the buffered transformed signal (52) comprises at least one spectral component associated with a non-zero frequency.

16. 16. Apparatus according to any one of claims 13 to 15, wherein the spatial-spectral transformation is a discrete wavelet transform.

17. 17. The apparatus of claim 13, wherein the apparatus is for encoding into the data stream syntax elements indicating the selection of buffered spectral components for the buffered transformed signal (52).

18. The apparatus, encoding the previous picture, - subjecting a further spatial representation of said previous picture to said spatial-spectral transformation to obtain a further transformed signal, said further transformed signal representing said previous picture in terms of spectral components; deriving a further residual transform signal (32*) based on a further buffered transform signal (52*) of an even earlier picture of the sequence and the further transform signal; encoding said further residual transform signal (32*) into said data stream; encoding the previous picture by deriving said buffered transform signal (52) from said further residual transform signal (32*); 18. The apparatus of claim 13, configured to:

19. The apparatus, combining said further residual transform signal (32*) with said further buffered transform signal (52) to obtain a reconstructed transform signal (22'*) of said previous picture; selecting, for said buffered transformed signal (52), a subset of the spectral components of said reconstructed transformed signal (22'*); setting a spectral component of the buffered transformed signal (52) to a predetermined value, the spectral component not being included in the selection; 19. The apparatus of claim 18, configured to derive the buffered transform signal (52) from the further residual transform signal (32*) by:

20. The apparatus, encoding the residual transform signal (32) by quantizing the residual transform signal (32) and encoding the quantized residual transform signal (62) into the data stream; encoding said further residual transform signal (32*) by quantizing said further residual transform signal and encoding said quantized further residual transform signal (62) into said data stream; deriving the buffered transform signal (52) by scaling the quantized further residual transform signal (62) and combining the scaled further residual transform signal (32'*) with the further buffered transform signal (52*); 20. The apparatus of claim 18 or 19, configured to:

21. The apparatus is configured to code the picture into the data stream in portions, each of the portions being associated with a spatial domain of the picture, the residual transform signal (32) comprising a plurality of spectral components for each of the portions of the picture, and the apparatus: deriving the buffered transform signal (52) from the further residual transform signal (32'*) by selecting for the buffered transform signal (52) a subset of spectral components of one or more portions of the previous picture; deriving the residual transform signal (32) of the picture by predicting spectral components of one of the portions of the transform signal (22) using corresponding spectral components of a corresponding portion of the buffered transform signal (52); 21. The apparatus of claim 18, configured to:

22. the apparatus is for encoding into the data stream a syntax element indicating the selection of buffered spectral components for the buffered transformed signal (52) in common for all parts of the picture; 22. The apparatus of claim 21, wherein the apparatus is for encoding the picture into the data stream in slices, each of the slices including one or more of the portions, and the apparatus is for encoding into the data stream a syntax element indicating the selection of buffered spectral components for the buffered transformed signal (52) in common for all portions of a slice of the picture.

23. The apparatus is configured to encode a sub-sequence of pictures, the sub-sequence comprising: suppressing, for each of the pictures of the subsequence, predicting one or more spectral components of each of one or more portions of the picture using corresponding spectral components of the buffered transformed signal (52); predicting one or more spectral components of each of one or more further portions of the picture using corresponding spectral components of the buffered transformed signal (52); a sub-sequence of said sequence of pictures according to 23. Apparatus according to claim 21 or 22, wherein the unbuffered portions of all of the pictures of the sub-sequences in combination cover a full picture area of ​​the picture.

24. The apparatus, deriving the buffered transform signal (52) from the further residual transform signal (32'*) by selecting and buffering one or more of a plurality of portions of the spectral components of the further residual transform signal (32'*); deriving the residual transform signal (32) of the picture by using the selected and buffered portions of the spectral components of the further residual transform signal to predict corresponding portions of corresponding spectral components of the transform signal (22); 24. Apparatus according to any one of claims 18 to 23, configured to:

25. the residual transform signal and the further residual transform signal comprise, for each spectral component, a transformed representation of the picture, the transformed representation comprising a plurality of coefficients of a sample array; each of the plurality of portions of the spectral component comprises one or more rows of the sample array of the transformed representation of the spectral component; 25. The apparatus of claim 24.

26. 1. A method for decoding a sequence of pictures from a data stream, the method comprising: Deriving (71) a residual transform signal (32') of said picture from said data stream; combining (31) said residual transform signal (32') with a buffered transform signal (52) of a previous picture of said sequence to obtain a transform signal (22') of said picture, said transform signal (22') representing said picture in terms of spectral components; subjecting (21) said transformed signal (22') to a spectral-spatial transformation; decoding the sequence of pictures by The method, wherein the buffered transformed signal (52) includes a selection from spectral components representing the previous picture.

27. 1. A method for encoding a sequence of pictures into a data stream, the method comprising the steps of: - subjecting (20) the spatial representation of the picture to a spatial-spectral transformation to obtain a transformed signal (22), the transformed signal (22) representing the picture in terms of spectral components; deriving (30) a residual transform signal (32) based on a buffered transform signal (52) of a previous picture of the sequence and on the transform signal (22); encoding (70) the residual transform signal (32) of the picture into the data stream; encoding pictures of the sequence by The method, wherein the buffered transformed signal (52) includes a selection from spectral components representing the previous picture.

28. A non-transitory digital storage medium having stored thereon a computer program, which, when executed by a computer, produces a method for decoding a sequence of pictures from a data stream, the method comprising the steps of: deriving a residual transform signal (32') of said picture from said data stream; combining said residual transform signal (32') with a buffered transform signal (52) of a previous picture of said sequence to obtain a transform signal (22') of said picture, said transform signal (22') representing said picture in terms of spectral components; subjecting said transformed signal (22') to a spectral-spatial transformation; decoding pictures of the sequence by A non-transitory digital storage medium implementing a method, wherein the buffered transformed signal (52) includes a selection from spectral components representing the previous picture.

29. A non-transitory digital storage medium having stored thereon a computer program, which, when executed by a computer, produces a method for encoding a sequence of pictures into a data stream, the method comprising the steps of: - subjecting the spatial representation of the picture to a spatial-spectral transformation to obtain a transformed signal (22), the transformed signal (22) representing the picture in terms of spectral components; deriving a residual transform signal (32) based on a buffered transform signal (52) of a previous picture of the sequence and on the transform signal (22); encoding the residual transform signal (32) of the picture into the data stream; encoding pictures of the sequence by A non-transitory digital storage medium implementing a method, wherein the buffered transformed signal (52) includes a selection from spectral components representing the previous picture.

30. A data stream having a sequence of pictures encoded therein, the sequence of pictures being encoded in a data stream, the method comprising: - subjecting the spatial representation of the picture to a spatial-spectral transformation to obtain a transformed signal (22), the transformed signal (22) representing the picture in terms of spectral components; deriving a residual transform signal (32) based on a buffered transform signal (52) of a previous picture of the sequence and on the transform signal (22); encoding the residual transform signal (32) of the picture into the data stream; encoding pictures of the sequence by The buffered transformed signal (52) is encoded into the data stream by a method that includes a selection from spectral components representing the previous picture.

Images