Decoder, encoder and corresponding methods for prediction-based residual coding

Abstract

A block-based prediction decoder for decoding an image from a data stream is configured to derive a prediction residual block for a currently decoded block based on a prediction block for the currently decoded block, and to reconstruct the currently decoded block based on the prediction residual block and the prediction block.

Description

Technical Field

[0001] Embodiments of the present invention relate to decoders, encoders, and corresponding methods for prediction-based residual coding and decoding, and particularly to prediction-based residual coding and prediction-based residual reconstruction. Background Technology

[0002] The most common video codecs (such as AVC, HEVC, and VVC) encode video images in a block-based manner, where the video signal within a block of images is first predicted based on other images (inter-frame prediction) or based on neighboring samples within the same image (intra-frame prediction). Additionally, residual signals are transmitted to compensate for prediction errors. The residual signals are typically obtained by subtracting the predicted signal from the original video signal. For efficient transmission, the residual signals are transformed; then the resulting transform coefficients are quantized and entropy-coded.

[0003] On the decoder side, the quantization index undergoes entropy decoding, dequantization, and inverse transform to obtain the reconstructed residual signal. Finally, the reconstructed residual signal is added to the prediction signal, which is derived in the same manner as on the encoder side. It should be noted that due to the lossy nature of quantization, it is impossible to perfectly reconstruct the residual signal. Typically, to allow for parallel execution of prediction and residual decoding, the residual decoding process is independent of the prediction process.

[0004] The aim is to provide solutions that make image encoding / decoding and / or video encoding / decoding more efficient. Alternatively, it is desirable to reduce the bitstream and thereby lower signaling overhead.

[0005] This is achieved through the subject matter of the independent claims of this application.

[0006] Other embodiments of the invention are defined by the subject matter of the dependent claims of this application. Summary of the Invention

[0007] According to one aspect of the invention, the inventors of this application recognized that if the prediction signal is taken into account, the prediction residual signal (e.g., the residual signal) can be encoded and decoded more efficiently. This is based on the idea that information in the prediction signal may be valuable for the encoding and decoding of the residual signal, for example, in terms of improving the prediction residual signal itself so that it can be encoded and decoded more efficiently or with reduced signaling overhead; or in terms of improving the residual encoding / decoding tools (such as transform modules) so that the prediction residual signal can be encoded and decoded more efficiently. The inventors found that the additional information provided by the prediction signal improves the encoding / decoding efficiency of the residual signal, although this consideration of the prediction signal may hinder the parallel execution of the prediction process and the residual encoding / decoding.

[0008] Therefore, according to this aspect of the application, an image or video decoder is configured to: derive one or more prediction signals (e.g., generate or predict one or more prediction signals, prediction blocks, or predicted blocks); reconstruct a residual signal from a bitstream / data stream (e.g., decode a quantized residual signal from a bitstream / data stream (e.g., a quantization index associated with the residual signal), inverse quantize the quantized residual signal to obtain an inverse quantized residual signal, and perform an inverse transform on the inverse quantized residual signal to obtain a reconstructed residual signal); and combine the residual signal (e.g., the reconstructed residual signal) with one or more prediction signals (e.g., to obtain a reconstructed signal, such as a block-based, image-based, or video-based reconstructed signal). The reconstruction of the residual signal depends on one or more prediction signals. For example, the image or video decoder is configured to reconstruct the residual signal based on one or more prediction signals.

[0009] A corresponding image or video encoder can be configured to: derive one or more predicted signals; encode a residual signal into a bitstream; and combine the residual signal with one or more predicted signals. The encoding of the residual signal depends on one or more predicted signals. The image or video encoder is based on the same considerations as the image or video decoder described above. The encoder performs the opposite operation to the decoder, where the prediction processes of the encoder and decoder are identical.

[0010] The corresponding image or video encoding / decoding method can be configured to: derive one or more predicted signals; encode the residual signal into a bitstream / reconstruct the residual signal from the bitstream based on the one or more predicted signals; and combine the residual signal with one or more predicted signals.

[0011] Therefore, according to this aspect of the application, a block-based predictive decoder for decoding an image from a data stream is configured to: derive a predictive residual block of the current decoded block based on (e.g., depending on) a predictive block of the current decoded block (e.g., directly based on the predictive block or based on one or more features derived from the predictive block), and reconstruct the current decoded block based on the predictive residual block and the predictive block. According to one embodiment, the predictive block may be one of two or more (e.g., multiple) predictive blocks of the current decoded block, and for example, the block-based predictive decoder is configured to reconstruct the current decoded block based on the predictive residual block and two or more predictive blocks. The decoder may be configured to derive the predictive residual block of the current decoded block based on one or more of the two or more predictive blocks (e.g., at least based on the predictive blocks).

[0012] The corresponding block-based predictive encoder for encoding an image into a data stream is configured to: derive (or encode) a predictive residual block of the current coding block based on (e.g., depending on) the predictive block of the current coding block (e.g., directly based on the predictive block or based on one or more features derived from the predictive block), such that the current coding block can be reconstructed based on the predictive residual block and the predictive block. The block-based predictive encoder is based on the same considerations as the block-based predictive decoder described above. The encoder performs the opposite operation to the decoder, where the prediction processes of the encoder and decoder are identical.

[0013] A corresponding method for encoding an image into / decoding an image from a data stream includes: deriving a prediction residual block for the current encoded / decoded block based on (e.g., depending on) a prediction block of the current encoded / decoded block. The current encoded / decoded block can be reconstructed based on the prediction residual block and the prediction block.

[0014] It should be noted that the terms "prediction block" and "prediction signal" are used interchangeably in this document. That is, a prediction block can be a prediction signal, and a prediction signal can be a prediction block.

[0015] The method described above is based on the same considerations as the encoder / decoder described above. Furthermore, the method can supplement all the features and functions described regarding the encoder / decoder.

[0016] One embodiment relates to a data stream having images or videos encoded therein using the encoding methods described herein.

[0017] One embodiment relates to a computer program having program code for implementing the methods described herein when run on a computer, wherein the methods are executed on the computer. Attached Figure Description

[0018] The accompanying drawings are not necessarily drawn to scale, but are generally intended to illustrate the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, wherein:

[0019] Figure 1 An embodiment of the encoder is shown;

[0020] Figure 2 An embodiment of the decoder is shown;

[0021] Figure 3 The reconstructed signal based on the combination of the predicted residual signal and the predicted signal is shown;

[0022] Figure 4a illustrates an example of residual decoding that depends on the predicted signal;

[0023] Figure 4b illustrates an example of residual coding that depends on the predicted signal;

[0024] Figure 5a illustrates an example of residual decoding that depends on the characteristics of the predicted signal;

[0025] Figure 5b illustrates an example of residual coding that depends on the characteristics of the predicted signal;

[0026] Figure 6 The unmodified residual decoding is shown;

[0027] Figure 7a illustrates residual decoding using a transform choice that depends on the predicted signal;

[0028] Figure 7b illustrates residual coding using a transform selection that depends on the predicted signal;

[0029] Figure 8a illustrates residual decoding using a transform output that depends on the predicted signal;

[0030] Figure 8b illustrates residual coding using a transform output that depends on the predicted signal;

[0031] Figure 9a illustrates residual decoding in the spatial domain using residual modifications that depend on the predicted signal;

[0032] Figure 9b illustrates residual coding in the spatial domain that uses residual modifications that depend on the predicted signal;

[0033] Figure 10a illustrates residual decoding in the spectral domain using residual modifications dependent on the predicted signal; and

[0034] Figure 10b illustrates residual coding in the spectral domain that uses residual modifications that depend on the predicted signal. Detailed Implementation

[0035] In the following description, identical or equivalent elements, or elements having the same or equivalent functions, are indicated by the same or equivalent reference numerals or identified by the same name; and, even if they appear in different figures, repeated descriptions of elements identified by the same reference numerals or the same name are generally omitted. Therefore, the descriptions given for elements identified by the same or similar reference numerals or the same name may be interchangeable or applicable to each other in different embodiments.

[0036] In the following description, numerous details are set forth to more fully illustrate embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention. Furthermore, unless explicitly stated otherwise, features of the different embodiments described below can be combined with each other.

[0037] The following accompanying drawings first describe the encoder and decoder of a block-based predictive codec for encoding and decoding images of video, as an example of an encoding and decoding framework in which embodiments of the present invention can be incorporated. (Refer to...) Figures 1 to 3 The corresponding encoder and decoder are described. Subsequently, embodiments of the inventive concept are described, and how these concepts can be incorporated respectively. Figure 1 and Figure 2 In the encoder and decoder; however, the embodiments described in conjunction with the following Figure 4a and the accompanying drawings can also be used to form not based on Figure 1 and Figure 2 The encoder and decoder operate on the encoding and decoding framework upon which they are based.

[0038] Figure 1 An apparatus (e.g., a video encoder or image encoder) is shown that, exemplarily, uses transform-based residual coding to predictively encode image 12 into data stream 14. This apparatus or encoder is indicated by reference numeral 10. Figure 2 The corresponding decoder 20 is shown, that is, the device 20 configured to predictively decode the image 12' from the data stream 14 using the same transform-based residual decoding, wherein the superscript apostrophe is used to indicate that the image 12' reconstructed by the decoder 20 deviates from the image 12 originally encoded by the device 10 due to the encoding and decoding loss introduced by the quantization of the predictive residual signal. Figure 1 and Figure 2 Transform-based predictive residual coding and decoding are used exemplarily, but embodiments of this application are not limited to this predictive residual coding and decoding. As will be outlined below, this also applies to references to Figure 1 and Figure 2 Other details of the description.

[0039] Encoder 10 is configured to perform a spatial-to-spectral transformation on the prediction residual signal and encode the resulting prediction residual signal into data stream 14. Similarly, decoder 20 is configured to decode the prediction residual signal from data stream 14 and perform a spectral-to-spatial transformation on the resulting prediction residual signal.

[0040] Internally, encoder 10 may include a prediction residual signal former 22 that generates a prediction residual 24 to measure the deviation of the prediction signal 26 relative to the original signal (i.e., relative to image 12). According to one embodiment of the invention, the prediction signal 26 may be understood as the prediction signal of a predictor block (prediction block), or as a linear combination of one or more predictor blocks (prediction blocks). For example, the prediction residual signal former 22 may be a subtractor that subtracts the prediction signal from the original signal (i.e., from image 12 or a current block of image 12). Encoder 10 also includes a transformer 28 that performs a space-to-spectral transformation on the prediction residual signal 24 to obtain a spectral domain prediction residual signal 24', which is then quantized by a quantizer 32 also included in encoder 10. The quantized prediction residual signal 24'' is thus encoded into bitstream 14. For this purpose, encoder 10 may optionally include entropy encoder 34, which entropy encodes the transformed and quantized prediction residual signal into data stream 14.

[0041] The prediction signal 26, or prediction block, is generated by the prediction stage 36 of encoder 10 based on the prediction residual signal 24'', which has been encoded into data stream 14 and can be decoded from data stream 14. For this purpose, as... Figure 1 As shown, prediction stage 36 internally includes a dequantizer 38 that dequantizes the prediction residual signal 24'' to obtain a spectral domain prediction residual signal 24''' corresponding to signal 24', excluding quantization loss. It then includes an inverse transformer 40 that performs an inverse transform on the latter's prediction residual signal 24''', i.e., a spectral-to-spatial transform, to obtain a prediction residual signal 24'''' corresponding to the original prediction residual signal 24, excluding quantization loss. Subsequently, combiner 42 of prediction stage 36, for example by addition, recombines the prediction signal 26 or prediction block with the prediction residual signal 24'''' or prediction residual block to obtain reconstructed signal 46, i.e., a reconstruction of the original signal 12 or a reconstruction of the current decoded / encoded block. However, it should be noted that more than one prediction signal 26 or prediction block can be combined with the prediction residual signal 24'''' or prediction residual block for reconstruction. Reconstructed signal 46 can correspond to signal 12' or a block of signal 12'. Subsequently, the prediction module 44 of the prediction stage 36 generates a prediction signal 26 based on the signal 46 using, for example, spatial prediction (i.e., intra-frame image prediction) and / or temporal prediction (i.e., inter-frame image prediction).

[0042] Similarly, such as Figure 2As shown, decoder 20 can internally be composed of components corresponding to and interconnected with those of prediction stage 36. Specifically, entropy decoder 50 of decoder 20 can entropy decode the quantized spectral domain prediction residual signal 24'' from the data stream. Subsequently, the dequantizer 52, inverse transformer 54, combiner 56, and prediction module 58, interconnected and cooperating in the manner described above with reference to the modules of prediction stage 36, recover the reconstructed signal based on the prediction residual signal 24'', such that... Figure 2 As shown, the output of combiner 56 generates the reconstructed signal, i.e., image 12'.

[0043] Although not specifically described above, it is evident that encoder 10 can set some coding parameters, including prediction modes and motion parameters, according to some optimization scheme, such as optimizing some bitrate and distortion-related standards (i.e., coding costs). For example, encoder 10, decoder 20, and corresponding modules 44 and 58 can each support different prediction modes, such as intra-frame coding / decoding modes and inter-frame coding / decoding modes. The granularity of switching between these prediction mode types by encoder and decoder can correspond to the subdivision of image 12 and image 12' into coding / decoding segments or blocks, respectively. For example, based on these coding / decoding segments, the image can be subdivided into blocks for intra-frame coding / decoding and blocks for inter-frame coding / decoding.

[0044] Intra-code blocks are predicted based on spatially encoded / decoded neighborhoods (e.g., the current template) of the corresponding block (e.g., the current block), as outlined in more detail below. Several intra-code modes may exist, and these modes may be selected for corresponding intra-code segments, including directional or angular intra-code modes. According to a directional or angular intra-code mode, the corresponding segment is populated by extrapolating sample values ​​of the neighborhood along a direction specific to the directional intra-code mode. For example, intra-code modes may also include one or more other modes, such as DC (Distributed DC) and / or planar intra-code modes. According to a DC mode, the prediction of the corresponding intra-code block assigns DC values ​​to all samples within the corresponding intra-code segment. According to a planar intra-code mode, the prediction of the corresponding block is approximated or determined as a spatial distribution of sample values ​​described by a two-dimensional linear function over the sample locations of the corresponding intra-code block, wherein the driving tilt and offset of the plane defined by the two-dimensional linear function are determined based on neighboring samples.

[0045] In contrast, for example, inter-frame codec blocks can be predicted temporally. For an inter-frame codec block, a motion vector can be signaled within data stream 14, indicating a spatial displacement of a portion of a previously encoded / decoded image (e.g., a reference image) of the video to which image 12 belongs, where the previously encoded / decoded image is sampled to obtain a prediction signal (e.g., a prediction block) for the corresponding inter-frame codec block. This means that, in addition to the residual signal encoding included in data stream 14 (such as entropy-encoded transform coefficient levels representing the quantized spectral domain prediction residual signal 24''), data stream 14 may also encode: codec mode parameters for assigning codec modes to the various blocks; prediction parameters for some blocks, such as motion parameters for inter-frame codec segments; and optional other parameters, such as parameters for controlling and signaling the subdivision of image 12 and image 12' into segments, respectively. Decoder 20 uses these parameters to subdivide the image in the same manner as the encoder, assigning the same prediction modes to the segments and performing the same predictions to obtain the same prediction signal.

[0046] Figure 3 This illustrates the relationship between the reconstructed signal (i.e., the reconstructed image 12') and the combination of the prediction residual signal 24'''' and the prediction signal 26 signaled in the data stream 14 on the other hand. As mentioned above, the combination can be additive. Figure 3 In the diagram, prediction signal 26 is shown as a subdivision of an image region into intra-frame and inter-frame codec blocks, wherein intra-frame codec blocks are exemplarily represented by shaded lines, while inter-frame codec blocks are exemplarily represented by unshaded lines. The subdivision can be any subdivision, such as regularly subdividing the image region into rows and columns of square or non-square blocks, or subdividing the image 12 from a root block into multiple leaf blocks of different sizes, such as a quadtree subdivision, etc. Figure 3 The diagram illustrates a blend of these subdivisions, where the image region is first subdivided into rows and columns of root blocks, which are then further subdivided into one or more leaf blocks according to a recursive multi-branch tree subdivision.

[0047] Furthermore, for intra-frame codec block 80, the data stream 14 may encode an intra-frame codec mode that assigns one of several supported intra-frame codec modes to the corresponding intra-frame codec block 80. For inter-frame codec block 82, the data stream 14 may encode one or more motion parameters. Generally, inter-frame codec block 82 is not limited to being time-coded. Alternatively, inter-frame codec block 82 may be any block predicted from a previously codec portion outside the current image 12 itself, such as a previously codec image of the video to which image 12 belongs, or, in the case of a scalable encoder and a scalable decoder, an image of another view or an image of a lower layer in a hierarchy.

[0048] Figure 3 The prediction residual signal 24'''' in the image is also shown as a subdivision of the image region into block 84. To distinguish these blocks from the encoder / decoder blocks, predictor blocks, or prediction blocks 80 and 82, these blocks can be referred to as transform blocks or prediction residual blocks. In fact, Figure 3 It is shown that encoder 10 and decoder 20 can subdivide images 12 and 12' into blocks using two different subdivisions: one subdivision into codec blocks 80 and 82, and the other subdivision into transform blocks 84. These two subdivisions can be the same, that is, each codec block 80 and 82 can simultaneously form transform block 84, but... Figure 3 This illustrates a situation where, for example, subdividing into transform blocks 84 constitutes an extension of subdividing into codec blocks 80 and 82, such that any boundary between any two blocks in blocks 80 and 82 coincides with the boundary between the two blocks 84; in other words, each block 80 and 82 either coincides with one of the transform blocks 84 or with a set of transform blocks 84. However, subdivisions can also be determined or selected independently of each other, such that transform blocks 84 can alternatively span the block boundaries between blocks 80 and 82. Similar statements regarding subdividing into transform blocks 84 apply to subdividing into blocks 80 and 82, i.e., block 84 can be the result of regularly subdividing an image region into blocks (arranged or not arranged in rows and columns), the result of recursive multi-branch tree subdivision of the image region, or a combination thereof, or any other type of block subdivision. Incidentally, it should be noted that blocks 80, 82, and 84 are not limited to squares, rectangles, or any other shape.

[0049] Figure 3 It is also shown that the combination of prediction signal 26 and prediction residual signal 24'''' directly yields the reconstructed signal 12'. However, it should be noted that, according to alternative embodiments, more than one prediction signal 26 can be combined with prediction residual signal 24'''' to obtain image 12'. Therefore, the current decoded block 18 of image 12' can be reconstructed based on prediction residual block (e.g., see 86) and prediction block (e.g., see 80), for example by combining prediction residual block and prediction block; or, it can be reconstructed based on prediction residual block (e.g., see 86) and multiple prediction blocks, for example by combining prediction residual block and multiple prediction blocks.

[0050] exist Figure 3In this context, transform block 84 should have the following meaning. Transformer 28 and inverse transformer 54 perform their transforms on a unit basis using these transform blocks 84. For example, many codecs use some kind of DST (Discrete Sine Transform) or DCT (Discrete Cosine Transform) for all transform blocks 84. Some codecs allow skipping transforms, so that for some transform blocks 84, the prediction residual signal is directly encoded and decoded in the spatial domain. However, according to the embodiments described below, encoder 10 and decoder 20 are configured such that they support several transforms. For example, the transforms supported by encoder 10 and decoder 20 may include:

[0051] - DCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform

[0052] - DST-IV, where DST represents Discrete Sine Transform

[0053] - DCT-IV

[0054] - DST-VII

[0055] - Identity transformation (IT).

[0056] Of course, transformer 28 will support all forward transform versions of these transforms, while decoder 20 or inverse transformer 54 will support their corresponding backward or inverse transform versions:

[0057] - Inverse DCT-II (or Inverse DCT-III)

[0058] - Inverse DST-IV

[0059] - Inverse DCT-IV

[0060] - Inverse DST-VII

[0061] - Identity transformation (IT).

[0062] In any case, it should be noted that the supported set of transforms may include only one transform, such as a spectrum-to-space transform or a space-to-spectrum transform; but it is also possible that the encoder or decoder does not use any transforms at all, or does not use any transforms for a single block 80, 82, 84.

[0063] As outlined above, Figures 1 to 3 As an example, this illustrates a specific example of an encoder and decoder according to this application, in which the inventive concepts described further below can be implemented. Figure 1 and Figure 2 The encoder and decoder can represent possible implementations of the encoder and decoder described below, respectively. However, Figure 1 and Figure 2This is merely an example. The encoder according to embodiments of this application can perform block-based encoding of image 12 using the concepts outlined in more detail below, and can differ from... Figure 1 The encoder, for example, differs in that it is different from... Figure 3 The illustrated method performs subdivision down to block 80, and / or uses no transform at all or no transform for a single block. Similarly, the decoder according to embodiments of this application can perform block-based decoding of image 12' from data stream 14 using the encoding / decoding concepts further outlined below, but may differ from, for example... Figure 2 Decoder 20 differs in that it uses a different method than the reference decoder. Figure 3 The described method subdivides the image 12' into blocks, and / or the decoder derives the prediction residual from the data stream 14 in the spatial domain, for example, instead of in the transform domain, and / or the decoder does not use any transform at all or for a single block. According to one embodiment, the inventive concept further described below can be implemented in the residual encoding module 110 of the video encoder 10 or in the residual decoding module 210 of the video decoder 20.

[0064] For example, one embodiment relates to a block-based prediction decoder 20 for decoding an image 12' from a data stream 14, wherein the block-based prediction decoder 20 is configured to derive a prediction residual block 86 of the current decoded block 18 based on a prediction block 80 of the current decoded block 18 (e.g., based on features extracted from the prediction block). The current decoded block 18 can be reconstructed based on the prediction residual block 86 and the prediction block 80, for example, as... Figure 3 The reconstruction is achieved by combining both. Alternatively, the block-based predictive decoder 20 is configured to derive a predictive residual block 86 of the current decoded block 18 based on multiple predictive blocks (including predictive block 80) (e.g., based on features extracted from multiple predictive blocks), and to reconstruct the current decoded block 18 based on the predictive residual block 86 and the multiple predictive blocks, for example, by combining both. The corresponding block-based predictive encoder 10 for encoding the image 12 into the data stream 14 is configured to derive or encode a predictive residual block of the current encoded block based on the predictive block or multiple predictive blocks (e.g., based on features extracted from the predictive block or multiple predictive blocks), such that the current encoded block can be reconstructed based on the predictive residual block and the predictive block or multiple predictive blocks. Details of such a block-based predictive encoder 10 and decoder 20 are described with reference to Figures 4a to 10b, wherein the corresponding encoder 10 or decoder 20 may include features of the individual figures or combinations of features described with reference to two or more figures in Figures 4a to 10b.

[0065] The basic concept of this invention is to relax the constraints of separate prediction and residual encoding / decoding, and instead encode / decode the residual signal based on the prediction signal. The rationale behind this is that the prediction signal may contain information that allows the residual signal to be encoded / decoded more efficiently. Specifically, this invention includes novel ideas regarding: 1) how to extract such information from the prediction signal; and 2) how to utilize this information for residual encoding / decoding.

[0066] Figures 4a and 4b illustrate a general overview of the invention. Ignoring dashed arrow 26, Figure 4a shows a conventional decoding process, and Figure 4b shows a conventional encoding process: a prediction signal 26 (e.g., a prediction block) is derived by a prediction module (see 44 in Figure 4b and 58 in Figure 4a), wherein the decoder 20 can use information 202 (e.g., motion vectors or prediction patterns) from bitstream 14, which can be encoded into bitstream 14 by the corresponding encoder 10, and wherein the decoder 20 and the corresponding encoder 10 can use information from previous image data 204 (e.g., motion compensation blocks from previously decoded / encoded images or neighboring samples in the current image).

[0067] Furthermore, the residual decoding module 210 derives the reconstructed residual signal 24'''' from the bitstream 14. Finally, the prediction signal 26 and the reconstructed residual signal 24'''' (e.g., the prediction block of the current decoding block and the reconstructed prediction residual block of the current decoding block) are combined to obtain the reconstructed video signal 206 (e.g., by deriving a weighted sum of the prediction signals and adding that weighted sum to the reconstructed residual signal). The corresponding encoder 10 performs the opposite operation to the decoder 20, wherein the residual signal 24 is formed based on the prediction signal 26 and the video signal 106 (e.g., by deriving a weighted sum of the prediction signals 26 and subtracting that weighted sum from the video signal 106). The residual encoding module 110 encodes the residual signal 24 into the bitstream 14.

[0068] The dashed arrow 26 represents the basic concept of the invention: the residual decoding module 210 and the residual encoding module 110 respectively use one or more prediction signals 26 (e.g., prediction blocks) to decode (e.g., determine the reconstructed residual signal 24'''') or encode the residual signal (e.g., prediction residual blocks).

[0069] Figures 5a and 5b illustrate another aspect of the invention. Here, the residual decoding module 210 or the residual encoding module 110 does not directly use the prediction signal 26 or the prediction block, but instead uses features 222 (i.e., quantized features) of the prediction signal 26 or the prediction block derived by the feature extraction module 220. In other words, the decoder 20 is configured to derive the prediction residual block of the current decoded block based on the prediction block of the current decoded block by: extracting one or more features 222 from the prediction block or from multiple prediction blocks (e.g., in the case of multiple prediction blocks, features can be extracted by comparing multiple prediction blocks, or one or more features can be extracted for each of the multiple prediction blocks); and deriving the prediction residual block of the current decoded block based on one or more features 222. The corresponding encoder 10 can be configured to encode the prediction residual block of the current coding block based on one prediction block (where prediction signal 26 may include or may represent a prediction block) or multiple prediction blocks (where prediction signal 26 may include multiple prediction blocks, or multiple prediction signals may represent multiple prediction blocks, or multiple prediction blocks may be predicted from multiple prediction signals) by: extracting one or more features 222 from the prediction block or from multiple prediction blocks; and encoding the prediction residual block of the current coding block based on one or more features 222, for example, the encoding performed by the residual coding module 110 may be adjusted based on one or more features 222.

[0070] In other words, Figures 4a and 4b illustrate residual decoding / encoding (see 210 and 110) and prediction (see 58 and 44), where residual decoding / encoding directly depends on prediction signal 26 (prediction block) (see dashed arrow 26); and Figures 5a and 5b illustrate residual decoding / encoding (see 210 and 110) and prediction (see 58 and 44), where residual decoding / encoding directly depends on feature 222 extracted from prediction signal 26 (prediction block). Apart from this difference, the decoder 20 shown in Figure 5a operates in the same manner as the decoder 20 shown in Figure 4a, and the encoder 10 shown in Figure 5b operates in the same manner as the encoder 10 shown in Figure 4b (as indicated by the corresponding reference numerals).

[0071] The prediction module (see 58 in Figures 4a and 5a and 44 in Figures 4b and 5b) can be configured to perform inter-frame image prediction and / or intra-frame image prediction.

[0072] The residual decoding module 210 and the residual encoding module 110 may include features and / or functions as described below with reference to Figures 7a to 10b.

[0073] The first aspect of the invention relates to the operation of the residual decoding module 210 and the residual encoding module 110. Specifically, different modifications of the residual decoding module 210 and the residual encoding module 110 employing the prediction signal 26 (e.g., the prediction block) or its feature 222 are given below.

[0074] Figure 6 An unmodified residual decoding module 210 is shown, which is similar to the residual decoding module employed by the most common decoder 20. The entropy decoder module 50 decodes the quantized residual signal coefficients from the bitstream 14. These coefficients are then dequantized 52 and (inverse) transformed by the transform module 54 to obtain the reconstructed residual signal 24'''', for example, using one or more transforms. The encoder 10 performs the opposite operation to the decoder 20, see, for example, see... Figure 1 The entropy encoder module 110 applies one or more transforms (e.g., using transform module 28) to the predicted residual signal 24 to obtain a spectral domain predicted residual signal 24', which is then quantized 32 and entropy encoded 34 into the bitstream 14. The present invention relates to advantageous modifications of such residual decoding module 210 and residual encoding module 110, which will be described in more detail below with reference to Figures 7a to 10b.

[0075] Figures 7a to 8b illustrate different embodiments of a residual decoding module 210 and a residual encoding module 110 with a modified transform module 54 / 28, which uses one or more prediction signals 26 (e.g., one or more prediction blocks) or features 222 extracted from them.

[0076] According to one aspect of the invention, referring to Figures 7a and 7b, the transform selector module 230 uses the prediction signal 26 (e.g., a prediction block) or features 222 derived therefrom to select one (or more) transforms from a plurality of predetermined (inverse) transforms 232, which are shown in Figures 7a and 7b as (inverse) transform 0, (inverse) transform 1, ..., (inverse) transform N. For example, the selected transform is then used at decoder 20 to derive the reconstructed residual signal 24'''' from the residual signal coefficients 24'''', and the selected transform is used at encoder 10 to derive the spectral domain prediction residual signal 24' from the prediction residual signal 24.

[0077] In other words, the block-based prediction decoder 20 can be configured to predict block 80 based on the current decoded block 18 in the following manner (e.g., see [reference]). Figure 3 (or multiple prediction blocks to derive the current decoded block (e.g., see [reference])) Figure 3 The predicted residual block of 18 in (e.g., see 18) Figure 386 in the figure): Based on prediction block 80 or one or more features 222 derived therefrom, or based on multiple prediction blocks or one or more features 222 derived therefrom, a predetermined transformation is selected from a set of transformations (see 232 in Figure 7a); the transformation representation is decoded from data stream 14 (see 24''' in Figure 7a); and the transformation representation 24''' is subjected to an inverse transformation (e.g., an inverse transformation) that reverses the predetermined transformation to obtain, for example, prediction residual block 86 (e.g., a block of reconstructed residual signal 24''''; or alternatively, the reconstructed residual signal 24'''' may represent prediction residual block 86). The corresponding block-based predictive encoder 10 can be configured to encode a predictive residual block of the current coding block (e.g., a block of residual signal 24; or alternatively, residual signal 24 may represent a predictive residual block) based on one or more predictive blocks of the current coding block by: selecting a predetermined transform from a transform set (see 232 in FIG. 7b) based on the predictive block or one or more features 222 derived therefrom, or based on multiple predictive blocks or one or more features 222 derived therefrom; performing the predetermined transform on the predictive residual block of the current coding block to obtain a transform representation (see 24' in FIG. 7b); and encoding the transform representation into a data stream 14, wherein encoding the transform representation may include quantizing the transform representation 32 (e.g., to obtain a quantized transform representation) and entropy encoding the quantized transform representation into the data stream 14.

[0078] For example, predetermined transform 232, or some of the transforms in predetermined transform 232, are obtained through machine learning methods. Their number may depend on the size or shape of the current block, or other information available at decoder 20 or encoder 10. In other words, transform 232 may include several machine learning-based transforms, wherein these machine learning-based transforms depend on the size or shape of the current decode / encode block.

[0079] In one embodiment of the invention, the transform selector 230 (see Figures 7a and 7b) may employ a neural network to select the transform; that is, the selection may be performed by a neural network. In other words, the decoder 20 / encoder 10 may be configured to feed prediction blocks to a neural network or a predetermined function (e.g., see...). Figure 3 The prediction block 80, the prediction signal 26, or one or more features 222 derived from the prediction block 80 or the prediction signal 26, select a predetermined transformation from the transformation set 232 (i.e., the transformation to be used by the transformation module 28 or the transformation to be reversed by the transformation module 54).

[0080] According to one embodiment of the invention, the transform selector 230 can select a transform based on different predefined ranges of feature values. For example, the decoder 20 / encoder 10 can be configured to select a predetermined transform from the transform set 232 by thresholding one or more features derived from the prediction block 80 or the prediction signal 26, or by comparing one or more features with different predefined values ​​or ranges of values.

[0081] Selecting a transformation from the transformation set 232 by thresholding one or more features derived from the prediction block 80 may include: for each of the one or more features, checking whether the feature value of the corresponding feature is higher or lower than a threshold; if the feature value is higher than the threshold, selecting the corresponding first transformation; and if the feature value is lower than the threshold, selecting the corresponding second transformation. The one or more features may include: the directionality of the prediction block 80 (where the feature value quantifies the directionality); and / or the spatial properties of the prediction block 80, such as the local variance within the prediction block 80 or the gradient magnitude of the prediction block 80; the similarity between the prediction block 80 and a predefined prediction block 80 (where the feature value quantifies the similarity); and / or the properties of the prediction block 80 in the frequency space (where the feature value quantifies the property).

[0082] Selecting a transformation from the transformation set 232 by comparing one or more features with different predefined values ​​or ranges of values ​​may include one or more of the following:

[0083] - Compare the positions of one or more edges in the predicted block 80 with the predefined positions of the edges within the block;

[0084] - Compare the positions of the (local) maximum and minimum values ​​in prediction block 80 with the predefined positions of the (local) maximum and minimum values ​​within the block;

[0085] - Compare prediction block 80 with predefined prediction block 80; and

[0086] - Compare the properties of the predicted block 80 in the frequency space with the properties of the predefined block in the frequency space.

[0087] In one embodiment of the invention, a scalar directional feature is derived from the predicted signal or prediction block 80, and this scalar directional feature is used to select a transform from a predetermined set of transforms, for example, obtained through machine learning. To derive this feature, the decoder 20 / encoder 10 may first compute the gradient of the predicted signal or prediction block. Subsequently, the decoder 20 / encoder 10 may determine a histogram of gradient angles, wherein the number of intervals in the histogram corresponds to the number of transforms in the predetermined set of transforms 232. Finally, the decoder 20 / encoder 10 determines the interval with the most entries in the histogram and selects the predetermined transform associated with that interval.

[0088] In another embodiment of the invention, a scalar directional feature is derived from the predicted signal or prediction block 80, and a transformation is selected from a predetermined set 232 of transformations, for example, obtained through machine learning, using this scalar directional feature. To derive this feature, the decoder 20 / encoder 10 first derives the gradient of the predicted signal or prediction block 80. Subsequently, the decoder 20 / encoder 10 creates a matrix based on the obtained gradient vector, which may be, for example, a covariance matrix. Then, the decoder 20 / encoder 10 derives the eigenvectors of this matrix and their directions. Finally, the decoder 20 / encoder 10 selects a predetermined transformation based on this direction.

[0089] As described above, Figures 7a and 7b illustrate residual decoding / encoding using a predetermined (inverse) transform selected based on the prediction signal 26 or its features 222 (or based on the prediction block or features derived therefrom).

[0090] According to another aspect of the invention, referring to Figures 8a and 8b, the transform derivation module 240 directly uses the prediction signal 26 or its features 222 (or the prediction block or features derived therefrom) to derive an arbitrary transform function (or transform) 242; the (inverse) transform module (see 54 in Figure 8a and 28 in Figure 8b) uses the arbitrary transform function 242 to derive the reconstructed residual signal 24'''' from the residual signal coefficients 24''' at the decoder 20, and to derive the spectral domain prediction residual signal 24' from the prediction residual signal 24 at the encoder 10. The transform derivation module 240 can be configured to derive one or more arbitrary transforms 242 based on the prediction signal 26 or the features 222 derived from the prediction signal 26 (or based on the prediction block or features derived therefrom).

[0091] In other words, the block-based prediction decoder 20 can be configured to predict block 80 based on the current decoded block 18 in the following manner (e.g., see [reference]). Figure 3 (80) or more prediction blocks to derive the current decoded block (e.g., see [reference]). Figure 3 The predicted residual block of 18 in (e.g., see 18) Figure 386 in the figure): Based on prediction block 80 (e.g., a block of prediction signal 26) or one or more features 222 derived therefrom, or based on multiple prediction blocks or one or more features 222 derived therefrom, determine the inverse transform (an example of transform function 242 in FIG. 8a); decode the transform representation from data stream 14 (see 24''' in FIG. 8a); and perform the inverse transform 242 on the transform representation 24''' to obtain, for example, prediction residual block 86 (e.g., a block of reconstructed residual signal 24''''; or alternatively, the reconstructed residual signal 24'''' may represent prediction residual block 86). The corresponding block-based predictive encoder 10 can be configured to encode a predictive residual block (e.g., a block of residual signal 24; or alternatively, residual signal 24 may represent a predictive residual block) of the current coding block based on one or more predictive blocks of the current coding block (e.g., a block of predictive signal 26) or multiple predictive blocks by: determining a transform 242 based on the predictive block or one or more features 222 derived therefrom, or based on multiple predictive blocks or one or more features 222 derived therefrom; applying the transform 242 to the predictive residual block of the current coding block to obtain a transform representation (see 24' in FIG. 8b); and encoding the transform representation 24' into the data stream 14, wherein the encoding of the transform representation may include quantizing the transform representation 32 (e.g., to obtain a quantized transform representation), and entropy encoding the quantized transform representation into the data stream 14 by entropy encoding 34.

[0092] In one embodiment of the invention, for example, the transformation function (or transformation) 242 can be determined by using the prediction signal 26 or its features 222 (or the prediction block or features derived therefrom) as input to a neural network (NN). That is, the transformation derivation module 240 can be configured to derive the transformation 242 using the neural network. In other words, the decoder 20 / encoder 10 can be configured to determine the transformation 242 based on the prediction block by feeding the prediction signal 26, the prediction block, one or more features 222 derived from the prediction block 80, or one or more features derived from the prediction signal 26 to the neural network or a predetermined function.

[0093] As described above, Figures 8a and 8b illustrate residual decoding / encoding using a (inverse) transform derived from the prediction signal 26 or its features 222 (or from the prediction block or features derived therefrom).

[0094] Figures 9a to 10b illustrate different embodiments of a residual decoding module 210 and a residual encoding module 110 having a residual modification module 250, which uses one or more prediction signals 26 (e.g., one or more prediction blocks) or features 222 extracted from them.

[0095] According to one aspect of the invention, referring to Figures 9a and 9b, the residual modification module 250 uses the prediction signal 26 or its feature 222 to modify the reconstructed residual signal 24'''' at the decoder 20 and thereby derive the modified reconstructed residual signal 24*'''' (e.g., the modified reconstructed residual signal 24*'''' constitutes the reconstructed residual signal when the reconstructed residual signal is combined with one or more prediction signals 26), and modifies the residual signal 24 at the encoder 10 to obtain the modified residual signal 24*. For example, the decoder 20 is configured to modify the residual signal (see reconstructed residual signal 24'''') based on the prediction signal 26 or its feature 222 after transform 54.

[0096] In other words, the block-based prediction decoder 20 can be configured to predict block 80 based on the current decoded block 18 in the following manner (e.g., see [reference]). Figure 3 (80) or more prediction blocks to derive the current decoded block (e.g., see 80) Figure 3 The predicted residual block of 18 in (e.g., see 18) Figure 3 86): Decode the transform representation 24''' from data stream 14; apply an inverse transform 54 (e.g., an inverse transform that reverses the transform applied by encoder 10) to the transform representation 24''' to obtain an inversely transformed block (e.g., a block of reconstructed residual signal 24''''; or alternatively, the reconstructed residual signal 24'''' may represent the inversely transformed block); and based on the inversely transformed block, form a prediction residual block of the current decoded block according to prediction block 80 or one or more features 222 derived therefrom, or according to multiple prediction blocks or one or more features 222 derived therefrom (e.g., see [link to relevant documentation]). Figure 386 in; for example, a block of modified reconstructed residual signal 24*'''' (e.g., by modifying the inverse-transformed block according to the prediction signal 26 or one or more features 222 derived therefrom to obtain the prediction residual block). The corresponding block-based predictive encoder 10 can be configured to encode a predictive residual block (e.g., a block of residual signal 24; or alternatively, residual signal 24 may represent a predictive residual block) of the current coding block based on one or more predictive blocks of the current coding block (where prediction signal 26 may include or may represent the predictive block) by: forming a predictive residual block of the current coding block (e.g., a block of modified residual signal 24*; or alternatively, modified residual signal 24* may represent a predictive residual block) based on the predictive block or one or more features 222 derived therefrom, or based on multiple predictive blocks or one or more features 222 derived therefrom); applying a transform 28 to the predictive residual block of the current coding block to obtain a transform representation (see 24' in FIG. 9b); and encoding the transform representation into a data stream 14, wherein encoding the transform representation may include quantizing the transform representation 32 (e.g., to obtain a quantized transform representation), and entropy encoding the quantized transform representation into the data stream 14 by entropy encoding 34.

[0097] In one embodiment of the invention, the modification (see 250) may include: at decoder 20, setting a portion of the reconstructed residual signal 24'''' (e.g., the reconstructed prediction residual block) to zero or a predetermined value; and at encoder 10, setting a portion of the residual signal 24 (e.g., the prediction residual block) to zero or a predetermined value. In another embodiment of the invention, the modification (see 250) may include: at decoder 20, pruning or shifting the reconstructed residual signal 24'''' (e.g., the reconstructed prediction residual block) relative to prediction signal 26 (e.g., the prediction block) based on prediction signal 26 or its feature 222 (or based on the prediction block or features derived therefrom); and at encoder 10, pruning or shifting the residual signal 24 (e.g., the prediction residual block) relative to prediction signal 26 (e.g., the prediction block) based on prediction signal 26 or its feature 222 (or based on the prediction block or features derived therefrom). In another embodiment of the invention, the modification (see 250) may include: at decoder 20, shifting the reconstructed residual signal 24'''' to a specific predetermined block partition of the prediction signal 26 based on the prediction signal 26 or its feature 222 (e.g., shifting to the location of a specific partition of the prediction signal, e.g., before addition, see...). Figure 2(See Figures 4a and 5a, 56); and at encoder 10, shifting the residual signal 24 to a specific predetermined block partition of the prediction signal 26 based on the prediction signal 26 or its feature 222. In another embodiment of the invention, the modification (see 250) may include: at decoder 20, reordering samples of the reconstructed residual signal 24'''' (e.g., the reconstructed prediction residual block) based on the prediction signal 26 or its feature 222 (or based on the prediction block or features derived therefrom); and at encoder 10, reordering samples of the residual signal 24 (e.g., the prediction residual block) based on the prediction signal 26 or its feature 222 (or based on the prediction block or features derived therefrom). In another embodiment of the invention, the modification may be applied by a neural network based on the prediction signal 26 or its feature 222 (or based on the prediction block or features derived therefrom) (see 250). In another embodiment of the invention, the modification may be applied by a filter based on the prediction signal 26 or its feature 222 (or the prediction block or features derived from the prediction block) (see 250).

[0098] In other words, the block-based prediction decoder 20 / encoder 10 can be configured to predict based on a prediction block (where the prediction signal 26 may include or may represent the prediction block) or one or more features derived therefrom, or based on multiple prediction blocks or one or more features 222 derived therefrom, [e.g., on the decoder side, based on the inversely transformed block (e.g., the block of the reconstructed residual signal 24''''; or alternatively, the reconstructed residual signal 24'''' may represent the inversely transformed block), and on the encoder side, based on the unmodified prediction residual block ( For example, a block of residual signal 24; or alternatively, residual signal 24 may represent an unmodified prediction residual block), the prediction residual block of the current decode / encode block is formed by one or more of the following (e.g., a modified prediction residual block; for example, for decoder 20, a block of modified reconstructed residual signal 24*'''' (or a block of modified reconstructed residual signal 24*'''' representing a prediction residual block); for example, for encoder 10, a block of modified residual signal 24* (or a block of modified residual signal 24* representing a prediction residual block)):

[0099] - Set one or more sample portions of the predicted residual block to a predetermined value (such as zero), wherein the position of one or more portions depends on the predicted block or one or more features derived therefrom, or depends on multiple predicted blocks or one or more features derived therefrom 222;

[0100] - Prune and / or shift the inversely transformed block (on the decoder side) or the unmodified prediction residual block (on the encoder side) in a manner that depends on the prediction block or one or more features derived therefrom, or depends on multiple prediction blocks or one or more features derived therefrom.

[0101] - Depending on the prediction block or one or more features derived therefrom, or depending on multiple prediction blocks or one or more features derived therefrom 222, (on the decoder side) the inversely transformed block is placed within the prediction residual block; or depending on the prediction block or one or more features derived therefrom, or depending on multiple prediction blocks or one or more features derived therefrom 222, (on the encoder side) the unmodified prediction residual block is placed within the prediction residual block.

[0102] - Samples of the inversely transformed block (on the decoder side) or the unmodified prediction residual block (on the encoder side) are placed in a relative spatial arrangement within the prediction residual block, the relative spatial arrangement depending on the prediction block or one or more features derived therefrom, or depending on multiple prediction blocks or one or more features derived therefrom 222.

[0103] - Filter the inversely transformed block (on the decoder side) or the unmodified prediction residual block (on the encoder side) by a filter parameterized according to the prediction block or one or more features derived therefrom, or according to a plurality of prediction blocks or one or more features derived therefrom.

[0104] - Make the inversely transformed block (on the decoder side) or the unmodified prediction residual block (on the encoder side) undergo a neural network that depends on the prediction block or one or more features derived from it, or depends on multiple prediction blocks or one or more features derived from them.

[0105] - Pass the inversely transformed block (on the decoder side) or the unmodified prediction residual block (on the encoder side) and the prediction block or one or more features derived from it (or multiple prediction blocks or one or more features derived from them 222) through the neural network.

[0106] As described above, Figures 9a and 9b illustrate residual decoding / encoding with modifications (see 250) to the reconstructed residual signal / residual signal (e.g., reconstructed prediction residual block / predictive residual block) based on the prediction signal 26 or its features 222 (or based on the prediction block or features derived therefrom).

[0107] According to another aspect of the invention, referring to FIG10a, the residual modification module 250 uses the prediction signal 26 or its feature 222 to modify the residual coefficient signal 24''', and obtains the modified residual signal coefficients 24*''', which are then (inversely) transformed at 54 to obtain the reconstructed residual signal 24''''. FIG10b illustrates this aspect with respect to the corresponding encoder 10, which is configured to transform the prediction residual signal 24 to obtain the spectral domain prediction residual signal 24', and modify the spectral domain prediction residual signal 24' to obtain the modified spectral domain prediction residual signal 24*' (e.g., using the residual modification module 250). For example, the decoder 20 is configured to modify the residual coefficient signal 24''' based on the prediction signal 26 or its feature 222 before transformation 54.

[0108] In other words, the block-based prediction decoder 20 can be configured to predict block 80 based on the current decoded block 18 in the following manner (e.g., see [reference]). Figure 3 The current decoded block is derived from one or more prediction blocks (e.g., see 80) or multiple prediction blocks, or based on one or more features derived from prediction blocks or multiple prediction blocks (e.g., see 80). Figure 3 The predicted residual block of 18 in (e.g., see 18) Figure 3 86): Decode the transform representation 24''' from the data stream 14; modify the transform representation 24''' based on the prediction block or one or more features derived therefrom (or based on multiple prediction blocks or one or more features derived therefrom 222) to obtain the modified transform representation 24*'''; and apply an inverse transform 54 (e.g., an inverse transform that reverses the transform applied by the encoder 10) to the modified transform representation 24*'''. The corresponding block-based predictive encoder 10 can be configured to encode the predictive residual block of the current coding block (e.g., a block of residual signal 24; or alternatively, residual signal 24 may represent the predictive residual block) based on the predictive blocks of the current coding block (where prediction signal 26 may include or may represent the predictive block) or multiple predictive blocks by: transforming the predictive residual block 28 to obtain a transform representation, such as the spectral domain predictive residual signal 24'; modifying the transform representation according to the predictive block or one or more features 222 derived therefrom (or according to multiple predictive blocks or one or more features 222 derived therefrom) to obtain a modified transform representation 24*'; and encoding the modified transform representation 24*' into the data stream 14, wherein the encoding of the modified transform representation 24*' may include quantizing the modified transform representation 24*' 32 (e.g., to obtain a quantized transform representation), and entropy encoding the quantized transform representation 34 into the data stream 14.

[0109] In one embodiment of the invention, the modification (see 250) may include setting a subset of the residual signal coefficients (e.g., for decoder 20, a subset of the residual signal coefficients of transform representation 24'''; for encoder 10, a subset of the residual signal coefficients of spectral domain predicted residual signal 24'') to zero or a predetermined value based on the prediction signal 26 or its feature 222 (or based on the prediction block or one or more features 222 derived therefrom). In another embodiment of the invention, the modification (see 250) may include applying a second transform to the residual signal coefficients (e.g., for decoder 20, the residual signal coefficients of transform representation 24'''; for encoder 10, the residual signal coefficients of spectral domain predicted residual signal 24''). In another embodiment of the invention, the modification (see 250) may be applied by a neural network based on the prediction signal 26 or its feature 222 (or based on the prediction block or one or more features 222 derived therefrom).

[0110] In other words, the block-based prediction decoder 20 / encoder 10 can be configured to modify the transform representation 24''' / 24' based on a prediction block (where the prediction signal 26 may include or may represent the prediction block) or one or more features 222 derived therefrom (or based on multiple prediction blocks or one or more features 222 derived therefrom):

[0111] - Set a subset of the coefficients of the transformation representation 24''' / 24' to a predetermined value (such as zero), wherein one or more coefficients or predetermined values ​​are determined based on a prediction block or one or more features 222 derived therefrom (or based on multiple prediction blocks or one or more features 222 derived therefrom);

[0112] - Apply a quadratic transformation to the transform representation 24''' / 24' based on the prediction block or one or more features 222 derived therefrom (or based on multiple prediction blocks or one or more features 222 derived therefrom);

[0113] - Apply a neural network to the transform representation 24''' / 24' based on the prediction block or one or more features 222 derived therefrom (or based on multiple prediction blocks or one or more features 222 derived therefrom).

[0114] As described above, Figures 10a and 10b illustrate residual decoding / encoding with modifications (see 250) to the residual signal coefficients (see 24''' / 24') based on the prediction signal 26 or its features 222 (or based on the prediction block or one or more features 222 derived therefrom).

[0115] Alternatively or in lieu of the above, a sub-partition of the block can be selected based on the prediction signal 26 or its feature 222 (or based on the prediction block or one or more features 222 derived therefrom), and the reconstructed residual signal 24'''' (e.g., the reconstructed prediction residual block) can be added to the selected sub-partition. In other words, a sub-partition of the block is selected based on the prediction signal 26 or its feature 222, and the reconstructed residual signal 24'''' is used to reconstruct the image / video signal of the selected sub-partition. In other words, the block-based prediction decoder 20 / encoder 10 can be configured to: select a sub-partition of the current decoding / encoding block based on the prediction block of the current decoding / encoding block or the feature 222 derived from the prediction block; and reconstruct the sub-partition of the current decoding / encoding block using the prediction residual block.

[0116] For example, the decoder 20 / encoder 10 described herein can perform partition selection based on the local differences of the predicted signal or the predicted block. In one embodiment of the invention, a sub-block (or sub-region or sub-partition) is selected from a set of sub-blocks (or sub-regions or sub-partitions) formed by partitioning the current block. To this end, the decoder 20 / encoder 10 can be configured to derive a feature value for each sub-block (or sub-region) of the current block, i.e., derive a feature value for each sub-partition. For a given current sub-block (or sub-region or sub-partition), this can be achieved by summing the squared differences of the two predicted signals 26 (e.g., two predicted blocks; e.g., obtained by B-prediction) within that current sub-block (or sub-region or sub-partition). The decoder 20 / encoder 10 can select the sub-block (or sub-region or sub-partition) with the highest feature value. Finally, the decoder 20 / encoder 10 can be configured to shift the decoded / encoded residual signal to the selected sub-block and / or use the residual signal to reconstruct the video signal within that block.

[0117] In one embodiment of the present invention, the sub-partition is a rectangular block.

[0118] The embodiments described herein (see Figures 4a to 10b) are based on the concept that the decoding / encoding of a residual signal (e.g., a prediction residual block) can be improved using one or more prediction signals 26 or features 222 derived therefrom (or one or more prediction blocks or one or more features 222 derived therefrom). This section discusses aspects of the operation of the feature extraction module 220 (see Figures 5a and 5b) in this invention. Specifically, this section provides different methods for deriving features 222 from prediction signals 26 or prediction blocks. Subsequently, features 222 can be used by the modified residual decoding module 210 and the modified residual encoding module 110 described with respect to Figures 7a to 10b.

[0119] One aspect of the invention is to derive features 222 that quantify the directionality of one or more prediction signals 26 or prediction blocks. In one embodiment of the invention, the directionality can be determined based on the gradient of the prediction signal 26 or prediction block. In another embodiment of the invention, the directionality can be determined based on a histogram of the gradient angles of the prediction signal 26 or prediction block. In yet another embodiment of the invention, the directionality can be determined based on the eigenvalues ​​and / or eigenvectors of the matrix derived from the gradient.

[0120] Another aspect of the invention lies in deriving features 222 for quantifying the spatial characteristics of one or more prediction signals 26 or one or more prediction blocks. In one embodiment of the invention, the spatial characteristic may be the local variance within the prediction signal 26 or prediction block. In another embodiment of the invention, the spatial characteristic may be the gradient magnitude of the prediction signal 26 or prediction block. In another embodiment of the invention, the spatial characteristic may be the edge location in the prediction signal 26 or prediction block. In another embodiment of the invention, the spatial characteristic may be the location of (local) maximum and minimum values ​​in the prediction signal 26 or prediction block.

[0121] Another aspect of the invention is to derive features 222 for quantifying the similarity between one or more prediction signals 26 or one or more prediction blocks and a specific predefined signal.

[0122] Another aspect of the invention is to derive features 222 for quantizing the characteristics of the prediction signal 26 or the prediction block in the frequency space.

[0123] Another aspect of the invention is to use a neural network to derive features 222 from one or more prediction signals 26 (e.g., from one or more prediction blocks), for example, by feeding prediction blocks (predicted blocks), multiple prediction blocks, or one or more prediction signals 26 into the neural network to derive features 222.

[0124] Another aspect of the invention is to compare the prediction signals 26 or prediction blocks with each other, thereby deriving feature 222.

[0125] In one embodiment of the invention, the squared difference of the prediction signal 26 or the prediction block is derived as feature 222.

[0126] Another aspect of the invention is to combine the prediction signal 26 or the prediction block and derive its feature 222.

[0127] Another aspect of the invention is to derive features 222 for the prediction signal 26 or the prediction block respectively, and then combine the features 222.

[0128] Another aspect of the invention yields a feature map of local features 222 including the prediction signal 26 (e.g., the prediction block). The feature map can define a mapping of its values ​​to positions within the prediction block (the predicted block) or the prediction signal 26.

[0129] In another aspect of the invention, for different rectangular sub-blocks of a block or different sub-regions of a region, the prediction signal 26 or the feature value of the predicted block (e.g., also understood herein as the prediction block) is derived respectively.

[0130] Another aspect of the invention yields a feature vector comprising different features 222 of the prediction signal 26 or the prediction block.

[0131] In summary, one or more features 222 include one or more of the following:

[0132] - A directional metric for a prediction block or two or more prediction blocks, for example, the corresponding directional metric for each of two or more prediction blocks;

[0133] - Data derived from the gradients of a prediction block or two or more prediction blocks, for example, data derived from the gradients of the respective prediction block for each of two or more prediction blocks;

[0134] - Data derived from histograms of gradient angles of a prediction block or two or more prediction blocks (e.g., information associated with the interval with the most entries), such as data derived from histograms of gradient angles of the respective prediction block for each of two or more prediction blocks.

[0135] - Data derived from the eigenvalues ​​and / or eigenvectors of a matrix (e.g., covariance matrix) derived from the gradient angle of a prediction block or two or more prediction blocks, for example, for each of two or more prediction blocks, data derived from the eigenvalues ​​and / or eigenvectors of a matrix (e.g., covariance matrix) derived from the gradient angle of the corresponding prediction block.

[0136] - Spatial measure of a prediction block or two or more prediction blocks, for example, the corresponding spatial measure for each of two or more prediction blocks;

[0137] - Local variance plots of a prediction block or two or more prediction blocks, for example, the local variance plot of the corresponding prediction block for each of two or more prediction blocks;

[0138] - The gradient magnitude of a prediction block or two or more prediction blocks, for example, the gradient magnitude of the corresponding prediction block for each of two or more prediction blocks;

[0139] - The positions of the maximum and minimum values ​​in a prediction block or two or more prediction blocks, for example, the positions of the maximum and minimum values ​​in the respective prediction block for each of two or more prediction blocks;

[0140] - Edge locations in a prediction block or two or more prediction blocks, for example, for each of two or more prediction blocks, the edge locations in the corresponding prediction blocks;

[0141] - The similarity between a prediction block or one or more prediction blocks and one or more predefined block templates. For example, for each of two or more prediction blocks, the similarity between the corresponding prediction block and one or more predefined block templates.

[0142] According to one embodiment, one or more features 222 may (additionally or alternatively) include one or more of the following:

[0143] - A similarity measure between two or more predicted blocks (or between two or more predicted blocks);

[0144] - A metric derived by comparing two or more prediction blocks, such as the squared difference between two or more prediction blocks;

[0145] - A metric derived from a combination of two or more prediction blocks;

[0146] - A metric obtained by first deriving individual metrics for each prediction block in two or more prediction blocks, and then combining the individual metrics (the individual metrics may include one or more of the features mentioned above).

[0147] When the prediction block is predicted from two or more prediction signals 26, one or more features 222 may include one or more of the following:

[0148] - A similarity measure between (or between) the predicted signals 26;

[0149] - A metric derived by comparing the predicted signal 26, for example, the squared difference of the predicted signal 26;

[0150] - A metric derived from the combination of predicted signals 26;

[0151] - A measure obtained by first deriving individual measures for the prediction signal 26 separately, and then combining the individual measures (the individual measures may include one or more of the features mentioned above).

[0152] According to one embodiment, feature 222 is derived in the frequency space.

[0153] According to one embodiment, the predicted block (e.g., the block to be predicted) is derived from reconstructed samples of the spatial neighborhood of the predicted block. For example, the predicted signal 26 or the predicted block is derived from reconstructed samples of the spatial neighborhood of the current region (e.g., the current block or the current decode / encode block).

[0154] As described herein, the residual signal (e.g., the predicted residual block) and the prediction signal 26 (or the prediction block) can be associated with the current region (e.g., the current block or the current decoded / encoded block) in the (reconstructed) image.

[0155] Although some aspects have been described in the context of the device, it is apparent that these aspects also represent a description of the corresponding method, where blocks or means correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of corresponding blocks, items, or features of the corresponding device. Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device. Similarly, the device may include hardware devices, such as microprocessors, programmable computers, or electronic circuits, configured to perform one or more method steps.

[0156] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. It can be implemented using a digital storage medium on which electronically readable control signals are stored, such as a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM, or flash memory, the electronically readable control signals cooperating with (or capable of cooperating with) a programmable computer system to enable the execution of the corresponding methods. Therefore, the digital storage medium can be computer-readable.

[0157] The encoded image or video signals of the present invention can be stored on a digital storage medium or transmitted on a transmission medium (such as a wireless transmission medium or a wired transmission medium, such as the Internet).

[0158] Some embodiments of the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0159] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, performs one of the methods. For example, the program code may be stored on a machine-readable medium.

[0160] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0161] In other words, therefore, one embodiment of the method of the present invention is a computer program having program code that, when the computer program is run on a computer, performs one of the methods described herein.

[0162] Therefore, another embodiment of the method of the present invention is a data carrier (or digital storage medium or computer-readable medium) on which a computer program for performing one of the methods described herein is recorded. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0163] Therefore, another embodiment of the method of the present invention represents a data stream or signal sequence for performing one of the methods described herein. For example, the data stream or signal sequence may be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0164] Another embodiment includes a processing means, such as a computer or programmable logic device configured or adapted to perform one of the methods described herein.

[0165] Another embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0166] Another embodiment of the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0167] In some embodiments, programmable logic devices (e.g., field-programmable gate arrays) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0168] The device described herein can be implemented using hardware devices, computers, or a combination of hardware devices and computers.

[0169] The device described herein, or any component thereof, may be implemented, at least in part, in hardware and / or software.

[0170] The methods described herein can be performed using hardware devices, computers, or a combination of hardware devices and computers.

[0171] The methods described herein or any component of the device described herein may be performed at least in part by hardware and / or software.

[0172] The above embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. Therefore, the invention is intended to be limited only by the scope of the appended claims, and not by the specific details given herein through the description and explanation of the embodiments.

Claims

1. A video decoder (20) configured to: derive one or more prediction signals (26); reconstruct a residual signal (24) from a bitstream (14); and combine the reconstructed residual signal (24'''', 24*'''') with the one or more prediction signals (26), wherein, The reconstruction of the residual signal (24) depends on the one or more prediction signals (26).

2. According to claim 1, wherein, The decoder (20) applies inter-frame image prediction and / or intra-frame image prediction.

3. According to claim 1 or 2, wherein, When reconstructing the residual signal (24), the decoder (20) applies one or more transformations to the residual coefficient signal.

4. According to claim 3, wherein, The decoder (20) selects one or more transforms (232) from a predetermined transform set (232) based on the predicted signal (26) or features (222) derived from the predicted signal (26).

5. According to claim 4, wherein, The predetermined transformation set (232) is obtained through machine learning methods.

6. According to any one of claims 4 to 5, wherein, The selection is performed by a neural network.

7. According to any one of claims 4 to 6, wherein, The selection is performed by comparing the feature value with different predefined values ​​or different ranges of predefined values.

8. According to claim 3, wherein, The decoder (20) derives one or more arbitrary transformations (242) based on the predicted signal (26) or features (222) derived from the predicted signal (26).

9. According to claim 8, wherein, The decoder (20) uses a neural network to derive the transformation (242).

10. According to claim 3, wherein, The decoder (20) modifies the residual signal after the transformation based on the prediction signal (26) or the characteristics (222) of the prediction signal (26).

11. According to claim 10, wherein, The modification includes setting a portion of the residual signal to zero or another predetermined value.

12. According to any one of claims 10 to 11, wherein, The modification includes shifting the residual signal to the location of a specific partition of the predicted signal (26) before addition.

13. According to any one of claims 10 to 12, wherein, The modification includes reordering the samples of the residual signal.

14. According to any one of claims 10 to 13, wherein, The modification includes cropping the residual signal.

15. According to any one of claims 10 to 13, wherein, The modification includes applying a filter to the residual signal.

16. According to claim 3, wherein, The decoder (20) modifies the residual coefficient signal based on the prediction signal (26) or the characteristics (222) of the prediction signal (26) before the transformation.

17. According to claim 16, wherein, The modification includes setting a portion of the residual coefficient signal to zero or another predetermined value.

18. According to any one of claims 16 to 17, wherein, The modification includes applying a second transformation to the residual coefficient signal.

19. According to any one of claims 16 to 18, wherein, The modifications include the application of neural networks.

20. According to claim 3, wherein, Sub-partitions of the block are selected based on the predicted signal or its characteristics (222), and the video signal of the selected sub-partition is reconstructed using the reconstructed residual signals (24'''', 24*'''').

21. According to claim 20, wherein, For each sub-partition, a feature value is derived.

22. According to any one of claims 20 to 21, wherein, Select the sub-partition with the highest or lowest feature value.

23. According to any one of claims 20 to 22, wherein, The sub-partition is a rectangular block.

24. According to any one of claims 4 to 23, wherein, The features (222) derived from the predicted signal quantify the directionality of the predicted signal (26).

25. According to claim 24, wherein, The feature (222) is derived from the gradient of the predicted signal.

26. According to claim 24, wherein, The feature (222) is derived from the histogram of the gradient angle of the predicted signal (26).

27. According to claim 24, wherein, The features are derived from the eigenvalues ​​and / or eigenvectors of the matrix derived from the gradient (222).

28. According to any one of claims 4 to 23, wherein, The spatial features (222) of the predicted signal (26) are quantified from the features (222) derived from the predicted signal (26).

29. According to claim 28, wherein, The spatial feature (222) represents the local variance in the predicted signal (26).

30. According to claim 28, wherein, The spatial feature (222) represents the edge position within the predicted signal (26).

31. According to claim 28, wherein, The spatial feature (222) represents the gradient magnitude of the predicted signal.

32. According to claim 28, wherein, The spatial feature (222) represents the location of the (local) maximum and minimum values ​​in the predicted signal.

33. According to any one of claims 4 to 23, wherein, The predicted signals (26) are compared with each other to derive the feature (222).

34. According to claim 33, wherein, The comparison yields the squared difference of the predicted signal (26).

35. According to any one of claims 4 to 23, wherein, The predicted signals (26) are combined to derive the feature (222).

36. According to any one of claims 24 to 32, wherein, For each prediction, a feature (222) is derived, and then the features (222) are combined.

37. According to any one of claims 4 to 23, wherein, The feature (222) quantifies the similarity between one or more predicted signals (26) and a specific predefined signal.

38. According to any one of claims 4 to 37, wherein, The feature (222) quantifies the characteristics of the predicted signal (26) in the frequency space.

39. According to any one of claims 4 to 38, wherein, The feature (222) is derived from a neural network.

40. According to any one of claims 4 to 39, wherein, The feature (222) is a feature map that defines the mapping of its value to the position in the predicted signal.

41. According to any one of claims 4 to 39, wherein, For different rectangular sub-blocks of a block or for different sub-regions of a region, the feature values ​​of the predicted signal are obtained respectively.

42. According to any one of claims 4 to 41, wherein, The feature (222) is a vector that includes different features (222) of the predicted signal.

43. According to any one of claims 1 to 42, wherein, The residual signal and the prediction signal (26) are related to the current region in the reconstructed image.

44. According to claim 43, wherein, The predicted signal is derived from the reconstructed samples of the spatial neighboring regions of the current region.

45. A block-based predictive decoder (20) for decoding an image (12') from a data stream (14), configured to: Based on the prediction block (80) of the current decoded block (18), the prediction residual block (86) of the current decoded block (18) is derived; and The current decoding block (18) is reconstructed based on the predicted residual block (86) and the predicted block (80).

46. ​​The block-based prediction decoder (20) according to claim 45 is configured to derive a prediction residual block (86) of the current decoded block (18) based on the prediction block (80) of the current decoded block (18): Extract one or more features (222) from the prediction block (80); and Based on one or more of the features (222), the predicted residual block (86) of the current decoded block (18) is derived.

47. The block-based prediction decoder (20) according to claim 45 or 46 is configured to derive a prediction residual block (86) of the current decoded block (18) based on the prediction block (80) of the current decoded block (18): Based on the predicted signal (26) or one or more features (222) derived therefrom, a predetermined transformation is selected from the transformation set (232); Decode the transformed representation (24''') from the data stream (14); and Apply an inverse transformation (54) to the transformation representation (24''') that reverses the predetermined transformation.

48. The block-based prediction decoder (20) according to claim 47, wherein, The transformation includes multiple machine learning-based transformations.

49. The block-based prediction decoder (20) according to claim 48, wherein, The multiple machine learning transformations depend on the size or shape of the current decoding block (18).

50. The block-based prediction decoder (20) according to any one of claims 47 to 49 is configured to select the predetermined transformation from the set of transformations (232) by feeding the prediction block (80) or one or more features (222) derived from the prediction block (80) to a neural network or a predetermined function.

51. The block-based prediction decoder (20) according to any one of claims 47 to 50 is configured to select the predetermined transformation from the set of transformations (232) by thresholding one or more features (222) derived from the prediction block (80) or by comparing the one or more features (222) with different predefined values ​​or value ranges.

52. The block-based prediction decoder (20) according to any one of claims 45 to 51 is configured to derive a prediction residual block (86) of the current decoded block (18) based on the prediction block (80) of the current decoded block (18): The inverse transformation (242) is determined based on the prediction block (80) or one or more features (222) derived therefrom. Decode the transformed representation (24''') from the data stream (14); and The inverse transformation (242) is applied to the transformation representation (24''').

53. The block-based predictive decoder (20) according to claim 52 is configured to determine the inverse transform (242) based on the predictive block (80) by feeding a predictive signal to a neural network or a predetermined function, one or more features (222) derived from the predictive block (80), or one or more features (222) derived therefrom.

54. The block-based prediction decoder (20) according to any one of claims 45 to 53 is configured to derive a prediction residual block (86) of the current decoded block (18) based on the prediction block (80) of the current decoded block (18): Decode the transformed representation (24''') from the data stream (14); Apply an inverse transformation (54) to the transformed representation (24''') to obtain the inversely transformed block; and Based on the inversely transformed block, a prediction residual block (86) of the current decoded block (18) is formed according to the prediction signal (26) or one or more features (222) derived therefrom.

55. The block-based predictive decoder (20) according to claim 54 is configured to form a predictive residual block (86) of the current decoded block (18) based on the predictive block (80) by one or more of the following: One or more sample portions of the predicted residual block (86) are set to predetermined values, such as zero, where, The position of one or more parts depends on the prediction block (80) or one or more features (222) derived therefrom. The inversely transformed block is trimmed and / or shifted in a manner that depends on the predicted block (80) or one or more features (222) derived therefrom; Depending on the prediction block (80) or one or more features (222) derived therefrom, the inversely transformed block is placed within the prediction residual block (86); The samples of the inversely transformed block are placed within the prediction residual block (86) in a relative spatial arrangement that depends on the prediction block (80) or one or more features (222) derived therefrom; The inversely transformed block is filtered by a filter parameterized according to the prediction block (80) or one or more features (222) derived therefrom; The block that has undergone the inverse transformation is subjected to a neural network that depends on the predicted block (80) or one or more features (222) derived therefrom; The inversely transformed block and the predicted block (80) or one or more features (222) derived therefrom are subjected to a neural network.

56. The block-based prediction decoder (20) according to any one of claims 45 to 55 is configured to derive a prediction residual block (86) of the current decoded block (18) based on the prediction block (80) of the current decoded block (18): Decode the transformed representation (24''') from the data stream (14); The transform representation (24''') is modified based on the prediction block (80) or one or more features (222) derived therefrom to obtain the modified transform representation; and Apply an inverse transformation (54) to the modified transformation representation.

57. The block-based prediction decoder (20) according to claim 56 is configured to modify the transform representation based on the prediction block (80) by one or more of the following: Set a subset of the coefficients of the transform representation (24''') to a predetermined value, such as zero, where, The one or more coefficients or predetermined values ​​are determined based on the prediction block (80); A quadratic transformation is applied to the transform representation (24''') based on the prediction block (80); A neural network is applied to the transformation representation (24''') based on the prediction block (80).

58. The block-based prediction decoder (20) according to any one of claims 45 to 57 is configured to: use the prediction residual block (86) to select a sub-partition of the current decoding block (18) based on the prediction block (80) of the current decoding block (18) or based on one or more features (222) derived from the prediction block (80) to reconstruct the sub-partition of the current decoding block (18).

59. The block-based predictive decoder (20) according to claim 58 is configured to derive feature values ​​for each sub-partition of the current decoded block (18).

60. The block-based prediction decoder (20) of claim 59 is configured to select a sub-partition with the highest or lowest feature value.

61. The block-based prediction decoder (20) according to claim 60, wherein, The sub-partition is a rectangular block.

62. The block-based prediction decoder (20) according to any one of claims 45 to 61, wherein, The one or more features (222) include one or more of the following: The directional metric of the prediction block (80); Data derived from the gradient of the prediction block (80); Data derived from the histogram of the gradient angles of the prediction block (80); Data derived from the eigenvalues ​​and / or eigenvectors of a matrix, such as the covariance matrix, obtained from the gradient angle of the prediction block (80); The spatial metric of the prediction block (80); Local variance plot of the prediction block (80); The gradient magnitude of the prediction block (80); The positions of the maximum and minimum values ​​in the prediction block (80); The edge positions in the prediction block (80); The similarity between the prediction block (80) and one or more predefined block templates.

63. The block-based prediction decoder (20) according to any one of claims 45 to 62, wherein, The prediction block (80) is predicted from two or more prediction signals (26), and the one or more features (222) include one or more of the following: A similarity measure between predicted signals (26) (or between predicted signals (26)); A metric derived by comparing the predicted signals (26), for example, the squared difference of the predicted signals (26); The measure derived from the combination of the predicted signals (26); The measure is obtained by first deriving individual measures for the predicted signal (26) and then combining the individual measures.

64. The block-based prediction decoder (20) according to any one of claims 45 to 62 is configured as follows: For the current decoded block (18), a plurality of prediction blocks (80) including the prediction block (80) are derived. Based on the prediction block (80), the prediction residual block (86) of the current decoding block (18) is derived; and The current decoding block (18) is reconstructed based on the predicted residual block (86) and the plurality of predicted blocks.

65. The block-based prediction decoder (20) according to any one of claims 45 to 64, wherein, The aforementioned feature (222) is derived in the frequency space.

66. The block-based prediction decoder (20) according to any one of claims 45 to 65, wherein, The features are obtained by feeding the prediction blocks into a neural network (222).

67. The block-based prediction decoder (20) according to any one of claims 45 to 66, wherein, The feature (222) is a feature map that defines the mapping of its value to the position in the prediction block.

68. The block-based prediction decoder (20) according to any one of claims 45 to 67, wherein, For different rectangular sub-blocks or different sub-regions, the feature values ​​of the predicted block are obtained respectively.

69. The block-based prediction decoder (20) according to any one of claims 45 to 68, wherein, The feature (222) is a vector of different features (222) including the prediction block (80).

70. The block-based prediction decoder (20) according to any one of claims 45 to 69, wherein, The prediction block is derived from the reconstruction samples of the spatial neighboring regions of the prediction block (80).

71. A block-based predictive encoder for encoding an image into a data stream (14), configured to: derive a predictive residual block of the current coding block based on a predictive block of the current coding block, such that the current coding block can be reconstructed based on the predictive residual block and the predictive block.

72. The block-based predictive encoder of claim 71, configured to derive a predictive residual block of the current coding block based on the predictive block of the current coding block by: Extract one or more features from the prediction block (222); and The predicted residual block of the current coding block is derived based on one or more of the features (222).

73. The block-based predictive encoder of claim 71 or 72, configured to derive a predictive residual block of the current coding block based on the predictive block of the current coding block by: Based on the predicted signal (26) or one or more features (222) derived therefrom, a predetermined transformation is selected from the transformation set (232); The predetermined transformation is applied to the prediction residual block of the current coded block to obtain a transformed representation; and The transformation representation is encoded into the data stream (14).

74. The block-based predictive encoder of claim 73, wherein, The transformation (232) includes multiple machine learning-based transformations.

75. The block-based predictive encoder of claim 74, wherein, The multiple machine learning transformations depend on the size or shape of the current encoded block.

76. The block-based predictive encoder according to any one of claims 73 to 75, configured to select the predetermined transform from the transform set (232) by feeding the predictive block or one or more features derived from the predictive block to a neural network or a predetermined function (222).

77. The block-based predictive encoder according to any one of claims 73 to 76, configured to select the predetermined transform from the transform set (232) by thresholding one or more features (222) derived from the predictive block or by comparing the one or more features (222) with different predefined values ​​or value ranges.

78. A block-based predictive encoder according to any one of claims 71 to 77, configured to derive a predictive residual block of the current coding block based on a predictive block of the current coding block by: The transformation (242) is determined based on the prediction block or one or more features derived therefrom (222); The transform (242) is applied to the prediction residual block of the current coded block to obtain the transform representation; and The transformation representation is encoded into the data stream (14).

79. The block-based predictive encoder of claim 78, configured to determine the transformation (242) based on the prediction block by feeding a prediction signal to a neural network or a predetermined function, one or more features (222) derived from the prediction block, or one or more features (222) derived therefrom.

80. A block-based predictive encoder according to any one of claims 71 to 79 is configured to derive a predictive residual block of the current coding block based on a predictive block of the current coding block by: Based on the prediction signal or one or more features derived therefrom (222), a prediction residual block of the current coded block is formed; A transform (28) is applied to the prediction residual block of the current coded block to obtain a transform representation (24'); and The transform representation (24') is encoded into the data stream (14).

81. The block-based predictive encoder of claim 80, configured to form a predictive residual block of the current coding block based on the predictive block by one or more of the following: One or more sample portions of the predicted residual block are set to predetermined values, such as zero, where, The location of one or more parts depends on the prediction block or one or more features derived therefrom (222); The prediction residual block is trimmed and / or shifted in a manner that depends on the prediction block or one or more features derived therefrom (222); The samples of the prediction residual block are placed within the prediction residual block in a relative spatial arrangement that depends on the prediction block or one or more features (222) derived therefrom; The prediction residual block is filtered by a filter that is parameterized based on the prediction block or one or more features (222) derived therefrom; The predicted residual block is subjected to a neural network that depends on the predicted block or one or more features (222) derived therefrom; The predicted residual block and one or more features (222) derived therefrom are subjected to a neural network.

82. The block-based predictive encoder according to any one of claims 71 to 81 is configured to derive a predictive residual block of the current coding block based on the predictive block of the current coding block by: A transformation (28) is applied to the predicted residual block to obtain a transformed representation (24'); The transform representation (24') is modified based on the prediction block or one or more features (222) derived therefrom to obtain the modified transform representation (24*'); and The modified transform representation (24*') is encoded into the data stream (14).

83. The block-based predictive encoder of claim 82, configured to modify the transform representation (24') based on the predictive block by one or more of the following: Set a subset of the coefficients of the transformation representation (24') to a predetermined value, such as zero, where, One or more coefficients or predetermined values ​​are determined based on the prediction block; A quadratic transformation is applied to the transformation representation (24') based on the prediction block; A neural network is applied to the transformation representation (24') based on the prediction block.

84. The block-based predictive encoder according to any one of claims 71 to 83 is configured to: use the predictive residual block, based on the predictive block of the current coding block or based on one or more features (222) derived from the predictive block, select a sub-partition of the current coding block to reconstruct the sub-partition of the current coding block.

85. The block-based predictive encoder of claim 84, configured to derive feature values ​​for each sub-partition of the current coding block.

86. The block-based predictive encoder of claim 85 is configured to select a sub-partition with the highest or lowest feature value.

87. The block-based predictive encoder of claim 86, wherein, The sub-partition is a rectangular block.

88. The block-based predictive encoder according to any one of claims 71 to 87, wherein, The one or more features (222) include one or more of the following: The directional metric of the prediction block; Data derived from the gradient of the prediction block; Data derived from the histogram of the gradient angles of the predicted block; Data derived from the eigenvalues ​​and / or eigenvectors of a matrix, such as the covariance matrix, derived from the gradient angle of the predicted block; The spatial metric of the prediction block; The local variance plot of the prediction block; The gradient magnitude of the prediction block; The positions of the maximum and minimum values ​​in the prediction block; The edge locations in the predicted block; The similarity between the predicted block and one or more predefined block templates.

89. The block-based predictive encoder according to any one of claims 71 to 88, wherein, The prediction block is predicted from two or more prediction signals (26), and the one or more features (222) include one or more of the following: A similarity measure between predicted signals (26) (or between predicted signals (26)); A metric derived by comparing the predicted signals (26), for example, the squared difference of the predicted signals (26); The measure derived from the combination of the predicted signals (26); The measure is obtained by first deriving individual measures for the predicted signal (26) and then combining the individual measures.

90. The block-based predictive encoder according to any one of claims 71 to 88 is configured to: For the current coded block (18), a plurality of prediction blocks (80) including the prediction block (80) are derived; and Based on the prediction block (80), the prediction residual block (86) of the current coding block (18) is derived. in, The current coding block (18) can be reconstructed based on the predicted residual block (86) and the plurality of predicted blocks.

91. The block-based predictive encoder according to any one of claims 71 to 90, wherein, The aforementioned feature (222) is derived in the frequency space.

92. The block-based predictive encoder according to any one of claims 71 to 91, wherein, The features are obtained by feeding the prediction blocks into a neural network (222).

93. The block-based predictive encoder according to any one of claims 71 to 92, wherein, The feature (222) is a feature map that defines the mapping of its value to the position in the prediction block.

94. The block-based predictive encoder according to any one of claims 71 to 93, wherein, For different rectangular sub-blocks or different sub-regions, the feature values ​​of the predicted block are obtained respectively.

95. The block-based predictive encoder according to any one of claims 71 to 94, wherein, The feature (222) is a vector that includes different features (222) of the prediction block.

96. The block-based predictive encoder according to any one of claims 71 to 95, wherein, The predicted block is derived from the reconstructed samples of the spatial neighboring regions of the predicted block.

97. A method performed by the apparatus of any one of claims 45 to 70.

98. A method for decoding an image from a data stream (14), comprising: Based on the prediction block of the current decoded block, the prediction residual block of the current decoded block is derived; and The current decoding block is reconstructed based on the predicted residual block and the predicted block.

99. A method performed by the apparatus according to any one of claims 71 to 96.

100. A method for encoding an image into a data stream (14), comprising: Based on the prediction block of the current coding block, a prediction residual block of the current coding block is obtained, so that the current coding block can be reconstructed based on the prediction residual block and the prediction block.

101. A data stream (14) generated by the method according to claim 98 or by a block-based predictive encoder according to any one of claims 71 to 96.

102. A computer program, when executed on a computer or signal processor, for implementing the method according to any one of claims 97 to 100.

Images