Video decoder and encoder using special neighborhood signals, video decoder and encoder applying post-processing only to specific inter-predicted blocks, picture processing tool and method - Patent Application 20070122997

Abstract

a video decoder and encoder that uses neighboring signals generated by using non-post-processed versions of the contributing signals and / or by replacing the contributing signals with replacement signals generated independently of spatial signal interdependencies; a picture processing tool configured to polyphase-decompose luma samples and subject a tensor of cascade matrices of polyphase components to a neural network or convolution; and a video decoder and encoder that apply post-processing only to specific inter-predicted blocks.

Description

[Technical Field]

[0001] Embodiments relate to video decoders and encoders that use special neighborhood signals, video decoders and encoders that apply post-processing only to specific inter-predicted blocks, picture processing tools and methods.

[0002] Introduction Video sequences generally have a high degree of both spatial and temporal redundancy. All relevant approaches to video signal compression (i.e., efficient representation) are based on exploiting these redundancies. Temporal redundancy is exploited by motion-compensated (or inter) prediction, a core component of all video coding standards. In the evolution of these standards, from H.261 [1], first ratified in November 1988, to Versatile Video Coding (VVC) [2], [3], inter prediction has been enhanced in many ways. Typically, these enhancements aimed to improve the motion-compensated prediction signal and thus the overall coding performance. For example, it is a well-established finding that overlapping two individual prediction signals can reduce the resulting prediction error variance [4]. Therefore, simple averaging of two predictors has been used since the introduction of the MPEG-1 standard [5] in 1991. The H.264 / AVC video coding standard [6] introduced so-called weighted prediction, where weighting coefficients can be transmitted at the slice level for each reference picture.

[0003] The current state-of-the-art standard, VVC, offers several further extensions to inter prediction. Simple averaging in bi-prediction can be replaced by CU-weighted bi-prediction (BCW) [7]. In the case of block-based bi-prediction, there is a sample-wise refinement called bidirectional optical flow (BDOF) [8], [9]. Furthermore, there is subblock-based inter prediction, where individual motion vectors are derived for each subblock. This includes subblock-based temporal motion vector prediction (SbTMVP) [9], decoder-side motion vector refinement (DMVR) [9], and affine motion compensation (AMC) [9]. In the latter case, there is also a sample-wise refinement called prediction refinement by optical flow (PROF) [9]. Furthermore, geometric partitioning mode (GPM) [7] adds support for non-rectangular partitions. To jointly exploit temporal and spatial redundancies, VVC introduces joint inter / intra prediction (CIIP) [7], which additionally uses neighboring samples from neighboring blocks.

[0004] During the development of VVC, another method for incorporating spatially neighboring samples into a temporally predicted block has been studied in detail. This method, known as Local Illumination Compensation (LIC)

[10] ,

[11] , is conceptually based on the Illumination Compensation (IC) coding tool in the 3D extension of the High Efficiency Video Coding (HEVC) standard

[12] ,

[13] . In LIC, scale and offset values ​​are derived at the decoder to adjust the luminance of an inter-predicted block to that of its reconstructed samples above and to the left. However, due to its impact on decoding complexity, LIC is not part of VVC.

[0005] Based on previous work

[14] , a spatiotemporal residual network (STRN) is proposed here. The main idea of ​​STRN is to refine the inter-prediction signal without additional signaling by using a convolutional neural network (CNN) that incorporates information from spatially neighboring blocks. Corresponding sample data are stitched together to form the input tensor of the CNN. The output tensor contains the refined prediction signal. STRN is integrated into the VVC Test Model (VTM), the reference software for the VVC standard.

[0006] The main contributions of this study are as follows: Polyphase decomposition is applied to picture or video signal representations, which has been shown to improve the trade-off between computational complexity and coding performance.

[0007] The CNN is moved out of the intra-decoding loop. This allows the decoder to apply the CNN in parallel to all blocks in a picture, regardless of the intra-predicted blocks. Otherwise, i.e., with the CNN in the intra-decoding loop, this is conceptually impossible and therefore forces sequential processing in the decoder, which is practically difficult.

[0008] CNNs are studied in detail within the context of low-latency prediction structures. It has been shown that for long prediction chains, repeatedly applying CNNs can in some cases have a negative impact on compression efficiency. For example, it is shown how this problem can be mitigated by using CNNs only for specific inter-predicted blocks, without affecting random access (RA) coding performance. [Background technology]

[0009] Deep learning-based approaches have been successfully applied in recent years to many image processing tasks, especially those generally encompassed under the term computer vision. A particularly important class of such approaches is the convolutional neural network (CNN). One of the earliest CNNs was the so-called LeNet, first proposed by Y. LeCun in 1989 for automatic recognition of zip codes

[15] . In the following decades, CNNs have been applied to a variety of picture processing tasks, such as object recognition, picture classification and segmentation, image restoration and denoising, and many others.

[0010] In recent years, CNNs have also been proposed for video coding. Here, two distinct categories must be distinguished. The first category is the so-called end-to-end optimization compression method, such as

[16] –

[18] , in which the classical architecture of a hybrid video codec is replaced by a combination of an encoder network and a decoder network that are jointly optimized according to a common rate-distortion loss function. In the second category, the basic framework of a conventional hybrid video codec is maintained, but neural networks are used for specific coding tools, such as interpolation filtering

[19] ,

[20] , intra-prediction

[21] ,

[22] , quantization

[23] , or loop filtering

[24] –

[26] . Since the method proposed here belongs to this second category, related work from this category will be discussed in more detail below, focusing on inter-prediction. An overview of various approaches to neural network-based video compression can be found in

[27] ,

[28] .

[0011] In

[29] , Huo et al. propose a CNN-based motion compensation refinement (CNNMCR) scheme. There are two variants of CNNMCR: in the simple variant, the inter-prediction signal is fed to the CNN, and the output of the CNN is the refined prediction signal; in the extended variant, an extended block of already reconstructed neighboring samples is used as the input of the CNN. For each quantization parameter (QP), a separate model is trained.

[0012] In

[30] , Wang et al. describe a neural network-based inter-prediction (NNIP) algorithm that employs a combination of a fully connected network (FCN) and a CNN. Similar to

[29] , the output of the network is a refined inter-prediction signal, and reconstructed neighboring samples are incorporated into the network's input. However,

[30] additionally uses neighboring samples of a temporal reference block for input. An improved version of NNIP is presented in

[31] . Here, the network architecture is modified so that three neural networks are used in combination instead of two. In

[30] and

[31] , a separate model is trained for each combination of QP and block shape.

[0013] In

[32] , Zhao et al. propose a CNN-based fusion scheme, which applies only to bi-prediction and replaces the averaging of two predictors. The inputs to the network are the two component motion-compensated prediction signals, and the output is a combined inter-prediction signal. A separate model is trained for each QP.

[0014] In

[33] , Mao et al. present a CNN-based bi-prediction method that utilizes spatial information, called SICNN. Conceptually, SICNN can be seen as a combination of concepts from

[30] ,

[31] , and

[32] . Similar to

[32] , the two component prediction signals of the bi-prediction are used as the input to the CNN. Similar to

[30] and

[31] , the corresponding block is expanded to include the upper / left spatial neighboring samples. The output of the CNN is a refined bi-prediction signal. Again, for each QP, a separate model is trained for SICNN. In

[34] , Mao and Yu extend the work of

[33] to also include temporal distance information in the input to the CNN.

[0015] In

[35] , Zhang et al. describe a CNN-based inter-prediction refinement method for the AVS3 standard

[36] . This work builds on the work

[30] , but uses CNN instead of FCN to enable the network to be applied to all block shapes. Furthermore, in

[35] , spatially neighboring samples are not used. Furthermore, a separate model is trained for each QP.

[0016] In

[37] , Jin et al. propose a deep affine motion compensation network (DAMC-Net) based on the AMC method for VVC. The inputs to the network are the AMC prediction, the initial motion vector field, and the reference block. The output of the network is a refined AMC prediction signal. Similar to

[29] -

[31] ,

[33] ,

[34] , the input block is expanded to include the upper / left neighboring samples. A separate network model is trained for each combination of block shape and QP.

[0017] Previous work

[14] presented an intra-inter prediction residual convolutional neural network (IPRN). The architecture of the IPRN is based on

[33] ,

[34] . Therefore, the input to the network includes the inter prediction signal along with the two component prediction signals of bi-prediction, similarly augmented by the top / left neighbor samples. Unlike most other related works, the IPRN is based on VVC and uses a single network model for all block shapes and QP values. Furthermore, different training loss functions were investigated in

[14] . The sum of absolute transform differences (SATD), i.e., the "1-norm" in the DCT domain, was found to yield better coding performance than the commonly used sum of squared differences (SSD) and sum of absolute differences (SAD), which operate in the spatial domain.

[0018] Most of the methods described above, namely

[29] -

[31] ,

[33] ,

[34] ,

[37] , as well as previous work

[14] , use reconstructed neighboring samples above / to the left for the neural network input. This has significant implications for the actual implementation of the decoder. First, and most importantly, the network cannot be applied in parallel to the affected blocks of a single picture. Instead, blocks must be input to the network sequentially. This is caused by the fact that the network's input depends on the reconstructed neighboring samples and, therefore, on the network's output for these blocks. Second, by referencing the reconstructed neighboring samples, a CNN-refinement inter block can depend on the output of intra prediction if at least one of its neighboring blocks above / to the left happens to be intra predicted. Both of the above aspects have the effect of making CNN-based inter prediction part of the so-called intra decoding loop. This is a complete break with existing video codec design principles. In all video coding standards, including VVC, inter prediction can be performed in parallel at the decoder for all inter blocks of a single picture after the corresponding motion vectors have been determined. This would be impossible with such a change.

[0019] This paper proposes a spatiotemporal residual CNN (STRN) for enhanced inter prediction. Its distinct feature is that the network is moved out of the intra-decoding loop while still incorporating neighboring samples. Therefore, the solution described herein enables parallel processing of the CNN for all affected blocks of a picture at the decoder. Intra prediction and CNN processing can also be performed in parallel at the decoder using STRN. This aspect, which significantly impacts practical implementations, has not previously been addressed in the literature related to CNN-based inter prediction. Furthermore, most of the above-mentioned methods use separate CNN models for each block shape and / or QP value. In contrast, STRN uses a single CNN model for all block sizes and QP values. Summary of the Invention [Problem to be solved by the invention]

[0020] It is therefore desirable to provide concepts for rendering picture coding and / or video coding more efficiently, e.g., in terms of compression efficiency, and / or for improving the trade-off between computational complexity and coding performance to support CNN-based inter-prediction. Additionally or alternatively, it is desirable to reduce the bitstream and thus the signaling cost.

[0021] This is achieved by the subject matter of the independent claims of the present application.

[0022] Further embodiments according to the invention are defined by the subject matter of the dependent claims of the present application. [Means for solving the problem]

[0023] According to a first aspect of the present invention, the inventors of the present application have realized that one problem encountered when processing a current picture portion dependent on a previous picture portion arises from the fact that neighboring picture portions must be processed before the current picture portion can be processed. According to the first aspect of the present application, this difficulty is overcome by generating a neighboring signal that is independent of spatial signal interdependence, for example, by excluding signals with spatial signal interdependence and / or by replacing signals with spatial signal interdependence with replacement signals that are independent of spatial signal interdependence, and / or by using a non-post-processed version of the signal that depends on spatial signal interdependence. The inventors have found that forming or generating a neighboring signal using constrained spatial reference samples is advantageous because it allows for decoupling the processing of the current picture portion from the sequential spatial processing of picture portions that depend on already reconstructed neighboring samples. Thus, multiple picture portions, each of whose neighboring signals is generated independently of spatial signal interdependence, can be processed in parallel rather than sequentially. The idea is that the neighborhood signal introduced here allows the processing of a current picture portion depending on its spatial neighborhood, without the spatial neighborhood having to be completely reconstructed before the current picture portion is processed. By being able to take spatial neighborhood into account in the parallel processing of picture portions, high encoding / decoding efficiency, and in particular high coding performance, can be achieved.

[0024] Thus, according to a first aspect of the present application, a video decoder / encoder including a plurality of decoding / encoding tools is configured to apply the plurality of decoding / encoding tools to a current picture of a video on a block-by-block basis, e.g., controlled by a data stream. A reconstructed signal of the currently decoded / encoded picture can be derived by a sample-by-sample combination of contributing signals generated by the plurality of decoding / encoding tools, e.g., the video decoder is configured to derive the reconstructed signal. The plurality of decoding / encoding tools includes a first predetermined decoding / encoding tool configured to perform post-processing of a signal associated with the current block or to perform generation of a signal associated with the current block based on neighboring signals in a spatial neighborhood of the current block. In the post-processing, the first predetermined decoding / encoding tool is configured to post-process contributing signals of one or more second predetermined decoding tools in the current block corresponding to a partial combination from the sample-by-sample combination, or to post-process an intermediate signal in the current block. The intermediate signal may, for example, correspond to a sample-by-sample combination of two or more contributing signals among the contributing signals generated by the plurality of decoding / encoding tools. These two or more contributing signals may be generated by one or more second decoding / encoding tools, but may also be generated by one or more other decoding / encoding tools of the plurality of decoding / encoding tools. In generating, the first predetermined decoding / encoding tool is configured to generate the contributing signals of the first predetermined decoding / encoding tool for the current block. Furthermore, the video decoder / encoder is configured to generate neighboring signals in the spatial neighborhood by using the contributing signals of the one or more second predetermined decoding / encoding tools or versions of intermediate signals in the spatial neighborhood that have not been post-processed by the first predetermined decoding / encoding tool, and / or by replacing the contributing signals of one or more third predetermined decoding / encoding tools in the spatial neighborhood with replacement signals generated independently of spatial signal interdependencies, and / or by excluding samples from the spatial neighborhood whose sample-by-sample combination for deriving the reconstructed signal includes the contributing signals of one or more third predetermined decoding / encoding tools.

[0025] This first aspect is applicable to different decoding / encoding tools, such as a spatio-temporal residual network (STRN) tool, a local illumination compensation (LIC) tool, a joint inter / intra prediction (CIIP) tool, a residual code prediction (RSP) tool, and / or a template matching (TM) tool, where two or more of these decoding / encoding tools may be used or comprised by a video decoder / encoder.

[0026] According to one embodiment, the first predetermined decoding / encoding tool may be a STRN tool configured to post-process the contribution signals, i.e., inter-prediction signals, of one or more second predetermined decoding / encoding tools, i.e., one or more inter-prediction tools, or to post-process intermediate signals, i.e., sample-by-sample combinations of two or more inter-prediction signals generated by one or more inter-prediction tools. The post-processing may be performed by using a neural network or convolution, for example, by subjecting tensors to a neural network or convolution. For example, the STRN tool may be configured to post-process the contribution signals or intermediate signals based on a 3D tensor that includes one or more matrices derived from corresponding portions in one or more reference pictures, and includes one or more matrices derived from contribution signals with neighboring signals or intermediate signals with neighboring signals. The 3D tensor may represent an input to the neural network or convolution. The corresponding portions in one or more reference pictures may represent, for example, portions similar to the current block, i.e., the current portion, that may be found in the reference pictures. The corresponding portions in one or more reference pictures may be indicated or derived using one or more motion vectors derived / encoded from / into the data stream. There may be only one corresponding portion in a reference picture, or there may be two or more corresponding portions in a reference picture. Furthermore, a video decoder / encoder including the STRN tool may: using the contribution signals of one or more second predetermined decoding / encoding tools, i.e. one or more inter prediction tools, i.e. the inter prediction signals, or a sample-by-sample combination of two or more inter prediction signals generated by one or more inter prediction tools within a spatial neighborhood of an intermediate signal, i.e. a version not post-processed by a STRN tool; and / or replacing the contribution signals, i.e. intra prediction signals, of one or more third predetermined decoding / encoding tools, i.e. one or more intra prediction tools, in the spatial vicinity with replacement signals generated independently of spatial signal interdependencies; and / or excluding from the spatial neighborhood samples whose sample-by-sample combination for deriving the reconstructed signal includes contribution signals of one or more third predetermined decoding / encoding tools, i.e., one or more intra-prediction tools, in the spatial neighborhood, i.e., intra-prediction signals; may be configured to generate a neighboring signal in the spatial vicinity of the current block by

[0027] The replacement signal may be generated using inter prediction, for example, one or more second predetermined decoding / encoding tools, i.e., one or more inter prediction tools, may be configured to generate the replacement signal. For example, the replacement signal may be obtained by extending the inter prediction signal of the current block.

[0028] One embodiment relates to a video decoder / encoder including a plurality of decoding tools configured to apply a plurality of decoding / encoding tools to a current picture of a video on a block-by-block basis, e.g., controlled by a data stream, the plurality of decoding / encoding tools including a first set of prediction tools, and the video decoder / encoder configured to perform block-by-block selection of exactly one prediction tool from the first set of prediction tools when applying the plurality of decoding / encoding tools to the current picture on a block-by-block basis. A reconstructed signal of the currently decoded picture is derivable by a sample-by-sample combination of a prediction signal generated by the first set of prediction tools and a prediction residual signal derived, e.g., from the data stream. The plurality of decoding / encoding tools include a first predetermined decoding / encoding tool configured to post-process prediction signals of one or more inter-prediction tools of the first set of prediction tools based on neighboring signals within a spatial neighborhood. The video decoder / encoder is configured to generate neighboring signals within the spatial neighborhood by using versions of the prediction signals of one or more inter-prediction tools that have not been post-processed by the first predetermined decoding / encoding tool and / or by replacing prediction signals of one or more intra-prediction tools of the plurality of prediction tools with a replacement signal generated by inter prediction. The video decoder / encoder is configured, for example, to ignore the prediction residual signal when generating the neighboring signal when replacing the prediction signal of one or more intra prediction tools among the plurality of prediction tools with a replacement signal generated by inter prediction.

[0029] A further embodiment includes decoding / encoding video from / to a data stream using block-based prediction and transform-based predictive residual coding; performing block-based prediction on a predictive block; and applying a post-processing tool to a given predictive block and neighboring blocks that are in a spatial neighborhood of and overlap the spatial neighborhood of the given predictive block, wherein the post-processing tool: post-processing a prediction signal of the predetermined prediction block based on neighboring signals in a spatial neighborhood of the predetermined prediction block to obtain a post-processed prediction signal of the predetermined prediction block; and post-processing a prediction signal of the neighboring block based on a further neighboring signal in a further spatial neighborhood of the neighboring block to obtain a post-processed prediction signal of the neighboring block, wherein the neighboring block overlaps with the spatial neighborhood.

[0030] The neighboring blocks can be reconstructed by a sample-by-sample combination of the post-processed prediction signal and a prediction residual signal, e.g., obtained from the data stream. The video decoder / encoder is configured to form the neighboring signals within the neighboring blocks by a sample-by-sample combination of the prediction signal and the prediction residual signal of the neighboring blocks.

[0031] Yet another video decoder / encoder is configured to decode / encode video from / to a data stream using block-based prediction and transform-based predictive residual coding, perform block-based prediction using motion-compensated prediction for inter-predicted blocks and intra-prediction for intra-predicted blocks, and apply a post-processing tool to the given inter-predicted block, the post-processing tool being configured to post-process the inter-predicted signal of the given inter-predicted block based on neighboring signals within spatial neighborhoods of the given inter-predicted block to obtain a post-processed inter-predicted signal for the given inter-predicted block. A neighboring block that overlaps with the spatial neighborhood and is one of the intra-predicted blocks can be reconstructed by sample-by-sample addition of the intra-predicted signal of the neighboring block and a prediction residual signal, for example, obtained from the data stream. The decoder / encoder is configured to form the neighboring signal within the neighboring block by generating a replacement signal within the spatial neighborhood and the neighboring block through inter prediction.

[0032] According to one embodiment, the first predetermined decoding / encoding tool may be one or more second predetermined decoding / encoding tools, i.e., LIC tools configured to post-process contribution signals of one or more inter-prediction tools, i.e., inter-prediction signals, or post-process intermediate signals, i.e., sample-by-sample combinations of two or more inter-prediction signals generated by one or more inter-prediction tools. The post-processing may be performed by adapting or generating scaling and offset values ​​based on neighboring signals and post-processing the inter-prediction signals within the current block or the intermediate signals within the current block using the scaling and offset values. Furthermore, a video decoder / encoder including a LIC tool may: using a sample-by-sample combination of two or more inter-prediction signals generated by one or more inter-prediction tools in a spatial neighborhood of one or more second predetermined decoding / encoding tools, i.e., the contribution signals of one or more inter-prediction tools, i.e., the inter-prediction signals, or intermediate signals, i.e., versions that have not been post-processed by the LIC tools; and / or replacing the contribution signals, i.e. intra prediction signals, of one or more third predetermined decoding / encoding tools, i.e. one or more intra prediction tools, in the spatial vicinity with replacement signals generated independently of spatial signal interdependencies; and / or excluding from the spatial neighborhood samples whose sample-by-sample combination for deriving the reconstructed signal includes contribution signals of one or more third predetermined decoding / encoding tools, i.e., one or more intra-prediction tools, in the spatial neighborhood, i.e., intra-prediction signals; may be configured to generate a neighboring signal in the spatial vicinity of the current block by

[0033] The replacement signal may be generated using inter prediction, for example, one or more second predetermined decoding / encoding tools, i.e., one or more inter prediction tools, may be configured to generate the replacement signal. For example, the replacement signal may be obtained by extending the inter prediction signal of the current block.

[0034] One embodiment relates to a video decoder / encoder including a plurality of decoding / encoding tools configured to decode / encode video from / to a data stream using block-based prediction and transform-based predictive residual coding, perform block-based prediction using intra prediction of intra-predicted blocks to obtain intra-predicted signals for the respective blocks, and perform motion-compensated prediction of inter-predicted blocks to obtain inter-predicted signals for the respective blocks. For a current block, which is one of the inter-predicted blocks, the video decoder / encoder is configured to: apply, to a sub-block of the current block, a post-processing tool configured to post-process the inter-predicted signal of the current block based on neighboring signals within a spatial neighborhood of the current block or the sub-block. Additionally, the video decoder / encoder is configured to form the neighboring signals by excluding from the spatial neighborhood samples associated with neighboring intra-predicted blocks that overlap the spatial neighborhood and / or by using, within the spatial neighborhood, inter-predicted signals of neighboring inter-predicted blocks that overlap the spatial neighborhood in a version that has not been post-processed by the post-processing tool.

[0035] According to one embodiment, the first predetermined decoding / encoding tool may be a CIIP tool configured to generate an inter-intra prediction signal as a contribution signal of the first predetermined decoding / encoding tool for the current block. The CIIP tool may be configured to generate the inter-intra prediction signal by a weighted combination of an intra prediction signal and an inter prediction signal in the current block. The CIIP tool may be configured to perform intra prediction using neighboring signals to obtain the intra prediction signal in the current block, and to perform inter prediction to obtain the inter prediction signal in the current block. The CIIP tool may, for example, include an intra prediction decoding tool configured to generate the intra prediction signal for the current block and an inter prediction decoding tool configured to generate the inter prediction signal for the current block. Furthermore, a video decoder / encoder including a CIIP tool may: by replacing the contribution signals, i.e. intra prediction signals, of one or more third predetermined decoding / encoding tools, i.e. one or more intra prediction tools, in the spatial vicinity with a replacement signal generated independently of the spatial signal interdependence, e.g. the first replacement signal, or Replacing a contribution signal, i.e., an inter-intra predicted signal, of a third predetermined decoding / encoding tool corresponding to a first predetermined decoding / encoding tool, i.e., a CIIP tool, in a spatial neighborhood with a replacement signal, e.g., a second replacement signal, generated independently of spatial signal interdependence. may be configured to generate a neighboring signal in the spatial vicinity of the current block by

[0036] The replacement signal may be generated using inter prediction, for example, one or more second predetermined decoding / encoding tools, i.e., one or more inter prediction tools, may be configured to generate the replacement signal, or an inter prediction component of a CIIP tool may be configured to generate the replacement signal. For example, the CIIP tool may be configured to extend the inter prediction signal of the current block to obtain the first replacement signal. The inter-intra prediction signal in the spatial neighborhood may be generated by the CIIP tool by, for example, a weighted combination of the intra prediction signal in the spatial neighborhood and the inter prediction signal in the spatial neighborhood, and the CIIP tool may be configured to use the inter prediction signal in the spatial neighborhood as the second replacement signal.

[0037] One embodiment relates to a video decoder / encoder configured to decode / encode video from / to a data stream using block-based prediction and transform-based predictive residual coding, perform block-based prediction using motion-compensated prediction for inter-predicted blocks and intra-prediction for intra-predicted blocks, apply inter-intra prediction tools to the inter-intra predicted blocks, and apply inter-intra prediction tools to a given inter-intra predicted block. The decoder / encoder is configured to generate an inter-intra predicted signal for the given inter-intra predicted block based on neighboring signals within a spatial neighborhood of the given inter-intra predicted block. A first neighboring block overlapping the spatial neighborhood and being one of the intra-predicted blocks is reconstructable by a sample-by-sample combination of the intra-predicted signal of the first neighboring block and a first predicted residual signal, for example, obtained from the data stream. The decoder / encoder is configured to form the neighboring signal within the first neighboring block by generating a first substitution signal within the spatial neighborhood and the first neighboring block via inter prediction.

[0038] According to one embodiment, the first predetermined decoding / encoding tool may be an RSP tool configured to generate a prediction residual signal as a contribution signal of the first predetermined decoding / encoding tool to the current block. The RSP tool may be configured to generate the prediction residual signal by deriving / generating residual values ​​of the current block and predicting the sign of the residual value based on neighboring signals in the spatial neighborhood of the current block. Furthermore, a video decoder / encoder including the RSP tool may be configured to generate a neighboring signal in the spatial neighborhood of the current block by replacing the contribution signals, i.e., intra-prediction signals, of one or more third predetermined decoding / encoding tools, i.e., one or more intra-prediction tools, in the spatial neighborhood with a replacement signal generated independently of spatial signal interdependence. The replacement signal may be generated using inter-prediction, for example, one or more second predetermined decoding / encoding tools, i.e., one or more inter-prediction tools, may be configured to generate the replacement signal. If the current block is an inter-predicted block, for example, one or more second predetermined decoding / encoding tools may be configured to extend the inter-prediction signal of the current block to obtain the replacement signal. If the current block is an intra-predicted block, for example, the one or more second predetermined decoding / encoding tools are configured to predict a motion vector and generate an inter-predicted signal within the spatial neighborhood using the motion vector.

[0039] One embodiment relates to a video decoder / encoder configured to decode / encode video from / to a data stream using block-based prediction and transform-based predictive residual coding, performing block-based prediction on predictive blocks, and performing transform-based predictive residual coding to generate a predictive residual signal for a given predictive block of the predictive block, e.g., by deriving residual values ​​for the given predictive block from the data stream and predicting signs of the derived residual values ​​based on neighboring signals within a spatial neighborhood of the given predictive block. Intra-predicted neighboring blocks overlapping the spatial neighborhood can be reconstructed by a sample-by-sample combination of the intra-prediction signals of the intra-predicted neighboring blocks and an intra-prediction residual signal, e.g., obtained from the data stream, and / or inter-predicted neighboring blocks overlapping the spatial neighborhood can be reconstructed by a sample-by-sample combination of the inter-prediction signals of the inter-predicted neighboring blocks and an inter-prediction residual signal, e.g., obtained from the data stream. Furthermore, the video decoder / encoder is configured to form a neighborhood signal within the intra-predicted neighboring block by generating a first substitution signal within the spatial neighborhood and first neighboring block by inter prediction and / or within the inter-predicted neighboring block by using the inter-predicted signal of the inter-predicted neighboring block, e.g., for the intra-predicted reconstructed signal of the intra-predicted neighboring block, i.e., for a sample-by-sample combination of the intra-predicted signal and the intra-predicted residual signal.

[0040] According to one embodiment, the first predetermined decoding / encoding tool may be a TM tool configured to generate a prediction signal as the contribution signal of the first predetermined decoding / encoding tool for the current block. The TM tool may be configured to generate the prediction signal using template matching, and a neighboring signal within the spatial neighborhood of the current block represents a template for template matching. Furthermore, a video decoder / encoder having a TM tool may be configured to generate a neighboring signal within the spatial neighborhood of the current block by replacing the contribution signal, i.e., intra prediction signal, of one or more third predetermined decoding / encoding tools, i.e., one or more intra prediction tools, within the spatial neighborhood with a replacement signal generated independently of spatial signal interdependence. The replacement signal may be generated using inter prediction, for example, one or more second predetermined decoding / encoding tools, i.e., one or more inter prediction tools, may be configured to generate the replacement signal. For example, the one or more second predetermined decoding / encoding tools may be configured to predict motion vectors and generate an inter prediction signal within the spatial neighborhood using the motion vectors.

[0041] One embodiment relates to a video decoder / encoder configured to decode / encode video from / to a data stream using block-based prediction and transform-based predictive residual coding, and to generate a prediction signal for a given predictive block of the given predictive block by performing template matching using neighboring signals in a spatial neighborhood of the given predictive block as a template to identify an error-minimizing template match, and using the template match block associated with the error-minimizing template match as a prediction signal for the given predictive block. Intra-predicted neighboring blocks overlapping the spatial neighborhood can be reconstructed by a sample-by-sample combination of the intra-prediction signals of the intra-predicted neighboring blocks and an intra-prediction residual signal, e.g., obtained from the data stream, and / or inter-predicted neighboring blocks overlapping the spatial neighborhood can be reconstructed by a sample-by-sample combination of the inter-prediction signals of the inter-predicted neighboring blocks and an inter-prediction residual signal, e.g., obtained from the data stream. Furthermore, the video decoder / encoder is configured to form a neighborhood signal within the intra-predicted neighborhood block by generating a first substitution signal within the first neighborhood block by spatial neighborhood and inter prediction and / or within the inter-predicted neighborhood block by generating a second substitution signal within the inter-predicted neighborhood block by spatial neighborhood and inter prediction in a manner independent of the generation of the inter-predicted signal of the inter-predicted neighborhood block and the sample-by-sample addition of the second substitution signal and the inter-predicted residual signal, for example by generating a first substitution signal for the intra-predicted reconstruction signal of the intra-predicted neighborhood block, i.e., for a sample-by-sample combination of the intra-predicted signal and the intra-predicted residual signal.

[0042] According to a second aspect of the present invention, the inventors of the present application have recognized that one problem encountered when processing pictures using neural networks stems from the fact that large matrices and / or tensors must undergo multiple convolutions, resulting in high computational complexity. According to a second aspect of the present application, this difficulty is overcome by a polyphase decomposition of the neural network's input. The inventors have found that, compared to using an input that is not polyphase-decomposed, the polyphase decomposition results in a dramatic reduction in complexity with a slightly lower coding gain (for the same number of feature channels) or a significant increase in coding gain with approximately the same complexity (for twice the number of feature channels). Thus, polyphase decomposition of samples of a picture portion into polyphase components improves the tradeoff between computational complexity and coding performance.

[0043] Thus, according to a second aspect of the present application, a picture processing tool including a neural network or convolution, such as a convolutional neural network, is configured to polyphase-decompose luma samples of a picture portion into polyphase components to obtain a matrix for each polyphase component, and form a tensor by cascading the matrices of the polyphase components. The picture processing tool is configured to subject the tensor to the neural network or convolution, associating the matrices as different channels, to obtain an output tensor consisting of a concatenation of output matrices, one output matrix for each polyphase component. Additionally, the picture processing tool is configured to form a processed picture portion based on the output tensor by inverse polyphase decomposition.

[0044] Naturally, both of the above outlined aspects can be combined in a preferred manner.

[0045] A third aspect of the present invention relates to a video codec that provides post-processing for inter-predicted blocks. The inventors of this application have recognized that one problem encountered when activating post-processing tools for certain inter-predicted blocks is that it actually reduces coding efficiency rather than improving it. In particular, the inventors have discovered a method for locating, distinguishing, or identifying blocks among inter-predicted blocks to which post-processing tools can be applied and for which disabling the processing tools is beneficial in terms of coding efficiency. More precisely, the inventors have discovered a method for performing this identification in a manner that does not require explicit transmission of a switching flag or the like to control the activation and deactivation of post-processing tools. The identification method is defined by rules that include disabling post-processing tools for certain inter-predicted blocks, such as inter-predicted blocks with one or more zero motion vectors, inter-predicted blocks with one or more full-pel motion vectors, inter-predicted blocks associated with uni-prediction mode, merge mode, or bi-prediction mode using coding unit weights, inter-predicted blocks with specific block shapes, and inter-predicted blocks associated with specific quantization parameters. These rules avoid the possibility that repeated application of post-processing tools may adversely affect compression efficiency, especially for long prediction chains. By this means, the inventors have found a way to avoid the frequent provision of random access points (RAPs), which represent another possibility for how this repeated application of processing tools can look, but which is disadvantageous in terms of coding efficiency.

[0046] Thus, according to a third aspect of the present application, a video decoder / encoder is configured to decode / encode video from / to a data stream using block-based prediction and transform-based predictive residual coding, and to perform block-based prediction using motion-compensated prediction controlled via motion vectors. The video decoder is configured to derive motion vectors from the data stream of inter-predicted blocks, and the video encoder is configured to encode the motion vectors into the data stream of inter-predicted blocks. The video decoder / encoder is also configured to apply a post-processing tool to post-process an inter-predicted signal of a given inter-predicted block. Additionally, the video decoder / encoder derives from a given inter-predicted block: For example, a first inter-predicted block having one or more associated motion vectors, the number of which meeting a first predetermined criterion being zero, according to the data stream; and / or a second inter-predicted block having one or more motion vectors associated therewith, the number of which meeting a second predetermined criterion being full-pel motion vectors, for example according to the data stream; and / or a third inter-predicted block having one of a set of predetermined inter-prediction modes associated therewith, for example, according to the data stream, the set of predetermined inter-prediction modes including one or more of a uni-prediction mode, a merge mode, and a bi-prediction mode using coding unit weights; and / or a fourth inter-predicted block whose block shape meets a predetermined criterion; and / or a fifth inter-predicted block whose quantization parameter has a value that satisfies a further predetermined criterion, the quantization parameter being signaled in the data stream; is configured to identify the predetermined inter-predicted block from the inter-predicted blocks by excluding

[0047] Again, even this latter aspect may be combined with either or both of the previously identified aspects of this application.

[0048] According to a fourth aspect of the present invention, the inventors of the present application have realized that one problem encountered when processing a current picture portion depending on a previous picture portion arises from the fact that neighboring picture portions must be processed before the current picture portion can be processed. According to the fourth aspect of the present application, this difficulty is overcome by generating a constrained neighborhood signal for intra-predicting intra-predicted blocks. The inventors have found that it is advantageous to apply post-processing of the inter-predicted signal, separating the CIIP prediction tool and / or the RSP tool from the intra-prediction loop. This allows intra-predicting intra-predicted blocks of a picture in parallel with the post-processing of the inter-predicted signal and / or the generation of the inter-intra prediction signal and / or residual code prediction. This is based on the idea that the neighborhood signal introduced here allows processing of intra-predicted blocks depending on their spatial neighbors, without the spatial neighbors needing to be completely reconstructed before the intra-predicted blocks are processed. By being able to consider spatial neighbors in the parallel processing of picture portions, high encoding / decoding efficiency, particularly high coding performance, can be achieved.

[0049] Thus, according to a fourth aspect of the present application, a video decoder / encoder is configured to decode / encode video from / to a data stream using block-based prediction and transform-based predictive residual coding, and to perform block-based prediction by using motion-compensated prediction for inter-predicted blocks and by using intra-prediction for intra-predicted blocks. The video decoder / encoder is configured to intra-predict the intra-predicted blocks using neighboring signals within a spatial neighborhood of the intra-predicted blocks. The video decoder / encoder is further configured to apply a post-processing tool to a first neighboring block that overlaps the spatial neighborhood and is one of the inter-predicted blocks, where the post-processing tool is configured to post-process the inter-predicted signal of the first neighboring block to obtain a post-processed inter-predicted signal. The first neighboring block can be reconstructed by a sample-by-sample combination of the post-processed inter-predicted signal of the first neighboring block and a first prediction residual signal, for example, obtained from the data stream. Additionally, the video decoder / encoder is configured to form a neighboring signal within the spatial neighborhood by using, within the first neighboring block, an inter-prediction signal of a version of the first neighboring block that has not been post-processed by a post-processing tool.

[0050] According to a fourth aspect of the present application, a further embodiment relates to a video decoder / encoder configured to decode / encode video from / to a data stream using block-based prediction and transform-based predictive residual coding, perform block-based prediction by using intra prediction for intra-predicted blocks and by using inter-intra prediction for inter-intra predicted blocks, and intra-predict the intra-predicted blocks using a neighboring signal within a spatial neighborhood of the intra-predicted blocks. Furthermore, the video decoder / encoder is configured to inter-intra predict a first neighboring block that overlaps the spatial neighborhood and is one of the inter-intra predicted blocks to obtain an inter-intra predicted signal of the first neighboring block, where the inter-intra predicted signal corresponds to a weighted combination of the intra-predicted signal and the inter-predicted signal of the first neighboring block. The first neighboring block can be reconstructed by a sample-by-sample combination of the inter-intra predicted signal of the first neighboring block and a first predicted residual signal, e.g., obtained from the data stream. Additionally, the video decoder / encoder is configured to form a neighborhood signal within a spatial neighborhood by using, within the first neighborhood block, an inter-prediction signal of the first neighborhood block rather than an intra-prediction signal of the first neighborhood block.

[0051] A further embodiment according to a fourth aspect of the present application relates to a video decoder / encoder configured to decode video from a data stream using block-based prediction and transform-based predictive residual coding, perform block-based prediction by using intra prediction for intra-predicted blocks, and intra-predict the intra-predicted blocks using a neighborhood signal within a spatial neighborhood of the intra-predicted blocks. The video decoder / encoder is further configured to apply a residual code prediction tool to a first neighboring block overlapping the spatial neighborhood to obtain a predicted residual signal of the first neighboring block. The first neighboring block is reconstructable by a sample-by-sample combination of the predicted signal of the first neighboring block and the predicted residual signal of the first neighboring block. Additionally, the video decoder / encoder is configured to form the neighborhood signal within the spatial neighborhood by using the predicted signal of the first neighboring block that is not combined with the predicted residual signal of the first neighboring block within the first neighboring block.

[0052] Again, even this latter aspect may be combined with any of the previously specified aspects of this application, or with two or more of the previously specified aspects.

[0053] The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings: [Brief explanation of the drawings]

[0054] [Figure 1] 1 illustrates one embodiment of encoding into a data stream. [Figure 2] 1 illustrates an embodiment of an encoder. [Figure 3] 1 illustrates one embodiment of picture reconstruction. [Figure 4] 1 illustrates an embodiment of a decoder. [Figure 5] 1 illustrates one embodiment of a relationship between a reconstructed signal and a combination of a prediction residual signal and a prediction signal. [Figure 6] 1 illustrates an embodiment of a decoder configured to constrain neighboring signals. [Figure 7] 1 illustrates an embodiment of a decoder / encoder with a STRN tool that uses constrained neighborhood signals. [Figure 8] 1 illustrates an embodiment of a decoder / encoder with an LIC tool that uses constrained neighborhood signals. [Figure 9] 1 illustrates an embodiment of a decoder / encoder with CIIP tools that uses constrained neighborhood signals. [Figure 10] 1 illustrates an embodiment of a decoder / encoder with RSP tools that uses constrained neighborhood signals. [Figure 11] 1 illustrates an embodiment of a decoder / encoder with a TM tool that uses constrained neighborhood signals. [Figure 12] 1 illustrates an embodiment of a decoder / encoder configured to constrain neighboring signals for intra prediction. [Figure 13] 1 illustrates an embodiment of a picture processing tool that performs polyphasic partitioning of picture portions. [Figure 14] 1 illustrates an embodiment of a decoder configured to apply post-processing only to specific inter-predicted blocks. [Figure 15] 1 illustrates one embodiment of a neural network that can be used by the decoder, encoder, and / or picture processing tools described herein. [Figure 16] A comparison is presented between the IPRN architecture without polyphase decomposition and the STRN architecture with polyphase decomposition. [Figure 17] Figure 1 shows the relationship between learning rate decay and average SATD training loss for the first 20 epochs (IPRN and STRN models with B=4, N=6, and F=64 and F=128, respectively, both using the same training dataset). [Figure 18]We show the effect of spatially referencing samples in the input on the output (a simple CNN example with N=6 layers and a kernel size of 3x3, where every output value depends on a 13x13 area of ​​input values). [Figure 19] Average positional MSE reduction after inference with 32x32 blocks for N=6 different models: (a) IPRN with B=4, (b) STRN with B=4, (c) STRN with B=0, and (d) color map (using logarithmic scale). [Figure 20] Inter-decoding process (prediction and reconstruction with residual R), i.e., (a) inter-slice with inter, intra and STRN blocks, (b) decoding process with STRN in the intra-loop, and (c) proposed decoding process with constrained spatial reference samples that separates STRN from the intra-loop, are shown. [Figure 21] Examples of model, training, and dataset configurations are given below. [Figure 22] Coding performance and complexity analysis of different IPRN and STRN configurations are presented. [Figure 23] We show the coding performance of STRN using constrained spatial reference samples. [Figure 24] 1 shows the overall coding performance of the STRN variant. [Figure 25] We show the coding performance of the STRN variants as average MAC and luma BD rates per pixel for VTM-15.0 CTC RA (N, F, B, and K variants are connected by lines, respectively, and variant labels are included in Table V). [Figure 26] We show the coding performance for STRN with and without zero MV constraint (ZMC) as picture order count (POC) and frame luma BD rate [0...POC] for VTM-15.0 CTC LP. [Figure 27] We show the coding performance difference of STRN without and with zero MV constraint. DETAILED DESCRIPTION OF THE INVENTION

[0055] In the following description, embodiments are discussed in detail, but it should be understood that the embodiments provide many applicable concepts that can be embodied in a wide variety of decoding, encoding, picture processing, and video processing applications. The specific embodiments described are merely illustrative of particular ways to implement and use the concepts and do not limit the scope of the embodiments.

[0056] In the following description of the embodiments, the same or similar elements or elements having the same functions are given the same reference numerals or names, and repeated descriptions of elements given the same reference numerals or names are typically omitted. Therefore, the descriptions provided for elements having the same or similar reference numerals or identified by the same names can be mutually interchangeable or can be applied to each other in different embodiments.

[0057] In the following description, numerous details are set forth to provide a more thorough explanation of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that other embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the examples described herein. Furthermore, features of different embodiments described herein may be combined with each other unless otherwise specified.

[0058] To facilitate understanding of the following examples of this application, the description begins with a presentation of possible encoders and decoders compatible with which the examples outlined later in this application can be built. Figure 1 shows an apparatus for block-wise encoding of a picture 10 into a data stream 12. This apparatus is indicated using reference numeral 14 and may be a still picture encoder or a video encoder. In other words, when encoder 14 is configured to encode a video 16 containing picture 10 into data stream 12, picture 10 may be the current picture from video 16, or encoder 14 may exclusively encode picture 10 into data stream 12.

[0059] As mentioned above, the encoder 14 performs encoding in a block-by-block manner, or on a block basis. To this end, the encoder 14 subdivides the picture 10 into blocks, which the encoder 14 encodes into the data stream 12. Examples of possible subdivisions of the picture 10 into blocks 18 are described in more detail below. In general, the subdivision can result in blocks 18 of a fixed size, such as an array of blocks arranged in rows and columns, or in blocks 18 of different block sizes, such as by using hierarchical multi-tree subdivision that starts from the entire picture area of ​​the picture 10 or from a pre-partition of the picture 10 into an array of tree blocks; these examples should not be treated as exclusive of other possible ways of subdividing the picture 10 into blocks 18.

[0060] Furthermore, the encoder 14 is a predictive encoder configured to predictively encode the picture 10 into the data stream 12. For a particular block 18, this means that the encoder 14 determines a prediction signal (see reference numeral 24 in FIG. 2) for the block 18 and encodes into the data stream 12 the prediction residual (see reference numeral 26 in FIG. 2), i.e., the prediction error, whereby the prediction signal deviates from the actual picture content in the block 18.

[0061] The encoder 14 may support different prediction modes for deriving a prediction signal for a particular block 18. The prediction mode of interest in the following example is intra-prediction mode, in which the interior of the block 18 is spatially predicted from previously coded samples in its neighborhood in the picture 10. The encoding of the picture 10 into the data stream 12, and therefore the corresponding decoding procedure, may be based on a particular coding order 20 defined among the blocks 18. For example, the coding order 20 may traverse the blocks 18 in a raster scan order, such as row-wise from top to bottom, with each row traversed from left to right, although other scan orders, such as a diagonal scan order, are also possible. In the case of hierarchical multi-tree-based subdivision, a raster scan ordering or another scan ordering may be applied within each hierarchical level, or a depth-first scan order may be applied, i.e., leaf nodes in a block at a particular hierarchical level may precede blocks at the same hierarchical level that have the same parent block according to the coding order 20. Depending on the coding order 20, previously coded samples in the neighborhood of the block 18 may typically be located on one or more sides of the block 18. In the example presented here, for example, the neighboring, already coded samples of block 18 are located above and to the left of block 18 .

[0062] Intra-prediction modes may not be the only ones supported by encoder 14. If encoder 14 is a video encoder, for example, encoder 14 may also support an inter-prediction mode in which block 18 is temporally predicted from a previously encoded picture of video 16. Such an inter-prediction mode may be a motion-compensated prediction mode, according to which a motion vector is signaled for such block 18 indicating the relative spatial offset of the portion from which the prediction signal of block 18 is derived as a copy. An inter-predicted block is inter-predicted from a reference picture by determining a motion vector and copying the prediction signal for this block from a location in the reference picture pointed to by the motion vector. Additionally or alternatively, other non-intra-prediction modes may also be available, such as inter-prediction modes in the case where encoder 14 is a multiview encoder, or non-prediction modes in which the interior of block 18 is coded as is, i.e., without prediction.

[0063] Additionally or alternatively, the encoder 14 may support a combined inter-intra prediction (CIIP) mode in which the block 18 is temporally predicted from previously coded pictures of the video 16 to obtain an inter prediction signal, the block 18 is spatially predicted using samples neighboring the block 18 to obtain an intra prediction signal, and the inter prediction signal and the intra prediction signal are combined by a weighted combination, e.g., a weighted averaging process is applied to combine both predictions.

[0064] Before beginning to focus the description of this application on constraining samples within the neighborhood of block 18 for processing of block 18 or post-processing of the inter- or intra-predicted signal of block 18, a more specific example of a possible block-based encoder, i.e., a possible implementation of encoder 14, as described with respect to Figure 2, is presented, and then two corresponding examples of decoders compatible with Figures 1 and 2, respectively, are presented in Figures 3 and 4.

[0065] 2 illustrates a possible implementation of the encoder 14 of FIG. 1 , i.e., an implementation in which the encoder 14 is configured to use transform coding to code the prediction residual 26, but this is largely by way of example and the present application is not limited to such prediction residual coding. According to FIG. 2 , the encoder 14 comprises a subtractor 22 configured to subtract a corresponding prediction signal 24 from an inbound signal, i.e., picture 10, or from a current block 18 on a block-by-block basis, to obtain a prediction residual signal 26, which is then encoded into the data stream 12 by a prediction residual encoder 28. The prediction residual encoder 28 comprises a lossy encoding stage 28 a and a lossless encoding stage 28 b. The lossy stage 28 a receives the prediction residual signal 26 and comprises a quantizer 30 that quantizes samples of the prediction residual signal 26. As already mentioned above, this example uses transform coding of the prediction residual signal 26, and therefore the lossy encoding stage 28a includes a transform stage 32 connected between the subtractor 22 and a quantizer 30, which transforms this spectrally decomposed prediction residual 26 by quantization performed by the quantizer 30 on the transformed coefficients representing the residual signal 26. The transform may be a DCT, DST, FFT, Hadamard transform, etc. The transformed and quantized prediction residual signal 34 is then losslessly coded by a lossless encoding stage 28b, which is an entropy coder that entropy codes the quantized prediction residual signal 34 into the data stream 12. The encoder 14 further includes a prediction residual signal reconstruction stage 36 connected to the output of the quantizer 30, so as to reconstruct the prediction residual signal from the transformed and quantized prediction residual signal 34 in a manner that is also usable by a decoder (see reference numeral 54 in Figures 3 and 4), i.e., taking into account the coding loss of the quantizer 30. To this end, the prediction residual reconstruction stage 36 comprises an inverse quantizer 38 that performs the inverse of the quantization of the quantizer 30, followed by an inverse transformer 40 that performs an inverse transform to the transform performed by the transformer 32, such as the inverse of the spectral decomposition, such as the inverse of any of the specific transform examples mentioned above.Encoder 14 includes an adder 42 that adds a reconstructed prediction residual signal as output by inverse transformer 40 with prediction signal 24 to output a reconstructed signal, i.e., reconstructed samples. This output is provided to a predictor 44 of encoder 14, which then determines prediction signal 24 based thereon. It is predictor 44 that supports all prediction modes described above with respect to Figure 1. Figure 2 also shows that, if encoder 14 is a video encoder, encoder 14 may also include an in-loop filter 46 that filters the fully reconstructed picture, which, after being filtered, forms the reference picture for predictor 44 for inter-predicted blocks.

[0066] As already mentioned above, the encoder 14 operates on a block basis. In the following description, the block-of-interest basis refers to a subdivision of the picture 10 into blocks for which an intra-prediction mode is selected from one or more intra-prediction modes supported by the predictor 44 or the encoder 14, respectively; the selected intra-prediction mode or block-of-interest basis, performed separately, refers to a subdivision of the picture 10 into blocks for which an inter-prediction mode is selected from one or more inter-prediction modes supported by the predictor 44 or the encoder 14, respectively; and the selected inter-prediction mode or block-of-interest basis, performed separately, refers to a subdivision of the picture 10 into blocks for which a CIIP mode is selected from a set or more CIIP modes supported by the predictor 44 or the encoder 14, respectively; the selected CIIP mode is performed separately. However, there may also be other types of blocks into which the picture 10 is subdivided. For example, the above-mentioned decision of whether the picture 10 is inter-coded, intra-coded, or CIIP-coded may be made at a granularity or on a block-by-block basis deviating from the block 18. For example, mode decisions may be performed at the level of coding blocks, into which picture 10 is subdivided, and each coding block is subdivided into predictive blocks. Predictor 44 or encoder 14 may support multiple inter-coding modes, multiple intra-coding modes, and / or multiple CIIP modes. At the level of coding blocks, for example, it is determined whether each block is inter-coded, intra-coded, or CIIP-coded, and at the level of predictive blocks, into which coding blocks are subdivided, it is individually determined which actual mode should be selected from multiple modes supported for each coding by predictor 44 or encoder 14. These predictive blocks form blocks 18 of interest here. Another block subdivision relates to the subdivision into transform blocks, in units on which transformations by transformer 32 and inverse transformer 40 are performed.The transformed blocks may be, for example, the result of further subdivision of the coding blocks. The subdivision into transform blocks may be different from the subdivision into predictive blocks. Of course, the examples described herein should not be treated as limiting, and other examples exist as well. For completeness only, it should be noted that the subdivision into coding blocks may, for example, use multi-tree subdivision, and that predictive blocks and / or transform blocks may also be obtained by further subdividing the coding blocks using multi-tree subdivision. In the particular embodiments described herein, predictive blocks are of primary interest.

[0067] A decoder 54 or device for block-wise decoding compatible with the encoder 14 of FIG. 1 is shown in FIG. 3. This decoder 54 performs the inverse of the encoder 14, i.e., decodes the picture 10 from the data stream 12 block-wise, and for this purpose supports multiple intra-, inter-, and / or CIIP modes. The decoder 54 may, for example, comprise a residual provider 52. All other possibilities discussed above with respect to FIG. 1 are also valid for the decoder 54. To this end, the decoder 54 may be a still picture decoder or a video decoder, and all prediction modes and prediction possibilities are also supported by the decoder 54. The difference between the encoder 14 and the decoder 54 mainly lies in the fact that the encoder 14 chooses or selects coding decisions according to some optimization, such as, for example, to minimize some cost function that may depend on the coding rate and / or coding distortion. One of these coding options or coding parameters may include selecting an intra-prediction mode to be used for the current block 18 from among available or supported intra-prediction modes, selecting an inter-prediction mode to be used for the current block 18 from among available or supported inter-prediction modes, or selecting a CIIP mode to be used for the current block 18 from among available or supported CIIP modes. The selected mode may then be signaled by encoder 14 for the current block 18 in data stream 12, and decoder 54 redoes the selection using this signaling in data stream 12 for block 18. Similarly, subdivision of picture 10 into blocks 18 may be optimized in encoder 14, corresponding subdivision information may be conveyed in data stream 12, and decoder 54 restores the subdivision of picture 10 into blocks 18 based on the subdivision information. To summarize the above, decoder 54 may be a predictive decoder operating on a block basis, and in addition to intra-prediction modes, decoder 54 may support other prediction modes, such as inter-prediction modes or CIIP modes, for example, if decoder 54 is a video decoder.1, and this coding order 20 is followed in both the encoder 14 and the decoder 54, so that the same neighboring samples are available for the current block 18 in both the encoder 14 and the decoder 54. Therefore, in order to avoid unnecessary repetition, the description of the operation modes of the encoder 14 shall also apply to the decoder 54 as far as the subdivision of the picture 10 into blocks is concerned, e.g., as far as the prediction is concerned and as far as the coding of the prediction residuals is concerned. The difference lies in the fact that the encoder 14 selects, by optimization, some coding options or coding parameters and signals in or inserts into the data stream 12 coding parameters derived from the data stream 12 by the decoder 54 in order to redo the prediction, subdivision, etc.

[0068] Figure 4 shows a possible implementation of decoder 54 of Figure 3, i.e., an implementation that matches the implementation of encoder 14 of Figure 1 shown in Figure 2. Many elements of encoder 54 of Figure 4 are the same as elements present in the corresponding encoder of Figure 2, and therefore the same reference numerals with apostrophes are used to indicate these elements in Figure 4. In particular, summer 42', optional in-loop filter 46', and predictor 44' are connected to the prediction loop in the same manner as in the encoder of Figure 2. 4. The inverse quantized and retransformed prediction residual signal 34″ applied to the adder 42′ is derived by a sequence of entropy decoders 56 which reverse the entropy coding of the entropy encoder 28b to obtain the quantized and transformed prediction residual signal 34′, followed by a residual signal reconstruction stage 36′ consisting of an inverse quantizer 38′ and an inverse transformer 40′, just as on the encoding side. The output of the decoder is a reconstruction of the picture 10, i.e. a reconstructed signal 58 or a part of the reconstructed signal 58. The reconstruction of the picture 10 may be available at the output of the adder 42′ or, alternatively, directly at the output of the in-loop filter 46′. Some post-filter may be arranged at the output of the decoder to perform some post-filtering on the reconstruction of the picture 10 to improve the picture quality, although this option is not shown in FIG. 4.

[0069] Again, with respect to Figure 4, the explanation given above with respect to Figure 2 is also valid for Figure 4, except that the encoder simply performs optimization tasks and related decisions regarding coding options. However, all of the descriptions regarding block subdivision, prediction, inverse quantization, and retransformation are also valid for the decoder 54 of Figure 4.

[0070] 5 shows the relationship between the reconstructed signal 58, i.e. the reconstructed picture, on the one hand, and the combination of the inverse quantized and retransformed prediction residual signal 34" and the prediction signal 24', on the other hand. As already mentioned above, the combination may be an addition performed, for example, by adders 42' or 42. Although the prediction signal 24' is shown in FIG. 5 as a subdivision of the picture area into prediction blocks 80 of various sizes, this is merely an example. The subdivision may be any subdivision, such as a regular subdivision of the picture area into rows and columns of blocks, or a multi-tree subdivision of the picture 10 into leaf blocks of various sizes, such as a quad-tree subdivision, a mixture of which is shown in FIG. 5, in which the picture area is first subdivided into rows and columns of tree root blocks 82, which are then further subdivided according to a recursive multi-tree subdivision to obtain the prediction blocks 80. It is also possible that one or more tree root blocks 82 are not further subdivided, in which case each block 82 represents a prediction block 80.

[0071] The prediction residual signal 34″ in FIG. 5 is also shown as a subdivision of the picture area into blocks 84 of various sizes. These blocks 84 may be referred to as transform blocks or transform coefficient blocks to distinguish them from the prediction blocks 80. In fact, FIG. 5 shows that the encoder 14 and decoder 54 may use two different subdivisions of the picture 10 into blocks: one subdivision into prediction blocks 80 and another subdivision into blocks 84. While both subdivisions may be the same, i.e., each prediction block 80 can simultaneously form a transform block 84, and vice versa, FIG. 5 also shows that, for example, the subdivision into transform blocks 84 forms an extension of the subdivision into prediction blocks 80, so that any boundary between two prediction blocks 80 overlays the boundary between two blocks 84, or, alternatively, each prediction block 80 coincides with one of the transform blocks 84, or with a cluster of transform blocks 84 (compare prediction block 801 with its corresponding tree-root block 86, which is further subdivided into blocks 84). 8 illustrates the case where the prediction blocks 80 and 84 are not necessarily square, rectangular, or of any other shape. Furthermore, the subdivision of the current picture 10 into prediction blocks 80, from which the prediction signal 24′ is formed, and the subdivision of the current picture 10 into blocks 84 from which the prediction residual 34″ is coded, may not be the only subdivisions used for coding / decoding.These subdivisions are from the granularity at which prediction signal determination and residual coding are performed, but firstly, the residual coding may alternatively be performed without subdivision, and secondly, at a granularity other than these subdivisions, the encoder and decoder may set specific coding parameters which may include some of the parameters mentioned above, such as prediction parameters, prediction signal configuration control signals, etc.

[0072] FIG. 5 shows that the combination of the prediction signal 24′ and the prediction residual signal 34″ directly results in the reconstructed signal 58. However, it should be noted that, according to alternative embodiments, for example, prediction signals obtained from other views or other coding layers that are coded / decoded in separate prediction loops with separate DPBs, two or more prediction signals 24′ can be combined with the prediction residual signal 34″ to result in the picture 10.

[0073] In Fig. 5, the transform blocks 84 shall have the following meaning: the transformer 32 and the inverse transformer 40 / 40' perform their transforms in units of these transform blocks 84. For example, many codecs use some kind of DST or DCT for all transform blocks 84. Some codecs allow for skipping the transform for some of the transform blocks 84, so that the prediction residual signal 34" is coded directly in the spatial domain. However, according to the embodiments described herein, the encoder 14 and the decoder 54 are configured so that they support several transforms.

[0074] The following describes embodiments that may improve coding efficiency and / or compression efficiency for block-based picture and / or video coding. The following embodiments primarily describe features and functionality from the perspective of the decoder 54. However, it will be apparent that the same or similar features and functionality may be included by the encoder 14; for example, the decoding performed by the decoder 54 may correspond to the encoding by the encoder 14. Furthermore, the encoder 14 may include the same features in a feedback loop, for example, in the prediction stage 36, as described with respect to the decoder 54.

[0075] FIG. 6 illustrates one embodiment of a video decoder 54 including a plurality of decoding tools 110 configured to apply the plurality of decoding tools 110 to a current picture 10 of a video 16 on a block-by-block basis. FIG. 6 illustratively illustrates, for example, a first predetermined decoding tool 1101, a second predetermined decoding tool 1102, and a third predetermined decoding tool 1103 included in the plurality of decoding tools 110. Similarly, the corresponding encoder 14 includes a plurality of encoding tools, including a first predetermined encoding tool, a second predetermined encoding tool, and a third predetermined encoding tool, having the functionality and / or characteristics described with respect to the decoding tools 1101, 1102, and 1103 of the plurality of decoding tools 110. Furthermore, both the video decoder 54 and the corresponding video encoder 14 illustrated in FIG. 6 may include a neighborhood signal generator 120 illustrated in FIG. 6. According to one embodiment, the neighborhood signal generator 120 may be part of the first predetermined decoding tool 1101 in the case of a video decoder, or part of the first predetermined encoding tool in the case of a video encoder.

[0076] The blocks to which the multiple decoding tools 110 are applied may have different granularity. The multiple decoding tools 110 may be applied to blocks of different dimensions.

[0077] Video decoder 54 may be configured to individually select one or more decoding tools from among multiple decoding tools 110 for a block or for a sub-block of a block. For example, one of one or more second predetermined decoding tools, including second predetermined decoding tool 1102, or one of one or more third predetermined decoding tools, including third predetermined decoding tool 1101, may be selected for each block or sub-block. First predetermined decoding tool 1101 may be selectable in addition to, or as an alternative to, one of the one or more second predetermined decoding tools or one of the one or more third predetermined decoding tools.

[0078] A plurality of decoding tools 110 can be configured to generate contributing signals (C 1 , C 2 , and C 3 6 shows, for example, a first predetermined decoding tool 1101, a second predetermined decoding tool 1102, and a third predetermined decoding tool 1103. However, the plurality of decoding tools 110 may also include two or more first predetermined decoding tools 1101, two or more second predetermined decoding tools 1102, and / or two or more third predetermined decoding tools 1103, where all the first predetermined decoding tools generate either the prediction signal P or the prediction residual signal R, all the second predetermined decoding tools generate either the prediction signal P or the prediction residual signal R, and all the third predetermined decoding tools generate either the prediction signal P or the prediction residual signal R. The decoding tools 1101, 1102, and 1103 shown in FIG. 6 can all generate a prediction signal P, in which case the plurality of decoding tools 110 includes, for example, at least one further decoding tool configured to generate a prediction residual signal R.

[0079] The reconstructed signal 58 of the currently decoded picture 10 can be derived by a sample-by-sample combination of the contributing signals generated by the multiple decoding tools 110, for example, using an adder 42'. Figure 6 illustrates the use of a predicted signal P associated with a current block 18 in the current picture 10 to reconstruct the current block 18. 18 and the prediction residual signal R 18 However, it will be apparent that multiple decoding tools 110 may generate further contribution signals for further blocks of the current picture, for example, so that a reconstructed signal 58 of the currently decoded picture 10 can be derived.

[0080] The first predetermined decoding tool 1101 detects the spatial neighborhood (100 102 , 100 104 and 100 106 11. The first predetermined decoding tool 1101 is configured to perform post-processing, e.g., using a post-processor 112, or generation, e.g., using a generator 114, based on the neighboring signal 100' in the current block 18. For example, the first predetermined decoding tool 1101 may generate a neighboring signal 100', e.g., P 18 or R 18 The contribution signal corresponding to Alternatively, the first predetermined decoding tool 1101 may be configured to generate TIFF2026506249000002.tif77. ... Post-process TIFF2026506249000003.tif77 or intermediate signal currently in block 18 TIFF2026506249000004.tif77 is post-processed, for example, to convert the post-processed contribution signal into a contribution signal of a first predetermined decoding tool 1101. It is configured to be obtained as TIFF2026506249000005.tif77.

[0081] The second predetermined decoding tool 1102 may, for example, generate a contribution signal for the current block 18. Generate TIFF2026506249000006.tif77 or use two or more contributing signals ( TIFF2026506249000007.tif79 and TIFF2026506249000008.tif79) and is currently configured to generate an intermediate signal in block 18. TIFF2026506249000009.tif77 illustrates the process of combining two or more contributing signals currently in block 18, for example, by performing sample-by-sample summation, weighting, shifting, and / or averaging operations. TIFF2026506249000010.tif79 and TIFF2026506249000011.tif79 Alternatively, the intermediate signal in the current block 18 corresponds to a sample-by-sample combination of two or more contributing signals generated by another decoding tool of the plurality of decoding tools 110, for example, a fourth predetermined decoding tool.

[0082] According to one embodiment, the plurality of decoding tools 110 includes two or more second predetermined decoding tools, including second predetermined decoding tool 1102. Video decoder 54 is configured to, for example, for each block of current picture 10 for which second decoding is selected, select one of the two or more second predetermined decoding tools to generate a contribution signal within the respective block. Thus, first predetermined decoding tool 1101 may, for example, select the contribution signals of the two or more second predetermined decoding tools within current block 18. TIFF2026506249000012.tif77, for example, a contribution signal of a second predetermined decoding tool 1102 included in two or more second predetermined decoding tools Configured to post-process TIFF2026506249000013.tif77.

[0083] According to one embodiment, the current block 18 is subdivided into sub-blocks. For example, the plurality of decoding tools 110 may be configured to generate contribution signals on a sub-block basis. According to one embodiment, a second decoding may be selected for the current block 18, and the video decoder 54 may be configured to select one of two or more second predetermined decoding tools for each sub-block of the current block 18 to determine a contribution signal for the respective sub-block. The determined contribution signals may be combined, for example, to form an intermediate signal within the current block 18.

[0084] The video decoder 54 receives the neighboring signals 100' (100 102 , 100 104 and 100 106 (see The spatial neighborhood (100 102 and 100 104 ) in the contribution signal ( TIFF2026506249000014.tif79 and TIFF2026506249000015.tif79) or intermediate signal ( TIFF2026506249000016.tif79 and TIFF2026506249000017.tif79) and / or Contribution signals of a third predetermined decoding tool 1103 within the spatial neighborhood 106 TIFF2026506249000018.tif79 is used to represent the permutation signal S generated independently of spatial signal interdependence. 106 , for example, by substituting 122 an inter prediction signal for the spatial neighborhood 106, and / or From the spatial neighborhood 100, the sample-by-sample combination for the derivation of the reconstructed signal 58 is the contribution signal of a third predetermined decoding tool 1103. 124, e.g., excluding samples containing TIFF2026506249000019.tif79, the contribution signal of the third predetermined decoding tool 1103. Exclude TIFF2026506249000020.tif79 124 The method is configured to generate the signal by

[0085] This generation of the proximity signal 100 ′ may be performed by a proximity signal generator 120 .

[0086] FIG. 6 shows a spatial neighborhood 100 102 Contribution signal associated with TIFF2026506249000021.tif79 or intermediate signal TIFF2026506249000022.tif79 and spatial neighborhood 100 104 Contribution signal associated with TIFF2026506249000023.tif79 or intermediate signal Although it is shown that TIFF2026506249000024.tif79 is considered for generating the neighborhood signal 100', it is clear that the neighborhood signal generator 120 may also consider only one non-post-processed contributing signal or intermediate signal, for example in the case of blocks 104 and 102 forming one common block.

[0087] Intermediate signals within spatial neighborhoods 102 and 104 TIFF2026506249000025.tif79 and Regarding the generation of TIFF2026506249000026.tif79, the intermediate signal currently in block 18 The same considerations as described for generating TIFF2026506249000027.tif77 can apply.

[0088] Optionally, the plurality of decoding tools 110 includes two or more third predetermined decoding tools, including third predetermined decoding tool 1103. Video decoder 54 is configured to, for example, for each block of current picture 10 for which third decoding is selected, select one of the two or more third predetermined decoding tools to generate a contributing signal within the respective block.

[0089] According to one embodiment, the third predetermined decoding tool 1103 may correspond to the first predetermined decoding tool 1101. This is the case, for example, when the first predetermined decoding tool 1101 generates a contribution signal for a block and does not perform post-processing.

[0090] 6 exemplarily shows a picture area of ​​the current picture 10 including two blocks 102 and 104 associated with a second decoding and one block 106 associated with a third decoding. A second predetermined decoding tool 1102 may be configured to generate a contribution signal for the entire block 102 and a contribution signal for the entire block 104, and a third predetermined decoding tool 1103 may be configured to generate a contribution signal for the entire block 106. However, the contribution signals ( TIFF2026506249000028.tif79, TIFF2026506249000029.tif710, TIFF2026506249000030.tif79, TIFF2026506249000031.tif79 and TIFF2026506249000032.tif79) shows the portion (100) of each block (102, 104, and 106) that overlaps the neighborhood 100 of the current block 18. 102 , 100 104 and 100 106 In other words, for the generation of the neighborhood signal 100', only the contributing signals associated with samples within the spatial neighborhood 100 of, for example, the current block 18 are taken into account.

[0091] The decoder 54 of Figure 6 and / or the corresponding encoder 14 may comprise the features and / or functionality described with respect to Figures 7 to 15. According to one embodiment, the first predetermined decoding tool 1101 and / or the corresponding first predetermined encoding tool may be, for example, a STRN tool as described in detail with respect to Figure 7, Figure 13 or Figure 15, a LIC tool as described in detail with respect to Figure 8, a CIIP tool as described in detail with respect to Figure 9, an RSP tool as described in detail with respect to Figure 10, and / or a TM tool as described in detail with respect to Figure 11. According to one embodiment, the first predetermined decoding tool 1101 and / or the corresponding first predetermined encoding tool may be configured to apply post-processing only to certain inter-predicted blocks, for example using a post-processor 112, as described with respect to Figure 14.

[0092] The second predetermined decoding tool 1102 receives the contribution signal ( TIFF2026506249000033.tif79, TIFF2026506249000034.tif711, TIFF2026506249000035.tif711, TIFF2026506249000036.tif710, TIFF2026506249000037.tif79, TIFF2026506249000038.tif711, TIFF2026506249000039.tif711, The second predetermined decoding tool may be an inter-prediction tool configured to generate an inter-prediction signal as a block for which inter prediction is selected (see TIFF2026506249000040.tif79). Thus, the block for which inter prediction is selected, i.e., the second predetermined decoding tool, may represent an inter-predicted block. In the case of two or more second predetermined decoding tools, i.e., two or more inter prediction tools, a selection from among the two or more inter prediction tools may be enabled for each block for which inter prediction is selected.

[0093] The third predetermined decoding tool 1103 receives the contribution signal ( The third decoding tool may be an intra prediction tool configured to generate an intra prediction signal as a third predetermined decoding tool, i.e., a block for which intra prediction is selected, such as a block for which intra prediction is selected (see TIFF2026506249000041.tif79). Thus, the block for which the third decoding tool, i.e., intra prediction, is selected may represent an intra predicted block. In the case of two or more third predetermined decoding tools, i.e., two or more intra prediction tools, a selection from among the two or more intra prediction tools may be enabled for each block for which intra prediction is selected.

[0094] Alternatively, the third predetermined decoding tool 1103 may be the first predetermined decoding tool 1101, for example, if the first predetermined decoding tool 1101 is a CIIP tool, an RSP tool, or a TM tool.

[0095] 9-11 illustrate both alternatives, being described with respect to neighboring block 106 when third predetermined decoding tool 1103 is an intra prediction tool, and with respect to neighboring block 102 when third predetermined decoding tool 1103 is a CIIP tool, an RSP tool, or a TM tool, respectively. However, it is clear that it is still possible for neighborhood signal generator 120 to consider both blocks 102 and 106 for neighborhood signal 100′. This is implemented, for example, by a video decoder having one or more fourth predetermined decoding tools that are one or more intra prediction tools, and third predetermined decoding tool 1103 that is a CIIP tool, an RSP tool, or a TM tool. In this case, video decoder 54 may generate neighborhood signal 100′ in spatial neighborhood 100 as Spatial Neighborhood 100 102 1103 in the spatial neighborhood 100 106 by a further replacement signal generated independently of the spatial signal interdependence. or From the spatial neighborhood 100, samples whose sample-by-sample combination for the derivation of the reconstructed signal 58 includes a contribution signal of a third predetermined decoding tool, i.e., neighborhood 100 102 and from the spatial neighborhood 100, samples whose sample-by-sample combination for the derivation of the reconstructed signal 58 includes the contribution signals of one or more fourth predetermined decoding tools, i.e., the neighborhood 100 106 Excluding samples within The signal is generated by:

[0096] The one or more fourth predetermined decryption tools may be included in the plurality of decryption tools 110 .

[0097] FIG. 7 shows an embodiment of the decoder 54 / encoder 14 having a STRN tool 1101 as the first predetermined decoding / encoding tool.

[0098] The STRN tool 1101 is currently searching the spatial neighborhood 100 (100 102 , 100 104 and 1006), the inter prediction signal P inter,18 is post-processed to obtain the post-processed inter-predicted signal of the current block 18. TIFF2026506249000042.tif715 is acquired. The inter-prediction signal P inter,18 is generated by an inter prediction tool 1102, which may correspond to a second predetermined decoding / encoding tool, for example. The current block 18 is a post-processed inter prediction signal TIFF2026506249000043.tif715 and the prediction residual signal R currently associated with block 18. The block to which the STRN tool is applied may be referred to below as the STRN block.

[0099] The STRN tool 1101 may, for example, use a neural network or convolution to generate an inter-predicted signal P of the current block 18 with one or more matrices derived from corresponding portions in one or more reference pictures and neighboring signals. inter,18 and an inter prediction signal P of the current block 18 based on a 3D tensor including one or more matrices derived from inter,18 Optionally, the one or more matrices derived from corresponding portions in one or more reference pictures may be derived from corresponding portions in one or more reference pictures with their respective spatial neighbors, i.e., spatial neighbors of the corresponding portions. Optionally, the 3D tensor may include one or more further matrices. The STRN tool 1101 may be configured to perform neural network-based predictive filtering. A prediction signal of a block, for example, an inter-prediction signal P of the current block 18, is inter,18 The application of a neural network to the inter prediction signal P improves the quality of the predicted signal and therefore improves the coding efficiency. The neighboring signals as additional inputs are the inter prediction signal P of the current block 18. inter,18 The STRN tool 1101 may be configured to perform post-processing as described in detail with respect to FIG.

[0100] 7 exemplarily illustrates a picture area of ​​a current picture 10 having an inter-predicted block 104 and an intra-predicted block 106 adjacent to the current block 10, i.e., located to the left and above the current block 10. The inter-predicted block 104 is located in a first spatial neighborhood 100. 102 and the second spatial neighborhood 100 104 , the current block 18 overlaps with its spatial neighbors 100, and the intra-predicted block 106 is located in the third spatial neighbor 100. 106 The inter-prediction block 104 overlaps with the spatial neighborhood 100 of the current block 18 at inter,104and the prediction residual signal of the inter prediction block 104, and the intra prediction block 106 may be reconstructed by a sample-by-sample combination of the intra prediction signal P intra and the prediction residual signal of the intra prediction block 106 .

[0101] The plurality of decoding / encoding tools of the decoder 54 / encoder 14 may comprise one or more inter prediction tools, e.g., as one or more second predetermined decoding / encoding tools, configured to generate an inter prediction signal, e.g., as a contribution signal of one or more second predetermined decoding / encoding tools. inter,104 is an inter-prediction signal of one or more inter-prediction tools. The plurality of decoding / encoding tools of the decoder 54 / encoder 14 may additionally or alternatively include one or more intra-prediction tools, e.g., as one or more third predetermined decoding / encoding tools, configured to generate an intra-prediction signal, e.g., as a contribution signal of one or more third predetermined decoding / encoding tools. The intra-prediction signal P of the intra-prediction block 106 intra are intra prediction signals of one or more intra prediction tools.

[0102] Optionally, the inter-predicted block 104 is further subdivided into sub-blocks, including the sub-block indicated in Figure 7 by reference numeral 102. According to one embodiment, post-processing by a STRN tool 1101 is enabled for the sub-block 102, which performs post-processing on the inter-predicted signal P inter,102 to obtain a post-processed inter-predicted signal for the sub-block 102. TIFF2026506249000044.tif717. The sub-block 102 is configured to obtain a post-processed inter-predicted signal of the sub-block 102. It can be reconstructed by a sample-by-sample combination of TIFF2026506249000045.tif717 and the prediction residual signal R of sub-block 102.

[0103] In the following, the generation of the neighborhood signals in the spatial neighborhood 100 of the current block 18 for post-processing by the STRN tool 1101 is described in more detail.

[0104] As can be seen in FIG. 7, within the spatial neighborhood, i.e., the first spatial neighborhood 100 104 and the second spatial neighborhood 100 104 P generated by one or more inter-predictors in inter,104 and an inter-prediction signal, e.g., P inter,102 are used in versions that are not post-processed by the STRN tool for generating the neighborhood signal. Thus, regardless of whether the complete inter-predicted block 104 or only a sub-block 102 of the inter-predicted block 104 is post-processed by the STRN tool, only the inter-predicted signal generated by one or more inter-prediction tools is used for generating the neighborhood signal, and not a post-processed version of this inter-predicted signal. Generally speaking, the contribution signals of one or more second predetermined decoding / encoding tools within the spatial neighborhood 100 are only taken into account in versions that are not post-processed by the STRN tool for generating the neighborhood signal.

[0105] Specifically, in the example shown in FIG. 7, the video decoder 54 / encoder 14 uses the following to generate the neighborhood signal for the current block 18: First spatial neighborhood 100 102 Inter prediction signal P of sub-block 102 in inter,102 using a first spatial neighborhood 100 102 corresponds to a portion of a sub-block 102 of an inter-predicted block 104 that has post-processing by the STRN tool 1101 enabled and that overlaps with the spatial neighborhood 100 of the current block 18; and Second spatial neighborhood 100 104 The inter-predicted signal P of the inter-predicted block 104 in inter,104 using a second spatial neighborhood 100 104corresponds to a portion of the inter-predicted block 104 that is not post-processed by the STRN tool 1101 and that overlaps with the spatial neighborhood 100 of the current block 18.

[0106] The video decoder 54 / encoder 14 may, when generating the neighboring signals for the current block 18, use an inter-prediction signal, e.g., P inter,104 P combined with inter,102 and ignore the prediction residual signal in the overlap region / area (100 102 and 100 104 ) or within a spatial neighborhood, i.e., the first spatial neighborhood 100 102 and the second spatial neighborhood 100 104 Inter-prediction signals generated by one or more inter-prediction tools in, for example, P inter,104 P combined with inter,102 7 illustrates, for example, a case in which the decoder 54 / encoder 14 is configured to use a sample-by-sample combination of the first spatial neighborhood 100 and the prediction residual signal R. 102 The first spatial neighborhood 100 combined with the prediction residual signal R of the sub-block 102 in 102 Inter prediction signal P of sub-block 102 in inter,102 and the second spatial neighborhood 100 104 the second spatial neighborhood 100 combined with the prediction residual signal R of the sub-block 104 in 104 Inter prediction signal P of sub-block 104 in inter,104 By using the above, it is indicated that the signal is configured to generate a proximity signal.

[0107] Spatial neighborhood 100 (third spatial neighborhood portion 100 106 For neighboring intra-predicted blocks (see intra-prediction block 106) that overlap with the third spatial neighborhood 100, the respective intra-predicted signal is either omitted 124 when generating the neighboring signal or is replaced 122 by a replacement signal. 106The video decoder 54 / encoder 14 may, for example, use the intra-prediction signal P to generate the neighboring signal. intra the sample containing the third spatial neighborhood 100 106 from the spatial neighborhood 100. Alternatively, the video decoder 54 / encoder 14 may, for example, exclude the enhanced inter-predicted signal TIFF2026506249000046.tif718, i.e., the replacement signal. Extended inter-prediction signal TIFF2026506249000047.tif718 is the current inter prediction signal P of block 18. inter,18 the third spatial neighborhood 100 106 and the third spatial neighborhood 100 106 corresponds to a portion of the intra-predicted block 106 that overlaps with the spatial neighborhood 100 of the current block 18. In other words, for example, samples whose sample-by-sample combination for the derivation of the reconstructed signal 58 includes intra-predicted signals, i.e., contribution signals of one or more third predetermined decoding / encoding tools in the spatial neighborhood, can be replaced by inter-prediction, e.g., samples generated by one or more second predetermined decoding / encoding tools. The inter-predicted signal P of the current block 18 inter,18 is a part 100 of the spatial neighborhood 100 in which the derivation of the reconstructed signal 58 includes an intra-predicted signal, e.g., a contribution signal of one or more third predetermined decoding / encoding tools within the spatial neighborhood. 106 This extended inter-prediction signal TIFF2026506249000048.tif718 can serve as a replacement signal.

[0108] Above, the neighboring blocks (see 102, 104, and 106) are discussed individually with regard to which signals associated with each block are considered for the generation of the neighboring signals. However, it is clear that the entire spatial neighborhood 100 is considered in the generation. Thus, for example, within the spatial neighborhood, all inter-predicted signals, i.e., all inter-predicted signals of the inter-predicted blocks and non-post-processed versions of the STRN blocks, are considered, and all intra-predicted signals are excluded (124) or replaced by replacement signals that are considered instead. Optionally, a deblocking filter can be applied within the spatial neighborhood 100, thereby eliminating edges between areas containing different signal types, e.g., the second spatial neighborhood 100. 104 and a third spatial neighborhood 100 106 The edges between are smoothed or reduced.

[0109] This special generation of neighborhoods allows for parallelization of the coding process and therefore increases coding efficiency. For example, excluding or substituting intra prediction signals within the neighborhood 100 of the current block 18 allows for processing STRN blocks independently from the processing of intra blocks and / or in parallel with the processing of intra blocks. Furthermore, the use of prediction signals of STRN blocks within the neighborhood 100 that are not post-processed by the STRN tool 1101, i.e., P, instead of P*, allows for processing all STRN blocks of the picture 10 in parallel.

[0110] The basic idea behind this concept is to apply a neural network or convolution to the predicted signal of a block, e.g., the inter-predicted signal P of the current block 18, in order to improve its quality. inter,18, thus improving coding efficiency, and using nearby reconstructed (i.e., above / left) samples as additional inputs to the neural network to further improve the quality of the predicted signal. However, the problem with this idea is that the neural network input for one block depends on the neural network output for the preceding block, since the reconstructed neighboring samples can only be obtained after the application of the network, e.g., the first spatial neighborhood 100 102 See the neighboring samples in.

[0111] This problem can be solved in the following way.

[0112] Instead of neighboring reconstructed samples within the boundary extension, for example, neighboring predicted samples are used before the application of the neural network, for example, a version of the predicted signal that has not been post-processed by the neural network, for example P inter,102 Use.

[0113] Also, it does not use samples from neighboring intra blocks, see intra prediction block 106. Instead, it uses, for example, an extended prediction signal of the current block, i.e. the extended inter prediction signal of the current block 18. Use TIFF2026506249000049.tif718.

[0114] In a different embodiment, instead of the reconstructed samples of the neighboring intra-predicted blocks, the extended current prediction signal, i.e. the extended inter-prediction signal, The decision to use TIFF2026506249000050.tif718 can be made at different granularities within the boundary extension region, i.e., within the spatial neighborhood 100: Decide for each (sub)block separately (e.g. using a 4x4 area).

[0115] Decide once for a larger contiguous area (e.g., distinguish only the top, left, and top-left areas).

[0116] This allows for a two-stage (inter) reconstruction process, where each stage can be performed in parallel for all blocks.

[0117] a) Perform regular inter prediction (i.e., motion compensation, sub-pel filtering, etc.); b) For each block, apply a NN to the output signal of the first stage.

[0118] An alternative solution is similar to the solution above, with the following differences:

[0119] - Initial neighborhood prediction signal (P inter,102 (see reference) (before applying NN) and the (neighborhood) residual signal R in the boundary extension region (first spatial neighborhood 100 102 ) and the resulting "intermediate reconstruction" signal may be closer to the actual reconstruction signal, but still allows for block-parallel application of the NN in a two-stage process.

[0120] For intra prediction, we also use the "intermediate reconstruction" signal as input, which allows us to perform intra prediction in parallel with the second step of the two-stage approach (see, for example, Figure 12).

[0121] Generally, the following may apply: -Consider the prediction process as a synthesis of a first low complexity stage (for example motion compensation using FIR filters) and a second higher complexity stage (for example application of neural networks).

[0122] - Making the input of the secondary stage depend only on the result of the primary stage of the preceding block.

[0123] - Generalization to a larger number of stages may also be possible. The input to stage N depends only on the result of stage N' from a previous block where N' < N.

[0124] Optionally, the use of two or more neural networks or convolutions may be allowed. For example, the STRN tool 1101 may be configured to select one neural network from a set of two or more neural networks or one convolution from a set of two or more convolutions for each STRN block. The STRN tool 1101 of FIG. 7, for example, for the current block 18, selects a neural network from a set of two or more neural networks or a convolution from a set of two or more convolutions and uses the selected neural network or the selected convolution to determine the post-processed inter-prediction signal of the current block 18, TIFF2026506249000051.tif715. The neural network or convolution used by the STRN tool is selected, for example, once per STRN block, per picture, per sequence of pictures, or once for the entire video.

[0125] The neural networks of a set of two or more neural networks may differ from each other in their parameters such as (learned) weights and / or (learned) biases, and / or in their structure such as the number of layers, the type of layer (e.g., 2D convolution, 3D convolution, fully connected layer, etc.), and / or the input tensor format (e.g., number of channels, type of channels, border size, etc.). Similarly, the convolutions of a set of two or more convolutions may differ from each other in their parameters such as (learned) weights and / or (learned) biases, and / or in their structure such as the type of each convolution (e.g., 2D convolution, 3D convolution, fully connected layer, etc.) and / or the input tensor format (e.g., number of channels, type of channels, border size, etc.).

[0126] The neural network or convolution selected for the current block 18 may be explicitly signaled in the data stream on a per-sequence basis, and / or per-segment (e.g., group of pictures, random access point, etc.) of a sequence, and / or per-picture basis, and / or per-slice basis, and / or per-block (e.g., CTU, predictive block, etc.) basis. In other words, video encoder 14 may be configured to select a neural network or convolution for the current block 18 and indicate it in the data stream, i.e., encode information indicating the selected neural network or convolution, e.g., information pointing to a neural network in a set of two or more neural networks or a convolution in a set of two or more convolutions. Video decoder 54 may be configured to be controlled by the data stream to select a neural network or convolution, e.g., by deriving information indicating the selected neural network or convolution from the data stream.

[0127] Alternatively, or in combination with explicit signaling, the video encoder 14 / video decoder 54 may: Block shape, e.g., number of samples in the current block 18, aspect ratio of the current block 18, max(width, height), min(width, height), etc.; and / or the coding / prediction mode currently associated with block 18 (e.g., neural networks or convolutions may be different for uni-prediction and bi-prediction, and may be different for tools that do not use simple averaging for bi-prediction, such as BIO or BCW); and / or the temporal layer of the current picture 10 (e.g., neural networks or convolutions may be different for reference and non-reference pictures); and / or the quantization parameter associated with the current block 18, e.g., slice QP or block QP, and / or Residual signal (e.g., neural networks or convolutions may be different for blocks with and without transmitted residual signal) and / or the POC difference between the current picture and the reference picture (e.g., neural networks or convolutions may differ for smaller and larger POC differences, and for symmetric and asymmetric POC differences); and / or - Motion vectors (e.g., the precision of the motion vectors, which may be different for blocks with zero and non-zero motion vectors, for example) The neural network or convolution for the current block 18 may be configured depending on

[0128] Optionally, all or part of the network / convolution parameters are transmitted in the bitstream / datastream. According to one embodiment, the video encoder 14 / video decoder 54 is configured to encode / decode one or more parameters of the neural network or convolution selected for the current block 18 to / from the datastream. On the decoder side, the STRN tool 1101 may be configured to reconstruct the neural network or convolution based on one or more parameters of the neural network or convolution. For example, a subset of parameters or the full set of parameters may be transmitted at the beginning of the bitstream at a sequence of pictures or at a random access point. According to one embodiment, parameter updates, i.e., updates of parameters indicative of the selected neural network or convolution, may be transmitted in the datastream. For example, a full update, e.g., a new full set of parameters, and / or partial updates, e.g., only biases, only weights, only parameters of one or more specific layers, etc., and / or differential updates, e.g., only corrections to current parameter values, may be transmitted in the datastream.

[0129] FIG. 8 illustrates one embodiment of a decoder 54 / encoder 14 having a LIC tool 1101 as a first predetermined decoding / encoding tool, which may include the features and / or functionality described with respect to the decoder 54 / encoder 14 of FIG.

[0130] The LIC tool 1101 currently has a spatial neighborhood 100 (100 102 , 100 104 and 1006), the inter prediction signal P inter,18 to obtain a post-processed inter-predicted signal for the current block 18. TIFF2026506249000052.tif715 is acquired. The inter-prediction signal P inter,18 is generated by an inter prediction tool 1102, which may correspond to a second predetermined decoding / encoding tool, for example. The current block 18 is a post-processed inter prediction signal TIFF2026506249000053.tif715 and the prediction residual signal R currently associated with block 18. The block to which the LIC tool is applied may be referred to below as the LIC block.

[0131] The LIC tool 1101 determines or adapts the scaling and offset values ​​based on neighboring signals and the inter-predicted signal P of the current block 18. inter,18 By using the scaling and offset values ​​to post-process the inter-predicted signal P of the current block 18, inter,18 The video decoder 54 / encoder 14 is configured to post-process the current block 18, e.g., using the neighbor signals to derive scaling and offset values ​​to adjust the luminance of the current block 18, e.g., an inter-predicted block, to the luminance of its above and left neighbors, e.g., reconstructed samples.

[0132] Inter prediction signal P of current block 18 inter,18 The neighboring signal for post-processing of the first spatial neighborhood 100 may be generated as described with respect to FIG. 7, except that the sub-block 102 represents an LIC block and thus the inter-predicted signal within the spatial neighborhood 100 (first spatial neighborhood 100 102 and / or the second spatial neighborhood 100 104The only difference is that the .DELTA..times ...

[0133] This special generation of neighborhoods allows for parallelization of the coding process and therefore increases coding efficiency. For example, excluding or substituting intra prediction signals within neighborhood 100 of current block 18 allows for processing LIC blocks independently from the processing of intra blocks and / or in parallel with intra blocks. Furthermore, the use of prediction signals of LIC blocks within neighborhood 100 that are not post-processed by LIC tool 1101, i.e., P, rather than P*, allows for processing all LIC blocks of picture 10 in parallel.

[0134] FIG. 9 shows an embodiment of a decoder 54 / encoder 14 having a CIIP tool 1101 as the first predetermined decoding / encoding tool.

[0135] The CIIP tool 1101 currently has a spatial neighborhood 100 (100 102 , 100 104 and 1006), the inter-intra predicted signal P CIIP,18 The current block 18 is configured to generate an inter-intra predicted signal P CIIP,18 and the prediction residual signal R currently associated with the block 18. The block to which the CIIP tool is applied may be referred to as a CIIP block in the following.

[0136] The CIIP tool 1101 may generate an inter-intra predicted signal P for the current block 18 using, for example, inter prediction (see, for example, the inter portion 116 of the CIIP tool 1101) and intra prediction (see, for example, the intra portion 118 of the CIIP tool 1101). CIIP,18The CIIP tool is configured to generate an inter-intra predicted signal P for the current block 18 by, for example, a weighted combination of a planar intra predictor (e.g., using a neighboring signal / signal in the vicinity) and a motion-compensated temporal predictor, i.e., an inter predictor of, for example, a selected merging candidate. CIIP,18 Generate.

[0137] 9 exemplarily illustrates a picture area of ​​a current picture 10 having a CIIP block 102, an inter-predicted block 104, and an intra-predicted block 106 located adjacent to a current block 18, i.e., to the left and above the current block 18. The CIIP block 102 is located in a first spatial neighborhood 100. 102 The current block 18 overlaps with the spatial neighborhood 100 at 104 The current block 18 overlaps with the spatial neighborhood 100 at 106 The CIIP block 102 overlaps with the spatial neighborhood 100 of the current block 18 at CIIP , 102 and the prediction residual signal in the CIIP block 102, and the inter prediction block 104 can be reconstructed by a sample-by-sample combination of the inter prediction signal P inter,104 and the prediction residual signal R in the inter prediction block 104, and the intra prediction block 106 can be reconstructed by a sample-by-sample combination of the intra prediction signal P intra and the prediction residual signal in the intra prediction block 106 .

[0138] The plurality of decoding / encoding tools of the decoder 54 / encoder 14 may comprise one or more inter prediction tools, e.g., as one or more second predetermined decoding / encoding tools, configured to generate an inter prediction signal, e.g., as a contribution signal of one or more second predetermined decoding / encoding tools. inter,104 is an inter-prediction signal of one or more inter-prediction tools. The plurality of decoding / encoding tools of the decoder 54 / encoder 14 may additionally or alternatively include one or more intra-prediction tools, e.g., as one or more third predetermined decoding / encoding tools or one or more fourth predetermined decoding / encoding tools, configured to generate an intra-prediction signal, e.g., as a contribution signal of one or more third predetermined decoding / encoding tools. The intra-prediction signal P of the intra-prediction block 106 intra are intra prediction signals of one or more intra prediction tools.

[0139] In the following, the generation of neighborhood signals in the spatial neighborhood 100 of the current block 18 for intra prediction by the CIIP tool 1101 is described in more detail.

[0140] Spatial neighborhood 100 (first spatial neighborhood portion 100 102 For example, in the case of a CIIP block (see CIIP block 102) that overlaps with a first spatial neighborhood (see CIIP block 102), each inter prediction signal in the spatial neighborhood generated by the inter unit 116 of the CIIP tool 1101 is 102 The inter-intra prediction signal P CIIP,102 In other words, the video decoder 54 / encoder 14 may use the inter-intra predicted signal P CIIP,102 , for example, a contribution signal of a third predetermined decoding / encoding tool corresponding to a first predetermined decoding / encoding tool is calculated within a spatial neighborhood, i.e., a first spatial neighborhood portion 100 102 The inter-prediction signal P generated by the CIIP tool 1101 in inter,102, for example, by substituting a substitution signal generated independently of the spatial signal interdependence. 102 The prediction residual signal R in may be ignored in generating the neighboring signals.

[0141] No special constraints apply to inter-predicted blocks, such as the inter-predicted block 104. For inter-predicted blocks, the respective inter-predicted signals P inter,104 can be used to generate the neighborhood signal. The video decoder 54 / encoder 14 may, for example, use the second spatial neighborhood 100 when generating the neighborhood signal. 104 In the inter prediction signal P inter,104 to ignore the prediction residual signal R or to use the inter prediction signal P inter,104 and the prediction residual signal R.

[0142] Spatial neighborhood 100 (third spatial neighborhood portion 100 106 For intra-predicted blocks (see, e.g., intra-prediction block 106) that overlap with neighboring signals, the respective intra-predicted signals are used in generating the neighboring signals. TIFF2026506249000054.tif718) 122. Also, a third spatial neighborhood 100 106 The prediction residual signal R in the current block 18 may be ignored when generating the neighboring signals. Configured to use TIFF2026506249000055.tif718. Enhanced inter prediction signal TIFF2026506249000056.tif718 is, for example, the third spatially neighboring portion 100 of the inter prediction signal of the current block 18 generated by the inter portion 116 of the CIIP tool 1101. 106 and the third spatial neighborhood 100 106corresponds to the portion of the intra-predicted block 106 that overlaps with the spatial neighborhood 100 of the current block 18. In other words, for example, samples whose sample-by-sample combination for the derivation of the reconstructed signal 58 involves intra-predicted signals, i.e., contribution signals of one or more third predetermined decoding / encoding tools in the spatial neighborhood, may be replaced 122 by inter-prediction, e.g., samples generated by the inter portion 116 of the CIIP tool 1101. The inter-predicted signal P inter,18 is a part 100 of a spatial neighborhood 100 in which the derivation of the reconstructed signal 58 includes intra-predicted signals, e.g., contribution signals of one or more third predetermined decoding / encoding tools within the spatial neighborhood. 106 and this extended inter-prediction signal TIFF2026506249000057.tif718 can serve as a replacement signal.

[0143] Above, the neighboring blocks (see 102, 104, and 106) are discussed individually with respect to which signals associated with each block are considered for the generation of the neighboring signals. However, it is clear that the entire spatial neighborhood 100 is considered in the generation. Thus, for example, within the spatial neighborhood, all inter-predicted signals are considered, i.e., the inter-predicted signals of the inter-predicted blocks and the inter-predicted signals generated by the inter portion 116 of the CIIP tool 1101 for the CIIP blocks, and all intra-predicted signals of the intra-predicted blocks are replaced by the extended inter-predicted signals that are considered instead 122. Optionally, a deblocking filter can be applied within the spatial neighborhood 100, thereby eliminating edges between areas containing different signal types, e.g., the second spatial neighborhood portion 100. 104 and a third spatial neighborhood 100 106 The edges between are smoothed or reduced.

[0144] This special generation of neighborhoods allows for parallelization of the coding process and therefore increases coding efficiency. For example, permuting the intra-prediction signals within neighborhood 100 of current block 18 allows for processing CIIP blocks independently of and / or in parallel with the processing of intra-blocks. Furthermore, by using only the inter-prediction signals of CIIP blocks and ignoring the intra-prediction signals of CIIP blocks within neighborhood 100, it becomes possible to process all CIIP blocks of picture 10 in parallel.

[0145] FIG. 10 shows an embodiment of a decoder 54 / encoder having an RSP tool 1101 as the first predetermined decoding / encoding tool.

[0146] The RSP tool 1101 is currently searching the spatial neighborhood 100 (100 102 , 100 104 and 1006), the prediction residual signal R 18 The current block 18 is configured to generate a prediction signal P, for example an inter-prediction signal, an intra-prediction signal, or an inter-intra-prediction signal, and a prediction residual signal R of the current block 18. 18 The block to which the RSP tool is applied may be referred to as an RSP block hereinafter.

[0147] The RSP tool 1101 may, for example, derive residual values ​​of the current block 18 from the data stream and predict the sign of the derived residual values ​​based on neighboring signals in a spatial neighborhood of the current block, thereby generating a predicted residual signal R of the current block 18 as a contribution signal of the first predetermined decoding / encoding tool. 18 The RSP tool 1101 is configured to generate a predicted residual signal R by, for example, estimating the sign of the residual block from neighboring signals. 18Optionally, the RSP tool 1101 is configured to generate a predicted residual signal R for the current block 18 by deriving the residual values ​​of the current block 18 and the difference between the predicted and true signs of the residual values, e.g., from the data stream, predicting the signs of the residual values ​​based on neighboring signals in a spatial neighborhood of the current block to obtain the predicted signs, and reconstructing the signs by combining the predicted signs and the difference. 18 If the code is well estimated, the difference will tend to zero and will be efficiently entropy coded by CABAC.

[0148] 10 exemplarily illustrates a picture area of ​​a current picture 10 having an RSP block 102, an inter-predicted block 104, and an intra-predicted block 106 located adjacent to, i.e., to the left and above, the current block 18. The RSP block 102 is located in a first spatial neighborhood 100. 102 The current block 18 overlaps with the spatial neighborhood 100 at 104 The current block 18 overlaps with the spatial neighborhood 100 at 106 The RSP block 102 overlaps with the spatial neighborhood 100 of the current block 18 at 102 , for example, can be reconstructed by a sample-by-sample combination of an inter-prediction signal, an intra-prediction signal, or an inter-intra-prediction signal with a prediction residual signal generated by, for example, an RSP tool 1101 in the RSP block 102, and the inter-prediction block 104 can be reconstructed by a sample-by-sample combination of an inter-prediction signal P inter,104 and the prediction residual signal R in the inter prediction block 104, and the intra prediction block 106 can be reconstructed by a sample-by-sample combination of the intra prediction signal P intra and the prediction residual signal in the intra prediction block 106 .

[0149] The plurality of decoding / encoding tools of the decoder 54 / encoder 14 may comprise one or more inter prediction tools, e.g., as one or more second predetermined decoding / encoding tools, configured to generate an inter prediction signal, e.g., as a contribution signal of one or more second predetermined decoding / encoding tools. inter,104 is an inter-prediction signal of one or more inter-prediction tools. The plurality of decoding / encoding tools of the decoder 54 / encoder 14 may additionally or alternatively include one or more intra-prediction tools, e.g., as one or more third predetermined decoding / encoding tools or one or more fourth predetermined decoding / encoding tools, configured to generate an intra-prediction signal, e.g., as a contribution signal of one or more third predetermined decoding / encoding tools. The intra-prediction signal P of the intra-prediction block 106 intra are intra prediction signals of one or more intra prediction tools.

[0150] In the following, the generation of the neighborhood signal in the spatial neighborhood 100 of the current block 18 for the RSP tool 1101 is described in more detail.

[0151] No special constraints apply to inter-predicted blocks, such as inter-predicted block 104. The following considerations apply regardless of whether current block 18 is an inter-predicted block or not. For inter-predicted blocks, each inter-predicted signal P inter,104 can be used to generate the neighborhood signal. The video decoder 54 / encoder 14 may, for example, use the second spatial neighborhood 100 in generating the neighborhood signal. 104 In the case where the inter-predicted block 104 is an RSP block, the inter-predicted signal P inter,104 and ignores the prediction residual signal R. The video decoder 54 / encoder 14 may, for example, use the second spatial neighborhood 100 when generating the neighborhood signal. 104 If the inter-predicted block 104 is not an RSP block, the inter-predicted signal Pinter,104 and the prediction residual signal R.

[0152] Spatial neighborhood 100 (third spatial neighborhood portion 100 106 For intra-predicted blocks (see, for example, intra-predicted block 106) that overlap with the current block 18, if the current block 18 is an inter-predicted block, then when generating the neighboring signals, the respective intra-predicted signal is replaced with a replacement signal ( TIFF2026506249000058.tif718). Also, the third spatial neighborhood 100 106 The prediction residual signal R in the current block 18 may be ignored when generating the neighboring signals. Configured to use TIFF2026506249000059.tif718. Enhanced inter prediction signal TIFF2026506249000060.tif718 is, for example, a third spatial neighboring portion 100 of the inter prediction signal of the current block 18 generated by the inter coding tool 1102. 106 and the third spatial neighborhood 100 106 corresponds to the portion of the intra-predicted block 106 that overlaps with the spatial neighborhood 100 of the current block 18. In other words, for example, samples whose sample-by-sample combination for the derivation of the reconstructed signal 58 includes intra-predicted signals, i.e., contribution signals of one or more third predetermined decoding / encoding tools within the spatial neighborhood, may be replaced with samples generated by inter prediction. inter,18 is a part 100 of a spatial neighborhood 100 in which the derivation of the reconstructed signal 58 includes intra-predicted signals, e.g., contribution signals of one or more third predetermined decoding / encoding tools within the spatial neighborhood. 106 and this extended inter-prediction signal TIFF2026506249000061.tif718 can serve as a replacement signal.

[0153] Alternatively, if the current block 18 is not an inter-predicted block, The video decoder 54 / encoder 14 may be configured to disable the RSP tool 1101 for the current block 18, and to disable the RSP tool 1101 for the current block 18 if an intra-predicted block (e.g., see intra-predicted block 106) is in the spatial neighborhood 100 of the current block 18 (e.g., the third spatial neighborhood 100 106 (see or The video decoder 54 / encoder 14 generates a third spatial neighborhood 100 106 and uses the motion vector to estimate the motion vector within the spatial neighborhood, i.e., the third spatial neighborhood 100 106 Inter prediction signal P inter,106 and in generating the neighboring signal, the determined inter-prediction signal P inter,106 is used as a replacement signal to generate a third spatial neighborhood 100 106 The predictive residual signal R may be configured to ignore the predictive residual signal R in the

[0154] Spatial neighborhood 100 (first spatial neighborhood portion 100 102 For an RSP block (see RSP block 102) that overlaps with a first spatial neighborhood (see RSP block 102), if the RSP block 102 is an inter-predicted block, each inter-predicted signal in the spatial neighborhood is used to generate the neighborhood signal. 102 The prediction residual signal R within the RSP block 102 may be ignored in generating the neighborhood signal. Alternatively, if the RSP block 102 is not an inter-predicted block, the RSP block 102 is considered like the intra-predicted block 106 described above in generating the neighborhood signal.

[0155] Above, the neighboring blocks (see 102, 104, and 106) are discussed individually with respect to which signals associated with each block are considered for the generation of the neighboring signal. However, it should be clear that the entire spatial neighborhood 100 is considered in the generation. Optionally, a deblocking filter can be applied within the spatial neighborhood 100 to eliminate edges between areas containing different signal types, e.g., the second spatial neighborhood 100. 104 and a third spatial neighborhood 100 106 The edges between are smoothed or reduced.

[0156] This special generation of neighborhoods allows for parallelization of the coding process and therefore increases coding efficiency. For example, replacing intra-prediction signals within neighborhood 100 of current block 18 or disabling RSP tool 1101 allows for processing RSP blocks independently from and / or in parallel with the processing of intra-blocks when one or more intra-blocks are included in neighborhood 100 or overlap with neighborhood 100. Furthermore, ignoring the prediction residual signals of inter-prediction blocks included in neighborhood 100 or overlapping neighborhood 100 allows for parallel processing of all RSP blocks of picture 10.

[0157] FIG. 11 shows an embodiment of a decoder 54 / encoder 14 having a TM tool 1101 as the first predetermined decoding / encoding tool.

[0158] The TM tool 1101 is currently working on a spatial neighborhood 100 (100 102 , 100 104 and 1006), a predicted signal P 18 , for example an inter-predicted signal, an intra-predicted signal or an inter-intra-predicted signal. 18and the prediction residual signal R of the current block 18. A block to which the TM tool is applied may be referred to below as a TM block or a TM prediction block. 18 is sometimes referred to as a TM predicted signal, and the predicted residual signal is sometimes referred to as a TM predicted residual signal.

[0159] The TM tool 1101 calculates, for example using template matching, for the current block 18, a predicted signal P 18 , where the neighboring signals within the spatial neighborhood 100 of the current block 18 represent the template for template matching.

[0160] 11 exemplarily illustrates a picture area of ​​a current picture 10 having an inter-predicted block 104 and an intra-predicted block 106 disposed adjacent to the current block 18, i.e., to the left and above the current block 18, as described with reference to FIGS. 9 and 10, and additionally having a TM block 102 disposed adjacent to the current block 18. The TM block 102 is a first spatial neighborhood 100. 102 TM block 102 overlaps with the spatial neighborhood 100 of current block 18 at . The TM block 102 is, for example, a predicted signal P 102 , for example, can be reconstructed by a sample-by-sample combination of an inter-prediction signal, an intra-prediction signal or an inter-intra-prediction signal with a prediction residual signal in the TM block 102 .

[0161] In the following, the generation of the neighborhood signal in the spatial neighborhood 100 of the current block 18 for the TM tool 1101 is described in more detail.

[0162] No special constraints apply to inter-predicted blocks, such as the inter-predicted block 104. For inter-predicted blocks, the respective inter-predicted signals P inter,104can be used to generate the neighborhood signal. The video decoder 54 / encoder 14 may, for example, use the second spatial neighborhood 100 when generating the neighborhood signal. 104 In the inter prediction signal P inter,104 to ignore the prediction residual signal R or to use the inter prediction signal P inter,104 and the prediction residual signal R.

[0163] The video decoder 54 / encoder 14 determines whether a TM block (see, e.g., TM block 102) is in a spatial neighborhood 100 of the current block 18 (first spatial neighborhood 100 102 11. If the TM tool 1101 of the current block 18 overlaps with the TM tool 1101 of the current block 18, the TM tool 1101 may be configured to be disabled.

[0164] The video decoder 54 / encoder 14 may additionally or alternatively determine whether an intra-predicted block (see, e.g., intra-predicted block 106) is within a spatial neighborhood 100 of the current block 18 (e.g., a third spatial neighborhood 100 106 11. If the TM tool 1101 for the current block 18 overlaps with the current block 18 (see ).

[0165] Alternatively, instead of disabling, the video decoder 54 / encoder 14 may select the third spatial neighborhood 100 106 (and / or the first spatial neighborhood 100 102 ) and use the motion vector to estimate the motion vector within the spatial neighborhood 100, i.e., the third spatial neighborhood portion 100 106 (and / or the first spatial neighborhood 100 102 ) in the inter-prediction signal P inter and in generating the neighboring signal, the determined inter-prediction signal P inter may be configured to be used as a replacement signal, for example. 106 Within (and / or within the first spatial neighborhood 100 102 The prediction residual signal R in the neighboring signal may be ignored in generating the neighboring signal.

[0166] Above, the neighboring blocks (see 102, 104, and 106) are discussed individually with respect to which signals associated with each block are considered for the generation of the neighboring signal. However, it should be clear that the entire spatial neighborhood 100 is considered in the generation. Optionally, a deblocking filter can be applied within the spatial neighborhood 100 to eliminate edges between areas containing different signal types, e.g., the second spatial neighborhood 100. 104 and a third spatial neighborhood 100 106 The edges between are smoothed or reduced.

[0167] This special generation of neighborhoods allows for parallelization of the coding process and thus increases coding efficiency. For example, replacing the intra prediction signal within neighborhood 100 of current block 18 or disabling TM tool 1101 allows for processing TM blocks independently from and / or in parallel with the processing of intra blocks if one or more intra blocks are included in or overlap neighborhood 100. Furthermore, disabling TM tool 1101 allows for parallel processing of all TM blocks of picture 10 if one or more TM blocks are included in or overlap neighborhood 100.

[0168] TM is a texture synthesis technique used in digital image processing and can be applied to intra-prediction and inter-prediction. The patch of already decoded / encoded samples above and to the left of the current block 18 is called a template. TM finds the best match for the template in the reconstructed frame by minimizing the error between the template and its match, usually measured as the sum of squared differences (SSD). Finally, in TM-based prediction, the TM block associated with the error-minimizing template match is used as a prediction for the current block. This TM-based prediction approach does not require side information to generate the corresponding prediction signal at the decoder / encoder, since the same search process is performed there as well.

[0169] Figure 12 illustrates one embodiment of a video decoder 54 / encoder 14 configured to decode / encode video from / to a data stream using block-based prediction and transform-based predictive residual coding. Figure 12 illustratively illustrates a video decoder 54 / encoder 14 configured to perform block-based prediction by using motion-compensated prediction for inter-predicted blocks, by using intra-prediction for intra-predicted blocks, and by using inter-intra prediction for inter-intra predicted blocks, i.e., CIIP blocks. However, it will be apparent that the video decoder 54 / encoder 14 may alternatively be configured to perform block-based prediction by using one or more of motion-compensated prediction, intra-prediction, and inter-intra prediction. Optionally, the video decoder 54 / encoder 14 is configured to perform transform-based predictive residual coding by using residual code prediction.

[0170] 12 exemplarily shows a picture area 11 or picture portion of a current picture 10 having an intra-predicted block 18, an inter-predicted block 102, and an inter-intra-predicted block, i.e., a CIIP block 104. The block indicated by reference numeral 106 can be an intra-predicted block, an inter-predicted block, or a CIIP block. Because residual code prediction is enabled for the block, the block is referred to hereinafter as an RSP block 106. The RSP block 106, the inter-predicted block 102, and the CIIP block 104 represent neighboring blocks of the intra-predicted block 18, i.e., they are located adjacent to, i.e., to the left and above, the intra-predicted block 18. This arrangement of blocks 102, 104, and 106 is for illustrative purposes only, and it will be apparent that the block types of the neighboring blocks that overlap with the spatial neighborhood of intra-predicted block 18, i.e., intra-predicted blocks, inter-predicted blocks, CIIP blocks, or RSP blocks, depend on the prediction types supported by video decoder 54 / encoder 14, i.e., motion-compensated prediction, intra-prediction, and inter-intra prediction, and / or the residual coding type, i.e., residual code prediction.For example, if the video decoder 54 / encoder 14 is configured to perform block-based prediction through the use of motion compensated prediction and intra prediction, the blocks 102, 104, and 106 may be intra-predicted blocks and / or inter-predicted blocks, or if the video decoder 54 / encoder 14 is configured to perform block-based prediction through the use of intra prediction and inter-intra prediction, the blocks 102, 104, and 106 may be intra-predicted blocks and / or CIIP blocks, or if the video decoder 54 / encoder 14 is configured to perform block-based prediction using intra prediction and transform-based predictive residual coding using residual code prediction. If video decoder 54 / encoder 14 is configured to perform block-based prediction using motion compensation prediction and intra prediction and transform-based predictive residual coding using residual code prediction, blocks 102, 104, and 106 may be intra-predicted blocks and / or RSP blocks, and the RSP blocks may belong to inter-predicted blocks, or if video decoder 54 / encoder 14 is configured to perform block-based prediction using motion compensation prediction and intra prediction and transform-based predictive residual coding using residual code prediction, blocks 102, 104, and 106 may be intra-predicted blocks, inter-predicted blocks, and RSP blocks, and the RSP blocks may belong to inter-predicted blocks and / or intra-predicted blocks, e.g., inter-predicted blocks and intra-predicted blocks may include RSP blocks.

[0171] In the embodiment shown in FIG. 12, the inter-predicted block 102 is a first spatial neighborhood 100 102 The CIIP block 104 overlaps with the spatial neighborhood 100 of the intra-predicted block 18 at 104 The RSP block 106 overlaps with the spatial neighborhood 100 of the intra-predicted block 18 at 106 overlaps with the spatial neighborhood 100 of the intra-predicted block 18 at

[0172] The CIIP block 104 receives the inter-intra predicted signal P CIIPand the prediction residual signal of the CIIP block 104, which can be obtained, for example, from the data stream. CIIP is the intra-predicted signal associated with the CIIP block 104 and the inter-predicted signal P inter,104 For example, the inter-intra predicted signal P CIIP may be generated by the CIIP tool 1101 of FIG. 9 (P CIIP,18 ), or may be generated by the first decoding tool 1101 / first encoding tool of FIG. 6, for example using the generator 114 (see See TIFF2026506249000062.tif77).

[0173] The RSP block 106 receives the predicted signal P 106 , for example, may be reconstructed by a sample-by-sample combination of an inter-predicted signal, an intra-predicted signal, or an inter-intra-predicted signal with a predicted residual signal of the RSP block 106 generated using residual code prediction. For example, the video decoder 54 may include a residual code prediction tool configured to derive residual values ​​of the RSP block 106 from the data stream and predict the signs of the residual values ​​based on the spatial neighborhood of the RSP block 106. The encoder 14 may also, for example, generate a predicted signal P 106and the original signal of the RSP block 106, determine a residual value for the RSP block 106, encode it into the data stream, and include a residual code prediction tool configured to predict the sign of the residual value based on a spatial neighborhood of the RSP block 106, and optionally encode the difference between the predicted code and the actual code into the data stream. Optionally, the residual code prediction tool of the decoder 54 is further configured to derive the difference from the data stream and reconstruct the sign of the residual value by combining / summing the predicted code with the difference. Optionally, the predicted residual signal R of the RSP block 106 is calculated by the RSP tool 1101 of FIG. 10 using (R 18 ), or by the first decoding tool 1101 / first encoding tool of FIG. 6, for example using the generator 114 ( TIFF2026506249000063.tif77) can be generated.

[0174] The video decoder 54 / encoder 14 receives the inter-predicted signal P inter,102 The video decoder 54 / encoder 14 may be configured to inter-predict the inter-predicted block 102 to obtain the inter-predicted signal P of the inter-predicted block 102. The video decoder 54 / encoder 14 may include post-processing tools, and the video decoder 54 / encoder 14 may be configured to apply the post-processing tools to the inter-predicted block 102. The post-processing tools may be configured to apply the post-processing tools to the inter-predicted block 102 to obtain the inter-predicted signal P of the inter-predicted block 102. inter,102 is post-processed to obtain a post-processed inter-prediction signal The inter prediction block 102 is configured to obtain a post-processed inter prediction signal TIFF2026506249000064.tif717. TIFF2026506249000065.tif717 and a prediction residual signal R, which can be obtained from the data stream, for example. Optionally, the post-processing tool can correspond to the STRN tool 1101 of Figure 7, the LIC tool 1101 of Figure 8 or the first decoding tool 1101 / first encoding tool of Figure 6, using, for example, a post-processor 112.

[0175] Video decoder 54 / encoder 14 is configured to intra-predict intra-prediction block 18, i.e., a current block belonging to the intra-prediction block, using neighboring signals in a spatial neighborhood 100 of intra-prediction block 18.

[0176] In the following, the generation of the neighborhood signal in the spatial neighborhood 100 of the intra-predicted block 18 is described in more detail.

[0177] Spatial neighborhood 100 (first spatial neighborhood portion 100 102 For a post-processed inter-predicted block such as the inter-predicted block 102 that overlaps with the inter-predicted signal P inter,102 is the post-processed inter-prediction signal of the inter-prediction block 102 TIFF2026506249000066.tif717 may be used to generate the neighboring signal, i.e., the inter-predicted signal P inter,102 may be used in its non-post-processed version by a post-processing tool for the generation of the neighboring signal, i.e., the post-processed inter-predicted signal TIFF2026506249000067.tif717 is an inter-prediction signal P of the inter-prediction block 102 for generating the neighboring signal. inter,102 Optionally, the video decoder 54 / encoder 14 selects a region within the spatial neighborhood 100, i.e., a first spatial neighborhood portion 100, for generating the neighborhood signal. 102 Inter prediction signal P inter,102and the prediction residual signal R within the spatial neighborhood 100.

[0178] Spatial neighborhood 100 (second spatial neighborhood portion 100 104 For a CIIP block (see CIIP block 104) that overlaps with an inter-predicted signal P inter,104 is the inter-intra predicted signal P CIIP In other words, video decoder 54 / encoder 14 may use the inter-intra predicted signal P CIIP The inter prediction signal P inter,104 In other words, video decoder 54 / encoder 14 may be configured to generate the neighborhood signal by replacing the inter-predicted signal P inter,104 and not use the intra prediction signal of the inter-intra prediction, i.e., the intra prediction signal of the inter-intra prediction associated with the CIIP block is ignored or left in the generation of the neighborhood signal.

[0179] Like the RSP block 106, the spatial neighborhood 100 (third spatial neighborhood portion 100 106 For RSP blocks that overlap with the spatial neighborhood 100, the predicted signal P 106 may be used to generate the neighboring signal. 106 can be used without being combined with the predicted residual signal R in the RSP-block 106. The predicted residual signal that is ignored or left for the generation of the neighboring signals is generated using a residual code prediction tool.

[0180] Additionally, it is possible for an intra block to at least partially overlap with its spatial neighborhood 100, and the intra prediction tool 110 may be configured to use the reconstructed intra signal within the overlap region of the intra block and the neighborhood 100, i.e., a sample-by-sample combination of the intra prediction signal and the prediction residual signal, or to use the intra prediction signal within the overlap region of the intra block and the neighborhood 100 and ignore the prediction residual signal of the intra block.

[0181] Above, the neighboring blocks (see 102, 104, and 106) are discussed individually with respect to which signals associated with each block are considered for generating the neighboring signal. However, it should be clear that the entire spatial neighborhood 100 is considered when generating the neighboring signal. Optionally, a deblocking filter can be applied within the spatial neighborhood 100, thereby eliminating edges between areas containing different signal types, e.g., the second spatial neighborhood 100. 104 and a third spatial neighborhood 100 106 and / or the first spatial neighborhood 100 102 and a second spatial neighborhood 100 104 The edges between are smoothed or reduced.

[0182] This special generation of neighborhoods allows for parallelization of the coding process and thus improves coding efficiency. For example, using the prediction signals of post-processed blocks, such as STRN blocks and / or LIC blocks in the neighborhood 100, that are not post-processed by the respective post-processing tools 1101, i.e., in versions using P instead of P*, allows for intra prediction of intra blocks to be performed in parallel with the post-processing of the post-processed blocks. For example, by using only the inter prediction signals of CIIP blocks and ignoring the intra prediction signals of CIIP blocks in the neighborhood 100, intra prediction of intra blocks can be performed in parallel with the prediction of CIIP blocks. For example, by using only the prediction signals of RSP blocks and ignoring the prediction residual signals of RSP blocks in the neighborhood 100, intra prediction of intra blocks can be performed in parallel with residual code prediction.

[0183] 12 exemplarily illustrates the generation of a neighborhood signal for an intra-prediction block 18. However, it is clear that a neighborhood signal for a CIIP block can be correspondingly generated, where the neighborhood signal is used to generate an intra-prediction signal associated with the CIIP block and combined with the inter-prediction signal associated with the CIIP block, for example, by weighted combining, to obtain an inter-intra-prediction signal for the CIIP block. In other words, the intra-prediction block 18 may instead be a CIIP block, and an inter-intra prediction tool may be used to generate the inter-intra-prediction signal instead of the intra prediction tool 110.

[0184] Figure 13 shows one embodiment of a picture processing tool 1101 that may correspond to the first predetermined decoding / encoding tool described with respect to Figures 6 and 7. Thus, the decoder 54 or encoder 14 described with respect to Figures 6 and 7 may include the features and / or functionality described with respect to the picture processing tool 1101 below.

[0185] The picture processing tool 1101 includes a neural network or convolution indicated by reference numeral 130 and is configured to polyphase-decompose 140 the luma samples of the picture portion 11 into polyphase components, to obtain a matrix (see 1421-1424) for each polyphase component, and to form 144 a tensor 146 by cascading the matrices 1421-1424 of the polyphase components. The picture processing tool 1101 is configured to associate the matrices 1421-1424 as different channels, to subject tensor 146 to the neural network or convolution (see 130), to obtain an output tensor 148 consisting of a concatenation of the output matrices, one output matrix per polyphase component, and to form a processed picture portion 11′ based on the output tensor 148 by inverse polyphase decomposition 150, i.e., to rearrange the samples accordingly.

[0186] FIG. 13 exemplarily illustrates a polyphase division of the picture portion 11 into four polyphase components. However, it is clear that the picture processing tool 1101 may alternatively be configured to polyphase divide the picture portion 11 into a different number of polyphase components. The luma samples of the picture portion 11 have a two-dimensional arrangement along a first direction x and a second direction y, where the second direction y is perpendicular to the first direction x. In the polyphase division, for example, the luma samples are alternately divided in the first x direction and the second y direction into different ones of the polyphase components. For example, the luma samples are divided along the first direction x and the second direction y into even and odd samples, e.g., even and odd with respect to the position index of the luma samples within the picture portion 11, to obtain the polyphase components. The input signal is polyphase divided horizontally and vertically (see 130), for example, before processing with a neural network or convolution.

[0187] The picture portion comprises, for example, a block 18 of a picture with its spatial neighborhood 100. The picture processing tool 1101 is configured to split the luma samples of the block 18 and the spatial neighborhood 100 into polyphase components in a polyphase splitting to obtain matrices (see 1421 to 1424) for each polyphase component. The processed picture portion 11′ has, for example, the same dimensions as the picture portion 11. The picture processing tool 1101 is configured, for example, to combine, sum or add a picture portion with the processed picture portion to obtain an intermediate signal and crop the intermediate signal to obtain the post-processed picture portion, or vice versa, i.e., to first perform a crop and then a combination. In a cropping, for example, a portion associated with the spatial neighborhood 100 is cut out.

[0188] The neural network or convolution (see 130) may include features and / or functionality as described with respect to Figure 15. In the embodiment of Figure 15, the input tensor 146 is larger because the picture portion 11 of the current picture, the corresponding picture portion 111 in the first reference picture, and the corresponding picture portion 112 in the second reference picture are polyphase-split (144). Nevertheless, the concepts described with respect to Figure 15 are also applicable to the picture processing tool 1101 of Figure 13.

[0189] The picture processing tool 1101 may be a post-processing tool for inter-predicted blocks, see also the examples described with respect to Figure 7, Figure 14 or Figure 15. In this case, the picture portion 11 may be an inter-prediction of a picture block received from an inter-prediction tool of a video decoder / encoder. For example, the STRN tool of the video decoder 54 / encoder 14 described with respect to Figure 7 receives an inter-prediction signal P of the current block 18 from the inter-prediction tool 1102. inter,18 The picture processing tool 1101 may receive the

[0190] The inter-prediction of a picture block may be obtained by uni-prediction, bi-prediction, etc., and at least one predictor in the reference picture is used for the inter-prediction. The predictor represents, for example, a corresponding block, i.e., a block similar to the picture block in the reference picture. The picture processing tool 1101 receives the inter-prediction of the picture block together with one or more predictors and can polyphase-decompose 140 the inter-prediction and one or more predictors to obtain a tensor 146. For example, the tensor 146 may be formed from 12 matrices of polyphase components when the inter-prediction represents normal bi-prediction, i.e., has two predictors. In the case of uni-prediction, the picture processing tool 1101 can be configured to fill the input with a double uni-prediction signal because the input is the inter-prediction of the picture block and has twice the predictor, i.e., a uni-predictor. The tensor 146 is also formed from 12 matrices of polyphase components. The same polyphase decomposition applied to the picture portion 11 is also applied to the one or more predictors.

[0191] Optionally, the picture portion 11 may be an inter-prediction, such as an intra-prediction, an inter-intra-prediction, or a bi-prediction, prediction of a picture block with neighboring reconstructed (i.e., above / left) samples, for example with a boundary extension width B, i.e., with a spatial neighborhood 100. In the case of bi-prediction, it also includes two correspondingly extended component prediction signals, i.e., two predictors, at the input of the neural network or convolution (see 130). Generally speaking, if the picture portion 11 is intra-predicted, one or more predictors are accompanied by their respective spatial neighbors. Optionally, the picture portion 11 may be a prediction of a picture block with a constrained spatial neighborhood 100, i.e., a neighborhood signal 100′, as described with reference to FIGS. 6 and 7. Constraints may not be applied to the spatial neighborhood of one or more predictors involved in the inter-prediction.

[0192] The picture portion includes, for example, inter-predicted luma samples of block 18 of the picture together with the spatial neighborhood 100 of block 18. The picture processing tool 1101 is configured to, for example, in a per-polyphase division, divide the inter-predicted luma samples of block 18 and the luma samples of the spatial neighborhood 100 into polyphase components to obtain matrices (see 1421 to 1424) for each polyphase component, and to divide the luma samples of a corresponding reference picture portion including a corresponding block in the reference picture and spatial neighborhood of the corresponding block into polyphase components to obtain reference matrices for each polyphase component. The processed picture portion 11′ has, for example, the same dimensions as the picture portion 11. The picture processing tool 1101 is configured, for example, to combine, sum, or add the picture portion with the processed picture portion to obtain an intermediate signal and crop the intermediate signal to obtain the post-processed picture portion, or vice versa, i.e., to first perform the crop and then the combination. Cropping, for example, involves cutting out the portion associated with the spatial neighborhood 100 .

[0193] The picture processing tools 1101 can be configured to allow the picture portion 11 to correspond to one of multiple picture portion dimensions, for example, by restricting the convolution of the tensor 146 with the neural network kernel 132 or convolution (see 130) to the dimension of the picture portion 11 and using the same kernel 132 for each of the multiple picture dimensions. The neural network or convolution can be applied to picture portions 11 of different sizes and shapes, and to picture portions 11 associated with different quantization parameters, for example.

[0194] The neural network or convolution (see 130) may, for example, include only a convolution layer. The neural network or convolution (see 130) may, for example, include N layers. The neural network or convolution may, for example, be configured to perform a convolution 134 for each layer, followed by a rectified linear unit activation 136, except for the last layer of the N layers, where the rectified linear unit activation 136 is skipped.

[0195] Figure 14 illustrates one embodiment of a decoder 54 / encoder 14 configured to decode / encode video from / to data stream 12 using block-based prediction and transform-based predictive residual coding. The decoder 54 / encoder 14 is configured to perform block-based prediction using motion-compensated prediction, i.e., inter-prediction, controlled via a motion vector 200, as in the case of block 18. The motion vector 200 indicates, for example, a corresponding block 210 in reference picture 111 for a block in current picture 10. Figure 14 exemplarily illustrates three reference pictures 111, 112, and 113 for block 18 in current picture 10, with each corresponding block in each reference picture 111, 112, and 113 indicated by a corresponding motion vector. The motion vector 210 indicates the offset of the corresponding block 210 relative to a co-located block 220 of block 18 in reference picture 111. Video encoder 14 is configured to encode one or more motion vectors 200 for inter-predicted blocks such as block 18, and video decoder 54 is configured to derive one or more motion vectors 200 for inter-predicted blocks such as block 18 from the data stream.

[0196] The decoder 54 / encoder 14 is configured to apply a post-processing tool 1101 for post-processing the inter-predicted signal of a given inter-predicted block and to identify the given inter-predicted block from the inter-predicted blocks by excluding the following blocks from the given inter-predicted block:

[0197] For example, a first inter-predicted block according to the data stream 12 has one or more motion vectors 200 associated therewith whose number is zero, satisfying a first predetermined criterion. For example, the first predetermined criterion may be that the number must be at least one, i.e., if one or more of the motion vectors 200 associated with the inter-predicted block are zero motion vectors, the inter-predicted block represents one of the first inter-predicted blocks. Alternatively, the first predetermined criterion may be that the number is at least two, at least three, etc. A motion vector 200 that is zero, i.e., a zero motion vector, indicates that a co-located block 200 in a reference picture, e.g., 111, 112, or 113, is used as a predictor for the respective first inter-predicted block 18.

[0198] For example, a second inter-predicted block having one or more associated motion vectors 200, a number of which satisfy a second predetermined criterion according to the data stream 12 are full-pel motion vectors. For example, the second predetermined criterion may be that the number must be at least 1, i.e., if one or more of the motion vectors 200 associated with the inter-predicted block are full-pel motion vectors, the inter-predicted block represents one of the second inter-predicted blocks. Alternatively, the second predetermined criterion may be that the number must be at least 2, at least 3, etc.

[0199] -For example, a third inter-prediction block having one of a set of predetermined inter-prediction modes associated therewith according to data stream 12, the set of predetermined inter-prediction modes including one or more of a uni-prediction mode, a merge mode, and a bi-prediction mode using coding unit weights.

[0200] a fourth inter-predicted block whose block shape meets a third predetermined criterion;

[0201] - A fifth inter-predicted block for which the data stream signals a quantization parameter having a value that meets a fourth predetermined criterion.

[0202] This embodiment avoids gradual signal degradation based on the inventors' discovery that repeated application of post-processing tools 1101, e.g., tools using NNs, can lead to gradual signal degradation, which is particularly relevant for low-delay prediction structures.

[0203] Gradual signal degradation can be mitigated by constraining the set of coding modes that can be applied by post-processing tools 1101. For example, applying post-processing tools 1101 can be disabled for any of the following cases:

[0204] Motion vectors equal to zero (including / excluding affine blocks).

[0205] - Motion vectors with full-pel accuracy (or resulting in full-pel positions).

[0206] - Blocks without a signaled residual (i.e. coded block ag[cbf] equals 0).

[0207] A particular prediction mode (e.g., uni-prediction, a particular merge mode, or CU-weighted bi-prediction [BCW]).

[0208] - A specific slice (e.g., implicitly based on a temporal layer, or explicitly via additional syntax elements).

[0209] -Specific block shapes.

[0210] -A specific quantization parameter (QP) value.

[0211] According to one embodiment, the post-processing tool 1101 may comprise features and / or functionality as described with respect to the picture processing tool 1101 of Figure 13 and / or as described with respect to the first predetermined decoding / encoding tool 1101 of Figures 6, 7 and 8. The post-processing tool 1101 may be configured to post-process the inter-predicted signals of the blocks comprised in the predetermined inter-predicted block, for example as described with respect to the block referenced by reference number 18 in Figures 6, 7, 8, 13 and 15.

[0212] In the following, a description of the neural network 130 for enhanced inter prediction is provided. This section introduces details of the network architecture and training process. The neural network 130 described below is referred to as the STRN network.

[0213] Figure 15 shows an overview of the proposed STRN network architecture, based on the architecture of

[14] . The approach of the present invention is to extract inter-blocks of the current block 18 in VVC, e.g., P, such that the input and output of the neural network 130 (shown in the top left and top right of Figure 15, respectively) represent the interface between the video codec and the STRN domain. inter,18 The aim is to improve the predicted signal of

[0214] Given an inter-block of size WxH, e.g., current block 18, bi-prediction in VVC uses two reference pictures, i.e., L0 and L1 prediction signals, i.e., motion compensated reference blocks 181 and 182 of size WxH of the predictor, to obtain a prediction signal P inter,18 Therefore, the STRN input consists of the sample array of the current picture, i.e., picture portion 11, and the L0 and L1 prediction signals, i.e., picture portions 111 and 112. However, for STRN, the block size is extended by an L-shaped B sample-wide area along the top / left boundary, i.e., the spatial neighborhood (see 100, 1001 and 1002), to a size W B ×HB = (W+B) × (H+B) input arrays are obtained. For the L0 and L1 predicted signals, the extended motion compensated reference blocks, i.e., picture portions 111 and 112, are derived using the same motion vectors as for regular bi-prediction, so that the input arrays C1 and C3 contain additional (interpolated) prediction samples along the top / left boundary, i.e., spatio-temporal reference samples, i.e., samples in the respective spatial neighborhoods 1001 and 1002. For the current picture, the input array C2 contains the regular bi-prediction P of the current block 18 in the corresponding WxH area, i.e., P inter,18 and additional reconstructed samples in an L-shaped B-wide area along the top / left border, i.e., spatial reference samples, i.e., samples in the spatial neighborhood 100. Unlike

[14] , these reconstructed samples may be subject to certain constraints that allow the STRN block and the intra blocks to be decoded independently and in parallel; for example, referring to Figure 7, the intra prediction block 106 and the STRN block 102 may be decoded / encoded in parallel.

[0215] Together, the three input arrays form a tensor C = [C1C2C3], which is used to derive the actual input tensor 146 of the neural network 130 via a polyphase decomposition 144

[38] . B ×H B and C with elements (c;x;y), the polyphase components are W B and H B requires that is a multiple of 2 TIFF2026506249000068.tif44149, which is obtained by splitting it into even and odd samples along the x and y directions. Then, the four polyphase components are By combining as TIFF2026506249000069.tif742, the size An input tensor 146 of TIFF2026506249000070.tif1332 is formed. Note that this decomposition only rearranges the tensor elements, but does not change the number of elements or their values. In the context of deep learning, such a multiphase operation is also sometimes used for addressing different problems, such as addressing differently sampled color components, and is referred to as pixel (un)shuffling

[39] . In terms of neural network structure, the proposed STRN essentially consists of N convolutional layers. As shown in Figure 15, all layers, except the last one, perform convolutions 134 with a kernel size of 3 × 3, followed by a rectified linear unit (ReLU) activation function 136. The operations of these convolutional layers are performed using a weight matrix W k and the bias vector Using TIFF2026506249000071.tif727 TIFF2026506249000072.tif15116. The input to the first layer L0 can correspond to the 12x3x3 subtensor of C* at location (x;y), i.e., tensor 146. For each layer, the last layer L N The output of The operation of equation (2) is applied to all positions (x; y) using zero padding to preserve the block shape, so that the size of the image is TIFF2026506249000073.tif1330. Each convolutional layer has a in ×3×3×c out The weight of each k size) and c out Individual bias (b k size), and c in and c out is the number of input and output channels for each layer. The first layer has 12 input channels, the last layer has 4 output channels, but all hidden layers have F feature channels. This means that for a STRN with N = 6 layers and F = 128 feature channels, there are a total of n w = 608256 weights and n b = 644 biases.

[0216] As shown in Figure 15, neural network 130 can have a skip connection 138 where input array C2 is added to the output. N is the polyphase representation of the output, and the four output channels correspond to the four polyphase components as in equation (1). Therefore, L N The element of W before the skip connection 138 B ×H B Output Array TIFF2026506249000074.tif75, which means that the four polyphase components are merged into one array by inverse polyphase decomposition 150. Then the final output, e.g., the processed picture portion 11′, i.e., the post-processed inter-predicted signal for the current block 18, is TIFF2026506249000075.tif715 is the output of input array C2 TIFF2026506249000076.tif75 and add it to the refined prediction of the current W × H block 18, P*, i.e., Using TIFF2026506249000077.tif715 TIFF2026506249000078.tif987 is obtained by cropping an L-shaped B sample wide area along the top / left border. The skip connections 138 turn the STRN, i.e., neural network 130, into a residual network, where the convolutional layers use spatial and temporal reference samples, i.e., samples in the spatial neighborhood 100, also called the neighborhood signal, to generate a regular prediction P of the current block 18 (contained in the input array C2), i.e., P inter,18 Residual or offset value to improve This has the effect of learning to produce TIFF2026506249000079.tif75.

[0217] The reason for including polyphase decomposition 144 in the STRN architecture is that it allows for a significant complexity reduction. Table I in Figure 16 shows a comparison between an IPRN architecture without polyphase decomposition 144

[14] and a STRN architecture with polyphase decomposition. The computational complexity of deep learning methods for video coding is often evaluated in terms of a multiply-accumulate (MAC) operation per output luma sample, commonly referred to as MAC per pixel (MAC / pxl). For both IPRN and STRN, this value is About TIFF2026506249000080.tif715 and STRN Input tensor shape equal to TIFF2026506249000081.tif1320 TIFF2026506249000082.tif717 TIFF2026506249000083.tif1587, depends on the number of weights and block shape. Thus, multiphase decomposition 144 reduces complexity by reducing the tensor shape of the input and all subsequent layers. The resulting minimum and maximum values ​​in Table I highlight that STRN has approximately one-quarter the complexity of IPRN for the same number of feature channels, or alternatively, has nearly the same complexity for twice the number of feature channels.

[0218] In the following, a possible training of the neural network 130 is described.

[0219] The training dataset for STRN consists of a set of so-called training samples. Similar to

[14] , these samples are divided into three input arrays C for the inter-blocks. i and storing them together with the corresponding original signal array O of a block, e.g., the current block 18. The values ​​of these arrays are in the range [0...2 bThe IPRN and STRN architectures in

[14] share the commonality that their architectures are completely independent of the block shape W×H, meaning that the same model can be trained and applied to all VVC interblock shapes. As a result, the dataset contains training samples with a variety of block shapes. During training, each forward and backward propagation cycle processes a batch of training samples at a time. All training samples in a batch must have the same shape, but each batch can have different shapes. As a result, a single model can be trained on all block shapes contained in the dataset.

[0220] The core of the training process is a gradient descent algorithm with a loss function and backpropagation of the loss, based on a learning rate and an optimizer. Regarding the loss function, the differences between the commonly used SSD, SAD, and SATD have been studied in

[14] , and it is concluded that SATD performs better than other loss functions calculated in the spatial domain. Therefore, the SATD loss function is also used for the STRN, i.e., neural network 130. Given the output P* of the W × H block and the corresponding original signal O, the loss TIFF2026506249000084.tif72 is the residual 2D DCT-II TIFF2026506249000085.tifEquals 74 norm. For the backpropagation of the loss, the widely used Adam optimizer

[40] is employed with a learning rate that exponentially decays by a factor of 0.8 every two epochs. Figure 17 shows the learning rate (10 -4 Figure 1 shows an example of the relationship between the number of feature channels (using initial values ​​of ) and the resulting loss. Both models have roughly the same complexity, but due to the multiphase decomposition 144, STRN has twice the number of feature channels and consequently a lower loss.

[0221] One effect of the block-shape-independent architecture is that the influence of spatial reference samples in the input C, i.e., picture portion 11, on the output P*, i.e., processed picture portion 11′, is limited. Given a simple CNN such as an IPRN with N layers and a 3×3 kernel size, the value of an output element depends only on the values ​​of input elements within a (2N+1)×(2N+1) area around the location of the output element, as shown in FIG. 18. As a result, spatial reference samples within an L-shaped B-wide area along the top / left boundary of the input, i.e., spatial neighborhood 100, only affect the output value within an L-shaped N-wide area above / to the left of the block. In the case of the STRN, the area of ​​P* affected by spatial reference samples in C, i.e., processed picture portion 11′, is actually 2N-wide due to the multiphase decomposition 144. FIG. 19 shows the results of an experimental evaluation of the described effect, comparing an IPRN and a STRN with and without spatial reference samples. For each of the three trained models, the position-wise MSE reduction is The r values ​​are evaluated during inference as TIFF2026506249000087.tif7100. The r values ​​correspond to the amount of improvement at each location, and the plot in Figure 19 shows the average value across the inference dataset. Comparing the results in Figures 4(a) and (b) with Figure 4(c) reveals that the improvement in the predicted signal is significantly higher when spatial reference samples are included in the input. Furthermore, the results in Figures 4(a) and (b) confirm that the influence of spatial reference samples is limited to the L-shaped top / left area of ​​the block, and that this area is twice as wide in the STRN case as in the IPRN case. Note that the cross-shaped structure in Figures 4(a)-(c) is caused by the DMVR coding tool in VVC.

[0222] The following sections describe how STRN is integrated into VVC inter-coding, including its interaction with other inter-coding tools in the prediction process, its integration in the decoding process with special attention to the intra-loop, the compilation of input arrays for application and training sample collection, and its efficient integration in the encoding process.

[0223] Both IPRN and STRN in

[14] are inter-block prediction signals, e.g., P of current block 18. inter,18 , and thus may be integrated into the VVC inter prediction process as a post-processing module, for example as the post-processor 112 of the first predetermined decoding tool 1101 in Fig. 6, or as the STRN tool 1101 in Fig. 7, or as the picture processing tool 1101 in Fig. 13, or as the post-processing tool 1101 in Fig. 14. k and bias b k Given a trained model with σ and σ, an input tensor C is compiled based on the regular VVC inter prediction P and propagated forward through the network to give a refined prediction P*. See, for example, the discussion above. As in the training case above, the value of C is set in the range [0...2 b -1] to floating-point values ​​in the range [-0.5,0.5]. Therefore, the values ​​of P* are converted to floating-point values ​​in the range [0…2 b −1] and then used as the final prediction signal for the block in VVC.

[0224] In the proposed solution, the STRN, i.e., the neural network 130, is applied only to the luma component of a block, e.g., the current block 18 or picture portion 11, and to all uni-predictive and bi-predictive inter-blocks, e.g., the corresponding blocks 181 and 182 or picture portions 111 and 112 in one or more reference pictures, which means that - not coded with CIIP, BCW, GPM, or SbTMVP, and - Not having motion vectors equal to zero unless coded with AMC (this is referred to as the zero MV constraint in the following).

[0225] For all these cases, the use of the STRN, i.e., neural network 130, is mandatory, i.e., it cannot be switched off for individual blocks. Therefore, no tool flags or other mode data are signaled in the bitstream, i.e., data stream 12. Furthermore, a single model is used for all applicable inter-blocks, including all block shapes and all QP values. The zero MV constraint is motivated by the observation that repeated application of the CNN, i.e., neural network 130, can lead to progressive signal degradation. This is particularly relevant for low-latency prediction structures and is described in more detail below.

[0226] The general process of generating and compiling the input tensor C, i.e., tensor 146, is the same for both collecting training data and applying the STRN, i.e., neural network 130, as a coding tool. However, in the latter case, the design of the VVC decoding process for inter-pictures (or slices) must be considered. Achieving real-time decoding for high frame rate and / or high resolution applications is very difficult. For intra-blocks, the prediction signal is a function of the top / left spatial reference samples, which means that the intra-block can be decoded only after all its respective neighboring blocks in the current picture have been decoded. For inter-blocks, the prediction signal is a function of the L0 and L1 temporal reference samples, which means that the inter-block can be decoded independently of other blocks in the current picture. This design allows for parallel decoding of inter-blocks and thus significantly reduces the implementation complexity for decoding inter-slices. For applications with higher frame rates and / or higher resolutions, all inter-blocks can be processed in parallel first, and then the remaining intra-blocks can be processed sequentially. In this context, CIIP blocks are considered part of the intra-decoding loop, since the final prediction is a weighted combination of planar intra-prediction (spatial reference samples) and inter-prediction (temporal reference samples).

[0227] Figure 20 illustrates the decoding process for an inter slice with inter, intra, and STRN blocks. For STRN, the prediction signal P* is a function of both spatial and temporal reference samples. As a result, without appropriate modifications, STRN becomes part of the intra decoding loop, as shown in Figure 20(b). Both STRN and intra blocks depend on the reconstructed samples of neighboring STRN and intra blocks. For example, as shown in Figure 7, the processing of the current block 18 by the STRN tool 1101 depends on the signals associated with the intra prediction block 106 and the STRN block 102. This poses a problem for decoder implementations that rely on parallel processing of inter blocks, because STRN post-processing is mandatory for most inter coding modes and the computational complexity of forward propagating the input C through the neural network 130 is very high. While a simple solution would be to remove the spatial reference samples in C by setting B = 0, the corresponding results in Figure 19 and Table III in Figure 22 show that the possibility of improving the prediction signal is limited if we cannot adapt to the reconstructed signals of neighboring blocks.

[0228] The solution of the present invention for B>0 is shown in Figure 20(c). For example, the STRN post-processing performed by the post-processor 112 in Figure 6, the STRN tool 1101 in Figure 7, the picture processing tool 1101 in Figure 13, or the post-processing tool 1101 in Figure 14 is separated from the intra-decoding loop by imposing the following constraint:

[0229] For the spatial reference samples of the STRN block corresponding to the intra block, the enhanced inter prediction signal of the current block is used instead of the reconstructed signal, i.e., the enhanced inter prediction signal TIFF2026506249000088.tif718 is the reconstructed signal P within the spatial neighborhood intra +R, for example, 100 in Figs. 106 Please refer to.

[0230] For the spatial reference samples of both the STRN and intra blocks corresponding to the STRN block, the intermediate reconstructed signal (P+R) without STRN post-processing or only the predicted signal P without STRN post-processing is used instead of the actual reconstructed signal (P*+R) or P*, where R is the residual transmitted in the bitstream, e.g., referring to FIG. 7, the inter predicted signal P inter,102 or intermediate reconstructed signal (P inter,102 +R) is used by the STRN tool 1101 and not the post-processed version, i.e., the post-processed inter-predicted signal of the STRN block 102 TIFF2026506249000089.tif717 or reconstructed signal ( TIFF2026506249000090.tif717+R).

[0231] The corresponding results in Table III of Figure 22 show that the coding performance for using constrained spatial reference samples, i.e., neighboring signals, such as 100' in Figure 6, is significantly better than for the simple solution with B = 0. Here, the inter-decoding process can be implemented as follows (see Figure 20(c)): (1) reconstruct all inter-blocks in parallel without STRN post-processing, (2) reconstruct the remaining intra-blocks consecutively, and (3) apply STRN post-processing to applicable inter-blocks in parallel. Steps (1) and (2) are the same as the normal VVC decoding process without STRN, and steps (2) and (3) are independent of each other and can be performed in reverse order or even simultaneously. Furthermore, the parallel processing of step (3) allows for efficient use of the GPU, which is not the case when STRN is part of the intra-decoding loop.

[0232] A W×H inter-block with prediction P to which STRN post-processing can be applied, e.g. P inter,18 (see, for example, current block 18 in Figures 6, 7, 13 and 14), the input array C i The process of generating the resulting input array C depends on the coding mode.i is required to be identical to each normal inter-predicted signal in the corresponding W × H area. While the prediction processes of BDOF, DMVR, and AMC operate on sub-blocks, STRN is applied to the entire W × H block, which includes the reference signals and prediction signals of all sub-blocks. B ×H B For the entire area, input array C i This means that is derived.

[0233] For input arrays C1 and C3, the L0 and L1 motion vectors available from regular inter prediction are used to generate extended W frames from the respective reference pictures, including additional spatiotemporal reference samples in an L-shaped B-wide area along the top / left border. B ×H B Obtain the area. Except for AMC and DMVR, this step is simple because for each reference picture, only one motion vector is used for the entire block. For uni-predictive blocks that use reference data from only one temporal reference picture, C1 and C3 are identical and both contain either L0 or L1 reference data, depending on the selected reference list. AMC and DMVR use motion vectors refined individually on a subblock-by-subblock basis, so deriving additional spatio-temporal reference samples requires extending the process accordingly. For AMC, the input arrays C1 and C3 are generated without PROF refinement by applying the motion vectors of the 4x4 subblocks along the top / left boundary to the extended subblock containing the adjacent B-wide area, using the same interpolation filters as for regular AMC subblocks. For DMVR, the WxH area is divided into sub-blocks of up to 16x16 samples, and the L-shaped B-width area along the top / left boundaries of C1 and C3 is derived by introducing an additional sub-block that inherits the refined motion vectors and horizontal and / or vertical dimensions of the sub-blocks along their top / left boundaries using the same sample padding process as for regular DMVR sub-blocks.

[0234] For input array C2, a normal prediction P, e.g., Pinter,18 are copied to the corresponding W×H area, and the remaining L-shaped top / left area, i.e., the spatial neighborhood 100, is filled with spatial reference samples. Depending on the application, i.e., whether the STRN post-processing needs to be separated from the intra-decoding loop or not, either constrained or normal reconstructed samples are used. For this purpose, a reference sample buffer is continuously filled during the encoding and decoding process to collect the necessary data of already processed blocks. In some cases, for blocks located along the top or left boundary of the picture and for intra-reference blocks, spatial reference samples are (partially) unavailable, in which case constrained spatial reference samples are used. These areas of C2 are then filled with simple bi-prediction without BDOF refinement, i.e., the average of the corresponding sample values ​​in C1 and C3.

[0235] The VVC encoding process attempts to minimize the rate-distortion (RD) cost by testing different combinations of block partitioning and coding modes on the original uncompressed picture. For a given W×H block in an inter-picture, several candidate coding modes are tested, including both inter-prediction and intra-prediction modes. Ultimately, the coding mode with the lowest RD cost is selected and used to determine block partitioning later.

[0236] STRN is integrated as a post-processing module. Because the computational complexity of forward propagating an input tensor C, e.g., tensor 146, through neural network 130 is very high, STRN is not used for all coding mode candidates, but only for the most promising ones. To this end, the best coding mode is first determined using the RD cost without STRN refinement. During this step, a list of length K is filled with coding modes for which STRN is applicable (e.g., see the discussion above). If STRN is applicable to fewer than K coding modes, the list is not completely filled. If STRN is applicable to more than K coding modes, the list contains those with the lowest RD cost without STRN refinement. In a second step, the RD costs of up to K coding modes in the list are updated using STRN refinement, and the final coding mode for the block is selected between the best coding mode in the list and the best coding mode for which STRN is not applicable.

[0237] The experimental results and evaluation of STRN are described below.

[0238] To evaluate the impact of STRN on VVC coding efficiency, we used the VVC Test Model 15 reference software (VTM-15.0)

[43] under the JVET Common Test Conditions (CTC)

[45] . Unless otherwise noted, the STRN model, i.e., neural network 130, was trained with the configuration and dataset specified in Table II of Figure 21, resulting in a model file containing the layer structure along with weight and bias values. For application in VTM, STRN post-processing was integrated into the software using the LibTorch 1.10 API, which provides the functionality to load the model file and forward propagate the input tensor C through the network. All VTM coding experiments were performed without GPU support, meaning that the runtimes presented in this section were obtained by running both VTM and STRN post-processing in a single thread on the CPU. Apart from the fast encoder search described above, our implementation is not optimized for runtime. In particular, the decoder operates without the parallel processing described above, which is intended for hardware implementations and real-time applications.

[0239] Table IV in Figure 23 shows the coding gain as a Bjöntegaard delta (BD)

[46] ,

[47] rate for CTC sequences, as well as the overall coding performance for RA, low-delay B (LB), and low-delay P (LP) configurations. While the training dataset contains only samples of specific block shapes and QPs under the RA configuration, the results in Table IV demonstrate that STRN achieves substantial coding gains for different coding structures and when applied to all block shapes (using the default QP range 22...37). Additional results in Table V in Figure 24 show that STRN performs equally well for a high QP range of 27...42, with an overall luma BD rate of over 4%.

[0240] Table III in Figure 22 shows the coding performance of IPRN and STRN, along with important intermediate steps in the development of the proposed solutions. To evaluate the effect of polyphase decomposition 144 and separate it more precisely from the intra-decoding loop, the table additionally includes an analysis of MAC operations and sample usage. The number of MAC operations may depend on the network configuration (number of weights) and block shape (if B>0), with theoretical minimum and maximum values ​​achieved for the largest and smallest block shapes, respectively. Both the average MAC per pixel and the average sample usage are measured for all blocks and all pictures of the decoded CTC bitstream, with sample usage indicating the portion of luma samples covered by blocks to which IPRN or STRN is applied. Comparing the overall results of IPRN with configurations (a) and (b) in Figure 22 highlights that polyphase decomposition 144 either results in a dramatic reduction in complexity with a slightly lower coding gain (for the same number of feature channels) or a significant increase in coding gain with roughly the same complexity (for twice the number of feature channels). Note that configuration (b) uses the same trained model as STRN, but is still part of the intra-decoding loop in VVC because input array C2 contains unconstrained spatial reference samples. Configuration (c) in Figure 22 is a solution for decoupling STRN from the intra-decoding loop by completely omitting spatial reference samples, i.e., B = 0. However, compared to configuration (b), this results in a substantial reduction in coding gain. In our proposed solution, STRN instead uses constrained spatial reference samples, which results in slightly less coding gain than configuration (b), but has the advantage that STRN post-processing is decoupled from the intra-decoding loop. All remaining results described below are based on STRN and therefore use constrained spatial reference samples.

[0241] The effect of the polyphase decomposition 144 and spatial reference samples on the improvement of the predicted signal has already been shown by the inference-based evaluation in Figure 19. Regarding these results, it is essential to mention the cross-shaped structure at the center of the blocks in all three configurations. Further investigation showed that this effect comes from DMVR, i.e., 16x16 sub-blocks that use sample padding instead of reconstructed samples along the boundaries of the L0 and L1 predicted signals. These areas tend to have higher MSE in the prediction P and, consequently, higher MSE reduction (improvement) after applying IPRN or STRN.

[0242] Figure 25 and Table V in Figure 24 compare the coding performance of the default STRN configuration with variations having different numbers of layers N, number of feature channels F, spatial reference size B, or encoder list length K. Table V presents the BD rate and relative runtime for both the default and high QP ranges, and the diagram in Figure 25 shows the luma BD rate versus average MAC per pixel, including additional variations of N and F. While decoder runtime and average MAC per pixel are strongly correlated, only the latter can be accurately reproduced for a given set of bitstreams, since it is independent of the simulation environment. Therefore, per-pixel MAC is more suitable for evaluating the complexity overhead of STRN. The results of varying N, F, and B confirm that our default STRN configuration offers a good tradeoff between coding gain and complexity. When targeting a configuration with lower complexity, reducing the number of feature channels F offers a better tradeoff than reducing the number of layers N or spatial reference size B. However, when targeting a configuration with higher coding gain, increasing the encoder list length K offers an interesting tradeoff. The additional coding gain is achieved without a significant increase in decoding complexity. Increasing K corresponds to testing additional coding modes with STRN post-processing in the encoder, which results in significantly higher encoder runtime. For example, the K=2 variant has nearly the same coding gain and encoder runtime as the F=192 variant, but with only half the decoding complexity.

[0243] Table VI in Figures 26 and 27 examines the effect of the zero MV constraint introduced above, focusing on the low-delay configurations LB and LP. Note that the zero MV constraint is also taken into account in the training process by using a dataset that includes or excludes training samples for blocks that satisfy the zero MV condition. Before adding the zero MV constraint to the conditions for applicable blocks, STRN was observed to result in significant coding loss for Class E sequences when using the LP configuration. Further investigation revealed that STRN post-processing can result in gradual signal degradation. This effect is illustrated by the diagram in Figure 26, which shows how the average luma BD rate varies over the length of the sequence. Both curves (dashed lines) without the zero MV constraint feature a gradual decline in coding efficiency that accumulates to a significant loss. The fact that Class E sequences are highly static, with large areas of constant background, and that the LP configuration is limited to uniprediction, leads to situations where STRN post-processing is repeatedly applied to the exact same signal. The example in Figure 26 shows that this effect is almost completely eliminated by adding the zero MV constraint, i.e., by omitting STRN post-processing for blocks with motion vectors equal to zero. The results in Table VI of Figure 27 further confirm that the zero MV constraint improves the coding performance of difficult sequences in the low-delay configuration without affecting the coding performance of other sequences or RA configurations.

[0244] This application presents an approach for refining interblock prediction signals in state-of-the-art video coding via a spatiotemporal residual neural network (STRN), e.g., the neural network 130 shown in Figure 13 or Figure 15. Starting from our previous work in

[14] , the architecture is improved by adding a multiphase decomposition 144 of the input tensor before the first convolutional layer. The multiphase decomposition 144 has been shown theoretically and experimentally to increase the area of ​​blocks, e.g., current block 18, that can benefit from spatial reference samples while reducing computational complexity (worst-case and effective MAC per pixel). Compared to an IPRN without the multiphase decomposition 144, this results in almost a quarter of the complexity and a slightly lower coding gain, or roughly the same complexity and a significantly higher coding gain by doubling the number of feature channels.

[0245] STRN includes the inter prediction signal of the block, i.e., the P of the current block 18, together with spatial and temporal reference samples. inter,18 is used to compile an input tensor 146, which is then forward propagated through the trained network 130, resulting in a refined prediction signal, i.e., a post-processed inter prediction signal for the current block 18. TIFF2026506249000091.tif715 is obtained. The same model, e.g., neural network 130, is used for all coding modes, block shapes, and QPs. Furthermore, STRN is supported for most inter prediction modes, with an average sample usage of approximately 68%, and is mandatory for all applicable blocks.

[0246] Including spatial reference samples in the inter prediction process is difficult. The additional dependency on reconstructed blocks within the same picture makes parallel decoding of STRN blocks impossible. They become part of the intra decoding loop, which contradicts the basic design of the VVC decoding process and is not feasible for real-time decoder implementations. In our solution, STRN is decoupled from the intra decoding loop by prohibiting reconstructed samples of intra blocks in input array C2 and using a special reference sample buffer containing intermediate reconstructed samples of the STRN block. These constraints achieve a slightly lower coding gain, but STRN blocks can be decoded independently and in parallel with the intra blocks.

[0247] STRN has been implemented under the VTM reference software and CTC, achieving an average coding gain of -4.07% of the luma BD rate for the RA configuration with approximately 3x encoder and 70x decoder runtimes. However, our implementation is not optimized for runtime, and coding experiments are performed single-threaded on a CPU. Experimental evaluation confirmed that the default STRN configuration (N=6, F=128, and B=4) is a good tradeoff between coding gain and complexity, and that additional coding gain can be achieved for K>1 without a significant increase in decoding complexity.

[0248] For low-delay prediction structures, STRN exhibits a gradual signal degradation effect. We show that this effect can be successfully mitigated by adding a zero-MV constraint to the conditions for blocks where STRN is applicable. As a result, STRN achieves consistent and substantial coding gains for all configurations.

[0249] Although some aspects have been described as features in the context of an apparatus, it will be apparent that such description may also be considered a description of the corresponding features of a method. Although some aspects have been described as features in the context of a method, it will be apparent that such description may also be considered a description of the corresponding features with respect to the functionality of the apparatus.

[0250] Some or all of the method steps may be performed by (or using) a hardware apparatus such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0251] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software, or at least partially in hardware or at least partially in software. Implementations can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored, which cooperates (or can cooperate) with a programmable computer system so that the respective methods are performed. Thus, the digital storage medium may be computer-readable.

[0252] Some embodiments according to the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0253] Generally, embodiments of the present invention can be implemented as a computer program product having program code that operates to perform one of the methods when the computer program product is run on a computer, and the program code may for example be stored on a machine-readable carrier.

[0254] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0255] In other words, an embodiment of the inventive methods is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0256] A further embodiment of the inventive methods is, therefore, a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0257] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals can for example be adapted to be transmitted via a data communication connection, for example via the Internet.

[0258] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0259] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0260] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0261] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to implement some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0262] The devices described herein may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0263] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0264] In the foregoing Detailed Description, it can be seen that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, subject matter may lie in fewer than all features of a single disclosed example. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate example. While each claim may stand on its own as a separate example, a dependent claim may refer to a specific combination with one or more other claims in the claim, and it should be noted that other examples may also include a combination of a dependent claim with the subject matter of each other dependent claim, or a combination of each feature with other dependent or independent claims. Such combinations are suggested herein unless it is stated that a specific combination is not intended. Furthermore, it is intended that features of a claim be included in any other independent claim, even if that claim is not directly dependent on that independent claim.

[0265] The above-described embodiments are merely illustrative for the principles of the present disclosure. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims, and not by the specific details presented by the description and explanation of the embodiments herein. References

[0266] [1] ITU-T, “H.261:Video codec for audiovisual services at px 384 kbit / s”, Mar.1993, available from ITU-T at https: / / www.itu.int / rec / T-REC-H.261

[0267] [2]ITU-T and ISO / IEC,“Versatile Video Coding”,Jul.2020,available from ITU-T at https: / / www.itu.int / rec / T-REC-H.266 and from ISO / IEC at https: / / www.iso.org / standard / 73022.html

[0268] [3]B.Bross,Y.-K.Wang,Y.Ye,S.Liu,J.Chen,G.J.Sullivan,and J.-R.Ohm,“Overview of the Versatile Video Coding(VVC)standard and its applications”,IEEE Transactions on Circuits and Systems for Video Technology,vol.31,no.10,pp.3736-3764,2021,doi:10.1109 / TCSVT.2021.3101953

[0269] [4]B.Girod,“Efficiency analysis of multihypothesis motion-compensated prediction for video coding”,IEEE Transactions on Image Processing,vol.9,no.2,pp.173-183,2000,doi:10.1109 / 83.821595

[0270] [5]ISO / IEC JTC / SC29,“Coded representation of picture,audio and multimedia / hypermedia information”,Committee Draft of standard ISO / IEC 11172,Dec.1991

[0271] [6]ITU-T and ISO / IEC,“Advanced Video Coding”,Aug.2004,available from ITU-T at https: / / www.itu.int / rec / T-REC-H.264 and from ISO / IEC at https: / / www.iso.org / standard / 61490.html

[0272] [7]W.-J.Chien,L.Zhang,M.Winken,X.Li,R.-L.Liao,H.Gao,C.-W.Hsu,H.Liu,and C.-C.Chen,“Motion vector coding and block merging in the Versatile Video Coding standard”,IEEE Transactions on Circuits and Systems for Video Technology,vol.31,no.10,pp.3848-3861,2021,doi:10.1109 / TCSVT.2021.3101212

[0273] [8]A.Alshin,E.Alshina,and T.Lee,“Bi-directional optical flow for improving motion compensation”,in 28th Picture Coding Symposium,2010,pp.422-425,doi:10.1109 / PCS.2010.5702525

[0274] [9]H.Yang,H.Chen,J.Chen,S.Esenlik,S.Sethuraman,X.Xiu,E.Alshina,and J.Luo,“Subblock-based motion derivation and inter prediction refinement in the Versatile Video Coding standard”,IEEE Transactions on Circuits and Systems for Video Technology,vol.31,no.10,pp.3862-3877,2021,doi:10.1109 / TCSVT.2021.3100744

[0275]

[10] H.Liu,Y.Chen,J.Chen,L.Zhang,and M.Karczewicz,“Local illumination compensation”,document VCEG-AZ06,ITU-T Q.6 / SG 16(VCEG),2015

[0276]

[11] C.-W.Seo and J.-K.Han,“Pixel based illumination compensation for inter prediction in HEVC”,Electronics letters,vol.47,no.23,pp.1278-1280,2011,doi:10.1049 / el.2011.2524

[0277]

[12] ITU-T and ISO / IEC,“High Efficiency Video Coding”,Aug.2021,available from ITU-T at https: / / www.itu.int / rec / T-REC-H.265 and from ISO / IEC at https: / / www.iso.org / standard / 75484.html

[0278]

[13] G.Tech,Y.Chen,K.M¨uller,J.-R.Ohm,A.Vetro,and Y.-K.Wang,“Overview of the multiview and 3D extensions of High Efficiency Video Coding”,IEEE Transactions on Circuits and Systems for Video Technology,vol.26,no.1,pp.35-49,2016,doi:10.1109 / TCSVT.2015.2477935

[0279]

[14] P.Merkle,M.Winken,J.Pfaff,H.Schwarz,D.Marpe,and T.Wiegand,“Intra-inter prediction for Versatile Video Coding using a residual convolutional neural network”,in 2022 IEEE International Conference on Image Processing(ICIP),2022,pp.1711-1715,doi:10.1109 / ICIP46576.2022.9897324

[0280]

[15] Y.LeCun,B.Boser,J.S.Denker,D.Henderson,R.E.Howard,W.Hubbard,and L.D.Jackel,“Backpropagation applied to handwritten zip code recognition”,Neural Computation,vol.1,no.4,pp.541-551,Dec.1989,doi:10.1162 / neco.1989.1.4.541

[0281]

[16] G.Lu,W.Ouyang,D.Xu,X.Zhang,C.Cai,and Z.Gao,“DVC:An end-to-end deep video compression framework”,in 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),2019,pp.10 998-11 007,doi:10.1109 / CVPR.2019.01126

[0282]

[17] A.Djelouah,J.Campos,S.Schaub-Meyer,and C.Schroers,“Neural inter-frame compression for video coding”,in 2019 IEEE / CVF International Conference on Computer Vision(ICCV),2019,pp.6420-6428,doi:10.1109 / ICCV.2019.00652

[0283]

[18] E.Agustsson,D.Minnen,N.Johnston,J.Ball,S.J.Hwang,and G.Toderici,“Scale-space flow for end-to-end optimized video compression”,in 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),2020,pp.8500-8509,doi:10.1109 / CVPR42600.2020.00853

[0284]

[19] N.Yan,D.Liu,H.Li,B.Li,L.Li,and F.Wu,“Convolutional neural network-based fractional-pixel motion compensation”,IEEE Transactions on Circuits and Systems for Video Technology,vol.29,no.3,pp.840-853,2019,doi:10.1109 / TCSVT.2018.2816932

[0285]

[20] L.Murn,S.Blasi,A.F.Smeaton,and M.Mrak,“Improved CNN-based learning of interpolation filters for low-complexity inter prediction in video coding”,IEEE Open Journal of Signal Processing,vol.2,pp.453-465,2021,doi:10.1109 / OJSP.2021.3089439

[0286]

[21] W.Cui,T.Zhang,S.Zhang,F.Jiang,W.Zuo,Z.Wan,and D.Zhao,“Convolutional neural networks based intra prediction for HEVC”,in 2017 Data Compression Conference(DCC),2017,pp.436-436,doi:10.1109 / DCC.2017.53

[0287]

[22] J.Pfaff,P.Helle,D.Maniry,S.Kaltenstadler,W.Samek,H.Schwarz,D.Marpe,and T.Wiegand,“Neural network based intra prediction for video coding,” in Applications of Digital Image Processing XLI,ser.Society of Photo-Optical Instrumentation Engineers(SPIE)Conference Series,vol.10752,Sep.2018,p.1075213,doi:10.1117 / 12.2321273

[0288]

[23] M.M.Alam,T.D.Nguyen,M.T.Hagan,and D.M.Chandler,“A perceptual quantization strategy for HEVC based on a convolutional neural network trained on natural images”,in Applications of Digital Image Processing XXXVIII,A.G.Tescher,Ed.,vol.9599,International Society for Optics and Photonics.SPIE,2015,p.959918,doi:10.1117 / 12.2188913

[0289]

[24] Y.Zhang,T.Shen,X.Ji,Y.Zhang,R.Xiong,and Q.Dai,“Residual highway convolutional neural networks for in-loop filtering in HEVC”,IEEE Transactions on Image Processing,vol.27,no.8,pp.3827-3841,2018,doi:10.1109 / TIP.2018.2815841

[0290]

[25] C.Jia,S.Wang,X.Zhang,S.Wang,J.Liu,S.Pu,and S.Ma,“Content-aware convolutional neural network for in-loop filtering in High Efficiency Video Coding”,IEEE Transactions on Image Processing,vol.28,no.7,pp.3343-3356,2019,doi:10.1109 / TIP.2019.2896489

[0291]

[26] Z.Huang,J.Sun,X.Guo,and M.Shang,“One-for-all:An efficient variable convolution neural network for in-loop filter of VVC”,IEEE Transactions on Circuits and Systems for Video Technology,vol.32,no.4,pp.2342-2355,2022,doi:10.1109 / TCSVT.2021.3089498

[0292]

[27] S.Ma,X.Zhang,C.Jia,Z.Zhao,S.Wang,and S.Wang,“Image and video compression with neural networks:A review”,IEEE Transactions on Circuits and Systems for Video Technology,vol.30,no.6,pp.1683-1698,2020,doi:10.1109 / TCSVT.2019.2910119

[0293]

[28] D.Ding,Z.Ma,D.Chen,Q.Chen,Z.Liu,and F.Zhu,“Advances in video compression system using deep neural network:A review and case studies”,Proceedings of the IEEE,vol.109,no.9,pp.1494-1520,2021,doi:10.1109 / JPROC.2021.3059994

[0294]

[29] S.Huo,D.Liu,F.Wu,and H.Li,“Convolutional neural network-based motion compensation refinement for video coding”,in IEEE International Symposium on Circuits and Systems(ISCAS),2018,doi:10.1109 / ISCAS.2018.8351609

[0295]

[30] Y.Wang,X.Fan,C.Jia,D.Zhao,and W.Gao,“Neural network based inter prediction for HEVC”,in IEEE International Conference on Multimedia and Expo(ICME),2018,doi:10.1109 / ICME.2018.8486600

[0296]

[31] Y.Wang,X.Fan,R.Xiong,D.Zhao,and W.Gao,“Neural network-based enhancement to inter prediction for video coding”,IEEE Transactions on Circuits and Systems for Video Technology,vol.32,no.2,pp.826-838,2022,doi:10.1109 / TCSVT.2021.3063165

[0297]

[32] Z.Zhao,S.Wang,S.Wang,X.Zhang,S.Ma,and J.Yang,“Enhanced bi-prediction with convolutional neural network for High-Efficiency Video Coding”,IEEE Transactions on Circuits and Systems for Video Technology,vol.29,no.11,pp.3291-3301,2019,doi:10.1109 / TCSVT.2018.2876399

[0298]

[33] J.Mao,H.Yu,X.Gao,and L.Yu,“CNN-based bi-prediction utilizing spatial information for video coding”,in IEEE International Symposium on Circuits and Systems(ISCAS),2019,doi:10.1109 / ISCAS.2019.8702552

[0299]

[34] J.Mao and L.Yu,“Convolutional neural network based bi-prediction utilizing spatial and temporal information in video coding”,IEEE Transactions on Circuits and Systems for Video Technology,vol.30,no.7,pp.1856-1870,2020,doi:10.1109 / TCSVT.2019.2954853

[0300]

[35] Z.Zhang,X.Fan,D.Zhao,and W.Gao,“CNN-based inter prediction refinement for AVS3”,in IEEE International Conference on Multimedia Expo Workshops(ICMEW),2020,doi:10.1109 / ICMEW46912.2020.9106017

[0301]

[36] J.Zhang,C.Jia,M.Lei,S.Wang,S.Ma,and W.Gao,“Recent development of AVS video coding standard:AVS3”,in Picture Coding Symposium(PCS),2019,doi:10.1109 / PCS48520.2019.8954503

[0302]

[37] D.Jin,J.Lei,B.Peng,W.Li,N.Ling,and Q.Huang,“Deep affine motion compensation network for inter prediction in VVC”,IEEE Transactions on Circuits and Systems for Video Technology,vol.32,no.6,pp.3923-3933,2022,doi:10.1109 / TCSVT.2021.3107135

[0303]

[38] J.Blackburn and M.N.Do,“Two-dimensional geometric lifting”,in 2009 16th IEEE International Conference on Image Processing(ICIP),2009,pp.3817-3820,doi:10.1109 / ICIP.2009.5414291

[0304]

[39] W.Shi,J.Caballero,F.Huszar,J.Totz,A.P.Aitken,R.Bishop,D.Rueckert,and Z.Wang,“Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network”,in 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),2016,pp.1874-1883,doi:10.1109 / CVPR.2016.207

[0305]

[40] D.P.Kingma and J.Ba,“Adam:A method for stochastic optimization”,in 3rd International Conference on Learning Representations(ICLR),May 2015,doi:10.48550 / arXiv.1412.6980

[0306]

[41] A.Paszke,S.Gross,F.Massa,A.Lerer,J.Bradbury,G.Chanan,T.Killeen,Z.Lin,N.Gimelshein,L.Antiga,A.Desmaison,A.Kopf,E.Yang,Z.DeVito,M.Raison,A.Tejani,S.Chilamkurthy,B.Steiner,L.Fang,J.Bai,and S.Chintala,“PyTorch:An imperative style,high-performance deep learning library”,in Advances in Neural Information Processing Systems 32.Curran Associates,Inc.,2019,pp.8024-8035,doi:10.48550 / arXiv.1912.01703

[0307]

[42] X.Glorot and Y.Bengio,“Understanding the difficulty of training deep feedforward neural networks”,in Proceedings of the 13th International Conference on Artificial Intelligence and Statistics(AISTATS),ser.JMLR Proceedings,vol.9.JMLR.org,May 2010,pp.249-256,available:http: / / proceedings.mlr.press / v9 / glorot10a / glorot10a.pdf,[Online;accessed Dec.2022]

[0308]

[43] “VVC reference software version 15.0”,Available:https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware VTM,[Online;accessed Dec.2022]

[0309]

[44] F.Zhang,D.Ma,and D.Bull,“BVI-DVC:A training database for deep video compression”,Apr.2020,doi:10.5523 / bris.3hj4t64fkbrgn2ghwp9en4vhtn

[0310]

[45] F.Bossen,J.Boyce,K.Suehring,X.Li,and V.Seregin,“VTM common test conditions and software reference configurations for SDR video”,document JVET-T2010,ITU-T / ISO / IEC Joint Video Experts Team(JVET),Oct.2020

[0311]

[46] G.Bjontegaard,“Calculation of average PSNR differences between RDcurves”,document VCEG-M33,ITU-T Q.6 / SG 16(VCEG),Apr.2001

[0312]

[47] ITU-T and ISO / IEC,“Working practices using objective metrics for evaluation of video coding efficiency experiments”,Jul.2020,available from ITU-T at http: / / handle.itu.int / 11.1002 / pub / 8160e8da-en and from ISO / IEC at https: / / www.iso.org / standard / 81591.html.

Claims

1. A video decoder (54) comprising a plurality of decoding tools (110), applying said plurality of decoding tools (110) to a current picture (10) of a video (16) on a block-by-block basis as controlled by a data stream (12); deriving a reconstructed signal (58) of the currently decoded picture (10) by a sample-by-sample combination of the contributing signals generated by the plurality of decoding tools (110); The plurality of decoding tools (110) are configured to, based on a neighborhood signal (100') within a spatial neighborhood (100) of the current block (18), One or more second predetermined decoding tools (110) in the current block (18) ２ ) contribution signal, or intermediate signals in the current block (18) corresponding to partial combinations from the sample-by-sample combinations; Post-processing the or The first predetermined decoding tool (110) for the current block (18) １ a first predetermined decoding tool (110) configured to generate a contribution signal of １ ), The video decoder (54) converts the neighborhood signal (100') within the spatial neighborhood (100) into said one or more second predetermined decoding tools (110 ２ ) or said first predetermined decoding tool (110 １ ) using a version of the intermediate signals in said spatial neighborhood (100) that has not been post-processed by One or more third predetermined decoding tools (110) within said spatial vicinity (100). ３ ) by a replacement signal generated independently of spatial signal interdependencies (122); and / or From the spatial neighborhood (100), the sample-by-sample combination for the derivation of the reconstructed signal (58) is transmitted to the one or more third predetermined decoding tools (110). ３ excluding (124) samples containing said contribution signal of a video decoder (54) configured to generate the

2. The first predetermined decoding tool (110 １ ) based on the neighborhood signals (100') within the spatial neighborhood (100) of the current block (18), The one or more second predetermined decoding tools (110) in the current block (18) ２ ) configured to post-process the contribution signals of The video decoder (54) controls the generation of the neighborhood signal (100') within the spatial neighborhood (100) by: The first predetermined decoding tool (110 １ the one or more second predetermined decoding tools (110) within the spatial neighborhood (100) in the version that has not been post-processed by ２ ) using said contribution signals of The first predetermined decoding tool (110 １ the one or more second predetermined decoding tools (110) within the spatial neighborhood (100) in the version that has not been post-processed by ２ ) combining said contribution signals with a prediction residual signal by sample-by-sample combination; 2. The video decoder of claim 1, configured to perform by:

3. The video decoder (54) controls the generation of the neighborhood signal (100') within the spatial neighborhood (100) by: The one or more third predetermined decoding tools (110) within the spatial vicinity (100) ３ ) by the replacement signal generated independently of spatial signal interdependencies, 3. The video decoder of claim 1, wherein the substitution signal represents a prediction signal, and the video decoder (54) is configured to refrain from combining the substitution signal with a prediction residual signal when generating the neighboring signal (100') within the spatial neighborhood (100) of the current block (18).

4. The video decoder (54) controls the generation of the neighborhood signal (100') within the spatial neighborhood (100) by: The one or more third predetermined decoding tools (110) within the spatial vicinity (100) ３ ) by the replacement signal generated independently of spatial signal interdependencies, The video decoder (54) uses the one or more second predetermined decoding tools (110 ２ 4. A video decoder according to claim 1, configured to generate the replacement signal using a

5. The video decoder (54) controls the generation of the neighborhood signal (100') within the spatial neighborhood (100) by: The one or more third predetermined decoding tools (110) within the spatial vicinity (100) ３ ) by the replacement signal generated independently of spatial signal interdependencies, The replacement signal is transmitted to the first predetermined decoding tool (110 １ the one or more second predetermined decoding tools (110) within the spatial neighborhood (100) of the current block (18) in a version that has not been post-processed by ２ 5. Video decoder according to claim 1, wherein the signal contributions of the input signal and the output signal are:

6. said one or more second predetermined decoding tools (110 ２ 6. The video decoder of claim 4, wherein the current block is a spatially neighboring block of the video decoder, and the replacement signal represents an inter prediction signal within the spatial neighborhood of the current block.

7. The first predetermined decoding tool (110 １ ) is configured to post-process the intermediate signals in the current block (18) corresponding to partial combinations from the sample-by-sample combinations based on the neighboring signals (100′) in a spatial neighborhood (100) of the current block (18); The video decoder (54) uses the first predetermined decoding tool (110 １ ) to generate the neighboring signals (100') within the spatial neighborhood (100) by using the intermediate signals within the spatial neighborhood (100) in a version that has not been post-processed by The intermediate signals within the current block (18) and the intermediate signals within the spatial neighborhood (100) of the current block (18) are respectively decoded by the one or more second predetermined decoding tools (110) within the current block (18) and the spatial neighborhood (100) of the current block (18). ２ 7. A video decoder according to claim 1, wherein the decoder corresponds to a sample-by-sample combination of two or more contributing signals of the order of 1 to 6.

8. 8. The video decoder of claim 1, wherein the video decoder (54) is a predictive decoder using transform-based residual decoding, and the contributing signals include a prediction signal and a prediction residual signal.

9. The video decoder of any one of claims 1 to 8, wherein the video decoder (54) is configured to subject the reconstructed signal (58) to an in-loop filter and / or a post-processing filter.

10. said one or more second predetermined decoding tools (110 ２ ) is a second predetermined decoding tool (110) ２ 10. The video decoder of claim 1, wherein the decoder is one or more inter-prediction decoding tools configured to generate an inter-prediction signal as the contributing signal of the first or second input signal.

11. When the plurality of decoding tools (110) are applied to the current picture (10) block-by-block as controlled by the data stream (12), for each of one or more first decoding tools, determining on a block-by-block basis whether to apply the respective first decoding tool as controlled by the data stream (12); for each of one or more second decoding tools, parameterizing the application of said respective second decoding tool on a block-by-block basis as controlled by said data stream (12); deriving, for each of one or more third decoding tools, from said data stream (12) a block subdivision for performing said block-by-block application of said respective third decoding tool controlled by said data stream (12); A video decoder according to any one of claims 1 to 10, configured to apply one or more of:

12. 12. The video decoder of claim 11, wherein the one or more decoding tools may belong to two or more, or all, of the one or more first decoding tools, the one or more second decoding tools, and the one or more third decoding tools.

13. The first predetermined decoding tool (110 １ ) in the current block (18) to the one or more second predetermined decoding tools (110) ２ ), and the video decoder (54) is configured to post-process the contributing signals (100′) within the spatial neighborhood (100) by: The first predetermined decoding tool (110 １ the one or more second predetermined decoding tools (110) within the spatial neighborhood (100) in the version that has not been post-processed by ２ ) and / or The one or more third predetermined decoding tools (110) within the spatial vicinity (100) ３ ) by said replacement signal generated independently of spatial signal interdependencies (122). A video decoder according to any one of claims 1 to 12, configured to generate by:

14. The first predetermined decoding tool (110 １ ) uses a neural network or convolution to generate a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the contributing signal with the neighboring signal (100′), based on the one or more second predetermined decoding tools (110). ２ 14. The video decoder of claim 13, configured to post-process the contribution signals of

15. The first predetermined decoding tool (110 １ ) selects said neural network from a set of two or more neural networks; or 15. The video decoder of claim 14 configured to select the convolution from a set of two or more convolutions.

16. The first predetermined decoding tool (110 １ 16. The video decoder of claim 15, wherein a neural network selector (14) is configured to select the neural network or the convolution under control of the data stream (12).

17. The first predetermined decoding tool (110 １ ) the neural network or the convolution is the block shape of the current block (18), and / or a prediction mode associated with the current block (18), and / or the temporal layer of the current picture (10), and / or a quantization parameter value associated with the current picture (10) or the current block (18), and / or a prediction residual signal associated with the current block (18), and / or if the current block (18) is associated with an inter-prediction mode, the picture order count difference between the current picture (10) and a reference picture; and / or If the current block (18) is associated with an inter prediction mode, a motion vector associated with the current block (18) 16. A video decoder as claimed in claim 15, configured to make the selection in dependence on

18. 18. A video decoder according to any one of claims 15 to 17, wherein the neural networks of the set of two or more neural networks differ from each other in terms of weights, biases, number of layers, layer types, and / or input tensor format.

19. 19. The video decoder of any one of claims 15 to 18, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, convolution type, and / or input tensor format.

20. The first predetermined decoding tool (110 １ 15. The video decoder of claim 14, wherein a neural network or a convolution is configured to derive the neural network or the convolution from the data stream (12).

21. The first predetermined decoding tool (110 １ 14. The video decoder of claim 13, wherein the local illumination compensation tool is a local illumination compensation tool.

22. The first predetermined decoding tool (110 １ ) in the current block (18) to the one or more second predetermined decoding tools (110) ２ ) said contribution signals, adapting scaling and offset values ​​based on said neighboring signals (100'); The one or more second predetermined decoding tools (110) in the current block (18) ２ using the scaling and offset values ​​to post-process the contribution signals of 22. The video decoder of claim 21, configured for post-processing by:

23. 23. The video decoder of claim 1, wherein the plurality of decoding tools (110) comprises a first set of decoding tools, and the video decoder (54) is configured to perform block-wise selection of exactly one decoding tool from the first set of decoding tools when applying the plurality of decoding tools (110) to the current picture (10) on a block-by-block basis.

24. The one or more third predetermined decoding tools (110 ３ 24. The video decoder of claim 23, wherein:

25. 25. The video decoder of claim 23 or claim 24, wherein the first set of decoding tools comprises one or more inter-prediction decoding tools configured to generate an inter-prediction signal and one or more intra-prediction decoding tools configured to generate an intra-prediction signal.

26. The one or more third predetermined decoding tools (110 ３ 26. The video decoder of claim 25, wherein the one or more intra-prediction decoding tools include:

27. said one or more second predetermined decoding tools (110 ３ 27. A video decoder according to claim 25 or 26, wherein the one or more inter-prediction decoding tools include:

28. The first predetermined decoding tool (110 １ ) is configured to post-process contribution signals of the one or more inter-prediction decoding tools in the current block (18) based on the neighborhood signal (100′), and the video decoder (54) processes the neighborhood signal (100′) in the spatial neighborhood (100) by: The first predetermined decoding tool (110 １ ) using the contribution signals of the one or more inter-prediction decoding tools in the spatial neighborhood (100) in a version that has not been post-processed by Replacing (122) a contributing signal of at least one of the one or more intra-prediction decoding tools within the spatial neighborhood (100) by a replacement signal generated independently of spatial signal interdependencies. A video decoder according to any one of claims 25 to 27, configured to generate by

29. The first predetermined decoding tool (110 １ 29. The video decoder of claim 28, wherein the one or more inter-prediction decoding tools are configured to post-process the contribution signals of the one or more inter-prediction decoding tools based on a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the contribution signals with the neighboring signals (100′), using a neural network or convolution.

30. The first predetermined decoding tool (110 １ ) selects said neural network from a set of two or more neural networks; or 30. The video decoder of claim 29, configured to select the convolution from a set of two or more convolutions.

31. The first predetermined decoding tool (110 １ 31. The video decoder of claim 30, wherein a neural network selector (14) is configured to select the neural network or the convolution under control of the data stream (12).

32. The first predetermined decoding tool (110 １ ) the neural network or the convolution is the block shape of the current block (18), and / or a prediction mode associated with the current block (18), and / or the temporal layer of the current picture (10), and / or a quantization parameter value associated with the current picture (10) or the current block (18), and / or a prediction residual signal associated with the current block (18), and / or if the current block (18) is associated with an inter-prediction mode, the picture order count difference between the current picture (10) and a reference picture; and / or If the current block (18) is associated with an inter prediction mode, a motion vector associated with the current block (18) 31. A video decoder as claimed in claim 30, configured to make the selection in dependence on

33. 33. A video decoder according to any one of claims 30 to 32, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

34. 34. A video decoder according to any one of claims 30 to 33, wherein the convolutions of the set of two or more convolutions differ from each other in terms of weights, biases, type of convolution and / or input tensor format.

35. The first predetermined decoding tool (110 １ 23. The video decoder of claim 22, wherein a neural network or a convolution is configured to derive the neural network or the convolution from the data stream (12).

36. The first predetermined decoding tool (110 １ 28. The video decoder (54) of claim 25, wherein one or more intra-prediction decoding tools are included in the first set of decoding tools and are configured to generate a prediction signal for the current block (18) using a combination of inter-prediction and intra-prediction based on the neighborhood signal (100′), and the video decoder (54) is configured to generate the neighborhood signal (100′) within the spatial neighborhood (100) by replacing (122) the contributing signal of at least one of the one or more intra-prediction decoding tools within the spatial neighborhood (100) with a replacement signal generated independently of spatial signal interdependencies.

37. 37. The video decoder of any one of claims 25 to 36, wherein the decoding tools in the first set of decoding tools are predictive decoding tools configured to generate a prediction signal.

38. A video decoder according to any preceding claim, configured to generate the replacement signal using inter prediction.

39. A video decoder according to any preceding claim, configured to generate the replacement signal using inter prediction using motion vectors according to the data stream (12).

40. The first predetermined decoding tool (110 １ ) in the current block (18) to the one or more second predetermined decoding tools (110) ２ ), and the video decoder (54) is configured to post-process the contribution signals of the first predetermined decoding tool (110). １ the one or more second predetermined decoding tools (110) within the spatial vicinity (100) of the version that has not been post-processed by ２ ) to generate the neighborhood signal (100') within the spatial neighborhood (100), by using the contributing signals of The video decoder (54) and determining whether the one or more third predetermined decoding tools (110) for a given block are based on further neighboring signals within the spatial neighborhood (100) of the given block. ３ ) contribution signal, The first predetermined decoding tool (110 １ the one or more second predetermined decoding tools (110) within the spatial vicinity (100) of the predetermined block in a version that has not been post-processed by ２ 40. The video decoder of any one of claims 1 to 39, further configured to generate the further neighbouring signals in the spatial neighbourhood (100) of the given block by using contribution signals of

41. The one or more third predetermined decoding tools (110 ３ ) corresponds to one or more intra prediction tools, and the one or more third predetermined decoding tools (110) for the predetermined block. ３ ) said contribution signal is an intra prediction signal, said one or more second predetermined decoding tools (110 ２ ) corresponds to one or more inter prediction tools, and the one or more second predetermined decoding tools (110) in the spatial neighborhood (100) of the predetermined block. ２ 41. The video decoder of claim 40, wherein the contribution signals of (a) and (b) correspond to inter prediction signals.

42. The first predetermined decoding tool (110 １ ) for the current block (18) the first predetermined decoding tool (110) １ ) a prediction residual signal as the contribution signal of deriving residual values ​​for the current block (18) from the data stream (12); predicting the sign of the derived residual value based on the neighboring signals (100') within the spatial neighborhood (100) of the current block (18); A video decoder according to any one of claims 1 to 41, configured to generate by

43. The video decoder (54) transmits the neighboring signals (100') within the spatial neighborhood (100) to one or more third predetermined decoding tools (110) within the spatial neighborhood (100). ３ 43. The video decoder of claim 42, configured to generate the spatial signal interdependence by replacing (122) the contribution signals of the first and second inputs of the first input signal with a replacement signal generated independently of the spatial signal interdependence.

44. The one or more third predetermined decoding tools (110 ３ 44. A video decoder according to claim 42 or 43, wherein the contributing signal of said first sub-frame is an intra-predicted signal and the substitution signal represents an inter-predicted signal.

45. The first predetermined decoding tool (110 １ ) uses template matching to determine the first predetermined decoding tool (110) for the current block (18). １ ) and the neighboring signal (100') within the spatial neighborhood (100) of the current block (18) represents a template for the template matching.

46. The first predetermined decoding tool (110 １ 46. ​​Video decoder according to any one of claims 1 to 45, comprising only one third predetermined decoding tool corresponding to:

47. The first predetermined decoding tool (110 １ ) determines the first predetermined decoding tool (110) for the current block (18) based on the neighboring signals (100') within the spatial neighborhood (100) of the current block (18). １ 47. The video decoder of claim 46, configured to generate the contribution signals:

48. 48. The video decoder of claim 1, wherein the video decoder is configured to generate the neighborhood signal within the spatial neighborhood by further applying a deblocking filter within the spatial neighborhood.

49. A picture processing tool according to any one of claims 81 to 103 is connected to the first predetermined decoding tool (110 １ 49. A video decoder according to any one of claims 1 to 48, comprising as

50. a video decoder (54) comprising a plurality of decoding tools (110) controlled by a data stream (12) and configured to apply the plurality of decoding tools (110) to a current picture (10) of video on a block-by-block basis, the plurality of decoding tools (110) including a first set of prediction tools, the video decoder (54) performing a block-by-block selection of exactly one prediction tool from the first set of prediction tools when applying the plurality of decoding tools (110) to the current picture (10); and deriving a reconstructed signal (58) of the currently decoded picture (10) by a sample-by-sample combination of a prediction signal generated by the first set of prediction tools and a prediction residual signal derived from the data stream (12); The plurality of decoding tools (110) includes a first predetermined decoding tool (110) configured to post-process prediction signals of one or more inter-prediction tools of the first set of prediction tools based on the neighboring signals (100') within a spatial neighborhood (100). １ ), The video decoder (54) uses the first predetermined decoding tool (110 １ ), and / or by replacing (122) the prediction signal of one or more intra prediction tools of the plurality of prediction tools with a replacement signal generated by inter prediction.

51. The first predetermined decoding tool (110 １ 51. The video decoder of claim 50, wherein the decoder is configured to post-process the prediction signals of the one or more inter prediction tools based on a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the prediction signals of the one or more inter prediction tools involving the neighboring signals (100′), using a neural network or convolution.

52. 52. The video decoder of claim 50 or claim 51, wherein the video decoder is configured to ignore a prediction residual signal when generating the neighboring signal when replacing the prediction signal of the one or more intra prediction tools of the plurality of prediction tools with the replacement signal generated by inter prediction.

53. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; applying a post-processing tool to a given predicted block and neighboring blocks that are in a spatial vicinity of the given predicted block and overlap the spatial vicinity (100) of the given predicted block, the post-processing tool comprising: post-processing a prediction signal of the predetermined prediction block based on neighboring signals in the spatial neighborhood (100) of the predetermined prediction block to obtain a post-processed prediction signal of the predetermined prediction block; - post-processing a prediction signal of the neighboring block based on further neighboring signals in a further spatial neighborhood (100) of the neighboring block to obtain a post-processed prediction signal of the neighboring block, the neighboring block overlapping with the spatial neighborhood (100); reconstructing said neighboring blocks by a sample-by-sample combination of said post-processed prediction signal and a prediction residual signal obtained from said data stream (12); forming the neighborhood signal (100') within the neighborhood block by using the predicted signal of a version of the neighborhood block that has not been post-processed by the post-processing tool; a video decoder (54) configured to:

54. 54. The video decoder of claim 53, wherein the video decoder is configured to perform a sample-by-sample addition of the prediction signal and the prediction residual signal of the neighboring block in forming the neighboring signal within the neighboring block.

55. 54. The video decoder of claim 53, wherein the video decoder is configured to ignore the prediction residual signal when forming the neighboring signal within the neighboring block.

56. The post-processing tool uses neural networks or convolution to configured to post-process the prediction signal of the predetermined prediction block based on a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the prediction signal of the predetermined prediction block involving the neighboring signals (100′); The post-processing tool may use further neural networks or further convolutions to 56. A video decoder according to any one of claims 53 to 55, configured to post-process the prediction signal of the neighboring block based on further 3D tensors (146) comprising one or more matrices derived from corresponding parts in one or more further reference pictures and one or more matrices derived from the prediction signal of the neighboring block involving the further neighboring signals (100').

57. 57. The video decoder of claim 56, wherein the neural network is equal to the further neural network and the convolution is equal to the further convolution.

58. The post-processing tool (110 １ ) selects the neural network and the further neural network from a set of two or more neural networks; or 57. A video decoder as claimed in claim 56, configured to select the convolution and the further convolution from a set of two or more convolutions.

59. The post-processing tool (110 １ 59. A video decoder according to claim 58, wherein a decoder (14) is configured to be controlled by the data stream (12) to select the neural network or the convolution and the further neural network or the further convolution.

60. The post-processing tool (110 １ ) the neural network or the convolution is the block shape of the predetermined predicted block, and / or a prediction mode associated with the given prediction block; and / or a temporal layer of a picture comprising the given predictive block and the neighboring blocks, and / or a quantization parameter value associated with the picture comprising the given predictive block or the given predictive block and the neighboring block; and / or a prediction residual signal associated with said predetermined prediction block, and / or if the given predictive block is associated with an inter prediction mode, a picture order count difference between a reference picture and the picture comprising the given predictive block and the neighboring blocks; and / or If the given prediction block is associated with an inter prediction mode, a motion vector associated with the given prediction block Depending on the configuration you choose, The post-processing tool (110 １ ) converts the further neural network or the further convolution into the block shape of the neighboring blocks, and / or a prediction mode associated with the neighboring block, and / or a temporal layer of the picture comprising the given prediction block and the neighboring blocks, and / or a quantization parameter value associated with the neighboring block or the picture comprising the given predictive block and the neighboring block; and / or a prediction residual signal associated with said neighboring block, and / or if the neighboring block is associated with an inter prediction mode, a picture order count difference between a reference picture and the picture comprising the given prediction block and the neighboring block; and / or If the neighboring block is associated with an inter prediction mode, a motion vector associated with the neighboring block 59. A video decoder as claimed in claim 58, configured to select in dependence on

61. 61. A video decoder according to any one of claims 58 to 60, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

62. 62. A video decoder according to any one of claims 58 to 61, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, type of convolution, and / or input tensor format.

63. The post-processing tool (110 １ 58. A video decoder according to claim 56 or 57, wherein a neural network or convolution and a further neural network or further convolution are configured to derive the neural network or the convolution and the further neural network or the further convolution from the data stream (12).

64. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and intra-prediction for intra-predicted blocks; applying a post-processing tool to a given inter-predicted block, the post-processing tool being configured to post-process an inter-predicted signal of the given inter-predicted block based on neighboring signals within a spatial neighborhood (100) of the given inter-predicted block to obtain a post-processed inter-predicted signal of the given inter-predicted block; reconstructing a neighboring block overlapping said spatial neighborhood (100) and being one of said intra-predicted blocks by sample-by-sample addition of an intra-predicted signal of said neighboring block and a prediction residual signal obtained from said data stream (12); and forming the neighborhood signal (100') within the neighborhood block by generating a replacement signal within an overlapping region of the spatial neighborhood (100) and the neighborhood block by inter-prediction.

65. 65. The video decoder of claim 64, wherein the post-processing tool is configured to post-process the inter-predicted signal of the given inter-predicted block based on a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the inter-predicted signal involving the neighboring signals (100′), using a neural network or convolution.

66. The post-processing tool (110 １ ) selects said neural network from a set of two or more neural networks; or 66. The video decoder of claim 65 configured to select the convolution from a set of two or more convolutions.

67. The post-processing tool (110 １ 67. The video decoder of claim 66, wherein a neural network selector (14) is configured to select the neural network or the convolution under control of the data stream (12).

68. The post-processing tool (110 １ ) the neural network or the convolution is the block shape of the predetermined inter-predicted block, and / or a prediction mode associated with the given inter-predicted block; and / or the temporal layer of the picture comprising the given inter-predicted block, and / or a quantization parameter value associated with the given inter-predicted block or the picture comprising the given inter-predicted block; and / or a prediction residual signal associated with the given inter-predicted block, and / or a picture order count difference between a reference picture and the picture comprising the given inter-predicted block; and / or A motion vector associated with the given inter-predicted block 67. A video decoder as claimed in claim 66, configured to select in dependence on

69. 69. A video decoder according to any one of claims 66 to 68, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

70. 70. A video decoder according to any one of claims 66 to 69, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, type of convolution, and / or input tensor format.

71. The post-processing tool (110 １ 66. The video decoder of claim 65, wherein a neural network or a convolution is configured to derive the neural network or the convolution from the data stream (12).

72. A video decoder (54) comprising a plurality of decoding tools (110), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; - intra-prediction of the intra-prediction block to obtain an intra-prediction signal for the respective block; motion compensated prediction for inter-predicted blocks to obtain an inter-predicted signal for the respective blocks; performing said block-based prediction by use of applying, for a current block (18) that is one of the inter-predicted blocks, a post-processing tool configured to post-process the inter-predicted signal of the current block (18) based on a neighboring signal (100′) within a spatial neighborhood (100) of the current block (18); The video decoder (54) converts the neighboring signal (100') into excluding (124) from said spatial neighborhood (100) samples associated with nearby intra-predicted blocks that overlap said spatial neighborhood (100); and / or using inter-predicted signals of nearby inter-predicted blocks within the spatial neighborhood (100) that overlap a version of the spatial neighborhood (100) that has not been post-processed by the post-processing tool; a video decoder (54) configured to form

73. 73. The video decoder of claim 72, wherein the post-processing tool is a local illumination compensation tool.

74. The post-processing tool may further include: adapting scaling and offset values ​​based on said neighboring signals (100'); using the scaling value and the offset value for post-processing the inter-predicted signal of the current block (18); 73. A video decoder according to claim 71 or 72, configured for post-processing by:

75. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing block-based prediction by using motion compensated prediction for inter-predicted blocks, and by using intra prediction for intra-predicted blocks, and by applying inter-intra prediction tools to inter-intra predicted blocks; applying the inter-intra prediction tool to a given inter-intra prediction block, the inter-intra prediction tool being configured to generate an inter-intra prediction signal for the given inter-intra prediction block based on neighboring signals within a spatial neighborhood (100) of the given inter-intra prediction block; reconstructing a first neighboring block overlapping the spatial neighborhood (100) and being one of the intra-predicted blocks by a sample-by-sample combination of an intra-predicted signal of the first neighboring block and a first prediction residual signal obtained from the data stream (12); forming the neighborhood signal (100') within the first neighborhood block by generating a first substitution signal within an overlapping region of the spatial neighborhood (100) and the first neighborhood block by inter prediction; a video decoder (54) configured to:

76. reconstructing a second neighboring block overlapping the spatial neighborhood (100) and being one of the inter-intra predicted blocks by a sample-by-sample combination of the inter-intra predicted signal of the second neighboring block and a second prediction residual signal obtained from the data stream (12); Within the second neighboring block, further by using the inter prediction component of the inter-intra prediction signal and ignoring the intra prediction component of the inter-intra prediction signal within an overlapping region between the spatial neighborhood (100) and the second neighboring block, and forming said proximity signal (100').

77. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction residual signal for a predetermined prediction block among the prediction blocks is performing the transform-based predictive residual coding to derive residual values ​​for the given predictive block from the data stream (12); predicting the sign of the derived residual value based on a neighboring signal (100') within a spatial neighborhood (100) of the given prediction block; performing the transform-based predictive residual coding to derive further residual values ​​from the data stream, and predicting signs of the derived further residual values ​​based on further neighboring signals in further spatial neighborhoods of the inter-predicted neighboring blocks, thereby reconstructing inter-predicted neighboring blocks that overlap the spatial neighborhood (100) by a sample-by-sample combination of inter-predicted signals of the inter-predicted neighboring blocks and inter-prediction residual signals obtained from the data stream (12); forming the neighborhood signal (100') in an overlapping region of the inter-predicted neighboring block and the spatial neighborhood (100) by using the inter-predicted signal of the inter-predicted neighboring block and ignoring the inter-predicted residual signal.

78. reconstructing a further inter-predicted neighboring block overlapping said spatial neighborhood (100) by a sample-by-sample combination of a further inter-predicted signal of said further inter-predicted neighboring block and said further inter-predicted residual signal generated without predicting the sign of the residual values ​​of said further inter-predicted residual signal; 78. The video decoder of claim 77, further configured to: when forming the neighborhood signal within the neighboring block, further use a sample-by-sample sum of the further inter-predicted signal of the further inter-predicted neighboring block and the further inter-predicted residual signal within an overlapping area of ​​the further inter-predicted neighboring block and the spatial neighborhood (100).

79. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction residual signal for a predetermined prediction block among the prediction blocks is performing the transform-based predictive residual coding to derive residual values ​​for the given predictive block from the data stream (12); predicting the sign of the derived residual value based on a neighboring signal (100') within a spatial neighborhood (100) of the given prediction block; reconstructing an intra-predicted neighboring block that overlaps with the spatial neighborhood (100) by a sample-by-sample combination of an intra-prediction signal of the intra-predicted neighboring block and an intra-prediction residual signal obtained from the data stream (12); and forming the neighboring signal (100') within the spatial neighborhood by generating a first replacement signal within an overlapping region of the spatial neighborhood (100) and the intra-predicted neighboring block by inter-prediction.

80. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction signal for a predetermined prediction block among the prediction blocks is performing template matching using a neighboring signal (100') within a spatial neighborhood (100) of the given predicted block as a template to find an error-minimizing template match; using a template match block associated with the error-minimizing template match as the prediction signal for the given prediction block; reconstructing an intra-predicted neighboring block overlapping with the spatial neighborhood (100) by a sample-by-sample combination of an intra-predicted signal of the intra-predicted neighboring block and the intra-prediction residual signal obtained from the data stream (12), and / or reconstructing the TM-predicted neighboring block overlapping with the spatial neighborhood (100) by a sample-by-sample combination of a TM-predicted residual signal obtained from the data stream (12) and a TM-predicted neighboring block obtained by performing template matching using a further neighboring signal in a further spatial neighborhood of the predetermined prediction block as a template for finding a further error-minimizing template match, and using a further template match block associated with the further error-minimizing template match as the TM-predicted neighboring block; within the intra-predicted neighboring block by generating a first substitution signal within an overlapping region of the spatial neighborhood (100) and the intra-predicted neighboring block by inter-prediction; and / or within the TM-predicted neighboring block by generating, by inter prediction, a second substitution signal within an overlapping region of the spatial neighborhood (100) and the TM-predicted neighboring block in a manner independent of the generation of the inter-predicted signal of the TM-predicted neighboring block; forming said proximity signal (100').

81. Picture Processing Tools (110 １ ) where Polyphase splitting (140) of the luma samples of the picture portion (11) into polyphase components and forming a matrix (142) for each polyphase component. １ ~142 ４ ) and The matrix of polyphase components (142 １ ~142 ４ ) to form the tensor (146); The tensor (146) is converted into the matrix (142 １ ~142 ４ ) as different channels to obtain an output tensor (148) consisting of a concatenation of output matrices, one output matrix for each polyphase component; forming a processed picture portion (11′) based on said output tensor (148) by an inverse polyphase decomposition (150), １ ).

82. 82. A picture processing tool (110) according to claim 81, configured to combine said picture portion (11) with said processed picture portion (11') to obtain a post-processed picture portion. １ ).

83. said picture portion (11) comprising a block of a picture with its spatial neighbours (100); The picture processing tool (110 １ ) divides the luma samples in the block and the spatial neighborhood into the polyphase components in the polyphase division (140) and divides the matrices (142) for each polyphase component. １ ~142 ４ 82. The picture processing tool (110) according to claim 81, configured to obtain １ ).

84. said processed picture portion has the same dimensions as said picture portion (11); The picture processing tool (110 １ )teeth, combining said picture portion (11) with said processed picture portion (11') to obtain an intermediate signal; and cropping the intermediate signal to obtain a post-processed picture portion. １ ).

85. said processed picture portion (11') has the same dimensions as said picture portion (11); The picture processing tool (110 １ )teeth, - cropping said picture portion (11) and said processed picture portion (11') to obtain a cropped picture portion and a cropped processed picture portion; and combining the cropped picture portion with the cropped processed picture portion to obtain a post-processed picture portion.

86. The picture processing tool (110 １ 86. A picture processing tool (110) according to any one of claims 81 to 85, wherein the picture processing tool (110) is a post-processing tool for inter-predicted blocks, and the picture portion (11) is an inter-prediction of a picture block received from an inter-prediction tool of a video decoder (54). １ ).

87. In the polyphase division (140), further dividing the luma samples of the corresponding portion in the reference picture into the polyphase components to further obtain a reference matrix for each polyphase component; In forming the tensor (146), the matrix (142 １ ~142 ４ 87. The picture processing tool (110) of claim 86, configured to: １ ).

88. the picture portion (11) comprises inter-predicted luma samples of a block with its spatial neighbors (100) of the picture, The picture processing tool (110 １ ) in the polymorphic division (140), 142. Separating the inter-predicted luma samples of the block and the luma samples in the spatial neighborhood into the polyphase components and １ ~142 ４ ) and and dividing luma samples of a reference picture portion including a corresponding block in a reference picture and a spatial neighborhood of the corresponding block into the polyphase components to obtain a reference matrix for each polyphase component. １ ).

89. said processed picture portion (11') has the same dimensions as said picture portion (11); The picture processing tool (110 １ )teeth, combining said picture portion (11) with said processed picture portion (11') to obtain an intermediate signal; 89. A picture processing tool according to claim 88, configured to: crop the intermediate signal to obtain a post-processed picture portion.

90. said processed picture portion (11') has the same dimensions as said picture portion (11); The picture processing tool (110 １ )teeth, - cropping said picture portion (11) and said processed picture portion (11') to obtain a cropped picture portion and a cropped processed picture portion; and combining the cropped picture portion with the cropped processed picture portion to obtain a post-processed picture portion.

91. the luma samples of the spatial neighborhood (100) of the block include intra-predicted samples and inter-predicted samples; The picture processing tool (110 １ ) before performing the polyphasic partitioning (140): replacing the intra-predicted samples of the spatial neighborhood (100) of the block with first replacement samples generated by inter-prediction; and / or The picture processing tool (110 １ 91. A picture processing tool (110) according to any one of claims 88 to 90, configured to use the inter-predicted samples of the spatial neighbours (100) of the block in a version that has not been post-processed by １ ).

92. the luma samples of the picture portion (11) have a two-dimensional arrangement along a first direction and a second direction, the second direction being perpendicular to the first direction; The picture processing tool (110 １ 92. A picture processing tool (110) according to any one of claims 81 to 91, wherein in said polyphase splitting (140), said luma samples are split into different ones of said polyphase components alternately in said first and second directions. １ ).

93. 93. The picture processing tool (110) of claim 92, wherein the luma samples are split into four polyphase components in the polyphase split (140). １ ).

94. the luma samples of the picture portion (11) have a two-dimensional arrangement along a first direction and a second direction, the second direction being perpendicular to the first direction; The picture processing tool (110 １ 85. A picture processing tool (110) according to any one of claims 81 to 84, wherein in the polyphase splitting (140), the luma samples are split into even and odd samples along the first and second directions to obtain four polyphase components. １ ).

95. A picture processing tool (110) according to any one of claims 81 to 94, configured to enable the picture portion (11) to correspond to one of a plurality of picture portion (11) dimensions. １ ).

96. a picture processing tool (110) configured to perform a convolution of said tensor (146) with said kernel of said neural network or said convolution; １ ) a picture processing tool (110) according to any one of claims 81 to 95, wherein the kernels are not different for different quantization parameter values ​​associated with the picture portion (11). １ ).

97. 99. A picture processing tool (110) according to any one of claims 81 to 98, wherein the neural network or the convolution comprises N layers, and the neural network or the convolution is configured to perform a layer-by-layer convolution (134) followed by a rectified linear unit activation (136) except for the last layer of the N layers, in which the rectified linear unit activation (136) is skipped. １ ).

98. selecting said neural network from a set of two or more neural networks; or A picture processing tool according to any one of claims 81 to 97, configured to select the convolution from a set of two or more convolutions.

99. 99. A picture processing tool according to claim 98, configured to be controlled by a data stream (12) to select the neural network or the convolution.

100. The neural network or the convolution, the shape of the picture portion (11), and / or a prediction mode associated with said picture portion (11), and / or the temporal layer of the picture containing said picture portion (11), and / or a quantization parameter value associated with said picture portion (11) or said picture containing said picture portion (11), and / or a prediction residual signal associated with said picture portion (11), and / or if said picture portion (11) is associated with an inter-prediction mode, the picture order count difference between a reference picture and said picture containing said picture portion (11); and / or If said picture portion (11) is associated with an inter prediction mode, a motion vector associated with said picture portion (11) 99. A picture processing tool according to claim 98, configured to select in dependence on

101. 101. A picture processing tool according to any one of claims 98 to 100, wherein the neural networks of the set of two or more neural networks differ from each other in terms of weights, biases, number of layers, layer types, and / or input tensor format.

102. 102. A picture processing tool according to any one of claims 98 to 101, wherein the convolutions of the set of two or more convolutions differ from each other in terms of weights, biases, type of convolution, and / or input tensor format.

103. A picture processing tool according to any one of claims 81 to 97, configured to derive said neural network or said convolution from a data stream (12).

104. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing, for inter-predicted blocks, the block-based prediction using motion compensated prediction controlled via motion vectors derived from the data stream (12); a post-processing tool (110) for post-processing an inter-predicted signal of a given inter-predicted block (18); １ ) and From the predetermined inter-predicted block (18), a first inter-predicted block having one or more associated motion vectors (200) whose number satisfying a first predetermined criterion is zero according to said data stream (12); and / or a second inter-predicted block having one or more motion vectors (200) associated therewith, according to said data stream (12), a number of which satisfy a second predetermined criterion being full-pel motion vectors; and / or a third inter-predicted block having one of a set of predetermined inter-prediction modes associated therewith according to the data stream (12), the set of predetermined inter-prediction modes including one or more of a uni-prediction mode, a merge mode, and a bi-prediction mode using coding unit weights; and / or a fourth inter-predicted block whose block shape satisfies a third predetermined criterion; and / or a fifth inter-prediction block in which the data stream (12) signals a quantization parameter having a value that satisfies a fourth predetermined criterion; and identifying the predetermined inter-predicted block (18) from among the inter-predicted blocks by filtering out (124) the predetermined inter-predicted block (18).

105. The post-processing tool (110 １ ) uses a neural network or convolution to generate one or more matrices (142) derived from the inter-predicted signal with neighboring signals in a spatial neighborhood (100) of the given inter-predicted block (18), and one or more matrices (142) derived from corresponding portions in one or more reference pictures. １ ~142 ４ 105. The video decoder of claim 104, configured to post-process the inter-predicted signal based on a 3D tensor (146) comprising:

106. The post-processing tool (110 １ ) selects said neural network from a set of two or more neural networks; or 106. The video decoder of claim 105 configured to select the convolution from a set of two or more convolutions.

107. The post-processing tool (110 １ 107. The video decoder of claim 106, wherein a neural network selector (14) is configured to select the neural network or the convolution under control of the data stream (12).

108. The post-processing tool (110 １ ) the neural network or the convolution is the block shape of the predetermined inter-predicted block, and / or a prediction mode associated with the given inter-predicted block; and / or the temporal layer of the picture comprising the given inter-predicted block, and / or a quantization parameter value associated with the given inter-predicted block or the picture comprising the given inter-predicted block; and / or a prediction residual signal associated with the given inter-predicted block, and / or a picture order count difference between a reference picture and the picture comprising the given inter-predicted block; and / or A motion vector associated with the given inter-predicted block 107. A video decoder according to claim 106, configured to select in dependence on:

109. 109. A video decoder according to any one of claims 106 to 108, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

110. 110. A video decoder according to any one of claims 106 to 109, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, type of convolution, and / or input tensor format.

111. The post-processing tool (110 １ 106. The video decoder of claim 105, wherein a neural network or a convolution is configured to derive the neural network or the convolution from the data stream (12).

112. The post-processing tool (110 １ )teeth, adapting scaling and offset values ​​based on neighboring signals within a spatial neighborhood (100) of the given inter-predicted block (18); using the scaling value and the offset value for post-processing the inter-predicted signal; 112. The video decoder of claim 111, configured to post-process the inter-predicted signal by:

113. The spatial neighborhood (100) includes intra-predicted and inter-predicted portions, and the video decoder (54) divides the neighborhood signal into Replacing (122) the intra-predicted signal in the intra-predicted portion with a replacement signal generated by inter-prediction, and / or The post-processing tool (110 １ ) using a version of the inter-predicted signal in the inter-predicted portion that has not been post-processed by removing (124) signals associated with the intra-predicted portion; 113. A video decoder according to claim 111 or 112, configured to form by

114. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and by using intra prediction for intra-predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; A post-processing tool (110) is applied to a first neighboring block that overlaps the spatial neighborhood (100) and is one of the inter-predicted blocks. １ ), wherein said post-processing tool (110) １ ) is configured to post-process the inter prediction signal of the first neighboring block to obtain a post-processed inter prediction signal; reconstructing the first neighboring block by a sample-by-sample combination of the post-processed inter prediction signal of the first neighboring block and a first prediction residual signal obtained from the data stream (12); The neighboring signals in the spatial neighborhood (100) are processed by the post-processing tool (110) in the first neighborhood block. １ by using the inter prediction signal of the first neighboring block in a version that has not been post-processed by

115. The post-processing tool (110 １ ) is generated by using a neural network or convolution to generate one or more matrices (142) derived from corresponding portions in one or more reference pictures. １ ~142 ４ ) and one or more matrices (142) derived from the inter-prediction signal. １ ~142 ４ 115. The video decoder of claim 114, configured to post-process the inter prediction signal of the first neighboring block based on a 3D tensor (146) comprising:

116. The post-processing tool (110 １ ) selects said neural network from a set of two or more neural networks; or 116. The video decoder of claim 115 configured to select the convolution from a set of two or more convolutions.

117. The post-processing tool (110 １ 117. The video decoder of claim 116, wherein a neural network selector (14) is configured to select the neural network or the convolution under control of the data stream (12).

118. The post-processing tool (110 １ ) the neural network or the convolution is the block shape of the first neighboring block, and / or a prediction mode associated with the first neighboring block, and / or a temporal layer of a picture comprising said first neighboring block, and / or a quantization parameter value associated with the first neighboring block or the picture containing the first neighboring block, and / or a prediction residual signal associated with said first neighboring block, and / or a picture order count difference between a reference picture and the picture comprising the first neighboring block, and / or a motion vector associated with the first neighboring block; 117. A video decoder according to claim 116, configured to select in dependence on:

119. 119. A video decoder according to any one of claims 116 to 118, wherein the neural networks of the set of two or more neural networks differ from each other in terms of weights, biases, number of layers, layer types, and / or input tensor format.

120. 120. A video decoder according to any one of claims 116 to 119, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, type of convolution, and / or input tensor format.

121. The post-processing tool (110 １ 116. The video decoder of claim 115, wherein a neural network or a convolution is configured to derive the neural network or the convolution from the data stream (12).

122. The post-processing tool (110 １ ) converts the inter prediction signal of the first neighboring block into determining a scaling value and an offset value based on a spatial neighborhood (100) of the first neighboring block; using the scaling value and the offset value for post-processing the inter-predicted signal of the first neighboring block; 112. A video decoder according to claim 111, configured for post-processing by:

123. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using intra prediction for intra-predicted blocks and by using inter-intra prediction for inter-intra predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; inter-intra predicting a first neighboring block that overlaps the spatial neighborhood (100) and is one of the inter-intra predicted blocks to obtain an inter-intra predicted signal of the first neighboring block, wherein the inter-intra predicted signal corresponds to a weighted combination of the intra predicted signal and the inter predicted signal of the first neighboring block; reconstructing the first neighboring block by a sample-by-sample combination of the inter-intra prediction signal of the first neighboring block and a first prediction residual signal obtained from the data stream (12); and forming the neighborhood signal within the spatial neighborhood (100) by using the inter-predicted signal of the first neighboring block rather than the intra-predicted signal of the first neighboring block within the first neighboring block.

124. A video decoder (54), decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by use of intra prediction for intra-predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; applying a residual code prediction tool to a first neighboring block overlapping said spatial neighborhood (100) to obtain a predicted residual signal of said first neighboring block; reconstructing the first neighboring block by a sample-by-sample combination of the prediction signal of the first neighboring block and the prediction residual signal of the first neighboring block; and forming the neighborhood signal within the spatial neighborhood (100) by using the prediction signal of the first neighboring block that is not combined with the prediction residual signal of the first neighboring block within the first neighboring block.

125. The residual code prediction tool deriving a residual value for the first neighboring block from the data stream (12); 125. The video decoder of claim 124, configured to: predict a sign of the residual value based on a spatial neighborhood (100) of the first neighboring block.

126. 126. A video decoder as described in claim 124 or claim 125, wherein the video decoder (54) is configured to intra predict the first neighboring block, which is one of the intra-predicted blocks, to obtain an intra-prediction signal as the prediction signal of the first neighboring block.

127. The video decoder (54) performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and by using the intra prediction for the intra-predicted blocks; and inter predicting the first neighboring block, the first neighboring block being one of the inter predicted blocks, to obtain an inter predicted signal as the predicted signal of the first neighboring block.

128. The video decoder (54) performing the block-based prediction by using inter-intra prediction for inter-intra predicted blocks and by using the intra prediction for the intra predicted blocks; and performing inter-intra prediction on the first neighboring block, the first neighboring block being one of the inter-intra predicted blocks, to obtain an inter-intra predicted signal as the predicted signal of the first neighboring block.

129. A video encoder (14) comprising a plurality of encoding tools (110), configured to apply the plurality of encoding tools (110) to a current picture (10) of a video (16) on a block-by-block basis; a reconstructed signal (58) of the currently coded picture (10) is derivable by a sample-by-sample combination of contributing signals generated by the plurality of coding tools (110); The plurality of encoding tools (110) are configured to, based on a neighborhood signal (100') within a spatial neighborhood (100) of the current block (18), One or more second predetermined encoding tools (110) in the current block (18). ２ ) contribution signal, or intermediate signals in the current block (18) corresponding to partial combinations from the sample-by-sample combinations; Post-processing the or The first predetermined encoding tool (110) for the current block (18) １ a first predetermined encoding tool (110) configured to generate a contribution signal of １ ), The video encoder (14) encodes the neighborhood signal (100') within the spatial neighborhood (100) as: said one or more second predetermined coding tools (110 ２ ) or said first predetermined coding tool (110 １ ) using intermediate signals within said spatial neighborhood (100) in a version that has not been post-processed by One or more third predetermined coding tools (110) within said spatial neighborhood (100). ３ ) by a replacement signal generated independently of spatial signal interdependencies (122); and / or From the spatial neighborhood (100), the sample-by-sample combination for the derivation of the reconstructed signal (58) is performed by the one or more third predetermined coding tools (110). ３ excluding (124) samples containing said contribution signal of a video encoder (14) configured to generate the

130. The first predetermined encoding tool (110 １ ) based on the neighborhood signals (100') within the spatial neighborhood (100) of the current block (18), The one or more second predetermined encoding tools (110) in the current block (18) ２ ) configured to post-process the contribution signals of The video encoder (14) controls the generation of the neighborhood signal (100') within the spatial neighborhood (100) by: The first predetermined encoding tool (110 １ the one or more second predetermined coding tools (110) in the spatial neighborhood (100) in the version that has not been post-processed by ２ ) using said contribution signals of The first predetermined encoding tool (110 １ the one or more second predetermined coding tools (110) in the spatial neighborhood (100) in the version that has not been post-processed by ２ ) combining said contribution signals with a prediction residual signal by sample-by-sample combination; 130. A video encoder (14) according to claim 129, configured to perform by

131. The video encoder (14) controls the generation of the neighborhood signal (100') within the spatial neighborhood (100) by: The one or more third predetermined coding tools (110) within the spatial neighborhood (100) ３ ) by the replacement signal generated independently of spatial signal interdependencies, A video encoder (14) as described in claim 129 or claim 130, wherein the substitution signal represents a prediction signal, and the video encoder (14) is configured to refrain from combining the substitution signal with a prediction residual signal when generating the neighboring signal (100') within the spatial neighborhood (100) of the current block (18).

132. The video encoder (14) controls the generation of the neighborhood signal (100') within the spatial neighborhood (100) by: The one or more third predetermined coding tools (110) within the spatial neighborhood (100) ３ ) by the replacement signal generated independently of spatial signal interdependencies, The video encoder (14) uses the one or more second predetermined encoding tools (110 ２ 132. A video encoder (14) according to any one of claims 129 to 131, configured to generate the replacement signal using

133. The video encoder (14) controls the generation of the neighborhood signal (100') within the spatial neighborhood (100) by: The one or more third predetermined coding tools (110) within the spatial neighborhood (100) ３ ) by the replacement signal generated independently of spatial signal interdependencies, The replacement signal is generated by the first predetermined encoding tool (110 １ the one or more second predetermined coding tools (110) in the spatial neighborhood (100) of the current block (18) in a version that has not been post-processed by ２ ) contribution signals.

134. said one or more second predetermined coding tools (110 ２ 134. A video encoder (14) according to claim 132 or claim 133, wherein one or more inter prediction tools are present, and the replacement signal represents an inter prediction signal within the spatial neighborhood (100) of the current block (18).

135. The first predetermined encoding tool (110 １ ) based on the neighboring signals (100') in the spatial neighborhood (100) of the current block (18), the intermediate signals in the current block (18) corresponding to a partial combination from the sample-by-sample combinations; configured to post-process the The video encoder (14) The first predetermined encoding tool (110 １ ) using the intermediate signals within the spatial neighborhood (100) in the version that has not been post-processed by configured to generate the proximity signal (100') within the spatial proximity (100) by The intermediate signals in the current block (18) and the intermediate signals in the spatial neighborhood (100) of the current block (18) are respectively encoded by the one or more second predetermined encoding tools (110) in the current block (18) and in the spatial neighborhood (100) of the current block (18). ２ 135. A video encoder (14) according to any one of claims 129 to 134, which corresponds to a sample-by-sample combination of two or more contributing signals of

136. A video encoder (14) according to any one of claims 129 to 135, wherein the video encoder (14) is a predictive encoder using transform-based residual coding, and the contributing signals comprise a prediction signal and a prediction residual signal.

137. A video encoder (14) according to any one of claims 129 to 136, wherein the video encoder (14) is configured to subject the reconstructed signal (58) to an in-loop filter and / or a post-processing filter.

138. said one or more second predetermined coding tools (110 ２ ) by said one or more second predetermined decoding tools (110 ２ 138. A video encoder (14) according to any one of claims 129 to 137, wherein the one or more inter-prediction coding tools are configured to generate an inter-prediction signal as the contributing signal of

139. When applying the plurality of coding tools (110) to the current picture (10) on a block-by-block basis, determining, for each of one or more first coding tools, on a block-by-block basis whether to apply the respective first coding tool; for each of one or more second coding tools, parameterizing on a block-by-block basis when applying the respective second coding tool; deriving, for each of one or more third coding tools, a block subdivision for performing the block-by-block application of the respective third coding tool; A video encoder (14) according to any one of claims 129 to 138, configured to apply one or more of the following:

140. 140. The video encoder (14) of claim 139, wherein one or more encoding tools may belong to two or more, or all, of the one or more first encoding tools, the one or more second encoding tools, and the one or more third encoding tools.

141. The first predetermined encoding tool (110 １ ) selects the one or more second predetermined encoding tools (110) in the current block (18). ２ ), and the video encoder (14) is configured to post-process the contributing signals of the neighboring signals (100′) within the spatial neighborhood (100): The first predetermined encoding tool (110 １ the one or more second predetermined coding tools (110) in the spatial neighborhood (100) in the version that has not been post-processed by ２ ) and / or The one or more third predetermined coding tools (110) within the spatial neighborhood (100) ３ ) by said replacement signal generated independently of spatial signal interdependencies (122). A video encoder (14) according to any one of claims 129 to 140, configured to generate by

142. The first predetermined encoding tool (110 １ ) uses a neural network or convolution to generate the one or more second predetermined encoding tools (110) based on a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the contributing signals with the neighboring signals (100'). ２ 142. The video encoder (14) of claim 141, configured to post-process the contribution signals of

143. The first predetermined encoding tool (110 １ ) selects said neural network from a set of two or more neural networks; or 143. The video encoder of claim 142 configured to select the convolution from a set of two or more convolutions.

144. The first predetermined encoding tool (110 １ 144. The video encoder of claim 143, configured to encode the neural network or the convolution selected by

145. The first predetermined encoding tool (110 １ ) the neural network or the convolution is the block shape of the current block (18), and / or a prediction mode associated with the current block (18), and / or the temporal layer of the current picture (10), and / or a quantization parameter value associated with the current picture (10) or the current block (18), and / or a prediction residual signal associated with the current block (18), and / or if the current block (18) is associated with an inter-prediction mode, the picture order count difference between the current picture (10) and a reference picture; and / or If the current block (18) is associated with an inter prediction mode, a motion vector associated with the current block (18) 144. A video encoder as claimed in claim 143, configured to select in dependence on:

146. 146. A video encoder according to any one of claims 143 to 145, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

147. 147. A video encoder according to any one of claims 143 to 146, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, convolution type, and / or input tensor format.

148. The first predetermined encoding tool (110 １ 142. The video encoder (14) of claim 141, wherein the local illumination compensation tool is a local illumination compensation tool.

149. The first predetermined encoding tool (110 １ ) selects the one or more second predetermined encoding tools (110) in the current block (18). ２ ) said contribution signals, adapting scaling and offset values ​​based on said neighboring signals (100'); The one or more second predetermined encoding tools (110) in the current block (18) ２ using the scaling and offset values ​​to post-process the contribution signals of 149. A video encoder (14) according to claim 148, configured for post-processing by:

150. A video encoder (14) as described in any one of claims 129 to 149, wherein the plurality of encoding tools (110) comprises a first set of encoding tools, and the video encoder (14) is configured to perform a block-wise selection of exactly one encoding tool from the first set of encoding tools when applying the plurality of encoding tools (110) to the current picture (10) on a block-by-block basis.

151. said one or more third predetermined coding tools (110 ３ 151. The video encoder (14) of claim 150, wherein the first set of encoding tools includes:

152. 152. A video encoder (14) according to claim 150 or claim 151, wherein the first set of encoding tools comprises one or more inter-prediction encoding tools configured to generate an inter-prediction signal and one or more intra-prediction encoding tools configured to generate an intra-prediction signal.

153. said one or more third predetermined coding tools (110 ３ 153. The video encoder (14) of claim 152, wherein the one or more intra-prediction coding tools include:

154. said one or more second predetermined coding tools (110 ３ 154. A video encoder as claimed in claim 152 or claim 153, wherein the one or more inter-prediction coding tools include:

155. The first predetermined encoding tool (110 １ ) is configured to post-process contribution signals of the one or more inter-prediction coding tools in the current block (18) based on the neighborhood signal (100′), and the video encoder (14) processes the neighborhood signal (100′) in the spatial neighborhood (100) by: The first predetermined encoding tool (110 １ ) using the contribution signals of the one or more inter-prediction coding tools within the spatial neighborhood (100) in a version that has not been post-processed by replacing (122) a contributing signal of at least one of the one or more intra-prediction coding tools within the spatial neighborhood (100) with a replacement signal generated independently of spatial signal interdependencies; A video encoder (14) according to any one of claims 152 to 154, configured to generate by

156. The first predetermined encoding tool (110 １ 156. The video encoder (14) of claim 155, wherein the one or more inter-prediction coding tools are configured to post-process the contribution signals of the one or more inter-prediction coding tools based on a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the contribution signals with the neighboring signals (100′), using a neural network or convolution.

157. The first predetermined encoding tool (110 １ ) selects said neural network from a set of two or more neural networks; or 157. The video encoder of claim 156 configured to select the convolution from a set of two or more convolutions.

158. The first predetermined encoding tool (110 １ 158. The video encoder of claim 157, configured to encode the neural network or the convolution selected by

159. The first predetermined encoding tool (110 １ ) the neural network or the convolution is the block shape of the current block (18), and / or a prediction mode associated with the current block (18), and / or the temporal layer of the current picture (10), and / or a quantization parameter value associated with the current picture (10) or the current block (18), and / or a prediction residual signal associated with the current block (18), and / or if the current block (18) is associated with an inter-prediction mode, the picture order count difference between the current picture (10) and a reference picture; and / or If the current block (18) is associated with an inter prediction mode, a motion vector associated with the current block (18) 159. A video encoder as claimed in claim 158, configured to select in dependence on

160. 160. A video encoder according to any one of claims 157 to 159, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

161. 161. A video encoder according to any one of claims 157 to 160, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, convolution type, and / or input tensor format.

162. The first predetermined encoding tool (110 １ 155. The video encoder (14) of claim 152, wherein one or more intra-prediction coding tools are included in the first set of coding tools and are configured to generate a prediction signal for the current block (18) using a combination of inter-prediction and intra-prediction based on the neighborhood signal (100′), and the video encoder (14) is configured to generate the neighborhood signal (100′) within the spatial neighborhood (100) by replacing (122) the contributing signal of at least one of the one or more intra-prediction coding tools within the spatial neighborhood (100) with a replacement signal generated independently of spatial signal interdependencies.

163. A video encoder (14) according to any one of claims 150 to 162, wherein the encoding tools in the first set of encoding tools are predictive encoding tools configured to generate a prediction signal.

164. A video encoder (14) according to any one of claims 129 to 163, configured to generate the replacement signal using inter prediction.

165. A video encoder (14) according to any one of claims 129 to 164, configured to generate the replacement signal using inter prediction using motion vectors.

166. The first predetermined encoding tool (110 １ ) selects the one or more second predetermined encoding tools (110) in the current block (18). ２ ), and the video encoder (14) is configured to post-process the contribution signals of the first predetermined encoding tool (110). １ the one or more second predetermined coding tools (110) in the spatial vicinity (100) of the version that has not been post-processed by ２ ) to generate the neighborhood signal (100') within the spatial neighborhood (100), by using the contributing signals of The video encoder (14) and determining the one or more third predetermined coding tools (110) for the predetermined block based on further neighboring signals within the spatial neighborhood (100) of the predetermined block. ３ ) contribution signal, The first predetermined encoding tool (110 １ the one or more second predetermined coding tools (110) in the spatial vicinity (100) of the predetermined block in a version that has not been post-processed by ２ 166. A video encoder (14) according to any one of claims 129 to 165, further configured to generate the further neighbourhood signal in the spatial neighbourhood (100) of the given block by using contribution signals of

167. said one or more third predetermined coding tools (110 ３ ) corresponds to one or more intra prediction tools, and the one or more third predetermined encoding tools (110) for the predetermined block. ３ ) said contribution signal is an intra prediction signal, said one or more second predetermined coding tools (110 ２ ) corresponds to one or more inter prediction tools and is used to predict the one or more second predetermined coding tools (110) in the spatial neighborhood (100) of the predetermined block. ２ 167. The video encoder (14) of claim 166, wherein the contribution signals of (a) correspond to inter prediction signals.

168. The first predetermined encoding tool (110 １ ) selects the first predetermined encoding tool (110) for the current block (18). １ ) a prediction residual signal as the contribution signal of determining a residual value of the current block (18); predicting the sign of the residual value based on the neighboring signals (100') within the spatial neighborhood (100) of the current block (18); A video encoder (14) according to any one of claims 129 to 176, configured to generate by

169. The video encoder (14) encodes the neighboring signals (100') within the spatial neighborhood (100) to one or more third predetermined encoding tools (110) within the spatial neighborhood (100). ３ 169. The video encoder (14) of claim 168, configured to generate the spatial signal interdependence by replacing (122) the contribution signals of the first and second inputs of the first input signal with a replacement signal generated independently of the spatial signal interdependence.

170. said one or more third predetermined coding tools (110 ３ 168. A video encoder as claimed in claim 167, wherein the contributing signal of said first sub-frame is an intra-predicted signal and the substitution signal represents an inter-predicted signal.

171. The first predetermined encoding tool (110 １ ) uses template matching to determine the first predetermined encoding tool (110) for the current block (18). １ ), wherein the neighborhood signal (100') within the spatial neighborhood (100) of the current block (18) represents a template for the template matching.

172. The first predetermined encoding tool (110 １ 172. A video encoder (14) according to any one of claims 129 to 171, comprising only one third predetermined encoding tool corresponding to:

173. The first predetermined encoding tool (110 １ ) determines the first predetermined coding tool (110) for the current block (18) based on the neighboring signals (100') within the spatial neighborhood (100) of the current block (18). １ 173. The video encoder (14) of claim 172, configured to generate the contribution signals:

174. A video encoder (14) according to any one of claims 129 to 173, wherein the video encoder (14) is configured to generate the neighbourhood signal (100') within the spatial neighbourhood (100) by further applying a deblocking filter within the spatial neighbourhood (100).

175. The picture processing tool according to any one of claims 47 to 63 is connected to the first predetermined encoding tool (110 １ 175. A video encoder (14) according to any one of claims 129 to 174, comprising:

176. A video encoder (14) comprising a plurality of encoding tools (110), the video encoder (14) is configured to apply the plurality of encoding tools (110) to a current picture (10) of video on a block-by-block basis, the plurality of encoding tools (110) including a first set of prediction tools, and the video encoder (14) is configured to perform a block-by-block selection of exactly one prediction tool from the first set of prediction tools when applying the plurality of encoding tools (110) to the current picture (10); a reconstructed signal (58) of the currently coded picture (10) is derivable by a sample-by-sample combination of a prediction signal and a prediction residual signal generated by the first set of prediction tools; The plurality of encoding tools (110) includes a first predetermined encoding tool (110) configured to post-process prediction signals of one or more inter-prediction tools of the first set of prediction tools based on the neighboring signals (100') within a spatial neighborhood (100). １ ), The video encoder (14) uses the first predetermined encoding tool (110 １ ), and / or by replacing (122) a prediction signal of one or more intra prediction tools among the plurality of prediction tools with a replacement signal generated by inter prediction.

177. The first predetermined encoding tool (110 １ 177. The video encoder (14) of claim 176, wherein the video encoder (14) is configured to post-process the prediction signals of the one or more inter prediction tools based on a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the prediction signals of the one or more inter prediction tools involving the neighboring signals (100′), using a neural network or convolution.

178. 178. A video encoder (14) according to claim 167 or claim 177, wherein the video encoder is configured to ignore a prediction residual signal when generating the neighboring signal when replacing the prediction signal of the one or more intra prediction tools among the plurality of prediction tools with the replacement signal generated by inter prediction.

179. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; applying a post-processing tool to a given predicted block and neighboring blocks that are in a spatial vicinity of the given predicted block and overlap the spatial vicinity (100) of the given predicted block, the post-processing tool comprising: post-processing a prediction signal of the predetermined prediction block based on neighboring signals in the spatial neighborhood (100) of the predetermined prediction block to obtain a post-processed prediction signal of the predetermined prediction block; post-processing a prediction signal of the neighboring block based on a further neighboring signal in a further spatial neighborhood (100) of the neighboring block to obtain a post-processed prediction signal of the neighboring block, the neighboring block overlapping with the spatial neighborhood (100); the neighboring blocks are reconstructable by a sample-by-sample combination of the post-processed prediction signal and a prediction residual signal; and forming the neighborhood signal (100') within the neighborhood block by using the predicted signal of a version of the neighborhood block that has not been post-processed by the post-processing tool; a video encoder (14) configured to:

180. 180. The video encoder of claim 179, wherein the video encoder is configured to perform a sample-by-sample addition of the prediction signal and the prediction residual signal of the neighboring block when forming the neighboring signal within the neighboring block.

181. 180. The video encoder of claim 179, wherein the video encoder is configured to ignore the prediction residual signal when forming the neighboring signal within the neighboring block.

182. The post-processing tool uses neural networks or convolution to configured to post-process the prediction signal of the predetermined prediction block based on a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the prediction signal of the predetermined prediction block involving the neighboring signals (100′); The post-processing tool may use further neural networks or further convolutions to 182. A video encoder according to any one of claims 179 to 181, configured to post-process the prediction signal of the neighboring block based on further 3D tensors (146) comprising one or more matrices derived from corresponding parts in one or more further reference pictures and one or more matrices derived from the prediction signal of the neighboring block involving the further neighboring signal (100').

183. 183. The video encoder of claim 182, wherein the neural network is equal to the further neural network and the convolution is equal to the further convolution.

184. The post-processing tool (110 １ ) selects the neural network and the further neural network from a set of two or more neural networks; or 183. The video encoder of claim 182 configured to select the convolution and the further convolution from a set of two or more convolutions.

185. The post-processing tool (110 １ 185. A video encoder according to claim 184, configured to encode the neural network or the convolution selected by (a) selecting the neural network or the convolution and the further neural network or the further convolution into the data stream.

186. The post-processing tool (110 １ ) the neural network or the convolution is the block shape of the predetermined predicted block, and / or a prediction mode associated with the given prediction block; and / or a temporal layer of a picture comprising the given predictive block and the neighboring blocks, and / or a quantization parameter value associated with the picture comprising the given predictive block or the given predictive block and the neighboring block; and / or a prediction residual signal associated with said predetermined prediction block, and / or if the given predictive block is associated with an inter prediction mode, a picture order count difference between a reference picture and the picture comprising the given predictive block and the neighboring blocks; and / or If the given prediction block is associated with an inter prediction mode, a motion vector associated with the given prediction block Depending on the configuration you choose, The post-processing tool (110 １ ) converts the further neural network or the further convolution into the block shape of the neighboring blocks, and / or a prediction mode associated with the neighboring block, and / or a temporal layer of the picture comprising the given prediction block and the neighboring blocks, and / or a quantization parameter value associated with the neighboring block or the picture comprising the given predictive block and the neighboring block; and / or a prediction residual signal associated with said neighboring block, and / or if the neighboring block is associated with an inter prediction mode, a picture order count difference between a reference picture and the picture comprising the given prediction block and the neighboring block; and / or If the neighboring block is associated with an inter prediction mode, a motion vector associated with the neighboring block 185. A video encoder as claimed in claim 184, configured to select depending on:

187. 187. A video encoder as claimed in any one of claims 184 to 186, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

188. 188. A video encoder according to any one of claims 184 to 187, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, convolution type, and / or input tensor format.

189. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and intra-prediction for intra-predicted blocks; applying a post-processing tool to a predetermined inter-predicted block, the post-processing tool being configured to post-process an inter-predicted signal of the predetermined inter-predicted block based on neighboring signals within a spatial neighborhood (100) of the predetermined inter-predicted block to obtain a post-processed inter-predicted signal of the predetermined inter-predicted block; a neighboring block overlapping the spatial neighborhood (100) and being one of the intra-predicted blocks can be reconstructed by a sample-by-sample addition of an intra-predicted signal of the neighboring block and a prediction residual signal of the neighboring block; and forming the neighborhood signal (100') within the neighborhood block by generating a replacement signal within an overlapping region of the spatial neighborhood (100) and the neighborhood block by inter prediction.

190. 190. The video encoder (14) of claim 189, wherein the post-processing tool is configured to post-process the inter-predicted signal of the given inter-predicted block based on a 3D tensor (146) including one or more matrices derived from corresponding portions in one or more reference pictures and one or more matrices derived from the inter-predicted signal involving the neighboring signal (100′), using a neural network or convolution.

191. The post-processing tool (110 １ ) selects said neural network from a set of two or more neural networks; or 191. The video encoder of claim 190 configured to select the convolution from a set of two or more convolutions.

192. The post-processing tool (110 １ 192. The video encoder of claim 191, configured to encode the neural network or the convolution selected by

193. The post-processing tool (110 １ ) the neural network or the convolution is the block shape of the predetermined inter-predicted block, and / or a prediction mode associated with the given inter-predicted block; and / or the temporal layer of the picture comprising the given inter-predicted block, and / or a quantization parameter value associated with the given inter-predicted block or the picture comprising the given inter-predicted block; and / or a prediction residual signal associated with the given inter-predicted block, and / or a picture order count difference between a reference picture and the picture comprising the given inter-predicted block; and / or A motion vector associated with the given inter-predicted block 192. A video encoder as claimed in claim 191, configured to select in dependence on:

194. 194. A video encoder according to any one of claims 191 to 193, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

195. 195. A video encoder according to any one of claims 191 to 194, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, convolution type, and / or input tensor format.

196. A video encoder (14) comprising a plurality of encoding tools (110), encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; - intra-prediction of the intra-prediction block to obtain an intra-prediction signal for the respective block; motion compensated prediction for inter-predicted blocks to obtain an inter-predicted signal for the respective blocks; performing said block-based prediction by use of applying, for a current block (18) that is one of the inter-predicted blocks, a post-processing tool configured to post-process the inter-predicted signal of the current block (18) based on a neighboring signal (100′) within a spatial neighborhood (100) of the current block (18); The video encoder (14) converts the neighboring signal (100') into excluding (124) from said spatial neighborhood (100) samples associated with nearby intra-predicted blocks that overlap said spatial neighborhood (100); and / or using inter-predicted signals of nearby inter-predicted blocks within the spatial neighborhood (100) that overlap a version of the spatial neighborhood (100) that has not been post-processed by the post-processing tool; a video encoder (14) configured to form a

197. 200. The video encoder (14) of claim 196, wherein the post-processing tool is a local illumination compensation tool.

198. The post-processing tool may further include: adapting scaling and offset values ​​based on said neighboring signals (100'); using the scaling value and the offset value for post-processing the inter-predicted signal of the current block (18); 198. A video encoder (14) according to claim 196 or 197, configured for post-processing by

199. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing block-based prediction by using motion compensated prediction for inter-predicted blocks, and by using intra prediction for intra-predicted blocks, and by applying inter-intra prediction tools to inter-intra predicted blocks; applying the inter-intra prediction tool to a given inter-intra prediction block, the inter-intra prediction tool being configured to generate an inter-intra prediction signal for the given inter-intra prediction block based on neighboring signals within a spatial neighborhood (100) of the given inter-intra prediction block; a first neighboring block overlapping the spatial neighborhood (100) and being one of the intra-predicted blocks can be reconstructed by a sample-by-sample combination of an intra-predicted signal of the first neighboring block and a first prediction residual signal of the first neighboring block; generating a first substitution signal in an overlapping region of the spatial neighborhood (100) and the first neighboring block by inter prediction, in the first neighboring block: forming said proximity signal (100'); a video encoder (14) configured to:

200. a second neighboring block overlapping the spatial neighborhood (100) and being one of the inter-intra predicted blocks can be reconstructed by a sample-by-sample combination of an inter-intra predicted signal of the second neighboring block and a second prediction residual signal of the second neighboring block; The video encoder (14) of claim 199, wherein the video encoder (14) is configured to further form the neighborhood signal (100') within the second neighborhood block by using the inter-prediction component of the inter-intra prediction signal and ignoring the intra-prediction component of the inter-intra prediction signal within an overlapping region between the spatial neighborhood (100) and the second neighborhood block.

201. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction residual signal for a predetermined prediction block among the prediction blocks is performing the transform-based predictive residual coding to determine a residual value for the given predictive block; predicting the sign of the residual value based on a neighboring signal (100') within a spatial neighborhood (100) of the given prediction block; an inter-predicted neighboring block overlapping the spatial neighborhood (100) can be reconstructed by a sample-by-sample combination of an inter-predicted signal of the inter-predicted neighboring block and an inter-predicted residual signal of the inter-predicted neighboring block generated by performing the transform-based predictive residual coding to determine further residual values ​​and predicting signs of the determined further residual values ​​based on further neighboring signals in further spatial neighborhoods of the inter-predicted neighboring block; The video encoder (14) is configured to form the neighborhood signal (100') within an overlapping region of the inter-predicted neighboring block and the spatial neighborhood (100) by using the inter-prediction signal of the inter-predicted neighboring block and ignoring the inter-prediction residual signal.

202. a further inter-predicted neighboring block overlapping said spatial neighborhood (100) can be reconstructed by a sample-by-sample combination of a further inter-predicted signal of said further inter-predicted neighboring block and a further inter-predicted residual signal generated without predicting the sign of the residual values ​​of said further inter-predicted residual signal, 202. The video encoder of claim 201, further comprising: a video encoder for generating a video signal for a neighboring block; a video encoder for generating a video signal for a neighboring block; a video encoder for generating a video signal for a neighboring block;

203. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction residual signal for a predetermined prediction block among the prediction blocks is performing the transform-based predictive residual coding to determine a residual value for the given predictive block; and predicting the sign of the determined residual value based on a neighboring signal (100') within a spatial neighborhood (100) of the given prediction block, an intra-predicted neighboring block overlapping the spatial neighborhood (100) can be reconstructed by a sample-by-sample combination of an intra-prediction signal and an intra-prediction residual signal of the intra-predicted neighboring block; The video encoder (14) is configured to form the neighborhood signal (100') within the spatial neighborhood by generating a first substitution signal within an overlapping region of the spatial neighborhood (100) and the intra-predicted neighborhood block by inter prediction.

204. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction signal for a predetermined prediction block among the prediction blocks is performing template matching using a neighboring signal (100') within a spatial neighborhood (100) of the given predicted block as a template to find an error-minimizing template match; and using a template match block associated with the error-minimizing template match as the prediction signal for the given prediction block, an intra-predicted neighboring block overlapping the spatial neighborhood (100) can be reconstructed by a sample-by-sample combination of an intra-predicted signal of the intra-predicted neighboring block and an intra-prediction residual signal of the intra-predicted neighboring block; and / or a TM-predicted neighboring block overlapping the spatial neighborhood (100) can be reconstructed by a sample-by-sample combination of a TM-predicted residual signal of the TM-predicted neighboring block and a TM-predicted neighboring block obtained by performing template matching using a further neighboring signal in a further spatial neighborhood of the given prediction block as a template to find a further error-minimizing template match, and using the further template match block associated with the further error-minimizing template match as the TM-predicted neighboring block; within the intra-predicted neighboring block by generating a first substitution signal within an overlapping region of the spatial neighborhood (100) and the intra-predicted neighboring block by inter-prediction; and / or within the TM-predicted neighboring block by generating, by inter prediction, a second substitution signal within an overlapping region of the spatial neighborhood (100) and the TM-predicted neighboring block in a manner independent of the generation of the TM-predicted signal of the TM-predicted neighboring block; forming said proximity signal (100').

205. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction using motion compensated prediction controlled via motion vectors; a post-processing tool (110) for post-processing an inter-predicted signal of a given inter-predicted block (18); １ ) and From the predetermined inter-predicted block (18), a first inter-predicted block associated with one or more motion vectors (200) whose number satisfying a first predetermined criterion is zero; and / or a second inter-predicted block having one or more associated motion vectors (200), the number of which meeting a second predetermined criterion being full-pel motion vectors; and / or a third inter-predicted block having one of a set of predetermined inter-prediction modes associated therewith, the set of predetermined inter-prediction modes including one or more of a uni-prediction mode, a merge mode, and a bi-prediction mode using coding unit weights; and / or a fourth inter-predicted block whose block shape satisfies a third predetermined criterion; and / or a fifth inter-predicted block whose quantization parameter has a value that satisfies a fourth predetermined criterion; and identifying the predetermined inter-predicted block (18) from among the inter-predicted blocks by filtering out (124) the predetermined inter-predicted block (18).

206. The post-processing tool (110 １ ) uses a neural network or convolution to generate one or more matrices (142) derived from the inter-predicted signal with neighboring signals in a spatial neighborhood (100) of the given inter-predicted block (18), and one or more matrices (142) derived from corresponding portions in one or more reference pictures. １ ~142 ４ 206. The video encoder (14) of claim 205, configured to post-process the inter-predicted signal based on a 3D tensor (146) comprising:

207. The post-processing tool (110 １ ) selects said neural network from a set of two or more neural networks; or 207. The video encoder of claim 206 configured to select the convolution from a set of two or more convolutions.

208. The post-processing tool (110 １ 208. The video encoder of claim 207, configured to encode the neural network or the convolution selected by

209. The post-processing tool (110 １ ) the neural network or the convolution is the block shape of the predetermined inter-predicted block, and / or a prediction mode associated with the given inter-predicted block; and / or the temporal layer of the picture comprising the given inter-predicted block, and / or a quantization parameter value associated with the given inter-predicted block or the picture comprising the given inter-predicted block; and / or a prediction residual signal associated with the given inter-predicted block, and / or a picture order count difference between a reference picture and the picture comprising the given inter-predicted block; and / or A motion vector associated with the given inter-predicted block 208. A video encoder as claimed in claim 207, configured to select in dependence on:

210. 210. A video encoder according to any one of claims 207 to 209, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

211. 211. The video encoder of any one of claims 207 to 210, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, convolution type, and / or input tensor format.

212. The post-processing tool (110 １ )teeth, adapting scaling and offset values ​​based on neighboring signals within a spatial neighborhood (100) of the given inter-predicted block (18); using the scaling value and the offset value for post-processing the inter-predicted signal; 206. The video encoder (14) of claim 205, configured to post-process the inter-predicted signal by:

213. The spatial neighborhood (100) includes intra-predicted and inter-predicted portions, and the video encoder (14) divides the neighborhood signal into Replacing (122) the intra-predicted signal in the intra-predicted portion with a replacement signal generated by inter-prediction, and / or The post-processing tool (110 １ ) using a version of the inter-predicted signal in the inter-predicted portion that has not been post-processed by removing (124) signals associated with the intra-predicted portion; A video encoder (14) according to any one of claims 206 to 212, configured to form by

214. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and by using intra prediction for intra-predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; A post-processing tool (110) is applied to a first neighboring block that overlaps the spatial neighborhood (100) and is one of the inter-predicted blocks. １ ), wherein said post-processing tool (110) １ ) is configured to post-process the inter prediction signal of the first neighboring block to obtain a post-processed inter prediction signal; The first neighboring block is reconstructable by a sample-by-sample combination of the post-processed inter prediction signal of the first neighboring block and a first prediction residual signal of the first neighboring block; and The neighboring signals in the spatial neighborhood (100) are processed by the post-processing tool (110) in the first neighborhood block. １ and forming the inter prediction signal of the first neighboring block in a version that is not post-processed by the first neighboring block.

215. The post-processing tool (110 １ ) is generated by using a neural network or convolution to generate one or more matrices (142) derived from corresponding portions in one or more reference pictures. １ ~142 ４ ) and one or more matrices (142) derived from the inter-prediction signal. １ ~142 ４ 215. The video encoder (14) of claim 214, configured to post-process the inter prediction signal of the first neighboring block based on a 3D tensor (146) comprising:

216. The post-processing tool (110 １ ) selects said neural network from a set of two or more neural networks; or 216. The video encoder of claim 215 configured to select the convolution from a set of two or more convolutions.

217. The post-processing tool (110 １ 217. The video encoder of claim 216, configured to encode the neural network or the convolution selected by

218. The post-processing tool (110 １ ) the neural network or the convolution is the block shape of the first neighboring block, and / or a prediction mode associated with the first neighboring block, and / or a temporal layer of a picture comprising said first neighboring block, and / or a quantization parameter value associated with the first neighboring block or the picture containing the first neighboring block, and / or a prediction residual signal associated with said first neighboring block, and / or a picture order count difference between a reference picture and the picture comprising the first neighboring block, and / or a motion vector associated with the first neighboring block; 217. A video encoder as claimed in claim 216, configured to select in dependence on:

219. 219. A video encoder according to any one of claims 216 to 218, wherein the neural networks of the set of two or more neural networks differ from each other with respect to weights, biases, number of layers, layer types, and / or input tensor format.

220. 220. The video encoder of any one of claims 216 to 219, wherein the convolutions of the set of two or more convolutions differ from each other with respect to weights, biases, convolution type, and / or input tensor format.

221. The post-processing tool (110 １ ) converts the inter prediction signal of the first neighboring block into determining a scaling value and an offset value based on a spatial neighborhood (100) of the first neighboring block; using the scaling value and the offset value for post-processing the inter-predicted signal of the first neighboring block; 216. A video encoder (14) according to claim 215, configured for post-processing by:

222. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using intra prediction for intra-predicted blocks and by using inter-intra prediction for inter-intra predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; inter-intra predicting a first neighboring block that overlaps the spatial neighborhood (100) and is one of the inter-intra predicted blocks to obtain an inter-intra predicted signal of the first neighboring block, wherein the inter-intra predicted signal corresponds to a weighted combination of the intra predicted signal and the inter predicted signal of the first neighboring block; the first neighboring block is reconstructable by a sample-by-sample combination of the inter-intra prediction signal of the first neighboring block and a first prediction residual signal of the first neighboring block; The video encoder (14) is configured to form the neighborhood signal within the spatial neighborhood (100) by using the inter-predicted signal of the first neighboring block rather than the intra-predicted signal of the first neighboring block within the first neighboring block.

223. A video encoder (14), comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by use of intra prediction for intra-predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; applying a residual code prediction tool to a first neighboring block overlapping the spatial neighborhood (100) to obtain a predicted residual signal of the first neighboring block; the first neighboring block is reconstructable by a sample-by-sample combination of a prediction signal of the first neighboring block and the prediction residual signal of the first neighboring block; The video encoder (14) is configured to form the neighborhood signal within the spatial neighborhood (100) by using the prediction signal of the first neighboring block that is not combined with the prediction residual signal of the first neighboring block within the first neighboring block.

224. The residual code prediction tool determining a residual value for the first neighboring block; 224. The video encoder (14) of claim 223, configured to: predict a sign of the residual value based on a spatial neighborhood (100) of the first neighboring block.

225. The video encoder (14) of claim 223 or claim 224, wherein the video encoder (14) is configured to intra-predict the first neighboring block, which is one of the intra-predicted blocks, to obtain an intra-predicted signal as the prediction signal of the first neighboring block.

226. The video encoder (14) performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and by using the intra prediction for the intra-predicted blocks; 226. A video encoder (14) as described in claim 223 or claim 225, configured to: inter predict the first neighboring block, which is one of the inter predicted blocks, to obtain an inter predicted signal as the predicted signal of the first neighboring block.

227. The video encoder (14) performing the block-based prediction by using inter-intra prediction for inter-intra predicted blocks and by using the intra prediction for the intra predicted blocks; and inter-intra predicting the first neighboring block, the first neighboring block being one of the inter-intra predicted blocks, to obtain an inter-intra predicted signal as the predicted signal of the first neighboring block.

228. 1. A method for decoding video, comprising: applying said plurality of decoding tools (110) to a current picture (10) of a video (16) on a block-by-block basis as controlled by a data stream (12); deriving a reconstructed signal (58) of the currently decoded picture (10) by a sample-by-sample combination of the contributing signals generated by the plurality of decoding tools (110); The plurality of decryption tools (110) includes a first predetermined decryption tool (110) １ ), the method comprising: using the first predetermined decoding tool, based on a neighboring signal (100') in a spatial neighborhood (100) of the current block (18), One or more second predetermined decoding tools (110) in the current block (18) ２ ) contribution signal, or intermediate signals in the current block (18) corresponding to partial combinations from the sample-by-sample combinations; post-processing, or The first predetermined decoding tool (110) for the current block (18) １ ) generating a contribution signal of The method comprises: said one or more second predetermined decoding tools (110 ２ ) or said first predetermined decoding tool (110 １ ) using a version of the intermediate signals in said spatial neighborhood (100) that has not been post-processed by One or more third predetermined decoding tools (110) within said spatial vicinity (100). ３ ) by a replacement signal generated independently of spatial signal interdependencies (122); and / or From the spatial neighborhood (100), the sample-by-sample combination for the derivation of the reconstructed signal (58) is transmitted to the one or more third predetermined decoding tools (110). ３ excluding (124) samples containing said contribution signal of generating a

229. 1. A method for decoding video, comprising: applying a plurality of decoding tools (110) to a current picture (10) of video on a block-by-block basis under control of a data stream (12), the plurality of decoding tools (110) including a first set of prediction tools, the method including performing a block-by-block selection of exactly one prediction tool from the first set of prediction tools when applying the plurality of decoding tools (110) to the current picture (10) on a block-by-block basis; deriving a reconstructed signal (58) of the currently decoded picture (10) by a sample-by-sample combination of a prediction signal generated by the first set of prediction tools and a prediction residual signal derived from the data stream (12); The plurality of decryption tools (110) includes a first predetermined decryption tool (110) １ ), the method comprising: １ post-processing prediction signals of one or more inter-prediction tools of said first set of prediction tools based on said neighboring signals (100') within a spatial neighborhood (100) using The first predetermined decoding tool (110 １ and generating the neighborhood signal (100') within the spatial neighborhood (100) by using a version of the prediction signal of the one or more inter prediction tools that has not been post-processed by inter prediction and / or by replacing (122) the prediction signal of one or more intra prediction tools of the plurality of prediction tools with a replacement signal generated by inter prediction.

230. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; applying a post-processing tool to a given predicted block and neighboring blocks that are in a spatial vicinity of the given predicted block and overlap the spatial vicinity (100) of the given predicted block; post-processing a prediction signal of the predetermined prediction block based on neighboring signals in the spatial neighborhood (100) of the predetermined prediction block to obtain a post-processed prediction signal of the predetermined prediction block; post-processing a prediction signal of the neighboring block based on a further neighboring signal in a further spatial neighborhood (100) of the neighboring block to obtain a post-processed prediction signal of the neighboring block, the neighboring block overlapping with the spatial neighborhood (100); reconstructing said neighboring blocks by a sample-by-sample combination of said post-processed prediction signal and a prediction residual signal obtained from said data stream (12); forming the neighborhood signal (100') within the neighborhood block by using the predicted signal of a version of the neighborhood block that has not been post-processed by the post-processing tool; A method comprising:

231. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and intra-prediction for intra-predicted blocks; applying a post-processing tool to a given inter-predicted block to post-process an inter-predicted signal of the given inter-predicted block based on neighboring signals within a spatial neighborhood (100) of the given inter-predicted block to obtain a post-processed inter-predicted signal of the given inter-predicted block; reconstructing a neighboring block overlapping said spatial neighborhood (100) and being one of said intra-predicted blocks by sample-by-sample addition of an intra-predicted signal of said neighboring block and a prediction residual signal obtained from said data stream (12); and forming the neighborhood signal (100') within the neighborhood block by generating a replacement signal within an overlapping region of the spatial neighborhood (100) and the neighborhood block by inter-prediction.

232. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; - intra-prediction of the intra-prediction block to obtain an intra-prediction signal for the respective block; motion compensated prediction for inter-predicted blocks to obtain an inter-predicted signal for the respective blocks; performing said block-based prediction by use of applying a post-processing tool to post-process the inter-predicted signal of a current block (18), the current block (18), based on a neighboring signal (100') within a spatial neighborhood (100) of the current block (18); The proximity signal (100') excluding (124) from said spatial neighborhood (100) samples associated with nearby intra-predicted blocks that overlap said spatial neighborhood (100); and / or using inter-predicted signals of nearby inter-predicted blocks within the spatial neighborhood (100) that overlap a version of the spatial neighborhood (100) that has not been post-processed by the post-processing tool; and forming by

233. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks, and by using intra prediction for intra-predicted blocks, and by applying inter-intra prediction tools to inter-intra predicted blocks; applying the inter-intra prediction tool to a given inter-intra prediction block to generate an inter-intra prediction signal for the given inter-intra prediction block based on neighboring signals within a spatial neighborhood (100) of the given inter-intra prediction block; reconstructing a first neighboring block overlapping the spatial neighborhood (100) and being one of the intra-predicted blocks by a sample-by-sample combination of an intra-predicted signal of the first neighboring block and a first prediction residual signal obtained from the data stream (12); forming the neighborhood signal (100') within the first neighborhood block by generating a first substitution signal within an overlapping region of the spatial neighborhood (100) and the first neighborhood block by inter prediction; A method comprising:

234. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction residual signal for a predetermined prediction block among the prediction blocks is performing the transform-based predictive residual coding to derive residual values ​​for the given predictive block from the data stream (12); predicting the sign of the derived residual value based on a neighboring signal (100') within a spatial neighborhood (100) of the given prediction block; performing the transform-based predictive residual coding to derive further residual values ​​from the data stream, and predicting signs of the derived further residual values ​​based on further neighboring signals in further spatial neighborhoods of the inter-predicted neighboring blocks, thereby reconstructing inter-predicted neighboring blocks that overlap the spatial neighborhood (100) by a sample-by-sample combination of inter-predicted signals of the inter-predicted neighboring blocks and inter-prediction residual signals obtained from the data stream (12); forming the neighborhood signal (100') within an overlapping region of the inter-predicted neighboring block and the spatial neighborhood (100) by using the inter-predicted signal of the inter-predicted neighboring block and ignoring the inter-predicted residual signal.

235. A method for decoding video, decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction residual signal for a predetermined prediction block among the prediction blocks is performing the transform-based predictive residual coding to derive residual values ​​for the given predictive block from the data stream (12); predicting the sign of the derived residual value based on a neighboring signal (100') within a spatial neighborhood (100) of the given prediction block; reconstructing an intra-predicted neighboring block that overlaps with the spatial neighborhood (100) by a sample-by-sample combination of an intra-prediction signal of the intra-predicted neighboring block and an intra-prediction residual signal obtained from the data stream (12); and forming the neighboring signal (100') within the spatial neighborhood by generating a first replacement signal within an overlapping region of the spatial neighborhood (100) and the intra-predicted neighboring block by inter-prediction.

236. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction signal for a predetermined prediction block among the prediction blocks is performing template matching using a neighboring signal (100') within a spatial neighborhood (100) of the given predicted block as a template to find an error-minimizing template match; using a template match block associated with the error-minimizing template match as the prediction signal for the given prediction block; reconstructing an intra-predicted neighboring block overlapping with the spatial neighborhood (100) by a sample-by-sample combination of an intra-predicted signal of the intra-predicted neighboring block and the intra-prediction residual signal obtained from the data stream (12), and / or reconstructing the TM-predicted neighboring block overlapping with the spatial neighborhood (100) by a sample-by-sample combination of a TM-predicted residual signal obtained from the data stream (12) and a TM-predicted neighboring block obtained by performing template matching using a further neighboring signal in a further spatial neighborhood of the predetermined prediction block as a template for finding a further error-minimizing template match, and using a further template match block associated with the further error-minimizing template match as the TM-predicted neighboring block; within the intra-predicted neighboring block by generating a first substitution signal within an overlapping region of the spatial neighborhood (100) and the intra-predicted neighboring block by inter-prediction; and / or within the TM-predicted neighboring block by generating, by inter prediction, a second substitution signal within an overlapping region of the spatial neighborhood (100) and the TM-predicted neighboring block in a manner independent of the generation of the inter-predicted signal of the TM-predicted neighboring block; forming said proximity signal (100').

237. 1. A method for processing a picture, comprising: Polyphase splitting (140) of the luma samples of the picture portion (11) into polyphase components and forming a matrix (142) for each polyphase component. １ ~142 ４ ) and The matrix of polyphase components (142 １ ~142 ４ ) to form the tensor (146); The tensor (146) is converted into the matrix (142 １ ~142 ４ ) as different channels to obtain an output tensor (148) consisting of a concatenation of output matrices, one output matrix for each polyphase component; forming a processed picture portion (11') based on said output tensor (148) by inverse polyphase decomposition (150).

238. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing, for inter-predicted blocks, the block-based prediction using motion compensated prediction controlled via motion vectors derived from the data stream (12); a post-processing tool (110) for post-processing an inter-predicted signal of a given inter-predicted block (18); １ ) and From the predetermined inter-predicted block (18), a first inter-predicted block having one or more associated motion vectors (200) whose number satisfying a first predetermined criterion is zero according to said data stream (12); and / or a second inter-predicted block having one or more motion vectors (200) associated therewith, according to said data stream (12), a number of which satisfy a second predetermined criterion being full-pel motion vectors; and / or a third inter-predicted block having one of a set of predetermined inter-prediction modes associated therewith according to the data stream (12), the set of predetermined inter-prediction modes including one or more of a uni-prediction mode, a merge mode, and a bi-prediction mode using coding unit weights; and / or a fourth inter-predicted block whose block shape satisfies a third predetermined criterion; and / or a fifth inter-prediction block in which the data stream (12) signals a quantization parameter having a value that satisfies a fourth predetermined criterion; and identifying the predetermined inter-predicted block (18) from among the inter-predicted blocks by excluding (124)

239. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and by using intra prediction for intra-predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; A post-processing tool (110) is applied to a first neighboring block that overlaps the spatial neighborhood (100) and is one of the inter-predicted blocks. １ ) to post-process the inter prediction signal of the first neighboring block to obtain a post-processed inter prediction signal; reconstructing the first neighboring block by a sample-by-sample combination of the post-processed inter prediction signal of the first neighboring block and a first prediction residual signal obtained from the data stream (12); The neighboring signals in the spatial neighborhood (100) are processed by the post-processing tool (110) in the first neighborhood block. １ and forming the inter prediction signal of the first neighboring block in a version that is not post-processed by

240. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using intra prediction for intra-predicted blocks and by using inter-intra prediction for inter-intra predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; inter-intra predicting a first neighboring block that overlaps the spatial neighborhood (100) and is one of the inter-intra predicted blocks to obtain an inter-intra predicted signal of the first neighboring block, wherein the inter-intra predicted signal corresponds to a weighted combination of the intra predicted signal and the inter predicted signal of the first neighboring block; reconstructing the first neighboring block by a sample-by-sample combination of the inter-intra prediction signal of the first neighboring block and a first prediction residual signal obtained from the data stream (12); and forming the neighborhood signal within the spatial neighborhood (100) by using the inter-predicted signal of the first neighboring block rather than the intra-predicted signal of the first neighboring block within the first neighboring block.

241. 1. A method for decoding video, comprising: decoding video from the data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by use of intra prediction for intra-predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; applying a residual code prediction tool to a first neighboring block overlapping said spatial neighborhood (100) to obtain a predicted residual signal of said first neighboring block; reconstructing the first neighboring block by a sample-by-sample combination of the prediction signal of the first neighboring block and the prediction residual signal of the first neighboring block; forming the neighborhood signal within the spatial neighborhood (100) by using the prediction signal of the first neighboring block that is not combined with the prediction residual signal of the first neighboring block within the first neighboring block.

242. 1. A method for encoding video, comprising: applying the plurality of encoding tools (110) to a current picture (10) of a video (16) on a block-by-block basis; a reconstructed signal (58) of the currently coded picture (10) is derivable by a sample-by-sample combination of contributing signals generated by the plurality of coding tools (110); The plurality of encoding tools (110) includes a first predetermined encoding tool (110) １ ), the method comprising: determining the first predetermined coding tool (110) based on a neighborhood signal (100') within a spatial neighborhood (100) of a current block (18). １ ) to One or more second predetermined encoding tools (110) in the current block (18). ２ ) contribution signal, or intermediate signals in the current block (18) corresponding to partial combinations from the sample-by-sample combinations; Post-processing the or The first predetermined encoding tool (110) for the current block (18) １ ) contribution signal, The neighboring signals (100') within the spatial neighborhood (100) are said one or more second predetermined coding tools (110 ２ ) or said first predetermined coding tool (110 １ ) using intermediate signals within said spatial neighborhood (100) in a version that has not been post-processed by One or more third predetermined coding tools (110) within said spatial neighborhood (100). ３ ) by a replacement signal generated independently of spatial signal interdependencies (122); and / or From the spatial neighborhood (100), the sample-by-sample combination for the derivation of the reconstructed signal (58) is performed by the one or more third predetermined coding tools (110). ３ excluding (124) samples containing said contribution signal of and generating the same by

243. 1. A method for encoding video, comprising: applying a plurality of encoding tools (110) to a current picture (10) of a video on a block-by-block basis, the plurality of encoding tools (110) comprising a first set of prediction tools, the method including performing a block-by-block selection of exactly one prediction tool from the first set of prediction tools when applying the plurality of encoding tools (110) to the current picture (10); a reconstructed signal (58) of the currently coded picture (10) is derivable by a sample-by-sample combination of a prediction signal and a prediction residual signal generated by the first set of prediction tools; The plurality of encoding tools (110) includes a first predetermined encoding tool (110) １ ), the method comprising: １ post-processing prediction signals of one or more inter-prediction tools of said first set of prediction tools based on said neighboring signals (100') within a spatial neighborhood (100) using The first predetermined encoding tool (110 １ and generating the neighborhood signal (100') within the spatial neighborhood (100) by using a version of the prediction signal of the one or more inter prediction tools that has not been post-processed by intra prediction and / or by replacing (122) the prediction signal of one or more intra prediction tools of the plurality of prediction tools with a replacement signal generated by inter prediction.

244. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; applying a post-processing tool to a given predicted block and neighboring blocks that are in a spatial vicinity of the given predicted block and overlap the spatial vicinity (100) of the given predicted block; post-processing a prediction signal of the predetermined prediction block based on neighboring signals in the spatial neighborhood (100) of the predetermined prediction block to obtain a post-processed prediction signal of the predetermined prediction block; post-processing a prediction signal of the neighboring block based on a further neighboring signal in a further spatial neighborhood (100) of the neighboring block to obtain a post-processed prediction signal of the neighboring block, the neighboring block overlapping with the spatial neighborhood (100); the neighboring blocks are reconstructable by a sample-by-sample combination of the post-processed prediction signal and a prediction residual signal; forming the neighborhood signal (100') within the neighborhood block by using the predicted signal of a version of the neighborhood block that has not been post-processed by the post-processing tool.

245. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and intra-prediction for intra-predicted blocks; applying a post-processing tool to a given inter-predicted block to post-process an inter-predicted signal of the given inter-predicted block based on neighboring signals within a spatial neighborhood (100) of the given inter-predicted block to obtain a post-processed inter-predicted signal of the given inter-predicted block; a neighboring block overlapping the spatial neighborhood (100) and being one of the intra-predicted blocks can be reconstructed by a sample-by-sample addition of an intra-predicted signal of the neighboring block and a prediction residual signal of the neighboring block; and forming the neighborhood signal (100') within the neighborhood block by generating a replacement signal within an overlapping region of the spatial neighborhood (100) and the neighborhood block by inter-prediction.

246. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; - intra-prediction of the intra-prediction block to obtain an intra-prediction signal for the respective block; motion compensated prediction for inter-predicted blocks to obtain an inter-predicted signal for the respective blocks; performing said block-based prediction by use of applying a post-processing tool to post-process the inter-predicted signal of a current block (18), the current block (18), based on a neighboring signal (100') within a spatial neighborhood (100) of the current block (18); The proximity signal (100') excluding (124) from said spatial neighborhood (100) samples associated with nearby intra-predicted blocks that overlap said spatial neighborhood (100); and / or using inter-predicted signals of nearby inter-predicted blocks within the spatial neighborhood (100) that overlap a version of the spatial neighborhood (100) that has not been post-processed by the post-processing tool; and forming by

247. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks, and by using intra prediction for intra-predicted blocks, and by applying inter-intra prediction tools to inter-intra predicted blocks; applying the inter-intra prediction tool to a given inter-intra prediction block to generate an inter-intra prediction signal for the given inter-intra prediction block based on neighboring signals within a spatial neighborhood (100) of the given inter-intra prediction block; a first neighboring block overlapping the spatial neighborhood (100) and being one of the intra-predicted blocks can be reconstructed by a sample-by-sample combination of an intra-predicted signal of the first neighboring block and a first prediction residual signal of the first neighboring block; generating a first substitution signal in an overlapping region of the spatial neighborhood (100) and the first neighboring block by inter prediction, in the first neighboring block: forming said proximity signal (100').

248. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction residual signal for a predetermined prediction block among the prediction blocks is performing the transform-based predictive residual coding to determine a residual value for the given predictive block; predicting the sign of the residual value based on a neighboring signal (100') within a spatial neighborhood (100) of the given prediction block, an inter-predicted neighboring block overlapping the spatial neighborhood (100) can be reconstructed by a sample-by-sample combination of an inter-predicted signal of the inter-predicted neighboring block and an inter-predicted residual signal of the inter-predicted neighboring block generated by performing the transform-based predictive residual coding to determine further residual values ​​and predicting signs of the determined further residual values ​​based on further neighboring signals in further spatial neighborhoods of the inter-predicted neighboring block; forming the neighborhood signal (100') within an overlapping region of the inter-predicted neighboring block and the spatial neighborhood (100) by using the inter-predicted signal of the inter-predicted neighboring block and ignoring the inter-predicted residual signal.

249. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction residual signal for a predetermined prediction block among the prediction blocks is performing the transform-based predictive residual coding to determine a residual value for the given predictive block; predicting the sign of the determined residual value based on a neighboring signal (100') within a spatial neighborhood (100) of the given prediction block, an intra-predicted neighboring block overlapping the spatial neighborhood (100) can be reconstructed by a sample-by-sample combination of an intra-prediction signal and an intra-prediction residual signal of the intra-predicted neighboring block; and forming the neighborhood signal (100') within the spatial neighborhood by generating a first substitution signal within an overlapping region of the spatial neighborhood (100) and the intra-predicted neighborhood block by inter-prediction.

250. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction on a prediction block; A prediction signal for a predetermined prediction block among the prediction blocks is performing template matching using a neighboring signal (100') within a spatial neighborhood (100) of the given predicted block as a template to find an error-minimizing template match; and using a template match block associated with the error-minimizing template match as the prediction signal for the given prediction block, an intra-predicted neighboring block overlapping the spatial neighborhood (100) can be reconstructed by a sample-by-sample combination of an intra-predicted signal of the intra-predicted neighboring block and an intra-prediction residual signal of the intra-predicted neighboring block; and / or a TM-predicted neighboring block overlapping the spatial neighborhood (100) can be reconstructed by a sample-by-sample combination of a TM-predicted residual signal of the TM-predicted neighboring block and a TM-predicted neighboring block obtained by performing template matching using a further neighboring signal in a further spatial neighborhood of the given prediction block as a template to find a further error-minimizing template match, and using the further template match block associated with the further error-minimizing template match as the TM-predicted neighboring block; within the intra-predicted neighboring block by generating a first substitution signal within an overlapping region of the spatial neighborhood (100) and the intra-predicted neighboring block by inter-prediction; and / or within the TM-predicted neighboring block by generating, by inter prediction, a second substitution signal within an overlapping region of the spatial neighborhood (100) and the TM-predicted neighboring block in a manner independent of the generation of the TM-predicted signal of the TM-predicted neighboring block; forming said proximity signal (100').

251. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction using motion compensated prediction controlled via a motion vector; a post-processing tool (110) for post-processing an inter-predicted signal of a given inter-predicted block (18); １ ) and From the predetermined inter-predicted block (18), a first inter-predicted block associated with one or more motion vectors (200) whose number satisfying a first predetermined criterion is zero; and / or a second inter-predicted block having one or more associated motion vectors (200), the number of which meeting a second predetermined criterion being full-pel motion vectors; and / or a third inter-predicted block having one of a set of predetermined inter-prediction modes associated therewith, the set of predetermined inter-prediction modes including one or more of a uni-prediction mode, a merge mode, and a bi-prediction mode using coding unit weights; and / or a fourth inter-predicted block whose block shape satisfies a third predetermined criterion; and / or a fifth inter-predicted block whose quantization parameter has a value that satisfies a fourth predetermined criterion; and identifying the predetermined inter-predicted block (18) from among the inter-predicted blocks by excluding (124)

252. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using motion compensated prediction for inter-predicted blocks and by using intra prediction for intra-predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; A post-processing tool (110) is applied to a first neighboring block that overlaps the spatial neighborhood (100) and is one of the inter-predicted blocks. １ ) to post-process the inter prediction signal of the first neighboring block to obtain a post-processed inter prediction signal; The first neighboring block is reconstructable by a sample-by-sample combination of the post-processed inter prediction signal of the first neighboring block and a first prediction residual signal of the first neighboring block; and Within the first neighborhood block, the post-processing tool (110 １ and forming the neighboring signal within the spatial neighborhood (100) by using a version of the inter prediction signal of the first neighboring block that has not been post-processed by

253. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by using intra prediction for intra-predicted blocks and by using inter-intra prediction for inter-intra predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; inter-intra predicting a first neighboring block that overlaps with the spatial neighborhood (100) and is one of the inter-intra predicted blocks to obtain an inter-intra predicted signal of the first neighboring block, wherein the inter-intra predicted signal corresponds to a weighted combination of the intra predicted signal and the inter predicted signal of the first neighboring block; The first neighboring block is reconstructable by a sample-by-sample combination of the inter-intra prediction signal of the first neighboring block and a first prediction residual signal of the first neighboring block; and and forming the neighborhood signal within the spatial neighborhood (100) by using the inter-predicted signal of the first neighboring block rather than the intra-predicted signal of the first neighboring block within the first neighboring block.

254. 1. A method for encoding video, comprising: encoding the video into a data stream (12) using block-based prediction and transform-based predictive residual coding; performing the block-based prediction by use of intra prediction for intra-predicted blocks; Intra-predicting an intra-predicted block using neighboring signals within a spatial neighborhood (100) of said intra-predicted block; applying a residual code prediction tool to a first neighboring block overlapping the spatial neighborhood (100) to obtain a predicted residual signal of the first neighboring block, The first neighboring block is reconstructable by a sample-by-sample combination of a prediction signal of the first neighboring block and the prediction residual signal of the first neighboring block; and forming the neighborhood signal within the spatial neighborhood (100) by using the prediction signal of the first neighboring block that is not combined with the prediction residual signal of the first neighboring block within the first neighboring block.

255. A data stream comprising video encoded by a video encoder according to any one of claims 129 to 227.

256. A computer program having a program code for performing the method according to any one of claims 228 to 254, when the computer program runs on a computer.

Images