Picture / video coding for supporting varying resolution and / or efficiently processing area-based packing

Abstract

To provide a method for video decoding that improves coding efficiency when using varying scene resolution and / or allows more efficient processing of region-based packing of one coded picture / a plurality of coded pictures.SOLUTION: A decoder 30 is a block-based video decoder for supporting an inter-prediction mode and an intra-prediction mode, performing residual transform coding for coding prediction residual, and performing in-loop filtering processing, pictures 34 of a video 32 varying in resolution. A post-processor is connected to an output portion of a decoder for resampling the pictures 34 of the video 32 so as to present a constant output sample rate for presentation of the video.SELECTED DRAWING: Figure 4a

Description

[Technical field]

[0001] The present application relates to video / picture coding with improved coding efficiency involving supporting varying resolutions and / or efficiently handling area-based packing. [Background technology]

[0002] The single-layer based version of HEVC does not allow picture resolution changes within a coded video sequence. It is only allowed at random access points (RAPs) (e.g., at IDR RAPs that completely reset the decoded picture buffer), where the resolution change may occur with the start of a new coded video sequence, which involves flushing the decoded picture buffer (DPB). However, flushing the DPB significantly reduces the achievable coding efficiency due to the fact that the exploitation of previously coded references is discontinued.

[0003] Therefore, there is a need for an improved video / picture codec that can more efficiently exploit resolution variations to improve coding efficiency.

[0004] There are also tools that allow conveying a scene, such as a panoramic scene, via coded pictures or coded videos, called area-based packing, where the picture / video codec is oblivious to how this scene is mapped area-wise onto the picture / pictures. The mapping is merely a pre-processing at the encoder's side and a post-processing at the decoder's side, but the codec operates on the packed picture / pictures without awareness of area-based packing. MPEG OMAF provides a framework for such conveyance of a scene, for example, via a packed picture / pictures. For example, to control the post-processing at the decoder's side to remap the coded picture / pictures onto scene regions, a scene partition / mapping is signaled as side information via an SEI message, which specifies the partition of the coded picture / pictures into picture regions, each of which is mapped onto a respective scene region of the scene. Although such a solution is efficient in terms of the reuse of existing coding tools, i.e., existing picture / video codecs, it would be advantageous to have an immediate concept that allows region-based packing to be handled more efficiently. Summary of the Invention [Problem to be solved by the invention]

[0005] It is therefore an object of the present application to provide an encoding / decoding tool that provides improved coding efficiency when using varying scene resolutions and / or allows for more efficient handling of region-based packing of a coded picture / pictures.

[0006] This object is achieved by the subject matter of the independent claims enclosed herewith. [Means for solving the problem]

[0007] According to a first aspect of the present application, a more efficient video codec is achieved by supporting the variation of the target spatial resolution at which the video is coded in the data stream on a time scale finer than the random access distance at which the video encoding is performed. The temporal variation of the target spatial resolution can be controlled as part of the rate control of the video encoder. When the variation of the target spatial resolution is performed on a time scale finer than the random access distance, the coding efficiency can be improved by allowing the encoder to adapt the target spatial resolution to the current video coding complexity. Different possibilities exist for varying the target spatial resolution, i.e., the number of samples per mutually corresponding picture content in the pictures of the video. For example, the video encoder and the video decoder may use a prediction loop in which the DPB is serially connected, and the already processed pictures are buffered in the DPB with varying target spatial resolution. In that case, if there is a target spatial resolution difference between the referenced picture and the reference picture that refers to the referenced picture, resampling of the pictures in the DPB may be performed on the fly, and the resampling provides a motion compensated predictor. Alternatively, the prediction loop may be fed by pictures at a reference spatial resolution, i.e. at some fixed spatial resolution that may be larger than any other possible value that the target spatial resolution may take. In such a case, resampling may be performed to fill the gap between the target spatial resolution and the reference spatial resolution. For example, to perform prediction correction of predictors obtained from pictures in the DPB at the reference spatial resolution rather than the target spatial resolution, prediction residual coding is performed at the target spatial resolution while performing resampling of the prediction residual towards the reference spatial resolution. Alternatively, to perform prediction correction at the target spatial resolution, resampling of predictors obtained from pictures in the DPB towards the target spatial resolution may be performed. Switching between these options may be supported by the encoder and the decoder, respectively.The variation of the target spatial resolution may be performed at the picture level, i.e., for each picture, the target spatial resolution may be set by the encoder and signaled in the data stream to the decoder. According to one embodiment, the signaling overhead associated with successive changes in the target spatial resolution and the associated picture size is mitigated by partitioning the pictures of the video according to the change in the target spatial resolution in a default manner, such as, for example, such that the partitions are proportionally constant, i.e., the pictures are mutually proportional to each other despite their varying size, i.e., the partitions match when scaled to equal spatial resolutions. According to yet a further embodiment, the later relative scaling of the partitions may be presented in the data stream as one option, while another option involves signaling the partitions at a predefined time scale that is finer than or equal to the time scale at which the target spatial resolution varies. The division just described may be used to encode videos such as videos resulting in blocks for which a selection between inter-prediction and intra-prediction modes is performed on the unit, or videos resulting in blocks for which zero prediction parameters for inter-prediction blocks and for intra-prediction blocks are performed on the unit, or videos resulting in blocks for which prediction residuals are subject to transformation or inverse transformation on the unit in the encoder and decoder, respectively, or videos resulting in blocks for which pictures are subdivided on the unit into tiles that are coded / decoded independently of each other, i.e., without intra-picture coding interdependence.

[0008] According to a further aspect of the present application, a video codec involves coding / decoding pictures of a video partitioned according to a scene partition into picture regions, each having a respective scene region of a scene mapped onto it, depending on this scene partition. The video encoder supports the adjustment of the scene partition, e.g., a kind of user setting, and signals this scene partition in a data stream from which the video decoder derives it. By rendering the coding / decoding of the video in a data stream dependent on the scene partition, the scene partition is not only a pre-processing / post-processing for the video codec, but also allows to achieve an improvement in coding efficiency by taking into account the scene partitions that define the partitions into picture regions, as well as their mapping onto the scene regions. For example, instead of signaling the scene partitions as side information in addition to coding of packed pictures, i.e. pictures partitioned into picture regions, the scene partition signaling can be used to at least partially save the signaling overhead that would be redundant for the scene partitions. For example, coding a packed picture may involve coding partitioning into blocks in which the packed picture is coded, such as coding units in which inter or intra prediction decisions are made, prediction units in which corresponding prediction parameters for the above-mentioned blocks are made, and / or transform units in which prediction residuals are transform coded, and / or tiles in which the packed picture is coded in a mutually independent manner. Partitioning decisions related to multi-tree subdivisions involved in the coding partitioning may be inferred, for example, based on scene partitioning, thereby saving signaling overhead. Alternatively, the video encoder may simply follow the restrictions in the coding partitioning with respect to the scene partitioning. Still further, some coding / decoding tools of the video encoder and video decoder may be operated depending on this scene partitioning. For example, derivation of a predictor may be performed depending on the scene partitioning.Scene partitioning may result in picture regions within one picture with different scene sampling resolutions, i.e., different numbers of samples per unit area or unit viewing angle of the scene, or, speaking differently, different ratios between picture region sizes, e.g., measured in units of numbers of samples, compared to corresponding scene region sizes, e.g., measured as fractions of the entire scene. Such differences in scene sampling resolutions may be taken into account by video encoders and video decoders when performing motion compensation prediction. For example, a predictor may be determined depending on the scene sampling resolution difference between the picture region in which the inter-prediction block is located and the picture region of the reference picture pointed to by the motion vector of the respective inter-prediction block. Still further, motion vector prediction may be performed in a manner that takes into account the scene sampling resolution difference. For example, instead of immediately adopting such motion vector of the neighboring block, a motion vector candidate may be derived based on the motion vector of the neighboring block through modification, and this motion vector may be duplicated in such a way that a version of this motion vector with its initial point set to the reference point of the neighboring block and another version of this motion vector with its initial point set to the reference point of the inter-prediction block have end points, and the images of such end points in the scene have the same distance from each other through the scene partition as the images of the neighboring blocks in the scene and the reference points of the inter-prediction block. Thus, the motion vector prediction is made via a detour across the scene region, thereby responding to the scene partition.

[0009] Furthermore, the coding / decoding of the video may be responsive to the scene partitions in terms of setting one or more parameters related to the coding partitions. For example, such parameters may be set and signaled in the data stream for picture regions separately. Alternatively or additionally, filtering of a given picture of the video may be performed using a filter, at least one parameter of which is varied across the given picture depending on the scene partition. For example, at least one parameter may be set at the interface between adjacent picture regions depending on the difference in scene sampling resolution between adjacent / contacting picture regions. In this manner, the filter characteristics may operate more precisely by taking the scene sampling resolution into account. Still further, the video encoder and the video decoder may locate the interface between picture regions that do not border each other in the scene, i.e., the scene discontinuity interface, based on the scene partition. The encoding and decoding may be responsive to these scene discontinuity interfaces. That is, for example, the encoder may treat these scene discontinuity interfaces like picture boundaries, for example. Decoders and encoders may perform filtering differently at such scene discontinuity boundaries compared to boundaries between picture regions that border each other in a scene. As a further alternative, picture region-based signaling of at least one parameter may be supported. Alternatively or additionally, the quantization parameter controlling the spectral shape of the spatial frequency spectrum of the quantization noise may be allowed to be set specifically on a picture region basis, i.e. in units of groups of one or more picture regions. The variations may be signaled in the data stream.

[0010] The latter aspect, i.e., the variation of at least one quantization parameter that controls the spectral shape of the spatial frequency spectrum of the quantization noise of the quantization, may be used independently of whether or not scene partitioning is used, according to further aspects of the present application. By adapting the quantization parameter to local characteristics of the picture, especially when having locally or spatially varying resolution, coding efficiency may be improved.

[0011] According to yet a further aspect of the present application, improved coding efficiency is achieved by having a video codec that allows a target spatial resolution at which video is coded in a data stream to be varied in units of picture regions of pictures, smaller than pictures, while signaling the variation in the data stream. This aspect may be combined with the first and third aspects described above, i.e., quantization parameter variation. The prediction loop may be fed with pictures at, for example, a reference spatial resolution. Again, the variation of the target spatial resolution within a picture may be a task performed by the rate control of the encoder.

[0012] Advantageous aspects of the present application are the subject matter of the dependent claims, preferred embodiments of which are described below with reference to the figures. [Brief description of the drawings]

[0013] [Figure 1a] 1 is a schematic block diagram of a video encoder according to one embodiment that uses video encoding that depends on temporal variation of a target spatial resolution; [Figure 1b] FIG. 1B is a block diagram of a decoder compatible with the video encoder according to FIG. 1A, according to one embodiment. [Figure 2a] FIG. 2 is a schematic diagram illustrating the difference in target spatial resolution according to the embodiments of FIGS. 1a and 1b. [Figure 2b] FIG. 1c is a schematic diagram of the reconstructed video output by the video decoder of FIG. [Diagram 3]Schematic diagram of an encoded video for explaining the time scale at which the target spatial resolution varies as compared to the random access distance / pitch. [Figure 4a] Schematic diagram of a decoder operating according to different deformation modes that can be operated by the decoder of FIG. 1b, where different deformation modes with different resolutions at which pictures are buffered in the DPB are shown. [Figure 4b] Schematic diagram of a decoder operating according to different deformation modes that can be operated by the decoder of FIG. 1b, where different deformation modes with different resolutions at which pictures are buffered in the DPB are shown. [Figure 5a] Block diagram of an encoder that forms a possible implementation form of the encoder of FIG. 1a and operates according to different deformation modes in which pictures are buffered in the decoded picture buffer at the reference spatial resolution, thereby conforming to the decoder of FIG. 4b, but the side or region where prediction correction is performed is different, i.e., at a varying target spatial resolution or reference spatial resolution. [Figure 5b] Block diagram of an encoder that forms a possible implementation form of the encoder of FIG. 1a and operates according to different deformation modes in which pictures are buffered in the decoded picture buffer at the reference spatial resolution, thereby conforming to the decoder of FIG. 4b, but the side or region where prediction correction is performed is different, i.e., at a varying target spatial resolution or reference spatial resolution. [Figure 6] Diagram showing a possible syntax for signaling a varying target spatial resolution. [Figure 7] Diagram showing an example for an SPS syntax including a flag for switching on and off the possibility of varying the target spatial resolution. [Figure 8] Diagram showing a syntax example for an SPS for transmitting different target spatial resolution settings on a larger scale, such as for a sequence of pictures, that allows pictures to reference any of these settings in the data stream by indexing. [Figure 9]9 is a diagram illustrating an example for slice segment header syntax, which references a predefined target spatial resolution setting, such as that signaled in FIG. 8, by an index syntax element. [Figure 10] 1 illustrates an example syntax for signaling an output aspect ratio for a resolution, such as a reference spatial resolution or one of a target spatial resolution. [Figure 11] A diagram showing an example for syntax for transmitting output aspect ratios for all possible target spatial resolution settings. [Figure 12] FIG. 13 illustrates an alternative syntax by which a number of different output aspect ratio settings may be signaled, the number being the same as or greater than the number of possible target spatial resolution settings. [Figure 13] FIG. 13 is a schematic diagram illustrating the possibility of signaling the relative tiling of pictures with their varying target spatial resolutions by one flag. [Figure 14a] FIG. 1 is a schematic diagram showing an example for scene partitioning or region-based packing. [Figure 14b] FIG. 1 is a schematic diagram showing an example for scene partitioning or region-based packing. [Figure 15a] 1A-1C are schematic block diagrams illustrating a video encoder according to an embodiment, the video encoder operating in response to scene partitioning or region-based packing, respectively; [Figure 15b] FIG. 15b is a schematic block diagram of a video decoder compatible with the video encoder of FIG. 15a, which operates similarly in response to or performs video decoding that is dependent on scene partitioning. [Figure 16] FIG. 15C is a schematic diagram showing the coding partitions used for video encoding and video decoding in the video encoder and decoder of FIG. 15a and FIG. 15b, and corresponding coding partition information conveyed in the data stream, to explain the possibilities for video encoding and video decoding dependencies in scene partitions. [Figure 17]FIG. 1 is a schematic diagram showing an example of a coding partition using a sequence of regular tree block partitions into an array of tree blocks, followed by a hierarchical multi-tree subdivision of the tree root blocks, to illustrate the parameterization of such a coding partition and to illustrate the possibility of taking scene partitions into account within such a coding partition. [Figure 18] 18 is a diagram illustrating an example of a syntax for conveying the coding segment parameterization of FIG. 17 in a data stream, here illustratively an SPS. [Figure 19a] 1A-1C are schematic diagrams illustrating different possibilities of using scene partitions to essentially refine the coding partitions conveyed in a data stream; [Figure 19b] 1A-1C are schematic diagrams illustrating different possibilities of using scene partitions to essentially refine the coding partitions conveyed in a data stream; [Figure 19c] FIG. 13 shows an example syntax for CU split syntax for splitting a block partially extending from its own RWP region into an adjacent RWP region while avoiding signaling split indicators for the split decision that are redundant or that could otherwise be inferred by inspecting the location of the region to be split relative to the boundary between the current RWP region and the adjacent RWP region. [Figure 19c-1] FIG. 13 shows an example syntax for CU split syntax for splitting a block partially extending from its own RWP region into an adjacent RWP region while avoiding signaling split indicators for the split decision that are redundant or that could otherwise be inferred by inspecting the location of the region to be split relative to the boundary between the current RWP region and the adjacent RWP region. [Figure 20a] 1 is a schematic diagram illustrating that a current inter-predicted block references a portion of a reference picture in a picture region corresponding to a dual scene sample density / resolution. [Figure 20b] FIG. 20b is a schematic diagram showing the opposite situation compared to FIG. 20a. [Figure 21] 1 is a schematic diagram illustrating a current inter-predicted block and the derivation of a motion vector predictor for this block based on neighboring blocks present in adjacent picture regions of different scene sampling densities / resolutions. [Figure 22] FIG. 1 is a schematic diagram illustrating the scaling of motion vector predictors derived from adjacent blocks that are both in the same picture region, but where the motion vectors of the adjacent blocks point to picture regions of different scene sampling resolutions / densities. [Diagram 23] FIG. 2 is a schematic diagram illustrating application of a filter across picture regions of different scene sampling resolutions / densities. [Figure 24a] 11A-11C show example syntax diagrams for syntax for signaling different quantization parameters that control noise colorness, for varying this quantization parameter within a picture and within a picture region, respectively. [Figure 24b] FIG. 24b is a schematic diagram of a transform block, such as a prediction residual transform block, to illustrate how the quantization parameters of FIG. 24a can affect noise colorness. [Figure 25a] FIG. 2 is a schematic block diagram of an encoder according to an embodiment, i.e., a possible video encoder, in which the encoder locally varies a quantization parameter that affects noise colorness in a coded picture. [Figure 25b] FIG. 25b is a schematic block diagram of a decoder compatible with the encoder of FIG. 25a; [Figure 26] FIG. 2 is a schematic diagram showing the variation of target spatial resolution within a coded picture and its sub-regions; [Figure 27] A diagram showing an example syntax for signaling varying target spatial resolution. [Figure 28]FIG. 2B is a schematic diagram illustrating the variation of target spatial resolution when the video encoder and decoder of FIG. 1A and FIG. 1B are implemented or modified to operate with a spatially varying target spatial resolution, with or without a temporal target spatial resolution change as described with respect to FIG. 2A. [Figure 29] FIG. 27 shows an example syntax diagram to illustrate that spatial differences in target spatial resolution within several pictures according to the examples of FIGS. 26-28 can be achieved using syntax elements, or resampling indicators, or some other indicators that indicate several target spatial resolution values ​​for each block into which the respective pictures are partitioned. [Diagram 30] A diagram showing an example syntax that differs from that of Figure 29 in that the transmission of syntax elements is performed at the coding unit level, i.e., for the block for which the prediction mode decision is made, rather than for the transform block as the exemplary Figure 20 refers to. [Diagram 31] FIG. 31 is a syntax example for an SPS showing the possibility of restricting the transmission of the syntax elements shown in FIGS. 29 and 30 to blocks within some predefined block size range, while assuming that for blocks outside that range a reference spatial resolution must be used for encoding / decoding. [Diagram 32] FIG. 30 illustrates an example syntax that differs from that of FIG. 29 in that the transmission of syntax elements is restricted to a range of predefined block sizes. [Diagram 33] 13 is a diagram illustrating an example syntax for an SPS signaling a quantization step size offset to be used for resampled blocks of a picture, i.e., blocks that are not coded / decoded at the reference spatial resolution. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The description begins with embodiments relating to the first aspect of the present application. Later, some further aspects are described. It should be noted that the aspects of the present application may be combined as outlined in more detail below.

[0015] A first aspect of the present application relates to dynamic variation of the target spatial resolution on a time scale finer than the random access pitch or distance at which the video coding is performed. It should be noted that later on further aspects of the present application are described, which may be used together to form further embodiments beyond those described herein, and such combinations are also described below.

[0016] For example, an embodiment of the present application relating to the first aspect may enable a hybrid block-based motion-compensated video encoder to dynamically change the spatial resolution, e.g., measured as an amount of samples per depicted viewing angle of a picture, the change being related to, e.g., the input spatial resolution. The coding exploits this fact in terms of coding efficiency. Later, there are also embodiments and aspects described in which such a hybrid block-based motion-compensated video encoder is given the opportunity to dynamically change the spatial resolution also with respect to picture subsections.

[0017] So far, the signaling needs of such codec systems remain unsolved. The single layer based version of HEVC, for example, does not allow picture resolution to change within a coded video sequence. It is only allowed at random access points that do not have leading pictures, i.e., IDR RAPs that completely reset the decoded picture buffer, where the resolution change may be done with the start of a new coded video sequence, including flushing of the decoded picture buffer. However, there are scenarios where it is desirable to have the resolution change at a different position in the bitstream than the IDR / BLA based RAPs. First, the open GOP coding structure uses a RAP type with a greater coding efficiency, i.e., with a leading picture following the RAP in bitstream order, but using a reference picture preceding the RAP. The RAP is here, for example, a CRA picture. A resampling of the reference picture before the RAP would be necessary, which is not foreseen in the codecs known so far. Going forward, the RAP rate should be kept beyond some limits to avoid coding efficiency loss due to prediction reservoir loss associated with such RAPs. Therefore, it may be desirable to change spatial resolution within a coded video sequence without the occurrence of a RAP, such as within the highest temporal layer of a hierarchical GOP, or within any other subset of coded pictures.

[0018] The embodiments described herein allow for resolution changes in a coded video sequence on a time scale finer than the random access pitch / distance. For example, there are three imaginable variants for implementing such a mid-CV resolution change. Here, the reference spatial resolution may define a highest spatial resolution, e.g., a maximum of the target spatial resolutions at which the video is coded in the data stream. However, before describing that, the general structure of the encoder and decoder is described with respect to Fig. 1a and Fig. 1b. Fig. 1a shows a video encoder 10 that receives a source video 12 and encodes the source video 12 in a target spatial resolution in a data stream 14 in such a way that the target spatial resolution varies on a time scale finer than the random access distance at which the source video is encoded in the data stream 14. Fig. 2a shows, for example, a subsequence of pictures of the video 12, here exemplarily three pictures 161, 162, and 163. All of these pictures show the same scene 18 and coincide in the field of view of the scene 18, e.g., shown by a rectangle 20 that surrounds the scene 18. That is, the pictures 161-163 of the video 12 may coincide in the scene sections depicted in these pictures. However, as illustrated by the dots in the pictures 161-163 in FIG. 2a, which are intended to represent a kind of spatial sampling of each picture 161-163, the pictures of the video 12 are coded by the encoder 10 into the data stream 14 at varying target spatial resolutions, i.e., at varying numbers of samples per scene section or region 20, or, speaking differently, at different numbers of samples per unit area of ​​the scene 18, e.g., per unit viewing angle. The following description of various implementation variations will clarify as to what "variation of target spatial resolution" may refer to. In particular, the varied target spatial resolution may relate to the prediction residual at which the video 12 is coded into the data stream 14.That is, the encoder 10 may be a predictive coder, such as, for example, a hybrid video coder supporting inter-prediction and intra-prediction modes, and the target spatial resolution variation may refer to the resolution at which the prediction residuals are coded into the data stream 14. That is, the source video 12 has some spatial resolution, or more precisely, its pictures at the time they enter the encoder 10, but the situation during coding at which the encoder 10 performs the transition from the input resolution to the target spatial resolution may depend on the implementation chosen. In particular, as outlined in more detail below, resampling may be performed as an initial process, naturally ahead of all other coding tasks of the encoder 10, which therefore all operate at this target spatial resolution. These tasks may include, for example, prediction as well as residual coding. The decoded picture buffer of the encoder 10, exemplarily shown at 22 in Fig. 1a, will thus buffer the pictures 161-163 of the video 12 at the varying target spatial resolution. This variant is referred to as variant a below. Alternatively, the decoded picture buffer 22 may buffer pictures of the video 12 at a reference spatial resolution, i.e. at some fixed spatial resolution, such as the input resolution of the video 12, with transitions between the target spatial resolution at which the residual coding is performed and the reference spatial resolution made with respect to the prediction signal or the residual signal, respectively; these variants are referred to as b and c below.

[0019] Figure 1b shows a video decoder that fits the encoder of figure 1a. The video decoder of figure 1b is indicated generally using the reference number 30 and is configured to decode a video 32 from a data stream 14, i.e. for the reconstruction of the video 12. The video decoder 30 is configured to derive from the data stream 14 a variation of a target spatial resolution at which the video 32 is encoded in the data stream 14, the variation being defined on a finer time scale than the random access distance at which the video 32 is encoded in the data stream 14. See figure 2b, which shows a reconstructed or decoded video 32 and its pictures 341-343 in a manner corresponding to the manner shown in figure 2a, when the source video 12 entering the encoder 10 is internally resampled in the encoder 10 for encoding. The pictures 341-343 all show or represent the scene 18, and may for example all have this same scene section encoded therein. Depending on the implementation for which examples are described in more detail below, the sample resolution of pictures 341-343 output by decoder 30 may coincide with the target spatial resolution exemplarily shown in Fig. 2a, or may be constant in time, e.g. corresponding to the above-mentioned reference sample resolution at which these pictures are buffered in decoded picture buffer 36 of decoder 30 just as they were in DPB 22 of encoder 10. Examples of these will become apparent from the following description of several variants for implementing the encoder and decoder, respectively.

[0020] As explained so far, the variation of the target spatial resolution occurs on a time scale finer than the random access distance / pitch at which the video coding is performed. As outlined in more detail below and as shown in FIG. 2a, the time scale at which the target spatial resolution varies may for example be a per-picture variation. That is, the target spatial resolution may vary on a picture-by-picture basis. Thus, the encoder 10 may use the signaling of the variation of the target spatial resolution at the picture level to decide on a picture-by-picture basis how the target spatial resolution should be set. However, this is not the only possibility. FIG. 3 shows a time axis t and uses hatching to indicate pictures of the video 12 at random access points. They occur at some temporal distance Δt, which may be constant as shown in FIG. 3 or may vary in time, in which case the random access distance may indicate for example the average distance between the random access point pictures of the video 12. The time scale at which the target spatial resolution occurs indicates the minimum duration at which the target spatial resolution may change between them. Thus, according to the example of Figures 1a and 1b, the random access distance Δt, 38 is greater than the minimum temporal distance δt, 40 over which the target spatial resolution can change. Figure 3 shows an example where the target spatial resolution can change from one picture to another of the video 12.

[0021] As already indicated above, there are different possibilities for implementing the encoder and the decoder of Fig. 1a and Fig. 1b, respectively. These possibilities are the variants a, b and c shown below. Fig. 4a and Fig. 4b show variant a, and variants b and c, respectively. Fig. 4a and Fig. 4b focus on the decoder 30, but it is clear how the encoder 10 should be brought into compatibility with each concept / variation. According to all variants, the prediction residuals of each picture are coded in the data stream 14 with varying target spatial resolutions, as illustrated by rectangles of different sizes on the left side of Fig. 4a and Fig. 4b. According to variant a, each picture is buffered in reconstructed form at the target spatial resolution. This is also illustrated by showing the pictures of the video in the form of rectangles of different sizes in the DPB 36. The DPB 36 is connected in series in a prediction loop 42, which is also present in the encoder to keep it synchronized with the decoder 30. According to variant a, when this reference picture of target resolution A is used as a reference picture by another picture currently being decoded with target spatial resolution B≠A, resampling 44 of the reference picture in the DPB 36 is performed on the fly. As shown in Fig. 4a, the decoder 30 may be a block-based video decoder, for example, supporting inter-prediction and intra-prediction modes, transform coding to code the prediction residual, i.e. residual transform coding, and in-loop filtering. As already mentioned, the pictures of the video output, i.e. the video 32, vary in resolution. It may be that a post-processor, not shown in Fig. 4a and not shown in Fig. 1b, is connected to the output of the decoder 30 in order to resample the pictures 34 of the video 32 to present a constant output sample rate for the presentation of the video.

[0022] Fig. 4b shows variants b and c. According to both variants, the pictures of the video are buffered in the DPB 36 at a reference spatial resolution. In other words, each coded picture is reconstructed in the DPB 36, and then the whole picture is resampled in the DPB 36 at the highest resolution if the highest reference spatial resolution is chosen. According to variant b, if necessary, when a resolution larger than or equal to any of the varying target spatial resolution values ​​is used as the reference target spatial resolution, an on-the-fly resampling is performed when referenced by pictures of a different resolution, such as a lower resolution. The stream of output pictures, i.e. the video 32, is output at a reference target spatial resolution, such as the highest resolution. According to variant c, the pictures are also stored in the DPB 36 at the reference target spatial resolution. If the picture resolution of the currently decoded picture is not equal to the reference spatial resolution, the stream of output pictures, i.e. video 32, is output at the reference spatial resolution as was the case with variant b, and the residual signal or prediction residual is resampled during reconstruction by bilinear filtering or the like. Thus, while in variant a, the DPB 36 is filled with pictures of varying resolution, each reference picture being resampled to the resolution of the currently coded / decoded picture at their reconstruction, in variants b and c, the DPB 36 is filled only with pictures of the highest resolution, which is achieved in two different ways.

[0023] Figures 5a and 5b focus on the difference between variants b and c. Figures 5a and 5b focus on the encoder side. As already mentioned with respect to Figures 4a and 4b, the prediction loop 42 coexists in the decoder and the encoder, and therefore all tasks and steps described with respect to the decoder are also performed on the encoder side, with the difference being that the encoder also chooses or selects the prediction mode, i.e. parameter settings such as inter-prediction mode or intra-prediction mode, and coding parameters, e.g. prediction parameters for each mode, quantization of transform prediction residual, etc. The encoder may do this by performing some rate control, and in particular by performing some cost minimization, i.e., aiming to minimize the cost depending on the distortion and coding rate, for example.

[0024] According to Figures 5a and 5b, the encoder 10 is implemented as a block-based predictive video coder, i.e. a subtractor 48 subtracts a prediction signal 50 from a source signal 52 to derive an undistorted prediction residual signal 54, which then undergoes transformation, scaling and quantization in a module 56 to give a prediction residual signal 58, which is finally coded into the data stream 14, via several stream multiplexers or modules 60 that perform formatting, entropy coding and setting up a header for this data. To keep synchronicity with the decoder side, the encoder 10 reconstructs the prediction residual signal in the spatial domain by subjecting the prediction residual 58, as derivable from the data stream 14, to a scaling and inverse transform module or an inverse quantization and inverse transform module 62. One input of an adder 64 is connected to the output of this module 62, which adds this prediction residual signal to the prediction signal 50, and the prediction signal thus modified is subjected to filtering by a filter control analysis module 66 followed by an in-loop filter 68. The result is buffered in the DPB 22 to buffer the reconstructed picture and to form a reservoir for the reference pictures referenced by the inter-prediction blocks of the picture to be subsequently coded. A motion estimation module 70 has access to the source signal 52 to estimate the motion field of the source video, thus determining the motion vectors used to code the video. The pictures stored in the DPB 22 are made accessible for an inter-prediction module 72, which generates the predictor, i.e. 74, of the inter-prediction block of the picture currently being coded. In the case of intra-prediction blocks, there is an intra-prediction module 76, which has access to the modified prediction signal output by the adder 64. The prediction parameters, ie the intra prediction mode, are determined by an intra picture estimation module 78 based on the analysis of a modified prediction signal, ie a signal previously reconstructed in the neighborhood of the currently coded block.Prediction parameters of inter-predicted blocks, such as motion vectors, i.e. 80, and prediction parameters of intra-predicted blocks, i.e. intra-prediction modes, such as angular modes, such as DC mode or planar mode, or non-angular modes, i.e. 82, are forwarded to the multiplexer 60 for signaling these prediction parameters together with the chosen prediction mode, i.e. inter / intra prediction mode decision, in the data stream 14. A selector 84 constructs the prediction signal 50 by selecting an inter-prediction predictor 74 or an intra-prediction predictor 86 depending on the chosen prediction mode for the various blocks of the currently coded picture, thus forming the prediction signal 50. The encoder 10 additionally comprises a generic coder control 88 which takes care of tasks such as rate control and cost minimization.

[0025] Fig. 5a focuses on variant b. As shown in Fig. 5a, pictures 90 of the inbound video signal 12 are resampled in an initial resampling stage 92 to pictures 94 of varying target spatial resolution before these pictures 94 enter the encoder loop, i.e. they are subject to subtraction by the subtractor 48. It should be noted that this task is transparent to the decoder side, which corresponds in its internal structure to the encoder 10, with regard to modules 62 and 64, 66, 68, the DPB 22 for which the reference number 36 is used as far as the decoder side is concerned, and modules 72, 76 and 84 with subsequent modules forming or connected in series in the prediction loop 42. The target spatial resolution variation can be controlled by a generic coder control 88 as shown in Fig. 5a, depending on the rate control and / or cost minimization. Since the pictures 94 forming the source signal 52 for the encoder 10 of FIG. 5a are of varying spatial resolution, the encoder 10 of FIG. 5a comprises a resampler that performs resampling to or from the target spatial resolution to a reference spatial resolution. For example, a resampler 96 is connected in the forward prediction loop 42 of the DPB 22 to resample the pictures before buffering in the DPB 22 at the reference spatial resolution. That is, resampling is performed whenever the target spatial resolution of the currently coded picture is not equal to the reference spatial resolution. Furthermore, a resampler 98 resamples the reference pictures referenced by the currently coded picture and is therefore connected between the DPB 22 and the inter prediction module 72, respectively, together with the motion estimator 70, which also has access to the sampled results. That is, the resampler 98 resamples the buffered pictures in the DPB 22 from the reference spatial resolution to the target spatial resolution. Other blocks and modules, indicated using hatching, are also configured to handle the varying target spatial resolution of the inbound pictures 94.

[0026] Fig. 5b focuses on variant c, where the source signal 52 is formed directly by the video 12. According to the example of Fig. 5b, the input resolution, i.e. the source resolution, of the source video 12 is used as the reference spatial resolution, but this example may differ in that a resampler, such as resampler 92, performs an initial resampling from the source resolution to the reference spatial resolution. According to variant c, as shown in Fig. 5b, a resampler 100 is connected between the output of the subtractor 48 and the input of the transformation, scaling and quantization module 56 in order to resample the undistorted prediction residual signal 54 to a varying target spatial resolution, before the transformation and quantization in module 56, and a resampler 102 performs the reverse process, i.e. resampling from the target spatial resolution to the reference spatial resolution, by being connected, on the one hand, between the output of the inverse transformation module 62 and, on the other hand, the first input of the adder 64, i.e. the prediction correction performed by the adder 64. That is to say, according to variant c shown in Fig. 5b, a resampling step is performed on the prediction residual signal, thereby allowing reconstruction in the highest resolution.

[0027] The following description focuses on several signaling possibilities to address the needs that arise according to variants a to c.

[0028] As already explained above, the target spatial resolution may be allowed to vary on a time scale corresponding to the picture pitch or frame rate. Fig. 6 shows, for example, that a target spatial resolution signaling 104 may be included in the picture parameter set, i.e. PPS 106, of the data stream 14. This may be done according to variants a-c. The signaling 104 according to the example of Fig. 6 indicates the picture size horizontally 108 and vertically 110. The possibility of using a varying target spatial resolution may be a mode signaled to be switched on or off via a flag 112 included in the PPS 106, so that, if activated, the target spatial resolution of the corresponding picture is indicated by using syntax elements 108 and 110 indicating the corresponding picture size, measured in number of samples in x and y, or, if not activated, not.

[0029] A sequence range constraint flag can trigger whether dynamic resolution is allowed or not. Such a flag can be included in the SPS (Sequence Parameter Set) or in a parameter set of a larger scope, such as a video-wide one in the VUI. Figure 7 shows an example where such a flag, flag 116, is included in the SPS 114. The latter flag 116, SPS_dynamic_resolution_enabled_flag, is set to 0 to signal that the value of flag 112, PPS_dynamic_resolution_enabled_flag, must not be set to 1.

[0030] An alternative would be that the picture size set Pie Size Set would be defined in a parameter set with a larger scope than the PPS, such as the SPS or VPS, and that an index into that set would be included or accompanied in every video slice NAL unit in the slice header. An example is shown in Figures 8 and 9. Then, beyond flag 116, SPS 114 includes target spatial resolution signaling 118 if the subsequent flag is set, which signals the picture size or target spatial resolution via syntax elements 108 and 110, once for each possible value or possible state of the target spatial resolution, and slice header 120, an example of which is shown in Figure 9, references one of these possible values ​​or states via index 122, thereby indicating a reference to the respective target spatial resolution indicated via signaling 118 for each picture to which the slice of slice header 120 belongs. It may be required that all slices in one picture indicate or index to the same target spatial resolution.

[0031] According to an embodiment, it is possible to signal in the data stream 14 as to which possibilities are used to allow for variation of the target spatial resolution. For example, a flag output_scaling_flag or some other signaling can indicate whether the resampling is performed as a pre-processing / post-processing step or as a canonical in-loop resampling step. This would make it possible to distinguish variant a, i.e. resampling outside the decoding loop, from variants b and c, in which the resampling is performed in the decoding loop 42. In this case, at least one of several modes, i.e. variant a on the one hand and variants b and c on the other hand, would be implemented in the codec, and the encoder, the decoder, respectively. In this way, the decoder is enabled to configure further down elements of the multimedia processing chain up to an indication of whether a dynamic adjustment of the resolution of the decoded video signal should be performed before presentation.

[0032] Further signaling that may be included in or inserted into the data stream 14 may include aspect ratio signaling. In particular, in HEVC, for example, there are some bitstream signaling aspects that are sequence specific and are not allowed to change in the CVS. As in the embodiments described so far, the resolution, i.e. the target spatial resolution, is allowed to change in the coded video sequence CVS, and such signaling should be dynamic, i.e. these aspects should be dynamically described. Taking for example the signaling aspect of the sample aspect ratio, reducing the resolution of the pictures of the source signal 12 can be achieved in multiple ways, i.e., by resampling horizontally or vertically, or by resampling in both directions. All three of the just mentioned options require different resampling ratios for a given target resolution, which will lead to different sample aspect ratios. Changing the resolution will therefore change the coded sample aspect ratio. One option is to specify a default way that the aspect ratio is applied to one of the picture sizes, such as the first or largest picture size specified in the SPS across all possible picture sizes. Another possibility is to specify a reference picture size to which the signaled aspect ratio is applied and leave it up to the decoder 30 or some renderer connected to the output of the decoder to derive which is the sample aspect ratio of the other format. See, for example, FIG. 10. FIG. 10 shows that the data stream may include information or signaling indicating the output aspect ratio at 126. FIG. 10 shows that this information 126 may be included in the vui 124, but this is just one example.That is, the signaling 126 indicates an aspect ratio 128 in x or horizontal and an aspect ratio 130 in y or vertical for one possible setting of a target spatial resolution, such as a reference spatial resolution, and a decoder or some device receiving the video 32 uses this aspect ratio to present the video 32 for pictures at the respective spatial resolution. When varying the picture size of the video 32, such as in variant a, the aspect ratio signaling 126 is used for pictures at the corresponding spatial resolution, such as the reference spatial resolution, and for pictures at another resolution, the aspect ratio for presentation is determined based on the signaling 126 and the respective target spatial resolution or picture size of the respective picture. That is, for another picture size with pic_width_in_luma_samples[i] and pic_height_in_luma_samples[i], the aspect ratio for that picture is sar_width[i]=sar_width_default / pic_width_in_luma_samples_default* pic_width_in_luma_samples[i], and sar_height[i]=sar_height_default / pic_height_in_luma_samples_default*pic_height_in_luma_samples[i] It can be derived as:

[0033] Note that the above example calculation assumes that there are no cropping parameters. pic_width_in_luma_samples[i] and pic_height_in_luma_samples[i] correspond to the entire decoded picture, while the cropped picture is a portion of what is output. Two different resolutions may use completely different cropping windows, and thus the above calculation would require performing cropping before calculating the relationship between sar_width and sar_height for the different resolutions. That is, croppedWidth[i]=pic_width_in_luma_samples[i]-(conf_win_right_offset[i]+conf_win_left_offset[i]), croppedHeight[i]=pic_height_in_luma_samples[i]-(conf_win_top_offset[i]+conf_win_left_offset[i]), therefore, sar_width[i]=sar_width_default / croppedWidth_default*croppedWidth[i], and sar_height[i]=sar_height_default / croppedHeight_default* croppedHeight[i] It is.

[0034] However, typically these sample aspect ratio values ​​are used in explicit interfaces for post-processing, where values ​​directly from the VUI are used to configure the post-processing step. While the above calculations are simple and can be easily performed, a simpler approach that would allow existing interfaces and APIs to be used with minimal modifications would be to directly signal different aspect ratios for each resolution, as is done today.

[0035] For example, a first option would be to describe several aspect ratios, one for each signaled picture size, as shown in FIG.

[0036] Here, signaling 126 indicates the aspect ratio for outputting pictures of video 32 for each possible setting of the target spatial resolution, the number of which is indicated by num_pic_in_luma_samples_minus1. Figure 11 shows that an alternative way or option of indicating the aspect ratio could be to index by index 132 to some of the predefined aspect ratios.

[0037] The signaling example of Fig. 11 may be further improved and may be made more efficient for some cases. It should be understood that in the example of Fig. 11 there are as many sample aspect ratios described as there are picture sizes. Speaking alternatively, Fig. 11 shows a sample aspect ratio for each possible setting of the target spatial resolution. However, it may happen that the sample aspect ratios are equal for all picture sizes or all possible settings of the target spatial resolution, or that the number of envisaged sample aspect ratios is at least less than the number of picture sizes or target spatial resolutions. Thus, the signaling of the aspect ratios may be made as shown in Fig. 12. Here, the flag 134 may indicate whether all aspect ratios are equal, i.e., all pictures of the video 32 have the same aspect ratio. If yes, the signaling 126 simply needs to be signaled once for all pictures. If not, the number of different aspect ratios is signaled at 136 and then the signaling 126 is included in the data stream as often as indicated by 136. For each picture, then an indication is made as to which target spatial resolution is to be used, i.e. via the picture size indication 104 or 122, and indirectly, this selection of the target spatial resolution to be used also indicates the aspect ratio to be applied and the presentation of the video 32 or respective picture, i.e. by associating the correct aspect ratio to each target spatial resolution.

[0038] As previously described, the following parameters may be included in the data stream 14:

[0039] Similar to above, signaling for the following parameters may be present in the data stream: min_spatial_segmentation_idc ue(v)

[0040] min_spatial_segmentation_idc indicates the parallel processing capability. For example, it constrains the maximum number of luma samples contained in a slice (when tiles are not used and wavefront processing is not enabled - entropy_coding_sync_enabled_flag is equal to 0). Otherwise, when tiles are in use, it constrains the maximum number of luma samples contained in a tile. Another parallel processing mode is when entropy_coding_sync_enabled_flag is equal to 1 (wavefront processing), where CtbSize and picture size determine the number of "wavefront" rows, and the value of min_spatial_segmentation_idc sets a constraint on the relationship between CtbSize and picture size. For any of these parallel processing modes (slice, tile, wavefront), the value of min_spatial_segmentation_idc sets a constraint as explained above that depends on the picture size. For better understanding, the HEVC specification text follows.

[0041] - Otherwise (min_spatial_segmentation_idc is not equal to 0), it is a requirement of bitstream conformance that exactly one of the following conditions must be true:

[0042] - For each PPS activated in CVS, tiles_enabled_flag is equal to 0 and entropy_coding_sync_enabled_flag is equal to 0, and (4*PicSizelnSamplesY) / minSpatialSegmentationTimes More than 4 luma samples There is no slice in the CVS that contains it.

[0043] - For each PPS activated in CVS, tiles_enabled_flag is equal to 1 and entropy_coding_sync_enabled_flag is equal to 0, and (4*PicSizelnSamplesY) / minSpatialSegmentationTimes More than 4 luma samples There are no tiles in the CVS that contains it.

[0044] - within each PPS activated in the CVS, tiles_enabled_flag is equal to 0, and entropy_coding_sync_enabled_flag is equal to 1, and the syntax elements pic_width_in_luma_samples, pic_height_in_luma_samples, and the variable CtbSizeY obey the following constraint: (2*pic_height_in_luma_samples+pic_width_in_luma_samples)*CtbSizeY≦(4*PicSizelnSamplesY) / minSpatialSegmentationTimes4.

[0045] Considering that the picture size or the target spatial resolution can vary from picture to picture, min_spatial_segmentation_idc can be signaled in a more flexible way. Similar to the aspect ratio, the signaling of min_spatial_segmentation_idc can be For a particular picture size, the same constraints on luma samples per slice / tile apply for other picture sizes. Using min_spatial_segmentation_idc[i] for each picture size, This can be done.

[0046] Another example is max_bytes_per_pic_denom ue(v) is related to.

[0047] max_bytes_per_pic_denom indicates the number of bits that cannot be exceeded by the sum of the sizes of VCL NAL units of a CVS coded picture ((PicSizelnMinCbsY*RawMinCuBits) / (8*max_bytes_per_pic_denom)). It also depends on the picture size.

[0048] So, similar to above, it works like this for some picture sizes: For a particular picture size, the same constraints apply as for any picture size. Using min_spatial_segmentation_idc[i] for each picture size, It should be shown.

[0049] Another example is max_bits_per_min_cu_denom ue(v) is related to.

[0050] max_bits_per_min_cu_denom specifies an upper bound on the number of coding bits in a coding_unit() in CVS. It depends on the Ctb size and minCbLog2SizeY as follows: (128+RawMinCuBits) / max_bits_per_min_cu_denom*(1<<(2*(log2CbSize-MinCbLog2SizeY)))

[0051] In a multi-resolution coding scheme as considered here, it may be assumed that the Ctb size and the minimum coding block size (minCbLog2SizeY) depend on the picture size. Taking that into account, different values ​​for different picture sizes may be assumed.

[0052] Another example is log2_max_mv_length_horizontal ue(v) log2_max_mv_length_vertical ue(v) is related to.

[0053] Similarly, constraints on the two syntax elements for MV can be used for parallelization. They express constraints on MV that apply to coded pictures. With this information, it is possible for the decoder to know if enough data has been reconstructed from a (still being decoded) picture to start decoding another picture that references it. Since picture sizes can vary, for the previous case, the following options are used: For a specific picture size, the MV is scaled accordingly. Using log2_max_mv_length_horizontal[i] and log2_max_mv_length_vertical[i] for each picture size, A display for each picture size is required.

[0054] (For example, in some scenarios such as tile streaming where the same content is presented to the client at varying overall resolutions (to handle devices with, for example, FullHD or 4K video decoders), it may be desirable to maintain a fixed tile structure with respect to the picture plane regardless of the picture size or aspect ratio. One embodiment is that, as illustrated in the upper part of FIG. 13 below, a flag (relative_tiling_flag) indicating the tiled structure signaled at the highest reference spatial resolution (resolution A) (for example, the default structure signaled with a certain sequence range) is to be used in the bitstream for pictures having a lower resolution (resolution B) as well, and the tiling structure is scaled to the lower resolution compared to each picture size. In the lower part of FIG. 13, an example for non-relative tiling is shown, where, illustratively, a picture at the resampled resolution B (<A) fits entirely within a single tile of the original resolution. In this embodiment, the tiling structure is signaled only for one resolution, but in the sequence range, for example, in the SPS. Other syntaxes with picture ranges, such as PPS, omit each syntax.)

[0055] First, this can potentially save bits by avoiding redundant tiling signaling for each resolution. Second, the fact that the amount and relative dimensions of the tiles remain constant over the course of the coded video sequence allows for more efficient resource planning on the decoder side, where the decoder can avoid reevaluating the degree of parallelization to be used in the decoding process for each individual picture. In particular, the fact that the relative dimensions of the tiles remain constant across pictures of different resolutions can enable an estimate of the decoder execution time for individual tiles, which can help simplify decoder resource planning.)

[0056] In another embodiment, the signaling may consist of a sequence-level guarantee included in the bitstream indicating that tiles are aligned with respect to all pictures in the same spatial layer even if the spatial resolution changes within the layer. The decoder may use this guarantee to reserve parallel decoding resources accordingly and to determine whether the decoder is capable of decoding the bitstream through parallel processing. Multiple tile structures may be sent in the bitstream, but all of these structures are identical or are subsampled equivalents of the same tile structure.

[0057] A further embodiment consists of a flag in the bitstream that indicates the guarantee that the tile structures are aligned in the same spatial layer when the resolution changes within that layer, in that for every tile boundary present in the lower resolution, an aligned tile boundary exists in the higher resolution. Pictures coded at the higher resolution may also contain additional tile boundaries. Multiple tile structures may be sent in the bitstream, but all structures at the lower resolution are identical and an equivalent tile structure exists at the higher resolution.

[0058] If the decoder performs random access (RA) to the coded bitstream, i.e. starts decoding from any RAP in the middle of the stream, leading pictures belonging to open GOP RAPs such as CRA are typically discarded and decoding / presentation starts from the RAP picture. Depending on the system variant a, b or c, the knowledge at the decoder side about the resolution in the bitstream is different.

[0059] In variant a), the decoder may be informed through a picture-level range syntax in the RAP about the resolution of the current picture that exists in the DPB at a particular resolution after decoding. Each reference picture in the DPB at that particular resolution that is needed for decoding of the current picture at another resolution is resampled to the resolution of the current picture.

[0060] In variants b) and c), in the RAP, the sequence-level range syntax may inform the decoder about the highest resolution in the bitstream, but the picture-level range syntax informs the decoder about the resolution of the current picture to be decoded, to allow the decoder to resample all pictures to the highest resolution.

[0061] The following description concerns an attached aspect of the present application, namely, area-based packing, i.e., enabling video coding to perform scene-based subdivision of a picture into picture regions, each of which is associated with several scene sections or scene regions of a scene. According to this aspect, a video codec is responsive to area-based packing or scene packing / partitioning. Scene packing / partitioning may be performed in such a way that different scene regions are conveyed in the data stream with different scene sampling densities or different scene resolutions, thereby resulting in a sub-picture scene resolution change. There are various reasons why it may be beneficial to vary the spatial scene resolution within a picture of a video sequence during coding, i.e., to represent individual subsections of a scene with different spatial resolutions. Such a measure may be used either to improve the coding efficiency of the codec, i.e., as another measure to reduce the bitrate span for a given picture area, or to reduce the codec memory requirements for processing samples of a decoded picture by reducing the amount of these samples before processing the decoded picture to a post-decoder resampling engine. However, video codecs available so far do not care about such measures and system standards that rely on such sub-picture resampling, such as MPEG OMAF, which does so with pre- and post-processing without codec awareness. Here, a picture is divided into sub-pictures or regions, and these regions are rearranged or packed within the coded picture. In state-of-the-art codecs, only metadata of these packing steps is carried in the video bitstream, i.e. in the form of so-called per-region packing SEI messages, to reverse the processing before presenting the video. According to an embodiment further described below, the idea is to facilitate knowledge of the packing format in the codec tool decision and coding process.For example, a canonical region-wise packing (RWP) syntax may be introduced, for example, in the PPS, that defines regions of a so-called unpacked picture, i.e., an original picture, where all regions are located with the correct neighboring context at equal spatial resolution and in correspondence to regions in the packed picture. The image transformations described between regions in unpacked and packed pictures may include translation, scaling, mirroring, etc.

[0062] Figures 14a and 14b provide examples of two exemplary RWP scenarios. In these figures, the left side shows an unpacked picture 200, i.e., a scene, and the right side shows a packed picture 202, i.e., a coded picture. The unpacked picture 200 and the packed picture 202 are shown at a uniform sample resolution. The packed picture 202 is subdivided into picture regions 204. Each picture region 204 has a corresponding scene section 206 of the unpacked picture 200 associated with it, the correspondence or association being indicated by the use of the letters a, b, and c. The partitioning of pictures 202 into picture regions 204, indicated by arrows 208, and the mapping of these picture regions 204 to scene sections 206 is signaled in the data stream via RWP information or scene packing / partitioning information, and this mapping 208 may involve different resampling or spatial scaling for the picture regions 204, as becomes apparent from Figures 14a and 14b.

[0063] Thus, according to an embodiment further described below, RWP syntax may be used to describe this scene partitioning / packing, which introduces a number of coding tool restrictions, modifications, and signaling that may be used to exploit this scene partitioning knowledge in the codec. To facilitate understanding of the following description, a schematic framework of an encoder and decoder is described with respect to Figures 15a and 15b. Figure 15a shows a video encoder 220. The video encoder 220 is configured to encode an inbound video 222 into a data stream 224, where each picture 202 of the video 222 is partitioned according to the scene partitions 208 into picture regions 204, each having a respective scene region 206 of the scene 200 mapped thereon via the scene partitions 208. The video encoder 220 supports adjustment of the scene partitions 208. For example, the video encoder 220 comprises an input for receiving information 226 about the scene partitions 208, which may be performed by a module connected to the input of the video encoder when providing a video 222 based on the video data or a video representing the scene 200. The video encoder 220 is configured to signal the scene partitions in the data stream 224. For example, the information of the scene partitions 208 is illustrated as being comprised in the data stream 224 at 228 in Fig. 15a. In a manner described in more detail below, the video encoder 220 is configured to encode the video 222 into the data stream 224 depending on the scene partitions 208 or the information 226, respectively.

[0064] A corresponding video decoder is shown in Figure 15b. The video decoder of Figure 15b, namely 230, is configured to receive a data stream 224 having video 222 encoded therein, and to derive scene partition information 228 from the data stream 224. As indicated by dashed line 232, the decoder 230 may optionally propagate the scene partition information, which it outputs and which represents a reconstruction of the video 222, forward to a recipient of the reconstructed video 234. Importantly, the video decoder 230 is configured to depend on the scene partition information 228 to decode the video 234 from the data stream 224. The following description provides an example of this dependency.

[0065] In detail, as explained so far, the scene partition information 228 may be conveyed in the data stream at several time scales, such as at the picture level, at the picture sequence level, or at the picture group level, or may be updated intermittently, as in the case of an SEI message, whose scope follows until the next update of the scene partition 208 via another SEI message. As can be seen from Figures 14a and 14b, due to the scene mapping 208, different picture regions 204 of a picture of the coded video 222 may depict objects in the scene 200 with different scene sampling densities or scene sampling resolutions, whose environment should be taken into account in the prediction process, as will be outlined in more detail below. Furthermore, different scene sampling densities for picture regions, i.e., different numbers of samples per unit area of ​​the scene 200, may result in different coding parameters that are optimal for efficiently coding the respective picture region 204. It may therefore be advantageous to allow for a per-picture-region setting of such parameters. Still further, due to scene partitions 208 and especially changes in spatial neighborhood relationships between picture regions, when comparing picture 202 on the one hand and scene 200 on the other hand, interfaces between adjacent picture regions 204 may represent interfaces where the picture content on both sides of these interfaces does not relate to immediately adjacent parts of scene 200. Thus, for example, it does not make sense to filter across them. The following description reveals examples of how this should be taken into account in the encoding and decoding process.

[0066] The following description relates, for example, to the relationship between coding partitions on the one hand and scene partitions on the other hand. To explain this, please refer to FIG. 16. FIG. 16 shows one picture 202 of a video 222. To encode the video 222 into a data stream 224, the video encoder 220 and the video decoder 230 partition the picture 202 of the video 222 into blocks 240. In units of these blocks 240, the video 222 is coded into the data stream 224. For example, in units of these blocks, the video encoder and the video decoder may perform predictive residual coding / decoding. Thus, the partition of the picture 202 into blocks 240 may be a partition into transform blocks. Alternatively, the video encoder and the decoder perform video encoding / decoding with switching between intra-prediction and inter-prediction modes in units of blocks 240. Thus, the blocks 240 may be coding blocks or coding units, and the data stream 224 may convey a prediction mode decision for each block 240. Alternatively, the video encoder and decoder may set prediction parameters for intra-predicted blocks and inter-predicted blocks on a block 240 basis. That is, the blocks 240 may be prediction units or prediction blocks. Alternatively, the blocks 240 may be tiles into which the picture 202 is partitioned, and the coding / decoding of these tiles 240 will be performed without coding interdependencies. The data stream 224 may include coding partition information 242 regarding how the picture 202 should be partitioned into the blocks 240.

[0067] In the next described possibility, the video encoder and decoder allow parameterization of the coding partition in such a way that the parameterization can be performed individually for the picture regions 204 of the picture 202. The coding partition can be defined, for example, as shown in FIG. 17. A pre-partitioning into an array of tree root blocks 244 can pre-partition the picture 202. These tree root blocks 244 can then be subject to hierarchical sub-tree partitioning as illustrated by the dotted lines in FIG. 17. The leaf blocks of this hierarchical multi-tree sub-partitioning, indicated using reference sign 246 in FIG. 17, can form, for example, block 240. The size of the tree root block 244, indicated at 248 in FIG. 17, will then define the maximum block size of the resulting block 246. The parameterization can, for example, define this maximum block size. The parameterization can also define a minimum block size 250 for the leaf blocks and / or a maximum number of hierarchical partitioning levels allowed in the hierarchical multi-tree sub-partitioning. A zero partition level would correspond, for example, to no partitioning at all, i.e., leaving the tree root block 244 as is and using it as block 240. A partition level of 1 would correspond to a first partitioning of the tree root block 244 into several regions of a first partitioning level. In the example of FIG. 17, several leaf blocks 246 are of the first partitioning level, marked with a 1 therein. A region of the first partitioning level that is further partitioned into sub-regions results in a sub-region of a second partitioning level, with leaf blocks 246 of such a hierarchical level exemplarily illustrated in FIG. 17, marked with a 2 therein. Thus, the maximum number of such hierarchical partitioning levels may be a further parameterization for the coding partition. And the difference between the minimum and maximum block sizes may also be indicated. The encoder may select these parameterizations for each picture region 204 and may signal the parameterization 252 for each picture region 204 in the data stream 224.The video decoder may use the parameterization 250 to perform coding partitioning based on the information 242. For example, based on the maximum block size 248, the decoder may perform pre-partitioning to the tree root block 244. Based on the indication of the minimum block size, or the maximum number of blocks in the hierarchical partition level, or the difference between the minimum and maximum block sizes 248 and 250, respectively, the decoder may infer that some partitioning is not feasible anyway. Therefore, in case of such partitioning, the encoder does not signal the corresponding partition flag in the coding partition information 242, and the decoder does not convey the respective partition flag from the data stream 224. Calling the CTU of the tree root block 244 and taking the syntax of HEVC as a base, FIG. 18 provides an example for partitioning parameters defined on a picture region 204 basis. According to FIG. 18, different maximum / minimum CTU sizes are defined for each picture region 204. Here, the CTU size defined in HEVC is used as an example. In HEVC, the CTU size is specified generically for the entire coded video sequence CVS and the entire picture, respectively, following the example of Fig. 18, but the syntax of HEVC is extended to signal on a per-region basis, i.e., per picture region 204. In particular, Fig. 18 shows an SPS indicating by flag 254 that picture regions 204 are specified, which may be of different scene resolutions or scene sampling densities, as previously explained. If partitioning to picture regions 204 is applied, i.e., scene partitioning is applied, the parameterization 250 is signaled multiple times, as many times as indicated in 256. The number 256 may be equal to or smaller than the number of picture regions, and the data stream 224 may include indices to different parameterization settings for each parameter region 204 in different pictures.

[0068] The parameters may alternatively be derived implicitly from a single set of default parameters using knowledge of the different sampling of regions within the picture.

[0069] The mapping of areas of the picture 202 may then be done in various ways, for example slices of the picture 202 may be mapped to resolution areas, i.e. to one of the number of parameterizations 250, which number is shown in 256. The mapping may be done in units of independently coded tiles as mentioned above, i.e. each tile may be associated with one of the parameterizations 250. Alternatively, at the sequence or picture parameter set level, depending on the desired codec flexibility, the picture 202 may be subdivided into resolution areas that are coarser or finer than the division of the picture 202 into slices or tiles.

[0070] Furthermore, the video encoder and the video decoder may use the scene partition 208 or the scene partition information 228, respectively, as a source for implicitly specifying the coding block partition. That is, both the video encoder and the decoder may partition the picture 202 into blocks 240 according to the coding partition and may encode / decode the video in units of these blocks 240. As described, these blocks may be transform blocks, prediction blocks, coding blocks, or tiles, just to name a few. Both the video encoder and the decoder will perform the coding partition according to the scene partition 208. This is explained using the tile partition as an example. That is, according to the embodiment just described, the RWP or the scene partition information 228 is used in partitioning the picture 202 into blocks 250 or other units of a block-based hybrid video codec, for example, using motion compensation prediction. For example, if the partition into blocks 240 signaled via information 228 does not align with the boundaries between picture regions 204 defined in RWP information 228, and the regions 204 adjacent to the RWP boundary have different characteristics, such as different spatial sampling densities or different scene resolutions, the corresponding block or unit structure may be aligned to the RWP boundary without the need for explicit signaling, i.e., an implicit partition is derived that coincides with the scene partition or region boundary signaled in RWP 228. See, for example, Figures 19a and 19b. According to Figure 19a, coding partition 242 signals the partition of picture 202 into blocks 240. Block 240 may be, for example, a tile. Coding partition information 242 will, for example, simply indicate the size of block 240, thereby defining the normal subdivision of picture 202 into an array of blocks 240 of that size. However, there is a block 240 that crosses a boundary 250 between two different picture regions 204a and 204b, possibly differing in scene sampling resolution, etc. The boundary 250 is derivable from the bitstream 224 for the decoder 230 based on the scene partition information 228.Thus, the encoder 220 and the decoder 230 implicitly split those blocks 240 that cross the boundary 250 into two blocks 240a and 240b along the boundary 250 without having to explicitly signal this division into coding partitions 242.

[0071] An alternative way of defining the coding partitions and coding partition information 242 depending on the scene partition 228 is shown in Fig. 19b. Here, the blocks or coding units 240 are aligned to the boundaries of the picture regions 204a and 204b. That is, the coding partition information 242 may, for example, simply indicate the size of the blocks 240, and the video encoder and video decoder individually subdivide each picture region 204a and 204b along the perimeter or boundary of each picture region 240a and 240b subdividing this picture region into blocks 240 of that size, starting at some location, such as starting at the top left corner of the respective region. Blocks 240 that extend beyond the boundaries of the respective coding regions to their right side and bottom may be cropped accordingly to end at the respective boundaries. The result is shown in Fig. 19b.

[0072] Similar to the normal subdivision into blocks 240, implicit block partitioning may also be applied to hierarchical multi-tree subdivision, as described with respect to Figure 17. For example, assume that a leaf block of hierarchical level 1 in Figure 17, to the right in the tree root block 244, crosses a boundary between picture regions. In such a case, the division of this region may be implicit between the video encoder and the video decoder, and this signaling of the respective bits in information 242 may be omitted.

[0073] Thus, as just mentioned, the use of RWP information 228 as a source for inferring some partitions that are not explicitly signaled in the coding partition 242 may also pertain to a coding partition 242 that includes hierarchical multi-tree subdivision. This may apply to partitioning blocks completely into leaf regions, but also to hierarchically partitioning blocks that overlap adjacent regions. For example, in FIG. 19b, blocks 240 to the right of region 204a cross the boundary 250. They are cropped. They are processed such that the region extends beyond the periphery of the picture. If such one block 240 is subject to further multi-tree subdivision into further leaf blocks, for each of a subset of partition decisions of the multi-tree subdivision of the multi-tree subdivision of the block that crosses the boundary 250, the encoder and decoder may infer whether it should be performed, in which case the corresponding region of the current partition level is partitioned, or not, so that the corresponding region subject to each partition decision becomes a leaf region of the hierarchical multi-tree subdivision. Without sending a split bit indicator for each split decision as part of the information 242, a guess is made depending on the scene partition of a given picture, i.e., based on whether the corresponding region of the current split level would cross the boundary 250 if not split. For each other split decision of the multi-tree subdivision, independent of those split decisions guessed based on the RWP information 228, the respective split decision will be signaled via a respective split bit indicator in the data stream as part of the information 242. Figure 19c presents an example syntax for CU quad-tree partitioning by taking the HEVC syntax of CU partitioning as a base and highlighting their amendments resulting from utilizing RWP tiling, where it is assumed that Tiled (pixel position) is a function resulting in different values ​​for pixels located in different RWP regions 204a and 204b, i.e., on different sides of the boundary 250, respectively.The syntax achieves splitting of CUs, i.e., CTBs of region 204a that would cross boundary 250 if not cropped, in a way that the leaf blocks are aligned with region boundary 250, while refraining from signaling split indicator split_cu_flag for splits that may be inferred based on the position of boundary 250. In 251, split_cu_flag is not signaled if, for example, the current region subject to splitting that would be signaled by split_cu_flag should be performed anyway when the region would cross boundary 250. In 251, hierarchical splitting is skipped for regions that lie entirely outside region 204a to which current hierarchically split block 240 belongs.

[0074] Another situation in which the scene partition 208 may affect the coding / decoding process is related to sub-pel interpolation filters and motion compensation. According to an embodiment of the present application, the idea here is to facilitate a sub-pel interpolation filter in a block-based hybrid video encoder / decoder to address potentially different spatial sampling characteristics of regions of RWP in a picture during motion compensated prediction whenever, on the one hand, a current block subject to motion compensated prediction and an area of ​​a reference picture referenced by the motion vector of the current block are located in two differently sampled regions, i.e., picture regions 204 associated with different scene sampling densities / resolutions.

[0075] In one embodiment, the following applies: a) When downsampling 2:1 (from reference to current), some full sample positions should be treated as sub-pel positions, or b) When upsampling 1:2 (from reference to current), sub-pel interpolation is performed and the resulting samples should be treated as full-pel samples, or c) Otherwise, no change, normal motion compensated prediction.

[0076] For example, taking the following illustration of a picture in Figures 20a and 20b having two regions (left and right), each 4x4 sample depicting a different but different viewing angle, i.e. the left side of the picture, depicts image content with a relatively smaller viewing angle but higher resolution / fidelity.

[0077] Take for example the following illustration in Fig. 20a of an area 260 among the picture areas 204 of the current picture 202 on the right side and a picture area 262 among the picture areas 204 of the reference picture. Both depicted areas 260 and 262 are 4x4 coded samples wide, the coded samples being indicated by crosses 264. Due to different scene sampling densities / resolutions, the areas 260 and 262 relate to different viewing angles. More precisely, both areas 260 and 262 are composed of the same number of coded samples, but the left-hand area 262 depicts image content with a 2:1 ratio and a relatively smaller viewing angle but higher resolution / fidelity than the right-hand area 260.

[0078] Every full-pel or coded sample position is shown as a cross in the figure. It is clear that the 2×2 subset of left-hand samples 266, i.e., the samples at the positions marked by the rotated dotted crosses, correspond to regions of the version at a lower sampling resolution, i.e., these 2×2 samples on the left, indicated by 268, describe the same viewing angle as the 4×4 samples on the right.

[0079] In this example, for case a), the dashed rectangle 268 on the right indicates the current block, and the motion vector 270 points to the sample positions on the left. Some of the coded full-pel positions are treated as half-pel positions in this case, and only a subset (the rotated dashed cross) is considered for reference (again, enclosed in the red dashed rectangle 266).

[0080] Conversely, in case b), as shown in FIG. 20b below, when the current block is located on the left side as marked through the red dashed rectangle 272 and the reference block is located on the right side with relatively low sampling (1:2), the sub-pel sample positions (indicated through the rotated cross 276) are treated as full sample positions for reference and duplicated parts as shown by 276.

[0081] Note that in FIG. 20b, area 260 is an area of ​​the reference picture, area 262 is an area of ​​the current picture, and the picture region area 260 is located at a lower scene sampling density / resolution than the one of the picture regions 204 where the picture area 262 is located. In other words, the existing samples at the higher resolution, i.e., the cross 264 on the left side, may be used and considered as sub-pels for the lower resolution block 276, for example, when the resolution ratio is 1:2. For example, when the resolution ratio of the block is 1:2, every second sample in the high resolution block 272 may be skipped if the motion vector would point to an integer sample position considering the lower resolution. Each of these skipped samples of the high resolution block would correspond to a half-pel sample when the lower resolution is considered.

[0082] According to further embodiments, the motion vector predictors may be scaled in the encoder and decoder based on characteristics of the region in the packed picture 202 that contains the current block for which the motion vector is to be predicted, and the predictor block that forms the source for some of the motion vector predictors. For example, Figure 21 shows a packed picture 202 having two picture regions 204a and 204b, the left region 204a being illustratively downscaled horizontally with respect to an unpacked picture, shown on the left side in Figure 21 using reference numeral 200, and subdivided into a scene region 206a corresponding to the picture region 204a and a scene region 206b corresponding to the picture region 204b. When the block 280 is located in the picture region 204b and inter prediction is used to code the dashed block 280, the exemplary motion vector candidate 282, i.e. the motion vector used to code the neighboring block 284 located in the picture region 204a, is scaled according to the resampling that the picture region 204 underwent before encoding, or more precisely, according to the scene sampling resolution difference between the picture regions 204a and 204b, respectively. In a specific example, the horizontal component of the motion vector 282 will be increased according to the scene sampling resolution difference and will then be used as the motion vector candidate 286 for the current block 280. Thus, the candidate 286 used will represent a scaled version of the motion vector 282 of the neighboring block 284.

[0083] Figure 22 below shows how the scaling is performed for two blocks 280, 284 belonging to the high resolution region 204b. Note that it is assumed that the low resolution region 204a of the picture 202 is downsampled only in the horizontal domain by a scaling factor of 1:2.

[0084] The scaling of the predictor in the packed picture region is the result of considering the MV predictor in the unpacked region 200. In FIG. 22, it is shown how the predictor MV (in the packed picture 290) is mapped to another vector 292 in the unpacked picture 200. More specifically, in the given example, we consider the horizontal component MV when the vertical component is not subject to any scaling from the RWP 208. x Focus on the horizontal component MV x is composed of two parts, namely, MV x1 (a portion of the motion vector 290 located in the low resolution region 204a) and MV x2 (a portion of the motion vectors 290 located in the high resolution region 204b), which results in 2×MV x1 +MV x2 The horizontal component is 2×MV x1 +MV x2 The predictor 294 in the unpacked region is the predictor used for the current block 280, whose image in the unpacked region 200 is shown in 280'. This predictor 294 also needs to be transformed back to the packed region at the same time, thereby giving a predictor vector 296. However, this motion vector 296 for the current block is generated in a different resolution region 206a, 206b, i.e., 2×sMV x1 and sMV x2 (where, that in the unpacked region 200, i.e., 2×MV x1 +MV x2 = 2 × sMV x1 +sMV x2 (As explained above), this means that the horizontal component is x1 +sMV x2 This results in a scaled motion vector 296 in the packed region 202 that is equal to:

[0085] With respect to the location of the center of the block and the location of the region, the enumerated motion vector components (and parts) can be calculated. For example, in the example considered above (with MVs towards the left having a positive sign), -M.V. x2 =Blockcenter-RegionBoundary -M.V. x1 =MV x -MV x2 - sMV x2 =Blockcenter-RegionBoundary (inside the unpacked region) - 2×MV x1 =2×MV x1 +MV x2 -sMV x2 It is.

[0086] Further embodiments of possible coding / decoding dependencies of the encoder and decoder of Fig. 15a and Fig. 15b relate to filtering parameters. In state-of-the-art video codecs such as HEVC or AVC, parameters of filters such as post-filters or in-loop filters or deblocking filters may be derived based on characteristics of neighboring blocks to be filtered. Filter parameters may relate, for example, to filter strength, transfer function of the filter, filter kernel size, or deblocking filter range. On the encoder and decoder side, the characteristic based on which such filter parameters are determined may be, for example, a prediction mode, such as intra block size or inter block size. If different resolution contents are mixed in the picture 202 before encoding, no special care is taken in the in-loop deblocking filtering step, which may lead to visual artifacts, for example, along the boundary between two such differentially sampled regions, such as between the two differentially sampled picture regions 204a and 204b shown in Fig. 23. This is due to the fact that between decoding and presentation on the display, the two regions 204a and 204b may be resampled to have equal resolution, i.e. the samples per order and the samples modified by the filter in both regions 204a and 204b, i.e. the samples within the filter range 300, may occupy different amounts of order and field of view. To prevent visual artifacts in such cases, according to one embodiment, further filter parameters such as the strength of such a filter or the filter range 300, i.e. the number of samples from the filtering boundary that are taken into account / modified by the filtering process, are modified relative to each other depending on the resolution of the neighboring picture regions 204a and 204b. This is illustrated in FIG. 23.

[0087] According to a further embodiment, the dependency of the encoding / decoding process in the scene partition 208 concerns the reference sample derivation for a filtering process, such as for example an in-loop deblocking filtering process.

[0088] When regions in a picture are packed (e.g., when different horizontal and vertical resamplings are used across the regions and the regions are rearranged to derive a rectangular picture plane), adjacent samples in the packed picture may not be adjacent in the unpacked picture. Therefore, it is undesirable to naively filter across boundaries with such properties. Instead, as part of the present invention, RWP information is used to derive samples in the packed picture that have adjacent relationships in the unpacked picture and use these samples for the filtering process.

[0089] See, for example, Fig. 14b. Here, the interface between picture regions 204c and 204b is an artificial or scene discontinuity interface. That is, the parts of picture regions 204c and 204b that abut each other at this interface 310 in the packed region, i.e., in picture 202, do not abut each other in the scene 200. It therefore becomes feasible to truly discontinue or avoid filtering across this interface 310. However, according to an alternative, and in accordance with the embodiment just outlined, the encoder and decoder fill the filter kernel of each filter that extends beyond this boundary surface 310 from a currently filtered picture area to an adjacent picture area, such as from picture area 204c to picture area 204d, with samples taken from a part of picture 202 that borders the currently filtered picture area at its boundary surface 310 in scene 200, such as the left hand edge side of picture area 204a in the case of the example just outlined. In the example of Figure 14b, if picture area 204b is currently filtered and the filter kernel extends beyond the boundary surface 310, it may be filled by using the right hand edge portion of picture area 204a, as can be seen by looking at scene area 200.

[0090] According to another embodiment, the video encoder and decoder exhibit a scene partition dependency with respect to encoding and decoding as far as quantization is concerned.

[0091] When the residual signal of a motion-compensated predicted picture is transformed in a video codec, the floating-point transform coefficients are usually subjected to lossy quantization at the encoder side. Apart from uniform quantization according to a given step size (quantization parameter or QP), state-of-the-art codecs allow modifying the quantization towards non-uniformity, for example by incorporating a scaling factor into the quantization according to the transform coefficient base frequency. This allows implementing subjectively optimized coding schemes. Such a scaling list may be known a priori at the decoder side and may be activated and indexed by bitstream signaling, but it is also possible to send custom scaling lists per picture in the bitstream.

[0092] When regions in a packed picture depict content with varying characteristics, coding efficiency gains can be achieved by organizing the scaling lists to the characteristics of the regions. For example, these characteristics may include resolution / sampling, but also the nature of the content, etc. Thus, it is feasible to signal these organized scaling lists on a per picture region 204 basis. Figure 24a illustrates one embodiment, where scaling_list_data()[i] is a syntax structure that conveys the coefficients of the scaling list specifically for the i-th region.

[0093] Individual areas or regions 204 in the coded picture 202 can then reference one of the signaled scaling lists, e.g., per index into the list scaling_list_data()[i] from the slice header or picture parameter set. Alternatively, num_scaling_lists_areas_minus1 may be equal to the number of regions in the RWP, which implicitly signal a scaling list for each individual region.

[0094] That is, according to the embodiment just outlined, the video encoder and decoder of Fig. 15a and Fig. 15b allow sub-picture-wise setting of the quantization parameter, and in particular picture region 204-wise setting. The quantization parameter controls the spectral shape of the spatial frequency spectrum of the quantization noise of the quantization. See, for example, Fig. 24b. Fig. 24b shows a transform block 350 as it may be coded into the data stream 224 by the video encoder 220 to encode one block of prediction residuals. The transform block 350 has been obtained by the video encoder using several transforms that spectrally decompose the corresponding residual prediction block in x and y. The prediction coefficients of the transform block 350 are divided into different horizontal frequencies f x and vertical frequency f y For example, the DC transform coefficient value is in the upper left corner, and the highest frequency components in x and y are located in the lower right corner, both of which are shown at 352 and 354, respectively. The just-described quantization parameters, which may be coded in the data stream 224 in such a way that they may vary between picture regions 204 within one picture 202, control the spectral shape of the spatial frequency spectrum of the quantization noise, for example, by controlling the quantization step size to be used to non-uniformly quantize and dequantize the transform coefficients of the transform 350. That is, rather than scaling the quantization step size for quantizing the transform coefficients of the transform 350 equally for all transform coefficients of the transform 350, the quantization parameters described herein vary the quantization step size to be applied in quantizing and dequantizing the transform coefficients of the transform 350 differently. Such scaling may be performed by transform coefficient scaling prior to quantization, while maintaining a uniform quantization step size. For example, the quantization parameters determine the slope or otherwise spectral variation of the quantization step size to be used for the transform coefficients of the transform 350.

[0095] That is, in other words, the encoder 15a and the decoder 15b may enable quantization and dequantization that varies spectrally, i.e., has a coarseness that varies across the transform coefficients 352 of the transform 350, so that the quantization noise in turn varies spectrally. x and f y The variation of the quantization noise over the spectrum may be performed by scaling the transform coefficients 352 before quantization and inverse-scaling the quantized transform coefficients at the decoder side. Even more precisely, at least one quantization parameter, which may be varied by the encoder for different regions of the picture, may define a scaling factor for each transform coefficient 352 of the transform 350. These scale factors define a non-uniform spectral shape. For each region, the at least one quantization parameter may, for example, indicate a scale factor for each transform coefficient 352 separately, or scale factors for different spectral domain transforms 350, such as sub-blocks of the transform 350 into which the transform 350 is subdivided in rows and columns, or into differently sized sub-blocks. Alternatively, the at least one quantization parameter may determine the spectral shape defined by the scale factor according to a parameterized two-dimensional function that defines a scale factor policy with respect to the transform coefficients 352. The encoder will divide each transform coefficient 352 by a respective scale factor before quantization using, for example, a uniform quantization function, i.e., a quantization function that is equal for all transform coefficients 352 of the transform 350. The decoder will receive the quantization parameters per region and thus the resulting scale factor for each transform coefficient 352, and will scale the received quantized transform coefficients 352 by multiplying the respective transform coefficients 352 by the received scale factor, thereby shaping the quantization noise spectrally, i.e., separately for each region of the current picture, according to the received quantization parameters.

[0096] Before proceeding to another aspect of the application and describing an embodiment with respect to this aspect, it is noted that the just-mentioned possibility of varying the quantization parameter to vary the spectral shape of the spatial frequency spectrum of the quantization noise across a picture can also be used independently of the scene partition dependency of the encoding and decoding as described so far with respect to Figures 15a and 15b. Figures 15a and 15b may thus relate to examples of an encoder and a decoder, as well as enabling the just-mentioned sub-picture-wise setting of the quantization parameter, independently of whether the picture 202 of the coded video sequence has undergone any scene partitioning or not, and whether the data stream 224 comprises a scene partition description 228 or not. However, the data stream 224 will in any case comprise information, such as 360 in Figure 24a, on the sub-picture-wise variation of the quantization parameter, via which the spectral shape of the quantization noise is varied as just outlined. In this regard, it is noted that there are also other possibilities to affect the noise coloriness than those described with respect to Figure 24b.

[0097] The latter environment is described in more detail with respect to Figures 25a and 25b, which show an encoder and a decoder, respectively. The encoder of Figure 25a may be, but is not necessarily, a video encoder. The encoder of Figure 25a, indicated using reference number 320, receives a picture 302, which may or may not be a picture of a video 322, and encodes the picture 302 into a data stream 324 using transform-based predictive coding. That is, the encoder 320 may predict blocks of the picture 302 using one or more prediction modes, such as a spatial prediction mode and, in the case of video coding, a temporal prediction mode, and may encode the prediction residual using transform coding. For this purpose, the encoder 320 may subdivide the picture 302 into transform blocks, namely blocks 304, which the encoder 320 then applies a transform 306 to provide the transform blocks 350, respectively. For quantization, the encoder 320 inverse scales this transform 350 using the above-mentioned spectral shape 309 defined by the above-mentioned scale factor and then subjects the result to quantization 311 to give quantized transform coefficients, i.e. quantized transform 313, for each transform block 304. The quantization function used in step 311 may be a uniform transform coefficient that does not vary across transform coefficient positions, but the scale factor defining the spectral shape 309 varies across transform coefficients in the transform 350 and is defined via the above-mentioned at least one quantization parameter. However, the encoder 320 varies this at least one quantization parameter within the picture 302 within a sub-region of the picture 302, such as the sub-region 304. In the data stream 324, the encoder 320 signals a quantization parameter 328 that defines the spectral shape 309 for each sub-region 304. That is, the information 328 specifies a spectral shape 309 for each sub-region 304 such that a different spectral shape is used for transform blocks 304 in one sub-region 304 compared to transform blocks 304 in another sub-region of the same picture 302 .The encoder 320 may decide on the setting of the quantization parameter 328 per region 304 as part of the rate control and / or cost minimization. Alternatively, the encoder 320 may set the quantization parameter 328 according to some other partition of the picture 302, such as according to a partition of the picture 302 into regions of different spatial resolutions at which the picture 302 is coded, or into regions forming different RWP regions as described with respect to Figures 24a and 24b, to which exemplary reference is made to the subsequent figures, in particular Figure 28. In that case, the signaling of such other partitions may be signaled in parallel with the partition of the picture 302 into regions 304, and optionally the setting of the quantization parameter per region 304 may also be inferred from other settings for the region 304, such as, for example, the target spatial resolution at which the respective portion of the picture 302, i.e., each region 304, is coded into the data stream 324.

[0098] Figure 25b shows a decoder 330 that matches the encoder 320 of Figure 25a. It is configured to receive a data stream 324 and reconstruct therefrom a picture 302'. For this purpose, the decoder uses transform-based decoding, a subdivision of the picture 302 into transform blocks 304, a subdivision of the picture 302 into regions 304, and uses the quantization parameter information 328 in the data stream 324 to use the same quantization parameter for each region during inverse quantization. That is, the decoder 330 takes the quantized transform coefficients of the transform 313 from the data stream 324, scales them using a spectral shape 309 determined by the quantization parameter for each region 304 in which the corresponding block 304 of each transform 313 is located, and performs an inverse transform to the spatial domain for reconstruction purposes.

[0099] As already explained, the spectral variation of the quantization coarseness may be controlled in other ways than combining a spectrally uniform quantizer with a spectrally varying scaling. The quantization parameters may be embodied by a list of scale coefficients for different parts of the transform 350, or may define the spectral shape 309 in other ways, such as using a parameterized function. Furthermore, it should be noted that the encoder and decoder need not be transform-based coders; that is, they may code the picture 302 into the data stream 324 in a way other than in the form of quantized transform coefficients. Quantization may, for example, be performed in the spatial domain instead. Furthermore, sub-band decomposition may also be used.

[0100] Picture 302 may be used instead.

[0101] Moreover, as also already explained above, the spectral shape 309 does not need to be newly transmitted in the data stream. Rather, it can be set in a default manner for the encoder and decoder, with the quantization parameter simply indicating the use of this default quantization parameter. For example, for some regions 304, the use of the default quantization parameter may be indicated as active, while for transform blocks in regions 304 for which the quantization parameter indicates inactivity, a spectrally uniform scaling is used, i.e., the quantization noise is spectrally flat. In yet further embodiments, some spectrally varying spectral shapes 309 may be pre-set or represent default settings in the encoder and decoder, where the quantization parameter indicates for each region 304 as to which of the default quantization parameter settings is used to define the spectral shape 309. Furthermore, for some regions 304, a mixture thereof is feasible, and the quantization parameter may explicitly define the spectral shape 309 in the data stream 324 without indexing any pre-defined default settings. In the latter case, the quantization parameters may then dictate the spectral shape, for example, according to some cost minimization and / or according to a manner determined by the encoder, for example, in a manner involved in rate control.

[0102] With respect to Figures 1a and 1b, the possibility is described that the video encoder and the video decoder allow for temporal variation of the target spatial resolution at which the video is coded. According to the embodiments described below, it is possible to allow for sub-picture resolutions at which some videos are coded. According to one embodiment, the sub-picture properties relate to predictive residual coding. For example, it is conceivable to implement a system as shown in Figure 5a on a sub-picture basis, similar to the codec system with full picture resampling (reconstructed picture resampling) according to variant b) described with respect to Figures 1a and 1b. In such a system, the regions on each picture are depicted with varying resolutions with respect to each other. After reconstruction, the images are resampled to equal sampling density.

[0103] Similarly, as shown in FIG. 5b, it is conceivable to implement a system based on variant c) (residual signal resampling) described with respect to FIG. 1a and FIG. 1b, where resampling is selectively performed on a sub-picture basis. In such a system, the residual signal is resampled on a sub-picture basis, for example, per CTU, tile, slice, or otherwise defined region (RWP). For reconstruction of the coded picture, the residual signal is resampled back to full resolution. Thus, the DBP 22 / 36 is filled with pictures of the reference spatial resolution, but the motion compensation and in-loop filtering are also performed at full resolution. In such a system, the resampling of the residual signal is a normative part of the codec, and therefore the filter parameters before / after resampling, as well as the size of the sample array, should be clearly specified, for example with respect to the CTU size.

[0104] Figure 26 shows a coded picture 400 in such a system, where residual resampling is performed on a CTU basis, i.e., at the unit of CTU 402. Thus, the block structure carrying the residual signal samples, and the number of samples obtained, are adjusted according to explicit signaling (resample_idc).

[0105] The syntax element resample_idc can be signaled per individual CTU 402 or aggregated per larger group of CTUs, such as a slice or tile or RWP region. The syntax table in Figure 27 gives an example of per-region signaling variants.

[0106] Similarly, it is necessary that the exact sub-picture filter parameters, such as the filter kernel, are known at the encoder and decoder sides to ensure avoidance of drift between the two. Since the characteristics of sub-pictures may vary across picture planes and different contents, it is prudent to allow signaling of region-specific filter parameters. Signaling conveying filter parameters, such as kernels, that have an association with picture regions may be derivable.

[0107] Thus, an embodiment according to the just outlined aspect may be a video encoder according to Fig. 1a and a video decoder according to Fig. 1b, in which pictures of a video are encoded / decoded in such a way that the encoding / decoding is performed at a target spatial resolution, which varies in units of picture regions into which the pictures are subdivided. Fig. 28 for example shows that pictures 16a-16c of video 12 are subdivided into picture regions 500, and exemplarily shows by dots for picture 16a that different target spatial resolutions are chosen for these regions 500. The variation is signaled in the data stream, such as using the syntax shown in Fig. 27. Internally, the decoded picture buffers 22 and 36 of the encoder and decoder, respectively, may buffer pictures of the video at a reference spatial resolution, such as the original spatial resolution of video 12, or at some other spatial resolution as described above with respect to Figs. 1a and 1b. Thus, the variation of the target spatial resolution in spatial terms may relate to predictive residual coding. Internally, the video encoder and the video decoder may operate according to variant b or variant c described above. As already indicated above, the embodiments described with respect to Figures 1a and 1b and Figures 26 to 28 may be combined with the above embodiments. Thus, the description of Figures 1a and 1b as further specified in Figures 2a to 14 shall also be applicable to the embodiment just outlined as far as the encoder and the internal structure of the encoder etc. are concerned. Furthermore, the varying quantization parameter functions and varying filter functions of Figures 15a to 24b may also be used in the encoder and the decoder according to the embodiment just outlined.

[0108] As explained with respect to Figs. 26-28, in some rate-distortion optimizations in the encoder 10, it is possible to implement the encoder 10 and the decoder 30 of Figs. 1a and 1b in such a way that a target spatial resolution is to be determined for each CTU or prediction unit into which each picture is subdivided. The encoder and the decoder will perform resampling for each CTU or for each prediction unit or for some other picture blocks, at least for those that do not have a reference spatial resolution as explained with respect to variants b and c with respect to Figs. 4b and 5b, respectively. Doing so will obtain or result in a higher coding efficiency. For example, when encoding a resolution with a target spatial resolution lower than the reference spatial resolution, such as the original full resolution of the inbound video 12, a residual of a sampled version is encoded, thereby reducing the number of coefficients. Such a mode can be signaled in the data stream 14 as a new mode or with an additional flag. The signaling may be done per block, such as per CTU, i.e. per tree root block, as shown in FIG. 26, where block 402 is the tree root block as shown on the right side of FIG. 26. FIG. 29 presents an example where a flag, i.e. res_subsampled_flag, signals whether resampling is applied or not, where there are only two possible values ​​for the target spatial resolution for the various blocks, for example. Alternatively, if the flag in FIG. 29 is set, a further syntax element not shown in FIG. 29 may be signaled in the data stream, which syntax element then indicates a target spatial resolution different from the reference spatial resolution, and not setting the flag in FIG. 29 indicates that the block to which this flag refers is of the reference spatial resolution, i.e. no resampling is required.

[0109] Figure 29 shows that this flag may be transmitted at the transform block level, but transmission at the coding unit level, where the prediction mode, i.e., intra / inter prediction mode, is varied and signaled, will also be possible. Thus, according to Figure 30, the flag in Figure 29 will not be repeated for smaller blocks, which would increase the signaling. All residual_coding() syntaxes in a CTU can use the same flag, for example.

[0110] Thus, the flags in Figures 20 and 30 represent the possibility for signaling variations in the target spatial resolution within each picture.

[0111] FIG. 31 shows that the above-mentioned flag res_subsampled_flag may not be transmitted for all blocks into which some pictures are partitioned. As explained in the above description, pictures may be subdivided by the encoder and decoder into blocks of varying size, and according to the example of FIG. 31, an indication of varying the target spatial resolution compared to some reference spatial resolution may be signaled for the whole video or for some sequences of those pictures, e.g., only for blocks of some size within some block size ranges. FIG. 31 provides an example, where it is assumed that residual subsampling is more beneficial for blocks of a given size and, e.g., provides a larger gain for large blocks. Therefore, syntax in data stream 14 is added to avoid avoiding syntax for varying the target spatial resolution for all blocks into which some pictures are partitioned. In other words, the example of FIG. 31 avoids that the signaling overhead spent for varying the target spatial resolution over some pictures is unfavorable to the rate-distortion gain. According to the example of Fig. 31, the data stream 14 signals for a sequence of pictures that a target spatial resolution is signaled for a picture of the sequence of pictures in the following way: the pictures are partitioned into blocks of varying size and all blocks outside a predefined range of block sizes are coded / decoded at a reference spatial resolution, and only for other blocks with block sizes within a predefined block size range it is signaled in the data stream 14 as to which target spatial resolution is chosen for the respective block.Thus, in Figure 31, the SPS includes a flag mixed_resolution_enabled_flag that switches on target spatial resolution variation spatially across the picture, and if it is set, two syntax elements, log2_min_res_subsampled_flag_present_block_size_minus3 and log_diff_max_min_res_subsampled_flag_present_coding_block_size, determine the block size range. Taking the CU-wise transmission of the target spatial resolution indication, i.e., exemplarily via res_subsampled_flag, Figure 32 shows that the respective syntax elements indicating the target spatial resolution for some blocks are only transmitted if the block size goes down to the signaled predefined block size range.

[0112] Fig. 33 concerns the following problem: when downsampling and upsampling only introduce distortion, the above-mentioned res_subsampled_flag cannot inherently provide a better rate-distortion performance, but it would be needed to describe how well the sampled blocks are coded. It would therefore be possible to add delta_qp, i.e. the variation of the quantization parameter compared to the non-resampled blocks, for those blocks with a target spatial resolution other than the reference spatial resolution. Fig. 33 represents an example where this is transmitted in the data stream 14 as part of the SPS, but transmission within the PPS would also be possible. That is, now, with regard to which quantization step size must be applied, or more precisely, which offset compared to the quantization step size is used for the non-resampled blocks coded at the reference spatial resolution, the syntax element delta_qp_4_subsampled_blocks would indicate for those blocks into which some pictures to be subdivided are subject to resampling, i.e. not equal to the target spatial resolution.

[0113] Similarly, a 2:1 subsampling factor may be implemented, or the subsampling factor may be flexible and signaled, more than one may be signaled, and may be fixed for different frames, per CTU size, or per temporal level.

[0114] Although some aspects are described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item, or a feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0115] The data stream of the present invention may be stored on a digital storage medium or transmitted over a transmission medium, such as a wireless or wired transmission medium, such as the Internet.

[0116] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. Implementations may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, having electronically readable control signals stored thereon, which cooperates (or is capable of cooperating) with a programmable computer system on which the respective methods are executed. Thus, the digital storage medium may be computer readable.

[0117] The same embodiment according to the invention comprises a data carrier having electronically readable control signals capable of cooperating with a programmable computer system such that one of the methods described herein is carried out.

[0118] Generally, embodiments of the present invention can be implemented as a computer program product having program code operable to perform one of the methods when the computer program product runs on a computer. The program code can be stored, for example, on a machine-readable carrier.

[0119] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0120] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0121] A further embodiment of the inventive method is therefore a data carrier (or a digital storage medium or a computer readable medium) comprising recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium or recording medium is typically tangible and / or non-transitory.

[0122] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be adapted to be transferred via a data communication connection, for example via the Internet.

[0123] A further embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform one of the methods described herein.

[0124] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0125] Further embodiments according to the invention comprise an apparatus or a system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may comprise, for example, a file server for transferring the computer program to the receiver.

[0126] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0127] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0128] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0129] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0130] Any of the methods described herein, or components of the apparatus described herein, may be implemented at least in part by hardware and / or software.

[0131] The above-described embodiments are merely examples for the principles of the present invention. It is understood that modifications and variations of the configurations and details described herein are obvious to others skilled in the art. It is therefore intended to be limited only by the scope of the forthcoming patent claims, and not by the specific details presented through the description and explanation of the embodiments herein. [Explanation of symbols]

[0132] 10. Video Encoder 12 Source video, inbound video signal 14 Data Stream 161, 162, 163 Pictures 18 Scenes 20 Scene Sections, Areas 22 Decoded Picture Buffer 30 Video Decoder 32 Videos 341, 342, 343 Pictures 36 Decoded Picture Buffer 38 Random Access Distance 40 Minimum Time Distance 42 Prediction Loop 44 Resampling 48 Subtractor 50 Predictive Signals 52 Source Signal 54 Prediction residual signal 56 Transformation, Scaling, and Quantization Modules 58 Prediction residual signal 60 Multiplexer 62 Scaling and inverse transformation module, inverse quantization and inverse transformation module 64 Adder 66 Filter Control Analysis Module 68 In-loop filter 70 Motion Estimation Module 72 Inter Prediction Module 74 Inter-prediction predictors 76 Intra Prediction Module 78 Intra-picture Estimation Module 80 Prediction parameters for inter-predicted blocks 82 Prediction parameters for intra-predicted blocks 84 Selector 86 Intra-prediction predictors 88 General purpose coder control 90 Pictures 92 Initial resampling stage, resampler 94 Inbound Picture 96, 98, 100, 102 Resampler 200 Unpacked Pictures, Scenes, and Unpacked Regions 202 Packed Picture, Current Picture, Packed Region 204 Picture Area 206 Scene section, Scene area 208 Scene mapping, scene division 220 Video Encoder 222 Inbound Video, Coded Video 224 Data Stream 226 Information about scene division 228 Scene division information 230 Video Decoder 234 Reconstructed Video 240 blocks 242 Coding Classification Information 244 Tree Root Block 246 Leaf Block 248 maximum block size 250 Minimum block size, parameterization, bounds 270 Motion Vectors 272 high resolution blocks 276 low resolution blocks 280 blocks 282 Motion Vectors, Motion Vector Candidates 284 adjacent blocks 286 motion vector candidates 290 Motion Vectors 294 Predictors 296 predictor vectors, scaled motion vectors 302 Pictures 304 Conversion Block, Sub-area 306 Conversion 309 Spectral Shape 310 Scene discontinuity boundary surface 311 Quantization 313 Conversion 320 Encoder 322 Videos 324 Data Stream 330 Decoder 350 conversion block, conversion 400 Coded Pictures 500 Picture Area

Claims

[Claim 1] 1. A method for video decoding, comprising: decoding, from a bitstream of encoded video data, a flag indicating whether reference picture resampling is possible and whether a reference picture has parameters that may differ from corresponding parameters of a current picture, where a portion of the current picture refers to a portion of the reference picture, and where the parameters include at least one of a picture height in luma samples and a picture width in luma samples; determining, based on the flag, that reference picture resampling is possible and that the parameters of the reference picture may differ from corresponding parameters of the current picture; resampling the portion of the reference picture to generate a resampled portion of the reference picture; and predicting the current picture using the resampled portion of the reference picture.

Images