VIDEO DECODER, METHOD FOR DECODING, VIDEO ENCODER, METHOD FOR ENCODING, AND NON-TRANSITORY COMPUTER READABLE MEDIUM - Patent application

Abstract

To provide a video decoder, a decoding method, a video encoder, and an encoding method for signaling sample aspect ratio (SAR).SOLUTION: A video decoder 151 comprises: an input step of receiving an encoded video signal comprising encoded picture data and a sequence parameter set (SPS) comprising an indication indicating that sample aspect ratio is changeable within a video sequence; a step of receiving a supplemental enhancement information (SEI) message in an encoded video signal comprising SAR values for a first group of one or more of a plurality of pictures of the video sequence; a step of decoding encoded picture data to obtain a plurality of decoded pictures; and a step of outputting the plurality of decoded pictures and sample aspect information for one or more of the plurality of decoded pictures.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a video decoder, a method for decoding, a video encoder, a method for encoding, and a non-transitory computer readable medium.

[0002] H.265 / HEVC (HEVC = High Efficiency Video Coding) is a video codec that already offers tools to improve or enable parallel processing in the encoder and / or decoder. For example, HEVC supports sub-division of a picture into an array of tiles that are coded independently of each other. Another concept supported by HEVC concerns WPP, according to which CTU rows or CTU lines of a picture may be processed in parallel from left to right, for example in stripes (CTU = coding tree unit), provided that some minimum CTU offset is obeyed in the processing of consecutive CTU lines. However, it would be preferable to have a video codec nearby that supports parallel processing capabilities of a video encoder and / or video decoder even more efficiently.

[0003] In the following we describe the state of the art of VCL partitioning (VCL = video coding layer).

[0004] Usually in video coding, the coding process of picture samples requires smaller partitions, and the samples are divided into several rectangular regions for joint processing such as predictive coding or transform coding. Thus, the picture is divided into blocks of a certain size that is constant during the coding of a video sequence. In H.264 / AVC, a standard fixed size block of 16x16 samples, the so-called macroblock, is used (AVC=Advanced Video Coding).

[0005] In the latest HEVC standard (see ISO / IEC, ITU-T. High efficiency video coding. ITU-T Recommendation H.265 | ISO / IEC 23008 10 (HEVC), edition 1, 2013; edition 2, 2014.), there are coding tree blocks (CTBs) or coding tree units (CTUs) of maximum size 64x64 samples. In further descriptions of HEVC, the more general term CTU is used for such kind of blocks.

[0006] The CTUs are processed in raster scan order starting with the top-left CTU and processing the CTUs down by picture lines to the bottom-right CTU.

[0007] The coded CTU data is organized in a kind of container called a slice. Originally, in previous video coding standards, a slice means a division of a picture comprising one or more consecutive CTUs. A slice is used for segmentation of coded data. From another point of view, a complete picture can also be defined as one large segment, and therefore, historically, the term slice still applies. Besides the coded picture samples, a slice also comprises additional information related to the coding process of the slice itself, which is put into a so-called slice header.

[0008] According to the state of the art, the VCL (video coding layer) also comprises techniques for fragmentation and spatial partitioning. Such partitioning can be applied, for example, to video coding for various reasons, including processing load balancing in parallelization, CTU size matching in network transmission, error mitigation, etc.

[0009] Other examples relate to RoI (RoI = Region of Interest) coding, where there is, for example, a region in the center of the picture that the viewer can select with a zoom-in operation (RoI-only decoding), or gradual decoder refresh (GDR), where intra data (typically placed in one frame of a video sequence) is distributed temporally over several consecutive frames, for example as a string of intra blocks that swipes on the picture plane and locally resets the temporal prediction chain in the same way that an intra picture does for the entire picture plane. In the latter case, two regions are present in each picture, one that has recently been reset and one that is potentially affected by errors and error propagation.

[0010] Reference Picture Resampling (RPR) is a technique used in video coding to adapt the quality / rate of video by using coarse quantization parameters, but also potentially adapting the resolution of each transmitted picture. Thus, the references used for inter prediction may have a different size than the picture currently being predicted for coding. Essentially, RPR requires a resampling process in the prediction loop, e.g., upsampling and downsampling filters to be defined.

[0011] Depending on the flavor, RPR can result in a change in the coded picture size at any picture, or it can be restricted to occur only at specific positions bounded by segment boundaries, such as adaptive HTTP streaming, only at specific pictures. Summary of the Invention

[0012] It is an object of the present invention to provide an improved concept for video encoding and decoding.

[0013] The object of the present invention is solved by the subject matter of the independent claims.

[0014] Preferred embodiments are provided in the dependent claims. [Brief description of the drawings]

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0016] [Figure 1] 1 illustrates a video encoder according to one embodiment. [Diagram 2] 1 illustrates a video decoder according to one embodiment. [Diagram 3] 1 illustrates a system according to one embodiment. [Figure 4] 1 shows region of interest (RoI) scalability versus expansion scalability. [Figure 5a] A first example of a content fragment (gray) moving within the picture over time is shown. [Figure 5b] A second example of a content fragment (gray) moving within the picture over time is shown. [Figure 6a] Indicates the current picture with mixed reference pictures. [Figure 6b] 13 shows an example of ignoring the cropping window of a reference picture. [Figure 7] 1 shows a video encoder. [Figure 8] 1 shows a video decoder. [Figure 9] It shows the relationship between a reconstructed signal, for example a reconstructed picture, on the one hand, and a combination of a prediction residual signal and a prediction signal as signalled in the data stream, on the other hand. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The following description of the figures begins with a description of an encoder and decoder of a block-based predictive codec for encoding pictures of a video to form an example of an encoding framework into which embodiments of the present invention may be incorporated. The respective encoders and decoders are described with respect to Figures 7 to 9. A description of embodiments of the inventive concepts is then given together with a description of how such concepts may be incorporated into the encoders and decoders of Figures 7 and 8, respectively. However, the embodiments of Figures 1 to 3 and described below may also be used to form encoders and decoders that do not operate according to the encoding framework underlying the encoders and decoders of Figures 7 and 8.

[0018] FIG. 7 shows a video encoder, typically an apparatus for predictively coding a picture 12 into a data stream 14 using transform-based residual coding. The apparatus, i.e. the encoder, is indicated with the reference number 10. FIG. 8 shows a corresponding video decoder 20, for example an apparatus 20 configured to predictively decode a picture 12' from the data stream 14 also using transform-based residual decoding, where the apostrophe is used to indicate that the picture 12' reconstructed by the decoder 20 deviates from the picture 12 originally encoded by the apparatus 10 in terms of coding losses introduced by quantization of the prediction residual signal. FIG. 7 and FIG. 8 exemplarily use transform-based predictive residual coding, although embodiments of the present application are not limited to this type of predictive residual coding. This also applies to other details described with respect to FIG. 7 and FIG. 8, as outlined below.

[0019] The encoder 10 is configured to subject the prediction residual signal to a spatial-spectral transformation and to encode the prediction residual signal thus obtained into a data stream 14. Similarly, the decoder 20 is configured to decode the prediction residual signal from the data stream 14 and to subject the prediction residual signal thus obtained to a spectral-spatial transformation.

[0020] Internally, the encoder 10 may comprise a prediction residual signal former 22 which generates a prediction residual 24, allowing to measure the deviation of a prediction signal 26 from an original signal, for example from the picture 12. The prediction residual signal former 22 may be a subtractor which subtracts a prediction signal from the original signal, for example from the picture 12. The encoder 10 then further comprises a transformer 28 which applies a spatial-spectral transformation to the prediction residual signal 24 to obtain a spectral domain prediction residual signal 24', which is quantized by a quantizer 32 also constituted by the encoder 10. Thus, the quantified prediction residual signal 24 is coded into the bitstream 14, while the encoder 10 optionally constitutes an entropy coder 34 which transforms and quantizes the prediction residual signal into a data stream by entropy coding, and the prediction signal 26 is generated on the basis of the prediction residual signal 24, which is coded into the data stream 14 or decodable from the data stream 14, by a prediction stage 36 of the encoder 10. For this purpose, the prediction stage 36 may internally comprise, as shown in FIG. 7, a dequantizer 38 which dequantizes the prediction residual signal 24″ to obtain a spectral domain prediction residual signal 24′″ corresponding to the signal 24′ excluding the quantization losses, after which an inverse transformer 40 inversely transforms the latter prediction residual signal 24′″, for example by subjecting it to a spectral-spatial transformation, to obtain a prediction residual signal 24″″ corresponding to the original prediction residual signal 24 excluding the quantization losses. A combiner 42 of the prediction stage 36 may then recombine, for example by addition, the prediction signal 26 and the prediction residual signal 24″″ to obtain a reconstructed signal 46, a reconstruction of an original signal, for example the picture 12. The reconstructed signal 46 may correspond to the signal 12′. A prediction module 44 of the prediction stage 36 then generates the prediction signal 26 based on the signal 46, for example by using spatial prediction, i.e. intra-picture prediction, and / or temporal prediction, i.e. inter-picture prediction.

[0021] Similarly, the decoder 20 may be internally constructed from components corresponding to the prediction stage 36 and interconnected in a corresponding manner, as shown in Figure 8. In particular, an entropy decoder 50 of the decoder 20 may entropy decode a quantized spectral domain prediction residual signal 24'' from the data stream, where an inverse quantizer 52, an inverse transformer 54, a combiner 56 and a prediction module 58 are interconnected and cooperate in the manner described above with respect to the modules of the prediction stage 36 to recover a reconstructed signal based on the prediction residual signal 24'', such that an output of the combiner 56 yields the reconstructed signal, i.e., picture 12', as shown in Figure 8.

[0022] Although not specifically mentioned above, it is readily apparent that the encoder 10 may set some coding parameters, including, for example, prediction mode, motion parameters, etc., according to, for example, some rate- and distortion-related criteria, for example, a method for optimizing coding cost, for example, according to some optimization scheme. For example, the encoder 10 and the decoder 20, and the corresponding modules 44, 58, may support different prediction modes, such as intra- and inter-coding modes, respectively. The granularity with which the encoder and the decoder switch between these prediction mode types may correspond to the subdivision of the pictures 12 and 12', respectively, into coding segments or coding blocks. In units of these coding segments, for example, the picture may be subdivided into intra-coded and inter-coded blocks. The intra-coded blocks are predicted based on the spatial, already coded / decoded neighborhood of the respective blocks, as outlined in more detail below. There may be several intra-coding modes, each of which may be selected for each intra-coding segment, including directional or angular intra-coding modes, in which the respective segments are filled by extrapolating neighboring sample values ​​along a certain direction specific to the respective directional intra-coding mode to the respective intra-coding segment. The intra-coding modes may also comprise one or more further modes, such as, for example, a DC coding mode in which the prediction for each intra-coded block assigns a DC value to all samples in the respective intra-coded segment, and / or a planar intra-coding mode in which the prediction for each block is approximated or determined to be a spatial distribution of sample values ​​described by a two-dimensional linear function over the sample positions of the respective intra-coded block with a planar driving slope and offset defined by the two-dimensional linear function based on neighboring samples. In comparison, inter-coded blocks may, for example, be predicted temporally.In the case of inter-coded blocks, motion vectors may be signaled in the data stream, which indicate the spatial displacement of parts of previously coded pictures of the video to which the picture 12 belongs, and the previously coded / decoded pictures are sampled to obtain a prediction signal for the respective inter-coded block. This means that in addition to the residual signal coding provided by the data stream 14, such as entropy-coded transform coefficient levels representing the quantized spectral domain prediction residual signal 24'', the data stream 14 may also code therein optional further parameters, such as coding mode parameters for assigning coding modes to the various blocks, prediction parameters for some blocks, such as motion parameters for inter-coded segments, and parameters for controlling and signaling the subdivision of the pictures 12 and 12′ into segments, respectively. The decoder 20 uses these parameters to subdivide the pictures in a similar way as the encoder did, assign the same prediction modes to the segments, and perform the same predictions to result in the same prediction signals.

[0023] FIG. 9 shows the relationship between a reconstructed signal, for example a reconstructed picture 12′, on the one hand, and a combination of a prediction residual signal 24″″ and a prediction signal 26 signaled in the data stream 14, on the other hand. As already mentioned above, the combination may be additive. The prediction signal 26 is shown in FIG. 9 as a subdivision of the picture area into intra-coded blocks, exemplarily shown with hatching, and inter-coded blocks, exemplarily shown without hatching. The subdivision may be any subdivision, such as a regular subdivision of the picture area into rows and columns of square or non-square blocks, or a multi-tree subdivision, subdividing the picture 12 from a tree root block into multiple leaf blocks of different sizes, such as a quadtree subdivision, where the picture area is first subdivided into rows and columns of tree root blocks, which are then further subdivided into one or more leaf blocks according to a recursive multi-tree subdivision, a mixture of which is shown in FIG. 9.

[0024] Also, data stream 14 may have intra-coding modes coded therein for intra-coded blocks 80, which assigns each intra-coded block 80 one of several supported intra-coding modes. For inter-coded blocks 82, data stream 14 may have one or more motion parameters coded therein. Generally speaking, inter-coded blocks 82 are not limited to being temporally coded. Alternatively, inter-coded blocks 82 may be any blocks predicted from previously coded portions beyond current picture 12 itself, such as previously coded pictures of the video to which picture 12 belongs, or from another view or hierarchically lower picture if the encoder and decoder are scalable encoder and decoder, respectively.

[0025] Also, the prediction residual signal 24'''' in FIG. 9 is illustrated as a subdivision of the picture region into blocks 84. These blocks are sometimes called transform blocks to distinguish them from the same ones as the coding blocks 80 and 82. In effect, FIG. 9 shows that the encoder 10 and the decoder 20 can respectively use two different subdivisions of the picture 12 and the picture 12' into blocks, namely one subdivision into coding blocks 80 and 82, respectively, and another subdivision into transform blocks 84. Although both subdivisions may be the same, e.g., each coding block 80 and 82 may simultaneously form a transform block 84, FIG. 9 shows the case where, for example, the subdivision into transform blocks 84 forms an extension of the subdivision into coding blocks 80, 82 such that any boundary between the two blocks 80 and 82 overlaps with the boundary between the two blocks 84, or, alternatively, such that each block 80, 82 either coincides with one of the transform blocks 84 or coincides with a cluster of transform blocks 84. However, the subdivisions may be determined or selected independently of each other, such that the transform block 84 may cross the block boundary between the blocks 80, 82. As far as the subdivision into the transform block 84 is concerned, it is similar to that presented with respect to the subdivision into the blocks 80, 82, e.g., the block 84 may be the result of a regular subdivision of the picture region into blocks (with or without arrangement into rows and columns), a recursive multi-tree subdivision of the picture region, or a combination thereof, or any other type of blocking. As an aside, it should be noted that the blocks 80, 82, and 84 are not limited to being square, rectangular, or any other shape.

[0026] 9 further illustrates that the combination of the prediction signal 26 and the prediction residual signal 24'''' directly results in the reconstructed signal 12'. However, it should be noted that according to alternative embodiments, more than one prediction signal 26 may be combined with the prediction residual signal 24'''' to produce the picture 12'.

[0027] In Fig. 9, the transform blocks 84 have the following importance: the transformer 28 and the inverse transformer 54 perform their transform in units of these transform blocks 84. For example, many codecs use some kind of DST or DCT for all transform blocks 84. Some codecs allow to skip the transform for some of the transform blocks 84, so that the prediction residual signal is directly coded in the spatial domain. However, according to the embodiment described below, the encoder 10 and the decoder 20 are configured to support several transforms. For example, the transforms supported by the encoder 10 and the decoder 20 can include: DCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform DST-IV, where DST stands for Discrete Sine Transform DCT-IV DST-VII Identity Transformation (IT)

[0028] Of course, the transformer 28 supports all of the forward transform versions of these transforms, while the decoder 20 or inverse transformer 54 supports the corresponding backward or inverse versions: Inverse DCT-II (or inverse DCT-III) Reverse DST-IV ·Inverse DCT-IV · Reverse DST-VII Identity Transformation (IT)

[0029] The following description provides further details on which transforms may be supported by the encoder 10 and the decoder 20. Note that in any case, the set of supported transforms may comprise only one transform, such as one spectral-to-spatial or spatial-to-spectral transform.

[0030] As already outlined above, Figures 7 to 9 are presented as examples in which the inventive concepts further described below can be implemented to form specific examples of encoders and decoders according to the present application. The encoders and decoders of Figures 7 and 8 may represent possible implementations of the encoders and decoders described below, respectively. However, Figures 7 and 8 are merely examples. However, an encoder according to an embodiment of the present application may perform block-based coding of the picture 12 differently from the encoder of Figure 7, as will be outlined in detail below, for example in that the same is a still image encoder rather than a video encoder, in that it does not support inter-prediction, or in that the subdivision into blocks 80 is performed in a different way than illustrated in Figure 9. Similarly, a decoder according to an embodiment of the present invention is capable of block-based decoding of pictures 12' from data stream 14 using the encoding concepts further outlined below, but may differ from the decoder 20 of FIG. 8 in, for example, being a still picture decoder rather than a video decoder, not supporting intra prediction, or subdividing pictures 12' into blocks in a different manner than described with respect to FIG. 9, and / or not deriving prediction residuals from data stream 14 in, for example, the transform domain.

[0031] In the following, a generic video encoder according to an embodiment is described in FIG. 1, a generic video decoder according to an embodiment is described in FIG. 2, and a generic system according to an embodiment is described in FIG.

[0032] FIG. 1 shows a generic video encoder 101 according to an embodiment.

[0033] The video encoder 101 is configured to encode a plurality of pictures of a video by generating an encoded video signal, each of the plurality of pictures comprising original picture data.

[0034] The video encoder 101 comprises a data encoder 110 configured to generate an encoded video signal comprising encoded picture data, the data encoder being configured to encode a number of pictures of a video into the encoded picture data.

[0035] Furthermore, the video encoder 101 comprises an output interface 120 configured to output the coded picture data for each of the multiple pictures.

[0036] FIG. 2 shows a generic video decoder 151 according to an embodiment.

[0037] The video decoder 151 is configured to decode an encoded video signal comprising encoded picture data to reconstruct a number of pictures of the video.

[0038] The video decoder 151 comprises an input interface 160 arranged to receive an encoded video signal.

[0039] Additionally, the video decoder comprises a data decoder 170 configured to reconstruct multiple pictures of the video by decoding the encoded picture data.

[0040] FIG. 3 illustrates a general system according to an embodiment.

[0041] The system includes a video encoder 101 of FIG. 1 and a video decoder 151 of FIG.

[0042] The video encoder 101 is configured to generate an encoded video signal.

[0043] The video decoder 151 is configured to decode the encoded video signal to reconstruct video pictures.

[0044] A first aspect of the present invention is set out in claims 1 to 33. The first aspect provides a sample aspect ratio signal.

[0045] A second aspect of the present invention is recited in claims 34 to 72. The second aspect provides Reference Picture Resampling restrictions to reduce the implementation burden.

[0046] A third aspect of the present invention is set out in claims 73 to 131. The third aspect provides flexible region-based referencing for zooming reference picture resampling, in particular providing more efficient address zoom use cases.

[0047] The first aspect of the present invention will now be described in detail.

[0048] In particular, the first aspect provides sample aspect ratio signalling.

[0049] Sample aspect ratio (SAR) is relevant to correctly presenting the encoded video to the consumer, so that when the aspect ratio of the encoded sample array changes over time via RPR (e.g., by subsampling in one dimension), the aspect ratio of the presented picture can remain constant as intended.

[0050] The latest SAR in the Video Usability Information (VUI) in a Sequence Parameter Set (SPS) like HEVC or AVC only allows setting a constant SAR for the entire signaled video sequence, e.g., SAR changes are only allowed at the beginning of an encoded video sequence (e.g., the sample aspect ratio is constant for each encoded video sequence).

[0051] Therefore, as part of this invention, a new mode of SAR signaling is introduced into video coding. The sequence level parameter set, or SPS, contains the following indications: RPR is used (so the encoded picture size may change) Actual SAR is not given in VUI, Alternatively, the SAR of a coded video may be indicated as dynamic and may vary within a CVS (coded video sequence). The actual SAR of a coded picture is indicated via a SEI (supplemental enhancement information) message at the resolution switching point. [Table 1]

[0052] Dynamic SAR Information SEI Message [Table 2]

[0053] Similarly, the vui_aspect_ratio_constant_flag may be employed, for example.

[0054] The vui_aspect_ratio_constant_flag may be, for example, an indication of whether the sample aspect ratio is constant for a video sequence or whether the sample aspect ratio is variable within a video sequence.

[0055] For example, if vui_aspect_ratio_constant_flag may be set to, for example, 0 (or may be set to, for example, FALSE, or may be set to, for example, -1), this may indicate, for example, that dynamic SAR information is present, for example, in the SEI message.

[0056] In an alternative embodiment, the SAR information in the VUI (e.g., SPS) is used as a default, which is used unless an SEI message is available, in which case the information in the SEI message is overwritten with the information in the SPS. [Table 3]

[0057] In another embodiment, the SAR information is associated with a picture resolution and signaled in the PPS (picture parameter set), and the picture resolution is signaled. A default SAR is signaled in the SPS, and if the SAR changes for a particular picture resolution, a different SAR is signaled, overriding the default SAR.

[0058] SPS VUI: [Table 4]

[0059] Also, for the SEI case, the SPS can further indicate that the SAR may change and that it will be updated to the PPS (similar to the previous aspect_ratio_dynamic_SEI_present_flag). Thus, in some applications it may be possible to constrain or limit the SAR to not change, which can facilitate enforcement or RPR / ARC.

[0060] PPS: [Table 5]

[0061] If pps_aspect_ratio_info_present_flag is set to 0, the default SAR is obtained from the SPS, otherwise the actual SAR is provided.

[0062] The second aspect of the present invention will now be described in detail.

[0063] In particular, the second aspect provides signaling regarding constraints for reference picture resampling.

[0064] The burden on implementations can be reduced by restricting the RPR scheme in various ways. A general RPR scheme that does not include any additional restrictions, such as the one in the following invention, would require implementers to over-provision their decoder hardware to do the following: Any current picture, worst case: resampling on every picture Resampling of DPB (decoded picture buffer), intermediate GOP (group of pictures) versus any picture at a defined position with fewer reference pictures Simultaneous resampling of multiple images of different resolutions to a target resolution Cascaded resampling chain of ref pic with loss of picture quality

[0065] The following invented restrictions make it possible to reduce the implementation costs of a codec featuring such a restricted RPR scheme compared to a non-restricted RPR codec.

[0066] In one embodiment, resolution changes are only allowed at RAPs (random access points), e.g., the maximum number of resampled pictures is the amount of RASL (random access decodable skipped picture) pictures at this RAP, which are typically one or more GOPs apart, e.g., tens of pictures, which reduces the worst-case rate at which such resampling operations must be supported.

[0067] In another embodiment, the resolution change is performed on key pictures in a hierarchical GOP, e.g. pictures: Belongs to the lowest time layer, and occurs once in every GOP, and All subsequent pictures in coding order have a lower POC (e.g. an earlier presentation timestamp), When a reference picture is resampled, none of the immediately following pictures in the GOP of a higher temporal layer require cascade up / downsampling.

[0068] According to another embodiment, a resolution change is only allowed in the picture that immediately follows the key picture in presentation order, ie the first picture of the next GOP in presentation order.

[0069] In another embodiment, the time distance between successive resolution changes is limited by a minimum POC (picture order count) distance in the level definition.

[0070] In another embodiment, the temporal distance between successive resolution changes is limited by a minimum number of intermediate coded pictures in the level definition.

[0071] In another embodiment, resolution changes can only occur for pictures marked as non-discardable or reference pictures with non_reference_picture_flag equal to 0.

[0072] In another embodiment, the speed of the resolution change is limited by the level definition.

[0073] In another embodiment, the resampling of reference pictures relative to the current picture is restricted to using a single resampling ratio, e.g., all reference pictures of the current picture that have a different resolution than the current picture are required to have the same resolution.

[0074] In another embodiment, when one reference picture of the current picture requires resampling, all reference pictures of the current picture need to use resampling, e.g., be of the same original resolution as the one reference picture.

[0075] In another embodiment, only one reference picture of the current picture is allowed to require resampling.

[0076] According to another embodiment, the maximum number of pictures requiring resampling at a resolution change point is optionally indicated in the coded video sequence / bitstream as a guarantee to the decoder, and if no indication is present, it is inferred or indicated by the level definition.

[0077] In another embodiment, the original (unresampled) reference picture is removed from the reference picture list and / or decoded picture buffer after being resampled, e.g., marked as not used for reference, such that only the resampled reference picture is available therefrom.

[0078] In another embodiment, the resampling ratios used within a coded video sequence are restricted to a set of resampling ratios contained in a parameter set having sequence or bitstream scope (decoding parameter set, DPS; sequence parameter set, SPS).

[0079] The third aspect of the present invention will now be described in detail.

[0080] In particular, the second aspect provides flexible region-based referencing for zooming for reference picture resampling.

[0081] As mentioned above, layered codecs such as SHVC and SVC accommodate two enhancement scalability modes: RoI scalability (where the area of ​​the lower layer picture is expanded at the higher layer) and enhancement scalability (where the lower layer picture is enhanced via additional content at the higher layer) as shown in Figure 4 below.

[0082] Enhanced scalability may refer, for example, to so-called zoom-out use cases, e.g. use cases that change over time in the sense that the video completely covers more content, e.g. a larger capture angle, a larger part of a scene, a larger area, etc.

[0083] FIG. 4 shows Region of Interest (RoI) scalability versus extended scalability.

[0084] In scenarios where zooming in and out is allowed, a zoom and movement region is defined that is used for prediction and predicted. This is known as RoI scalability (typically zooming in) or extension scalability (typically zooming out). In RoI scalability with scalable coding, a region is usually defined in the reference picture that is upscaled to the dimension of the reference picture. However, in scalable coding, the upper layer picture and the lower layer picture on which prediction is performed represent the same time.

[0085] In the case of SHVC and SVC, this is done due to layered coding, and in these cases the juxtaposed base layer does not represent any motion, e.g., the corresponding samples in the base layer are known, so it was possible to completely upscale a known region in the base layer and operate on that upscaled basis.

[0086] However, in RPR applications, the two pictures between which prediction is performed do not depict the same time instance, and therefore some content outside the defined region may move from time instance A (low resolution) to the zoomed-in / out region at time instance B (high resolution). Forbidding reference to these regions for prediction is detrimental for coding efficiency.

[0087] However, in the case of RPR, the reference can point to some area outside the corresponding reference region, for example by zooming in the area of ​​an object moving into the RoI. This is shown in Figure 5a without actually changing the coding resolution:

[0088] FIG. 5a depicts a first example of a content fragment (gray) moving within the picture over time.

[0089] In a first embodiment, a reference region is defined that contains an area larger than the area of ​​the RoI, such that the grey boxes in the figure that fall in the RoI zoom region are in the reference:

[0090] FIG. 5b depicts a second example of a content fragment (gray) moving within the picture over time.

[0091] This leads to reconstructing a region slightly larger than the RoI for the picture corresponding to the RoI, and the additional region is removed by indicating a cropping window. The problem arises from the fact that the scaling factors used to upsample the reference are calculated in Versative Video Coding (VVC) from the cropped-out picture. First, assuming there is no RoI, the horizontal scale factor HorScale and the vertical scale factor VerScale are calculated as follows: HorScale = CroppedOutWidthPic / CroppedOutWidthRefPic VerScale = CroppedOutHeightPic / CroppedOutHeightRefPic

[0092] The reason for showing the ratio based on the cropped out picture is that depending on the picture size of interest, some additional samples need to be decoded, since the codec needs to be a multiple of the minimum size (in VVC 8 samples). So if either Pic or RefPic is not a multiple of 8, some samples are added to the input picture, making them a multiple of 8, leading to different ratios and incorrect scaling factors. This problem can be even worse if the bitstream is desired to be coded as "mergeable", e.g., if the picture size needs to be a multiple of the CTU size up to 128, then the correct scaling factor needs to take into account the cropping window.

[0093] In the described scenario (combining RPR with RoI), where a cropping window is used to include some additional references, the use of a cropping window is insufficient.

[0094] As mentioned above, the RoI in the reference picture can be defined a little larger, and the RoI is used for reference but discarded along with the cropping window in the current reconstructed picture. However, if the horizontal scale factor HorScale and the vertical scale factor VerScale are calculated as follows: HorScale = CroppedOutWidthPic / WidthEnlargedRefRoI VerScale = CroppedOutHeightPic / HeightEnlargeRefRoI Some of the samples within the expanded RoI actually correspond to samples within the cropped out region, leading to incorrect results.

[0095] In the following, a cropping window based concept according to a first group of embodiments is described.

[0096] Thus, in the first group of embodiments above, the calculation may be, for example, as follows: HorScale = CodedPicWidth / RefRoIWidth VerScale = CodedPicHeight / RefRoIHeight Such calculations include samples being cropped for the purposes of calculating the scale factor.

[0097] Regarding signaling, in one embodiment, the signaling of the extended RoI indicates that the cropping window information should be ignored in the scaling factor calculation.

[0098] In another embodiment, whether the cropping window needs to be taken into account in the calculation of the scaling factor is indicated in the bitstream (eg, in the parameter set or slice header). [Table 6]

[0099] The cropping window may also be referred to as, for example, an adaptive cropping window. The offsets for the cropping window / adaptive cropping window may also be referred to as, for example, pps_conf_win_left_offset, pps_conf_win_top_offset, pps_conf_win_right_offset, and pps_conf_win_botton_offset.

[0100] Instead of using use_cropping_for_scale_factor_derivation_flag to determine whether to ignore information in the coded video signal on the cropping window for upscaling a region in a reference picture (or to determine whether information in the coded video signal on the cropping window is used for upscaling a region in a reference picture), for example, pps_scaling_window_explicit_signalling_flag may be used.

[0101] For example, if pps_scaling_window_explicit_signalling_flag is set to 0 (or, e.g., set to FALSE, or, e.g., set to −1), information in the coded video signal on the cropping window may be used, e.g., to upscale regions in a reference picture, and, for example, if pps_scaling_window_explicit_signalling_flag is set to 1 (or, e.g., set to TRUE), information in the coded video signal on the cropping window may be ignored, e.g., to upscale regions in a reference picture.

[0102] One drawback of the above approach is that a larger area is decoded for the current picture to allow for referencing samples outside the RoI, e.g., samples on the expanded RoI. More specifically, samples are decoded in the area outside the RoI and later discarded in the cropping window. This leads to additional sample overhead and reduced coding efficiency that can potentially counteract the coding efficiency gain of allowing for referencing outside the corresponding RoI in the reference picture.

[0103] A more efficient approach would be to decode only the RoI (omitting the additional samples needed to create an 8x picture or CTU, as described above), but allow for reference to samples within the expanded RoI.

[0104] In the following, a bounding box based concept according to a second group of embodiments is described.

[0105] In the second group of embodiments above, samples that are outside the red rectangle but inside the green box (RoI offset plus an additional RoI offset) are used to determine the resampled ref pic instead of using only the red RoI.

[0106] The size of the bounding box for the MV around the red cutout is defined / signaled with the advantage that it limits memory access / line buffer requirements and also enables implementation with a pic-wise upsampling approach.

[0107] Such signaling can be included in the PPS(additional_roi_X): [Table 7]

[0108] Therefore, the derivation of the scaling factors is as follows: HorScale = CroppedOutWidthPic / RefRoIWidth VerScale = CroppedOutHeightPic / RefRoIHeight

[0109] In one embodiment, the reference sample is identified by finding the aligned sample using the RoI_X_offset and applying a MV that is clipped if the reference sample is outside the extended RoI indicated by additional_RoI_x. Alternatively, samples outside this extended RoI are padded with the last sample within the extended RoI.

[0110] In another embodiment, this expanded RoI is used only as a limit or constraint that can be used for implementation optimization. For example, if the reference picture is fully upsampled first when necessary, rather than on-the-fly (block-based), only the expanded RoI is resampled instead of the whole picture, saving a lot of processing.

[0111] A further problem is when two or more reference pictures are used simultaneously. In this case, it is necessary to identify the picture to which the RoI region information applies. In such cases, instead of adding information to the PPS, the slice header indicates that some of the entries in the reference list refer to a part of it, rather than the whole picture. For example, [Table 8]

[0112] In further embodiments, additional constraints are in place: Only reference pictures with lower POC can have RoI information. Typically, RoI switching, with the described features, is applied in open GOP switching scenarios, and thus POCs with higher POC already represent RoI scenes. Only one reference picture can have RoI information.

[0113] In another embodiment, RoIInfo() is carried in the picture parameter set, and the slice header carries only a flag (RoI_flag) for each reference picture, indicating whether the RoI information should be applied for resampling (deriving the scaling factor). The following figure shows the principle for four coded pictures, two before and two after the switching point. At the switching point, the total resolution remains constant, but an upsampling of the RoI is performed. Two PPS are defined, and the PPS of the latter two pictures indicates the RoI in the reference picture. Furthermore, the slice headers of the latter two pictures carry RoI_flag[i] for each of their reference pictures, the value of which is denoted in the figure as "RoI_flag" or "RF = x".

[0114] Additionally, the slice header may carry, for each reference picture, not only the RoI_flag as described above, but also, if the flag is true, an additional index into the array of RoIInfo() carried in the parameter set to identify the RoI information to apply to that particular reference picture.

[0115] FIG. 6a shows a current picture with mixed reference pictures.

[0116] In the following, the case of zooming out according to the third group of embodiments will be described.

[0117] Instead of RoI scalability, the above third group of embodiments may consider extended scalability, for example scalability going from the RoI picture to a larger area. In such a case, the cropping window of the reference picture should also be ignored, especially if an area of ​​the currently decoded picture is identified as an area for scalability such as zooming.

[0118] FIG. 6b shows an example of ignoring the cropping window of a reference picture for an identified region in the current picture.

[0119] HorScale = IdentifiedRegionInPicWidth / CodedRefPicWidth VerScale = IdentifiedRegionInPicHeight / CodedRefPicHeight

[0120] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of the corresponding method, in which blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0121] Depending on the specific implementation requirements, the embodiments of the present invention can be implemented in hardware, or in software, or at least partly in hardware, or at least partly in software. The implementation can be carried out using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that the respective method is executed. The digital storage medium may therefore be computer readable.

[0122] Some embodiments according to the invention comprise a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0123] In general, embodiments of the present invention can be implemented as a computer program product having program code operable to perform one of the methods when the computer program product is run on a computer. The program code may for example be stored on a machine readable carrier.

[0124] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0125] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0126] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) comprising recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0127] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, the data stream or the sequence of signals for example being adapted to be transferred via a data communication connection, for example via the Internet.

[0128] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0129] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0130] Further embodiments according to the invention comprise an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example comprise a file server for transferring the computer program to the receiver.

[0131] In some embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functions of the methods described herein.

[0132] In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0133] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0134] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0135] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the configurations and details described herein will be apparent to others skilled in the art. It is therefore intended to be limited only by the scope of the impending claims, and not by the specific details presented by the description and illustration of the embodiments herein.

Claims

1. 1. A video decoder for decoding an encoded video signal comprising encoded picture data to decode a plurality of pictures of a video sequence of a video, the video decoder comprising: At least one memory; at least one processor communicatively coupled to the at least one memory; Equipped with The at least one processor reads instructions from the at least one memory; receiving the encoded video signal comprising encoded picture data and a sequence parameter set (SPS) comprising an indication that a sample aspect ratio (SAR) is variable within the video sequence; receiving a supplemental enhancement information (SEI) message in the encoded video signal, the SEI message comprising a SAR value for a first group of one or more of the plurality of pictures of the video sequence; decoding the encoded picture data to obtain a plurality of decoded pictures; outputting the plurality of decoded pictures and sample aspect information for one or more of the plurality of decoded pictures; configured to perform operations comprising: the sample aspect information for the one or more of the plurality of decoded pictures is dependent on whether the indication indicates that the SAR value is constant for the video sequence or is variable within the video sequence. Video decoder.

2. The SPS includes information on whether the SPS has a default SAR value; 2. A video decoder according to claim 1.

3. The operations further comprise receiving a second SEI message in the encoded video signal; the second SEI message comprises a second SAR value for a second group of one or more of the plurality of pictures of the video sequence; the second group is different from the first group; 2. A video decoder according to claim 1.

4. the first group of one or more of the plurality of pictures of the video sequence comprising the one or more of the plurality of pictures of the video sequence that are encoded within coded picture data received after an SEI message comprising the SAR value is received and before a second SEI message comprising the second SAR value is received; the second group of the one or more of the plurality of pictures of the video sequence comprises the one or more of the plurality of pictures of the video sequence that are encoded within the coded picture data received after the second SEI message comprising the second SAR value is received.

4. A video decoder according to claim 3.

5. 1. A method for decoding an encoded video signal comprising encoded picture data to decode a plurality of pictures of a video sequence of a video, the method comprising: receiving the encoded video signal comprising encoded picture data and a sequence parameter set (SPS) comprising an indication that a sample aspect ratio (SAR) is variable within the video sequence; receiving a supplemental enhancement information (SEI) message in the encoded video signal, the SEI message comprising a SAR value for a first group of one or more of the plurality of pictures of the video sequence; decoding the encoded picture data to obtain a plurality of decoded pictures; outputting the plurality of decoded pictures and sample aspect information for one or more of the plurality of decoded pictures; Equipped with the sample aspect information for the one or more of the plurality of decoded pictures is dependent on whether the indication indicates that the SAR value is constant for the video sequence or is variable within the video sequence. method.

6. The SPS includes information on whether the SPS has a default SAR value; The method according to claim 5.

7. receiving a second SEI message in the encoded video signal; the second SEI message comprises a second SAR value for a second group of one or more of the plurality of pictures of the video sequence; the second group is different from the first group; The method according to claim 5.

8. the first group of one or more of the plurality of pictures of the video sequence comprising the one or more of the plurality of pictures of the video sequence that are encoded within coded picture data received after an SEI message comprising the SAR value is received and before a second SEI message comprising the second SAR value is received; the second group of the one or more of the plurality of pictures of the video sequence comprises the one or more of the plurality of pictures of the video sequence that are encoded within the coded picture data received after the second SEI message comprising the second SAR value is received. The method of claim 7.

9. 6. A method for executing a method for a computer-implemented computer-implemented program comprising: Non-transitory computer-readable medium.

10. 1. A video encoder for encoding a plurality of pictures of a video sequence of a video, the video encoder comprising: At least one memory; at least one processor communicatively coupled to the at least one memory; Equipped with The at least one processor reads instructions from the at least one memory; encoding the plurality of pictures of the video sequence into coded picture data; generating a sequence parameter set (SPS) comprising an indication that a sample aspect ratio (SAR) is allowed to vary within the video sequence; generating a supplemental enhancement information (SEI) message comprising a SAR value for a first group of one or more of the plurality of pictures of the video sequence; generating an encoded video signal such that the encoded video signal comprises the encoded picture data, the SPS, and the SEI message; outputting the encoded video signal; configured to perform operations comprising: Video encoder.

11. The operations further comprise generating the SPS such that the SPS comprises information as to whether the SPS comprises a default SAR value. The video encoder of claim 10.

12. The operations further comprise generating a second SEI message comprising second SAR values ​​for a second group of one or more of the plurality of pictures of the video sequence; the second group is different from the first group; The video encoder of claim 10.

13. The encoded video signal comprises: The SPS is before the SEI message, prior to encoding of the one or more first groups of the plurality of pictures of the video sequence, the SEI message the encoding of the one or more of the first group of pictures of the video sequence occurs before the second SEI message; the second SEI message is sent prior to encoding of the second group of one or more pictures of the video sequence; The arrangement is such that The video encoder of claim 12.

14. 1. A method for encoding a plurality of pictures of a video sequence of a video, the method comprising: encoding the plurality of pictures of the video sequence into coded picture data; generating a sequence parameter set (SPS) comprising an indication that a sample aspect ratio (SAR) is allowed to vary within the video sequence; generating a supplemental enhancement information (SEI) message comprising a SAR value for a first group of one or more of the plurality of pictures of the video sequence; generating an encoded video signal such that the encoded video signal comprises the encoded picture data, the SPS, and the SEI message; outputting the encoded video signal; A method comprising:

15. generating the SPS such that the SPS includes information on whether the SPS includes a default SAR value; The method of claim 14.

16. generating a second SEI message comprising a second SAR value for a second group of one or more of the plurality of pictures of the video sequence; the second group is different from the first group; The method of claim 14.

17. The encoded video signal comprises: The SPS is before the SEI message, prior to encoding of the one or more first groups of the plurality of pictures of the video sequence, the SEI message the encoding of the one or more of the first group of pictures of the video sequence occurs before the second SEI message; the second SEI message is sent prior to encoding of the second group of one or more pictures of the video sequence; The arrangement is such that 17. The method of claim 16.

18. 15. The method of claim 14, comprising: Non-transitory computer-readable medium.

Images