VIDEO DECODER, METHOD FOR DECODING, VIDEO ENCODER, METHOD FOR ENCODING, AND NON-TRANSITORY COMPUTER-READABLE MEDIUM - Patent application

Abstract

To provide a decoder and a decoding method for decoding a plurality of pictures having different SAR values.SOLUTION: A generic video decoder 151 executes operations including the steps of: receiving an encoded video signal having a sequence parameter set (SPS) with an indication that encoded picture data and sample aspect ratio (SAR) can be modified in a video sequence; receiving a supplemental enhancement information (SEI) message within the encoded video signal having a SAR value for one or more first groups of multiple pictures of the video sequence; decoding the encoded picture data for obtaining a plurality of decoded pictures; and outputting the plurality of decoded pictures and sample aspect information for one or more of the decoded pictures.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a video decoder, a method for decoding, a video encoder, a method for encoding, and a non-transitory computer-readable medium.

[0002] H.265 / HEVC (HEVC = High Efficiency Video Coding) is a video codec that already provides tools to improve or enable parallel processing in the encoder and / or decoder. For example, HEVC supports sub-division of a picture into an array of tiles that are coded independently of each other. Another concept supported by HEVC relates to WPP, according to which CTU rows or CTU lines of a picture may be processed in parallel from left to right, for example in stripes (CTU = coding tree unit), provided that some minimum CTU offset is observed in the processing of consecutive CTU lines. However, it would be preferable to have a video codec nearby that more efficiently supports parallel processing capabilities of a video encoder and / or video decoder.

[0003] In the following, we will explain the state-of-the-art VCL partitioning (VCL = video coding layer).

[0004] Typically, in video coding, the coding process of picture samples requires smaller partitions, and the samples are divided into several rectangular regions for joint processing such as predictive coding or transform coding. Therefore, a picture is divided into blocks of a certain size that remains constant during the coding of a video sequence. In H.264 / AVC, a standard fixed-size block of 16x16 samples, the so-called macroblock, is used (AVC = Advanced Video Coding).

[0005] In the latest HEVC standard (see ISO / IEC, ITU-T. High efficiency video coding. ITU-T Recommendation H.265 | ISO / IEC 23008 10 (HEVC), edition 1, 2013; edition 2, 2014.), there are coding tree blocks (CTBs) or coding tree units (CTUs) with a maximum size of 64x64 samples. In further descriptions of HEVC, the more general term CTU will be used for such kind of blocks.

[0006] The CTUs are processed in raster scan order starting with the top left CTU and processing the CTUs down by picture lines to the bottom right CTU.

[0007] The coded CTU data is organized into a kind of container called a slice. Originally, in previous video coding standards, a slice meant a division of a picture comprising one or more consecutive CTUs. A slice is used for segmentation of coded data. From another perspective, a complete picture can also be defined as one large segment, and therefore, historically, the term slice still applies. In addition to the coded picture samples, a slice also comprises additional information related to the coding process of the slice itself, which is placed in a so-called slice header.

[0008] According to the state of the art, the VCL (video coding layer) also comprises techniques for fragmentation and spatial partitioning. Such partitioning can be applied to video coding for various reasons, such as processing load balancing in parallelization, CTU size matching in network transmission, error mitigation, etc.

[0009] Other examples relate to RoI (RoI = Region of Interest) coding, where there is, for example, a region in the center of a picture that the viewer can select with a zoom-in operation (RoI-only decoding), or gradual decoder refresh (GDR), where intra data (typically placed in one frame of a video sequence) is temporally distributed over several consecutive frames, for example as a sequence of intra blocks that swipe across the picture plane and locally reset the temporal prediction chain in the same way that an intra picture does for the entire picture plane. In the latter case, two regions exist in each picture: one that has recently been reset and one that is potentially affected by errors and error propagation.

[0010] Reference Picture Resampling (RPR) is a technique used in video coding to adapt the quality / rate of video by using coarse quantization parameters as well as potentially adapting the resolution of each transmitted picture. Thus, the reference used for inter-prediction may have a different size than the picture currently being predicted for encoding. Essentially, RPR requires a resampling process in the prediction loop, e.g., upsampling and downsampling filters to be defined.

[0011] Depending on the flavor, RPR can result in a change in the coded picture size at any picture, or it can be restricted to occur only at specific positions bounded by segment boundaries, such as adaptive HTTP streaming, and only at specific pictures. Summary of the Invention

[0012] It is an object of the present invention to provide an improved concept for video encoding and video decoding.

[0013] The object of the present invention is solved by the subject matter of the independent claims.

[0014] Preferred embodiments are provided in the dependent claims. [Brief explanation of the drawings]

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0016] [Figure 1] 1 illustrates a video encoder according to one embodiment. [Figure 2] 1 illustrates a video decoder according to one embodiment. [Figure 3] 1 illustrates a system according to one embodiment. [Figure 4] Region of Interest (RoI) scalability versus expansion scalability. [Figure 5a] A first example of a content fragment (gray) moving within a picture over time is shown. [Figure 5b] A second example of a content fragment (gray) moving within the picture over time is shown. [Figure 6a] Indicates the current picture with mixed reference pictures. [Figure 6b] An example of ignoring the cropping window of a reference picture is shown below. [Figure 7] 1 shows a video encoder. [Figure 8] 1 shows a video decoder. [Figure 9] It shows the relationship between a reconstructed signal, for example a reconstructed picture, on the one hand, and a combination of a prediction residual signal and a prediction signal as signaled in the data stream, on the other hand. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following description of the figures begins with a description of an encoder and decoder of a block-based predictive codec for encoding pictures of video to form an example of an encoding framework into which embodiments of the present invention can be incorporated. Respective encoders and decoders are described with reference to Figures 7 through 9. A description of embodiments of the inventive concepts is then presented, along with a discussion of how such concepts may be incorporated into the encoders and decoders of Figures 7 and 8, respectively. However, the embodiments of Figures 1 through 3 and described below may also be used to form encoders and decoders that do not operate according to the encoding framework underlying the encoders and decoders of Figures 7 and 8.

[0018] FIG. 7 illustrates a video encoder, typically an apparatus for predictively encoding a picture 12 into a data stream 14 using transform-based residual coding. The apparatus, i.e., the encoder, is indicated with the reference numeral 10. FIG. 8 illustrates a corresponding video decoder 20, e.g., an apparatus 20 configured to predictively decode a picture 12′ from the data stream 14 also using transform-based residual decoding, where an apostrophe is used to indicate that the picture 12′ reconstructed by the decoder 20 deviates from the picture 12 originally encoded by the apparatus 10 in terms of coding loss introduced by quantization of the prediction residual signal. While FIGS. 7 and 8 illustratively use transform-based predictive residual coding, embodiments of the present application are not limited to this type of predictive residual coding. This also applies to other details described with respect to FIGS. 7 and 8, as outlined below.

[0019] The encoder 10 is configured to subject the prediction residual signal to a spatial-spectral transformation and to encode the prediction residual signal thus obtained into a data stream 14. Similarly, the decoder 20 is configured to decode the prediction residual signal from the data stream 14 and to subject the prediction residual signal thus obtained to a spectral-spatial transformation.

[0020] Internally, the encoder 10 may include a prediction residual signal former 22 that generates a prediction residual 24, allowing the deviation of a prediction signal 26 from, for example, picture 12, from the original signal to be measured. The prediction residual signal former 22 may be a subtractor that subtracts a prediction signal from, for example, picture 12, from the original signal. The encoder 10 then includes a transformer 28 that applies a spatial-spectral transformation to the prediction residual signal 24 to obtain a spectral-domain prediction residual signal 24', which is then quantized by a quantizer 32 also implemented by the encoder 10. While the quantized prediction residual signal 24 is thus coded into the bitstream 14, the encoder 10 may optionally include an entropy coder 34 that transforms and quantizes the prediction residual signal into a data stream by entropy coding, and a prediction signal 26 is generated based on the prediction residual signal 24 by a prediction stage 36 of the encoder 10, which is coded into the data stream 14 or can be decoded from the data stream 14. For this purpose, as shown in FIG. 7 , the prediction stage 36 may internally comprise an inverse quantizer 38 that inversely quantizes the prediction residual signal 24″ to obtain a spectral-domain prediction residual signal 24′″ that corresponds to the signal 24′ excluding quantization losses. An inverse transformer 40 then inversely transforms the latter prediction residual signal 24′″, e.g., subjects it to a spectral-to-spatial transformation, to obtain a prediction residual signal 24′″ that corresponds to the original prediction residual signal 24 excluding quantization losses. A combiner 42 of the prediction stage 36 can then recombine the prediction signal 26 and the prediction residual signal 24′″, for example by addition, to obtain a reconstructed signal 46, a reconstruction of the original signal, e.g., picture 12. The reconstructed signal 46 may correspond to signal 12′. A prediction module 44 of the prediction stage 36 then generates the prediction signal 26 based on the signal 46, for example by using spatial prediction, i.e., intra-picture prediction, and / or temporal prediction, i.e., inter-picture prediction.

[0021] Similarly, decoder 20 may be internally composed of components corresponding to, and interconnected in a corresponding manner with, prediction stage 36, as shown in Figure 8. In particular, entropy decoder 50 of decoder 20 may entropy decode a quantized spectral domain prediction residual signal 24" from the data stream, whereby inverse quantizer 52, inverse transformer 54, combiner 56 and prediction module 58 are interconnected and cooperate in the manner described above with respect to the modules of prediction stage 36 to recover a reconstructed signal based on prediction residual signal 24" such that the output of combiner 56 yields the reconstructed signal, i.e., picture 12', as shown in Figure 8.

[0022] Although not specifically described above, it is readily apparent that the encoder 10 may set certain coding parameters, including, for example, prediction modes, motion parameters, etc., according to certain rate- and distortion-related criteria, e.g., a method for optimizing coding cost, e.g., a certain optimization scheme. For example, the encoder 10 and the decoder 20 and corresponding modules 44, 58 may support different prediction modes, such as intra-coding and inter-coding modes, respectively. The granularity with which the encoder and decoder switch between these prediction mode types may correspond to the subdivision of the pictures 12 and 12′, respectively, into coding segments or coding blocks. At these coding segment levels, for example, the picture may be subdivided into intra-coded and inter-coded blocks. The intra-coded blocks are predicted based on the respective blocks' spatial, already coded / decoded neighborhoods, as outlined in more detail below. Several intra-coding modes may exist, and each intra-coding mode may be selected for each intra-coding segment, including a directional or angular intra-coding mode in which the respective segment is filled by extrapolating neighboring sample values ​​along a direction specific to the respective directional intra-coding mode. The intra-coding modes may also include one or more additional modes, such as a DC coding mode in which prediction for each intra-coded block assigns a DC value to all samples in the respective intra-coded segment, and / or a planar intra-coding mode in which prediction for each block is approximated or determined based on neighboring samples as a spatial distribution of sample values ​​described by a two-dimensional linear function over the sample positions of the respective intra-coded block with a planar driving slope and offset defined by the two-dimensional linear function. In comparison, inter-coded blocks may be predicted temporally, for example.For inter-coded blocks, motion vectors may be signaled within the data stream, indicating the spatial displacement of portions of previously coded pictures of the video to which picture 12 belongs, and the previously coded / decoded pictures are sampled to obtain a prediction signal for each inter-coded block. This means that in addition to the residual signal coding provided by data stream 14, such as entropy-coded transform coefficient levels representing the quantized spectral-domain prediction residual signal 24″, data stream 14 may also encode optional further parameters therein, such as coding mode parameters for assigning coding modes to various blocks, prediction parameters for some blocks, such as motion parameters for inter-coded segments, and parameters for controlling and signaling the subdivision of pictures 12 and 12′ into segments, respectively. Decoder 20 uses these parameters to subdivide the pictures in a similar manner as the encoder did, assign the same prediction modes to the segments, and perform the same predictions, resulting in the same prediction signals.

[0023] 9 illustrates the relationship between, on the one hand, a reconstructed signal, e.g., a reconstructed picture 12′, and, on the other hand, a combination of a prediction residual signal 24″″ and a prediction signal 26 signaled in the data stream 14. As already mentioned above, the combination may be additive. The prediction signal 26 is illustrated in FIG. 9 as a subdivision of the picture region into intra-coded blocks, exemplarily shown with hatching, and inter-coded blocks, exemplarily shown without hatching. The subdivision may be any subdivision, such as a regular subdivision of the picture region into rows and columns of square or non-square blocks, or may be a multi-tree subdivision that subdivides the picture 12 from a tree root block into multiple leaf blocks of various sizes, such as a quadtree subdivision, where the picture region is first subdivided into rows and columns of tree root blocks and then further subdivided into one or more leaf blocks according to a recursive multi-tree subdivision, a mixture of which is illustrated in FIG. 9.

[0024] Data stream 14 may also have intra-coding modes coded therein for intra-coded blocks 80, which assign each intra-coded block 80 one of several supported intra-coding modes. For inter-coded blocks 82, data stream 14 may have one or more motion parameters coded therein. Generally speaking, inter-coded blocks 82 are not limited to being temporally coded. Alternatively, inter-coded blocks 82 may be any blocks predicted from previously coded portions beyond current picture 12 itself, such as a previously coded picture of the video to which picture 12 belongs, or another view or hierarchically lower picture if the encoder and decoder are scalable encoder and decoder, respectively.

[0025] 9 also illustrates a subdivision of the picture region into blocks 84. These blocks are sometimes referred to as transform blocks to distinguish them from the coding blocks 80 and 82. In effect, FIG. 9 illustrates that the encoder 10 and decoder 20 can use two different subdivisions of the picture 12 and the picture 12′, respectively: one subdivision into coding blocks 80 and 82, respectively, and another subdivision into transform blocks 84. While both subdivisions may be the same, e.g., each coding block 80 and 82 may simultaneously form a transform block 84, FIG. 9 illustrates the case where the subdivision into transform blocks 84 forms an extension of the subdivision into coding blocks 80, 82, such that any boundary between the two blocks 80 and 82 overlaps the boundary between the two blocks 84, or, alternatively, such that each block 80, 82 either coincides with one of the transform blocks 84 or with a cluster of transform blocks 84. However, the subdivisions may be determined or selected independently of one another, such that the transform blocks 84 can cross the block boundaries between the blocks 80, 82. As far as the subdivision into transform blocks 84 is concerned, it is similar to that presented with respect to the subdivision into blocks 80, 82; for example, the blocks 84 may be the result of a regular subdivision of the picture region into blocks (with or without arrangement into rows and columns), a recursive multi-tree subdivision of the picture region, or a combination thereof, or any other type of blocking. As an aside, it should be noted that the blocks 80, 82, and 84 are not limited to being square, rectangular, or any other shape.

[0026] 9 further illustrates that the combination of the prediction signal 26 and the prediction residual signal 24'''' directly results in the reconstructed signal 12'. However, it should be noted that, according to alternative embodiments, more than one prediction signal 26 may be combined with the prediction residual signal 24'''' to produce the picture 12'.

[0027] In Figure 9, the transform blocks 84 have the following significance: The transformer 28 and the inverse transformer 54 perform their transforms in units of these transform blocks 84. For example, many codecs use some kind of DST or DCT for all transform blocks 84. Some codecs allow for skipping the transform for some of the transform blocks 84, so that the prediction residual signal is directly coded in the spatial domain. However, according to the embodiments described below, the encoder 10 and the decoder 20 are configured to support several transforms. For example, the transforms supported by the encoder 10 and the decoder 20 may include: DCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform DST-IV, where DST stands for Discrete Sine Transform DCT-IV DST-VII Identity Transformation (IT)

[0028] Of course, the transformer 28 supports all of the forward transform versions of these transforms, while the decoder 20 or inverse transformer 54 supports the corresponding backward or inverse versions: Inverse DCT-II (or Inverse DCT-III) Reverse DST-IV ·Inverse DCT-IV Reverse DST-VII Identity Transformation (IT)

[0029] The following description provides further details on which transforms may be supported by the encoder 10 and decoder 20. Note that in any case, the set of supported transforms may comprise only one transform, such as one spectral-to-spatial or spatial-to-spectral transform.

[0030] As already outlined above, Figures 7 to 9 are presented as examples in which the inventive concepts further described below can be implemented to form specific examples of encoders and decoders according to the present application. The encoders and decoders of Figures 7 and 8 may represent possible implementations of the encoders and decoders described below, respectively. However, Figures 7 and 8 are merely examples. However, an encoder according to an embodiment of the present application, as outlined in more detail below, may perform block-based coding of picture 12 differently from the encoder of Figure 7 in that it is a still image encoder rather than a video encoder, does not support inter-prediction, or the subdivision into blocks 80 is performed differently from the way illustrated in Figure 9. Similarly, a decoder according to an embodiment of the present invention may perform block-based decoding of pictures 12′ from data stream 14 using the encoding concepts further outlined below, but may differ from decoder 20 of FIG. 8 in that, for example, it is a still picture decoder rather than a video decoder, it does not support intra prediction, or it subdivides pictures 12′ into blocks in a different way than described with respect to FIG. 9, and / or it does not derive prediction residuals from data stream 14 in, for example, the transform domain.

[0031] In the following, a generic video encoder according to an embodiment is described in FIG. 1, a generic video decoder according to an embodiment is described in FIG. 2, and a generic system according to an embodiment is described in FIG.

[0032] FIG. 1 shows a general video encoder 101 according to an embodiment.

[0033] The video encoder 101 is configured to encode a plurality of pictures of a video by generating an encoded video signal, each of the plurality of pictures comprising original picture data.

[0034] The video encoder 101 comprises a data encoder 110 configured to generate an encoded video signal comprising encoded picture data, the data encoder being configured to encode a plurality of pictures of a video into the encoded picture data.

[0035] Furthermore, the video encoder 101 comprises an output interface 120 configured to output the coded picture data for each of the plurality of pictures.

[0036] FIG. 2 shows a generic video decoder 151 according to an embodiment.

[0037] The video decoder 151 is configured to decode the coded video signal comprising coded picture data to reconstruct a plurality of pictures of the video.

[0038] The video decoder 151 comprises an input interface 160 configured to receive an encoded video signal.

[0039] Additionally, the video decoder comprises a data decoder 170 configured to reconstruct multiple pictures of the video by decoding the coded picture data.

[0040] FIG. 3 illustrates a general system according to an embodiment.

[0041] The system includes a video encoder 101 of FIG. 1 and a video decoder 151 of FIG.

[0042] The video encoder 101 is configured to generate an encoded video signal.

[0043] The video decoder 151 is configured to decode the encoded video signal to reconstruct the video pictures.

[0044] A first aspect of the present invention is set out in claims 1 to 33. The first aspect provides a sample aspect ratio signal.

[0045] A second aspect of the present invention is set forth in claims 34 to 72. The second aspect provides Reference Picture Resampling restrictions to reduce the implementation burden.

[0046] A third aspect of the present invention is set out in claims 73 to 131. The third aspect provides flexible region-based referencing for zooming reference picture resampling, and in particular provides more efficient address zoom use cases.

[0047] The first aspect of the present invention will be described in detail below.

[0048] In particular, the first aspect provides sample aspect ratio signaling.

[0049] Sample aspect ratio (SAR) is relevant to correctly presenting encoded video to the consumer so that when the aspect ratio of the encoded sample array changes over time via RPR (e.g., by subsampling in one dimension), the aspect ratio of the presented picture can remain constant as intended.

[0050] The latest SAR in the Video Usability Information (VUI) in a Sequence Parameter Set (SPS) like HEVC or AVC only allows setting a constant SAR for the entire signaled video sequence, e.g., SAR changes are only allowed at the start of a coded video sequence (e.g., the sample aspect ratio is constant for each coded video sequence).

[0051] Therefore, as part of this invention, a new mode of SAR signaling is introduced into video coding: The sequence level parameter set, or SPS, contains the following indications: RPR is used (which may result in a change in the coded picture size) Actual SAR is not given in VUI, Alternatively, the SAR of a coded video may be designated as dynamic and may change within a CVS (coded video sequence). The actual SAR of a coded picture is indicated via the SEI (supplemental enhancement information) message at the resolution switch point. [Table 1]

[0052] Dynamic SAR Information SEI Message [Table 2]

[0053] Similarly, the vui_aspect_ratio_constant_flag may be employed, for example.

[0054] The vui_aspect_ratio_constant_flag may be, for example, an indicator of whether the aspect ratio of the sample is constant for a video sequence or whether the aspect ratio of the sample can change within a video sequence.

[0055] For example, if vui_aspect_ratio_constant_flag may be set to, for example, 0 (or may be set to, for example, FALSE, or may be set to, for example, -1), this may indicate, for example, that dynamic SAR information is present, for example, in the SEI message.

[0056] In an alternative embodiment, the SAR information in the VUI (e.g., SPS) is used as a default, and this is used unless an SEI message is available. The information in the SEI message is overwritten with the information in the SPS. [Table 3]

[0057] In another embodiment, SAR information is associated with a picture resolution and signaled in a PPS (picture parameter set), where the picture resolution is signaled. A default SAR is signaled in the SPS, and if the SAR changes for a particular picture resolution, a different SAR is signaled, overriding the default SAR.

[0058] SPS VUI: [Table 4]

[0059] Also, for the SEI case, the SPS can further indicate that the SAR may change and that it will be updated to the PPS (similar to the previous aspect_ratio_dynamic_SEI_present_flag). Thus, in some applications, it may be possible to constrain or limit the SAR to not change, which can facilitate enforcement or RPR / ARC.

[0060] PPS: [Table 5]

[0061] If pps_aspect_ratio_info_present_flag is set to 0, the default SAR is obtained from the SPS, otherwise the actual SAR is provided.

[0062] The second aspect of the present invention will now be described in detail.

[0063] In particular, the second aspect provides signaling regarding constraints for reference picture resampling.

[0064] The implementation burden can be reduced by restricting the RPR scheme in various ways. A general RPR scheme that does not include additional restrictions, such as the invention below, would require implementers to over-provision their decoder hardware to do the following: Any current picture, worst case: resampling on all pictures DPB (decoded picture buffer), intermediate GOP (group of pictures) pair, resampling of any picture at a defined position with fewer reference pictures Simultaneous resampling of multiple images with different resolutions to a target resolution Cascaded resampling chain of ref pic with loss of picture quality

[0065] The following invented restrictions make it possible to reduce the implementation costs of a codec featuring such a restricted RPR scheme compared to an unrestricted RPR codec.

[0066] In one embodiment, resolution changes are only allowed at RAPs (random access points), e.g., the maximum number of resampled pictures is the amount of RASL (random access decodable skipped picture) pictures at this RAP, which are typically at a distance of one or more GOPs, e.g., tens of pictures, apart, which reduces the worst-case rate at which such resampling operations must be supported.

[0067] In another embodiment, the resolution change is performed on key pictures within a hierarchical GOP, e.g., pictures: Belongs to the lowest time layer, and occurs once in every GOP, and All subsequent pictures in coding order have a lower POC (e.g., an earlier presentation timestamp), When a reference picture is resampled, none of the immediately following pictures in the GOP of a higher temporal layer require cascade up / downsampling.

[0068] According to another embodiment, a resolution change is only allowed in the picture that immediately follows the key picture in presentation order, ie the first picture of the next GOP in presentation order.

[0069] In another embodiment, the time distance between successive resolution changes is limited by a minimum POC (picture order count) distance in the level definition.

[0070] In another embodiment, the temporal distance between successive resolution changes is limited by a minimum number of intermediate coded pictures in the level definition.

[0071] In another embodiment, resolution changes can only occur for pictures marked as non-discardable or reference pictures with non_reference_picture_flag equal to 0.

[0072] In another embodiment, the speed of the resolution change is limited by the level definition.

[0073] In another embodiment, the resampling of reference pictures relative to the current picture is restricted to using a single resampling ratio, e.g., all reference pictures of the current picture that have a different resolution than the current picture are required to have the same resolution.

[0074] In another embodiment, when one reference picture of the current picture requires resampling, all reference pictures of the current picture must use resampling, e.g., be of the same original resolution as the one reference picture.

[0075] In another embodiment, only one reference picture of the current picture is allowed to require resampling.

[0076] According to another embodiment, the maximum number of pictures requiring resampling at a resolution change point is optionally indicated in the coded video sequence / bitstream as a guarantee to the decoder, and if no indication is present, it is inferred or indicated by the level definition.

[0077] In another embodiment, the original (unresampled) reference picture is removed from the reference picture list and / or decoded picture buffer after being resampled, e.g., marked as not being used for reference, so that only the resampled reference picture is available from there.

[0078] In another embodiment, the resampling ratios used within a coded video sequence are restricted to the set of resampling ratios contained in a parameter set with sequence or bitstream scope (decoding parameter set, DPS; sequence parameter set, SPS).

[0079] The third aspect of the present invention will now be described in detail.

[0080] In particular, the second aspect provides flexible region-based referencing for zooming for reference picture resampling.

[0081] As mentioned above, layered codecs such as SHVC and SVC accommodate two enhancement scalability modes: RoI scalability (where the area of ​​the lower layer picture is expanded in the upper layer) and enhancement scalability (where the lower layer picture is enhanced via additional content in the upper layer) as shown in Figure 4 below.

[0082] Enhanced scalability may refer, for example, to so-called zoom-out use cases, e.g., use cases that change over time in the sense that the video completely covers more content, e.g., a larger capture angle, a larger part of the scene, a larger area, etc.

[0083] Figure 4 shows the region of interest (RoI) scalability versus the extended scalability.

[0084] In scenarios where zooming in and out is allowed, a zoom and movement region that is used for prediction and predicted is defined. This is known as RoI scalability (typically zooming in) or extension scalability (typically zooming out). In RoI scalability with scalable coding, a region that is upscaled to the dimension of the reference picture is usually defined in the reference picture. However, in scalable coding, the upper layer picture and the lower layer picture on which prediction is performed represent the same time.

[0085] In the case of SHVC and SVC, this is done for layered coding, and in these cases the juxtaposed base layer does not represent any motion, e.g., the corresponding samples in the base layer are known, so it was possible to completely upscale a known region in the base layer and operate on that upscaled basis.

[0086] However, in RPR applications, the two pictures between which prediction is performed do not depict the same time instance, and therefore some content outside the defined region may move from time instance A (low resolution) to the zoomed-in / out region at time instance B (high resolution). Forbidding reference to these regions for prediction is detrimental to coding efficiency.

[0087] However, in the case of RPR, the reference can point to some area outside the corresponding reference region, for example, by zooming in on an area where an object moving into the RoI is located. This is shown in Figure 5a without actually changing the coding resolution:

[0088] FIG. 5a depicts a first example of a content fragment (gray) moving within a picture over time.

[0089] In a first embodiment, a reference region is defined that includes an area larger than the area of ​​the RoI, such that the grey box in the figure that falls within the RoI zoom region is within the reference:

[0090] FIG. 5b depicts a second example of a content fragment (gray) moving within the picture over time.

[0091] This leads to reconstructing a region slightly larger than the RoI for the picture corresponding to the RoI, and the additional region is removed by indicating a cropping window. The problem arises from the fact that the scaling factors used to upsample the reference are calculated in VVC (Versative Video Coding) from the cropped-out picture. First, assuming there is no RoI, the horizontal scale factor HorScale and the vertical scale factor VerScale are calculated as follows: HorScale = CroppedOutWidthPic / CroppedOutWidthRefPic VerScale = CroppedOutHeightPic / CroppedOutHeightRefPic

[0092] The reason for showing the ratio based on the cropped picture is that the codec requires a multiple of the minimum size (in VVC 8 samples), so depending on the picture size of interest, some additional samples need to be decoded. Therefore, if either Pic or RefPic is not a multiple of 8, some samples will be added to the input picture, making them a multiple of 8, resulting in different ratios and incorrect scaling factors. This problem can be even worse if the bitstream is desired to be coded as "mergeable," i.e., if the bitstream can be merged with other bitstreams, such as when the picture size needs to be a multiple of the CTU size up to 128. Therefore, the correct scaling factor needs to take the cropping window into account.

[0093] In the described scenario (combining RPR with RoI), where a cropping window is used to include some additional references, the use of a cropping window is insufficient.

[0094] As mentioned above, the RoI in the reference picture can be defined a little larger, and the RoI is used for reference but discarded along with the cropping window in the current reconstructed picture. However, if the horizontal scale factor HorScale and the vertical scale factor VerScale are calculated as follows: HorScale = CroppedOutWidthPic / WidthEnlargedRefRoI VerScale = CroppedOutHeightPic / HeightEnlargeRefRoI Some of the samples within the expanded RoI actually correspond to samples within the cropped-out region, resulting in incorrect results.

[0095] In the following, a cropping window based concept according to a first group of embodiments is described.

[0096] Thus, in the first group of embodiments above, the calculation may be, for example, as follows: HorScale = CodedPicWidth / RefRoIWidth VerScale = CodedPicHeight / RefRoIHeight Such calculations include samples being clipped for the calculation of the scale factor.

[0097] Regarding signaling, in one embodiment, the signaling of the extended RoI indicates that the cropping window information should be ignored in the scaling factor calculation.

[0098] In another embodiment, whether the cropping window needs to be taken into account in the calculation of the scaling factor is indicated in the bitstream (eg, in the parameter set or slice header). [Table 6]

[0099] The cropping window may also be referred to as, for example, an adaptive cropping window. The offsets for the cropping window / adaptive cropping window may also be referred to as, for example, pps_conf_win_left_offset, pps_conf_win_top_offset, pps_conf_win_right_offset, and pps_conf_win_botton_offset.

[0100] Instead of using use_cropping_for_scale_factor_derivation_flag to determine whether to ignore information in the coded video signal on the cropping window for upscaling a region in a reference picture (or to determine whether information in the coded video signal on the cropping window is used for upscaling a region in a reference picture), for example, pps_scaling_window_explicit_signalling_flag may be used.

[0101] For example, if pps_scaling_window_explicit_signalling_flag is set to 0 (or, e.g., set to FALSE, or, e.g., set to −1), information in the coded video signal on the cropping window may be used, e.g., to upscale regions in reference pictures. And, for example, if pps_scaling_window_explicit_signalling_flag is set to 1 (or, e.g., set to TRUE), information in the coded video signal on the cropping window may be ignored, e.g., to upscale regions in reference pictures.

[0102] One drawback of the above approach is that a larger area is decoded for the current picture to allow for referencing samples outside the RoI, e.g., samples over an expanded RoI. More specifically, samples are decoded in the area outside the RoI and later discarded by the cropping window. This leads to additional sample overhead and reduced coding efficiency, which can potentially counteract the coding efficiency gain of allowing for referencing outside the corresponding RoI in the reference picture.

[0103] A more efficient approach is to decode only the RoI (omitting the additional samples required to create the 8x picture or CTU, as described above), but allow for reference to samples within the expanded RoI.

[0104] In the following, a bounding box based concept according to a second group of embodiments is described.

[0105] In the second group of embodiments above, samples that are outside the red rectangle but inside the green box (RoI offset plus an additional RoI offset) are used to determine the resampled ref pic instead of using only the red RoI.

[0106] The size of the bounding box for the MV around the red cutout is defined / signaled with the benefit of limiting memory access / line buffer requirements and also allowing implementation with a pic-wise upsampling approach.

[0107] Such signaling can be included in the PPS(additional_roi_X): [Table 7]

[0108] Therefore, the derivation of the scaling factors is as follows: HorScale = CroppedOutWidthPic / RefRoIWidth VerScale = CroppedOutHeightPic / RefRoIHeight

[0109] In one embodiment, the reference sample is identified by finding the aligned sample using the RoI_X_offset and applying MV that is clipped if the reference sample is outside the extended RoI indicated by additional_RoI_x. Alternatively, samples outside this extended RoI are padded with the last sample within the extended RoI.

[0110] In another embodiment, this extended RoI is only used as a limit or constraint that can be used for implementation optimization. For example, if the reference picture is first fully upsampled as needed, rather than on-the-fly (block-based), then only the extended RoI is resampled instead of the entire picture, saving a lot of processing.

[0111] A further problem occurs when two or more reference pictures are used simultaneously. In this case, it is necessary to identify the picture to which the RoI region information applies. In such cases, instead of adding information to the PPS, the slice header indicates that some of the entries in the reference list refer to a part of the picture, rather than the whole picture. For example, [Table 8]

[0112] In further embodiments, additional constraints are in place: Only reference pictures with lower POC can have RoI information. Typically, RoI switching, with the described features, is applied in open GOP switching scenarios, and therefore POCs with higher POC already represent RoI scenes. Only one reference picture can have RoI information.

[0113] In another embodiment, RoIInfo() is carried in the picture parameter set, and the slice header carries only a flag (RoI_flag) for each reference picture, indicating whether the RoI information should be applied for resampling (deriving the scaling factor). The following figure shows the principle for four coded pictures, two before and two after the switching point. At the switching point, the total resolution remains constant, but upsampling of the RoI is performed. Two PPSs are defined, and the PPSs of the latter two pictures indicate the RoI in the reference picture. Furthermore, the slice headers of the latter two pictures carry RoI_flag[i] for each of their reference pictures, and the value is denoted in the figure as "RoI_flag" or "RF = x".

[0114] Additionally, the slice header may carry not only the RoI_flag as described above, but also, if the flag is true, an additional index for each reference picture into the array of RoIInfo() carried in the parameter set to identify the RoI information that applies to that particular reference picture.

[0115] FIG. 6a shows a current picture with mixed reference pictures.

[0116] The following describes the case of zooming out according to the third group of embodiments.

[0117] Instead of RoI scalability, the above third group of embodiments can consider extended scalability, for example, scalability going from the RoI picture to a larger area. In such a case, the cropping window of the reference picture should also be ignored, especially if an area of ​​the currently decoded picture is identified as an area for scalability such as zooming.

[0118] FIG. 6b shows an example of ignoring the cropping window of the reference picture for an identified region in the current picture.

[0119] HorScale = IdentifiedRegionInPicWidth / CodedRefPicWidth VerScale = IdentifiedRegionInPicHeight / CodedRefPicHeight

[0120] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, in which a block or device corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or electronic circuitry. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0121] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware, or in software, or at least partially in hardware, or at least partially in software. This implementation can be carried out using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that the respective methods are performed. Thus, the digital storage medium may be computer-readable.

[0122] Some embodiments according to the invention comprise a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0123] Generally, embodiments of the present invention may be implemented as a computer program product having program code operable to perform one of the methods when the computer program product is run on a computer, which may for example be stored on a machine-readable carrier.

[0124] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0125] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0126] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) comprising, recorded thereon, a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0127] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals may for example be adapted to be transmitted via a data communication connection, for example via the Internet.

[0128] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0129] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0130] Further embodiments according to the invention comprise an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may for example be a computer, a mobile device, a memory device, etc. The apparatus or system may for example include a file server for transferring the computer program to the receiver.

[0131] In some embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functions of the methods described herein.

[0132] In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0133] The devices described herein may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0134] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0135] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is therefore intended to be limited only by the scope of the impending claims, and not by the specific details presented by the description and illustration of the embodiments herein.

Claims

1. 1. A video decoder for decoding an encoded video signal comprising encoded picture data to decode a plurality of pictures of a video sequence of video, the video decoder comprising: at least one memory; at least one processor communicatively coupled to the at least one memory; Equipped with The at least one processor reads instructions from the at least one memory and receiving a coded video signal comprising coded picture data and a sequence parameter set (SPS) comprising an indication that a sample aspect ratio (SAR) can be changed within a coded video sequence (CVS); receiving a first supplemental enhancement information (SEI) message in the coded video signal, the first SEI message comprising a first SAR value for a first group of one or more pictures of the plurality of pictures in the CVS; receiving a second SEI message in the coded video signal, the second SEI message comprising a second SAR value for a second group of one or more pictures of the plurality of pictures in the CVS, the second SAR value being different from the first SAR value and the second group being different from the first group; decoding the coded picture data corresponding to the first and second group of one or more pictures of the plurality of pictures to obtain a plurality of decoded pictures; outputting the plurality of decoded pictures, wherein one or more pictures of the plurality of decoded pictures have a different SAR value than one or more other pictures of the plurality of decoded pictures; configured to perform operations comprising: Video decoder.

2. The SPS includes information on whether the SPS has a default SAR value; 2. The video decoder of claim 1.

3. the first group of one or more pictures of the plurality of pictures in the CVS comprises the one or more pictures of the plurality of pictures in the CVS that are encoded within the coded picture data received after the first SEI message with the first SAR value is received and before the second SEI message with the second SAR value is received; the second group of one or more pictures of the plurality of pictures in the CVS comprises the one or more pictures of the plurality of pictures in the CVS that are encoded in the coded picture data received after the second SEI message comprising the second SAR value is received.

2. The video decoder of claim 1.

4. 1. A method of decoding an encoded video signal comprising coded picture data to decode a plurality of pictures of a video sequence of video, the method comprising: receiving a coded video signal comprising coded picture data and a sequence parameter set (SPS) comprising an indication that a sample aspect ratio (SAR) can be changed within a coded video sequence (CVS); receiving a first supplemental enhancement information (SEI) message in the coded video signal, the first SEI message comprising a first SAR value for a first group of one or more pictures of the plurality of pictures in the CVS; receiving a second SEI message in the coded video signal, the second SEI message comprising a second SAR value for a second group of one or more pictures of the plurality of pictures in the CVS, the second SAR value being different from the first SAR value and the second group being different from the first group; decoding the coded picture data corresponding to the first and second group of one or more pictures of the plurality of pictures to obtain a plurality of decoded pictures; outputting the plurality of decoded pictures, wherein one or more pictures of the plurality of decoded pictures have a different SAR value than one or more other pictures of the plurality of decoded pictures; A method comprising:

5. The SPS includes information on whether the SPS has a default SAR value; The method of claim 4.

6. the first group of one or more pictures of the plurality of pictures in the CVS comprises the one or more pictures of the plurality of pictures in the CVS that are encoded within the coded picture data received after the first SEI message with the first SAR value is received and before the second SEI message with the second SAR value is received; the second group of one or more pictures of the plurality of pictures in the CVS comprises the one or more pictures of the plurality of pictures in the CVS that are encoded in the coded picture data received after the second SEI message comprising the second SAR value is received. The method of claim 4.

7. comprising instructions that, when executed, perform the method of claim 4, Non-transitory computer-readable medium.

8. 1. A video encoder for encoding a plurality of pictures of a video sequence of a video, the video encoder comprising: at least one memory; at least one processor communicatively coupled to the at least one memory; Equipped with The at least one processor reads instructions from the at least one memory and encoding the plurality of pictures of a coded video sequence (CVS) into coded picture data; generating a sequence parameter set (SPS) comprising an indication that a sample aspect ratio (SAR) is variable within the CVS; generating a first supplemental enhancement information (SEI) message comprising a first SAR value for a first group of one or more pictures of the plurality of pictures in the CVS; generating a second SEI message comprising a second SAR value for a second group of one or more pictures of the plurality of pictures in the CVS, the second SAR value being different from the first SAR value and the second group being different from the first group; generating an encoded video signal such that the encoded video signal comprises the encoded picture data, the SPS, the first SEI message, and the second SEI message; transmitting the encoded video signal; configured to perform operations comprising: Video encoder.

9. The operations further comprise generating the SPS such that the SPS includes information on whether the SPS includes a default SAR value. The video encoder of claim 8 .

10. The encoded video signal: the SPS is before the first SEI message; the first SEI message is sent before encoding of the first group of the one or more pictures of the plurality of pictures of the CVS; the encoding of the first group of the one or more pictures of the plurality of pictures of the CVS occurs before the second SEI message; the second SEI message is sent before encoding of the second group of the one or more pictures of the plurality of pictures of the CVS; are arranged so that The video encoder of claim 8 .

11. 1. A method for encoding a plurality of pictures of a video sequence of a video, the method comprising: encoding the plurality of pictures of a coded video sequence (CVS) into coded picture data; generating a sequence parameter set (SPS) comprising an indication that a sample aspect ratio (SAR) is variable within the CVS; generating a first supplemental enhancement information (SEI) message comprising a first SAR value for a first group of one or more pictures of the plurality of pictures in the CVS; generating a second SEI message comprising a second SAR value for a second group of one or more pictures of the plurality of pictures in the CVS, the second SAR value being different from the first SAR value and the second group being different from the first group; generating an encoded video signal such that the encoded video signal comprises the encoded picture data, the SPS, the first SEI message, and the second SEI message; transmitting the encoded video signal; A method comprising:

12. generating the SPS such that the SPS includes information on whether the SPS includes a default SAR value; The method of claim 11.

13. The encoded video signal: the SPS is before the first SEI message; the first SEI message is sent before encoding of the first group of the one or more pictures of the plurality of pictures of the CVS; the encoding of the first group of the one or more pictures of the plurality of pictures of the CVS occurs before the second SEI message; the second SEI message is sent before encoding of the second group of the one or more pictures of the plurality of pictures of the CVS; are arranged so that The method of claim 11.

14. comprising instructions that, when executed, perform the method of claim 11 ; Non-transitory computer-readable medium.

Images