Video decoder, method for decoding, and non-transient computer-readable media

Abstract

The system provides a video decoder in which sample aspect ratio information for each decoded picture is determined based on control information and instructions indicating that the sample aspect ratio (SAR) can be changed within the video sequence. [Solution] The video decoder performs an operation comprising: receiving an encoded video signal comprising an encoded picture data and a sequence parameter set (SPS) comprising an indication that the sample aspect ratio (SAR) can be changed within the video sequence; receiving control information within the encoded video signal that specifies the SAR value to be applied to the picture in the video sequence after it has been received; decoding the encoded picture data in order to obtain a plurality of decoded pictures; and outputting a plurality of decoded pictures and sample aspect information for the decoded pictures.

Description

Technical Field

[0003] , , , ,

[0005] ,

[0004] , , ,

[0001] The present invention relates to a video decoder, a method for decoding, and a non - transient computer - readable medium.

[0002] H.265 / HEVC (HEVC = High Efficiency Video Coding) is a video codec that already provides tools for improving or enabling parallel processing in an encoder and / or decoder. For example, HEVC supports the sub - division of a picture into an array of tiles that are encoded independently of each other. Another concept supported by HEVC is related to WPP, according to which, for example in stripes, CTU rows or CTU lines of a picture may be processed in parallel from left to right, following some minimum CTU offsets in the processing of consecutive CTU lines (CTU = coding tree unit). However, it is preferable to have a video codec in the near future that more efficiently supports the parallel processing capabilities of video encoders and / or video decoders.

[0003] Hereinafter, VCL partitioning according to the prior art will be described (VCL = video coding layer).

[0004] Normally, in video coding, the encoding process of picture samples requires smaller partitions, and the samples are divided into several rectangular regions for combined processing such as predictive coding or transform coding. Therefore, a picture is divided into blocks of a specific size that is constant during the encoding of a video sequence. In H.264 / AVC, standard fixed - size blocks of 16×16 samples, so - called macroblocks, are used (AVC = Advanced Video Coding).

[0005] The latest HEVC standard (see ISO / IEC, ITU-T. High efficiency video coding. ITU-T Recommendation H.265 | ISO / IEC 23008 10 (HEVC), edition 1, 2013; edition 2, 2014) has coded tree blocks (CTBs) or coded tree units (CTUs) with a maximum size of 64x64 samples. In further explanations of HEVC, the more general term CTU is used for such types of blocks.

[0006] The CTUs are processed in raster scan order, starting from the top left CTU and working their way down to the bottom right CTU, processing each picture line individually.

[0007] Encoded CTU data is organized into a type of container called a slice. Originally, in earlier video encoding standards, a slice referred to a division containing one or more consecutive CTUs of a picture. Slices are used for segmenting encoded data. From another perspective, a complete picture can also be defined as one large segment, and therefore, historically, the term slice is still applicable. In addition to the encoded picture sample, a slice also contains additional information related to the encoding process of the slice itself, which is placed in a so-called slice header.

[0008] According to the latest technology, the Video Coding Layer (VCL) also incorporates techniques for fragmentation and spatial partitioning. Such partitioning can be applied to video coding for various reasons, including processing load balancing in parallelization, CTU size matching in network transmission, and error mitigation.

[0009] Another example relates to RoI (Region of Interest) coding, in which there is, for example, a central region of a picture that the viewer can select with a zoom-in operation (decoding of RoI only), or a gradual decoder refresh (GDR) distributed over several consecutive frames as a series of intrablocks that, for example, swipe across the picture plane and locally reset the time prediction chain in the same way that the intrapicture does across the entire picture plane. In the latter case, there are two regions in each picture, one recently reset and one potentially affected by errors and error propagation.

[0010] Reference Picture Resampling (RPR) is a technique used in video coding to adapt the quality / rate of video by not only using coarser quantization parameters but also potentially adapting the resolution of each transmitted picture. Therefore, the reference used for interpretation may have a different size than the picture currently being predicted for coding. Essentially, RPR requires a resampling process in the prediction loop, such as defined upsampling and downsampling filters.

[0011] Depending on the flavor, RPR can result in a change in the encoded picture size of any picture, or it can be restricted to occur only in specific pictures, for example, at specific locations bounded by segment boundaries, like adaptive HTTP streaming. [Overview of the project]

[0012] The object of the present invention is to provide an improved concept for video encoding and video decoding.

[0013] The object of the present invention is solved by the subject matter of the independent claim.

[0014] Preferred embodiments are provided in the dependent claims. [Brief explanation of the drawing]

[0015] Embodiments of the present invention will be described in detail below with reference to the drawings:

[0016] [Figure 1] A video encoder according to one embodiment is shown. [Figure 2] This shows a video decoder according to one embodiment. [Figure 3] This shows a system according to one embodiment. [Figure 4] This shows Region of Interest (RoI) scalability versus scalability based on expansion. [Figure 5a] This shows the first example of content fragments (gray) that move within a picture over time. [Figure 5b] Here is a second example of content fragments (gray) that move within the picture over time. [Figure 6a] This shows the current picture, which has a mixed reference picture. [Figure 6b] This example shows how to ignore the cropping window of a reference picture. [Figure 7] This shows the video encoder. [Figure 8] This shows a video decoder. [Figure 9] This shows the relationship between a combination of a reconstructed signal, such as a reconstructed picture, and a predicted residual signal, such as one signalized in a data stream, and the predicted signal. [Modes for carrying out the invention]

[0017] The following description of the drawings begins with a description of an encoder and decoder of a block-based predictive codec for encoding video pictures, in order to form an example of an encoding framework into which embodiments of the present invention can be incorporated. Each encoder and decoder is described with respect to Figures 7 to 9. Subsequently, embodiments of the concept of the present invention are described, along with a description of how such concepts are incorporated into the encoders and decoders of Figures 7 and 8, respectively. However, the embodiments described in Figures 1 to 3 and below may also be used to form encoders and decoders that do not operate according to the underlying encoding framework of the encoders and decoders of Figures 7 and 8.

[0018] Figure 7 shows a device for predictively encoding picture 12 into a data stream 14 using a video encoder, typically using transform-based residual coding. The device, i.e., the encoder, is indicated by reference numeral 10. Figure 8 shows a corresponding video decoder 20, for example, a device configured to predictively decode picture 12' from the data stream 14, also using transform-based residual decoding, where an apostrophe is used to indicate that the picture 12' reconstructed by the decoder 20 deviates from the picture 12 originally encoded by the device 10 with respect to coding loss introduced by the quantization of the predictive residual signal. While Figures 7 and 8 use transform-based predictive residual coding as an example, embodiments of the present application are not limited to this type of predictive residual coding. This also applies to other details described with respect to Figures 7 and 8, as outlined below.

[0019] The encoder 10 is configured to apply a spatial spectral transform to the predicted residual signal and encode the resulting predicted residual signal into a data stream 14. Similarly, the decoder 20 is configured to decode the predicted residual signal from the data stream 14 and apply a spectral spatial transform to the resulting predicted residual signal.

[0020] Internally, the encoder 10 may include a prediction residual signal former 22 that generates a prediction residual 24, thereby measuring the deviation of the prediction signal 26 from the original signal, for example, from the picture 12. The prediction residual signal former 22 may also be a subtractor that subtracts the prediction signal from the picture 12 from the original signal. The encoder 10 further includes a transformer 28 that applies a spatial spectral transform to the prediction residual signal 24 to obtain a spectral domain prediction residual signal 24', and this residual signal is quantized by a quantizer 32 also configured by the encoder 10. Thus, the quantified prediction residual signal 24 is encoded in a bitstream 14, but the encoder 10 may optionally configure an entropy coder 34 that converts and quantizes the prediction residual signal into a data stream by entropy coding, and the prediction signal 26 is generated by the prediction stage 36 of the encoder 10 based on the prediction residual signal 24 that is encoded in the data stream 14 or decodeable from the data stream 14. For this purpose, the prediction stage 36 may internally include a dequantizer 38 that dequantizes the prediction residual signal 24'' to obtain a spectral domain prediction residual signal 24'''' corresponding to the signal 24' excluding quantization losses, as shown in Figure 7. Then, an inverse transformer 40 inverse transforms the latter prediction residual signal 24'''', for example, by a spectral-spatial transform, to obtain a prediction residual signal 24'''' corresponding to the original prediction residual signal 24 excluding quantization losses. The combiner 42 of the prediction stage 36 can then recombine the prediction signal 26 and the prediction residual signal 24'''', for example by addition, to obtain a reconstructed signal 46, the original signal, for example, a reconstruction of picture 12. The reconstructed signal 46 may correspond to signal 12'. The prediction module 44 of the prediction stage 36 then generates the prediction signal 26 based on signal 46, for example by using spatial prediction, i.e., intra-picture prediction, and / or temporal prediction, i.e., inter-picture prediction.

[0021] Similarly, as shown in FIG. 8, the decoder 20 may be internally configured from components corresponding to the prediction stage 36 and interconnected in a corresponding manner. In particular, the entropy decoder 50 of the decoder 20 can entropy-decode the quantized spectral region prediction residual signal 24'' from the data stream, where the inverse quantizer 52, the inverse transformer 54, the combiner 56, and the prediction module 58 are interconnected and cooperate in the manner described above with respect to the modules of the prediction stage 36 to recover the reconstructed signal based on the prediction residual signal 24'', whereby, as shown in FIG. 8, the output of the combiner 56 results in the reconstructed signal, i.e., the picture 12'.

[0022] Although not specifically stated above, it is readily apparent that encoder 10 may set several coding parameters, including, for example, prediction mode, motion parameters, etc., according to certain rate and distortion-related criteria, for example, a method for optimizing coding cost, for example, a certain optimization scheme. For example, encoder 10 and decoder 20, and corresponding modules 44 and 58 may support different prediction modes, such as intra-coding mode and inter-coding mode, respectively. The granularity at which the encoder and decoder switch between these prediction mode types may correspond to the subdivision of picture 12 and 12' into coding segments or coding blocks, respectively. At the unit of these coding segments, for example, the picture may be subdivided into intra-coding blocks and inter-coding blocks. The intra-coding blocks are predicted based on the spatial, already coded / decoded neighborhood of each block, as outlined in more detail below. Several intra-coding modes may exist, each intra-coding mode being selected for each intra-coding segment, including directional or angular intra-coding modes, in which each segment is satisfied by extrapolating neighboring sample values ​​along a certain direction specific to each directional intra-coding mode to each intra-coding segment. The intra-coding modes may also comprise one or more further modes, such as a DC coding mode in which the prediction for each intra-coded block assigns DC values ​​to all samples within each intra-coded segment, and / or a planar intra-coding mode in which the prediction for each block is approximated or determined to be a spatial distribution of sample values ​​described by a two-dimensional linear function over the sample positions of each intra-coded block, having a plane-driven slope and offset defined by a two-dimensional linear function based on adjacent samples. In contrast, inter-coded blocks may be predicted, for example, in time.In the case of an inter-coded block, the motion vector may be signaled within the data stream, where the motion vector indicates a spatial displacement of a portion of a previously encoded picture of the video to which picture 12 belongs, and the previously encoded / decoded pictures are sampled to obtain the prediction signal for each inter-coded block. This means that in addition to the residual signal coding provided by the data stream 14, such as the entropy-coded transform coefficient levels representing the quantized spectral region prediction residual signal 24'', the data stream 14 may also encode optional further parameters, such as coding mode parameters for assigning coding modes to various blocks, prediction parameters for some blocks such as motion parameters for inter-coded segments, and parameters for controlling and signaling the subdivision of pictures 12 and 12' into segments. The decoder 20 uses these parameters to subdivide the pictures in a similar way as the encoder did, assign the same prediction modes to the segments, and perform the same predictions to result in the same prediction signals.

[0023] Figure 9 shows the relationship between, on the one hand, a reconstructed signal, such as the reconstructed picture 12', and on the other hand, a combination of the predicted residual signal 24'''' signaled in the data stream 14 and the predicted signal 26. As already mentioned above, the combination may be additional. The predicted signal 26 is shown in Figure 9 as a subdivision of the picture region into intra-coded blocks, shown exemplary with hatching, and inter-coded blocks, shown exemplary without hatching. The subdivision may be any subdivision, such as regularly subdividing the picture region into rows and columns of square or non-square blocks, or it may be a multi-tree subdivision, where the picture 12 is subdivided from a tree root block into multiple leaf blocks of various sizes, such as a quad-tree subdivision, and the picture region is first subdivided into rows and columns of a tree root block, and then further subdivided into one or more leaf blocks according to a recursive multi-tree subdivision, a mixture of these shown in Figure 9.

[0024] Furthermore, the data stream 14 may have an intra-encoding mode encoded therein for the intra-encoding block 80, which assigns one of several supported intra-encoding modes to each intra-encoding block 80. For the inter-encoding block 82, the data stream 14 may have one or more motion parameters encoded therein. Generally speaking, the inter-encoding block 82 is not limited to being encoded in time. Alternatively, the inter-encoding block 82 may be any block predicted from a previously encoded portion beyond the current picture 12 itself, such as a previously encoded picture of the video to which the picture 12 belongs, or another view or hierarchically lower-level picture if the encoder and decoder are scalable encoder and decoder, respectively.

[0025] Furthermore, the predicted residual signal 24'''' in Figure 9 is illustrated as a subdivision of the picture region into block 84. These blocks are sometimes called transform blocks to distinguish them from the same as the encoded blocks 80 and 82. In effect, Figure 9 shows that the encoder 10 and decoder 20 can use two different subdivisions of picture 12 and picture 12', one that subdivides into blocks, i.e., encoded blocks 80 and 82, respectively, and another that subdivides into transform block 84. Both subdivisions may be the same, for example, each encoded block 80 and 82 may simultaneously form transform block 84, but Figure 9 shows, for example, the case where the subdivision into transform block 84 forms an extension of the subdivision into encoded blocks 80, 82 such that any boundary between the two blocks 80 and 82 coincides with the boundary between the two blocks 84, or alternatively, each block 80, 82 coincides with one of the transform blocks 84 or coincides with a cluster of transform blocks 84. However, the subdivisions may be determined or selected independently of each other so that the transformed block 84 can intersect the block boundaries between blocks 80 and 82. With respect to the subdivision to transformed block 84, the same applies as presented for the subdivision to blocks 80 and 82, for example, block 84 may be the result of regular subdivision of the picture region into blocks (with or without row and column arrangement), the result of recursive multi-tree subdivision of the picture region, or a combination thereof, or any other type of blockization. Incidentally, it should be noted that blocks 80, 82, and 84 are not limited to being square, rectangular, or any other shape.

[0026] Figure 9 further shows that the combination of the prediction signal 26 and the prediction residual signal 24'''' directly yields the reconstructed signal 12'. However, it should be noted that, according to an alternative embodiment, two or more prediction signals 26 may be combined with the prediction residual signals 24'''' to obtain the picture 12'.

[0027] In Figure 9, the transformation blocks 84 have the following importance: The converters 28 and inverse converters 54 perform their transformations in units of these transformation blocks 84. For example, many codecs use some kind of DST or DCT for all transformation blocks 84. Some codecs allow skipping transformations for some of the transformation blocks 84 so that the predicted residual signal is directly encoded in the spatial domain. However, according to the embodiments described below, the encoder 10 and decoder 20 are configured to support several transformations. For example, the transformations supported by the encoder 10 and decoder 20 may include: • DCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform. • DST-IV, where DST represents the Discrete Sine Transform. · DCT-IV · DST-VII • Identity transformation (IT)

[0028] Naturally, the converter 28 supports all forward conversion versions of these conversions, while the decoder 20 or inverse converter 54 supports their corresponding reverse or reverse versions: • Inverse DCT-II (or Inverse DCT-III) · Reverse DST-IV ·Inverse DCT-IV · Reverse DST-VII • Identity transformation (IT)

[0029] The following explanation provides further details on which transformations may be supported by the encoder 10 and decoder 20. Note that in any case, the set of supported transformations may consist of only one transformation, such as a single spectral-to-spatial transformation or a spatial-to-spectral transformation.

[0030] As outlined above, Figures 7 to 9 are presented as examples that can implement the concepts of the present invention, which will be further described below, in order to form specific examples of encoders and decoders according to this application. The encoders and decoders in Figures 7 and 8 may represent possible implementations of the encoders and decoders described below, respectively. However, Figures 7 and 8 are merely examples. However, the encoders according to embodiments of this application may perform block-based coding of picture 12 differently from the encoder in Figure 7, for example, in that they are still image encoders rather than video encoders, do not support mutual prediction, or the subdivision into blocks 80 is performed in a different manner than illustrated in Figure 9. Similarly, a decoder according to an embodiment of the present invention can perform block-based decoding of picture 12' from data stream 14 using the coding concepts further outlined below, but may differ from the decoder 20 in Figure 8, for example, in that it is a still picture decoder rather than a video decoder, does not support intra-prediction, subdivides picture 12' into blocks in a different manner than described with respect to Figure 9, and / or does not derive prediction residuals from data stream 14 in the transformation region, for example.

[0031] In the following, a general-purpose video encoder according to an embodiment is shown in Figure 1, a general-purpose video decoder according to an embodiment is shown in Figure 2, and a general-purpose system according to an embodiment is shown in Figure 3.

[0032] Figure 1 shows a typical video encoder 101 according to an embodiment.

[0033] The video encoder 101 is configured to encode multiple pictures of a video by generating an encoded video signal, and each of the multiple pictures contains the original picture data.

[0034] The video encoder 101 includes a data encoder 110 configured to generate an encoded video signal comprising encoded picture data, the data encoder being configured to encode multiple pictures of the video into encoded picture data.

[0035] Furthermore, the video encoder 101 includes an output interface 120 configured to output encoded picture data for each of multiple pictures.

[0036] Figure 2 shows a general-purpose video decoder 151 according to an embodiment.

[0037] The video decoder 151 is configured to decode an encoded video signal containing encoded picture data in order to reconstruct multiple pictures of a video.

[0038] The video decoder 151 includes an input interface 160 configured to receive encoded video signals.

[0039] Furthermore, the video decoder includes a data decoder 170 configured to reconstruct multiple pictures of the video by decoding encoded picture data.

[0040] Figure 3 shows a typical system according to an embodiment.

[0041] The system includes a video encoder 101 shown in Figure 1 and a video decoder 151 shown in Figure 2.

[0042] The video encoder 101 is configured to generate an encoded video signal.

[0043] The video decoder 151 is configured to decode the encoded video signal and reconstruct the video picture.

[0044] A first aspect of the present invention is described in claims 1 to 33. The first aspect provides a sample aspect ratio signal.

[0045] A second aspect of the present invention is described in claims 34 to 72. The second aspect provides reference picture resampling restrictions to reduce the implementation burden.

[0046] A third aspect of the present invention is described in claims 73 to 131. The third aspect provides a flexible region-based reference for zooming reference picture resampling, and in particular provides a more efficient address zoom use case.

[0047] The first aspect of the present invention will be described in detail below.

[0048] In particular, the first embodiment provides sample aspect ratio signaling.

[0049] The sample aspect ratio (SAR) is related to correctly presenting encoded video to the consumer, and as a result, if the aspect ratio of the encoded sample array changes over time via RPR (e.g., by subsampling in one dimension), the aspect ratio of the presented picture can be kept constant as intended.

[0050] This allows setting a constant SAR for the entire latest SAR-signaled video sequence in the Video Usability Information (VUI) within a Sequence Parameter Set (SPS) such as HEVC or AVC. For example, SAR changes are only permitted at the beginning of the encoded video sequence (e.g., the sample aspect ratio is constant for each encoded video sequence).

[0051] Accordingly, as part of the present invention, a new mode of SAR signaling is introduced to video coding. The sequence level parameter set, i.e., SPS, includes the following instructions: • RPR is used (therefore, the encoded picture size may change). • Actual SAR is not provided in VUI. Instead, the SAR of encoded video is shown to be dynamic and can change within the CVS (coded video sequence). The actual SAR of the encoded picture is indicated via SEI (Supplemental Enhancement Information) messages at resolution switching points. [Table 1]

[0052] Dynamic SAR Information SEI Message [Table 2]

[0053] Similarly, vui_aspect_ratio_constant_flag may be adopted, for example.

[0054] The vui_aspect_ratio_constant_flag may, for example, be an indicator that shows whether the aspect ratio of a sample is constant with respect to the video sequence, or whether the aspect ratio of a sample can be changed within the video sequence.

[0055] For example, if vui_aspect_ratio_constant_flag can be set to 0 (or FALSE, or -1), this may indicate that dynamic SAR information is present in the SEI message.

[0056] In an alternative embodiment, SAR information within the VUI (e.g., SPS) is used as the default unless an SEI message is available. The information in the SEI message is overwritten by the information in the SPS. [Table 3]

[0057] In another embodiment, SAR information is associated with the picture resolution and signaled in the PPS (Picture Parameter Set), and the picture resolution is signaled. The default SAR is signaled in the SPS, and if the SAR changes for a particular picture resolution, a different SAR is signaled and overrides the default SAR.

[0058] SPS VUI: [Table 4]

[0059] Furthermore, in the case of SEI, SPS can also indicate that SAR may be changed and that SAR will be updated in PPS (similar to the previous aspect_ratio_dynamic_SEI_present_flag). Therefore, in some applications, it may be possible to constrain or limit the change to SAR, which can facilitate implementation or RPR / ARC.

[0060] PPS: [Table 5]

[0061] If pps_aspect_ratio_info_present_flag is set to 0, the default SAR is obtained from SPS; otherwise, the actual SAR is provided.

[0062] A second aspect of the present invention will be described in detail below.

[0063] In particular, the second aspect provides signaling regarding constraints for reference picture resampling.

[0064] The implementation burden can be reduced by restricting the RPR scheme in various ways. In a general RPR scheme that does not include additional restrictions, such as the invention described below, the implementer needs to overprovision its decoder hardware to perform the following: • Any current picture, worst case: resampling of all pictures • DPB (Decoded Picture Buffer), Intermediate GOP (Group of Pictures) pair, resampling of any picture at a defined location with fewer reference pictures. • Simultaneous resampling of multiple images at various resolutions to the target resolution • (Reference) Cascaded resampling chain of ref pic with loss of picture quality

[0065] The following invented limitations make it possible to reduce the implementation cost of codecs featuring such limited RPR schemes compared to unlimited RPR codecs.

[0066] In one embodiment, resolution changes are permitted only at a random access point (RAP), for example, the maximum number of resampled pictures is the number of random access decodable skipped pictures (RASL) at this RAP, and the RAP is typically located one or more GOPs, for example, several tens of pictures apart, which reduces the worst-case rate at which such resampling operations must be supported.

[0067] In another embodiment, the resolution change affects key pictures in the hierarchical GOP, for example, the picture: · Belonging to the lowest time layer, • Occurs once in every GOP, and • All subsequent pictures in the encoding order have a lower POC (e.g., an earlier presentation timestamp). When a reference picture is resampled, none of the immediately following pictures in the GOP of a higher time layer require cascading up / downsampling.

[0068] According to another embodiment, resolution changes are permitted only in the picture that immediately follows the key picture in the presentation order, i.e., the first picture of the next GOP in the presentation order.

[0069] In another embodiment, the time distance between consecutive resolution changes is limited by the minimum POC (picture order count) distance in the level definition.

[0070] In another embodiment, the temporal distance between consecutive resolution changes is limited by the minimum number of intermediate encoded pictures in the level definition.

[0071] In another embodiment, a change in resolution can only occur in a picture marked as non-discardable, or in a reference picture where non_reference_picture_flag is equal to 0.

[0072] In another embodiment, the rate of resolution change is limited by the level definition.

[0073] In another embodiment, the resampling of the reference picture relative to the current picture is restricted to using a single resampling ratio, for example, requiring that all reference pictures of the current picture having a different resolution than the current picture have the same resolution.

[0074] In another embodiment, when one reference picture of the current picture requires resampling, all reference pictures of the current picture must also use resampling, for example, to the same original resolution as one of the reference pictures.

[0075] In another embodiment, only one reference picture of the current picture is permitted to require resampling.

[0076] According to another embodiment, the maximum number of pictures requiring resampling at resolution change points is optionally indicated within the encoded video sequence / bitstream as a decoder guarantee, and if no such indication exists, inference or indication is made by level definition.

[0077] In another embodiment, the original (unresampled) reference picture is removed from the reference picture list and / or decoded picture buffer after resampling, and is marked, for example, as not being used for reference, so that only the resampled reference picture is available from it.

[0078] In another embodiment, the resampling ratio used within an encoded video sequence is limited to a set of resampling ratios contained in a parameter set having a sequence or bitstream scope (decoding parameter set, DPS; sequence parameter set, SPS).

[0079] A third aspect of the present invention will be described in detail below.

[0080] In particular, the second embodiment provides a flexible region-based reference for zooming for reference picture resampling.

[0081] As described above, layered codecs such as SHVC and SVC address two modes of extended scalability: RoI scalability (where the area of ​​the lower layer picture is expanded in the upper layer) and extended scalability (where the lower layer picture is expanded in the upper layer through additional content), as shown in Figure 4 below.

[0082] Scalability can also refer to use cases that change over time, such as the so-called zoom-out use case, where the video fully covers more content, such as a wider capture angle, a larger portion of the scene, or a larger area.

[0083] Figure 4 shows the region of interest (RoI) scalability relative to scalability.

[0084] In scenarios where zooming in and zooming out are permitted, the zoom and movement regions used for prediction and predicted are defined. This is known as RoI scalability (typically zooming in) or augmented scalability (typically zooming out). In RoI scalability with scalable coding, the region to be upscaled to the dimensions of the reference picture is usually defined within the reference picture. However, in scalable coding, the upper and lower picture on which the prediction is performed represents the same time.

[0085] In the case of SHVC and SVC, this is done for layered coding, in which case the juxtaposed base layers do not represent any movement; for example, since the corresponding samples in the base layers are known, it was possible to completely upscale the known regions in the base layers and operate based on that upscaled criterion.

[0086] However, in RPR applications, the two pictures on which predictions are performed do not depict the same time instance, and therefore some content outside the defined area can move from time instance A (low resolution) to time instance B (high resolution) in a zoom-in / out region. Prohibiting the referencing of these areas for prediction is detrimental to encoding efficiency.

[0087] However, in the case of RPR, for example, by zooming in on an object moving to a RoI within the area, the reference can point to an area outside the corresponding reference region. This is illustrated in Figure 5a without actually changing the encoding resolution:

[0088] Figure 5a illustrates the first example of content fragments (gray) moving within the picture over time.

[0089] In the first embodiment, a reference region is defined that includes an area larger than the area of ​​the RoI, such that the gray box in the figure that falls within the RoI zoom region is within the reference:

[0090] Figure 5b illustrates a second example of content fragments (gray) moving within the picture over time.

[0091] This results in reconstructing a slightly larger area than the RoI for the picture corresponding to the RoI, with the additional area being removed by indicating a cropping window. This problem arises from the fact that the scaling factors used to upsample the reference are calculated in VVC (Versative Video Coding) from the cropped picture. First, assuming there is no RoI, the horizontal scaling factor HorScale and vertical scaling factor VerScale are calculated as follows: HorScale = CroppedOutWidthPic / CroppedOutWidthRefPic VerScale = CroppedOutHeightPic / CroppedOutHeightRefPic

[0092] The reason for indicating the ratio based on the cropped picture is that the codec needs to be a multiple of the minimum size (in VVC 8 samples), so depending on the size of the picture of interest, some additional samples need to be decoded. Therefore, if either Pic or RefPic is not a multiple of 8, some samples are added to the input picture, making them multiples of 8, resulting in a different ratio and an incorrect scaling factor. This problem can be further exacerbated if the bitstream is to be encoded as "mergeable," meaning that the bitstream can be merged with other bitstreams, for example, if the picture size needs to be a multiple of the CTU size up to 128. Therefore, the correct scaling factor must take the cropping window into account.

[0093] In the described scenario (combining RPR with RoI), where a cropping window is used to include some additional references, the use of a cropping window is insufficient.

[0094] As mentioned above, the RoI in the reference picture can be defined to be slightly larger, and the RoI will be used for the reference but will be discarded along with the cropping window in the current reconstructed picture. However, if the horizontal scale factor HorScale and vertical scale factor VerScale are calculated as follows: HorScale = CroppedOutWidthPic / WidthEnlargedRefRoI VerScale = CroppedOutHeightPic / HeightEnlargeRefRoI Some of the samples within the expanded RoI actually correspond to samples within the cropped-out region, so the results are incorrect.

[0095] The following describes the cropping window-based concept according to the first group of embodiments.

[0096] Therefore, in the embodiments of the first group described above, the calculation may be, for example, as follows: HorScale = CodedPicWidth / RefRoIWidth VerScale = CodedPicHeight / RefRoIHeight Such calculations involve samples that are cut out for the purpose of calculating the scaling factor.

[0097] Regarding signaling, in one embodiment, the signaling of the expanded RoI indicates that the cropping window information should be ignored in the scaling factor calculation.

[0098] In another embodiment, whether or not a cropping window needs to be considered in the calculation of the scaling factor is indicated by the bitstream (e.g., parameter set or slice header). [Table 6]

[0099] The cropping window is sometimes called the adaptive cropping window, for example. The offsets relative to the cropping window / adaptive cropping window are sometimes called pps_conf_win_left_offset, pps_conf_win_top_offset, pps_conf_win_right_offset, and pps_conf_win_botton_offset, for example.

[0100] Instead of using `use_cropping_for_scale_factor_derivation_flag` to determine whether to ignore information in the encoded video signal on the cropping window in order to upscale the area in the reference picture (or to determine whether to use information in the encoded video signal on the cropping window in order to upscale the area in the reference picture), you may use, for example, `pps_scaling_window_explicit_signalling_flag`.

[0101] For example, if pps_scaling_window_explicit_signalling_flag is set to 0 (or, for example, set to FALSE, or, for example, set to -1), information in the encoded video signal on the cropping window may be used, for example, to upscale a region in the reference picture. And, for example, if pps_scaling_window_explicit_signalling_flag is set to 1 (or, for example, set to TRUE), information in the encoded video signal on the cropping window may be ignored, for example, to upscale a region in the reference picture.

[0102] One drawback of the above approach is that a larger area is decoded for the current picture to allow referencing samples outside the RoI, for example, samples on an expanded RoI. More specifically, samples are decoded in areas outside the RoI and later discarded in the cropping window. This leads to additional sample overhead and a reduction in coding efficiency, which can potentially counteract the coding efficiency gain that allows referencing outside the corresponding RoI in the reference picture.

[0103] A more efficient approach would be to decode only the RoI (omitting the additional samples needed to create an 8x picture or CTU, as mentioned above), but allowing us to reference the samples within the expanded RoI.

[0104] The following describes the bounding box-based concept according to the second group of embodiments.

[0105] In the embodiment of the second group described above, a sample that is outside the red rectangle but inside the green box (with an additional RoI offset added to the RoI offset) is used to determine the resampled ref pic instead of using only the red RoI.

[0106] The size of the bounding box for MV around the red cutout is defined / signaled with the advantage of limiting memory access / line buffer requests and enabling implementation by a pic-wise upsampling approach.

[0107] Such signaling can be included in PPS(additional_roi_X): [Table 7]

[0108] Therefore, the derivation of the scaling factor is as follows: HorScale = CroppedOutWidthPic / RefRoIWidth VerScale = CroppedOutHeightPic / RefRoIHeight

[0109] In one embodiment, a reference sample is identified by finding a sample arranged using RoI_X_offset and applying a clipping MV if the reference sample lies outside the expanded RoI indicated by additional_RoI_x. Alternatively, this sample outside the expanded RoI is padded with the last sample within the expanded RoI.

[0110] In another embodiment, this expanded RoI is used solely as a limitation or constraint that can be used for implementation optimization. For example, if the reference picture is to be fully upsampled first as needed, rather than on-the-fly (block-based), only the expanded RoI is resampled instead of the entire picture, saving a lot of processing.

[0111] A further problem arises when two or more referenced pictures are used simultaneously. In this case, it is necessary to identify the picture to which the RoI area information applies. Instead of adding information to the PPS in such cases, the slice header indicates that some entries in the reference list refer to only a part of the picture, not the entire picture. For example, [Table 8]

[0112] In further embodiments, additional constraints are in place: Only reference pictures with a lower POC can have RoI information. Typically, RoI switching is applied to open GOP switching scenarios using the described features, and therefore, POCs with a higher POC already represent RoI scenes. Only one reference picture can have RoI information.

[0113] In another embodiment, RoIInfo() is carried with a picture parameter set, and the slice header carries only a flag (RoI_flag) for each reference picture, indicating whether or not RoI information should be applied for resampling (derivation of scaling factors). The following figure illustrates the principle in four encoded pictures two before and two after a switching point. At the switching point, the total resolution remains constant, but RoI upsampling is performed. Two PPS are defined, and the PPS of the latter two pictures indicate the RoI in the reference picture. Furthermore, the slice header of the latter two pictures carries RoI_flag[i] for each of those reference pictures, the value of which is shown in the figure as "RoI_flag" or "RF = x".

[0114] Furthermore, the slice header can carry, for each referenced picture, not only RoI_flag as described above, but also an additional index to the array of RoIInfo(), which is carried in the parameter set to identify the RoI information to apply to a particular referenced picture, if the flag is true.

[0115] Figure 6a shows the current picture with mixed reference pictures.

[0116] The following describes the case of zooming out according to the third group of embodiments.

[0117] Instead of RoI scalability, in embodiments of the third group described above, extended scalability, such as scalability from an RoI picture to a larger area, can be considered. In such cases, the cropping window of the reference picture should also be ignored, especially if the area of ​​the currently decoded picture is identified as the area for scalability such as scaling.

[0118] Figure 6b shows an example of ignoring the cropping window of a reference picture when a region is identified within the current picture.

[0119] HorScale = IdentifiedRegionInPicWidth / CodedRefPicWidth VerScale = IdentifiedRegionInPicHeight / CodedRefPicHeight

[0120] While some embodiments have been described in the context of the apparatus, it is clear that these embodiments also represent descriptions of the corresponding methods, where a block or device corresponds to a method step or a feature of a method step. Similarly, embodiments described in the context of a method step also represent descriptions of the corresponding blocks, items, or features of the corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0121] Depending on the specific implementation requirements, embodiments of the present invention may be implemented in hardware, in software, or at least partially in hardware, or at least partially in software. This implementation may be carried out using digital storage media, such as floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs, EPROMs, EEPROMs, or flash memory, which store electronically readable control signals that cooperate (or can cooperate) with a programmable computer system to perform the respective methods. Thus, the digital storage media may be computer-readable.

[0122] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system so that one of the methods described herein is performed.

[0123] Generally, embodiments of the present invention can be implemented as a computer program product having program code, the program code operable to perform one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0124] Other embodiments include a computer program stored in a machine-readable carrier for performing one of the methods described herein.

[0125] In other words, one embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

[0126] Accordingly, a further embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) having a computer program for performing one of the methods described herein recorded thereon. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-temporary.

[0127] Accordingly, a further embodiment of the method of the present invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0128] Further embodiments include processing means, such as a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0129] Further embodiments include a computer on which a computer program for performing one of the methods described herein is installed.

[0130] Further embodiments of the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0131] In some embodiments, programmable logic devices (e.g., field-programmable gate arrays) can be used to perform some or all of the functions of the methods described herein.

[0132] In some embodiments, a field-programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is performed preferably by any hardware device.

[0133] The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0134] The methods described herein may be performed using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0135] The embodiments described above are merely for illustrating the principles of the present invention. Modifications and variations of the configurations and details described herein will be obvious to those skilled in the art. Accordingly, the invention is intended to be limited only by the impending claims and not by the descriptions of the embodiments herein and by the specific details presented herein.

Claims

1. A video decoder for decoding an encoded video signal comprising encoded picture data in order to decode multiple pictures of a video sequence, wherein the video decoder: At least one memory, At least one processor that is communicably connected to the at least one memory, Equipped with, The at least one processor reads an instruction from the at least one memory, The steps include receiving an encoded video signal comprising encoded picture data and a sequence parameter set (SPS) having an indication that the sample aspect ratio (SAR) can be changed within the video sequence, The step of receiving control information specifying a SAR value within the encoded video signal, wherein the control information is applied to the picture of the video sequence after the control information has been received, To obtain multiple decoded pictures, the process involves the steps of decoding the encoded picture data and The steps include outputting the plurality of decoded pictures and sample aspect information for the decoded pictures, It is configured to perform an operation that includes the following: The sample aspect information for each of the decoded pictures is determined based on the control information and the instructions indicating that the SAR can be modified within the video sequence. Video decoder.

2. The control information is included in the supplemental enhancement information (SEI) message associated with the video sequence. The video decoder according to claim 1.

3. The control information is included in the video usability information (VUI) parameter set, or in another syntax structure within the encoded video signal. The video decoder according to claim 1.

4. The at least one processor is further configured to apply the new SAR value to a decoded picture acquired after the updated control information is received, in response to receiving updated control information specifying a new SAR value. The video decoder according to claim 1.

5. The updated control information is received in a subsequent SEI message that follows the SEI message specifying the previous SAR value. The video decoder according to claim 4.

6. The SPS further includes a default SAR value that is applied to the decoded picture when the control information specifying an alternative SAR value does not exist. The video decoder according to claim 1.

7. The at least one processor is configured to output metadata for each decoded picture that identifies the SAR value applied to the picture. The video decoder according to claim 1.

8. The control information specifying the SAR value is received in association with an encoded picture or GOP (Group of Pictures) and is applied to all decoded pictures of the encoded picture or GOP. The video decoder according to claim 1.

9. The at least one processor is configured to determine the SAR value for each decoded picture based on both the control information and the temporal ordering of the access units in the encoded video signal. The video decoder according to claim 1.

10. The at least one processor is further configured to perform picture rescaling or pixel aspect ratio adjustment of the decoded picture according to the determined SAR value before outputting. The video decoder according to claim 1.

11. A method for decoding an encoded video signal comprising encoded picture data in order to decode multiple pictures of a video sequence, wherein the method is: The steps include receiving an encoded video signal comprising encoded picture data and a sequence parameter set (SPS) having an indication that the sample aspect ratio (SAR) can be changed within the video sequence, The step of receiving control information specifying a SAR value within the encoded video signal, wherein the control information is applied to the picture of the video sequence after the control information has been received, To obtain multiple decoded pictures, the process involves the steps of decoding the encoded picture data and The steps include outputting the plurality of decoded pictures and sample aspect information for the decoded pictures, Equipped with, The sample aspect information for each of the decoded pictures is determined based on the control information and the instructions indicating that the SAR can be modified within the video sequence. method.

12. The control information is included in the supplemental enhancement information (SEI) message associated with the video sequence. The method according to claim 11.

13. The control information is included in the video usability information (VUI) parameter set, or in another syntax structure within the encoded video signal. The method according to claim 11.

14. The further step includes, in response to receiving updated control information specifying a new SAR value, applying the new SAR value to a decoded picture acquired after the updated control information has been received. The method according to claim 11.

15. The updated control information is received in a subsequent SEI message that follows the SEI message specifying the previous SAR value. The method according to claim 14.

16. The SPS further includes a default SAR value that is applied to the decoded picture when the control information specifying an alternative SAR value does not exist. The method according to claim 11.

17. The further step includes outputting metadata for each decoded picture that identifies the SAR value applied to that picture. The method according to claim 11.

18. The control information specifying the SAR value is received in association with an encoded picture or GOP (Group of Pictures) and is applied to all decoded pictures of the encoded picture or GOP. The method according to claim 11.

19. The process further includes, before outputting, performing picture rescaling or pixel aspect ratio adjustment of the decoded picture according to the determined SAR value, The method according to claim 11.

20. When executed by one or more processors, the one or more processors: The steps include receiving encoded picture data and an encoded video signal comprising a sequence parameter set (SPS) with an indication that the sample aspect ratio (SAR) can be changed within the video sequence, The step of receiving control information specifying a SAR value within the encoded video signal, wherein the control information is applied to the picture of the video sequence after the control information has been received, To obtain multiple decoded pictures, the process involves the steps of decoding the encoded picture data and The steps include outputting the plurality of decoded pictures and sample aspect information for the decoded pictures, A non-temporary computer-readable medium that stores instructions for performing an operation comprising, The sample aspect information for each of the decoded pictures is determined based on the control information and the instruction that the SAR can be modified within the video sequence. A non-temporary computer-readable medium.

Images