Video coding concept with drift suppression

Abstract

To provide a video decoding method that more effectively suppresses an adverse effect on video quality due to drift caused by formation of a video bitstream composed of segments under switching between different video bitstream representations.SOLUTION: The decoder A20 comprises at least one processor and is configured to decode, from a datastream A14, a supplemental enhancement information (SEI) message associated with a sequence of pictures including at least one random access skipped leading (RASL) picture and a clean random access (CRA) picture associated with the at least one RASL picture, determine that the SEI message indicates that coding of all the RASL pictures in the sequence of pictures was constrained by excluding at least one coding tool, and decode the at least one RASL picture without using the at least one coding tool.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] Detailed Description The present application relates to video coding and concepts suitable for reducing drift.

[0002] HTTP streaming of coded video has become an important conduit for video distribution over the past decade, and today it can be reached by OTT service providers over the public Internet by hundreds of millions of users. Standard protocols such as Dynamic Adaptive Streaming over HTTP (DASH) [1] enable service providers to stream media to clients by having the server provide the media in a temporally segmented format at various bitrates. Client devices can then download and play successive segments in a dynamic and adaptive manner by selecting among the offered variants of a particular segment according to their available network bandwidth and their decoding capabilities. In practice, content is provided as multiple so-called representations generated by an optimized bitrate ladder, often including multiple resolutions and fidelities, to optimize the perceived quality for a particular bitrate and thereby the user experience [2]. Each segment is typically coded independently of previous segments using a so-called closed group of pictures (GOP) coding structure [2], so that the downloaded and depacketized segment data can be concatenated into a suitable bitstream and fed to the decoder. In contrast to such closed GOP structures, segments using so-called open GOP coding structures contain some pictures that use inter-prediction from pictures in the previous segment, which is advantageous for coding efficiency. Because pictures using inter-prediction from the previous segment come first in presentation order, they can be skipped from the output without playback problems or visual artifacts when the segment is accessed randomly. However, problems arise when a resolution switch occurs during continuous playback, as these pictures are skipped in such a non-seamless switch. Even a pure bitrate switch can result in some pictures being dropped or significant visual artifacts if the segment is not properly encoded for the switch.

[0003] Popular older generation codecs, such as AVC [4] and HEVC [5], do not provide the reference picture resampling (RPR) functionality required to use reference pictures of different resolutions. Therefore, when a resolution switch occurs in such an open GOP structure, some pictures in the segment cannot be decoded correctly after the resolution switch because reference pictures from the previous segment are not available at the required resolution, resulting in non-constant frame rate playback at the segment switch from the dropped pictures. In [6], the authors present approaches to overcome the open GOP resolution switching problem by adopting standard modifications to the HEVC decoding process or by using the less widely used scalable HEVC extension (SHVC) that provides RPR functionality. However, available solutions have not yet enabled substantial adoption of open GOP coding in HTTP streaming.

[0004] The recently completed Versatile Video Coding (VVC) standard [7], Version 1, is the latest video coding standard resulting from a collaboration between the ITU-T Video Coding Experts Group and ISO / IEC Subcommittee 29, also known as the Moving Pictures Experts Group. VVC not only offers significantly improved coding efficiency compared to previous generation codecs [8], but also includes many application-driven features in its initial Main 10 profile, such as RPR. During the development of VVC, RPR was primarily studied in the context of interactive scenarios using low-latency coding structures [9], where practical requirements on latency and buffer size set severe limits on the feasibility of inserting intra-coded pictures for resolution switching.

[0005] However, RPR in VVC can also bring significant benefits to coding efficiency in video encoding in the streaming domain.

[0006] It would be advantageous to have a concept at hand that allows open GOP resolution switching to be used in HTTP streaming using codecs such as VVC, in which case the problems outlined above arise not only with respect to RPR, but also when, for example, concatenating segments associated with different SNRs to form a video bitstream. DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0007] It is therefore an object of the present invention to provide a video coding concept that allows to more effectively suppress the negative effects on video quality due to drift caused, for example, by the formation of a video bitstream consisting of segments under switching between different video bitstream representations. [Means for solving the problem]

[0008] This object is achieved by the subject matter of the independent claims of the present application.

[0009] An embodiment according to a first aspect of the present invention provides a video decoder for decoding video from a data stream, the decoder being configured to decode an indication (e.g., gci_rasl_pictures_tool_constraint_flag) valid for a picture sequence of the video, indicating that RASL pictures within the picture sequence are coded in a manner that excludes at least one coding tool. This ensures that RASL pictures do not experience excessive drift when switching in an open GOP structure. For example, the indication may serve as a kind of promise, ensuring that the decoder knows that an open GOP switch by concatenating separately coded open GOP versions of the video coded at different spatial resolutions and / or different SNRs will not cause excessive drift in the RASL pictures. A further embodiment provides a video encoder for encoding video into a data stream, the video encoder being configured to encode the indication into the data stream. For example, an RASL picture may point to a picture that follows an intra-coded picture (e.g., a CRA) in a picture sequence in decoding order but precedes it in presentation order, and this picture may use a reference picture that precedes the intra-coded picture in decoding order. For example, the preceding picture may belong to a previous picture sequence, e.g., the intra-coded picture may be the first picture of the sequence in coding order. Due to the reference to a picture that precedes the intra-coded picture in decoding order, the above-mentioned drift artifacts or other types of artifacts may occur when switching resolutions at an intra-coded picture with respect to a previous segment of video that includes the previous intra-coded picture. Signaling the above indication in the data stream can ensure a decoder that a resolution switch in a picture sequence compared to a previous picture sequence does not involve excessive drift within the RASL picture.The decoder can then decide based on that indication whether a resolution switch is desirable or not.

[0010] A further embodiment according to the invention is a video decoder for decoding video from a data stream, configured to decode, for each picture of a sequence of pictures of the video, an indication (e.g. sps_extra_ph_bit_present_flag, gc_rasl_pictures_tool_constraint_flag) that indicates whether each picture is coded in a way that excludes at least one coding tool, and in particular to ensure that a cross-component linear model based prediction tool is used so that the effect of drift within RASL pictures is low.

[0011] Advantageous aspects of the present application are the subject matter of the dependent claims. Preferred embodiments of the present application are described below with reference to the figures. [Brief explanation of the drawings]

[0012] [Figure 1] 1 illustrates a video encoder according to an embodiment; [Figure 2] 1 illustrates a video decoder according to an embodiment; [Figure 3] 1 illustrates a block-based residual coding scheme according to an embodiment. [Figure 4] 1 illustrates a video data stream including two segments according to an embodiment. [Figure 5] 1 illustrates a method of operation of an optical flow tool according to an embodiment. [Figure 6] 1 illustrates how an encoding tool is applied according to an embodiment. [Figure 7] 1 illustrates how a temporal motion vector prediction tool operates according to an embodiment; [Figure 8] 1 illustrates the operation of a decoder-side motion vector refinement tool according to an embodiment. [Figure 9]1 illustrates how a cross-component linear model tool operates, according to an embodiment. [Figure 10] 1 illustrates how a luminance mapping and chroma scaling tool operates, according to an embodiment. [Figure 11] 10 illustrates another manner of operation of a luminance mapping and chroma scaling tool according to an embodiment. [Figure 12] 1 shows an example of prediction error in an open GOP scenario. DETAILED DESCRIPTION OF THE INVENTION

[0013] The following description of the drawings begins with presenting a description of an encoder and decoder of a block-based predictive codec for encoding pictures of video, to form an example of an encoding framework in which embodiments of the present invention may be incorporated. Each encoder and decoder is described with reference to Figures 1-3. Below, a description of embodiments of the inventive concepts is presented along with an explanation of how such concepts may be incorporated into the encoder and decoder of Figures 1 and 2, respectively, although the embodiments described in Figure 4 and thereafter may also be used to form encoders and decoders that do not operate according to the underlying encoding framework of the encoder and decoder of Figures 1 and 2.

[0014] FIG. 1 illustrates an apparatus for predictively encoding picture A12 into data stream A14, illustratively using transform-based residual coding. The apparatus, or encoder, is indicated using the reference symbol A10. FIG. 2 illustrates a corresponding decoder A20, i.e., apparatus A20, which is also configured to predictively decode picture A12′ from data stream A14 using transform-based residual decoding; the apostrophe is used to indicate that picture A12′ reconstructed by decoder A20 deviates from picture A12 originally encoded by apparatus A10 in terms of coding loss introduced by quantization of the prediction residual signal. While FIGS. 1 and 2 illustratively use transform-based predictive residual coding, embodiments of the present application are not limited to this type of predictive residual coding. This also applies to other details described with respect to FIGS. 1 and 2, as outlined below.

[0015] Encoder A10 is configured to subject the prediction residual signal to a spatial-to-spectral transform and encode the prediction residual signal into data stream A14, thereby obtaining the prediction residual signal therein. Similarly, decoder A20 is configured to decode the prediction residual signal from data stream A14 and subject the prediction residual signal to a spectral-to-spatial transform, thereby obtaining the prediction residual signal.

[0016] Internally, encoder A10 may include a prediction residual signal former A22, which generates a prediction residual A24 to measure the deviation of a prediction signal A26 from an original signal, i.e., picture A12. Prediction residual signal former A22 may be a subtractor, i.e., subtracting a prediction signal from the original signal, e.g., picture A12. Encoder A10 then further includes a transformer A28, which subjects prediction residual signal A24 to a spatial-to-spectral transform to obtain a spectral-domain prediction residual signal A24'. This spectral-domain prediction residual signal is then quantized by quantizer A32 and included by encoder A10. Thus, the quantized prediction residual signal A24'' is coded into bitstream A14. To this end, encoder A10 may optionally include an entropy coder A34, which entropy codes the transformed and quantized prediction residual signal into data stream A14. A prediction signal A26 is generated by a prediction stage A36 of encoder A10 based on a prediction residual signal A24" encoded into and decodable from data stream A14. To this end, prediction stage A36 may internally include an inverse quantizer A38, as shown in FIG. 1, which inverse quantizes prediction residual signal A24" to obtain a spectral-domain prediction residual signal A24'" that corresponds to signal A24' except for quantization losses, and then an inverse transformer A40 which subjects the latter prediction residual signal A24'" to an inverse transform, i.e., a spectral-to-spatial transform, to obtain a prediction residual signal A24"" that corresponds to the original prediction residual signal A24 except for quantization losses. A combiner A42 of the prediction stage A36 then recombines, such as by adding, the prediction signal A26 and the prediction residual signal A24"", to obtain a reconstructed signal A46, i.e., a reconstruction of the original signal A12. The reconstructed signal A46 may correspond to signal A12'. A prediction module A44 of the prediction stage A36 then generates a prediction signal A26 based on signal A46, for example by using spatial prediction, i.e., intra-picture prediction, and / or temporal prediction, i.e., inter-picture prediction.

[0017] Similarly, decoder A20 may be internally constructed from components that correspond to prediction stage A36 and are interconnected in a manner corresponding to the prediction stages, as shown in Figure 2. In particular, entropy decoder A50 of decoder A20 may entropy decode a quantized spectral domain prediction residual signal A24" from the data stream, after which inverse quantizer A52, inverse transformer A54, synthesizer A56 and prediction module A58 are interconnected and cooperate as described above with respect to the modules of prediction stage A36 to recover a reconstructed signal based on prediction residual signal A24" such that the output of synthesizer A56 is a reconstructed signal, i.e., picture A12', as shown in Figure 2.

[0018] Although not specifically described above, it is readily apparent that encoder A10 may set some encoding parameters, including prediction modes, motion parameters, etc., according to some optimization schemes, such as a method for optimizing some rate- and distortion-related criteria, such as coding cost. For example, encoder A10 and decoder A20 and corresponding modules A44 and A58 may each support different prediction modes, such as intra-coding and inter-coding modes. The granularity at which the encoder and decoder switch between their prediction mode types may correspond to the subdivision of pictures A12 and A12′, respectively, into coding segments or coding blocks. In units of these coding segments, for example, a picture may be subdivided into intra-coded blocks and inter-coded blocks. The intra-coded blocks are predicted based on their spatial, already coded / decoded neighbors, as outlined in more detail below. Several intra-coding modes may be present and selectable for each intra-coded segment, including directional or angular intra-coding modes, according to which each segment is filled by extrapolating neighboring sample values ​​along a particular direction specific to each directional intra-coding mode into each intra-coded segment. These intra-coding modes may also include one or more further modes, such as a DC coding mode, according to which prediction of each intra-coded block assigns a DC value to all samples in each intra-coded segment, and / or a planar intra-coding mode, according to which prediction of each block is approximated and determined to be a spatial distribution of sample values ​​described by a two-dimensional linear function over the sample positions of each intra-coded block, with driving of the slope and offset of the plane defined by the two-dimensional linear function based on neighboring samples.In comparison, inter-coded blocks may be predicted, for example temporally. For inter-coded blocks, motion vectors may be signaled within the data stream, indicating the spatial displacement of portions of previously coded pictures of the video to which picture A12 belongs, where the previously coded / decoded pictures are sampled to obtain a prediction signal for each inter-coded block. This means that in addition to the residual signal coding included in data stream A14, such as entropy-coded transform coefficient levels representing the quantized spectral-domain prediction residual signal A24", data stream A14 may also optionally code further parameters therein, such as coding mode parameters for assigning coding modes to various blocks, prediction parameters for some blocks, such as motion parameters for inter-coded segments, and parameters for controlling and signaling the respective division of pictures A12 and A12′ into segments. Decoder A20 uses these parameters to subdivide the picture in the same way as the encoder did, assign the same prediction modes to the segments, and perform the same prediction, thereby resulting in the same prediction signal.

[0019] FIG. 3 illustrates the relationship between, on the one hand, the reconstructed signal, i.e., the reconstructed picture A12′, and, on the other hand, the combination of the prediction residual signal A24″″ signaled in the data stream A14 and the prediction signal A26. As already indicated above, this combination may be additive. The prediction signal A26 is illustrated in FIG. 3 as a subdivision of the picture region into intra-coded blocks, exemplarily shown using hatching, and inter-coded blocks, exemplarily shown without hatching. The subdivision may be any subdivision, such as a regular subdivision of the picture region into rows and columns of square or non-square blocks, or a multi-tree subdivision of the picture A12 from a tree root block into multiple leaf blocks, such as a quadtree subdivision. FIG. 3 illustrates these combinations, where the picture region is first subdivided into rows and columns of tree root blocks, which are then further subdivided into one or more leaf blocks according to a recursive multi-tree subdivision.

[0020] Again, data stream A14 may have an intra-coding mode coded therein for intra-coded blocks A80, such that one of several supported intra-coding modes is assigned to each intra-coded block A80. For inter-coded blocks A82, data stream A14 may have one or more motion parameters coded therein. In general, inter-coded blocks A82 are not restricted to being temporally coded. Alternatively, inter-coded blocks A82 may be any block predicted from previously coded portions beyond current picture A12 itself, such as previously coded pictures of the video to which picture A12 belongs, or pictures of another view, or pictures of a hierarchically lower layer if the encoder and decoder are scalable encoder and decoder, respectively.

[0021] In Figure 3, the prediction residual signal A24"" is also shown as a subdivision of the picture domain into blocks A84. These blocks may be referred to as transform blocks to distinguish them from the coding blocks A80 and A82. In fact, Figure 3 shows that the encoder A10 and the decoder A20 may use two different subdivisions of pictures A12 and A12' into blocks, respectively: one into coding blocks A80 and A82, respectively, and another into transform blocks A84. Although both subdivisions may be the same, i.e., coding blocks A80 and A82 respectively, and may simultaneously form transform blocks A84, Figure 3 shows, for example, the case where the subdivision into transform blocks A84 forms an extension of the subdivision into coding blocks A80, A82, so that any boundary between the two blocks A80 and A82 overlaps with the boundary between the two blocks A84, or in other words, each block A80, A82 coincides with one of the transform blocks A84 or coincides with a cluster of transform blocks A84. However, these subdivisions may also be determined or selected independently of each other, so that the transform blocks A84 may alternately cross the block boundary between blocks A80 and A82. Thus, as far as the subdivision into transformation blocks A84 is concerned, similar statements are true as those made with regard to the subdivision into blocks A80, A82, i.e., blocks A84 may be the result of a regular subdivision of a picture region into blocks (with or without arrangement into rows and columns), a result of a recursive multi-tree subdivision of a picture region, or a combination thereof or any other kind of block result. As an aside, it is noted that blocks A80, A82 and A84 are not limited to being square, rectangular or of any other shape.

[0022] 3 further illustrates that the combination of prediction signal A26 and prediction residual signal A24'''' directly results in reconstructed signal A12'. Note, however, that according to alternative embodiments, more than one prediction signal A26 can be combined with prediction residual signal A24'''' to result in picture A12'.

[0023] In Fig. 3, the transform block A84 has the following significance: the transformer A28 and the inverse transformer A54 perform their transforms in units of their transform block A84. For example, many codecs use some kind of DST or DCT for all transform blocks A84. Some codecs allow to skip transforms, so for some transform blocks A84, the prediction residual signal is directly coded in the spatial domain. However, according to the embodiment below, the encoder A10 and the decoder A20 are configured in such a way that they support several transforms. For example, the transforms supported by the encoder A10 and the decoder A20 are: o DCT-II (or DCT-III) (DCT stands for Discrete Cosine Transform), o DST-IV (DST stands for Discrete Sine Transform), o DCT-IV, o DST-VII, o Identity Transformation (IT), Support.

[0024] Naturally, the transformer A28 supports all of the forward transform versions of those transforms, but the decoder A20 or inverse transformer A54 supports their corresponding backward or inverse versions, o Inverse DCT-II (or Inverse DCT-III), o Reverse DST-IV, o Inverse DCT-IV, o Inverted DST-VII, o Identity Transformation (IT), Support.

[0025] Subsequent descriptions provide further details on which transforms may be supported by encoder A10 and decoder A20. Note that in any case, the supported transform set may include only one transform, such as one spectral-to-spatial or spatial-to-spectral transform.

[0026] As already outlined above, Figures 1 to 3 are presented as an example in which the inventive concepts described further below may be implemented to form specific examples of encoders and decoders according to the present application. Thus far, the encoders and decoders of Figures 1 and 2, respectively, may represent possible implementations of the encoders and decoders described below in this specification. However, Figures 1 and 2 are merely examples. However, an encoder according to an embodiment of the present application may use concepts outlined in more detail below to perform block-based encoding of picture A12 that differs from the encoder of Figure 1, for example, in that the subdivision into blocks A80 is performed differently from the way illustrated in Figure 3. Similarly, a decoder according to an embodiment of the present application may perform block-based decoding of picture A12′ from data stream A14 using the coding concepts further outlined below, but may differ from decoder A20 of FIG. 2, for example, in that it does not support intra prediction, or in that it subdivides picture A12′ into blocks in a different manner than described with respect to FIG. 3, and / or in that it derives prediction residuals from data stream A14 in the spatial domain rather than the transform domain, for example.

[0027] As discussed, Figures 1-3 are merely intended to provide a rough overview of video codecs that may serve as the basis for embodiments of the present application outlined below. For example, VVC is one example of a video codec to which the video decoders and video encoders of Figures 1 and 2 may conform.

[0028] The following description is structured as follows: Preliminarily, VVC is used as an example video codec environment. Based on this example, the following description reports experiments investigating the impact of open GOP coding structures on general coding efficiency and the impact of segment switches on picture quality. Again, the embodiments described below are not limited to VVC, and the coding tools discussed with respect to these embodiments are not limited to those discussed with respect to VVC. However, these experiments and the presentation of their results provide motivation for the embodiments described below. Furthermore, the following description provides an overview of GOP coding structures and segmentation, and then presents constrained encoding that enables open GOP switches, such as open GOP resolution switches, and effectively suppresses drift associated with the switch. Afterwards, several embodiments of the present application that emerge from the discussion of VVC are presented.

[0029] The following provides an overview of the structure within a VVC bitstream and media segmentation for streaming. Media segments are typically aligned with Intra Random Access Point (IRAP) pictures using only intra-coding tools. IRAP pictures frequently appear in coded video bitstreams to enable functions such as seeking or fast-forwarding, and can also serve as switching points for adaptive HTTP streaming. Video-on-Demand (VoD) streaming systems typically align segments with IRAP picture durations. That is, IRAP pictures are typically placed at the start of a segment, and the desired segment duration determines the temporal distance between IRAP pictures. However, there are use cases, such as very low-latency streaming, where not all media segments contain IRAP pictures, allowing for the transmission of small segments without the need to wait for IRAP pictures, thereby reducing latency on the content generation side. Segment sizes may vary in length depending on the target application. For example, in VoD services, players can build larger buffers (e.g., 30 seconds) to compensate for throughput fluctuations, where segment sizes of up to a few seconds (e.g., 5 seconds) can be a reasonable design choice.[3] However, live services that require tighter end-to-end delays cannot use such large buffers on the client side, and therefore require more frequent switch points and short segments of 1 second or less.

[0030] Pictures between two IRAP pictures are typically encoded in a bidirectionally predicted hierarchical GOP structure with pre-presentation reordering, as introduced in AVC

[10] , because such a structure offers significant advantages in coding efficiency. A hierarchical GOP structure can be used for temporal scalability, where decoding all pictures up to a given layer corresponds to a given frame rate, and a corresponding temporal ID (Tid) value is assigned to each picture for a GOP size of 8 pictures, as shown in Figure 1. A GOP can be defined as all pictures from the first Tid0 picture in decoding order up to, but not including, the next Tid0 picture. A segment typically contains one or more GOP structures, depending on the IRAP period and GOP size. While HEVC allowed for a typical 16-picture GOP size due to the amount of reference picture slots in the decoded picture buffer (DBP), VVC's increased DPB capacity allows for hierarchical GOP sizes of up to 32 pictures.

[0031] A picture that follows an IRAP picture in decoding order but precedes it in presentation order is introduced in HEVC and is called a leading picture. Furthermore, these can be distinguished into random access skip leading (RASL) and random access decodable leading (RADL). While an RADL picture can only use reference pictures following the IRAP picture in decoding order, an RASL picture can also use reference pictures preceding the IRAP. An immediate random access (IDR) type IRAP picture can only have leading pictures that are RADL pictures, which reset the DBP and lead to a so-called closed GOP structure. Furthermore, on the other hand, a clean random access (CRA) type IRAP picture does not reset the DPB. Therefore, reconstructed pictures from before the CRA in decoding order can be used as references for future pictures, i.e., RASL pictures, enabling a so-called open GOP coding structure. RASL pictures offer improved coding efficiency compared to RADL pictures, but can be undecodable if no reference picture is available, e.g., during random access in the associated IRAP at the beginning of a segment without decoding the previous segment. A more detailed overview of the high-level syntax of VVC can be found in

[11] .

[0032] For example, FIG. 4 illustrates the formation of a video data stream from the concatenation of two consecutive segments with different resolutions, where the second segment employs an open GOP coding structure with reference pictures from the first segment. In particular, the referenced reference pictures are rectangles with arrows emerging from them in FIG. 4. The arrows themselves indicate a prediction interdependency, i.e., pointing from the reference picture to the picture it references. Each picture is associated with a specific temporal ID Tid, and as shown in the figure, the coding order is out of the picture output / presentation order. As shown in the figure, pictures with output order ranks 9 through 15 are RASL pictures that directly or indirectly reference the CRA pictures of segment 1, the segment to which they belong, and pictures from the previous segment, segment 0, primarily pictures with output order rank 8. For example, a video segment may also be referred to as a picture sequence, which may include, for example, one GOP.

[0033] If a reference picture for a RASL picture is located in a previous segment and a streaming client switches representations after such a previous segment, the client decoder will decode the RASL picture using a different variant of at least some of the reference pictures compared to the encoder. This situation, if content is not generated properly, can cause bitstream mismatch or significant inconsistency in the reconstructed RASL picture, and this drift can propagate to all RASL pictures up to the CRA picture, except for the associated CRA picture. Below, we describe the generation of appropriate content that allows the use of an open GOP structure while maintaining bitstream conformance during segment switches and avoiding undesired drift that can adversely affect visual quality during the switch.

[0034] Many inter-prediction tools in VVC exhibit various possibilities for drift when open GOP switching is performed, while the use of the tools is limited by compatibility constraints. We then analyze the possibility of drift of inter-prediction tools in VVC during open GOP resolution switching, and propose a constrained encoding method that overcomes the severe artifacts of open GOP resolution switching while ensuring VVC compatibility.

[0035] Regarding the potential for drift in VVC coding tools, the initial set of coding tools in VVC can be categorized as either inter-sample prediction, e.g., the usual block-based translational motion compensation (CMC) known from many previous versions of VVC, or a newly introduced inter-prediction mode in VVC called affine motion compensation (AMC), which decomposes a prediction block into smaller sub-blocks that are individually motion-compensated to emulate affine motion compensation

[12] . Optical flow prediction refinement (PROF) or bidirectional optical flow (BDOF), an optional component of AMC, is a further newly introduced inter-prediction tool in VVC that relies on optical flow-based methods to modify predicted samples to emulate sample-by-sample inter-prediction. When using such inter-sample prediction tools, if a different representation is used as a reference for reconstruction, the visual quality of the reconstructed picture will tend toward the visual quality of the representation and away from the visual quality of the original representation. However, such inter-sample prediction is less likely to cause visually disturbing artifacts, and instead results in gradual quality transitions in a given sequence of RASL pictures, as predicted source samples of a first visual quality are gradually updated through residual information of a second visual quality.

[0036] For example, Figure 5 illustrates an optical flow tool 300 and its functionality. An inter-predicted block 10c in picture 12 is shown. The inter-predicted block 10c is associated with a motion vector 302. That is, the motion vector 302 is signaled within the data stream for the inter-predicted block 10c. The motion vector 302 indicates the translational displacement of the inter-predicted block 10c from which a reference picture 304 is to be sampled / copied to generate a translational inter-predicted signal for the inter-predicted block 10c. When the optical flow 300 is used for the inter-predicted block 10c, the optical flow tool 300 improves the translational inter-predicted signal through optical flow-based analysis. More precisely, rather than simply sampling the reference picture 304 in the footprint of block 10c displaced according to motion vector 302, optical flow tool 300 uses an area slightly larger than the footprint of picture 10c in reference picture 304, i.e., area 306, to determine the inter-predicted signal of inter-predicted block 10c by examining gradients within area 306. In other words, each sample of the inter-predicted signal of block 10c is determined by using optical flow tool 300, possibly by using gradient-sensitive FIR filtering in addition to interpolation filtering in case the samples of block 10c displaced according to motion vector 302 are located at sub-pel positions in reference picture 304.

[0037] It should be noted that although FIG. 5 only shows one reference picture and one motion vector, the optical flow tool can also perform optical flow analysis with respect to two reference pictures and two motion vectors, with picture 12 being between both reference pictures and the picture containing inter-prediction block 10c.

[0038] Later, when describing embodiments of the present application, the optical flow tool 300 may form an example of an encoding tool that is excluded from encoding to avoid drift. Therefore, according to some embodiments described below, a video decoder and / or video encoder supports such an optical flow tool. As FIGS. 1 and 2 show possible implementations of a video decoder and encoder, the video decoder and encoder according to FIGS. 1 and 2 supporting the optical flow tool according to FIG. 5 may represent an exemplary basis for embodiments of the present application. In any case, different possibilities exist for determining whether to apply the optical flow tool 300 to the inter-prediction block 10c on the decoder side and the encoder side, respectively. For example, the optical flow tool 300 may be an inherently applied encoding tool. For example, whether the optical flow tool is applied to block 10c may depend on one or more encoding options signaled in the data stream of block 10c in relation to further encoding tools other than the optical flow tool. Alternatively, the optical flow tool 300 may be an inherently applied coding tool, where the decision of whether the optical flow tool 300 is applied to the block 10c depends on the size of the block 10c. Of course, both dependencies may apply. Alternatively, the optical flow tool 300 may be an explicitly applied coding tool, where a syntax element is encoded into the data stream that exclusively signals whether the optical flow tool 300 is applied to the block 10c. In other words, this syntax element is specific to the block 10c. Note that this syntax element may not be a flag or a binary value that simply toggles between non-application and application of the tool 350. Rather, the syntax element may be an m-ary syntax element, one of m of whose states is associated with, for example, application of the tool 350.Alternatively, the syntax element may be an m-ary syntax element, one of whose m states is associated with, for example, non-application of tool 350 .

[0039] In the embodiments described subsequently, the encoder signals to the decoder that certain coding tools, such as the optical flow tool 300, are excluded from encoding the RASL picture. This signaling, described later, need not function as an actual control of the availability of these coding tools for certain picture blocks or picture slices. Rather, this signaling or indication, presented in the embodiments described further below, may function as a kind of additional signaling or promise from the encoder to the decoder that these certain coding tools (or simply one coding tool), described further below, are excluded from encoding a particular picture, such as the RASL picture. In the latter case, the indication or signaling, described later, is redundant with or in addition to configuration signaling in the data stream that deactivates a particular coding tool, making it unavailable for application to picture blocks within a particular picture or picture slice. The encoder follows the guarantees provided by the configuration signaling and, if applicable, the syntax elements associated with block-based tool application, i.e., preventing a tool(s) from being used in the RASL picture. Thus, the optical flow tool may be a deactivatable coding tool that can be deactivated on a picture or slice basis by configuration data in the data stream for application to an inter-predicted block such as block 10c. For example, such configuration signaling may be configured by a slice header or picture header. Thus, whether or not the optical flow tool 300 is applied to a particular inter-predicted block 10c is determined based on the configuration signaling, and the above-mentioned explicit signaling or inherent decision regarding its application is made simply if the configuration signaling indicates that the optical flow tool 300 is activated, i.e., available, for the picture 12 (or slice of picture 12) of which the inter-predicted block 10c is a part.

[0040] The latter situation is shown again in FIG. 6, since the general possibilities for deciding whether or not to apply a particular coding tool are similar to those for the coding tools described herein. Thus, FIG. 6 shows a picture 12 currently being coded and a block 10 that is part of the picture 12. The picture block 10 is shown merely for illustrative purposes and is, in reality, just one picture block into which the picture 12 is divided. The block level to which the block 10 belongs could correspond, for example, to the block at which the intra / inter prediction mode decision is made, although other possibilities exist as well, such as smaller blocks than the latter. Possible examples of dividing the picture 12 into blocks such as the block 10 are described with reference to FIG. 3. A representative coding tool 350 is shown in FIG. 6. Whether this coding tool is actually applied to the block 10 can be controlled by the following circumstances: The application can depend on the size of the block, indicated by arrow 352, and / or the coding options signaled in the data stream for the given block 10, such as whether the block 10 is coded using intra or inter block mode. The latter dependency is indicated by arrow 354 in FIG. 6. Instead of or in addition to the block size dependency, a dependency on the block aspect ratio or tree partitioning level of the corresponding recursive multi-tree-based partitioning tree may be applied. The coding option may relate to one or more of the following: the block is intra-coded, the block is inter-coded, the block is bi-predicted, the block is bi-predicted with an equally spaced opposite reference picture, etc. Both information entities, i.e., block size / aspect ratio / partitioning level and coding option, are signaled in the data stream, i.e., both the block size / aspect ratio / partitioning level via partitioning information regarding the partitioning of picture 12 into blocks including block 10, and coding options specific to block 10, for example. Instead of relying on an implicit block-level determination, explicit signaling in the form of syntax element 353 may be used to control the application of tool 350 with respect to block 10.The syntax elements are specific to the block 10 and are specific with respect to the tool decision of the tool 350. This application decision 356 is made at the block level, but at a larger level, the application decision may further depend on activation decisions 358 made at a larger level, such as with respect to the entire picture 12 or to slices into which the picture 12 is subdivided along the block coding order in which the picture 12 is coded into a data stream. The activation decisions 358 may be controlled by the aforementioned configuration signaling, such as settings in the slice header of the slice to which the block 10 belongs, or settings in the picture header of the picture 12, or settings in a parameter set associated with the picture 12, such as a picture parameter set. Not all of the decisions 356 and 358 apply. While it is possible that none of them apply, the indications described below act like configuration signaling, as a kind of promise or redundant signaling, indicating the activation or deactivation of a particular coding tool 350 for a particular slice, picture, or RASL picture.

[0041] Thus, the encoding tool 350 of FIG. 6 may be the optical flow tool 300 of FIG. 5, but FIG. 6 representatively illustrates any of the encoding tools that are subject to encoder constraint determination as further described below.

[0042] The second coding toolset of VVC is used for syntax (i.e., model parameter) prediction from picture syntax or samples. Like previous versions, VVC enables block-based motion vector (MV) prediction using temporal MV candidates from so-called co-located reference pictures via Temporal Motion Vector Prediction (TMVP)

[13] . This capability was extended in VVC by introducing a finer-granularity sub-block-based TMVP variant (SBTMVP), which adds a displacement step when finding the corresponding motion information in the co-located reference picture.

[0043] Figure 7 shows a temporal motion vector prediction tool 500. This tool 500 is another example of the encoding tool 350 of Figure 6 and a further example of an encoding tool described further below with respect to the encoder constraint indication. The temporal motion vector prediction tool is for predicting a motion vector 508 for an inter-predicted block 10c of picture 12 based on a motion vector 510 associated with a block 506 in a reference picture 502. While the tool 500 could use the motion vector of a co-located block in the reference picture 502 as a predictor 508, according to the example of Figure 7, the tool 500 first derives a displacement vector 504, such as a spatial prediction motion vector, for block 10c and uses this displacement vector 504 to locate the "co-located block 506" in the reference picture 502. The motion vector 510 of this co-located block 506 is then used for the candidate temporal prediction motion vector 508. 7 further illustrates that the tool 500 may be operable to determine the temporal predictor motion vector 508 only for insertion into the motion vector candidate list 512 for the block 10c from which one motion vector predictor is ultimately selected, such as by using an index signaled in the data stream for the block 10c. Alternatively, the list 512 may be interpreted in a particular way that leads to a particular ordering of the motion vector candidates in the list 512, with the one with the highest rank simply ultimately chosen / selected for use in the inter-predicted block 10c.

[0044] It should be noted that the motion vector prediction tool of Figure 6 is to be understood as being broad enough to also cover the case of temporal motion vector prediction in the merge sense, i.e. providing a motion vector predictor from a co-located block together with other motion prediction settings such as reference picture index.

[0045] All options described with respect to Figure 6 are available to tool 500 to determine whether tool 500 actually applies to block 10c. More precisely, tool 500 may be an example of encoding tool 350 of Figure 6, in which case explicit signaling in the form of syntax element 353 may be used to control the application of tool 500 with respect to block 10c. That is, the index signaled in the data stream of block 10c controls whether or not to select TMVP candidate 508 from list 512. If not selected, tool 500 remains disabled with respect to block 10c, which is interpreted as tool 500 not applying to block 10c. However, higher-level configuration signaling via decision 358 may be used to more globally deactivate tool 500, such that for blocks residing within the region (picture or slice) for which tool 500 is signaled to be deactivated, list 512 is interpreted in any case to exclude TMVP candidates 508, and the list index no longer serves as the block-level decision control discussed above. The encoder can decide which method to use in order to adhere to the promise given to the decoder to avoid RASL picture drift, as taught herein.

[0046] Further tools in the second set mentioned above can be characterized as sample-to-syntax prediction tools. An entirely new inter-prediction tool introduced in VVC is decoder-side motion vector refinement (DMVR), which refines the accuracy of MV in bidirectional prediction based on the mirroring property of two reference pictures with equal and opposite temporal distances to the current picture.

[0047] FIG. 8 shows a decoder-side motion vector refinement tool 400. When applied to an inter-predicted block 10d of picture 12, the tool 400 refines the coded / signaled motion vector 402 in the data stream of this block 10d by using a best match search to inter-predict this block 10d from a reference picture 404. The best match search may be performed at the highest resolution supported by the decoder and encoder, such as 1 / 16 pel resolution. The signaled motion vector may be of lower resolution and merely serves to "substantially" indicate the refined motion vector finally determined by the DMVR tool. There are various possibilities regarding what needs to be matched with the reference picture to perform the best match search. One possibility is to use an already decoded portion adjacent to the inter-predicted block 10d. This portion undergoes motion vector displacement using the signaled motion vector and its surrounding motion vector candidates, and the best matched candidate is selected as the refined motion vector. Alternatively, block 10d may be a bidirectionally predicted block, and there may be a pair of signaled motion vectors to be refined by tool 400. That is, if inter-predicted block 10d were a bidirectionally predicted block. Picture 12 may be present between the reference pictures in presentation order, i.e., the reference pictures may be temporally located before and after picture 12. Optionally, the two reference pictures are temporally spaced equally apart from picture 12. Tool 400 may be dedicated to this case. That is, tool 400 would essentially be activated at the block level in response to a block coding option indicating that block 10d is bi-predicted based on a reference picture that is equally spaced from current picture 10 and has picture 10 between them. In that case, the signaled motion vector pair, one of which is vector 402, is refined by performing a best match search among motion vector pair candidates that include the signaled motion vector pair.The best match search can be performed, for example, by testing the similarity between sampled reference pictures in parts called motion vector pair candidates. The motion vector pair candidates can be limited to those in which one motion vector candidate for a reference picture deviates from the corresponding signal motion vector for this reference picture in the opposite way that the other motion vector candidate for another reference picture deviates from the other signal motion vector for this other reference picture. SAD or SSD can be used to obtain the similarity. The best-matching motion vector pair candidate is then used as the signal motion vector replacement 406. That is, vector 402 of reference picture 404 is replaced by 406, and the other signal vector of the other reference picture is replaced by the other vector of the motion vector pair candidate. Other possibilities may exist as well. For example, both signaled motion vectors of bi-predicted block 10d can be individually refined by performing a best match search of the average of sampled patches in the two reference pictures in the signaled motion vector and its two surrounding signaled motion vectors.

[0048] Regarding the decision whether to apply the MVR tool 400 to block 10d, all the alternatives described with respect to Figure 6 are applicable, i.e. the MVR tool 400 may be the encoding tool 350.

[0049] A newer tool in VVC is the Cross-Component Linear Model (CCLM). This allows intra-prediction of the chroma components of a block from their respective luma components using a linear model. The model parameters are derived from the reconstructed luma sample values. The linear model is then applied to the subsampled luma samples rec' L is converted to a saturation prediction by the following means: P(i,j)=a·rec' L (i,j)+b where the parameters a and b are derived from adjacent luma and chroma samples as follows: Xl and X s and Y are the averages of the two largest and two smallest adjacent samples, respectively. l and Y s and denote the average of the corresponding saturation sample pairs. The parameters are derived as follows: a=(Y l -Y s ) / (X l -X s ) b=Y s -a·X s Because the parameter derivation process only considers extreme values ​​of neighboring samples, this process tends to produce extensive drift even when there is a single-sample drift outlier in a neighboring block. Furthermore, due to the linear model, luminance drift can be amplified when a is large. For other intra-prediction modes that consider all neighboring samples, drift propagation is less pronounced and cannot be linearly amplified. Due to this inherent instability, special care must be taken when applying this mode to applications where constrained drift is acceptable, such as open GOP switching in HTTP adaptive streaming. Additionally, in the context of the described application, drift can only occur in RASL frames, i.e., motion-predicted frames. When an encoder decides to use CCLM, i.e., intra-prediction mode, this is usually due to the lack of a suitable motion-compensated predictor, implying regions of high temporal activity. In such regions, the expected reconstruction drift of open GOP switching is expected to be large, further contributing to the instability effects discussed.

[0050] FIG. 9 illustrates the operating mode of the cross-component linear model tool 100. The tool 100 is applied to a block 10a. The luma component of this block 10a is reconstructed by decoding 122 a residual signal from the data stream using any type of prediction 120 and combining the prediction signal with the luma residual signal. The tool 100 aims to predict the chroma component(s) based on the reconstructed luma component 124. This is done using a linear model or linear mapping 106. This linear mapping 106 uses a scalar linear function to predict, for each sample 126 of the block 10a, the chroma component of the sample 126 based on the reconstructed luma component of the sample 126. The linear parameters of the scalar linear function, i.e., a and b, are determined for the block 10a by analyzing the statistics of the luma and chroma components of already reconstructed samples in the neighborhood 112 of the block 10a, across the entire block. In particular, the statistical analysis 124 performed by the tool 100, shown at 128 in FIG. 9, can determine, for each component, the external luma and chroma values ​​occurring within the reconstructed samples of the neighborhood 112. For example, the average of the two largest luma values ​​and the average of the two smallest luma values ​​may be used. The same is done for the chroma component to be predicted. Based on the resulting four average values, a measure of the range of luma and chroma values ​​within the neighborhood 112 is determined, and the ratio between them is used as the slope of the scalar-linear function of the linear mapping 106. The average of the minimum luma value minus the slope times the average of the minimum chroma value is used to determine the intercept of the scalar-linear function. The parameter derivation 108 thus performed generates a scalar-linear function, and each luma sample of the reconstructed luma component 124 is used to predict the value of the corresponding chroma sample in the block 10a, thereby generating the inter-chroma component prediction signal for the block 10a.Although not shown in FIG. 9, the data stream may possibly comprise an encoded chroma residual signal to correct the inter-chroma component prediction signals of the chroma components C1 and / or C2.

[0051] Again, tool 100 is another example of coding tool 350 of FIG. 6 . In other words, whether a coding tool is applied to block 10 a of a picture can be determined according to any of the options described with respect to FIG. 6 . It is noteworthy that VVC does not provide any means for deactivating tool 100 picture-wide, or at least slice-wide, although, according to the embodiments described below, such signaling is proposed, thereby avoiding harmful drift caused by tool 100 itself. More precisely, tool 100 may be an example of coding tool 350 of FIG. 6 , in which explicit signaling in the form of syntax element 353 is used to control the application of tool 100 with respect to block 10 a. The syntax element can be, for example, a flag that switches tool 100 on or off for block 10. Regarding the discussion of coding constraint indications described with respect to the embodiments described below, two possibilities exist. That is, it can only inform the receiver, i.e., decoder, of the fact that all syntax elements 353 for a particular picture 12 indicate non-application of tool 100; alternatively, it can also act as configuration signaling to deactivate tool 100 for a picture 12 as a result of the data stream not conveying syntax elements 353 for blocks 10 within that picture 12. For example, taking VVC as an example, there is no configuration signaling to deactivate tool 100 at the granularity of a picture or slice. In VVC, such configuration signaling exists only to control activation of tool 100 for a picture sequence. Therefore, deactivation taking RASL into account is not feasible.

[0052] Furthermore, a new tool is introduced in the loop filtering stage of VVC, called Luminance Mapping and Chroma Scaling (LMCS), where chroma sample values ​​are scaled using parameters derived from the luma samples, as shown in Figure 10.

[0053] Here, the chroma-to-luminance dependency still exists, but it is less pronounced than in CCLM. In the chroma scaling portion of the process, the transformed and dequantized chroma residual is scaled according to model parameters derived from the luma samples of adjacent virtual pipeline data units (VPDUs). CCLM relies on samples from adjacent VPDUs to reduce pipeline latency. However, in LMCS, all adjacent luma samples are considered, allowing for averaging of drift outliers across adjacent samples in a VPDU. The model parameters are also used to scale the residual signal. The residual signal does not aggregate drift, but is directly signaled. For these reasons, the tool is much less likely to amplify drift, but it still needs to be considered when encoding in drift-control applications.

[0054] The operation mode of the LMCS tool 200 is shown in Figure 11. The idea here is to perform luma component prediction 202 and luma component residual decoding 204 for a given picture 12 on an encoding luma tone scale 208 rather than a presentation luma tone scale 210. More precisely, the reconstructed luma values ​​may represent the luma component of the reconstructed signal of the picture 12 on a linear scale for a particular bit depth, while the luma tone mapping 212 may map this scale 210 to the coding scale 208 using a particular tone mapping function, such as a linear tone mapping function for the picture or some other tone mapping function. The tone mapping function may be signaled in a data stream, such as a picture parameter set for the picture 12. The function is appropriately determined by the encoder. Thus, the inter-predicted signal obtained by inter prediction 202 for block 10b of the picture 12 undergoes luma tone mapping 212 to obtain the reconstructed luma component of the picture 12 before being combined with the residual signal 204 on the coding scale 208. For intra-predicted blocks, inter prediction 206 is used. Intra prediction is performed in the coding scale 208 domain. A further goal of the tool 200 is to control the quantization error of the chroma components according to a luma tone mapping 212. That is, the quantization error of the chroma components is controlled separately for each block and adapted to the influence of the luma component by the luma tone mapping 212. For this purpose, a chroma residual scale factor 216 is determined for block 10b of picture 12 from an average 220 of the coded luma tone scale versions of the reconstructed luma components of picture 12 within a neighborhood 222 of picture block 10b. A chroma residual signal 224 decoded for picture block 10b from the data stream is scaled 226 according to the thus determined chroma residual scale factor 216, and this scale is used for correction 228 of picture block 10b and for intra chroma prediction signal 230.The intra chroma prediction signal 230 can use the same prediction tools, or a subset thereof, as compared to the luma component. By using the neighborhood 222 to determine the average 220, the luma and chroma components of block 10b can be reconstructed in parallel instead of serially. The reconstructed luma and chroma components of picture 12 then undergo inverse luma tone mapping 240 to obtain the final reconstructed result of picture 12 and to obtain the basis for the next picture to be encoded / decoded, i.e., to serve as a reference picture for this subsequently encoded / decoded picture.

[0055] The same notes are valid with respect to encoding tool 200 as were made with respect to the other encoding tools described with respect to previous figures: encoding tool 200 is an example of encoding tool 350 of Figure 6, or the options for determining application of this encoding tool to particular blocks described with respect to Figure 6 may be used. As an example, block-by-block decision 356 may be omitted with respect to encoding tool 300, but configuration signaling may be used to control application on a picture-by-picture or slice-by-slice basis.

[0056] Errors in predicted motion vectors by inter-syntax and inter-sample predictors are relatively likely to cause severe artifacts in subsequent inter-sample predictors that use these incorrectly predicted motion vectors as spatial or temporal motion vector candidates. This is particularly true for (SB)TMVP, which exhibits the most noticeable artifacts during open GOP transitions. It is also true for DMVR, since incorrect motion vector errors can significantly propagate to subsequent pictures. However, the same applies to other prediction models, such as CCLM and / or LMCS, that operate based on parameters derived from reconstructed sample values. Figure 12 shows the impact of common syntax or parameter prediction errors on the visual and objective quality of RASL pictures using a GOP size of 32 pictures in conventional open GOP coding. It is clear that RASL pictures suffer from significant artifacts in the luma and chroma components of the reconstructed picture.

[0057] A third issue with open GOP switching can arise from the use of adaptive parameter sets (APSs) in VVC, which store the adaptive loop filter (ALF) filter coefficients, the luma mapping with chroma scaling (LMCS) parameters, and the quantization scaling list. A RAS picture may point to an APS transmitted before its respective CRA in decoding order, which is available during continuous decoding. However, this is unavailable during random access at the CRA picture, since the associated RAS picture is dropped in this case. Therefore, open GOP resolution switching can result in references to missing APSs that crash non-error-tolerant decoders, or visual artifacts if the wrong APS parameters are used with coincident identifier values. As with syntax predictors, this issue is likely to cause visual glitches, ranging from complete decoder failure.

[0058] To avoid the above problems when performing open GOP resolution switching, a constrained VVC encoding method can be used, which consists of three pillars described below.

[0059] First, a RASL picture associated with a CRA is constrained to prevent pictures preceding the CRA in decoding order from being selected as co-located reference pictures for performing inter-syntax prediction, i.e., (SB)TMVP. This ensures that the exact same reference pictures and motion information are used as on the encoder side, preventing syntax prediction errors due to inaccurate source motion information from previous segment pictures. In a possible implementation, the first RASL picture in decoding order is restricted to use only its associated CRA picture as a co-located reference picture, which naturally hosts only zero motion vectors, while subsequent RASL pictures have access to non-zero temporal MV candidates between the first RASL picture in decoding order and subsequent pictures. Regarding sample-to-syntax prediction tools, DMVR is disabled for all RASL pictures that have an active reference picture preceding the associated CRA in decoding order. In another alternative, DMVR is disabled for all RASL pictures regardless of reference picture, in a further alternative, DMVR is disabled only for RASL pictures that serve as co-located reference pictures for subsequent pictures, and in yet another alternative, DMVR is disabled for all RASL pictures except for RASL pictures that are not used as references because they are the highest temporal layer, so that erroneous sample values ​​of reference pictures from previous segments that are different from encoder-side samples of other RASL pictures or samples affected by drift do not cause errors in sample-to-syntax prediction.

[0060] To ensure VVC conformance after a segment switch, further tools must be restricted for all RASL pictures that have a reference picture preceding the associated CRA. This means that the use of the optical flow-related tools BDOF and PROF is disabled. Alternatively, for simplicity, BDOF and PROF can be disabled for all RASL pictures. Furthermore, VVC's new feature of independently coded subpictures within a video, useful for 360-degree viewport-dependent video streaming, must be disabled for RPR to be used. All of the above tool constraints are also part of the conformance constraints defined in the VVC specification to enable RPR. Beyond the conformance constraints related to the use of RPR in VVC, prediction techniques that use parameter estimation from reconstructed samples can also introduce noticeable artifacts, requiring further tool constraints. Therefore, because the current VVC syntax only allows disabling CCLM on a sequence-by-sequence basis, which significantly reduces overall coding efficiency, our implementation disables CCLM through a block-by-block constraint in the encoder-side search algorithm. This effectively enables drift to be reliably avoided on the encoder side, but the drift is not easily visible on the decoder side without extensive low-level analysis. Also, because the tool is enabled but not used, unnecessary bits (i.e., coding unit-level flags for CCLM usage such as cclm_mode_flag or cclm_mode_idx) are being sent to signal the encoding's decision not to use the tool.

[0061] Second, in an open GOP coding structure, i.e., when a CRA picture is followed in decoding order by its associated RASL picture, the APSs required for all pictures must also be present in the segment. Note that when randomly accessing an open GOP stream, no such constraint is necessary, and an RASL picture can reference an APS transmitted in the bitstream before the associated CRA picture in decoding order. This reference is not an issue when randomly accessing such a CRA picture, because the RASL picture is dropped. Also, if no switching is performed, such APSs are available for continuous decoding. However, these APSs may not be available in streaming with open GOP switching and should therefore not be referenced. In our implementation, the processes related to the ALF, LMCS, and quantization scaling list are reset in a manner similar to that of a closed GOP coding structure.

[0062] Third, from the perspective of the VVC high-level syntax, the individual encoding of variants within the bitstream ladder must be coordinated, keeping in mind the target of open GOP switching at the decoder side. Therefore, the sequence parameter sets (SPS) of all segment variants must be adjusted so that a segment switch does not trigger the start of a new coding layer video sequence due to an SPS change. For example, when properly adjusted, the SPS should indicate the maximum resolution in the bitstream ladder, matching block size and saturation format, appropriate matching level indicators, and related constraint flags such as gci_no_res_change_in_clvs_constraint_flag, sps_ref_pic_resampling_enabled_flag, and sps_res_change_in_clvs_allowed_flag with appropriate configuration to enable the use of RPR at the decoder side. Devices with capabilities lower than those required for the specified maximum resolution or level should use the adjusted SPS through a system mechanism.

[0063] As is evident from the tool constraints discussion above, VVC's RPR is designed to be constrained to limit its implementation and runtime complexity. A key aspect of this complexity consideration is that the memory bandwidth required to access scaled reference samples when using RPR must be within acceptable limits and not significantly higher than when not using RPR. Each VVC-encoded picture is accompanied by a so-called scaling window, which is used to determine the scaling factor between two pictures. To set a limit on the memory bandwidth requirements of RPR, the relationship between the scaling window of a picture using RPR and the scaling window of its reference picture is constrained to allow up to 8x upscaling and 2x downscaling. In other words, assuming each scaling window matches the picture size of that representation, RPR is permitted when switching to a representation with 8x picture size. However, for downscaling, RPR can only be used if the picture size is reduced by more than half in each dimension.

[0064] Typically, in adaptive streaming scenarios, up-switching is performed in a progressive manner, i.e., gradually increasing the resolution or quality. However, with regard to down-switching, it is possible that the player may switch to the lowest quality to avoid buffer underrun when its buffer is insufficient. This means that down-switching is unlikely to occur gradually. One way to mitigate this limitation of RPR in VVC is to encode the lowest quality representation in a closed GOP structure to serve as a fallback if the picture size is reduced by less than half during such non-progressive down-switching events.

[0065] The following describes embodiments of decoders and encoders that support one, several, or all of the above-described encoding tools 100, 200, 300, 400, and 500. The decoders and encoders described below can be implemented in a manner consistent with FIGS. 1 and 2. While tools 100, 200, 300, 400, and 500 are described above primarily with respect to the decoder side, it is clear that the description of the corresponding tools can easily be transferred to the encoder side, with the difference being that the encoder inserts relevant information into the data stream rather than decoding it from the data stream. Each supported encoding tool represents one encoding tool 350. An encoding tool that uses block-based explicit syntax element control 353 involves the encoder encoding syntax elements on a block basis and the decoder decoding them from the data stream. Encoding tool 100 may be the only one that uses this explicit block-based syntax element. Other encoding tools 200, 300, 400, and 500 may use the inherent block-based adaptation decision 356 in conjunction with picture-based or slice-based configuration signaling to deactivate the tool entirely.

[0066] The following embodiments relate to an indication or signaling that indicates to a decoder whether a specific coding constraint regarding the use of one or more of the above-mentioned sets of coding tools is being observed. The encoder signals this indication in the data stream and limits its encoding accordingly by adhering to the corresponding encoding constraint. The decoder uses the indication and interprets it as a guarantee or instruction for drift limitation in the case of segment switching. According to alternative embodiments, the indication / signaling described below can also be used to actually deactivate one or more coding tools. For example, VVC has so far not been able to deactivate tools 100 on a picture or slice basis. In addition to the promise function, the indication / signaling described below can take on the function of configuration signaling to deactivate tools 100 for specific pictures / slices. The block-based syntax element for block-based decision 356 can then be omitted, and no encoding or decoding to or from the data stream can occur.

[0067] Next, a combination is presented that enables open GOP resolution switching using the proposed constraint signaling. This means that a CRA RASL picture can be decoded with acceptable drift when performing a stream switch because certain coding tools are not active for the RASL picture. While the state of the art allows such indication for some tools that are part of the presented method, such as TMVP, SBTMVP, BDOF, PROF, and DMVR, the presented method has notable additional constraints that are necessary to avoid severe artifacts due to sample-to-syntax prediction tools, namely CCLM and / or LMCS. Therefore, an encoder needs to be able to indicate in the bitstream that such tools are not active for a particular picture, i.e., a CRA RASL picture.

[0068] VVC provides an extension mechanism to add bit flags to the Picture Header (PH) and Slice Header (SH) syntax in a backward-compatible way. For this purpose, each SPS indicates the number of additional bits for such purposes in the PH or SH syntax. These must be parsed when reading the syntax, and a derivation is used to assign this additional bit to a flag or variable value. The following table shows the respective SPS and PH syntax along with their respective semantics. The SH syntax and semantics are similar to the PH syntax and semantics. [Table 1] sps_extra_ph_bit_present_flag[i] equal to 1 specifies that the i-th extra bit is present in the PH syntax structure that references the SPS. sps_extra_ph_bit_present_flag[i] equal to 0 specifies that the i-th extra bit is not present in the PH syntax structure that references the SPS.

[0069] The variable NumExtraPhBits is derived as follows: NumExtraPhBits=0 for(i=0;i<(sps_num_extra_ph_bytes*8);i++) if(sps_extra_ph_bit_present_flag[i]) (1) NumExtraPhBits++ [Table 2] ph_extra_bit[i] may have any value. Decoders conforming to this version of this specification shall ignore the presence and value of ph_extra_bit[i]. Its value has no effect on the decoding process specified in this version of this specification.

[0070] While an unaware decoder will at least be able to parse and decode the bitstream correctly, a decoder that is aware of the meaning of the additional bits can further interpret the indication of the additional bits and act accordingly. For example, it can advise the client that the constraints of the presented method are met and that stream switching is possible without significant drift. Similarly, file format packagers, HTTP streaming servers, and even RTP streaming servers can take this bitstream indication into account when packaging, presenting, and delivering content in a manner that uses bitstream switching.

[0071] An embodiment of the present invention is to carry an indication of the presented method in an extra bit of the PH or SH syntax of an RASL picture or an associated CRA picture as follows: In SPS semantics, a specified extra bit flag is identified through the index i of sps_extra_ph_bit_present_flag[i] as indicating the presence of a PH / SH extra bit flag indicating the presented method. For example, the presence of the first extra bit in the PH can be identified by the first SPS PH extra bit (i=0) of the first SPS PH extra byte as follows:

[0072] The value of the variable ConstraintMethodFlagPresentFlag is set equal to sps_extra_ph_bit_present_flag[0]. Note that index 0 is used, but another index could be used instead. That is, the bits in sps_extra_ph_bit_present_flag[i] are chosen to mean that the RASL picture is constrained with respect to the tool used.

[0073] In the PH semantics, each variable that indicates the flavor of the proposed constraint method is derived as follows:

[0074] The value of the variable ConstrainedRASLFlagEnabledFlag / ConstrainedCRAFlagEnabledFlag is set equal to (ConstraintMethodFlagPresentFlag&&ph_extra_bit[0]). Note that although index 0 is used, depending on the value indicated by sps_extra_ph_bit_present_flag[i] and which index is used to constrain the RASL picture ph_extra_bit[j], the j-th flag of the extra flags in PH indicates whether the constraint is applied to the RASL picture or not.

[0075] Alternative 1 (RASL picture carries signaling): If ConstrainedRASLFlagEnabledFlag is equal to 1, the current picture is encoded without using CCLM. The PH / SH control flags and the sequence level constraint flags for BDOF, DMVR, PROF, (SB)TMVP, and LMCS are already in VVC Version 1, but CCLM is missing picture or slice scope control flags.

[0076] Alternative 2 (carrying signaling in associated CRA picture): If ConstrainedCRAFlagEnabledFlag is equal to 1, the RASL picture associated with the current picture is encoded without using TOOLSET, where TOOLSET refers to CCLM and / or LMCS and / or BDOF and / or PROF and / or DMVR and / or RASL pictures, without using any collocated pictures for TMVP preceding (sb) the current picture (i.e., the CRA picture).

[0077] An alternative embodiment similar to the above can be constructed for SH signaling for slices of RASL or CRA pictures.

[0078] In another alternative embodiment, the above constraints are indicated as CVS, CLVS and / or bitstream properties / constraints by adding a general constraint flag (e.g., general gci_rasl_pictures_tool_constraint_flag or gci_cra_pictures_tool_constraint_flag) with the same meaning to either the DCI, VPS, or SPS, as follows:

[0079] Corresponds to Alternative 1: If gci_rasl_pictures_tool_constraint_flag is equal to 1, it specifies that all RASL pictures in the OlsInScope have their ConstrainedRASLFlagEnabledFlag equal to 1. If gci_rasl_pictures_tool_constraint_flag is equal to 0, no such constraint is imposed.

[0080] Corresponds to Alternative 2: If gci_cra_pictures_tool_constraint_flag is equal to 1, it specifies that all CRA pictures in OlsInScope have their ConstrainedCRAFlagEnabledFlag equal to 1. If gci_cra_pictures_tool_constraint_flag is equal to 0, no such constraint is imposed.

[0081] That is, when such a general constraint flag is set, all RASL pictures associated with CVS, CLVS, and / or CRA in the bitstream are encoded without using TOOLSET, where TOOLSET refers to CCLM and / or LMCS and / or BDOF and / or PROF and / or DMVR and / or RASL pictures, without using any co-located pictures for TMVP preceding (sb) the current picture (i.e., the CRA picture).

[0082] In another alternative embodiment, the above constraints are indicated in a PPS extension syntax: RASL pictures associated with a CRA may reference a PPS that indicates that the above constraints are active, while other pictures in the bitstream reference PPSs that do not indicate the above constraints.

[0083] In another alternative embodiment, the above constraint signaling is performed through an SEI message either for the CRA picture, or for the associated RASL picture, or for the entire coding layer video sequence of the picture.

[0084] In another alternative embodiment, the above constraint signaling is used to conditionally transmit CCLM flags at the coding unit level.

[0085] In another alternative embodiment, the above constraint signaling is not applied to all RASL pictures associated with the CRA, but is limited to a subset of RASL pictures depending on the actual tool, taking into account the tolerable drift and the resulting coding efficiency penalty.

[0086] DMVR: (same as above description): DMVR is disabled for all RASL pictures that have an active reference picture before the associated CRA in decoding order. Note that the current picture can have active reference pictures (actually used for prediction) in its reference picture list (RPL) and inactive reference pictures that are not used to predict the sample or syntax of the current picture but are subsequent pictures (in decoding order) and therefore not ready to be deleted from the decoded picture buffer (DPB). In another alternative, DMVR is disabled for all RASL pictures regardless of the reference picture; in a further alternative, DMVR is disabled only for RASL pictures that serve as co-located reference pictures for subsequent pictures; in yet another alternative, DMVR is disabled for RASL pictures that do not belong to the highest temporal layer. The alternatives may be combined.

[0087] - BDOF&PROF: Disabled for all RASL pictures, or only for RASL pictures that have an active reference picture before the associated CRA in decoding order.

[0088] According to an embodiment, this indication is signaled in the data stream in the form of an SEI message. As mentioned above, the SEI message may be valid for all pictures of a picture sequence, e.g., a coded video sequence, CVS. For example, the SEI message may be signaled within the sequence. Thus, a decoder can infer from the presence of the SEI message or from the indication within the SEI message that all RASL pictures of the sequence have been coded in a manner that excludes a predetermined set of coding tools. For example, according to an embodiment, the set of coding tools includes at least a cross-component linear model-based prediction tool 100 and a decoder-side motion vector refinement tool 400.

[0089] Further embodiments of the above invention are described below. D1.1. A video decoder for decoding video from a data stream, comprising: The video decoder is configured to decode from the data stream an indication (e.g., gci_rasl_pictures_tool_constraint_flag) that is valid for a picture sequence of the video and indicates that RASL pictures within the picture sequence have been coded in a manner that excludes a predetermined set of one or more coding tools (e.g., as a kind of promise that enables the decoder to recognize that open GOP switching by concatenating separately coded open GOP versions of the video coded at different spatial resolutions and / or different SNRs will not cause excessive drift in the RASL pictures).

[0090] D1.2. The set of one or more coding tools: The video decoder of any preceding embodiment D1.#, comprising a cross-component linear model-based prediction tool (100).

[0091] D1.3. Pursuant to the cross-component linear model-based forecasting tool, A video decoder as described in embodiment D1.2, wherein the chroma component (102) of a picture block (10a) is predicted from the luma component (104) of the picture block (10a) using a linear model (106) whose parameters are determined (108) from extreme luma and chroma values ​​(110) in an already decoded neighborhood (112) of the picture block.

[0092] D1.4. The set of one or more coding tools: The video decoder of any preceding embodiment D1.#, comprising a luma tone mapping and chroma residual scaling prediction tool (200).

[0093] D1.5. According to said luma tone mapping and chroma residual scaling prediction tool, Luminance component prediction (202) (e.g., inter-prediction) and luma component residual decoding (204) for a given picture (12) are performed with an encoded luma tone scale (208), whereupon a presentation luma tone scale (210) is mapped by a luma tone mapping (212) to obtain an encoded luma tone scale version (214) of the reconstructed luma component of the given picture; a chroma residual scale factor (216) for a picture block (10b) of the given picture is determined from an average (220) of the coded luma tone scale versions of the reconstructed luma component of the given picture within a neighborhood (222) of the picture block; A video decoder as described in embodiment D1.4, wherein a chroma residual signal (224) decoded for the picture block from the data stream is scaled (226) according to the chroma residual scale factor and used to correct (228) an intra chroma prediction signal (230) for the picture block.

[0094] D1.6. The set of one or more coding tools: The video decoder of any preceding embodiment D1.#, comprising an optical flow tool (300).

[0095] D1.7. The optical flow tool: The video decoder of embodiment D1.6, for improving the translational inter-prediction signal of a given inter-prediction block (10c) by optical flow-based analysis.

[0096] D1.8. The set of one or more coding tools: The video decoder of any preceding embodiment D1.#, comprising a decoder-side motion vector refinement tool (400).

[0097] D1.9. The decoder-side motion vector refinement tool: A video decoder as described in embodiment D1.8, for improving the signaled motion vector (402) encoded in the data stream for inter-predicting a given inter-predicted block (10d) from a reference picture (404) by improving the signaled motion vector by performing a best match search among motion vector candidates at and around the signaled motion vector.

[0098] D1.9a. The decoder-side motion vector refinement tool: A video decoder as described in embodiment D1.9, configured to perform the best match search using already decoded neighbors of the inter-predicted block with respect to the reference picture.

[0099] D1.9b. The decoder-side motion vector refinement tool: A video decoder as described in embodiment D1.8, configured to refine a pair of signaled motion vectors (402) encoded in a data stream to inter-predict a given inter-bidirectionally predicted block (10d) from a pair of reference pictures (404) located temporally before and after the given inter-bidirectionally predicted block (10d) [in presentation order] by performing a best match search among candidate pairs of motion vectors including and surrounding the signaled motion vector pair.

[0100] D1.10. The set of one or more coding tools: The video decoder of any preceding embodiment D1.#, comprising a temporal motion vector prediction tool (500).

[0101] D1.11. A video decoder as described in embodiment D1.10, wherein forming a motion vector candidate list for an inter-prediction block by the temporal motion vector prediction tool includes recruiting motion vector candidates from a previously decoded picture (502).

[0102] D1.12. A video decoder as described in embodiment D1.11, wherein forming a motion vector candidate list for an inter-predicted block by the temporal motion vector prediction tool includes recruiting motion vector candidates from blocks (506) of the previously decoded picture pointed to by a motion vector predictor (504).

[0103] D1.13. The video decoder of embodiment D1.12, wherein the motion vector predictor includes a temporal motion vector predictor.

[0104] D1.14. The indication: a decoder capability information section of the data stream; and a video or sequence parameter set for said data stream, and The video decoder of any preceding embodiment D1.#, wherein the information is included in one of the supplemental enhancement information messages.

[0105] D1.15. A video decoder as described in any preceding embodiment D1.#, wherein the indication includes one bit collectively indicating the exclusion of all encoding tools of the predetermined set of one or more encoding tools with respect to the encoding of the RASL picture in the picture sequence.

[0106] D1.16. The video decoder of any preceding embodiment D1.#, wherein the decoder is configured to support reference picture resampling.

[0107] D1.17. A video decoder as described in embodiment D1.16, wherein, following resampling of the reference picture, the reference picture of an inter-predicted block is subject to the sample resampling to fill in any deviation in scaling window size or sample resolution between the reference picture and the picture in which the inter-predicted block is included to provide an inter-predicted signal for the inter-predicted block.

[0108] D1.18. The set of one or more coding tools [e.g., 200, 300, 400, 500]: one or more first inherently applied coding tools, each of which is applied for a given block in accordance with one or more coding options signaled in the data stream for the given block, and which relate to further coding tools other than the respective further coding tools; and / or A video decoder as described in any of the preceding embodiments D1.#, including one or more second inherently applied encoding tools, each of which is applied to a given block depending on the size of the given block.

[0109] D1.19. The set of one or more coding tools [e.g., 100, 500]: A video decoder as described in any of the preceding embodiments D1.#, comprising one or more explicitly applied coding tools, each of which is applied to a given block in accordance with a syntax element encoded in the data stream to exclusively signal the application of the respective coding tool to the given block.

[0110] D1.20. The video decoder of embodiment D1.19, wherein the decoder is configured to decode the syntax elements from the data stream for blocks within the RASL picture and for blocks of pictures other than the RASL picture.

[0111] D1.21. A video decoder as described in embodiment D1.19, wherein the decoder is configured to decode the syntax elements from the data stream for blocks within pictures excluding only RASL pictures (e.g., this saves bits within RASL pictures).

[0112] D1.22. The video decoder of any preceding embodiment D1.#, wherein the decoder is configured to support intra-prediction block decoding modes and inter-prediction block decoding modes.

[0113] D1.23. The sequence of pictures is It consists of pictures beginning with, including, and ending with one CRA picture, up to and including the picture immediately preceding the CRA picture in coding order, or The video decoder of any of the preceding embodiments D1.#, wherein the video decoder is composed of pictures that are consecutive in coding order and include multiple CRAs.

[0114] D1.24. The set of one or more coding tools: A video decoder as described in any of the preceding embodiments D1.#, comprising one or more deactivatable coding tools, each of which, for application to picture blocks, can be deactivated on a picture or slice basis by configuration signaling in the data stream.

[0115] D1.25. The video decoder of any preceding embodiment D1.#, configured to use the indication to determine whether an open GOP switch results in tolerable drift.

[0116] D1.25a. A video decoder as described in any of the preceding embodiments D1.#, configured to use the indication to determine whether an open GOP switch causes a sample mismatch but preserves syntax and parameter settings.

[0117] D1.26. The indication indicates that all RASL pictures in the sequence of pictures are coded in the manner that excludes the predetermined set of one or more coding tools; and / or the indication indicates that all RASL pictures in the picture sequence are coded in the manner that excludes a predetermined set of the one or more coding tools, and that this coding tool has a reference picture that precedes, in decoding order, the CRA picture to which the same picture is associated; and / or the indication indicates that all RASL pictures in the sequence of pictures have been coded in the manner that excludes a predetermined set of the one or more coding tools that serve as temporal motion vector prediction reference pictures for subsequent pictures; and / or A video decoder as described in any of the preceding embodiments D1.#, wherein the indication indicates that all RASL pictures in the picture sequence are coded in the manner that excludes a predetermined set of the one or more coding tools that do not belong to the highest temporal layer.

[0118] D1.27. The indication indicates that the RASL picture in the sequence of pictures is coded in the manner that excludes a predetermined set of the one or more coding tools; for a first subset of one or more coding tools from the predetermined set of one or more coding tools, all RASL pictures to which a first property applies are coded in the manner that excludes the first subset of one or more coding tools from the predetermined set of one or more coding tools; A video decoder as described in any of the preceding embodiments D1.#, wherein for a second subset of one or more encoding tools from the predetermined set of one or more encoding tools, all RASL pictures to which a second characteristic is applied, or all RASL pictures, are coded in the manner that excludes the second subset of one or more encoding tools from the predetermined set of one or more encoding tools, and the first subset and the second subset are disjoint.

[0119] D1.28. The first characteristic and / or the second characteristic are: has a reference picture that precedes, in decoding order, the CRA picture with which it is associated; serves as a temporal motion vector prediction reference picture for subsequent pictures; and A video decoder as described in embodiment D1.27, selected from those that do not belong to the highest temporal layer.

[0120] D1.29. The video decoder of embodiment D1.27 onwards, wherein the first subset includes one or more of a decoder-side motion vector refinement tool and a temporal motion vector prediction tool.

[0121] D1.30. The first characteristic is: has a reference picture that precedes, in decoding order, the CRA picture with which it is associated; serves as a temporal motion vector prediction reference picture for subsequent pictures; and A video decoder as described in embodiment D1.29, selected from those that do not belong to the highest temporal layer.

[0122] D2.1. A video decoder for decoding video from a data stream, comprising: The video decoder is configured to decode from the data stream, for each picture of a picture sequence of the video, either globally for each picture or slice-by-slice, an indication (e.g., using sps_extra_ph_bit_present_flag and ph_extra_bit, or using gci_rasl_pictures_tool_contraint_flag) of whether each picture has been coded in a manner that excludes a predetermined set of one or more coding tools, wherein the predetermined set includes a cross-component linear model-based prediction tool (e.g., as an indication for a type of picture that can ensure that potential drift in RASL pictures is sufficiently low).

[0123] D2.2. The video decoder of any preceding embodiment D2.#, wherein the decoder is configured to support intra-prediction block decoding modes and inter-prediction block decoding modes.

[0124] D2.3. According to the cross-component linear model-based forecasting tool, A video decoder as described in any preceding embodiment D2.#, wherein the chroma component of a picture block is predicted from the luma component of the picture block using a linear model whose parameters are determined from extreme luma and chroma values ​​in an already decoded neighborhood of the picture block.

[0125] D2.4. The set of one or more coding tools further comprises: The video decoder of any preceding embodiment D2.#, comprising a luma tone mapping and chroma residual scaling prediction tool.

[0126] D2.5. According to said luma tone mapping and chroma residual scaling prediction tool, Luminance component prediction and luma component residual decoding for a given picture are performed with an encoded luma tone scale, and then the presentation luma tone scale is mapped by a luma tone mapping to obtain an encoded luma tone scale version of the reconstructed luma component of the given picture; a chroma residual scale factor for a picture block of the given picture is determined from an average of the coded luma tone scale versions of the reconstructed luma component of the given picture within a neighborhood of the picture block; A video decoder as described in embodiment D2.4, wherein a chroma residual signal decoded for the picture block from the data stream is scaled according to the chroma residual scale factor and used to correct the intra chroma prediction signal of the picture block.

[0127] D2.6. The set of one or more coding tools further comprises: The video decoder of any preceding embodiment D2.#, including an optical flow tool.

[0128] D2.7. The optical flow tool: The video decoder of embodiment D2.6 is for improving the translational inter-prediction signal of a given inter-prediction block by optical flow-based analysis.

[0129] D2.8. The set of one or more coding tools further comprises: The video decoder of any preceding embodiment D2.#, comprising a decoder-side motion vector refinement tool.

[0130] D2.9. The decoder-side motion vector refinement tool: A video decoder as described in embodiment D2.8, for improving the signaled motion vector (402) encoded in the data stream for inter-predicting a given inter-predicted block (10d) from a reference picture (404) by improving the signaled motion vector by performing a best match search among motion vector candidates at and around the signaled motion vector.

[0131] D2.9a. The decoder-side motion vector refinement tool: The video decoder of embodiment D2.9, configured to perform the best match search using already decoded neighbors of the inter-predicted block with respect to the reference picture.

[0132] D2.9b. The decoder-side motion vector refinement tool: A video decoder as described in embodiment D2.8, configured to refine a pair of signaled motion vectors (402) encoded in a data stream to inter-predict a given inter-bidirectionally predicted block (10d) from a pair of reference pictures (404) located temporally before and after the given inter-bidirectionally predicted block (10d) by performing a best match search between candidate pairs of motion vectors including and surrounding the signaled motion vector pair.

[0133] D2.10. The set of one or more coding tools further comprises: The video decoder of any preceding embodiment D2.#, comprising a temporal motion vector prediction tool.

[0134] D2.11. A video decoder as described in embodiment D2.10, wherein forming a motion vector candidate list for an inter-predicted block by the temporal motion vector prediction tool includes recruiting motion vector candidates from previously decoded pictures.

[0135] D2.12. A video decoder as described in embodiment D2.11, wherein forming a motion vector candidate list for an inter-predicted block by the temporal motion vector prediction tool includes recruiting motion vector candidates from blocks of the previously decoded picture pointed to by a motion vector predictor.

[0136] D2.13. The video decoder of embodiment D2.12, wherein the motion vector predictor includes a temporal motion vector predictor.

[0137] D2.14. The indication: one or more picture parameter sets referenced by said pictures of said picture sequence; a picture header of the picture of the picture sequence, and A video decoder as described in any preceding embodiment D1.#, wherein the information is included in one of the slice headers of the slices of the picture of the picture sequence.

[0138] D2.15. The indication is: picture parameter sets referenced by the pictures of the picture sequence, the picture parameter sets including: at least one first picture parameter set indicating that pictures referencing the at least one first picture parameter set are to be coded in a manner that excludes the one or more predetermined set of coding tools; and at least one second picture parameter set indicating that pictures referencing the at least one second picture parameter set are to be coded in a manner that potentially uses the one or more predetermined set of coding tools; or A video decoder as described in any preceding embodiment D2.#, wherein the picture parameter sets referenced by the pictures of the picture sequence include: at least one first picture parameter set indicating that RASL pictures associated with pictures referencing at least one first picture parameter set are to be coded in a manner that excludes a predetermined set of the one or more coding tools; and at least one second picture parameter set indicating that RASL pictures associated with pictures referencing at least one second picture parameter set are to be coded in a manner that potentially uses the predetermined set of the one or more coding tools.

[0139] D2.16. The video decoder of embodiment D2.15, wherein the indication includes syntax elements in an extended syntax portion of the picture parameter set (eg, using sps_extra_ph_bit_present_flag and ph_extra_bit).

[0140] D2.17. The video decoder of embodiment D2.16, wherein the length of the extension syntax portion of the picture parameter set is indicated within a sequence of the data stream or a video parameter set.

[0141] D2.18. The indication: a picture header of said picture of said picture sequence, and / or a slice header of a slice of the picture of the picture sequence; The video decoder of any preceding embodiment D2.#, wherein the length of the extension portion (e.g., NumExtraPhBits) is indicated in a picture, or sequence, or video parameter set of the data stream.

[0142] D2.18a. The syntax element is: whether the picture to which the syntax element belongs (e.g., the picture to which the picture header or slice header relates) is coded in a manner that excludes a predetermined set of the one or more coding tools; or The video decoder of embodiment D2.18, indicating whether the RASL picture associated with the picture to which the syntax element belongs is coded in a manner that excludes a predetermined set of the one or more coding tools.

[0143] D2.19. The video decoder of any preceding embodiment D2.#, wherein the decoder is configured to support reference picture resampling.

[0144] D2.22. A video decoder as described in embodiment D2.19, wherein, following resampling of the reference picture, the reference picture of an inter-predicted block is subject to the sample resampling to fill in any deviation in scaling window size or sample resolution between the reference picture and the picture in which the inter-predicted block is included to provide an inter-predicted signal for the inter-predicted block.

[0145] D2.23. The set of one or more coding tools (e.g., 200, 300, 400, 500) one or more first inherently applied coding tools, each of which is applied for a given block in accordance with one or more coding options signaled in the data stream for the given block, and which relate to further coding tools other than the respective further coding tools; and / or A video decoder as described in any of the preceding embodiments D2.#, including one or more second inherently applied encoding tools, each of which is applied to a given block depending on the size of the given block.

[0146] D2.24. The set of one or more coding tools (e.g., 100, 500) A video decoder as described in any of the preceding embodiments D2.#, comprising one or more explicitly applied coding tools, each of which is applied to a given block in accordance with a syntax element encoded in the data stream to exclusively signal the application of the respective coding tool to the given block.

[0147] D2.25. A video decoder as described in embodiment D2.24, wherein the decoder is configured to decode the syntax elements from the data stream for blocks within a picture or slice for which the predetermined set of one or more coding tools is signaled to be excluded from encoding, and for blocks within a picture or slice for which the predetermined set of one or more coding tools is not signaled to be excluded from encoding.

[0148] D2.26. A video decoder as described in embodiment D2.24, wherein the decoder is configured to decode the syntax elements from the data stream only for blocks within pictures or slices for which a predetermined set of the one or more encoding tools is signaled to be excluded from encoding.

[0149] D2.27. A video decoder as described in any preceding embodiment D2.24 or later, wherein the cross-component linear model-based prediction tool belongs to the one or more encoding tools that are explicitly applied.

[0150] D2.28. A video decoder as described in any preceding embodiment D2.16 or later, wherein the syntax element is a single bit that collectively indicates the exclusion of all encoding tools of the predetermined set of one or more encoding tools.

[0151] D2.29. The set of one or more coding tools: A video decoder as described in any of the preceding embodiments D2.#, comprising one or more deactivatable coding tools, each of which, for application to picture blocks, can be deactivated on a picture or slice basis by configuration signaling in the data stream.

[0152] D2.30. The video decoder of any preceding embodiment D2.#, configured to use the indication to determine whether an open GOP switch results in tolerable drift.

[0153] D2.30a. A video decoder as described in any of the preceding embodiments D2.#, configured to use the indication to determine whether an open GOP switch causes a sample mismatch but preserves syntax and parameter settings.

[0154] E1.1. A video encoder for encoding video into a data stream, comprising: The video encoder is configured to encode into the data stream an indication (e.g., gci_rasl_pictures_tool_constraint_flag) that is valid for a sequence of pictures of the video and indicates that RASL pictures within the sequence of pictures have been coded in a manner that excludes a predetermined set of one or more coding tools (e.g., as a kind of promise that enables the encoder to recognize that open GOP switching by concatenating separately coded open GOP versions of the video coded at different spatial resolutions and / or different SNRs will not cause excessive drift in the RASL pictures).

[0155] E1.2. The set of one or more coding tools: The video decoder of any preceding embodiment E1.#, comprising a cross-component linear model-based prediction tool (100).

[0156] E1.3. According to the cross-component linear model-based forecasting tool, A video encoder as described in embodiment E1.2, wherein the chroma component (102) of a picture block (10a) is predicted from the luma component (104) of the picture block (10a) using a linear model (106) whose parameters are determined (108) from extreme luma and chroma values ​​(110) in an already encoded neighborhood (112) of the picture block.

[0157] E1.4. The set of one or more coding tools: The video encoder of any preceding embodiment E1.#, comprising a luma tone mapping and chroma residual scaling prediction tool (200).

[0158] E1.5. According to said luma tone mapping and chroma residual scaling prediction tool, Luminance component prediction (202) (e.g., inter-prediction) and luma component residual encoding (204) for a given picture (12) are performed with an encoded luma tone scale (208), whereupon a presentation luma tone scale (210) is mapped by a luma tone mapping (212) to obtain an encoded luma tone scale version (214) of the reconstructed luma component of the given picture; a chroma residual scale factor (216) for a picture block (10b) of the given picture is determined from an average (220) of the coded luma tone scale versions of the reconstructed luma component of the given picture within a neighborhood (222) of the picture block; A video encoder as described in embodiment E1.4, wherein a chroma residual signal (224) encoded into the data stream for the picture block is scaled (226) according to the chroma residual scale factor and used to correct (228) an intra chroma prediction signal (230) for the picture block.

[0159] E1.6. The set of one or more coding tools: The video encoder of any preceding embodiment E1.#, comprising an optical flow tool (300).

[0160] E1.7. The optical flow tool: The video encoder of embodiment E1.6, wherein the video encoder is for improving the translational inter-prediction signal of a given inter-prediction block (10c) by optical flow-based analysis.

[0161] E1.8. The set of one or more coding tools: The video encoder of any preceding embodiment E1.#, comprising a decoder-side motion vector refinement tool (400).

[0162] E1.9. The decoder-side motion vector refinement tool: A video encoder as described in embodiment E1.8, for improving the signaled motion vector (402) encoded in the data stream for inter-predicting a given inter-predicted block (10d) from a reference picture (404) by improving the signaled motion vector by performing a best match search among motion vector candidates at and around the signaled motion vector.

[0163] E1.9a. The decoder-side motion vector refinement tool: A video encoder as described in embodiment E1.9, configured to perform the best match search using already decoded neighbors of the inter-predicted block with respect to the reference picture.

[0164] E1.9b. The decoder-side motion vector refinement tool: A video encoder as described in embodiment E1.8, configured to refine a pair of signaled motion vectors (402) encoded in a data stream to inter-predict a given inter-bidirectionally predicted block (10d) from a pair of reference pictures (404) located temporally before and after the given inter-bidirectionally predicted block (10d) by performing a best match search between candidate pairs of motion vectors including and surrounding the signaled motion vector pair.

[0165] E1.10. The set of one or more coding tools: The video encoder of any preceding embodiment E1.#, comprising a temporal motion vector prediction tool (500).

[0166] E1.11. A video encoder as described in embodiment E1.10, wherein forming a motion vector candidate list for an inter-prediction block by the temporal motion vector prediction tool includes recruiting motion vector candidates from a previously encoded picture (502).

[0167] E1.12. A video encoder as described in embodiment E1.11, wherein forming a motion vector candidate list for an inter-predicted block by the temporal motion vector prediction tool includes recruiting motion vector candidates from blocks (506) of the previously encoded picture pointed to by a motion vector predictor (504).

[0168] E1.13. The video encoder of embodiment E1.12, wherein the motion vector predictor includes a temporal motion vector predictor.

[0169] E1.14. The indication is: a decoder capability information section of the data stream; and a video or sequence parameter set for said data stream, and The video encoder of any preceding embodiment E1.#, wherein the information is included in one of the supplemental enhancement information messages.

[0170] E1.15. A video encoder as described in any preceding embodiment E1.#, wherein the indication includes one bit collectively indicating the exclusion of all encoding tools of the predetermined set of one or more encoding tools with respect to the encoding of the RASL picture in the picture sequence.

[0171] E1.16. The video encoder of any preceding embodiment E1.#, wherein the encoder is configured to support reference picture resampling.

[0172] E1.17. A video encoder as described in embodiment E1.16, wherein, following the resampling of the reference picture, the reference picture of an inter-predicted block is subject to the sample resampling to fill in the deviation in scaling window size or sample resolution between the reference picture and the picture in which the inter-predicted block is included to provide an inter-predicted signal for the inter-predicted block.

[0173] E1.18. The set of one or more coding tools (e.g., 200, 300, 400, 500) one or more first inherently applied coding tools, each of which is applied for a given block in accordance with one or more coding options signaled in the data stream for the given block, and which relate to further coding tools other than the respective further coding tools; and / or A video encoder as described in any of the preceding embodiments E1.#, comprising one or more second inherently applied encoding tools, each of which is applied to a given block depending on the size of the given block.

[0174] E1.19. The set of one or more coding tools (e.g., 100) A video encoder as described in any of the preceding embodiments E1.#, comprising one or more explicitly applied encoding tools, each of which is applied to a given block in accordance with a syntax element encoded in the data stream to exclusively signal the application of the respective encoding tool to the given block.

[0175] E1.20. The video encoder of embodiment E1.19, wherein the encoder is configured to encode the syntax elements into the data stream for blocks within the RASL picture and for blocks of pictures other than the RASL picture.

[0176] E1.21. A video encoder as described in embodiment E1.19, wherein the encoder is configured to encode the syntax elements into the data stream for blocks within pictures excluding only RASL pictures (e.g., this saves bits within RASL pictures).

[0177] E1.22. The video encoder of any preceding embodiment E1.#, wherein the encoder is configured to support intra-prediction block encoding modes and inter-prediction block encoding modes.

[0178] E1.23. The sequence of pictures is It consists of pictures beginning with, including, and ending with one CRA picture, up to and including the picture immediately preceding the CRA picture in coding order, or The video encoder of any preceding embodiment E1.#, wherein the video encoder comprises pictures that are consecutive in coding order and include multiple CRAs.

[0179] E1.24. The set of one or more coding tools: A video encoder as described in any of the preceding embodiments E1.#, comprising one or more deactivatable encoding tools, each of which, for application to picture blocks, can be deactivated on a picture or slice basis by configuration signaling in the data stream.

[0180] E1.25. The video encoder of any preceding embodiment E1.#, configured to follow the indication as an encoding constraint when encoding the video into the data stream.

[0181] E1.26. The indication indicates that all RASL pictures in the sequence of pictures are coded in the manner that excludes the predetermined set of one or more coding tools; and / or the indication indicates that all RASL pictures in the picture sequence are coded in the manner that excludes a predetermined set of the one or more coding tools, and that this coding tool has a reference picture that precedes, in decoding order, the CRA picture to which the same picture is associated; and / or the indication indicates that all RASL pictures in the sequence of pictures have been coded in the manner that excludes a predetermined set of the one or more coding tools that serve as temporal motion vector prediction reference pictures for subsequent pictures; and / or A video encoder as described in any of the preceding embodiments E1.#, wherein the indication indicates that all RASL pictures in the picture sequence have been coded in the manner that excludes a predetermined set of the one or more coding tools that do not belong to the highest temporal layer.

[0182] E1.27. The indication indicates that the RASL picture in the sequence of pictures is coded in the manner that excludes a predetermined set of the one or more coding tools; for a first subset of one or more coding tools from the predetermined set of one or more coding tools, all RASL pictures to which a first property applies are coded in the manner that excludes the first subset of one or more coding tools from the predetermined set of one or more coding tools; A video encoder as described in any of the preceding embodiments E1.#, wherein for a second subset of one or more encoding tools from the predetermined set of one or more encoding tools, all RASL pictures to which a second property is applied, or all RASL pictures, are encoded in the manner that excludes the second subset of one or more encoding tools from the predetermined set of one or more encoding tools, and the first subset and the second subset are disjoint.

[0183] E1.28. The first characteristic and / or the second characteristic are: has a reference picture that precedes, in decoding order, the CRA picture with which it is associated; serves as a temporal motion vector prediction reference picture for subsequent pictures; and A video encoder as described in embodiment E1.27, selected from: those that do not belong to the highest temporal layer.

[0184] E1.29. A video encoder as in embodiment E1.27 onwards, wherein the first subset includes one or more of a decoder-side motion vector refinement tool and a temporal motion vector prediction tool.

[0185] E1.30. The first characteristic is: has a reference picture that precedes, in decoding order, the CRA picture with which it is associated; serves as a temporal motion vector prediction reference picture for subsequent pictures; and A video encoder as described in embodiment E1.29, selected from those that do not belong to the highest temporal layer.

[0186] E2.1. A video encoder for encoding video into a data stream, comprising: The video encoder is configured to encode into the data stream, for each picture of a picture sequence of the video, either globally for each picture or slice by slice, an indication (e.g., using sps_extra_ph_bit_present_flag and ph_extra_bit, or using gci_rasl_pictures_tool_contraint_flag) of whether the each picture is coded in a manner that excludes a predetermined set of one or more coding tools, wherein the predetermined set includes a cross-component linear model-based prediction tool (e.g., as an indication for a type of picture that can ensure that potential drift in RASL pictures is sufficiently low).

[0187] E2.2. The video encoder of any preceding embodiment E2.#, wherein the encoder is configured to support intra-prediction block encoding modes and inter-prediction block encoding modes.

[0188] E2.3. According to the cross-component linear model-based forecasting tool, A video encoder as described in any preceding embodiment E2.#, wherein the chroma component of a picture block is predicted from the luminance component of the picture block using a linear model whose parameters are determined from extreme luminance and chroma values ​​in an already encoded neighborhood of the picture block.

[0189] E2.4. The set of one or more encoding tools further comprises: The video encoder of any preceding embodiment E2.#, comprising a luma tone mapping and chroma residual scaling prediction tool.

[0190] E2.5. According to the luma tone mapping and chroma residual scaling prediction tool, Luminance component prediction and luminance component residual encoding for a given picture are performed with an encoded luminance tone scale, and then the presentation luminance tone scale is mapped by a luminance tone mapping to obtain an encoded luminance tone scale version of the reconstructed luminance component of the given picture; a chroma residual scale factor for a picture block of the given picture is determined from an average of the coded luma tone scale versions of the reconstructed luma component of the given picture within a neighborhood of the picture block; A video encoder as described in embodiment E2.4, wherein a chroma residual signal encoded for the picture block from the data stream is scaled according to the chroma residual scale factor and used to correct the intra chroma prediction signal of the picture block.

[0191] E2.6. The set of one or more coding tools further comprises: The video encoder of any preceding embodiment E2.#, including an optical flow tool.

[0192] E2.7. The optical flow tool: The video encoder of embodiment E2.6, wherein the video encoder is for improving a translational inter-prediction signal of a given inter-prediction block by optical flow-based analysis.

[0193] E2.8. The set of one or more coding tools further comprises: The video encoder of any preceding embodiment E2.#, comprising a decoder-side motion vector refinement tool.

[0194] E2.9. The decoder-side motion vector refinement tool: A video encoder as described in embodiment E2.8, for improving the signaled motion vector (402) encoded in the data stream for inter-predicting a given inter-predicted block (10d) from a reference picture (404) by improving the signaled motion vector by performing a best match search among motion vector candidates at and around the signaled motion vector.

[0195] E2.9a. The decoder-side motion vector refinement tool: A video encoder as described in embodiment E2.9, configured to perform the best match search using already decoded neighbors of the inter-predicted block with respect to the reference picture.

[0196] E2.9b. The decoder-side motion vector refinement tool: A video encoder as described in embodiment E2.8, configured to refine a pair of signaled motion vectors (402) encoded in a data stream to inter-predict a given inter-bidirectionally predicted block (10d) from a pair of reference pictures (404) located temporally before and after the given inter-bidirectionally predicted block (10d) by performing a best match search between candidate pairs of motion vectors including and surrounding the signaled motion vector pair.

[0197] E2.10. The set of one or more encoding tools further comprises: The video encoder of any preceding embodiment E2.#, comprising a temporal motion vector prediction tool.

[0198] E2.11. A video encoder as described in embodiment E2.10, wherein forming a motion vector candidate list for an inter-prediction block by the temporal motion vector prediction tool includes recruiting motion vector candidates from previously encoded pictures.

[0199] E2.12. A video encoder as described in embodiment E2.11, wherein forming a motion vector candidate list for an inter-predicted block by the temporal motion vector prediction tool includes recruiting motion vector candidates from blocks of the previously encoded picture indicated by a motion vector predictor.

[0200] E2.13. The video encoder of embodiment E2.12, wherein the motion vector predictor includes a temporal motion vector predictor.

[0201] E2.14. The indication is: one or more picture parameter sets referenced by said pictures of said picture sequence; a picture header of the picture of the picture sequence, and A video encoder as in any preceding embodiment D1.#, wherein the information is included in one of the slice headers of the slices of the picture of the picture sequence.

[0202] E2.15. The indication is: picture parameter sets referenced by the pictures of the picture sequence, the picture parameter sets including: at least one first picture parameter set indicating that pictures referencing the at least one first picture parameter set are to be coded in a manner that excludes the one or more predetermined set of coding tools; and at least one second picture parameter set indicating that pictures referencing the at least one second picture parameter set are to be coded in a manner that potentially uses the one or more predetermined set of coding tools; or A video encoder as described in any preceding embodiment E2.#, wherein the picture parameter sets referenced by the pictures of the picture sequence include: at least one first picture parameter set indicating that RASL pictures associated with pictures referencing at least one first picture parameter set are to be coded in a manner that excludes a predetermined set of the one or more coding tools; and at least one second picture parameter set indicating that RASL pictures associated with pictures referencing at least one second picture parameter set are to be coded in a manner that potentially uses the predetermined set of the one or more coding tools.

[0203] E2.16. The video encoder of embodiment E2.15, wherein the indication includes syntax elements in an extended syntax portion of the picture parameter set (eg, using sps_extra_ph_bit_present_flag and ph_extra_bit).

[0204] E2.17. The video encoder of embodiment E2.16, wherein the length of the extension syntax portion of the picture parameter set is indicated within a sequence of the data stream or a video parameter set.

[0205] E2.18. The indication is: a picture header of said picture of said picture sequence, and / or a slice header of a slice of the picture of the picture sequence; A video encoder as in any preceding embodiment E2.#, wherein the length of the extension portion (e.g., NumExtraPhBits) is indicated in a picture, or sequence, or video parameter set of the data stream.

[0206] E2.18a. The syntax element is: whether the picture to which the syntax element belongs (e.g., the picture to which the picture header or slice header relates) is coded in a manner that excludes a predetermined set of the one or more coding tools; or The video encoder of embodiment E2.18, indicating whether a RASL picture associated with the picture to which the syntax element belongs is coded in a manner that excludes a predetermined set of the one or more coding tools.

[0207] E2.19. The video encoder of any preceding embodiment E2.#, wherein the encoder is configured to support reference picture resampling.

[0208] E2.22. A video encoder as described in embodiment E2.19, wherein, following the resampling of the reference picture, the reference picture of an inter-predicted block is subject to the sample resampling to fill in the deviation in scaling window size or sample resolution between the reference picture and the picture in which the inter-predicted block is included to provide an inter-predicted signal for the inter-predicted block.

[0209] E2.23. The set of one or more coding tools (e.g., 200, 300, 400, 500) one or more first inherently applied coding tools, each of which is applied for a given block in accordance with one or more coding options signaled in the data stream for the given block, and which relate to further coding tools other than the respective further coding tools; and / or A video encoder as described in any of the preceding embodiments E2.#, comprising one or more second inherently applied encoding tools, each of which is applied to a given block depending on the size of the given block.

[0210] E2.24. The set of one or more coding tools: A video encoder as described in any of the preceding embodiments E2.#, comprising one or more explicitly applied encoding tools, each of which is applied to a given block in accordance with a syntax element encoded in the data stream to exclusively signal the application of the respective encoding tool to the given block.

[0211] E2.25. A video encoder as described in embodiment E2.24, wherein the encoder is configured to encode the syntax elements into the data stream for blocks within a picture or slice for which the predetermined set of one or more encoding tools is signaled to be excluded from encoding, and for blocks within a picture or slice for which the predetermined set of one or more encoding tools is not signaled to be excluded from encoding.

[0212] E2.26. A video encoder as described in embodiment E2.24, wherein the encoder is configured to encode the syntax elements into the data stream only for blocks within a picture or slice for which a predetermined set of the one or more encoding tools has been signaled to be excluded from encoding.

[0213] E2.27. A video encoder as in any preceding embodiment E2.24 or later, wherein the cross-component linear model-based prediction tool belongs to the one or more encoding tools that are explicitly applied.

[0214] E2.28. A video encoder as described in any preceding embodiment E2.16 or later, wherein the syntax element is a single bit that collectively indicates the exclusion of all encoding tools of the predetermined set of one or more encoding tools.

[0215] E2.29. The set of one or more coding tools: A video encoder as described in any of the preceding embodiments E2.#, comprising one or more deactivatable encoding tools, each of which, for application to picture blocks, can be deactivated on a picture or slice basis by configuration signaling in the data stream.

[0216] E2.30. The video encoder of any preceding embodiment E2.#, configured to follow the indication as an encoding constraint when encoding the video into the data stream.

[0217] B1.1. A video-encoded data stream, The data stream includes an indication (e.g., gci_rasl_pictures_tool_constraint_flag) that is valid for a sequence of pictures of the video and indicates that RASL pictures within the sequence of pictures have been coded in a manner that excludes a predetermined set of one or more coding tools (e.g., as a kind of promise that enables the encoder to know that open GOP switching by concatenating separately coded open GOP versions of the video coded at different spatial resolutions and / or different SNRs will not cause excessive drift in the RASL pictures).

[0218] B1.2. The data stream of embodiment B1.1 produced by an encoder according to any of embodiments E1.#.

[0219] B2.1. A video-encoded data stream, The data stream includes an indication (e.g., using sps_extra_ph_bit_present_flag and ph_extra_bit, or using gci_rasl_pictures_tool_contraint_flag) for each picture of a picture sequence of the video, either globally for each picture or per slice, indicating whether each picture has been coded in a manner that excludes a predetermined set of one or more coding tools, wherein the predetermined set includes a cross-component linear model-based prediction tool (e.g., as an indication for a type of picture that can be confirmed to have sufficiently low potential drift in RASL pictures).

[0220] B2.2. The data stream of embodiment B2.1 produced by an encoder according to any of embodiments E2.#.

[0221] M. A method performed by any of the decoders and encoders described above.

[0222] P. A computer program having a program code for performing the method of embodiment M when the program is run on a computer.

[0223] While some aspects are described in terms of apparatus, it will be apparent that these aspects also represent descriptions of corresponding methods, where blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in terms of method steps also represent descriptions of corresponding blocks or items or features of corresponding apparatus. Some or all method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or electronic circuitry. In some embodiments, one or more of the most significant method steps may be performed by such an apparatus.

[0224] The inventive data stream may be stored on a digital storage medium or transmitted over a transmission medium, such as a wireless or wired transmission medium, such as the Internet.

[0225] Depending on specific implementation requirements, embodiments of the invention may be implemented in hardware or software. Implementations may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, having electronically readable control signals stored thereon that cooperate (or are capable of cooperating) with a programmable computer system so that the respective methods are performed. Thus, the digital storage medium may be computer-readable.

[0226] Some embodiments according to the invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0227] Generally, embodiments of the present invention may be implemented as a computer program product having program code operable to perform one of the methods when the computer program product runs on a computer, which may for example be stored on a machine-readable carrier.

[0228] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0229] In other words, an embodiment of the inventive methods is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0230] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, digital storage medium or recorded medium is typically tangible and / or non-transitory.

[0231] A further embodiment of the inventive method is, therefore, a data stream or sequence of signals representing the computer program for performing one of the methods described herein. The data stream or sequence of signals can be adapted to be transferred via a data communication connection, for example via the Internet.

[0232] A further embodiment comprises a processing means, such as for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0233] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0234] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0235] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0236] The devices described herein may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0237] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least partially in hardware and / or software.

[0238] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0239] The methods described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0240] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore the intention to be limited only by the scope of the appended claims and not by the specific details presented by way of description and illustration of the embodiments herein.

[0241] References [1] ISO / IEC JTC 1, Information technology - Dynamic adaptive streaming over HTTP (DASH) - Part 1: Media presentation description and segment formats, ISO / IEC 23009-1, 2012 (and subsequent editions). [2] J. De Cock, Z. Li, M. Manohara, A. Aaron. “Complexity-based consistent-quality encoding in the cloud.” 2016 IEEE International Conference on Image Processing (ICIP). IEEE, 2016 [3] DASH Industry Forum Implementation Guidelines. [Online]. Available: https: / / dashif.org / guidelines / [4] ITU-T and ISO / IEC JTC 1, Advanced Video Coding for generic audio-visual services, Rec. ITU-T H.264 and ISO / IEC 14496-10 (AVC), May 2003 (and subsequent editions). [5] ITU-T and ISO / IEC JTC 1, “High Efficiency Video Coding,” Rec. ITU-T H.265 and ISO / IEC 23008-2 (HEVC), April 2013 (and subsequent editions). [6] Y. Yan, M. Hannuksela, and H. Li. “Seamless switching of H. 265 / HEVC-coded dash representations with open GOP prediction structure.” 2015 IEEE International Conference on Image Processing (ICIP). IEEE, 2015. [7] ITU-T and ISO / IEC JTC 1, “Versatile video coding”, Rec. ITU-T H.266 and ISO / IEC 23090-3 (VVC), August 2020. [8] V. Baroncini and M. Wien, “VVC verification test report for UHD SDR video content”, doc. JVET-T2020 of ITU-T / ISO / IEC Joint Video Experts Team (JVET), 21th meeting: October 2020. [9] D. Luo, V. Seregin, W. Wan. “Description of Core Experiment 1 (CE1): Reference picture resampling filters“, doc. JVET-Q2021 of ITU-T / ISO / IEC Joint Video Experts Team (JVET), 15th meeting: July 2019

[10] H. Schwarz, D. Marpe, and T. Wiegand, “Analysis of hierarchical B pictures and MCTF”, ICME 2006, IEEE International Conference on Multimedia and Expo, Toronto, Ontario, Canada, July 2006.

[11] Y.-K. Wang et al., “The High-Level Syntax of the Versatile Video Coding (VVC) Standard” IEEE Trans. Circuits Syst. Video Technol., in press

[12] H. Yang et al., “Subblock based Motion Derivation and Inter-Prediction Refinement in Versatile Video Coding Standard”, IEEE Trans. Circuits Syst. Video Technol., in press

[13] W.-J. Chien et al., “Motion Vector Coding and Block Merging in Versatile Video Coding Standard”, IEEE Trans. Circuits Syst. Video Technol., in press

Claims

1. 1. A decoder for video decoding, the decoder comprising at least one processor, the at least one processor comprising: decoding, from the data stream, a supplemental enhancement information (SEI) message associated with a sequence of pictures including at least one random access skip leading (RASL) picture and a clean random access (CRA) picture associated with the at least one RASL picture; determining that the SEI message indicates that encoding of all RASL pictures in the sequence of pictures has been constrained by excluding at least one encoding tool; decoding the at least one RASL picture without using the at least one encoding tool; A decoder configured to:

2. The decoder of claim 1 , wherein the at least one encoding tool comprises a cross-component linear model-based prediction tool.

3. 3. The decoder of claim 2, wherein the cross-component linear model-based prediction tool is for predicting chroma components of a picture block, the chroma components being predicted from the luma components of the picture block using a linear model with parameters based on luma and chroma values ​​of already decoded blocks neighboring the picture block.

4. The decoder of claim 1 , wherein the at least one encoding tool comprises a decoder-side motion vector refinement tool.

5. the decoder-side motion vector refinement tool modifies the signaled motion vector using at least one motion vector candidate selected from a plurality of motion vector candidates associated with previously decoded blocks neighboring the picture block; The decoder of claim 4 , wherein the signaled motion vectors are encoded within the data stream for inter-predicting inter-predicted blocks from reference pictures.

6. The at least one processor further comprises:

2. The decoder of claim 1, configured to determine that the SEI message further indicates that encoding of the at least one RASL picture is constrained so that a co-located reference picture used for temporal motion vector prediction (TMVP) or sub-block temporal motion vector prediction (sbTMVP) of a motion vector of the at least one RASL picture does not precede the CRA picture associated with the RASL picture in decoding order.

7. The at least one processor further comprises: resampling the reference picture of the inter-predicted block to fill in (i) a deviation in sample resolution between a reference picture and a current picture in which the inter-predicted block is included, or (ii) a deviation in scaling window size; generating an inter prediction signal for the inter prediction block based on the resampled reference picture; 2. A decoder according to claim 1, configured to:

8. 1. A method of video decoding, comprising: decoding, from the data stream, a supplemental enhancement information (SEI) message associated with a sequence of pictures including at least one random access skip leading (RASL) picture and a clean random access (CRA) picture associated with the at least one RASL picture; determining that the SEI message indicates that encoding of all RASL pictures in the sequence of pictures has been constrained by excluding at least one encoding tool; decoding the at least one RASL picture without using the at least one encoding tool; A method comprising:

9. A non-transitory computer-readable medium that, when executed, causes at least one processor to: decoding, from the data stream, a supplemental enhancement information (SEI) message associated with a sequence of pictures including at least one random access skip leading (RASL) picture and a clean random access (CRA) picture associated with the at least one RASL picture; determining that the SEI message indicates that encoding of all RASL pictures in the sequence of pictures has been constrained by excluding at least one encoding tool; decoding the at least one RASL picture without using the at least one encoding tool; A non-transitory computer-readable medium containing instructions for causing a

10. The non-transitory computer-readable medium of claim 9 , wherein the at least one encoding tool comprises a cross-component linear model-based prediction tool.

11. The non-transitory computer-readable medium of claim 9 , wherein the at least one encoding tool comprises a decoder-side motion vector refinement tool.

12. When executed, the at least one processor: determining that the SEI message further indicates that coding of the at least one RASL picture is constrained such that a co-located reference picture used for temporal motion vector prediction (TMVP) or sub-block temporal motion vector prediction (sbTMVP) of a motion vector of the at least one RASL picture does not precede the CRA picture associated with the RASL picture in decoding order; 10. The non-transitory computer-readable medium of claim 9, further comprising instructions to cause the execution of:

13. When executed, the at least one processor: resampling the reference picture of the inter-predicted block to fill (i) a deviation in sample resolution between the reference picture and the current picture in which the inter-predicted block is included, or (ii) a deviation in scaling window size; generating an inter-prediction signal for the inter-prediction block based on the resampled reference picture; 10. The non-transitory computer-readable medium of claim 9, further comprising instructions to cause the execution of:

14. 1. An encoder for video encoding, the encoder comprising at least one processor, the at least one processor comprising: determining that all random access skip leading (RASL) pictures in a sequence of pictures are encoded without using at least one encoding tool, the sequence of pictures including at least one random access skip leading (RASL) picture and a clean random access (CRA) picture associated with the at least one RASL picture; encoding the RASL pictures in the sequence of pictures into a data stream without using the at least one encoding tool; encoding a supplemental enhancement information (SEI) message associated with the sequence of pictures into the data stream, the SEI message indicating that encoding of all RASL pictures in the sequence of pictures is constrained by excluding the at least one encoding tool; an encoder configured to:

15. 1. A method of video decoding, comprising: determining that all random access skip leading (RASL) pictures in a sequence of pictures are encoded without using at least one encoding tool, the sequence of pictures including at least one random access skip leading (RASL) picture and a clean random access (CRA) picture associated with the at least one RASL picture; encoding the RASL pictures in the sequence of pictures into a data stream without using the at least one encoding tool; encoding a supplemental enhancement information (SEI) message associated with the sequence of pictures into the data stream, the SEI message indicating that encoding of all RASL pictures in the sequence of pictures is constrained by excluding the at least one encoding tool; A method comprising:

16. A non-transitory computer-readable medium that, when executed, causes at least one processor to: determining that all random access skip leading (RASL) pictures in a sequence of pictures are encoded without using at least one encoding tool, the sequence of pictures including at least one random access skip leading (RASL) picture and a clean random access (CRA) picture associated with the at least one RASL picture; encoding the RASL pictures in the sequence of pictures into a data stream without using the at least one encoding tool; encoding a supplemental enhancement information (SEI) message associated with the sequence of pictures into the data stream, the SEI message indicating that encoding of all RASL pictures in the sequence of pictures is constrained by excluding the at least one encoding tool; A non-transitory computer-readable medium containing instructions for causing a

Images