Video codec allowing sub-picture or region wise random access and concept for video composition using video codec

Abstract

To provide the concept for a video codec that effectively implements video composition / merging or gradual decoding refresh, along with the concept that effectively enables video composition / merging.SOLUTION: A decoder 22 decodes sub-picture related information 40 from a data stream 10 for each sub-picture 24 of each picture 18 in a set 42 of one or more pictures of a video, and identifies the respective sub-pictures: when the sub-picture related information is in a first state, as being coded in a random access point manner; or when the sub-picture related information is in a second state, not coded in the random access point manner. The decoder further decodes from the data stream a random access point signaling granularity indication 44. The granularity indication indicates that the sub-picture related information assumes a common state for all sub-pictures of each picture at the first granularity, and may assume different states at the second granularity.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] This application relates to video encoding and video composition. [Background technology]

[0002] Various applications make use of sub-pictures, i.e., sub-dividing a video picture into sub-pictures for separate encoding and decoding. Among these applications are 360° video and gradual decoding refresh.

[0003] For 360° video, sub-pictures (tiles in case of HEVC) are provided separately in different resolutions, as shown in the left side of Fig. 10. Users select some tiles in high resolution and others in low resolution according to their viewing orientation, as shown in the center of Fig. 10, and these are merged into one common bitstream, as shown in the right side of Fig. 10. Whenever a user changes their viewing orientation, the set of tiles with high resolution and the set of tiles with low resolution also change. However, not all tiles with high resolution change to tiles with low resolution and vice versa. This means that a switch point is needed only for the subset of tiles whose resolution changes. This switch point is usually a Random Access Point (RAP) of that particular sub-picture bitstream.

[0004] Gradual Decoding Refresh (GDR) is a mechanism that allows starting the decoding of a bitstream at a non-RAP and after decoding a given number of pictures, displaying the video with similar quality as if the decoding process had started at a RAP, as shown in Figure 11.

[0005] In Fig. 11, for example, four consecutive non-RAP pictures (consecutive in the coding order 900) are shown to be evenly sub-divided into three tile groups tile_group0, tile_group1, tile_group2, i.e. into regions of the respective pictures. The pictures of the video 902 are indicated by reference symbols 904a-904d, and the regions into which they are sub-divided are indicated using reference symbols 906. Each region 906 is indicated to be coded in a self-contained manner, i.e., forming a random access point, by being inscribed therein with "IRAP", or alternatively, to be coded in a manner that refers to other pictures, by being inscribed therein with "INTER". In none of the four pictures 904a-904d shown in Fig. 11 are all the regions 906 coded as random access points. Rather, in the example of Fig. 11, the random access point regions of three consecutive pictures, such as pictures 904a-904c, spatially completely cover the picture region of the picture of the video 902, although only one of the regions is a random access point. Thus, after such a triplet of pictures, the last picture of such a triplet, here picture 904c, is available in a decoder without artifacts under certain conditions: the encoder managed that the non-RAP regions of this picture 904c merely referenced regions that are - for example in terms of prediction and / or entropy context derivation, i.e. in terms of coding interdependencies - RAP regions, or non-RAP regions that themselves have coding dependencies directly or indirectly only on RAP regions. That is, a clear decoded picture is not immediately achieved as when decoding starts with RAP. Rather, the decoded picture is gradually refreshed to eliminate the missing reference artifacts, and after some time a clear decoded picture is achieved, which is picture 904c.Typically, GDR is achieved by dividing the content into regions (eg, tiles) and encoding the regions with unaligned sub-picture RAPs.

[0006] Either of the above two scenarios results in a change of references across different sub-bitstreams, and RAP marking of different sub-pictures is not performed explicitly since they are only signaled as I_SLICES type (no prediction to previously decoded pictures is applied), and prediction of consecutive sub-pictures of the same sub-picture bitstream is interrupted only by not including the sub-picture before a picture of I_SLICE type in the reference picture list.

[0007] Notwithstanding the existing video coding concepts discussed above, a need remains for a video coding environment that allows, for example, more efficient video combining and / or more efficient implementation of gradual decoding refresh. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] WO 2015 / 059194 [Patent Document 2] WO 2016 / 026526 Summary of the Invention [Problem to be solved by the invention]

[0009] It is therefore an object of the present invention to provide a video codec that allows a more effective video compositing / integration and / or a more effective implementation of gradual decoding refresh, and / or a concept that allows video compositing / integration to be made more effective, e.g. in terms of the complexity of the compositing / integration task. [Means for solving the problem]

[0010] This object is achieved by the subject matter of the independent claims of the present application.

[0011] A first aspect of the present application aims to make the implementation of video synthesis / merging and / or gradual decoding refresh in video codecs more effective by introducing a random access point signaling granularity indication, which indicates to a video decoder at what granularity within a picture of a video the sub-picture related information may vary spatially. The sub-picture related information identifies whether the respective sub-picture is coded in a random access point manner or not in a random access point manner. That is, if the random access point signaling granularity indication signaled in the data stream separately or in addition to the sub-picture related information exhibits a first granularity, this represents, for each picture of a set of one or more pictures forming a scope of the random access point signaling granularity indication, that the sub-picture related information is in a state common to all sub-pictures of the respective picture. That is, in that case, all sub-pictures are either in a first state signaling random access point coding of the associated sub-picture or in a second state signaling non-random access point coding of the associated sub-picture. However, if the random access point signaling granularity indication presents a second granularity, this indicates that for each picture of a set of one or more pictures, the sub-picture related information may be in a different state for the sub-pictures of the respective picture. As a result of the introduction of such a random access point signaling granularity indication, the task of synthesizing a data stream having a video encoded therein in such a way that the pictures of the video are subdivided into sub-pictures is reduced in terms of the requirements imposed on the reservoir of one or more input video streams, sometimes referred to herein as sub-streams, having sub-pictures encoded therein. In particular, these input video streams do not have to be encoded in an interdependent manner, i.e. in such a way that if one sub-picture in one sub-stream is RAP coded, the temporally aligned sub-pictures of the other sub-streams are RAP coded as well.If several substreams are coded into one common input video stream, they may be coded in a normal way, where each picture of each input video data stream is coded such that all subpictures of the respective picture have one subpicture related information associated with them in the input video stream, which is either a first state for all subpictures of the respective picture, or a second state for all subpictures of the respective picture, except that temporally aligned subpictures in other substreams may not be coded in the RAP manner. When compositing video data streams containing random access point signaling granularity indications, a composer / merger is free to combine into a composite data stream pictures of videos coded such that there are pictures divided into subpictures, whose subpicture related information is adopted or inherited from the respective input video data streams, at least one of which is a RAP and at least one of which is not a RAP. Thus, for example, there is no need to introduce one or more new states for the sub-picture related information present in the composite data stream for each sub-picture, in other words, the sub-picture related information may simply be inherited from the respective reservoirs of one or more input video data streams.

[0012] A sub-picture may be a tile or a group of tiles, i.e. a sub-picture may be coded independently of other sub-pictures of the same picture. Alternatively, a sub-picture may be a tile or a group of tiles forming an MCTS, i.e. a motion constrained tile set. This means that a sub-picture may be coded independently of spatially offset sub-pictures of other pictures, in addition to being coded independently within the same picture. In that case, pictures may be divided into sub-pictures in a certain way, such that the sub-pictures of different pictures are aligned with each other or with spatially coincident sub-picture boundaries. Thereby, such a data stream including such a random access point signaling granularity indication would be able to signal sub-picture related information by using the same alphabet as used in the reservoir of sub-streams that would have been coded in the picture granularity regime. Sub-picture related information may include integer-valued syntax elements that distinguish between multiple types, such as NAL unit types. Each sub-picture may be coded into one or more NAL units, each NAL unit having, for example, such an integer-valued syntax element. Thus, regardless of whether the random access point signaling granularity indication indicates a first or a second granularity, a first subset of the one or more NAL unit types of the plurality of NAL unit types distinguished by the integer-valued syntax element will correspond to a first state of the sub-picture related information, while a second subset of the one or more NAL unit types will correspond to a second state of the sub-picture related information.

[0013] The random access point signaling granularity indication may be contained within an SPS or VPS of the data stream, and a set of one or more pictures forming a scope of the random access point signaling granularity indication including a series of pictures of a video or covering an entire video. Alternatively, the random access point signaling granularity indication may be conveyed per access unit in the data stream, and a set of one or more pictures forming a scope of the random access point signaling granularity indication including only one picture.

[0014] A video decoder that is not capable of parsing the random access point signaling granularity indication may identify a data stream as a non-compliant data stream if, for a given picture, the sub-picture related information is in a different state for different sub-pictures of this picture. Other video decoders that are capable of parsing and decoding the random access signaling granularity indication may adapt their decoding operations in response to this indication. That is, they may identify a data stream as a non-compliant data stream if the random access point signaling granularity indication exhibits a first granularity and, for a given picture of a set of one or more pictures that form the scope of the random access point signaling granularity indication, the sub-picture related information is in a different state for different sub-pictures of this picture.

[0015] Additionally or alternatively, more features of the video decoder can be activated when the random access point signaling granularity indication exhibits a second granularity, i.e. sub-picture granularity, and these features can likewise be advantageously used to achieve a more efficient video codec irrespective of the use of a combined use of sub-picture related information on the one hand and a random access point signaling granularity indication on the other hand, as will be described below with respect to further aspects of the present application, i.e. a decoder according to the following aspects can only operate in the sub-picture granularity regime and does not need to signal the use of this regime.

[0016] According to a second aspect of the present application, it is an object to make a video codec more flexible with respect to per sub-picture coding, e.g. with respect to use for representing composited video. The picture is divided into sub-pictures. This object is achieved by rendering the video decoder depending on an evaluation of sub-picture related information conveyed in the data stream for each sub-picture. In particular, according to the second aspect, the data stream with such video coded comprises, for each sub-picture of each picture, sub-picture related information, which, in a first state, identifies the respective sub-picture as coded in a random access point manner and, in a second state, identifies the respective sub-picture as not coded in a random access point manner. The video decoder checks the picture with respect to the sub-picture related information of its sub-picture. If the sub-picture related information is in a first state for all sub-pictures of a given picture for any picture, i.e. if all its sub-pictures represent random access points, the video decoder flushes its decoded picture buffer. Such a concept allows for synthesizing data streams from substreams actually generated for different video codec domains, where, similar to the first aspect, subpictures in one picture were required to match in subpicture related information, i.e., all subpictures of one picture must be in the same state. However, when synthesizing such data streams, the state of the subpicture related information of the various subpictures used to synthesize the data streams may simply be adopted or inherited from there. Furthermore, a video decoder may identify true random access points, in that by identifying pictures whose subpicture related information assumes a first state for all subpictures of the respective picture, the entire picture can be recovered without artifacts.When combined with the first aspect, a video decoder may apply this feature of identifying random access point pictures, in which case the decoded picture buffer may be flushed in response to determining that the sub-picture related information of such pictures assumes a first state for all sub-pictures of the respective picture when the random access point signaling granularity indication exhibits a second granularity, i.e., sub-picture granularity. When combined with the first aspect, such a video decoder may even change the schedule for flushing the decoded picture buffer. That is, when the random access point signaling granularity indication exhibits a first granularity, i.e., per-picture granularity, the video decoder may flush the decoded picture buffer in response to the sub-picture related information exhibiting a first state for a first sub-picture of a particular picture, according to the decoding / encoding order between the sub-pictures, since the sub-picture related information relating to other sub-pictures should in any case exhibit the same state, i.e., the first state. Alternatively, when, for a particular picture, the random access point signaling granularity indication exhibits a second granularity, i.e., sub-picture granularity, a flush of the decoded picture buffer responsive to the sub-picture related information being in the first state for all sub-pictures of a given picture may be scheduled to be performed upon decoding of the last sub-picture of that picture.

[0017] The third and fourth aspects of the present application relate to decoder-side decoded picture buffer management. These aspects of the present application can be combined with any of the first and second aspects and have similar objectives as outlined above with respect to aspects 1 and 2. That is, the third and fourth aspects of the present application aim to provide a video codec that allows a composite data stream to be composited from sub-picture related sub-streams, so that the compositer / synthesizer does not need to harmonize or unify the reference picture buffer descriptions that describe the set of reference pictures that should remain in the DPB signaled in the various sub-streams that contribute to the composite data stream. Thereby, the composite data stream can immediately signal at the start of each picture which pictures form the set of reference pictures for the current picture, so that the decoder may remove all other pictures from the decoded picture buffer at the start of this current picture. Rather, according to the third and fourth aspects, the video codec allows the data stream to signal a reference picture buffer description, which provides information for each sub-picture of the currently coded / decoded picture individually about the set of reference pictures required for decoding the current picture or for decoding with the respective next picture, and the reference picture buffer description may be different between the sub-pictures of the current picture. Such a reference picture buffer description describes, for a current sub-picture of the current picture, the picture containing the sub-picture that serves as a reference for the current sub-picture or for the sub-picture of the next picture that is related to the current reference picture through its origin, for example, because it is related to one sub-video. If coded as a separate sub-video, the buffer description of the sub-picture actually indicates the reference pictures that contain the sub-pictures that are sub-pictures of the same sub-video and should be kept in the DPB, because they serve as references for the current or subsequent sub-pictures of that sub-video.According to a third aspect of the present application, the decoder has the capability to form a union of a set of reference pictures to remain in the decoded picture buffer for a sub-picture of this current picture according to the reference picture buffer description of the sub-picture, thereby marking those pictures to remain in the decoded picture buffer, while other pictures are deleted from the decoded picture buffer, such as at the end of the decoding of the current picture. According to a fourth aspect of the present application, the decoder can perform the decoded picture emptying operation on a sub-picture basis, rather than on a complete picture basis. The marking and deletion of sub-pictures is performed for each sub-picture separately. In other words, according to the fourth aspect, the decoded picture buffer is managed in units of one sub-picture decoded picture buffer for each sub-picture of the currently decoded picture, and the marking of sub-pictures to remain in the decoded picture buffer of each sub-picture and the operation of deleting other pictures from the decoded picture buffer is performed based on the reference picture buffer description transmitted in the data stream for each sub-picture of the currently decoded picture. As already mentioned above, the latter third and fourth aspects may for example be combined with the first aspect, so that the decoder may have the functionality as described above with respect to the third and fourth aspects, for example when the random access point signaling granularity indication exhibits the second granularity, while managing / emptying the decoded picture buffer on a picture-by-picture basis, when the random access point signaling granularity indication exhibits the first granularity, i.e. picture granularity, in which case the decoder may be configured to empty and perform marking of the decoded picture buffer based on the first encountered reference picture buffer description for the currently decoded picture, i.e. based on the first signaled reference picture buffer description for the current picture.

[0018] A fifth aspect of the present application relates to POC processing. Similar to the third and fourth aspects, the fifth aspect aims to provide a video codec that allows a data stream to be composed of substreams that are separately coded using different random access point durations, groups of pictures, or different frame rates, etc., thereby resulting in time-aligned subpictures to contribute to the composite data stream, which are actually signaled in various substreams with different POC values. The fifth aspect of the present application can be combined with any of the first to fourth aspects and aims to alleviate such composition / merging procedures by freeing the compositor / merger from the need to harmonize the POC values ​​of substreams associated with different subpictures and with different origins. Rather, the video codec allows the data stream to signal different POC values ​​for sub-pictures that actually belong to the same picture of the data stream, and the decoder is configured to derive, for each sub-picture of the currently decoded picture, a final POC value for the respective sub-picture based on the decoded POC value of the respective sub-picture, and one or more inter-sub-picture POC compensation parameters associated with each sub-picture and kept updated by the decoder such that the final POC values ​​of the respective sub-pictures for each picture of the video are equal to each other. The one or more inter-sub-picture POC compensation parameters may compensate for frame rate differences between contributing sub-streams, different sizes of groups of pictures, different POC minimum numbers, etc. The one or more inter-sub-picture POC compensation parameters may include a POC offset value and a POC step size difference. Some parameters, such as the POC offset values ​​at random access point subpictures, may be determined by the decoder itself based on the difference between their signaled POC value and the final POC value of at least one other subpicture of the same picture that does not form a random access point.On the other hand, hints regarding the POC step size differences may be conveyed in the data stream to guide the decoder and ease the decision of these POC step size differences. These hints may be inserted into the data stream by the combiner / synthesizer, which has access to this knowledge based on higher level parameters in the contributing sub-streams.

[0019] A sixth aspect of the present application relates to the idea of ​​a video codec implementing gradual decoding refresh, where the decoder is aware of the gradual decoding refresh not only in terms of a time or picture from which the gradual decoding refresh will be completed in the future, but also in terms of satisfying certain constraints in performing prediction derivation based on coding parameters signaled in the data stream during the gradual decoding refresh. In particular, according to the sixth aspect of the present application, the video decoder is configured to log, for each reference picture of the video, a subdivision of the reference picture into refresh picture regions and non-refresh picture regions. The decoder decodes from the data stream information about the refresh update region in the currently decoded picture and decodes the refresh update region from the data stream using inter prediction. Furthermore, the decoder determines a first region of the currently decoded picture that does not overlap with the refresh update region and is to be coded / decoded independently from the non-refresh picture region of the reference picture, and decodes the first region from the data stream by deriving a prediction of the first region from the reference picture based on an encoding parameter that is a signal in the data stream for the first region. In so doing, the prediction is derived in a manner that depends on the location of the refresh picture region of the reference picture such that the prediction is independent of the non-refresh picture region of the reference picture. For example, the motion vectors may be clipped to reach the non-refresh picture region, and / or padding may be used to fill portions of the motion compensated prediction blocks in the reference picture that reach beyond the refresh picture region into the non-refresh picture region. Alternatively, the construction of the motion vector candidate list may exclude motion vectors that originate from outside the refresh picture region. The second region may be processed differently by the decoder. The second region may be separated from the first region and the refresh update region and decoded by the decoder including predictions from the refresh picture region and the non-refresh picture region.By this method, the refresh picture region grows continuously from picture to picture. The decoder is aware of the growth, since it needs to keep track of the growth of the refresh picture region and needs to perform prediction derivation for the first region accordingly. Making the decoder aware of the gradual decoding refresh has several advantages: the decoder is aware of the current refresh status, possible random access periods, etc. The effort associated with the prediction derivation and the need to make this derivation dependent on the location of the refreshed picture region of the reference picture costs very little, since the decoder may need to implement similar measures, for example, at the outer boundaries of the picture or at the outer boundaries of the independently coded picture tiles. On the other hand, the encoder can code the coding parameters with a wider range of values, since the decoder redirects some signalable states of the coding parameters from predictions that may result in a dependency on non-refreshed picture regions of the reference picture to predictions that only depend on the refreshed picture regions. However, expanding the signalable states of the coding parameters may lead to higher compression efficiency with respect to the coding of these coding parameters.

[0020] Advantageous aspects of the present application are the subject matter of the dependent claims. Preferred embodiments of the present application are described below with reference to the accompanying drawings. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a schematic diagram showing a situation for synthesizing a data stream based on several input substreams or for combining substreams into a composite data stream, where a decoder is shown decoding the composite data stream and corresponding encoders for forming the substreams and the synthesizer / combiner itself; the embodiments shown with respect to FIGS. 2 to 8 are used to explain preferred functions and concepts that may be applied by the synthesizer / combiner and / or the decoder. [Diagram 2]FIG. 2 is a schematic diagram illustrating the concept of a video codec allowing indication of whether sub-picture granularity or picture granularity is used for random access point signaling; [Diagram 3] 1 shows a schematic flow diagram illustrating a function that a decoder may have for flushing the decoded picture buffer. [Figure 4] 1 shows a schematic flow diagram illustrating a preferred functionality of a decoder when performing an operation to empty a decoded picture buffer. [Diagram 5] FIG. 2 is a schematic diagram illustrating a decoder configured to perform an operation to empty a unit of a sub-picture in a decoded picture buffer; [Figure 6] FIG. 2 is a schematic diagram illustrating a possible means for dealing with sub-picture relocation of sub-pictures belonging to successive sub-videos within a picture of a composite video data stream; [Figure 7] FIG. 2 is a schematic diagram illustrating a preferred means of a video decoder for handling different POC domains for different sub-pictures of a video. [Figure 8] FIG. 8 is a schematic diagram illustrating the concept of FIG. 7 with respect to POC offset compensation in a random access point sub-picture according to one embodiment. [Figure 9] 1 is a schematic diagram illustrating an embodiment of an encoder and a decoder that allows efficient gradual decoding refresh; [Figure 10] FIG. 1 is a schematic diagram illustrating an example of 360° streaming, in which a video data stream is composed of sub-streams associated with different MCTSs of a panoramic scene. [Figure 11] FIG. 2 is a schematic diagram illustrating gradual decoding refresh. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] With respect to FIG. 1, the task of synthesizing a data stream from a reservoir of substreams is described. This illustrates the problems associated with it, namely problems related to the different origins of the substreams and differences such as the placement of random access points, the associated POC values, different GOP structures, etc., which will be described in a little later. These problems are then solved according to the embodiments described later. These embodiments relate to different aspects of the present application, which can be combined with each other to yield further embodiments, as will be further described later. However, some of the assumptions made in FIG. 1 are not necessary for the embodiments described later, or in other words, some of the features described and discussed with respect to FIG. 1 may not apply to the embodiments described later, while still achieving advantages from the aspects addressed by the respective embodiments. Such situations are also described below.

[0023] Fig. 1 shows a scenario in which a data stream 10 is generated by synthesis based on multiple substreams 12, or a reservoir of substreams 12. This synthesis or synthesis process is performed by a synthesizer / synthesizer 14. In particular, the data stream 10 is generated to have a video 16 of pictures 18 encoded therein. The arrows 20 in Fig. 1 indicate a presentation time order defined between the pictures 18 of the video 16, i.e. the order in which the pictures 18 are output by the decoder 22 when the decoder 22 decodes the video 16 from the data stream 10. As indicated by the dashed lines 26 in Fig. 1, the pictures 18 are spatially subdivided into subpictures 24, which therefore represent the boundaries between the subpictures. In particular, the data stream 10 has a video 16 encoded therein in such a way that the subpictures 24 are coded independently of each other. The independent coding of the sub-pictures into the data stream 10 and the consequent possibility of decoding the sub-pictures 24 from the data stream 10 in a mutually independent manner relates at least to the sub-pictures within one picture 18, i.e. picture-internal. For example, in Fig. 1 the pictures 18 are shown subdivided into sub-pictures 24, each picture 18 comprising six sub-pictures, each with a reference number. Thus, for example, sub-picture 3 is coded into the data stream 10 in an independent manner from the other sub-pictures of the same picture 18. Thus, sub-picture 3, shown hatched in Fig. 1, can be decoded from the data stream 10 by the decoder 22 in an independent manner from the other hatched sub-pictures 1, 2, 4, 5 and 6 of the same picture 18. However, a further feature of the data stream 10 according to Fig. 1 is that the pictures 18 of the video 16 are spatially subdivided into sub-pictures 24 in a constant manner over the sequence of pictures, i.e. their boundaries 26 coincide spatially as compared to different pictures.Moreover, the co-located or spatially aligned sub-pictures, i.e. the sub-pictures 24 with the same reference number in Fig. 1, form one sub-video in which each sub-picture 24 is coded not only independently from other sub-pictures of the same picture, but also independently from sub-pictures of other pictures that are spatially offset with respect to the respective sub-picture, i.e. from all sub-pictures belonging to other sub-videos or all sub-pictures with other reference numbers in Fig. 1. For example, in HEVC terminology, each sub-picture 24 in Fig. 1 may be an MCTS. In the absence of the latter property of coding independence from spatially offset sub-pictures of different pictures of the video 16, the sub-pictures 24 may be, for example, one or more tiles, using HEVC terminology.

[0024] The individual sub-videos formed by the spatially aligned sub-pictures 24 of the video 16 are also indicated in FIG. 1 with the number 28. That is, all these sub-videos consist of a sequence of sub-pictures 24. The encoder 30 independently encodes these sub-videos 28 into a number of sub-streams 32 or a reservoir of sub-streams 32. In fact, the sub-videos 28 may be treated by the encoder 30 in groups. A group of the sub-videos 28 may be encoded into an input data stream, which comprises a video in which the group of sub-videos 28 are encoded independently of each other, so that such an input video data stream consists of one sub-stream 32 for each sub-video 28. The sub-streams 32 are received by the combiner / synthesizer 14, which then combines the data stream 10 based on the sub-streams 32. Depending on the application, only a subset of the sub-videos 28 or only a subset of the sub-streams 32 may actually contribute to the data stream 10, and this subset may further vary or differ between pictures 18 of the video 16.

[0025] The objective of the synthesis / merger operation as performed by the synthesizer / synthesizer 14 is to perform the operation in the coded domain, i.e. requantization of prediction residuals or re-performing motion compensation is avoided by the synthesizer / synthesizer 14. Each sub-picture 24 of the sub-video 28 is coded, for example, into one or more NAL units 34 of the corresponding sub-stream 32. If the sub-picture 24 belongs to the one that should contribute to the video 16 of the synthesized data stream 10, the synthesizer / synthesizer 14 inserts the respective one or more NAL units into the data stream 10, as indicated using the same reference number, i.e. 34, but with an apostrophe to indicate that the synthesizer / synthesizer 14 may need to adapt other coding parameters in these NAL units, other than the motion vector information or the prediction residual data. According to an embodiment described herein below, the synthesizer / synthesizer 14 is given the opportunity to reduce the portion of the sub-stream 32 that is to be adapted when forming the data stream 10 based on the sub-stream 32. This also relates to the parameter set of the sub-stream 32 or the input video data stream in which it is included, if the combiner / synthesizer 14 aims to form a corresponding parameter set 36 in the data stream 10 .

[0026] The situation of FIG. 1 can be further explained by referring to FIG. 10. FIG. 10 shows the case where the reservoir of sub-videos 28 forms a sub-section of a 360° video. In the example of FIG. 10, the entire scene was divided into 24 sub-videos. In the example of FIG. 10, many sub-videos were provided, i.e. 24 sub-videos representing the scene in high resolution and encoded by the encoder 30 into two input video data streams 38a and 38b, sometimes more efficiently using a large RAP distance and other times resulting in a lower compression efficiency using a small RAP distance, for example input video data stream 38b. Similarly, two further input video data streams are provided, each having a complete scene with 24 sub-videos encoded, again one 38c with a large RAP distance and the other 38d with a small RAP distance. Picture 16 of the combined video data stream was shown as showing the complete scene. In particular, each picture 16 is composed of 12 mutually temporally aligned sub-pictures of the sub-videos encoded in the data streams 38a and 38b, and 12 mutually temporally aligned sub-pictures and temporally aligned to the high-resolution sub-pictures obtained from the data streams 38c and 38d. That is, in the example of FIG. 10, the viewport was assumed to be 12 sub-pictures or tiles wide. In the center of FIG. 10, the hatching indicates a selection of sub-pictures 24 in the picture 16 of the composite data stream, which contribute in high resolution and low resolution to the picture 16 of the composite video. Another selection is made for different RAP distance versions. For each high-resolution sub-picture 24 and each low-resolution sub-picture 24, for a particular picture instant, the data streams 38a and 38c each have one or more NAL units efficiently coded using a large RAP distance, and the other data streams 38b and 38d each have NAL units coded using a small RAP distance. Preferably, the composite data stream 10 is constructed using NAL units of the efficiently coded data streams 38a and 38c.All data streams 38a-38d may have sub-streams 38 encoded therein using synchronized random access points within each data stream 38a-38d individually, i.e., the video pictures encoded in each of the data streams 38a-38d are each divided into 24 sub-pictures 24, and for each of those pictures, all or none of the sub-pictures are encoded into the respective input video data streams 38a-38d in a random access point fashion.

[0027] A problem arises whenever the viewport changes, i.e. the selection of contributing substreams that contribute to the composition / integration of the data stream 10 changes. In the example of FIG. 10, 96 substreams are available, but for a particular picture, only 24 contribute to the composite data stream 10. For example, assume that the viewport changes by two subpictures or tiles 24. Thus, four of the subpictures 24 of the picture 16 change the substream from which they originate. That is, two of these four originating from the high resolution data stream 38a are in this case taken from one of the low resolution data streams 38c and 38d, and the other two subpictures 24 originating from the low resolution data stream 38c are in this case taken from the low resolution data stream 38d. In particular, in order to reduce the time to encounter the next random access point, the low RAP distance versions of the data streams 38b and 38d, respectively, are used preliminarily for these four subpictures 24. The four subpictures change their position within the picture 16. However, all other sub-pictures 24 remain downloaded in their respective more efficient versions of data streams 38a and 38c. They do not require random access points. It should be noted that, as a variant, the synthesizer / synthesizer 14 may insert corresponding NAL units of the more efficiently coded input video data stream 38c into the data stream 10 for those sub-pictures 24 that change from high resolution to low resolution using downsampling from high resolution to low resolution, thereby deriving reference pictures for the corresponding non-RAP sub-pictures in low resolution.

[0028] That is, the example scenario described in combination with Fig. 1 and Fig. 10 presents the following problems related to 360° streaming: In particular, when a user joins the service, i.e. tunes in to the service, i.e. wants to watch a panoramic video, a random access point picture is required. That is, without the embodiments described below, the aggregated data stream 10 would require each sub-picture in the start picture to be coded in the random access point manner. That is, all NAL units of that picture would need to be coded in the random access point manner. In HEVC language, for example, they would need to be NAL unit types IDR, CRA, or BLA. That is, without the embodiments described further below, the decoder would be forced to encounter a RAP picture to start decoding. However, as discussed above with respect to FIG. 10, when a user switches viewports, only some sub-bitstreams 32 are newly added to the subset of bitstreams 32 that contribute to or are merged into data stream 10, so it would be highly inefficient to require all sub-bitstreams 32, i.e., even those that remain in the subset of contributing bitstreams, to have their sub-picture 24 coded in the starting picture with the random access point method. Rather, as outlined with respect to FIG. 10, it would be preferable if only some sub-picture bitstreams 32 effectively had RAPs and others did not. However, this would mean that the NAL unit type indicated by NAL unit 34′ in data stream 10 would need to be changed compared to the setting of the underlying NAL unit 34. In particular, the NAL type of NAL unit 34 of sub-picture 24 of the starting picture, set to IDR, CRA, or BLA, may not be associated with the same NAL unit type of the corresponding version 34′ and data stream 10.Because the requirements of the data stream require that if one of the sub-picture bitstreams has one of these NAL unit types, all the others have the same NAL unit type. In particular, when a decoder as defined so far finds an IDR or CRA (previously with EOS NALU) NAL unit for a particular access unit of the data stream 10, i.e. for a particular picture 18 of the video 16, it will restart the decoding process and flush its decoder picture buffer (DPB) since none of the pictures there are available for reference or are used. This means, for example, that when merging the sub-bitstreams 32 together, provisions must be made for the case that for a certain picture 18 of the video 16, not all contributing sub-bitstreams 32 contain the NAL unit type corresponding to random access point coding, but only a small part of them do. Rather, RAP NAL units in such pictures 18 would have to be rewritten to non-RAP NAL units, a process that would also require modification of other headers, such as slice, tile, and / or tile group headers, because their content depends on whether the NAL unit in question is a RAP or not.

[0029] Therefore, the embodiment described below with respect to FIG. 2 attempts to improve the signaling of sub-picture RAPs beyond the previous techniques. For example, there are already proposals to move away from the above-mentioned constraint of assigning NAL unit types to NAL units of the data stream such that the NAL type with which the sub-picture is coded is constant within each picture. For example, US Pat. No. 6,233,933 has already proposed to use NAL unit-based signaling, for example in the NALU header, more specifically a dedicated NALU type indicating the sub-picture random access point type, or a recovery point SEI message containing substitution information substituting all the parameters required to obtain a HEVC compliant data stream. However, solving the above problem in the way proposed in that document still requires the synthesizer / synthesizer 14 to perform the tedious task of "stream translations". The embodiment outlined below therefore seeks to overcome this problem and avoid the need for the synthesizer / synthesizer 14 to expend additional RAP type signaling overhead.

[0030] Fig. 2 shows a concept of a video codec according to an embodiment of the present application with respect to the first aspect of the present application. Fig. 2 shows an embodiment of a corresponding data stream 10 and a video decoder 22 that can be adapted to a new video codec that addresses the above-mentioned problems. The reference symbol 10 of Fig. 1 is reused since the data stream 10 of Fig. 1 can also correspond to the description described below with respect to Fig. 2, and thus can at least partially address the above-mentioned problems.

[0031] That is, the video decoder 22 of FIG. 2 receives a data stream 10 having video of coded pictures. The data stream 10 has video coded therein in such a way that sub-pictures, into which a video picture is spatially subdivided, are coded independently of each other. Each sub-picture is coded in one or more NAL units 34', respectively, of the data stream 10. That is, the NAL units 34' convey, among other things, motion information and prediction residual information. Besides the explanation given above with respect to FIG. 1, the independence of the sub-picture coding may only relate to a sub-picture within one picture 18 of the video coded in the data stream 10. Thus, FIG. 2 exemplarily shows one picture 18, its sub-division into sub-pictures 24, and the corresponding NAL units 34' in which the sub-pictures 24 are coded in the data stream 10. Of course, this coding may also be done to generate separately coded sub-videos, as described above with respect to FIG. 1.

[0032] The decoder 22 decodes sub-picture related information 40 for each sub-picture 24 of each picture 18 from the data stream 10. The sub-picture related information for a particular sub-picture 24 may be, for example, a syntax element indicating the NAL unit type of the one or more NAL units 34' in which the respective sub-picture 24 is coded. When a first condition is present, the sub-picture related information 40 identifies the sub-picture 24 as being coded in the one or more NAL units 34' in a random access point manner, i.e., it forms a random access of the sub-picture. That is, the sub-picture will be coded in the one or more NAL units 34' in a self-contained manner without reference to other sub-blocks. When a second condition is present, the sub-picture related information will identify the sub-picture 24 as not being coded in a random access point manner, i.e., a NAL unit 34' having such sub-picture related information 40 may have associated sub-blocks 24 coded therein using coding dependencies to other sub-blocks. In particular, sub-picture related information 40 may include an integer-valued syntax element for each sub-picture 24 that distinguishes, from the plurality of NAL unit types, a first subset of one or more NAL unit types that correspond to a first state, i.e., a sub-picture RAP state, and that distinguishes a second subset of one or more NAL unit types that correspond to a second state, i.e., a non-RAP state.

[0033] However, the decoder 22 of Fig. 2 is further configured to read a random access point signaling granularity indication 44 from the data stream 10 with respect to a set 42 of one or more pictures. The indication may be conveyed in the data stream 10 as some kind of high level signaling related to a plurality of pictures. That is, in that case the set 42 of one or more pictures 18 would comprise a sequence of pictures such as a GOP, or even the entire video 16. The indication 44 may for example be included in a sequence parameter set or a video parameter set of the data stream 10. On the other hand, the indication 44 may relate only to one access unit, i.e. one picture 18 of the video 16. In other words, the set 42 may comprise one picture. Such per-access-unit signaling may be included in a PPS, i.e. picture parameter set, of the data stream 10, or in an access unit delimiter, etc. This indication 44 would have at least two signallable states / granularities. When exhibiting or signaling a first granularity, the indication 44 will indicate that for each picture 18 of the set of one or more pictures 42, the sub-picture related information 40 will exhibit a common state for all sub-pictures 24 of the respective picture 18. In other words, the sub-picture related information 40 will exhibit one equal state for all sub-pictures 24 within one picture 18 in the set 42. This sameness may also be true when considering the level of NAL unit types. That is, the NAL unit types of the NAL units 34' belonging to the sub-pictures 24 of one picture 18 may be required to be identical, i.e. equal to each other, in case of the indication 44 signaling a first granularity, i.e. picture granularity.

[0034] However, when the instruction 44 indicates a second granularity, which may be called sub-picture granularity, this indicates that for each picture 18 in the set 42, the sub-picture related information 40 may have different states for the sub-pictures 24 of the respective picture 18.

[0035] 2 can decode instructions 44 from the data stream and, based thereon, determine whether data stream 10 is of a type that requires sub-picture related information 40 to be the same in each picture 18, or whether data stream 10 is of a type where such a restriction does not apply. Decoder 22 can operate differently in response to instructions 44 with respect to one or more decoding operations, which are described in more detail below.

[0036] In any case, however, said synthesizer / synthesizer 14 may utilize the possibility of setting the indication 44 to sub-picture granularity in the following way: The synthesizer / synthesizer 14 may synthesize an access unit of the data stream 10 by aggregating NAL units 34 of the reservoir of the coded sub-stream 32, which are related to the temporally aligned sub-picture 24 of the current picture to which the access unit refers, such as 46 in FIG. 2, and in so doing may inherit, adopt or keep without modification the NAL unit type of the NAL units, such that the NAL units 34′ of the data stream 10 in this access unit 46 match the aggregated NAL units in the sub-picture related information 40.

[0037] In the following, several embodiments are described that explain the functionality of the decoder 22 of FIG. 2 for the case where the instructions 44 indicate sub-picture granularity. However, it should be understood that all these embodiments and their descriptions can also be applied to decoders that are not configured to read or decode the instructions 44 from the data stream 10. In other words, the embodiments described below are not limited to the case where the instructions 44 are present in the data stream and the decoder 22 is configured to read the instructions from the data stream 10. Rather, the embodiments described below of further aspects of the present application should also be interpreted as a description of a decoder that can deal with the fact that certain settings, such as random access point coding or non-random access point coding of sub-pictures, change within one picture. However, these decoders may not have any particular alternative functionality, as is true for the decoder 22 of FIG. 2 that operates differently depending on the instructions 44.

[0038] To briefly summarize what has been explained so far, the idea illustrated in Fig. 2 is that for a CVS that is a concatenation of coded video sequences, an additional indication 44 in the data stream 10, such as a higher level signaling, such as a signaling at the coded video sequence level or at the bitstream level, can reinterpret a NAL unit level signaling, such as a NAL unit type signaling, as a sub-picture random access indication. For example, the indication 44 can be signaled in the form of a flag that can be referred to as sub_picture_random_access_process_enabled_flag. For example, if this flag is equal to zero, the NAL unit types indicating the random access capability will be interpreted by the decoder 22 as a whole picture level random access. On the other hand, if the value of the flag is equal to one, the respective NAL unit type will be interpreted as a sub-picture random access point.

[0039] Alternatively, the indication 44 can be implemented as per-access-unit signaling. For example, an access unit delimiter NAL unit can be used to indicate what type of random access is allowed within the access unit starting from that access unit delimiter NAL unit. A syntax element can be added to such an access unit delimiter NAL unit to indicate one of the following conditions: (1) all sub-pictures 24 of the picture of the access unit to which the delimiter belongs contain one random access picture (IRAP), (2) some sub-pictures 24 may contain one random access picture (IRAP), or (3) none of the sub-pictures 24 contains a random access picture (IRAP).

[0040] Using the instruction 44, the decoder can easily identify what type of random access function can be applied to the pictures 18 of the set 42 that define the scope of the instruction 44, i.e. the scope to which the instruction 44 relates, and whether one of the following processes described in relation to the embodiments below should be performed by the decoder 22:

[0041] Before resuming the description of what the decoder 22 does when the instruction 44 indicates sub-picture granularity, it should be noted that if the instruction 44 indicates picture granularity and, for any picture 18 of the set 42 to which the instruction 44 belongs, the sub-picture related information 40 exhibits different states for different sub-pictures 24 of that picture, the decoder 22 of FIG. 2 may be configured to identify the data stream 10 as a non-compliant data stream, i.e., as not compliant with the video codec.

[0042] Fig. 3 shows an embodiment of the behaviour of the decoder 22 with regard to flushing the decoded picture buffer, that is, the buffer in which the reference pictures are buffered, that is, the pictures on the basis of which the not yet decoded pictures can be predicted. The solid lines in Fig. 3 show the functionality of the decoder 22 in the case of sub-picture granularity, and the dashed lines show the corresponding functionality in the case of picture granularity. It should be noted that Fig. 3 should again be treated as a description of both cases: an extension of the functionality described above for the decoder described with reference to Fig. 2, capable of decoding the indication 44 and operating according to any of the modes shown in Fig. 3, or a description of an alternative decoder 22 in the case where the data stream 10 does not include the indication 44 and the decoder 22 necessarily operates according to the sub-picture granularity mode.

[0043] As shown in Fig. 3, the decoder 22 according to the embodiment of Fig. 3 is configured to check, for a particular access unit or a particular picture 18, whether the sub-picture related information 40 is in a random access point state for all sub-pictures 24 of that picture, as indicated in step 50. If the answer is true, the decoder flushes the decoded picture buffer 48 accordingly in step 52. Again, the decoder 22 can apply this functionality of steps 50 and 52 for pictures 18 for which the indication 44 indicates sub-picture granularity, whereas, if the indication 44 indicates picture granularity, it can apply an alternative approach, indicated by the dashed lines in Fig. 3. According to this alternative approach, the decoder 22 checks, for a particular access unit or picture 18, whether the sub-picture related information 40 for the first encountered sub-picture 24 within the access unit or of the respective picture is random access point coded in step 54, and if the answer is true, the decoder flushes 56 the decoded picture buffer 48 accordingly. A different behavior of the decoder 22 for flushing the decoded picture buffer 48, i.e., on the one hand, when the indication 44 indicates sub-picture granularity, and on the other hand, when the indication 44 indicates picture granularity, may result in a different scheduling of the flushing of the decoded picture buffer in steps 52 or 56, i.e., relative to the time when the decoder 22 starts to decode the individual NAL units 34′ of an access unit 46, such as a scheduling relative to the time when the decoder 22 retrieves the NAL units 34′ of that access unit 46 from the coded picture buffer. If it is clear that all sub-pictures of the current access unit are random access points, the flushing of the decoded picture buffer 48, i.e., flushing in the sub-picture granularity domain, may be performed in step 52 when decoding one or more NAL units 34′ of the last sub-picture 24 in the decoding order 58.On the other hand, in the case of step 56, i.e. flushing of the decoded picture buffer 48 in the picture granularity domain, may be performed when decoding one or more NAL units 34' of the first sub-picture 24 in the encoding / decoding order 58. That is, flushing may be performed even before decoding the second sub-picture 24 in the encoding / decoding order 58 of the current picture or access unit, if it is clear that the sub-picture related information 40 present in the data stream for that sub-picture indicates random access point encoding. Again, the functionality described in relation to the picture granularity mode with reference to FIG. 3 may not be implemented in the decoder 22 according to alternative embodiments in which the instruction 44 is not present.

[0044] That is, Figure 3 illustrates an embodiment in which the decoding process is modified to flush the decoded picture buffer 48 of the decoder 22 and perform a random access point operation only if it is clear that all sub-pictures 24 of a particular picture in the data stream 10 are sub-picture random access points, and therefore the entire picture is a random access point itself. Thus, the decoder 22 may be configured to decode up to the last NAL unit 34' in the current access unit to determine whether this access unit is a complete random access point.

[0045] The decoder embodiment described next with respect to Fig. 4 focuses on the decoder's functionality / operational modes related to the deletion of pictures stored in the decoded picture buffer 48 that are not required for individual reference pictures or pictures not yet decoded. That is, while Fig. 3 focuses on flushing the decoded picture buffer 48, i.e., deleting all pictures currently buffered in the respective decoded picture buffer, Fig. 4 focuses on another manner in which the data stream 10 may convey a reference picture buffer description, i.e., information about reference pictures that should remain in the decoded picture buffer 48 to serve as a possible basis for inter-prediction of pictures not yet decoded. For this purpose, the data stream 10 conveys information about a set of reference pictures, the information of which is updated per picture or per access unit. Similar to the sub-picture related information 40, this information regarding the reference picture set to be maintained in the decoded picture buffer for future use, i.e. the reference picture buffer description, is signaled in the data stream 10 for each sub-picture 24, i.e. in its corresponding one or more NAL units 34'. According to a variant illustrated in Fig. 4 using solid and dashed lines, this reference picture buffer description may be information subject to similar constraints depending on the indication 44, as has been explained so far for the sub-picture related information 40 for indicating RAP and non-RAP coding. Therefore, the reference picture buffer description is illustrated in Fig. 2 using the reference symbol 40' to indicate the similarity regarding the constraints depending on the indication 44, or to represent as an analogous indication that may be present alternatively or additionally to the indication 44.In particular, it may be required that the set of reference pictures indicated by the buffer description 40' for the first sub-picture 24 of a particular picture 18 is equal to the set of reference pictures indicated by the buffer description 40' for any other sub-picture 24 of the same picture 18, or at least includes all reference pictures indicated by the reference picture buffer description 40' for the second and subsequent sub-pictures in the encoding / decoding order 58 of that picture 18. In the case of an indication 44 indicating sub-picture granularity (or a corresponding indication for a reference picture buffering process), the sets of reference picture sets indicated by the buffer description 40' for various sub-pictures 24 of one picture 18 may freely differ from each other in any way, i.e. may indicate mutually different sets of reference pictures. Mutually different means that the sub-pictures in one picture and their corresponding successor sub-pictures in a subsequent picture require different reference pictures, i.e. different sets of reference pictures. For example, their corresponding predecessor sub-pictures, which originate from predictive references, are included in mutually different sets of reference pictures. The mutual correspondence of the sub-pictures is due, for example, to their origin, i.e., they could have been coded separately in the corresponding sub-stream 32 as sub-video 28. The latter degree of freedom allows the synthesizer / aggregator to synthesize the data stream 10 based on the NAL units 34 of the reservoir of the sub-stream 32 by simply taking over the reference picture buffer description 40' without modification when putting together NAL units to form one access unit of the data stream 10. Thus, the reference picture buffer description 40' will be identical between the NAL units 34' in the final data stream 10 and the NAL units 34 in the reservoir of the sub-stream 32.

[0046] Before resuming the description of Fig. 4, it should be noted that this means that the data stream of the embodiment of the figure does not necessarily include the sub-picture related information 40. The embodiment described with respect to Fig. 2 may or may not include the reference picture buffer description 40'. Then, similar to what has been described above with respect to Fig. 3, the description of Fig. 4 may be interpreted as describing a possible extension / modification of the embodiment described so far with respect to Figs. 2 and 3, and as a self-contained embodiment in which the decoder only performs the functions indicated using solid lines in Fig. 4. That is, the instructions 44 in particular may not be present in the data stream.

[0047] According to Fig. 4, the decoder 22 is configured to perform picture deletion from the decoded picture buffer 48 as follows. In particular, the decoder performs the operations of marking, in step 60, pictures in the decoded picture buffer 48 that should remain in the decoded picture buffer 48 to serve as prediction references for pictures not yet decoded, and deleting, in step 62, unmarked reference pictures in the decoded picture buffer 48, i.e. reference pictures that are not marked and are not required for future display / output by the decoder, by performing these steps 60 and 62 cyclically for each picture. However, to perform the marking 60, the decoder decodes (64) a buffer description 40' indicating a reference picture set RefPi for each sub-picture i of the current access unit or current picture. The pictures in the decoded buffer 48 that should be marked in step 60 are then identified by forming a union of the sets RefPi for all sub-pictures of the current access unit or picture (66). Thus, the removal of unmarked reference pictures in the decoded picture buffer in step 62, shown in Figure 4, may be performed when decoding the last sub-picture 24 in the encoding / decoding order of the current access unit or picture. That is, the removal 62 may be performed after parsing or decoding all sub-pictures 24 of the current picture, or after obtaining at least all sub-pictures 28, more precisely all NAL units 34', from the decoded picture buffer. Of course, unmarked pictures are only removed if they are not needed for output, either because they are not output pictures or because they have already been output.

[0048] In the case of a decoder 22 capable of operating in the sub-picture granularity mode described with respect to steps 60-66, as well as in the picture granularity mode shown in dashed lines in Fig. 4, i.e. when the decoder 22 is able to switch between both modes in response to an indication 44, the decoder 22 may alternatively operate as follows when deleting pictures from the decoded picture buffer. In particular, when the picture granularity mode is active, the decoder 22 may still perform the marking 68 of reference pictures to be kept in the decoded picture buffer 48 and the deletion of unmarked reference pictures from the decoded picture buffer 48 in step 70 on an access unit / picture basis, but different behaviors may be applied for determining the reference pictures to be marked and for scheduling the deletion 70. In particular, the decoder may decode in step 72 a buffer description informing about the set of reference pictures of the sub-picture 28 first encountered in the encoding / decoding order 58, and may identify (74) in step 68 the reference picture to be marked as part of that set, i.e. as RefP1 in the case of an index corresponding to the decoding / coding order. Of course, the decoder 22 can also read / decode reference picture buffer descriptions 40' from the data stream 10 for other sub-pictures than the first one in the encoding / decoding order 58 of the current access unit or current picture, but this information is not used for the marking in step 68. Furthermore, the removal of unmarked reference pictures in step 70 can be done before decoding the second sub-picture in the encoding / decoding order 58, or in other words after parsing or decoding the first sub-picture in the encoding / decoding order 58. Of course, unmarked pictures are only removed if they are not needed for output, either because they are not output pictures or because they have already been output.

[0049] In accordance with the possibilities described above, if the decoder 22 also reads reference picture buffer descriptions for a subpicture 28 other than the first subpicture in the encoding / decoding order 58 for a particular access unit, the decoder 22 may be configured to identify the data stream 10 as a non-conforming data stream, in which case the set of reference pictures indicated by the buffer description 40' of any subpicture 28 other than the first subpicture in the encoding / decoding order 58 contains a reference picture that is not included in the set of reference pictures indicated by the buffer description 40' of the first subpicture 24 of that picture.

[0050] Figure 5 shows an alternative to the concept outlined above with respect to figure 4. According to figure 4, the decoded picture buffer 48 has been emptied in units of pictures. According to the sub-picture granularity mode, measures have been taken to remove only those pictures that are no longer required either for output or for reference to any sub-picture. According to the embodiment of figure 5, the decoder 22 can perform an operation of emptying the decoded picture buffer 48 in units of sub-pictures 24. With regard to the possible combination of this embodiment with the previous embodiment, the same modes apply as provided above with respect to figure 4. That is to say, the embodiment now described with respect to figure 5 can be combined with the description of the decoder of figure 2 or figure 3, and thus a decoder implemented according to figure 5 may or may not be configured to alternatively operate in a picture granularity mode, e.g. in response to an instruction 44.

[0051] The concept according to Fig. 5 is now explained. In particular, the data stream 10 decoded by the decoder 22 has a video 16 encoded therein in such a way that the video picture 18 is always sub-divided into spatially non-overlapping sub-pictures 24 or sub-pictures that are aligned with each other and are encoded such that the sub-pictures that belong to different sub-videos are encoded independently of each other. That is, this encoding is performed within the sub-videos 28, thereby resulting in a reservoir of sub-streams 32. As described above with respect to Fig. 1, each sub-stream 32 has a respective sub-video 28 encoded therein, each sub-video 28 containing a possible candidate for a particular sub-picture, i.e., a sub-picture 24 within a picture 18 of the video 16 of the data stream 10. The sub-streams 32 are encoded independently of each other. They contain, for each sub-picture 24, one or more NAL units 34. As part of the encoding process, the NAL units 34 may already possess a sub-picture identifier 80. That is, each NAL unit 34 may be tagged with such a sub-picture identifier 80 to allow differentiation between the reservoirs of the sub-stream 32. As explained with respect to FIG. 1, simply one subset, or more precisely a suitable subset of the reservoirs of the sub-stream 32, may be selected to use the NAL units 34 with the corresponding sub-pictures of the current timestamp encoded therein, and may form the access unit 46 of the data stream 10 by combining them. Upon combining them, the NAL units 34' resulting in the data stream 10 still have the sub-picture identifier 80 therein. In FIG. 5, this is shown in the following way: the identifiers 80 of the reservoirs of the bitstream 32 are differentiated by integers 1...N. The pictures 18 of the video 16 encoded in the data stream 10 each have six sub-pictures 24.For each sub-picture 24 in a picture 18, the data stream 10 has one or more NAL units 34' in the corresponding access unit 46, which are tagged with sub-picture identifiers 80Ai, i=1...6, Ai∈{1...N}. Which sub-videos 28 or sub-streams 32 are selected to synthesize the data stream 10 may be application dependent, or may be constant or variable in time. Furthermore, each NAL unit 34 and their versions 34' compiled in the data stream 10 are shown as including a reference picture buffer description 40' as described above. As is evident due to the separate encoding of the sub-streams 32, this information 40' is different for the different sub-streams 32.

[0052] In performing the operation of emptying the decoded picture buffer 48 on a sub-picture 24 basis, the decoder 22 operates as follows. Generally speaking, the decoder 22 decodes from the data stream 10, for the currently decoded picture 18, a buffer description 40' for a set of reference pictures for each sub-picture 24 of the currently decoded picture 18. Then, for each sub-picture 24 of the currently decoded picture 18, the decoder uses the reference picture buffer description 40' for each sub-picture 24 to identify sub-pictures in the decoded picture buffer 48 that are spatially collocated with the respective sub-picture, i.e., that belong to the same sub-video 28, that should remain in the decoded picture buffer 48. The other sub-pictures are deleted. Thus, this deletion is performed on a sub-picture by sub-picture basis.

[0053] As shown in Figure 5, the video decoder may, for this purpose, logically divide the decoder picture buffer 48 into sub-picture DPBs 481-486, one for each sub-picture 24 into which picture 18 of video 16 is currently spatially subdivided. As a side note, while picture 18 is shown in the figures of this application as being divided into six sub-pictures, this is for illustrative purposes only; other numbers may be feasible as well. i Each of the sub-picture DPBs 48 is used to buffer sub-pictures 24 of a picture 18 emerging from a NAL unit 34' in the data stream 10 that is tagged with a particular sub-picture identifier 80 associated with the respective sub-picture DPB. i Let A be a given subpicture identifier A. i Therefore, each subpicture DPB48 i is the given subpicture identifier A i For the currently decoded picture, the decoder 22 performs the following: i , the decoder 22 determines whether the currently decoded picture 18 is a sub-picture DPB 48 i A given subpicture identifier A associated with i If so, decoder 22 uses the set of reference pictures indicated by its buffer description 40' for the corresponding subpicture to determine whether its subpicture DPB 48 contains a corresponding subpicture 24 with a subpicture identifier 80 equal to i and the sub-picture DPB48 of the decoded picture buffer 48 i Each subpicture should remain in DPB48 i Identifies the subpictures in DPB48. iAll these subpictures in the subpicture DPB48 i By collecting only those sub-pictures encoded in NAL units 34' that are tagged with a particular identifiable sub-picture identifier within a particular sub-video 28, however, a particular sub-picture DPB 48 may belong to the same sub-video 28. i For example, if NAL unit 34' is not present in the current access unit and therefore the corresponding sub-picture 24 is not included in the current picture 18, then decoder 22 derives the corresponding sub-picture DPB 48. i Flash and each subpicture DPB48 i to another predefined sub-picture identifier of the non-corresponding sub-picture 24 of the currently decoded picture 18, the other predefined sub-picture identifier being unequal to the predefined sub-picture identifiers of all the sub-picture DPBs 481-486. Instead of flushing, depending on the application, the decoder may subject the reference sub-pictures contained therein to sub-sampling or the like in order to reuse them as reference sub-pictures for the sub-picture 24 of the predefined sub-picture identifier to which the sub-picture DPB was reallocated.

[0054] Before proceeding with the description of the present application with the description of further embodiments, the above-mentioned embodiments with respect to Fig. 4 and Fig. 5 are summarized again in different words. In particular, Fig. 4 showed the concept of full picture dumping, while Fig. 5 presented the embodiment of partial picture dumping. Both embodiments have in common that they deal with the situation where the operation of emptying the decoded picture buffer is performed without responding to a random access point, but according to the reference picture buffer description, i.e., information about reference pictures / sub-pictures that are no longer required for reference. As mentioned above, a random access point is not the only case where picture information is determined to be no longer required for reference and can therefore be deleted from the respective decoded picture buffer (if it has already been output). In AVC or HEVC, pictures in the decoded picture buffer are marked as "used for reference" or "not used for reference" during the decoding process. For example, in HEVC, there is a signaling called Reference Picture Set (RPS). The RPS indicates which pictures in the decoded picture buffer are used for reference of the current picture or the pictures that follow in the decoding order. That is, the RPS (RefP) signals which pictures should be marked as being used for reference: those that do not appear in the RPS are marked as "not used for reference" and can therefore be removed from the decoded picture buffer if they are not needed for output (e.g., if they have already been output).

[0055] As explained above, in the 360° case where only some tiles / subpictures are switched from high resolution to low resolution or vice versa, the random access points may be aligned since the reference pictures may be different.

[0056] As for the RAP case, when several bitstreams are spliced ​​together, i.e. converted into sub-picture bitstreams in a new bitstream 10, the RPS needs to be rewritten, a process which would increase the complexity of such operations to be performed by the combiner / synthesizer 14.

[0057] Although new video codecs may have different ways of marking reference pictures, the following problem will always apply: different references may apply to different sub-pictures in the bitstream, and if the marking process is performed for the first slice / tile of an access unit as shown with dashed lines in Fig. 4, and this needs to carry all the information of all slices / tiles in an access unit, a header rewrite will be required when merging streams.

[0058] The idea of ​​the embodiments of Figures 4 and 5 is therefore to modify the process of reference picture marking in order to allow random access or provision and removal of sub-pictures from the decoded picture buffer.

[0059] According to the embodiment of Fig. 4, the deletion of the decoded picture buffer is still done on a picture-by-picture basis, but random access of sub-pictures is still allowed. According to a possibility of implementing the embodiment of Fig. 4, signaling at sequence or picture level, such as instruction 44, defines whether the marking process is performed in a picture-granular manner, as indicated using dashed lines in Fig. 4, or whether the marking process is performed only after the last sub-picture of the current access unit, such as after parsing or decoding a NAL unit or slice or tile of the last sub-picture of the current access unit. Signaling to switch between both methods may be included in a parameter set, such as SPS or PPS, in the form of sub_picture_management_process_enabled_flag.

[0060] In the case of sub-picture processing, which may be the only mode in which switching between both modes is off as outlined above, the process may include, according to one implementation, generating a temporally marked picture list for each sub-picture 24 of the current picture 18. Finally, it is then checked which ones are marked as "not used for reference". For example, using the RPS signaling 40' from HEVC as an example of reference picture indication, the following pseudocode could be used to identify pictures that should remain in the decoded picture buffer:

[0061] N=number of pics in DPB. Mark N pics as “not used for reference” For i=0…NumSubPicBitstream { If Pic in active RPS => mark that picture as “as used for reference” }

[0062] When all sub-picture bitstreams for the current access unit have been decoded, remove from the DPB all pictures that are marked as "unused for reference" and have already been output.

[0063] However, compared to the approach of rewriting all the RPS information, this approach requires more memory in the decoded picture buffer.

[0064] In an alternative implementation of the embodiment of Fig. 4 for the sub-picture DPB deletion mode, a first process is performed at the beginning of each picture, i.e., when processing the first slice / tile or sub-picture, by marking all pictures as "not used for reference", in addition, on the other hand, for each analyzed slice / tile / sub-picture, another process is performed to mark the indicated pictures for each slice / tile / sub-picture as "used for reference". Reference picture dumping according to the final result, i.e., when analyzing / decoding the last slice / tile / sub-picture, is performed after the reference pictures have already been output and all slices / tiles / sub-pictures of the current picture have been transmitted / received.

[0065] 5 relates to the case where the content is coded with motion constrained tiles, or MCTS, or more generally, sub-pictures coded independently of spatially offset sub-pictures, or sub-pictures belonging to different sub-videos, which contain sub-pictures of the same picture as well as sub-pictures of other pictures. In such cases, a sub-picture identifier can be used to mark the sub-picture-specific region so that it can be removed from the decoded picture buffer. An MCTS coded bitstream can only refer to itself.

[0066] In Fig. 5 the possibility of separately marking sub-picture regions of picture 18 is exploited. If a sub-picture is marked as "unused for reference" and the picture has already been output, its region may be removed from the decoded picture buffer. Thus, instead of having to wait until parsing all NAL units of an access unit, the process of marking each sub-picture separately could be performed while parsing the NAL unit type and RPS in the slice / tile group or sub-picture header.

[0067] This is done by dividing multiple sub-pictures into DPBs 48 based on information in the data stream, e.g., some parameter set conveyed in the data stream. i 5 by initializing the subpicture DPB, or by reallocating the subpicture DPB, or by allocating DPB space for each newly occurring subpicture identifier (for each new sub-video). After the initialization, a picture (meaning the reconstructed samples and corresponding referenceable data for prediction such as motion vectors) is assigned the subpicture identifier A. i Corresponding sub-picture DPB48 i The subpicture identifier A i could be, for example, one tile group ID.

[0068] Next, the reference picture set, i.e. the list of which pictures need to be kept in the decoded picture buffer 48 with their corresponding IDs for prediction, is analyzed, which affects whether a sub-picture in its corresponding sub-picture DPB can be removed. Obviously, in some scenarios, the tile configuration in a picture may change. For example, in a sub-picture bitstream merging scenario, it may happen that sub-pictures with IDs 0 and 1 are merged for a period of time, and then sub-pictures with IDs 1 and 2 are merged. In such a case, the sub-picture DPB is flushed based on the parameter set instructions. Whenever the activated parameter set does not contain the previous sub-picture ID, the sub-picture DPB corresponding to that sub-picture is flushed. Obviously, the removal of pictures is only done if they are not intended for output, i.e. they have already been output or are no longer needed for output.

[0069] Some remarks are necessary with respect to Figures 4 and 5. For example, in addition to what has been discussed above with respect to Figure 5, it would be possible to modify the management of the sub-picture DPBs to the extent that the number of sub-picture DPBs managed simultaneously exceeds the number of sub-pictures 24 in the pictures 18 of the video 16. For example, for each sub-picture identifier 80 encountered in the reservoir of the sub-stream 32, or more precisely, encountered in the data stream 10, the decoder 22 could provide a separate sub-picture DPB.

[0070] It should further be noted that the description of Figure 5 focuses on sub-picture granularity processing with respect to decoded picture buffer emptying operations. However, while Figure 5 describes a decoder that can operate accordingly, Figure 5 should also be seen as a description of a decoder that can operate in the manner shown in Figure 5, i.e., in sub-picture processing mode, as well as in a picture-by-picture processing mode, such as in response to instruction 44 shown in Figure 2 or equivalent signaling. In that case, the decoder 22 of Figure 5 would be able to manage the decoded picture buffer 48 on a picture-by-picture basis in the manner shown in Figure 4 using dashed lines.

[0071] With respect to the above buffer descriptions 40', it should be noted that they may indicate the reference pictures that should remain in the DPB in any manner, such as in list form or in analytical terms indicating the corresponding reference pictures using POCs, etc. The buffer descriptions 40' may either actively reference these pictures or may reference pictures that should be removed from the DPB.

[0072] Moreover, while Figures 4 and 5 focus on hollowing out the decoded picture buffer, it is clear that the decoder 22 in these figures uses the just decoded picture content to fill the decoded picture buffer. For example, in the case of Figure 4, the filling of the decoded picture buffer 48 can be performed picture by picture in the case of sub-picture processing, i.e. in both cases of sub-picture processing and picture by picture processing. While Figure 5 focuses on hollowing out the DBP per sub-picture, the filling is performed per sub-picture as soon as a particular sub-picture is decoded and the sub-picture is inserted into the corresponding sub-picture DPB. Of course, there can be an alternative approach in which the filling of the decoded picture buffer 48 is performed picture by picture, in which case the decoded sub-pictures 24 of the picture 18 currently being decoded would be inserted into their corresponding sub-picture DPB at the same time at the end of the decoding of the current picture 18. For the per-picture DPB hollowing mode, the decoder of FIG. 5 may operate as one of the corresponding modes of FIG. 4 (ie, as indicated by the dashed lines in FIG. 4).

[0073] Furthermore, some brief remarks are necessary regarding the above embodiment related to the processing of the decoded picture buffer. In particular, it has been mentioned so far that the sub-pictures 24 of the pictures 18 of the video 16 can be coded like sub-videos in a certain way, in which the sub-pictures can depend on other sub-pictures of other pictures as long as they belong to the same sub-video, while being independent of any other sub-pictures, i.e. other sub-pictures of the same picture 18 or sub-pictures of other pictures 18 belonging to different sub-videos. Such sub-pictures belonging to the same sub-video may change the sub-picture position in the picture 18 of the video 16 in time. Figure 6 illustrates this case. To deal with such a situation, various possibilities exist. In Figure 6, two pictures 18 of the video 16 are exemplarily illustrated. In one picture 18, there are sub-pictures 24 with sub-picture IDs 2, 7, 3, 4, 5 and 6 that belong to the sub-video. It can be seen that in the next picture, sub-pictures 24 of the Sub Video with sub-picture IDs 2, 7, 4, and 5 are still present, but the sub-picture positions are different: instead of sub-picture 24 with sub-picture IDs 3 and 6, two new sub-pictures, or more precisely, sub-pictures with new sub-picture IDs 9 and 12, are present in the following picture.

[0074] Now assume that sub-pictures 24 of the later picture 18b, belonging to sub-picture identifiers that also occur in the earlier picture 18a, are not coded with the Random Access Point scheme and furthermore refer to sub-pictures of the picture 18a, i.e. the correct sub-pictures 24 of the picture 18a will be present in the decoded picture buffer 48 of the decoder 28 when decoding the picture 18b, but measures must be taken to correctly perform the referencing, i.e. motion compensated, prediction for the decoding of the sub-pictures 24 of the picture 18b of these sub-picture identifiers.

[0075] One possibility to solve this problem could be for the compositor / synthesizer 14 to take part in such situations. It could use the technique presented in US Pat. No. 5,399,494, according to which a non-output picture 18c is inserted by the compositor / synthesizer 14 into the data stream 10, and the compositor / synthesizer 14 rearranges the picture content of such picture 18a by using a motion vector field that is constant piece by piece, i.e. constant within each sub-picture position, so that the sub-pictures 24 still required in the following picture 18b are placed in their new sub-picture positions that they assume in this following picture 18b. Furthermore, the compositor / synthesizer 14 ensures that for such picture 18b, instead of referring to picture 18a as a reference picture, it is redirected to assert picture 18c as a reference picture. Furthermore, motion vector prediction should be prohibited for a sub-picture 24 of a picture 18b belonging to a sub-video, the sub-picture of which exists in picture 18a, and which is redirected by picture 18c that is not to be output.

[0076] Another possibility would be that when sub-pictures change their sub-picture position, such as the sub-pictures of picture 18b with sub-picture IDs 2, 7, 4, 5 in Fig. 6, the decoder 22 rearranges the picture in the decoded picture buffer to generate new rearranged reference pictures. According to such a procedure, the reference pictures in the decoded picture buffer would be spatially split into their sub-pictures and the sub-pictures would be rescrambled. As described above with respect to Fig. 1, such a process could also include spatial resampling if a sub-picture 24 in picture 18b is indicated to belong to the same picture content, but with reduced spatial resolution compared to the sub-pictures of the previous picture, such as picture 18a. In this way, the transition from the sub-picture layout of one picture 18a to the sub-picture layout of the new picture 18b could be considered without inserting an output picture 18c.

[0077] And yet another alternative could be that the decoder 22 redirects the motion vectors used to code the sub-pictures of the picture 18b. The decoder 22 can generate a motion vector offset of the motion vector of the relocated sub-picture in the picture 18b in order to compensate for the change in position of the sub-picture with respect to the reference picture 18a. The motion vector offset is added to the coded motion vector of the sub-stream belonging to the sub-picture 24 of the picture 18b, or more precisely of the sub-stream in which the sub-picture is also present in the picture 18a, in the motion compensated prediction involved in the decoding of the sub-picture 24 of the picture 18b. Thereby, the correct predictor of the inter-predicted blocks of these sub-pictures of the picture 18a is derived, which predictor points to the correct position in the reference picture 18a, i.e. to a different sub-picture position, but to the correct position in the sub-pictures belonging to these sub-videos shown in both pictures 18a and 18b.

[0078] Sub-picture relocation in the decoded picture buffer with respect to reference pictures, and motion vector redirection that may be performed by decoder 22, may be triggered by observed changes in the association of sub-picture identifiers to individual sub-picture positions in pictures 18 of video 16, which associations accordingly describe the sub-picture relocation in data stream 10 of reference pictures and their sub-pictures. Alternatively, these processes may be triggered by explicit data stream instructions in data stream 10 written into the data stream by combiner / synthesizer 14, such as in a PPS.

[0079] The embodiment described next with respect to FIG. 7 is intended to describe a video codec environment that mitigates the application described above with respect to FIG. 1 in terms of picture order count (POC) processing. Thus, FIG. 7 describes a possible functionality of a decoder for processing a data stream having video encoded therein, in which a picture 18 is subdivided into sub-pictures 24, and in which the sub-pictures 24 are encoded separately from each other as part of a sub-video 28, as described above with respect to FIG. 5, etc. The functionality described with respect to FIG. 7 may be a specific mode of the decoder 22, triggered by a sub-picture indication, such as the indication 44 described with respect to FIG. 2. In particular, FIG. 7 allows different POC values ​​to be transmitted in the data stream 10 for the sub-pictures 24 of a picture 18, such that different POC values ​​may occur even within one picture 18. If the functionality of FIG. 7 represents only one mode of the decoder 22, another mode may be that only one POC value is allowed for one picture 18.

[0080] When the data stream 10 is synthesized based on the reservoir of sub-streams 32 as described with respect to FIG. 1, for example, the picture order count (POC) expansion in the display time order 20 of the sub-pictures 24 in the corresponding sub-video 28 may differ. For example, the difference may be due to different frame rates between the sub-videos 24. The video 16 will naturally have a minimum frame rate of the sub-videos 28 that contribute to the video 16. Therefore, the corresponding pictures of the high frame rate sub-video 28 will have to be skipped by the synthesizer / integrator 14 when synthesizing the data stream 10. It should be noted that when encoding such a high frame rate sub-video 28 into the corresponding sub-stream 32, the encoding process will naturally not need such pictures, because they should remain reference pictures for other P or B pictures and contribute to the synthesis of the video 16 together with the sub-pictures 24 of the low frame rate sub-video 28. Additionally or alternatively, such differences in the temporal evolution of the picture order count values ​​of the sub-pictures 24 in the various sub-videos 28 and corresponding sub-streams 32, respectively, may be due to different GOP structures underlying these sub-streams 32. For example, different GOP lengths, i.e., different numbers of sub-pictures 24 per GOP, may be applied to different sub-streams 32. Naturally, the generation / encoding process resulting in the sub-streams 32 must be performed in such a way that, at least as far as those sub-pictures 24 of the corresponding sub-videos 28 that contribute to the composition of the picture 18 of the video 16 are concerned, their order of occurrence within the respective sub-videos 28 and sub-streams 32 is consistent both in the display time order 20 and in the encoding / decoding order 58.

[0081] Thus, FIG. 7 shows one or more NAL units 34 having sub-pictures 24 of a sub-video 28 encoded therein, which have been selected by the compositor / synthesizer 14 to be used to join a corresponding access unit 46 of the data stream 10 by stitching them together, and which are given a POC value 90. The POC value orders the sub-pictures 24 in the respective sub-video 28, for example in a GOP (group of pictures) based display time order 20. That is, the GOP value 90 in the NAL unit 34 may start the order anew every time a new GOP is started. Advantageously, the concept of FIG. 7 allows the compositor / synthesizer 14 to simply carry over the POC value 90, without modification, when stitching together the NAL units 34 of the contributing sub-streams 32 to result in a corresponding NAL unit 34′ in the access unit 46 of the data stream 10. 7 illustrates by hatching the sub-pictures 24 of the various sub-videos 28 and their corresponding NAL units 34 in their corresponding sub-streams 32, which correspond to a common time point in the encoding / decoding order 58 and to a common encoding / decoding rank, so as to together form a corresponding access unit 46 in the stream 10. To take into account the differences in POC values ​​90 between the NAL units 34' corresponding to different sub-pictures of the currently decoded picture 18 or the currently decoded access unit 46, the decoder 22 operates as follows: In particular, the decoder 22 decodes from the data stream the POC values ​​90 contained in the NAL units 34' for each sub-picture 24 of the currently decoded picture 18. Furthermore, the decoder 22 manages, for each sub-picture 24 of the currently decoded picture 18, or more precisely, for each sub-video 28 from which the picture 24 in the currently decoded picture 18 originates, one or more inter-sub-picture POC compensation parameters 92 which parameterize a POC correction function 94, which, when applied to the POC value 90 of the corresponding sub-video 28, results in a final POC value 96 that is the same for each picture 18 of the video 16.According to an example, the POC values ​​90 carried by the NAL units 34' in one access unit 46 of the data stream may be different due to different GOP lengths used for encoding the sub-videos to which the sub-pictures 24 belong and are encoded in these NAL units 34'. In that case, as long as the contributing sub-videos 28 and sub-streams 32 are not changed during synthesis / integration, the mutual offsets between the POC values ​​90 in one access unit 46 tend to be constant across successive access units 46 and data stream 10, as long as none of the GOPs of the contributing sub-streams 32 start a new GOP. The inter-sub-picture POC compensation parameters 92 may include, for example, a POC compensation offset added by a function 94 to the POC value 90 of the corresponding sub-video of the current sub-picture 24 of the sub-video 28 corresponding to the most recent start of a GOP, i.e. to account for differences between the various sub-streams 32 of the IRAP sub-pictures 24 in that sub-video 28. Whenever a new sub-video 28 starts contributing to a picture 18 of the video 16, or whenever one of the active sub-videos 28 starts a new GOP by having a sub-picture whose information 40 signals a RAP, the decoder 22 updates, calculates, or resets the POC compensation offset of that sub-video or sub-stream 32 based on the comparison, i.e., by calculating the difference, respectively. Any final POC value 96 of any sub-video 28 or sub-stream 32 is then maintained within the current video 16, i.e., within the same access unit 46, which does not have any random access point encoding therein. Due to frame rate differences between the sub-streams 32, the inter-sub-picture POC compensation parameters 92 may also include a factor by which the POC value 90 is scaled using a function 94 to generate the final POC value 96. Due to such frame rate differences, the mutual differences between POC values ​​90 within an access unit 46 of data stream 10 tend to correlate with the mutual differences of POC values ​​90 after the POC values ​​have been offset compensated using the aforementioned POC compensation offsets, and the mutual differences scale in time according to the POC step size differences.This scaled difference is compensated for by inversely scaling these POC values. The decoder 22 may derive such scaling factors from the data stream 10 as part of parameters 92, which may have been written into the data stream by the combiner / synthesizer 14 based on an estimate of the underlying frame rates of the sub-video 28 and the corresponding sub-stream 32, respectively.

[0082] Thus, in brief summary, the decoder 22 may manage POC compensation parameters 92 to parameterize a function 94. The function 94 so parameterized, when applied to the POC values ​​90 corresponding to a particular sub-picture 24 of the current picture 18, results in a final POC value 96. The final POC values ​​96 thus obtained are equal to each other for each picture 18 of the video 16. The offset between the parameters 92 may be updated by the decoder 22 in the random access point sub-picture 24 in the manner described above, provided that a non-RAP sub-picture 24 is present in that picture 18 or access unit 46, respectively. If all sub-pictures 24 of the current picture 18 are random access point coded, the decoder 22 may set the offset to any default value equal to the sub-picture 24. Additionally or alternatively, the RAP sub-picture may itself be set to the same default POC value, such as by appropriately parameterizing the encoder 30. This value may be zero. The decoder would not have to do anything under this encoder constraint. However, encoder 30 would be free to set the POC of RAP sub-pictures 24 of one picture 18 to different values, and in such a case the decoder may be configured to set the POC values ​​of these RAP sub-pictures to a default POC value, such as zero, or to a value equal to the POC value 90 signaled for the first encountered RAP sub-picture. The scaling factor between parameters 92 may be determined by decoder 22 each time a sub-video 28 makes a new contribution to the composition of video 16.

[0083] As further outlined below, the decoder 22 may need to take care to ensure that the motion vector predictor scaling performed by the decoder 22 to preserve the sub-pictures 24 from the corresponding NAL units 34' is performed correctly. The motion predictor scaling is performed when a motion vector predictor associated with a reference picture A having a POC distance a to the picture containing the block for which the respective motion vector predictor was used is scaled to reference some required POC distance b for the currently predicted block of the current picture. Since the scaling should be performed with respect to the POC distance associated with the POC definition of the respective sub-stream 32, this scaling may still need to be performed by the decoder 22 based on the POC values ​​90 of the individual sub-streams. For example, according to HEVC motion vector predictor scaling, different motion vector scaling may result if the frame rate scale factor between parameters 92 was not a power of two.

[0084] As already mentioned above, the above embodiments can be combined with each other. This is not only true when each of the embodiments described with respect to Figures 3, 4, 5 and 7 is combined with the embodiment of Figure 2, but also true for any combination of two, three or all of the embodiments described with respect to Figures 3, 4, 5 and 7. Each of these concepts eases the use of the resulting video codec concept in performing the task of video synthesis / merger outlined with respect to Figure 1. Naturally, the example outlined in Figure 1 should not be treated as limiting the example of video encoding codecs shown in the above embodiments.

[0085] Before starting the description of further embodiments of the present application, which relate to further aspects of the present application, but on a slightly different topic, namely gradual decoding refresh, the embodiment of FIG. 7 is described again, or in other words, drawn attention to. In particular, as already outlined above, the embodiment of FIG. 7 relates to POCdelta derivation. As explained for the sub-pictures 24, each sub-picture can be, for example, a tile, or a set of tiles (that fit into the joint bitstream 10) can have unaligned random access points or different reference or GOP structures. As outlined above, such use cases can arise, for example, when initially separately coded bitstreams 32 are merged together into a single data stream 10.

[0086] In such a case, the POC (Picture Order Count) indicated in the sub-bitstream 32 of each sub-picture 24 may be different, despite relating to one common time point or contributing to one common access unit 46 in the aggregated data stream 10. If this is the case, this leads to the burden that the POC values ​​signaled in the bitstream 32, i.e. in the NAL units 34, must be rewritten by the synthesizer / aggregator in order to be able to aggregate in the resulting composite data stream to produce one POC value for one access unit. The embodiment of Figure 7 alleviates this burden and allows for the aggregation of such sub-picture sub-streams 32 without adjusting the POC values ​​coded in the NAL units such as the headers, i.e. tile group headers, of each sub-picture 24.

[0087] The POC is used to derive the boundaries of the access units. That is, typically, the POC is used to associate a group of slices or a group of tiles or a NAL unit with an access unit 46. This may be an alternative behavior when the decoder 22 of FIG. 7 is combined with the embodiment of FIG. 2. That is, the concept described in FIG. 7 will be applied by the decoder 22 when the indication 44 indicates sub-picture granularity, whereas, when the indication 44 indicates picture granularity, the decoder 22 will assume that each access unit 46 has only a single POC value that is equal within that access unit 46. In the case of picture granularity, the decoder 22 will recognize the start of the next access unit 46 in response to a change in the POC value from one NAL unit to another. In the case of the sub-picture granularity functionality described in FIG. 7, the identification of the access unit interface separating consecutive access units in the data stream 10 may be performed by the decoder 22, for example, based on an access unit delimiter inserted in the data stream 10 by the combiner / synthesizer 14.

[0088] Among the various processes where the POC value is used in the decoding process is picture output, i.e. pictures are output in ascending or POC order for picture reference, i.e. to determine whether short-term or long-term reference pictures are used for reference.

[0089] The embodiment described with respect to FIG. 7 allows different signal values ​​of POC in the NAL units 34′ of each sub-picture 24 in the same access unit 46, while still allowing the correct derivation of the POC value 96 that is ultimately used in the above operations of picture output and picture referencing when multiple sub-pictures 24 are decoded from the same data stream 10. Typically, a random access point results in a reset of the POC value, which allows the decoder to start the decoding process at the beginning of the bitstream. When the decoder finds an IDR, it typically assigns a POC value of zero for that picture and derives the POC values ​​of subsequent pictures based on that. The next access unit or the NAL units within that access unit contain information such as (tile group) header bits that signal the POC of the access unit to which they belong. Typically, the POC is signaled in the LSB bits in the slice / tile group header, and the MSB bits are derived by the decoder. This process, if applied to the scenario outlined in FIG. 7, may lead to erroneous derivation of access unit boundaries, reference pictures, and output order when sub-pictures of one access unit contain different POCLSB values ​​or different POC LSB lengths within one access unit.

[0090] The embodiment described with respect to FIG. 7 used a per-subpicture deltaPOC calculation that tracks the difference between different subpicture bitstreams 32. For example, if the decoding process of the decoder 22 starts with a regular full-picture IRAP, all NAL units 34' of the access unit 46 carry the same POC value (NALPOC LSB). At this point the POC MSB is set to 0. Now, during the course of the CVS, a subpicture RAP may occur and may carry a different NAL POC LSB, e.g. a zero value, while other NALPOC LSBs in that access unit remain unchanged (not equal to 0). Whenever a subpicture RAP is recognized as described above or via up-to-date signaling, such as by NAL unit type or SEI message, a deltaPOC is derived for this subpicture as the difference between the IRAP NALU POC LSB and the other NALU POC LSBs in that access unit. A full-picture POC 96 is derived for all NALUs as the basis for the above functions, such as picture output, picture reference, etc. When dealing with sub-picture IRAP NALUs, the full picture POC 96 incorporates a non-zero delta POC, while for the remaining non-sub-picture IRAP NALUs, incorporating the respective zero delta POCs in the calculation of the full picture POC 96 results in the same full picture POC value 96 for all NALUs of one access unit 46. Figure 8 illustrates POC compensation according to Figure 7 using an example, where four consecutive pictures 18a-18d of a video 16 are shown, each sub-divided into three sub-pictures 24, with the first picture 18a being an IRAP picture, i.e. all its sub-pictures 24 being random access points, and the second picture 18b and the fourth picture 18d having all their sub-pictures 24 non-RAP coded. The third picture 18c has one sub-picture 24, i.e. the top picture 24 of Figure 8, coded in a random access point manner, while the other sub-pictures 24 are not coded in such a manner.As can be seen, Figure 8 shows a simple implementation of a function 94, i.e., summation with a constant representing the POC compensation parameter, i.e., the second addend of the summation equation, represented within each subpicture 24. The first addend represents the signaled POC value, i.e., the input to the parameterizable function, and the resulting sum corresponds to a final POC value 96. As shown in Figure 8, the decoder obtains the POC compensation offset by subtracting 98 the transmitted POC value of the RAP subpicture 24 from the final POC value for one of the non-RAP subpictures 24 of picture 18c.

[0091] The introduction of the described full-picture POC scheme may be gated via an SPS / PPS level flag. Alternatively, instead of an implicit derivation of a deltaPOC value, i.e., a POC compensation offset, one parameter set in data stream 10 may indicate a deltaPOC to be subtracted / added to each sub-picture POC value 90 by a function 94, with the resulting full-picture POC 96 being aligned for each sub-picture 24 of the currently decoded picture 18 or current access unit 46 in data stream 10. If a sub-picture IRAPNALU is within an access unit 46, additional constraints may be necessary to be able to derive a full-picture POC.

[0092] For example, the requirements for data stream 10, i.e., bitstream conformance requirements, may be: at least one sub-picture non-IRAP NALU is present in access unit 46, i.e., at least one sub-picture 24 of each picture 18 is coded in a non-RAP manner, which allows the derivation of the current full-picture POC 96, and according to which deltaPOC, i.e., POC compensation offsets, of all sub-picture IRAP NALUs should be derived. In other words, a new NAL unit type, a sub-RAP that does not allow a different NAL unit type, is signaled using a flag. In such a case, such signaling can be used only if one of the NAL units does not indicate that the NAL unit is a sub-RAP.

[0093] Another solution could be as follows: Once all sub-pictures in one access unit 46 are recognized to be sub-picture IRAPs, POCMSB is reset and deltaPOC for each sub-picture 24 is calculated such that all sub-pictures have the same full-picture POC. For example, the first sub-picture deltaPOC in this case is set to 0 and all other sub-picture deltaPOC are set accordingly resulting in full-picture POC 96 for the first sub-picture IRAPNALU in the access unit.

[0094] Also as explained above, the substreams 32 involved in the merging of the data stream 10 may be of different POC step sizes. This may occur, for example, in a conferencing scenario. One stream may be encoded at 30 FPS and the other substream 32 at 60 FPS. In such a case, as mentioned above, one option is to merge at the lowest common frame rate, i.e. 30 FPS. For this purpose, the topmost temporal layer of the 60 FPS substream 32 may be dropped to reduce the frame rate to 30 FPS. However, the POC step sizes between temporally adjacent pictures in the two streams will be different, since a 60 FPS encoder will have to use a larger POC step size to accommodate more frames compared to a 30 FPS encoder. In the above scenario where multiple streams are merged into one common bitstream, care must be taken to ensure a correct POC derivation. If the POCs of the subpicture bitstreams have different step sizes, the embodiment of FIG. 7 is capable of forcing a scaling of the POC values ​​90 of the individual subpictures 24 in the encoded video sequence 10. For example, the integrator / compositor 14 will be informed of, or will derive itself, the differences in POC step sizes from the input sub-picture bitstreams 32 and will write the respective POC scaling factors into the parameter set for all sub-picture bitstreams that contribute to the composition of the data stream 10 in order to align all full-picture POCs.

[0095] Further note: As mentioned above, the scaled full picture POC is used for reference picture derivation and output order, while the motion vector scaling should still be performed according to the difference in the unscaled POC, so that the exact motion vector scaling as expected by the original encoder is performed.

[0096] At the decoder side, instead of tracking all POC values ​​90 of each sub-picture 24, i.e., unscaled POC, the decoder can derive the difference in POC values ​​used at the encoder side to perform motion vector candidate scaling from the POC values ​​scaled using the respective POC scaling coefficients when deriving the POC difference for motion vector scaling.

[0097] Further, although the POC offset values ​​between parameters 92 have been described as being automatically determined by the decoder, alternatively or additionally, the POC offset values ​​may be inserted into the data stream 10 by the synthesizer / integrator 14, as described above with respect to the POC scaling factors.

[0098] With reference to Fig. 9, the present application proceeds with a description of an example of a video codec that allows gradual decoding refresh. Fig. 9 illustrates the concept underlying this embodiment.

[0099] In particular, FIG. 9 shows consecutive pictures 100 of a video 102, i.e., four consecutive pictures ordered according to their encoding / decoding order 58. FIG. 9 shows an encoder that encodes the video 102 into a data stream 106, and a decoder 108 that decodes or reconstructs the video 102 from the data stream 106. The concept is as follows: To enable gradual decoding refresh, the video decoder 108 logs, for each of the reference pictures of the video 102, a subdivision of the respective reference picture into a refresh picture region and a non-refresh picture region. In particular, a reference picture is a picture that precedes the currently decoded picture in the decoding order 58 and that can be used as a prediction reference for the currently decoded picture or any subsequent picture. The above-mentioned refresh picture region and non-refresh picture region will become clear from the following description. In particular, the decoder 108 decodes from the data stream 106 information regarding the refresh update region 110, indicated by cross-hatching in FIG. 9, for a particular picture 100a. In the example of FIG. 9, the refresh update region encompasses the left third of the picture 100a. The encoder, for example, encodes each picture 100a-100d into a respective access unit 112a-112d and signals within the access unit 112a for the picture 100a information 114 locating the refresh update region 110 within the picture 100a. The refresh update region indication 114 may initiate a gradual decoding refresh, whereby special measures are taken for the encoding and decoding of the picture 100a and several subsequent pictures until the gradual decoding refresh is completed on both sides of the encoder 104 and the decoder 108, as will be explained in more detail below.

[0100] In particular, the region 110 located by the information 114 represents a region that has been coded by the encoder 104 by intra coding only, i.e. coded in a random access point manner, and the decoder 108 therefore decodes the refresh update region 110 accordingly, i.e. using intra prediction. The coding of the other regions of the picture 100a, i.e. the remaining region 116, is not further restricted and can be coded by the encoder 104 using inter and / or intra prediction based on previous pictures and decoded by the decoder 108 accordingly.

[0101] For the next picture 100b in the encoding / decoding order 58, the gradual decoding refresh is not yet completed. The encoder 104 encodes this picture 100b into a corresponding access unit 112b, in which it again signals information 114 relating to the refresh update region 110 in the picture 100b. In the example of FIG. 9, the refresh update region of the picture 100b covers the central horizontal third of the picture 100b and is therefore spatially adjacent to the refresh update region 110 of the previous picture 100a. While the information 114 in the access unit 112a initiated the gradual decoding refresh, the information 114 relating to the picture 112b represents the continuation of this initiated gradual decoding refresh. The region 110 of the picture 100b is again encoded by the encoder 104 by intra prediction only and decoded accordingly by the decoder 108. However, as part of the initiated gradual decoding refresh, the decoder 108 and the encoder 104 perform the following tasks: First, the encoder 104 and the decoder 108 perform the aforementioned logging of refresh picture regions of the reference pictures. Since the gradual decoding refresh has been started in the picture 100a and the picture 100a forms a reference picture for the following picture 100b, the refresh update region 110 of the picture 100a represents the refresh picture region of the picture 100a, while the other region 116 is a non-refresh picture region. The refresh picture region is indicated in Fig. 9 by being surrounded by a solid line 118. Secondly, the decoder 108 and the encoder 104 determine within the currently coded / decoded picture, i.e. within the current description picture 100b, a first region 120 to be coded and decoded separately from the refresh update region 110 of the same picture 100b and independently from the non-refresh picture region of the reference picture (here the picture 100a).Encoder 104 may code explicit signaling as to where region 120 is, or, more preferably, encoder 104 and decoder 108 may determine region 120 as a region of picture 100b that is juxtaposed relative to refresh picture region 118 of picture 100a, the immediately preceding picture in encoding / decoding order 58. To distinguish region 120 from refresh update region 110, refresh update region 110 is shown using cross-hatching in FIG.

[0102] The encoder and decoder take special measures to code and decode the region 120 into the corresponding access unit 112b. In particular, the encoder 104 does not restrict the signalable domain of the coding parameters, for example, on the basis of which the prediction of the region 120 is derived by the encoder and the decoder. In other words, the encoder 104 codes pictures into the corresponding access unit using coding parameters that determine the prediction of these pictures from previous pictures, such as motion compensated prediction, and using prediction residual information that describes the prediction residual to use when this prediction should be corrected. The coding parameters and the prediction residual information are coded into the access unit to describe the corresponding picture. However, within the refresh update region 110, the encoder 104 exclusively uses intra prediction, i.e., prediction from any other picture is not used. For region 120, the encoder 104 uses inter prediction and does not, for example, restrict the signalable domain of coding parameters, but compared to other regions such as region 116, a different method of deriving a prediction for region 120 based on the signaled coding parameters is used by the encoder 104 and the decoder 108. In particular, the position of the respective reference picture, here the refresh picture region 110 of picture 100a, is taken into account. For example, the encoder 104 is free to code, for region 120, a motion vector into the data stream 106 that may result in a dependency of region 120 on region 116 if the prediction derivation is not changed. That is, a motion vector may be coded that actually points to a portion of reference picture 100a that extends into region 116. Similarly, the encoder 104 may use motion vector prediction to encode the motion vectors of the region 120, but in interpreting / constructing the corresponding motion vector prediction list of motion vector predictor candidates formed by the motion vectors used to encode the picture 100a, the encoder 104 exclusively uses the motion vectors used for blocks in the refresh picture region 110 for exclusive inclusion in these motion vector predictor candidate lists.The possibility that the encoder 104 may use a signalable state that is redirected to a prediction derivation result for the region 120 that could have actually been signaled by other states of the coding parameters of the region 120 offers the encoder 104 the possibility of increasing compression efficiency, since for example a coding parameter state that is later redirected by the encoder 104 and the decoder 108 to another state and that leads to a prediction that depends only on the updated picture region 110 may include a state that can be coded using fewer bits compared to a coding parameter state that directly describes this prediction without redirection by the encoder 104 and the decoder 108. That is, the encoder 104 and the decoder 108 may, for example, clip the motion vectors that were actually signaled in the data stream 106 for the region 120 so that they do not extend beyond the boundaries of the refresh picture region 110 of the reference picture 100a. Alternatively or additionally, the encoder and decoder 108 may use padding for portions of the reference picture 100a pointed to by the motion vector signaled in the data stream 106 for the region 120 that exceed the boundaries of the refresh picture region 110. Further alternatively or additionally, the construction of the motion vector candidate list performed by the encoder 104 and the decoder 108 may be performed in such a way as to exclude motion vectors from inclusion in such list that are outside the refresh picture region 110 in the reference picture 100a. The encoder 104 may signal the selected motion vector predictor, for example by signaling it in the data stream 106, to the decoder 108 and index such motion vector candidate list that is interpreted / constructed accordingly. The remaining area of ​​picture 100b, i.e., in the case of FIG. 9, the right third of picture 100b, represents a region 116 separate from the first region 120 and the refresh update region 110 of picture 100b, which is encoded by encoder 104, including unrestricted inter prediction with respect to the refresh picture region 110 of picture 100a.That is, region 116 of picture 100b is an unrestricted inter prediction region.

[0103] The subsequently coded / decoded picture 100c, in the example of Fig. 9, represents the completion of the gradual decoding refresh started with the picture 100a. For this picture 100c, the encoder 104 encodes in the corresponding access unit 112c information 114 indicating the position of a refresh update region 110 of the picture 100c, which in the case of Fig. 9 covers the right third of the picture 100c. That is to say, Fig. 9 shows that the refresh update regions 110 of the successive pictures 100a-100c starting from the picture 100a representing the start of the gradual decoding refresh represent mutually separated portions of the picture region of the pictures of the video 102. This picture 110 is also intra coded by the encoder 104 and decoded by the decoder 108 accordingly.

[0104] For picture 100c, picture 100b also represents a reference picture. Its refresh picture area 118 is determined by the encoder and decoder as the union of the first area 120 and the refresh update area 110 of picture 100b. This also means that during gradual decoding refresh, the refresh picture area 118 grows continuously. In this case too, the encoder 104 and the decoder 108 determine the first area 120 of picture 100c to be, for example, an area that spatially overlaps with the refresh picture area 110 of the previous picture, i.e. picture 100b. This area 120 is coded / decoded as described above for the area 120 of picture 100b, with the difference that this area 120 must have two reference picture candidates, namely picture 100a with refresh picture area 118 and picture 100b with refresh picture area 118. Although inter prediction is used for region 120, i.e., it is an allowed coding mode in addition to the intra prediction modes, the coding parameters for region 120 of picture 100c are redirected to a state such that the resulting prediction does not result in coding / decoding dependency of region 120 from unrefreshed regions 116 of pictures 100a and 100b.

[0105] For pictures 100c onwards, the gradual decoding refresh initiated by the encoder 104 at picture 100a is completed and the refresh picture region 118 covers the complete pictures of the video 102 from picture 100c onwards. Intermittently or periodically, the encoder 104 may initiate another gradual decoding refresh, and the refresh picture region will collapse to correspond to the refresh update region 110 of the initial starting picture of that subsequent gradual decoding refresh.

[0106] The gradual decoding refresh as described with respect to FIG. 9 may be described as sub-picture intra refresh padding. It may be implemented using tiles or tile groups or without tiles. In the embodiment of FIG. 9, some parts of the picture, i.e. areas 110, which may be called sub-pictures, are refreshed by applying intra coding, while the following pictures are inter coded and inter decoded in a constrained manner. According to the example of FIG. 9, the picture is divided into columns, here exemplarily divided into N=3 columns. However, it should be made clear that the number of columns N can be chosen differently and even different forms of growth of the refresh picture area 118 between such gradual decoding refreshes, i.e. by blocks instead of by columns of pictures. As described with respect to FIG. 9, in the first sub-picture RAP, i.e. the picture of access unit 0, picture 100a in the case of FIG. 9, the first column is coded only in intra blocks, taking it as colldx=0.

[0107] In the next access unit AU=1, i.e. the second picture 100b in the encoding / decoding order 58 of FIG. 9, the second column, i.e. coldx=1, is coded using only intra blocks, and the first column is coded using inter-coded blocks that can only refer to samples decoded in column coldx=0 of the previous picture. However, instead of constraining the motion vectors that the encoder can code in the data stream, the embodiment of FIG. 9 is also free to code motion vectors that actually result in dependencies on other regions when column coldx=0. In particular, according to the embodiment of FIG. 9, an intra-refresh region, i.e. the first column in this example, is shown, and padding such as orthogonal extrapolation of the outermost pixel lines is used for the reference blocks of the inter-coded blocks, which are blocks beyond column coldx=0 of the picture of access unit 1, if the motion vector indicates so. Alternatively or additionally, motion vector clipping based on the region indication can be applied to ensure that the motion vectors of the refreshed regions do not depend on the non-refreshed regions. As pictures are successively decoded, the refreshed region 118 increases, thus indicating the refreshed region 110 for each picture.

[0108] It should be noted that an alternative to Fig. 9 would be for the encoder 104 to signal the successive positions of the update refresh regions 110 of successive pictures 100a-100c, i.e. the positions of the sequence of pictures forming a gradual decoding refresh, by indicating the pattern of these regions 110. This pattern would account for the refreshing. Regions per sequence could also be indicated. Thus, this kind of information could be signaled once for the picture 100a, i.e. the start picture of the GDR sequence 100a-100c, thereby signaling the GDR start and also signaling the regions 110 for the pictures 100a-c. The former indication could be something that could be indicated in the PPS. Alternatively, a GDR start indication may be signaled with respect to picture 100a, such as within its PPS, while the pattern of positions of regions 110 in a GDR picture sequence starting from such a GDR start picture may be something that is signaled at a higher level, such as within an SPS.

[0109] Further remarks regarding the above embodiments are given below: The above embodiments present advantageous video codec concepts. Many of them allow advantageous video synthesis as outlined above. However, it should be noted that an encoder may form any of the above inventive data streams 10 directly, i.e. without synthesis of pre-encoded sub-streams. In such a task, the encoder may take advantage of advantageous properties of the data stream 10 present in the above description, such as the possibility to select the RAP time instant and the reference picture buffer description per sub-picture instead of the whole picture.

[0110] Thus, the above embodiments also relate to a video decoder (and corresponding video encoder and data stream) for decoding a video 16 of a picture 18 from a data stream 10 having video encoded therein in such a way that sub-pictures 24 into which the picture is spatially subdivided are encoded independently of each other. The video decoder is configured to decode, from the data stream, for each sub-picture of each picture of a set 42 of one or more pictures of the video, sub-picture related information 40, such as a syntax element nal_unit_type, which identifies the respective sub-picture as being coded in a random access point manner when in a first state, such as state IDR_N_LP, and identifies the respective sub-picture as not being coded in a random access point manner when in a second state, such as state TRAIL_NUT. The decoder decodes from the data stream, for a set of one or more pictures 42 whose PPS includes the syntax element mixed_nalu_types_in_pic_flag, a random access point signaling granularity indication 44, which may also be referred to as mixed_nalu_types_in_pic_flag, which when it is of a first granularity or a first state, such as 0, indicates that for each picture of the set of one or more pictures, the sub-picture related information is in one common state, such as IDR_N_LP for all sub-pictures of the respective picture, and when it is of a second granularity or a second state, such as 1, indicates that for each picture of the set of one or more pictures, the sub-picture related information may be in different states for the sub-pictures of the respective picture, e.g. IDR_N_LP for at least one sub-picture and TRAIL_NUT for at least one other sub-picture within the same picture.In other words, when mixed_nalu_types_in_pic_flag is in a first state, such as equal to 0, it may indicate that the value of nal_unit_type is the same for all coded slice NAL units of one picture, i.e., for all subpictures in one picture. But otherwise, when mixed_nalu_types_in_pic_flag is in another state, such as equal to 1, this indicates that the VCL NAL units of one or more subpictures of that picture all have a distinct value of nal_unit_type, such as one from a NAL unit set consisting of one or more of STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, and CRA_NUT, while all other VCL NAL units of that picture have a different distinct value of nal_unit_type, i.e., one from another set consisting of one or more of TRAIL_NUT, RADL_NUT, and RASL_NUT. In further other words, when mixed_nalu_types_in_pic_flag assumes a second state, such as 1, it may specify that each picture that references a PPS with mixed_nalu_types_in_pic_flag has multiple VCL NAL units that do not have the same value of nal_unit_type and that the picture is not an IRAP picture, while when mixed_nalu_types_in_pic_flag assumes another state, such as 0, it may indicate that each picture that references the PPS has one or more VCL NAL units and that the VCL NAL units of each picture that references the PPS have the same value of nal_unit_type. By this means, mixed_nalu_types_in_pic_flag may be set to a first state, such as 1, to indicate to pictures that reference a PPS, such as those resulting from a subpicture bitstream merging operation, that they contain slices of different NAL unit types.

[0111] Although some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item, or a feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0112] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-Ray™, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that the respective method is performed. Thus, the digital storage medium may be computer readable.

[0113] Some embodiments according to the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0114] The inventive encoded video signal or data stream, respectively, can be stored on a digital storage medium or transmitted over a transmission medium, such as a wireless transmission medium or a wired transmission medium, such as the Internet.

[0115] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code operable to perform one of the methods when the computer program product is run on a computer. The program code can for example be stored on a machine readable carrier.

[0116] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0117] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0118] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0119] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may be adapted to be transferred through a data communication connection, for example via the Internet.

[0120] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0121] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0122] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0123] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0124] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0125] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or software.

[0126] The methods described herein can be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0127] The methods described herein, or any components of the apparatus described herein, may be performed at least in part by hardware and / or software.

[0128] The above-described embodiments merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims, and not by the specific details presented as descriptions and illustrations of the embodiments herein.

Claims

1. 1. A method for decoding a picture from a data stream, comprising: decoding, for a picture, a flag from the data stream indicating whether a NAL unit type differs between at least two sub-pictures of the picture; determining that the NAL unit type is the same for all the sub-pictures of the picture if the flag is a first value; if the flag is a second value different from the first value, decoding a first NAL unit type of a first sub-picture of the picture and a second NAL unit type of a second sub-picture of the picture, the second NAL unit type being different from the first NAL unit type; A method for providing the above.

2. the first NAL unit type is random access; the second NAL unit type is not random access; The method of claim 1.

3. the flag is included in a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS) of the data stream; Each NAL unit type is associated with a slice of a subpicture. The method of claim 1.

4. if the flag is the second value, storing at least one of the sub-pictures in a buffer while decoding the sub-picture; flushing the buffer after all the sub-pictures of the picture have been parsed or decoded. The method of claim 1 further comprising:

5. and distinguishing between two subsets of NAL unit types based on an integer-valued syntax element corresponding to the NAL unit type. a first subset of the two NAL unit type subsets corresponding to random access; a second subset of the two NAL unit types corresponding to non-random access; The method of claim 1.

6. Decoding, for a picture, a flag from the data stream indicating whether a NAL unit type differs between at least two sub-pictures of the picture; if the flag is a first value, determining that the NAL unit type is the same for all the sub-pictures of the picture; if the flag is a second value different from the first value, decoding a first NAL unit type of a first sub-picture of the picture and a second NAL unit type of a second sub-picture of the picture, the second NAL unit type being different from the first NAL unit type; A decoder configured to:

7. the first NAL unit type is random access; the second NAL unit type is not random access; A decoder according to claim 6.

8. the flag is included in a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS) of the data stream; Each NAL unit type is associated with a slice of a subpicture. A decoder according to claim 6.

9. if the flag is the second value, storing at least one of the sub-pictures in a buffer while decoding the sub-picture; flushing the buffer after all the sub-pictures of the picture have been parsed or decoded.

7. The decoder of claim 6 further configured to:

10. further configured to distinguish between two subsets of NAL unit types based on an integer-valued syntax element corresponding to the NAL unit type; a first subset of the two NAL unit type subsets corresponding to random access; a second subset of the two NAL unit types corresponding to non-random access; A decoder according to claim 6.

11. When executed, the method includes the steps of: decoding, for a picture, a flag from the data stream indicating whether a NAL unit type differs between at least two sub-pictures of said picture; determining that the NAL unit type is the same for all the sub-pictures of the picture if the flag is a first value; if the flag is a second value different from the first value, decoding a first NAL unit type of a first sub-picture of the picture and a second NAL unit type of a second sub-picture of the picture, the second NAL unit type being different from the first NAL unit type; containing instructions to execute Non-transitory computer-readable medium.

12. the first NAL unit type is random access; the second NAL unit type is not random access; The non-transitory computer-readable medium of claim 11.

13. the flag is included in a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS) of the data stream; Each NAL unit type is associated with a slice of a subpicture. The non-transitory computer-readable medium of claim 11.

14. When executed, the at least one processor is caused to: storing at least one of the sub-pictures in a buffer while decoding the sub-picture if the flag is the second value; flushing the buffer after all the sub-pictures of the picture have been parsed or decoded; [0033] The non-transitory computer-readable medium of claim 11.

15. When executed, the at least one processor is caused to: and further comprising instructions for distinguishing between two subsets of NAL unit types based on an integer-valued syntax element corresponding to the NAL unit type. a first subset of the two NAL unit type subsets corresponding to random access; a second subset of the two NAL unit types corresponding to non-random access; The non-transitory computer-readable medium of claim 11.

16. 1. A method for encoding a picture into a data stream, comprising the steps of: determining whether the NAL unit type is the same for all sub-pictures of a picture; encoding a flag with a first value in response to determining that the NAL unit type is the same for all the sub-pictures of the picture; In response to determining that the NAL unit type is not the same for all of the sub-pictures of the picture, encoding the flag with a second value different from the first value; encoding a first NAL unit type of a first sub-picture of the picture and a second NAL unit type of a second sub-picture of the picture, the second NAL unit type being different from the first NAL unit type; A method comprising:

17. determining whether the NAL unit type is the same for all subpictures of a picture; In response to determining that the NAL unit type is the same for all the sub-pictures of the picture, encode a flag with a first value; In response to determining that the NAL unit type is not the same for all of the sub-pictures of the picture, encoding the flag with a second value different from the first value; encoding a first NAL unit type of a first sub-picture of the picture and a second NAL unit type of a second sub-picture of the picture, the second NAL unit type being different from the first NAL unit type; An encoder configured as follows.

18. When executed, the method includes the steps of: determining whether the NAL unit type is the same for all sub-pictures of a picture; encoding a flag with a first value in response to determining that the NAL unit type is the same for all the sub-pictures of the picture; In response to determining that the NAL unit type is not the same for all of the sub-pictures of the picture, encoding the flag with a second value different from the first value; encoding a first NAL unit type of a first sub-picture of the picture and a second NAL unit type of a second sub-picture of the picture, the second NAL unit type being different from the first NAL unit type; A non-transitory computer-readable medium comprising instructions for executing

Images