Video codec allowing sub-picture or region wise random access and concept for video composition using the same

Abstract

For example, in terms of the complexity of synthesis / merging operations, a video codec that enables more effective implementation of more effective video synthesis / merging and / or progressive decoding refreshing, and / or a concept that allows for more effective video synthesis / merging are described.

Description

Technology Field

[0001] This application relates to video coding and video synthesis. Background Technology

[0002] Various applications use subpictures, which subdivide a video picture into subpictures for separate encoding and decoding. Among these applications are 360° video and progressive decoding refresh.

[0003] In the case of 360° video, subpictures (tiles in the case of HEVC) are provided separately at different resolutions, as shown on the left side of Fig. 1. The user selects high-resolution tiles and low-resolution tiles depending on the viewing direction, as illustrated in the middle of Fig. 1, and they are merged into a common bitstream, as shown on the right side of Fig. 10. However, the user changes the viewing direction of the tile set when changing between high and low resolutions. However, not all high-resolution tiles are changed to low-resolution tiles, and vice versa. This means that a switching point is required only for a subset of tiles that change their resolution, which typically becomes a Random Access Point (RAP) for a specific subpicture bitstream.

[0004] Progressive Decoding Refresh (GDR) is a mechanism that enables the decoding process to start decoding a bitstream from a non-RAP as it started from a RAP as illustrated in Fig. 11, and to decode a given number of pictures to display a video of the same quality.

[0005] FIG. 11 illustrates, for example, four consecutive non-RAP pictures (consecutive in coding order 900) being equally subdivided into three tile groups, tile_group0, tile_group1, and tile_group2, i.e., regions of each picture. Pictures of video (902) are denoted by reference numerals 904a through 904d, and regions into which they are subdivided are denoted by reference numeral 906. Each region (906) is indicated as being coded in a self-contained manner, that is, in a manner forming a random access point depending on whether "IRAP" is written, i.e., coded in a manner where "inter" is written to refer to another picture. None of the four pictures (904a through 904d) illustrated in FIG. 11 have all regions (906) coded as random access points. Rather, in the example of FIG. 11, only one of the regions is a random access point, but in the example of FIG. 11, the random access point regions of three consecutive pictures, such as pictures (904a to 904c), spatially completely cover the picture regions of the pictures of the video (902), and thus after the triple of these pictures, the last picture (here, picture (904c)) is available in an artifact-free decoder if the encoder takes note that the non-RAP region of the picture (904c) is referenced as a RAP region or a non-RAP region that has a coding dependency only on the RAP region itself, directly or indirectly, for the purpose of deriving the prediction and / or entropy context, e.g., in terms of coding interdependency. That is, a clean decoded picture is not achieved immediately as when starting decoding with RAP, but after a while, the decoded picture is gradually refreshed to remove the artifact of missing reference. At this time, this is called a picture (904c). Generally, GDR can be achieved by dividing the content into regions (e.g., tiles) and encoding them into unaligned subpicture RAPs.

[0006] As a result of one of the two scenarios mentioned, references to different subbitstreams differ. Additionally, RAP marking for different subpictures is not explicitly performed because these subpictures are signaled only as type I_SLICES (predictions for previously decoded pictures are not applied), and breaking prediction for consecutive subpictures of the same subpicture bitstream is performed by not including subpictures preceding the picture with type I_SLICE in the reference picture list.

[0007] Despite the existing video coding concepts mentioned above, a video coding environment is still needed that allows for, for example, more effective implementation of video composition and / or progressive decoding refresh. The problem to be solved

[0008] Accordingly, the object of the present invention is to provide a video codec that enables more effective implementation of video synthesis / merging and / or progressive decoding refresh, for example in terms of the complexity of synthesis / merging operations, and / or a concept that allows for more effective video synthesis / merging. means of solving the problem

[0009] The above objective is achieved by the subject of the independent claim of the present application.

[0010] A first aspect of the present application aims to provide a more effective implementation of video synthesis / merging and / or progressive decoding refresh in a video codec by introducing a random access point signaling granularity indication that indicates to a video decoder at what granularity subpicture-related information may vary spatially within a picture of the video. The subpicture-related information identifies each subpicture as being coded in a random access point manner or not coded in a random access point manner. That is, where a random access point signaling granularity indication signaled in a data stream separately from or additionally to the subpicture-related information assumes a first granularity, for each picture of one or more sets of pictures forming the range of the random access point signaling granularity indication, the subpicture-related information assumes a common state for all subpictures of each picture. That is, all subpictures assume a first state signaling the random access point coding of the associated subpicture or a second state signaling the non-random access point coding of the associated subpicture. However, if the random access point signaling granularity indication assumes a second granularity, this allows the subpicture-related information to assume a different state for each picture of one or more picture sets for each picture of each picture. As a result of the introduction of this random access point signaling granularity indication, the task of constructing such a data stream having video encoded in such a way that the picture is subdivided into subpictures is relaxed in terms of the requirements raised in storage of one or more input video streams (sometimes referred to herein as substreams) in which the subpictures are coded. In particular, these input video streams do not need to be coded in an interdependent manner such that if one subpicture in one substream is RAP coded, the temporally aligned subpictures in another substream are also coded in the RAP manner.If some substreams are coded into a single common input video stream, this can be coded in a general manner such that each picture of each input video datastream is coded in such a way that all subpictures of each picture have their associated subpicture-related information in the input video datastream, which is either a first state for all subpictures of each picture or a second state for all subpictures of each picture. However, temporally aligned subpictures of other substreams may not be coded in a RAP manner. In synthesizing a video datastream containing random access point signaling granularity indications, the synthesizer / merger may freely configure the pictures of the video coded into the synthesized datastream having pictures subdivided into subpictures, and the subpicture-related information is adopted or taken from each input video datastream, even though at least one is RAP while at least the other is not. Thus, for example, there is no need to introduce one or more new states for the subpicture-related information present in the synthesized datastream for each subpicture. In other words, subpicture-related information can simply be transferred from each repository of one or more input video data streams.

[0011] A subpicture may be a tile or a group of tiles, meaning that a subpicture may be coded independently from other subpictures of the same picture. Alternatively, a subpicture may be an MCTS, that is, a tile or a group of tiles forming a set of tiles with restricted movement, which implies that a subpicture may also be coded independently from subpictures of other pictures that are spatially offset, whereby the picture may be subdivided into subpictures in a consistent manner such that, in that case, the subpictures of different pictures are aligned with spatially matching subpicture boundaries or are spatially aligned. Such a data stream containing these random access point signaling granularity indications will therefore be able to signal subpicture-related information using the same alphabet used in the repository of substreams that may have been coded in the picture granularity regime. The subpicture-related information may include integer value syntax elements that distinguish between multiple types, such as NAL unit types, for example. Each subpicture may be coded by one or more NAL units, and each NAL unit may have, for example, these integer value syntax elements. Accordingly, regardless of whether the random access point signaling granularity indication represents a first or second granularity, a first subset of one or more NAL unit types among a plurality of NAL unit types distinguished by integer value syntax elements corresponds to a first state of subpicture-related information, while a second subset of one or more NAL unit types corresponds to a second state.

[0012] Random access point signaling granularity markings may be included in the SPS or VPS of a data stream, and a set of one or more pictures may include a sequence of pictures of a video or form a range of random access points signaling granularity markings that cover the entire video. Alternatively, random access point signaling granularity markings may be delivered to the data stream per access unit, and a set of one or more pictures may form a range of random access point signaling granularity markings that include only one picture.

[0013] For any picture, in the case of subpicture-related information assuming a different state for different subpictures of this picture, a video decoder that cannot parse the random access point signaling granularity mark may identify the data stream as an unsuitable data stream. Other video decoders capable of parsing and decoding the random access point signaling granularity mark may adjust their decoding behavior according to this mark: they may identify the data stream as an unsuitable data stream for any picture in the case of subpicture-related information assuming a different state for different subpictures of this picture, for any picture in a set of one or more pictures forming a range of the random access point signaling granularity mark assuming a first granularity.

[0014] Additionally or alternatively, more functions of the video decoder may be enabled when the random access point signaling granularity indication assumes a second granularity, namely, subpicture granularity, which is discussed subsequently in relation to additional aspects of the present application, as these functions can be used advantageously to achieve a more effective video codec, regardless of whether the juxtaposition of subpicture-related information is used on the one hand and the random access point signaling granularity indication is used on the other. That is, a decoder according to the following aspect may operate only in the subpicture granularity regime without needing to signal the use of this regime.

[0015] According to a second aspect of the present application, the goal is to make the video codec more flexible in terms of use for subpicture unit coding, and for use, for example, for representing synthetic video. This picture is subdivided into subpictures. The objective is achieved by providing a video decoder in response to an evaluation of subpicture-related information transmitted in a data stream for each subpicture. In particular, according to the second aspect, the data stream in which such video is encoded contains information related to each subpicture of each picture, which identifies each subpicture as coded in a random access point manner under a first state and identifies each subpicture as not coded in a random access point manner under a second state. The video decoder examines the picture for the subpicture-related information of its subpictures. If the subpicture-related information assumes a first state for all subpictures of a predetermined picture for any picture, that is, that all subpictures represent random access points, the video decoder flushes the decoded picture buffer. Similar to the first aspect, this concept allows for the synthesis of data streams from actually generated substreams for different video codec domains where subpictures within a picture must match in subpicture-related information, that is, all subpictures of a picture must assume the same state. Nevertheless, when synthesizing these data streams, the state of the subpicture-related information for the various subpictures used to construct the data stream may simply be adopted or taken from therein. Furthermore, the video decoder can nevertheless identify true random access points in that the entire picture can be recovered without artifacts by identifying the picture in which the subpicture-related information assumes a first state for all subpictures of each picture.In combination with the first aspect, the video decoder may apply this function to identify a random access point picture when the random access point signaling granularity indication assumes a second granularity, i.e., a subpicture granularity, and the decoded picture buffer may be flushed in response to seeing that the subpicture-related information for such picture assumes a first state for all subpictures of each picture. In combination with the first aspect, this video decoder may change the scheduling when flushing the decoded picture buffer: in the case of a random access point signaling granularity indication assuming a first granularity, i.e., a picture unit granularity, the video decoder may flush the decoded picture buffer in response to the subpicture-related information assuming a first state for a specific picture's first subpicture according to the decoding / coding order between subpictures, since the subpicture-related information regarding other subpictures needs to assume the same state, i.e., the first state anyway. In the case of a random access point signaling granularity indication assuming a second granularity, for a specific picture, the flushing of the decoded picture buffer in response to subpicture-related information assuming a first state for all subpictures of a predetermined picture, i.e., the subpicture granularity, can be scheduled to occur sequentially when decoding the last subpicture of the picture.

[0016] The third and fourth aspects of the present application relate to the management of decoded picture buffers on the decoder side. These aspects of the present application may be combined with any of the first and second aspects and may have a purpose similar to that outlined above in relation to the first and second aspects. That is, the third and fourth aspects of the present application aim to provide a video codec that allows a synthesized data stream to be synthesized from subpicture-related substreams in such a way that the synthesizer / merger does not need to harmonize or integrate reference picture buffer descriptions describing the set of reference pictures that must remain in the DPB signaled from various substreams contributing to the synthesized data stream, so that the synthesized data stream can be synthesized from subpicture-related substreams in such a manner that the decoder can remove all other pictures from the decoded picture buffer at the start of this current picture, with respect to which picture forms the set of reference pictures for the current picture at the start of each picture. Rather, according to aspects 3 and 4, since the video codec can signal a reference picture buffer description that provides information about the set of reference pictures required for the data stream to decode the current picture, or enables decoding in the next picture for each subpicture of the currently coded / decoded picture individually, the reference picture buffer description may differ between the subpictures of the current picture. Such reference picture buffer descriptions describe, for example, through a source associated with a single subvideo, the picture containing the subpicture that serves as a reference to the current subpicture of the current picture, or the subpicture of the next picture associated with the current reference picture. When coded as separate subvideos, the buffer description of the subpicture is actually the same subvideo and indicates a reference picture containing the subpicture that must be maintained in the DPB when the subpicture serves as a reference to the current subpicture or the next subpicture of that subvideo.According to a third aspect of the present application, the decoder has the function of forming a combination of reference picture sets for the subpictures of this current picture that must remain in the DPB according to the reference picture buffer description of the subpicture, in order to mark the picture to remain in the decoded picture buffer while other pictures are removed from the decoded picture buffer, as at the end of the current picture decoding. According to a fourth aspect of the present application, the decoder may perform the emptying of the decoded picture in units of subpictures rather than complete pictures. The marking and removal of subpictures are performed separately for each subpicture. In other words, according to the fourth aspect, the decoded picture buffer is managed in units of one sub-picture decoded picture buffer per sub-picture of the currently decoded picture, and the marking of the sub-picture that must remain in each sub-picture decoded picture buffer when another one is to be removed is performed based on the reference picture buffer description submitted to the data stream for each sub-picture of the currently decoded picture. As already mentioned above, for example, in the case of a random access point signaling a particle size indication assuming a second particle size, the third and fourth aspects of the latter manage / empty the picture buffer decoded on a picture-by-picture basis when the random access point signaling particle size indication assumes a first particle size, i.e., picture particle size, and, for example, combined with the first aspect, the decoder may have the function just described in relation to the third and fourth aspects, in this case the decoder may be configured to perform emptying and marking of the decoded picture buffer for the currently decoded picture, i.e., based on the reference picture buffer description that was first signaled for the current picture.

[0017] A fifth aspect of the present application relates to POC processing. Similar to the third and fourth aspects, the fifth aspect aims to provide a video codec that allows a data stream to be composed of individually coded substreams, such as using different random access point periods, picture groups, or even different frame rates, so that temporally aligned subpictures contributing to a composite data stream signaled from various substreams having different POC values ​​are effectively resulted. The fifth aspect of the present application aims to alleviate such composite / merging procedures by being combinable with any of the prior ones, thereby freeing the composite / merger from having to harmonize the POC values ​​of substreams associated with different subpictures and having different sources. Rather, the video codec enables the data stream to signal different POC values ​​for sub-pictures that actually belong to the same picture of the data stream, and the decoder is configured to leak, for each sub-picture of the currently decoded picture, a final POC value for each sub-picture based on the decoded POC value for that sub-picture, and one or more inter-subpicture POC compensation parameters that are continuously updated by the decoder so that for each sub-picture and for each picture associated with each sub-picture, the final POC value of the sub-picture is the same for each picture of the video. One or more inter-subpicture POC compensation parameters can compensate for frame rate differences between contributing substreams, different sizes of picture groups, different minimum POC numbers, etc. One or more inter-subpicture POC compensation parameters may include POC offset values ​​and POC step size differences. Some parameters, such as the POC offset value in the random access point subpicture, can be determined by the decoder itself based on the difference between the signaled POC value of at least one other subpicture of the same picture that does not form a random access point and the final POC value.Hints regarding POC step size differences can be passed in turn from the data stream to guide the decoder and ease the determination of these POC step size differences. These hints can be inserted into the data stream by a synthesizer / merger that accesses this knowledge based on the higher-level parameters of the contributing substream.

[0018] A sixth aspect of the present application relates to the idea of ​​a video codec that implements a progressive decoding refresh in such a manner that the decoder recognizes the progressive decoding refresh, in terms of the point at which the progressive decoding refresh is completed or in terms of the picture, as well as in terms of performing prediction derivation based on coding parameters transmitted from the data stream during the progressive decoding refresh. In particular, according to the sixth aspect of the present application, the video decoder is configured to log, for each reference picture of the video, the subdivision of the reference picture into a refreshed picture region and a non-refreshed picture region. The decoder decodes information regarding the refresh update region within the currently decoded picture from the data stream and decodes the refresh update region using inter-prediction from the data stream. Additionally, the decoder decodes the first region from the data stream by deriving a prediction for the first region from the reference picture based on a signal of the data stream for the first region, by deriving a prediction for the first region from the reference picture, thereby determining that the first region of the currently decoded picture is separated from the refresh update region and coded / decoded independently of the non-refreshed picture region of the reference picture, based on coding parameters in a manner that depends on the location of the refreshed picture region of the reference picture so that the prediction is independent of the non-refreshed picture region of the reference picture. For example, motion vectors may be clipped to reach the non-refreshed picture region to fill the portion of the motion-compensated prediction block in the reference picture that exceeds the refreshed picture region with the non-refreshed picture region, and / or padding may be used. Alternatively, the motion vector candidate list construction may exclude motion vectors originating outside the refreshed picture region. The second region may be processed differently by the decoder. The second region can be decoded by a decoder that is separated from the first region and the refresh update region and includes predictions from the refreshed picture region as well as the non-refreshed picture region.The picture area refreshed by this method continues to grow for each picture. Since the decoder must track the growth of the refreshed picture area and perform prediction induction for the first area accordingly, the decoder becomes aware of the growth. Rendering so that the decoder recognizes the gradual decoding refresh offers several advantages: the decoder recognizes the current refreshing situation, possible random access periods, etc. The effort associated with the need to render prediction induction and this induction based on the location of the refreshed picture area of ​​the reference picture is almost cost-effective, as the decoder may need to implement similar measures, for example, at the outer boundary of a picture or the outer boundary of an independently coded picture tile. The encoder can encode coding parameters into a wider range of values ​​when the decoder redirects some signalable states of coding parameters from predictions that would result in dependency on the unrefreshed picture area of ​​the reference picture to predictions that depend solely on the refreshed picture area. However, expanding the signaling possible states of coding parameters can lead to higher compression efficiency in relation to the coding of these coding parameters. Brief explanation of the drawing

[0019] The advantageous aspects of the present application are the subject of the dependent claims. Preferred embodiments of the present application are described below in conjunction with the drawings: FIG. 1 shows a schematic diagram illustrating an environment for synthesizing a data stream based on multiple input substreams or merging substreams into a composed data stream. In this case, a decoder decodes the synthesized data stream; the synthesizer / merger itself, as well as the corresponding encoder for forming the substreams, are illustrated; and embodiments described in connection with FIGS. 2 through 8 are used to explain desirable functions and concepts that may be applied by the synthesizer / merger and / or decoder; FIG. 2 illustrates a schematic diagram illustrating a video codec concept indicating whether subpicture granularity or picture granularity is used for random access point signaling; FIG. 3 illustrates a schematic flowchart illustrating possible functions of a decoder for flushing a decoded picture buffer; FIG. 4 illustrates a schematic flowchart to illustrate an advantageous function for the decoder when performing decoded picture buffer flushing; FIG. 5 illustrates a schematic diagram illustrating a decoder configured to perform a decoded picture buffer that empties a subpicture unit; FIG. 6 illustrates a schematic diagram illustrating possible measures for processing the rearrangement of subpictures belonging to consecutive subvideos within a picture of a synthesized video data stream; FIG. 7 illustrates a schematic diagram illustrating a preferred action for a video decoder to process different POC domains for different subpictures of a video; FIG. 8 illustrates a schematic diagram illustrating the concept of FIG. 7 regarding POC offset compensation in a random access point subpicture according to one embodiment; FIG. 9 illustrates a schematic diagram illustrating an embodiment of an encoder and a decoder that allows for efficient progressive decoding refresh; FIG. 10 illustrates a schematic diagram illustrating a 360° streaming example in which a video data stream is composed of substreams related to MCTS between panoramic scenes; and FIG. 11 illustrates a schematic diagram illustrating a progressive decoding refresh. Specific details for implementing the invention

[0020] With respect to FIG. 1, the operation of constructing a data stream from a repository of a substream is described. This is done to explain the problems associated therewith, namely, problems related to various sources of the substream and differences, such as random access point localization, related POC values, and different GOP structures. These problems are resolved according to the embodiments described later. These embodiments relate to different aspects of the present application and can also be combined to create additional embodiments as further described below. However, it should be noted that some of the assumptions in FIG. 1 are not necessarily required for the embodiments described later; in other words, some of the features described and discussed with respect to FIG. 1 may be omitted with respect to the embodiments, and nevertheless, since the advantages from the aspects covered by each embodiment are achieved, they are subsequently described. This situation will also be discussed below.

[0021] FIG. 1 illustrates a scenario in which a data stream (10) is generated by configuring the data stream (10) based on a repository of multiple or substreams (12). This synthesis or merging process is performed by a synthesizer / merger (14). In particular, the data stream (10) is generated in such a way that it has a video (16) of an encoded picture (18). The arrow (20) in FIG. 1 describes the display time order defined between the pictures (18) of the video (16), that is, the order in which the pictures (18) are output by the decoder (22) when the decoder (22) decodes the video (16) from the data stream (10). The picture (18) is spatially subdivided into subpictures (24) as exemplified by the dotted line (26) in FIG. 1 to indicate the boundaries of the subpictures. In particular, the data stream (10) has a video (16) encoded in such a way that the subpictures (24) are encoded independently of each other. The independent encoding of a subpicture into a data stream (10) and the consequent decoding of a subpicture (24) from the data stream (10) are related in an independent manner to a subpicture within at least one picture (18), i.e., within the picture. For example, in FIG. 1, the picture (18) is shown as being subdivided into subpictures (24) and containing six subpictures listed in FIG. 1, with each picture (18) having a number written inside. Thus, for example, subpicture 3 is encoded into the data stream (10) in an independent manner from any other subpicture of the same picture (18). Thus, subpicture 3, indicated by hatching in FIG. 1, can be decoded from the data stream (10) by the decoder (22) in an independent manner from other subpictures 1, 2, 4, 5, and 6 of the same picture (18), which are also indicated by hatching. However, another feature of the data stream (10) according to Fig. 1 is that the picture (18) of the video (16) is spatially subdivided into sub-pictures (24) at a constant rate over the picture sequence.That is, the boundaries (26) of these are spatially aligned when comparing different pictures. In addition, since each subpicture (24) is encoded independently of other subpictures of the same picture as well as subpictures of other pictures spatially offset from each subpicture, co-placed or spatially aligned subpictures, i.e., subpictures (24) of FIG. 1 with the same number written on them, form a single subvideo, that is, all subpictures belong to other subvideos in FIG. 1 or have different numbers written on them internally. For example, in terms of HEVC, each subpicture (24) of FIG. 1 may be an MCTS. Without the coding independence of different pictures of the video (16) or the latter characteristic of spatially offset subpictures, the subpicture (24) may be, for example, one or more tiles, in terms of HEVC.

[0022] Individual sub-videos formed by spatially aligned sub-pictures (24) of the video (16) are illustrated again in FIG. 1 as 28. Thus, all these sub-videos are composed of a sequence of sub-pictures (24). The encoder (30) independently encodes these sub-videos (28) into a plurality of substreams (32) or their storage. In fact, the sub-videos (28) can be processed in groups by the encoder (30). Groups of sub-videos (28) can be coded into a single input data stream in a manner that is coded independently of each other so that these input video data streams are composed of substreams (32) for each sub-video (28), thereby allowing a video composed of groups of sub-videos (28) to be encoded therein. The substreams (32) are received by a synthesizer / merger (14), which in turn forms a data stream (10) based on the substreams (32). Depending on the application, only a subset of subvideos (28) or a subset of substreams (32) may actually contribute to the data stream (10), and this subset may change or vary between the pictures (18) of the video (16) beyond this.

[0023] The purpose of these synthesis / merging operations performed by the synthesizer / merger (14) is to perform operations in the encoded domain. That is, requantization of prediction residuals or re-execution motion compensation is avoided by the synthesizer / merger (14). Each subpicture (24) of the subvideo (28) is coded, for example, with one or more NAL units (34) of the corresponding substream (32), and if the subpicture (24) belongs to the ones that must contribute to the video (16) of the synthesized datastream (10), the synthesizer / merger (14) inserts each one or more NAL units into the datastream (10) as illustrated using the same reference sign, i.e., 34, but nevertheless, to indicate that the synthesizer / merger (14) may need to adapt other coding parameters other than motion vector information or prediction residual data to these NAL units, it is indicated by an apostrophe. According to the embodiment described below, the synthesizer / merger (14) is given the opportunity to reduce the portion of the substream (32) that must be adapted when forming the data stream (10) based on the substream (32). This is also related to the parameter set (36) of the substream (32) or the input video data stream to which the substream is a part, when the synthesizer / merger (14) aims to form a parameter set (36) corresponding to the data stream (10).

[0024] The situation of FIG. 1 can be illustrated with further reference to FIG. 10. FIG. 10 illustrates a case where a repository of sub-videos (28) forms a subsection of a 360° video. In the example of FIG. 10, the entire scene is subdivided into 24 sub-videos. Many sub-videos are provided in the example of FIG. 10: the 24 sub-videos are encoded by the encoder (30) into two input video data streams (38a and 38b), such that the scene is shown in high resolution and used more efficiently once using a larger RAP distance, and at other times using a lower RAP distance in relation to the input video data stream (38b), thus resulting in less efficient compression. Likewise, two additional input video data streams are provided, each having a complete scene encoded with 24 sub-videos, one data stream (38c) having a high RAP distance, and the other data stream (38d) having a low RAP distance. A picture (16) of a synthesized video datastream is exemplified as showing the entire scene. In particular, each picture (16) consists of 12 mutually temporally aligned subpictures of subvideos coded by datastreams (38a and 38b), and the 12 subpictures are mutually temporally aligned and temporally aligned with respect to high-resolution subpictures taken from datastreams (38c, 38d). That is, in the example of FIG. 10, the viewport is assumed to be 12 subpictures or tile widths. In the middle of FIG. 10, hatching indicates the selection of subpictures (24) within the picture (16) of the composed datastream, which contributes to the picture (16) of the video composed of high resolution and low resolution. Another selection is made in relation to different RAP distance versions.For a specific picture time instance, for each high-resolution subpicture (24) and each low-resolution subpicture (24), there is one or more NAL units in the data streams (38a and 38c) which are each more efficiently coded using a high RAP distance, and in other data streams (38b and 38d) which are each coded using a low RAP distance. Preferably, the configured data stream (10) is configured using the NAL units of the more efficiently coded data streams (38a, 38c). All data streams (38a to 38d) may have individually encoded substreams (38) using synchronous random access points within each data stream (38a to 38d). That is, the video picture coded in each of the data streams (38a to 38d) is subdivided into 24 subpictures (24), and for each picture, all of the subpictures are coded or none are coded in each input video data stream (38a to 38d) in a random access point manner.

[0025] A problem arises whenever the viewport changes. That is, the selection of contributing substreams that contribute to the compositing / merging of the datastream (10) changes. In the example of FIG. 8, 96 substreams are available, but only 24 contribute to the composited datastream (10) for a specific picture. For example, assume that the viewport is changed by two subpictures or tiles (24). Thus, four of the subpictures (24) of the picture (16) change their outgoing substreams: two of these four originating from the high-resolution datastream (38a) are now taken from one of the low-resolution datastreams (38c and 38d), and the other two subpictures (24) originating from the low-resolution datastream (38c) are now taken from the low-resolution datastream (38d). In particular, to reduce the time to encounter the next random access point, lower RAP distance versions of the data stream (38b and 38d) are used in advance for each of these four subpictures (24). The four subpictures mentioned change their positions within the picture (16). However, all other subpictures (24) remain downloaded from the more efficient versions of the data stream (38a and 38c) respectively. They do not require random access points. As a variation, it should be noted that by using downsampling from high resolution to low resolution to derive a reference picture of the corresponding non-RAP subpicture of lower resolution, the synthesizer / merger (14) may be allowed to insert the corresponding NAL unit of the more efficiently coded input video data stream (38c) into the data stream (10) for these subpictures (24) that change from high resolution to low resolution.

[0026] That is, in this example of the scenario just exemplified in relation to Fig. 1 in combination with Fig. 10, the following problem related to 360° streaming is described. In particular, when a user subscribes to the service, that is, when tuning into the service, that is, when wanting to view a panoramic image, a random access point picture is required. That is, without the embodiment described later, the compiled data stream (10) would require that each sub-picture within the starting picture be coded in a random access point manner. That is, all NAL units of the picture must be coded in a random access point manner. In HEVC language, for example, they must be NAL unit types IDR, CRA, or BLA. That is, without the embodiment further described below, the decoder is obligated to encounter the RAP picture to start decoding. However, when the user switches the viewport, as previously described in relation to FIG. 10, only some sub-bitstreams (32) are newly added to the subset of bitstreams (32) that contribute to or are merged into the data stream (10), so it is very inefficient to require all sub-bitstreams (32), that is, those remaining within the subset of contributing bitstreams, to have a starting picture (24) coded in a random access point manner. Rather, as schematically described in relation to FIG. 10, it would be desirable for only some sub-picture bitstreams (32) to effectively have RAP and the rest not. However, this means the following: the NAL unit type displayed in the NAL unit (34') within the data stream (10) must be changed according to the settings of the base NAL unit (34).In particular, since it would be a requirement of the data stream that if one of the subpicture bitstreams has one of these NAL unit types, all others must have the same NAL unit type, the NAL type of the NAL unit (34) of the subpicture (24) of the starting picture set to IDR, CRA, or BLA may not be associated with the same NAL unit type in the corresponding version (34') and data stream (10). In particular, when the decoder defined so far looks for a specific access unit of the data stream (10), that is, an IDR or CRA (previously including EOS NALU) NAL unit for a specific picture (18) of the video (16), the decoding process restarts and the decoder picture buffer (DPB) is flushed because neither picture is available or used for reference. This means that, for example, if the previously described scenario is implemented using HEVC for the current version of the data stream (10), when merging sub-bitstreams (32) together, for some pictures (18) of the video (16), measures must be taken if all contributing sub-bitstreams (32) contain only some of the NAL unit types corresponding to random access point coding, rather than all of them. Rather, in that case, the RAP NAL unit(s) within such pictures (18) must be rewritten as non-RAP NAL units, which also require changes to other headers such as slice, tile, and / or tile group headers. This is because the content depends on whether the NAL unit in question is RAP or not.

[0027] Accordingly, the embodiment described below in relation to FIG. 2 aims to improve the signaling of subpicture RAP beyond the technology currently at hand. For example, there have already been proposals to escape the obligation just described to assign a NAL unit type to the NAL unit of the data stream so that the NAL type to which the subpicture is coded is constant within each image. For example, WO 2015 / 059194 proposes using NAL unit-based signaling, e.g., a NALU header, more specifically a dedicated NALU type, to indicate a recovery point SEI message or a subpicture random access point type containing replacement information that replaces all parameters required to acquire an HEVC-compliant data stream. However, to solve the problem described above in the manner proposed in the latter document, the synthesizer / merger (14) must perform the cumbersome task of "stream translation," and therefore, the embodiment outlined below attempts to overcome this problem and avoid the need for the synthesizer / merger (14) to consume additional RAP type signaling overhead.

[0028] FIG. 2 illustrates a concept of a video codec according to an embodiment of the present application relating to the first aspect of the present application. FIG. 2 illustrates an embodiment of a corresponding data stream (10) and a video decoder (22) that can follow a corresponding new video codec. Since the data stream (10) of FIG. 1 can correspond to the description now presented in relation to FIG. 2 and consequently can solve at least partially the problem described above, reference numeral 10 of FIG. 1 has been reused.

[0029] That is, the video decoder (22) of FIG. 2 receives a data stream (10) containing video of an encoded picture. The data stream (10) contains video encoded in such a way that the video picture is spatially subdivided into subpictures that are encoded independently of each other. Each subpicture is coded into one or more NAL units (34') of the data stream (10). That is, the NAL units (34') carry motion information and prediction residual information, in particular. Note that, in addition to the description given above in relation to FIG. 1, the subpicture coding independence may only relate to subpictures within a single picture (18) of the video encoded in the data stream (10). Accordingly, FIG. 2 illustrates, by example, a single picture (18), subdivision into subpictures (24), and corresponding NAL units (34') in which the subpictures (24) are coded within the data stream (10). Naturally, coding can also be performed to generate separately coded sub-videos as outlined above in relation to Fig. 1.

[0030] The decoder (22) decodes subpicture-related information (40) for each subpicture (24) of each picture (18) from the data stream (10). The subpicture-related information for a specific subpicture (24) may be, for example, a syntax element indicating the NAL unit type of one or more NAL units (34') to which each subpicture (24) is coded. Assuming a first state, the subpicture-related information (40) identifies the subpicture (24) as being coded into one or more NAL units (34') in a random access point manner. That is, this forms a subpicture random access. That is, the subpicture will be coded into one or more NAL units (34') in a self-contained manner without referencing any other subblock. Assuming a second state, the subpicture-related information will identify the subpicture (24) as not being coded in a random access point manner. That is, a NAL unit (34') having such subpicture-related information (40) may have an associated subblock (24) coded therein using coding dependencies for other subblocks. In particular, the subpicture-related information (40) may include an integer value syntax element for each subpicture (24) that distinguishes between a plurality of NAL unit types, a first subset of one or more NAL unit types corresponding to a first state, i.e., a subpicture RAP state, and a second subset of one or more NAL unit types corresponding to a second state, i.e., a non-RAP state.

[0031] However, the decoder (22) of FIG. 2 is further configured to read a random access point signaling granularity indication (44) from the data stream (10) in relation to a set (42) of one or more pictures. It may be transmitted from the data stream (10) as a kind of high-level signaling associated with one or more pictures. That is, in that case, the set (42) of one or more pictures (18) will include a sequence of pictures such as a GOP or the entire video (16). The indication (44) may consist, for example, of a set of sequence parameters or a set of video parameters of the data stream (10). On the other hand, the indication (44) may be associated with only one access unit, that is, one picture (18) of the video (16). In other words, the set (42) may include one picture. Such access unit-specific signaling may be included in the PPS, set of picture parameters, or access unit identifier, etc. of the data stream (10). This indication (44) will have at least two signalable states / grain. When assuming or signaling the first particle size, the indication (44) will indicate that for each picture (18) of one or more picture sets (42), the subpicture-related information (40) assumes a common state for all subpictures (24) of each picture (18). In other words, the subpicture-related information (40) will assume the same state for all subpictures (24) within one picture (18) in the set (42). The same state may be true when considering the level of the NAL unit type. That is, the NAL unit type of the NAL unit (34') belonging to the subpicture (24) of one picture (18) may be the same for the first particle size, i.e., the picture particle size, in the case of the indication (44) signaling the picture particle size.

[0032] However, if the indication (44) represents a second particle size that can be called a subpicture particle size, this indicates that for each picture (18) of the set (42), the subpicture-related information (40) is allowed to assume a different state for each picture (18)'s subpicture (24).

[0033] However, in any case, the aforementioned synthesizer / merger (14) may utilize the possibility of setting the display (44) to the subpicture grain size in the following manner; it is associated with a subpicture (24) that is temporally aligned with the current picture referenced by this access unit, such as 46 in FIG. 2, and thus constitutes an access unit of the data stream (10) by combining together the NAL unit (34) of the repository of the coded substream (32), so that the NAL unit type of the NAL unit may be not acquired, adopted, or modified, and thus the NAL unit (34') of the data stream (10) within this access unit (46) matches the NAL unit placed together and the subpicture related information (40).

[0034] Hereinafter, some embodiments describing the function of the decoder (22) of FIG. 2 are described with respect to cases where the indication (44) represents subpicture particle size. However, all such embodiments and descriptions should be understood to apply to decoders that are not configured to read or decode the indication (44) from the data stream (10). In other words, the embodiments described subsequently should not be limited to cases where the indication (44) is present in the data stream and the decoder (22) is configured to read the same thing from the data stream (10). Rather, embodiments described subsequently in additional aspects of the present application will be interpreted as descriptions of decoders capable of handling the fact that certain settings, such as random access point coding or non-random access point coding of subpictures, vary within a single picture. However, such decoders do not have specific alternative functions, as this is true for the decoder (22) of FIG. 2 which operates differently depending on the indication (44).

[0035] To briefly summarize what has been explained so far, FIG. 2 illustrates a concept in which, for the connection of a coded video sequence (CVS), an additional indication (44) of a data stream (10), such as signaling at a higher level, such as signaling at the coded video sequence level or bitstream level, enables NAL unit level signaling, such as NAL unit type signaling, to be reinterpreted as a subpicture random access indication. For example, the indication (44) may be signaled in the form of a flag that can be called sub_picture_random_access_process_enabled_flag. For example, when this flag is equal to 0, the NAL unit type indicating the random access function will be interpreted by the decoder (22) as full picture level random access. However, when the flag value is equal to 1, each NAL unit type is interpreted as a subpicture random access point.

[0036] Alternatively, the indication (44) may be implemented as access unit signaling. For example, an access unit identifier NAL unit may be used to indicate what type of random access is allowed in an access unit starting from the access unit identifier NAL unit. A syntax element may be added to an access unit identifier NAL unit indicating one of the following states: 1) all subpictures (24) of the picture in the access unit to which the identifier belongs contain random access pictures (IRAP), 2) some subpictures (24) contain random access pictures (IRAP), and 3) none of the subpictures (24) contain random access pictures (IRAP).

[0037] By using the mark (44), the decoder can easily identify what kind of random access function can be applied to the picture(s) (18) of the set (42), which defines the range of the mark (44), i.e., the range to which the mark (44) is related, and which of the following processes described in the next embodiment should be performed by the decoder (22).

[0038] Before resuming the explanation of what the decoder (22) does when the indication (44) signals the subpicture grain size, it should be noted that when the indication (44) indicates the picture grain size and the subpicture related information (40) assumes a different state for any picture (18) of the set (42) to which the indication (44) belongs, and for a different subpicture (24) of that picture, the decoder (22) of FIG. 2 may be configured to identify the data stream (10) as an unsuitable data stream, that is, a stream that does not follow the video codec.

[0039] FIG. 3 illustrates an example of the behavior of a decoder (22) in relation to flushing a decoded picture buffer, that is, a buffer in which a reference picture is buffered based on a picture in which a picture that has not yet been decoded can be predicted. In solid lines, FIG. 3 illustrates the function of the decoder (22) for the case of subpicture granularity, while dotted lines illustrate the corresponding function for the case of picture granularity. Again, it should be noted that FIG. 3 should be treated as an extension of the function described for the decoder described in relation to FIG. 2, which can decode the indication (44) and operate according to the mode shown in FIG. 3, or as an alternative description of the decoder (22) for the case where the data stream (10) does not contain the indication (44) and the decoder (22) inevitably operates according to the subpicture granularity mode, and both.

[0040] As illustrated in FIG. 3, the decoder (22) according to the embodiment of FIG. 3 is configured to check, for a specific access unit or a specific picture (18), whether the subpicture-related information (40), as illustrated in 50, assumes a random access point state for all subpictures (24) of the picture. If so, that is, in response thereto, the decoder flushes the picture buffer (48) decoded in 52. Again, the decoder (22) may apply this function of steps 50 and 52 to a picture (18) where the indication (44) indicates the subpicture grain size, whereas in the case of an indication (44) indicating the picture grain size, an alternative approach illustrated by the dashed line in FIG. 3 may be applied. According to this alternative approach, the decoder (22) checks whether there is random access point coding in 54 for a specific access unit or picture (18), for subpicture-related information (40) for the first subpicture (24) encountered in the access unit, or for each picture, and if so, that is, in response to this, the decoder is configured to flush (56) the decoded picture buffer (48). Different operations of the decoder (22) for flushing the decoded picture buffer (48) in the case of an indication (44) showing subpicture particle size on one side and picture particle size on the other may appear as different scheduling of flushing the decoded picture buffer in 52 or 56, as a result, in relation to the time when the decoder (22) starts decoding the individual NAL unit (34') of the access unit (46), such as in relation to the time when the decoder (22) retrieves the NAL unit (34') of the access unit (46) from the coded picture buffer.If it is evident that all subpictures of the current access unit are random access points, flushing of the picture buffer (48) decoded in step 52, i.e., in the subpicture granularity domain, may occur when decoding one or more NAL units (34') of the last subpicture (24) in the decoding order (58); however, in the case of step 56, i.e., flushing of the picture buffer (48) decoded in the picture granularity domain, may occur when decoding one or more NAL units (34') of the first subpicture (24) in the coding / decoding order (58), i.e., when it is evident that the subpicture-related information (40) present in the data stream for that subpicture indicates random access point coding, i.e., before decoding the second subpicture (24) of the current picture or access unit in the coding / decoding order (58). Again, the functions described in relation to FIG. 3 with respect to the picture grain mode may not be implemented in the decoder (22) according to an alternative embodiment in which the indication (44) is not present.

[0041] That is, FIG. 3 describes an embodiment in which the decoding process is modified to flush the decoded picture buffer (48) of the decoder (22) and perform a random access point operation only when it is clear that all subpictures (24) of the data stream (10) for a specific picture are subpicture random access points and therefore the entire picture is a random access point. Accordingly, the decoder (22) may be configured to decode up to the last NAL unit (34') of the current access unit to determine whether this access unit is a whole random access point.

[0042] The embodiment of the decoder described below in relation to FIG. 4 focuses on the function / mode of operation of the decoder related to the removal of individual reference pictures or pictures stored in the decoded picture buffer (48) that are no longer needed for the picture yet to be decoded. That is, while FIG. 3 focuses on the flushing of the decoded picture buffer (48), i.e., the removal of all pictures currently buffered in each decoded picture buffer, FIG. 4 focuses on another aspect in which the data stream (10) can convey information about the reference picture buffer description, i.e., the reference picture that must remain in the decoded picture buffer (48), serving as a possible basis for inter-prediction regarding the picture yet to be decoded. To this end, the data stream (10) conveys information about the set of reference pictures to which the information is updated to the picture unit or access unit. Similar to the subpicture-related information (40), this information regarding the set of reference pictures to be retained in the decoded picture buffer for future use—namely, the reference picture buffer description—is something signaled in the data stream (10) for each subpicture (24), i.e., one or more corresponding NAL units (34'). According to the variation illustrated in FIG. 4 using solid and dotted lines, this reference picture buffer description may be information subject to similar constraints according to the indication (44) as described so far in relation to the subpicture-related information (40) for indicating RAP and non-RAP coding. Thus, the reference picture buffer description is shown in FIG. 2 using the reference symbol (40') to explain the similarity in terms of constraint according to the indication (44), or is shown as an alternative or additionally similar to the indication (44).In particular, the set of reference pictures indicated by the buffer description (40') for the first subpicture (24) of a specific picture (18) may be identical to the set of reference pictures indicated by the buffer description (40') for another subpicture (24) of the picture (18), or may include at least all reference pictures indicated by the reference picture buffer description (40') for the second and next subpictures of the picture (18) in the coding / decoding order (58). In the case of the indication (44) for subpicture granularity (or the corresponding indication for reference picture buffering processing), the set of reference pictures indicated by the buffer description (40') for various subpictures (24) of one picture (18) may differ from one another in any way, that is, may indicate mutually different sets of reference pictures. The difference between them means that a subpicture within one picture and a corresponding subsequent subpicture in the next picture require a different reference picture, that is, a different set of reference pictures. The predicted references are contained in different sets of reference pictures. For example, the mutual correspondence of subpictures originates from their origin, that is, they may have been separately coded as a corresponding substream (32) as a subvideo (28). The latter freedom allows the synthesizer / merger to synthesize the data stream (10) based on the NAL unit (34) in the substream (32) storage by simply taking, without any modification, the reference picture buffer description (40') that is identical in the NAL unit (34') of the final data stream (10) on one hand and in the NAL unit (34) in the storage of the substream (32) on the other hand when combining such NAL units to form one access unit of the data stream (10).

[0043] Therefore, before resuming the description of FIG. 4, this implies that the data stream for the embodiment of FIG. 4 does not necessarily contain subpicture-related information (40). The embodiment described in relation to FIG. 2 may or may not include a reference picture buffer description (40'). Then, similar to the description above in relation to FIG. 3, the description of FIG. 4 may be interpreted as describing possible extensions / modifications of self-contained embodiments in which the decoder implements the function exemplified in FIG. 4 using a continuous line, as well as embodiments described so far for FIG. 2 and 3. That is, in particular, the indication (44) may not be present in the data stream.

[0044] According to FIG. 4, the decoder (22) is configured to perform picture removal from the decoded picture buffer (48) as follows. In particular, the decoder periodically performs steps 60 and 62 for each picture to mark the picture in the decoded picture buffer (48) that must remain in the decoded picture buffer (48) to function as a predictive reference for the picture not yet decoded in step (60), and to remove 62 unmarked reference pictures from the decoded picture buffer (48) that are not marked and are not required for future display / output by the decoder. However, to perform the marking (60), the decoder performs decoding (64) for each subpicture i of the current access unit or current picture buffer description (40') representing the reference picture set RefPi. The picture in the decoded buffer (48) to be marked in step (60) is the set RefP for all subpictures of the current access unit or picture. iIt is identified (66) by forming a union of. As illustrated in FIG. 4, the removal of unmarked reference pictures from the decoded picture buffer in step 62 may therefore occur when decoding the last subpicture (24) of the picture in the current access unit or coding / decoding order (58). That is, removal (62) may occur after parsing or decoding all subpictures (24) of the current picture, or after at least all subpictures (28) or, more precisely, all NAL units (34') are taken from the decoded picture buffer. Naturally, unmarked pictures are removed only if they are not output pictures or are not needed for output because they have already been output.

[0045] In the case of a decoder (22) capable of operating in the picture grain size mode illustrated by the dashed line in FIG. 4 as well as the subpicture grain size mode described in relation to steps 60 through 66, while the decoder (22) can switch according to the indication (44), the decoder (22) may alternatively operate as follows when removing a picture from the decoded picture buffer. In particular, when the picture grain size mode is active, the decoder (22) performs the marking (68) of a reference picture to be retained in the decoded picture buffer (48) and the removal of an unmarked reference picture from the decoded picture buffer (48) in an access unit / picture unit in step 70, but other operations may be applied in relation to determining the reference picture to be marked and scheduling of the removal (70). In particular, the decoder can decode a buffer description in step (72) that indicates a reference picture set for the subpicture (28) first encountered in the coding / decoding order (58), and in step (68), for the index corresponding to the decoding / coding order, identify the reference picture marked in step 68 as that set, i.e., RefP1 (74). Naturally, the decoder (22) can also read / decode a reference picture buffer description (40') from a data stream (10) for a current access unit or another subpicture of the current picture that is not the first in the coding / decoding order (58), but this information is not used for marking in step 68. Additionally, the removal of the unmarked reference picture in step (70) may occur before decoding the second subpicture in the coding / decoding order (58), or, in other words, after parsing or decoding the first subpicture in the coding / decoding order (58). Naturally, the unmarked picture is removed only if it is not an output picture or is not needed for output because it has already been output.

[0046] According to the possibility just mentioned, if the decoder (22) also reads a reference picture buffer description for a subpicture (28) other than the first subpicture of the coding / decoding order (58) for a specific access unit, the decoder (22) may be configured to identify the data stream (10) as an unsuitable data stream if the reference picture set indicated by the buffer description (40') for any subpicture (28) other than the first one in the coding / decoding order (58) contains a reference picture that is not included in the reference picture set indicated by the buffer description (40') for the first subpicture (24) of that picture.

[0047] FIG. 5 illustrates an alternative to the concept outlined above in relation to FIG. 4. According to FIG. 4, the decoded picture buffer (48) is emptied on a picture-by-picture basis. Actions are taken according to the subpicture granularity mode to remove only the pictures that are no longer needed. According to the embodiment of FIG. 5, the decoder (22) can perform the emptiness of the decoded picture buffer (48) on a subpicture (24) basis. In relation to the previously described embodiment and possible combinations of this embodiment, the same mode provided above in relation to FIG. 4 applies. That is, the embodiment described below in relation to FIG. 5 can be combined with the description of the decoder of FIG. 2 or FIG. 3. Thus, the decoder implemented according to FIG. 5 may or may not be configured to operate alternatively in the picture granularity mode in response to, for example, the indication (44).

[0048] The concept according to FIG. 5 is as follows. In particular, the video (16) is encoded such that the data stream (10) decoded by the decoder (22) is consistently subdivided into spatially non-overlapping subpictures (24) or co-aligned subpictures (24) in which subpictures belonging to different sub-videos are coded independently of each other. Thus, coding occurs in the sub-video (28) and leads to the storage of the substream (32). As described above in relation to FIG. 1, each substream (32) is encoded with each sub-video (28), and each sub-video (28) contains possible candidates for a specific subpicture, i.e., a subpicture (24), in the picture (18) of the video (16) of the data stream (10). The substreams (32) are encoded independently of each other. They include one or more NAL units (34) per subpicture (24). As part of the encoding process, the NAL unit (34) may already be provided with a subpicture identifier (80). That is, each NAL unit (34) may be tagged with this subpicture identifier (80) to enable distinction between the repositories of the substream (32). As described in relation to FIG. 1, to form the access unit (46) of the data stream (10) together, a subset of the repositories of the substream (32), or more precisely, only an appropriate subset, may be selected to use the NAL unit (34) in which the subpicture corresponding to the current timestamp is encoded. When combined, the NAL unit (34') generating the data stream (10) still has the subpicture identifier (80). In FIG. 5, this is exemplified in the following way: the identifier (80) of the repositories of the bitstream (32) is distinguished by integers 1 … N. Each picture (18) of the video (16) coded by the data stream (10) has six subpictures (24).For each subpicture (24) within a picture (18), the data stream (10) is a subpicture identifier (80A) in the corresponding access unit (46). i It includes one or more NAL units (34') tagged using ), where i = 1 … 6, and A i ∈ {1 … N}. Which subvideo (28) or which substream (32) is selected to form the data stream (10) may vary depending on the application and may be constant or change over time. Additionally, each NAL unit (34) and its version (34') that together form the data stream (10) is indicated as containing the aforementioned reference picture buffer description (40'). As is evident from the separate encoding of the substream (32), this information (40') differs for each substream (32).

[0049] In performing emptying of the picture buffer (48) decoded in units of subpictures (24), the decoder (22) operates as follows. Generally speaking, the decoder (22) decodes the buffer description (40') for the reference picture set for each subpicture (24) of the currently decoded picture (18) from the data stream (10). For each subpicture (24) of the currently decoded picture (18), the decoder uses the reference picture buffer description for each subpicture (24) to identify the corresponding subpicture that must remain in the decoded picture buffer (48) that is spatially placed in each subpicture, that is, that belongs to the same subvideo (28). Other subpictures are removed. Thus, removal is performed in units of subpictures.

[0050] As illustrated in FIG. 5, the video decoder may, for this purpose, logically divide the decoder picture buffer (48) into subpicture DPBs (481 to 486), one for each subpicture (24) in which the picture (18) of the video (16) is currently spatially subdivided. Meanwhile, it should be noted that the picture (18) is illustrated in the drawings of this application as being subdivided into six subpictures merely for illustrative purposes. Other numbers are also possible. Each subpicture DPB (48 i ) is used to buffer the subpicture (24) of the picture (18) coming from the NAL unit (34') in the data stream (10) tagged with a specific subpicture identifier (80) associated with each subpicture DPB. In other words, the decoder (22) buffers each subpicture DPB (48 i ) a predetermined subpicture identifier A i Associate with, and accordingly, each subpicture DPB (48 i ) is a predetermined subpicture identifier A i It is used exclusively to buffer subpictures of a previously coded picture (18) coded by a NAL unit (34') tagged with the same subpicture identifier (80). For the currently decoded picture, the decoder (22) performs the following: each subpicture DPB (48 i For ), the decoder (22) checks whether the currently decoded picture (18) includes a corresponding subpicture (24), and if so, uses the set of reference pictures indicated for the corresponding subpicture by the latter buffer description (40') to each subpicture DPB (48 i Identifies the subpictures within each subpicture DPB (48i) that must remain in the ) and the decoded picture buffer (48), respectively. All such subpictures in the subpicture DPB (48i) are identified in the subpicture DPB (48 iA subpicture coded with a NAL unit (34') tagged with a specific distinguishable subpicture identifier in the same subvideo (28) is simply collected to belong to the same subvideo (28). However, for a specific subpicture DPB (48i), if the NAL unit (34') does not exist in the current access unit and therefore the corresponding subpicture (24) is not included in the current picture (18), the decoder (22) flushes the corresponding subpicture DPB (48i) and reassigns each subpicture DPB (48i) to another predetermined subpicture identifier of a non-corresponding subpicture (24) of the current decoded picture (18) that is not identical to the predetermined subpicture identifier of all subpicture DPBs (481 to 486). Instead of flushing, depending on the application, the decoder may process the subpicture contained within by sub-sampling, etc., instead of flushing the reassigned subpicture DPB, in order to reuse the reference subpicture contained within as a reference subpicture for the subpicture (24) of the predetermined subpicture identifier for which the subpicture DPB was reassigned.

[0051] Before proceeding with the description of the present application along with the description of additional embodiments, the embodiments outlined above in relation to FIGS. 4 and 5 are briefly summarized again. In particular, FIG. 4 presents the concept of total picture dumping, whereas FIG. 5 illustrates an embodiment for partial picture dumping. Both embodiments have in common that they handle situations where the decoded picture buffer flushing is performed not in response to a random access point, but in response to a reference picture buffer description, that is, information regarding a reference picture / subpicture that is no longer needed for reference. As described above, since a random access point is not the only case where it is determined that picture information is no longer needed for reference, it can be removed from each decoded picture buffer (if already output). In AVC or HEVC, pictures in a decoded picture buffer are marked as "used for reference" or "not used for reference" during the decoding process. For example, in HEVC, there is signaling for what is called the Reference Picture Set (RPS). RPS indicates which pictures in the decoded picture buffer are used as references for the current picture or subsequent pictures in the decoding order. If not needed for output (e.g., if already outputted), those not listed in the RPS are marked as "not used as reference" and can be removed from the decoded picture buffer.

[0052] As discussed above, in the case of 360° where only some tiles / subpictures are switched from high resolution to low resolution or vice versa, the reference picture may differ, so random access points may be aligned.

[0053] In the case of RAP, when multiple bitstreams are concatenated together, that is, converted into a sub-picture bitstream of a new bitstream (10), RPS must rewrite which process increases complexity for the work performed by the synthesizer / merger (14).

[0054] The new video codec may have a different method for marking reference pictures, but the problem always applies: different references may be applied to different subpictures within the bitstream, and as illustrated with dotted lines in Fig. 4, the marking process is performed for the first slice / tile of the access unit, and if this needs to convey all information for all slices / tiles of the access unit, then the header needs to be rewritten when the streams are merged together.

[0055] The idea of ​​the embodiments of FIGS. 4 and 5 is to change the reference picture marking process to allow random access to subpictures or subpicture supply and removal to the decoded picture buffer.

[0056] According to the embodiment of FIG. 4, decoding of the picture buffer is still performed on a picture-by-picture basis, but subpicture random access is still allowed. Depending on the feasibility of implementing the embodiment of FIG. 4, signaling for a sequence or picture level, such as the mark (44), defines whether the marking process is performed in the picture granularity manner exemplified by the dashed line in FIG. 4, or only after the last subpicture of the current access unit, such as after parsing or after decoding the NAL unit or slice or tile of the last subpicture in the current access unit. The signaling switching between the two methods may be included in a set of parameters such as SPS or PPS in the form of sub_picture_management_process_enabled_flag.

[0057] In the case of subpicture processing where the only mode in which the transition between the two modes is turned off, as described above, the process may include, depending on the implementation example, a step of generating a temporally marked picture list for each subpicture (24) of the current picture (18), and finally, checking which one is marked as "not for reference." For example, using RPS signaling (40') from HEVC as an example for the reference picture marking, the following pseudocode may be used to identify the picture that must remain in the decoded buffer.

[0058] N=number of pic in DPB.

[0059] Mark N pics as "not used for reference"

[0060] For i=0.. NumSubPicBitstream

[0061] {

[0062] If Pic in active RPS => mark that picture as "as used for reference"

[0063] }

[0064] When all subpicture bitstreams are decoded for the current access unit, they are marked as "not used for reference" in the DPB and all already output pictures are removed.

[0065] However, compared to the method of rewriting all RPS information, this method requires more memory for the decoded picture buffer.

[0066] In an alternative implementation of the embodiment of FIG. 4 regarding the subpicture DPB removal mode, the first process is performed at the start of each picture, i.e., when processing the first slice / tile or subpicture, by marking all pictures as "not used for reference," whereas for each parsed slice / tile / subpicture, another process is performed to mark the picture marked on a slice / tile / subpicture unit as "used for reference." Reference picture dumping according to the final result, i.e., the last slice / tile / subpicture at the time of parsing / decoding, is performed after the reference picture has already been output and all slices / tiles / subpictures of the current picture have been delivered / received.

[0067] FIG. 5 corresponds to a case where the content is encoded as a motion-constrained tile, i.e., MCTS, or more generally, as a subpicture coded independently from a spatially offset subpicture or a subpicture belonging to a different subvideo, wherein the subpicture includes not only that of the same picture but also that of a different picture. In such a case, it is possible to mark a specific area of ​​the subpicture using a subpicture identifier so that it can be removed from the decoded picture buffer. A bitstream encoded with MCTS can refer only to itself.

[0068] In FIG. 5, the possibility of individually marking the subpicture area of ​​the picture (18) was used. If the subpicture is marked as “not for reference” and the picture has already been output, the area can be removed from the decoded picture buffer. Thus, it is possible to execute a process of individually marking each subpicture while parsing the NAL unit type and RPS from the slice / tile group or subpicture header without having to wait until all NAL units of the access unit have been parsed.

[0069] This is a multi-subpicture DPB (48) based on information from some set of parameters transmitted to the data stream, for example, according to FIG. 5. iThis is done by initializing ) or by reallocating the subpicture DPB, or by allocating DPB space for each newly occurring subpicture identifier (for each new subvideo). After initialization, the picture (meaning reconstructed samples for prediction, such as motion vectors, and corresponding referenceable data) is subpicture identifier A i Subpicture DPB (48) corresponding to i It will be stored in ). For example, subpicture identifier A i can be a single tile group ID.

[0070] Then, a reference picture set, that is, a list representing the pictures to be retained for prediction in the picture buffer (48) decoded with their corresponding IDs, is parsed and will affect whether the subpictures of the corresponding subpicture DPB can be removed. Obviously, in some scenarios, the tile settings within a picture may change. For example, in a subpicture bitstream merging scenario, it may happen that subpictures with IDs 0 and 1 are merged together, and later subpictures with IDs 1 and 2 are merged together. In such cases, the subpicture DPB is flushed based on the parameter set instructions. Whenever the activated parameter set does not contain the previous subpicture ID, the subpicture DPB corresponding to that subpicture is flushed. Obviously, picture removal is performed only when the picture is not for output, that is, when it has already been output or is no longer needed for output.

[0071] There are a few notes regarding FIGS. 4 and 5. For example, in addition to what is described above with respect to FIG. 5, the management of subpicture DPBs may be changed to such an extent that the number of subpicture DPBs managed simultaneously exceeds the number of subpictures (24) of the pictures (18) of the video (16). For example, for each subpicture identifier (80) encountered in the storage of the substream (32) or, more precisely, in the data stream (10), the decoder (22) may provide a separate subpicture (DPB).

[0072] Additionally, it should be noted that the description of FIG. 5 focuses on subpicture granularity processing for emptying the decoded picture buffer. However, while FIG. 5 describes a decoder that can operate accordingly, FIG. 5 is also illustrated as a description of a decoder that can operate in the manner shown in FIG. 5, that is, in a picture unit processing mode, such as responding to the indication (44) or some equivalent signaling shown in FIG. 2, as well as in the subpicture processing mode. In that case, the decoder (22) of FIG. 5 can manage the picture buffer (48) decoded in picture units using a dotted line in the manner shown in FIG. 4.

[0073] Regarding the buffer description (40') mentioned above, it should be noted that they may represent reference pictures that must remain in the DPB in the form of a list, such as in the form of analysis terms representing corresponding reference pictures, as in POC, etc. The buffer description (40') may positively cite these pictures or cite the pictures to be removed from the DPB.

[0074] Additionally, although FIGS. 4 and 5 focus on emptying the decoded picture buffer, it is evident that the decoder (22) in these figures fills the decoded picture buffer using the picture content just decoded. For example, in the case of FIG. 4, filling the decoded picture buffer (48) can be performed on a picture-by-picture basis, in the case of subpicture processing, or in both cases, in both subpicture processing and picture-by-picture processing. In FIG. 5, which focuses on emptying the subpicture-by-picture DBP, filling can be performed on a subpicture basis and is inserted into the corresponding subpicture DPB as soon as a specific subpicture is decoded. Naturally, an alternative approach may be to fill the decoded picture buffer (48) on a picture-by-picture basis, that is, the decoded subpicture (24) of the currently decoded picture (18) is inserted into their corresponding subpicture DPB simultaneously with the completion of decoding the current picture (18). In the case of the picture unit DPB empty mode, the decoder of FIG. 5 can operate as if it were in the corresponding mode of FIG. 4 (i.e., shown by the dashed line in FIG. 4).

[0075] Additionally, regarding the processing of decoded picture buffers, there are a few brief notes regarding the aforementioned embodiments. In particular, in cases where a subpicture belongs to the same subvideo but is independent of another subpicture—that is, another subpicture of the same picture (18) and a subpicture of another picture (18) belonging to a different subvideo—it has been noted so far that the subpicture (24) of the picture (18) of the video (16) can be coded like a subvideo in such a way that the subpicture can depend on another subpicture of another picture. Such a subpicture belonging to the same subvideo can change the position of the subpicture within the picture (18) of the video (16) in time. FIG. 6 illustrates this case. Various possibilities exist to handle this situation. In FIG. 6, two pictures (18) of the video (16) are illustrated as examples. In one picture (18), there is a subpicture (24) belonging to a sub-video having subpicture IDs 2, 7, 3, 4, 5, and 6. In a subsequent picture, it can be seen that the subpicture (24) of the sub-video having subpicture IDs 2, 7, 4, and 5 still exists, but is in a different subpicture location. Instead of the subpicture (24) having subpicture IDs 3 and 6, two new subpictures, or more precisely, subpictures of new subpicture IDs 9 and 12, exist in the latter image.

[0076] Now, it is assumed that the subpicture (24) of the latter picture (18b), which belongs to the subpicture identifier occurring in the previous picture (18a), is not coded in a random access point manner and additionally references the next subpicture (18a). That is, the correct subpicture (24) of the picture (18a) exists in the decoded picture buffer (48) of the decoder (28) when decoding the picture (18b), but in order to correctly perform the prediction for the decoding of the subpictures (24) of the picture (18b) of these subpicture identifiers, measures must be taken to perform the reference, i.e., the motion compensation prediction.

[0077] One possibility for solving this problem is for the synthesizer / merger (14) to take note of such a situation. This can be done using the technique set forth in WO 2016 / 026526, according to which the unoutput picture (18c) is inserted into the data stream (10) by the synthesizer / merger (14), which rearranges the picture contents of this picture (18a) using a constant motion vector field, that is, a constant motion vector field within each subpicture position, so that the subpicture (24) still needed in the subsequent picture (18b) is arranged in their new subpicture positions assumed in this subsequent picture (18b). Additionally, the synthesizer / merger (14) is redirected to declare that, for this picture (18b), the picture (18c) is referenced as the reference picture instead of the picture (18a) being referenced as the reference picture. Additionally, motion vector prediction for the sub-picture (24) of the picture (18b) belonging to the sub-video that was directed by the non-output picture (18c), which was present in the picture (18a), must be prohibited.

[0078] Another possibility is that as the subpictures change their subpicture positions, such as the subpictures of picture (18b) having subpicture IDs 2, 7, 4, and 5 in FIG. 6, the decoder (22) rearranges its own picture in the decoded picture buffer to create a new rearranged reference picture. According to this procedure, the reference picture in the decoded picture buffer is spatially divided into its subpictures, and the subpictures will be re-scrambled. As described above in relation to FIG. 1, this processing may also include spatial resampling if a specific subpicture (24) of picture (18b) is indicated to belong to the same picture content, but the spatial resolution is reduced compared to the subpictures of the previous picture, such as picture (18a). In this way, the transition from the subpicture layout of picture (18a) to the subpicture layout of the new picture (18b) can be considered without the insertion of the output picture (18c).

[0079] And another alternative may be for the decoder (22) to redirect the motion vector used to code the subpicture of the picture (18b). The decoder (22) may generate a motion vector offset for the motion vector of the subpicture redirected in the picture (18b) to compensate for the subpicture position change relative to the reference picture (18a). In motion compensation prediction related to decoding the subpicture (24) of the picture (18b), the motion vector offset is added to the coded motion vector of the substream belonging to the subpicture (24) of the picture (18b), more precisely, the substream in which that subpicture also exists in the picture (18a). By this, the correct predictor of the inter-prediction block of the sub-picture of the picture (18a) is derived, which indicates the correct location in the reference picture (18a), that is, the correct location within the sub-picture belonging to the sub-video shown in both pictures (18a and 18b), but is located in a different sub-picture location.

[0080] The redirection of subpictures in the decoded picture buffer in relation to the reference picture and the motion vector redirection enabled by the decoder (22) may be triggered by the observed change in the subpicture identifier for the individual subpicture positions within the picture (18) of the video (16), which describes the redirection of the reference picture's subpictures and their subpictures in the data stream (10) accordingly. Alternatively, this process may be triggered by an explicit data stream indication of the data stream (10) written to the data stream by the synthesizer / merger (14), as in, for example, within the PPS.

[0081] The embodiment described below in relation to FIG. 7 is intended to describe a video codec environment that mitigates the application described above in relation to FIG. 1 in terms of picture order count (POC) processing. Accordingly, FIG. 7 describes possible functions of a decoder for processing a video-encoded data stream, wherein a picture (18) is subdivided into subpictures (24), and the subpictures (24) are coded as parts of separate sub-videos (28) as described in relation to FIG. 5 and the drawings above. The functions described in relation to FIG. 7 may be specific modes of the decoder (22) triggered by a subpicture indication, such as the indication (44) described in relation to FIG. 2, for example. In particular, FIG. 7 allows different POC values ​​to occur even within a single picture (18) by being transmitted in the data stream (10) for the subpictures (24) of the picture (18). If the function of FIG. 7 represents only one mode of the decoder (22), the other mode may be one in which only one POC value is allowed for one picture (18).

[0082] For example, if the data stream (10) is configured based on the storage of the substream (32) as described above in relation to FIG. 1, the picture order count development of the subpicture (24) within the corresponding subvideo (28) in the display time order (20) may differ. For example, this difference may result from different frame rates between the subvideos (24). The video (16) will naturally have the minimum frame rate of the subvideo (28) contributing to the video (16). Therefore, the corresponding picture of the higher frame rate subvideo (28) will be skipped by the synthesizer / merger (14) when configuring the data stream (10). In encoding such a higher frame rate subvideo (28) into its corresponding substream (32), the encoding process must naturally be noted that this picture is not needed as a reference picture for other P or B pictures that must contribute to the synthesis of the video (16) together with the subpicture (24) of the lower frame rate subvideo (28). Additionally or alternatively, this difference in the temporal development of the picture order count value of the subpicture (24) in each of the various subvideos (28) and the corresponding substream (32) may result from different GOP structures underlying these substreams (32). For example, different GOP lengths, i.e., different numbers of subpictures (24) per GOP, may be applied to different substreams (32). Naturally, the generation / encoding process for generating the substream (32) is performed in such a way that the video contributing to the synthesis of the picture (18) of the video (16) in terms of display time order (20) and coding / decoding order (58) matches in a sequential order within each subvideo (28) and substream (32), at least with respect to the subpicture (24) of the corresponding subvideo (28).

[0083] Accordingly, FIG. 7 shows one or more NAL units (34) having subpictures (24) of subvideos (28) selected by a synthesizer / merger (14) to be used for participating in corresponding access. A unit (46) of the data stream (10) is provided with a POC value (90) by combining these. The POC value arranges the subpictures (24) within each subvideo (28) in display time order (20), for example, based on a GOP (picture group). That is, the GOP value (90) within the NAL unit (34) can be reordered whenever a new GOP is started. Preferably, the concept of FIG. 7 allows the synthesizer / merger (14) to simply take the POC value (90) without modification when combining the NAL units (34) of the contributing substream (32), so that a corresponding NAL unit (34') is resulting in an access unit (46) of the data stream (10). By hatching, FIG. 7 illustrates subpictures (24) of various subvideos (28) and their corresponding NAL units (34) within corresponding substreams (32) corresponding to one common time moment and one common coding / decoding order in terms of coding / decoding order (58) to form corresponding access units (46) in stream (10). To explain the difference in POC values ​​(90) between NAL units (34') corresponding to different subpictures of the currently decoded picture (18) or the currently decoded access unit (46), the decoder (22) operates as follows. In particular, the decoder (22) decodes the POC values ​​(90) contained in the NAL units (34') from the data stream for each subpicture (24) of the currently decoded picture (18).Additionally, the decoder (22) is derived from one or more inter-subpicture POC compensation parameters (92) that must parameterize a POC correction function (94) which produces the same final POC value (96) for each picture (18) of the video (16) when the picture (24) of the currently decoded picture (18) is applied to the POC value (90) of the corresponding subvideo (28). According to one example, the POC value (90) carried by the NAL unit (34') within one access unit (46) of the data stream may differ due to different GOP lengths used in the coding of the subvideo to which the subpicture (24) coded by this NAL unit (34') belongs. In that case, unless the contributing sub-video (28) and sub-stream (32) are changed during synthesis / merging, the mutual offset between POC values ​​(90) within one access unit (46) tends to be constant across consecutive access units (46) and data streams (10) unless any of the 32 GOPs of the contributing sub-stream start a new GOP. The inter-subpicture POC compensation parameter (92) may include, for example, a POC compensation offset added by a function (94) to the POC value (90) of the corresponding sub-video to account for the difference between the various sub-streams (32) of the IRAP sub-picture (24) within the sub-video (28), that is, the current sub-picture (24) of the sub-video (28) corresponding to the most recent start of the GOP.Whenever a new sub-video (28) begins to contribute to a picture (18) of the video (16), or whenever one of the running sub-videos (28) starts a new GOP by having a sub-picture that signals the information (40) RAP, the decoder (22) updates, calculates, or resets the POC compensation offset of that sub-video or substream (32) based on a comparison, that is, by calculating the difference, and at this time any of the final POC values ​​(96) of any sub-video (28) or substream (32) are still maintained in the video (16) at that point in time, that is, in the same access unit (46), and any access point is coded. Due to the frame rate difference between the substreams (32), the POC compensation parameter (92) between sub-pictures may also include a factor that is scaled using a function (94) so ​​that the POC value (90) yields the final POC value (96). Due to these frame rate differences, the mutual difference between POC values ​​(90) within the access unit (46) of the data stream (10) tends to be related by the mutual difference between POC values ​​(90) that are offset compensated using the aforementioned POC compensation offset and then scaled temporally according to the POC step size difference. This scaled difference is compensated by inversely scaling these POC values. The decoder (22) can derive a scaling factor, such as a part of the parameter (92), from the data stream (10) that may have been written to the data stream (10) by the synthesizer / merger (14), based on the evaluation of the frame rate under each of the sub-video (28) and the corresponding sub-stream (32).

[0084] Therefore, to summarize briefly, the decoder (22) can manage the POC compensation parameter (92) to parameterize the function (94). Thus, the parameterized function (94) yields a final POC value (96) when applied to the POC value (90) corresponding to a specific subpicture (24) of the current picture (18). The final POC value (96) obtained in this way is the same for each picture (18) of the video (16). If a non-RAP subpicture (24) exists in the picture (18) or access unit (46), respectively, the offset between the parameters (92) can be updated by the decoder (22) in the random access point subpicture (24) in the manner described above. If all subpictures (24) of the current picture (18) are random access point coded, the decoder (22) can set the offset to the same arbitrary default value for the subpicture (24). Additionally or alternatively, a RAP subpicture can be set to the same default POC value itself, for example, by appropriately parameterizing the encoder (30). This value may be 0. The decoder does not need to do anything with this encoder constraint. However, the encoder (30) may be free to set the POC of a RAP subpicture (24) of a picture (18) to a different value, and in such case, the decoder may be configured to set the POC value of this RAP subpicture to a default POC value such as 0 or to the same value as the POC value (90) signaled for the first RAP subpicture encountered. The scaling factor between the parameters (92) may be determined by the decoder (22) whenever a subvideo (28) newly contributes to the composition of the video (16).

[0085] As further explained below, the decoder (22) may need to ensure that the motion vector predictor scaling performed by the decoder (22) is correctly performed in order to retain the subpicture (24) from the corresponding NAL unit (34'). Motion predictor scaling is completed when the motion vector predictor with respect to reference picture A, having a POC distance a for the picture containing the block in which each motion vector predictor is used, is scaled to refer to some required POC distance b for the currently predicted block of the current picture. Since scaling must be performed in relation to the POC distance associated with the POC definition of each substream (32), this scaling may still need to be performed by the decoder (22) based on the POC value (90) of the individual substream. According to HEVC motion vector predictor scaling, for example, if the frame rate scale factor between the parameters (92) is not a power of 2, a different motion vector scaling may result.

[0086] As already mentioned above, the embodiments described above can be combined with one another. This is true not only by combining each of the embodiments described for FIGS. 3, 4, 5, and 7 with the embodiment of FIG. 2, but also with combining them two, three, or all together with the embodiments described for FIGS. 3, 4, 5, and 7. Each of these concepts relaxes the use of the resulting video codec concept when performing the video synthesis / merging operation described in relation to FIG. 1. Consequently, the example outlined in FIG. 1 should not be treated as being limited to the video coding codec examples described in the embodiments above.

[0087] Before beginning the description of additional embodiments of the present application related to a slightly different subject, namely, additional aspects of the present application related to progressive decoding refresh, the embodiment of FIG. 7 will be described again. In particular, as already described above, the embodiment of FIG. 7 relates to POC delta induction. As described, subpictures (24), each subpicture may have, for example, a single tile or a set of tiles (corresponding to the joint bitstream (10)) that are not aligned, random access points or different reference structures or GOP structures. As outlined above, such use cases may occur, for example, when bitstreams (32) that were initially encoded separately are merged together into a single datastream (10).

[0088] In this case, the data streams (10) related to or merged at a single common time moment contribute to a single common access unit (46), but the picture order count (POC) displayed within the sub-bitstream (32) of each subpicture (24) may differ. In this case, this results in the burden of having to rewrite the POC value signaled within the bitstream (32), i.e., within the NAL unit (34), by the synthesizer / merger, so that it can be merged into a resulting data stream to generate a single POC value for a single access unit. The embodiment of FIG. 7 reduces this burden and enables the merging of these subpicture substreams (32) without adjusting the POC value coded into the header of each subpicture (24), i.e., the tile group header, which is a NAL unit.

[0089] The POC is used to derive the access unit boundary. That is, generally, the POC is used to associate a slice or tile group or NAL unit with the access unit (46). This may be an alternative operation of the decoder (22) of FIG. 7 when combined with the embodiment of FIG. 2: the concept described in FIG. 7 is applied by the decoder (22) in the case of the indication (44) representing the subpicture particle size, whereas the decoder (22) will expect each access unit (46) to have only one identical POC value within the access unit (46) in the case of the indication (44) representing the picture particle size. The decoder (22) will identify the start of the next access unit (46) in response to a change in the POC value from one NAL unit to another in the case of the picture particle size. For the subpicture granularity function described in FIG. 7, the identification of an access unit interface separating consecutive access units (46) from a data stream (10) can be performed by a decoder (22), for example, based on an access unit separator inserted into the data stream (10) by a synthesizer / merger (14).

[0090] During the various processes in which the POC value is used during the decoding process—specifically, to determine whether the short-term or long-term reference picture is used for reference—there is a picture output; that is, for picture reference, the pictures are output in increments or sequences of the POC.

[0091] The embodiment described in relation to FIG. 7 allows different signaled values ​​of the POC in each NAL unit (34') of the subpicture (24) within the same access unit (46), while still allowing for the accurate derivation of the POC value (96) finally used for the aforementioned operations on picture output and picture reference when multiple subpictures (24) are decoded from the same data stream (10). A random access point generally leads to a reset of the POC value, at which the decoder starts the decoding process at the beginning of the bitstream. When the decoder finds an IDR, it generally assigns a POC value of 0 to the corresponding picture and derives the POC value of the next picture based on this. The next access unit or NAL unit within the access unit contains information such as (tile group) header bits that signal the POC of the access unit to which they belong. Typically, the POC is signaled by the LSB bit in the header of the slice / tile group, and the MSB bit is derived by the decoder. When applied to the scenario summarized in Figure 7, this process will lead to incorrect derivation of access unit boundaries, reference pictures, and output order if the subpictures of the access unit contain different POC LSB values ​​or different POC LSB lengths within a single access unit.

[0092] The embodiment described in relation to FIG. 7 used a subpicture-specific deltaPOC calculation to track the difference between different subpicture bitstreams (32). For example, when the decoding process of the decoder (22) starts at a regular full picture IRAP, all NAL units (34') of the access unit (46) carry the same POC value (NAL POC LSB). The POC MSB is set to 0 in this instance and the newly introduced deltaPOC. Now, during the CVS process, a subpicture RAP may occur and carry a different NAL POC LSB, for example, a value of 0, while other NAL POC LSBs of the access unit remain unchanged (not equal to 0). Whenever a subpicture RAP is recognized as described above or through state-of-the-art signaling such as a NAL unit type or SEI message, a deltaPOC is derived for this subpicture as the difference between the subpicture IRAP NALU POC LSB of the access unit and other NALU POC LSBs. The entire picture POC (96) is derived for all NALUs as the basis for the above functions, such as picture output and picture reference. When processing subpicture IRAP NALUs, the entire picture POC (96) incorporates a non-zero deltaPOC, whereas for the remaining non-subpicture IRAP NALUs, each zero deltaPOC is incorporated into the calculation of the entire picture POC (96), resulting in the entire picture POC value 96 for all NALUs for the access unit (46). FIG. 8 illustrates, for example, a POC compensation according to FIG. 7, wherein four consecutive pictures (18a to 18d) of a video (16) are shown, each subdivided into three subpictures (24), wherein the first picture (18a) is an IRAP picture, that is, all subpictures (24) are random access points, whereas the second picture (18b) and the fourth picture (18d) have all subpictures (24) non-RAP coded.The third picture (18c) is coded in a random access point manner, with one subpicture (24), i.e., the upper picture of FIG. 8, being coded in this manner, while the other is not coded in this manner. As can be seen, FIG. 8 illustrates a simple example of a function (94), namely, an addition having a second add-end of the sum recorded in the individual subpicture (24), which is a constant representing the POC compensation parameter. The first add-end represents the signaled POC value, i.e., the input to the parameterizable function, and the resulting sum corresponds to the final POC value (96). As illustrated in FIG. 8, the decoder obtains the POC compensation offset by subtracting (98) the final POC value for one of the non-RAP subpictures (24) of the picture (18c) along with the transmitted POC value of the RAP subpicture (24).

[0093] The introduction of the described whole picture POC method can be gated via SPS / PPS level flags. Alternatively, instead of the implicit derivation of deltaPOC values, i.e., POC compensation offset, a set of parameters within the data stream (10) indicates the deltaPOC to be subtracted / added to each subpicture POC value (90) via the function (94), so that the resulting whole picture POC (96) is aligned with the currently decoded picture (18) or each subpicture (24) of the current access unit (46) of the data stream (10). If a subpicture IRAP NALU is present in the access unit (46), additional constraints may be required to enable the derivation of the whole picture POC.

[0094] For example, this may be a requirement of the data stream (10), namely, a requirement of bitstream conformity in which at least one subpicture non-IRAP NALU exists in the access unit (46), that is, at least one subpicture (24) of each picture (18) is coded in a non-RAP manner to allow the derivation of the current entire picture POC (96) according to the deltaPOC of all subpicture IRAP NALUs, i.e., the POC compensation offset, which must be derived. In other words, when a sub-RAP is signaled as a new NAL unit type, a flag is used to not allow other NAL unit types. In such a case, such signaling may be used only when one of the NAL units does not indicate that the NAL unit is a sub-RAP.

[0095] Another solution may be as follows. When all subpictures of access (46) are recognized as subpicture IRAPs, the POC MSB is reset and a deltaPOC per subpicture (24) is calculated so that all subpictures result in the same overall picture POC. For example, in this case, the first subpicture deltaPOC is set to 0 and all other subpicture deltaPOCs are set accordingly so that they result in the overall picture POC (96) for the first subpicture IRAP NALU of the access unit.

[0096] Additionally, as described above, it is possible for the substreams (32) participating in configuring the data stream (10) to have different POC step sizes. For example, this may occur in a meeting scenario. One stream may be coded at 30 FPS and the other substream (32) may be coded at 60 FPS. In this case, as described above, merging to the lowest common frame rate, namely 30 FPS, is one option. For this purpose, the highest time layer of the 60 FPS substream (32) may be dropped, thereby reducing the frame rate to 30 FPS. However, since the 60 FPS encoder must accommodate a larger amount of frames using a relatively larger POC step size compared to the 30 FPS encoder, the POC step sizes between temporally adjacent pictures in the two streams differ. In the scenario described above, where the streams must be merged together into a common bitstream, care must be taken to ensure the correct POC induction. When the POCs of the subpicture bitstreams have different step sizes, the embodiment of FIG. 7 allows for scaling of the POC values ​​(90) of individual subpictures (24) within the coded video sequence (10). For example, the synthesizer / merger (14) may be notified of or self-induced by the difference in POC step sizes from the input subpicture bitstream (32) and writes each POC scaling factor to a parameter set for all subpicture bitstreams contributing to the configuration of the data stream (10) to align all the whole picture POCs.

[0097] While it was additionally noted above that the scaled full picture POC is used for reference picture derivation and output order, motion vector scaling may need to be performed based on the difference from the unscaled POC. By doing so, the correct motion vector scaling expected by the initial encoder is performed.

[0098] Instead of tracking all POC values ​​(90) of individual subpictures (24), i.e., unscaled POCs, on the decoder side, the decoder can derive a difference in POC values ​​used on the encoder side to perform motion vector candidate scaling from scaled POC values ​​by using each POC scaling factor when deriving the POC difference for motion vector scaling.

[0099] Additionally, although it has been described that the POC offset value among the parameters (92) is automatically determined by the decoder, alternatively or additionally, the POC offset value can be inserted into the data stream (10) by the synthesizer / merger (14) as described above in relation to the POC scaling factor.

[0100] With reference to FIG. 9, the present application describes an example of a video codec that allows for progressive decoding refresh. FIG. 9 illustrates the concept on which this embodiment is based.

[0101] In particular, FIG. 9 illustrates a sequence of pictures (100) of a video (102), namely four sequences of pictures aligned along a coding / decoding order (58). FIG. 9 illustrates an encoder that encodes the video (102) into a data stream (106) and a decoder (108) that decodes or reconstructs the video (102) from the data stream (106). The concept is as follows: To enable progressive decoding refresh, the video decoder (108) records, for each reference picture of the video (102), a subdivision into a refreshed picture area and a non-refreshed picture area of ​​each reference picture. In particular, the reference picture is a picture prior to the currently decoded picture in the decoding order (58) that can be used as a predictive reference for the currently decoded picture or any next picture. The refreshed picture and the just-mentioned non-refreshed picture area will become clear from the following description. In particular, the decoder (108) decodes information about the refresh update area (110) illustrated in cross-hatching in FIG. 9 from the data stream (106) for a specific picture (100a). In the example of FIG. 9, this includes the left third of the picture (100a). The encoder encodes, for example, each picture (100a-100d) into respective access units (112a-112d) and signals information (114) for the picture (100a) regarding the location of the refresh update area (110) within the picture (100a) to the access unit (112a). The indication of the refresh update area (114) may initiate a progressive decoding refresh that causes special measures for coding and decoding picture (100a) and some subsequent pictures to complete the progressive decoding refresh at both the encoder (104) and the decoder (108). This is explained in more detail below.

[0102] In particular, the region (110) located by the information (114) represents a region encoded by intra-coding only by the encoder (104), that is, coded in a random access point manner, and the decoder (108) decodes the refresh update region (110) accordingly, that is, using intra-prediction. The coding of other regions of the picture (100a), i.e., the remaining region (116), is no longer restricted and can be encoded by the encoder (104) and accordingly can be decoded by the decoding (108) using intra-prediction and / or inter-prediction based on the previous picture.

[0103] For the next picture (100b) in the coding / decoding sequence (58), the progressive decoding refresh has not yet been completed. The encoder (104) encodes this picture (100b) into the corresponding access unit (112b) and internally signals information (114) for the refresh update area (110) within the picture (100b). In the example of FIG. 9, the refresh update area of ​​the picture (100b) covers the horizontal middle third of the picture (100b) and is therefore spatially adjacent to the refresh update area (110) of the previous picture (100a). While the information (114) within the access unit (112a) initiates the progressive decoding refresh, the information (114) for the picture (112b) indicates a continuation of this initiated progressive decoding refresh. The area (110) of the picture (100b) is again encoded by the encoder (104) by intra-prediction only and decoded by the decoder (108) accordingly. However, as part of the initiated progressive decoding refresh, the decoder (108) and the encoder (104) perform the following operations: First, the encoder (104) and the decoder (108) perform the aforementioned logging of the refreshed picture area of ​​the reference picture. As the progressive decoding refresh is initiated in the picture (100a) and the picture (100a) forms the reference picture for the subsequent picture (100b), the refresh update area (110) of the picture (100a) represents the refreshed picture area of ​​the picture (100a), while the other area (116) is the unrefreshed picture area. In FIG. 9, the updated picture area is shown as being surrounded by a continuous line (118). Secondly, the decoder (108) and the encoder (104) determine a first area (120) to be coded and decoded independently of the refreshed updated area (110) of the same picture (100b) and the unrefreshed picture area of ​​the reference picture (here, picture (100a)).The encoder (104) may code an explicit signaling of where the region (120) is located, or more preferably, the encoder (104) and the decoder (108) determine the region (120) as the refreshed picture region (118) of the picture (100a), that is, the region of the picture (100b) that is juxtaposed with the immediately preceding picture in the coding / decoding order (58). To distinguish the region (120) from the refreshed updated region (110), the refreshed updated region (110) is illustrated in FIG. 9 using cross-hatching, whereas simple hatching is used for the region (120).

[0104] Both the encoder and the decoder take special measures to encode / decode the region (120) to / from the corresponding access unit (112b). In particular, the encoder (104) does not restrict the signalable domain of the coding parameters on which the prediction of the region (120) is derived by the encoder and the decoder, for example. In other words, the encoder (104) uses coding parameters that determine the prediction of these pictures from previous pictures, such as using motion compensation prediction, and uses prediction residual information that describes the prediction residuals to which this prediction is corrected to encode the picture to the corresponding access unit. The coding parameters and prediction residual information are coded to the access unit to describe the corresponding picture. However, within the refresh update region (110), the encoder (104) uses intra prediction exclusively, that is, the prediction of other pictures is not used. For region (120), the encoder (104) uses inter-prediction and, for example, does not restrict the signalable domain of the coding parameters, but other methods of deriving predictions for region (120) based on the signaled coding parameters are used by the encoder (104) and decoder (108) compared with other regions such as region (116). In particular, the location of the refresh picture region (110) of each reference picture, here, picture (100a), is considered. For example, the encoder (104) may freely encode into the data stream (106) a motion vector for region (120) that would result in the region (120) having a dependency on region (116) if the prediction derivation is not changed, that is, a motion vector that actually points to the part of the reference picture (100a) reaching region (116).Similarly, the encoder (104) may use motion vector prediction to code motion vectors for the region (120), but in constructing a corresponding motion vector prediction list of motion vector predictor candidates formed by motion vectors used to code the picture (100a), the encoder (104) exclusively fills this motion vector predictor candidate list using motion vectors that are exclusively used for blocks within the refreshed picture region (110). The possibility that the encoder (104) can use a signalable state that is redirected as a result of inducing a prediction for the region (120) that would actually have been signalable by a different state of coding parameters for the region (120) provides the possibility of increasing compression efficiency for the encoder (104) because, for example, this coding parameter state that induces a prediction relying only on the updated picture region (110) which is sometimes redirected to a different state by the encoder (104) and decoder (108) later may be encodingable using fewer bits than the coding parameter state that immediately describes this prediction without redirection by the encoder (104) and decoder (108). That is, the encoder (104) and decoder (108) can clip the motion vector that is actually signaled in the data stream (106) for the region (120) so as not to reach beyond the boundary of the refresh picture region (110) of the reference picture (100a), for example. Alternatively or additionally, the encoder and decoder (108) may use padding for the portion of the reference picture (108) indicated by the motion vector signaled in the data stream (106) for the area (120) that extends beyond the boundary of the refreshed picture area (110). Alternatively or additionally, the construction of the motion vector candidate list performed by the encoder (104) and decoder (108) may be performed in such a way that motion vectors are excluded from filling such a list outside the refreshed picture area (110) within the reference picture (100a).The encoder (104) may, for example, signal selected motion vector predictors to the decoder (108) by signaling within the data stream (106) and index them into a list of such motion vector candidates that are interpreted accordingly. The remaining area of ​​the picture (100b), that is, in the case of FIG. 9, the right third of the picture (100b), represents an area coded by the encoder (104) that includes inter-predictions without any restriction on the first area (120) and the refresh update area (110) of this picture (100b), and the area (116) separated from the refresh picture area (110) of the picture (100a). That is, the area (116) of the picture (100b) is an inter-predicted area without any restriction. The subsequently coded / decoded picture (100c) represents the completion of the progressive decoding refresh that started from the picture (100a) in the example of FIG. 9. The encoder (104) codes information (114) for this picture (100c) into the corresponding access unit (112c) indicating the location of the refresh update area (110) of the picture (100c) that covers the right third of the picture (100c) in the case of FIG. 9. That is, FIG. 9 illustrates that the refresh update area (110) of the successive pictures (100a to 100c), starting from the picture (100a) that represents the start of the progressive decoding refresh, assumes mutually separated parts of the picture area of ​​the picture of the video (102). This picture (110) is then intra-coded by the encoder (104) and decoded by the decoder (108) accordingly.

[0105] In the case of picture (100c), picture (100b) also represents a reference picture. The refreshed picture area (118) is determined by the encoder and decoder as the union of the first area (120) of picture (100b) and the refresh update area (110). This means that, in turn, during progressive decoding refresh, the refreshed picture area (118) is continuously expanded. Again, the encoder (104) and decoder (108) determine the first area (120) of picture (100c). For example, it is made to be an area that spatially overlaps with the refreshed picture area (110) of the immediately preceding picture, i.e., picture (100b). This region (120) is coded / decoded as previously described in relation to the region (120) of the picture (100b), the difference being that the region (120) has two reference picture candidates, namely, a picture (100a) having a refreshed picture region (118) and a picture (100b) having its refreshed picture region (118). Inter-prediction is used for the region (120), that is, a coding mode allowed in addition to the inter-prediction mode, but the coding parameters for the region (120) of the picture (100c) are redirected to the state so that the resulting prediction does not cause any dependency of the coding / decoding of the region (120) from the unrefreshed region (116) of the pictures (100a and 100b).

[0106] Continuing from picture (100c), the progressive decoding refresh initiated by the encoder (104) in picture (100a) is completed, and the refreshed picture area (118) covers the complete picture that has been the picture of the video (102) since that picture (100c). Intermittently or periodically, the encoder (104) initiates another progressive decoding refresh so that the refreshed picture area collapses to correspond to the refresh update area (110) of the first initiated picture of the subsequent progressive decoding refresh.

[0107] The progressive decoding refresh described in relation to FIG. 9 can be described as subpicture intra-refresh padding. It can be implemented using tiles or groups of tiles, or without using tiles. In the embodiment of FIG. 9, a portion of the picture, i.e., the region (110), which may be called a subpicture, is refreshed by applying intracoding, and the next picture is intercoded and inter-decoded in a limited manner. According to the embodiment of FIG. 9, the picture is subdivided into columns, here exemplarily N = 3 columns. However, it is clear that the number of columns N can be chosen differently. And this implies that even other forms of growth of the refreshed picture region (118) during this progressive decoding refresh can be chosen, such as growth in blocks rather than columns of the picture. As described in relation to FIG. 9, in the first subpicture RAP, i.e., the picture in access unit 0, in the case of FIG. 9, picture (100a), the first column, colldx=0, is encoded only within the intra-block.

[0108] In the second picture (100b) of the next access unit AU=1, i.e., the encoding / decoding sequence (58) of FIG. 9, the second column, i.e., colldx=1, is encoded only as an intra-block, and the first column is encoded as an inter-coded block that can refer only to samples decoded from the previous picture in the column colldx=0. However, instead of limiting the motion vectors that the encoder can encode in the data stream, the embodiment of FIG. 9 can also freely encode motion vectors that would actually cause dependencies on other regions when column colldx=0. In particular, according to the embodiment of FIG. 9, an intra-refresh region, i.e., the first column in this example, is indicated, and padding such as orthogonal extrapolation of the outermost pixel line is used in the reference block of the inter-coded block that exceeds column colldx=0 of the picture of access unit 1 when the motion vector points to it. Motion vector clipping based on the indication of the area may be applied alternatively or additionally to ensure that the motion vector of the refreshed area does not depend on the non-refreshed area. As the pictures are decoded sequentially, the refreshed area (118) increases and accordingly, the refreshed area (110) per picture is indicated.

[0109] An alternative to FIG. 9 is that the encoder (104) signals the successive locations of the update refresh regions (110) of the picture sequence forming the successive pictures (100a to 100c), i.e., the progressive coding refresh, by displaying the pattern of these regions (110). This pattern describes the refresh. Regions per sequence may also be displayed. Thus, this type of information may be signaled once for the picture (100a), i.e., the starting picture of the GDR sequence (100a-100c), thereby signaling the GDR start and the regions (110) for the pictures (100a-c). The former display may be one that can be displayed in the PPS. Alternatively, the GDR start display may be signaled for the picture (100a) as in the PPS. The location pattern of the region (110) in the GGR picture sequence starting from the GDR start picture may be something that is signaled at a higher level, such as in the SPS.

[0110] In relation to the above embodiments, the following is further noted. The above embodiments present advantageous video codec concepts. Many of them allow for advantageous video configurations as described above. However, it should be noted that the encoder can directly form any of the data streams (10) of the present invention described above, that is, without the configuration of pre-encoded substreams. In this operation, the encoder can utilize the advantageous characteristics of the data streams (10) existing above, such as the possibility of selecting RAP time instances and reference picture buffer descriptions in a sub-picture manner instead of a picture-whole manner.

[0111] Accordingly, the above embodiment also relates to a video decoder—and a corresponding video encoder and data stream—for decoding the video (16) of a picture (18) from a data stream (10) in which the video is encoded in such a way that the subpictures (24) in which the picture is spatially subdivided are encoded independently of each other. The video decoder is configured to decode subpicture-related information (40), such as a syntax element nal_unit_type, from the data stream for each subpicture of each picture of a set (42) of one or more pictures of the video, such that for each subpicture, each subpicture is identified as being coded in a random access point manner, such as when assuming a first state, such as when assuming an IDR_N_LP state, and each subpicture is identified as not being coded in a random access point manner, such when assuming a second state, such as when assuming a TRAIL_NUT state. The decoder decodes a random access point signaling granularity indicator (44) that can be named mixed_nalu_types_in_pic_flag from a data stream for a set (42) of one or more pictures such that the syntax element mixed_nalu_types_in_pic_flag is included in the PPS, wherein when assuming a first granularity or first state such as 0, for each picture of one or more picture sets, the subpicture-related information indicates that a common state such as IDR_N_LP is assumed for all subpictures of each picture, and when assuming a second granularity or second state such as 1, for each picture of one or more picture sets, the subpicture-related information indicates that a different state such as IDR_N_LP is assumed for at least one subpicture within the same picture and TRAIL_NUT is assumed for at least one other subpicture.In other words, mixed_nalu_types_in_pic_flag assuming a first state such as 0 may indicate that the value of nal_unit_type must be the same for all coded slice NAL units of the picture, and thus for all subpictures within a single picture. However, if mixed_nalu_types_in_pic_flag is assumed to be in a different state, for example, 1, this indicates that VCL NAL units of one or more subpictures of the picture all have a specific value of nal_unit_type that is one of the sets of NAL unit types consisting of one or more of STSA_NUT, RADL_NUT, RASL_NUT, IDR_W_RADL, IDR_N_LP, and CRA_NUT, whereas other VCL NAL units of the picture all have a different specific value of nal_unit_type, namely one of the additional sets consisting of one or more of TRAIL_NUT, RADL_NUT, and RASL_NUT. That is, mixed_nalu_types_in_pic_flag assuming a second state such as 1 can specify that each picture referencing a PPS containing mix_nalu_types_in_pic_flag has one or more VCL NAL units, that the VCL NAL units do not have the same value of nal_unit_type, and that the picture is not an IRAP picture, whereas mixed_nalu_types_in_pic_flag assuming another state such as 0 can indicate that each picture referencing that PPS has one or more VCL NAL units, and that the VCL NAL units of each picture referencing the PPS have the same value of nal_unit_type.By means of such means, mixed_nalu_types_in_pic_flag can be set to a first state such as 1 to indicate that the picture referencing the PPS, such as that resulting from the subpicture bitstream merging operation, contains slices having different NAL unit types.

[0112] Although some aspects are described in the context of a device, it is clear that these aspects represent a description of the method, whereby a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding device. Some or all of the method steps may be executed by (or using) a hardware device, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be executed by such a device.

[0113] Depending on specific implementation requirements, embodiments of the present invention may be implemented in hardware or software. Implementation may be performed using a digital storage medium, for example, a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, which stores electronically readable control signals that cooperate (or can cooperate) with a programmable computer system to perform each method.

[0114] Some embodiments according to the present invention include a data carrier having an electronically readable control signal capable of cooperating with a programmable computer system, so as to enable one of the methods described herein to be performed.

[0115] Generally, embodiments of the present invention may be implemented as a computer program product having program code, and the program code operates to perform one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, in a machine-readable carrier.

[0116] Another embodiment includes a computer program for performing one of the methods described herein stored in a machine-readable carrier.

[0117] That is, an embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

[0118] Accordingly, additional embodiments of the method of the present invention are a data carrier or digital storage medium, or a computer-readable medium, comprising a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is generally tangible and / or non-temporal.

[0119] Accordingly, an additional embodiment of the method of the present invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein.

[0120] A data stream or signal sequence can be configured to be transmitted, for example, through a data communication connection, for example, over the Internet.

[0121] Additional embodiments include processing means configured or adapted to perform one of the methods described herein, such as a computer or a programmable logic device.

[0122] Additional embodiments include a computer installed with a computer program for performing one of the methods described herein.

[0123] Further embodiments according to the present invention include a device or system configured to transmit (e.g., electronically or optically) a computer program to perform one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The device or system may include, for example, a file server for transmitting the computer program to the receiver.

[0124] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the method described herein. In some embodiments, the field-programmable gate array may work with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0125] The device described in this specification may be implemented using a hardware device, a computer, or a combination of a hardware device and a computer.

[0126] Any component of the device described in this specification or any component of the device described in this specification may be implemented at least partially in hardware and / or software.

[0127] The method described in this specification may be performed using a hardware device, a computer, or a combination of a hardware device and a computer.

Claims

Claim 1 A method for decoding pictures from a data stream, comprising: decoding a flag indicating whether, for a picture, the NAL unit type differs between at least two subpictures of the picture from the data stream; determining that the NAL unit type is the same for all subpictures of the picture when the flag is a first value; and decoding a first NAL unit type for a first subpicture of the picture and a second NAL unit type for a second subpicture of the picture when the flag is a second value different from the first value, wherein the first NAL unit type is different from the second NAL unit type. Claim 2 A method for decoding pictures from a data stream, wherein the first NAL unit type is random access and the second NAL unit type is not random access. Claim 3 A method for decoding pictures from a data stream, wherein, in claim 1, the flag is included in the picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS) of the data stream, and each NAL unit type is associated with a slice of a subpicture. Claim 4 A method for decoding pictures from a data stream according to claim 1, further comprising: storing at least one of the subpictures in a buffer while decoding the subpictures when the flag is the second value; and flushing the buffer after all of the subpictures of the picture have been parsed or decoded. Claim 5 A method for decoding pictures from a data stream according to claim 1, further comprising the step of distinguishing between two subsets of NAL unit types based on an integer value syntax element corresponding to the NAL unit type, wherein the first subset of the two subsets of NAL unit types corresponds to random access, and the second subset of the two subsets of NAL unit types corresponds to non-random access. Claim 6 A decoder configured to decode a flag indicating whether the NAL unit type differs between at least two sub-pictures of a picture from a data stream for a picture; to determine that the NAL unit type is the same for all sub-pictures of the picture when the flag is a first value; and to decode a first NAL unit type for a first sub-picture of the picture and a second NAL unit type for a second sub-picture of the picture when the flag is a second value different from the first value, wherein the first NAL unit type is different from the second NAL unit type. Claim 7 In claim 6, the first NAL unit type is random access, and the second NAL unit type is a decoder that is not random access. Claim 8 In claim 6, the flag is included in the picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS) of the data stream, and each NAL unit type is associated with a slice of a subpicture, a decoder. Claim 9 A decoder according to claim 6, wherein, when the flag is the second value, at least one of the subpictures is stored in a buffer while decoding the subpictures; and further configured to flush the buffer after all of the subpictures of the picture have been parsed or decoded. Claim 10 A decoder according to claim 6, further configured to distinguish between two subsets of NAL unit types based on an integer value syntax element corresponding to the NAL unit type, wherein the first subset of the two NAL unit type subsets corresponds to random access, and the second subset of the two NAL unit type subsets corresponds to non-random access. Claim 11 A non-transient computer-readable medium comprising instructions, wherein when the instructions are executed, at least one processor causes: to decode a flag indicating whether, for a picture, from a data stream, the NAL unit type differs between at least two subpictures of the picture; to determine that, when the flag is a first value, the NAL unit type is the same for all subpictures of the picture; and when the flag is a second value different from the first value, to decode a first NAL unit type for a first subpicture of the picture and a second NAL unit type for a second subpicture of the picture, wherein the first NAL unit type is different from the second NAL unit type. Claim 12 In claim 11, the first NAL unit type is random access, and the second NAL unit type is a non-random access, non-transient computer-readable medium. Claim 13 In claim 11, the flag is included in the picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS) of the data stream, and each NAL unit type is associated with a slice of a subpicture, a non-transient computer-readable medium. Claim 14 A non-transient computer-readable medium according to claim 11, further comprising instructions that, when executed, cause at least one processor to: store at least one of the subpictures in a buffer while decoding the subpictures when the flag is the second value; and flush the buffer after all of the subpictures of the picture have been parsed or decoded. Claim 15 A non-transient computer-readable medium according to claim 11, further comprising instructions that, when executed, cause at least one processor to distinguish between two subsets of NAL unit types based on an integer value syntax element corresponding to the NAL unit type, wherein the first subset of the two subsets of NAL unit types corresponds to random access, and the second subset of the two subsets of NAL unit types corresponds to non-random access. Claim 16 A method for encoding pictures into a data stream, comprising: determining whether a NAL unit type is the same for all subpictures of a picture; encoding a flag with a first value in response to a determination that the NAL unit type is the same for all subpictures of the picture; and encoding the flag with a second value in response to a determination that the NAL unit type is not the same for all subpictures of the picture — the second value being different from the first value —, and encoding a first NAL unit type for a first subpicture of the picture and a second NAL unit type for a second subpicture of the picture, wherein the first NAL unit type is different from the second NAL unit type. Claim 17 An encoder, wherein it determines whether the NAL unit type is the same for all subpictures of a picture; in response to the determination that the NAL unit type is the same for all subpictures of the picture, it encodes a flag with a first value; and in response to the determination that the NAL unit type is not the same for all subpictures of the picture: it encodes the flag with a second value — the second value being different from the first value — and is configured to encode a first NAL unit type for a first subpicture of the picture and a second NAL unit type for a second subpicture of the picture, wherein the first NAL unit type is different from the second NAL unit type. Claim 18 A non-transient computer-readable medium comprising instructions, wherein when the instructions are executed, at least one processor causes: to determine whether a NAL unit type is the same for all sub-pictures of a picture; in response to the determination that the NAL unit type is the same for all sub-pictures of the picture, to encode a flag with a first value; and in response to the determination that the NAL unit type is not the same for all sub-pictures of the picture, to encode the flag with a second value — the second value being different from the first value — and to encode a first NAL unit type for a first sub-picture of the picture and a second NAL unit type for a second sub-picture of the picture, wherein the first NAL unit type is different from the second NAL unit type.

Images