Decoder, encoder, and method for mixing NAL units of different NAL unit types in a video stream

Abstract

To provide a decoder, an encoder, and a corresponding method for processing a video data stream (11) including a first sub-bitstream (11-1) and a second sub-bitstream (11-2).SOLUTION: In a solution for mixing different network abstraction layer (NAL) unit types within an access unit, a non-intra random access point (IRAP) NAL unit type is mixed with a different IRAP NAL unit type or a non-IRAP NAL unit type, and a non-IRAP NAL unit type is mixed with a different non-IRAP NAL unit type.SELECTED DRAWING: None

Description

Technical Field

[0001] Embodiments of the present disclosure relate to decoders and corresponding encoders for decoding / encoding video streams, and particularly to video streams comprising at least two video sub-streams. The video stream may comprise a plurality of access units, and each access unit may comprise at least one picture comprised in each of the two or more video sub-streams. Each picture may be associated with a NAL (Network Abstraction Layer) unit of a particular NAL unit type. The innovative concepts described herein introduce a solution for mixing NAL units of different NAL unit types.

Background Art

[0002] NAL units can be classified into VCL (Video Coding Layer) and non-VCL NAL units according to whether they contain encoded pictures or other related data, respectively. The HEVC standard includes several VCL NAL unit types that identify picture categories for decoder initialization and random access purposes. Using the content of the 2-byte NAL unit header, the purpose of the associated payload data can be easily identified.

[0003] Video coding techniques enable random access and bitstream coupling. Regarding random access, a bitstream can begin with an IDR (Instantaneous Decoding Refresh) access unit. An IDR access unit contains an independently coded picture, i.e., a coded picture that can be decoded without decoding previous pictures in the NAL unit stream. The presence of an IDR access unit indicates that subsequent pictures in the bitstream do not require references to pictures preceding the picture they contain in order to be decoded. IDR pictures are used within coded structures known as Closed GOPs (Group of Pictures). An alternative to the IDR syntax described above is the use of independently coded pictures at RAP or IRAP ((Intra)Random Access Point) locations, i.e., the CRA (Clean Random Access) picture syntax, which specifies a location within the bitstream. This syntax supports an efficient temporal coding order known as the open GOP operation, specifying a position in the bitstream where the decoder can begin successfully decoding a picture without having to decode any previously appearing pictures in the bitstream.

[0004] Good support for random access can be important to enable channel switching, seek operations, and dynamic streaming services. Several pictures that follow a CRA picture in the decoding order and precede it in the display order may contain interpicture prediction references to pictures that are not available in the decoder. Therefore, these undecodeable pictures may be discarded by the decoder, which begins the decoding process at the CRA point. For this purpose, such undecodeable pictures may be identified as RASL pictures (Random Access Skipped Leading). An IRAP picture may be an IDR or CRA picture, followed by a RASL picture in the bitstream. RASL pictures may be discarded by the decoder because, for example, they may contain references to pictures that do not actually exist in the bitstream, for concatenation operations.

[0005] A further type of picture that can follow an IRAP picture in decoding order and precede an IRAP picture in output order is a RADL picture (Random Access Decordable Leading), which may not contain any references to any picture that precedes an IRAP picture in decoding order. RADL and RASL pictures are sometimes collectively called Leading Pictures (LPs). Pictures that follow an IRAP picture in both decoding and output order are known as trailing pictures (TRAILs). They may not contain any references to LPs for interpicture prediction.

[0006] Modern video coding can also provide temporal sublayering support. Therefore, a temporal identifier indicating a level in a hierarchical temporal prediction structure can be specified in the NAL unit header. This helps achieve temporal scalability without the need to parse the rest of the bitstream outside the NAL unit header. Under certain circumstances, the number of decoded temporal sublayers can be adjusted during the decoding process of a single encoded video sequence. The location of a point in the bitstream where sublayer switching is possible, allowing the decoding of several higher temporal layers to begin, can be indicated by the presence of an STSA picture (Stepwise Temporal Sublayer Access). At the location of an STSA picture, it may be possible to switch from decoding lower temporal sublayers to decoding one specific higher temporal sublayer (but not further layers above it, unless they also contain an STSA picture).

[0007] When managing multiple reference pictures, a specific set of previously decoded pictures may exist in the DPB (Decorded Picture Buffer) for decoding the rest of the pictures in the bitstream. To identify these pictures, a list of POC (Picture Order Count) identifiers may be sent in each slice header. The set of reference pictures to be retained is called the RPS (Reference Picture Set). For each picture in the bitstream, the POC may include the least significant bit portion (LSB) and the most significant bit portion (MSB) to indicate its current picture order count (POC value), and the picture order may differ from the decoding order of each picture. Therefore, decoding a picture after an IRAP picture may refer to a reference picture that preceded the IRAP picture, but this can be difficult.

[0008] Therefore, it is desirable to improve existing encoders and decoders that can provide proper handling of pictures and sequences of pictures in a bitstream, especially when they appear after IRAP pictures.

[0009] Accordingly, it is proposed to provide a decoder having at least the features of an independent claim, and a corresponding apparatus for encoding and / or merging having at least the features of an independent claim. Furthermore, it is proposed to provide corresponding methods for encoding and decoding, as well as corresponding computer program products and bitstreams, respectively. Advantageous embodiments are presented in the dependent claims.

[0010] As described above, a bitstream may contain a picture associated with an IRAP NAL unit type, which may also be called an IRAP picture, and a picture not associated with an IRAP NAL unit type, i.e., a picture associated with a non-IRAP NAL unit type, may also be called a non-IRAP picture. [Overview of the project] [Means for solving the problem]

[0011] In one embodiment, a decoder is provided for decoding video content, the decoder is configured to decode a bitstream comprising a first subbitstream associated with a first spatial segment (131) of the picture of the video content and a second subbitstream associated with a second spatial segment (132) of the picture of the video content. In this case, the bitstream comprises at least one access unit in which at least one first NAL unit of the first subbitstream is of a non-IRAP NAL unit type, e.g., RASL, RADL, TRAIL, STSA. The at least one access unit further comprises at least one different second NAL unit of the second subbitstream, which is of a non-IRAP NAL unit type, e.g., RASL, RADL, TRAIL, STSA, and the first NAL unit of the first subbitstream and the second NAL unit of the second subbitstream are mixed. In other words, non-IRAP NAL units are mixed with different non-IRAP NAL units. For example, a NAL unit of the TRAIL picture NAL unit type can be mixed with a NAL unit of the RASL picture NAL unit type. According to this non-restrictive example, within one access unit, the TRAIL picture of the first sub-bitstream may be mixed with the RASL picture of the second sub-bitstream. Another non-restrictive example may allow the TRAIL picture of the first sub-bitstream to be mixed with the RADL picture of the second sub-bitstream within one access unit. Several further non-restrictive examples may allow the TRAIL picture of the first sub-bitstream to be mixed with the STSA picture of the second sub-bitstream within one access unit. Several further non-restrictive examples may allow the STSA picture of the first sub-bitstream to be mixed with the RADL picture of the second sub-bitstream within one access unit. Several further non-restrictive examples may allow the STSA picture of the first sub-bitstream to be mixed with the RASL picture of the second sub-bitstream within one access unit.Some further non-restrictive examples may allow mixing a RASL picture in the first sub-bitstream with a RADL picture in the second sub-bitstream within a single access unit.

[0012] In a further embodiment, a decoder is provided for decoding video content, the decoder is configured to decode a bitstream comprising a first subbitstream relating to a first spatial segment of a picture of the video content and a second subbitstream relating to a second spatial segment of a picture of the video content. In this case, the bitstream comprises at least one access unit of an IRAP NAL unit type in which at least one first NAL unit of the first subbitstream is mixed with at least one second NAL unit of the second subbitstream, which is a further IRAP NAL unit type. In other words, the first IRAP NAL unit is mixed with a further second IRAP NAL unit. The further second IRAP NAL unit may be of the same NAL unit type as the first IRAP NAL unit, or it may be of a different NAL unit type than the first IRAP NAL unit. For example, a NAL unit of the CRA picture NAL unit type may be mixed with further pictures of the same NAL unit type, for example, further CRA picture NALs, or with further pictures of a different NAL unit type, for example, a NAL unit of the IDR picture NAL unit type with a preceding picture, i.e., an IDR-with-RADL picture unit type (IDR_W_RADL). According to this non-limiting example, within a single access unit, a CRA picture in the first sub-bitstream may be mixed with further CRA pictures in the second sub-bitstream or with an IDR_W_RADL picture in the second sub-bitstream.

[0013] In a further embodiment, a decoder is provided for decoding video content, the decoder is configured to decode a bitstream comprising a first subbitstream relating to a first spatial segment of the picture of the video content and a second subbitstream relating to a second spatial segment of the picture of the video content. In this case, the bitstream comprises at least one access unit which is an IRAP NAL unit type in which at least one first NAL unit of the first subbitstream is mixed with at least one second NAL unit of the second subbitstream which is either a non-IRAP reading picture NAL unit type (e.g., RASL or RADL) or an STSA NAL unit type. In other words, within one access unit, an IRAP NAL unit (IDR_W_RADL or CRA) is mixed with a non-IRAP NAL unit, and the non-IRAP NAL unit is one of a RASL picture, a RADL picture, or an STSA picture.

[0014] In a further embodiment, a decoder is provided for decoding video content, the decoder is configured to decode a bitstream containing a first subbitstream and a second subbitstream, the decoder is configured to derive from slice headers of NAL units of the first and second subbitstreams, the NAL units being IRAP NAL unit type (e.g., CRA, IDR_W_RADL), and the information is for setting the top-level POC portion of the POC of the picture referenced by the NAL unit. In this example, the top-level POC portion needs to be set to a predetermined value for all NAL units of IRAP NAL unit type in the access unit of the bitstream, if at least one NAL unit in the access unit is of non-instantaneous Decoding Refresh (non-IDR) unit type. In other words, if at least one NAL unit in the access unit references a non-IDR picture, e.g., a CRA picture, the top-level POC portion needs to be set to a predetermined value. As a non-limiting example, the given value may be greater than or equal to the top-level POC portion of the POC associated with the picture preceding the picture referenced by the NAL unit. In other words, if a non-IDR picture exists within the access unit, a POC reset may not occur.

[0015] In a further embodiment, a decoder is provided for decoding video content, the decoder is configured to decode a bitstream containing a first sub-bitstream and a second sub-bitstream, the decoder is configured to derive from slice headers of NAL units of the first and second sub-bitstreams, the NAL units being IRAP NAL unit types (e.g., CRA, IDR_W_RADL) and information for setting the top-level POC portion of the POC of the picture referenced by the NAL unit. In this example, no picture reference is permitted to traverse any access unit of the bitstream in the decoding order, where all NAL units are of IRAP NAL unit type and have slice headers with top-level POC portions set to a predetermined value. In other words, if each picture in an access unit may reference an IRAP NAL unit type (e.g., CRA or IDR_W_RADL) and have a top-level POC portion set to a predetermined value, no picture referenced from a reference picture preceding the access unit may be permitted. As an unrestricted example, the predetermined value may be smaller than the top-level POC portion of the POC associated with the picture preceding the picture referenced by the NAL unit. The predetermined value may be even smaller, for example, if a reset of the top-level POC portion occurs. Therefore, according to the unrestricted example above, the access unit may only have IRAP pictures, and if a reset of the top-level POC portion occurs, picture referencing by referencing a preceding reference picture may not be permitted.

[0016] In a further embodiment, a decoder is provided for decoding video content, the decoder is configured to decode a bitstream containing a first subbitstream and a second subbitstream, the decoder is configured to derive from slice headers of NAL units of the first and second subbitstreams, the NAL units being IRAP NAL unit types (e.g., CRA, IDR_W_RADL) and information for setting the top-level POC portion of the POC of the picture referenced by the NAL unit. In this example, it is not permitted for a referenced picture of a first access unit having a slice header preceding a second access unit of a bitstream having a slice header where the top-level POC portion is set to a first predetermined value, and in bitstream order, all NAL units are of IRAP NAL unit type, and the top-level POC portion is set to a second predetermined value, to be referenced by a picture following the second access unit of the bitstream in bitstream order. In a non-limiting example, the first predetermined value may be greater than the second predetermined value. In other words, if the first predetermined value of the top-level POC portion of the (preceding) first access unit is greater than the second predetermined value of the top-level POC portion of the second access unit, then a picture following the second access unit may not be permitted to reference the reference picture preceding the second access unit.

[0017] In a further embodiment, corresponding encoders, methods for encoding, and methods for decoding are proposed.

[0018] In a further embodiment, a computer program is provided, each of which is configured to implement one of the methods described herein when executed on a computer or signal processor, thereby the methods described herein are implemented by one of the computer programs. [Brief explanation of the drawing]

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the drawings. [Figure 1] Figure 1 is a schematic diagram of a bitstream that can be processed by the decoder and encoding / merging apparatus of the present invention according to an embodiment. [Figure 2] Figure 2 is a schematic diagram of a bitstream having an access unit with the IDR_N_LP NAL unit type and all other access units with the same NAL unit type. [Figure 3] Figure 3 is a schematic diagram of a bitstream having an access unit with the IDR_W_LP NAL unit type and a corresponding RADL unit type mixed with the TRAIL unit type. [Figure 4] Figure 4 is a schematic diagram of a bitstream having an access unit with a CRA NAL unit type and a corresponding RASL unit type mixed with a TRAIL unit type. [Figure 5] Figure 5 shows a complete 360-degree low-resolution thumbnail with selective high-resolution tiles. [Figure 6] Figure 6 shows an exemplary coding structure in which, in one tile (top example), the picture uses the STSA NAL unit type, which indicates a stepwise temporal sublayer access, and in another tile (bottom example), the same picture uses the RASL NAIL unit type. [Figure 7] Figure 7 shows exemplary coding structures in a high-resolution region using a closed GOP structure (top) and a high-resolution region using an open GOP structure (bottom). [Figure 8] Figure 8 shows an exemplary coding structure of a multilayer bitstream with a POC reset with layer-aligned RAP pictures. [Figure 9] Figure 9 shows an exemplary coding structure for a multilayer bitstream with a POC MSB reset when all NAL unit IDRs are not permitted. [Figure 10] FIG. 10 is a diagram showing an exemplary coding structure of a multi-layer bit stream with POC MSB reset when all NAL unit IDs are permitted. [Figure 11] FIG. 11 is a diagram showing an exemplary coding structure of a multi-layer bit stream when POC MSB reset and LT reference are not permitted. [Figure 12] FIG. 12 is a diagram showing an exemplary coding structure of a multi-layer bit stream when all POC MSBs are equal to 0 and LT reference is not permitted. [Figure 13] FIG. 13 is a diagram showing an exemplary coding structure of a multi-layer bit stream when all POC MSBs are equal to 0 and LT reference is permitted. [Figure 14] FIG. 14 is a diagram showing an exemplary coding structure of a multi-layer bit stream when the POC MSB is not reset and LT reference is permitted.

BEST MODE FOR CARRYING OUT THE INVENTION

[0020] Elements having equivalent or equivalent elements or equivalent or equivalent functions are indicated by equivalent or equivalent reference numerals in the following description.

[0021] Method steps depicted by block diagrams and described with reference to the block diagrams may also be executed in an order different from the order depicted and / or described. Further, method steps regarding specific features of the apparatus may be replaceable with the features of the apparatus or may be other methods.

[0022] Mixing of NAL unit types in pictures and access units 1. Mixed unit types within an encoded picture Figure 1 shows, for introductory purposes, the structure of an exemplary video bitstream 11 that can be processed by a decoder 100 and apparatus 10 for encoding and / or merging a video bitstream 11 according to an embodiment of the present invention. The video bitstream 11 comprises a sequence of consecutive pictures 12a, 12b, 12c, two or more of the consecutive pictures 12a, 12b, 12c can be grouped together in a GOP (Group of Pictures) 12 that can represent at least a portion of the intended video content.

[0023] For coding purposes, each picture 12a, 12b, and 12c can be subdivided into spatial segments, such as slices, tiles, or blocks. In this non-limiting example, each picture 12a, 12b, and 12c is subdivided into two spatial segments, namely, a first (e.g., left) spatial segment 131 and a second (e.g., right) spatial segment 132.

[0024] Bitstream 11 can be subdivided into a first sub-bitstream 11-1 associated with the first spatial segment 131 of pictures 12a, 12b, and 12c, and a second sub-bitstream 11-2 associated with the second spatial segment 132 of pictures 12a, 12b, and 12c.

[0025] Some pictures 12a, 12b can be coded dependently, for example by predictive coding. Some of these pictures 12b can be coded by single prediction, for example by referring to a single preceding picture, while some other pictures 12a can be coded by biprediction, for example by referring to a preceding and succeeding picture. Some other pictures 12c can be coded independently, i.e., these pictures 12c cannot refer to a preceding or succeeding picture. Independently coded pictures 12c can provide an (intra) random access point ((I)RAP) representing a position in the bitstream 11, so that the decoder can begin decoding a succeeding picture without having to decode an earlier-appearing picture in the bitstream 11. For example, picture 12c is an independently coded picture which may also be called an (I)RAP picture. In this disclosure, the terms RAP and IRAP may be used as synonyms.

[0026] Bitstream 11 may include one or more access units 30 for accessing bitstream 11. In the non-limiting example of Figure 1, the access unit 30 can align (with respect to presentation order) picture segments 131 and 132 contained in the first and second sub-bitstreams 11-1 and 11-2. Pictures 12a, 12b, and 12c containing these aligned picture segments 131 and 132 can each be grouped into different categories, for example, IRAP pictures and non-IRAP pictures (as described above).

[0027] A Network Abstraction Layer (NAL) can provide a high-level syntax for defining the picture segments 131, 132 and the pictures 12a, 12b, 12c. In particular, an NAL unit may be associated with each of the pictures 12a, 12b, 12c, with respect to the picture segments 131, 132. Thus, in this disclosure, the picture segments 131, 132 and the first and second sub-bitstreams 11-1, 11-2 can also be referred to as NAL units. Therefore, the first and second sub-bitstreams 11-1, 11-2 are also referred to as NAL unit streams. Each of the NAL units 131, 132 may contain a specific NAL unit type. The NAL unit type can then identify the aforementioned category of each picture contained within the NAL unit, i.e., whether each NAL unit is of IRAP type or non-IRAP type.

[0028] There are multiple applications that rely on mixing IRAP types within a picture, where the IRAP type is a random access property of an Access Unit (AU) 30 or a portion thereof, indicated through the NAL unit type. Use cases are in single-layer or multi-layer codec domains, i.e., a single-layer codec bitstream consisting of slices where the picture is IRAP while other slices of the same picture are not, or a multi-layer codec bitstream where the Access Unit contains IRAP in one layer and non-IRAP pictures in other layers.

[0029] Currently, support for mixed NAL unit types is provided in the prior art where a single type of IRAP NAL unit can be mixed within a picture together with a TRAIL picture. However, this does not cover use cases that adequately motivate the need for features such as those described below, for which the present invention provides a solution.

[0030] The primary use case motivating the need for mixed NAL unit types is 360-degree video, where tile streaming is used, with only a subset of the entire 360 ​​degrees transmitted in high resolution and the rest in low resolution. When a change in viewing direction occurs, some of the high-resolution tiles are no longer needed, but new tiles shown in low resolution will be needed in high resolution from this point onward. These tiles are the only tiles that encounter the change (from high resolution to low resolution, and vice versa). These tiles are downloaded using an IRAP-based representation, while other tiles do not require this property. When merging bitstreams corresponding to different tiles into a single bitstream, some of the tiles will contain NAL unit types corresponding to IRAP pictures, while others will contain NAL unit types for non-IRAP pictures.

[0031] Furthermore, there are several other use cases where it is desirable to allow AUs to have different types of NAL units. For example, in a scenario where there is a 1080p picture with an RoI (Region of Interest) inside a 720p picture, different parts of the video (RoI or others) can be assumed to have different RAP periods. For example, the same bitstream may be used to feed to two receivers, one of which is interested in the entire video and the other only in the RoI.

[0032] In the case of 360-degree video streaming where different bitstreams merge, or RoI streaming where the video is originally encoded into different parts having different RAP periods for different regions, the decoder will encounter bitstreams where the NAL units in the encoded picture are not necessarily the same. Therefore, the present invention provides a solution for mixing NAL units of different NAL unit types.

[0033] 1.1 Trailing Picture Mixing (Prior Art) The solution currently proposed in the prior art is that the PPS flag indicates whether RAP pictures are mixed with non-RAP pictures, and then the NAL unit types that can be encountered in the NAL unit of the mixed AU are the TRAILNAL unit type for non-IRAP types (maximum VCL_RSV_6 or GDR), and a single type IDR_W_RADL (IDR with Leading Pictures), IDR_N_LP (IDR with No Leading Pictures), and CRA (Clean Random Access). Therefore, the prior art suggests mixing IRAP pictures with TRAIL pictures.

[0034] Figure 2 shows a prior art example in which an IRAP NAL unit type NAL unit 201 is mixed with a trailing picture type (TRAIL) NAL unit 202. The IRAP NAL unit type in this example is an IDR (IDR_N_LP) without a leading picture. All other access units have NAL units of the same NAL unit type (i.e., TRAIL).

[0035] 1.2 Mixing Reading Pictures However, these means are insufficient for the envisioned applications in which an open GOP structure may be introduced and applied to a leading picture (LP). Figure 3 shows an exemplary case in which an IRAP NAL unit 301 of type IDR_W_RADL is shown with its associated leading picture 302, which may be an IRAP NAL unit of type RADL (Random Access Decordable Leading Picture). An IRAP type NAL unit 301 (e.g., IDR_W_RADL) may be mixed with a non-IRAP NAL unit type NAL unit 303 (e.g., TRAIL). An associated NAL unit 302 of a non-IRAP type leading picture type (RADL) may be mixed with a different non-IRAP type NAL unit 304 (e.g., TRAIL).

[0036] Figure 4 shows a further exemplary case in which an IRAP NAL unit type (e.g., CRA) NAL unit 301 is shown with its associated reading picture 302, which could be an IRAP NAL unit type RASL (Random Access skipped Reading Picture). Alternatively, although not shown, the associated reading picture 302 could be an IRAP NAL unit type RADL (Random Access Decodeable Reading Picture) NAL unit 302. An IRAP type NAL unit 301 (e.g., CRA) can be mixed with a non-IRAP NAL unit type NAL unit 303 (e.g., TRAIL). An associated NAL unit 302 of a non-IRAP type reading picture type (e.g., RASL) can be mixed with a different non-IRAP type NAL unit 304 (e.g., TRAIL).

[0037] Therefore, it is necessary to handle encoded pictures with mixed NAL unit types, not only when IRAP NAL units are included, but also when only non-IRAP NAL units are included in the mixed picture.

[0038] In short, the prior art proposes mixing IRAP NAL units with TRAIL-type non-IRAP NAL units, or mixing non-IRAP NAL units of the same type, i.e., TRAIL-type NAL units, with TRAIL-type NAL units.

[0039] The present invention proposes, instead, mixing a first non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA) with at least one different second non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA).

[0040] Accordingly, according to this embodiment, a decoder 100 is provided for decoding the video content 12, and the decoder 100 is configured to decode a bitstream 11 which includes a first subbitstream 11-1 associated with a first spatial segment 131 of pictures 12a, 12b, and 12c of the video content 12, and a second subbitstream 11-2 associated with a second spatial segment 132 of pictures 12a, 12b, and 12c of the video content 12. According to this embodiment, the bitstream 11 includes at least one access unit 30 which is a non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA) in which at least one first NAL unit 302 of the first subbitstream 11-1 is mixed with at least one different second NAL unit 304 of the second subbitstream 11-2 which is a different non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA).

[0041] For example, a TRAIL picture can be mixed with a RADL picture or a RASL picture. TRAIL + RADL TRAIL + RASL

[0042] Therefore, according to one embodiment, the decoder 100 may be configured to process a bitstream 11 in which the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access decodeable reading picture unit type (RADL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a trailing picture unit type (TRAIL).

[0043] In a further embodiment, the decoder 100 may be configured to process a bitstream 11 in which the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access skip reading picture unit type (RASL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a trailing picture unit type (TRAIL).

[0044] In one embodiment of the present invention, the bitstream 11 has an instruction as a flag in a parameter set (PPS, SPS, VPS) which can mix, for example, the NAL unit types in the encoded picture as follows: TRAIL + RADL TRAIL + RASL

[0045] Such signaling can be carried in PPS to indicate that TRAIL+(CRA xor IDR_W_LD xor IDR_N_P) is mixable.

[0046] For example, if the flag is not set (i.e., flag=0), all NAL units in a picture may have the same NAL unit type, and a picture or PU is said to have the same NAL unit type as the encoded slice NAL units of the picture or PU.

[0047] Otherwise, if the flag is set (i.e., flag = 1), the picture may contain at least two subpictures. Additionally or alternatively, the NAL units of the picture may comprise at least two different NAL unit types. Furthermore, additionally or alternatively, the NAL units of the picture may not comprise any NAL unit types of the GDR. Further additionally or alternatively, if one of the NAL units of the picture is at least one of the following NAL unit types, • Instant decoder refresh using random access decode-reading picture (IDR_W_RADL) • Instant decoder refresh without a reading picture (IDR_N_LP) • Clean Random Access (CRA)

[0048] In that case, all other available NAL units of the picture may have one of the three unit types (IDR_W_RADL, IDR_N_LP, CRA) or a trailing picture unit type (TRAIL).

[0049] Alternatively, since RADL and RASL pictures do not have a different normative decoding process than TRAIL pictures, as IRAP pictures do, for example, this instruction can be made at the SPS level. In one embodiment, for example, the instruction may take the form of a constraint flag, for example, a general constraint flag. A general constraint flag may indicate whether a particular mix of NAL unit types is permitted. If such a general constraint flag is set (i.e., general constraint flag = 1), it may indicate that there is no mix of NAL unit types in the picture and that the above flag is not set in the parameter set (PPS, SPS, VPS) (i.e., flag = 0). If the general constraint flag is not set (i.e., general constraint flag = 0), the above constraint may not be imposed.

[0050] Therefore, according to one embodiment, the decoder 100 may be configured to derive an instruction from the bitstream 11, which explicitly informs the decoder 100 that a non-IRAP NAL unit type NAL unit 304 of the second sub-bitstream 11-2 (which is a trailing picture unit type (TRAIL)) is mixed with a non-IRAP NAL unit type NAL unit 302 of the first sub-bitstream 11-1, and that this NAL unit 302 is either a random access skip reading picture unit type (RASL) or a random access decodeable reading picture unit type (RADL).

[0051] According to one embodiment, the instruction may include syntax elements in a parameter set. For example, the syntax elements may be included in at least one of a Picture Parameter Set (PPS) or a Sequence Parameter Set (SPS).

[0052] However, assuming a 360-degree video tile-based streaming scenario, when tiles with RASL properties and TRAIL picture tiles are mixed within a picture, it can occur that the RASL tiles cannot be correctly reconstructed because the tiles change position at partial RAP points, and therefore the reference picture used is incorrect (i.e., still depicting the content of other tiles). For applications, recognizing these defective reconstructed picture regions is critical, while dropping the entire mixed reading picture (including TRAIL+RASL tiles) is not an option for reasons of playout continuity. Instead, the client recognizes the defective regions in the decoded and outputted picture and does not use them for subsequent processing.

[0053] Alternatively, given a configuration such as that shown in Figure 5, for example, the client can use a lower-resolution version 51 contained within the picture for content that is incorrectly reconstructed within those mixed reading pictures. In one embodiment, areas corresponding to potentially defective reconstructed tiles are indicated for subsequent processing, for example, through SEI messages or by external means, such as an application. The instructions described mean that areas corresponding to NAL units having a RASL of type NAL unit are ignored / not output.

[0054] Therefore, according to one embodiment, if the NAL unit 302 of the first sub-bitstream 11-1 is a non-IRAP NAL unit type and the NAL unit 304 of the second sub-bitstream 11-2 is a non-IRAP NAL unit type and the NAL unit 304 is a trailing picture unit type (TRAIL), the decoder 100 may be configured to derive instructions from the bitstream 11.

[0055] In one example, this instruction can be directed to the decoder 100 to ignore or not output the first spatial segment 131 in pictures 12a, 12b, and 12c of the video content 12 of the first subbitstream 11-1, which corresponds to the NAL unit 302 of the random access skip reading picture unit type of the first subbitstream 11-1.

[0056] In another example, this instruction could be directed to the decoder 100 to output the first spatial segment 131 of the video content 12a, 12b, and 12c of the first subbitstream 11-1 corresponding to the NAL unit 302 of the Random Access Skip Read Picture Unit Type (RASL) of the first subbitstream 11-1, and further instruct to mark the first spatial segment 131 as a corrupted spatial segment. For example, the decoder 100 may be configured to mark the aforementioned corrupted spatial segment 133 for subsequent processing chains via an SEI (Supplemental Enhancement information) message.

[0057] Furthermore, another option is to avoid mixing RASL pictures with trailing pictures (TRAIL) at all, and thus allow only RADL pictures to be mixed with TRAIL pictures. Thus, in one embodiment, there is a constraint as shown in SPS, and when a CRA is mixed with a TRAIL picture, the associated leading picture must be of type RADL only.

[0058] Accordingly, according to one embodiment, the decoder 100 may be configured to decode constraint instructions from a parameter set, the constraint instructions indicating to the decoder 100 that the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access decodeable reading (RADL) picture unit type only if its associated IRAP NAL unit 301 of the first sub-bitstream 11-1 is a clean random access unit type (CRA), when the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a non-reading picture NAL unit type (e.g., TRAIL or STSA).

[0059] This may be particularly true in embodiments where the non-IRAP NAL unit type of the NAL unit 304 of the second bitstream 11-2 is a trailing picture unit type (TRAIL), i.e., when mixing the associated RADL reading picture with TRAIL+CRA.

[0060] The previous embodiments focused primarily on TRAIL NAL unit types. However, STSA NAL unit types are also often possible, enabling step-wise temporal sublayer access. That is, while the above examples mainly concerned non-IRAP NAL units of the TRAIL NAL unit type, the present invention can also provide solutions for mixing non-IRAP NAL units of the STSA (Stepwise Temporal Sublayer Access) unit type with at least one of the RASL unit type, RADL unit type, and TRAIL unit type. STSA + RASL STSA + RADL STSA + TRAIL

[0061] Therefore, according to one embodiment, the decoder 100 may be configured to decode a bitstream 11 in which the non-IRAP NAL unit type of the NAL unit 302 of the first subbitstream 11-1 is a stepwise temporal sublayer access unit type (STSA), and the non-IRAP NAL unit type of the NAL unit 304 of the second subbitstream 11-2 is a trailing picture unit type (TRAIL).

[0062] In a further embodiment, the decoder 100 may be configured to decode the bitstream 11, where the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access decodeable reading picture unit type (RADL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a stepwise temporal sublayer access unit type (STSA).

[0063] In a further embodiment, the decoder 100 may be configured to decode bitstream 11 in which the non-IRAP NAL unit type of the NAL unit 302 of the first subbitstream 11-1 is a Random Access Skip Reading Picture Unit Type (RASL) and the non-IRAP NAL unit type of the NAL unit 304 of the second subbitstream 11-2 is a Stepwise Temporal Sublayer Access Unit Type (STSA).

[0064] Figure 6 shows an exemplary encoding structure in which, in one picture segment 132 (e.g., a tile), picture 12a uses NAL unit 304 of STSA NAL unit type to exhibit stepwise temporal sublayer access, while the same picture 12a in another picture segment 131 (e.g., a tile) uses NAL unit 302 of RASL NAL unit type.

[0065] Since STSA NAL unit types cannot be placed at the lowest time level 0, STSA NAL units do not mix with IRAP NAL units. However, they can mix with RADL or RASL NAL unit types, similar to the TRAIL NAL units mentioned above.

[0066] In one embodiment, the flag is included in the parameter set, for example, in the PPS, indicating that the STSA NAL unit is mixed with the RASL or RADL NAL unit type.

[0067] Therefore, in one embodiment of the present invention, there is a directive in the bitstream 11 that the NAL unit types in the encoded picture can be mixed as follows, for example, as flags in the parameter set (PPS, SPS, VPS): STSA + RADL STSA + RASL STSA + TRAIL

[0068] In further embodiments, whether such mixing is permitted may be indicated, for example, by using constraint flags at the sequence level or picture level.

[0069] Accordingly, according to one embodiment, the decoder 100 may be configured to derive instructions from the bitstream 11, where the instructions explicitly inform the decoder 100 that a non-IRAP NAL unit type NAL unit 304 of the second sub-bitstream 11-2, which is a stepwise temporal sublayer access unit type (STSA), is mixed with a non-IRAP NAL unit type NAL unit 302 of the first sub-bitstream 11-1, and that the NAL unit 302 is either a random access skip reading picture unit type (RASL) or a random access decodeable reading picture unit type (RADL).

[0070] According to one embodiment, the instruction includes syntax elements within a parameter set. According to a further embodiment, the syntax elements are included in at least one of a picture parameter set or a sequence parameter set.

[0071] Furthermore, bitstream 11 may be given instructions that effectively indicate that all NAL units 302, 304 of picture 12a, including the STSA NAL unit 304, satisfy the constraints related to the STSA NAL unit type, even if they have another NAL unit type, such as RASL (see, for example, NAL unit 302). This may be indicated by a flag in the SPS indicating that the bitstream 11-1 of the (spatial) subset containing the RASL picture (see, for example, NAL unit 302) satisfies the constraint that each RASL picture is also an STSA picture. Such signaling is even more important for merge use cases, as it can indicate whether the original bitstream containing RASL or RADL NAL units satisfies the characteristics of an STSA picture, and as a result, it can be easily derived whether the mixed-encoded picture has STSA characteristics when mixed with STSA.

[0072] Accordingly, according to one embodiment, the decoder 100 may be configured to decode a constraint flag from the parameter set of the bitstream 11, the constraint flag indicating to the decoder 100 that any access unit 30 of the bitstream 11, comprising a mixture of one or more NAL units 302 of a non-IRAP reading picture NAL unit type (e.g., RASL or RADL) and one or more NAL units 304 of a stepwise temporal sublayer access unit type (STSA), satisfies the constraint associated with the stepwise temporal sublayer access unit type (STSA), i.e., each RASL picture 302 may be treated as an STSA picture.

[0073] The above constraints can be used to indicate to the decoder 100 that, according to the decoding order, any picture associated with a higher temporal sublayer than the Stepwise Temporal Sublayer Access (STSA) unit type is not permitted to reference a picture in the above-mentioned next higher temporal sublayer if the above-mentioned next higher temporal sublayer precedes the above-mentioned Stepwise Temporal Sublayer Access (STSA) unit type.

[0074] Alternatively, a bitstream 11 having an encoded picture 12a with a mixed NAL unit type (e.g., STSA + RASL or RADL) may include a flag indicating that the picture 12a, when flagged, may contain NAL units 304 of the STSA NAL unit type, but (total) does not adhere to the constraints associated with the STSA NAL unit type. This indicates that the encoded picture 12a may have a NAL unit type equal to STSA, but the properties implied by STSA (i.e., further time levels can be decoded from its AU onward) do not apply. Therefore, STSA parsing is ignored, and stepwise temporal sublayer access is inferred.

[0075] Therefore, according to one embodiment, the decoder 100 may be configured to decode a flag from a parameter set, which indicates to the decoder 100 that any access unit 30 comprising a mixture of one or more NAL units 302 of a non-IRAP reading picture NAL unit type (RASL or RADL) and one or more NAL units 304 of a stepwise temporal sublayer access unit type (STSA) does not adhere to the constraints associated with the stepwise temporal sublayer access (STSA) unit type. Furthermore, the decoder 100 may be configured to refrain from analyzing one or more NAL units 304 of the stepwise temporal sublayer access (STSA) unit type in response to the aforementioned flag, and implicitly infer the decoding of the stepwise temporal sublayer access, i.e., the STSA constraints do not apply to each access unit.

[0076] This concept can also be applied independently of mixing NAL unit types, as described above. Accordingly, according to one embodiment, a decoder 100 for decoding video content 12 is proposed, the decoder 100 is configured to decode a bitstream 11 and decode constraint flags from a parameter set of the bitstream 11, the aforementioned constraint flags indicating to the decoder 100 that any access unit 30 having NAL units 302 of a non-IRAP reading picture NAL unit type (e.g., RASL or RADL) satisfies constraints related to stepwise temporal sublayer access (STSA) unit types within the bitstream 11.

[0077] 1.3 Mixing various RAP types A further important case not made possible by prior art is the ability to mix pictures in an open GOP (Group of Pictures) structure with pictures in a closed GOP structure. For example, mixing two different types of IRAP NAL unit types (e.g., IDR, CRA), e.g., IDR_W_RADL and CRA NAL unit type may be permitted, or mixing NAL units of the same NAL unit type, e.g., CRA and CRA may be permitted. · IDR_W_RADL + CRA ·CRA + CRA

[0078] As described above, this embodiment allows for the mixing of IRAP NAL units of the NAL unit type associated with open GOP structures and IRAP NAL units of the NAL unit type associated with closed GOP structures. For example, a CRA having RADL may be associated with a closed GOP structure, while a CRA having RASL may be associated with an open GOP structure.

[0079] Accordingly, according to the embodiment, a decoder 100 for decoding the video content 12 may be provided, the decoder 100 being configured to decode a bitstream 11 which includes a first subbitstream 11-1 associated with a first spatial segment 131 of pictures 12a, 12b, 12c of the video content 12 and a second subbitstream 11-2 associated with a second spatial segment 132 of pictures 12a, 12b, 12c of the video content 12. According to this example, the bitstream 11 includes at least one access unit 30 which is an IRAP NAL unit type in which at least one first NAL unit 302 of the first subbitstream 11-1 is mixed with at least one second NAL unit 304 of the second subbitstream 11-2 which is a further (same or different) IRAP NAL unit type.

[0080] There is at least one use case, and it is very useful. For example, one approach to 360-degree video transmission using tiled streaming is when the entire low-resolution video content is being transmitted (whether or not some of it is also transmitted as high-resolution tiles). In such a case, the change in viewing direction and the corresponding change in high-resolution tile selection from one viewport to another are shown in Figure 5.

[0081] Since the entire low-resolution video content is always available on the client side and no per-tile stream switching is required for this region, low-resolution content can be encoded with CRA, and therefore can take advantage of higher encoding efficiency compared to a closed GOP structure using IDR. Also, the longer the random access point period required for stream switching, the more sufficient this region is, again benefiting the encoding efficiency of low-resolution video. On the other hand, high-resolution tiles can change frequently in fast transitions from one viewport to another, as shown in Figure 5, and therefore, as with low-resolution video, using CRA and associated RASL pictures does not allow for smooth viewport switching. This is because RASL regions in pictures representing repositioned or newly added tile streams cannot be properly decoded (due to lack of references) and are discarded when they encounter a viewport change.

[0082] Figure 7 shows an example in which NAL unit 304 associated with a closed GOP structure is mixed with NAL unit 302 associated with an open GOP structure. In this particular, but non-limiting example, CRA-type NAL unit 304 having a leading RADL 314 (closed GOP) is mixed with CRA-type NAL unit 302 having a leading RASL 312 (open GOP). This non-limiting example shows an encoded structure for a high-resolution region in a second sub-bitstream 11-2 that uses a closed GOP structure with CRA in combination with a RADL-type leading picture, and a low-resolution region in a first sub-bitstream 11-1 that uses an open GOP structure with CRA in combination with a RASL-type leading picture.

[0083] According to one embodiment, the decoder 100 is configured to process the bitstream 11, wherein the IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is an IRAP NAL unit type associated with a closed GOP structure, and the IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is an IRAP NAL unit type associated with an open GOP structure.

[0084] In this particular non-limiting embodiment, as shown in Figure 7, the IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a CRA unit type having a random access decodeable picture unit type (CRA with RADL), and the IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a CRA unit type having a random access skip reading picture unit type (CRA with RASL).

[0085] Note that Figure 7 shows the sequence of access units and the pictures contained within them in their display order. However, the display order may differ from the encoding order, also known as the bitstream order. As can be seen, access unit 30 contains a mixed picture 12a that includes a first VCL NAL unit 302 representing the first spatial segment 131a of picture 12a in the first sub-bitstream 11-1, and a second VCL NAL unit 304 representing the second spatial segment 132a of picture 12a in the second sub-bitstream 11-2. In this case, the first and second NAL units are IRAP NAL units. In this particular, non-limiting example, both the first and second NAL units 302 and 304 are IRAP NAL unit type CRA. Also, Figure 7 shows the display order of consecutive pictures.

[0086] Therefore, in the display order, a further access unit 31 may precede the access unit 30 described above. The leading access unit 31 described above includes a mixed picture 12b that includes a first VCL NAL unit 312 representing the first spatial segment 131b of the picture 12b in the first sub-bitstream 11-1, and a second VCL NAL unit 314 representing the second spatial segment 132b of the picture 12b in the second sub-bitstream 11-2. The first NAL unit 312 of this AU31 is of non-IRAP NAL unit type RASL, and the second NAL unit 314 of this AU31 is of non-IRAP NAL unit type RADL. Figure 7 also shows the display order of consecutive pictures.

[0087] Optionally, and in terms of display order, one or more further access units, such as the access unit 32 described exemplarily, may precede the access unit 31 described above. More generally, access unit 31, which includes non-IRAP NAL units 312, 314 (e.g., RASL or RADL), may precede, in terms of display order, at least one further access unit 32, which may also include non-IRAP NAL units 322, 324 (e.g., RASL or RADL). To maintain the open GOP structure in the first sub-bitstream 11-1, the RASL NAL unit 312 of access unit 31 may precede (in terms of display order) by a NAL unit 322, which may be of type RASL and is included in the preceding access unit 32. Next, to maintain the closed GOP structure in the second sub-bitstream 11-2, the RADL NAL unit 314 of access unit 31 may precede (in terms of display order) by a NAL unit 324, which may also be of type RADL and is included in the preceding access unit 32.

[0088] The encoding order, also known as the bitstream order, may differ from the display order shown in Figure 7. For example, RADL and RASL pictures 131b, 132b, 131c, and 132c are encoded dependently, which means that CRA pictures 131a and 132a extract differential encoding information from their respective CRA pictures 13 (in display order). For example, RADL and RASL pictures 131b and 132b included in the access unit 31 refer to CRA pictures 131a and 132a included in the access unit 30 (in display order), even though the access unit 30 follows the access unit 31 (in display order).

[0089] RASL picture 131b may optionally refer to further pictures (in display and encoding order), such as the preceding RASL picture 131c illustrated in Figure 7. Therefore, the preceding RASL picture 131c of the preceding access unit 32 must be encoded before the RASL picture 13b of access unit 31. However, the even preceding RASL picture 131c of the preceding access unit 32 may refer to the preceding picture in encoding order, although the display order follows, as is the case with the IRAP picture 131a of access unit 30. That is, the IRAP(CRA) picture 131a of access unit 30 is the last picture in the sequence (i.e., CRA) picture 13 described above, but since both RASL pictures 131b and 131c refer to the IRAP(CRA) picture 131a during encoding, the IRAP(CRA) picture 131a of access unit 30 is the first picture in encoding order, i.e., it must be encoded first.

[0090] The same applies to RADL pictures. RADL picture 132b can arbitrarily reference preceding pictures (in display and encoding order), as exemplified by preceding RADL picture 132c in Figure 7. Therefore, further preceding and succeeding RADL pictures 132c of the preceding access unit 32 must be encoded before the RADL picture 132b of access unit 31. However, even further preceding RADL pictures 132c of the preceding access unit 32 may reference preceding pictures in encoding order, although their display order follows, as exemplified by IRAP picture 132a of access unit 30. In other words, the IRAP(CRA) picture 132a of the access unit 30 is the last picture in the order (i.e., CRA) picture 13 as described above, but the IRAP(CRA) picture 132a of the access unit 30 is the first picture in the encoding order, that is, it must be encoded first, because both RADL pictures 132b and 132c refer to the IRAP(CRA) picture 132a during encoding.

[0091] More generally, non-IRAP pictures of the Reading Picture (LP) type (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) can refer to following IRAP pictures (e.g., CRA pictures 131a, 132a) (e.g., in display order), and the IRAP pictures (e.g., CRA pictures 131a, 132a) must be encoded first, i.e., before the Reading Picture (LP) type reading non-IRAP pictures (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) (in display order). In other words, IRAP pictures (CRA pictures 131a, 132a, etc.) follow the display order of non-IRAP pictures of the Reading Picture (LP) type (RASL and RADL pictures 131b, 131c, 132b, 132c, etc.), but IRAP pictures (CRA pictures 131a, 132a, etc.) are encoded first. In other words, non-IRAP pictures of the Reading Picture (LP) type (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) read IRAP pictures (e.g., CRA pictures 131a, 132a) in display order, and IRAP pictures (e.g., CRA pictures 131a, 132a) read non-IRAP pictures of the Reading Picture (LP) type (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) in bitstream order (encoded order). In other words, non-IRAP pictures of type Leading Picture (LP) (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) lead IRAP pictures (e.g., CRA pictures 131a, 132a) in display order, while non-IRAP pictures of type Loading Picture (LP) (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) follow IRAP pictures (e.g., CRA pictures 131a, 132a) in encoded / bitstream order.

[0092] Considering the above examples and encoding structures, one of the advantages of the present invention is that a mixture of CRAs with reading RASL (open GOP) and CRAs with reading RADL (closed GOP) in the picture of the access unit 30 is permitted. When such a mixed NALU access unit 30 is encountered in a coded video sequence (CVS) during normal playout (decoding the entire bitstream 11 from the beginning), all the necessary reference pictures become available. Therefore, pictures following a mixed picture with a RASL NAL unit are decoded and output successfully.

[0093] However, when such a mixed NALU is encountered during a seek operation, or when a decryption process is initiated from such an AU during random access, the process for generating an unavailable reference picture must be called, and is one of the following: • Subsequent processing chains need to be informed that the RASL region was not correctly decoded and that the corresponding low-resolution region is used, for example, through the SEI instruction. • Affected pictures will be completely removed from the output.

[0094] This means that when starting to decode a mixed picture, one option is to treat it as a GDR picture, where some parts are decodeable and can be shown, while other parts cannot be shown and are refreshed over time (until all RASL NAL units have finished). Such behavior relates, for example, to a 360 scenario where the RAP type is selected in the opposite way to how it is shown in Figure 7, i.e., the low resolution uses a closed GOP structure (e.g., CRA with RADL) and the high resolution content uses an open GOP structure (e.g., CRA with RASL). Then, it can show the low resolution because it uses a closed GOP structure and the player waits until the high resolution using an open GOP structure is neatly decoded to show it.

[0095] Another option for handling such a situation is to not display any pictures that are affected (or not even partially affected) by the missing references, and to drop them, i.e., drop any mixed pictures that have a NAL unit type equal to RASL.

[0096] In other words, any mixed pictures 12b, 12c having NAL unit types 312, 322 equivalent to RASL can be marked to be dropped / discarded to ensure continuous decoding behavior.

[0097] Accordingly, according to one embodiment, the decoder 100 may be configured to decode and present, in bitstream order, the jointly placed spatial segments 132b, 132c, ... of one or more pictures 12b, 12c, ... of access units 31, 32, ..., following the NAL unit 304 of the second sub-bitstream 11-2 to the associated access unit 30, such that the jointly placed spatial segments 132b, 132c, ... of the second spatial segment 132a of the picture 12a of the at least one access unit 30 to which the NAL unit 304 of the second sub-bitstream 11-2 is associated. Furthermore, the decoder 100 may be configured to mark the co-located spatial segments 131b, 131c, ... of one or more pictures 12b, 12c, ... of access units 31, 32, ..., which follow the NAL unit 302 of the first subbitstream 11-1 in which the access unit 30 is associated, so that the co-located segments 131b, 131c, ... of the first spatial segment 131a of the picture 12a of the at least one access unit 30 associated with the NAL unit 302 of the first subbitstream 11-1 are aligned together.

[0098] According to a further embodiment, the decoder 100 may be configured to operate depending on whether the bitstream 11 is to be decoded, in the case of continuous decoding during a normal playout (decoding the entire bitstream 11 from the beginning), or in the case of the start of decoding (for example, during a seek operation or when the decoding process starts from such an AU during random access).

[0099] During a normal playout (decoding the entire bitstream 11 from the beginning), the decoder 100 can be configured to decode and present the co-located spatial segments 132b, 132c, ... of one or more pictures 12b, 12c, ... of access units 31, 32, ..., which follow in bitstream order the NAL unit 304 of the second subbitstream 11-2 associated with at least one access unit 30 (e.g., CRA with RADL), where the co-located spatial segments 132b, 132c, ... of the second spatial segment 132a of the picture 12a of at least one access unit 30 associated with the NAL unit 304 of the second subbitstream 11-2. Furthermore, in the event of decoding initiation (for example, if decoding is initiated from such an AU during a seek operation or random access), the decoder 100 may be configured to discard one or more pictures 12b, 12c, ... of subsequent access units 31, 32, ... in bitstream order to at least one access unit 30, and to resume picture output after one or more discarded pictures 12b, 12c, ... In other words, any mixed pictures 12b, 12c having at least one NAL unit 312, 322 equal to RASL are dropped / discarded.

[0100] In a further embodiment, the decoder 100 may be configured to mark the aforementioned corrupted spatial segments 131b, 131c, ... for subsequent processing chains by an SEI message.

[0101] In a further embodiment, regardless of whether the IDR_W_RADL NAL unit type exists, there is an instruction in the bitstream that the AU is treated as a CRA, i.e., an IRAP with NoIncorrectPicOutputFlag equal to 1. This indicates that the AU is a mixed NALU but has the mentioned properties of IRAP and open GOP structure. In other words, the decoder 100 may be configured to derive an instruction from the bitstream 11 that points toward the decoder 100 to decode at least one access unit 30 associated with a first subbitstream 11-1 and a second subbitstream 11-2 in a manner corresponding to a clean random access NAL unit type.

[0102] The above description described an embodiment in which the decoder 100 is configured to process the bitstream 11, but here A) Non-IRAP NAL unit type non-IRAP NAL unit (e.g., RASL, RADL, TRAIL, STSA) is mixed with at least one different second NAL unit of a different non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA), or B) A NAL unit of an IRAP NAL unit type (e.g., IDR_W_RADL, CRA) is mixed with at least one NAL unit of a further IRAP NAL unit type (e.g., IDR_W_RADL, CRA).

[0103] However, there may be further embodiments in which the decoder 100 can be configured to process the bitstream 11, and here C) NAL units of IRAP NAL unit type (IDR_W_RADL, CRA) are mixed with at least one NAL unit from either a non-IRAP reading picture NAL unit type (e.g., RASL, RADL) or an STSA NAL unit type.

[0104] Further embodiments also propose to provide a corresponding apparatus for processing the video bitstream 11 during the encoding of the bitstream 11 and / or the merging of the first and second sub-bitstreams 11-1, 11-2. The apparatus 10 described above may be at least one of an encoder, merger, or network node for processing the bitstream 11 according to the innovative principles described herein.

[0105] Accordingly, according to the embodiment, an apparatus 10 for processing video content 12 is proposed, the apparatus 10 is configured to provide a first sub-bitstream 11-1 related to a first spatial segment 131 of pictures 12a, 12b, 12c of the video content 12, and a second sub-bitstream 11-2 related to a second spatial segment 132 of pictures 12a, 12b, 12c of the video content 12, the first sub-bitstream 11-1 and the second sub-bitstream 11-2 being part of bitstream 11. A) The bitstream 11 includes at least one access unit 30 in which at least one first NAL unit 302 of the first sub-bitstream 11-1 is a non-IRAP NAL unit type that is mixed with at least one different second NAL unit 304 of the second sub-bitstream 11-2, which is a different non-IRAP NAL unit type, or B) The bitstream 11 includes at least one access unit 30, wherein at least one first NAL unit 302 of the first sub-bitstream 11-1 is an IRAP NAL unit type mixed with at least one different second NAL unit 304 of the second sub-bitstream 11-2, which is a further IRAP NAL unit type, C) The bitstream 11 includes at least one access unit (30) in which at least one first NAL unit 302 of the first sub-bitstream 11-1 is an IRAP NAL unit type mixed with at least one different second NAL unit 304 of the second sub-bitstream 11-2, which is either a non-IRAP reading picture NAL unit type (RASL or RADL) or an STSA NAL unit type.

[0106] 1.4 Summary and Syntax Examples The following outlines the types of mixtures permitted within the present invention. [Table 1]

[0107] In the new mixed types described in the present invention, there are two types of mixed types: one that includes the IRAP type and the other that does not include the IRAP type.

[0108] Option 1 As mentioned above in the previous section, one option is to communicate only mixes with IRAP types using flags within the PPS. For example, reuse an existing mix of IDR or CRA and TRAIL up to VCL_RSV_6 (mixed_nalu_types_in_pic_flag). Non-IRAP types can be communicated using constraint flags in the SPS / VPS / DPS.

[0109] A mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture referencing a PPS has one or more VCL NAL units, that the VAL NAL units do not have the same nal_unit_type value and have a nal_unit_type value in the range from IDR_W_RADL to CRA_NUT, and that the picture is not an IRAP IDR picture. A mixed_nalu_types_in_pic_flag equal to 0 specifies that each picture referencing a PPS has one or more VCL NAL units, and if the VCL NAL units have a nal_unit_type value in the range from IDR_W_RADL to CRA_NUT, then the VCL NAL units of each picture referencing a PPS have the same value as nal_unit_type.

[0110] Next, if the AU is the first AU in the bitstream or the first AU after a Sequence (EOS) NAL unit that references a PPS with a mixed_nalu_types_in_pic_flag equal to 1, the NAL unit type of the encoded picture must be CRA_NUT and IDR_W_RADL. Mixed pictures with IDR / CRA having non-IRAP types are only permitted if the AU is not the first picture in the bitstream or sequence.

[0111] A value equal to 1 for no_non_irap_mixed_nalu_types_in_pic_constraint_flag specifies that for a picture whose sequence contains TRAIL, STSA, RASL, or RADL, it is a bitstream conformance requirement that the VCL NAL units have the same value as nal_unit_type. A value equal to 0 does not impose this constraint; that is, a picture may have two different values ​​for nal_unit_type among TRAIL, STSA, RASL, and RADL.

[0112] Option 2 Another option would require specifying an instruction (idc) indicating a different combination instead of the PPS flag (mixed_nalu_types_in_pic_flag).

[0113] A no_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that mixed_nalu_types_in_pic_idc must be equal to 0 for bitstream conformance to be required. A no_mixed_nalu_types_in_pic_constraint_flag equal to 0 means no such constraint is imposed.

[0114] A mixed_nalu_types_in_pic_idc value equal to 0 specifies that each picture referencing the PPS has one or more VCL NAL units, and that each VCL NAL unit in each picture referencing the PPS has the same value as nal_unit_type.

[0115] A mixed_nalu_types_in_pic_idc equal to 1 specifies that each picture referencing a PPS has multiple VCL NAL units, and all VCL NAL units have two specific values ​​of nal_unit_type ranging from TRAIL_NUT to RSV_VCL_6. [Table 2]

[0116] A mixed_nalu_types_in_pic_idc equal to 2 specifies that each picture referencing the PPS has multiple VCL NAL units, with one or more VCL NAL units all having a specific value of nal_unit_type ranging from IDR_W_RADL to CRA_NUT, and all other VCL NAL units having a specific value of nal_unit_type ranging from TRAIL_NUT to RSV_VCL_6, or being equal to GRA_NUT.

[0117] This value corresponds to the permitted mixing of NAL units and existing flags. [Table 3]

[0118] A mixed_nalu_types_in_pic_idc equal to 3 specifies that each picture referencing the PPS has multiple VCL NAL units with a nal_unit_type value of IDR_W_RADL and one or more VCL NAL units with a nal_unit_type value of CRA_NUT. [Table 4]

[0119] For any specific picture's VCL NAL units, the following applies: If mixed_nalu_types_in_pic_idc is equal to 0, the value of nal_unit_type will be the same for all coded slice NAL units in the picture. A picture or PU is said to have the same NAL unit type as the coded slice NAL units of the picture or PU. If mixed_nalu_types_in_pic_idc is 1 or 2, the following applies: If one of the VCL NAL units in a picture has a nal_unit_type value equal to TRAIL_NUT, the picture or PU is called a trailing picture or trailing PU. Otherwise (there are no TRAIL_NUT VCL NAL units in the picture), the picture or PU is called the leading picture or leading PU. If one of the VCL NAL units within a picture has a nal_unit_type value equal to RADL_NUT, the picture or PU is called a RADL picture or RADL PU. Otherwise (if the VCL NAL units in the picture do not have a nal_unit_type value equal to RADL_NUT), the picture or PU is called a RASL picture or RASL PU. Otherwise (where mixed_nalu_types_in_pic_idc is equal to 3), the picture is called a CRA picture or CRA PU.

[0120] The aspect of handling major NAL unit types as STSA pictures can be implemented using constraint flags as follows: [Table 5]

[0121] A leading_stsa_pictures_flag equal to 1 means that interpretation references of RADL and RASL pictures are constrained as follows: A no_mixed_nalu_types_in_pic_constraint_flag equal to 0 means that no such constraint is imposed.

[0122] If leading_stsa_pictures_flag is equal to 1, the following applies: When the current picture is a RASL picture or a RADL picture, there are no active entries in RefPicList[0] or RefPicList[1] that have a TemporalId equal to the TemporalId of the current picture. If the current picture is a picture that, in the decoding order, follows RASL or RADL and precedes an associated IRAP picture having the same TemporalId as the current picture, then there are no pictures with the same TemporalId as the current picture included as active entries in RefPicList[0] or RefPicList[1] that precede the RASL or RADL picture in the decoding order.

[0123] In other words, the restriction related to STSA pictures prohibits, according to the decoding order, pictures related to the next higher temporal sub-layer than STSA from using any references in the next higher temporal sub-layer preceding STSA, meaning the decoder can begin decoding the next temporal sub-layer from STSA onward. Such restrictions can, of course, also be applied to RASL / RADL pictures as described above, and therefore the signaling described above can serve as an indication of these characteristics.

[0124] 2. Mixing picture types within a multi-layer access unit This subsection may relate, as a non-limiting example, to multilayer access units in a multilayer bitstream. However, the features described herein may already be applicable to a single layer. Furthermore, this subsection illustrates the principles of the invention by referring to a long-term reference picture as a non-limiting example. However, the concepts of the invention are equally applicable to short-term reference pictures; that is, the features described herein may generally be applicable to any reference picture. Furthermore, this subsection may illustrate a POC MSB reset by setting the POC value to zero, i.e., by setting MSB = 0. However, the value zero acts only as a placeholder symbolizing a decreasing value for the MSB, and it is also possible that an MSB reset could occur to any predetermined value that is not equal to zero but is smaller than the current MSB at each bitstream position.

[0125] Conventional techniques can provide support for mixing different RAP type pictures within a layered access unit by signaling the top-level POC (Picture Order Count) bit (poc_msb_val), i.e., for time instances in which the access unit simultaneously contains both RAP and non-RAP pictures within its layer. However, problems remain with reference pictures of the leading picture NAL unit type, which involve references to pictures preceding their associated IRAP pictures in encoding order, as illustrated below by the example of Long Term Reference Pictures, but are also applicable to Short Term Reference Pictures, and for which the present invention provides a solution.

[0126] The current syntax for POC signaling is as follows: [Table 6] [Table 7]

[0127] An exemplary use of the POC MSB signaling described above is to set the POC MSB to 0 when all layers have the same IRAP NAL unit type, for example, when all layers are IDR_W_RADL or CRA, as shown in Figure 8.

[0128] Figure 8 shows a multi-layer bitstream 11, which consists of a first sub-bitstream 11-1 of the first layer L0 and a second sub-bitstream 11-2 of the second layer L1. Each layer, L0, may have one or more temporal sublayers T0, T1, T2. Thus, the spatial segments of a picture can be distributed across the different layers L0, L1 and temporal sublayers T0, T1, T2 described above.

[0129] Figure 8 further illustrates some exemplary multilayer access units 30, 31, and 32. In this example, the multilayer access units 30, 31, and 32 may comprise a first NAL unit 302 of the first sub-bitstream 11-1 in the first layer L0, which is mixed with a second NAL unit 304 of the second sub-bitstream 11-2 in the second layer L1. The NAL units may comprise different NAL unit types, particularly IRAP and non-IRAP NAL unit types, as described above. Furthermore, the NAL units may comprise a slice header to which the picture order count (POC) is signaled, as described above. The concepts described herein relate to the most significant POC portion, e.g., the most significant POC bits, or the POC MSB. Thus, in the figure, the POC MSB is shown for each IRAP NAL unit.

[0130] For example, the multilayer access units 30 and 32 represent access units where the POC MSB of the first NAL unit 302 in the first sub-bitstream 11-1 is aligned with the POC MSB of the second NAL unit 304 in the second sub-bitstream 11-2, and both NAL units are of IRAP NAL unit type.

[0131] The multilayer access unit 31 represents an unaligned access unit, i.e., the POC MSB of the first NAL unit 302 in the first subbitstream 11-1 is the POC MSB of the second NAL unit 304 in the second subbitstream 11-2, which is of the non-IRAP NAL unit type (e.g., MSB = 0), and thus represents an unaligned access unit. Therefore, the POC MSB of the first NAL unit 302 in the first subbitstream 11-1 can be increased by a predetermined value, for example, by 1.

[0132] Figure 8 illustrates how the aforementioned MSB signaling in the slice header can be used in such multi-layer cases. The purpose of the signaling is to correctly maintain the POC value in unaligned RAPs, such as in the unaligned access unit 31 of CRA w MSB=1, i.e., to avoid the POC (LSB and MSB) being reset to zero when not all layers have IRAP in the same access unit.

[0133] However, Figure 8 also shows that there are problems with the reference picture 12r (e.g., long-term and / or short-term) when the POC is aligned. For ease of understanding, the following figures may refer only to the long-term reference picture. However, this concept is also applicable to the short-term reference picture.

[0134] It should be noted that the pictures 12a, 12b, 12c, ... prior to the last CRA access unit 32, i.e., the second aligned RAP, have POC MSB > 0 and POC LSB > 0. Therefore, when resetting the MSB in the last CRA access unit 32 to 0 (as illustrated in Figure 8), it is impossible to refer to any of the previous pictures 12a, 12b, 12c, ... using the POC MSB or its delta (used for long-term reference pictures). One such example is shown in Figure 8 by the green arrow marked "LT ref". Thus, the example shown in Figure 8 shall demonstrate that picture referencing may be impossible in this case.

[0135] Therefore, the present invention provides a solution to this problem so that picture referencing is possible.

[0136] In one embodiment, the constraint is expressed or indicated in the specification through bitstream flags (SPS, VPS, DPS) that allow the MSB to be set to 0 only when all pictures in the access unit are of type IDR, i.e., IDR_W_RADL or IDR_N_LP. Thereafter, when CRA is used, no POC reset occurs and LT referencing is permitted (see Figures 9 and 10 for examples).

[0137] As seen in Figure 9, the access unit 32 could be an aligned access unit 32 where both NAL units 302 and 304 are of IRAP NAL unit type. However, since the first NAL unit 302 of the first sub-bitstream 11-1 is of CRA type, a POC reset does not occur. In other words, Figure 9 shows a scenario in which a POC reset is not permitted.

[0138] Instead, Figure 10 shows a scenario in which a POC reset is permitted because both the first NAL unit 302 of the first sub-bitstream 11-1 and the second NAL unit 304 of the second sub-bitstream 11-2 are of IDR unit type.

[0139] Accordingly, according to the embodiment, a decoder 100 is provided for decoding video content 12, the decoder 100 is configured to decode a bitstream 11 comprising a first subbitstream 11-1 and a second subbitstream 11-2, the decoder 100 is configured to derive from slice headers of NAL units 302, 304 of the first and second subbitstreams 11-1, 11-2, where the NAL units 302, 304 are IRAP NAL unit types (e.g., CRA, IDR_W_LP, IDR_N_LP), information for setting the top-level POC portion (POC MSB) of the POC of the picture 12r referenced by the NAL units 302, 304. According to this example, the top-level POC portion (POC MSB) needs to be set to a predetermined value for all NAL units 302, 304 of the IRAP NAL unit type within the access unit 32 of the bitstream 11, if at least one of the NAL units 302, 304 within the access unit 32 is of a non-instantaneous decoding refresh unit type, such as a CRA unit type.

[0140] As described above, this subsection can describe a POC MSB reset by setting the POC value to zero, i.e., by setting MSB = 0, as a non-restrictive example. However, the value zero acts only as a placeholder symbolizing a decreasing value for the MSB, and it is also possible that an MSB reset could occur to any predetermined value that is not equal to zero but is smaller than the current MSB at each bitstream position. Thus, the predetermined value described above may be greater than or equal to the top-level POC portion of the POC associated with the pictures 12a, 12b, ... preceding picture 12r referenced by the NAL unit 302. In other words, if all NAL units 302, 304 within the access unit 32 are not of IDR unit type, no POC reset will occur.

[0141] In some examples, the predetermined value described above may be equal to zero, i.e., MSB = 0. In some other examples, the predetermined value described above may be equal to the implicitly derived top-level POC portion, for example, the implicit derivation of the normal SOTA POC MSB.

[0142] In another embodiment, there may be constraints, expressed in the specification or indicated via bitstream flags (SPS, VPS, DPS), that whenever an aligned RAP access unit 32 occurs and the POC MSB of the access unit 32 is set to 0, that the reference pictures 12a, 12b, 12c, ... preceding the aligned RAP access unit 32 (long-term or short-term) are permitted to be referenced by the picture 12s following the aligned RAP access unit 32 in bitstream order.

[0143] For example, Figure 11 shows that picture 12s may precede picture 12r in presentation order, but picture 12s may follow picture 12r in decoding order.

[0144] Figures 11 and 12 show that any picture 12r with an MSB equal to 0 does not have reading pictures 12a, 12b, 12c, ... that have an LT reference picture (whereas Figure 11 represents a reset from MSB = 1 to MSB = 0, and Figure 12 represents an MSB equal to 0, for example, because POC_LSB is long enough that no increase occurs). Thus, Figure 11 shows a scenario where POC MSB reset and picture referencing are not allowed, and Figure 12 shows a scenario where all POC MSBs are equal to zero but picture referencing is not allowed. Figure 13 shows that if the MSB is not equal to 0, LT reference pictures are allowed for reading pictures.

[0145] Accordingly, according to the embodiment, a decoder 100 is provided for decoding video content 12, the decoder 100 is configured to decode bitstream 11 including a first subbitstream 11-1 and a second subbitstream 11-2, the decoder 100 is configured to derive from slice headers of NAL units 302, 304 of the first and second subbitstreams 11-1, 11-2, where the NAL units 302, 304 are IRAP NAL unit types (e.g., CRA, IDR_W_LP, IDR_N_LP) and information for setting the top-level POC portion (POC MSB) of the picture 12r referenced by the NAL units 302, 304. According to this example, no arbitrary picture reference is permitted to intersect with any access unit 32 of bitstream 11 having slice headers where all NAL units 302, 304 are IRAP NAL unit types and the top-level POC portion (POC MSB) is set to a predetermined value in the decoding order.

[0146] Again, as mentioned above, this subsection can illustrate a POC MSB reset by setting the POC value to zero, i.e., by setting MSB = 0, as a non-restrictive example. However, the value zero acts only as a placeholder symbolizing a decreasing value for the MSB, and it is also possible that an MSB reset could occur at any predetermined value that is not equal to zero but is smaller than the current MSB at each bitstream position. Thus, the predetermined value in the example described with reference to Figures 11-13 may be smaller than the top-level POC portion (POC MSB) of the POC associated with the picture (12a, 12b, 12c, ...) preceding the picture (12r) referenced by the NAL units (302, 304), i.e., the value at which the POC reset occurred.

[0147] In some examples, the predetermined value described above may be equal to zero, i.e., MSB = 0. In some other examples, the predetermined value described above may be equal to the implicitly derived top-level POC portion, e.g., the implicit derivation of the normal SOTA POC MSB.

[0148] In another embodiment, there may be a constraint, as expressed in the specification or indicated through bitstream flags (SPS, VPS, DPS), that whenever an aligned RAP access unit 32 occurs and the POC MSB of the access unit 32 is set to 0 (POC reset), that reference pictures 12a, 12b, 12c, ... (short-term or long-term) having an MSB different from 0 before the aligned RAP access unit 32 are allowed to be referenced in bitstream order by the picture 12s that follows the aligned RAP access unit 32.

[0149] Since the LT reference picture is referenced solely by the POC_LSB value, please note that in the example shown in Figure 8 above, if there is no MSB increase, using the LT reference picture will not cause any problems.

[0150] Figure 14 shows that no POC MSB reset occurred in this example. A POC MSB reset cannot be indicated by either increasing the POC MSB value of access unit 32 (in the example shown in Figure 14, the POC MSB value is increased, i.e., MSB = 3) or by leaving the POC MSB value of access unit 32 the same as that of the previous access unit 31.

[0151] Accordingly, according to one embodiment, a decoder 100 is provided for decoding video content 12, the decoder 100 is configured to decode a bitstream 11 comprising a first subbitstream 11-1 and a second subbitstream 11-2, the decoder 100 is configured to derive from slice headers of NAL units 302, 304 of the first and second subbitstreams 11-1, 11-2, where the NAL units 302, 304 are IRAP NAL unit types (e.g., CRA, IDR_W_LP, IDR_N_LP), information for setting the top-level POC portion (POC MSB) of the POC of the picture 12r referenced by the NAL units 302, 304. In this example, any picture reference of reference pictures 12a, 12b, 12c, ... of a first access unit 31 consisting of NAL units 301, 303 having a slice header preceding the bitstream order and with the topmost POC portion set to a first predetermined value is not permitted by picture 12s following the second access unit 32 of bitstream 11 in the bitstream order, where all NAL units 302, 304 are of IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP) and the topmost POC portion (POC MSB) has a slice header set to a second predetermined value.

[0152] Again, as mentioned above, this subsection can describe a POC MSB reset by setting the POC value to zero, i.e., by setting MSB = 0, as a non-restrictive example. However, the value zero acts only as a placeholder symbolizing a decreasing value for the MSB, and it is also possible that an MSB reset could occur at any predetermined value that is not equal to zero but is smaller than the current MSB at each bitstream position. Therefore, with respect to the example described with reference to Figure 14, the first predetermined value of the POC MSB of the preceding first access unit 31 may be greater than the second predetermined value of the POC MSB of the succeeding second access unit 32. In this case, picture referencing is not permitted. Conversely, if the second predetermined value of the MSB of the succeeding second access unit 32 is greater than the first predetermined value of the POC MSB of the preceding first access unit 31, then picture referencing is permitted. In other words, if no POC reset occurs, picture referencing is permitted.

[0153] For example, the second predetermined value may be equal to zero to indicate a POC reset. Therefore, if a POC reset occurs, picture referencing is not permitted. In some other examples, the second predetermined value described above may be equal to an implicitly derived top-level POC portion, e.g., the implicit derivation of a normal SOTA POC MSB.

[0154] While several embodiments have been described in the context of apparatus, these embodiments also represent descriptions of corresponding methods, and it is clear that blocks or devices correspond to method steps or features of method steps. Similarly, embodiments described in the context of method steps also represent descriptions of corresponding blocks, items, or features of the corresponding apparatus.

[0155] Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0156] Depending on specific implementation requirements, embodiments of the present invention may be implemented in hardware, in software, or at least partially in hardware or at least partially in software. Implementations may be carried out using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH® memory, which stores electronically readable control signals that cooperate (or can cooperate) with a programmable computer system to carry out each method. Thus, the digital storage medium may be computer-readable.

[0157] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system so that one of the methods described herein is performed.

[0158] In general, embodiments of the present invention can be implemented as a computer program product having program code, the program code operates to perform one of the methods when the computer program product runs on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0159] Other embodiments include a computer program stored in a machine-readable carrier for performing one of the methods described herein.

[0160] In other words, one embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

[0161] Accordingly, a further embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) recording a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-temporary.

[0162] Therefore, a further embodiment of the method of the present invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0163] Further embodiments include processing means, such as a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0164] Further embodiments include a computer on which a computer program for performing one of the methods described herein is installed.

[0165] Further embodiments of the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0166] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field-programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0167] The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0168] The methods described herein may be performed using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0169] This disclosure has been described with reference to exemplary embodiments, but this description is not intended to be constrained. Various modifications and combinations of the exemplary embodiments, as well as other embodiments of this disclosure, will become apparent to those skilled in the art by reference to the description. Accordingly, the appended claims are intended to encompass any such modifications or embodiments.

Claims

1. A decoder for decoding video content, Equipped with a processor, The aforementioned processor, A bitstream comprising a first subbitstream associated with a first spatial segment of the encoded picture of the video content and a second subbitstream associated with a second spatial segment of the encoded picture of the video content, wherein each network abstraction layer (NAL) unit of the encoded picture is configured to decode the bitstream, and is of a non-intra random access point (IRAP) NAL unit type. The value of a general constraint flag, which indicates whether mixing of NAL unit types in the encoded picture is permitted, is decoded. Based in part on the value of the general constraint flag, it is configured to identify that at least one first NAL unit of the first sub-bitstream is mixed with at least one different second NAL unit of the second sub-bitstream. The non-IRAP NAL unit type of the first NAL unit of the first sub-bitstream is a random access decodeable reading unit type (RADL), The non-IRAP NAL unit type of the second NAL unit in the second sub-bitstream is a trailing picture unit type (TRAIL). decoder.

2. The aforementioned processor, If the aforementioned general constraint flag indicates that mixing of NAL unit types in the picture is not permitted, then it is determined that there are no mixed NAL unit types in the picture. If the aforementioned general constraint flag indicates that mixing of NAL unit types within the picture is permitted, then it is determined that the picture is permitted to contain mixed NAL unit types. It is configured in such a way. The decoder according to claim 1.

3. A method for decrypting video content, A bitstream comprising a first sub-bitstream associated with a first spatial segment of the encoded picture of the video content, and a second sub-bitstream associated with a second spatial segment of the encoded picture of the video content, wherein each network abstraction layer (NAL) unit of the encoded picture is of a non-intra random access point (IRAP) NAL unit type, for decoding the bitstream. Decoding a general constraint flag value that indicates whether mixing of NAL unit types in the encoded picture is permitted, Based in part on the values ​​of the general constraint flags, it is identified that at least one first NAL unit of the first sub-bitstream is mixed with at least one different second NAL unit of the second sub-bitstream, Includes, The non-IRAP NAL unit type of the first NAL unit of the first sub-bitstream is a random access decodeable reading unit type (RADL), The non-IRAP NAL unit type of the second NAL unit in the second sub-bitstream is a trailing picture unit type (TRAIL). method.

4. The aforementioned method, If the aforementioned general constraint flag indicates that mixing of NAL unit types in the picture is not permitted, then it is determined that there are no mixed NAL unit types in the picture. If the aforementioned general constraint flag indicates that mixing of NAL unit types in the picture is permitted, then it is determined that the picture is permitted to have mixed NAL unit types. including, The method according to claim 3.

5. The general constraint flag further indicates that the encoded picture includes at least two subpictures. The decoder according to claim 1.

6. The general constraint flag further indicates that the encoded picture includes at least two subpictures. The method according to claim 3.

Images