Decoder, encoder, and method for mixing NAL units of different NAL unit types in video stream

Abstract

To provide a decoder, an encoder, and a corresponding method for processing a video data stream (11) including a first sub-bitstream (11-1) and a second sub-bitstream (11-2).SOLUTION: In a solution for mixing different network abstraction layer (NAL) unit types within an access unit, a non-intra random access point (IRAP) NAL unit type is mixed with a different IRAP NAL unit type or a non-IRAP NAL unit type, and a non-IRAP NAL unit type is mixed with a different non-IRAP NAL unit type.SELECTED DRAWING: None

Description

Technical Field

[0001] Embodiments of the present disclosure relate to decoders and corresponding encoders for decoding / encoding video streams, and more particularly to video streams comprising at least two video sub-streams. The video stream may comprise a plurality of access units, each access unit may comprise at least one picture included in each of the two or more video sub-streams. Each picture may be associated with a NAL (Network Abstraction Layer) unit of a particular NAL unit type. The innovative concept described herein introduces a solution for mixing NAL units of different NAL unit types.

Background Art

[0002] NAL units can be classified into VCL (Video Coding Layer) and non-VCL NAL units according to whether they contain encoded pictures or other related data, respectively. The HEVC standard includes several VCL NAL unit types that identify the category of pictures for decoder initialization and random access purposes. Using the contents of the 2-byte NAL unit header, the purpose of the associated payload data can be easily identified.

[0003] Video coding techniques enable the combination of random access and bitstreams. Regarding random access, the bitstream can start with an IDR (Instantaneous Decoding Refresh) access unit. An IDR access unit contains an encoded picture that can be decoded without decoding previous pictures in the NAL unit stream, i.e., an independently encoded picture. The presence of an IDR access unit indicates that subsequent pictures in the bitstream do not require a reference to a picture before the picture it contains for it to be decoded. IDR pictures are used within an encoding structure known as a closed GOP (Group Of Pictures). As an alternative to the above IDR syntax, there is the use of independently encoded pictures at the position of a RAP or IRAP ((Intra)Random Access Point), i.e., a CRA (Clean Random Access) picture syntax that specifies a position within the bitstream. This syntax specifies a position within the bitstream where the decoder can start to normally decode a picture without the need to decode pictures that appear earlier in the bitstream, supporting an efficient temporal encoding order known as open GOP operation.

[0004] Good support for random access can be important to enable channel switching, seek operations, and dynamic streaming services. Some pictures that follow a CRA picture in decoding order and precede it in display order may include interpicture prediction references to pictures that are not available in the decoder. Thus, these non-decodable pictures may be discarded by a decoder that starts the decoding process at a CRA point. For this purpose, such non-decodable pictures may be identified as RASL pictures (Random Access Skipped Leading). An IRAP picture can be an IDR or CRA picture, and RASL pictures in the bitstream may follow a CRA picture. Since RASL pictures may include references to pictures that do not actually exist in the bitstream, for example, for splicing operations, they may be discarded by the decoder.

[0005] A further type of picture that can follow an IRAP picture in decoding order and precede it in output order is a RADL picture (Random Access Decordable Leading), which may not include references to any picture that precedes the IRAP picture in decoding order. RASL and RADL pictures are sometimes collectively referred to as Leading Pictures (LP). Pictures that follow an IRAP picture in both decoding order and output order are known as trailing pictures (TRAIL). They may not include any references to LPs for interpicture prediction.

[0006] In today's video coding, Temporal Sublayering Support can also be provided. Therefore, a temporal identifier indicating the level in the hierarchical temporal prediction structure can be specified in the NAL unit header. This helps to achieve temporal scalability without the need to analyze parts of the bitstream other than the NAL unit header. Under certain circumstances, the number of decoded temporal sublayers can be adjusted during the decoding of one coded video sequence. The position in the bitstream where the switching of sublayers can start decoding some of the higher temporal layers can be indicated by the presence of STSA pictures (Stepwise Temporal Sublayer Access). At the position of an STSA picture, it may be possible to switch from decoding one particular lower temporal sublayer to decoding one higher temporal sublayer (however, further layers above are not so unless they also include STSA pictures).

[0007] In the case of managing multiple reference pictures, a specific set of previously decoded pictures may be present in the DPB (Decoded Picture Buffer) for decoding the remaining parts of the pictures in the bitstream. To identify these pictures, a list of POC (Picture Order Count) identifiers may be sent in each slice header. The set of reference pictures to be retained is called the RPS (Reference Picture Set). The POC may include the least significant bit part (LSB) and the most significant bit part (MSB) for indicating the respective current picture order count (POC value) for each picture in the bitstream, and the picture order may be different from the decoding order of each picture. Therefore, decoding a picture after an IRAP picture, which refers to the reference pictures before the IRAP picture, can be difficult.

[0008] Therefore, it is desirable to improve existing encoders and decoders that can provide appropriate processing of pictures and picture sequences in the bitstream, particularly when they appear after an IRAP picture.

[0009] Therefore, it is proposed to provide a decoder having at least the features of the independent claims and a corresponding apparatus for encoding and / or merging having at least the features of the independent claims. Furthermore, it is proposed to provide corresponding methods for encoding and decoding, as well as corresponding computer program products and bitstreams. Advantageous embodiments are presented in the dependent claims.

[0010] As described above, the bitstream may comprise a picture associated with an IRAP NAL unit type, which picture may also be referred to as an IRAP picture, and pictures not associated with an IRAP NAL unit type, i.e., pictures associated with a non-IRAP NAL unit type, may also be referred to as non-IRAP pictures. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0011] According to one aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream including a first sub-bitstream associated with a first spatial segment (131) of a picture of the video content and a second sub-bitstream associated with a second spatial segment (132) of the picture of the video content. In this case, the bitstream includes at least one access unit in which at least one first NAL unit of the first sub-bitstream is a non-IRAP NAL unit type, such as RASL, RADL, TRAIL, STSA. The at least one access unit further includes at least one different second NAL unit of the second sub-bitstream that is a non-IRAP NAL unit type, such as RASL, RADL, TRAIL, STSA, and the first NAL unit of the first sub-bitstream and the second NAL unit of the second sub-bitstream are mixed. In other words, non-IRAP NAL units are mixed with different non-IRAP NAL units. For example, a NAL unit of the TRAIL picture NAL unit type can be mixed with a NAL unit of the RASL picture NAL unit type. According to this non-limiting example, within one access unit, a TRAIL picture of the first sub-bitstream can be mixed with a RASL picture of the second sub-bitstream. Other non-limiting examples can enable mixing a TRAIL picture of the first sub-bitstream with a RADL picture of the second sub-bitstream within one access unit. Some further non-limiting examples can enable mixing a TRAIL picture of the first sub-bitstream with an STSA picture of the second sub-bitstream within one access unit. Some further non-limiting examples can enable mixing an STSA picture of the first sub-bitstream with a RADL picture of the second sub-bitstream within one access unit. Some further non-limiting examples can enable mixing an STSA picture of the first sub-bitstream with a RASL picture of the second sub-bitstream within one access unit.Some further non-limiting examples may enable mixing of a RASL picture of a first sub-bitstream with a RADL picture of a second sub-bitstream within one access unit.

[0012] According to a further aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream comprising a first sub-bitstream associated with a first spatial segment of a picture of the video content and a second sub-bitstream associated with a second spatial segment of a picture of the video content. In this case, the bitstream comprises at least one access unit of an IRAP NAL unit type in which at least one first NAL unit of the first sub-bitstream is mixed with at least one second NAL unit of the second sub-bitstream which is a further IRAP NAL unit type. In other words, a first IRAP NAL unit is mixed with a further second IRAP NAL unit. The further second IRAP NAL unit may be of the same NAL unit type as the first IRAP NAL unit or may be of a different NAL unit type from the first IRAP NAL unit. For example, a NAL unit of a CRA picture NAL unit type may be mixed with a further picture of the same NAL unit type, for example a further CRA picture NAL, or with a further picture of a different NAL unit type, for example a NAL unit of an IDR picture NAL unit type having a previous picture, i.e., an IDR-with-RADL picture unit type (IDR_W_RADL). According to this non-limiting example, within one access unit, a CRA picture of a first sub-bitstream can be mixed with a further CRA picture of a second sub-bitstream or an IDR_W_RADL picture of a second sub-bitstream.

[0013] According to a further aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream comprising a first sub-bitstream associated with a first spatial segment of a picture of the video content and a second sub-bitstream associated with a second spatial segment of a picture of the video content. In this case, the bitstream comprises at least one access unit of an IRAP NAL unit type in which at least one first NAL unit of the first sub-bitstream is mixed with at least one second NAL unit of the second sub-bitstream which is of a non-IRAP reading picture NAL unit type (e.g., RASL or RADL) or an STSA NAL unit type. In other words, within one access unit, an IRAP NAL unit (IDR_W_RADL or CRA) is mixed with non-IRAP NAL units, and the non-IRAP NAL units are any one of a RASL picture, a RADL picture, or an STSA picture.

[0014] According to a further aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream including a first sub-bitstream and a second sub-bitstream, the decoder being configured to derive from slice headers of NAL units of the first and second sub-bitstreams, the NAL units being of an IRAP NAL unit type (e.g., CRA, IDR_W_RADL), and being information for setting a most significant POC part of the POC of the picture referred to by the NAL unit. In this example, the most significant POC part needs to be set to a predetermined value for all NAL units of the IRAP NAL unit type within an access unit of the bitstream when at least one NAL unit within the access unit is of a non-instantaneous decoding refresh (non-IDR) unit type. In other words, when at least one NAL unit within the access unit refers to a non-IDR picture, e.g., a CRA picture, the most significant POC part needs to be set to a predetermined value. As a non-limiting example, the predetermined value can be greater than or equal to the most significant POC part of the POC associated with the picture preceding the picture referred to by the NAL unit. In other words, when a non-IDR picture is present within an access unit, a POC reset may not occur.

[0015] According to a further aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream including a first sub-bitstream and a second sub-bitstream, the decoder being configured to derive from slice headers of NAL units of the first and second sub-bitstreams, the NAL units being of an IRAP NAL unit type (e.g., CRA, IDR_W_RADL), and being information for setting the most significant POC part of the POC of the picture referred to by the NAL unit. In this example, any picture reference is not permitted to cross any access unit of the bitstream in the decoding order where all NAL units are of the IRAP NAL unit type and have a slice header in which the most significant POC part is set to a predetermined value. In other words, if each picture within an access unit can refer to an IRAP NAL unit type (e.g., CRA or IDR_W_RADL) and can have a most significant POC part set to a predetermined value, any picture that refers from a reference picture preceding the access unit is not permitted. As a non-limiting example, the predetermined value may be smaller than the most significant POC part of the POC associated with the picture preceding the picture referred to by the NAL unit. The predetermined value may be smaller, for example, when a reset of the most significant POC part occurs. Thus, according to the above non-limiting example, there may be a case where an access unit includes only IRAP pictures, and when a reset of the most significant POC part occurs, picture reference by referring to a preceding reference picture may not be permitted.

[0016] According to a further aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream including a first sub-bitstream and a second sub-bitstream, the decoder being configured to derive from slice headers of NAL units of the first and second sub-bitstreams, the NAL units being of an IRAP NAL unit type (e.g., CRA, IDR_W_RADL), and being information for setting a most significant POC portion of a POC of a picture referred to by the NAL unit. In this example, the most significant POC portion is set to a first predetermined value, and in bitstream order, for all NAL units of an IRAP NAL unit type, it is not permitted for a reference picture of a first access unit having a slice header that precedes a second access unit of the bitstream having a slice header in which the most significant POC portion is set to a second predetermined value to be referred to by a picture following the second access unit of the bitstream. According to a non-limiting example, the first predetermined value may be greater than the second predetermined value. In other words, if the first predetermined value of the most significant POC portion of the (preceding) first access unit is greater than the second predetermined value of the most significant POC portion of the second access unit, it may not be permitted for a picture following the second access unit to refer to a reference picture preceding the second access unit.

[0017] According to a further aspect, respective corresponding encoders, a method for encoding, and a method for decoding are proposed.

[0018] According to a further aspect, a computer program is provided, where each of the computer programs is configured to implement one of the methods described herein when executed on a computer or a signal processor, whereby the methods described herein are implemented by one of the computer programs.

Brief Description of the Drawings

[0019] Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the drawings.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

[0020] Elements that are equivalent or have equivalent elements or equivalent functions are denoted by equivalent or equivalent reference numerals in the following description.

[0021] Method steps depicted by a block diagram and described with reference to the block diagram may also be executed in an order different from the order in which they are depicted and / or described. Further, method steps regarding specific features of the apparatus may be replaceable with the features of the apparatus or may be other methods.

[0022] Mixing of NAL unit types in pictures and access units 1. Mixed unit types within an encoded picture FIG. 1 shows, for introduction purposes, the structure of an exemplary video bitstream 11 that can be processed by a decoder 100 and an apparatus 10 for encoding and / or merging a video bitstream 11 according to an embodiment of the present invention. The video bitstream 11 comprises a sequence of consecutive pictures 12a, 12b, 12c, and two or more of the consecutive pictures 12a, 12b, 12c can be grouped together in a GOP (Group of Pictures) 12 that can represent at least a part of the intended video content.

[0023] For coding, each picture 12a, 12b, 12c can be subdivided into spatial segments, such as slices, tiles, or blocks. In this non-limiting example, pictures 12a, 12b, 12c are each subdivided into two spatial segments, namely a first (e.g., left) spatial segment 131 and a second (e.g., right) spatial segment 132.

[0024] The bitstream 11 can be subdivided into a first sub-bitstream 11-1 related to the first spatial segment 131 of pictures 12a, 12b, 12c and a second sub-bitstream 11-2 related to the second spatial segment 132 of pictures 12a, 12b, 12c.

[0025] Some of the pictures 12a, 12b can be encoded dependently, for example by predictive coding. Some of these pictures 12b can be predicted by a single prediction, for example by referring to a single previous picture, while some other pictures 12a can be predicted by dual prediction, for example by referring to both a previous picture and a subsequent picture. Some other pictures 12c can be encoded independently, i.e., these pictures 12c cannot refer to previous or subsequent pictures. The independently encoded pictures 12c can provide an (Intra) Random Access Point ((I)RAP) that represents a position in the bitstream 11, which allows the decoder to start decoding subsequent pictures without having to decode earlier pictures in the bitstream 11. For example, picture 12c is an independently encoded picture that can also be called an (I)RAP picture. In the present disclosure, the terms RAP and IRAP can be used synonymously.

[0026] The bitstream 11 can comprise one or more access units 30 for accessing the bitstream 11. In the non-limiting example of FIG. 1, the access unit 30 can align the picture segments 131, 132 included in the first and second sub-bitstreams 11-1, 11-2 (with respect to the presentation order). The pictures 12a, 12b, 12c including these aligned picture segments 131, 132 can each be grouped into different categories, for example, (as described above) IRAP pictures and non-IRAP pictures.

[0027] The Network Abstraction Layer (NAL) can provide High Level Syntax for defining those picture segments 131, 132 and pictures 12a, 12b, 12c. In particular, the NAL unit may be associated with the picture segments 131, 132 for each of the pictures 12a, 12b, 12c. Thus, in the present disclosure, the above-described picture segments 131, 132, and the first and second sub-bitstreams 11-1, 11-2 can be called synonymous with the NAL unit. Thus, the first and second sub-bitstreams 11-1, 11-2 are also called NAL unit streams. Each of the NAL units 131, 132 can include a specific NAL unit type. The NAL unit type can then identify the aforementioned category of each picture included in the NAL unit, i.e., whether each NAL unit is of the IRAP type or the non-IRAP type.

[0028] There are multiple applications that depend on mixing IRAP types within a picture, and the IRAP type is the random access property of an access unit (AU) 30 or a part thereof indicated through the NAL unit type. The use case can be either a single-layer or multi-layer codec domain, i.e., a single-layer codec bitstream composed of slices where the picture is IRAP while other slices of the same picture are not, or a multi-layer codec bitstream where the access unit includes IRAP in one layer and the other layer includes non-IRAP pictures.

[0029] Currently, support for mixed NAL unit types is given in the prior art where a single type of IRAP NAL unit type can be mixed within a picture together with TRAIL pictures. However, this does not cover use cases that properly motivate the need for features such as those provided by the present invention as described below.

[0030] The main use case that motivates the need for a mixed NAL unit type is 360-degree video, where tile streaming is used and only a subset of the entire 360 degrees is transmitted at high resolution and the rest at low resolution. When a change in the viewing direction occurs, some of the high-resolution tiles are no longer needed, but new tiles shown at low resolution will be needed at high resolution from this time forward. These tiles are the only tiles that encounter the change (from high resolution to low resolution and vice versa). These tiles are downloaded using an expression starting with an IRAP, but this property is not required for other tiles. When merging bitstreams corresponding to different tiles into a single bitstream, some of the tiles contain NAL unit types corresponding to IRAP pictures and others contain NAL unit types of non-IRAP pictures.

[0031] Furthermore, there are several other use cases where it is desirable to enable an AU to have different types of NAL units. For example, in a scenario where there is a 1080p picture with an RoI (Region of Interest) inside 720p within the RoI, it can be assumed that different parts of the video (RoI or otherwise) have different RAP periods. For example, when the same bitstream is used to feed two receivers, one is interested in the whole video and the other is only interested in the RoI.

[0032] In either the case of 360-degree video streaming where merging of different bitstreams occurs, or RoI streaming where the video is originally encoded in different parts with different RAP periods for different regions, the decoder will encounter bitstreams where the NAL units within the encoded pictures are not necessarily the same. Therefore, the present invention provides a solution for mixing NAL units of different NAL unit types.

[0033] 1.1 Mixing of Trailing Pictures (Prior Art) In the prior art, the currently proposed solution is that the PPS flag indicates whether the RAP picture is mixed with non - RAP pictures, and then the NAL unit types that can be encountered in the NAL unit of the mixed AU are TRAIL NAL unit types (up to VCL_RSV_6 or GDR) for non - IRAP types, and a single type of IDR_W_RADL (IDR with Leading Pictures), IDR_N_LP (IDR with No Leading Pictures), and CRA (Clean Random Access). Therefore, the prior art suggests mixing IRAP pictures with TRAIL pictures.

[0034] Figure 2 shows an example according to the prior art, where the NAL unit 201 of the IRAP NAL unit type is mixed with the NAL unit 202 of the trailing picture type (TRAIL). The IRAP NAL unit type in this example is IDR_N_LP (IDR with No Leading Pictures). All other access units have NAL units of the same NAL unit type (i.e., TRAIL).

[0035] 1.2 Mixing of Leading Pictures However, these means are not sufficient for the assumed applications where the open GOP structure can be applied by introducing a leading picture (LP). Figure 3 shows an exemplary case where a NAL unit 301 of an IRAP NAL unit type (e.g., IDR_W_RADL) is shown together with its associated leading picture 302, and in this case, it can be a NAL unit 302 of the NAL unit type RADL (Random Access Decordable Leading Picture). The NAL unit 301 of the IRAP type (e.g., IDR_W_RADL) can be mixed with a NAL unit 303 of a non-IRAP NAL unit type (e.g., TRAIL). The associated NAL unit 302 of the leading picture type (RADL) which is of the non-IRAP type can be mixed with a NAL unit 304 of a different non-IRAP type (e.g., TRAIL).

[0036] Figure 4 shows a further exemplary case where a NAL unit 301 of an IRAP NAL unit type (e.g., CRA) is shown together with its associated leading picture 302, and in this case, it can be a NAL unit 302 of the NAL unit type RASL (Random Access skipped Leading Picture). Alternatively, although not shown, the associated leading picture 302 can be a NAL unit 302 of the NAL unit type RADL (Random Access Decordable Leading Picture). The NAL unit 301 of the IRAP type (e.g., CRA) can be mixed with a NAL unit 303 of a non-IRAP NAL unit type (e.g., TRAIL). The associated NAL unit 302 of the leading picture type (e.g., RASL) which is of the non-IRAP type can be mixed with a NAL unit 304 of a different non-IRAP type (e.g., TRAIL).

[0037] Therefore, it is necessary to handle coded pictures with mixed NAL unit types not only when an IRAP NAL unit is included, but also for non-IRAP NAL units included in a mixed picture, targeting only non-IRAP NAL units.

[0038] Briefly summarized, the prior art can propose mixing an IRAP NAL unit with a non-IRAP NAL unit of TRAIL type, or mixing non-IRAP NAL units of the same type, i.e., NAL units of TRAIL type with NAL units of TRAIL type.

[0039] Instead, the present invention proposes mixing a first non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA) with at least one different second non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA).

[0040] Accordingly, according to an embodiment, a decoder 100 is provided for decoding video content 12, and the decoder 100 is configured to decode a bitstream 11 including a first sub-bitstream 11-1 related to a first spatial segment 131 of pictures 12a, 12b, 12c of the video content 12 and a second sub-bitstream 11-2 related to a second spatial segment 132 of pictures 12a, 12b, 12c of the video content 12. According to this embodiment, the bitstream 11 includes at least one access unit 30 of a non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA) in which at least one first NAL unit 302 of the first sub-bitstream 11-1 is mixed with at least one different second NAL unit 304 of the second sub-bitstream 11-2 that is of a different non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA).

[0041] For example, a TRAIL picture can be mixed with a RADL picture or a RASL picture. ·TRAIL + RADL ·TRAIL + RASL

[0042] Thus, according to one embodiment, decoder 100 may be configured to process bitstream 11 in which the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access decoder readable picture unit type (RADL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a trailing picture unit type (TRAIL).

[0043] According to a further embodiment, decoder 100 may be configured to process bitstream 11 in which the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access skip readable picture unit type (RASL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a trailing picture unit type (TRAIL).

[0044] In one embodiment of the present invention, there is an indication as a flag in a parameter set (PPS, SPS, VPS) in bitstream 11 that can mix, for example, the NAL unit types within an encoded picture as follows. ·TRAIL + RADL ·TRAIL + RASL

[0045] Such signaling can be carried in the PPS to indicate that TRAIL+(CRA xor IDR_W_LD xor IDR_N_P) is mixable.

[0046] For example, when the flag is not set (i.e., flag = 0), all NAL units of a picture may have the same NAL unit type, and the picture or PU is said to have the same NAL unit type as the encoded slice NAL unit of the picture or PU.

[0047] Otherwise, when the flag is set (i.e., flag = 1), the picture may include at least two sub-pictures. Additionally or alternatively, the NAL units of the picture may have at least two different NAL unit types. Further additionally or alternatively, the NAL units of the picture may not have the NAL unit type of GDR. Further additionally or alternatively, when one NAL unit of the picture is at least one of the following NAL unit types: · Instant decoder refresh by random access decode reading picture (IDR_W_RADL) · Instant decoder refresh without reading picture (IDR_N_LP) · Clean random access (CRA)

[0048] In that case, all other available NAL units of the picture may have one of the above three unit types (IDR_W_RADL, IDR_N_LP, CRA) or the trailing picture unit type (TRAIL).

[0049] Alternatively, since RADL pictures and RASL pictures do not have a canonical decoding process different from that of, for example, IRAP pictures, this indication can be done at the SPS level. In one embodiment, for example, the indication can be in the form of a constraint flag, for example, a general constraint flag. The general constraint flag may indicate whether a particular mix of NAL unit types is permitted. When such a general constraint flag is set (i.e., general constraint flag = 1), it may indicate that there is no mixing of NAL unit types within the picture and that the above flags within the parameter sets (PPS, SPS, VPS) are not set (i.e., flag = 0). When the general constraint flag is not set (i.e., general constraint flag = 0), the above constraints may not be imposed.

[0050] Thus, according to one embodiment, decoder 100 may be configured to derive an indication from bitstream 11 that explicitly informs decoder 100 that the NAL unit 304 of the non-IRAP NAL unit type in the second sub-bitstream 11-2 (this NAL unit 304 is a trailing picture unit type (TRAIL)) is mixed with the NAL unit 302 of the non-IRAP NAL unit type in the first sub-bitstream 11-1, and this NAL unit 302 is either a random access skip reading picture unit type (RASL) or a random access decoderable reading picture unit type (RADL).

[0051] According to one embodiment, the indication may comprise a syntax element in the parameter set. For example, the syntax element may be included in at least one of a Picture Parameter Set (PPS) or a Sequence Parameter Set (SPS).

[0052] However, assuming a 360-degree video tile-based streaming scenario, when tiles with RASL properties and tiles of TRAIL pictures are mixed within a picture, the tiles change position at partial RAP points, and thus, the reference pictures used are incorrect (i.e., still depicting the content of other tiles), which may cause the RASL tiles not to be correctly reconstructed. For an application, it is extremely important to recognize these defective reconstructed picture regions. On the other hand, dropping the entire mixed leading picture (including TRAIL+RASL tiles) is not an option for reasons of playout continuity. Instead, the client recognizes the defective regions within the decoded and output picture and does not use them for subsequent processing.

[0053] Instead, for example, when a setting as shown in FIG. 5 is given, the client can use the low-resolution version 51 included in the picture for the content that is inaccurately reconstructed within those mixed leading pictures. In one embodiment, the regions corresponding to potentially defective reconstructed tiles are indicated to subsequent processing, for example, through an SEI message or by external means, such as an application. The instructions described mean that the regions corresponding to NAL units with the NAL unit type of RASL are ignored / not output.

[0054] Thus, according to one embodiment, when the NAL unit 302 of the non-IRAP NAL unit type in the first sub-bitstream 11-1 is of the random access skip leading picture unit type (RASL) and the non-IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is of the trailing picture unit type (TRAIL), the decoder 100 may be configured to derive an instruction from the bitstream 11.

[0055] In one example, this instruction can be directed to the decoder 100 to ignore or not output the first spatial segment 131 in pictures 12a, 12b, 12c of the video content 12 of the first sub-bitstream 11-1 corresponding to the NAL unit 302 of the random access skip reading picture unit type of the first sub-bitstream 11-1.

[0056] In another example, this instruction can be directed to the decoder 100 to output the first spatial segment 131 of pictures 12a, 12b, 12c of the video 12 content of the first sub-bitstream 11-1 corresponding to the NAL unit 302 of the random access skip read picture unit type (RASL) of the first sub-bitstream 11-1, and further, to instruct to mark the first spatial segment 131 as a damaged spatial segment. For example, the decoder 100 may be configured to mark the above-mentioned damaged spatial segment 13 for the subsequent processing chain by means of an SEI (Supplemental Enhancement information) message.

[0057] Furthermore, another option is to avoid the RASL picture being completely mixed with the trailing picture (TRAIL), and thus only allow the RADL picture to be mixed with the TRAIL picture. Therefore, in one embodiment, there are constraints shown in the SPS, and when the CRA is mixed with the TRAIL picture, the related reading picture must be of the RADL type only.

[0058] Accordingly, according to one embodiment, decoder 100 may be configured to decode a constraint instruction from a parameter set, where the above-described constraint instruction indicates to decoder 100 that the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a non-leading picture NAL unit type (e.g., TRAIL or STSA), and the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access decodable leading (RADL) picture unit type only when its associated IRAP NAL unit 301 of the first sub-bitstream 11-1 is a clean random access unit type (CRA).

[0059] This may particularly apply to embodiments where the non-IRAP NAL unit type of the NAL unit 304 of the second bitstream 11-2 is a trailing picture unit type (TRAIL), i.e., when mixing the associated RADL leading pictures with TRAIL + CRA.

[0060] In the previous aspect, the focus was mainly on the TRAIL NAL unit type. However, in many cases, there may also be an STSA NAL unit type, which enables step-wise temporal sublayer access. That is, while the above example mainly related to non-IRAP NAL units of the TRAIL NAL unit type, the present invention can also provide solutions for mixing non-IRAP NAL units of the STSA (Stepwise Temporal Sublayer Access) unit type with at least one of the RASL unit type, RADL unit type, and TRAIL unit type. ·STSA + RASL ·STSA + RADL ·STSA + TRAIL

[0061] Accordingly, according to one embodiment, decoder 100 may be configured to decode a bitstream 11 in which the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a stepwise temporal sublayer access unit type (STSA), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a trailing picture unit type (TRAIL).

[0062] In a further embodiment, decoder 100 may be configured to decode bitstream 11, where the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access decoder-decodable reading picture unit type (RADL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a stepwise temporal sublayer access unit type (STSA).

[0063] In a further embodiment, decoder 100 may be configured to decode a bitstream 11 in which the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access skip reading picture unit type (RASL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a stepwise temporal sublayer access unit type (STSA).

[0064] FIG. 6 shows an exemplary encoding structure in which in one picture segment 132 (e.g., tile), picture 12a performs stepwise temporal sublayer access using the NAL unit 304 of the STSA NAL unit type, while the same picture 12a in another picture segment 131 (e.g., tile) uses the NAL unit 302 of the RASL NAL unit type.

[0065] Since the STSA NAL unit type cannot be placed at the lowest temporal level 0, the STSA NAL unit does not mix with the IRAP NAL unit. However, these can mix with the RADL or RASL NAL unit type, similar to the TRAIL NAL unit described above.

[0066] In one embodiment, the flag is included in the parameter set, for example, in the PPS that indicates that the STSA NAL unit is mixed with the RASL or RADL NAL unit type.

[0067] Therefore, in one embodiment of the present invention, there is an instruction in the bitstream 11 that the NAL unit types within the coded picture can be mixed as follows, for example, as a flag within the parameter set (PPS, SPS, VPS). ·STSA + RADL ·STSA + RASL ·STSA + TRAIL

[0068] In a further embodiment, whether such mixing is permitted can be indicated, for example, using a constraint flag for the sequence level or picture level.

[0069] Therefore, according to one embodiment, the decoder 100 may be configured to derive an instruction from the bitstream 11, where the above instruction is that the NAL unit 304 of the non-IRAP NAL unit type in the second sub-bitstream 11-2, where the NAL unit 304 is a stepwise temporal sublayer access unit type (STSA), is mixed with the NAL unit 302 of the non-IRAP NAL unit type in the first sub-bitstream 11-1, and the above NAL unit 302 is explicitly informed to the decoder 100 that it is either a random access skip reading picture unit type (RASL) or a random access decodable reading picture unit type (RADL).

[0070] According to one embodiment, the indication includes syntax elements within a parameter set. According to a further embodiment, the syntax elements are included in at least one of a picture parameter set or a sequence parameter set.

[0071] Furthermore, an indication that effectively indicates that all NAL units 302, 304 of picture 12a including the STSA NAL unit 304 satisfy the constraints associated with the STSA NAL unit type, even if they have another NAL unit type, e.g., RASL (see, e.g., NAL unit 302), can be added to the bitstream 11. This can be indicated by a flag in the SPS indicating that a (spatial) subset 11-1 of the bitstream including RASL pictures (see, e.g., NAL unit 302) satisfies the constraint that each RASL picture is also an STSA picture. Such signaling can indicate whether the original bitstream including RASL or RADL NAL units satisfies the characteristics of STSA pictures for the result that when mixed with STSA, it is possible to easily derive whether the mixed-coded pictures have STSA characteristics, which is more important for the use case of merge.

[0072] Thus, according to one embodiment, decoder 100 may be configured to decode a constraint flag from the parameter set of bitstream 11, which indicates to decoder 100 that any access unit 30 of bitstream 11 comprising a mix of one or more NAL units 302 of a non-IRAP leading picture NAL unit type (e.g., RASL or RADL) and one or more NAL units 304 of a stepwise temporal sublayer access unit type (STSA) satisfies the constraints associated with the stepwise temporal sublayer access unit type (STSA), i.e., each RASL picture 302 may be treated as an STSA picture.

[0073] The above constraints can indicate that, for the decoder 100, any picture associated with a temporal sublayer higher than the stepwise temporal sublayer access (STSA) unit type, in decoding order, is not allowed to reference a picture within the above-mentioned next higher temporal sublayer if the above-mentioned next higher temporal sublayer precedes the above-mentioned stepwise temporal sublayer access unit type (STSA).

[0074] Alternatively, a bitstream 11 having an encoded picture 12a with a hybrid NAL unit type (e.g., STSA + RASL or RADL) may include a flag indicating that the picture 12a for which the flag is set may contain a NAL unit 304 of the STSA NAL unit type but does not (in total) conform to the constraints associated with the STSA NAL unit type. This indicates that the encoded picture 12a may have a NAL unit type equal to STSA, but the characteristics implied by STSA (i.e., that further temporal levels can be decoded from after its AU) do not apply. Thus, the syntax analysis of STSA is ignored and stepwise temporal sublayer access is assumed.

[0075] Thus, according to one embodiment, decoder 100 may be configured to decode a flag from a parameter set, the flag indicating to decoder 100 that any access unit 30 comprising a mix of one or more NAL units 302 of non-IRAP reading picture NAL unit type (RASL or RADL) and one or more NAL units 304 of stepwise temporal sublayer access unit type (STSA) does not comply with the constraints associated with the stepwise temporal sublayer access (STSA) unit type. Further, decoder 100 may be configured to refrain from parsing one or more NAL units 304 of stepwise temporal sublayer access (STSA) unit type in response to the flag described above and implicitly infer the decoding of stepwise temporal sublayer access, i.e., the STSA constraints are not applied to each access unit.

[0076] This concept may also be applied regardless of mixing NAL unit types as described above. Thus, according to one embodiment, a decoder 100 for decoding video content 12 is proposed, the decoder 100 being configured to decode a bitstream 11 and decode a constraint flag from a parameter set of the bitstream 11, the constraint flag described above indicating to decoder 100 that any access unit 30 comprising a NAL unit 302 of non-IRAP reading picture NAL unit type (e.g., RASL or RADL) within the bitstream 11 complies with the constraints associated with the stepwise temporal sublayer access (STSA) unit type.

[0077] 1.3 Mixing of Different RAP Types A further important case not enabled by the prior art is to enable mixing pictures with an open GOP (Group of Pictures) structure with pictures with a closed GOP structure. For example, mixing of two different types of IRAP NAL unit types (e.g., IDR, CRA), e.g., IDR_W_RADL and CRA NAL unit types, may be permitted, or mixing of NAL units of the same NAL unit type, e.g., CRA and CRA, may be permitted. ·IDR_W_RADL + CRA ·CRA + CRA

[0078] As described above, this embodiment is to enable mixing of an IRAP NAL unit of a NAL unit type related to an open GOP structure and an IRAP NAL unit of a NAL unit type related to a closed GOP structure. For example, a CRA with RADL may be associated with a closed GOP structure, while a CRA with RASL may be associated with an open GOP structure.

[0079] Accordingly, according to an embodiment, a decoder 100 for decoding video content 12 may be provided, and the decoder 100 is configured to decode a bitstream 11 including a first sub-bitstream 11-1 related to a first spatial segment 131 of pictures 12a, 12b, 12c of the video content 12 and a second sub-bitstream 11-2 related to a second spatial segment 132 of pictures 12a, 12b, 12c of the video content 12. According to this example, the bitstream 11 includes at least one access unit 30 of an IRAP NAL unit type in which at least one first NAL unit 302 of the first sub-bitstream 11-1 is mixed with at least one second NAL unit 304 of the second sub-bitstream 11-2 that is a further (same or different) IRAP NAL unit type.

[0080] There is at least one use case which is very useful. For example, one approach to 360-degree video transmission using tiled streaming is the case where the entire low-resolution video content is transmitted (regardless of whether a part of it is also transmitted as high-resolution tiles). In such a case, the change in the viewing direction and the corresponding change in the high-resolution tile selection from one viewport to another are shown in FIG. 5.

[0081] The entire low-resolution video content is always available on the client side, and since tile-by-tile stream switching is not required for this area, the low-resolution content can be encoded with CRA and thus higher encoding efficiency can be utilized compared to a closed GOP structure using IDR. Also, the longer the random access point period required for stream switching, the more sufficient it is for this area, which again benefits the encoding efficiency of the low-resolution video. On the other hand, the high-resolution tiles, as shown in FIG. 5, may change frequently during fast progression from one viewport to another, and thus, as in the case of low-resolution video, using CRA and the related RASL pictures does not enable smooth switching of viewports. This is because the RASL area in the picture representing the newly added tile stream cannot be properly decoded (due to lack of reference) when the position is changed, and is discarded when encountering a viewport change.

[0082] FIG. 7 shows an example where the NAL unit 304 associated with the closed GOP structure is mixed with the NAL unit 302 associated with the open GOP structure. In this particular, but non-limiting example, the CRA type NAL unit 304 having a reading RADL314 (closed GOP) is mixed with the CRA type NAL unit 302 having a reading RASL312 (open GOP). This non-limiting example shows the coding structure of the high-resolution region in the second sub-bitstream 11-2 that uses a closed GOP structure with CRA in combination with a RADL type reading picture, and the low-resolution region in the first sub-bitstream 11-1 that uses an open GOP structure with CRA in combination with a RASL type reading picture.

[0083] According to one embodiment, the decoder 100 is configured to process the bitstream 11, the IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is an IRAP NAL unit type associated with the closed GOP structure, and the IRAP NAL unit type of the NAL unit 302 in the first sub-bitstream 11-1 is an IRAP NAL unit type associated with the open GOP structure.

[0084] In this particular non-limiting embodiment as shown in FIG. 7, the IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is a CRA unit type having a random access decoder-decodable picture unit type (CRA with RADL), and the IRAP NAL unit type of the NAL unit 302 in the first sub-bitstream 11-1 is a CRA unit type having a random access skip reading picture unit type (CRA with RASL).

[0085] Note that FIG. 7 shows the sequence of access units and the pictures contained therein in their display order. However, the display order may be different from the encoding order, which is also called the bitstream order. As can be seen, access unit 30 includes a hybrid picture 12a that includes a first VCL NAL unit 302 representing a first spatial segment 131a of picture 12a in the first sub-bitstream 11-1, and a second VCL NAL unit 304 representing a second spatial segment 132a of picture 12a in the second sub-bitstream 11-2. In this case, the first and second NAL units are IRAP NAL units. In this non-limiting example, both the first and second NAL units 302, 304 are of the IRAP NAL unit type CRA. Also, FIG. 7 shows the display order of consecutive pictures.

[0086] Therefore, in the display order, a further access unit 31 may precede the above-described access unit 30. The above-described leading access unit 31 includes a hybrid picture 12b that includes a first VCL NAL unit 312 representing a first spatial segment 131b of picture 12b in the first sub-bitstream 11-1, and a second VCL NAL unit 314 representing a second spatial segment 132b of picture 12b in the second sub-bitstream 11-2. The first NAL unit 312 of this AU31 is a non-IRAP NAL unit of type RASL, and the second NAL unit 314 of this AU31 is a non-IRAP NAL unit of type RADL. Also, FIG. 7 shows the display order of consecutive pictures.

[0087] Optionally, and in display order, one or more additional access units, such as the exemplary access unit 32, may precede the access unit 31 described above. More generally, an access unit 31 comprising non-IRAP NAL units 312, 314 (e.g., RASL or RADL) may be in front of at least one additional access unit 32 which may preferably also comprise non-IRAP NAL units 322, 324 (e.g., RASL or RADL) in display order. To maintain the open GOP structure in the first sub-bitstream 11-1, the RASL NAL unit 312 of the access unit 31 may be included in a preceding access unit 32 and may be preceded (in display order) by a NAL unit 322 which may be of RASL type. Next, to maintain the closed GOP structure in the second sub-bitstream 11-2, the RADL NAL unit 314 of the access unit 31 may be included in a preceding access unit 32 and may be preceded (in display order) by a NAL unit 324 which may also be of RADL type.

[0088] The encoding order, also called the bitstream order, may be different from the display order shown in FIG. 7. For example, the RADL and RASL pictures 131b, 132b, 131c, 132c are encoded dependently, which means taking differential encoding information from the respective CRA pictures 131a, 132a (in display order) of the CRA pictures 13. For example, the RADL and RASL pictures 131b, 132b included in the access section 31 refer to the CRA pictures 131a, 132a included in the access unit 30 (in display order), even though the access unit 30 follows the access unit 31 (in display order).

[0089] The RASL picture 131b can optionally further reference a picture such as the preceding RASL picture 131c illustrated in FIG. 7 (in the order of display and encoding). Thus, the preceding RASL picture 131c of the preceding access unit 32 must be encoded before the RASL picture 131b of the access unit 31. However, an even more preceding RASL picture 131c of the preceding access unit 32 may itself reference a picture that follows the display order but precedes the encoding order, such as the IRAP picture 131a of the access unit 30. That is, the IRAP (CRA) picture 131a of the access unit 30 is the last picture in the order (i.e., CRA) picture 13 described above. However, the IRAP (CRA) picture 131a of the access unit 30 is the first picture in the encoding order, that is, it must be encoded first, because both RASL pictures 131b and 131c reference the IRAP (CRA) picture 131a during encoding.

[0090] The same applies to the RADL picture. The RADL picture 132b can optionally further reference a preceding picture (in the order of display and encoding), such as the preceding RADL picture 132c illustrated in FIG. 7. Thus, the further preceding or following RADL picture 132c of the preceding access unit 32 must be encoded before the RADL picture 132b of the access unit 31. However, an even more preceding RADL picture 132c of the preceding access unit 32 may itself reference a picture that follows the display order but precedes the encoding order, such as the IRAP picture 132a of the access unit 30. That is, the IRAP (CRA) picture 132a of the access unit 30 is the last picture in the order (i.e., CRA) picture 13 described above. However, the IRAP (CRA) picture 132a of the access unit 30 is the first picture in the encoding order, that is, it must be encoded first, because both RADL pictures 132b and 132c reference the IRAP (CRA) picture 132a during encoding.

[0091] More generally, a leading picture (LP) type non-IRAP picture (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) can reference a following IRAP picture (e.g., CRA pictures 131a, 132a) (e.g., in display order), and the IRAP picture (e.g., CRA pictures 131a, 132a) must be encoded first, i.e., before the leading non-IRAP picture of the leading picture (LP) type (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) in (display order). That is, the IRAP picture (such as CRA pictures 131a, 132a) follows the display order of the non-IRAP picture of the leading picture (LP) type (such as RASL and RADL pictures 131b, 131c, 132b, 132c), but the IRAP picture (such as CRA pictures 131a, 132a) is encoded first. In other words, the non-IRAP picture of the leading picture (LP) type (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) reads the IRAP picture (e.g., CRA pictures 131a, 132a) in display order, and the IRAP picture (e.g., CRA pictures 131a, 132a) reads the non-IRAP picture of the leading picture (LP) type (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) in bitstream order (encoding order). In other words, the leading non-IRAP picture of the leading picture (LP) type (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) reads the IRAP picture (e.g., CRA pictures 131a, 132a) in display order, but the non-IRAP picture of the leading picture (LP) type (e.g., RASL and RADL pictures 131b, 131c, 132b, 132c) follows in the encoding / bitstream order after the IRAP picture (e.g., CRA pictures 131a, 132a).

[0092] Considering the above examples and coding structures, it is one of the advantages of the present invention that a mixture of a CRA having a reading RASL (open GOP) and a CRA having a reading RADL (closed GOP) within a picture of the access unit 30 is allowed. When such a mixed NALU access unit 30 is encountered within a coded video sequence (CVS) during normal playout (decoding the entire bitstream 11 from the beginning), all necessary reference pictures become available. Therefore, the pictures following a mixed picture having a RASL NAL unit will be decoded and output normally.

[0093] However, when such a mixed NALU is encountered during a seek operation, or when decoding processing starts from such an AU during random access, the process for generating unavailable reference pictures needs to be invoked and is one of the following. · The subsequent processing chain needs to be informed that the RASL region is not correctly decoded and the corresponding low-resolution region is used, for example, through SEI indication. · The affected pictures are completely removed from the output.

[0094] This means that when starting to decode a mixed picture, one option is to treat it as a GDR picture where some parts are decodable and can be shown, and other parts cannot be shown and are refreshed over time (until all RASL NAL units are finished). Such an operation is related to, for example, a 360 scenario where the RAP type is selected contrary to the method shown in FIG. 7, that is, the low resolution uses a closed GOP structure (e.g., CRA having RADL), and the high-resolution content uses an open GOP structure (e.g., CRA having RASL). Then, it uses the closed GOP structure and the player waits until the high resolution using the open GOP structure is decoded cleanly to show it, so the low resolution can be shown.

[0095] Another option for handling such situations is to not display any of the pictures affected (even partially) by the missing reference and drop them, i.e., drop any mixed pictures having an NAL unit type equal to RASL.

[0096] In other words, any mixed pictures 12b, 12c having NAL unit types 312, 322 equal to RASL can be marked to be dropped / discarded in order to guarantee continuous decoding behavior.

[0097] Thus, according to one embodiment, decoder 100 may be configured to decode and present co-located spatial segments 132b, 132c,... of one or more pictures 12b, 12c,... of access units 31, 32,... that follow at least one access unit 30 associated with NAL unit 304 of the second sub-bitstream 11-2 in bitstream order, where the co-located spatial segments 132b, 132c,... are arranged together with the second spatial segment 132a of picture 12a of at least one access unit 30 associated with NAL unit 304 of the second sub-bitstream 11-2 in the at least one access unit 30. Further, decoder 100 may be configured to mark for discarding co-located spatial segments 131b, 131c,... of one or more pictures 12b, 12c,... of access units 31, 32,... that follow at least one access unit 30 associated with NAL unit 302 of the first sub-bitstream 11-1, where the co-located segments 131b, 131c,... are arranged together with the first spatial segment 131a of picture 12a of at least one access unit 30 associated with NAL unit 302 of the first sub-bitstream 11-1 in the at least one access unit 30.

[0098] According to a further embodiment, the decoder 100 can be configured to operate depending on whether the bitstream 11 is to be decoded, in the case of continuous decoding during normal playback (decoding the entire bitstream 11 from the beginning), or at the start of decoding (for example, when the decoding process starts from such an AU during a seek operation or during random access).

[0099] During normal playback (decoding the entire bitstream 11 from the beginning), the decoder 100 can be configured to decode and present one or more pictures 12b, 12c, ··· of access units 31, 32, ··· that follow in bitstream order at least one access unit 30 (for example, a CRA having a RADL) to which the NAL unit 304 of the second sub-bitstream 11-2 is related, and in these access units 31, 32, ···, the co-located spatial segments 132b, 132c, ··· are arranged together with the second spatial segment 132a of the picture 12a of at least one access unit 30 to which the NAL unit 304 of the second sub-bitstream 11-2 is related. Further, in the case of starting decoding (for example, when the decoding process starts from such an AU during a seek operation or during random access), the decoder 100 can be configured to discard one or more pictures 12b, 12c, ··· of access units 31, 32, ··· that follow the at least one access unit 30 in bitstream order, and resume picture output after one or more discarded pictures 12b, 12c, ···. In other words, any hybrid pictures 12b, 12c having at least one NAL unit 312, 322 equal to RASL are dropped / discarded.

[0100] According to a further embodiment, the decoder 100 may be configured to mark the above-mentioned damaged spatial segments 131b, 131c, … for the subsequent processing chain by means of an SEI message.

[0101] In a further embodiment, regardless of whether there is an IDR_W_RADL NAL unit type, an indication that the AU is treated as a CRA, i.e., an IRAP where the NoIncorrectPicOutputFlag is equal to 1, is present in the bitstream. The AU is a mixed NALU, but has the mentioned characteristics of IRAP and open GOP structures. In other words, the decoder 100 can be configured to derive from the bitstream 11 an indication that indicates towards the decoder 100 to decode at least one access unit 30 associated with the first sub-bitstream 11-1 and the second sub-bitstream 11-2 in a manner corresponding to a clean random access NAL unit type.

[0102] In the above description, an embodiment in which the decoder 100 is configured to process the bitstream 11 has been described, but here A) A non-IRAP NAL unit of a non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA) is mixed with at least one different second NAL unit of a different non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA), or B) A NAL unit of an IRAP NAL unit type (e.g., IDR_W_RADL, CRA) is mixed with at least one NAL unit of a further IRAP NAL unit type (e.g., IDR_W_RADL, CRA).

[0103] However, there may be a further embodiment in which the decoder 100 can be configured to process the bitstream 11, where C) A NAL unit of an IRAP NAL unit type (IDR_W_RADL, CRA) is mixed with at least one NAL unit of one of a non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or an STSA NAL unit type.

[0104] Further embodiments also propose to provide a corresponding apparatus for processing the video bitstream 11 during the encoding of the bitstream 11 and / or the merging of the first and second sub-bitstreams 11-1, 11-2. The apparatus 10 described above can be at least one of an encoder, a merger, or a network node for processing the bitstream 11 according to the innovative principles described herein.

[0105] Accordingly, an embodiment proposes an apparatus 10 for processing video content 12, the apparatus 10 being configured to provide a first sub-bitstream 11-1 related to a first spatial segment 131 of pictures 12a, 12b, 12c of the video content 12 and a second sub-bitstream 11-2 related to a second spatial segment 132 of pictures 12a, 12b, 12c of the video content 12, the first sub-bitstream 11-1 and the second sub-bitstream 11-2 being part of the bitstream 11, A) the bitstream 11 includes at least one access unit 30 of a non-IRAP NAL unit type in which at least one first NAL unit 302 of the first sub-bitstream 11-1 is mixed with at least one different second NAL unit 304 of the second sub-bitstream 11-2 that is of a different non-IRAP NAL unit type, or, B) the bitstream 11 includes at least one access unit 30 of an IRAP NAL unit type in which at least one first NAL unit 302 of the first sub-bitstream 11-1 is mixed with at least one different second NAL unit 304 of the second sub-bitstream 11-2 that is of a further IRAP NAL unit type, or, C) The bitstream 11 includes at least one access unit (30) in which at least one first NAL unit 302 of the first sub-bitstream 11-1 is of an IRAP NAL unit type that is mixed with at least one different second NAL unit 304 of a second sub-bitstream 11-2 that is either of a non-IRAP leading picture NAL unit type (RASL or RADL) or of an STSA NAL unit type.

[0106] 1.4 Summary and Syntax Examples The following shows an overview of the mixed types allowed within the present invention. [Table 1]

[0107] In the new mixed types described in the present invention, there are two types of mixing, one with an IRAP type and the other without an IRAP type.

[0108] Option 1 As described above in the previous section, one option is to signal only the mixing with an IRAP type with a flag in the PPS. For example, reuse the existing one that mixes IDR or CRA with TRAIL up to VCL_RSV_6 (mixed_nalu_types_in_pic_flag). Types other than IRAP can be signaled using constraint flags in the SPS / VPS / DPS.

[0109] A mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture referring to the PPS has one or more VCL NAL units, the VCL NAL units do not have the same nal_unit_type value, have nal_unit_type values in the range from IDR_W_RADL to CRA_NUT, and the picture is not an IRAP IDR picture. A mixed_nalu_types_in_pic_flag equal to 0 specifies that each picture referring to the PPS has one or more VCL NAL units, and when the VCL NAL units have nal_unit_type values in the range from IDR_W_RADL to CRA_NUT, the VCL NAL units of each picture referring to the PPS have the same value as nal_unit_type.

[0110] Next, if the AU is the first AU in the bitstream or the first AU after a sequence (EOS) NAL unit referring to a PPS with a mixed_nalu_types_in_pic_flag equal to 1, the NAL unit type of the coded picture shall be CRA_NUT and IDR_W_RADL. A mixed picture with IDR / CRA having a non-IRAP type is only permitted if the AU is not the first picture in the bitstream or sequence.

[0111] A no_non_irap_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that it is a bitstream conformity requirement that the VCL NAL units of pictures containing TRAIL, STSA, RASL, RADL in the sequence have the same value as nal_unit_type. A value equal to 0 imposes no such constraint. That is, the pictures may have two different values of nal_unit_type among TRAIL, STSA, RASL, RADL.

[0112] Option 2 Another option would be to specify an indication (idc) indicating different combinations instead of the PPS flag (mixed_nalu_types_in_pic_flag).

[0113] The no_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that it is a bitstream compliance requirement that the mixed_nalu_types_in_pic_idc be equal to 0. The no_mixed_nalu_types_in_pic_constraint_flag equal to 0 does not impose such a constraint.

[0114] The mixed_nalu_types_in_pic_idc equal to 0 specifies that each picture referring to the PPS has one or more VCL NAL units, and the VCL NAL units of each picture referring to the PPS have the same value as nal_unit_type.

[0115] The mixed_nalu_types_in_pic_idc equal to 1 specifies that each picture referring to the PPS has multiple VCL NAL units, and all the VCL NAL units have two specific values of nal_unit_type in the range from TRAIL_NUT to RSV_VCL_6.

Table 2

[0116] The mixed_nalu_types_in_pic_idc equal to 2 specifies that each picture referring to the PPS has multiple VCL NAL units, one or more of the VCL NAL units all have a specific value of nal_unit_type in the range from IDR_W_RADL to CRA_NUT, and the other VCL NAL units all have a specific value of nal_unit_type in the range from TRAIL_NUT to RSV_VCL_6 or are equal to GRA_NUT.

[0117] This value corresponds to the permitted mixing of NAL units and existing flags.

Table 3

[0118] A mixed_nalu_types_in_pic_idc equal to 3 specifies that each picture referring to the PPS has multiple VCL NAL units with the nal_unit_type value of IDR_W_RADL and one or more VCL NAL units with the nal_unit_type value of CRA_NUT.

Table 4

[0119] For the VCL NAL units of any particular picture, the following applies. If mixed_nalu_types_in_pic_idc is equal to 0, the nal_unit_type value is the same for all coded slice NAL units of the picture. A picture or PU is said to have the same NAL unit type as the coded slice NAL units of the picture or PU. If mixed_nalu_types_in_pic_idc is 1 or 2, the following applies. If one of the VCL NAL units within the picture has a nal_unit_type value equal to TRAIL_NUT, the picture or PU is called a trailing picture or trailing PU. Otherwise (if there is no TRAIL_NUT VCL NAL unit within the picture), the picture or PU is called a leading picture or leading PU. If one of the VCL NAL units within the picture has a nal_unit_type value equal to RADL_NUT, the picture or PU is called a RADL picture or RADL PU. In other cases (when the VCL NAL unit in the picture does not have a nal_unit_type value equal to RADL_NUT), the picture or PU is called a RASL picture or a RASL PU. In other cases (when mixed_nalu_types_in_pic_idc is equal to 3), the picture is called a CRA picture or a CRA PU.

[0120] Aspects related to handling the main NAL unit type as an STSA picture can be implemented using constraint flags as follows.

Table 5

[0121] When leading_stsa_pictures_flag is equal to 1, the reference for inter prediction of RADL and RASL pictures is constrained as follows. When no_mixed_nalu_types_in_pic_constraint_flag is equal to 0, such constraints are not imposed.

[0122] When leading_stsa_pictures_flag is equal to 1, the following applies. When the current picture is a RASL picture or a RADL picture, there is no active entry in RefPicList[0] or RefPicList[1] that has a TemporalId equal to the TemporalId of the current picture. When the current picture is a picture that follows a RASL or RADL in decoding order and precedes a related IRAP picture that has the same TemporalId as the current picture, there shall be no picture in RefPicList[0] or RefPicList[1] that precedes the RASL or RADL picture in decoding order and has a TemporalId equal to that of the current picture and is included as an active entry.

[0123] In other words, the restrictions related to the STSA picture prohibit, in decoding order, a picture related to the temporal sub-layer immediately above the STSA from using any reference in the immediately above temporal sub-layer that precedes the STSA, i.e., the decoder can start decoding the next temporal sub-layer after the STSA. Such restrictions can, of course, also be applied to RASL / RADL pictures as described above, and thus the above signaling can serve as an indication of these characteristics.

[0124] 2. Mixing of picture types within the multi-layer access unit This subsection may relate, by way of non-limiting example, to a multi-layer access unit in a multi-layer bitstream. However, the features described herein may already be applied to a single layer. Further, this subsection explains the principles of the present invention by way of non-limiting example by referring to a long-term reference picture. However, the concepts of the present invention are equally applicable to short-term reference pictures, i.e., the features described herein can generally be applied to the pictures being referred to. Further, this subsection can explain POC MSB reset, by way of non-limiting example, by setting the POC value to zero, i.e., setting MSB = 0. However, the value zero acts only as a placeholder symbolizing a decreasing value of the MSB, and for example, it is also possible that the MSB reset occurs at any predetermined value that is not equal to 0 but is smaller than the current MSB at each bitstream position.

[0125] The prior art may provide support for mixing different RAP type pictures within a layered access unit by signaling the top POC (Picture Order Count) bit (poc_msb_val), i.e., for a time instance where an access unit contains both RAP pictures and non-RAP pictures in that layer simultaneously. However, as shown below by examples of Long Term Reference Pictures, although it involves references to pictures preceding their associated IRAP pictures in the coding order, it is also applicable to Short Term Reference Pictures, and there may remain problems with reference pictures of the leading picture NAL unit type, which the present invention provides a solution for.

[0126] The current syntax related to POC signaling is as follows.

Table 6

Table 7

[0127] An exemplary usage of the above POC MSB signaling is to set the POC MSB to 0 when all layers have the same IRAP NAL unit type, for example, when all layers are IDR_W_RADL or CRA as shown in FIG. 8.

[0128] FIG. 8 shows a multi-layer bitstream 11 composed of a first sub-bitstream 11-1 of a first layer L0 and a second sub-bitstream 11-2 of a second layer L1. Each layer, L0, may include one or more temporal sub-layers T0, T1, T2. Therefore, the spatial segments of the pictures may be distributed across the different layers L0, L1 and temporal sub-layers T0, T1, T2 described above.

[0129] FIG. 8 further shows some exemplary multi-layer access units 30, 31, 32. In this example, the multi-layer access units 30, 31, 32 may comprise a first NAL unit 302 of a first sub-bitstream 11-1 in a first layer L0 that is mixed with a second NAL unit 304 of a second sub-bitstream 11-2 in a second layer L1. The NAL units may comprise different NAL unit types, particularly IRAP and non-IRAP NAL unit types, as described above. Further, the NAL units may comprise a slice header in which a picture order count (POC) is signaled, as described above. The concepts described herein relate to the most significant part of the POC, e.g., the most significant POC bit, or POC MSB. Thus, in the figure, the POC MSB is shown for each IRAP NAL unit.

[0130] For example, the multi-layer access units 30, 32 represent an access unit in which the POC MSB of the first NAL unit 302 in the first sub-bitstream 11-1 is aligned with the POC MSB of the second NAL unit 304 of the second sub-bitstream 11-2, and both NAL units are of the IRAP NAL unit type.

[0131] The multi-layer access unit 31 represents a misaligned access unit, i.e., the POC MSB of the first NAL unit 302 within the first sub-bitstream 11-1 is misaligned with the POC MSB (e.g., MSB = 0) of the second NAL unit 304 of the second sub-bitstream 11-2 of the non-IRAP NAL unit type. Thus, the POC MSB of the first NAL unit 302 of the first sub-bitstream 11-1 may be incremented by a predetermined value, e.g., incremented by 1.

[0132] FIG. 8 shows how the MSB signaling in the aforementioned slice header can be used in such a multi-layer case. The purpose of the signaling is, for example, to correctly maintain the POC value in a misaligned RAP such as the misaligned access unit 31 with CRA w MSB = 1, that is, to avoid the POC (LSB and MSB) being reset to zero when not all layers have an IRAP in the same access unit.

[0133] However, FIG. 8 also shows that there is a problem with the reference picture 12r (e.g., long-term and / or short-term) when the POC is aligned. For ease of understanding, the following figures may refer only to the long-term reference picture. However, this concept is also applicable to short-term reference pictures.

[0134] Note that the pictures 12a, 12b, 12c,... before the last CRA access unit 32, that is, the second aligned RAP, have POC MSB>0 and POC LSB>0. Therefore, when resetting the MSB to 0 in the last CRA access unit 32 (as exemplarily shown in FIG. 8), it is impossible to refer to any of the previous pictures 12a, 12b, 12c,... using the POC MSB or its delta (used for the long-term reference picture). One such example is shown in FIG. 8 by the green arrow marked "LT ref". Therefore, the example shown in FIG. 8 shall show that picture reference may be impossible in this case.

[0135] Therefore, the present invention provides a solution to this problem so that picture reference becomes possible.

[0136] In one embodiment, the constraint can be expressed or indicated in the specification through a bitstream flag (SPS, VPS, DPS) that can set the MSB to 0 only when all pictures in the access unit are of type IDR, i.e., IDR_W_RADL or IDR_N_LP. Thereby, when CRA is used, no POC reset occurs and LT references are permitted (see, for example, FIGS. 9 and 10).

[0137] As seen in FIG. 9, the access unit 32 can be an aligned access unit 32 where both NAL units 302, 304 are of the IRAP NAL unit type. However, since the first NAL unit 302 of the first sub-bitstream 11-1 is of the CRA type, no POC reset occurs. That is, FIG. 9 shows a scenario where POC reset is not permitted.

[0138] Instead, FIG. 10 shows a scenario where POC reset is permitted since both the first NAL unit 302 of the first sub-bitstream 11-1 and the second NAL unit 304 of the second sub-bitstream 11-2 are of the IDR unit type.

[0139] Accordingly, according to an embodiment, a decoder 100 for decoding video content 12 is provided, and the decoder 100 is configured to decode a bitstream 11 including a first sub-bitstream 11-1 and a second sub-bitstream 11-2. The decoder 100 is configured to derive from slice headers of NAL units 302, 304 of the first and second sub-bitstreams 11-1, 11-2, where the NAL units 302, 304 are information for setting the most significant POC part (POC MSB) of the POC of a picture 12r referred to by the NAL units 302, 304 and are of an IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP). According to this example, the most significant POC part (POC MSB) needs to be set to a predetermined value for all NAL units 302, 304 of the IRAP NAL unit type within an access unit 32 of the bitstream 11 when at least one of the NAL units 302, 304 within the above-mentioned access unit 32 is of a non-Instantaneous Decoding Refresh unit type, e.g., a CRA unit type.

[0140] As described above, this subsection can describe POC MSB reset by setting the POC value to zero, i.e., setting MSB = 0, as a non-limiting example. However, the value zero only acts as a placeholder symbolizing a decreasing value of the MSB, and for example, it is also possible that the MSB reset occurs at an arbitrary predetermined value that is not equal to 0 but is smaller than the current MSB at each bitstream position. Accordingly, the above-mentioned predetermined value may be equal to or greater than the most significant POC part of the POC associated with pictures 12a, 12b,... preceding the picture 12r referred to by the NAL unit 302. In other words, if not all NAL units 302, 304 within the access unit 32 are of the IDR unit type, the POC reset is not performed.

[0141] In some examples, the above-mentioned predetermined value may be equal to zero, i.e., the MSB = 0 may also be the case. In some other examples, the above-mentioned predetermined value may be equal to the implicitly derived most significant POC part, for example, the implicit derivation of the normal SOTA POC MSB.

[0142] In another embodiment, whenever an aligned RAP access unit 32 occurs and the POC MSB of the access unit 32 is set to 0, the (long-term or short-term) reference pictures 12a, 12b, 12c, ··· before the aligned RAP access unit 32 are permitted to be referenced by a picture 12s following the aligned RAP access unit 32 in bitstream order, which may be expressed in the specification or indicated via bitstream flags (SPS, VPS, DPS).

[0143] For example, FIG. 11 shows that the picture 12s may precede the picture 12r in the presentation order, but the picture 12s may follow the picture 12r in the decoding order.

[0144] FIGS. 11 and 12 show that any picture 12r having an MSB equal to 0 does not have leading pictures 12a, 12b, 12c, ··· having LT reference pictures (while FIG. 11 represents a reset from MSB = 1 to MSB = 0, and FIG. 12 represents an MSB equal to 0, for example, because the POC_LSB is long enough that no increase occurs). Therefore, FIG. 11 shows a scenario where POC MSB reset and picture reference are not permitted, and FIG. 12 shows a scenario where all POC MSBs are equal to zero but picture reference is not permitted. FIG. 13 shows that when the MSB is not equal to 0, LT reference pictures are permitted for leading pictures.

[0145] Accordingly, according to an embodiment, a decoder 100 for decoding video content 12 is provided, the decoder 100 being configured to decode a bitstream 11 including a first sub-bitstream 11-1 and a second sub-bitstream 11-2, the decoder 100 being configured to derive from slice headers of NAL units 302, 304 of the first and second sub-bitstreams 11-1, 11-2, where the NAL units 302, 304 are information for setting the most significant POC part (POCMSB) of the POC of the picture 12r referred to by the NAL units 302, 304 and are of an IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP). According to this example, any picture reference is not permitted to intersect any access unit 32 of the bitstream 11 that has a slice header in the decoding order in which all NAL units 302, 304 are of an IRAP NAL unit type and the most significant POC part (POC MSB) is set to a predetermined value.

[0146] Again, as described above, this subsection can describe POC MSB reset by, as a non-limiting example, setting the POC value to zero, i.e., setting MSB = 0. However, the value zero only acts as a placeholder symbolizing a decreasing value of the MSB, and it is also possible, for example, that the MSB reset occurs at any predetermined value that is not equal to 0 but is smaller than the current MSB at each bitstream position. Accordingly, the predetermined value in the example described with reference to FIGS. 11 to 13 can be the most significant POC part (POC MSB) of the POC associated with the pictures (12a, 12b, 12c, ···) preceding the picture (12r) referred to by the NAL units (302, 304), i.e., can be smaller than that at which the POC reset occurred.

[0147] In some examples, the above-mentioned predetermined value may be equal to zero, i.e., the MSB = 0 may also be possible. In some other examples, the above-mentioned predetermined value may be equal to the implicitly derived most significant POC part, for example, equal to the implicit derivation of the normal SOTA POC MSB.

[0148] In another embodiment, whenever an aligned RAP access unit 32 occurs and the POC MSB of the access unit 32 is set to 0 (POC reset), there may be a constraint expressed in the specification or indicated through bitstream flags (SPS, VPS, DPS) that allows a (short-term or long-term) reference picture 12a, 12b, 12c,... having an MSB different from 0 to be referenced in bitstream order by a picture 12s following the aligned RAP access unit 32.

[0149] Note that since the reference to the LT reference picture is made only by the POC_LSB value, in the above example shown in FIG. 8, if there is no MSB increase, using the LT reference picture will not be a problem.

[0150] FIG. 14 shows that no POC MSB reset has occurred in this example. The POC MSB reset may not be indicated by either increasing the POC MSB value of the access unit 32 (in the example shown in FIG. 14, the POC MSB value is increased, i.e., MSB = 3) or keeping the POC MSB value of the access unit 32 the same as that of the previous access unit 31.

[0151] Accordingly, according to one embodiment, a decoder 100 for decoding video content 12 is provided, the decoder 100 being configured to decode a bitstream 11 comprising a first sub-bitstream 11-1 and a second sub-bitstream 11-2, the decoder 100 being configured to derive from slice headers of NAL units 302, 304 of the first and second sub-bitstreams 11-1, 11-2, where the NAL units 302, 304 are information for setting the most significant POC part (POC MSB) of the POC of a picture 12r referred to by the NAL units 302, 304 and an IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP). According to this example, the most significant POC part is set to a first predetermined value, and any picture reference of reference pictures 12a, 12b, 12c,... of a first access unit 31 consisting of NAL units 301, 303 having slice headers preceding in bitstream order is not allowed by a picture 12s following the second access unit 32 of the bitstream 11 in bitstream order, where all NAL units 302, 304 are of the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP) and the slice headers have the most significant POC part (POC MSB) set to a second predetermined value.

[0152] Again, as described above, this sub-section can illustrate the POC MSB reset by, as a non-limiting example, setting the POC value to zero, i.e., setting the MSB = 0. However, the value zero only acts as a placeholder symbolizing a decreasing value of the MSB, and for example, it is also possible that the MSB reset occurs at any predetermined value that is not equal to 0 but smaller than the current MSB at each bitstream position. Thus, regarding the example described with reference to FIG. 14, the above-mentioned first predetermined value of the POC MSB of the preceding first access unit 31 may be larger than the above-mentioned second predetermined value of the POC MSB of the subsequent second access unit 32. In this case, picture reference is not permitted. Conversely, it can be stated that when the second predetermined value of the MSB of the subsequent second access unit 32 is larger than the above-mentioned first predetermined value of the POC MSB of the previous first access unit 31, picture reference is permitted. That is, when the POC reset does not occur, picture reference is permitted.

[0153] For example, the second predetermined value may be equal to zero to indicate a POC reset. Thus, when a POC reset occurs, picture reference is not permitted. In some other examples, the above-mentioned second predetermined value may be equal to the implicitly derived most significant POC part, for example, equal to the implicit derivation of the normal SOTA POC MSB.

[0154] Although some aspects have been described in the context of an apparatus, these aspects also represent a description of the corresponding method, and it is clear that a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of the corresponding block or item or feature of the corresponding apparatus.

[0155] Some or all of the method steps may be performed (or used) by a hardware device such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0156] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware, or in software, or at least partially in hardware, or at least partially in software. The implementation can be carried out using a digital storage medium, such as a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH (registered trademark) memory, in which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that each method is carried out. Thus, the digital storage medium may be computer-readable.

[0157] Some embodiments according to the present invention comprise a data carrier having electronically readable control signals that can cooperate with a programmable computer system so that one of the methods described herein is carried out.

[0158] Generally, embodiments of the present invention can be implemented as a computer program product having program code, the program code operating to execute one of the methods when the computer program product operates on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0159] Other embodiments include a computer program for executing one of the methods described herein, stored on a machine-readable carrier.

[0160] In other words, one embodiment of the method according to the present invention is thus a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

[0161] Thus, a further embodiment of the method according to the present invention is a data carrier (or digital storage medium, or computer-readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0162] Thus, a further embodiment of the method according to the present invention is a sequence of data streams or signals representing a computer program for performing one of the methods described herein. The sequence of data streams or signals may be configured to be transferred, for example, via a data communication connection, such as via the Internet.

[0163] A further embodiment comprises processing means, such as a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0164] A further embodiment includes a computer having installed thereon a computer program for performing one of the methods described herein.

[0165] A further embodiment according to the present invention comprises an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0166] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0167] The devices described herein can be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0168] The methods described herein can be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0169] Although this disclosure has been described with reference to exemplary embodiments, this description is not intended to be construed in a limiting sense. Various modifications and combinations of the exemplary embodiments, as well as other embodiments of this disclosure, will be apparent to those skilled in the art by reference to this description. Accordingly, the appended claims are intended to embrace any such modifications or embodiments.

Claims

1. 1. A decoder for decoding video content, comprising: A processor is provided. The processor, configured to decode a bitstream comprising: a first sub-bitstream associated with a first spatial segment of a coded picture of the video content; and a second sub-bitstream associated with a second spatial segment of a coded picture of the video content, wherein each Network Abstraction Layer (NAL) of the coded picture is a Non-Intra Random Access Point (IRAP) NAL unit type; configured to identify that at least one first NAL unit of the first sub-bitstream is mixed with at least one different second NAL unit of the second sub-bitstream; the non-IRAP NAL unit type of the first NAL unit of the first sub-bitstream is a random access decodable reading unit type (RADL); the non-IRAP NAL unit type of the second NAL unit of the second sub-bitstream is a trailing picture unit type (TRAIL); decoder.

2. The processor, Decoding a general constraint flag associated with a parameter set, the general constraint flag indicating whether mixing of NAL unit types within a picture is allowed; If the general constraint flag indicates that mixing of NAL unit types in the picture is not allowed, determining that the picture does not have mixed NAL unit types; if the general constraint flag indicates that mixing of NAL unit types in the picture is allowed, determine that the picture has mixed NAL unit types; It is configured as follows: A decoder according to claim 1.

3. 1. A method for decoding video content, comprising: decoding a bitstream including a first sub-bitstream associated with a first spatial segment of a coded picture of the video content and a second sub-bitstream associated with a second spatial segment of a coded picture of the video content, wherein each Network Abstraction Layer (NAL) of the coded pictures is a Non-Intra Random Access Point (IRAP) NAL unit type; identifying at least one first NAL unit of the first sub-bitstream to be mixed with at least one different second NAL unit of the second sub-bitstream; Including, the non-IRAP NAL unit type of the first NAL unit of the first sub-bitstream is a random access decodable reading unit type (RADL); the non-IRAP NAL unit type of the second NAL unit of the second sub-bitstream is a trailing picture unit type (TRAIL); method.

4. The method comprises: decoding a general constraint flag associated with a parameter set, the general constraint flag indicating whether mixing of NAL unit types within a picture is allowed; determining that the picture does not have mixed NAL unit types if the general constraint flag indicates that mixing of NAL unit types in the picture is not allowed; determining that the picture has mixed NAL unit types if the general constraint flag indicates that mixing of NAL unit types in the picture is allowed; Including, The method according to claim 3.

Images