Decoder, encoder and method for mixing NAL units of different NAL unit types in a video stream

Abstract

The invention relates to a decoder, an encoder and a method for mixing NAL units of different NAL unit types in a video stream, the video data stream (11) comprising a first sub-bitstream (11-1) and a second sub-bitstream (11-2). The concepts described herein provide a solution for mixing different NAL units (301, 302, 303, 304) of different NAL unit types within an access unit (30, 31, 32). For example, an IRAP NAL unit type may be mixed with a different IRAP NAL unit type or a non-IRAP NAL unit type, and a non-IRAP NAL unit type may be mixed with a different non-IRAP NAL unit type.

Description

[0001] This application is a divisional application of the invention patent application with the application number 202080097218.7, the application date of December 16, 2020, and the invention title of "Decoder, Encoder, and Method for Mixing NAL Units of Different NAL Unit Types in a Video Stream". Technical Field

[0002] Embodiments of the present disclosure relate to decoders and corresponding encoders for decoding / encoding video streams and in particular video streams comprising at least two video sub-streams. The video stream may comprise a plurality of access units, where each access unit may comprise at least one picture included in each of the two or more video sub-streams. Each picture may be associated with a NAL (Network Abstraction Layer) unit of a specific NAL unit type. The innovative concepts described herein introduce solutions for mixing NAL units of different NAL unit types. Background Art

[0003] Depending on whether the NAL unit contains an encoded picture or other associated data respectively, the NAL unit can be classified into VCL (Video Coding Layer) and non-VCL NAL units. In the HEVC standard, there are several VCL NAL unit types, and the VCL NAL unit types identify the types of pictures for decoder initialization and random-access purposes. Using the content of the two-byte NAL unit header, it is possible to easily identify the purpose of the associated payload data.

[0004] Video coding techniques enable random access and bitstream splicing. Regarding random access, the bitstream may start with an IDR (Instantaneous Decoding Refresh) access unit. The IDR access unit contains independently encoded pictures, i.e., encoded pictures that can be decoded without decoding any previous pictures in the NAL unit stream. The presence of an IDR access unit indicates that subsequent pictures in the bitstream will not require reference to pictures before the pictures it contains for decoding. Within an encoding structure called a closed GOP (Group of Pictures), IDR pictures are used. An alternative to the IDR syntax discussed above is provided by the CRA (Clean Random Access) picture syntax, which specifies the use of independently encoded pictures at the position of a RAP or IRAP ((Intra) Random Access Point), i.e., at a position in the bitstream where the decoder can start successfully decoding a picture without having to decode any pictures that appear earlier in the bitstream, which supports an efficient temporal coding order called open GOP operation.

[0005] Good support for random access can be crucial for enabling channel switching, seeking operations, and dynamic streaming services. Some pictures that follow a CRA picture in decoding order and precede a CRA picture in display order may contain inter-picture prediction references to pictures that are not available at the decoder. Thus, these non-decodable pictures can be discarded by the decoder, which starts its decoding process at a CRA point. For this purpose, such non-decodable pictures can be identified as RASL pictures (Random Access Skipped Leading). An IRAP picture can be an IDR or a CRA picture, where a CRA picture can be followed by RASL pictures in the bitstream. RASL pictures can be discarded by the decoder because, for example, due to a splicing operation, a RASL picture may contain references to pictures that do not actually exist in the bitstream.

[0006] Another type of picture that can follow an IRAP picture in decoding order and precede an IRAP picture in output order is a RADL picture (Random Access Decodable Leading), which may not contain references to any pictures that precede the IRAP picture in decoding order. RASL pictures and RADL pictures can be collectively referred to as leading pictures (LP). Pictures that follow an IRAP picture in both decoding order and output order are called trailing pictures (TRAIL). Trailing pictures may not contain any references to LPs for inter-picture prediction.

[0007] In current video coding, temporal sub-layer support can also be provided. Thus, a temporal identifier can be specified in the NAL unit header, which indicates the level in a hierarchical temporal prediction structure. This helps to achieve temporal scalability without the need to parse parts of the bitstream other than the NAL unit header. In some cases, it is possible to adjust the number of decoded temporal sub-layers during the decoding process of an encoded video sequence. The position of a point in the bitstream where a sub-layer switch may start decoding some higher temporal layers can be indicated by the presence of an STSA picture (Stepwise Temporal Sub-layer Access). At the position of an STSA picture, it may be possible to switch from decoding a lower temporal sub-layer to decoding a specific higher temporal sub-layer (but not to other layers above it, unless those other layers also contain an STSA picture).

[0008] For multi-reference picture management, a specific set of previously decoded pictures may exist in the DPB (decoded picture buffer) for decoding the remainder of the pictures in the bitstream. To identify these pictures, a list of POC (picture order count) identifiers may be conveyed in each slice header. The set of reserved reference pictures is referred to as the RPS (reference picture set). The POC may include a least significant bit portion (LSB) and a most significant bit portion (MSB) to indicate the corresponding current picture order count (POC value) for each picture in the bitstream, where the picture order may be different from the decoding order of the corresponding picture. Thus, decoding pictures after an IRAP picture (the pictures of which reference reference pictures preceding the IRAP picture) may be challenging.

[0009] Thus, it is desirable to improve existing encoders and decoders such that proper handling of pictures and picture sequences in the bitstream can be provided, particularly in the case where they occur after an IRAP picture. Summary of the Invention

[0010] Accordingly, it is proposed to provide a decoder having at least the features of the independent claims and a corresponding device for encoding and / or combining having at least the features of the independent claims. Further, it is proposed to provide corresponding methods for encoding and decoding, as well as corresponding computer program products and bitstreams, respectively. Advantageous embodiments are set forth in the dependent claims.

[0011] As mentioned above, the bitstream may include: pictures associated with the IRAP NAL unit type, which pictures may also be referred to as IRAP pictures; and pictures not associated with the IRAP NAL unit type (i.e., associated with a non-IRAP NAL unit type), which pictures may also be referred to as non-IRAP pictures.

[0012] According to one aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream that includes a first sub-bitstream related to a first spatial segment of a picture of the video content and a second sub-bitstream related to a second spatial segment of the picture of the video content. In this case, the bitstream includes at least one access unit, wherein at least one first NAL unit of the first sub-bitstream belongs to a non-IRAP NAL unit type, such as RASL, RADL, TRAIL, STSA. The at least one access unit further includes at least one different second NAL unit of the second sub-bitstream that belongs to a non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA), wherein the first NAL unit of the first sub-bitstream and the second NAL unit of the second sub-bitstream are mixed. In other words, non-IRAP NAL units are mixed with different non-IRAP NAL units. For example, a NAL unit of the TRAIL picture NAL unit type can be mixed with a NAL unit of the RASL picture NAL unit type. According to this non-limiting example, within one access unit, a TRAIL picture of the first sub-bitstream can be mixed with a RASL picture of the second sub-bitstream. Other non-limiting examples can allow a TRAIL picture of the first sub-bitstream to be mixed with a RADL picture of the second sub-bitstream within one access unit. Some additional non-limiting examples can allow a TRAIL picture of the first sub-bitstream to be mixed with a STSA picture of the second sub-bitstream within one access unit. Some additional non-limiting examples can allow a STSA picture of the first sub-bitstream to be mixed with a RADL picture of the second sub-bitstream within one access unit. Some additional non-limiting examples can allow a STSA picture of the first sub-bitstream to be mixed with a RASL picture of the second sub-bitstream within one access unit. Some additional non-limiting examples can allow a RASL picture of the first sub-bitstream to be mixed with a RADL picture of the second sub-bitstream within one access unit.

[0013] According to another aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream comprising a first sub-bitstream associated with a first spatial segment of a picture of the video content and a second sub-bitstream associated with a second spatial segment of a picture of the video content. In this case, the bitstream comprises at least one access unit, within which at least one first NAL unit of the first sub-bitstream belongs to the IRAP NAL unit type and is mixed with at least one second NAL unit of the second sub-bitstream belonging to another IRAP NAL unit type. In other words, a first IRAP NAL unit is mixed with another second IRAP NAL unit. The said another second IRAP NAL unit may belong to the same NAL unit type as the first IRAP NAL unit, or it may belong to a different NAL unit type from the first IRAP NAL unit. For example, a NAL unit of the CRA picture NAL unit type may be mixed with another picture of the same NAL unit type, e.g., with another CRA picture NAL or with another picture of a different NAL unit type, e.g., with a NAL unit of the IDR picture NAL unit type with a leading picture (i.e., IDR picture unit type with RADL (IDR_W_RADL)). According to this non-limiting example, within one access unit, a CRA picture of the first sub-bitstream may be mixed with another CRA picture of the second sub-bitstream or with an IDR_W_RADL picture of the second sub-bitstream.

[0014] According to another aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream comprising a first sub-bitstream associated with a first spatial segment of a picture of the video content and a second sub-bitstream associated with a second spatial segment of a picture of the video content. In this case, the bitstream comprises at least one access unit, within which at least one first NAL unit of the first sub-bitstream belongs to the IRAP NAL unit type and is mixed with at least one second NAL unit of the second sub-bitstream belonging to either a non-IRAP leading picture NAL unit type (e.g., RASL or RADL) or a STSA NAL unit type. In other words, within one access unit, an IRAP NAL unit (IDR_W_RADL or CRA) is mixed with a non-IRAP NAL unit, where the non-IRAP NAL unit is any one of a RASL picture, a RADL picture or a STSA picture.

[0015] According to a further aspect, there is provided a decoder for decoding video content, the decoder being configured to decode a bitstream including a first sub-bitstream and a second sub-bitstream, and wherein the decoder is configured to derive information for setting the most significant POC part of the POC of a picture referred to by a NAL unit from the slice headers of the NAL units of the first and second sub-bitstreams, the NAL unit belonging to the IRAP NAL unit type (e.g., CRA, IDR_W_RADL). In this example, if at least one NAL unit within the access unit belongs to a non-instantaneous decoding refresh (non-IDR) unit type, it is required that the most significant POC part be set to a predetermined value for all NAL units of the IRAP NAL unit type within the access unit of the bitstream. In other words, if at least one NAL unit within the access unit refers to a non-IDR picture, e.g., a CRA picture, it is required that the most significant POC part be set to the predetermined value. As a non-limiting example, the predetermined value may be equal to or greater than the most significant POC part of the POC associated with a picture preceding the picture referred to by the NAL unit. In other words, if there is a non-IDR picture within the access unit, POC reset may not occur.

[0016] According to a further aspect, there is provided a decoder for decoding video content, the decoder being configured to decode a bitstream including a first sub-bitstream and a second sub-bitstream, and wherein the decoder is configured to derive information for setting the most significant POC part of the POC of a picture referred to by a NAL unit from the slice headers of the NAL units of the first and second sub-bitstreams, the NAL unit belonging to the IRAP NAL unit type (e.g., CRA, IDR_W_RADL). In this example, any picture reference across any access unit of the bitstream in decoding order is prohibited, within which all NAL units belong to the IRAP NAL unit type and have slice headers in which the most significant POC part is set to a predetermined value. In other words, if each picture within the access unit can refer to the IRAP NAL unit type (e.g., CRA or IDR_W_RADL) and can have the most significant POC part set to the predetermined value, no picture reference from a reference picture preceding the access unit may be allowed. As a non-limiting example, the predetermined value may be less than the most significant POC part of the POC associated with a picture preceding the picture referred to by the NAL unit. For example, if reset of the most significant POC part occurs, the predetermined value may be even smaller. Thus, according to the non-limiting example mentioned above, if the access unit can only include IRAP pictures and, if reset of the most significant POC part occurs, picture reference by referring to a preceding reference picture may be prohibited.

[0017] According to a further aspect, there is provided a decoder for decoding video content, the decoder being configured to decode a bitstream comprising a first sub-bitstream and a second sub-bitstream, and wherein the decoder is configured to derive information for setting a most significant POC part of a picture referred to by a NAL unit from slice headers of NAL units of the first sub-bitstream and the second sub-bitstream, the NAL unit belonging to an IRAP NAL unit type (e.g., CRA, IDR_W_RADL). In this example, any picture reference by a picture of a second access unit following the bitstream in bitstream order to a reference picture of a first access unit, the first access unit being composed of NAL units having slice headers in which the most significant POC part is set to a first predetermined value and the first access unit preceding the second access unit of the bitstream in bitstream order, is prohibited, within the second access unit, all NAL units belong to an IRAP NAL unit type and have slice headers in which the most significant POC part is set to a second predetermined value. According to a non-limiting example, the first predetermined value may be greater than the second predetermined value. In other words, if a first predetermined value of a most significant POC part of a first (preceding) access unit is greater than a second predetermined value of a most significant POC part of a second access unit, the following may not be allowed: a picture following the second access unit may refer to a reference picture preceding the second access unit.

[0018] According to a further aspect, a corresponding encoder, a method for encoding, and a method for decoding are proposed.

[0019] According to a further aspect, there is provided a computer program, wherein each computer program in the computer program is configured to, when executed on a computer or a signal processor, implement one of the methods described herein, such that one of the computer programs in the computer program implements the method described herein.

[0020] A decoder for decoding video content according to the present invention, the decoder being configured to: decode a bitstream including a first sub-bitstream related to a first spatial segment of a picture of the video content and a second sub-bitstream related to a second spatial segment of the picture of the video content, wherein, A) the bitstream includes at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to a non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), and at least one first NAL unit is mixed with at least one different second NAL unit of the second sub-bitstream (11-2) belonging to a different non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), or B) the bitstream includes at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to an IRAP NAL unit type (e.g., IDR, CRA), and at least one first NAL unit is mixed with at least one second NAL unit of the second sub-bitstream belonging to another IRAP NAL unit type (e.g., IDR, CRA), or C) the bitstream includes at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to an IRAP NAL unit type (e.g., IDR, CRA), and at least one first NAL unit is mixed with at least one second NAL unit of the second sub-bitstream belonging to either a non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or an STSA NAL unit type.

[0021] According to a preferred embodiment of the decoder, the spatial segment of a picture relates to independently encoded sub - pictures of a picture of the video content, and the decoder is configured to use boundary extensions for motion - compensated prediction across sub - picture edges. In case A), the non - IRAP NAL unit type of the NAL units of the first sub - bitstream is a Random Access Decodable Leading Picture Unit type (RADL), and the non - IRAP NAL unit type of the NAL units of the second sub - bitstream is a Trailing Picture Unit type (TRAIL). Or in case A), the non - IRAP NAL unit type of the NAL units of the first sub - bitstream is a Random Access Skipped Leading Picture Unit type (RASL), and the non - IRAP NAL unit type of the NAL units of the second sub - bitstream is a Trailing Picture Unit type (TRAIL). Or in case A), the non - IRAP NAL unit type of the NAL units of the first sub - bitstream is a Random Access Skipped Leading Picture Unit type (RASL), and the non - IRAP NAL unit type of the NAL units of the second sub - bitstream is a Random Access Decodable Leading Picture Unit type (RADL). Or in case A), the non - IRAP NAL unit type of the NAL units of the first sub - bitstream is a Step - by - Step Temporal Sub - layer Access Unit type (STSA), and the non - IRAP NAL unit type of the NAL units of the second sub - bitstream is a Trailing Picture Unit type (TRAIL).

[0022] Preferably, the decoder is configured to derive an indication from the bitstream, wherein the indication explicitly signals to the decoder that the NAL units of the non-IRAP NAL unit type of the second sub-bitstream are mixed with the NAL units of the non-IRAP NAL unit type of the first sub-bitstream, the NAL units belong to the trailing picture unit type (TRAIL), and the NAL units are any one of the leading picture unit types of the random access skip leading picture unit type (RASL) or the random access decodable leading picture unit type (RADL). The indication includes a syntax element in a parameter set. The syntax element is included in at least one of the picture parameter set or the sequence parameter set. If the NAL unit of the non-IRAP NAL unit type of the first sub-bitstream is of the random access skip leading picture unit type (RASL), and the NAL unit of the non-IRAP NAL unit type of the second sub-bitstream is of the trailing picture unit type (TRAIL), then the decoder is configured to derive from the bitstream an indication indicated to the decoder as follows: a) ignore or suppress the output, or b) output and mark as a discontinuous spatial segment those first spatial segments of the pictures of the video content of the first sub-bitstream that correspond to the NAL units of the random access skip leading picture unit type (RASL) of the first sub-bitstream.

[0023] In case b), the decoder is configured to mark the discontinuous spatial segment to a subsequent processing chain by means of an SEI (Supplemental Enhancement Information) message. The decoder is configured to decode a constraint indication from a parameter set, wherein the constraint indication indicates to the decoder that if the non-IRAP NAL unit type of the NAL unit of the second sub-bitstream belongs to a non-leading picture NAL unit type (e.g., TRAIL or STSA), then the non-IRAP NAL unit type of the NAL unit of the first sub-bitstream exclusively belongs to the random access decodable leading picture unit type (RADL) in the case where the IRAP-NAL unit of the first sub-bitstream belongs to the clean random access unit type (CRA).

[0024] Preferably, the non-IRAP NAL unit type of the NAL unit of the second bitstream belongs to the trailing picture unit type (TRAIL). In case A), the non-IRAP NAL unit type of the NAL unit of the first sub-bitstream is the random access decodable leading picture unit type (RADL), and the non-IRAP NAL unit type of the NAL unit of the second sub-bitstream is the stepwise temporal sub-layer access unit type (STSA). Or in case A), the non-IRAP NAL unit type of the NAL unit of the first sub-bitstream is the random access skipped leading picture unit type (RASL), and the non-IRAP NAL unit type of the NAL unit of the second sub-bitstream is the stepwise temporal sub-layer access unit type (STSA). The decoder is configured to derive an indication from the bitstream, wherein the indication explicitly signals to the decoder that the NAL unit of the non-IRAP NAL unit type of the second sub-bitstream is mixed with the NAL unit of the non-IRAP NAL unit type of the first sub-bitstream, the NAL unit belongs to the stepwise temporal sub-layer access unit type (STSA), and the NAL unit belongs to any one of the leading picture unit types of the random access skipped leading picture unit type (RASL) or the random access decodable leading picture unit type (RADL).

[0025] Preferably, the indication includes a syntax element in a parameter set. The syntax element is included in at least one of a picture parameter set or a sequence parameter set. The decoder is configured to decode a constraint flag of a parameter set from the bitstream, and the constraint flag indicates to the decoder that any access unit (30) of the bitstream, the access unit (30) includes a mixture of one or more NAL units of the non-IRAP leading picture NAL unit type (RASL or RADL) and one or more NAL units (304) of the stepwise temporal sub-layer access unit type (STSA), satisfies the constraints associated with the stepwise temporal sub-layer access unit type (STSA).

[0026] Preferably, the constraint indicates to the decoder that if a temporal sub-layer that is higher than the stepwise temporal sub-layer access (STSA) unit type precedes the stepwise temporal sub-layer access unit type (STSA), any picture that is allowed to follow in decoding order and is associated with the next higher temporal sub-layer is not allowed to reference pictures in the next higher temporal sub-layer. The decoder is configured to decode a constraint flag of a parameter set from the bitstream, the constraint flag indicating to the decoder that any access unit of the bitstream that includes one or more NAL units of a non-IRAP leading picture NAL unit type (RASL or RADL) and one or more NAL units of the stepwise temporal sub-layer access unit type (STSA) does not comply with the constraint associated with the temporal sub-layer access (STSA) unit type, and wherein the decoder is further configured to, in response to the constraint flag, refrain from parsing the one or more NAL units of the stepwise temporal sub-layer access (STSA) unit type and implicitly infer stepwise temporal sub-layer access decoding.

[0027] A decoder for decoding video content according to the present invention, the decoder being configured to: decode a bitstream and decode a constraint flag of a parameter set from the bitstream, the constraint flag indicating to the decoder that, within the bitstream, any access unit that includes NAL units of a non-IRAP leading picture NAL unit type (e.g., RASL or RADL) complies with the constraint associated with the stepwise temporal sub-layer access (STSA) unit type.

[0028] According to an improvement of the decoder described above, in case B), the IRAP NAL unit type of the NAL units of the second sub-bitstream belongs to the IRAP NAL unit type associated with a closed GOP structure, and the IRAP NAL unit type of the NAL units of the first sub-bitstream belongs to the IRAP NAL unit type associated with an open GOP structure.

[0029] The IRAP NAL unit type of the NAL units of the second sub-bitstream is a CRA unit type (CRA_with_RADL) having a randomly accessible decodable leading picture unit type, and wherein the IRAP NAL unit type of the NAL units of the first sub-bitstream is a CRA unit type (CRA_with_RASL) having a randomly accessible skipped leading picture unit type.

[0030] Or in case B), the IRAP NAL unit type of the NAL unit (304) of the second sub-bitstream is Instantaneous Decoding Refresh with Random Access Decodable Leading Picture Unit type (IDR_W_RADL), and the IRAP NAL unit type of the NAL unit of the first sub-bitstream is Clean Random Access unit type (CRA).

[0031] The decoder is configured to: decode and present a co-located spatial segment of one or more pictures of an access unit that follows, in bitstream order, the at least one access unit associated with the NAL unit of the second sub-bitstream, the co-located spatial segment being co-located with a second spatial segment of the picture of the at least one access unit associated with the NAL unit of the second sub-bitstream, and mark as discarded a co-located spatial segment of one or more pictures of an access unit that follows, in bitstream order, the at least one access unit associated with the NAL unit of the first sub-bitstream, the co-located segment being co-located with a first spatial segment of the picture of the at least one access unit associated with the NAL unit of the first sub-bitstream.

[0032] The decoder is configured to: decode and present a co-located spatial segment of one or more pictures of an access unit that follows, in bitstream order, the at least one access unit associated with the NAL unit of the second sub-bitstream, the co-located spatial segment being co-located with a second spatial segment of the picture of the at least one access unit associated with the NAL unit of the second sub-bitstream, and in case of starting decoding, discard the one or more pictures of the access unit that follows, in bitstream order, the at least one access unit, and resume picture output after the one or more discarded pictures.

[0033] The decoder is configured to, during continuous decoding during normal playback, decode and present co-located spatial segments of one or more pictures of an access unit that follows, in bitstream order, the at least one access unit associated with the NAL unit of the second sub-bitstream, the co-located spatial segments being co-located with the second spatial segment of the picture of the at least one access unit associated with the NAL unit of the second sub-bitstream, and mark the co-located spatial segments of one or more pictures of the access unit that follows, in bitstream order, the at least one access unit associated with the NAL unit of the first sub-bitstream, as discarded, the co-located segments being co-located with the first spatial segment of the picture of the at least one access unit associated with the NAL unit of the first sub-bitstream, and, in the case of starting decoding, discard the one or more pictures of the access unit that follows, in bitstream order, the at least one access unit, and resume picture output after the one or more discarded pictures.

[0034] The decoder is configured to mark the discontinuous spatial segments to a subsequent processing chain by means of an SEI (Supplemental Enhancement Information) message. The decoder is configured to derive from the bitstream an indication that indicates to the decoder that the at least one access unit associated with the first sub-bitstream and the second sub-bitstream is decoded in a manner corresponding to the clean random access NAL unit type, even if the access unit includes a NAL unit of an IRAP NAL unit type (e.g., IDR_W_RADL) different from the clean random access NAL unit type.

[0035] The present invention also provides a device for processing video content, the device being configured to: provide a first sub-bitstream related to a first spatial segment of a picture of the video content, and provide a second sub-bitstream related to a second spatial segment of the picture of the video content, wherein the first sub-bitstream and the second sub-bitstream are parts of a bitstream, and wherein: A) the bitstream includes at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to a non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), which is mixed with at least one different second NAL unit of the second sub-bitstream belonging to a different non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), or B) the bitstream includes at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to an IRAP NAL unit type (e.g., IDR, CRA), which is mixed with at least one second NAL unit of the second sub-bitstream belonging to another IRAP NAL unit type (e.g., IDR, CRA), or C) the bitstream includes at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to an IRAP NAL unit type (e.g., IDR, CRA), which is mixed with at least one second NAL unit of the second sub-bitstream belonging to either a non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or an STSA NAL unit type.

[0036] According to the improvement of the device, the spatial segment of the picture relates to independently encoded sub-pictures of the picture of the video content, and the decoder is configured to use boundary extension for motion-compensated prediction across sub-picture edges. In case A), the non-IRAP NAL unit type of the NAL unit of the first sub-bitstream is a Random Access Decodable Leading Picture Unit type (RADL), and the non-IRAP NAL unit type of the NAL unit of the second sub-bitstream is a Trailing Picture Unit type (TRAIL). Or in case A), the non-IRAP NAL unit type of the NAL unit of the first sub-bitstream is a Random Access Skipped Leading Picture Unit type (RASL), and the non-IRAP NAL unit type of the NAL unit of the second sub-bitstream is a Trailing Picture Unit type (TRAIL). Or in case A), the non-IRAP NAL unit type of the NAL unit of the first sub-bitstream is a Random Access Skipped Leading Picture Unit type (RASL), and the non-IRAP NAL unit type of the NAL unit of the second sub-bitstream is a Random Access Decodable Leading Picture Unit type (RADL). Or in case A), the non-IRAP NAL unit type of the NAL unit of the first sub-bitstream is a Stepwise Temporal Sub-layer Access Unit type (STSA), and the non-IRAP NAL unit type of the NAL unit of the second sub-bitstream is a Trailing Picture Unit type (TRAIL).

[0037] Preferably, the device is configured to set an indication in the bitstream, which is used to indicate that: the NAL units of the non-IRAP NAL unit type in the second sub-bitstream are mixed with the NAL units of the non-IRAP NAL unit type in the first sub-bitstream, the NAL units belong to the trailing picture unit type (TRAIL), and the NAL units are any one of the leading picture unit types of the random access skip leading picture unit type (RASL) or the random access decodable leading picture unit type (RADL). The indication includes a syntax element in a parameter set. The syntax element is included in at least one of the picture parameter set or the sequence parameter set. If the NAL unit of the non-IRAP NAL unit type in the first sub-bitstream is of the random access skip leading picture unit type (RASL), and the NAL unit of the non-IRAP NAL unit type in the second sub-bitstream is of the trailing picture unit type (TRAIL), then the device is configured to set an indication in the bitstream for indicating one of the following: a) ignore or suppress the output, or b) output and mark as a discontinuous spatial segment those first spatial segments of the pictures of the video content in the first sub-bitstream that correspond to the NAL units of the random access skip leading picture unit type (RASL) in the first sub-bitstream. Or in case b), the device is configured to mark the discontinuous space to a subsequent processing chain by means of an SEI (Supplemental Enhancement Information) message. The device is configured to set a constraint indication in the parameter set, which is used to indicate that: if the non-IRAP NAL unit type of the NAL unit in the second sub-bitstream belongs to a non-leading picture NAL unit type (e.g., TRAIL or STSA), then the non-IRAP NAL unit type of the NAL unit in the first sub-bitstream exclusively belongs to the random access decodable leading picture unit type (RADL) in the case that the IRAP-NAL unit in the associated first sub-bitstream belongs to the clean random access unit type (CRA).

[0038] Preferably, the non-IRAP NAL unit type of the NAL unit of the second bitstream belongs to the trailing picture unit type (TRAIL). In case A), the non-IRAP NAL unit type of the NAL unit of the first sub-bitstream is the random access decodable leading picture unit type (RADL), and the non-IRAP NAL unit type of the NAL unit of the second sub-bitstream is the progressive temporal sub-layer access unit type (STSA). Or in case A), the non-IRAP NAL unit type of the NAL unit of the first sub-bitstream is the random access skipped leading picture unit type (RASL), and the non-IRAP NAL unit type of the NAL unit of the second sub-bitstream is the progressive temporal sub-layer access unit type (STSA).

[0039] Preferably, the device is configured to set an indication in the bitstream for indicating that: the NAL unit of the non-IRAP NAL unit type of the second sub-bitstream is mixed with the NAL unit of the non-IRAP NAL unit type of the first sub-bitstream, the NAL unit (304) belongs to the progressive temporal sub-layer access unit type (STSA), and the NAL unit belongs to any one of the leading picture unit types of the random access skipped leading picture unit type (RASL) or the random access decodable leading picture unit type (RADL). The indication includes a syntax element in a parameter set. The syntax element is included in at least one of a picture parameter set or a sequence parameter set. The device is configured to set a constraint flag in the parameter set of the bitstream, and the constraint flag indicates that any access unit of the bitstream, the access unit including a mixture of one or more NAL units of the non-IRAP leading picture NAL unit type (RASL or RADL) and one or more NAL units (304) of the progressive temporal sub-layer access unit type (STSA), satisfies the constraints associated with the progressive temporal sub-layer access unit type (STSA).

[0040] Preferably, the constraint indicates that if the next higher temporal sub-layer than the progressive temporal sub-layer access unit type (STSA) precedes the progressive temporal sub-layer access unit type (STSA), then any picture following in decoding order and associated with the next higher temporal sub-layer is not allowed to reference pictures in the next higher temporal sub-layer.

[0041] The device is configured to set a constraint flag in the parameter set of the bitstream, the constraint flag indicating that any access unit of the bitstream, the access unit including a mixture of one or more NAL units of a non-IRAP leading picture NAL unit type (RASL or RADL) and one or more NAL units of the stepwise temporal sublayer access unit type (STSA), does not comply with the constraints associated with the temporal sublayer access (STSA) unit type, and implicitly infers stepwise temporal sublayer access decoding rather than parsing the one or more NAL units of the stepwise temporal sublayer access unit type (STSA).

[0042] The present invention also provides a device for processing video content, the device being configured to: process a bitstream, and set a constraint flag in the parameter set of the bitstream, the constraint flag indicating that, within the bitstream, any access unit including NAL units of a non-IRAP leading picture NAL unit type (RASL or RADL) complies with the constraints associated with the stepwise temporal sublayer access unit type (STSA).

[0043] In case B), the IRAP NAL unit type of the NAL units of the second sub-bitstream belongs to the IRAP NAL unit type associated with a closed GOP structure, and the IRAP NAL unit type of the NAL units of the first sub-bitstream belongs to the IRAP NAL unit type associated with an open GOP structure. The IRAP NAL unit type of the NAL units (304) of the second sub-bitstream is a CRA unit type (CRA_with_RADL) having a random access decodable leading picture unit type, and, wherein, the IRAP NAL unit type of the NAL units of the first sub-bitstream is a CRA unit type (CRA_with_RASL) having a random access skipped leading picture unit type.

[0044] Or in case B), the IRAP NAL unit type of the NAL units of the second sub-bitstream is an instantaneous decoding refresh (IDR_W_RADL) having a random access decodable leading picture unit type, and the IRAP NAL unit type of the NAL units of the first sub-bitstream is a clean random access unit type (CRA). The device is configured to set an indication in the bitstream for indicating that, even if the access unit includes NAL units of an IRAP NAL unit type different from the clean random access NAL unit type (e.g., IDR_W_RADL), the at least one access unit related to the first sub-bitstream and the second sub-bitstream will be processed in a manner corresponding to the clean random access NAL unit type. The device includes at least one of an encoder, a combiner, and a network node.

[0045] According to another method for decoding video content according to the present invention, the method includes: decoding a bitstream, the bitstream including a first sub-bitstream related to a first spatial segment of a picture of the video content and a second sub-bitstream related to a second spatial segment of the picture of the video content, wherein A) the bitstream includes at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to a non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), which is mixed with at least one different second NAL unit of the second sub-bitstream belonging to a different non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), or B) the bitstream includes at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to an IRAP NAL unit type (e.g., IDR, CRA), which is mixed with at least one second NAL unit of the second sub-bitstream belonging to another IRAP NAL unit type (e.g., IDR, CRA), or C) the bitstream includes at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to an IRAP NAL unit type (e.g., IDR, CRA), which is mixed with at least one second NAL unit of the second sub-bitstream belonging to either a non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or an STSA NAL unit type.

[0046] A method for processing video content according to the present invention, the method comprising: providing a first sub-bitstream associated with a first spatial segment of a picture of the video content, and providing a second sub-bitstream associated with a second spatial segment of the picture of the video content, wherein the first sub-bitstream and the second sub-bitstream are parts of a video bitstream, and wherein A) the bitstream comprises at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to a non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), which is mixed with at least one different second NAL unit of the second sub-bitstream belonging to a different non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), or B) the bitstream comprises at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bitstream belongs to an IRAP NAL unit type (e.g., IDR, CRA), which is mixed with at least one second NAL unit of the second sub-bitstream belonging to another IRAP NAL unit type (e.g., IDR, CRA), or C) the bitstream comprises at least one access unit, within the at least one access unit, at least one first NAL unit of the first sub-bit belongs to an IRAP NAL unit type (e.g., IDR, CRA), which is mixed with at least one second NAL unit of the second sub-bitstream belonging to either a non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or an STSA NAL unit type. The method may comprise combining the first sub-bitstream and the second sub-bitstream into the video bitstream.

[0047] The present invention may provide a data stream, which can be obtained by the above method. The present invention also provides a computer-readable digital storage medium, on which a computer program has been stored, the computer program having program code for implementing the method according to the above when run on a computer.

[0048] A decoder for decoding video content according to the present invention, the decoder being configured to: decode a bitstream including a first sub-bitstream and a second sub-bitstream, wherein the decoder is configured to derive information for setting the most significant POC part (POC MSB) of the POC of a picture referred to by the NAL unit from the slice headers of the NAL units of the first sub-bitstream and the second sub-bitstream, the NAL unit belonging to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP), and, A) wherein, if at least one NAL unit within the access unit belongs to a non-instantaneous decoding refresh unit type (e.g., CRA), then it is required to set the most significant POC part (POC MSB) to a predetermined value for all NAL units of the IRAP NAL unit type within the access unit of the bitstream, or B) wherein any picture reference across any access unit of the bitstream is prohibited, within the access unit, all NAL units belong to the IRAP NAL unit type, and have slice headers in which the most significant POC part (POC MSB) is set to a predetermined value, or C) wherein any picture reference by a picture of a second access unit following the bitstream in the bitstream order to a picture of a first access unit is prohibited, the first access unit being composed of NAL units having slice headers in which the most significant POC part (POC MSB) is set to a first predetermined value, and the first access unit preceding the second access unit of the bitstream in the bitstream order, within the second access unit, all NAL units belong to the IRAP NAL unit type, and have slice headers in which the most significant POC part (POC MSB) is set to a second predetermined value.

[0049] According to the decoder described above, in case A), the predetermined value is equal to or greater than the most significant POC part (POC MSB) of the POC associated with a picture preceding the picture referred to by the NAL unit. The predetermined value is equal to zero. The predetermined value is equal to the implicitly derived most significant POC part. In case B), the predetermined value is less than the most significant POC part (POC MSB) of the POC associated with a picture preceding the picture referred to by the NAL unit. The predetermined value is equal to zero. The predetermined value is equal to the implicitly derived most significant POC part. In case C), the first predetermined value is greater than the second predetermined value. The second predetermined value is equal to zero.

[0050] or the second predetermined value is equal to the most significant POC part derived implicitly. The bitstream is a multi-layer bitstream, and the first sub-bitstream belongs to the first layer, and the second sub-bitstream belongs to a different second layer. The picture reference includes at least one picture reference among a short-term picture reference and a long-term picture reference.

[0051] The present invention also provides an encoder for encoding video content, the encoder being configured to: encode a bitstream including a first sub-bitstream and a second sub-bitstream, wherein the encoder is configured to set, in a slice header of a NAL unit of the first sub-bitstream and the second sub-bitstream, the most significant POC part (POC MSB) of the POC of the picture referred to by the NAL unit, the NAL unit belonging to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP), A) wherein, if at least one NAL unit within the access unit belongs to a non-instantaneous decoding refresh unit type (e.g., CRA), then it is required to set the most significant POC part (POC MSB) to a predetermined value for all NAL units of the IRAP NAL unit type within the access unit of the bitstream, or B) wherein any picture reference is prohibited from spanning any access unit of the bitstream in decoding order, within the access unit, all NAL units belong to the IRAP NAL unit type and have a slice header in which the most significant POC part (POC MSB) is set to a predetermined value, or C) wherein any picture reference by a picture of a second access unit following the bitstream in the bitstream order to a picture of a first access unit is prohibited, the first access unit being composed of NAL units having a slice header in which the most significant POC part (POC MSB) is set to a first predetermined value, and the first access unit preceding the second access unit of the bitstream in the bitstream order, within the second access unit, all NAL units belong to the IRAP NAL unit type and have a slice header in which the most significant POC part (POC MSB) is set to a second predetermined value.

[0052] According to the encoder, in case A), the predetermined value is equal to or greater than the most significant POC part (POC MSB) of the POC associated with the picture preceding the picture referred to by the NAL unit. The predetermined value is equal to zero. The predetermined value is equal to the implicitly derived most significant POC part. In case B), the predetermined value is less than the most significant POC part (POC MSB) of the POC associated with the picture preceding the picture referred to by the NAL unit. The predetermined value is equal to zero. The predetermined value is equal to the implicitly derived most significant POC part. In case C), the first predetermined value is greater than the second predetermined value. The second predetermined value is equal to zero. The second predetermined value is equal to the implicitly derived most significant POC part. The bitstream is a multi-layer bitstream, and wherein the first sub-bitstream belongs to the first layer, and the second sub-bitstream belongs to a different second layer. The picture reference includes at least one of a short-term picture reference and a long-term picture reference.

[0053] According to another method for decoding video content according to the present invention, the method includes: decoding a bitstream including a first sub-bitstream and a second sub-bitstream, and deriving information for setting the most significant POC part (POC MSB) of the POC of the picture referred to by the NAL unit from the slice headers of the NAL units of the first sub-bitstream and the second sub-bitstream, the NAL unit belonging to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP), and A) wherein, if at least one NAL unit in the access unit belongs to the non-instantaneous decoding refresh unit type (e.g., CRA), then it is required to set the most significant POC part (POC MSB) to a predetermined value for all NAL units of the IRAP NAL unit type in the access unit of the bitstream, or B) wherein any picture reference is prohibited from crossing any access unit of the bitstream in decoding order, in the access unit, all NAL units belong to the IRAP NAL unit type and have slice headers in which the most significant POC part (POC MSB) is set to a predetermined value, or C) wherein any picture reference of the picture of the first access unit by the picture of the second access unit following the bitstream in the bitstream order is prohibited, the first access unit is composed of NAL units having slice headers in which the most significant POC part (POC MSB) is set to a first predetermined value, and the first access unit precedes the second access unit of the bitstream in the bitstream order, in the second access unit, all NAL units belong to the IRAP NAL unit type and have slice headers in which the most significant POC part (POC MSB) is set to a second predetermined value.

[0054] According to another method for encoding video content according to the present invention, the method includes: encoding a bitstream, the bitstream including a first sub-bitstream and a second sub-bitstream, and setting, in slice headers of NAL units of the first sub-bitstream and the second sub-bitstream, a most significant POC part (POC MSB) of a picture referenced by the NAL unit, where, if the NAL unit belongs to a non-instantaneous decoding refresh unit type (e.g., CRA), it is required to set the most significant POC part (POC MSB) to a predetermined value for all NAL units of the IRAP NAL unit type within an access unit of the bitstream, or B) where any picture reference is prohibited from crossing any access unit of the bitstream in decoding order, within the access unit, all NAL units belong to the IRAP NAL unit type and have slice headers in which the most significant POC part (POC MSB) is set to a predetermined value, or C) where any picture reference from a picture of a second access unit following the bitstream in the bitstream order to a picture of a first access unit is prohibited, the first access unit being composed of NAL units having slice headers in which the most significant POC part (POC MSB) is set to a first predetermined value, and the first access unit preceding the second access unit of the bitstream in the bitstream order, within the second access unit, all NAL units belong to the IRAP NAL unit type and have slice headers in which the most significant POC part (POC MSB) is set to a second predetermined value.

[0055] The present invention also provides a data stream that can be obtained by the above method. A computer-readable digital storage medium is also provided, on which a computer program is stored, the computer program having program code for implementing the above method when run on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Hereinafter, embodiments of the present disclosure are described in more detail with reference to the accompanying drawings, in which: Figure 1 A schematic diagram showing a bitstream that can be processed by an inventive decoder and device for encoding / merging according to an embodiment. Figure 2 A schematic diagram showing a bitstream with an access unit having an IDR_N_LP NAL unit type, while all other access units have the same NAL unit type. Figure 3A schematic diagram showing a bitstream with access units having an IDR_W_LP NAL unit type and corresponding RADL unit types mixed with TRAIL unit types. Figure 4 A schematic diagram showing a bitstream with access units having a CRA NAL unit type and corresponding RASL unit types mixed with TRAIL unit types. Figure 5 Showing a full 360-degree low-resolution thumbnail with selective high-resolution tiles. Figure 6 Showing an exemplary coding structure where, in one tile (top example), the picture uses the STSA NAL unit type to indicate progressive temporal sub-layer access, while the same picture in another tile (bottom example) uses the RASL NAL unit type. Figure 7 Showing an exemplary coding structure for high-resolution regions using a closed GOP structure (top) and a high-resolution region using an open GOP structure (bottom). Figure 8 Showing an exemplary coding structure of a multi-layer bitstream where POC MSB reset is performed at layer-aligned RAP pictures. Figure 9 Showing an exemplary coding structure of a multi-layer bitstream where POC MSB reset is prohibited if not all NAL units are IDR. Figure 10 Showing an exemplary coding structure of a multi-layer bitstream where POC MSB reset is allowed because all NAL units are IDR. Figure 11 Showing an exemplary coding structure of a multi-layer bitstream where POC MSB reset is performed and LT reference is prohibited. Figure 12 Showing an exemplary coding structure of a multi-layer bitstream where all POC MSBs are equal to 0 and LT reference is prohibited. Figure 13 Showing an exemplary coding structure of a multi-layer bitstream where all POC MSBs are equal to 0 and LT reference is allowed, and Figure 14 Showing an exemplary coding structure of a multi-layer bitstream where POC MSB reset is not performed and LT reference is allowed. Detailed Description

[0057] In the following description, equal or equivalent elements or elements having equal or equivalent functionality are referred to by equal or equivalent reference numerals.

[0058] Method steps depicted with and described with reference to block diagrams can also be executed in an order different from the depicted and / or described order. In addition, method steps involving specific features of a device can be replaced by those features of the device, and vice versa.

[0059] Mix the picture and the NAL unit type in the access unit 1. Mix the unit types within the encoded picture For introduction purposes, Figure 1 The structure of an exemplary video bitstream 11 according to an embodiment of the present invention is shown. The video bitstream 11 can be processed by a decoder 100 and a device 10 for encoding and / or combining the video bitstream 11. The video bitstream 11 includes a sequence of consecutive pictures 12a, 12b, 12c, where two or more of the consecutive pictures 12a, 12b, 12c can be grouped together in a GOP (Group of Pictures) 12, and the GOP (Group of Pictures) 12 can represent at least part of the expected video content.

[0060] For encoding purposes, each picture 12a, 12b, 12c can be subdivided into spatial segments, such as slices, tiles, or blocks. In this non-limiting example, each of the pictures 12a, 12b, 12c is subdivided into two spatial segments, namely, a first (e.g., left) spatial segment 131 and a second (e.g., right) spatial segment 132.

[0061] The bitstream 11 can be subdivided into a first sub-bitstream 11-1 related to the first spatial segment 131 of the pictures 12a, 12b, 12c, and into a second sub-bitstream 11-2 related to the second spatial segment 132 of the pictures 12a, 12b, 12c.

[0062] Some of the pictures 12a, 12b can be encoded dependently, for example, by predictive coding. Some of these pictures 12b can be predicted by single prediction (e.g., by referring to a single preceding picture), while some other pictures 12a can be predicted by double prediction (e.g., by referring to a preceding picture and a subsequent picture). Some other pictures 12c can be encoded independently, i.e., these pictures 12c do not refer to preceding or subsequent pictures. The independently encoded pictures 12c can provide an (intra) random access point ((I)RAP), which represents a position in the bitstream 11 at which the decoder can start decoding subsequent pictures without having to decode any pictures that occur earlier in the bitstream 11. For example, picture 12c is an independently encoded picture, which can also be referred to as an (I)RAP picture. In the present disclosure, the terms RAP and IRAP can be used synonymously.

[0063] The bitstream 11 may include one or more access units 30 for accessing the bitstream 11. In Figure 1 a non-limiting example, the access unit 30 may include picture segments 131, 132 (aligned with respect to the presentation order) included in the first sub-bitstream 11-1 and the second sub-bitstream 11-2. Pictures 12a, 12b, 12c respectively including these aligned picture segments 131, 132 may be grouped into different types, such as being grouped into IRAP pictures (as mentioned above) and non-IRAP pictures.

[0064] The Network Abstraction Layer (NAL) may provide a high-level syntax for defining those picture segments 131, 132 and pictures 12a, 12b, 12c. In particular, the NAL unit may be associated with the picture segments 131, 132 for each of the pictures 12a, 12b, 12c. Thus, in the present disclosure, the above-mentioned picture segments 131, 132 of the first sub-bitstream 11-1 and the second sub-bitstream 11-2 may also be synonymously referred to as NAL units. Therefore, the first sub-bitstream 11-1 and the second sub-bitstream 11-2 may also be referred to as NAL unit streams. Each NAL unit 131, 132 may include a specific NAL unit type. Further, the NAL unit type may identify the above-mentioned type of the corresponding picture included in the NAL unit, that is, whether the corresponding NAL unit belongs to the IRAP type or the non-IRAP type.

[0065] There are multiple applications that rely on mixing IRAP types within a picture, where the IRAP type is the random access attribute of the access unit (AU) 30 or a part thereof indicated by the NAL unit type. Use cases can be found in the single-layer or multi-layer codec domain, that is: a single-layer codec bitstream, whose pictures are composed of slices that are IRAP, while other slices of the same picture are not IRAP; or a multi-layer codec bitstream, whose access units include IRAP in one layer, while other layers include non-IRAP pictures.

[0066] Currently, support for mixed NAL unit types is given in the prior art, where a single type of IRAP NAL unit type can be mixed with TRAIL pictures within a picture. However, this does not properly cover the use cases that motivate the need for features as described below, and the present invention provides a solution for such use cases.

[0067] The main use case that motivates the need for mixed NAL unit types is 360° video, where tile streaming is used and only a subset of the entire 360° is transmitted at high resolution and the rest at low resolution. When a change in the viewing orientation occurs, then some tiles that were at high resolution are no longer required, and new tiles that were previously shown at low resolution are required to be at high resolution from that time instance forward. These tiles are the only ones that encounter a change (from high resolution to low resolution and vice versa). These tiles are then downloaded using a representation starting with an IRAP, while other tiles do not need to have this property. When the bitstreams corresponding to different tiles are merged into a single bitstream, some of the tiles contain NAL unit types corresponding to IRAP pictures, while other tiles contain NAL unit types of non-IRAP pictures.

[0068] In addition, there are some other use cases where it is desirable to allow an AU to carry NAL units of different types. For example, in a scenario where there is a Region of Interest (RoI) in a 1080p picture with a 720p RoI inside, for the case of using the same bitstream to feed two types of receivers (e.g., one type of receiver is interested in the whole video and the other type of receiver is only interested in the RoI), one can envision different parts of the video (the RoI or the rest) having different RAP periods.

[0069] In any case of 360° video streaming (where merging of different bitstreams occurs) or RoI streaming (where the video is initially encoded for different regions with different RAP periods), the decoder will encounter a bitstream for which the NAL units within the encoded pictures are not necessarily the same. Therefore, the present invention provides a solution for mixing NAL units of different NAL unit types.

[0070] 1.1. Mixing trailing pictures (prior art) Currently, the following solution is proposed in the prior art: the PPS flag indicates whether an RAP picture is mixed with a non-RAP picture, and then the NAL unit types that can be encountered in the NAL units of the mixed AU are the TRAIL NAL unit types for non-IRAP types (up to VCL_RSV_6 or GDR) and a single type of IDR_W_RADL (IDR with leading picture), IDR_N_LP (IDR without leading picture), and CRA (clean random access). Therefore, the prior art proposes to mix IRAP pictures with TRAIL pictures.

[0071] Figure 2An example according to the prior art is shown, in which a NAL unit 201 of the IRAP NAL unit type is mixed with a NAL unit 202 of the trailing picture type (TRAIL). The IRAP NAL unit type in this example is an IDR (IDR_N_LP) without a leading picture. All other access units have NAL units of the same NAL unit type (i.e., TRAIL).

[0072] 1.2. Mixing leading pictures However, these measures are not sufficient for the envisaged applications where an open GOP structure can be applied to introduce a leading picture (LP). Figure 3 An exemplary situation is shown, in which a NAL unit 301 of the IRAP NAL unit type (e.g., IDR_W_RADL) is shown together with its associated leading picture 302. In this case, the leading picture 302 can be a NAL unit 302 of the NAL unit type RADL (random access decodable leading picture). The NAL unit 301 of the IRAP type (e.g., IDR_W_RADL) can be mixed with a NAL unit 303 of a non-IRAP NAL unit type (e.g., TRAIL). The associated NAL unit 302 of the leading picture type (RADL) of the non-IRAP type can be mixed with a NAL unit 304 of a different non-IRAP type (e.g., TRAIL).

[0073] Figure 4 Another exemplary situation is shown, in which a NAL unit 301 of the IRAP NAL unit type (e.g., CRA) is shown together with its associated leading picture 302. In this case, the leading picture 302 can be a NAL unit 302 of the NAL unit type RASL (random access skipped leading picture). Alternatively, but not shown, the associated leading picture 302 can be a NAL unit 302 of the NAL unit type RADL (random access decodable leading picture). The NAL unit 301 of the IRAP type (e.g., CRA) can be mixed with a NAL unit 303 of a non-IRAP NAL unit type (e.g., TRAIL). The associated NAL unit 302 of the leading picture type (e.g., RASL) of the non-IRAP type can be mixed with a NAL unit 304 of a different non-IRAP type (e.g., TRAIL).

[0074] Therefore, it is necessary to process the encoded pictures of the NAL unit types that need to be mixed not only for the cases including IRAP NAL units, but also for the non-IRAP NAL units only included in the mixed pictures.

[0075] In short, the prior art may suggest mixing IRAP NAL units with non-IRAP NAL units of the TRAIL type or mixing non-IRAP NAL units of the same type, i.e., mixing NAL units of the TRAIL type with NAL units of the TRAIL type.

[0076] In contrast, the present invention suggests mixing a first non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA) with at least one different second non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA).

[0077] Thus, according to an embodiment, there is provided a decoder 100 for decoding video content 12. The decoder 100 is configured to decode a bitstream 11 that includes a first sub-bitstream 11-1 related to a first spatial segment 131 of pictures 12a, 12b, 12c of the video content 12 and a second sub-bitstream 11-2 related to a second spatial segment 132 of pictures 12a, 12b, 12c of the video content 12. According to this embodiment, the bitstream 11 includes at least one access unit 30. Within the at least one access unit 30, at least one first NAL unit 302 of the first sub-bitstream 11-1 belongs to a non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA), and at least one first NAL unit 302 of the first sub-bitstream 11-1 is mixed with at least one different second NAL unit 304 of the second sub-bitstream 11-2 that belongs to a different non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA).

[0078] For example, a TRAIL picture may be mixed with a RADL picture or a RASL picture: ·TRAIL + RADL ·TRAIL + RASL Thus, according to an embodiment, the decoder 100 may be configured to process the bitstream 11, where the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a Random Access Decodable Leading Picture Unit type (RADL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a Trailing Picture Unit type (TRAIL).

[0079] According to another embodiment, the decoder 100 may be configured to process a bitstream 11, wherein the non-IRAP NAL unit type of the NAL units 302 of the first sub-bitstream 11-1 is a random access skipped leading picture unit type (RASL), and the non-IRAP NAL unit type of the NAL units 304 of the second sub-bitstream 11-2 is a trailing picture unit type (TRAIL).

[0080] In one embodiment of the present invention, in the bitstream 11, for example, as a flag in a parameter set (PPS, SPS, VPS), there is an indication that the NAL unit types within an encoded picture can be mixed as follows: ·TRAIL+RADL ·TRAIL+RASL Regarding the indication that TRAIL+(CRA xor IDR_W_LD xor IDR_N_P) can be mixed, such signaling can be carried in the PPS.

[0081] For example, if the flag is not set (i.e., flag = 0), then all NAL units of a picture may include the same NAL unit type, and the picture or PU is said to have the same NAL unit type as the encoded slice NAL units of the picture or PU.

[0082] In other cases, if the flag is set (i.e., flag = 1), then the picture may include at least two sub-pictures. Additionally or alternatively, the NAL units of the picture may include at least two different NAL unit types. Further additionally or alternatively, the NAL units of the picture may not include the NAL unit type of GDR. Even further additionally or alternatively, if a NAL unit of the picture belongs to at least one of the following NAL unit types: · Instantaneous decoder refresh with random access to a decoded leading picture (IDR_W_RADL) · Instantaneous decoder refresh without a leading picture (IDR_N_LP) · Clean random access (CRA) Then all other available NAL units of the picture may include one of the three unit types (IDR_W_RADL, IDR_N_LP, CRA) noted above or a trailing picture unit type (TRAIL).

[0083] Alternatively, since the RADL and RASL pictures do not have a canonical decoding process different from that of, for example, a TRAIL picture such as an IRAP picture, this indication can be made at the SPS level. In one embodiment, for example, this indication can be in the form of a constraint flag (e.g., a general constraint flag). The general constraint flag can indicate whether certain mixes of NAL unit types are allowed. If such a general constraint flag is set (i.e., general constraint flag = 1), it can indicate that there are no mixed NAL unit types in the picture and that the flags mentioned above (i.e., flag = 0) in the parameter sets (PPS, SPS, VPS) are not set. If the general constraint flag is not set (i.e., general constraint flag = 0), the constraints mentioned above may not be imposed.

[0084] Thus, according to an embodiment, the decoder 100 can be configured to derive an indication from the bitstream 11, where the indication explicitly signals to the decoder 100 the mixing of a NAL unit 304 of a non-IRAP NAL unit type of the second sub-bitstream 11-2 (the NAL unit 304 belongs to the trailing picture unit type (TRAIL)) with a NAL unit 302 of a non-IRAP NAL unit type of the first sub-bitstream 11-1 (the NAL unit 302 is either a random access skipped leading picture unit type (RASL) or a random access decodable leading picture unit type (RADL)).

[0085] According to an embodiment, the indication can include a syntax element in the parameter set. For example, the syntax element can be included in at least one of the picture parameter set (PPS) or the sequence parameter set (SPS).

[0086] However, when tiles with RASL attributes and tiles of TRAIL pictures are mixed within a picture, assuming a streaming scenario based on 360-degree video tiles, it is possible for tiles to change position at partial RAP points and, therefore, the RASL tiles cannot be correctly reconstructed because the reference pictures used will be incorrect (i.e., still depicting the content of other tiles). Identifying those incorrectly reconstructed picture regions is crucial for applications, and deleting the entire mixed leading picture (containing TRAIL + RASL tiles) is not an option for the sake of playback continuity. Instead, the client will identify the error regions in the decoded and output pictures and will not use those regions for subsequent processing.

[0087] Instead, considering, for example, Figure 5With the settings described in [reference], the client can use the low-resolution version 51 contained in the picture for the content that is incorrectly reconstructed in those mixed leading pictures. In one embodiment, the area corresponding to the potentially incorrect reconstructed tile is indicated to subsequent processing, for example, through an SEI message or by an external means (e.g., an application). The described indication will mean that the area corresponding to the NAL unit with the NAL unit type of RASL will be ignored / not output.

[0088] Thus, according to an embodiment, if the NAL unit 302 of the non-IRAP NAL unit type in the first sub-bitstream 11-1 is of the random access skipped leading picture unit type (RASL), and the non-IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is of the trailing picture unit type (TRAIL), then the decoder 100 can be configured to derive an indication from the bitstream 11.

[0089] In one example, the indication can direct the decoder 100 to ignore or refrain from outputting those first spatial segments 131 of pictures 12a, 12b, 12c of the video content 12 of the first sub-bitstream 11-1, where the first spatial segments 131 correspond to the NAL units 302 of the random access skipped leading (RASL) picture unit type of the first sub-bitstream 11-1.

[0090] In an alternative example, the indication can direct the decoder 100 to: output those first spatial segments 131 of pictures 12a, 12b, 12c of the video 12 content of the first sub-bitstream 11-1, where the first spatial segments 131 correspond to the NAL units 302 of the random access skipped leading picture unit type (RASL) of the first sub-bitstream 11-1; and additionally mark the first spatial segments 131 as broken spatial segments. For example, the decoder 100 can be configured to mark the broken spatial segments 131 for the subsequent processing chain by means of an SEI (Supplemental Enhancement Information) message.

[0091] Moreover, as another option, mixing of RASL pictures with trailing pictures (TRAIL) is completely avoided, and thus only mixing of RADL pictures with TRAIL pictures is allowed. Therefore, in one embodiment, there is the following constraint indicated in the SPS: when a CRA is mixed with a TRAIL picture, the associated leading picture must only belong to the RADL type.

[0092] Thus, according to an embodiment, the decoder 100 may be configured to decode a constraint indication from a parameter set, wherein the constraint indication indicates to the decoder 100 that if the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 belongs to a non-leader picture NAL unit type (e.g., TRAIL or STSA), then in the case where the IRAP-NAL unit 301 of the associated first sub-bitstream 11-1 of the NAL unit 302 of the first sub-bitstream 11-1 belongs to a clean random access unit type (CRA), the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 exclusively belongs to a random access decodable leading RADL picture unit type.

[0093] This may apply in particular to an embodiment where the non-IRAP NAL unit type of the NAL unit 304 of the second bitstream 11-2 belongs to a trailing picture unit type (TRAIL), i.e., in the case of mixing TRAIL + CRA with an associated RADL leading picture.

[0094] In the previous aspect, the focus was mainly on the TRAIL NAL unit type. However, in many cases, there may also be an STSA NAL unit type, thus allowing progressive temporal sub-layer access. That is, even though the examples mentioned above mainly involve non-IRAP NAL units of the TRAIL NAL unit type, the present invention may also provide a solution for mixing non-IRAP NAL units of the STSA (progressive temporal sub-layer access) unit type with at least one of the RASL unit type, RADL unit type, and TRAIL unit type, namely: · STSA + RASL · STSA + RADL · STSA + TRAIL Thus, according to an embodiment, the decoder 100 may be configured to decode the bitstream 11, wherein the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a progressive temporal sub-layer access unit type (STSA), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a trailing picture unit type (TRAIL).

[0095] In a further embodiment, the decoder 100 may be configured to decode the bitstream 11, wherein the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 is a random access decodable leading picture unit type (RADL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is a progressive temporal sub-layer access unit type (STSA).

[0096] In another embodiment, the decoder 100 may be configured to decode the bitstream 11, wherein the non-IRAP NAL unit type of the NAL units 302 of the first sub-bitstream 11-1 is the Random Access Skipped Leading Picture Unit type (RASL), and the non-IRAP NAL unit type of the NAL units 304 of the second sub-bitstream 11-2 is the Stepwise Temporal Sub-layer Access Unit type (STSA).

[0097] Figure 6 An exemplary coding structure is shown, where, in a picture segment 132 (e.g., a tile), the picture 12a uses the NAL units 304 of the STSA NAL unit type to indicate stepwise temporal sub-layer access, while the same picture 12a in another picture segment 131 (e.g., a tile) uses the NAL units 302 of the RASL NAL unit type.

[0098] Since the STSA NAL unit type cannot be placed in the lowest temporal level 0, the STSA NAL units will never be mixed with the IRAP NAL units. However, they can be mixed with the RADL or RASL NAL unit types as the TRAIL NAL units mentioned above.

[0099] In one embodiment, a flag is included in the parameter set, e.g., in the PPS indicating the mixing of the STSA NAL units with the RADL or RASL NAL unit types.

[0100] Thus, in one embodiment of the present invention, in the bitstream 11, e.g., as a flag in the parameter set (PPS, SPS, VPS), there is an indication that the NAL unit types within the coded pictures can be mixed as follows: · STSA + RADL · STSA + RASL · STSA + TRAIL In another embodiment, it can be indicated whether such mixing is allowed, e.g., using a constraint flag at the sequence level or picture level.

[0101] Thus, according to an embodiment, the decoder 100 may be configured to derive an indication from the bitstream 11, wherein the indication explicitly signals to the decoder 100 a NAL unit 304 of a non-IRAP NAL unit type in the second sub-bitstream 11-2 (the NAL unit 304 belonging to a stepwise temporal sub-layer access unit type (STSA)) being mixed with a NAL unit 302 of a non-IRAP NAL unit type in the first sub-bitstream 11-1 (the NAL unit 302 being any one of a random access skipped leading picture unit type (RASL) or a random access decodable leading picture unit type (RADL)).

[0102] According to an embodiment, the indication includes a syntax element in a parameter set. According to a further embodiment, the syntax element is included in at least one of a picture parameter set or a sequence parameter set.

[0103] Additionally, an indication may be added to the bitstream 11 that effectively indicates that even if all NAL units 302, 304 of a picture 12a containing the STSA NAL unit 304 have another NAL unit type, e.g., RASL (see, e.g., NAL unit 302), they all satisfy the constraints associated with the STSA NAL unit type. This can be indicated by a flag in the SPS that indicates that the (spatial) subset of the bitstream 11-1 containing RASL pictures (see, e.g., NAL unit 302) satisfies the constraint that each RASL picture is also an STSA picture. Such signaling is even more important for use cases where the original bitstream containing RASL or RADL NAL units can indicate whether the RADL and RASL NAL units satisfy the properties of an STSA picture, such that if mixed with STSA, it can be easily derived whether the mixed encoded picture has STSA properties.

[0104] Thus, according to an embodiment, the decoder 100 may be configured to decode a constraint flag in a parameter set from the bitstream 11, the constraint flag indicating to the decoder 100 that any access unit 30 of the bitstream 11 (which includes a mixture of one or more NAL units 302 of a non-IRAP leading picture NAL unit type (e.g., RASL or RADL) and one or more NAL units 304 of a stepwise temporal sub-layer access unit type (STSA)) satisfies the constraints associated with the stepwise temporal sub-layer access unit type (STSA), i.e., each RASL picture 302 can be considered an STSA picture.

[0105] The constraint may indicate to the decoder 100 that if the next temporal sublayer higher than the step - temporal sublayer access (STSA) unit type precedes the step - temporal sublayer access unit type (STSA), any picture that is not allowed to follow in decoding order and is associated with the next higher temporal sublayer shall not reference pictures in the next higher temporal sublayer.

[0106] Alternatively, a bitstream 11 with an encoded picture 12a having a hybrid NAL unit type (e.g., STSA + RASL or RADL) may include a flag indicating that the picture 12a (for which the flag is set) may contain NAL units 304 of the STSA NAL unit type but (generally) does not comply with the constraints associated with the STSA NAL unit type. This will indicate that even though the encoded picture 12a may have a NAL unit type equal to STSA, the properties implied by STSA (i.e., the ability to decode additional temporal levels forward from this AU) do not apply. Thus, the parsing of STSA will be ignored, and step - temporal sublayer access will be inferred.

[0107] Thus, according to an embodiment, the decoder 100 may be configured to decode a flag from a parameter set that indicates to the decoder 100 that any access unit 30 (which includes a mixture of one or more NAL units 302 of a non - IRAP leading picture NAL unit type (RASL or RADL) and one or more NAL units 304 of a step - temporal sublayer access unit type (STSA)) does not comply with the constraints associated with the step - temporal sublayer access (STSA) unit type. In addition, the decoder 100 may be configured to refrain from parsing one or more NAL units 304 of the step - temporal sublayer access (STSA) unit type in response to the flag and implicitly infer step - temporal sublayer access decoding, i.e., the STSA constraints do not apply to the corresponding access unit.

[0108] This concept may also be applied independently of mixing NAL unit types as described above. Thus, according to an embodiment, a decoder 100 proposed for decoding video content 12 is configured to decode a bitstream 11 and decode a constraint flag from a parameter set of the bitstream 11, where the constraint flag indicates to the decoder 100 that within the bitstream 11, any access unit 30 that includes NAL units 302 of a non - IRAP leading picture NAL unit type (e.g., RASL or RADL) complies with the constraints associated with the step - temporal sublayer access (STSA) unit type.

[0109] 1.3. Mixing various RAP types Another important scenario that cannot be achieved by the prior art is to allow mixing of pictures with an open GOP (Group of Pictures) structure and pictures with a closed GOP structure. For example, it may be allowed to mix two different types of IRAP NAL unit types (e.g., IDR, CRA) (e.g., IDR_W_RADL and CRA NAL unit types), or it may be allowed to mix NAL units of the same NAL unit type (e.g., CRA and CRA): ·IDR_W_RADL + CRA ·CRA + CRA As mentioned above, this embodiment should be able to achieve mixing of IRAP NAL units of a NAL unit type associated with an open GOP structure and IRAP NAL units of a NAL unit type associated with a closed GOP structure. For example, a CRA with RADL may be associated with a closed GOP structure, while a CRA with RASL may be associated with an open GOP structure.

[0110] Thus, according to an embodiment, a decoder 100 for decoding video content 12 may be provided. The decoder 100 is configured to decode a bitstream 11 that includes a first sub-bitstream 11-1 related to a first spatial segment 131 of pictures 12a, 12b, 12c of the video content 12 and a second sub-bitstream 11-2 related to a second spatial segment 132 of pictures 12a, 12b, 12c of the video content 12. According to this example, the bitstream 11 includes at least one access unit 30, within which at least one first NAL unit 302 of the first sub-bitstream 11-1 belongs to an IRAP NAL unit type that is mixed with at least one second NAL unit 304 of the second sub-bitstream 11-2, and the at least one second NAL unit 304 belongs to another (same or different) IRAP NAL unit type.

[0111] There is at least one use case where this is very helpful. For example, one approach for 360° video transmission using tiled streaming is a case where the entire low-resolution video content is transmitted (regardless of whether parts of it are also transmitted as high-resolution tiles). In such a case, Figure 5 illustrates the change in viewing orientation and the corresponding change in high-resolution tile selection from one viewport to another.

[0112] Since the entire low-resolution video content is always available on the client side and, for this region, tile-wise stream switching is not required, the low-resolution content can be encoded with CRA and thus with a higher coding efficiency compared to a closed GOP structure using IDRs. Also, for this region, a random access point period longer than the random access point period necessary for stream switching will be sufficient, which in turn benefits the coding efficiency of the low-resolution video. On the other hand, as Figure 5 shown, high-resolution tiles may change frequently, etc. during a rapid progression from one viewport to another, and, thus, using CRA and associated RASL pictures as for low-resolution video will not allow for smooth switching of viewports. This is because the changed positions within the picture or RASL regions representing newly added tile streams cannot be decoded properly (due to missing references) and are thus discarded when a viewport change is encountered.

[0113] Figure 7 An example is shown where NAL unit 304 associated with a closed GOP structure and NAL unit 302 associated with an open GOP structure are mixed. In this particular but non-limiting example, NAL unit 304 of type CRA with a leading RADL 314 (closed GOP) is mixed with NAL unit 302 of type CRA 302 with a leading RASL 312 (open GOP). This non-limiting example shows the coding structure for: a high-resolution region in a second sub-bitstream 11-2 that uses a closed GOP structure with CRA combined with a leading picture of RADL type; and a low-resolution region in a first sub-bitstream 11-1 that uses an open GOP structure with CRA combined with a leading picture of RASL type.

[0114] According to an embodiment, decoder 100 is configured to process bitstream 11, where the IRAP NAL unit type of NAL unit 304 in the second sub-bitstream 11-2 belongs to the IRAP NAL unit type associated with a closed GOP structure, and the IRAP NAL unit type of NAL unit 302 in the first sub-bitstream 11-1 belongs to the IRAP NAL unit type associated with an open GOP structure.

[0115] In this particular but non-limiting embodiment as Figure 7 shown, the IRAP NAL unit type of NAL unit 304 in the second sub-bitstream 11-2 is a CRA unit type with a random access decodable leading picture unit type (CRA with RADL), and the IRAP NAL unit type of NAL unit 302 in the first sub-bitstream 11-1 is a CRA unit type with a random access skipped leading picture unit type (CRA with RASL).

[0116] It will be noted that Figure 7 the sequence of access units and the pictures contained therein are shown in their display order. However, the display order may be different from the coding order, which may also be referred to as the bitstream order. As can be seen, access unit 30 includes hybrid picture 12a, and hybrid picture 12a includes: a first VCL NAL unit 302, which represents a first spatial segment 131a of picture 12a in the first sub-bitstream 11-1; and a second VCL NAL unit 304, which represents a second spatial segment 132a of picture 12a in the second sub-bitstream 11-2. In this case, the first NAL unit and the second NAL unit are IRAP NAL units. In this particular but non-limiting example, both the first NAL unit 302 and the second NAL unit 304 belong to the IRAP NAL unit type CRA. Furthermore, Figure 7 the display order of consecutive pictures is shown.

[0117] Thus, in the display order, additional access unit 31 may precede access unit 30 discussed above. The preceding access unit (AU) 31 includes hybrid picture 12b, and hybrid picture 12b includes: a first VCL NAL unit 312, which represents a first spatial segment 131b of picture 12b in the first sub-bitstream 11-1; and a second VCL NAL unit 314, which represents a second spatial segment 132b of picture 12b in the second sub-bitstream 11-2. The first NAL unit 312 of this AU 31 belongs to the non-IRAP NAL unit type RASL, and the second NAL unit 314 of this AU 31 belongs to the non-IRAP NAL unit type RADL. Furthermore, Figure 7 the display order of consecutive pictures is shown.

[0118] Optionally, and further in display order, one or more additional access units (such as, exemplarily depicted access unit 32) may precede the access unit 31 discussed above. More generally, in display order, at least one additional access unit 32 may precede the access unit 31 including non-IRAP NAL units 312, 314 (e.g., RASL or RADL), and the at least one additional access unit 32 may also preferably include non-IRAP NAL units 322, 324 (e.g., RASL or RADL). To maintain the open GOP structure in the first sub-bitstream 11-1, the NAL unit 322 may precede (in display order) the RASL NAL unit 312 of the access unit 31, the NAL unit 322 is included in the preceding (in display order) access unit 32, and may also belong to the RASL type. Further, to maintain the closed GOP structure in the second sub-bitstream 11-2, the NAL unit 324 may precede (in display order) the RADL NAL unit 314 of the access unit 31, the NAL unit 324 is included in the preceding (in display order) access unit 32, and may also belong to the RADL type.

[0119] The coding order (which is also referred to as the bitstream order) may deviate Figure 7 from the display order depicted in. For example, the RADL pictures 132b, 132c and the RASL pictures 131b, 131c are coded dependently, which means that they retrieve differential coding information from their respective following (in display order) CRA pictures 131a, 132a. For example, even though the access unit 30 follows (in display order) the access unit 31, the RADL picture 132b and the RASL picture 131b included in the access unit 31 also refer to the following (in display order) CRA pictures 131a, 132a included in the access unit 30.

[0120] The RASL picture 131b may optionally further refer to preceding (in display order and coding order) pictures, such as, in Figure 7The preceding RASL picture 131c depicted exemplarily in. The additional RASL picture (correspondingly, the additional and preceding RASL picture 131c of the preceding access unit 32) must be encoded before the RASL picture 131b of the access unit 31. However, the additional and preceding RASL picture 131c of the preceding access unit 32 itself may refer to a picture that is subsequent in display order but preceding in encoding order, such as the IRAP (CRA) picture 131a of the access unit 30. This means that even though the IRAP (CRA) picture 131a of the access unit 30 is the last picture (in display order) of the sequence discussed above (i.e., the IRAP (CRA) picture 131a of the access unit 30 follows both the RASL pictures 131b and 131c in display order), the IRAP (CRA) picture 131a of the access unit 30 is also the first picture in encoding order, i.e., it must be encoded first because, during encoding, both the RASL pictures 131b and 131c refer to the IRAP (CRA) picture 131a.

[0121] This also applies to RADL pictures. The RADL picture 132b may optionally further refer to a preceding (in both display order and encoding order) picture, such as the preceding RADL picture 132c depicted exemplarily in Figure 7 . Thus, the additional and preceding RADL picture 132c of the preceding access unit 32 must be encoded before the RADL picture 132b of the access unit 31. However, the additional and preceding RADL picture 132c of the preceding access unit 32 itself may refer to a picture that is subsequent in display order but preceding in encoding order, such as the IRAP (CRA) picture 132a of the access unit 30. This means that even though the IRAP (CRA) picture 132a of the access unit 30 is the last picture (in display order) of the sequence discussed above (i.e., the IRAP (CRA) picture 132a of the access unit 30 follows both the RADL pictures 132b and 132c in display order), the IRAP (CRA) picture 132a of the access unit 30 is also the first picture in encoding order, i.e., it must be encoded first because, during encoding, both the RADL pictures 132b and 132c refer to the IRAP (CRA) picture 132a.

[0122] More generally, non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 131b, 131c and RADL pictures 132b, 132c) may refer to following (in display order) IRAP pictures (e.g., CRA pictures 131a, 132a), where the IRAP pictures (e.g., CRA pictures 131a, 132a) must be encoded first (i.e., before their leading (in display order) non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 131b, 131c and RADL pictures 132b, 132c)). This means that even though the IRAP pictures (e.g., CRA pictures 131a, 132a) follow (in display order) non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 131b, 131c and RADL pictures 132b, 132c), the IRAP pictures (e.g., CRA pictures 131a, 132a) are encoded first. In other words, non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 131b, 131c and RADL pictures 132b, 132c) are immediately in front of the IRAP pictures (e.g., CRA pictures 131a, 132a) in display order, while the IRAP pictures (e.g., CRA pictures 131a, 132a) are immediately in front of non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 131b, 131c and RADL pictures 132b, 132c) in bitstream order (encoding order). In yet other words, non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 131b, 131c and RADL pictures 132b, 132c) are in front of the IRAP pictures (e.g., CRA pictures 131a, 132a) in display order, but non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 131b, 131c and RADL pictures 132b, 132c) follow the IRAP pictures (e.g., CRA pictures 131a, 132a) in encoding / bitstream order.

[0123] Taking into account the above-described situation and encoding structure, one advantage of the present invention is that it allows mixing of CRAs with leading RASL (open GOP) and CRAs with leading RADL (closed GOP) within a picture of access unit 30. When such a mixed NALU access unit 30 is encountered within an encoded video sequence (CVS) during normal playback (decoding the entire bitstream 11 from the beginning), all required reference pictures are available. Thus, pictures following the mixed picture with RASL NAL units will be decoded and output normally.

[0124] However, when such a mixed NALU is encountered during a lookup operation or when starting the decoding process from such an AU during random access, a process for generating unavailable reference pictures needs to be called, and: · What needs to be notified to the subsequent processing chain, for example, through SEI indication, is that the RASL region has not been correctly decoded, and the corresponding low-resolution region will be used.

[0125] · The affected pictures can be completely deleted from the output.

[0126] This means that when starting to decode a mixed picture, one option is to treat it as a GDR picture, where some parts are decodable and can be shown, while other parts cannot be shown and will be refreshed over time (until all RASL NAL units are finished). For example, in a 360 scenario, when the RAP type is selected contrary to the way shown in Figure 7 (i.e., the low resolution uses a closed GOP structure (e.g., CRA with RADL), and the high-resolution content uses an open GOP structure (e.g., CRA with RASL)), such an operation is relevant. Then, the low resolution can be shown because the low resolution uses a closed GOP configuration, and the player will wait until the high resolution using the open GOP structure is cleanly decoded to show it.

[0127] Another option for handling such a situation would be not to show any of the pictures in the picture affected by the missing reference (even not partially) and delete them, that is, delete any mixed picture with a NAL unit type equal to RASL.

[0128] In other words, any mixed pictures 12b, 12c with NAL unit types 312, 322 equal to RASL can be marked as deleted / discarded to ensure continuous decoding behavior.

[0129] Thus, according to an embodiment, the decoder 100 may be configured to decode and present co-located spatial segments 132b, 132c, ... of one or more pictures 12b, 12c, ... accessed by access units 31, 32, ..., where the access units 31, 32, ... follow, in bitstream order, at least one access unit 30 (e.g., a CRA with RADL) involved in the NAL unit 304 of the second sub-bitstream 11-2, and the co-located spatial segments 132b, 132c, ... are co-located with a second spatial segment 132a of a picture 12a of at least one access unit 30 involved in the NAL unit 304 of the second sub-bitstream 11-2. Further, the decoder 100 may be configured to mark as discarded co-located spatial segments 131b, 131c, ... of one or more pictures 12b, 12c, ... accessed by access units 31, 32, ..., where the access units 31, 32, ... follow, in bitstream order, at least one access unit 30 (e.g., a CRA with RASL) involved in the NAL unit 302 of the first sub-bitstream 11-1, and the co-located segments 131b, 131c, ... are co-located with a first spatial segment 131a of a picture 12a of at least one access unit 30 involved in the NAL unit 302 of the first sub-bitstream 11-1.

[0130] According to a further embodiment, the decoder 100 may be configured to operate depending on whether the bitstream 11 is to be decoded during normal playback (decoding the entire bitstream 11 from the start) or during start-up decoding (e.g., during a seek operation or during a decoding process starting from such an AU during random access).

[0131] In the case of normal playback (decoding the entire bitstream 11 from the beginning), the decoder 100 may be configured to decode and present the co-located spatial segments 132b, 132c,... of one or more pictures 12b, 12c,... of access units 31, 32,... The access units 31, 32,... follow, in bitstream order, at least one access unit 30 involved in the NAL unit 304 of the second sub-bitstream 11-2 (e.g., a CRA with RADL). The co-located spatial segments 132b, 132c,... are co-located with the second spatial segment 132a of the picture 12a of at least one access unit 30 involved in the NAL unit 304 of the second sub-bitstream 11-2. Additionally, in the case of starting decoding (e.g., during a seek operation or during the decoding process starting from such an AU during random access), the decoder 100 may be configured to discard one or more pictures 12b, 12c,... of access units 31, 32,... that follow, in bitstream order, at least one access unit 30, and resume picture output after one or more discarded pictures 12b, 12c,.... In other words, any hybrid pictures 12b, 12c having at least one NAL unit 312, 322 equal to RASL are deleted / discarded.

[0132] According to a further embodiment, the decoder 100 may be configured to mark the discontinuous spatial segments 131b, 131c,... for a subsequent processing chain by means of SEI (Supplemental Enhancement Information) messages.

[0133] In a further embodiment, there is an indication in the bitstream that, regardless of whether there is an IDR_W_RADL NAL unit type, the AU is considered a CRA, i.e., an IRAP with a NoIncorrectPicOutputFlag equal to 1. There is an indication that the AU is a hybrid NALU but has the mentioned characteristics and IRAP for an open GOP structure. In other words, the decoder 100 may be configured to derive from the bitstream 11 an indication that instructs the decoder 100 to decode at least one access unit 30 related to the first sub-bitstream 11-1 and the second sub-bitstream 11-2 in a manner corresponding to a clean random access NAL unit type.

[0134] In the foregoing description, embodiments have been discussed in which the decoder 100 is configured to process the bitstream 11, where A) NAL units of non-IRAP NAL unit types (e.g., RASL, RADL, TRAIL, STSA) are mixed with at least one different second NAL unit of a different non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA), or, B) NAL units of the IRAP NAL unit type (e.g., IDR_W_RADL, CRA) are mixed with at least one NAL unit of another IRAP NAL unit type (e.g., IDR_W_RADL, CRA).

[0135] However, there may be additional embodiments according to which the decoder 100 may be configured to process the bitstream 11, wherein: C) NAL units of the IRAP NAL unit type (IDR_W_RADL, CRA) are mixed with at least one NAL unit of either the non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or the STSA NAL unit type.

[0136] Additional embodiments also propose to provide corresponding devices for processing the video bitstream 11 during the encoding of the bitstream 11 and / or the merging of the first sub-bitstream 11-1 and the second sub-bitstream 11-2. The device 10 may be at least one of an encoder, a merger, or a network node for processing the bitstream 11 according to the innovative principles described herein.

[0137] Thus, according to an embodiment, a device 10 for processing video content 12 is proposed, the device 10 being configured to provide a first sub-bitstream 11-1 related to a first spatial segment 131 of pictures 12a, 12b, 12c of the video content 12, and to provide a second sub-bitstream 11-2 related to a second spatial segment 132 of pictures 12a, 12b, 12c of the video content 12, wherein the first sub-bitstream 11-1 and the second sub-bitstream 11-2 are parts of the bitstream 11, wherein: A) The bitstream (11) includes at least one access unit (30), within which at least one first NAL unit (302) of the first sub-bitstream (11-1) belongs to a non-IRAP NAL unit type and is mixed with at least one different second NAL unit (304) of the second sub-bitstream (11-2) belonging to a different non-IRAP NAL unit type, or, B) The bitstream (11) includes at least one access unit (30), within which at least one first NAL unit (302) of the first sub-bitstream (11-1) belongs to an IRAP NAL unit type and is mixed with at least one second NAL unit (304) of the second sub-bitstream (11-2) belonging to another IRAP NAL unit type, or, C) The bitstream 11 includes at least one access unit 30. Within the at least one access unit 30, at least one first NAL unit 302 of the first sub-bitstream 11-1 belongs to the IRAP NAL unit type, and is mixed with at least one second NAL unit 304 of the second sub-bitstream 11-2 that belongs to either the non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or the STSA NAL unit type.

[0138] 1.4. Summary and Exemplary Syntax In the following, a summary of the allowed hybrid types within the present invention is shown: NAL unit type Will be mixed with the following CRA_NUT IDR_W_RADL TRAIL RADL or RASL or STSA STSA RADL or RASL RASL TRAIL or RADL RADL TRAIL or RASL In the new hybrid types described in the present invention, there are two types of hybrids, one type that is an essential IRAP type and another type that does not have the IRAP type.

[0139] Option 1: As discussed in the previous section in the text above, one option is to signal using the flag in the only PPS mix of the essential IRAP type. For example, using again the existing option of mixing IDR or CRA with TRAIL until VCL_RSV_6 (mixed_nalu_types_in_pic_flag). The non-IRAP type can be signaled using the constraint flag at SPS / VPS / DPS.

[0140] The mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture referring to the PPS has more than one VCL NAL unit, and the VAL NAL units do not have the same nal_unit_type value, there is at least one VCL NAL unit having a nal_unit_type value within the range of IDR_W_RADL to CRA_NUT, and the picture is not an IRAP IDR picture. The mixed_nalu_types_in_pic_flag equal to 0 specifies that when the VCL NAL unit has a nal_unit_type value within the range of IDR_W_RADL to CRA_NUT, each picture referring to the PPS has one or more VCL NAL units, and the VCL NAL units of each picture referring to the PPS have the same nal_unit_type value.

[0141] Thus, if the AU is the first AU of the bitstream or the first AU at the end of a sequence (EOS) NAL unit of a PPS following a reference with mixed_nalu_types_in_pic_flag equal to 1, the NAL unit type in the coded picture must be CRA_NUT and IDR_W_RADL. Mixed pictures with IDR / CRA of non-IRAP type are only allowed for AUs that are not the first AU in the bitstream or sequence.

[0142] The no_non_irap_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that the requirement for bitstream conformance is that the VCL NAL units of pictures with TRAIL, STSA, RASL, and RADL in the sequence have the same nal_unit_type value. A value of 0 does not impose such a constraint, i.e., a picture can have two different nal_unit_type values among TRAIL, STSA, RASL, and RADL.

[0143] Option 2: Another option would be to have an indication (idc) indicating different combinations instead of the flag (mixed_nalu_types_in_pic_flag) in the PPS.

[0144] The no_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that the requirement for bitstream conformance is that mixed_nalu_types_in_pic_idc should be equal to 0. A no_mixed_nalu_types_in_pic_constraint_flag equal to 0 does not impose such a constraint.

[0145] The mixed_nalu_types_in_pic_idc equal to 0 specifies that each picture referring to the PPS has one or more VCL NAL units, and the VCL NAL units of each picture referring to the PPS have the same nal_unit_type value.

[0146] The mixed_nalu_types_in_pic_idc equal to 1 specifies that each picture referring to the PPS has more than one VCL NAL unit, and the VCL NAL units should all inclusively have two specific nal_unit_type values in the range from TRAIL_NUT to RSV_VCL_6. NAL unit type Will be mixed with the following TRAIL RADL or RASL or STSA STSA RADL or RASL RASL TRAIL or RADL RADL TRAIL or RASL

[0147] A `mixed_nalu_types_in_pic_idc` equal to 2 specifies that each picture referred to by the reference PPS has more than one VCL NAL unit, and one or more of the VCL NAL units shall all inclusively have a specific `nal_unit_type` value within the range from IDR_W_RADL to CRA_NUT, and the other VCL NAL units shall all inclusively have a specific `nal_unit_type` value within the range from TRAIL_NUT to RSV_VCL_6 or equal to GRA_NUT.

[0148] This value corresponds to the allowed mixing of NAL units with existing flags.

[0149] A `mixed_nalu_types_in_pic_idc` equal to 3 specifies that each picture referred to by the reference PPS has more than one VCL NAL unit with a `nal_unit_type` value of IDR_W_RADL and one or more VCL NAL units with a `nal_unit_type` value of CRA_NUT. NAL unit type Will be mixed with the following CRA_NUT IDR_W_RADL

[0150] For the VCL NAL units of any specific picture, the following applies: - If `mixed_nalu_types_in_pic_idc` is equal to 0, then for all coded slice NAL units of the picture, the `nal_unit_type` values shall all be the same. The picture or PU is said to have the same NAL unit type as the coded slice NAL units of the picture or PU.

[0151] - If `mixed_nalu_types_in_pic_idc` is equal to 1 or 2, then the following applies.

[0152] - If one of the VCL NAL units in the picture has a `nal_unit_type` value equal to TRAIL_NUT, then the picture or PU is said to be a trailing picture or trailing PU.

[0153] - In other cases (where there is no TRAIL_NUT VCL NAL unit in the picture), the picture or PU is said to be a leading picture or leading PU.

[0154] - If one of the VCL NAL units in the picture has a `nal_unit_type` value equal to RADL_NUT, then the picture or PU is said to be a RADL picture or RADL PU.

[0155] - In other cases (where none of the VCL NAL units in the picture have a nal_unit_type value equal to RADL_NUT), the picture or PU is called a RASL picture or RASL PU.

[0156] - In other cases (where mixed_nalu_types_in_pic_idc is equal to 3), the picture is called a CRA picture or CRA PU.

[0157] Aspects related to treating the leading NAL unit type as an STSA picture can be implemented using the constraint flags as follows.

[0158] The leading_stsa_pictures_flag equal to 1 specifies that the inter-prediction reference for RADL pictures and RASL pictures is the constraint specified below. The no_mixed_nalu_types_in_pic_constraint_flag equal to 0 does not impose such a constraint.

[0159] When the leading_stsa_pictures_flag is equal to 1, the following applies: - When the current picture is a RASL or RADL picture, there shall not be an active entry in RefPicList[0] or RefPicList[1] with a TemporalId equal to the TemporalId of the current picture.

[0160] - When the current picture is a picture that follows a RASL or RADL in decoding order and precedes an associated IRAP picture with a TemporalId equal to the TemporalId of the current picture, there shall not be a picture with a TemporalId equal to the TemporalId of the current picture, which is included as an active entry in RefPicList[0] or RefPicList[1], that precedes the RASL or RADL picture in decoding order.

[0161] In other words, the constraints associated with an STSA picture prohibit a picture that follows in decoding order and is associated with the next higher temporal sublayer than STSA from using any reference in that next higher temporal sublayer before STSA, i.e., the decoder can start decoding the next temporal sublayer from STSA forward. Of course, such constraints can also be enforced for RASL / RADL pictures as described above, and thus, the signaling above can be used as an indication of these properties.

[0162] 2. Mixing Picture Types within a Multi - layer Access Unit By way of non - limiting example, this subsection may relate to multi - layer access units within a multi - layer bitstream. However, the features described herein may already be applied to a single layer. Additionally, by way of non - limiting example, this subsection describes inventive principles with reference to long - term reference pictures. However, the inventive concept also applies to short - term reference pictures, i.e., the features described herein generally may be applied to picture references. Additionally, by way of non - limiting example, this subsection may describe resetting the POC MSB by setting the POC value to zero, i.e., MSB = 0. However, the zero value only serves as a placeholder to denote a decreased MSB value, e.g., it may also be possible that any predetermined value not equal to 0 but less than the current MSB at the corresponding bitstream position may occur for MSB resetting.

[0163] The prior art may provide support for mixing pictures of different RAP types within a hierarchical access unit (i.e., for time instances where the access unit simultaneously contains RAP pictures and non - RAP pictures in its layer) by signaling the most significant POC (Picture Order Count) bit (poc_msb_val). However, there may still be problems regarding reference pictures belonging to the leading picture NAL unit type, which have a reference to pictures of associated IRAP pictures that precede the reference picture in coding order as shown by way of example of long - term reference pictures below, but also apply to short - term reference pictures, and the present invention provides a solution for such reference pictures.

[0164] The current POC signaling - related syntax is as follows.

[0165] SPS Syntax Slice Header Syntax An exemplary use of the POC MSB signaling above is to set the POC MSB to 0 when all layers have the same IRAP NAL unit type (e.g., IDR_W_RADL or CRA in all layers as illustrated in Figure 8 .

[0166] Figure 8 FIG. 11 shows a multi - layer bitstream 11, which includes a first sub - bitstream 11 - 1 in a first layer L0 and a second sub - bitstream 11 - 2 in a second layer L1. Each of the layers L1, L0 may include one or more temporal sub - layers T0, T1, T2. Thus, the spatial segments of pictures may be distributed over the different layers L0, L1 and temporal sub - layers T0, T1, T2.

[0167] Figure 8 Some exemplary multi-layer access units 30, 31, 32 are further shown. In this example, the multi-layer access units 30, 31, 32 may include a first NAL unit 302 of a first sub-bitstream 11-1 in a first layer L0, which is mixed with a second NAL unit 304 of a second sub-bitstream 11-2 in a second layer L1. As described above, the NAL unit may include different NAL unit types, in particular, the IRAP NAL unit type and the non-IRAP NAL unit type. Additionally, as mentioned above, the NAL unit may include a slice header, wherein the picture order count (POC) is signaled. The concepts described herein relate to the most significant POC part, for example, to the most significant POC bit or POC MSB. Thus, in the figure, the POC MSB is depicted at each IRAP NAL unit.

[0168] For example, the multi-layer access units 30, 32 represent aligned access units, wherein the POC MSB of the first NAL unit 302 in the first sub-bitstream 11-1 is aligned with the POC MSB of the second NAL unit 304 in the second sub-bitstream 11-2, where the two NAL units belong to the IRAP NAL unit type.

[0169] The multi-layer access unit 31 represents an unaligned access unit, that is, wherein the POC MSB of the first NAL unit 302 in the first sub-bitstream 11-1 is not aligned with the POC MSB (e.g., MSB = 0) of the second NAL unit 304 in the second sub-bitstream 11-2 that belongs to the non-IRAP NAL unit type. Thus, the POC MSB of the first NAL unit 302 of the first sub-bitstream 11-1 can be increased by a predetermined value, for example, by one.

[0170] Figure 8 It is shown how the MSB signaling in the previously described slice header can be used in such a multi-layer case. The purpose of this signaling is to keep the POC value correct at an unaligned RAP (e.g., the unaligned access unit 31 with CRA w MSB = 1), that is, to avoid resetting the POC (LSB and MSB) to zero when not all layers have an IRAP at the same access unit.

[0171] However, Figure 8 It is also shown that there are problems with respect to (e.g., long-term and / or short-term) reference pictures 12r when the POC is aligned. For ease of understanding, the following figures may only relate to long-term reference pictures. However, the present concepts also apply to short-term reference pictures.

[0172] Note that pictures 12a, 12b, 12c, … prior to the last CRA access unit 32 (i.e., the second aligned RAP) have POC MSB > 0 and POC LSB > 0. Thus, when the MSB is reset to 0 at the last CRA access unit 32 (as exemplarily shown in Figure 8 ), then it will not be possible to use the POC MSB or its increment (such as for long-term reference pictures) to reference any of the previous pictures 12a, 12b, 12c, …. In Figure 8 , an example is indicated by the green arrow marked with ‘LT ref’. Thus, Figure 8 the example shown should demonstrate that picture reference may not be possible in this case.

[0173] Therefore, the present invention provides a solution to this problem such that picture reference will become possible.

[0174] In one embodiment, the following constraint is expressed in the specification or indicated by bitstream flags (SPS, VPS, DPS): The MSB can only be set to 0 if all pictures within an access unit belong to type IDR (i.e., IDR_W_RADL or IDR_N_LP). Thus, when using CRA, no POC reset occurs and LT reference is allowed (see for example Figure 9 and Figure 10 ).

[0175] As can be seen in Figure 9 , the access unit 32 can be an aligned access unit 32 where two NAL units 302, 304 belong to the IRAP NAL unit type. However, since the first NAL unit 302 of the first sub-bitstream 11-1 belongs to the CRA type, no POC reset should occur. This means that Figure 9 shows a scenario where POC reset is prohibited.

[0176] Conversely, Figure 10 shows a scenario where POC reset is allowed because both the first NAL unit 302 of the first sub-bitstream 11-1 and the second NAL unit 304 of the second sub-bitstream 11-2 belong to the IDR unit type.

[0177] Accordingly, according to an embodiment, a decoder 100 for decoding video content 12 is provided. The decoder 100 is configured to decode a bitstream 11 that includes a first sub-bitstream 11-1 and a second sub-bitstream 11-2. Among them, the decoder 100 is configured to derive information for setting the most significant POC part (POC MSB) of the picture 12r referred to by the NAL units 302, 304 from the slice headers of the NAL units 302, 304 of the first sub-bitstream 11-1 and the second sub-bitstream 11-2, where the NAL units 302, 304 belong to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP). According to this example, if at least one of the NAL units 302, 304 within the access unit 32 belongs to the non-Instantaneous Decoding Refresh unit type, e.g., belongs to the CRA unit type, it is required to set the most significant POC part (POC MSB) to a predetermined value for all NAL units 302, 304 of the IRAP NAL unit type within the access unit 32 of the bitstream 11.

[0178] As mentioned above, as a non-limiting example, this subsection may describe the POC MSB reset by setting the POC value to zero, i.e., MSB = 0. However, the zero value only serves as a placeholder for representing a reduced MSB value. For example, it is also possible that any predetermined value not equal to 0 but less than the current MSB at the corresponding bitstream position may cause the MSB reset. Therefore, the predetermined value mentioned above may be equal to or greater than the most significant POC part of the POC associated with the pictures 12a, 12b,... preceding the picture 12r referred to by the NAL unit 302. In other words, if not all of the NAL units 302, 304 within the access unit 32 belong to the IDR unit type, POC reset is not allowed.

[0179] In some examples, the predetermined value may be equal to zero, i.e., MSB = 0. In some other examples, the predetermined value may be equal to the implicitly derived most significant POC part, e.g., equal to the conventional SOTA POC MSB implicit derivation.

[0180] In another embodiment, there may be the following constraints expressed in the specification or indicated by bitstream flags (SPS, VPS, DPS): Whenever an aligned RAP access unit 32 appears and the POC MSB of the access unit 32 is set to 0, it is not allowed for the picture 12s following the aligned RAP access unit 32 in bitstream order to reference the (long-term or short-term) reference pictures 12a, 12b, 12c,... preceding the aligned RAP access unit 32.

[0181] For example, Figure 11 shows that even if picture 12s can precede picture 12r in the presentation order, picture 12s can follow picture 12r in the decoding order.

[0182] Figure 11 and Figure 12 shows that any picture 12r having an MSB equal to 0 does not have leading pictures 12a, 12b, 12c,... with LT reference pictures (although Figure 11 represents a reset from MSB = 1 to MSB = 0, but Figure 12 represents that the MSB is equal to 0, for example, because the POC_LSB is long enough so that no increment occurs). Therefore, Figure 11 shows a scenario where POC MSB reset and picture reference are prohibited, and, Figure 12 shows a scenario where all POC MSBs can be equal to zero, but picture reference is not allowed. Figure 13 shows that if the MSB is not equal to 0, LT reference pictures are allowed for leading pictures.

[0183] Thus, according to an embodiment, a decoder 100 for decoding video content 12 is provided. The decoder 100 is configured to decode a bitstream 11 including a first sub-bitstream 11-1 and a second sub-bitstream 11-2. The decoder 100 is configured to derive information for setting the most significant POC part (POC MSB) of the POC of a picture 12r referenced by NAL units 302, 304 (where the NAL units 302, 304 belong to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP)) of the first sub-bitstream 11-1 and the second sub-bitstream 11-2 from slice headers. According to this example, any picture reference is prohibited from crossing any access unit 32 of the bitstream 11 in the decoding order, within which all NAL units 302, 304 belong to the IRAP NAL unit type and have slice headers in which the most significant POC part (POC MSB) is set to a predetermined value.

[0184] Furthermore, as mentioned above, as a non-limiting example, this subsection may describe the POC MSB reset by setting the POC value to zero, i.e., MSB = 0. However, the zero value only serves as a placeholder for symbolically representing a reduced MSB value. For example, it is also possible that any predetermined value not equal to 0 but less than the current MSB at the corresponding bitstream position may occur for MSB reset. Therefore, referring to Figures 11 to 13The predetermined value of the described example may be less than the most significant POC portion (POC MSB) of the POC associated with pictures (12a, 12b, 12c, ……) that precede the picture (12r) referenced by the NAL units (302, 304), i.e., a POC reset occurs.

[0185] In some examples, the predetermined value may be equal to zero, i.e., MSB = 0. In some other examples, the predetermined value may be equal to the implicitly derived most significant POC portion, for example, equal to the implicit derivation of the conventional SOTA POC MSB.

[0186] In another embodiment, there may be the following constraints expressed in the specification or indicated by bitstream flags (SPS, VPS, DPS): Whenever an aligned RAP access unit 32 appears and the POC MSB of the access unit 32 is set to 0 (POC reset), it is not allowed for picture 12s that follows the aligned RAP access unit 32 in bitstream order to reference (short-term or long-term) reference pictures 12a, 12b, 12c, …… that precede the aligned RAP access unit 32 and have an MSB different from 0.

[0187] Note that, if in the above example shown in Figure 8 there is no increase in the MSB yet, using the LT reference picture will not be a problem because the reference to the LT reference picture is only made by the POC_LSB value.

[0188] Figure 14 It is shown that: in the example, no POC MSB reset occurs. By increasing the POC MSB value of the access unit 32 (in the example shown in Figure 14 the POC MSB value increases, i.e., MSB = 3) or making the POC MSB value of the access unit 32 the same as that in the previous access unit 31, the POC MSB reset may not be indicated.

[0189] Accordingly, according to an embodiment, there is provided a decoder 100 for decoding video content 12. The decoder 100 is configured to decode a bitstream 11, which includes a first sub-bitstream 11-1 and a second sub-bitstream 11-2. Among them, the decoder 100 is configured to derive information for setting the most significant POC part (POC MSB) of the POC of the picture 12r referred to by the NAL units 302, 304 from the slice headers of the NAL units 302, 304 of the first sub-bitstream 11-1 and the second sub-bitstream 11-2 (wherein the NAL units 302, 304 belong to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP)). According to this example, any picture reference to the reference pictures 12a, 12b, 12c,... of the first access unit 31 by the picture 12s of the second access unit 32 following the bitstream 11 in bitstream order is prohibited. The first access unit 31 consists of NAL units 301, 303, and the NAL units 301, 303 have slice headers in which the most significant POC part is set to a first predetermined value, and the first access unit 31 precedes the second access unit 32 of the bitstream 11 in bitstream order. In the second access unit 32, all NAL units 302, 304 belong to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP), and have slice headers in which the most significant POC part (POC MSB) is set to a second predetermined value.

[0190] Furthermore, as mentioned above, as a non-limiting example, this subsection may describe the POC MSB reset by setting the POC value to zero, i.e., MSB = 0. However, the zero value only serves as a placeholder for representing a reduced MSB value. For example, it is also possible that any predetermined value not equal to 0 but less than the current MSB at the corresponding bitstream position may cause an MSB reset. Therefore, regarding the example described with reference to Figure 14 the first predetermined POC MSB value of the preceding first access unit 31 may be greater than the second predetermined POC MSB value of the subsequent second access unit 32. In this case, picture reference will be prohibited. On the contrary, it can be stated that if the second predetermined MSB value of the subsequent second access unit 32 can be greater than the first predetermined POC MSB value of the preceding first access unit 31, picture reference will be allowed. In other words, if no POC reset occurs, picture reference will be allowed.

[0191] For example, the second predetermined value may be equal to zero to indicate a POC reset. Thus, if a POC reset occurs, picture reference will be prohibited. In some other examples, the second predetermined value may be equal to the most significant POC part implicitly derived, such as being equal to the conventional SOTAPOC MSB implicitly derived.

[0192] Although some aspects have been described in the context of a device, it is clear that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of method steps also represent a description of corresponding blocks or items or features of a corresponding device.

[0193] Some or all of the method steps can be performed by (or with) a hardware device such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps can be performed by such a device.

[0194] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or in software or at least partly in hardware or at least partly in software. This implementation can be carried out using a digital storage medium (e.g., a floppy disk, a DVD, a Blu-ray disc, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a flash memory) having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the corresponding method is carried out. Thus, the digital storage medium can be computer-readable.

[0195] Some embodiments according to the present invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system such that one of the methods described herein is carried out.

[0196] Generally speaking, embodiments of the present invention can be realized as a computer program product having program code that is operative to carry out one of the methods described herein when the computer program product runs on a computer. The program code can be stored, for example, on a machine-readable carrier.

[0197] Other embodiments include a computer program stored on a machine-readable carrier for carrying out one of the methods described herein.

[0198] In other words, thus, embodiments of the inventive method are computer programs having program code for carrying out one of the methods described herein when the computer program runs on a computer.

[0199] Accordingly, a further embodiment of the inventive method is a data carrier (or digital storage medium or computer-readable medium), which comprises a computer program recorded thereon for performing one of the methods described herein. The data carrier, digital storage medium or recording medium is typically tangible and / or non-transitory.

[0200] Accordingly, a further embodiment of the inventive method is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence can be configured, for example, to be transferred via a data communication connection (e.g., via the Internet).

[0201] A further embodiment includes a processor device, such as a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0202] A further embodiment includes a computer on which a computer program for performing one of the methods described herein has been installed.

[0203] A further embodiment according to the invention includes a device or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver can be, for example, a computer, a mobile device, a memory device, etc. The device or system can include, for example, a file server for transferring the computer program to the receiver.

[0204] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array can cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0205] The devices described herein can be implemented using a hardware device or using a computer or using a combination of a hardware device and a computer.

[0206] The methods described herein can be performed using a hardware device or using a computer or using a combination of a hardware device and a computer.

[0207] Although this disclosure has been described with reference to illustrative embodiments, this description is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of this disclosure, will be apparent to persons skilled in the art upon reference to this description. Accordingly, it is intended that the appended claims include any such modifications or embodiments.

Claims

1. A decoder (100) for decoding video content (12), the decoder (100) being configured to: decode a bitstream (11), the bitstream (11) including a first sub-bitstream (11-1) associated with a first spatial segment (131) of pictures (12a, 12b, 12c, ……) of the video content (12) and a second sub-bitstream (11-2) associated with a second spatial segment (132) of the pictures (12a, 12b, 12c, ……) of the video content (12), wherein, A) the bitstream (11) includes at least one access unit (30), within the at least one access unit (30), at least one first NAL unit (302) of the first sub-bitstream (11-1) belongs to a non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), at least one first NAL unit (302) is mixed with at least one different second NAL unit (304) of the second sub-bitstream (11-2) that belongs to a different non-IRAP NAL unit type (e.g., RADL, RASL, TRAIL, STSA), or, B) the bitstream (11) includes at least one access unit (30), within the at least one access unit (30), at least one first NAL unit (302) of the first sub-bitstream (11-1) belongs to an IRAP NAL unit type (e.g., IDR, CRA), at least one first NAL unit (302) is mixed with at least one second NAL unit (304) of the second sub-bitstream (11-2) that belongs to another IRAP NAL unit type (e.g., IDR, CRA), or, C) the bitstream (11) includes at least one access unit (30), within the at least one access unit (30), at least one first NAL unit (302) of the first sub-bitstream (11-1) belongs to an IRAP NAL unit type (e.g., IDR, CRA), at least one first NAL unit (302) is mixed with at least one second NAL unit (304) of the second sub-bitstream (11-2) that belongs to either a non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or an STSA NAL unit type.

2. The decoder (100) according to claim 1, Among them, the spatial segments (131, 132) of the pictures (12a, 12b, 12c, ……) relate to independently encoded sub-pictures (20c) of the pictures (12a, 12b, 12c, ……) of the video content (12), and the decoder (100) is configured to use a boundary extension for motion-compensated prediction across sub-picture edges.

3. The decoder (100) according to claim 1 or 2, Among them, in case A), The non-IRAP NAL unit type of the NAL unit (302) of the first sub-bitstream (11-1) is a random access decodable leading picture unit type (RADL), and, The non-IRAP NAL unit type of the NAL unit (304) of the second sub-bitstream (11-2) is a trailing picture unit type (TRAIL).

4. The decoder (100) according to claim 1 or 2, Among them, In case A), The non-IRAP NAL unit type of the NAL unit (302) of the first sub-bitstream (11-1) is a random access skipped leading picture unit type (RASL), and, The non-IRAP NAL unit type of the NAL unit (304) of the second sub-bitstream (11-2) is a trailing picture unit type (TRAIL).

5. The decoder (100) according to claim 1 or 2, Among them, In case A), The non-IRAP NAL unit type of the NAL unit (302) of the first sub-bitstream (11-1) is a random access skipped leading picture unit type (RASL), and, The non-IRAP NAL unit type of the NAL unit (304) of the second sub-bitstream (11-2) is a random access decodable leading picture unit type (RADL).

6. The decoder (100) according to claim 1 or 2, Among them, In case A), The non-IRAP NAL unit type of the NAL unit (302) of the first sub-bitstream (11-1) is a progressive temporal sub-layer access unit type (STSA), and, The non-IRAP NAL unit type of the NAL unit (304) of the second sub-bitstream (11-2) is a trailing picture unit type (TRAIL).

7. The decoder (100) according to claim 3 or 4, Among them, The decoder (100) is configured to derive an indication from the bitstream (11), wherein the indication explicitly signals to the decoder (100) that the NAL unit (304) of the non-IRAP NAL unit type of the second sub-bitstream (11-2) is mixed with the NAL unit (302) of the non-IRAP NAL unit type of the first sub-bitstream (11-1), the NAL unit (304) belongs to the trailing picture unit type (TRAIL), and the NAL unit (302) is either the random access skipped leading picture unit type (RASL) or the random access decodable leading picture unit type (RADL).

8. The decoder (100) according to claim 7, Among them, The indication includes a syntax element in a parameter set.

9. The decoder (100) according to claim 8, Among them, The syntax element is included in at least one of a picture parameter set or a sequence parameter set.

10. The decoder (100) according to any one of claims 1 to 9, wherein, If the NAL unit (302) of the non-IRAP NAL unit type of the first sub-bitstream (11-1) is of the random access skipped leading picture unit type (RASL), and the NAL unit (304) of the non-IRAP NAL unit type of the second sub-bitstream (11-2) is of the trailing picture unit type (TRAIL), then the decoder (100) is configured to derive from the bitstream (11) an indication to the decoder (100) as follows: a) ignore or suppress output, or b) output and mark as discontinuous space segment Those first space segments (131) of the pictures (12a, 12b, 12c) of the video content (12) of the first sub-bitstream (11-1) correspond to the NAL unit (302) of the random access skipped leading picture unit type (RASL) of the first sub-bitstream (11-1).

Images