Decoder, encoder, and method for mixing NAL units of different NAL unit types in a video stream

Abstract

The present invention relates to a decoder, an encoder, and a corresponding method for processing a video data stream (11) that includes a first sub-bitstream (11-1) and a second sub-bitstream (11-2). The concepts described herein provide solutions for mixing different NAL units (301, 302, 3030, 304) of different NAL unit types within access units (30, 31, 32). For example, IRAP NAL unit types can be mixed with different IRAP NAL unit types or non-IRAP NAL unit types, and non-IRAP NAL unit types can be mixed with different non-IRAP NAL unit types.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to decoders and corresponding encoders for decoding / encoding video streams and, in particular, video streams including at least two video sub-streams. The video stream may include a plurality of access units, where each access unit may include at least one picture included in each of the two or more video sub-streams. Each picture may be associated with a NAL (Network Abstraction Layer) unit of a particular NAL unit type. The innovative concepts described herein introduce solutions for mixing NAL units of different NAL unit types. Background Art

[0002] NAL units may be classified into VCL (Video Coding Layer) and non-VCL NAL units according to whether they contain an encoded picture or other associated data, respectively. In the HEVC standard, there are several VCL NAL unit types that identify the kind of picture for decoder initialization and random-access purposes. Using the content of the two-byte NAL unit header, it is possible to easily identify the purpose of the associated payload data.

[0003] Video coding techniques enable random access and bitstream splicing. Regarding random access, the bitstream may start with an IDR (Instantaneous Decoding Refresh) access unit. An IDR access unit contains independently encoded pictures, i.e., encoded pictures that can be decoded without decoding any previous pictures in the NAL unit stream. The presence of an IDR access unit indicates that subsequent pictures in the bitstream will not require reference to pictures before those contained in it for decoding. Within an encoding structure called a closed GOP (Group of Pictures), IDR pictures are used. An alternative to the IDR syntax discussed above is provided by the CRA (Clean Random Access) picture syntax, which specifies the use of independently encoded pictures at the position of a RAP or IRAP ((Intra) Random Access Point), i.e., at a position in the bitstream where the decoder can start successfully decoding a picture without decoding any pictures that occurred earlier in the bitstream, which supports an efficient temporal coding order called open GOP operation.

[0004] Good support for random access can be crucial for enabling channel switching, seeking operations, and dynamic streaming services. Some pictures that follow a CRA picture in decoding order and precede a CRA picture in display order may contain inter-picture prediction references to pictures that are not available at the decoder. Thus, these undecodable pictures can be discarded by the decoder, which starts its decoding process at a CRA point. For this purpose, such undecodable pictures can be identified as RASL pictures (Random Access Skipped Leading). An IRAP picture can be an IDR or a CRA picture, where a CRA picture can be followed by RASL pictures in the bitstream. RASL pictures can be discarded by the decoder because, for example, due to a splicing operation, a RASL picture may contain references to pictures that do not actually exist in the bitstream.

[0005] Another type of picture that can follow an IRAP picture in decoding order and precede an IRAP picture in output order is a RADL picture (Random Access Decodable Leading), which may not contain references to any pictures that precede the IRAP picture in decoding order. RASL pictures and RADL pictures can be collectively referred to as leading pictures (LP). Pictures that follow an IRAP picture in both decoding order and output order are called trailing pictures (TRAIL). Trailing pictures may not contain any references to LPs for inter-picture prediction.

[0006] In current video coding, temporal sub-layer support can also be provided. Thus, a temporal identifier can be specified in the NAL unit header, which indicates the level in a hierarchical temporal prediction structure. This helps to achieve temporal scalability without the need to parse parts of the bitstream other than the NAL unit header. In some cases, it is possible to adjust the number of decoded temporal sub-layers during the decoding process of an encoded video sequence. The position of a point in the bitstream where a sub-layer switch may start decoding some higher temporal layers can be indicated by the presence of a STSA picture (Stepwise Temporal Sub-layer Access). At the position of a STSA picture, it may be possible to switch from decoding a lower temporal sub-layer to decoding a particular higher temporal sub-layer (but not to another layer above it, unless that other layer also contains a STSA picture).

[0007] For multi-reference picture management, a specific set of previously decoded pictures may be present in the DPB (Decoded Picture Buffer) for decoding the remaining pictures in the bitstream. To identify these pictures, a list of POC (Picture Order Count) identifiers may be transmitted in each slice header. The set of retained reference pictures is referred to as the RPS (Reference Picture Set). The POC may include a least significant bit part (LSB) and a most significant bit part (MSB) to indicate the corresponding current picture order count (POC value) for each picture in the bitstream, where the picture order may be different from the decoding order of the corresponding picture. Thus, decoding pictures after an IRAP picture (where the picture references a reference picture preceding the IRAP picture) may be challenging.

[0008] Thus, it is desirable to improve existing encoders and decoders such that proper handling of pictures and picture sequences in the bitstream can be provided, particularly in cases where they occur after an IRAP picture. SUMMARY OF THE INVENTION

[0009] Accordingly, it is proposed to provide a decoder having at least the features of the independent claims and a corresponding apparatus for encoding and / or combining having at least the features of the independent claims. Additionally, corresponding methods of encoding and decoding, as well as corresponding computer program products and bitstreams, are proposed. Advantageous embodiments are set forth in the dependent claims.

[0010] As mentioned above, the bitstream may include: pictures associated with the IRAP NAL unit type, which may also be referred to as IRAP pictures; and pictures not associated with the IRAP NAL unit type (i.e., associated with a non-IRAP NAL unit type), which may also be referred to as non-IRAP pictures.

[0011] According to one aspect, there is provided a decoder for decoding video content, the decoder being configured to decode a bitstream comprising a first sub-bitstream associated with a first spatial segment of a picture of the video content and a second sub-bitstream associated with a second spatial segment of the picture of the video content. In this case, the bitstream comprises at least one access unit, wherein at least one first NAL unit of the first sub-bitstream belongs to a non-IRAP NAL unit type, such as RASL, RADL, TRAIL, STSA. The at least one access unit further comprises at least one different second NAL unit of the second sub-bitstream belonging to a non-IRAP NAL unit type (such as, RASL, RADL, TRAIL, STSA), wherein the first NAL unit of the first sub-bitstream and the second NAL unit of the second sub-bitstream are mixed. In other words, non-IRAP NAL units are mixed with different non-IRAP NAL units. For example, a NAL unit of the TRAIL picture NAL unit type may be mixed with a NAL unit of the RASL picture NAL unit type. According to this non-limiting example, within one access unit, a TRAIL picture of the first sub-bitstream may be mixed with a RASL picture of the second sub-bitstream. Other non-limiting examples may allow a TRAIL picture of the first sub-bitstream to be mixed with a RADL picture of the second sub-bitstream within one access unit. Some additional non-limiting examples may allow a TRAIL picture of the first sub-bitstream to be mixed with a STSA picture of the second sub-bitstream within one access unit. Some additional non-limiting examples may allow a STSA picture of the first sub-bitstream to be mixed with a RADL picture of the second sub-bitstream within one access unit. Some additional non-limiting examples may allow a STSA picture of the first sub-bitstream to be mixed with a RASL picture of the second sub-bitstream within one access unit. Some additional non-limiting examples may allow a RASL picture of the first sub-bitstream to be mixed with a RADL picture of the second sub-bitstream within one access unit.

[0012] According to a further aspect, there is provided a decoder for decoding video content, the decoder being configured to decode a bitstream comprising a first sub-bitstream related to a first spatial segment of a picture of the video content and a second sub-bitstream related to a second spatial segment of a picture of the video content. In this case, the bitstream comprises at least one access unit, within which at least one first NAL unit of the first sub-bitstream belongs to the IRAP NAL unit type and is mixed with at least one second NAL unit of the second sub-bitstream belonging to another IRAP NAL unit type. In other words, a first IRAP NAL unit is mixed with another second IRAP NAL unit. The said another second IRAP NAL unit may belong to the same NAL unit type as the first IRAP NAL unit, or it may belong to a different NAL unit type from the first IRAP NAL unit. For example, a NAL unit of the CRA picture NAL unit type may be mixed with another picture of the same NAL unit type, for example, with another CRA picture NAL or with another picture of a different NAL unit type, for example, with a NAL unit of the IDR picture NAL unit type with a leading picture (i.e., the IDR picture unit type with RADL (IDR_W_RADL)). According to this non-limiting example, within one access unit, a CRA picture of the first sub-bitstream may be mixed with another CRA picture of the second sub-bitstream or with an IDR_W_RADL picture of the second sub-bitstream.

[0013] According to a further aspect, there is provided a decoder for decoding video content, the decoder being configured to decode a bitstream comprising a first sub-bitstream related to a first spatial segment of a picture of the video content and a second sub-bitstream related to a second spatial segment of a picture of the video content. In this case, the bitstream comprises at least one access unit, within which at least one first NAL unit of the first sub-bitstream belongs to the IRAP NAL unit type and is mixed with at least one second NAL unit of the second sub-bitstream belonging to either a non-IRAP leading picture NAL unit type (e.g., RASL or RADL) or a STSA NAL unit type. In other words, within one access unit, an IRAP NAL unit (IDR_W_RADL or CRA) is mixed with a non-IRAP NAL unit, where the non-IRAP NAL unit is any one of a RASL picture, a RADL picture or a STSA picture.

[0014] According to another aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream including a first sub-bitstream and a second sub-bitstream, and wherein the decoder is configured to derive information for setting the most significant POC part of the POC of a picture referred to by a NAL unit from slice headers of NAL units of the first sub-bitstream and the second sub-bitstream, the NAL unit belonging to the IRAP NAL unit type (e.g., CRA, IDR_W_RADL). In this example, if at least one NAL unit within the access unit belongs to a non-instantaneous decoding refresh (non-IDR) unit type, it is required that the most significant POC part be set to a predetermined value for all NAL units of the IRAP NAL unit type within the access unit of the bitstream. In other words, if at least one NAL unit within the access unit refers to a non-IDR picture, e.g., refers to a CRA picture, it is required that the most significant POC part be set to the predetermined value. As a non-limiting example, the predetermined value may be equal to or greater than the most significant POC part of the POC associated with a picture preceding the picture referred to by the NAL unit. In other words, if there is a non-IDR picture within the access unit, POC reset may not occur.

[0015] According to another aspect, a decoder for decoding video content is provided, the decoder being configured to decode a bitstream including a first sub-bitstream and a second sub-bitstream, and wherein the decoder is configured to derive information for setting the most significant POC part of the POC of a picture referred to by a NAL unit from slice headers of NAL units of the first sub-bitstream and the second sub-bitstream, the NAL unit belonging to the IRAP NAL unit type (e.g., CRA, IDR_W_RADL). In this example, any picture reference across any access unit of the bitstream in decoding order is prohibited, within which all NAL units belong to the IRAP NAL unit type and have slice headers in which the most significant POC part is set to a predetermined value. In other words, if each picture within the access unit can refer to the IRAP NAL unit type (e.g., CRA or IDR_W_RADL) and can have the most significant POC part set to the predetermined value, any picture reference from a reference picture preceding the access unit is not allowed. As a non-limiting example, the predetermined value may be less than the most significant POC part of the POC associated with a picture preceding the picture referred to by the NAL unit. For example, if reset of the most significant POC part occurs, the predetermined value may be even smaller. Thus, according to the non-limiting example mentioned above, if the access unit can only include IRAP pictures and if reset of the most significant POC part occurs, picture reference by referring to a preceding reference picture may be prohibited.

[0016] According to a further aspect, there is provided a decoder for decoding video content, the decoder being configured to decode a bitstream comprising a first sub-bitstream and a second sub-bitstream, and wherein the decoder is configured to derive information for setting a most significant POC part of a POC of a picture referred to by a NAL unit from slice headers of NAL units of the first sub-bitstream and the second sub-bitstream, the NAL unit belonging to an IRAP NAL unit type (e.g., CRA, IDR_W_RADL). In this example, any picture reference by a picture of a second access unit following the bitstream in bitstream order to a reference picture of a first access unit, the first access unit being composed of NAL units having slice headers in which the most significant POC part is set to a first predetermined value, and the first access unit preceding the second access unit of the bitstream in bitstream order, is prohibited, within the second access unit, all NAL units belonging to an IRAP NAL unit type and having slice headers in which the most significant POC part is set to a second predetermined value. According to a non-limiting example, the first predetermined value may be greater than the second predetermined value. In other words, if a first predetermined value of a most significant POC part of a first (preceding) access unit is greater than a second predetermined value of a most significant POC part of a second access unit, the following may not be allowed: a picture following the second access unit may refer to a reference picture preceding the second access unit.

[0017] According to a further aspect, a corresponding encoder, a method for encoding, and a method for decoding are proposed.

[0018] According to a further aspect, a computer program is provided, wherein each computer program in the computer program is configured to implement one of the methods described herein when executed on a computer or a signal processor, such that one of the computer programs in the computer program implements the methods described herein. Description of the Drawings

[0019] Hereinafter, embodiments of the present disclosure are described in more detail with reference to the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram showing a bitstream that can be processed by an inventive decoder and device for encoding / merging according to an embodiment,

[0021] Figure 2 A schematic diagram showing a bitstream with an access unit having an IDR_N_LP NAL unit type while all other access units have the same NAL unit type,

[0022] Figure 3Schematic diagram showing a bitstream with an access unit having an IDR_W_LP NAL unit type and a corresponding RADL unit type mixed with a TRAIL unit type

[0023] Figure 4 Schematic diagram showing a bitstream with an access unit having a CRA NAL unit type and a corresponding RASL unit type mixed with a TRAIL unit type

[0024] Figure 5 Showing a full 360 - degree low - resolution thumbnail with selective high - resolution tiles

[0025] Figure 6 Showing an exemplary coding structure where, in one tile (top example), the picture uses the STSA NAL unit type to indicate progressive temporal sub - layer access, while the same picture in another tile (bottom example) uses the RASL NAL unit type

[0026] Figure 7 Showing an exemplary coding structure for high - resolution regions using a closed GOP structure (top) and a high - resolution region using an open GOP structure (bottom)

[0027] Figure 8 Showing an exemplary coding structure of a multi - layer bitstream where POC MSB reset is performed at layer - aligned RAP pictures

[0028] Figure 9 Showing an exemplary coding structure of a multi - layer bitstream where POC MSB reset is prohibited if not all NAL units are IDR

[0029] Figure 10 Showing an exemplary coding structure of a multi - layer bitstream where POC MSB reset is allowed because all NAL units are IDR

[0030] Figure 11 Showing an exemplary coding structure of a multi - layer bitstream where POC MSB reset is performed and LT reference is prohibited

[0031] Figure 12 Showing an exemplary coding structure of a multi - layer bitstream where all POC MSBs are equal to 0 and LT reference is prohibited

[0032] Figure 13 Showing an exemplary coding structure of a multi - layer bitstream where all POC MSBs are equal to 0 and LT reference is allowed, and

[0033] Figure 14An exemplary coding structure for a multi-layer bitstream is shown, where POC MSB reset is not performed and LT references are allowed. Detailed Description

[0034] In the following description, equal or equivalent elements or elements having equal or equivalent functionality are referred to by equal or equivalent reference numerals.

[0035] Method steps depicted by and described with reference to block diagrams may also be executed in an order different from the order depicted and / or described. In addition, method steps relating to specific features of a device may be replaced with those features of the device, and vice versa.

[0036] Mix the NAL unit types in pictures and access units

[0037] 1. Mix the unit types within an encoded picture

[0038] For introduction purposes, Figure 1 The structure of an exemplary video bitstream 11 according to an embodiment of the present invention is shown. The video bitstream 11 can be processed by a decoder 100 and a device 10 for encoding and / or combining the video bitstream 11. The video bitstream 11 includes a sequence of consecutive pictures 12a, 12b, 12c, where two or more of the consecutive pictures 12a, 12b, 12c can be grouped together in a GOP (Group of Pictures) 12, and the GOP (Group of Pictures) 12 can represent at least a part of the expected video content.

[0039] For encoding purposes, each picture 12a, 12b, 12c can be subdivided into spatial segments, such as slices, tiles, or blocks. In this non-limiting example, each of the pictures 12a, 12b, 12c is subdivided into two spatial segments, i.e., into a first (e.g., left) spatial segment 13 1 and a second (e.g., right) spatial segment 13 2 .

[0040] The bitstream 11 can be subdivided into a first sub-bitstream 11-1 related to the first spatial segment 13 of the pictures 12a, 12b, 12c 1 and into a second sub-bitstream 11-2 related to the second spatial segment 13 of the pictures 12a, 12b, 12c 2 .

[0041] Some pictures 12a, 12b may be dependently encoded, for example, by means of predictive coding. Some of these pictures 12b may be predicted by uni-prediction (e.g., by referring to a single preceding picture), while some other pictures 12a may be predicted by bi-prediction (e.g., by referring to a preceding picture and a subsequent picture). Some other pictures 12c may be independently encoded, i.e., these pictures 12c may not involve preceding pictures or subsequent pictures. The independently encoded pictures 12c may provide an (intra-frame) random access point ((I)RAP), which represents the following position in the bitstream 11: at this position, the decoder can start decoding the subsequent picture without decoding any picture that appeared earlier in the bitstream 11. For example, picture 12c is an independently encoded picture, which may also be referred to as an (I)RAP picture. In the present disclosure, the terms RAP and IRAP may be used synonymously.

[0042] The bitstream 11 may include one or more access units 30 for accessing the bitstream 11. Figure 1 In a non-limiting example, the access unit 30 may include aligned (with respect to presentation order) picture segments 13 contained in the first sub-bitstream 11-1 and the second sub-bitstream 11-2. 1 , 13 2 . These aligned image segments 13 are respectively included 1 , 13 2 The pictures 12a, 12b, 12c may be grouped into different categories, for example into IRAP pictures (as mentioned above) and non-IRAP pictures.

[0043] The Network Abstraction Layer (NAL) may provide a means to define those picture segments 13 1 , 13 2 and high-level syntax of the pictures 12a, 12b, 12c. In particular, the NAL unit may be associated with the picture segment 13 for each of the pictures 12a, 12b, 12c. 1 , 13 2 Thus, in the present disclosure, the picture segments 13 mentioned above of the first sub-bitstream 11-1 and the second sub-bitstream 11-2 are 1 , 13 2 Therefore, the first sub-bitstream 11-1 and the second sub-bitstream 11-2 can also be referred to as NAL unit streams. Each NAL unit 13 1 , 13 2 A specific NAL unit type may be included. In turn, the NAL unit type may identify the aforementioned kind of the corresponding picture contained in the NAL unit, ie, whether the corresponding NAL unit belongs to an IRAP type or a non-IRAP type.

[0044] There are multiple applications that rely on mixing IRAP types within a picture, where the IRAP type is the random access property of an access unit (AU) 30 or a part thereof indicated by the NAL unit type. Use cases can be found in the single-layer or multi-layer codec domain, namely: a single-layer codec bitstream where the picture consists of slices that are IRAP, while other slices of the same picture are not IRAP; or a multi-layer codec bitstream where the access unit contains IRAP in one layer and non-IRAP pictures in other layers.

[0045] Currently, support for mixed NAL unit types is given in the prior art, where a single type of IRAP NAL unit type can be mixed with TRAIL pictures within a picture. However, this does not properly cover use cases such as those that motivate the need for features described hereinafter, and the present invention provides a solution for such use cases.

[0046] The main use case that motivates the need for mixed NAL unit types is 360° video, where tile streaming is used and only a subset of the entire 360° is transmitted at high resolution, and the rest at low resolution. When a change in the viewing orientation occurs, then some tiles that were at high resolution are no longer required, and new tiles that were previously shown at low resolution are required to be at high resolution from that time instance forward. These tiles are the only ones that encounter a change (from high resolution to low resolution and vice versa). These tiles are then downloaded using a representation that starts with an IRAP, while other tiles do not need to have this property. When the bitstreams corresponding to different tiles are merged into a single bitstream, some of these tiles contain NAL unit types corresponding to IRAP pictures, while other tiles contain NAL unit types of non-IRAP pictures.

[0047] In addition, there are some other use cases where it is desirable to allow an AU to carry NAL units with different types. For example, in a scenario where there is a Region of Interest (RoI) in a 1080p picture with 720p inside the RoI, for the case of using the same bitstream to feed two types of receivers (e.g., one type of receiver is interested in the entire video, while the other type of receiver is only interested in the RoI), one can envision different parts of the video (the RoI or the rest) having different RAP periods.

[0048] In any case of 360° video streaming, where merging of different bitstreams occurs, or RoI streaming, where the video is initially encoded into different parts with different RAP periods for different regions, the decoder will encounter a bitstream where the NAL units within the encoded pictures are not necessarily the same. Therefore, the present invention provides a solution for mixing NAL units of different NAL unit types.

[0049] 1.1. Mixing trailing pictures (prior art)

[0050] Currently, the following solution is proposed in the prior art: The PPS flag indicates whether a RAP picture is mixed with a non-RAP picture, and then, the NAL unit types that can be encountered in the NAL unit of the mixed AU are the TRAIL NAL unit types for non-IRAP types (up to VCL_RSV_6 or GDR) and a single type of IDR_W_RADL (IDR with leading picture), IDR_N_LP (IDR without leading picture), and CRA (clean random access). Therefore, the prior art proposes to mix IRAP pictures with TRAIL pictures.

[0051] Figure 2 An example according to the prior art is shown, where a NAL unit 201 of the IRAP NAL unit type is mixed with a NAL unit 202 of the trailing picture type (TRAIL). The IRAP NAL unit type in this example is IDR without a leading picture (IDR_N_LP). All other access units have NAL units of the same NAL unit type (i.e., TRAIL).

[0052] 1.2. Mixing leading pictures

[0053] However, these measures are not sufficient for the following envisioned applications: where an open GOP structure can be applied to introduce leading pictures (LP). Figure 3 An exemplary situation is shown, where a NAL unit 301 of the IRAP NAL unit type (e.g., IDR_W_RADL) is shown together with its associated leading picture 302. In this case, the leading picture 302 can be a NAL unit 302 of the NAL unit type RADL (random access decodable leading picture). The NAL unit 301 of the IRAP type (e.g., IDR_W_RADL) can be mixed with a NAL unit 303 of a non-IRAP NAL unit type (e.g., TRAIL). The associated NAL unit 302 of the leading picture type (RADL) of the non-IRAP type can be mixed with a NAL unit 304 of a different non-IRAP type (e.g., TRAIL).

[0054] Figure 4 Another exemplary case is shown, in which a NAL unit 301 of an IRAP NAL unit type (e.g., CRA) is shown together with its associated leading picture 302. In this case, the leading picture 302 can be a NAL unit 302 of the NAL unit type RASL (Random Access Skip Leading Picture). Alternatively, but not shown, the associated leading picture 302 can be a NAL unit 302 of the NAL unit type RADL (Random Access Decodable Leading Picture). The NAL unit 301 of the IRAP type (e.g., CRA) can be mixed with NAL units 303 of a non-IRAP NAL unit type (e.g., TRAIL). The associated NAL unit 302 of a non-IRAP type leading picture type (e.g., RASL) can be mixed with NAL units 304 of a different non-IRAP type (e.g., TRAIL).

[0055] Therefore, it is necessary to process the encoded pictures of the NAL unit types that need to be mixed not only for the case where IRAP NAL units are included, but also for the non-IRAP NAL units only included in the mixed pictures.

[0056] In short, the prior art can suggest mixing IRAP NAL units with non-IRAP NAL units of the TRAIL type or mixing non-IRAP NAL units of the same type, i.e., mixing NAL units of the TRAIL type with NAL units of the TRAIL type.

[0057] On the contrary, the present invention suggests mixing a first non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA) with at least one different second non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA).

[0058] Thus, according to an embodiment, a decoder 100 for decoding video content 12 is provided. The decoder 100 is configured to decode a bitstream 11 that includes a first sub-bitstream 11-1 related to a first spatial segment 13 of pictures 12a, 12b, 12c of the video content 12 1 and a second sub-bitstream related to a second spatial segment 13 of pictures 12a, 12b, 12c of the video content 12 2Associated second sub-bitstream 11-2. According to this embodiment, the bitstream 11 includes at least one access unit 30. Within the at least one access unit 30, at least one first NAL unit 302 of the first sub-bitstream 11-1 belongs to a non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA), and at least one first NAL unit 302 of the first sub-bitstream 11-1 is mixed with at least one different second NAL unit 304 of the second sub-bitstream 11-2 that belongs to a different non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA).

[0059] For example, a TRAIL picture can be mixed with a RADL picture or a RASL picture:

[0060] TRAIL+RADL

[0061] TRAIL+RASL

[0062] Thus, according to an embodiment, the decoder 100 can be configured to process the bitstream 11, wherein the non-IRAP NAL unit type of the NAL unit 302 in the first sub-bitstream 11-1 is a random access decodable leading picture unit type (RADL), and the non-IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is a trailing picture unit type (TRAIL).

[0063] According to another embodiment, the decoder 100 can be configured to process the bitstream 11, wherein the non-IRAP NAL unit type of the NAL unit 302 in the first sub-bitstream 11-1 is a random access skipped leading picture unit type (RASL), and the non-IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is a trailing picture unit type (TRAIL).

[0064] In an embodiment of the present invention, in the bitstream 11, for example, as a flag in the parameter set (PPS, SPS, VPS), there is an indication that the NAL unit types within the encoded picture can be mixed as follows:

[0065] TRAIL+RADL

[0066] TRAIL+RASL

[0067] Regarding the indication that TRAIL + (CRA xor IDR_W_LD xor IDR_N_P) can be mixed, such signaling can be carried in the PPS.

[0068] For example, if the flag is not set (i.e., flag = 0), all NAL units of a picture can include the same NAL unit type, and the picture or PU is said to have the same NAL unit type as the coded slice NAL units of the picture or PU.

[0069] In other cases, if the flag is set (i.e., flag = 1), the picture can include at least two sub - pictures. Additionally or alternatively, the NAL units of the picture can include at least two different NAL unit types. Further additionally or alternatively, the NAL units of the picture can not include the NAL unit type of GDR. Still further additionally or alternatively, if a NAL unit of the picture belongs to at least one of the following NAL unit types:

[0070] Instantaneous decoder refresh with random access to the decoded leading picture (IDR_W_RADL)

[0071] Instantaneous decoder refresh without a leading picture (IDR_N_LP)

[0072] Clean random access (CRA)

[0073] then all other available NAL units of the picture can include one of the three unit types (IDR_W_RADL, IDR_N_LP, CRA) noted above or a trailing picture unit type (TRAIL).

[0074] Alternatively, since RADL and RASL pictures do not have a different canonical decoding process from, for example, TRAIL pictures as IRAP pictures do, this indication can be made at the SPS level. In one embodiment, for example, this indication can be in the form of a constraint flag (e.g., a general constraint flag). The general constraint flag can indicate whether certain mixtures of NAL unit types are allowed. If such a general constraint flag is set (i.e., general constraint flag = 1), it can indicate that there are no mixed NAL unit types in the picture and the flags mentioned above (i.e., flag = 0) in the parameter sets (PPS, SPS, VPS) are not set. If the general constraint flag is not set (i.e., general constraint flag = 0), the constraints mentioned above may not be imposed.

[0075] Thus, according to an embodiment, the decoder 100 may be configured to derive an indication from the bitstream 11, where the indication explicitly signals to the decoder 100 a mixing of NAL units 304 of a non-IRAP NAL unit type of the second sub-bitstream 11-2 (the NAL unit 304 belonging to the trailing picture unit type (TRAIL)) and NAL units 302 of a non-IRAP NAL unit type of the first sub-bitstream 11-1 (the NAL unit 302 being any one of the leading picture unit types of the random access skipped leading picture unit type (RASL) or the random access decodable leading picture unit type (RADL)).

[0076] According to an embodiment, the indication may include a syntax element in a parameter set. For example, the syntax element may be included in at least one of a picture parameter set (PPS) or a sequence parameter set (SPS).

[0077] However, when tiles with RASL attributes and tiles of TRAIL pictures are mixed within a picture, assuming a streaming scenario based on 360-degree video tiles, it is possible for tiles to change position at partial RAP points, and thus the RASL tiles cannot be correctly reconstructed because the reference pictures used will be incorrect (i.e., still depicting the content of other tiles). Identifying those incorrect reconstructed picture regions is crucial for applications, and deleting the entire mixed leading picture (containing TRAIL+RASL tiles) is not an option for the reason of playback continuity. Instead, the client will identify the incorrect regions in the decoded and output pictures and will not use those regions for subsequent processing.

[0078] Instead, considering a setting such as, for example, Figure 5 as illustrated in, the client can use the low-resolution version 51 included in the picture for the content that is incorrectly reconstructed in those mixed leading pictures. In one embodiment, the region corresponding to the potentially incorrect reconstructed tile is indicated to subsequent processing, for example, via an SEI message or by an external means (e.g., an application). The described indication will mean that the region corresponding to the NAL unit of the NAL unit type with RASL will be ignored / not output.

[0079] Thus, according to an embodiment, if the NAL unit 302 of a non-IRAP NAL unit type of the first sub-bitstream 11-1 is of the random access skipped leading picture unit type (RASL), and the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 is of the trailing picture unit type (TRAIL), then the decoder 100 may be configured to derive an indication from the bitstream 11.

[0080] In one example, the indication may indicate to decoder 100 to ignore or suppress those first spatial segments 13 of pictures 12a, 12b, 12c of the video content 12 of the first sub-bitstream 11-1 1 , the first spatial segment 13 1 corresponds to the NAL unit 302 of the random access skip leading (RASL) picture unit type of the first sub-bitstream 11-1.

[0081] In an alternative example, the indication may indicate to decoder 100: those first spatial segments 13 of pictures 12a, 12b, 12c of the video 12 content of the first sub-bitstream 11-1 1 , the first spatial segment 13 1 corresponds to the NAL unit 302 of the random access skip leading picture unit type (RASL) of the first sub-bitstream 11-1; and, additionally, mark the first spatial segment 13 1 as a broken spatial segment. For example, decoder 100 may be configured to mark the broken spatial segment 13 to a subsequent processing chain by means of an SEI (Supplemental Enhancement Information) message 1 .

[0082] Moreover, as another option, mixing of RASL pictures with trailing pictures (TRAIL) is completely avoided, and thus only mixing of RADL pictures with TRAIL pictures is allowed. Thus, in one embodiment, there is a constraint as indicated in the SPS: when a CRA is mixed with a TRAIL picture, the associated leading picture must only belong to the RADL type.

[0083] Thus, according to an embodiment, decoder 100 may be configured to decode a constraint indication from a parameter set, wherein the constraint indication indicates to decoder 100 that: if the non-IRAP NAL unit type of the NAL unit 304 of the second sub-bitstream 11-2 belongs to a non-leading picture NAL unit type (e.g., TRAIL or STSA), then in the case where the IRAP-NAL unit 301 of the associated first sub-bitstream 11-1 of the NAL unit 302 of the first sub-bitstream 11-1 belongs to the clean random access unit type (CRA), the non-IRAP NAL unit type of the NAL unit 302 of the first sub-bitstream 11-1 exclusively belongs to the random access decodable leading RADL picture unit type.

[0084] This may particularly apply to an embodiment where the non-IRAP NAL unit type of the NAL unit 304 of the second bitstream 11-2 belongs to the trailing picture unit type (TRAIL), i.e., in the case of mixing TRAIL + CRA with an associated RADL leading picture.

[0085] In previous aspects, the focus was mainly on the TRAIL NAL unit type. However, in many cases, there can also be STSA NAL unit types, allowing for progressive temporal sublayer access. That is, even though the examples mentioned above mainly involve non-IRAP NAL units of the TRAIL NAL unit type, the present invention can also provide a solution for mixing non-IRAP NAL units of the STSA (progressive temporal sublayer access) unit type with at least one of the RASL unit type, RADL unit type, and TRAIL unit type, namely:

[0086] STSA + RASL

[0087] STSA + RADL

[0088] STSA + TRAIL

[0089] Thus, according to an embodiment, the decoder 100 can be configured to decode the bitstream 11, wherein the non-IRAP NAL unit type of the NAL unit 302 in the first sub-bitstream 11-1 is the progressive temporal sublayer access unit type (STSA), and the non-IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is the trailing picture unit type (TRAIL).

[0090] In another embodiment, the decoder 100 can be configured to decode the bitstream 11, wherein the non-IRAP NAL unit type of the NAL unit 302 in the first sub-bitstream 11-1 is the randomly accessible decodable leading picture unit type (RADL), and the non-IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is the progressive temporal sublayer access unit type (STSA).

[0091] In another embodiment, the decoder 100 can be configured to decode the bitstream 11, wherein the non-IRAP NAL unit type of the NAL unit 302 in the first sub-bitstream 11-1 is the randomly accessible skipped leading picture unit type (RASL), and the non-IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is the progressive temporal sublayer access unit type (STSA).

[0092] Figure 6 Shows an exemplary coding structure, where in one picture segment 13 2 (e.g., a tile), the picture 12a uses NAL units 304 of the STSA NAL unit type to indicate progressive temporal sublayer access, while the same picture 12a in another picture segment 13 1 (e.g., a tile) uses NAL units 302 of the RASL NAL unit type.

[0093] Since the STSA NAL unit type cannot be placed in the lowest temporal level 0, the STSA NAL unit will never be mixed with IRAP NAL units. However, they can be mixed with RADL or RASL NAL unit types just like the TRAIL NAL units mentioned above.

[0094] In one embodiment, a flag is included in the parameter set, for example, in the PPS indicating the mixing of STSA NAL units with RASL or RADL NAL unit types.

[0095] Thus, in one embodiment of the present invention, in the bitstream 11, for example, as a flag in the parameter set (PPS, SPS, VPS), there is an indication that the NAL unit types within the coded picture can be mixed as follows:

[0096] STSA+RADL

[0097] STSA+RASL

[0098] STSA+TRAIL

[0099] In additional embodiments, it is possible to indicate whether such mixing is allowed, for example, using a constraint flag at the sequence level or picture level.

[0100] Thus, according to an embodiment, the decoder 100 can be configured to derive an indication from the bitstream 11, where the indication explicitly signals to the decoder 100 that the NAL unit 304 of the non-IRAP NAL unit type in the second sub-bitstream 11-2 (the NAL unit 304 belongs to the stepwise temporal sub-layer access unit type (STSA)) is mixed with the NAL unit 302 of the non-IRAP NAL unit type in the first sub-bitstream 11-1 (the NAL unit 302 is any one of the random access skip leading picture unit type (RASL) or the random access decodable leading picture unit type (RADL)).

[0101] According to an embodiment, the indication includes a syntax element in the parameter set. According to an additional embodiment, the syntax element is included in at least one of the picture parameter set or the sequence parameter set.

[0102] Alternatively, an indication may be added to the bitstream 11 that effectively indicates that even if all NAL units 302, 304 of picture 12a containing the STSA NAL unit 304 have another NAL unit type, e.g., RASL (see, e.g., NAL unit 302), they all satisfy the constraints associated with the STSA NAL unit type. This can be indicated by a flag in the SPS that indicates that the bitstream 11-1 containing a (spatial) subset of RASL pictures (see, e.g., NAL unit 302) satisfies the constraint that each RASL picture is also an STSA picture. Such signaling is even more important for use cases where the original bitstream containing RASL or RADL NAL units can indicate whether the RADL and RASL NAL units satisfy the properties of an STSA picture, such that if mixed with STSA, it can be easily deduced whether the mixed encoded picture has STSA properties.

[0103] Thus, according to an embodiment, the decoder 100 may be configured to decode a constraint flag of a parameter set from the bitstream 11 that indicates to the decoder 100 that any access unit 30 of the bitstream 11, which includes a mixture of one or more NAL units 302 of a non-IRAP leading picture NAL unit type (e.g., RASL or RADL) and one or more NAL units 304 of a stepwise temporal sublayer access unit type (STSA), satisfies the constraints associated with the stepwise temporal sublayer access unit type (STSA), i.e., each RASL picture 302 can be considered an STSA picture.

[0104] The constraint may indicate to the decoder 100 that if the next higher temporal sublayer than the stepwise temporal sublayer access (STSA) unit type precedes the stepwise temporal sublayer access unit type (STSA), then no picture following in decoding order and associated with the next higher temporal sublayer is allowed to reference pictures in the next higher temporal sublayer.

[0105] Alternatively, the bitstream 11 with the encoded picture 12a having a mixed NAL unit type (e.g., STSA+RASL or RADL) may include a flag indicating that the picture 12a (for which the flag is set) may contain NAL units 304 of the STSA NAL unit type but does not (overall) comply with the constraints associated with the STSA NAL unit type. This will indicate that even if the encoded picture 12a may have an NAL unit type equal to STSA, the properties implied by STSA (i.e., the ability to decode additional temporal levels forward from this AU) do not apply. Thus, the parsing of STSA will be ignored, and stepwise temporal sublayer access will be inferred.

[0106] Thus, according to an embodiment, the decoder 100 may be configured to decode a flag from a parameter set that indicates to the decoder 100 that any access unit 30 that includes one or more NAL units 302 of a non-IRAP leading picture NAL unit type (RASL or RADL) and one or more NAL units 304 of a stepped temporal sub-layer access unit type (STSA) does not comply with the constraints associated with the stepped temporal sub-layer access (STSA) unit type. In addition, the decoder 100 may be configured to refrain from parsing one or more NAL units 304 of the stepped temporal sub-layer access (STSA) unit type in response to the flag, and implicitly infer stepped temporal sub-layer access decoding, i.e., the STSA constraints do not apply to the corresponding access unit.

[0107] This concept may also be applied independently of mixing NAL unit types as described above. Thus, according to an embodiment, a decoder 100 proposed for decoding video content 12 is configured to decode a bitstream 11 and to decode a constraint flag from a parameter set of the bitstream 11, wherein the constraint flag indicates to the decoder 100 that, within the bitstream 11, any access unit 30 that includes NAL units 302 of a non-IRAP leading picture NAL unit type (e.g., RASL or RADL) complies with the constraints associated with the stepped temporal sub-layer access (STSA) unit type.

[0108] 1.3. Mixing various RAP types

[0109] Another important case that cannot be achieved by the prior art is to allow mixing of pictures with an open GOP (Group of Pictures) structure and pictures with a closed GOP structure. For example, mixing of two different types of IRAP NAL unit types (e.g., IDR, CRA) (e.g., IDR_W_RADL and CRA NAL unit types) may be allowed, or mixing of NAL units of the same NAL unit type (e.g., CRA and CRA) may be allowed:

[0110] IDR_W_RADL + CRA

[0111] CRA + CRA

[0112] As mentioned above, this embodiment should be able to achieve mixing of IRAP NAL units of NAL unit types associated with an open GOP structure and IRAP NAL units of NAL unit types associated with a closed GOP structure. For example, a CRA with RADL may be associated with a closed GOP structure, while a CRA with RASL may be associated with an open GOP structure.

[0113] Thus, according to an embodiment, a decoder 100 for decoding video content 12 can be provided. The decoder 100 is configured to decode a bitstream 11, which includes a first sub-bitstream 11-1 related to a first spatial segment 13 of pictures 12a, 12b, 12c of the video content 12 1 and a second sub-bitstream 11-2 related to a second spatial segment 13 of pictures 12a, 12b, 12c of the video content 12 2 According to this example, the bitstream 11 includes at least one access unit 30. Within the at least one access unit 30, at least one first NAL unit 302 of the first sub-bitstream 11-1 belongs to an IRAP NAL unit type mixed with at least one second NAL unit 304 of the second sub-bitstream 11-2, and the at least one second NAL unit 304 belongs to another (same or different) IRAP NAL unit type.

[0114] There is at least one use case where this is very helpful. For example, one approach for 360° video transmission using tiled streaming is a case where the entire low-resolution video content is transmitted (regardless of whether parts of it are also transmitted as high-resolution tiles). In such a case, Figure 5 illustrates changes in viewing orientation and corresponding changes in high-resolution tile selection from one viewport to another.

[0115] Since the entire low-resolution video content is always available on the client side and, for this area, tile-wise stream switching is not required, the low-resolution content can be encoded with CRA and thus with a higher encoding efficiency compared to a closed GOP structure using IDR. Also, for this area, a random access point period longer than the random access point period required for stream switching will be sufficient, which in turn benefits the encoding efficiency of the low-resolution video. On the other hand, as Figure 5 shown, high-resolution tiles may change frequently, etc., during a rapid progression from one viewport to another, and thus using CRA and associated RASL pictures for the low-resolution video will not allow for smooth switching of viewports. This is because the changed positions within the picture or the RASL regions representing newly added tile streams cannot be properly decoded (due to missing references) and are thus discarded when a viewport change is encountered.

[0116] Figure 7An example is shown below, in which the NAL unit 304 associated with the closed GOP structure and the NAL unit 302 associated with the open GOP structure are mixed. In this particular but non-limiting example, the NAL unit 304 of type CRA with a leading RADL 314 (closed GOP) is mixed with the NAL unit 302 of type CRA 302 with a leading RASL 312 (open GOP). This non-limiting example shows the coding structure for: the high-resolution region in the second sub-bitstream 11-2, which uses a closed GOP structure with CRA combined with a leading picture of RADL type; and the low-resolution region in the first sub-bitstream 11-1, which uses an open GOP structure with CRA combined with a leading picture of RASL type.

[0117] According to an embodiment, the decoder 100 is configured to process the bitstream 11, wherein the IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 belongs to the IRAP NAL unit type associated with the closed GOP structure, and the IRAP NAL unit type of the NAL unit 302 in the first sub-bitstream 11-1 belongs to the IRAP NAL unit type associated with the open GOP structure.

[0118] In this particular but non-limiting embodiment as Figure 7 shown, the IRAP NAL unit type of the NAL unit 304 in the second sub-bitstream 11-2 is a CRA unit type with a random access decodable leading picture unit type (CRA with RADL), and the IRAP NAL unit type of the NAL unit 302 in the first sub-bitstream 11-1 is a CRA unit type with a random access skipped leading picture unit type (CRA with RASL).

[0119] It will be noted that Figure 7 the sequence of access units and the pictures contained therein are shown in their display order. However, the display order may be different from the coding order, which may also be referred to as the bitstream order. As can be seen, the access unit 30 includes the mixed picture 12a, and the mixed picture 12a includes: a first VCL NAL unit 302, which represents the first spatial segment 13 1 a of the picture 12a in the first sub-bitstream 11-1; and a second VCL NAL unit 304, which represents the second spatial segment 13 2 a of the picture 12a in the second sub-bitstream 11-2. In this case, the first NAL unit and the second NAL unit are IRAP NAL units. In this particular but non-limiting example, both the first NAL unit 302 and the second NAL unit 304 belong to the IRAP NAL unit type CRA. Furthermore, Figure 7Shows the display order of consecutive pictures.

[0120] Thus, in the display order, an additional access unit 31 may precede the access unit 30 discussed above. The preceding access unit (AU) 31 includes a hybrid picture 12b, and the hybrid picture 12b includes: a first VCL NAL unit 312, which represents a first spatial segment 13 of the picture 12b in the first sub-bitstream 11-1 1 b; and a second VCL NAL unit 314, which represents a second spatial segment 13 of the picture 12b in the second sub-bitstream 11-2 2 b. The first NAL unit 312 of this AU 31 belongs to the non-IRAP NAL unit type RASL, and the second NAL unit 314 of this AU 31 belongs to the non-IRAP NAL unit type RADL. Furthermore, Figure 7 Shows the display order of consecutive pictures.

[0121] Optionally, and further in the display order, one or more additional access units (such as the access unit 32 depicted exemplarily) may precede the access unit 31 discussed above. More generally, in the display order, at least one additional access unit 32 may precede the access unit 31 including non-IRAP NAL units 312, 314 (e.g., RASL or RADL), and the at least one additional access unit 32 may also preferably include non-IRAP NAL units 322, 324 (e.g., RASL or RADL). To maintain the open GOP structure in the first sub-bitstream 11-1, the NAL unit 322 may precede (in the display order) the RASL NAL unit 312 of the access unit 31. The NAL unit 322 is included in the preceding (in the display order) access unit 32 and may also belong to the RASL type. Furthermore, to maintain the closed GOP structure in the second sub-bitstream 11-2, the NAL unit 324 may precede (in the display order) the RADL NAL unit 314 of the access unit 31. The NAL unit 324 is included in the preceding (in the display order) access unit 32 and may also belong to the RADL type.

[0122] The coding order (which is also referred to as the bitstream order) may deviate from Figure 7 the display order depicted therein. For example, the RADL pictures 13 2 b, 13 2 c and the RASL pictures 13 1 b, 13 1 c are encoded dependently, which means they are encoded from their respective following (in the display order) CRA pictures 13 1 a, 13 2a Retrieve differential encoded information. For example, even if access unit 30 follows (in display order) access unit 31, the RADL picture 13 included in access unit 31 2 b and the RASL picture 13 1 b also refers to the CRA picture 13 that follows (in display order) and is included in access unit 30 1 a, 13 2 a.

[0123] The RASL picture 13 1 b may optionally further refer to a preceding (in display order and encoding order) picture, such as the preceding RASL picture 13 Figure 7 exemplarily depicted in 1 c. The additional RASL picture (correspondingly, the additional and preceding RASL picture 13 of the preceding access unit 32 1 c) must be encoded before the RASL picture 13 of access unit 31 1 b. However, the additional and preceding RASL picture 13 of the preceding access unit 32 1 c itself may refer to a picture that is following in display order but leading in encoding order, such as the IRAP (CRA) picture 13 of access unit 30 1 a. This means that even if the IRAP (CRA) picture 13 of access unit 30 1 a is the last picture (in display order) of the sequence discussed above (i.e., the picture IRAP (CRA) 13 of access unit 30 1 a follows the RASL pictures 13 1 b, 13 1 c both in display order), the IRAP (CRA) picture 13 of access unit 30 1 a is also the first picture in encoding order, i.e., it must be encoded first because, during encoding, the RASL pictures 13 1 b, 13 1 c both refer to the IRAP (CRA) picture 13 1 a.

[0124] This also applies to the RADL picture. The RADL picture 13 2 b may optionally further refer to a preceding (in display order and encoding order) picture, such as the preceding RADL picture 13 Figure 7 exemplarily depicted in 2 c. Therefore, the additional and preceding RADL picture 13 of the preceding access unit 32 2 c must be encoded before the RADL picture 13 of access unit 31 2b was encoded previously. However, said further and preceding RADL picture 13 of the preceding access unit 32 2 c itself can refer to pictures that are following in display order but leading in encoding order, such as the IRAP (CRA) picture 13 of access unit 30 2 a. This means that even though said IRAP (CRA) picture 13 of access unit 30 2 a is the last picture (in display order) of the sequences discussed above (i.e., the IRAP (CRA) picture 132a of access unit 30 follows the RADL picture 13 2 b, 13 2 c both), said IRAP (CRA) picture 13 of access unit 30 2 a is also the first picture in encoding order, i.e., it must be encoded first because, during encoding, the RADL pictures 13 2 b, 13 2 c both refer to the IRAP (CRA) picture 13 2 a.

[0125] More generally, non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 13 1 b, 13 1 c and RADL pictures 13 2 b, 13 2 c) can refer to IRAP pictures that follow (in display order) (e.g., CRA pictures 13 1 a, 13 2 a), where the IRAP pictures (e.g., CRA pictures 13 1 a, 13 2 a) must be encoded first (i.e., before their leading (in display order) non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 13 1 b, 13 1 c and RADL pictures 13 2 b, 13 2 c). This means that even though the IRAP pictures (e.g., CRA pictures 13 1 a, 13 2 a) follow (in display order) non-IRAP pictures of the leading picture (LP) type (e.g., RASL pictures 13 1 b, 13 1 c and RADL pictures 13 2 b, 13 2 c), the IRAP pictures (e.g., CRA pictures 13 1 a, 13 2a) is also encoded first. In other words, the leading picture (LP) type (e.g., RASL picture 13 1 b, 13 1 c and RADL picture 13 2 b, 13 2 c) of non-IRAP pictures is immediately before the IRAP picture (e.g., CRA picture 13 1 a, 13 2 a) in display order, while the IRAP picture (e.g., CRA picture 13 1 a, 13 2 a) is immediately before the non-IRAP pictures of the leading picture (LP) type (e.g., RASL picture 13 1 b, 13 1 c and RADL picture 13 2 b, 13 2 c) in bitstream order (encoding order). In other words, the non-IRAP pictures of the leading picture (LP) type (e.g., RASL picture 13 1 b, 13 1 c and RADL picture 13 2 b, 13 2 c) are before the IRAP picture (e.g., CRA picture 13 1 a, 13 2 a) in display order, but the non-IRAP pictures of the leading picture (LP) type (e.g., RASL picture 13 1 b, 13 1 c and RADL picture 13 2 b, 13 2 c) follow the IRAP picture (e.g., CRA picture 13 1 a, 13 2 a) in encoding / bitstream order.

[0126] Considering the above situation and the encoding structure, one advantage of the present invention is that it allows mixing of CRAs with leading RASL (open GOP) and CRAs with leading RADL (closed GOP) within the pictures of the access unit 30. When such a mixed NALU access unit 30 is encountered during normal playback (decoding the entire bitstream 11 from the beginning) within the encoded video sequence (CVS), all the required reference pictures are available. Therefore, the pictures following the mixed picture with the RASL NAL unit will be decoded and output normally.

[0127] However, when such a mixed NALU is encountered during a seek operation or when the decoding process starts from such an AU during random access, a process for generating unavailable reference pictures needs to be invoked, and:

[0128] What needs to be notified to the subsequent processing chain, for example, through SEI indication, is that the RASL region has not been correctly decoded, and the corresponding low-resolution region will be used.

[0129] The affected pictures can be completely deleted from the output.

[0130] This means that when starting to decode the mixed picture, one option is to treat it as a GDR picture, where some parts are decodable and can be shown, while other parts cannot be shown and will be refreshed over time (until all RASL NAL units are finished). For example, in a 360 scenario, when the RAP type is selected contrary to the way shown in Figure 7 (i.e., the low resolution uses a closed GOP structure (e.g., CRA with RADL), and the high-resolution content uses an open GOP structure (e.g., CRA with RASL)), such an operation is relevant. Then, the low resolution can be shown because the low resolution uses a closed GOP configuration, and the player will wait until the high resolution using the open GOP structure is cleanly decoded to show it.

[0131] Another option for handling such a situation would be not to show any of the pictures in the picture affected by the missing reference (even partially) and delete them, i.e., delete any mixed pictures with a NAL unit type equal to RASL.

[0132] In other words, any mixed pictures 12b, 12c with NAL unit types 312, 322 equal to RASL can be marked as deleted / discarded to ensure continuous decoding behavior.

[0133] Thus, according to an embodiment, the decoder 100 may be configured to decode and present the co-located spatial segments 13 2 b, 13 2 c,... of one or more pictures 12b, 12c,... of the access units 31, 32,... The access units 31, 32,... follow, in bitstream order, at least one access unit 30 (e.g., CRA with RADL) involved in the NAL unit 304 of the second sub-bitstream 11-2, and the co-located spatial segments 13 2 b, 13 2 c,... are co-located with the second spatial segment 13 2 a of the picture 12a of at least one access unit 30 involved in the NAL unit 304 of the second sub-bitstream 11-2. In addition, the decoder 100 may be configured to mark the co-located spatial segments 13 1 b, 131 c, …… are discarded, and access units 31, 32, …… follow, in bitstream order, at least one access unit 30 (e.g., a CRA with RASL) associated with the NAL unit 302 of the first sub-bitstream 11-1, and the co-located segment 13 1 b, 13 1 c, …… and the first spatial segment 13 of picture 12a of at least one access unit 30 associated with the NAL unit 302 of the first sub-bitstream 11-1 1 a are co-located.

[0134] According to another embodiment, the decoder 100 may be configured to operate depending on whether the bitstream 11 is to be decoded during normal playback (decoding the entire bitstream 11 from the beginning) or during the start of decoding (e.g., during a seek operation or during the decoding process starting from such an AU during random access).

[0135] In the case of normal playback (decoding the entire bitstream 11 from the beginning), the decoder 100 may be configured to decode and present the co-located spatial segments 13 of one or more pictures 12b, 12c, …… of access units 31, 32, ……, and access units 31, 32, …… follow, in bitstream order, at least one access unit 30 (e.g., a CRA with RADL) associated with the NAL unit 304 of the second sub-bitstream 11-2, and the co-located spatial segments 13 2 b, 13 2 c, …… and the co-located spatial segments 13 are co-located with the second spatial segment 13 of picture 12a of at least one access unit 30 associated with the NAL unit 304 of the second sub-bitstream 11-2. Further, in the case of the start of decoding (e.g., during a seek operation or during the decoding process starting from such an AU during random access), the decoder 100 may be configured to discard one or more pictures 12b, 12c, …… of access units 31, 32, …… that follow, in bitstream order, at least one access unit 30, and resume picture output after one or more discarded pictures 12b, 12c, ……. In other words, any hybrid pictures 12b, 12c having at least one NAL unit 312, 322 equal to RASL are deleted / discarded. 2 b, 13 2 c, …… and the second spatial segment 13 of picture 12a of at least one access unit 30 associated with the NAL unit 304 of the second sub-bitstream 11-2 2 a are co-located. Also, in the case of the start of decoding (e.g., during a seek operation or during the decoding process starting from such an AU during random access), the decoder 100 may be configured to discard one or more pictures 12b, 12c, …… of access units 31, 32, …… that follow, in bitstream order, at least one access unit 30, and resume picture output after one or more discarded pictures 12b, 12c, ……. In other words, any hybrid pictures 12b, 12c having at least one NAL unit 312, 322 equal to RASL are deleted / discarded.

[0136] According to another embodiment, the decoder 100 may be configured to mark the discontinuous spatial segments 13 to a subsequent processing chain by means of SEI (Supplemental Enhancement Information) messages 1 b, 13 1 c, …….

[0137] In another embodiment, there is an indication in the bitstream that, regardless of whether an IDR_W_RADL NAL unit type exists, an AU is considered a CRA, i.e., an IRAP with a NoIncorrectPicOutputFlag equal to 1. There is an indication that the AU is a hybrid NALU but for the mentioned features and IRAP of an open GOP structure. In other words, the decoder 100 can be configured to derive from the bitstream 11 an indication that instructs the decoder 100 to decode at least one access unit 30 associated with the first sub-bitstream 11-1 and the second sub-bitstream 11-2 in a manner corresponding to a clean random access NAL unit type.

[0138] In the above description, embodiments were discussed in which the decoder 100 is configured to process a bitstream 11, where,

[0139] A) A NAL unit of a non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA) is mixed with at least one different second NAL unit of a different non-IRAP NAL unit type (e.g., RASL, RADL, TRAIL, STSA), or,

[0140] B) A NAL unit of an IRAP NAL unit type (e.g., IDR_W_RADL, CRA) is mixed with at least one NAL unit of another IRAP NAL unit type (e.g., IDR_W_RADL, CRA).

[0141] However, there can be additional embodiments according to which the decoder 100 can be configured to process a bitstream 11, where:

[0142] C) A NAL unit of an IRAP NAL unit type (IDR_W_RADL, CRA) is mixed with at least one NAL unit of either a non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or an STSA NAL unit type.

[0143] Additional embodiments also propose providing corresponding devices for processing a video bitstream 11 during the encoding of the bitstream 11 and / or the merging of the first sub-bitstream 11-1 and the second sub-bitstream 11-2. The device 10 can be at least one of an encoder, a merger, or a network node for processing the bitstream 11 according to the innovative principles described herein.

[0144] Thus, according to an embodiment, a device 10 for processing video content 12 is proposed, the device 10 being configured to provide a first spatial segment 13 of pictures 12a, 12b, 12c of the video content 12 1A first sub-bitstream 11-1 related thereto, and providing a second spatial segment 13 of pictures 12a, 12b, 12c of the video content 12 2 A second sub-bitstream 11-2 related thereto, wherein the first sub-bitstream 11-1 and the second sub-bitstream 11-2 are parts of the bitstream 11, wherein:

[0145] A) The bitstream (11) includes at least one access unit (30), within the at least one access unit (30), at least one first NAL unit (302) of the first sub-bitstream (11-1) belongs to a non-IRAP NAL unit type, which is mixed with at least one different second NAL unit (304) of the second sub-bitstream (11-2) belonging to a different non-IRAP NAL unit type, or,

[0146] B) The bitstream (11) includes at least one access unit (30), within the at least one access unit (30), at least one first NAL unit (302) of the first sub-bitstream (11-1) belongs to an IRAP NAL unit type, which is mixed with at least one second NAL unit (304) of the second sub-bitstream (11-2) belonging to another IRAP NAL unit type, or,

[0147] C) The bitstream 11 includes at least one access unit 30, within the at least one access unit 30, at least one first NAL unit 302 of the first sub-bitstream 11-1 belongs to an IRAP NAL unit type, which is mixed with at least one second NAL unit 304 of the second sub-bitstream 11-2 belonging to either a non-IRAP leading picture NAL unit type (e.g., RASL, RADL) or an STSA NAL unit type.

[0148] 1.4. Summary and Exemplary Syntax

[0149] In the following, a summary of the allowed hybrid types within the present invention is shown:

[0150]

[0151] In the new hybrid types described in the present invention, there are two types of hybrids, one type of required IRAP type and another type without an IRAP type.

[0152] Option 1:

[0153] As discussed in the previous section of the text above, one option is to signal using flags in the PPS-only mix of the required IRAP types. For example, again using the existing option of mixing IDR or CRA with TRAIL until VCL_RSV_6 (mixed_nalu_types_in_pic_flag). Non-IRAP types can be signaled using constraint flags at SPS / VPS / DPS.

[0154] A mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture referring to the PPS has more than one VCL NAL unit, and the VAL NAL units do not have the same nal_unit_type value, there is at least one VCL NAL unit with a nal_unit_type value in the range from IDR_W_RADL to CRA_NUT, and the picture is not an IRAP IDR picture. A mixed_nalu_types_in_pic_flag equal to 0 specifies that each picture referring to the PPS has one or more VCL NAL units when the VCL NAL units have a nal_unit_type value in the range from IDR_W_RADL to CRA_NUT, and the VCL NAL units of each picture referring to the PPS have the same nal_unit_type value.

[0155] Thus, if the AU is the first AU of the bitstream or the first AU following the end of a sequence (EOS) NAL unit referring to a PPS with a mixed_nalu_types_in_pic_flag equal to 1, the NAL unit type in the encoded picture must be CRA_NUT and IDR_W_RADL. Mixed pictures with IDR / CRA of non-IRAP types are only allowed for AUs that are not the first AU in the bitstream or sequence.

[0156] A no_non_irap_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that the requirement for bitstream consistency is that the VCL NAL units of pictures in the sequence with TRAIL, STSA, RASL, and RADL have the same nal_unit_type value. A value equal to 0 does not impose such a constraint, i.e., the picture can have two different nal_unit_type values in TRAIL, STSA, RASL, and RADL.

[0157] Option 2:

[0158] Another option would be to have an indication (idc) indicating different combinations instead of the flag (mixed_nalu_types_in_pic_flag) in the PPS.

[0159] The no_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that the requirement for bitstream conformance is that the mixed_nalu_types_in_pic_idc shall be equal to 0. The no_mixed_nalu_types_in_pic_constraint_flag equal to 0 does not impose such a constraint.

[0160] The mixed_nalu_types_in_pic_idc equal to 0 specifies that each picture referring to the PPS has one or more VCL NAL units, and the VCL NAL units of each picture referring to the PPS have the same nal_unit_type value.

[0161] The mixed_nalu_types_in_pic_idc equal to 1 specifies that each picture referring to the PPS has more than one VCL NAL unit, and the VCL NAL units shall all inclusively have two specific nal_unit_type values in the range from TRAIL_NUT to RSV_VCL_6.

[0162] NAL unit type to be mixed with the following TRAIL RADL or RASL or STSA STSA RADL or RASL RASL TRAIL or RADL RADL TRAIL or RASL

[0163] The mixed_nalu_types_in_pic_idc equal to 2 specifies that each picture referring to the PPS has more than one VCL NAL unit, and one or more of the VCL NAL units shall all inclusively have specific nal_unit_type values in the range from IDR_W_RADL to CRA_NUT, and the other VCL NAL units shall all inclusively have specific nal_unit_type values in the range from TRAIL_NUT to RSV_VCL_6 or equal to GRA_NUT.

[0164] This value corresponds to the allowed mixing of NAL units with the existing flags.

[0165] NAL unit type to be mixed with the following CRA_NUT, IDR_W_RADL, IDR_N_LP TRAIL, STSA, RADL, RASL, …… VCL_RSV_6, GRA_NUT

[0166] A mixed_nalu_types_in_pic_idc equal to 3 specifies that each picture that references the PPS has more than one VCL NAL unit with a nal_unit_type value of IDR_W_RADL and one or more VCL NAL units with a nal_unit_type value of CRA_NUT.

[0167] NAL unit type to be mixed with the following CRA_NUT IDR_W_RADL

[0168] For the VCL NAL units of any particular picture, the following applies:

[0169] - If mixed_nalu_types_in_pic_idc is equal to 0, then for all coded slice NAL units of the picture, the nal_unit_type values shall all be the same. The picture or PU is said to have the same NAL unit type as the coded slice NAL units of the picture or PU.

[0170] - If mixed_nalu_types_in_pic_idc is equal to 1 or 2, then the following applies.

[0171] - If one of the VCL NAL units in the VCL NAL units of a picture has a nal_unit_type value equal to TRAIL_NUT, then the picture or PU is said to be a trailing picture or trailing PU.

[0172] - In other cases (where there is no TRAIL_NUT VCL NAL unit in the picture), the picture or PU is said to be a leading picture or leading PU.

[0173] - If one of the VCL NAL units in the VCL NAL units of a picture has a nal_unit_type value equal to RADL_NUT, then the picture or PU is said to be a RADL picture or RADL PU.

[0174] - In other cases (where none of the VCL NAL units in the picture have a nal_unit_type value equal to RADL_NUT), the picture or PU is said to be a RASL picture or RASL PU.

[0175] - In other cases (mixed_nalu_types_in_pic_idc equal to 3), the picture is said to be a CRA picture or CRA PU.

[0176] Aspects related to treating the leading NAL unit type as an STSA picture can be implemented using the constraint flag as follows.

[0177] general_constraint_info( ) { descriptor general_progressive_source_flag u(1) … leading_stsa_pictures_flag u(1) … while( !byte_aligned( ) ) gci_alignment_zero_bit f(1) }

[0178] The leading_stsa_pictures_flag equal to 1 specifies that the inter-prediction reference for RADL pictures and RASL pictures is subject to the constraints specified below. The no_mixed_nalu_types_in_pic_constraint_flag equal to 0 does not impose such a constraint.

[0179] When the leading_stsa_pictures_flag is equal to 1, the following applies:

[0180] - When the current picture is a RASL or RADL picture, there shall not be an active entry in RefPicList[0] or RefPicList[1] with a TemporalId equal to the TemporalId of the current picture.

[0181] - When the current picture is a picture that follows a RASL or RADL in decoding order and precedes the associated IRAP picture with a TemporalId equal to the TemporalId of the current picture, there shall not be a picture with a TemporalId equal to the TemporalId of the current picture, which is included as an active entry in RefPicList[0] or RefPicList[1], and that precedes the RASL or RADL picture in decoding order.

[0182] In other words, the constraints associated with STSA pictures prohibit pictures that follow in decoding order and are associated with the next higher temporal sublayer than STSA from using any reference in that next higher temporal sublayer prior to STSA, i.e., the decoder can start decoding the next temporal sublayer from STSA forward. Of course, such constraints can also be applied to RASL / RADL pictures as described above, and thus, the signaling above can be used as an indication of these properties.

[0183] 2. Mixing picture types within a multi-layer access unit

[0184] By way of non - limiting example, this subsection may relate to multi - layer access units in a multi - layer bitstream. However, the features described herein may already be applied to a single layer. Additionally, by way of non - limiting example, this subsection describes inventive concepts with reference to long - term reference pictures. However, the inventive concept also applies to short - term reference pictures, i.e., the features described herein generally may be applied to picture references. Additionally, by way of non - limiting example, this subsection may describe resetting the POC MSB by setting the POC value to zero, i.e., MSB = 0. However, the zero value only serves as a placeholder to denote a reduced MSB value. For example, it is also possible that any predetermined value that is not equal to 0 but is less than the current MSB at the corresponding bitstream position may cause MSB reset.

[0185] The prior art may support the mixing of pictures of different RAP types within a hierarchical access unit (i.e., for time instances where the access unit simultaneously contains RAP pictures and non - RAP pictures in its layer) by signaling the most significant POC (Picture Order Count) bit (poc_msb_val). However, there may still be problems regarding reference pictures belonging to the leading picture NAL unit type, which have a reference to a picture of an associated IRAP picture that precedes the reference picture in coding order as shown by way of example of long - term reference pictures below, but also applies to short - term reference pictures, and the present invention provides a solution for such reference pictures.

[0186] The current POC signaling - related syntax is as follows.

[0187] SPS syntax

[0188] seq_parameter_set_rbsp( ) { descriptor ... log2_max_pic_order_cnt_lsb_minus4 ue(v) poc_msb_in_rap_pics_flag u(1) if( poc_msb_in_rap_pics_flag > 0 ) poc_msb_len_minus1 ue(v) ... }

[0189] slice header syntax

[0190]

[0191] An exemplary usage of the POC MSB signaling above is to set the POC MSB to 0 when all layers have the same IRAP NAL unit type (e.g., IDR_W_RADL or CRA in all layers as illustrated in Figure 8 ).

[0192] Figure 8The multi-layer bitstream 11 is shown, which includes a first sub-bitstream 11-1 in the first layer L0 and a second sub-bitstream 11-2 in the second layer L1. Each of the layers L1, L0 may include one or more temporal sub-layers T0, T1, T2. Thus, the spatial segments of the picture may be distributed over the different layers L0, L1 and temporal sub-layers T0, T1, T2.

[0193] Figure 8 Some exemplary multi-layer access units 30, 31, 32 are further shown. In this example, the multi-layer access units 30, 31, 32 may include a first NAL unit 302 of the first sub-bitstream 11-1 in the first layer L0, which is mixed with a second NAL unit 304 of the second sub-bitstream 11-2 in the second layer L1. As described above, the NAL unit may include different NAL unit types, in particular, the IRAP NAL unit type and the non-IRAP NAL unit type. Additionally, as mentioned above, the NAL unit may include a slice header, where the picture order count (POC) is signaled. The concepts described herein relate to the most significant POC part, e.g., relate to the most significant POC bit or POC MSB. Thus, in the figure, at each IRAP NAL unit, the POC MSB is depicted.

[0194] For example, the multi-layer access units 30, 32 represent aligned access units, where the POC MSB of the first NAL unit 302 in the first sub-bitstream 11-1 is aligned with the POC MSB of the second NAL unit 304 in the second sub-bitstream 11-2, where the two NAL units belong to the IRAP NAL unit type.

[0195] The multi-layer access unit 31 represents an unaligned access unit, i.e., where the POC MSB of the first NAL unit 302 in the first sub-bitstream 11-1 is not aligned with the POC MSB (e.g., MSB = 0) of the second NAL unit 304 in the second sub-bitstream 11-2 that belongs to the non-IRAP NAL unit type. Thus, the POC MSB of the first NAL unit 302 of the first sub-bitstream 11-1 may be incremented by a predetermined value, e.g., by one.

[0196] Figure 8 It is shown how the MSB signaling in the previously described slice header can be used in such a multi-layer case. The purpose of this signaling is to keep the POC value correct at unaligned RAPs (e.g., unaligned access unit 31 with CRA w MSB = 1), i.e., to avoid resetting the POC (LSB and MSB) to zero when not all layers have an IRAP at the same access unit.

[0197] However, Figure 8There is also a problem with respect to a reference picture 12r (e.g., long-term and / or short-term) when POC alignment occurs. For ease of understanding, the following drawings may only relate to long-term reference pictures. However, the present concept also applies to short-term reference pictures.

[0198] Note that pictures 12a, 12b, 12c,... prior to the last CRA access unit 32 (i.e., the second aligned RAP) have POC MSB > 0 and POC LSB > 0. Thus, when the MSB is reset to 0 at the last CRA access unit 32 (as exemplarily shown in Figure 8 ), then it will not be possible to use the POC MSB or its increment (as for long-term reference pictures) to reference any of the previous pictures 12a, 12b, 12c,.... An example of this is indicated by the green arrow marked 'LT ref' in Figure 8 . Thus, Figure 8 the example shown in

[0199] should demonstrate that picture reference may not be possible in this case.

[0200] Therefore, the present invention provides a solution to this problem such that picture reference will become possible.

[0200] In one embodiment, the following constraint is expressed in the specification or indicated by a bitstream flag (SPS, VPS, DPS): The MSB can only be set to 0 if all pictures within an access unit belong to type IDR (i.e., IDR_W_RADL or IDR_N_LP). Thus, when using CRA, no POC reset occurs and LT reference is allowed (see for example Figure 9 and Figure 10 ).

[0201] As can be seen in Figure 9 , the access unit 32 can be an aligned access unit 32 where two NAL units 302, 304 belong to the IRAP NAL unit type. However, since the first NAL unit 302 of the first sub-bitstream 11-1 belongs to the CRA type, no POC reset should occur. This means that Figure 9 shows a scenario where POC reset is prohibited.

[0202] Conversely, Figure 10 shows a scenario where POC reset is allowed because both the first NAL unit 302 of the first sub-bitstream 11-1 and the second NAL unit 304 of the second sub-bitstream 11-2 belong to the IDR unit type.

[0203] Thus, according to an embodiment, a decoder 100 for decoding video content 12 is provided. The decoder 100 is configured to decode a bitstream 11, which includes a first sub-bitstream 11-1 and a second sub-bitstream 11-2. Among them, the decoder 100 is configured to derive information for setting the most significant POC part (POC MSB) of the POC of a picture 12r referred to by NAL units 302, 304 from slice headers of the NAL units 302, 304 of the first sub-bitstream 11-1 and the second sub-bitstream 11-2, where the NAL units 302, 304 belong to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP). According to this example, if at least one of the NAL units 302, 304 within the access unit 32 belongs to the non-Instantaneous Decoding Refresh unit type, e.g., belongs to the CRA unit type, it is required to set the most significant POC part (POC MSB) to a predetermined value for all NAL units 302, 304 of the IRAP NAL unit type within the access unit 32 of the bitstream 11.

[0204] As mentioned above, as a non-limiting example, this subsection may describe the POC MSB reset by setting the POC value to zero, i.e., MSB = 0. However, the zero value only serves as a placeholder for representing a reduced MSB value symbolically. For example, it is also possible that any predetermined value not equal to 0 but less than the current MSB at the corresponding bitstream position may cause the MSB reset. Therefore, the predetermined value mentioned above may be equal to or greater than the most significant POC part of the POC associated with pictures 12a, 12b,... preceding the picture 12r referred to by the NAL unit 302. In other words, if not all of the NAL units 302, 304 within the access unit 32 belong to the IDR unit type, POC reset is not allowed.

[0205] In some examples, the predetermined value may be equal to zero, i.e., MSB = 0. In some other examples, the predetermined value may be equal to the implicitly derived most significant POC part, e.g., equal to the conventional SOTA POC MSB implicit derivation.

[0206] In another embodiment, there may be the following constraints expressed in the specification or indicated by bitstream flags (SPS, VPS, DPS): Whenever an aligned RAP access unit 32 appears and the POC MSB of the access unit 32 is set to 0, it is not allowed for the picture 12s following the aligned RAP access unit 32 in bitstream order to refer to the (long-term or short-term) reference pictures 12a, 12b, 12c,... preceding the aligned RAP access unit 32.

[0207] For example, Figure 11 shows that even if picture 12s can precede picture 12r in the presentation order, picture 12s can follow picture 12r in the decoding order.

[0208] Figure 11 and Figure 12 shows that any picture 12r having an MSB equal to 0 does not have leading pictures 12a, 12b, 12c,... with LT reference pictures (although Figure 11 represents a reset from MSB = 1 to MSB = 0, but Figure 12 represents that the MSB is equal to 0, for example, because the POC_LSB is long enough so that no increment occurs). Therefore, Figure 11 shows a scenario where POC MSB reset and picture reference are prohibited, and, Figure 12 shows a scenario where all POC MSBs can be equal to zero, but picture reference is not allowed. Figure 13 shows that if the MSB is not equal to 0, LT reference pictures are allowed for leading pictures.

[0209] Thus, according to an embodiment, a decoder 100 for decoding video content 12 is provided, the decoder 100 being configured to decode a bitstream 11 including a first sub-bitstream 11-1 and a second sub-bitstream 11-2, wherein the decoder 100 is configured to derive information for setting the most significant POC part (POC MSB) of the POC of a picture 12r referred to by NAL units 302, 304 of the first sub-bitstream 11-1 and the second sub-bitstream 11-2 (wherein the NAL units 302, 304 belong to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP)) from the slice headers. According to this example, any picture reference is prohibited from spanning any access unit 32 of the bitstream 11 in the decoding order, within which all NAL units 302, 304 belong to the IRAP NAL unit type and have slice headers in which the most significant POC part (POC MSB) is set to a predetermined value.

[0210] Furthermore, as mentioned above, as a non-limiting example, this subsection may describe the POC MSB reset by setting the POC value to zero, i.e., MSB = 0. However, the zero value only serves as a placeholder for representing a reduced MSB value. For example, it is also possible that any predetermined value not equal to 0 but less than the current MSB at the corresponding bitstream position may occur for MSB reset. Therefore, referring to Figures 11 to 13The predetermined value of the described example can be less than the most significant POC part (POC MSB) of the POC associated with the pictures (12a, 12b, 12c, ……) that precede the picture (12r) referenced by the NAL units (302, 304), that is, a POC reset occurs.

[0211] In some examples, the predetermined value can be equal to zero, that is, MSB = 0. In some other examples, the predetermined value can be equal to the implicitly derived most significant POC part, for example, equal to the implicit derivation of the conventional SOTA POC MSB.

[0212] In another embodiment, there can be the following constraints expressed in the specification or indicated by bitstream flags (SPS, VPS, DPS): Whenever an aligned RAP access unit 32 appears and the POC MSB of the access unit 32 is set to 0 (POC reset), it is not allowed for the picture 12s following the aligned RAP access unit 32 in bitstream order to reference the (short-term or long-term) reference pictures 12a, 12b, 12c, …… that precede the aligned RAP access unit 32 and have an MSB different from 0.

[0213] Note that if in the above example shown in Figure 8 there is no increase in the MSB yet, using LT reference pictures will not be a problem because the reference of LT reference pictures is only based on the POC_LSB value.

[0214] Figure 14 It is shown that: in the example, no POC MSB reset occurs. By increasing the POC MSB value of the access unit 32 (in the example shown in Figure 14 the POC MSB value increases, that is, MSB = 3) or making the POC MSB value of the access unit 32 the same as that in the previous access unit 31, the POC MSB reset can not be indicated.

[0215] Thus, according to an embodiment, a decoder 100 for decoding video content 12 is provided. The decoder 100 is configured to decode a bitstream 11, which includes a first sub-bitstream 11-1 and a second sub-bitstream 11-2. Among them, the decoder 100 is configured to derive information for setting the most significant POC part (POC MSB) of the POC of a picture 12r referred to by NAL units 302, 304 from the slice headers of the NAL units 302, 304 of the first sub-bitstream 11-1 and the second sub-bitstream 11-2 (where the NAL units 302, 304 belong to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP)). According to this example, any picture reference by a picture 12s of a second access unit 32 following the bitstream 11 in bitstream order to the reference pictures 12a, 12b, 12c,... of a first access unit 31 is prohibited. The first access unit 31 is composed of NAL units 301, 303, and the NAL units 301, 303 have slice headers in which the most significant POC part is set to a first predetermined value, and the first access unit 31 precedes the second access unit 32 of the bitstream 11 in bitstream order. In the second access unit 32, all NAL units 302, 304 belong to the IRAP NAL unit type (e.g., CRA, IDR_W_LP, IDR_N_LP), and have slice headers in which the most significant POC part (POC MSB) is set to a second predetermined value.

[0216] Furthermore, as mentioned above, as a non-limiting example, this subsection may describe a POC MSB reset by setting the POC value to zero, i.e., MSB = 0. However, the zero value only serves as a placeholder for representing a reduced MSB value. For example, it is also possible that any predetermined value not equal to 0 but less than the current MSB at the corresponding bitstream position may occur for MSB reset. Therefore, regarding the example described with reference to Figure 14 the first predetermined POC MSB value of the preceding first access unit 31 may be greater than the second predetermined POC MSB value of the subsequent second access unit 32. In this case, picture reference will be prohibited. On the contrary, it can be stated that if the second predetermined MSB value of the subsequent second access unit 32 can be greater than the first predetermined POC MSB value of the preceding first access unit 31, then picture reference will be allowed. In other words, if no POC reset occurs, then picture reference will be allowed.

[0217] For example, the second predetermined value may be equal to zero to indicate a POC reset. Thus, if a POC reset occurs, picture reference will be prohibited. In some other examples, the second predetermined value may be equal to the most significant POC part implicitly derived, e.g., equal to the conventional SOTA POC MSB implicitly derived.

[0218] Although some aspects have been described in the context of a device, it is apparent that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of method steps also represent a description of corresponding blocks or items or features of a corresponding device.

[0219] Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0220] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software or at least partly in hardware or at least partly in software. This implementation can be carried out using a digital storage medium (e.g., a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a flash memory) having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the corresponding method is carried out. Thus, the digital storage medium can be computer-readable.

[0221] Some embodiments according to the invention include a data carrier having electronically readable control signals which are capable of cooperating with a programmable computer system such that one of the methods described herein is carried out.

[0222] In general, embodiments of the invention can be implemented as a computer program product having program code that is operative to carry out one of the methods described herein when the computer program product is run on a computer. The program code may, for example, be stored on a machine-readable carrier.

[0223] Other embodiments include a computer program stored on a machine-readable carrier for carrying out one of the methods described herein.

[0224] In other words, thus, embodiments of the inventive method are computer programs having program code for carrying out one of the methods described herein when the computer program is run on a computer.

[0225] Accordingly, a further embodiment of the inventive method is a data carrier (or digital storage medium or computer-readable medium) which comprises a computer program recorded thereon for performing one of the methods described herein. The data carrier, digital storage medium or recording medium is typically tangible and / or non-transitory.

[0226] Accordingly, a further embodiment of the inventive method is a data stream or signal sequence which represents a computer program for performing one of the methods described herein. The data stream or signal sequence can be configured, for example, to be transferred via a data communication connection (e.g., via the Internet).

[0227] A further embodiment includes a processor device, such as a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0228] A further embodiment includes a computer on which a computer program has been installed for performing one of the methods described herein.

[0229] A further embodiment according to the invention includes a device or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver can be, for example, a computer, a mobile device, a memory device, etc. The device or system can include, for example, a file server for transferring the computer program to the receiver.

[0230] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array can cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0231] The devices described herein can be implemented using a hardware device or using a computer or using a combination of a hardware device and a computer.

[0232] The methods described herein can be performed using a hardware device or using a computer or using a combination of a hardware device and a computer.

[0233] Although the present disclosure has been described with reference to illustrative embodiments, the description is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of the present disclosure, will be apparent to those skilled in the art upon reference to this description. Accordingly, it is intended that the appended claims include any such modifications or embodiments.

Claims

1. A decoder, comprising a microprocessor and a memory, the memory including program code which, when read by the microprocessor, causes the decoder to: decode from a bitstream a general constraint no_mixed_nalu_types_in_pic_constraint_flag associated with a plurality of pictures; and decode from the bitstream a plurality of picture parameter sets, each of the plurality of picture parameter sets having a mixed_nalu_types_in_pic_idc, and each of the plurality of picture parameter sets being referenced by a picture among the plurality of pictures, wherein the general constraint no_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that each of the plurality of pictures is constrained to reference a picture parameter set having a mixed_nalu_types_in_pic_idc equal to 0; the general constraint no_mixed_nalu_types_in_pic_constraint_flag equal to 0 specifies that each of the plurality of pictures is not constrained to reference a picture parameter set having a mixed_nalu_types_in_pic_idc equal to 0; and a picture parameter set having a mixed_nalu_types_in_pic_idc equal to 0 specifies that each picture referencing the picture parameter set having the mixed_nalu_types_in_pic_idc equal to 0 has one or more VCL NAL units, and each of the one or more VCL NAL units has the same nal_unit_type value.

2. The decoder according to claim 1, wherein, the general constraint no_mixed_nalu_types_in_pic_constraint_flag has a value equal to 1, and each of the plurality of picture parameter sets has a mixed_nalu_types_in_pic_idc having a value of 0.

3. The decoder according to claim 1, wherein, the general constraint no_mixed_nalu_types_in_pic_constraint_flag has a value equal to 0, and the plurality of picture parameter sets includes a first picture parameter set having a mixed_nalu_types_in_pic_idc having a value of 1.

4. The decoder according to claim 3, wherein, the plurality of pictures includes the first picture referencing the first picture parameter set and having one of a RADL NAL unit, a RASL NAL unit, or a STSA NAL unit and a TRAIL NAL unit.

5. The decoder according to claim 3, wherein, The plurality of pictures includes a first picture that refers to the first picture parameter set and has one of a RADLNAL unit or a RASLNAL unit and a STSANAL unit.

6. The decoder according to claim 3, wherein, The plurality of pictures includes a first picture that refers to the first picture parameter set and has one of a TRAILNAL unit or a RADLNAL unit and a RASLNAL unit.

7. The decoder according to claim 3, wherein, The plurality of pictures includes a first picture that refers to the first picture parameter set and has one of a TRAILVCLNAL unit or a RASLVCLNAL unit and a RADLVCLNAL unit.

8. An encoder comprising a microprocessor and a memory, the memory including program code which, when read by the microprocessor, causes the encoder to: encode the general constraint no_mixed_nalu_types_in_pic_constraint_flag associated with a plurality of pictures into a bitstream; and encode a plurality of picture parameter sets into the bitstream, each picture parameter set in the plurality of picture parameter sets having a mixed_nalu_types_in_pic_idc, and each picture parameter set in the plurality of picture parameter sets being referred to by a picture in the plurality of pictures, wherein the general constraint no_mixed_nalu_types_in_pic_constraint_flag equal to 1 specifies that each of the plurality of pictures is constrained to refer to a picture parameter set having a mixed_nalu_types_in_pic_idc equal to 0, the general constraint no_mixed_nalu_types_in_pic_constraint_flag equal to 0 specifies that each of the plurality of pictures is not constrained to refer to a picture parameter set having a mixed_nalu_types_in_pic_idc equal to 0, and a picture parameter set having a mixed_nalu_types_in_pic_idc equal to 0 specifies that each picture referring to the picture parameter set having the mixed_nalu_types_in_pic_idc equal to 0 has one or more VCL NAL units, and each of the one or more VCL NAL units has the same nal_unit_type value.

9. The encoder according to claim 8, wherein, the general constraint no_mixed_nalu_types_in_pic_constraint_flag has a value equal to 1, and each picture parameter set of the plurality of picture parameter sets has a mixed_nalu_types_in_pic_idc having a value of 0.

10. The encoder according to claim 8, wherein, The general constraint no_mixed_nalu_types_in_pic_constraint_flag has a value equal to 0, and the plurality of picture parameter sets includes a first picture parameter set having a mixed_nalu_types_in_pic_idc with a value of 1.

11. The encoder according to claim 10, wherein, the plurality of pictures includes the first picture that references the first picture parameter set and has one of a RADL NAL unit, a RASL NAL unit, or a STSA NAL unit and a TRAIL NAL unit.

12. The encoder according to claim 10, wherein, the plurality of pictures includes the first picture that references the first picture parameter set and has one of a RADL NAL unit or a RASL NAL unit and a STSA NAL unit.

13. The encoder according to claim 10, wherein, the plurality of pictures includes the first picture that references the first picture parameter set and has one of a TRAIL NAL unit or a RADL NAL unit and a RASL NAL unit.

14. The encoder according to claim 10, wherein, the plurality of pictures includes the first picture that references the first picture parameter set and has one of a TRAIL VCL NAL unit or a RASL VCL NAL unit and a RADL VCL NAL unit.

15. A decoder (100) according to any one of claims 1 - 7, for decoding video content (12), the decoder (100) being configured to: Decode a bitstream (11), the bitstream (11) including a first sub-bitstream (11-1) associated with a first spatial segment (13 1 ) of pictures (12a, 12b, 12c, ……) of the video content (12), a second sub-bitstream (11-2) associated with a second spatial segment (13 2 ) of the pictures (12a, 12b, 12c, ……) of the video content (12), and at least one coded picture, within which at least one first NAL unit (302) of the first sub-bitstream (11-1) belongs to a non-IRAP NAL unit type (RADL, RASL, TRAIL, STSA), the at least one first NAL unit (302) being mixed with at least one different second NAL unit (304) of the second sub-bitstream (11-2) belonging to a different non-IRAP NAL unit type (RADL, RASL, TRAIL, STSA), wherein the non-IRAP NAL unit type of the first NAL unit (302) of the first sub-bitstream (11-1) is a random access decodable leading unit type (RADL), and wherein the non-IRAP NAL unit type of the second NAL unit (304) of the second sub-bitstream (11-2) is a trailing picture unit type (TRAIL).

16. A method for operating a decoder according to any one of claims 1 to 7, for decoding video content (12), the method comprises: Decode a bitstream (11), the bitstream (11) including a first sub-bitstream (11-1) associated with a first spatial segment (13 1 ) of pictures (12a, 12b, 12c, ……) of the video content (12), a second sub-bitstream (11-2) associated with a second spatial segment (13 2 ) of the pictures (12a, 12b, 12c, ……) of the video content (12), and at least one coded picture, within which at least one first NAL unit (302) of the first sub-bitstream (11-1) belongs to a non-IRAP NAL unit type (RADL, RASL, TRAIL, STSA), and the at least one first NAL unit (302) is mixed with at least one different second NAL unit (304) of the second sub-bitstream (11-2) belonging to a different non-IRAP NAL unit type (RADL, RASL, TRAIL, STSA), wherein the non-IRAP NAL unit type of the first NAL unit (302) of the first sub-bitstream (11 - 1) is a random access decodable leading unit type (RADL), and wherein the non-IRAP NAL unit type of the second NAL unit (304) of the second sub-bitstream (11 - 2) is a trailing picture unit type (TRAIL).

Images