Video decoder, video decoding method, computer-readable storage medium, computer program

Abstract

To provide a video decoding method, a video decoder, and a program for extracting a bitstream such as a sub-bitstream from a video bitstream.SOLUTION: An encoder 10 encodes a video sequence 20 into a video bitstream 14 in units of pictures 26. Each picture 26 belongs to a time instant, for example, a frame of the video sequence. Encoded video data belonging to a common time instant may be called an access unit (AU) 22. An extractor 30 receives the video bitstream 14 and selects an OLS from one or more OLSs indicated in the video bitstream 14 based on an indication 32 provided to the extractor 30. A decoder 50 decodes a sub-bitstream 12 to obtain a decoded video sequence.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Description Embodiments of the present invention relate to an apparatus for encoding video into a video bitstream, an apparatus for decoding a video bitstream, and an apparatus for processing a video bitstream, such as an apparatus for extracting a bitstream such as a sub-bitstream from a video bitstream. Further embodiments relate to an encoding method, a decoding method, a processing method (e.g., an extraction method), and a video bitstream. Further embodiments relate to a video bitstream.

Background Art

[0002] In the newly emerged VVC codec, hierarchical coding for temporal, fidelity, and spatial scalability is assumed to be supported from the beginning. That is, an encoded video bitstream structured into so-called layers and (temporal) sub-layers, and the encoded picture data corresponding to a time, i.e., a so-called access unit (AU), can include pictures that can predict each other within each layer, and some of which are output after decoding. The concept of the so-called output layer set (OLS) indicates to the decoder the reference relationship and which layer is output when the bitstream is decoded. The OLS can also be used to identify the corresponding HRD-related timing / buffer information in the form of SEI messages of buffering period, picture timing, and decode unit information carried in a bitstream encapsulated in a so-called scalable nesting SEI message.

Summary of the Invention

[0003] It is desirable to have a concept for handling output layer sets that enable the extraction of sub-bitstreams from a video bitstream, a concept that provides an improved trade-off between a precise definition of the extractable sub-bitstreams by the output layer set (in that case, precisely describing which parts of the video bitstream are extracted), efficient utilization of decoder resources (for example, in that it avoids extracting parts unnecessary for decoding the selected sub-bitstreams, or in that it provides precise information about the decoder settings or requirements for decoding the selected sub-bitstreams), and low signaling overhead.

[0004] A first aspect of the present invention provides a concept for displaying, extracting, and / or decoding a randomly accessible subbitstream from a multilayer video bitstream. According to the first aspect, the extracted randomly accessible subbitstream selectively includes, from the bitstream portions of the access units of the multilayer video bitstream, the bitstream portions associated with the output layers of the randomly accessible subbitstream, as indicated by the output layer set display of the randomly accessible subbitstream, or the bitstream portions necessary to decode the randomly accessible bitstream portions of the output layers.

[0005] A second aspect of the present invention provides the concept of a multilayer video bitstream having multiple layers and multiple time layers. The multilayer video bitstream comprises a display of an output layer set containing one or more layers of the multilayer video bitstream, and a reference layer display showing inter-layer references of the layers of the output layer set. The multilayer video bitstream, in combination with the method by which the multilayer video bitstream is encoded, includes a display, e.g., a time layer display or an in-time layer display, that enables the identification of bitstream portions of layers belonging to the output layer set. This concept makes it possible to identify bitstream portions of an OLS by the type of bitstream portion and / or by inter-layer dependencies of the OLS indicated by the reference layer display. Thus, embodiments of the second aspect enable accurate extraction of sub-bitstreams while avoiding unnecessarily high signaling overhead.

[0006] A third aspect of the present invention provides a concept that enables a decoder that decodes a video bitstream to determine the set of output layers to decode based on the attributes of the video bitstream provided to the decoder. Thus, this concept enables the decoder to select an OLS without any instruction to the decoder which OLS to decode. A decoder that can select an OLS in the absence of instruction can ensure that the bitstream decoded by the decoder satisfies level requirements known to the decoder, for example, by representation in the video bitstream.

[0007] A fourth aspect of the present invention provides a concept for extracting sub-bitstreams from a multilayer video data stream, wherein within the extracted sub-bitstream, one of a predetermined set of bitstream subtypes or picture types (e.g., randomly accessible or independently encoded bitstream subtypes or picture types), for example, the same, access units comprising only picture or bitstream subtypes, are indicated by a sequence start indicator, even if each access unit is not a sequence start access unit in the original multilayer video bitstream from which the sub-bitstream was extracted. Thus, the frequency of sequence start access units in the sub-bitstream may be higher than in the multilayer video data stream, and accordingly, the decoder may benefit from avoiding unnecessarily long waiting times before decoding of the video sequence can begin by making more sequence start access units available.

[0008] A fifth aspect of the present invention provides a concept that allows for the extraction of sub-bitstreams from a multilayer video bitstream such that the sub-bitstreams consist only of pictures belonging to one or more time sublayers associated with an output layer set describing the sub-bitstreams to be extracted. For this purpose, syntax elements in the multilayer video bitstream are used to indicate a given time sublayer in the OLS in a way that identifies different states, including states where a given time sublayer is below the maximum of the time sublayers in an access unit, where at least one picture of a subset of layers is present. By avoiding the transfer of unnecessary sublayers of the multilayer video bitstream, the size of the sub-bitstreams can be reduced, potentially decreasing the decoder requirements for decoding the sub-bitstreams.

[0009] According to one embodiment, the decoder function-related parameters of a subbitstream that exclusively constitute the pictures of the time sublayer belonging to the OLS describing the subbitstream are signaled in the subbitstream and / or multilayer video data stream. Therefore, when determining the decoder-related function parameters, pictures that do not belong to the OLS can be omitted, allowing for efficient utilization of the decoder function.

[0010] A sixth aspect of the present invention provides a concept for handling time sublayers in signaling video parameters for output layer sets of a multilayer video bitstream. According to the embodiment, the OLS is associated with one or more bitstream adaptation sets, one or more buffer requirement sets, and one or more decoder requirement sets signaled by the video bitstream, each of which is valid for one or more time sublayers indicated by the constraint of a maximum time sublayer (e.g., hierarchically ordered time sublayers). The embodiment provides a concept of the relationship between the bitstream adaptation sets, buffer requirement sets, and decoder requirement sets associated with the OLS with respect to the maximum time sublayer to which they are associated, so that a decoder can easily determine the parameters of the OLS associated with the bitstream adaptation sets, buffer requirement sets, and decoder requirement sets. For example, the embodiment may enable a decoder to conclude that the parameters given in the bitstream adaptation sets, buffer requirement sets, and decoder requirement sets of the OLS are fully valid for the OLS. In other embodiments, the decoder can determine how effective the parameters given in the bitstream matching set, buffer requirements set, and decoder requirements set are for the OLS.

[0011] According to the embodiment, the maximum time sublayer indicated by the decoder requirements set associated with the OLS is less than or equal to the maximum time sublayer indicated by the buffer requirements set and bitstream adaptation set, respectively, associated with the OLS, and the parameters in the buffer requirements set and bitstream adaptation set are the same with respect to the same or lower time layers as the maximum time sublayer indicated by the decoder requirements set associated with the OLS. As a result, if the maximum time sublayer indicated by the decoder requirements set associated with the OLS is less than or equal to the maximum time sublayer indicated by the buffer requirements set and bitstream adaptation set, respectively, the decoder can infer that the parameters of the buffer requirements set and bitstream adaptation set associated with the OLS are valid only for the OLS insofar as they are equal to the maximum time sublayer indicated by the decoder requirements set associated with the OLS and relate to the lower time layers. Thus, the embodiment may enable the decoder to determine the video parameters of the OLS based on instructions regarding the maximum time sublayer constraints signaled for each set of parameters, thereby avoiding complex analysis of the OLS and video parameters. Furthermore, this concept allows for the association of the OLS with the buffer requirements set and the bitstream adaptation set. This enables signaling of a maximum time sublayer constraint that is greater than the constraint of the decoder requirements set associated with the OLS. Since this maximum time sublayer constraint is greater than the constraint of the decoder requirements set associated with the OLS, signaling of dedicated buffer requirements sets and dedicated bitstream adaptation sets associated with the same maximum time sublayer as the decoder requirements set can be omitted, potentially reducing the overhead of signaling the video parameter set.

[0012] A seventh aspect of the present invention provides a concept for handling the loss of pictures in a multilayer video bitstream, for example, due to bitstream errors or transmission loss, where the pictures are encoded using interlayer prediction. If a picture that is part of a first layer is lost, that picture is replaced by another picture in a second layer, and the picture in the second layer is used for interlayer prediction of the picture in the first layer. This concept includes replacing a picture with a further picture based on the agreement between the scaling window defined for the picture and the picture boundary of the picture, and the agreement between the scaling window defined for the further picture and the picture boundary of the further picture. If the scaling window defined for the picture and the picture boundary of the picture match, and the scaling window defined for the further picture and the picture boundary of the further picture match, replacing a picture with a further picture may not result in a change in the display window of the presented content, such as a change from detail view to overview. Further embodiments and advantageous embodiments of this disclosure are described below in more detail with respect to the figures. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 shows an example of an encoder, extractor, decoder, and multilayer video bitstream according to an embodiment. [Figure 2] Figure 2 shows an example of an output layer set for a randomly accessible subbitstream. [Figure 3] Figure 3 shows an example of an extracted, randomly accessible subbitstream containing unused pictures. [Figure 4] Figure 4 shows an example of a 3-layer bitstream in which pictures independently encoded across three layers are aligned. [Figure 5] Figure 5 shows an example of a 3-layer bitstream with unaligned, independently encoded pictures. [Figure 6] Figure 6 shows an example of a four-layer bitstream in which an access unit containing independently encoded pictures also contains pictures that reference other time sublayers. [Figure 7] Figure 7 shows an example of a decoder according to an embodiment. [Figure 8] Figure 8 shows an example of a multilayer video data stream with access units that have randomly accessible and non-randomly accessible pictures. [Figure 9] Figure 9 shows an example of a subbitstream of the multilayer video data stream from Figure 8. [Figure 10] Figure 10 shows an example of a multilayer video bitstream with two layers having different picture rates. [Figure 11] Figure 11 shows an encoder, extractor, multilayer video bitstream, and subbitstream according to an embodiment. [Figure 12] Figure 12 shows an example of the mapping between the video parameter set and the output layer set. [Figure 13] Figure 13 shows an example of sharing video parameters between different output layer sets. [Figure 14] Figure 14 shows an example of sharing video parameters between different OLSs according to an embodiment. [Figure 15] Figure 15 shows the sharing of video parameters between different OLSs according to another embodiment. [Modes for carrying out the invention]

[0014] In the following, embodiments will be described in detail, but it should be understood that the embodiments provide many applicable concepts that can be embodied in a wide variety of video encoding concepts. The specific embodiments described are merely illustrative of specific ways to implement and use the present concept and do not limit the scope of the embodiments. In the following description, a plurality of details are shown to provide a more complete description of embodiments of the present invention. However, it will be apparent to those skilled in the art that other embodiments can be implemented without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the examples described herein. Further, the features of different embodiments described herein can be combined with each other unless otherwise stated.

[0015] In the following description of the embodiments, elements that are the same or similar, or elements having the same function, are given the same reference numerals or are identified by the same name, and the repeated description of elements given the same reference numeral or identified by the same name is usually omitted. Therefore, the descriptions provided for elements having the same or similar reference numerals or identified by the same name are interchangeable with each other or can be applied to each other in different embodiments.

[0016] 0. Encoder 10, Extractor 30, Decoder 50, and Video Bitstreams 12, 14 According to FIG. 1 The embodiments described in this section provide examples of frameworks that can incorporate embodiments of the present invention. Hereinafter, embodiments of the concept of the present invention are presented, along with a description of how such concepts may be incorporated into the encoder and extractor of Figure 1. However, embodiments described with respect to Figure 2 and thereafter may be used to form encoders and extractors that do not operate according to the framework described with respect to Figure 1. Furthermore, it should be noted that although the encoder, extractor, and decoder are described together for illustrative purposes in Figure 1, they may be implemented separately from one another. It should also be noted that the extractor and decoder may be combined within a single device, or one of the two may be implemented as part of the other.

[0017] Figure 1 shows an example of an encoder 10, an extractor 30, a decoder 50, a video bitstream 14 (also called a video data stream or data stream), and a sub-bitstream 12. The encoder 10 is for encoding a video sequence 20 into a video bitstream 14. The encoder 10 encodes the video sequence 20 into the video bitstream 14 in units of pictures 26, each picture 26 belonging to a time, for example, a frame of the video sequence. Encoded video data belonging to a common time is sometimes called an access unit (AU) 22. Figure 1 shows the exemplary operation of three access units 221, 222, and 223 of the video sequence 20. Note that a description referring to an access unit 22 may refer to any of the exemplary access units 221, 222, or 223. Each access unit 22 contains or encodes one or more pictures 26, each picture 26 associated with one of several layers of the video bitstream 14. In Figure 1, an example of picture 26 is represented by picture 261 and picture 262. Picture 261 is associated with the first layer 241 of the video bitstream 14, and picture 262 is associated with the second layer 242 of the video bitstream 14. Note that in Figure 1, each of the access units 22 contains pictures for both the first layer 241 and the second layer 242, but the video bitstream 14 may contain access units 22 that do not necessarily contain pictures 26 for each of the layers 24 of the video bitstream 14. Furthermore, the video bitstream 14 may contain additional layers in addition to the first and second layers shown in Figure 1. The encoder 10 is configured to encode each of the pictures 26 into one or more bitstream portions of the video sequence 14, such as NAL units. For example, each of the bitstream portions 16 into which the picture 26 is encoded may encode a portion of the picture 26, such as a slice of the picture 26.The bitstream portion 16 in which picture 26 is actually encoded is sometimes called a Video Encoding Layer (VCL) NAL unit. Bitstream 14 may further include descriptive data, such as non-VCL NAL units, that indicate information describing the encoded video data. For example, each of the access units 22 may include a bitstream portion that signals the descriptive data for its respective access unit, in addition to the bitstream portion that signals the decoded video data. Video bitstream 14 may further include descriptive data that points to multiple access units, or parts of one or more access units. For example, video bitstream 14 may encode an Output Layer Set (OLS) indication 18 that indicates one or more output layer sets.

[0018] The OLS can be a display of a sub-bitstream extractable from the video bitstream 14. The OLS can indicate one or more or all of the multiple layers of the video bitstream 14 as the output layer of the sub-bitstream described by each OLS. Note that the set of layers indicated by the OLS may not necessarily be a proper subset of the layers of the video bitstream 14. In other words, all the layers of the video bitstream 14 may be included in the OLS. The OLS may optionally further include a description of the sub-bitstream described by the OLS and / or decoder requirements for decoding the sub-bitstream indicated by the OLS. Note that the sub-bitstream described by the OLS can be defined by further parameters other than layers, such as a temporal sub-layer or a sub-picture. For example, picture 26 of layer 24 can be associated with one of one or more temporal sub-layers of layer 24. The temporal sub-layer can include pictures at the times associated with each temporal sub-layer. For example, pictures of the first temporal sub-layer can be associated with the times when they form a sequence at the first frame rate, and pictures of the second temporal sub-layer can be associated with times located between the times when the pictures of the first temporal sub-layer are associated, so that the combination of the first temporal sub-layer and the second temporal sub-layer can provide a video sequence with a higher frame rate than in the case of one of the first temporal sub-layer and the second temporal sub-layer. The OLS may optionally indicate a temporal sub-layer for describing which bitstream portion or picture 26 belongs to the sub-bitstream described by the OLS. The temporal sub-layers of the bitstream or the encoded video sequence can be hierarchically ordered, for example, by indexing. For example, the hierarchical order can mean that decoding of pictures of the bitstream including a particular temporal sub-layer requires all the lower temporal sub-layers in the hierarchical order.

[0019] It should be noted that an OLS can include one or more output layers and optionally one or more non-output layers. In other words, an OLS can designate one or more of its layers as output layers and optionally designate one or more of its layers as non-output layers. For example, since a picture in a non-output layer may be needed to decode a picture in an output layer of an OLS, a layer containing a reference picture for a picture in an output layer of an OLS may be included in the OLS as a non-output layer.

[0020] The OLS may further include level information about the bitstream described by the OLS, which indicates or is associated with one or more bitstream constraints, such as one or more maximum values ​​among bitrate, picture size, and frame rate.

[0021] Optionally, the bitstream 14 may further include an extractability representation 19 of the OLS. For example, the extractability representation may be part of the OLS representation. The extractability representation may indicate a (not necessarily appropriate) subset of the bitstream portions 16 that form a decodeable subbitstream associated with the OLS. That is, the extractability representation may indicate which of the bitstream portions 16 belong to the OLS.

[0022] Picture 26 may be encoded into the video bitstream 14 by referencing other pictures, for example, for predicting residuals, motion vectors, and / or syntax elements. For example, a picture may reference another picture in the same access unit (called a reference picture of the picture), and the reference picture may be associated with another layer, which may be called an inter-layer reference picture. As an addition or alternative, a picture may reference a reference picture that is part of the same layer but is in a different access unit than the picture.

[0023] The extractor 30 may receive the video bitstream 14 and, for example, select an OLS from one or more OLSs indicated in the video bitstream 14 based on an indication 32 provided to the extractor 30. The extractor 30 may provide the subbitstream 12 indicated by the selected OLS by transferring at least one bitstream portion 16 belonging to the selected OLS to the subbitstream 12. Note that the transferred bitstream portion may not necessarily correspond exactly to the bitstream portion 16 signaled in the video bitstream 14, because the extractor 30 may modify or adapt one or more of the bitstream portions 16. In Figure 1, an apostrophe is used in the bitstream portion of the subbitstream 12, for example, reference numeral 16', to show the potential changes in the bitstream portion of the video bitstream 14 when it is transferred to the subbitstream 12.

[0024] The sub-bitstream 12 can be decoded by the decoder 50 to obtain a decoded video sequence represented by the sub-bitstream 12. Note that the decoded video sequence may differ from the video sequence 20 in that it may represent only a portion of the video sequence 20 in terms of resolution, fidelity, frame rate, picture size, and video content (when focusing on the extraction of subpictures), and that the decoded video sequence may have distortion due to quantization loss.

[0025] Picture 26 of video sequence 20 may include an independently encoded picture that does not reference pictures of other access units. That is, for example, an independently encoded picture is encoded into the video bitstream 14 without inter-prediction (although, with respect to time prediction, an independently encoded picture may optionally be encoded using inter-layer prediction). Independent encoding allows the decoder to begin decoding the video sequence at the access unit of the independently encoded picture. Independently encoded pictures are sometimes called instantaneous random access points (IRAPs). Examples of IRAP pictures are IDR and CRA pictures. In contrast, a trailing picture may refer to a picture of another access unit that may precede the trailing picture encoding order (the order in which picture 26 is encoded into the video stream 14). The bitstream portion 16 to which independently encoded pictures are encoded may be called an independently encoded bitstream portion, for example, an IRAP NAL unit, while the bitstream portion 16 to which dependently encoded pictures 26 are encoded may be called a dependent bitstream portion, for example, a non-IRAP NAL unit. Furthermore, it should be noted that not all bitstream portions of a single picture are necessarily encoded in the same way, not limited to independent and dependent encoding. For example, the first portion of picture 26 of a first access unit, for example, access unit 221, may be independently encoded, and the second portion of picture 26 of the first access unit may be dependently encoded. In this case, for picture 26 of a second access unit, such as access unit 222, the first portion of picture 26 of the second access unit may be dependently encoded, and the second portion may be independently encoded. In this way, a higher data rate for independent encoding than for dependent encoding can be distributed across multiple access units.Such encoding may be called a general decoder refresh (GDR) because the decoder 50 may have to decode several access units before decoding an entire picture independent of the access units preceding the GDR cycle, i.e., a sequence of pictures in which independently encoded portions covering the entire picture are distributed.

[0026] Several concepts and embodiments are described below with respect to Figure 1. It is noted that features described with respect to an encoder, video bitstream, extractor, or decoder are to be understood as descriptions of other entities. For example, a feature described as being present in a video data stream is to be understood as a description of an encoder configured to encode this feature into a video bitstream, and a decoder or extractor configured to read the feature from the video bitstream. It is further noted that the estimation of information based on the representation encoded in the video bitstream can be performed equally on the encoder side and the decoder side. It should be further noted that the embodiments described in the following sections can be combined with each other.

[0027] 1. Randomly accessible subbitstream display This section describes an embodiment according to a first aspect with reference to Figure 1. The details described in Section 0 can be optionally applied to the embodiment according to the first aspect. Furthermore, the details described in further aspects can be optionally implemented in the embodiments described in this section.

[0028] The randomly accessible bitstream portion may refer to an independently encoded bitstream portion, as explained with respect to Figure 1. Accordingly, a randomly accessible picture (or bitstream portion) can be called an independently encoded picture (or bitstream portion), as explained with respect to Figure 1.

[0029] Some embodiments of the first aspect can refer to a full IRAP level representation for non-aligned IRAP. Embodiments can refer to the effect of IRAP alignment where max_tid_il_ref_pics_plus1 == 0 (e.g., reference to IDR only, or reference to any one of IDR, CRA, or GRD, or one of them, where ph_recovery_poc_cnt is equal to 0).

[0030] Figure 2 shows an example of an output layer set for a video sequence, such as video sequence 20. In other words, the video sequence in Figure 2 can represent a video sequence formed by layers of an OLS, including an output layer L1 and a non-output layer L0. In access unit 22*, the multi-layer OLS bitstream in Figure 2 includes an example of a non-aligned IRAP. That is, access unit 22* includes a non-randomly accessible bitstream portion in one of the layers of the OLS, i.e., L1 in Figure 2.

[0031] Displaying level information for all IRAP sub-bitstreams, i.e., a bitstream containing the result of dropping all non-IRAP NAL units from the bitstream, can be useful for trick mode playback such as fast forward based only on IRAP pictures. This level information refers to the level_idc display, which points to a list of defined limits for parameters such as maximum picture size, maximum picture rate, maximum bitrate, maximum buffer size, maximum slice / tile / subpicture per picture, and minimum compression ratio. However, in the case of multi-layers, it is not uncommon for IRAP pictures not to be aligned across layers. For example, higher (dependent) layers have longer IRAP distances, so higher layers do not have as many IRAPs as lower (reference) layers. This is shown in Figure 2. Here, in the illustrated OLS with two layers, the upper layer L1 contains a trailing NAL unit at the location of POC (Picture Order Count) 3, i.e., within access unit 22*, while the lower layer L0 contains an IRAP NAL unit at the same location.

[0032] Figure 3 shows an example of the extracted full IRAP sub-bitstream, which can be extracted from the multi-layer video bitstream of Figure 2 and executed conventionally. In light of the fact that not all layers of the OLS are output by the decoder, for example, in the example of Figure 2, only the upper layer L1 is marked as the output layer, so maintaining the lower (non-output) layer IDR in the bitstream is a waste of decoder resources unless there is an IDR at the corresponding position of the upper (output) layer, as in the case of picture 260*. This is because the decoder output is the same regardless of whether the L0 IDR of POC3 is present in the full IRAP sub-bitstream, so the decoder will not output any pictures when decoding the full IRAP sub-bitstream of POC3. Furthermore, if the full IRAP level display is used to estimate the maximum playback speed of the full IRAP presentation (for example, in relation to the level and playback speed of the complete OLS bitstream), decoding the L0 IRAP of POC3 as described above will reduce the maximum achievable playback speed of the full IRAP sub-bitstream.

[0033] Therefore, omitting decoding / dropping from such bitstreams, and thereby excluding all IRAP NAL units in non-output layers of access units that do not have IRAP NAL units in all corresponding output layers in the OLS of all such IRAP subbitstreams from consideration by their respective level representations, is part of an embodiment of the present invention.

[0034] According to an embodiment of the first aspect, the video bitstream 14 represents an encoded video sequence 20, as described with respect to Figure 1, for example, and the video bitstream 14 includes a sequence of access units 22, each of which includes one or more bitstream portions 16, each of which is associated with one of a plurality of layers 24 of the video bitstream 14. Each bitstream portion 16 is one of the bitstream portion types, which includes independently encoded bitstream portion types such as randomly accessible bitstream portion types, such as IRAP types. The video bitstream 14 includes, for example, an OLS representation 18 of the OLS of the video bitstream 14, which is encoded by the encoder 10 and to be detected by the extractor 30, and an extractability representation 19 of randomly accessible sub-bitstreams described by the OLS, the OLS including one or more output layers and one or more non-output layers. For example, the randomly accessible sub-bitstreams may be all IRAP sub-bitstreams. For example, the OLS representation may include a level representation for all IRAP sub-bitstreams. It should be noted that the term "all IRAP subbitstream" should not be understood as necessarily exclusively containing either a randomly accessible bitstream portion or an IRAP bitstream portion. Rather, a randomly accessible subbitstream, or all IRAP subbitstream, may in some cases include a non-IRAP bitstream portion or a non-randomly accessible bitstream portion, such as the bitstream portion of a reference picture in the randomly accessible bitstream portion. In other examples, a randomly accessible subbitstream may contain only the randomly accessible bitstream portion.

[0035] According to the embodiment, since the encoder 10 provides a video bitstream 14, for each layer of the OLS, for each access unit 22 beyond the bitstream portion of each access unit 22, the bitstream portions of all output layers are randomly accessible bitstream portions, if each access unit 22 includes one of the randomly accessible bitstream portions.

[0036] In other words, in one embodiment, a requirement for bitstream compliance is that the bitstream shown by all IRAP level displays does not contain any access units without output pictures at any output layer.

[0037] There are use cases where not all output layers have pictures in every access unit of the original bitstream, such as stereo video where the frame rate differs depending on the eye. In such cases, the bitstream requirements are not as strict.

[0038] In another embodiment, a requirement for bitstream compliance is that the bitstream indicated by the full IRAP level display does not contain any access units without output pictures at any output layer.

[0039] Therefore, according to the embodiment, since the encoder 10 according to the first embodiment provides a video bitstream 14, for each layer indicated by the OLS of randomly accessible subbitstreams, for each access unit 22 beyond the bitstream portion of each access unit 22, the bitstream portion 16 of at least one output layer of the output layers is a randomly accessible bitstream portion, if each access unit includes one of the randomly accessible bitstream portions.

[0040] As a result of both outcomes (the randomly accessible bitstream portion of at least one or all output layers), access units that do not satisfy the bitstream constraints will either not be created by the encoder or will be dropped during extraction. In other words, a bitstream containing only IRAP levels is constrained to have no AUs with IRAP in the non-output layers and no AUs with non-IRAP NAL units in the picture at the same temporal position in the output layers.

[0041] Alternatively, the randomly accessible subbitstreams described by the OLS display 18 and the extractability display 19 (which are encoded into the video bitstream 14 by the encoder 10) selectively include each bitstream portion of each access unit if one of the following two conditions is met: The first condition is met if each bitstream portion is a randomly accessible bitstream portion and each bitstream portion is associated with one or more output layers, for example, the bitstream portion of picture 26* in Figure 2. The second condition is met if each bitstream portion is associated with a reference layer of one of the output layers, and each bitstream portion is associated with one or more non-output layers, and furthermore, at least one bitstream portion of an output layer is a randomly accessible bitstream portion beyond the bitstream portion of each access unit (as is the case with the bitstream portion of picture 26** in Figure 2, where the corresponding access unit includes the bitstream portion of picture 26*, which is part of the OLS output layer), or alternatively, all bitstream portions of output layers are randomly accessible bitstream portions beyond the bitstream portion of each access unit (as is also the case with picture 26** in Figure 2).

[0042] Therefore, an apparatus for extracting a sub-bitstream 12 from a video bitstream 14, such as the apparatus 30 in Figure 1 according to a prior embodiment of the first aspect, is configured to provide the sub-bitstream 12 as shown by the OLS display of the randomly accessible sub-bitstream.

[0043] In other words, as an alternative to the optional bitstream constraint that the bitstream indicated by the full IRAP level indication does not contain access units without output pictures at any output layer, according to the embodiment, the indicated level does not contain AUs in which this IRAP NAL unit and non-IRAP NAL unit are "mixed". Therefore, if a bitstream having only the indicated level IRAP is desired, such AUs must be dropped.

[0044] A similar case is considered when the output layer has IRAP NAL units but the reference layer does not. In a further embodiment, as long as the output layer has IRAP NAL units, NAL units in the concurrently referenced layer are not dropped and are considered for the indicated level. For this to work, there is a bitstream constraint that the temporal reference of a reference layer located in the same place but without IRAP NAL units refers only to pictures that are also concurrent with the IRAP NAL units of the output layer. Alternatively, the indicated level applies only to AUs where all NAL units are IRAP NAL units, and if such a bitstream of such a level (IRAP only) is considered, all others are discarded.

[0045] In a further embodiment, instead of referring to a layer selected for the OLS, the requirement that only AUs having all NAL unit types of IRAP type are considered for IRAP level representation is applied to the entire bitstream. In such a case, such an AU includes an access unit delimiter with aud_irap_or_gdr_au_flag = 1, so if there are IRAP NAL units (i.e., not considering the GDR case), the presence of an access unit delimiter with aud_irap_or_gdr_au_flag = 1 is used to determine whether the AU is subject to IRAP-only level representation.

[0046] According to an example of the first embodiment, the video bitstream 14 includes a level representation of a randomly accessible subbitstream, for example, a randomly accessible subbitstream described as extractable according to extractability information 19. The level representation (also called level information) may indicate levels related to bitstream constraints, as described in Section 0, for example, by pointing to a list of levels. For example, the level representation may be associated with one or more of the following: CPD size, DPB size, picture size, picture rate, minimum compression ratio, picture segmentation limits (e.g., tile / slice / subpicture), (e.g., HRD timing such as access unit / DU removal time, DPB output time).

[0047] In other words, when considering an extracted bitstream that has only an IRAP access unit in addition to the level display, further parameters become relevant. Such parameters are the DPB parameter and the HRD parameter.

[0048] According to an embodiment of the first aspect, a decoder such as decoder 50 is configured to check whether the picture buffer conforms to a randomly accessible subbitstream according to extractability information 19. For example, in addition to the parameters described above, decoder 50 may check the extractability information, the HRD parameter, and the level indication in the DPB parameter. The picture buffer may refer to the encoded picture buffer and / or the decoder's decoded picture buffer. Optionally, decoder 50 may be configured to derive from the video bitstream 12, for example, timing information for the picture buffer and timing information for a randomly accessible subbitstream indicated by the OLS indication 18. Based on the timing information, decoder 50 can decode the randomly accessible subbitstream.

[0049] In other words, an all-IRAP variant that omits IRAP in non-output layers without IRAP in the output layer within the same access unit also omits decoding of non-output pictures not used in the aforementioned reference, thus enabling a reduction in DPB requirements (i.e., DPB size in units of picture slots). In particular, this is a separate part of the bitstream level limit and is not directly related to the limit defined by the bitstream's level_idc. Together with the bitstream's picture size, the level sets a limit on the maximum number of pictures that can be held in the DPB. On the other hand, DPB parameters also include more information, such as the maximum number of picture reorders when outputting pictures, i.e., the number of pictures that may precede another picture in the output order but may follow it in the output order. Such information may differ if the extracted bitstream contains only IRAP pictures. Therefore, it is part of the present invention that the decoder signals additional DPB parameters for this representation in order to make more efficient use of its resources. One embodiment of the present invention is shown in Table 1 below. [Table 1-1] [Table 1-2] Table 1 shows that level_indication_for_all_irap_present indicates the presence of level information for all IRAP representations, excluding non-output IRAP pictures that do not have an output layer IRAP in each access unit.

[0050] For example, vps_ols_dpb_params_all_irap_idx[i] specifies the index of the dpb_parameters() syntax structure to be applied to the i-th multilayer OLS, considering only IRAP subbitstreams, within the list of dpb_parameters() syntax structures in the VPS. If it exists, the value of vps_ols_dpb_params_idx[i] is assumed to be within the range of 0 to VpsNumDpbParams-1 (inclusive). If vps_ols_dpb_params_all_irap_idx[i] does not exist, it is assumed to be equal to vps_ols_dpb_params_idx[i]. For a single-layer OLS, the corresponding dpb_parameters() syntax structure resides in the SPS referenced by the layer within the OLS. Each dpb_parameters() syntax structure in VPS shall be referenced by at least one value of vps_ols_dpb_params_idx[i] or vps_ols_dpb_params_all_irap_idx[i] for i in the range from 0 to NumMultiLayerOlss-1 (including both ends).

[0051] As noted, the additional information that may be needed for an extracted bitstream containing only IRAP NAL units is the HRD parameter. The HRD parameter may include one or more of the following, for example, the required CPB size, the time at which access units are removed from the CPB, the bitrate at which the CPB is supplied, or whether the resulting bitstream after extraction corresponds to a constant bitrate representation.

[0052] 2. Reference Picture Alignment Section 2 describes embodiments according to a second aspect with reference to Figure 1, and the details described in Section 0 may be optionally applied to embodiments according to a second aspect. Furthermore, the details described in further aspects may be optionally implemented in the embodiments described in this section.

[0053] In VVC, the output layer set defines the prediction dependencies between layers in the bitstream. The syntax element vps_max_tid_il_ref_pics_plus1[i][j], which is signaled for all direct reference layers of a given layer, allows for further restriction of the amount of reference layer pictures used for prediction, as follows:

[0054] A vps_max_tid_il_ref_pics_plus1[i][j] equal to 0 specifies that, for decoding the i-th layer's picture, the j-th layer's picture will not be used as an ILRP if it is neither an IRAP picture nor a GDR picture and has ph_recovery_poc_cnt = 0. A vps_max_tid_il_ref_pics_plus1[i][j] greater than 0 specifies that, for decoding the i-th layer's picture, the j-th layer's picture will not be used as an ILRP if its TemporalId is greater than vps_max_tid_il_ref_pics_plus1[i][j]-1. If it does not exist, the value of vps_max_tid_il_ref_pics_plus1[i][j] is assumed to be equal to vps_max_sublayers_minus1+1.

[0055] If it does not exist, the value is estimated to be vps_max_sublayer_minus1+1, where vps_max_sublayer_minus1 is the maximum number of sublayers that can exist in any layer within the bitstream. However, for certain layers, the value for the maximum number of sublayers may be smaller.

[0056] This syntax element indicates not only that inter-layer references are not used for some sublayers, or that some sublayers of the reference layer are not needed for decoding, but also a special mode (vps_max_tid_il_ref_pics_plus1[i][j] equal to 0) where only IRAP NAL units or GDR NAL units where ph_recovery_poc_cnt is equal to 0 are needed from the reference layer for decoding. Furthermore, the output layer set describing the bitstream passed to the decoder does not contain the unnecessary NAL units indicated by this syntax element vps_max_tid_il_ref_pics_plus1[i][j], or such NAL units are dropped by the specific decoder implementation performing the extraction process defined in the specification.

[0057] The syntax element vps_max_tid_il_ref_pics_plus1[i][j] exists only in directly referenced layers. For example, imagine an OLS with three layers as shown in Figure 4. Figure 4 shows an example of three layers L0, L1, and L2 where vps_max_tid_il_ref_pics_plus1 is equal to 0 for all referenced layers. Since L2 has L1 as its directly referenced layer and L1 uses L0 as its directly referenced layer, L2 has L0 as its indirectly referenced layer. In such a case, if vps_max_tid_il_ref_pics_plus1[2][1] is equal to 0, then only IRAP NAL units or GDR NAL units where ph_recovery_poc_cnt is equal to 0 are retained from L1 and consequently from L0, as indicated in the specification. More specifically, for each layer of the OLS, a variable is derived that indicates the number of sublayers it holds, namely NumSubLayersInLayerInOLS[i][j] (where i is the OLS index and j is the layer index). If the layer is an output layer, the value of this variable is set to the largest possible temporalId required for the bitstream. If it is not an output layer, but a reference layer for each layer k of the OLS that uses layer j as a reference, the value of NumSubLayersInLayerInOLS[i][k],vps_max_tid_il_ref_pics_plus1[k][j] is set to the maximum value of min(NumSubLayersInLayerInOLS[i][k],vps_max_tid_il_ref_pics_plus1[k][j]). In other words, for each layer k, the minimum value between the number of sublayers required for layer k (NumSubLayersInLayerInOLS[i][k]) and the number of sublayers required for layer j when considering all sublayers of layer k (vps_max_tid_il_ref_pics_plus1[k][j]) is checked. If layer k requires fewer sublayers than the value indicated by vps_max_tid_il_ref_pics_plus1[k][j], then layer j requires the same number of sublayers as layer k, and the smallest of the two values ​​is selected.Then, any further layers k that also use j as a reference are checked, and if other layers indicate that more sublayers are needed, a higher value is obtained, which is the maximum value needed after all layers that use layer j as a reference have been checked.

[0058] Problems arise when IRAP NAL units are not aligned between L0 and L1. For example, as shown in Figure 5, which illustrates a 3-layer example with unaligned IRAP in a lower layer, imagine that at some point L1 there is an IRAP AU but L0 does not, and the IRAP AU in L1 is using the unaligned IRAP AU in L0 as a reference. In such a case, the IRAP-based extraction process will discard the unaligned IRAP in L0 (picture 260* in Figure 5), and therefore will not be able to decode the IRAP AU in L1 (picture 261 in Figure 5).

[0059] In the embodiment, when vps_max_tid_il_ref_pics_plus1[i][j] is 0 for layer i, it is required that any direct or indirect layers of such layer i have aligned IRAP or GDR NAL units where ph_recovery_poc_cnt is equal to 0. In other words, for any indirect reference layer, the NAL units of the direct reference layer on which that indirect reference layer depends must be either IRAP NAL units or GDR NAL units where ph_recovery_poc_cnt is equal to 0.

[0060] According to an embodiment of the second aspect, the video bitstream 14 includes a sequence of access units 22, each of which includes one or more bitstream portions 16. Each bitstream portion 16 is associated with one of a plurality of layers 24 of the video bitstream 14 and one of a plurality of time layers of the video bitstream, for example, a time sublayer as described with respect to Figure 1. Bitstream portions 16 within the same access unit 22 are associated with the same time layer. Furthermore, each bitstream portion is one of the bitstream portion types, which includes a predetermined set of bitstream portion types. For example, the predetermined set of bitstream portion types may include independently encoded bitstream portion types such as IDRs and, optionally, types that depend only on other access units in the predetermined set of bitstream portion types.

[0061] According to an embodiment of the second aspect, the encoder 10 is configured to provide an OLS display of the video bitstream 14, where the OLS comprises one or more layers of the video bitstream. Furthermore, the encoder 10 provides a reference layer display for each layer of the OLS in the video bitstream, showing the set of reference layers on which each layer depends. Furthermore, the encoder 10 provides a time layer display (e.g., vps_max_tid_il_ref_pics_plus1[i][j]) for each layer of the OLS (e.g., i) in the video bitstream 14, showing whether the entire bitstream portion of each reference layer on which each layer depends is one of a predetermined set of bitstream portion types, or, if not, the bitstream portion up to which time layer each layer depends (e.g., the maximum index indexing the time layer on which each layer depends).

[0062] The encoder 10 according to this embodiment is configured such that, for each layer of the OLS (e.g., i), the time layer display shows that the entire bitstream portion of a given reference layer on which each layer depends (of the reference layer of each layer) is one of a predetermined set of bitstream portion types (e.g., the same one), and the access unit containing the bitstream portion of a given reference layer which is one of the predetermined set of bitstream portion types provides a video bitstream such that for each further reference layer on which the given reference layer directly or indirectly depends, it does not contain any bitstream portions other than those of the predetermined set of bitstream portion types (for example, direct dependency or direct reference is a dependency between the (dependent) layer and its reference layer, as shown, for example, in the reference layer display, and indirect dependency or reference is a dependency between the (dependent) layer and a direct or indirect reference layer of the (dependent) layer's reference layer that is not shown in the reference layer display).

[0063] In practice, this is more restrictive than necessary. Such an indirect reference layer (L0) can also be a reference layer for other layers, as shown in Figure 6, which illustrates a 4-layer example that directly references a sublayer with Tid1 (imagine the case of a fourth L3 layer where sublayers 0 and 1 of L0 are required and vps_max_tid_il_ref_pics_plus1[3][0] is 2).

[0064] In such cases, the non-IRAP NAL units of Layer 0 required for the IRAP NAL units of L1 are held in the OLS bitstream corresponding to L0+L1+L2+L3, so the IRAP alignment constraint described in the previous embodiment should become unnecessary. Therefore, to express the constraint, the variable NumSubLayersInLayerInOLS[i][j] can be used instead. This variable indicates the number of sublayers (each with a temporary ID) held in the i-th OLS for the j-th layer (0 means that only IRAP or GDRs with ph_recovery_poc_cnt equal to 0 are held).

[0065] In the embodiment, within the i-th OLS, two layers k and j such that k > j are aligned when NumSubLayersInLayerInOLS[i][j] and NumSubLayersInLayerInOLS[i][k] are equal to 0, and j is a reference layer of k (directly or indirectly), then the IRAP NAL unit or GDR where ph_recovery_poc_cnt is equal to 0.

[0066] Therefore, as an alternative to the embodiments described with respect to Figures 4 and 5, the encoder 10 may supply a bitstream 14 to further reference layers 20D on which a given reference layer 20C depends directly or indirectly, for each layer (e.g., i) 20B of the OLS on which the time layer representation indicates that all bitstream portions of a given reference layer 20C (of the reference layer of each layer) on which each layer 20B depends are one of a predetermined set of bitstream portion types (e.g., the same one). The following two criteria are met: firstly, an access unit 40A, 40B containing a bitstream portion of a given reference layer that is one of a predetermined set of bitstream portion types has no (40A) or does not have (40B) bitstream portions other than the predetermined set of bitstream portion types. Secondly, each further reference layer 20D is, according to the reference layer representation, a reference layer of a direct reference layer 20A on which each layer 20B depends.

[0067] According to an alternative embodiment, the encoder 10 is configured to provide, in addition to the OLS display and reference layer display described with respect to the previous embodiment, an in-layer time layer display [e.g., NumSubLayersInLayerInOLS[i][j]] for each layer (e.g., j) of the OLS (e.g., i) that indicates whether the OLS requires only the bitstream portion of each layer which is one of a predetermined set of bitstream portion types [e.g., indicated by NumSubLayersInLayerInOLS[i][j]=0], or otherwise, a subset of time layers [e.g., Maximum Time Layer Index] that contains the bitstream portion of each layer which the OLS requires.

[0068] According to this embodiment, the encoder 10 is configured to provide each bitstream 14 to each bitstream portion of an access unit that includes one bitstream portion from a set of predetermined bitstream portion types, for each layer 24 of the OLS where the in-layer time layer display indicates that the OLS requires only one bitstream portion from a set of predetermined bitstream portion types. The following conditions are met: if each bitstream portion belongs to a layer of the OLS and its in-layer time display indicates that the OLS requires only the bitstream portion of each layer which is one of a set of predetermined bitstream portion types, then each bitstream portion is one of a set of predetermined bitstream portion types, or, according to the reference layer display, each layer is independent of the layer of each bitstream portion.

[0069] According to a further embodiment, if IRAP is not aligned, inter-layer prediction is not used for layers where non-RAP NAL units reside in the same AU.

[0070] Accordingly, according to another embodiment of the second aspect, the encoder 10 is configured to provide OLS display, reference layer display, and time layer display in the video bitstream 14, as described with respect to the earlier embodiments of Section 2. Furthermore, according to this embodiment, the encoder 10 is configured such that, for each layer of the OLS (e.g., i), the time layer display therefor indicates that all bitstream portions of a predetermined reference layer (of the reference layer of each layer) on which each layer depends are one of a predetermined set of bitstream portion types, and for each layer, for each reference layer on which the predetermined reference layer depends directly or indirectly, there are not any bitstream portions other than the predetermined set of bitstream portion types, and the encoder 10 is configured such that, for each layer, the bitstream portion of the access unit containing the bitstream portion of the predetermined reference layer on which the predetermined reference layer depends is one of a predetermined set of bitstream portion types, without using mutual prediction means for bitstream portions belonging to layers that have a direct or indirect reference to one of further reference layers where there are no bitstream portions other than the predetermined set of bitstream portion types.

[0071] For example, a given set of bitstream subtypes may include one or all of the IRAP types and GDR types where ph_recovery_poc_cnt is equal to zero.

[0072] An embodiment of the encoder 10 according to a second aspect may be configured to provide a level representation of the bitstream 12 that can be extracted from the video bitstream according to the OLS within the video bitstream 14. For example, the level representation may include one or more of the following: encoded picture buffer size, decoded picture buffer size, picture size, picture rate, minimum compression ratio, picture subdivision limits (e.g., tile / slice / subpicture), bitrate, and buffer scheduling (e.g., HRD timing (AU / DU removal time, DPB output time)).

[0073] 3. Bitstream-based OLS determination Section 3 describes an embodiment according to a third aspect of the present invention with reference to Figure 1, and the details described in Section 0 may be optionally applied to the embodiment according to the third aspect. Furthermore, the details described in further aspects may be optionally implemented in the embodiments described in this section.

[0074] Embodiments of the third aspect can provide identification of the OLS corresponding to the bitstream. In other words, according to embodiments of the third aspect, it becomes possible to estimate the OLS of a video bitstream from a video bitstream, which should be decoded or extracted. A decoder that receives the bitstream to be decoded may be given additional information about the operating point to be decoded via its API. For example, in the current VVC draft specification, two variables are set by external means as follows:

[0075] The variable TargetOlsIdx, which identifies the OLS index of the target OLS to be decoded, and the variable Htid, which identifies the topmost time sublayer to be decoded, are set by external means not specified in this specification. The bitstream BitstreamToDecode does not include layers other than those contained in the target OLS and does not include NAL units where TemporalId is greater than Htid.

[0076] This specification does not specify what to do if these variables are not set, because in such cases, the decoder is expected to simply decode the entire bitstream given, rather than, for example, decoding a subset of the bitstream with respect to the time sublayer.

[0077] However, regarding output layer sets, the following problem arises. If a decoder is given a bitstream containing multiple layers, and the parameter set defines multiple OLSs containing all layers in the bitstream (e.g., variants with different output layers), the decoder cannot simply determine which output layer set to decode from the bitstream itself. Depending on the characteristics of the OLS, the OLS to be selected may result in various level requirements due to various DPS parameters, etc. Therefore, it is important that the decoder can select the OLS even when there is no external signal via an API. In other words, a fallback method is needed, as is the case in other situations where there is no external means, such as selecting the topmost time sublayer in the bitstream to decode.

[0078] In one embodiment, there is a bitstream compatibility constraint that a bitstream corresponds to only a single OLS, so that the decoder can clearly determine which OLS to decode from a given bitstream. This property can be instantiated, for example, by a syntax element indicating that all OLS are clearly determinable by the layers present in the bitstream, i.e., there is a unique mapping from the number of layers to the OLS.

[0079] According to an embodiment of the third aspect, an encoder 10 for providing a multilayer video bitstream 14 is configured to include multiple OLSs within the multilayer video bitstream 14, as shown, for example, in the OLS display 18 of Figure 1. Each OLS represents a subset of the layers of the multilayer video bitstream 14. Note that the subset of layers is not necessarily a suitable subset of layers; that is, the subset of layers may include all the layers of the multilayer video bitstream 14. The encoder 10 according to this embodiment provides the multilayer video bitstream 14 such that, for each OLS, the sub-bitstreams, such as the sub-bitstream 12 of the multilayer video bitstream 14 defined by each OLS, are distinguishable from the sub-bitstreams of the multilayer video bitstream defined by any other OLS among the multiple OLSs. For example, the encoder 10 can provide the multilayer video bitstream such that the OLS represent mutually different subsets of the layers of the multilayer video bitstream 14, so that the OLS are distinguishable by subsets of layers.

[0080] For example, each OLS may be defined by indicating a subset of layers in the OLS representation 18 by layer index. Optionally, the OLS representation may include further parameters that define a subset of the bitstream portions of the layers in the OLS whose bitstream portions belong to the OLS. For example, the OLS representation may indicate which time sublayers belong to the OLS.

[0081] In the example, encoder 10 can indicate within the multilayer video bitstream 14 that it uniquely belongs to one of the OLSs. For example, encoder 10 can indicate multiple OLSs such that each OLS has a subset of layers that is distinct from any other OLS. Thus, in the example, encoder 10 can indicate within the multilayer video bitstream 14 that a set of layers in the multilayer video bitstream 14, which may be indicated, for example, by a set of indices, uniquely belongs to one of the OLSs.

[0082] According to one embodiment, the encoder 10 is configured to check the suitability of the multilayer video bitstream 14 by checking, for each of the multiple OLSs, whether the sub-bitstreams of the multilayer video bitstream 14 defined by each OLS are distinguishable from or different from the sub-bitstreams of the multilayer video bitstream defined by any of the other OLSs. For example, if not, the encoder 10 can reject the suitability of the bitstream.

[0083] Therefore, an embodiment of a decoder for decoding a video bitstream, such as the decoder 50 in Figure 1, can derive one or more OLS from the video bitstream to be decoded, each representing a subset of the layers of the video bitstream, when decoding a video bitstream, such as video bitstream 14 or the video bitstream 12 extracted therefrom in Figure 1. The decoder can detect indications within the video bitstream that the video bitstream uniquely belongs to one of the OLS and decode one of the OLS that belongs to the video bitstream.

[0084] For example, the decoder 50 can identify the OLS that belongs to the video bitstream by identifying the layers contained in the video bitstream, and can decode the OLS that accurately identifies the layers contained in the video bitstream. Therefore, an indication that the video bitstream uniquely belongs to one of the OLS may indicate that the set of layers in the video bitstream uniquely belongs to one of the OLS.

[0085] For example, decoder 50 can determine one of the OLSs belonging to the video bitstream by examining the first access unit of the encoded video sequence. The first access unit may refer to the first one received, the first in temporal order, the first in decode order, or the first in output order. Alternatively, decoder 50 can determine one of the OLSs by examining whether the first of the access units is a sequence start access unit type, such as a CVSS access unit, which is defined, for example, by receiving the receive order, the temporal order, or the decode order. For example, decoder 50 can examine whether the first of the access units in the encoded video sequence, or the first of the access units, is a sequence start access unit type with respect to the layer contained in each access unit.

[0086] For example, the decoder 50 can determine one of the OLS such that each access unit contains a picture from exactly one OLS layer, when the first access unit of the encoded video sequence, or the first of the access units, is of the sequence start access unit type.

[0087] Some of the embodiments described above may have the drawback that certain combinations of OLS are prohibited, for example, in a multi-view two-layer scenario having an OLS that outputs both views and an OLS that outputs only the dependently encoded view. To mitigate this limitation, another embodiment of the present invention has a selection algorithm that selects from among the OLS corresponding to the bitstream or its first access unit or its CVSS AU, for example, one or more combinations such as the following:

[0088] • Select an OLS with the highest or lowest index. • Select the OLS with the most output layers. Figure 7 shows a decoder 50 according to one embodiment, which can optionally follow the latter embodiment having a selection algorithm. The decoder 50 according to Figure 7 can optionally correspond to the decoder 50 according to Figure 1. The decoder 50 according to Figure 7 is configured to decode a video bitstream 12, for example, a video bitstream 12 extracted from a multilayer video bitstream 14. Alternatively, the video bitstream 12 in Figure 7 can correspond to the video bitstream 14 in Figure 1. The video bitstream 12 in Figure 7 may be a multilayer video bitstream, but is not necessarily so; i.e., in this example, it may be a single-layer video bitstream. The video bitstream 12 includes access units 22 of the encoded video sequence 20, for example, access units 221, 222, each access unit including one or more pictures of the encoded video sequence, for example, pictures 261, 262. Each of the pictures 26 belongs to one or more layers 24 of the video bitstream 12, as described, for example, in Section 0. The decoder 50 shown in Figure 7 is configured to derive one or more OLS from the video bitstream 12. For example, the video bitstream 12 includes an OLS display 18, as described with respect to Figure 1, for example, and the OLS display 18 shows one or more OLS, such as OLS181 and OLS182 as shown in Figure 7. Each of the one or more OLS shows one or more (not necessarily appropriate) sets of one or more layers 24 of the video bitstream 12. In other words, each OLS shows one or more layers that should be part of each OLS. The decoder 50 determines one of the OLS based on one or more attributes of each OLS and decodes the one OLS determined from the OLS.

[0089] As described in relation to the previous embodiment, the decoder 50 may determine a subset of OLSs and one OLS by checking that the first access unit in the encoded video sequence, or the first access unit, is of the sequence start access unit type. For example, the access unit 221 shown in Figure 7 may be the first access unit in the encoded video sequence of the video bitstream 12 (e.g., the first one received, or the first in encoding order, the first in temporal order, or the first in output order). In another example, the encoded video sequence 20 may include further access units preceding access unit 221, and since the preceding access units are not sequence start access units, access unit 221 is the first sequence start access unit of sequence 20. The decoder 50 may check access unit 221 to detect picture 261 of the first layer 241 and picture 262 of the second layer 242. Based on this finding, the decoder 50 can conclude that the video bitstream 12 comprises a first layer 241 and a second layer 242.

[0090] According to the embodiment of Figure 7, the decoder 50 determines which OLS to decode based on one or more attributes of the OLS. These one or more attributes may include one or more of the following: the index of each OLS (i.e., the OLS index) and / or the number of layers in the OLS and / or the number of output layers in each OLS. For example, the decoder may determine which OLS has the highest or lowest OLS index and / or the largest number of layers and / or the most output layers. In other words, the decoder 50 can evaluate which OLS has the highest or lowest OLS index and / or the largest number of layers. Additionally or alternatively, the decoder 50 can evaluate which OLS contains the output layer with the highest or lowest index. By selecting an OLS based on the number of output layers and / or the number of layers, a bitstream that provides the highest quality output to the video sequence may be selected for decoding.

[0091] For example, the decoder 50 can select as a single OLS the OLS with the most output layers, the OLS with the most layers exceeding the OLS with the most output layers, and the OLS with the lowest OLS index exceeding the OLS with the most layers exceeding the OLS with the most output layers.

[0092] According to one embodiment, the decoder 50 determines one OLS by evaluating which OLS has the largest number of layers. If there are multiple OLS with the largest number of layers, the decoder 50 may evaluate the one with the most output layers among the OLS with the largest number of layers, or it may select the OLS with the most output layers among the OLS with the largest number of layers as one OLS.

[0093] In other words, if there is no instruction for an OLS to encode, the decoder 50 can decode an OLS that uses most of the layers present in the bitstream, where all the necessary layers of the OLS shown in the OLS display 18 are present, and provide a high-fidelity video output.

[0094] Further embodiments of the decoder 50 are described below with reference to Figure 7.

[0095] Furthermore, the decoder 50 of this further embodiment may optionally follow the selection algorithm described above, and may optionally correspond to the decoder 50 of Figure 1. According to this further embodiment, the decoder 50 is configured to decode a video bitstream 12, for example, a video bitstream 12 extracted from a multilayer video bitstream 14, as shown in Figure 1. Alternatively, the video bitstream 12 of Figure 7 may correspond to the video bitstream 14 of Figure 1. The video bitstream 12 of Figure 7 may be a multilayer video bitstream, but is not necessarily so; i.e., in this example, it may be a single-layer video bitstream. The video bitstream 12 includes access units 22 of the encoded video sequence 20, for example, access units 221, 222, each access unit including one or more pictures of the encoded video sequence, for example, pictures 261, 262. Each of the pictures 26 belongs to one or more layers 24 of the video bitstream 12, as described, for example, in Section 0. The decoder 50 according to this further embodiment is configured to derive one or more OLS from the video bitstream 12. For example, the video bitstream 12 includes an OLS display 18, as described with respect to Figure 1, for example, and the OLS display 18 shows one or more OLS, such as OLS181 and OLS182, as shown in Figure 7. Each of the one or more OLS shows a (not necessarily appropriate) set of one or more layers 24 of the video bitstream 12. In other words, each OLS shows one or more layers that should be part of each OLS. The decoder 50 according to this further embodiment determines a subset of OLS from (or among) the OLS such that each of the OLS of the subset of OLS belongs to the video bitstream 12. The decoder 50 according to this further embodiment further determines one of the subsets of OLS based on one or more attributes of each of the subsets of OLS and decodes one OLS determined from the subset of OLS.

[0096] For example, an OLS attributed to a video bitstream may mean that one or more sets of layers present in the video bitstream 12 correspond to the set of layers indicated in each OLS. In other words, the decoder 50 may determine a subset of OLS attributed to the video bitstream 12 based on the set of layers present in the video bitstream 12. That is, for each subset of OLS, the decoder 50 may determine the subset of OLS such that the subset of layers indicated by each OLS corresponds to the set of layers in the video bitstream 12. For example, in Figure 7, the video bitstream 12 includes, exemplarily, a picture 261 on a first layer 241 and a picture 262 on a second layer 242. OLS 181 indicates that the first layer 241 and the second layer 242 are part of OLS 181. Also, OLS 182 indicates that both the first and second layers are part of OLS 182. Therefore, as shown in the example in Figure 7, the decoder 50 can attribute both OLS 181 and 182 to a portion of the OLS subscripts that belong to the video bitstream 12.

[0097] Optionally, the decoder 50 may consider only those OLS that are decodeable by the decoder 50, according to the level information of the OLS.

[0098] As described in relation to the previous embodiment, the decoder 50 may determine a subset of the attributed OLS and one OLS by checking that the first access unit of the encoded video sequence, or the first access unit, is of the sequence start access unit type. For example, the access unit 221 shown in Figure 7 may be the first access unit of the encoded video sequence of the video bitstream 12 (e.g., the first one received, or the first in encoding order, the first in temporal order, or the first in output order). In another example, the encoded video sequence 20 may include further access units preceding access unit 221, and since the preceding access units are not sequence start access units, access unit 221 is the first sequence start access unit of sequence 20. The decoder 50 may check access unit 221 to detect picture 261 of the first layer 241 and picture 262 of the second layer 242. Based on this finding, the decoder 50 can conclude that the video bitstream 12 comprises a first layer 241 and a second layer 242.

[0099] According to one embodiment, the decoder 50 can determine the subset of the OLS such that the first access unit of the encoded video sequence, or, if the first access unit is a sequence start access unit type, for example, access unit 221, each access unit accurately contains the pictures of each layer of the subset of the OLS.

[0100] According to the embodiment of Figure 7, the decoder 50 determines which OLS to decode based on one or more attributes of a subset of OLS. One or more attributes may include one or more of the following: the index and / or number of output layers of each OLS, the highest or lowest index, i.e., the index of the highest or lowest layer, and the number of highest layers. In other words, the decoder 50 can evaluate which OLS in the subset of OLS belonging to the video bitstream 12 contains layers indexed with the highest or lowest layer index and / or which OLS has the most layers. Additionally or alternatively, the decoder 50 can evaluate which OLS in the subset of OLS has the maximum or minimum number of output layers and / or which OLS in the subset of OLS constitutes the output layer with the highest or lowest index.

[0101] According to one embodiment, the decoder 50 determines one OLS by evaluating which OLS has the largest number of layers among the subset of OLS. If there are multiple OLS with the largest number of layers, the decoder 50 may evaluate the one with the most output layers among the OLS with the largest number of layers, or it may select the OLS with the most output layers among the OLS with the largest number of layers as one OLS.

[0102] In other words, if there is no instruction for an OLS to encode, the decoder 50 can decode an OLS that uses most of the layers present in the bitstream, where all the necessary layers of the OLS shown in the OLS display 18 are present, and provide a high-fidelity video output.

[0103] In other words, an embodiment of the third aspect includes a decoder 50 for decoding video bitstreams 12, 14, the video bitstream 14 including access units 22 of encoded video sequence 20, each access unit 22 including one or more pictures 26 of the encoded video sequence, each picture belonging to one or more layers 24 of the video bitstream 14. The decoder is configured to derive one or more output layer sets (OLS) 181, 182 from the video bitstream 14, each representing one or more sets of layers of the video bitstream 14; determine subsets of OLS 181, 182 from the OLS 181, 182, each of which subsets of OLS belong to the video bitstream 14; determine one subset of OLS based on one or more attributes of each subset of OLS; and decode the OLS.

[0104] According to one embodiment, the decoder 50 is configured to determine a subset of the OLS such that, for each subset of the OLS, the subset of layers indicated by each OLS corresponds to a set of layers in the video bitstream 14.

[0105] According to one embodiment, the decoder 50 is configured to determine a subset of OLS and one OLS by checking that the first access unit of the encoded video sequence, or the first access unit 22, is of the sequence start access unit 22 type.

[0106] According to one embodiment, the decoder 50 is configured to determine a subset of the OLS such that the first of the access units 22 of the encoded video sequence, or, if the first of the access units 22 is of the sequence start access unit type, each access unit contains exactly the pictures of each layer of the subset of the OLS.

[0107] According to one embodiment, the decoder 50 is configured to determine one of the OLS by evaluating one criterion for each of one or more attributes of a subset of the OLS.

[0108] 4. Sub-bitstream sequence start access unit Section 4 describes embodiments of a fourth aspect of the present invention with reference to Figure 1. The descriptions provided in Section 0 can be optionally applied to embodiments of the fourth aspect. Further details described in this section can be optionally implemented in the embodiments described here.

[0109] Some embodiments of the fourth aspect may relate to access unit delimiters (AUDs) in supplemental enhancement information (SEI) that enable coded video sequence start access units (CVSS AUs) in a video bitstream, such as video bitstream 14, from which video bitstream 12 is extracted. For example, a CVSS AU may be a randomly accessible AU, such as an AU with randomly accessible or independently encoded pictures in each layer of the video bitstream, or an AU that can be decoded independently of the previous AU in the video bitstream.

[0110] The current specification mandates that an Encoded Video Sequence Start (CVSS) AU has either an IRAP NAL unit type or a GDR NAL unit type at each layer, and that the IRAP NAL unit types are the same within the CVSS AU. Furthermore, the presence of an Access Unit Delimiter (AUD) indicating that the CVSS AU is an IRAP AU or a GDR AU is mandatory.

[0111] Figure 8 shows an example of a multilayer bitstream where IRAPs are not aligned across different layers, i.e., all IRAPs are not aligned across layers, and the CVSS AU identification.

[0112] AU2, AU4, and AU6 have IRAP-type NAL unit types in their two lowest layers, but since not all layers in these AUs have the same IRAP type, we can see that these AUs are not CVSS AUs. It is not necessary to analyze all NAL units in an AU, and to easily identify CVSS AUs, AUD NAL units are used and are easily identified. In other words, AU0 and AU8 will contain AUD with a flag indicating that these AUs are CVSS AUs (IRAP AUs).

[0113] Figure 9 shows an example of the extracted video bitstream, for example, video bitstream 12. However, if the bitstream is extracted using only L0 and L1, new CVSS AUs will be present in the extracted bitstream, as indicated by reference numeral 22*. In other words, after extraction, some AUs (e.g., AU22*) are converted into CVSS AUs.

[0114] Since AU2 and AU6 will be CVSS AUs or IRAP AUs, the bitstream of such AUs that indicate the IRAP AU property must contain the AUD.

[0115] An embodiment according to the fourth aspect includes a device for extracting sub-bitstreams from a multilayer video bitstream, for example, an extractor 30 for extracting sub-bitstreams 12 from a multilayer video bitstream 14 as described with respect to Figure 1. According to the fourth aspect, the multilayer video bitstream 14 represents an encoded video sequence, such as an encoded video sequence 20, and the multilayer video bitstream includes access units 22 of the encoded video sequence. Each of the access units 22 includes one or more bitstream portions 16 of the multilayer video bitstream 14, and each bitstream portion belongs to one of the layers 24 of the multilayer video bitstream. According to the fourth aspect, the extractor 30 is configured to derive one or more OLS from the multilayer video bitstream 14, each representing a (not necessarily appropriate) subset of the layers of the multilayer video bitstream 14. For example, the multilayer video bitstream includes an OLS display 18 which includes information or a description of one or more OLS, such as OLS 181, OLS 182, as shown in Figure 7.

[0116] The extractor 30 according to the fourth embodiment is configured to provide within the subbitstream 12 a layer 24 of a multilayer video bitstream 14 that is indicated by one of a predetermined OLS, i.e., that is indicated to be part of a predetermined OLS. In other words, the extractor 30 can provide within the subbitstream 12 bitstream portions 16 belonging to each layer of a predetermined OLS. For example, a predetermined OLS may be provided to the extractor 30 by external means, for example by an OLS instruction 32 as shown in Figure 1. According to the embodiment of the fourth embodiment, the extractor 30 provides within the subbitstream 12 a sequence start indicator that is the start access unit of a subsequence of an encoded video sequence, for each access unit of the subbitstream 12, such as L0 and L1 as described with respect to Figures 8 and 9, where all bitstream portions of each access unit are bitstream portions of the same type from a predetermined set of bitstream portions.

[0117] In other words, the extractor 30 may provide an access unit that includes or provides to the sub-bitstream 12 exclusively includes a bitstream portion of the same set of predetermined bitstream portion types that have a sequence start indicator.

[0118] For example, a given set of bitstream subtypes may include one or more IRAP NAL unit types and / or GDR NAL unit types. For example, a given set of bitstream subtypes may include NAL unit types IDR_NUT, CRA_NUT, or GDR_NUT.

[0119] For example, the extractor 30 may determine, for each access unit that does not have a sequence start indicator in the multilayer video bitstream, or alternatively, for each access unit of the sub-bitstream 12, whether all bitstream portions of each access unit are the same bitstream portions from a predetermined set of bitstream portion types. For example, the extractor 30 can analyze the relevant information in each access unit or the multilayer video bitstream to determine whether all bitstream portions of each access unit are the same bitstream portions from a predetermined set of bitstream portion types.

[0120] In other words, in one embodiment, the bitstream extraction process removes unnecessary layers if they are not present in the AUs that require AUD after extraction, and adds AUD NUTs for such use.

[0121] According to one embodiment, the extractor 30 is configured to infer from the display in the multilayer video bitstream 14 that, for a given OLS, one of the access units is the starting access unit of a subsequence of the encoded video sequence represented by the given OLS, and to provide a sequence start display in the subbitstream indicating that one of the access units is the starting access unit.

[0122] In other words, in another embodiment, in such an AU (i.e., one that becomes a CVSS AU), there is a representation in the bitstream, and such an AU becomes a CVSS AU or IRAP AU when the layers are extracted, making the insertion (addition) of the AU simpler and requiring less analysis.

[0123] In a further embodiment, the extractor 30 is configured to extract nested information, such as nested SEI, indicating that for a given OLS, one or more access units, for example, access units other than the start access unit of the multilayer video bitstream 14, are start access units for the OLS. In this embodiment, the extractor 30 provides a sequence start instruction in the sub-bitstream indicating that one or more access units indicated in the nested information are start access units. For example, the extractor 30 may provide a sequence start indication for each of the indicated access units as described above. Alternatively, the device may provide a common indication of the indicated access units in the sub-bitstream.

[0124] In other words, in another embodiment, there exists a nesting SEI that can encapsulate an AUD so that when a particular OLS is extracted, such as one that converts an AU to an IRAP / GDR AU, the encapsulated AUD is deencapsulated and added to the bitstream. Currently, the specification only includes nesting of other SEI messages. Therefore, non-VCL non-SEI payloads need to be allowed within nested SEIs. One way to do this is to extend an existing nested SEI and indicate that it contains non-SEIs. Another way is to add a new SEI that contains other non-VCL payloads within the SEI.

[0125] Table 2 shows the first option for implementation. [Table 2-1] [Table 2-2] A nesting SEI will encapsulate another SEI (sei_message()) and a given number of non-VCL NAL units (nesting_num_nonVclNuts_minus1) that are not SEIs. The length (length_minus1) of such encapsulated non-VCL NAL units is written immediately before the nested SEI (nonVclNut) so that the boundary within the nested SEI can be found.

[0126] Table 3 shows another option, option 2. [Table 3] Option 2 allows for the addition of a type that also allows other non-VCL NAL units, so that the nonVclNutPayloadSEI message can be used if it becomes necessary to include other non-VCL NAL units in the nesting SEI in the future.

[0127] In such cases, a new SEI message is defined that directly contains a single non-VCL NAL unit (in this case, the access unit delimiter (AUD_rbsp())), and therefore such an encapsulated SEI can be added directly without modifying the nesting SEI (which already contains other SEIs within itself).

[0128] According to a further embodiment of the fourth aspect, the extractor 30 is configured to provide a sequence start indicator for each of the access units of a subbitstream, indicating that each access unit is the start access unit of the encoded video sequence, if all bitstream portions of each access unit are bitstream portions of the same type from a predetermined set of bitstream portion types, and each access unit includes bitstream portions of two or more layers.

[0129] In another embodiment, AUD is also required for access units that may become IRAP AU or GDR AU in the case of OLS extraction (layer drop), and the extractor can ensure they are present when needed and easily rewrite aud_irap_or_gdr_au_flag to 1 when appropriate (i.e., the AU changes to IRAP or GDR AU after extraction). One way to express this constraint in the specification is to extend existing text that requires AUD for IRAP or GDR AU in the current bitstream.

[0130] An AU can have a maximum of one EOB NAL unit. If vps_max_layers_minus1 is greater than 0, each IRAP or GDR AU is assumed to have only one AUD NAL unit.

[0131] This will be changed as follows:

[0132] An AU can have a maximum of one EOB NAL unit. If vps_max_layers_minus1 is greater than 0, each AU containing at least two layers, each containing only IRAP or GDR NAL units, is assumed to have only one AUD NAL unit.

[0133] In other words, instead of using all NAL units, use the terms IRAP or GDR picture.

[0134] An AU can have a maximum of one EOB NAL unit. If vps_max_layers_minus1 is greater than 0, each AU containing at least two IRAP or GDR pictures is assumed to have exactly one AUD NAL unit.

[0135] Accordingly, further embodiments according to the fourth aspect include an encoder 10 for providing a multilayer video bitstream, for example, the encoder 10 described with respect to Figure 1. Embodiments of the encoder 10 according to the fourth aspect are configured to provide a multilayer video bitstream representing an encoded video sequence, such as video sequence 20. The multilayer video bitstream 14 provided by the encoder 10 includes a sequence of access units 22, each containing one or more pictures, each associated with one of a plurality of layers 24 of the video bitstream. For each access unit of the multilayer video bitstream 14, the encoder 10 is configured to provide a sequence start indicator indicating whether all pictures in each access unit are one of a predetermined set of picture types, given that in the sub-bitstream 12 each access unit contains at least two pictures from one of a predetermined set of picture types. Optionally, the encoder 10 is configured to provide an OLS output layer set (OLS) display 18 in a multilayer video bitstream 14, where the OLS includes one or more layers of the multilayer video bitstream 14, and if each access unit includes at least two pictures from a set of predetermined access unit types, the encoder 10 provides a sequence start indicator for each access unit.

[0136] For example, a picture type may include one or more IRAP picture types and / or GDR picture types. A picture that is a picture type may mean that each of the one or more bitstream portions into which the picture is encoded is one of a given set of bitstream portion types, such as one or more IRAP NAL unit types or GDR NAL units.

[0137] For example, the sequence start indicators mentioned in the overall embodiments of Section 4 may be provided in the form of, or as part of, an access unit delimiter (AUD), which may be a bitstream portion provided within each access unit. For example, an indication that an access unit is a sequence start access unit may be indicated by setting a flag in the AUD, e.g., aud_irap_or_gdr_au_flag, to a value of 1, for example, to indicate that each access unit is a sequence start access unit.

[0138] Alternatively, the encoder 10 may provide a sequence start indicator for each access unit that includes at least two pictures from a predetermined set of picture types, but the encoder 10 may provide a sequence start indicator for each access unit of the multilayer video bitstream 14, where each access unit includes at least two pictures from a predetermined set of picture types, and each picture belongs to one of the layers of the OLS shown in the OLS display provided by the encoder 10 within the multilayer video bitstream 14.

[0139] As an alternative to the above embodiment in which AUD is mandatory for access units that may become IRAP AU or GDR AU during OLS extraction (layer drop), the extractor can verify their presence when needed and easily rewrite aud_irap_or_gdr_au_flag to 1 when appropriate (i.e., the AU changes to IRAP or GDR AU after extraction). The encoder only needs to write an AUD NAL unit if the access unit corresponds to a CVSS AU of at least one OLS. An example of the specification is as follows:

[0140] An AU can have a maximum of one EOB NAL unit. If vps_max_layers_minus1 is greater than 0, then each AU that contains only IRAP or GDR NAL units in all layers of at least one multilayer OLS will have exactly one AUD NAL unit.

[0141] In other words, using the terms IRAP or GDR picture, it would look like this:

[0142] An AU can have a maximum of one EOB NAL unit. If vps_max_layers_minus1 is greater than 0, then each AU containing only IRAP or GDR pictures in all layers of at least one multilayer OLS will have exactly one AUD NAL unit.

[0143] In this embodiment, the relevant OLS extraction process will be extended through the following steps.

[0144] [...] The output subbitstream (OutBitstream) is derived as follows: - The bitstream outBitstream is set to be the same as the bitstream inBitstream. - Remove all NAL units with a TemporalId greater than tIdTarget from the outBitstream. - Remove all NAL units from the outBitstream whose nal_unit_type is not equal to VPS_NUT, DCI_NUT, or EOB_NUT, and whose nuh_layer_id is not included in LayerIdInOls[targetOlsIdx]. - If the AU has two or more layers and contains only NAL units whose nal_unit_type is equal to a single type, such as IDR_NUT, CRA_NUT, or GDR_NUT, rewrite the AU's AUD flag aud_irap_or_gdr_au_flag to 1.

[0145] Therefore, the extractor 30 according to the fourth embodiment can provide a sequence start indicator by setting a value for the sequence start indicator for each access unit 22*. For example, the sequence start indicator may be a syntax element signaled by the access unit of the multilayer video bitstream 14, and the extractor 30 may modify or retain the value of the sequence start indicator when transferring the access unit in the sub-bitstream 12.

[0146] In other words, according to the embodiment, the extractor 30 described above in the fourth aspect may, for each of the access units 22 of the subbitstream, if all bitstream portions of each access unit are bitstream portions of the same type from a predetermined set of bitstream portion types, e.g., access unit 22*, provide a sequence start indicator in the subbitstream 12 when transferring each access unit 22* of the multilayer video bitstream 14 in the subbitstream 12, by setting the value of the sequence start indicator present in each access unit of the multilayer video bitstream (e.g., present in the AUD NAL unit of each access unit 22*), e.g., aud_irap_or_gdr_flag, to a predetermined value, e.g., 1, thereby indicating that each access unit is the starting access unit of the encoded video sequence. The predetermined value is a value that indicates that each access unit is the first access unit of the encoded video sequence. For example, the extractor 30 may change the value of the sequence start indicator to a predetermined value if the multilayer video bitstream 14 does not have the predetermined value.

[0147] Accordingly, an embodiment of the encoder 10 for providing a multilayer video bitstream 14 according to a fourth aspect provides an OLS indicator 18 in the multilayer video bitstream 14 that shows an OLS containing layers (i.e., at least two layers) of the video bitstream 14. For each access unit containing a picture of one of a predetermined picture types (e.g., the same type or not necessarily the same type) for a layer of the OLS, the encoder 10 may provide a sequence start indicator in the multilayer video bitstream 14, the sequence start indicator indicating whether all pictures in the access unit, i.e., those not part of the OLS, are one of the predetermined picture types, for example, by the value of the sequence start indicator.

[0148] In other words, encoder 10 may signal a sequence start indicator for an access unit, for example, access unit 22*, where the picture of the access unit is a picture belonging to one of the layers of the OLS and is of one of a predetermined type.

[0149] 5. Handling of time sublayers in the extraction process of output layer sets Section 5 describes a fifth embodiment of the present invention. The fifth embodiment may optionally follow the embodiments of the encoder 10 and extractor 30 described with respect to Figure 1. Further details described in this section may be optionally implemented in the embodiments described in this section.

[0150] Some embodiments according to the fifth aspect relate to the extraction process of OLS and vps_ptl_max_temporal_id[i][j]. Some embodiments according to the fifth aspect may relate to the derivation of NumSublayerInLayer[i][j].

[0151] To extract the Output Layer Set (OLS), you need to drop or remove any layers that do not belong to the OLS from the bitstream. However, note that layers belonging to the OLS may have different numbers of sublayers (time layer TLx in Figure 10).

[0152] Figure 10 shows an example of a two-layer bitstream, each having a different frame rate. The bitstream in Figure 10 includes an access unit 221 associated with the first time layer TL0 and an access unit 222 associated with the second time layer TL1. The first layer 241 contains pictures of both time sublayers TL0 and TL1, while the second layer 242 contains pictures of TL0 only. Thus, the first layer 241 has twice the frame rate or picture rate of the second layer 242.

[0153] The bitstream in Figure 10 may have two OLSs. One consists only of L0 and has two operating points (e.g., TL0 at 30fps and TL0+TL1 at 60fps). The other OLS may consist of L0 and L1, but may also consist of TL0 alone.

[0154] The current specification allows signaling of OLS profiles and levels using L0's TL0 and TL1, but the extraction process cannot generate a bitstream of TL0 only.

[0155] Currently, NumSublayerInLayer[i][j], which represents the largest sublayer contained in the i-th OLS of layer j, is set to vps_max_sublayers_minus1+1 if vps_max_tid_il_ref_pics_plus1[m][k] does not exist, or if layer j is the output layer of the i-th OLS.

[0156] Figure 11 shows an encoder 10 and extractor 30 according to an embodiment of the fifth aspect. The encoder 10 and extractor 30 according to Figure 11 may optionally correspond to the encoder 10 and extractor 30 described with respect to Figure 1. Thus, the description of Figure 1 may optionally apply to the elements shown in Figure 11. The encoder 10 according to Figure 11 is configured to encode an encoded video sequence 20 into a multilayer video bitstream 14. The multilayer video bitstream 14 includes access units 22, for example, access units 221, 222, and 223 in Figure 11 (or also in Figure 1), each of which constitutes one or more pictures 26 of the encoded video sequence. For example, each of the access units 22 contains one or more pictures related to one common moment or frame of the encoded video sequence, as described with respect to Figure 1. Each of the pictures 26 belongs to one of the layers 24 of the multilayer video bitstream. For example, in the exemplary example of Figure 11, the multilayer video bitstream 14 includes a first layer 241 and a second layer 242, where the first layer includes a picture 261 and the second layer includes a picture 262. According to a fifth embodiment, each of the access units 22 belongs to a time sublayer of the set of time sublayers TL0, TL1 of the encoded video sequence 20. For example, access units 221 and 223 in Figure 11 may belong to the first time sublayer TL0, and access unit 222 may belong to the second time sublayer TL1, as described with respect to Figure 10, for example. Time sublayers are sometimes called time subsets or time layers. For example, each time sublayer may be indicated by or indexed by a time identifier and characterized by, for example, a time relationship with respect to the frame rate and / or other time subsets. For example, each of the access units may include or be associated with a time identifier that associates each access unit with one of the time sublayers.

[0157] According to a fifth aspect, the encoder 10 is configured to provide a syntax element to the multilayer video bitstream 14, for example, max_tid_within_ols as described above, the syntax element indicating a predetermined time sublayer for the OLS, the OLS containing or indicating a (not necessarily appropriate) subset of the layers of the multilayer video bitstream 14. The syntax element indicates a predetermined time sublayer for the OLS in such a way that it identifies between different states, including a state in which the predetermined time sublayer is less than the maximum time sublayer in an access unit which is at least one picture of the subset of layers. For example, the predetermined time sublayer is the maximum time sublayer included in the OLS.

[0158] For example, the encoder 10 may provide an OLS representation 18 within the multilayer video data stream 14, as described with respect to Figure 1, for example. The OLS representation 18 may include a description or representation of one or more OLSs, for example, the OLS 181 shown in Figure 11. Each OLS may be associated with a set of layers in the multilayer video data stream 14. In the example shown in Figure 11, the OLS 181 is associated with the first layer 241 and the second layer 242. Note that the multilayer video bitstream 14 may optionally include further layers.

[0159] For example, in the example shown in Figure 11, OLS181 may include a first time sublayer (to which access units 221 and 223 may belong), but a second time sublayer to which access unit 222 may belong is not included in the OLS in this example. Therefore, according to this example, TL0 may be the largest time sublayer included in the OLS. Note that time sublayers may have a hierarchical order. In the example in Figure 11, among the time sublayers in access unit 22, at least one picture of a subset of layers (layer 241, which is part of the OLS), for example, picture 261 in the case of access unit 221, is the largest, which is TL1. Therefore, the largest time sublayer included in the OLS is below the largest time sublayer TL1. Thus, a given time sublayer may be indicated, for example, by indicating that it is below the largest time sublayer present in an access unit that is at least partially included in the OLS. In further examples, a given time sublayer may be indicated by indicating an index that identifies the given time sublayer.

[0160] Therefore, in the OLS181 example, the given time sublayer may be the first time sublayer. The syntax element provided to the multilayer video bitstream 14, for example max_tid_within_ols or vps_ptl_max_temporal_id, indicates the given time sublayer for the OLS. The syntax element distinguishes different states. According to one of the states, the given time sublayer is below the largest time sublayer in the access unit that contains at least one picture from the subset of layers. For example, in Figure 11, the OLS181 example described above includes layers 241 and 242. The largest time sublayer in the access unit of the subset of layers of OLS181 is the second time sublayer to which access unit 222 belongs. In an example where the second time sublayer does not belong to OLS181, the syntax element may indicate this state.

[0161] The extractor 30 according to the fifth embodiment can derive syntax elements from the multilayer video bitstream 14 and provide a subbitstream 12 by selectively transferring the pictures of the multilayer video bitstream 14 in the subbitstream 12 if each picture belongs to one of the layers of the OLS 181 and the picture belongs to an access unit 221, 223 that is equal to or belongs to a predetermined time sublayer.

[0162] In other words, the extractor 30 can provide the bitstream portion of each picture 26 in the subbitstream 12 if the picture is equal to or belongs to a predetermined time sublayer, or a time sublayer below it; otherwise, it can drop the picture, i.e., not transfer it.

[0163] In other words, the extractor 30 can use syntax elements in the construction of the subbitstream 12 to exclude pictures belonging to time sublayers that are not included in the OLS to be decoded but are included in any of the layers indicated by the OLS from being transferred in the subbitstream 12.

[0164] According to one embodiment, the multilayer video bitstream 14 indicates a syntax element for each layer of the OLS, which represents a predetermined time sublayer included in each OLS, for example, the maximum time sublayer. Based on the syntax elements for the layers of the OLS, the extractor 30 identifies whether a bitstream portion belonging to a time sublayer of the OLS belongs to a time sublayer not belonging to the OLS, and may consider the bitstream portion belonging to a time sublayer of the OLS as a target for transfer to the subbitstream 12.

[0165] For example, a syntax element may be part of OLS display 18, or it may be part of OLS 181 that it references. For example, a syntax element may be part of the video parameter set for each OLS.

[0166] In one embodiment, signaling (e.g., for the maximum time sublayer) is provided to the bitstream to indicate that the OLS has a maximum sublayer different from vps_max_sublayers_minus1+1, or is the largest of all layers present in the OLS. For this purpose, an existing syntax element vps_ptl_max_temporal_id[i][j] may be reused to indicate the maximum sublayer present in the OLS.

[0167] According to some embodiments, NumSublayerInLayer[i][j], which represents the largest sublayer included in the i-th OLS for layer j, is further changed to vps_ptl_max_temporal_id[i][j] if vps_max_tid_il_ref_pics_plus1[m][k] does not exist or layer j is the output layer of the i-th OLS.

[0168] Alternatively, you could add a new syntax element, such as max_tid_within_ols, to indicate the largest sublayer within the OLS.

[0169] According to the embodiment, the encoder 10 and / or extractor 30 are configured to derive decoder function-related parameters for each picture of the multilayer video bitstream 14, for a substream, e.g., a subbitstream 12, obtained by selectively taking over each picture, if each picture belongs to one of the layers of the OLS 181, and if the picture is equal to or belongs to an access unit below a predetermined time sublayer. In other words, the encoder 10 and / or extractor 30 can derive decoder function-related parameters for a subbitstream that exclusively contains pictures belonging to time sublayers belonging to the OLS describing the subbitstream. The encoder 30 or extractor 30 can signal the function-related parameters in the subbitstream 12. Thus, the encoder 10 can signal function-related parameters in the multilayer video bitstream 14. For example, decoder function-related parameters may include parameters such as those described in Section 6.

[0170] 6. Handling of Time Sublayers in Video Parameter Signaling Section 6 describes a sixth embodiment of the present invention with reference to Figures 11 and 11. Therefore, Figures 1 and 11 can be optionally applied to the sixth embodiment. Further details regarding the embodiments described in this section can be optionally implemented in the embodiments described here.

[0171] Some examples of the sixth aspect relate to constraints on vps_ptl_max_temporal_id[i], vps_dpb_max_temporal_id[i], and vps_hrd_max_tid[i], as are consistent for a given OLS.

[0172] The multilayer video bitstream 14 and / or sub-bitstream 12 described with respect to Figure 1 and / or Figure 11 may optionally include a video parameter set 81. The video parameter set 81 may include one or more decoder requirement sets, e.g., profile-tier-level-sets (PTL sets), and / or one or more buffer requirement sets, e.g., DPB parameter sets, and / or one or more bitstream adaptation sets, e.g., hypothetical reference decoder (HRD) parameter sets. For example, each of the OLS shown in the OLS representation 18 may be associated with each of the decoder requirement sets, buffer requirement sets, and bitstream adaptation sets to which the bitstream described by each OLS applies. The video parameter set may indicate, for each of the decoder requirement sets, buffer requirement sets, and bitstream adaptation sets, the maximum time sublayer referenced by each set, i.e., the maximum time sublayer of the video bitstream or video sequence referenced by each set.

[0173] Figure 12 shows an example of a video parameter set 81, which includes a first decoder requirements set 821, a first buffer requirements set 841, and a first bitstream adaptation set 861, related to the first OLS1 of the OLS display 18. Furthermore, according to Figure 12, the video parameter set 81 includes a second decoder requirements set 822, a second buffer requirements set 842, and a second bitstream adaptation set 862, related to the second output layer set OLS2.

[0174] For example, each OLS described by the OLS representation 18 may be associated with one of the decoder requirement set 82, buffer requirement set 84, and bitstream compatibility set 86 by associating each OLS with an index pointing to the decoder requirement set, buffer requirement set, and bitstream compatibility set, respectively. According to an embodiment of the sixth aspect, the multilayer video bitstream includes access units, each of which belongs to one of the sets of time sublayers of the encoded video sequence encoded in the multilayer video bitstream 14. The multilayer video bitstream 14 according to the sixth aspect further comprises a video parameter set 81 and the OLS representation 18. For each of the bitstream compatibility set 86, buffer requirement set 84, and decoder requirement set 82, the time subset representation indicates a constraint on the maximum time sublayer, e.g., the maximum time sublayer referenced by each bitstream compatibility set / buffer requirement set / decoder requirement set. For example, each of the bitstream compatibility set 86, buffer requirements set 84, and decoder requirements set 82 signals a syntax element that indicates their respective time subset representations (e.g., vps_ptl_max_temporal_id for the PTL set, vps_dpb_max_temporal_id for the DPB parameter set, and vps_hrd_max_tid for the bitstream compatibility set).

[0175] As shown in Figure 12, each of the bitstream adaptation set 86, buffer requirements set 84, and decoder requirements set 82 may include one or more sets of parameters for each time sublayer present in the set of layers containing the bitstream portion of the video bitstream referenced by the respective bitstream adaptation set 86, buffer requirements set 84, or decoder requirements set 82. For example, in Figure 12, OLS1 includes layer L0 containing the bitstream portion of time layer TL0, and OLS2 includes layers L0 and L1 containing the bitstream portions of time layers TL0 and TL1. The bitstream adaptation set 862 and decoder requirements set 822 associated with OLS2 include parameters for L0, and therefore, according to this example, parameters for time layer TL0, and further include parameters for L1, and therefore parameters for time layer TL1. The buffer requirements set 842 includes sets of parameters DPB0 for time sublayer TL0 and DPB1 for time sublayer TL1.

[0176] Typically, a VPS has three syntax structures, which are generally defined and then mapped to a specific OLS. • Profile hierarchy level (PTL), for example, one or more decoder requirement sets • DPB parameters, e.g., one or more buffer requirement sets • HRD parameters, e.g., one or more bitstream matching sets The mapping from PTL to OLS is performed for all OLS (single-layer or multi-layer) VPSs. However, the mapping of DPB and HRD parameters to OLS is performed only for VPSs of OLS with multiple layers. As shown in Figure 12, the PTL, DPB, and HRD parameters are first written to the VPS, and then mapped to indicate which parameters the OLS will use.

[0177] In the example shown in Figure 12, there are two OLSs and two parameters for each of them. However, the definitions and mappings are specified so that multiple OLSs can share the same parameters, so it is not necessary to repeat the same information multiple times, as shown in Figure 13, for example.

[0178] Figure 13 shows an example where OLS2 and OLS3 have the same PTL and DBP parameters but different HRD parameters.

[0179] In the examples in Figures 12 and 13, the values ​​of vps_ptl_max_temporal_id[i], vps_dpb_max_temporal_id[i], and vps_hrd_max_tid[i] for a given OLS are matched (TL0 for OLS1, TL1 for OLS2, and TL1 for OLS3), but this is no longer necessary. These three values ​​associated with the same OLS are no longer restricted to having the same value. Currently, there are no constraints on any of these values ​​to match the number of sublayers in the bitstream. For example, in the above examples, the values ​​are defined for two sublayers, but the bitstream could have a single sublayer for OLS2 and OLS3. Therefore, as matching becomes more complex, the decoder will no longer be able to easily find what the features of the bitstream are.

[0180] In the first embodiment, the bitstream signals the maximum number of sublayers present in the OLS (not necessarily the bitstream, as some may be dropped), but can be understood as at least an upper limit, i.e., no more sublayers can exist in the OLS within the bitstream than the signal value, e.g., vps_ptl_max_temporal_id[i]. Therefore, the DPB and HRD parameters are also used by the decoder.

[0181] If the values ​​of vps_dpb_max_temporal_id[i] and vps_hrd_max_tid[i] differ from vps_ptl_max_temporal_id[i], the decoder needs to perform a more complex mapping. Therefore, in one embodiment, when the OLS indexes the PTL structure, DPB structure, and HRD parameter structure by vps_ptl_max_temporal_id[i], vps_dpb_max_temporal_id[i], and vps_hrd_max_tid[i], respectively, there is a bitstream constraint that vps_dpb_max_temporal_id[i] and vps_hrd_max_tid[i] must be equal to vps_ptl_max_temporal_id[i].

[0182] According to the embodiment, the encoder 10, for example, the encoder 10 in Figure 1 or Figure 11, is configured to form an OLS display 18 such that the maximum time sublayers indicated by the bitstream adaptation set 86, buffer requirements set 84, and decoder requirements set 82 related to the OLS are equal to each other, and the parameters in the bitstream adaptation set 86, buffer requirements set 84, and decoder requirements set 82 are fully valid for the OLS.

[0183] However, as seen in the example in Figure 12, the parameters of OLS1 with a single sublayer (TL0), namely level 0 (PTL0's L0(TL0)), DPB parameter 0 (DPB0's DPB0) and HRD parameter 0 (HRD0's HRD0), are also described in PTL1 822, DPB1 842, and HRD1 862. One option to avoid repeating many parameters is to obtain the values ​​of OLS1 from DPB and HRD parameters that include more sublayers, excluding DPB0841 and HRD0861. An example of this is shown in Figure 14.

[0184] Figure 14 shows the definitions of PTL, DPB, and HRD, and an example of sharing between different OLSs that have sublayer information unrelated to some OLSs. Since PTL0 indicates that there is only one sublayer, only the parameters of TL0 for DPB1 and HRD1 are used.

[0185] Therefore, in another embodiment, if OLS indexes the PTL structure, DPB structure, and HRD parameter structure by vps_ptl_max_temporal_id[i], vps_dpb_max_temporal_id[i], and vps_hrd_max_tid[i] respectively, there is a bitstream constraint that vps_dpb_max_temporal_id[i] and vps_hrd_max_tid[i] must be greater than or equal to vps_ptl_max_temporal_id[i], and that large values ​​corresponding to higher sublayers of the DPB and HRD parameters are ignored by OLS.

[0186] Accordingly, according to a further embodiment, the encoder 10 is configured to form an OLS representation 18 and / or video parameter set 81 (or generally, a multi-layer video bitstream 14) that is valid only for OLS, insofar as the maximum time sublayer indicated by the decoder requirements set 82 associated with the OLS is less than or equal to the maximum time sublayer indicated by the buffer requirements set 84 and bitstream adaptation set 86, respectively, and the parameters in the buffer requirements set 84 and bitstream adaptation set 86 are the same with respect to the same or lower time layers as the maximum time sublayer indicated by the decoder requirements set 82 associated with the OLS.

[0187] In other words, the encoder 10 can provide an OLS display 18 and / or video parameter set 81 such that the maximum time sublayer indicated by the OLS-related buffer requirements set 84 is greater than or equal to the maximum time sublayer indicated by the OLS-related decoder requirements set 82, and the maximum time sublayer indicated by the OLS-related bitstream adaptation set 86 is greater than or equal to the maximum time sublayer indicated by the OLS-related decoder requirements set 82.

[0188] For example, Figure 14 shows an example of a video parameter set 81 that includes a first decoder requirements set 821 for a first set of layers, such as layer L0 which contains the access units for the first time sublayer TL0. The video parameter set 81 further includes a second decoder requirements set 822 which refers to a second set of layers, the second set of layers including layers L0 and L1, and the second set of layers including the access units for the first and second time sublayers, i.e., TL0 and TL1. Thus, the largest time sublayer of the second set of layers is the second time sublayer TL1. The video parameter set 81 further includes a DPB parameter set 842 which refers to a second set of layers, a bitstream adaptation set 862 which refers to a second set of layers, and a bitstream adaptation set 863 which refers to a set of layers including a third layer L2 which has the access units for the first and second time sublayers. The first OLS, OLS1, is associated with a first set of layers, and its decoder requirements are described by a first set of decoder requirements 821, which indicates that the largest time sublayer of the first set of layers is the first time sublayer. The first time sublayer is less than or equal to the largest time sublayer indicated by the DPB parameter set 842 associated with OLS1 and the bitstream adaptation set 862 associated with OLS1 (note that time sublayers are hierarchically ordered), so the DPB parameter set 842 and the bitstream adaptation set 862 contain information about the first set of layers. Thus, the DPB parameter set 842 and the bitstream adaptation set 862 are valid for OLS1 insofar as they relate to the first set of layers, and their access units belong to the first time sublayer. For example, as explained with respect to Figure 12, and also shown in Figures 13 and 14, the decoder requirements set 822, the buffer requirements set 842, and the bitstream adaptation set 862 each contain a set of parameters for each time sublayer included in the bitstream they refer to.According to this embodiment, the parameters relating to the first time sublayer TL0 are valid for OLS1 because the first time sublayer is less than or equal to the maximum time sublayer indicated by the decoder requirement set 822.

[0189] In other words, the decoder 50 may, for the OLS to be decoded, use (for example) only the parameters of the decoder requirements set 82, buffer requirements set 84, and bitstream adaptation set 86 associated with the OLS that are the same as or less than the maximum time sublayer associated with the decoder requirements set 82 for the OLS.

[0190] Figure 15 shows another example of the video parameter set 81 and OLS display 18. Figure 15 shows alternatives that may occur when the parameters of TL1 and TL0 are the same, or when the value of TL1 is the maximum allowable value for a certain level. In such cases, instead of not including DPB0 and HRD0 in the VPS as previously shown, both DPB1 and HRD1 can be included but not included. The value of the upper sublayer of OLS1 can then be derived as equal to the value signaled in TL0 or the maximum allowable value for the level. Thus, Figure 15 illustrates the definitions of PTL, DPB, and HRD and their sharing between different OLSs, illustrating sublayer information that needs to be inferred if it does not exist for a given OLS.

[0191] Therefore, in another embodiment, there are no bitstream constraints on the values ​​vps_ptl_max_temporal_id[i], vps_dpb_max_temporal_id[i], and vps_hrd_max_tid[i], but for values ​​i > vps_dpb_max_temporal_id[i] that are greater than vps_ptl_max_temporal_id[i], the vps_hrd_max_tid[i] DPB and HRD parameters up to vps_ptl_max_temporal_id[i] are assumed to be either the maximum value defined by the profile lever or equal to the highest signal DPB and HRD parameters.

[0192] Therefore, according to another embodiment, the encoder 10 is configured to form an OLS display and / or video parameter set 81 such that the maximum time sublayer indicated by the decoder requirement set 82 associated with the OLS is greater than or equal to the maximum time sublayer indicated by each of the buffer requirement set 84 and bitstream adaptation set 86 associated with the OLS. According to these embodiments, parameters relating to time sublayers greater than or equal to the maximum time sublayer indicated by each of the buffer requirement set 84 and bitstream adaptation set 86, which are missing in the buffer requirement set 84 and bitstream adaptation set 86 associated with the OLS, for example OLS2 in Figure 15, will be set to equal to a fourth parameter or to equal to a parameter in the buffer requirement set 84 and bitstream adaptation set 86 associated with the OLS relating to the maximum time sublayer indicated by each of the buffer requirement set 84 and bitstream adaptation set 86.

[0193] Therefore, an embodiment of a decoder that decodes a multilayer video bitstream, such as the decoder 50 in Figure 1, may be configured such that, when the maximum time sublayer indicated by the decoder requirements set 82 related to the OLS is greater than or equal to the maximum time sublayer indicated by the buffer requirements set 84 and bitstream adaptation set 86, i.e., the OLS being decoded, the OLS-related parameters of the buffer requirements set 84 and bitstream adaptation set 86 are set to equal to default values, such as the maximum value of each parameter indicated in the decoder requirements set 82, for each parameter of the buffer requirements set 84 and bitstream adaptation set 86, or to estimate the value of each parameter in the buffer requirements set 84 or bitstream adaptation set 86 related to the OLS, in relation to the maximum time sublayer indicated by the buffer requirements set 84 and bitstream adaptation set 86, respectively. For example, the choice of whether to use default values ​​or to use the values ​​of each parameter in the buffer requirements set or bitstream adaptation set related to the maximum time sublayer indicated by the buffer requirements set and bitstream adaptation set, respectively, may be made differently for each parameter of the buffer requirements set 84 and bitstream adaptation set 86.

[0194] 7. Selecting the output layer in a domain of interest application Section 7 describes an embodiment according to the seventh aspect with reference to Figure 1. Therefore, the description in Figure 1 can be optionally applied to the embodiment according to the seventh aspect. Furthermore, details describing further aspects can be optionally implemented in the embodiments described in this section.

[0195] Some embodiments according to the seventh aspect relate to the derivation of PicOutputFlag in RoI applications.

[0196] When multi-layer bitstreams such as video bitstream 14 are used, and the picture on the specified output layer is unavailable to the decoder (e.g., due to bitstream errors or transmission loss), a suboptimal user experience may result if certain considerations are not followed. Typically, if the access unit does not include a picture on the output layer, it is possible, depending on the implementation, to select and output a picture from a non-output layer to compensate for the error or loss, as is evident from the annotations in the derivation of the variable PicOutputFlag below.

[0197] - The current picture variable PictureOutputFlag is derived as follows: - PictureOutputFlag is set to 0 if sps_video_parameter_set_id is greater than 0 and the current layer is not an output layer (i.e., nuh_layer_id is not equal to OutputLayerIdInOls[TargetOlsIdx][i] for any value i in the range of 0 to NumOutputLayersInOls[TargetOlsIdx]-1), or if any of the following conditions are true: - The current picture is a RASL picture, and the NoOutputBeforeRecoveryFlag of the associated IRAP picture is equal to 1. - The current picture is either a GDR picture with NoOutputBeforeRecoveryFlag equal to 1, or a recovery picture of a GDR picture with NoOutputBeforeRecoveryFlag equal to 1. - Otherwise, PictureOutputFlag will be set to the same value as ph_pic_output_flag.

[0198] Note - In implementation, the decoder may output pictures that do not belong to the output layer. For example, if there is only one output layer, and in an AU, the output layer's pictures are unavailable, for example, due to loss or layer-down switching, the decoder may set PictureOutputFlag to 1 for the picture with the highest nuh_layer_id value and ph_pic_output_flag equal to 1 among all the pictures available to the decoder in the AU, and set PictureOutputFlag to 0 for all other pictures available to the decoder in the AU.

[0199] However, when the bitstream is created for a region of interest (RoI) application, that is, when the upper layers depict only a subset of the lower layer pictures (by using a scaling window), the switching between overview and detail views becomes very fast, making it undesirable to switch between layers of the decoder output in short frames. Therefore, as part of the present invention, in one embodiment, when a scaling window that does not cover the entire picture plane is used, the decoder implementation is not allowed to freely select the output layer, as follows:

[0200] In one implementation, the decoder can output pictures that do not belong to the output layer, as long as the scaling window covers the entire picture plane. For example, if there is only one output layer, and in an AU, the output layer's pictures are unavailable, for example, due to loss or layer-down switching, the decoder can set PictureOutputFlag to 1 for the picture with the highest nuh_layer_id value and ph_pic_output_flag equal to 1 among all the pictures available to the decoder in the AU, and set PictureOutputFlag to 0 for all other pictures available to the decoder in the AU.

[0201] According to an embodiment of the seventh aspect, a decoder 50 for decoding a multilayer video bitstream, e.g., a multilayer video bitstream 14 or a sub-bitstream 12, is configured to use vector-based inter-layer predictions from a reference picture 261 of a second layer 241 to a predicted picture 262 of a first layer 242, where the prediction vectors are scaled and offset according to the relative size and relative position of the predicted and reference pictures defined in the multilayer video bitstream 14. For example, picture 262 of layer 242 in Figure 1 can be encoded into the multilayer video datastream 14 using inter-layer predictions from picture 261 of layer 241, e.g., picture 261 of the same access unit 221. According to the seventh aspect, the multilayer video bitstream 14 may include an OLS representation 18 of an OLS showing a subset of the layers of the multilayer video bitstream 14, the OLS including one or more output layers including a first layer 241 and one or more non-output layers including a second layer.

[0202] If a predetermined picture 262 in the first layer 242 of the OLS, for example picture 262, is lost, the decoder 50 according to the seventh embodiment is configured to replace the predetermined picture with a further predetermined picture in the second layer 241 of the OLS, which is in the same access unit 22 as the predetermined picture, if the scaling window defined for the predetermined picture coincides with the picture boundary of the predetermined picture 262, and the scaling window defined for a further predetermined picture coincides with the picture boundary of a further predetermined picture. If at least one of the scaling pictures defined for the predetermined picture does not coincide with the picture boundary of the predetermined picture, and the scaling window defined for the predetermined picture does not coincide with the picture boundary of a further predetermined picture, the decoder 50 is configured to replace the predetermined picture by other means or not replace it at all.

[0203] 8. Further Embodiments In the previous section, several embodiments were described as features in the context of the apparatus, but it is clear that such descriptions may also be considered descriptions of corresponding features of the method. While several embodiments were described as features in the context of the method, it is also clear that such descriptions may also be considered descriptions of corresponding features relating to the function of the apparatus.

[0204] Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0205] The encoded image signal of the invention may be stored in a digital storage medium, or it may be transmitted over a transmission medium such as a wireless transmission medium such as the Internet or a wired transmission medium.

[0206] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software or at least partially in hardware or at least partially in software. This implementation may be carried out using a digital storage medium such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory having electronically readable control signals, where these control signals may be stored and, in cooperation with (or capable of cooperating with) a programmable computer system, the respective methods may be performed. Thus, the digital storage medium may be computer-readable.

[0207] In some embodiments of the present invention, one of the methods described herein is performed by including a data carrier having electronically readable control signals, wherein these control signals can cooperate with a programmable computer system.

[0208] In general, embodiments of the present invention may be implemented as a computer program product having program code, which is operable to perform one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, in a machine-readable carrier.

[0209] Other embodiments include a computer program that performs one of the methods described herein and is stored in a machine-readable carrier.

[0210] In other words, one embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

[0211] Further embodiments of the methods of the present invention are, therefore, a data carrier (or digital storage medium, or computer-readable medium), which includes a computer program recorded thereon for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-temporary.

[0212] A further embodiment of the method of the present invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. The data stream or signal sequence may be configured to be transmitted over a data communication connection, such as over the Internet.

[0213] Further embodiments include, for example, processing means such as a computer or programmable logic device configured or adapted to perform one of the methods described herein.

[0214] Further embodiments include a computer on which a computer program for performing one of the methods described herein is installed.

[0215] Further embodiments of the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0216] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the method herein. In some embodiments, a field-programmable gate array may cooperate with a microprocessor to perform one of the methods herein. Generally, the method is preferably performed by any hardware device.

[0217] The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0218] The methods described herein may be performed using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0219] As can be seen in the detailed description above, various features are grouped into examples for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting the intention that the examples described in the claims require more features than are explicitly stated in each claim. Rather, as reflected in the following claims, the subject matter may consist of fewer features than all of the single disclosed examples combined. Therefore, the following claims are incorporated within the detailed description herein, and each claim can stand alone as a separate example. While each claim can stand alone as a separate example, dependent claims may refer in a particular combination with one or more other claims, but it should be noted that other examples may also include combinations of a dependent claim with the subject matter of each other dependent claim, or combinations of each feature with other dependent or independent claims. Such combinations are proposed herein unless otherwise stated that a particular combination is not intended. Furthermore, even if a claim is not directly dependent on an independent claim, it is intended to include features of the claim relative to other independent claims.

[0220] The embodiments described above are merely illustrative of the principles of this disclosure. It will be understood that modifications and variations of the configurations and details described herein will be obvious to those skilled in the art. Accordingly, it is intended that the invention be limited only by the pending claims and not by the specific details presented as part of the description and explanation of the embodiments herein.

Claims

1. A video decoder, Equipped with a processor, The aforementioned processor, Obtain a bitstream containing multiple layers and access units, Decode an instruction indicating that the bitstream includes multiple output layer sets (OLS), and each of the multiple OLS has a unique number of layers. If it is determined that external information for identifying the OLS to be decoded from among the aforementioned multiple OLS is unavailable, In response to the determination, the first access unit among the access units is configured to select the OLS having the largest number of layers among the multiple OLS as the target for decoding. Video decoder.

2. The processor is further configured to decode the selected OLS. The video decoder according to claim 1.

3. The processor, in order to select the OLS, selects from among the plurality of OLS, Having the largest number of layers among the aforementioned multiple layers, and Having the smallest OLS index value, It is further configured to select OLS as the target for decoding. The video decoder according to claim 1.

4. The processor selects the OLS to be decoded, From among the multiple OLSs, identify the set of OLSs that has the largest number of layers among the multiple layers. From the aforementioned set, identify the OLS having the smallest OLS index value. The system is configured to select the OLS having the smallest identified OLS index value as the OLS to be decoded. The video decoder according to claim 3.

5. The aforementioned external information is external information that is not included in the bitstream. The video decoder according to claim 1.

6. A video decoding method, A step of obtaining a bitstream that includes multiple layers and access units, A step of decoding an instruction indicating that the bitstream includes a plurality of output layer sets (OLS), wherein each of the plurality of OLS has a unique number of the plurality of layers, The steps include determining that external information for identifying the OLS to be decoded from among the multiple OLS is unavailable, and In response to the determination, the first access unit among the access units selects the OLS having the largest number of layers among the multiple OLS as the target for decoding. including, Video decoding methods.

7. The step further includes decoding the selected OLS, The video decoding method according to claim 6.

8. When selecting the OLS, from among the multiple OLS, Having the largest number of layers among the aforementioned multiple layers, and Having the smallest OLS index value, The step further includes selecting the OLS to be decoded, The video decoding method according to claim 6.

9. The step of selecting the aforementioned OLS as the target for decoding is: The steps include identifying the set of OLS having the largest number of layers among the multiple OLS, From the aforementioned set, the step of identifying the OLS having the smallest OLS index value, The steps include selecting the OLS having the smallest OLS index value identified as the OLS to be decoded, including, The video decoding method according to claim 8.

10. The aforementioned external information is external information that is not included in the bitstream. The video decoding method according to claim 6.

11. When executed by at least one processor of an electronic device, it stores instructions that cause the method according to any one of claims 6 to 10 to be performed. Computer-readable storage medium.

12. When executed on a computer or signal processor, it causes the method described in any one of claims 6 to 10 to be performed. Computer program.

Images