Network entity for processing data streams

Abstract

A network entity for processing a data stream into which media content is encoded, the data stream comprising packets, each packet comprising a packet type identifier identifying a packet type associated with the respective packet from a plurality of packet types, wherein each packet having a packet type associated therewith from a first set of packet types from the plurality of packet types comprises an operation point identifier identifying an operation point associated with the respective packet from a plurality of operation points within a scalability space spanned by n scalability axes, wherein each packet having a packet type associated therewith from a second set of packet types from the first set of packet types additionally carries data. The network entity is configured to read a scalability axes descriptor from packets having a predetermined packet type associated therewith that is disjoint from the second set and to interpret the operation point identifier in accordance with the scalability axes descriptor.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 201910876391.6, the contents of which are incorporated herein in their entirety. TECHNICAL FIELD

[0002] The present application relates to scalable data streams such as scalable video streams, and network entities such as decoders or MANEs (Media Aware Network Elements) that process such scalable data streams. BACKGROUND

[0003] From a transmission point of view, there is a strong need to adapt video bitstreams in time or other dimensions, as identified and addressed within the standardization of H.264 / AVC. This understanding is reflected by the design decision to signal many important but rather constant parameters outside the video bitstream (in so-called parameter sets). The Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions of H.264 / AVC allow adaptation beyond the time dimension, but the lack of extensibility in the base specification of H.264 / AVC led to a complex approach to extend the H.264 / AVC high-level syntax in a backwards-compatible way. In H.264 / AVC and its extensions, the signaling of operation points along the scalability dimension is done by NAL unit header extensions, which include additional bytes for this purpose. The same information is provided by so-called prefix NAL units of NAL units, which contain pure H.264 / AVC video data and constitute the so-called base layer. The mechanism to extend the information provided via the Sequence Parameter Set (SPS) of the base layer (which is coded by the extensions of H.264 / AVC) for the enhancement layers is established via so-called Subset Sequence Parameter Sets (SSPS).

[0004] While the development of the HEVC base specification is still ongoing, efforts towards a 3D video coding extension have been made to first ensure extensible syntax in the base specification. The so-called scalability hooks in the base specification need to be carefully designed to be future-proof. The following paragraphs give an overview of the current status of the HEVC high-level (HL) syntax and the concepts currently discussed.

[0005] The current status of the HEVC standardization is as follows:

[0006] During the current HEVC base specification and 3D extension, many participants have proposed how to continue from the HL syntax specified in H.264 / AVC. The results are reflected in the current working draft of the specification and in many contributions by individual participants. The following paragraphs give an overview of the current discussions.

[0007] As mentioned above, the operation points within the scalability dimension in an SVC or MVC video bitstream require (specific) extensions of the H.264 / AVC NAL unit header. This is considered a suboptimal solution, which leads to additional effort (e.g., to parse multiple different NAL unit header structures) and requires prefix NAL units to signal the base layer. Therefore, efforts were made to ensure that the base HEVC NAL unit header syntax is generic enough to meet the needs of future extensions of the base specification.

[0008] In the syntax of the NAL unit as in the current working draft. The current consensus is to use a two-byte NAL unit header. In the first byte, one bit is used to signal nal_ref_flag, instead of the two bits of nal_ref_idc in H.264 / AVC, because this HL feature is not widely used in applications yet. The syntax element nal_unit_type thus has one more bit to signal the type of the NAL unit, which allows for a total of 64 distinguishable types.

[0009] The second byte of the NAL unit header is split into two parts, where 1C bits are used to signal the temporal_id of the NAL unit, because temporal scalability has been enabled in the base specification. The remaining 5 bits of the second byte are reserved to be equal to one in a conforming HEVC bitstream. The current understanding of the use of the remaining 5 bits is that they can be used to signal a scalability identifier such as the layer_id syntax element in future extensions.

[0010] While the picture parameter set (PPS) and sequence parameter set as defined in the current HEVC base specification are relatively similar to the parameter sets already specified in H.264 / AVC previously, two new parameter sets, called adaptation parameter set (APS) and video parameter set (VPS), have been introduced to HEVC, where only the VPS is relevant to the content of this document.

[0011] The video parameter set shall signal parameters such as the number of (e.g., temporal) levels / layers present in the video bitstream and the profile and level of all operation points inside. Other parameters to be signaled include the dependencies between scalable layers, as signaled in the SVC scalability information SEI message.

[0012] The following presents a brief overview of the semantics of the NAL unit and video parameter set syntax.

[0013] profile_idc and level_idc indicate the profile and level to which the coded video sequence conforms.

[0014] max_temporal_layers_minus1 + 1 specifies the maximum number of temporal layers present in the sequence. The value of max_temporal_layers_minus1 shall be in the range of 0 to 7, inclusive. more_rbsp_data( ) is specified as follows.

[0015] - If there is no more data in the RBSP, the return value of more_rbsp_data( ) is equal to FALSE.

[0016] - Otherwise, search the RBSP data for the last (least significant, rightmost) bit equal to one that is present in the RBSP. Taking into account the position of this bit, which is the first bit of the rbsp_trailing_bits( ) syntax structure (rbsp_stop_one_bit), the following applies.

[0017] - If there is more data in the RBSP before the rbsp_trailing_bits( ) syntax structure, the return value of more_rbsp_data( ) is equal to TRUE.

[0018] - Otherwise, the return value of more_rbsp_data( ) is equal to FALSE.

[0019] nal_ref_flag equal to one specifies that the content of the NAL unit contains a sequence parameter set, a picture parameter set, an adaptation parameter set, or a slice of a reference picture.

[0020] For coded video sequences conforming to one or more of the profiles specified in Annex 10 (coded using the coding process specified in clauses 2 through 9), a slice NAL unit with nal_ref_flag equal to zero indicates that the slice is part of a non-reference picture.

[0021] For sequence parameter set, picture parameter set, or adaptation parameter set NAL units, nal_ref_flag shall be equal to one. When nal_ref_flag is equal to zero for one NAL unit with nal_unit_type equal to 1 or 4 for a particular picture, nal_ref_flag shall be equal to zero for all NAL units with nal_unit_type equal to 1 or 4 for that picture.

[0022] nal_ref_flag shall be equal to one for NAL units with nal_unit_type equal to 5.

[0023] nal_ref_flag shall be equal to 0 for all NAL units with nal_unit_type equal to 6, 9, 10, 11, or 12.

[0024] nal_unit_type specifies the type of RBSP data structure contained in the NAL unit, as specified in Table 1.

[0025] Decoders shall ignore (remove from the bitstream and discard) the content of all NAL units that use the reserved value of nal_unit_type.

[0026] Table 1 - Possible NAL unit type codes and NAL unit type categories (which is not an exhaustive list)

[0027]

[0028]

[0029] A "profile" is a subset of the overall bitstream syntax. Within the limits imposed by the syntax of a given profile, there can still be a wide variation in the performance of encoders and decoders, depending on the values taken by the syntax elements in the bitstream, such as the specified size of the coded pictures. In many applications, it is currently neither feasible nor cost-effective to implement a decoder capable of handling all the hypothetical uses of the syntax within a particular profile.

[0030] To handle this problem, "tiers" are specified within each profile. A tier is a specified set of constraints on the values of the syntax elements in the bitstream. Such a constraint can be a simple limit on a value. Alternatively, such a constraint can be in the form of a constraint on an arithmetic combination of values (e.g., picture width times picture height times number of pictures per second).

[0031] Tier: A defined set of constraints on the values taken by syntax elements and variables. The same set of tiers is defined for all profiles, with most aspects of the definition of each tier being common across different profiles. Within the specified constraints, individual implementations can support different tiers for each supported profile. In different contexts, tiers are values of transform coefficients that are transformed before scaling.

[0032] Profile: A specified subset of the syntax.

[0033] In the development of the 3D video coding extension of HEVC, it has been proposed to shift certain parameters from the tile header to the access unit delimiter (AUD), which is a NAL unit that is optionally located at the beginning of a new access unit (AU) in H.264 / AVC.

[0034] Another proposal during the process of specification of the HEVC 3D video coding extension is to show the dependency between operation points by an indirect method between SPS. The NAL unit header should carry a reference to an SPS, and within each SPS there is a reference to a related base SPS. This (possibly overlapping) indirect method has to be resolved before the lowest (temporal...) level SPS. This method imposes a high burden on devices such as MANEs to deeply inspect the bitstream and save a large amount of information available to identify operation points.

[0035] In any case, it would be advantageous to have a solution now that facilitates or makes more efficient the handling of scalable data streams by network entities.

[0036] This object is achieved by the subject-matter of any of the appended independent claims. SUMMARY

[0037] The basic finding of the present invention is that the handling of scalable data streams by network entities can be made less complex if the packets actually carrying data in the data stream are accompanied by packets of different packet types, which have scalability axis descriptors defining the number of scalability axes and their semantic meaning. By this measure, it is sufficient for the data carrying packets to include an operation point identifier, which identifies the operation point associated with the respective packet within the scalability space spanned by the scalability axes. By this measure, the operation point identifier provides sufficient information to the network entity to associate such a packet with its operation point in the scalability space, assuming that the network entity has access to this scalability axis descriptors provided by packets of a predetermined packet type. In fact, the present technique allows to reduce the amount of data for most packets, since the operation point identifier only has to fulfill the identification task. The scalability axis descriptors can be transmitted less frequently. At the same time, the network entity does not have to perform the heavy task of parsing / reading the scalability axis descriptors less frequently, reducing the burden of the network entity. On the other hand, according to embodiments of the present invention, the network entity is able to interpret the operation point identifier at least to the extent that it can distinguish between packets that are the lowest operation point in the order of the operation points (i.e. base layer packets) and other packets (i.e. packets belonging to operation points located higher in the order of the operation points, i.e. "enhancement" layer packets).

[0038] According to another aspect of the application, which can be combined with the first aspect or can be implemented alone, the basic idea of the application is that if packets different from data carrying packets are used for transmitting the hierarchy and / or the profile, the handling of the scalable data stream by the network entity can be made more efficient, wherein the profile and / or the hierarchy descriptor is split into a first set explicitly showing the definition of the set of available coding options and / or the range of available syntax element values for the respective operating point and a second set of the profile and / or the hierarchy descriptor showing the definition of the set of available coding options and / or the range of available syntax element values for its respective operating point by referring to another profile and / or hierarchy descriptor. By this measure, the length of the complete sequence of profiles and / or hierarchy descriptors is kept low, saving transmission burden and parsing burden. BRIEF DESCRIPTION OF DRAWINGS

[0039] The preferred embodiments of the present application will be described hereinafter with reference to the drawings. Advantageous embodiments form the subject of the dependent claims. In the drawings:

[0040] FIG. 1A A schematic diagram showing a data stream in which media content is encoded is shown;

[0041] FIG. 1B The composition of a scalable data stream is shown schematically;

[0042] FIG. 1C The composition of a scalable data stream according to another example is shown schematically;

[0043] FIG. 2 A schematic diagram showing the situation in which a network entity receives a complete data stream is shown;

[0044] FIG. 3A Operating points within a two-dimensional scalability (e.g. time and space) are shown;

[0045] FIG. 3B The new_profile_level_flag concept is shown;

[0046] FIG. 4 The NAL unit header syntax for HEVC is shown;

[0047] FIG. 5 The VPS syntax is shown;

[0048] FIG. 6 The vps_extension_data() syntax is shown;

[0049] FIG. 7 A schematic diagram showing a data stream according to an example is shown;

[0050] FIG. 8a flowchart showing an example of a process to partition fixed bit length syntax elements or bit sequences, respectively;

[0051] FIG. 9 a flowchart showing an example of an operational mode of a network entity;

[0052] FIG. 10 showing another operational mode of a network entity;

[0053] FIG. 11 showing NAL unit syntax;

[0054] FIG. 12A to FIG. 12C showing SPS syntax;

[0055] FIG. 13 showing VPS syntax; and

[0056] FIG. 14 showing VPS extension data syntax. DETAILED DESCRIPTION

[0057] To facilitate an easy understanding of the embodiments outlined below, details of a scalable data stream and problems resulting from such details in handling the scalable data stream are first described with respect to FIG. 1A to FIG. 1C and FIG. 2 FIG. 1A A data stream 10 is shown into which media content, such as audio content, video content, picture content, 3D mesh content or the like, is encoded. The media content can be time-variant, wherein the data stream represents the media content in a sequence of time-stamped, such as pictures of a video, transform blocks or frames of an audio signal or the like.

[0058] As shown in FIG. 1A the data stream 10 comprises or consists of packets 12. The packets can be equal in size or can differ in size. Each packet comprises a packet type identifier 14. Each packet type identifier 14 identifies a packet type associated with the respective packet 12 from a plurality of packet types. The plurality of packet types comprises a data carrying packet type. In FIG. 1A for example, packets within a part of the data stream 10 exemplarily shown in FIG. 1A are exemplarily shown as being of this data carrying packet type, here exemplarily denoted as "A". In addition to this data carrying packet type, there can be other packets of other packet types, which do not carry data actually contributing to the encoded representation of the media content, but carry other data, such as supplemental data or information about the origin of the media content or the like.

[0059] ​At least each packet of the data carrying packet type, such as A, comprises an operation point identifier 16 identifying an operation point associated with the respective packet from a plurality of operation points within a scalability space spanned by n scalability axes, where 0 < n, the operation points having an ordered operation point order defined therein. For clarity, reference is made to FIG. 1B The operation points are illustrated by circles 18. The operation points are scattered or arranged within a scalability space 20 spanned by the scalability axes, where in FIG. 1B For illustration purposes, only two such scalability axes 22a and 22b are shown.

[0060] FIG. 1B The composition of a scalable data stream, such as scalable data stream 10, is illustrated. Packets 12, whose operation point identifiers identify operation point 1, carry data which together form an encoded representation of the media content at this operation point 1. Here, as an example, operation point 1 forms a basic representation, i.e. the least amount of information about the media content is provided by the packet data of operation point 1. In the case of a video, for example, such packet data represents the encoded representation of the media content at, for example, a low spatial resolution and a lowest SNR. By combining the data carried by the packets belonging to operation point 1 with the data carried by the packets 12 of operation point 2, as exemplified by the loop 24, the data thus derived results in an encoded representation of the media content which is reconstructed in a way, such as in terms of SNR or spatial resolution, which exceeds the representation obtained from the reconstruction of the media content from the packets of operation point 1 alone. By adding the data of the packets of operation point 2 to the data of the packets of operation point 1, the encoded representation is extended, the characteristics of the media content in relation to which are FIG. 1B the semantic meaning of axis 2 in

[0061] In FIG. 1B For example, axis 22b relates to spatial resolution, while axis 22a relates to SNR quality. However, other semantic meanings of scalability axes are equally possible. For example, the additional data of another operation point's packets can add information to the reconstructable version of the media content which was not available before, i.e. before the addition of this data, such as another view, another audio channel, a depth map or the like.

[0062] Thus, FIG. 1BAlso shown by the further loop 24, the base encoded representation of the media content can be extended along the axis 22b and along 22a. In other words, adding the data of a package 12 of a certain operation point to the data of a package of another operation point or a combination of the data of packages of other operation points 24 results in a new combination of data which exceeds the representation obtained without this addition according to one of the scalability axes 22a or 22b and this sequential effect is shown by the arrow 26 which thus forms a sequential operation point order among the operation points 18 which passes through the scalability space 20.

[0063] Now, assume that the network entity receives this complete data stream. FIG. 2 This situation is shown. The network entity is shown as block 28 and is for example a transcoder or a MANE. In FIG. 2 The network entity 28 receives the complete data stream 10, i.e. the data stream comprising all data packages including all data carrying data packages of all operation points. The task of the network entity 28 is to provide the media content to a sink 30. In FIG. 2 As an example, the network entity 28 uses a wireless transmission 32 to forward the media content to the sink 30. The network entity 28 is for example a mobility management entity. In this situation, the network entity 28 has to cope with time-variant transmission capabilities to the sink 30. That is, for example, there are time instances in which the transmission capabilities are not sufficient to transfer the complete data stream 10 to the sink 30 for real-time reproduction. Therefore, the network entity 28 has to "reduce" the data stream 10 in order to provide a trimmed version or representation of the media content to the sink 30. While for example the transmission capabilities are not sufficient to transfer the complete data stream 10, the transmission capabilities can be sufficient to transmit the data stream 10 omitting for example the data of the packages 12 of operation point 5.

[0064] It should be noted that there are also other situations in which the network entity 28 has to discard some packages 12 to meet some external driven constraints. In this respect, FIG. 2 This is only used as an example.

[0065] However, in order to perform the just mentioned task of reducing the data stream, the network entity 28 has to inspect at least a part of the data stream. This part to be inspected comprises the operation point identifiers 16. This imposes a high burden on the network entity 28 and it is therefore advantageous to keep this burden low. The embodiments described next achieve this goal.

[0066] Another aspect dealt with by the embodiments described in the following is related to the fact that the capabilities of the decoders assigned to decode the data stream can increase when extending the order of operation points. For example, a low level decoder can have a lower capability, i.e. have a lower profile and / or a lower level as explained in the introduction of the specification. Thus, the addition of some data of the packets of a certain operation point can thus be restricted to decoders meeting a certain minimum profile and / or minimum level criteria, and thus, it is advantageous if the network entity, such as a decoder or a certain network entity forwarding the data stream to the decoder, is able to understand the profile and / or level requirements associated with the operation point 18 in a simple way, but on the other hand at a low rate of consumption.

[0067] However, before starting to describe such embodiments, it should be noted that the following applies. In detail, the nomenclature applied in the embodiments described in the following differs from the nomenclature used in the description of the FIG. 1A to FIG. 2 The description made in the following is made in the context of the HEVC standard. However, it should be noted that the embodiments described in the following are not limited to the HEVC standard, but can be applied to other types of codecs as well. Thus, the following embodiments should be understood as being exemplary only.

[0068] Similarly, although FIG. 1B The schematic combinations 24 only form proper subsets of each other, thus each combination is a proper subset of all other combinations 24, but a proper subset or a superset of one or all other combinations 24 of the proper superset of the other combinations 24 is optional. For example, FIG. 1C It is shown that a case, in which the orderliness of the operation points 18, i.e. the property of having an order of operation points defined therein, should be understood to include the case, in which a combination also includes a case, in which a combination is neither a proper subset nor a proper superset of the other combinations. In FIG. 1C In detail, for example, the combination 24 resulting from the combination of the data of the packets of the operation point 4 to the data of the packets of the operation points 1 and 2 only partially overlaps with the combination 24 resulting from the combination of the data of the packets of the operation point 1C to the data of the packets of the operation points 1 and 2, wherein the intersection is formed by the latter combination, i.e. the combination of the operation points 1 and 2. Thus, the order of operation points includes a branching at the operation point 2.

[0069] The following embodiments present high level syntax including NAL unit header and VPS syntax that allows to signal scalability information in a way that can be used by a video coder, a video bitstream extractor or a network device for adaptation. The NAL unit header includes a scalability identifier that allows to signal up to three scalability dimensions.

[0070] Before describing the VPS syntax in more detail, some background information is provided regarding FIG. 3A and FIG. 3B An overview is presented. This concept can be used, for example, in the HEVC standard.

[0071] The new_profile_level_flag presented below is used to signal that a certain operation point uses a different profile or level than the previously described operation point within a scalability dimension of the video (e.g. time, space or any other dimension). FIG. 3A Different operation points within a spatially and temporally scalable video sequence are shown according to the scalability identifiers (temporal_id and layer_id). FIG. 3A Operation points within two-dimensional scalability (e.g. time and space) are shown.

[0072] The profile and level required to describe the decoder capabilities in terms of supported coding tools, macroblock throughput, etc. can change from operation point to operation point, which depends to a large extent on the used scalability. The new_profile_level_flag is used in a loop that signals the profile and level of a certain operation point or signals that the profile and level of a certain operation point has to be inherited from a previously described operation point by explicit reference, or signals that a new profile and level exists with respect to the previously described operation point.

[0073] Another aspect described below relates to an adaptable list of profile and level syntax elements. The profile, level and optionally other parameters of the operation points in a parameter set are given via a while loop that allows direct adaptation, i.e. excluding or deleting certain operation points and adding certain operation points without changing other syntax elements within a given syntax (i.e. within the VPS). This is achieved using the more_rbsp_data() syntax element that allows a while loop that does not necessarily signal the absolute number of layers and thus makes adaptation of a parameter set easy. FIG. 3B The new_profile_level_flag concept is shown.

[0074] Another aspect described below relates to layer status. Layer status represents a way to interpret the signaling of five reserved bits in the second byte of the NAL unit header that can be used as a scalable layer identifier layer_id in the context of scalability in the HEVC extension.

[0075] In the layer scenario approach, a scenario identifier is signaled in the NAL unit header, which allows the decoder or MANE to interpret the five reserved bits as an independent one- or multi-dimensional identifier for video coding, which is a characteristic of one or more scalability dimensions, such as temporal and spatial, spatial and multiview, or others. The scalable dimensions and the detailed description of the operation points within are given by the VPS, while the NAL unit header only provides information on how to interpret the five reserved bits of the NAL unit header syntax in order to derive one or more scalability identifiers.

[0076] The VPS syntax described next provides the means to signal the profile and level of each operation point and is designed to allow direct real-time adaptation. The following sections give details on this syntax.

[0077] The proposed NAL unit header syntax is described first.

[0078] The second byte of the NAL unit header is used to signal the so-called layer scenario and a scalability identifier, layer_id, which identifies the operation point in the context of the layer scenario.

[0079] FIG. 4 The NAL unit header syntax for HEVC extensions is given, with the new syntax elements layer_scenario and layer_id. The syntax element layer_scenario signals the scenario of the scalability dimension, e.g., one scalability dimension, to use either temporal scalability only (e.g., as in the temporal scalability in the HEVC base specification) or a combination of two types of scalability (e.g., temporal and spatial). The syntax element layer_id describes the specific operation point of the NAL unit within the signaled scalability dimension and has to be interpreted according to the layer_scenario syntax element.

[0080] In the case of no scalability or one scalability dimension (i.e., the value of layer_scenario is zero), the value of layer_id can be interpreted as an unsigned integer, which lists the operation points in consecutive order according to the coding dependency and importance. In the case of no scalability, all NAL units should have a layer_id value of zero to signal that there is no meaningful tier adaptation procedure and all NAL units have the same importance.

[0081] In the case of two scalability dimensions (i.e. the value of layer_scenario is one), the value of layer_id can be interpreted as two 3-bit unsigned variables. The first 3 bits give the operation point of the NAL unit in the first scalability dimension (e.g. time) and the second 3 bits give the operation point of the NAL unit in the second scalability dimension (e.g. space).

[0082] When the value of layer_scenario is two, the value of layer_id can be interpreted as a 2-bit unsigned integer variable giving the operation point in the first scalability dimension, followed by a 4-bit unsigned integer variable describing the operation point in the second scalability dimension. This can be beneficial when the number of layers in one dimension is larger than in the other dimension.

[0083] Considering three scalability dimensions, the corresponding value of layer_scenario is three and the value of the layer_id syntax element can be interpreted as three 2-bit unsigned integer variables giving the operation point according to the three scalability dimensions used (e.g. time, space and multi-view scalability).

[0084] Table 2 gives an overview of the values of layer_scenario and presents the corresponding interpretation of layer_id.

[0085] Table 2: Overview of the values of layer_scenario.

[0086]

[0087] The corresponding video parameter set syntax is described next.

[0088] The syntax of the video parameter set has to be adjusted to signal the profile and level of each operation point in the possible three-dimensional scalability space and to provide means to adapt it in a straightforward manner to reflect the current adaptation level of the video bitstream. This is achieved using the vps_extension_data() syntax element for profile and level signaling of NAL units of any kind of scalability (e.g. temporal, spatial or others). In addition, it provides the details to interpret the layer_id syntax element in the NAL unit header.

[0089] FIG. 5 The VPS syntax is given in Table 3.

[0090] FIG. 5 The profile and level of NAL units not using any kind of scalability, i.e. video data coded according to the HEVC base specification, with the lowest temporal level, is signaled.

[0091] Furthermore, a syntax element layer_id_type is given to allow a more detailed understanding of the scalability dimensions described by the layer_id syntax element within the NAL unit header syntax. Table 1C specifies for the layer_id_type syntax element the actual scalability dimensions indicated in layer_id. Corresponding to the HEVC base specification, only temporal scalability has to be defined. Additional HEVC scalability extensions will thus specify additional layer_id_type values for specific needs, e.g. 1 : temporal scalability, 2: spatial scalability for a SVC-like extension of HEVC.

[0092] This allows any device having access to the VPS to fully interpret the operation points of NAL units within the known scalability dimensions, while devices not having access to the VPS can still perform a correct adaptation of the video bitstream, but without knowing the actual scalability dimensions.

[0093] Table 3: layer_id_type and detailed description of scalability dimensions in layer_id

[0094]

[0095] The description of operation points using scalability in any dimension (e.g. temporal hierarchy > 0, spatial scalability,...) is given by vps_extension_data() as specified in FIG. 6 Each operation point is identified via its layer_id. The new_profile_level_flag allows an explicit indication of the profile and level of the described operation, or an implicit indication by referencing another operation point description (via its layer_id) using the syntax element ref_layer.

[0096] The use of the vps_extension_data() syntax allows an adaptation in a straightforward manner, as explicit references to operation points are made and the operation point can be omitted by excluding a fixed number of bytes. As long as other operation point descriptions do not reference the omitted vps_extension_data() syntax, no bits of the VPS have to be changed (except the omitted ones), which greatly simplifies real-time adaptation.

[0097] Thus, using the nomenclature applied with respect to FIG. 1A to FIG. 1C The specific embodiments just outlined describe, using the nomenclature applied with respect to FIG. 7. The data stream, generally indicated using the reference numeral 10, has media content encoded therein, such as audio, video, or other media content as indicated above. The data stream 10 includes packets 12, referred to as NAL units. Each packet 12 includes a packet type identifier, namely, nal_unit_type, which identifies the packet type associated with the corresponding packet 12 from a plurality of packet types. Each packet 12 (such as all NAL units) having a packet type associated therewith from a first set of the plurality of packet types includes an operation point identifier 34, namely, a combination of layer_scenario and layer_id, which identifies the operation point 18 associated with the corresponding packet 12 from a plurality of operation points 18 within the scalability space 20 spanned by n scalability axes 22a, 22b, where 0>n, the operation points 18 having a sequential operation point order defined therein. In this regard, it should be noted that not every NAL unit or packet 12 necessarily has an operation point identifier 34. In contrast, there may be packets 12 of a certain packet type which are of a general nature insofar as they relate to operation points and which therefore do not include any operation point identifier.

[0098] However, in FIG. 7 In the embodiment of , each packet 12 having associated therewith a packet type from the second set of packet types (i.e., a non-VPS NAL unit) additionally carries data 36 that, along with the packets 12 having associated therewith any operation point, contributes to the encoded representation of the media content at the operation point 18 associated with the corresponding packet, the arbitrary operation point being before or equal to the operation point associated with the corresponding packet according to the sequential operation point order. In this regard, reference is made to FIG. 1B and FIG. 1C Description for details.

[0099] Each packet having associated therewith a predetermined packet type that is disjoint from the second set of packet types has a scalability axis descriptor 38 defining the number n of one or more scalability axes and their syntactic meaning, wherein the scalability axis descriptor 38 is represented by the layer_id_type present in the VPS NAL unit, wherein the rbsp data portion 40 is populated according to the VPS syntax 42.

[0100] As also described above, each VPS packet can also include a sequence of profile and / or level descriptors 44, i.e., vps_extension_data, associated with at least a subset of the plurality of operation points 18, such as the leading operation points in the order of operation point sequence. Each profile and / or level descriptor 44 defines a set of available coding options, i.e., a minimum profile to be supported for reconstruction purposes, and / or a range of available syntax element values, i.e., a minimum level required for reconstruction, with which the syntax of the packets 12 of the operation points associated with the respective profile and / or level descriptor 44 complies. In other words, a data combination resulting from adding the data of a packet of a certain operation point 18 to the data of a packet associated with one or a combination of other operation points 18 can only be reconstructed for decoders supporting the respective profile and / or level.

[0101] A first set of profile and / or level descriptors 44 explicitly signals the definition of the set of available coding options and / or the range of available syntax element values, i.e., the descriptors with the syntax element new_profile_level_flag, i.e., flag 46, equal to 1. A second, disjoint, non-empty set of profile and / or level descriptors 44 signals the definition of the set of available coding options and / or the range of available syntax element values by reference to another, previous profile and / or level descriptor 44, i.e., the descriptors with the syntax element new_profile_level_flag equal to 0. In FIG. 6 In the case of profiles, the explicit signaling is done using the syntax element profile_idc located at 48, while the syntax element level_idc represents the level indicator 50. The reference to another profile and / or level descriptor 44 in the case of new_profile_level_flag = 0 is done using the syntax element ref_layer_id 52. Obviously, the use of reference profile and / or level descriptors 44 reduces the size required for transmitting the sequence 54 of profile and / or level descriptors, and as further outlined below, this aspect can be used independently of the common use of operation point identifiers 34 and scalability axis descriptors 38.

[0102] While the sequence 54 of profile and / or level descriptors 44 can be associated with at least a subset of the plurality of operation points in a sequential operation point order, this is not necessarily the case. Rather, each profile and / or level descriptor can have a field 56, namely a layer_id, which defines the operation point 18 to which the respective profile and / or level descriptor 44 is associated. A network entity processing the data stream 10 can determine the sequential operation point order defined among the operation points 18 from the order of the fields of the profile and / or level descriptors 44 in the sequence 54. However, or the sequential operation point order can be known to the network entity a priori, depending on, for example, the scalability axes descriptor. For more precision, the network entity, such as the network entity 28 of FIG. 2 the following described embodiments, the bit sequence for the syntax element ref_layer_id 52 uses the same bit sequence as the layer_id field 56 of the profile and / or level descriptors 44. In other words, the bit sequence for the syntax element ref_layer_id 52 is equal to the bit sequence of the layer_id field 56 of the profile and / or level descriptors 44.

[0103] The field 56 identifies the respective operation point using a bit sequence which is common to the operation point identifiers 34. In the following described embodiments, for example, all operation point identifiers 34 have a fixed bit length syntax element, namely a layer_id, and a context syntax element, namely a layer_scenario, wherein the context syntax element indicates n and the association of n partitions of the fixed bit length syntax element with n scalability axes, which is common to the bit sequence of the field 56 or is built equally to the bit sequence of the field 56. FIG. 8 An example is exemplarily shown to partition the fixed bit length syntax element or the bit sequence 60 of the field 56 or the fixed bit length syntax element of the operation point identifiers 34 into different numbers of partitions depending on three different states 62 of the context syntax element, wherein the partitions are exemplarily shown with a dashed line for the first state, a dash-dotted line for the second state and a dotted line for the third state. That is, a network entity interpreting the fixed bit length syntax element or the bit sequence 60 can identify the correct operation point identified by the individual fixed bit length syntax element or the bit sequence 60 as a component in the n-dimensional scalability space 20 by using the bits contained in the respective partition.

[0104] In the following described embodiments, the bit sequence for the syntax element ref_layer_id 52 uses the same bit sequence as the layer_id field 56 of the profile and / or level descriptors 44. In other words, the bit sequence for the syntax element ref_layer_id 52 is equal to the bit sequence of the layer_id field 56 of the profile and / or level descriptors 44. FIG. 8the same bit sequence as shown in the middle. Alternatively, it can have used related pointers defined in rank position order with sequential operation point order. For example, the network entity can have added the ref_layer_id bit by bit to the syntax element layer_id 56 in order to access or obtain the layer_id of the operation point represented by the current profile and / or level descriptor 44 in order to employ the profile identifier profile_idc 48 and / or the level identifier level_idc 50 of the current operation point.

[0105] In the above-described embodiments, the scalability axis descriptor 38, i.e. layer_id_type, forms a syntax element whose possible states are associated with different combinations of values of n and corresponding semantic meanings of the n scalability axes, respectively. That is, the possible states of the scalability axis descriptor are mapped to different (n) meanings (1),..., meaning(n), respectively (where reference is made to Table 3). That is, using the scalability axis descriptor 38, the network entity reading the data stream knows the semantics of the meaning of the property, FIG. 8 The values in the partition of the bit sequence 60 of the operation point identifier 34 (as obtained using the partitioning according to the context syntax element 64) can, however, at least rely on the ordinal property of the values in the partition in order to decide on a possible reduction of the data stream in the course of processing the data stream.

[0106] Before proceeding to the next embodiment outlined in the specific summary, it is noted that according to the above-described further aspect of the application, the scalability axis descriptor 38 can be omitted and its semantics, for example, included in the operation point identifier 34, which, however, would have to consume more bits than the presently described embodiments. However, such an alternative embodiment would still be advantageous, as there is the above-described sequence 54 of the profile and / or level descriptors 44, which refers to the explicit indication of the profile and / or level descriptors 44 and the indirect indication / reference to the profile and / or level descriptors 44.

[0107] However, with regard to the above description, it is noted that various modifications can be made to the above-described embodiments outlined. For example, the number of bits chosen in the above example for the fixed length syntax element or the bit sequence 60 (i.e. 6) can be made larger or smaller. A similar note is reasonable in view of the range of possible scalability axes n that can be adjusted by the context syntax element 64: the set of possible states is (1, 2, 3), but different sets of possible integers of n can also be implemented.

[0108] For the sake of completeness only, FIG. 9The operation mode of the network entity showing e.g. the sequence of read profile and / or level descriptors 44. First, the fields of the first profile and / or level descriptor 44 in the list 54 are read in step 66. Then, the network entity checks in step 68 whether the current profile and / or level descriptor 44 (whose fields have been read in step 66) explicitly indicates a profile and / or level or indirectly indicates a profile and / or level. According to the above embodiment, step 68 is designed to read the syntax element reserved_zero_one_bit, wherein the actual decision depends on the value of the syntax element: new_profile_level_flag equal to 1 indicates explicit indication and the syntax element equal to 0 indicates indirect indication.

[0109] In case of explicit indication, the network entity reads the profile and / or level identifier 58 / 50 in step 70 and associates the profile and / or level identifier 58 / 50 with the operation point identified by the fields read in step 66. However, in case of indirect indication, the network entity reads the reference field 52 from the data stream in step 72 and employs in step 74 the profile and / or level identifier associated with the operation point identified by this reference field 52 as the profile and / or level identifier of the operation point identified by the fields read in step 66. This step is then repeated until the sequence of profile and / or level descriptors 44 has been scanned.

[0110] Similarly, FIG. 10 The operation mode of the network entity utilizing the above embodiment is shown, wherein the scalability axis descriptor 38 refines the information provided by the operation point identifier 34 in the respective packet. As FIG. 10As shown in the middle, first the operation point identifier of the current packet is read in step 76. This reading is done for all packets 12. In addition, the network entity checks the VPS NAL unit in order to read the scalability axis descriptor therefrom in step 78. Steps 76 and 78 are done whenever the network entity encounters a respective packet of a respective packet type. Whenever the operation point identifier is read in step 76, the network entity distinguishes in step 80 between the case that the scalability axis descriptor has not been read yet, and thus is not available, or has been read, and thus is available. In the case of availability, the network entity interprets the operation point identifier of the current packet in terms of the scalability axis descriptor read in step 78 in step 82. After the interpretation, the network entity knows the meaning of the scalability axis and can act accordingly. In detail, the network entity selects a packet from the packets 12 in step 84, such as a packet to be decoded or forwarded to a sink, such as the sink 30, regardless of availability or non-availability. Step 84 can depend on external data, such as data indicating the capabilities of the sink to reproduce media data, such as a maximum reproducible spatial resolution, a maximum pixel bit depth of a display, etc. In the case of non-availability of the interpretation due to the non-availability of the scalability axis descriptor, the network entity 28 can be configured to act defensively, e.g. by only further processing packets following the order of operation points in the sequence that belong to the lowest operation point.

[0111] Likewise, the network entity 28 can be a decoder, a transcoder, a media-aware network entity such as a mobility management entity, or the like.

[0112] In addition, an entity generating packets for any of the embodiments of the data stream described above, such as an encoder or the like, can be configured to intersperse VPS NAL units between the data carrying packets frequently enough to keep the duration low, wherein the network entity is faced with the problem of non-availability occurring when the scalability axis descriptor is not available yet, as described with respect to FIG. 10 .

[0113] For a network entity reading the sequence of profile and / or level descriptors 44, it should be noted that this network entity can be configured to select among the packets 12 based on the set of availability coding options and / or the range of available syntax element values associated with the respective operation point in addition to or as an alternative to the processing procedure FIG. 10 , e.g. in the case that the network entity is a decoder or a sink (with which the network entity communicates), e.g. in the case that the network entity is an MME, the network entity can exclude operation points for which the required profile and / or level exceeds the maximum profile and / or maximum level of itself.

[0114] In other words, with respect to FIG. 4 to FIG. 6The described bitstream, in which media content is encoded, includes packets (e.g., NAL packets), each packet including a packet type identifier nal_unit_type that identifies a packet type associated with the respective packet from a plurality of packet types, wherein each packet (e.g., all NAL units) having a packet type associated therewith from a first set of the plurality of packet types includes an operation point identifier layer_scenario and layer_id that identifies an operation point associated with the respective packet from a plurality of operation points spanning a scalability space of n scalability axes, where 0 < n, the operation point having an ordered operation point order defined therein, wherein each packet 12 (e.g., non-VPS NAL units) having an arbitrary packet type associated therewith from a second set of the first set of packet types additionally carries data that, together with packets having an arbitrary operation point associated therewith, contributes to an encoded representation of the media content at the operation point associated with the respective packet, the arbitrary operation point preceding or equal to the operation point associated with the respective packet, wherein each packet having a predetermined packet type associated therewith that is disjoint from the second set (i.e., outside the second set) (e.g., VPS NAL units) has a scalability axes descriptor (e.g., layer_id_type) that defines a number of one or more scalability axes and their syntax meaning.

[0115] In another example of a bitstream, each of the packets having the predetermined packet type associated therewith further includes a sequence of profile and / or level descriptors vps_extension_data that are respectively associated with at least a subset of the plurality of operation points (e.g., leading operation points in the ordered operation point order), each profile and / or level descriptor defining a set of available coding options (e.g., a profile) and / or a range of available syntax element values (e.g., a level), the syntax of packets having an operation point associated therewith associated with the respective profile and / or level descriptor adhering to the set of available coding options and / or the range of available syntax element values, wherein a first set of the profile and / or level descriptors explicitly indicates a definition of the set of available coding options and / or the range of available syntax element values (e.g., new_profile_level_flag = 1), and a second, disjoint, non-empty set of the profile and / or level descriptors indicates a definition of the set of available coding options and / or the range of available syntax element values by reference to another, previous profile and / or level descriptor (e.g., new_profile_level_flag = 0).

[0116] In another example of a bitstream, the sequence of profile and / or level descriptors is sequentially associated with at least a subset of the plurality of operation points (e.g., leading operation points in the ordered operation point order).

[0117] In another example of the data stream, each profile and / or level descriptor has a field (e.g., layer_id) that identifies the operation point to which the respective profile and / or level descriptor is associated (i.e., along with 4), and the order operation point order can be derived from the sequence of profiles and / or level descriptors.

[0118] In another example of the data stream, in each profile and / or level descriptor, the field identifies the operation point to which the respective profile and / or level descriptor is associated using a bit sequence that is common to all operation point identifiers, the bit sequence identifying the operation point to which the respective profile and / or level descriptor is associated.

[0119] In another example of the data stream, all operation point identifiers have a fixed bit length syntax element (e.g., layer_id) and a context syntax element (e.g., layer_scenario) that indicates the association of n and the n partitions of the fixed bit length syntax element with the n scalability axes (but without semantic meaning), where the bit sequence is common to the fixed bit length syntax element.

[0120] In another example of the data stream, a second, disjoint, non-empty set of profiles and / or level descriptors indicates the definition of a set of available coding options and / or available syntax element value ranges by referring to another, previous profile and / or level descriptor, using an explicit indication of a field of the referred profile and / or level descriptor, or using a related indicator that is defined in terms of rank position in the order operation point order (where the second alternative is the alternative described above).

[0121] According to another aspect of the data stream, all operation point identifiers have a fixed bit length syntax element and a context syntax element that indicates the association of n and the n partitions of the fixed bit length syntax element with the n scalability axes, so that the n-th partition is a sequential measure of the position of the respective operation point along the n-th scalability axis.

[0122] According to another aspect of the data stream, the scalability axis descriptor (e.g., layer_id_type) includes a syntax element whose possible states are associated with different combinations of values of n and corresponding semantic meanings of the n scalability axes (i.e., a function maps {n, meaning(l),..., meaning(n)} to {1...max. layer_id_type possible states}).

[0123] In another data stream, media content is encoded therein, the data stream comprising packets, each packet comprising a packet type identifier identifying a packet type associated with the respective packet from a plurality of packet types, wherein each packet having a packet type associated therewith from a first set of the plurality of packet types comprises an operation point identifier identifying an operation point associated with the respective packet from a plurality of operation points within a scalability space spanned by n scalability axes, where 0 < n, the operation points having an order operation point order defined therein, wherein each packet having a packet type associated therewith from a second set of the first set of packet types additionally carries data that, together with packets having an operation point associated therewith that precede or are equal to the operation point associated with the respective packet, contribute to an encoded representation of the media content at the operation point associated with the respective packet, wherein each packet having a predetermined packet type associated therewith that is disjoint from the second set (i.e., outside the second set) comprises a sequence of profile and / or level descriptors associated with at least a subset of the plurality of operation points (e.g., leading operation points in the order operation point order), respectively, each profile and / or level descriptor defining a set of available coding options and / or a range of available syntax element values, the syntax of packets having an operation point associated therewith associated with the respective profile and / or level descriptor adhering to the set of available coding options and / or the range of available syntax element values, wherein a first set of the profile and / or level descriptors explicitly specify the definition of the set of available coding options and / or the range of available syntax element values, and a second, disjoint, non-empty set of the profile and / or level descriptors specify the definition of the set of available coding options and / or the range of available syntax element values by reference to another, previous profile and / or level descriptor.

[0124] In another example of a data stream, the sequence of profile and / or level descriptors is sequentially associated with at least a subset of the plurality of operation points in the order operation point order (e.g., leading operation points in the order operation point order).

[0125] In another example of a data stream, each profile and / or level descriptor has a field identifying the operation point to which the respective profile and / or level descriptor is associated.

[0126] In another example of a data stream, in each profile and / or level descriptor, the field identifying the operation point to which the respective profile and / or level descriptor is associated uses a bit sequence common to the operation point identifiers to identify the operation point to which the respective profile and / or level descriptor is associated, the bit sequence identifying the operation point to which the respective profile and / or level descriptor is associated.

[0127] In another example of a data stream, all operation point identifiers have a fixed bit length syntax element and a context syntax element that indicates n and the association of n partitions of the fixed bit length syntax element with n scalability axes, wherein the bit sequence is common to the fixed bit length syntax element.

[0128] In another example of a data stream, a second, disjoint, non-empty set of profiles and / or level descriptors indicates the definition of the set of available coding options and / or the range of available syntax element values by referencing another, previous profile and / or level descriptor, using explicit indications of fields of the referenced profile and / or level descriptor, or using a related indicator that is defined in terms of rank position in the order of operation points.

[0129] In another example of a data stream, all operation point identifiers have a flag that indicates whether they belong to a first set of profiles and / or level descriptors that explicitly indicates the definition of the set of available coding options and / or the range of available syntax element values and a second, disjoint, non-empty set of profiles and / or level descriptors that indicates the definition of the set of available coding options and / or the range of available syntax element values by referencing another, previous profile and / or level descriptor.

[0130] Another aspect has a network entity (e.g., a decoder or a MANE) to process a data stream in which media content is encoded, the data stream comprising packets, each packet comprising a packet type identifier that identifies a packet type associated with the respective packet from a plurality of packet types, wherein each packet having a packet type associated therewith from a first set of the plurality of packet types comprises an operation point identifier that identifies an operation point associated with the respective packet from a plurality of operation points within a scalability space spanned by n scalability axes, where 0 < n, the operation point having an order of operation points defined therein, wherein each packet having a packet type associated therewith from a second set of the first set of packet types additionally carries data that, together with packets having an arbitrary operation point associated therewith that precedes or is equal to the operation point associated with the respective packet, contributes to an encoded representation of the media content at the operation point associated with the respective packet, wherein the network entity is configured to read a scalability axes descriptor from a packet having a predetermined packet type associated therewith that is disjoint (i.e., outside) from the second set and to interpret the operation point identifier in dependence on the scalability axes descriptor, wherein the scalability axes descriptor defines the number of one or more scalability axes and their semantic meaning.

[0131] In another example, a network entity is configured to read a sequence of profiles and / or level descriptors having predetermined packet types associated therewith that are disjoint from (i.e., outside of) a second set, each profile and / or level descriptor defining a set of available coding options and / or a range of available syntax element values for a packet having an operation point associated therewith that is associated with at least a subset of the plurality of operation points (e.g., leading operation points in an order of operation point sequence), wherein the network entity is configured to explicitly read a definition of the set of available coding options and / or the range of available syntax element values from a first set of the profiles and / or level descriptors in response to a cue in the data stream, and to read a reference to another, previous profile and / or level descriptor for a second, disjoint, non-empty set of the profiles and / or level descriptors, and to derive the definition of the set of available coding options and / or the range of available syntax element values for the second, disjoint, non-empty set of the profiles and / or level descriptors from the corresponding referenced previous profile and / or level descriptor.

[0132] In another example of a network entity, the sequence of profiles and / or level descriptors are sequentially associated with at least a subset of the plurality of operation points in an order of operation point sequence (e.g., leading operation points in an order of operation point sequence).

[0133] In another example of a network entity, each profile and / or level descriptor has a field that identifies an operation point with which the corresponding profile and / or level descriptor is associated.

[0134] In another example of a network entity, in each profile and / or level descriptor, the field identifies the operation point with which the corresponding profile and / or level descriptor is associated using a bit sequence common to the operation point identifiers that identifies the operation point with which the corresponding profile and / or level descriptor is associated.

[0135] In another example of a network entity, all operation point identifiers have a fixed bit length syntax element and a context syntax element that cues n and an association of n partitions of the fixed bit length syntax element with n scalability axes, wherein the bit sequence is common to the fixed bit length syntax element.

[0136] In another example of the network entity, a second, disjoint, non-empty set of profiles and / or hierarchy descriptors indicates a definition of a set of available coding options and / or a range of available syntax element values using explicit signaling of fields of the referenced profile and / or hierarchy descriptor, or using a related indicator defined in rank position units in the order of the operation points.

[0137] In another example of the network entity, all operation point identifiers have a fixed bit length syntax element and a context syntax element, wherein the network entity is configured to use the context syntax element to determine n and to associate n partitions of the fixed bit length syntax element with n scalability axes, such that the nth portion is a sequential measure of the position of the corresponding operation point along the nth scalability axis.

[0138] In another example, the network entity is configured to select among the packets having associated therewith any packet type from a first set of packet types based on the interpretation.

[0139] Another example can have a network entity (e.g., a transcoder or MANE) to process a data stream in which media content is encoded, the data stream comprising packets, each packet comprising a packet type identifier identifying a packet type associated with the respective packet from a plurality of packet types, wherein each packet having a packet type associated therewith from a first set of the plurality of packet types comprises an operation point identifier identifying an operation point associated with the respective packet from a plurality of operation points within a scalability space spanned by n scalability axes, and 0 < n, the operation point having an order operation point order defined therein, wherein each packet having a packet type associated therewith from a second set of the first set of packet types additionally carries data that, together with packets having an operation point associated therewith, contributes to an encoded representation of the media content at the operation point associated with the respective packet, the operation point associated with the respective packet being either prior to an operation point associated with the respective packet, or equal to the operation point associated with the respective packet, wherein the network entity is configured to read, from packets having a predetermined packet type associated therewith that is disjoint (i.e., outside) of the second set, a sequence of profiles and / or level descriptors in order operation point order sequentially associated with at least a subset of the plurality of operation points (e.g., leading operation points in order operation point order), each profile and / or level descriptor defining a set of available coding options and / or a range of available syntax element values, the syntax of packets having an operation point associated therewith associated with the respective profile and / or level descriptor adhering to the set of available coding options and / or the range of available syntax element values, wherein a first set of the profiles and / or level descriptors explicitly specifies a definition of a set of available coding options and / or a range of available syntax element values, wherein the network entity is configured to explicitly read the definition of the set of available coding options and / or the range of available syntax element values from the first set of profiles and / or level descriptors in response to a specification in the data stream, and to read a reference to another, previous profile and / or level descriptor for a second, disjoint, non-empty set of the profiles and / or level descriptors, and to derive the definition of the set of available coding options and / or the range of available syntax element values for the second, disjoint, non-empty set of profiles and / or level descriptors from the respective referenced previous profile and / or level descriptor.

[0140] In another embodiment, the network entity is configured to select among packets having a packet type associated therewith from a first set of the plurality of packet types based on the associated set of available coding options and / or range of available syntax element values.

[0141] The following description of embodiments begins with a further introduction, however it can also shift to the embodiments described so far.

[0142] Besides the recording, storage and playback of video, its transmission is the main application in today's communication networks. The unstable nature of specific communication channels requires adaptation means provided by the high level syntax of HEVC to perform well in different environments. Direct adaptation of the video bitstream within the network during transmission, e.g. by a Media Aware Network Element (MANE), is one of the main challenges. Such devices usually have only limited resources in terms of processing power and have to operate with very low delay, which hinders an in-depth bitstream analysis.

[0143] The need for this setting has been recognized within the standardization of H.264 / AVC. The encapsulation of video data in Network Abstraction Layer (NAL) units and the design decision to signal many important but rather constant parameters outside the video bitstream (in so-called parameter sets) reflects this understanding. The lack of extensibility in the base specification of H.264 / AVC led to a complex approach to extend the high level syntax of H.264 / AVC in a backwards compatible way when specifying extensions. For HEVC, a flexible and extensible high level syntax shall be handled in the base specification. Therefore, the high level syntax has to provide the necessary information for devices such as MANEs in an accessible form (e.g. not entropy coded) and suitable to signal adaptation in a direct way.

[0144] The following adopts the concept of a video parameter set (VPS) as defined in the current draft specification to provide the information necessary for adapting the video bitstream, such as operation point description and coding dependencies. It combines information that in H.264 and its extensions is scattered over layer specific sequence parameter sets (SPS) and on top of the SPS in the parameter set hierarchy.

[0145] The current definition of VPS in the HEVC draft normative specification defines the VPS as a redundant copy of certain syntax only related to the sub-layer level requirements for temporal scalability, which were located in the SPS in H.264 and its extensions. Thus, the VPS is not necessary for the decoding of a coded video sequence. According to the following embodiments, however, the redundant signaling is omitted and the syntax elements are kept only in the VPS. Furthermore, according to the following embodiments, however, the signaling of the profile, level and sub-layer requirements is changed in such a way that the inherent hierarchy is reflected in the position of the requirements within the hierarchy of parameter sets (VPS, SPS and PPS). This means that the most essential and basic parameters for bitstream adaptation and capability exchange (i.e. profile and level information) are provided at the top of the parameter set hierarchy, i.e. in the VPS, while less important sub-layer requirements (i.e. sps_max_latency_increase[i], vps_num_reorder_pics[i], vps_max_dec_pic_buffering[i]) are signaled as in earlier versions of the draft specification. In this way, the most important information will be made available to the MANE in a straightforward way without the need to compile information from several SPSs. Applications relying on lower level requirements such as sub-layer requirements for temporal scalability can still find the required information in uncompiled form in the lower steps of the parameter set hierarchy. In general, a non-specific application MANE is hardly interested in this low level capability information of the end device. It is expected that for most applications the profile and level requirements are fine enough and corner case applications can still retrieve this information from the lower level parameter sets.

[0146] A VPS defined in this way allows the MANE to be fully aware of the scalability dimensions and the profile and level associated with each operation point. The operation points within the scalability dimensions can be signaled by reference (with redundant profile and level identifiers) to reduce the redundant signaling. The signaling of the respective operation point description with the VPS extension data syntax allows a straightforward modular structure, which can easily be built by the encoder.

[0147] A SEI message is proposed to allow the MANE and the receiver to be aware of the video sequence that can be adapted, where not all operation points described in the VPS and its extensions are present. It is important to allow the MANE to be aware of the actual (adapted) content of the video sequence to facilitate further adaptation (e.g. in an overlapping adaptation process of multiple MANEs).

[0148] To signal operation points of NAL units in general in the scalability dimensions that appear at the NAL unit header level, the reserved bits in the second byte of the NAL unit header are used to signal layer identifiers to identify enhancement layers or views from the base layer or view. The information necessary to interpret the layer identifiers of NAL units in terms of the scalability dimensions of the coded video sequence is provided by the VPS. This signaling includes the type of scalability and information on how the corresponding bits of the layer identifier correspond to the applicable scalability dimensions. This allows the MANE or any other device to correctly establish the matrix of scalable operation points, which can be multi-dimensional (temporal, quality, spatial, multi-view, depth...), from the information provided in the NAL unit header and the VPS.

[0149] FIG. 11 The NAL unit syntax of the described embodiment is shown in Table 1, where layer_id_plusl minus 1 indicates the layer identifier. A layer_id_plusl equal to 1 indicates that the layer identifier is equal to 0 and the NAL unit belongs to the base layer or base view. Higher values of layer_id_plusl indicate that the corresponding NAL unit belongs to an enhancement layer or view associated with the specified layer_id_plusl.

[0150] FIG. 12A to FIG. 12C The corresponding SPS syntax is shown in Table 2, and FIG. 13 The corresponding VPS syntax is shown in Table 3, where: scalability_type indicates the type of scalability used in the coded video sequence and the dimensions are signaled via layer_id_plusl in the NAL unit header. When scalability_type is equal to 0, the coded video sequence conforms to the base HEVC specification, therefore layer_id_plusl is equal to 1 for all NAL units and no NAL unit belongs to an enhancement layer or view. Higher values of scalability_type have to be interpreted according to FIG. 4

[0151] profile_space, profile_idc, constraint_flags and profile_compatibility_flag signal information as specified in the current WD and the indicated values are valid for all NAL units with layer_id_plusl equal to 1.

[0152] level_idc[i] signals information as specified in the current WD and the indicated values are valid for all NAL units with layer_id_plusl equal to 1 and temporal_id equal to i.

[0153] ​layer_id_plus1_dim_len[ dim ] is according to FIG. 13 indicates the length of layer_id_plus1 for the n-th scalability dimension. For n = 0, it starts at the MSB, or for n > 0, it ends at the end of the (n-1)-th signaled scalability dimension.

[0154] Table 4: Proposed scalability_type mapping table

[0155] scalability_type max_dim dimension 0 1 none 1 2 spatial and quality scalability 2 2 multiview and depth scalability 3 4 multiview, spatial, quality and depth scalability 4...15 reserved reserved

[0156] The exemplary defined scalability types in Table 4 represent a set of scalabilities that are expected at this point in time, but can be extended by new dimensions and / or combinations once work on HEVC extensions has progressed.

[0157] FIG. 14 The corresponding VPS extension data syntax is shown in Table 5, where:

[0158] vps_extension_type indicates the type of VPS extension. When the value of vps_extension_type is equal to 0, the syntax given above applies.

[0159] layer_id_plus1 indicates the layer_id for which the signaled syntax elements in the VPS extension are valid.

[0160] max_temporal_layers_minus1 plus 1 specifies the maximum number of temporal sub-layers that can exist in NAL units with layer_id_plus1 equal to the signaled layer_id_plus1 in the VPS.

[0161] new_profile_level_flag equal to 0 indicates that the profile and level associated with the value of layer_id_plus1 (equal to the value specified by profile_reference_layer_id_plus1) are signaled to be valid for NAL units with layer_id_plus1 specified in the VPS extension. new_profile_level_flag equal to 1 indicates that the profile and level for the specified value of layer_id_plus1 are explicitly signaled in the VPS extension.

[0162] profile_reference_layer_id_plus1 indicates the value of layer_id_plus1 for which the associated profile and level indications are valid for NAL units with layer_id_plus1 equal to the value of layer_id_plus1 indicated in the VPS extension.

[0163] profile_space, profile_idc, constraint_flags, and profile_compatibility_flag indicate information as specified in the current WD and the indicated values are valid for all NAL units with layer_id_plus1 equal to the value of layer_id_plus1 specified in the VPS extension data.

[0164] level_idc[i] indicates information as specified in the current WD and the indicated values are valid for all NAL units with layer_id_plus1 equal to the value of layer_id_plus1 specified in the VPS extension data and temporal_id equal to i.

[0165] num_reference_layers specifies the number of layers identified by layer_id that are to be used as reference in the decoding process.

[0166] direct_coding_dependency_layer_id_plus1 indicates the layer_id_plus1 of the NAL units that are to be used as reference in the decoding process.

[0167] The embodiments just described are very similar to the embodiments described with respect to FIG. 4 to FIG. 6 In this regard, FIG. 11 to FIG. 14 the same reference signs have been used in the description to identify corresponding syntax elements. In this regard, FIG. 7 the description of the embodiments of

[0168] In detail, according to the embodiments of FIG. 11 to FIG. 14 the operation point identifier 34 has been fixedly split into two components, namely, a first component 90 identifying the level of the temporal scalability axis in the form of the syntax element temporal_id, and a second component of this or the level being processed according to a further scalability axis, which is denoted 92. In this regard, the embodiments described with respect to FIG. 11 to FIG. 14 correspond to the previous embodiments, the second component 92 is to be further specified by a scalability axis descriptor.

[0169] The next difference concerns the knowledge about the number of scalability axes. In the above described embodiment of the VPS NAL unit 42, the network entity is informed about the number of scalability axes by means of the syntax element 94, scalability_type. In the above described embodiment of the VPS NAL unit 42, the network entity is informed about the number of scalability axes by means of the syntax element 94, scalability_type. FIG. 4 to FIG. 6 In the above described embodiment of the VPS NAL unit 42, the network entity is informed about the number of scalability axes by means of the syntax element 94, scalability_type. In the above described embodiment of the VPS NAL unit 42, the network entity is informed about the number of scalability axes by means of the syntax element 94, scalability_type.

[0170] As described in the following, all differences now highlighted can apply respectively and correspondingly, if the syntax element temporal_id 90 is not used, the network entity can derive from the fact that layer_id_plusl is equal to 1 that the operation point of the current packet represents a base layer according to all scalability axes.

[0171] In this regard, it should be noted that the selection of layer_id_plusl equal to 1 to represent the lowest level with respect to the other scalability axes than the temporal scalability axis is chosen only as an example and can for instance become layer_id_plusl equal to 0.

[0172] With respect to the above described embodiment of the VPS NAL unit 42, another difference concerns the scalability axes descriptor 38. The syntax element 94, scalability_type, of one scalar value identifies the number of scalability axes (above the temporal scalability axis) and the corresponding meaning of such (additional) scalability axes, whereas for each of this indicated number of (additional) scalability axes, the syntax portion 96 reveals a partitioning of the fixed length syntax element 60 under the components 92, the fields 56 and the reference field 52, this by assigning the bit order of this fixed length syntax element 60 to the individual (additional) scalability axes. FIG. 4 to FIG. 6 Another difference concerns the VPS packet and the fact that the VPS packet additionally comprises an operation point identifier reference list 98. For each field 56, this operation point identifier reference list 98 identifies the operation point identifier or the value of layer_id_plusl, which forms the necessary basis for the reconstruction of the operation point the field 56 is about. In other words, the operation point identifier reference list 98 identifies the combination 24 (see Fig. 2) of the operation point identifier or layer_id_plusl and the field 56, which is necessary for the reconstruction of the operation point the field 56 is about.

[0173] FIG. 1B or FIG. 3A and FIG. 3B ​all operation points contained in the individual combination 24 identified by the field 56. Instead, it is sufficient that the operation point reference list lists one or more of the reference operation points, whose corresponding combination 24 collectively results in the combination 24 of operation points identified by the individual field 56.

[0174] In other words, in this aspect, it is advantageous for efficient encoding of the hierarchy of layers and / or profile constraints that are assigned to the scalability layers, the data stream (in which the media content is encoded) comprises packets (e.g. NAL units), each packet comprising a packet type identifier nal_unit_type, which identifies a packet type associated with the respective packet from a plurality of packet types, wherein each packet (e.g. all NAL units) having a packet type associated therewith from a first set of the plurality of packet types comprises an operation point identifier layer_id_plusl, which identifies an operation point associated with the respective packet from a plurality of operation points within a scalability space spanned by n scalability axes, and 0 < n, the operation point having an ordered operation point order defined therein, wherein each packet (e.g. non-VPS NAL units) having a packet type associated therewith from a second set of the first set of packet types additionally carries data, which together with the packets having an operation point associated therewith facilitates an encoded representation of the media content at the operation point associated with the respective packet, the operation point being prior to or equal to the operation point associated with the respective packet, wherein each packet (e.g. VPS NAL units) having a predetermined packet type associated therewith disjoint (i.e. outside) from the second set, has a scalability axes descriptor (e.g. scalability_type and layer_id_plusl_dim_len), which defines the number of one or more scalability axes and their syntax meaning.

[0175] In another example of a data stream, each of the packets having a packet type associated therewith further comprises a sequence of profile and / or level descriptors vps_extension_data associated with at least a subset of the plurality of operation points (e.g., leading operation points in the order operation point order), each profile and / or level descriptor defining a set of available coding options (e.g., a profile) and / or a range of available syntax element values (e.g., a level), the syntax of the packet having an operation point associated therewith associated with the respective profile and / or level descriptor adhering to the set of available coding options and / or the range of available syntax element values, wherein a first set of the profile and / or level descriptors explicitly indicates a definition of the set of available coding options and / or the range of available syntax element values (e.g., new_profile_level_flag = 1), and a second, disjoint, non-empty set of the profile and / or level descriptors indicates a definition of the set of available coding options and / or the range of available syntax element values by reference to another, previous profile and / or level descriptor (e.g., new_profile_level_flag = 0).

[0176] In another example of a data stream, the sequence of profile and / or level descriptors is sequentially associated with at least a subset of the plurality of operation points (e.g., leading operation points in the order operation point order).

[0177] In another example of a data stream, each profile and / or level descriptor has a field (e.g., layer_id_plusl) that identifies the operation point (i.e., with 4) to which the respective profile and / or level descriptor is associated, the order operation point order being derivable from the sequence of profile and / or level descriptors.

[0178] In another example of a data stream, in each profile and / or level descriptor, the field identifies the operation point to which the respective profile and / or level descriptor is associated using a bit sequence common to the operation point identifiers, the bit sequence identifying the operation point to which the respective profile and / or level descriptor is associated.

[0179] In another example of a data stream, all operation point identifiers have a fixed bit length syntax element (e.g., layer_id_plusl), and the scalability access descriptor indicates n max_dim and an association of n partitions of the fixed bit length syntax element layer_id_plusl_dim_len and n scalability axes, wherein the bit sequence is common to the fixed bit length syntax element.

[0180] In another example of the data stream, a second, disjoint, non-empty set of profiles and / or hierarchy descriptors indicates the definition of the set of available coding options and / or the range of available syntax element values by referencing another, previous profile and / or hierarchy descriptor, using explicit signaling of the fields of the referenced profile and / or hierarchy descriptor, or using a related indicator defined in rank position units in the order of the operation points (where the second alternative is the one described above).

[0181] According to another aspect of the data stream, all operation point identifiers have a fixed bit length syntax element, and the scalability axes descriptor indicates n and the association of n partitions of the fixed bit length syntax element with n scalability axes, so that the n-th part is a sequential measure of the position of the corresponding operation point along the n-th scalability axis.

[0182] According to another aspect of the data stream, the scalability axes descriptor (e.g. scalability_type) includes a syntax element whose possible states are respectively associated with different combinations of values of n and corresponding semantic meanings of the n scalability axes (i.e. a function maps {n, meaning(l),..., meaning(n)} to {1... possible states of max.scalability_type}).

[0183] In another data stream, media content is encoded therein, the data stream comprising packets, each packet comprising a packet type identifier identifying a packet type associated with the respective packet from a plurality of packet types, wherein each packet having a packet type associated therewith from a first set of the plurality of packet types comprises an operation point identifier identifying an operation point associated with the respective packet from a plurality of operation points spanning a scalability space of n scalability axes, and 0 < n, the operation points having an order operation point order defined therein, wherein each packet having a packet type associated therewith from a second set of the first set of packet types additionally carries data that, together with packets having an operation point associated therewith, contributes to an encoded representation of the media content at the operation point associated with the respective packet, the operation point being prior to, or equal to, the operation point associated with the respective packet, wherein each packet having a predetermined packet type associated therewith that is disjoint from the second set (i.e., outside the second set) comprises a sequence of profile and / or level descriptors associated with at least a subset of the plurality of operation points (e.g., leading operation points in the order operation point order) respectively, each profile and / or level descriptor defining a set of available coding options and / or a range of available syntax element values, the syntax of packets having an operation point associated therewith associated with the respective profile and / or level descriptor adhering to the set of available coding options and / or the range of available syntax element values, wherein a first set of the profile and / or level descriptors explicitly show definitions of sets of available coding options and / or ranges of available syntax element values, and a second, disjoint, non-empty set of the profile and / or level descriptors show definitions of sets of available coding options and / or ranges of available syntax element values by reference to another, prior profile and / or level descriptor.

[0184] In another example of a data stream, the sequence of profile and / or level descriptors is sequentially associated with at least a subset of the plurality of operation points (e.g., leading operation points in the order operation point order) in the order operation point order.

[0185] In another example of a data stream, each profile and / or level descriptor has a field identifying the operation point with which the respective profile and / or level descriptor is associated.

[0186] In another example of a data stream, in each profile and / or level descriptor, the field identifies the operation point with which the respective profile and / or level descriptor is associated using a bit sequence common to the operation point identifiers, the bit sequence identifying the operation point with which the respective profile and / or level descriptor is associated.

[0187] In another example of a data stream, all operation point identifiers have a fixed bit length syntax element, and the scalability access descriptor indicates n max_dim and the association of n partitioning layer_id_plusl_dim_len fixed bit length syntax elements and n scalability axes, where the bit sequence is common to the fixed bit length syntax elements.

[0188] In another example of a data stream, a second, disjoint, non-empty set of profiles and / or level descriptors indicates the definition of the set of available coding options and / or the range of available syntax element values by referencing another, previous profile and / or level descriptor, using explicit signaling of fields of the referenced profile and / or level descriptor, or using a related indicator defined in rank position units of the order of operation points.

[0189] In another example of a data stream, all operation point identifiers have a flag indicating whether they belong to a first set of profiles and / or level descriptors explicitly signaling the definition of the set of available coding options and / or the range of available syntax element values and a second, disjoint, non-empty set of profiles and / or level descriptors indicating the definition of the set of available coding options and / or the range of available syntax element values by referencing another, previous profile and / or level descriptor.

[0190] Another aspect has a network entity (e.g., a decoder or MANE) configured to process a data stream into which media content is encoded, the data stream comprising packets, each packet comprising a packet type identifier identifying a packet type associated with the respective packet from a plurality of packet types, wherein each packet having a packet type associated with a first set of packet types from the plurality of packet types comprises an operation point identifier identifying an operation point associated with the respective packet from a plurality of operation points within a scalability space spanned by n scalability axes, and 0 < n, the operation points having an order of operation points defined therein, wherein each packet having a packet type associated with any packet type from a second set of the first set of packet types additionally carries data that, together with packets having an arbitrary operation point associated therewith, contributes to an encoded representation of the media content at the operation point associated with the respective packet, the arbitrary operation point being prior to the operation point associated with the respective packet, or equal to the operation point associated with the respective packet, wherein the network entity is configured to read a scalability axes descriptor from packets having a predetermined packet type disjoint (i.e., outside) from the second set and interpret the operation point identifier dependent on the scalability axes descriptor, wherein the scalability axes descriptor defines the number of one or more scalability axes and their semantic meaning.

[0191] In another example, a network entity is configured to read from a sequence of profiles and / or hierarchy descriptors having predetermined packet types associated therewith that are disjoint (i.e., outside of) a second set, each profile and / or hierarchy descriptor defining a set of available coding options and / or a range of available syntax element values, packets having operation points associated therewith associated with at least a subset of the plurality of operation points (e.g., leading operation points in a sequential operation point order), wherein the network entity is configured to explicitly read the definition of the set of available coding options and / or the range of available syntax element values from a first set of the profiles and / or hierarchy descriptors in response to a cue in the data stream, and to read a reference to another, previous profile and / or hierarchy descriptor for a second, disjoint, non-empty set of the profiles and / or hierarchy descriptors, and to derive the definition of the set of available coding options and / or the range of available syntax element values for the second, disjoint, non-empty set of the profiles and / or hierarchy descriptors from the individually referenced previous profile and / or hierarchy descriptor.

[0192] In another example of a network entity, the sequence of profiles and / or hierarchy descriptors are sequentially associated with at least a subset of the plurality of operation points (e.g., leading operation points in a sequential operation point order).

[0193] In another example of a network entity, each profile and / or hierarchy descriptor has a field that identifies the operation point to which the respective profile and / or hierarchy descriptor is associated.

[0194] In another example of a network entity, in each profile and / or hierarchy descriptor, the field identifies the operation point to which the respective profile and / or hierarchy descriptor is associated using a bit sequence common to the operation point identifiers that identifies the operation point to which the respective profile and / or hierarchy descriptor is associated.

[0195] In another example of a network entity, all operation point identifiers have a fixed bit length syntax element, and the scalability access descriptor indicates n max_dim and an association of n partitioning regions layer_id_plusl_dim_len and n scalability axes for the fixed bit length syntax element, wherein the bit sequence is common to the fixed bit length syntax element.

[0196] In another example of the network entity, a second, disjoint, non-empty set of profiles and / or hierarchy descriptors indicates the definition of the set of available coding options and / or the range of available syntax element values by referencing another, previous profile and / or hierarchy descriptor, using explicit notation of fields of the referenced profile and / or hierarchy descriptor, or using a related indicator defined in rank position units in order of operation point order.

[0197] In another example of the network entity, all operation point identifiers have a fixed bit length syntax element, and the scalability access descriptor indicates n max_dim and the association of n partitions layer_id_plus1_dim_len of the fixed bit length syntax element and n scalability axes, so the n-th part is a sequential measure of the position of the corresponding operation point along the n-th scalability axis.

[0198] In another example, the network entity is configured to select among packets having any packet type from a first set of packet types based on interpretation.

[0199] Another example can have a network entity (e.g., a transcoder or MANE) to process a data stream in which media content is encoded, the data stream comprising packets, each packet comprising a packet type identifier identifying a packet type associated with the respective packet from a plurality of packet types, wherein each packet having a packet type associated therewith from a first set of the plurality of packet types comprises an operation point identifier identifying an operation point associated with the respective packet from a plurality of operation points spanning a scalability space of n scalability axes, and 0 < n, the operation points having an order of operation point order defined therein, wherein each packet having a packet type associated therewith from a second set of the first set of packet types additionally carries data that, together with packets having an operation point associated therewith of any operation point preceding or equal to the operation point associated with the respective packet, contributes to an encoded representation of the media content at the operation point associated with the respective packet, wherein the network entity is configured to read from packets having a predetermined packet type associated therewith disjoint (i.e., outside) the second set a sequence of profiles and / or level descriptors in order of operation point order sequentially associated with at least a subset of the plurality of operation points (e.g., leading operation points in order of operation point order), each profile and / or level descriptor defining a set of available coding options and / or a range of available syntax element values, syntax of packets having an operation point associated therewith associated with the respective profile and / or level descriptor adhering to the set of available coding options and / or the range of available syntax element values, wherein a first set of the profiles and / or level descriptors explicitly show definitions of sets of available coding options and / or ranges of available syntax element values, wherein the network entity is configured to explicitly read definitions of sets of available coding options and / or ranges of available syntax element values from the first set of profiles and / or level descriptors in response to a signal in the data stream, and to read references to another, previous profile and / or level descriptor for a second, disjoint, non-empty set of the profiles and / or level descriptors, and to derive definitions of sets of available coding options and / or ranges of available syntax element values for the second, disjoint, non-empty set of profiles and / or level descriptors from the individually referenced previous profile and / or level descriptor.

[0200] In another embodiment, the network entity is configured to select among packets having a packet type associated therewith from a first set of the plurality of packet types based on the associated set of available coding options and / or range of available syntax element values.

[0201] The following differences to the above embodiments would be feasible. For example, instead of having one profile / generic descriptor for exactly one operation point or layer ID, it can be signaled in the data stream, such as in the VPS, to group this operation point or layer ID into a set of layers, where each profile / generic descriptor is associated with an individual one of the sets. For example, a field of flags can be signaled to signal the grouping: for each set, there can be one flag for each operation point to signal whether the respective operation point belongs to the respective set. Thus, instead of referring to any layer ID in field 56, field 56 of the profile / generic descriptor can be indexed to one of the signaled sets from layer IDs, i.e., from sets of operation points. Thus, a reference to another profile / generic descriptor would be done via signaling another set of layers in field 52, instead of signaling an individual layer ID therein.

[0202] Furthermore, with respect to scalability axis descriptor 38, the above can be signaled differently: a scalability type field consisting of a sequence of flags, each associated with a certain scalability type, can be present in the data stream, e.g., in the VPS, and signal whether an individual scalability type is present or not. In the order of the flags within this field, the flags being set will thus signal the number of scalability axes and their semantic meaning. Subdivision of the bit sequence of layer ID field 34 into fragments can be done using a length indicator for each scalability type, i.e., for each flag being set in the scalability type field, just as is done by layer_id_plusl_dim_len. The scalability type field and layer_id_plusl_dim_len together form scalability axis descriptor 38.

[0203] In the above, a list of reference operation point identifiers 98 and a field 56 identifying the operation points (or sets of operation points) to which the reference operation point identifiers 98 of the list 98 are associated are used to indicate the order of operation points, i.e. the dependencies between the layers. This allows for the possibility that an operation point depends on more than one previous operation point (in the order of their operation point identifiers) of the operation points. In use cases with multiple camera views, for example, it is beneficial to signal multiple reference views (i.e. multiple values of nuh layer id) for each coded view (i.e. individual values of nuh layer id) because depending on the translation of the multiple cameras with respect to each other, occlusion and exposure of corresponding image regions in neighboring camera views can occur. Thus, referencing multiple camera views can lead to higher coding efficiency because exposed image regions (i.e. texture) can be predicted rather than intra coded from any of the referenced camera views. The coding benefits are not limited to referencing within the same scalability dimension, e.g. a depth view (with respect to the currently coded camera view or another camera view) can be utilized to remove the motion vectors of the currently coded dependent camera view. However, the signaling that one operation point depends on more than one operation point with lower operation point identifiers associated therewith can also be implemented in a different way. For example, for each operation point, a flag can signal which of the operation points with layer IDs ahead (i.e. having lower operation point identifiers) the respective operation point depends on. The field can be signaled in a compressed form: N operation points occurring in the bitstream have an order defined in them, which is defined by their operation point identifiers. Thus the sequence of operation points can be scanned in the sense of operation point identifiers from the second but lowest operation point to the highest operation point, where each operation point with a lower operation point identifier inserts one flag into the VPS, requiring N(N-1) / 2 bits. That is, for each of the multiple operation points (or at least a subset of the operation points), there can be syntax in the VPS identifying the other operation points of the multiple operation points identified by lower operation point identifiers, such as a list of reference layer IDs 98 or a sequence of flags with respect to previous operation points. In other words, the syntax identifies those operation points for which the respective operation point the syntax belongs to is an extension of, thus identifying the order of operation points.

[0204] While some aspects have been described in the context of a device, it is clear that such aspects also represent a description, for corresponding methods, wherein a block or device corresponds to a method block or a method step. Similarly, aspects described in the context of a method step also represent a description for a corresponding block or item or feature of a corresponding device. Some or all of the method steps can be performed by (or using) a hardware apparatus, like, for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, one or more of the most important method steps can be performed by such an apparatus.

[0205] The encoded data stream of the present invention can be stored on a digital storage medium, or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium (such as the Internet).

[0206] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or in software. This can be performed using a digital storage medium (e.g., a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory) on which electronically readable control signals are stored, which cooperates (or is capable of cooperating) with a programmable computer system so that the respective method is performed. Thus, the digital storage medium can be computer-readable.

[0207] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0208] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine-readable carrier.

[0209] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0210] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0211] A further embodiment of the methods according to the invention is therefore a data carrier (or a digital storage medium or a computer-readable medium) on which is recorded the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are generally tangible and / or non-transitory.

[0212] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection, for example via the Internet.

[0213] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0214] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0215] According to another embodiment of the application comprises an apparatus or a system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0216] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array can cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware-implemented device.

[0217] The embodiments described above are merely illustrative of the principles of this application. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the appended claims and not by the specifics of the description and illustrations presented herein.

[0218] The application can also be configured in the following way.

[0219] 1. A data stream into which media content is encoded, the data stream comprising packets (12), each packet (12) comprising a packet type identifier (16) identifying a packet type associated with the respective packet from a plurality of packet types,

[0220] wherein each packet having a packet type from a first set of packet types associated with each packet (12) from the plurality of packet types comprises an operation point identifier (34) identifying an operation point (18) associated with the respective packet from a plurality of operation points within a scalability space (20) spanned by n scalability axes (22a, 22b), 0 < n, the operation points having an ordered operation point order defined among the operation points,

[0221] wherein each packet having a packet type from a second set of packet types of the first set of packet types associated with each packet (12) additionally carries data (36) which, together with the packet having an arbitrary operation point associated with the packet, contributes to an encoded representation of the media content at the operation point associated with the respective packet, the arbitrary operation point preceding or being equal to the operation point associated with the respective packet,

[0222] wherein each of the packets having the predetermined packet type associated with each of the packets (44) has a scalability axes descriptor (38) defining the number n of the one or more scalability axes and semantic implications.

[0223] 2. The data stream of 1, wherein each of the packets having the predetermined packet type associated with each of the packets further comprises a sequence (54) of profile and / or level descriptors (44) associated with one or more of at least a subset of the plurality of operation points, each profile and / or level descriptor (44) defining a set of available coding options and / or available syntax element value ranges, syntax of a packet having associated therewith an operation point associated with the respective profile and / or the respective level descriptor adhering to the set of available coding options and / or the available syntax element value ranges, wherein a first set of the profile and / or level descriptors explicitly specifies definitions of the set of available coding options and / or the available syntax element value ranges, and a second, disjoint, non-empty set of the profile and / or level descriptors specifies definitions of the set of available coding options and / or the available syntax element value ranges by reference to another profile and / or level descriptor.

[0224] 3. The data stream of 2, wherein each profile and / or level descriptor (44) has a field (56) identifying the operation point (18) associated with the respective profile and / or the respective level descriptor (44).

[0225] 4. The data stream of 3, wherein the second, disjoint, non-empty set of the profile and / or level descriptors (44) specifies definitions of the set of available coding options and / or the available syntax element value ranges by reference to another profile and / or level descriptor and using the explicit specification (52) of the field of the referenced profile and / or level descriptor.

[0226] 5. The data stream of 1, wherein for each of at least a subset of the plurality of operation points, each of the packets having the predetermined packet type associated with each of the packets further comprises syntax (98) identifying other operation points of the plurality of operation points identified by a lower operation point identifier, the respective operation point being an extension of the plurality of operation points, thereby indicating the sequential operation point order.

[0227] 6. The data stream of 3, wherein each field (56) identifies the respective operation point using a bit sequence (60) common to the operation point identifiers, the bit sequence identifying the respective operation point.

[0228] 7. The data stream of 6, wherein all operation point identifiers have a fixed bit length syntax element (60), and the scalability axes descriptor (38) indicates n and the association of n bins of the fixed bit length syntax element (60) with the n scalability axes, wherein the bit sequence is common to the fixed bit length syntax element.

[0229] 8. The data stream of 6, wherein all operation point identifiers (34) have a fixed bit length syntax element (60) and a context syntax element (64) that indicates n and the association of n bins of the fixed bit length syntax element with the n scalability axes, wherein the bit sequence is common to the fixed bit length syntax element.

[0230] 9. The data stream of 1, wherein all operation point identifiers (34) have a fixed bit length syntax element (60), and the scalability axes descriptor (38) indicates n and the association of n bins of the fixed bit length syntax element with the n scalability axes such that the nth portion is a sequential measure of the position of the respective operation point along the nth scalability axis.

[0231] 10. The data stream of 1, wherein all operation point identifiers (34) have a fixed bit length syntax element (60) and a context syntax element (64) that indicates n and the association of n bins of the fixed bit length syntax element with the n scalability axes such that the nth portion is a sequential measure of the position of the respective operation point along the nth scalability axis.

[0232] 11. The data stream of 1, wherein the scalability axes descriptor (38) includes a syntax element (94) whose possible states are individually associated with different combinations of a value of n and a semantic meaning of a corresponding semantic meaning of the n scalability axes.

[0233] 12. A data stream into which media content is encoded, the data stream comprising packets (12), each packet including a packet type identifier (16) that identifies a packet type associated with the respective packet from among a plurality of packet types,

[0234] wherein each of the packets (12) having associated therewith a packet type from the first set of packet types identifies an operation point (18) associated with the respective packet from a plurality of operation points within a scalability space (20) spanned by n scalability axes (22a, 22b), 0 < n, the operation points having an ordinal operation point order defined among the operation points,

[0235] wherein each of the packets (12) having associated therewith a packet type from the first set of packet types includes an operation point identifier (34) identifying an operation point (18) associated with the respective packet from a plurality of operation points within a scalability space (20) spanned by n scalability axes (22a, 22b), 0 < n, the operation points having an ordinal operation point order defined among the operation points,

[0236] wherein each of the packets (12) having associated therewith a packet type from the first set of packet types includes an operation point identifier (34) identifying an operation point (18) associated with the respective packet from a plurality of operation points within a scalability space (20) spanned by n scalability axes (22a, 22b), 0 < n, the operation points having an ordinal operation point order defined among the operation points,

[0237] 13. The data stream of 12, wherein each of the profile and / or level descriptors (44) has a field identifying the operation point associated with the respective profile and / or the respective level descriptor.

[0238] 14. The data stream of 13, wherein the second, disjoint, non-empty set of the profile and / or level descriptors (44) reference another profile and / or level descriptor and use the explicit signaling (52) of the field of the referenced profile and / or level descriptor to signal the definition of the set of available coding options and / or the range of available syntax element values.

[0239] 15. The data stream of 12, wherein, for each of at least a subset of the plurality of operation points, each of the packets having the predetermined packet type associated with each of the packets further comprises a syntax (98) that identifies other operation points of the plurality of operation points identified by a lower operation point identifier, the respective operation point being an extension of the plurality of operation points, thereby indicating the sequential operation point order.

[0240] 16. The data stream of 12, wherein each field (56) identifies the respective operation point using a bit sequence (60) common to the operation point identifiers, the bit sequence identifying the respective operation point.

[0241] 17. The data stream of 16, wherein all operation point identifiers have a fixed bit length syntax element (60), and the scalability axis descriptor (38) indicates n and an association of n partitions of the fixed bit length syntax element (60) with the n scalability axes, wherein the bit sequence is common to the fixed bit length syntax element.

[0242] 18. The data stream of 16, wherein all operation point identifiers (34) have a fixed bit length syntax element (60) and a context syntax element (64) that indicates n and an association of n partitions of the fixed bit length syntax element with the n scalability axes, wherein the bit sequence is common to the fixed bit length syntax element.

[0243] 19. The data stream of 12, wherein all operation point identifiers (34) have a flag that indicates whether belonging to the first set of the profile and / or the level descriptor explicitly indicating the definition of the set of available coding options and / or the available syntax element value ranges and / or the second, disjoint, non-empty set of the profile and / or the level descriptor indicating the definition of the set of available coding options and / or the available syntax element value ranges by reference to another, previous profile and / or level descriptor.

[0244] 20. A network entity for processing a data stream into which media content is encoded, the data stream comprising packets (12), each packet (12) comprising a packet type identifier (16) that identifies a packet type associated with the respective packet from a plurality of packet types,

[0245] wherein each of the packets (12) having associated therewith a packet type from a first set of packet types (22a, 22b) includes an operation point identifier (34) identifying an operation point (18) associated with the respective packet from a plurality of operation points within a scalability space (20) spanned by n scalability axes (22a, 22b), 0 < n, the operation points having an ordinal operation point order defined among the operation points,

[0246] wherein each of the packets (12) having associated therewith an arbitrary packet type from a second set of packet types (22a, 22b) of the first set of packet types additionally carries data (36) which, together with the packet having associated therewith an arbitrary operation point preceding or equal to the operation point associated with the respective packet, contributes to an encoded representation of the media content at the operation point associated with the respective packet,

[0247] wherein the network entity is configured to:

[0248] read a scalability axes descriptor (38) from a packet (12) having a predetermined packet type disjoint from the second set associated therewith, the scalability axes descriptor defining the number n and semantic meaning of the one or more scalability axes, and

[0249] interpret the operation point identifier according to the scalability axes descriptor.

[0250] 21. The network entity of 20, wherein the network entity is configured to read from the packets having the predetermined packet type disjoint from the second set associated therewith a sequence (54) of profile and / or level descriptors (44) individually associated with at least a subset of the plurality of operation points, each profile and / or level descriptor (44) defining a set of available coding options and / or available syntax element value ranges, syntax of a packet having associated therewith the operation point associated with the respective profile and / or the respective level descriptor adhering to the set of available coding options and / or the available syntax element value ranges, wherein the network entity is configured to explicitly read the definition of the set of available coding options and / or the available syntax element value ranges from a first set of the profile and / or the level descriptors in response to a hint in the data stream and to read a reference to another profile and / or level descriptor for a second, disjoint, non-empty set of the profile and / or the level descriptors and to derive the definition of the set of available coding options and / or the available syntax element value ranges for the second, disjoint, non-empty set of the profile and / or the level descriptors from the respective referenced profile and / or level descriptor.

[0251] 22. The network entity of 21, wherein each profile and / or hierarchy descriptor (44) has a field (56) that identifies the operating point (18) associated with the respective profile and / or the respective hierarchy descriptor (44).

[0252] 23. The network entity of 22, wherein the second, disjoint, non-empty set of profiles and / or hierarchy descriptors (44) reference another profile and / or hierarchy descriptor and use an explicit indication (52) of the field of the referenced profile and / or hierarchy descriptor to indicate the definition of the set of available coding options and / or the available syntax element value ranges.

[0253] 24. The network entity of 21, wherein the network entity is configured to further read, for each of at least a subset of the plurality of operating points, a syntax (98) from the packet having the predetermined packet type associated with the packet, the syntax identifying other operating points of the plurality of operating points identified by a lower operating point identifier, the respective operating point being an extension of the plurality of operating points, thereby indicating the sequential operating point order.

[0254] 25. The network entity of 24, wherein each field (54) identifies the respective operating point using a sequence of bits (60) common to the operating point identifiers, the sequence of bits identifying the respective operating point.

[0255] 26. The network entity of 25, wherein all operating point identifiers have a fixed bit length syntax element (60) and the scalability axis descriptor (38) indicates n and n partitions of the fixed bit length syntax element (60) and an association with the n scalability axes, wherein the sequence of bits is common to the fixed bit length syntax element.

[0256] 27. The network entity of 25, wherein all operating point identifiers (34) have a fixed bit length syntax element (60) and a context syntax element (64) indicating n and n partitions of the fixed bit length syntax element and an association with the n scalability axes, wherein the sequence of bits is common to the fixed bit length syntax element.

[0257] 28. The network entity of 20, wherein all operation point identifiers (34) have a fixed bit length syntax element (60), and the scalability axes descriptor (38) indicates n and an association of n bins of the fixed bit length syntax element with the n scalability axes such that the nth bin is a sequential quantity measure of a position of the respective operation point along the nth scalability axis.

[0258] 29. The network entity of 20, wherein all operation point identifiers (34) have a fixed bit length syntax element (60) and a context syntax element (64), wherein the network entity is configured to use the context syntax element to determine n and to associate n bins of the fixed bit length syntax element with the n scalability axes such that the nth bin is a sequential quantity measure of a position of the respective operation point along the nth scalability axis.

[0259] 30. The network entity of 20, wherein the network entity is configured to select among the packets (12) having associated therewith any packet type from the first set of the plurality of packet types based on interpretation.

[0260] 31. A network entity for processing a data stream into which media content is encoded, the data stream comprising packets (12), each packet (12) comprising a packet type identifier (16) identifying a packet type associated with the respective packet from a plurality of packet types,

[0261] wherein each packet having associated therewith a packet type from a first set of the plurality of packet types comprises an operation point identifier (34) identifying an operation point (18) associated with the respective packet from a plurality of operation points within a scalability space (20) spanned by n scalability axes (22a, 22b), and 0 < n, the operation points having a sequential operation point order defined among the operation points,

[0262] wherein each packet having associated therewith any packet type from a second set of the first set of packet types additionally carries data (36) that, together with the packet having associated therewith any operation point that precedes or is equal to the operation point associated with the respective packet, contributes to an encoded representation of the media content at the operation point associated with the respective packet,

[0263] wherein the network entity is configured to:

[0264] reading, from a package having a package type associated with the second set disjoint from the first set of package types, a sequence of configuration files and / or level descriptors associated with one or more of at least a subset of the plurality of operation points, each configuration file and / or level descriptor defining a set of available coding options and / or a range of available syntax element values for a package associated with the respective configuration file and / or the respective level descriptor to comply with, wherein the network entity is configured to explicitly read the definition of the set of available coding options and / or the range of available syntax element values from a first set of the configuration files and / or the level descriptors in response to a hint in the data stream, and to read a reference to another configuration file and / or level descriptor for a second, disjoint, non-empty set of the configuration files and / or the level descriptors, and to derive the definition of the set of available coding options and / or the range of available syntax element values for the second, disjoint, non-empty set of the configuration files and / or the level descriptors from the respective referenced configuration file and / or level descriptor.

[0265] 32. The network entity of 31, wherein the network entity is configured to select among the packages having an arbitrary package type from the first set of package types associated with the packages based on the associated set of available coding options and / or the associated range of available syntax element values.

[0266] 33. A method for processing a data stream into which media content is encoded, the data stream comprising packages (12), each package (12) comprising a package type identifier (16) identifying a package type associated with the respective package from a plurality of package types,

[0267] wherein each package having a package type from a first set of package types from the plurality of package types associated with each package (12) comprises an operation point identifier (34) identifying an operation point (18) associated with the respective package from a plurality of operation points within a scalability space (20) spanned by n scalability axes (22a, 22b) and 0 < n, the operation points having an ordinal operation point order defined among the operation points,

[0268] wherein each of the packets (12) having associated therewith a second set of packet types from the first set of packet types carries additionally data (36) with the packet having associated therewith any operation point prior to or equal to the operation point associated with the respective packet that contributes to the encoded representation of the media content at the operation point associated with the respective packet,

[0269] wherein the method comprises:

[0270] reading a scalability axes descriptor (38) from a packet having associated therewith a predetermined packet type disjoint from the second set of packet types (12), the scalability axes descriptor defining a number n of the one or more scalability axes and semantic meaning, and

[0271] interpreting the operation point identifier in accordance with the scalability axes descriptor.

[0272] 34. A method for processing a data stream into which media content is encoded, the data stream comprising packets (12), each packet (12) comprising a packet type identifier (16) identifying a packet type associated with the respective packet from a plurality of packet types,

[0273] wherein each of the packets (12) having associated therewith a packet type from a first set of packet types of the plurality comprises an operation point identifier (34) identifying an operation point (18) associated with the respective packet from a plurality of operation points within a scalability space (20) spanned by n scalability axes (22a, 22b), and 0 < n, the operation points having an ordered operation point order defined among the operation points,

[0274] wherein each of the packets (12) having associated therewith a second set of packet types from the first set of packet types carries additionally data (36) with the packet having associated therewith any operation point prior to or equal to the operation point associated with the respective packet that contributes to the encoded representation of the media content at the operation point associated with the respective packet,

[0275] wherein the method comprises:

[0276] reading, from a package having a predetermined package type associated therewith that is disjoint from the second set, a sequence of configuration files and / or level descriptors associated with one or more of at least a subset of the plurality of operation points, each configuration file and / or level descriptor defining a set of available coding options and / or a range of available syntax element values, syntax of a package having associated therewith an operation point associated with the respective configuration file and / or the respective level descriptor adhering to the set of available coding options and / or the range of available syntax element values, wherein the network entity is configured to explicitly read, responsive to a hint in the data stream, a definition of the set of available coding options and / or the range of available syntax element values from a first set of the configuration files and / or the level descriptors, and to read, for a second, disjoint, non-empty set of the configuration files and / or the level descriptors, a reference to another configuration file and / or level descriptor, and to derive, from the respective referenced configuration file and / or level descriptor, the definition of the set of available coding options and / or the range of available syntax element values for the second, disjoint, non-empty set of the configuration files and / or the level descriptors.

[0277] 35. A computer program for performing the processing method according to 33 and 34 when the computer program is run on a computer or processor.

[0278] References

[0279] [1] Thomas Wiegand, Gary J. Sullivan, Gisle Bjontegaard, Ajay Luthra, "Overview of the H.264 / AVC Video Coding Standard", IEEE Transactions on Circuits and Systems for Video Technology, Volume 13, Issue 7, July 2003.

[0280] [2] JCT-VC, "High-Efficiency Video Coding (HEVC) text specification Working Draft 6", JCTVC-H1003, February 2012.

[0281] [3] ISO / IEC 13818-1 : MPEG-2 Systems Specification.

Claims

1. A decoder for decoding a layered video data stream, the layered video data stream comprising NAL units, each of the NAL units comprising a NAL unit header having a layer identifier, the layer identifier associating the corresponding NAL unit with one of a plurality of layers, the decoder being configured to: reading a status identifier from the layered video data stream; and Where the status identifier indicates a split, the tier identifier is partitioned by associating bits of the tier identifier with different scalability dimensions.

2. An encoder for encoding a layered video data stream, the layered video data stream comprising NAL units, each of the NAL units comprising a NAL unit header having a layer identifier, the layer identifier associating the respective NAL unit with one of a plurality of layers, the encoder being configured to: A status identifier is written in the layered video data stream that, if splitting is indicated, indicates to a coder the layer identifier to be partitioned by associating bits of the layer identifier with different scalability dimensions.

3. A decoding method for decoding a layered video data stream, the layered video data stream comprising NAL units, each of the NAL units comprising a NAL unit header having a layer identifier, the layer identifier associating the corresponding NAL unit with one of a plurality of layers, the method comprising: reading a status identifier from the layered video data stream; as well as Where the status identifier indicates a split, the tier identifier is partitioned by associating bits of the tier identifier with different scalability dimensions.

4. A method for encoding a layered video data stream, the layered video data stream comprising NAL units, each of the NAL units comprising a NAL unit header having a layer identifier, the layer identifier associating the corresponding NAL unit with one of a plurality of layers, the method comprising: A status identifier is written in the layered video data stream that, if splitting is indicated, indicates to a coder the layer identifier to be partitioned by associating bits of the layer identifier with different scalability dimensions.

5. A method of storing video, comprising storing a layered video data stream, the layered video data stream comprising NAL units, each of the NAL units comprising a NAL unit header having a layer identifier, the layer identifier associating the corresponding NAL unit with one of a plurality of layers, and the layered video data stream comprising a status identifier that, in the case of indicating splitting, indicates to a decoder the layer identifier to be split by associating bits of the layer identifier with different scalability dimensions.

Images