Methods, apparatuses, and systems for signaling preselection

Abstract

A method of processing a media stream is described. The method comprises receiving a packetized media stream according to a predefined transport format, wherein the packetized media stream comprises a plurality of layered boxes, the plurality of layered boxes each being associated with a respective box type identifier, and wherein the plurality of boxes comprises one or more 5-track boxes referencing respective tracks indicating media components of the media stream; determining whether the media stream comprises a preselected preselected related box indicating a preselected predefined type, wherein the preselection corresponds to a media presentation to a user; and if it is determined that the media stream comprises the preselected related box: analyzing metadata information corresponding to the preselected related box, the metadata information indicating characteristics of the preselection; identifying one or more tracks in the packetized media stream that contribute to the preselection based on the metadata information; and providing the one or more tracks for downstream processing according to the given preselection.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to the following prior applications: U.S. Provisional Application No. 63 / 216,029 (reference number: D21064USP1), filed June 29, 2021, and U.S. Provisional Application No. 63 / 297,473 (reference number: D21064USP2), filed January 7, 2022, which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the general field of audio and / or video encoding and / or decoding, and more specifically to methods, apparatus and systems for preselection and processing corresponding to media presentation to a user via signal transmission. Background Technology

[0004] Generally, with the advent of Next Generation Audio (NGA) or similar video technologies, the overall audio experience is no longer delivered as a pre-generated, single instance of media component. Instead, individual semantic objects can be delivered separately, providing an efficient way for end users to customize content according to their preferences.

[0005] For example, dialogues can be provided in multiple languages ​​because additional optional audio components and language selections can be implemented by combining different audio components or by different balances between components.

[0006] Broadly speaking, for various reasons, including the possibility of saving transmission bandwidth for users who do not need certain parts of the media assets, modern video compression schemes can often take advantage of the possibility of distributing the total available media data across multiple streams.

[0007] As technical personnel can understand and recognize, in any case, media players will typically rely on accompanying metadata to guide how to render components to produce a predefined user experience.

[0008] For transport, multiple content components (or CCs) can be multiplexed into a single primary stream, or these components can be distributed across multiple primary streams.

[0009] Typically, not all available components should be presented at the same time; instead, only certain combinations of these components can provide the desired user experience.

[0010] Metadata in various multiplexing and transport layers provides clients (or end users) with the necessary knowledge about all available components. Additionally, in some cases, data may be required to determine which underlying streams to download and decode.

[0011] The International Organization for Standardization (ISO) includes and specifies a basic media file format commonly known as ISOBMFF. Specifically, ISOBMFF is specified by ISO through ISO / IEC 14496-12 MPEG-4 Part 12. Generally, it defines a common structure for time-based multimedia files such as video and / or audio. While most existing multiplexing formats may already provide a means of annotating components with the corresponding properties of the components within the file, ISO BMFF multiplexing appears to lack, to some extent, the ability to transmit the overall experience composed of combinations of content components as signals. In short, it appears to fall short compared to some other standards that seem to introduce pre-selection concepts, such as MPEG-DASH (ISO / IEC 23009-1).

[0012] In view of this, generally speaking, and more specifically in the context of ISOBMFF, it seems necessary to use a technique to signal to the user such pre-selected information (and possibly instructions on how to process such pre-selected information). Summary of the Invention

[0013] In view of the foregoing, this disclosure generally provides a method, apparatus, program, and computer-readable storage medium for processing media streams having the features of the respective independent claims.

[0014] According to a first aspect of this disclosure, a method for processing a media stream is provided. The media stream may be an audio stream, a video stream, or a combination thereof. The method may be performed on the user side or, in some cases, in a user (decoding) side environment. Depending on various implementations, the environment may include, but is not limited to, a TV, a soundbar, a web browser, a media player, a plugin, etc.

[0015] Specifically, the method may include receiving media streams packetized according to a predefined transmission format. The predefined transmission format may be the Basic Media File Format (ISOBMFF) specified by ISO through ISO / IEC 14496-12 MPEG-4 Part 12, or any other suitable (transmission) format. The packetized media stream may include multiple layered boxes, each associated with a corresponding box type identifier. It is noteworthy that, as used herein, in some possible cases, the term "box" may generally be used to refer to an object-oriented building block defined by a unique (box) type identifier (and possibly also, of appropriate length), such as that described in ISO 14496-12. Of course, the term "box" as used throughout this disclosure should not be construed as limited to such specifications. Rather, the term "box" should generally be understood as any suitable data structure that can be used as a placeholder for media data or other data in a packetized media stream. Furthermore, as those skilled in the art will understand and recognize, such a "box" may be referred to by any other suitable term. An example could be that in some possible specifications (including the first definition of MP4), in certain cases, a "box" may be alternatively referred to as an "atom". Furthermore, as indicated (“layering”), multiple boxes may have the same or different levels (or positions), may be nested (child / sub box with parent box), etc., depending on various implementations and / or requirements, as those skilled in the art will understand and recognize. More specifically, among other possibilities, multiple boxes may include one or more track boxes that refer to (or in other words, indicate) a corresponding track that indicates a media (content) component of a media stream. In a broad sense, a media (content) component can generally refer to a single / separate consecutive component of media content (and can also generally be associated with a corresponding (e.g., assigned) media content component type, which may include, but is not limited to, audio, video, text, etc.). Examples for understanding the concept of a “media (content) component” can be found, for example, as defined / described in MPEG-DASH (ISO / IEC 23009-1).

[0016] The method may further include determining whether the media stream includes preselected related boxes of a predefined type indicating preselection, wherein preselection may correspond to media presentation to a user. More specifically, as used herein, the term / phrase “preselection” is generally used to refer to a set of media content components (of a media stream) intended for (e.g., by a user-side device) common use and, more particularly, generally to a version of media presentation that can be selected by the end user for simultaneous decoding and / or presentation. Examples for understanding the concept of “preselection” can be found, for example, as described in MPEG-DASH (ISO / IEC 23009-1). Of course, as those skilled in the art will understand and recognize, in some other possible technical contexts, the term “preselection” may also be known (or mentioned) by using any other suitable (equivalent) term, such as (but not limited to) “presentation” as described, for example, in ETSI TS103190-2 or “preset” as described, for example, in ISO / IEC 23008-3, etc. Thus, a preselected related box can be a specific box among multiple boxes in a media stream that has a specific predefined (or predetermined) type. As the technician will understand and recognize, the specific type of this instruction preselection can be predefined (or predetermined) by any suitable means, as will be described in more detail below.

[0017] If the media stream is determined to include preselected related boxes, the method may further include: analyzing metadata information corresponding to the preselected related boxes, the metadata information indicating the characteristics of the preselection; identifying one or more tracks in the grouped media stream that contribute to the preselection based on the metadata information; and providing one or more tracks for downstream processing according to the given preselection. As those skilled in the art will understand and recognize, according to various embodiments, the metadata information described above may be (directly) included in the media stream (or more specifically, multiple boxes of the (grouped) media stream) or may be (indirectly) obtained from the media stream using any suitable means. For example, the metadata information may be included or contained in a head box (or a sub-box of another box), which may be associated with or linked to the preselected related box (e.g., as its sub-box). As mentioned above, preselection generally refers to a set of media content components intended for use, for example, by one or more suitable downstream devices (e.g., media decoders, media players, etc.). In some possible cases, the downstream device may also be simply referred to as a "sink". Therefore, depending on the implementation and / or requirements, downstream processing may include, but is not limited to, multiplexing (or, in some cases, re-multiplexing), sorting, merging, decoding, or rendering of those contributing tracks, as will be described in more detail below.

[0018] As configured as described above, the proposed method generally provides an efficient and flexible way to identify / identify tracks within a media stream configured to contribute to a specific preselection, and subsequently signal these tracks, thereby enabling (e.g., by one or more suitable downstream devices) further appropriate downstream processing of such contributing tracks. Therefore, broadly speaking, the proposed method can be viewed as providing the possibility and capability to signal information indicating preselection (and possibly processing of preselection) in a uniform manner within a transport layer file (e.g., ISOBMFF), which can be considered beneficial in various use cases or scenarios. For example, such a uniform representation of preselection (or in other words, format / type-independent) can be used to implement a uniform data structure for a media playback application programming interface (API) (e.g., to be used by the application or as a plugin for a web browser), eliminating the need for format-specific implementations in the media player, thereby reducing implementation and / or testing workload while improving reliability. As another example, this pre-selected uniform representation can also enable format-independent implementations of pre-selected data processing in manifest generators (e.g., MPEG HTTP-based Dynamic Adaptive Streaming (DASH) format files or HTTP Real-Time Streaming (HLS) format files), thereby avoiding the need for computationally more expensive operations on binary data and again reducing implementation effort and improving reliability. As used herein, the term "format / type independent" generally means that the proposed representation is universal for all data types (formats).

[0019] In some example implementations, the media stream may further include processing information indicating how (e.g., by (multiple) downstream devices) the preselected track will be processed. Similar to metadata information as described above, processing information may also be (directly) included in the media stream (or more specifically, multiple boxes of a (grouped) media stream) or may be (indirectly) obtained from the media stream using any suitable means, depending on the implementation. For example, processing information may also be included in a specific box (e.g., of a particular (predefined) type), which may be associated with or linked to a preselected or preselection-related box (e.g., as a sub-box).

[0020] In some example implementations, the processing information may include sorting information indicating the order in which one or more tracks are processed (e.g., decoding, merging, etc.). For example, in some possible cases, the track order may indicate the order in which tracks will be provided to downstream devices (e.g., decoding devices). Similar to the above, this sorting information may be implemented as being included in a (sub)box associated with or related to the processing information.

[0021] In some example implementations, the processing information may include merging information indicating whether one or more tracks should be merged with one or more other tracks for joint (downstream) processing. That is, depending on the implementation of such merging information, in some cases, a track(s) may be merged with one or more other tracks(s) for downstream processing; while in other cases, a track(s) may be processed independently (e.g., routed to a separate decoding instance). It is worth noting that in some possible cases, this merging may also be referred to as multiplexing or any other suitable term. As those skilled in the art will understand and recognize, track merging (multiplexing) can be achieved using any suitable means (e.g., by appending a subsequent track to the end of a previous track).

[0022] In some example implementations, the method may further include merging one or more tracks based on merging information and sorting information.

[0023] In some example implementations, for each track contributing to the preselection, the sorting information may include a corresponding track order value for defining the track's order. As those skilled in the art will understand and recognize, various suitable rules for defining the track order can be determined using the corresponding track order value. As mentioned above, in some possible cases, the track order may indicate the order in which tracks will be provided to downstream devices (e.g., decoding devices). In this case, a possible exemplary implementation (but not any kind of limitation) may be that a track with a smaller track order value (e.g., 1) will be provided to the decoding device earlier than another track with a larger track order value (e.g., 3). In some cases, if multiple tracks have the same value order, the sorting of these tracks may no longer be relevant or important. Further, in a similar manner, the merging information may include a corresponding merging flag for each track contributing to the preselection. Specifically, a first setting value (e.g., '1') of the merge flag can indicate that the corresponding track will be merged (or multiplexed) with adjacent tracks (e.g., the preceding or following track, depending on various implementations of such merge flags) in track order, and a second setting value (e.g., '0') of the merge flag can correspondingly indicate that the corresponding track will be processed separately (e.g., fed or routed to a separate downstream decoding device). Therefore, merging one or more tracks based on merge and ordering information can include scanning tracks sequentially (or sequentially) according to track order; and merging tracks according to the corresponding merge flag. For example, in some possible cases, if the merge flag [i] of track i is set to '1', each sample of track i can be appended to (multiple) samples of the track(s) with the next lower (or higher) track order value (e.g., track i-1 or i+1); while on the other hand, if the merge flag of a track is set to '0', track i can be provided to a separate decoder instance. As another possible example, in the extreme case where all the merge flags of the tracks are set to '0', applying the above concept, all tracks will be distributed to several (separate) downstream devices (sinks).

[0024] In some example implementations, the method may further include decoding one or more tracks for playing back a media stream based on media presentation indicated by a preselection.

[0025] In some example implementations, one or more tracks can be decoded by downstream devices (e.g., media players, TVs, soundbars, plug-ins, etc.).

[0026] In some example implementations, merging one or more tracks and decoding one or more tracks can be performed by a single device. In other words, there may be use cases where merging and decoding work together. More specifically, an example (but not limited to) of such a use case could be a JavaScript API that wants to work using only a single merged stream, without using multiple streams. In this case, an entity might typically take multiple incoming streams, merge / multiplex them as described above, and then send them as a single merged stream over a single-stream API for decoding on the other side of the API. In some possible cases, the single-stream format could be a Common Media Application Format (CMAF) byte stream format. Of course, as those skilled in the art will understand and recognize, in some other cases, track merging and decoding can be performed by different (separate) devices. As an illustrative example (but not limited to), a TV might be implementing the merging of streams as described above, but sending the merged stream to a separate downstream device (such as a soundbar) for subsequent decoding.

[0027] In some example implementations, the media stream may include multiple (rather than just one) pre-selected related boxes of a predefined type. Therefore, the method may further include selecting (or determining) a pre-selected related box among the multiple pre-selected related boxes. As those skilled in the art will understand and recognize, such selection (or determination) of a particular pre-selected related box among the multiple pre-selected related boxes can be performed in any suitable manner.

[0028] In some example implementations, the pre-selected relevant boxes can be selected (or determined) by an application (e.g., an application that controls a media player / decoder). For example, in some possible cases, the application can be configured (e.g., based on a predefined algorithm) to (automatically) select (or determine) pre-selected relevant boxes, for example, those corresponding to specific settings (e.g., decoding or rendering environment).

[0029] In some example implementations, the media stream may include one or more tab boxes (which may be associated with or linked in some way to a corresponding preselection or a corresponding preselection-related box), each tab box including descriptive information about the corresponding media presentation to the user corresponding to the corresponding preselection. Therefore, in this case, the selection (or determination) of the preselection-related box can be performed based on user input. As an example (but not limited to), the tab box may include descriptive information indicating optional subtitles in various languages ​​(e.g., English, German, Chinese, etc.), each language being considered a corresponding preselection (presentation) so that (e.g., a user controlling an application) can appropriately select (e.g., by clicking a mouse or keyboard) the corresponding language setting.

[0030] In some example implementations, preselection-related boxes can be considered independent of the media codec used to encode the media stream before it is grouped. In other words, generally, preselection-related boxes may only include the necessary information for the respective preselection, without including information that can be associated with the media codec (i.e., codec-specific information). In other words, in the preselection-related boxes(s), there is typically no information related to the media stream such as how it is encoded (e.g., by a particular media encoder) and / or how it should be decoded (e.g., by a particular media decoder).

[0031] In some example implementations, the metadata information corresponding to the preselection-related box may include track identification information indicating one or more track identifiers, each associated with a corresponding track, wherein the track associated with the one or more track identifiers in the metadata information may be related to media presentation. As those skilled in the art will understand and recognize, such track identification information indicating one or more track identifiers can be implemented in any suitable manner, such as as simple as an array, wherein each element in the array uniquely indicates a corresponding track identifier (which itself can be represented using an integer value or any other suitable form). In this case, in some possible instances, the metadata information corresponding to the preselection-related box may optionally further include a counter (e.g., an integer value) indicating the number of tracks required for (or contributing to) the preselection.

[0032] In some example implementations, the metadata information corresponding to the preselection-related box may include preselection identification information, which indicates a preselection identifier used to identify the preselection. That is, the metadata information corresponding to the preselection-related box may include necessary information (e.g., represented by integers) that makes the preselection uniquely identifiable to external (e.g., downstream) applications and / or devices, for example, to assist in the selection / determination of the corresponding preselection.

[0033] In some example implementations, the metadata information corresponding to the preselected associated box may include unique preselect-specific data for configuring downstream devices (e.g., downstream media players / decoders) to decode tracks according to the preselection. Depending on various implementations and / or requirements, this preselect-specific data may include any suitable information (e.g., codec-specific information in some cases) and may be implemented (represented) in any suitable manner (e.g., integers, arrays, strings, etc.).

[0034] According to a second aspect of this disclosure, a method for processing media streams is provided. The media stream may be an audio stream, a video stream, or a combination thereof. The method may be performed on the user side or, in some cases, in a user (decoding) side environment. Depending on the implementation, the environment may include, but is not limited to, a TV, a soundbar, a web browser, a media player, a plugin, etc.

[0035] Specifically, the method may include receiving media streams packetized according to a predefined transmission format. Similar to the first aspect above, the predefined transmission format may be the Basic Media File Format (ISOBMFF) specified by ISO through ISO / IEC 14496-12 MPEG-4 Part 12 or any other suitable (transmission) format. The packetized media stream may include multiple layered boxes, each associated with a corresponding box type identifier. As mentioned above, depending on various implementations and / or requirements, the multiple boxes (or referred to by any other suitable term) may have the same or different levels (or positions), may be nested (child boxes and parent boxes), etc., as will be understood and recognized by those skilled in the art. More specifically, among other possibilities, the multiple boxes may include one or more track boxes that reference (or in other words, indicate) corresponding tracks that indicate media (content) components of the media stream. Additionally, the multiple boxes may also include one or more track group boxes (or referred to by any other suitable term / name), each associated with a corresponding pair consisting of a track group identifier and a track group type that jointly identify a corresponding track group within the media stream. In other words, tracks with the same track group identifier and the same track group type (e.g., identified by or associated with the same track group identifier and the same track group type) can be considered to belong to the same track group. Each such track group can typically identify a corresponding preselection for the appropriate media presentation to the user. As already explained above, the term / phrase preselection (or referred to by any other suitable term / name) is generally used to refer to a set of media content components (of the media stream) intended for (e.g., by the user-side device) to be used together, and more particularly generally indicates a version of media presentation that can be selected by the end user for simultaneous decoding / presentation.

[0036] The method may further include examining (e.g., accessing, looping, etc.) trackboxes in the media stream to determine a complete (or total) set of preselections present in the media stream. Specifically, determining the complete set of preselections may include: identifying a unique set of track group identifiers and track group type pairs; and addressing the preselections by the corresponding track group identifiers. As mentioned above, each preselection is associated with a corresponding track group, which itself is identified by a corresponding track group identifier and a corresponding track group type pair. Therefore, a preselection can be addressed (or identified) by the corresponding track group identifier associated with / linked to it.

[0037] The method may further include selecting preselections from the complete set of preselections. In particular, preselections may be selected based on the attributes (e.g., represented as metadata or any other suitable form) of the corresponding preselections included in orbital group boxes with the same orbital group identifier.

[0038] The method may also include identifying (e.g., identifying) one or more track boxes that contribute to a selected preselected group. In particular, one or more track boxes that contribute to the (same) preselected group can be identified (by) the presence of (corresponding) track group boxes having the same track group identifier.

[0039] Furthermore, the method may further include identifying the orbital referenced in each member (element) of a set of one or more orbital boxes as defined above as contributing to one or more preselected orbitals.

[0040] Finally, the method may include providing one or more tracks for downstream processing based on a preselection. As mentioned above, a preselection typically refers to a set of media content components intended for use by one or more suitable downstream devices (or, in some cases, a sink) such as a media decoder, media player, etc. Therefore, depending on various implementations and / or requirements, downstream processing may include, but is not limited to, multiplexing (or, in some cases, re-multiplexing), sorting, merging, decoding, or rendering those contributing tracks, as will be described in more detail below.

[0041] As configured as described above, the proposed method typically provides an efficient and flexible way to identify / identify tracks within a media stream configured to contribute to a specific preselection, and subsequently signal these tracks, thereby enabling (e.g., by one or more downstream devices) further appropriate downstream processing of such contributing tracks. In particular, it should be noted that the method proposed in the first aspect typically seeks to provide relevant information in a preselection-related box for (all) tracks configured to contribute to a specific preselection, thereby enabling indexing (or identification) of all contributing tracks. In this sense, such indexing of tracks as described in the first aspect can be considered a kind of forward (direct) indexing. In contrast, in the method proposed in this second aspect, tracks contributing to a specific preselection can be jointly determined by a pair consisting of a track group type and a track group identifier. More specifically, tracks having (e.g., containing) a track group box with a specific (e.g., predefined or predetermined) track group type typically indicate that the track contributes to the preselection. Furthermore, tracks having the same track group identifier typically indicate which tracks belong to (contribute to) the same preselection. In this sense, contrary to that proposed in the first aspect, the indexing of tracks described herein can be viewed as a kind of reverse index. In any case, similar to the first aspect, the method proposed in the second aspect can also provide the possibility and capability of signaling information indicating preselection (and possibly processing of preselection) in a uniform manner within a transport layer file (e.g., ISOBMFF), which can be considered beneficial in various use cases or scenarios. For example, such a uniform representation of preselection (or in other words, format-independent) can be used to implement a uniform data structure for media playback APIs (e.g., to be used by applications or as a plugin for web browsers), eliminating the need for format-specific implementations in media players, thereby reducing implementation and / or testing effort while improving reliability. As another example, such a uniform representation of preselection can also enable format-independent implementations of preselection data processing in manifest generators (e.g., MPEG HTTP-based dynamically adaptive streaming DASH format files or HTTP real-time streaming HLS format files), thereby avoiding the need for computationally more expensive operations on binary data and again reducing implementation effort and improving reliability.

[0042] In some example implementations, each preselection may be associated with a corresponding preselection-related box of a predefined type. Specifically, the preselection-related box may be instantiated (e.g., inherent, extended, etc.) with a track group box having a predefined track group type associated with the preselection. As those skilled in the art will understand and recognize, such a predefined track group type associated with a preselection can be implemented using any suitable means, such as a specific (predefined) string (e.g., 'preselect' or 'pres'), a specific (predefined) value (e.g., '3'), etc. Generally, in the method proposed in the second aspect, each preselection and each track corresponding to (contributing to) the preselection typically has one such preselection-related box. In contrast, in the method proposed in the first aspect above, each preselection typically has one such preselection-related box.

[0043] In some example implementations, a preselected related box may be associated with a preselected processing box, which includes processing information indicating how to process tracks contributing to the preselection. According to various implementations, the processing information may be (directly) included in the media stream (or more specifically, multiple boxes of a (grouped) media stream) or may be (indirectly) obtained from the media stream using any suitable means. For example, the processing information may also be included in a specific box (e.g., of a particular (predefined) type), which may be associated with or linked to the preselected or preselected related box (e.g., as a sub-box).

[0044] In some example implementations, a preselected related box may be associated with a preselected information box, which includes semantic (or descriptive) information (e.g., attributes, characteristics, etc.) indicating the preselection.

[0045] In some example implementations, the processing information may include unique preselected specific data for configuring downstream devices (e.g., downstream media players / decoders) to decode tracks according to a preselection. Depending on various implementations and / or requirements, this preselected specific data may include any suitable information (e.g., codec-specific information in some cases) and may be implemented (represented) in any suitable manner (e.g., integers, arrays, strings, etc.).

[0046] In some example implementations, the processing information may include sorting information indicating the order in which tracks are ordered for further downstream processing (e.g., decoding, merging, etc.). For example, in some possible cases, the track order may indicate the order in which tracks will be provided to downstream devices (e.g., decoding devices). Similar to the above, this sorting information may be implemented as being included in a (sub)box associated with or related to the processing information.

[0047] In some example implementations, the processing information may include merging information indicating whether one or more tracks should be merged with one or more other tracks for, for example, joint (downstream) processing. That is, depending on the implementation of such merging information, in some cases, a track(s) may be merged with another track(s) for downstream processing; while in other cases, a track(s) may be processed independently (e.g., routed to a separate decoding instance). It is worth noting that in some possible cases, this merging may also be referred to as multiplexing or any other suitable term. As those skilled in the art will understand and recognize, track merging (multiplexing) can be achieved using any suitable means (e.g., by appending a subsequent track to the end of a previous track).

[0048] In some example implementations, the method may further include merging one or more tracks based on merging information and sorting information.

[0049] In some example implementations, for each track contributing to the preselection, the sorting information may include a corresponding track order value for defining the track's order. As those skilled in the art will understand and recognize, various suitable rules for defining the track order can be determined using the corresponding track order value. As mentioned above, in some possible cases, the track order may indicate the order in which tracks will be provided to downstream devices (e.g., decoding devices). In this case, a possible example implementation (but not any kind of limitation) may be that a track with a smaller track order value (e.g., 1) will be provided to the decoding device earlier than another track with a larger track order value (e.g., 3). In some cases, if multiple tracks have the same track order value, such track sorting may no longer be relevant or important. Further, in a similar manner, the merging information may include a corresponding merging flag for each track contributing to the preselection. Specifically, a first setting value (e.g., '1') of the merge flag can indicate that the corresponding track will be merged (or multiplexed) with adjacent tracks (e.g., the preceding or following track, depending on the various implementations of such merge flag) in track order, and a second setting value (e.g., '0') of the merge flag can correspondingly indicate that the corresponding track will be processed separately (e.g., fed / routed to a separate downstream decoding device). Therefore, merging one or more tracks based on merge and ordering information can include scanning tracks sequentially (or sequentially) according to track order; and merging tracks according to the corresponding merge flag. For example, in some possible cases, if the merge flag [i] of track i is set to '1', each sample of track i can be appended to (multiple) samples of the track(s) with the next lower (or higher) track order value (e.g., track i-1 or i+1); while on the other hand, if the merge flag of a track is set to '0', track i can be provided to a separate decoder instance. As another possible example, in the extreme case where all the merge flags of the tracks are set to '0', applying the above concept, all tracks would be distributed to several (separate) downstream devices (e.g., sinks).

[0050] In some example implementations, the method may further include decoding one or more tracks for playing back a media stream based on media presentation indicated by a preselection.

[0051] In some example implementations, one or more tracks can be decoded by downstream devices (e.g., media players, TVs, plugins, etc.).

[0052] In some example implementations, merging one or more tracks and decoding one or more tracks can be performed by a single device. In other words, there may be use cases where merging and decoding work together. More specifically, an example (but not limited to) of such a use case could be a JavaScript API that wants to work using only a single merged stream, without using multiple streams. In this case, an entity might typically take multiple incoming streams, merge / multiplex them as described above, and then send them as a single merged stream over a single-stream API for decoding on the other side of the API. In some possible cases, the single-stream format could be a Common Media Application Format (CMAF) byte stream format. Of course, as those skilled in the art will understand and recognize, in some other cases, track merging and decoding can be performed by different (separate) devices. As an illustrative example (but not limited to), a TV might be implementing the merging of streams as described above, but sending the merged stream to a separate downstream device (such as a soundbar) for subsequent decoding.

[0053] In some example implementations, pre-selection can be chosen (or determined) by an application (e.g., an application that controls a media player / decoder). For example, in some possible cases, the application can be configured (e.g., based on a predefined algorithm) to (automatically) select (or determine) pre-selection corresponding to, for example, specific settings (e.g., decoding or rendering environment).

[0054] In some example implementations, the media stream may include one or more tab boxes (which may be associated with or linked to a corresponding preselected or preselection-related box), each tab box including descriptive information about the corresponding media presentation to the user corresponding to the corresponding preselection. Therefore, in this case, the selection (or determination) of the preselection can be performed based on user input. As an example (but not limited to), the tab box may include descriptive information indicating optional subtitles in various languages ​​(e.g., English, German, Chinese, etc.), each language being considered a corresponding preselection (presentation) so that a user (e.g., controlling an application) can appropriately select (e.g., by clicking a mouse or keyboard) the corresponding language setting. Of course, as those skilled in the art will understand and recognize, the selection (or determination) of the preselection can also be performed in any other suitable manner.

[0055] In some example implementations, as described above, the media stream may include at least one of an audio stream or a video stream (or a combination thereof). Specifically, some possible scenarios (or use cases) where the methods proposed in this disclosure can be applied may be, for example, multi-person video conferencing (where viewers may have the ability to select one or more video streams); or a TV with picture-in-picture capability (e.g., where one frame has a higher bitrate / resolution and another frame has a lower bitrate / resolution).

[0056] According to a third aspect of this disclosure, a method for processing media streams is provided. The media stream may be an audio stream, a video stream, or a combination thereof. The method can be executed on an encoding-side environment (e.g., a media encoder). In some scenarios (or use cases), such an encoder may also be referred to as a (media) packer (i.e., configured to pack / group media input).

[0057] Specifically, the method may include encapsulating one or more elementary streams according to a predefined transport format to generate packetized media streams, wherein the packetized media streams comprise multiple layered boxes, each associated with a corresponding box type identifier. Similar to the above, the predefined transport format may be the Elementary Media File Format (ISOBMFF) specified by ISO through ISO / IEC 14496-12 MPEG-4 Part 12, or any other suitable (transport) format. Broadly speaking, as those skilled in the art will understand and recognize, elementary streams can be considered as compressed binary representations of (media) data (audio or video) flowing from a single media encoder to a media decoder. When multiplexed (or packetized) into a predefined transport format (e.g., ISOBMFF), these elementary streams may be referred to as “tracks,” where “track boxes” describe the properties (or attributes) of each track in the file header. Furthermore, as already described above, depending on various implementations and / or requirements, multiple boxes (or referred to by any other suitable term) may have the same or different levels (or positions), may be nested (child boxes with parent boxes), etc., as those skilled in the art will understand and recognize.

[0058] More specifically, encapsulating one or more elementary streams may include: grouping media data of one or more elementary streams according to a transport format to generate one or more track boxes that reference (or indicate) the corresponding tracks of one or more elementary streams; and generating one or more preselected related boxes of a predefined type based on header information of one or more elementary streams, wherein each of the one or more preselected related boxes indicates a corresponding preselection corresponding to media presentation to the user.

[0059] As configured as described above, the proposed method generally provides an efficient and flexible way to group media inputs (e.g., elementary streams) according to a predefined transport format (e.g., ISOBMFF). More specifically, by generating and including one or more preselection-related boxes (each preselection-related box indicating a corresponding preselection) alongside the grouped media streams, the proposed method also enables the representation of preselections in a codec-independent uniform manner, thereby further enabling appropriate downstream processing of tracks contributing to the corresponding preselections (e.g., according to the methods proposed in the first and second aspects above). Additionally, as mentioned above, this uniform representation of preselections also enables format-independent implementations of preselection data processing in manifest generators (e.g., MPEG HTTP-based dynamically adaptive streaming DASH format files or HTTP real-time streaming HLS format files), thereby avoiding the need for computationally more expensive operations on binary data, while simultaneously reducing implementation effort and improving reliability.

[0060] In some example implementations, each of one or more preselected related boxes may include metadata information indicating characteristics of the corresponding preselection. More specifically, the metadata information may include information indicating one or more tracks in the media stream that contribute to the corresponding preselection. As those skilled in the art will understand and recognize, according to various implementations, the metadata information described above may be (directly) included in the media stream (or more specifically, multiple boxes of a (grouped) media stream) or may be (indirectly) obtained from the media stream using any suitable means. For example, the metadata information may be included or contained in a head (or similar) box, which may be associated with or linked to the preselected related boxes in some way (e.g., as a sub-box).

[0061] In some example implementations, the metadata information corresponding to the corresponding preselected associated box may further include at least one of the following: preselection identification information indicating a preselection identifier for identifying the corresponding preselection; or unique preselection-specific data for decoding tracks based on the preselection (e.g., for configuring downstream devices such as downstream media decoders).

[0062] In some example implementations, encapsulating the basic media stream may further include generating one or more track group boxes, each track group box being associated with a corresponding track group identifier and a corresponding track group type (a pair consisting of these) within the media stream that jointly identifies the grouped track group. Specifically, tracks having the same track group identifier and the same track group type can be considered to belong to the same track group. Further, generating one or more preselection-related boxes may include: assigning a first unique identifier to each preselection; and for each track contributing to the corresponding preselection, generating a corresponding preselection-related box associated with the corresponding preselection and setting the track group identifier to the first unique identifier, wherein the preselection-related box instantiates a track group box having a predefined track group type associated with the preselection. As those skilled in the art will understand and recognize, such a predefined track group type associated with a preselection can be implemented using any suitable means, such as a specific (predefined) string (e.g., 'preselect' or 'pres'), a specific (predefined) value (e.g., 3), etc. Generally, in the methods proposed herein, there is typically one such preselection-related box for each preselection and for each track corresponding to (contributing to) the preselection.

[0063] In some example implementations, track groups can be generated by grouping tracks based on the corresponding media type contributing to a pre-selected track.

[0064] In some example implementations, the media type may include at least one of the following: audio, video, and subtitles. Of course, any other suitable media type may also be used, as those skilled in the art will understand and recognize.

[0065] In some example implementations, generating one or more preselected related boxes may further include generating one or more preselected processing boxes, each preselected processing box including processing information indicating how to process tracks contributed to a corresponding preselection. According to various implementations, the processing information may be (directly) included in the media stream (or more specifically, multiple boxes of a (grouped) media stream) or may be (indirectly) obtained from the media stream using any suitable means. For example, the processing information may also be included in a specific box (e.g., of a particular (predefined) type), which may be associated with or linked to a preselected or preselected related box in some way (e.g., as a sub-box).

[0066] In some example implementations, the processing information may include at least one of the following: sorting information indicating the order of tracks used for processing; or merging information indicating whether one or more tracks are to be merged with one or more other tracks (e.g., for joint (downstream) processing). For example, in some possible cases, the track order may indicate the order in which tracks will be provided to downstream devices (e.g., decoding devices). Further, depending on the implementation of the merging information, in some cases, a track(s) may be merged with some other tracks(s) for downstream processing; while in other cases, a track(s) may be processed separately (e.g., routed to a separate decoding instance). Similar to the above, the sorting information and the merging information may be implemented as being included in one (or more) boxes associated with or related to the processing information.

[0067] In some example implementations, the method may further include: receiving at least one input medium; and processing (e.g., encoding) the input medium to generate one or more elementary streams, wherein the one or more elementary streams include media data of the input medium and corresponding header information. For example, the input medium can be processed (e.g., encoded) using a suitable media encoder to appropriately generate the corresponding elementary streams.

[0068] In some example implementations, the method may further include generating a manifest file based on one or more preselected related boxes. Generally, the manifest file typically includes various information, such as information about the media stream (e.g., media type, codec attributes, media-specific properties, etc.). In addition to such information (i.e., information related to the media stream itself), the proposed method may further include information associated with preselections. Preselection-related information may include, but is not limited to, metadata information, processing information, etc. Compared to conventional techniques, the proposed method for generating manifest files typically provides a format-independent implementation of preselection-related data processing, thereby avoiding computationally more expensive operations (e.g., operations that must be performed on binary data in conventional techniques), reducing implementation and / or testing workload, and improving reliability.

[0069] In some example implementations, as those skilled in the art will understand and recognize, the manifest file may be an MPEG-based HTTP Dynamic Adaptive Streaming (DASH) format file, an HTTP Real-Time Streaming (HLS) format file, or any other suitable manifest format file.

[0070] According to a fourth aspect of this disclosure, a method for processing media streams is provided. The media stream may be an audio stream, a video stream, or a combination thereof. For example, the method may be performed by a manifest generator.

[0071] Specifically, the method may include receiving a media stream packetized according to a predefined transmission format. Specifically, the packetized media stream may include a plurality of layered boxes, each associated with a corresponding box type identifier, wherein the plurality of boxes may include one or more track boxes and one or more preselected associated boxes of a predefined type, the one or more track boxes referencing (e.g., indicating) a corresponding track of a media component of the media stream, and wherein each preselected associated box indicates a corresponding preselection corresponding to media presentation to the user.

[0072] Furthermore, the method may also include generating an inventory file based on one or more pre-selected related boxes.

[0073] As configured as described above, the proposed method can typically provide an efficient and flexible approach to generating manifest files by also considering pre-selected relevant information (e.g., descriptive information and / or processing-related information). More specifically, in addition to information related to the media stream(s), the proposed method can further include information associated with pre-selection. Pre-selected relevant information can include, but is not limited to, metadata information, processing information, etc. Compared to conventional (manifest generation) techniques, the proposed method for generating manifest files can typically provide a format-independent implementation of pre-selected relevant data processing, thereby avoiding computationally more expensive operations (e.g., operations that must be performed on binary data in conventional techniques), reducing implementation and / or testing workload, and improving reliability.

[0074] In some example implementations, as those skilled in the art will understand and recognize, the manifest file may be an MPEG-based HTTP Dynamic Adaptive Streaming (DASH) format file, an HTTP Real-Time Streaming (HLS) format file, or any other suitable manifest format file.

[0075] In some example implementations, the media presentation to a user can be characterized by appropriate configurations associated with the language, genre, and / or one or more media-specific attributes of the media stream. Of course, any other suitable configuration may be used depending on the various implementations.

[0076] In some example implementations, the predefined transmission format may be the ISO Basic Media File Format (ISOBMFF) or any other suitable transmission format.

[0077] According to a fifth aspect of the present invention, a media streaming processing apparatus is provided, the media streaming processing apparatus including a processor and a memory coupled to the processor. The processor may be adapted to cause the media streaming processing apparatus to perform all steps of any of the exemplary methods described in the foregoing aspects.

[0078] According to a sixth aspect of the invention, a computer program is provided. The computer program may include instructions that, when executed by a processor, cause the processor to perform all the steps of the method described throughout the disclosure.

[0079] According to a seventh aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium can store the aforementioned computer program.

[0080] It will be understood that the features of the apparatus and the steps of the method can be interchanged in various ways. In particular, the details of the disclosed methods (various methods) can be implemented by the corresponding apparatus (or system), and vice versa, as will be recognized by those skilled in the art. Furthermore, any statement made above with respect to the methods (various methods) is to be understood to apply equally to the corresponding apparatus (or system), and vice versa. Attached Figure Description

[0081] The following explanation of exemplary embodiments of this disclosure refers to the accompanying drawings, wherein the same reference numerals indicate the same or similar elements, and wherein,

[0082] Figure 1A This is a schematic diagram illustrating an exemplary implementation of a media playback application programming interface (API).

[0083] Figure 1B This is a schematic diagram illustrating an exemplary implementation of a media playback API according to embodiments of the present disclosure.

[0084] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a system for processing media streams according to embodiments of the present disclosure.

[0085] Figure 3A This is a schematic diagram illustrating an example implementation of the packer.

[0086] Figure 3B This is a schematic diagram illustrating an example implementation of a packer according to an embodiment of the present disclosure.

[0087] Figure 4A This is a schematic diagram illustrating an example implementation of the manifest generator.

[0088] Figure 4B This is a schematic diagram illustrating an example implementation of a manifest generator according to embodiments of the present disclosure.

[0089] Figure 5 This is a schematic flowchart illustrating an example of a method for processing a media stream according to an embodiment of the present disclosure.

[0090] Figure 6 This is a schematic flowchart illustrating another example of a method for processing a media stream according to an embodiment of the present disclosure.

[0091] Figure 7 This is a schematic flowchart illustrating yet another example of a method for processing a media stream according to an embodiment of the present disclosure.

[0092] Figure 8 This is a schematic flowchart illustrating yet another example of a method for processing a media stream according to embodiments of the present disclosure, and

[0093] Figure 9 This is a schematic block diagram of an example apparatus for performing a method according to an embodiment of the present disclosure. Detailed Implementation

[0094] As indicated above, unless otherwise indicated, the same or similar reference numerals in this disclosure may indicate the same or similar elements, such that repeated descriptions thereof may be omitted for the sake of brevity.

[0095] In particular, the accompanying drawings and the following description are for illustrative purposes only and relate to preferred embodiments. It should be noted that, based on the discussion below, alternative embodiments of the structures and methods disclosed herein will be readily recognized as feasible alternatives that can be employed without departing from the claimed principles.

[0096] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. It should be noted that, where feasible, similar or identical reference numerals may be used in the drawings, and these reference numerals may indicate similar or identical functions. The drawings depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0097] Furthermore, in drawings where connecting elements (such as solid or dashed lines or arrows) are used to illustrate connections, relationships, or associations between or among two or more other schematic elements, the absence of any such connecting element does not imply the absence of connections, relationships, or associations. In other words, some connections, relationships, or associations between elements are not shown in the drawings to avoid obscuring the invention. Additionally, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, in the case where a connecting element represents communication of signals, data, or instructions, those skilled in the art will understand that such an element represents one or more signal paths that may be necessary to influence the communication.

[0098] As indicated above, this document generally attempts to propose techniques for implementing preselection concepts in the context of ISOBMFF files, thereby enabling the availability of MPEG-DASH and various other media compression formats.

[0099] Therefore, the example embodiments described herein may relate to methods, apparatus, and processes for various "pre-selected" use cases.

[0100] Before describing the example embodiments in detail, it may still be worthwhile, for ease of understanding, to describe some possible terms that may be used throughout this disclosure, even though some of these terms may have already been (briefly) discussed above.

[0101] Specifically, for the purposes of this document, the term “preselection” generally refers to a group or more media (content) components that are simultaneously decoded / presented, as in, for example, MPEG-DASH (ISO / IEC 23009-1). As those skilled in the art will understand and recognize, in some other possible technical contexts, “preselection” may also be known (or mentioned) by using any other suitable (equivalent) term, such as (but not limited to) “presentation” as described in, for example, ETSI TS103190-2 or “preset” as described in, for example, ISO / IEC 23008-3, etc.

[0102] Regarding the term "AdaptationSet", this term should generally be defined as similarly described in MPEG-DASH (ISO / IEC 23009-1), and similarly, it can also be referred to as other terms, such as "SwitchingSet" as defined in the Common Media Application Format (CMAF).

[0103] Regarding “presentation”, this term should generally be defined as described in MPEG-DASH (ISO / IEC 23009-1), and may also be referred to as “AudioTrack” (e.g., as defined by the W3C API), “Tracklist”, or “Track” (e.g., as defined by ISOBMFF).

[0104] Regarding "media content component", this phrase / term should generally refer to a single, continuous component of media content having an assigned media content component type, as described in MPEG-DASH (ISO / IEC 23009-1).

[0105] Regarding "media content component type," this phrase / term should generally refer to a single type of media content, such as that described in MPEG-DASH (ISO / IEC 23009-1). As those skilled in the art will understand and recognize, examples of media content component types may include, but are not limited to, audio, video, text, or any other suitable type.

[0106] Regarding the term "box," this term should generally refer to an object-oriented building block defined by a unique type identifier and length, as described in, for example, ISO 14496-12. Similarly, it can be noted that in some specifications (including the first definition of MP4), the term "box" may be alternatively referred to as "atom," or any other suitable similar term.

[0107] Regarding the term "container box," this phrase / term should generally refer to a box whose sole purpose is to contain and group a set of related (sub)boxes, as described in ISO 14496-12, for example. However, it should be noted that a container box is not usually derived from a "fullbox."

[0108] Finally, regarding "ISO basic media file", this phrase / term should generally refer to a file that conforms to a file format specified in ISO 14496-12 (i.e., ISOBMFF).

[0109] Therefore, as will be understood and recognized by those skilled in the art, unless otherwise expressly instructed, the foregoing terms and / or phrases may be used interchangeably.

[0110] Before going into the technical details of the corresponding exemplary embodiments, it may be worthwhile to first describe some possible use cases (scenarios) related to the preselection, especially from an abstract and broad overall perspective.

[0111] Refer to the attached diagram. Figure 1A This is a schematic diagram illustrating an exemplary implementation of a media playback application programming interface (API). In other words, Figure 1A This can be seen as an example use case for presenting (pre-selected) choices in a media player. More specifically, Figure 1A Examples are often considered as a possible implementation method that supports pre-selection by using conventional techniques.

[0112] In a broader sense, while some media content can be delivered using methods that typically utilize manifest files, such as MPEG DASH, some player frameworks may only use content in ISOBMFF format. Therefore, media players based on this architecture will be required to make presentation selections based solely on metadata information contained within the ISOBMFF, and thus can no longer rely on signaling from manifest files(s).

[0113] like Figure 1A As shown, media player 1200 receives input media (or files) 1000 to generate corresponding media output / presentation 1400. Input media 1000 can be (grouped) container format files or streams (e.g., ISOBMFF files or streams). Media player 1200 is controlled by a suitable application 1300. Such application 1300 and corresponding conventional controls of media player 1200 may include, but are not limited to, "play," "pause," "stop," "forward," "rewind," etc. However, for next-generation audio (or similar video technologies), media player 1200 may be required to present a list of available experiences (or presentations) to application 1300 and, in exchange, obtain desired selections from the application. This is typically implemented in API 1290 between media player 1200 and application 1300.

[0114] Generally, an incoming ISOBMFF 1000 provides (including) metadata 1010 in a container format-specific representation (e.g., implemented as one or more boxes or in any other suitable form) and media base stream(s) 1020. Specifically, such container format metadata 1010 can be arranged as byte data, which is typically codec-specific. Further, the base stream 1020 itself may include header information data (or in some cases, some kind of metadata) 1030 (which is typically in binary data form) and encoded (or compressed) media data 1040 (which is similar to the binary metadata 1030 and is typically in binary data form). As those skilled in the art will understand and recognize, broadly speaking, the base stream data 1040 can typically be represented using binary information (bit data), while the container data 1010 can typically utilize byte data.

[0115] In addition, such as Figure 1A As shown, the media player 1200 itself may include multiple decoder implementations (or instances) 1220 for various media formats (or in other words, format or codec specific). These multiple decoder implementations require the media player 1200 to correspondingly provide appropriate selection elements / components (or simply selectors) 1210 to unpack the underlying stream from the container format data and determine which media decoder 1220 to use. This decision is typically based on the container format metadata 1010 as described above.

[0116] like Figure 1AAs further shown, the preselection-supporting decoder 1220 provides (e.g., generates) preselection-related data 1230. Such codec-specific data 1230 must be converted by the corresponding format converter 1240 into a codec-independent general format 1250 so that it can be used by the application 1300 via the corresponding API 1290 (which is typically codec-independent).

[0117] Typically, operations related to unpacking the basic stream and converting codec / format-specific data into a codec-independent generic format can introduce (undesired) extra work into the entire program (e.g., for implementation, for testing, etc.) and may therefore be considered inefficient to some extent.

[0118] In order to solve the problem regarding Figure 1A Some or all of the issues discussed Figure 1B An exemplary implementation of a media playback API according to embodiments of the present disclosure is illustrated schematically. It is worth noting that, as previously indicated, unless otherwise indicated, Figure 1B The same or similar reference numerals in the figures can indicate Figure 1A For the sake of brevity, repeated descriptions of elements that are the same or similar to those in the text can be omitted.

[0119] In particular, with Figure 1A compared to, Figure 1B The media player 1201 shown no longer needs to read the corresponding API of each decoder 1221, reformat its output, and provide data in a format-independent manner.

[0120] Instead, by utilizing a new data structure 1051 (typically in byte format) associated with preselections (as will be discussed in more detail below) within the incoming ISOBMFF file or stream 1001, the media player 1201 can present the data 1051 to the application 1301 only at its corresponding API 1291. The main reason is that such preselection-related data 1051 is codec-independent.

[0121] As proposed, this approach typically does not require the implementation of multiple format-dependent metadata format converters (e.g., ...). Figure 1A(See format converter 1240). Furthermore, the proposed method implicitly defines a unified data structure that can be used to render preselected related metadata (e.g., the new data structure 1051) at the media player API 1291. In other words, utilizing the newly proposed data structure, the media player can now expose the contained information through the appropriate API to request the application to select the desired rendering (preselection). More specifically, the method operates in a codec-independent manner and provides a lightweight approach because it avoids the need to parse the encoded media nature.

[0122] In summary, as those skilled in the art will understand and recognize, the methods presented above for use cases of media player APIs generally offer at least the following benefits:

[0123] • The preselection is uniformly represented at the top level in data transmission formats such as ISO / BMFF;

[0124] • A unified data structure for media player APIs;

[0125] • No format-specific implementation is required in the media player; and

[0126] • Reduced workload for implementers, reduced testing workload, improved reliability, and faster industry acceptance.

[0127] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a system for processing media streams according to an embodiment of the present invention. Specifically, it can be seen that... Figure 2 This schematically illustrates the possible overall workflow of such a media processing system.

[0128] In particular, such as Figure 2 As shown, and regarding Figure 1A and Figure 1B Similar to the media player use case described, the (automatic) manifest generator 2500 may also need access to the experience (presentation) contained in the incoming media data. Typically, since a large number of codecs may have to be processed in such a processing system, these devices 2500 will preferably operate on codec-independent metadata. Furthermore, since data-side paths are generally undesirable for robust operation, any metadata structure to be used will likely need to be comprehensive and provide all the details that should be displayed in the manifest (or manifest file).

[0129] exist Figure 2In the example system workflow shown, media (e.g., audio and / or video) encoders 2100-A, 2100-B, and 2100-C typically generate encoded media streams 2200-A, 2200-B, and 2200-C in different formats (e.g., MP3, MP4, AC-4, etc.). These basic streams 2200-A–C are then fed to packers 2300-A, 2300-B, and 2300-C to encapsulate them into a uniform transport format, such as ISOBMFF.

[0130] Subsequently, each of these ISOBMFF files or streams 2400-A, 2400-B, and 2400-C can be distributed to the corresponding media player (not shown), or distributed to the manifest generator 2500 to generate the corresponding manifest file 2600 as output. Such files (e.g., MPEG DASH format files or HLS format files) can typically be represented in a machine- and / or human-readable format such as XML (Extensible Markup Language).

[0131] Now, for reference Figure 3A Example diagrams illustrating possible implementations of a packer (e.g., packer 2300-A–C) are provided.

[0132] In particular, such as regarding Figure 3A As shown, the incoming media basic stream 3200 consists of (binary) media header information or metadata 3210 and (binary) encoded / compressed (raw) media information / data 3220. Figure 2 The basic stream 2400-A–C (exemplified in the example) is then divided into container format output files or streams (e.g., ISOBMFF) 3400.

[0133] More specifically, during the process, the packer 3300 typically passes the incoming stream data 3200 to the corresponding output representation 3420 in an unchanged manner. A suitable internal device (or component) 3310 of the packer 3300 can read the binary header information / metadata 3210 to generate corresponding descriptive metadata information (but in byte format) 3410 for the media output 3400. Such a device (or component) 3310 of the packer 3300 can be, for example, a decoder-specific information (DSI) generator or any other suitable component. The data 3410 can later be processed by the corresponding media player (e.g., such as...). Figure 1A The media player 1200 shown is used to select the appropriate decoder (e.g., such as...). Figure 1AThe decoder 1220 is shown. In the exemplary use case of ISOBMFF, the descriptive byte data 3410 may include decoder-specific information (DST for short) and may also include some general annotations about the basic stream. As previously explained, when encapsulated in certain file formats such as ISOBMFF, the basic stream may be referred to as a track.

[0134] and Figure 3A Compared to the implementation method, Figure 3B Possible example implementations of a packer according to embodiments of the present invention are schematically illustrated. Similar to the above, unless otherwise indicated, Figure 3B The same or similar reference numerals in the figures can indicate Figure 3A For the sake of brevity, repeated descriptions of identical or similar elements can be omitted.

[0135] More specifically, such as regarding Figure 3B As shown, the packetizer 3301 may now include an additional (internal) device / component 3321 (which may be referred to as a preselected data generator in some cases) that also provides the device / component with the binary header information 3211 of the incoming media stream 3201. In some cases, the functionality of device 3321 may also be implemented as an additional function as part of the existing (internal) device 3311. However, as Figure 3B As clearly shown, in this case, an additional byte format-specified data structure 3451 is generated and embedded into the output stream 3400. It is worth noting that, as explained above, data structure 3451 typically carries pre-selected relevant information (e.g., descriptive information and / or processing-related information) and a media format-independent representation (codec-independent).

[0136] Figure 4A This is a schematic diagram of an example implementation of the manifest generator.

[0137] In particular, such as regarding Figure 4A As shown, the manifest generator 4500 typically reads an input file or stream 4400 (e.g., an ISOBMFF file) to generate the necessary information for the corresponding manifest file 4600. This required data may include information 4620 about the media stream, such as the appropriate media type (sometimes also called MIME type), codec attributes, and media-specific properties. All information can be generated independently of each media file or stream, based on data available from the container-level metadata structure 4410.

[0138] In addition to the information associated with these individual media files or streams, it may also be necessary to generate pre-selected related information 4610. In conventional techniques, the required information may typically need to be transmitted via, for example, some kind of outbound data path (such as...). Figure 4A(As shown) to provide or by parsing the binary header data of the encapsulated basic stream.

[0139] Especially in the latter case, the corresponding device / component 4510 (which may be referred to as a preselected data generator or, in some possible cases, any other suitable term) is therefore not only codec format-dependent but also requires computationally expensive binary data parsing and, depending on the media format used, additional knowledge about how to generate the corresponding preselected format from the available content component metadata. In some cases, the steps additionally require jointly reading the payloads of multiple media files or streams, making the process difficult and error-prone. In some possible cases, as those skilled in the art will understand and recognize, the manifest generator 4500 may also include another (internal) device / component 4520 that can be responsible for performing operations such as media-specific signaling (e.g., "AdaptationSet").

[0140] In order to solve the problem regarding Figure 4A Some or all of the issues discussed Figure 4B An exemplary implementation of a manifest generator according to embodiments of the present disclosure is illustrated schematically. It is worth noting that, as previously indicated, unless otherwise indicated, Figure 4B The same or similar reference numerals in the figures can indicate Figure 4A For the sake of brevity, repeated descriptions of identical or similar elements can be omitted.

[0141] In particular, such as regarding Figure 4B As shown, by utilizing the proposed pre-selected relevant data structure 4451, not only is the need for a codec-specific binary parser avoided, but the manifest generator 4501 can also rely on the pre-generated information of the proposed data structure 4451 from the incoming stream 4401. As those skilled in the art will understand and recognize, the data structure 4451 is already aligned with the desired output data 4611. Therefore, the manifest generator 4501 proposed herein typically only requires a pre-selected data converter 4531 that responds to converting byte information into a representation of the manifest file (e.g., XML), rather than... Figure 4A The preselected data generator 4510 shown is codec specific.

[0142] In summary, as those skilled in the art will also understand and recognize, the methods presented above for use cases in manifest generators generally offer at least the following benefits:

[0143] • Format-independent implementation method for pre-selected data processing in the manifest generator;

[0144] The implementation of the byte format only avoids computationally more expensive operations on binary data;

[0145] • Due to the high degree of overlap with existing functionality, implementing pre-selected metadata in the packer incurs low overhead;

[0146] • The manifest generator operates identically for all media formats; and

[0147] • Reduced workload for implementers, reduced testing workload, improved reliability, and faster industry acceptance.

[0148] As described above and elsewhere in this disclosure, the techniques presented herein can be applied to the processing of audio streams and video streams (or combinations thereof). Specifically, when video processing is involved, it may be worthwhile to discuss some potential use cases in more detail to further understand the invention.

[0149] One possible use case could include frame rate / resolution scalable video processing. More specifically, some possible MPEG codecs (e.g., in the context of Universal Video Codec (VCC) as defined in ISO / IEC 23090-3 (“MPEG-I Part 3”, also known as ITU-T H.266, etc.) could provide the option to create multiple display frame rates from the same stream. For example, a stream could be decoded to 50Hz or, at the cost of greater complexity, to 100Hz. As those skilled in the art will understand and recognize, the possibility of decoding a stream to any frame rate can be covered by the preselection concept presented in this disclosure. In a relevant use case, one stream would carry only the 50Hz preselected bits, while a second stream would provide the additional 100Hz preselected bits—such that to decode to 100Hz, the 100Hz preselection would reference both streams. Similarly, the resolution of the image can be scaled, rather than the frame rate, so the above description also applies to multi-resolution streams.

[0150] Another conceivable use case could involve joint SDR / HDR streaming. A specific use case could involve the concept of High Dynamic Range (HDR), which offers higher contrast and sometimes a wider color gamut (WCG) compared to traditional video in Standard Dynamic Range (SDR). Systems such as those providing backward-compatible enhancement layers could be used to create streams that can be decoded for SDR displays or HDR-capable displays. The possibility of obtaining different experiences from the same or multiple streams can be signaled through pre-selection associated with the different experiences.

[0151] Another video-related use case involves picture-in-picture (PIP) technology. Specifically, in this use case, a portion of a video component can be replaced by different content (e.g., through the concept of "sub-pictures" in VCC or any other suitable method). An example is a news program where a portion of the display is replaced by a video capture of a sign language interpreter explaining the content to a hearing-impaired audience. Here's another example where the final experience consists of distinct parts, which are pre-selected (collectively).

[0152] Having discussed the above possible use cases, now refer to Figures 5 to 8 The flowchart illustrating an example of a method for processing a media stream according to an embodiment of the present invention is shown in the illustration.

[0153] In particular, from a broad perspective, Figure 5 and Figure 6 The method stream processing approach can be viewed as two possible solutions for processing (via signal transmission) and pre-selected related information in a transmission format file (e.g., ISOBMFF). Depending on various implementations and / or circumstances, the two possible solutions can be implemented together or alternatively. On the other hand, Figure 7 and Figure 8 The method stream processing methods can be considered as corresponding to the above references respectively. Figure 3B and Figure 4B Possible use cases for the described packer and manifest generator.

[0154] More specifically, such as Figure 5 The method 5000 shown can begin from step S5100 by receiving a media stream packetized according to a predefined transmission format. As previously explained, the media stream can be an audio stream, a video stream, or a combination thereof, as described in the reference. Figure 3BThe packetizer 3301 prepares the media stream 3401. Method 5000 can be performed on the user side, or in other words, in a user (decoding) side environment. Depending on the implementation, the environment may include, but is not limited to, a TV, soundbar, web browser, media player, plugin, etc. The predefined transmission format may be the Basic Media File Format (ISOBMFF) specified by ISO through ISO / IEC 14496-12 MPEG-4 Part 12, or any other suitable (transmission) format. Specifically, the packetized media stream may include multiple layered boxes, each associated with a corresponding box type identifier. As mentioned above, the term "box" as used throughout this disclosure should not be construed as limited to such a specific term. Rather, the term "box" should generally be understood as any suitable data structure that can be used as a placeholder for media data in a packetized media stream and therefore can be referred to by other suitable terms. Further, depending on the implementation and / or requirements, multiple boxes may have the same or different levels (or positions), may be nested (child boxes and parent boxes), etc., as will be understood and recognized by those skilled in the art. Among other possibilities, multiple boxes can include one or more track boxes that reference (or in other words, indicate) the corresponding track of a media (content) component of the media stream. Broadly speaking, a media (content) component can generally refer to a single / separate consecutive component of media content (and can also typically be associated with corresponding (assigned) media content component types such as audio, video, and text). Examples for understanding the concept of a "media (content) component" can be found, for example, as defined / described in MPEG-DASH (ISO / IEC 23009-1).

[0155] Subsequently, in step S5200, method 5000 may include determining whether the media stream includes a preselected associated box of a predefined type indicating preselection, wherein preselection may correspond to media presentation to a user. As mentioned above, the term / phrase “preselection” is generally used to refer to a set of media content components (of the media stream) intended for common use (e.g., by user-side devices) and more particularly generally indicates a version of media presentation that can be selected by the end user for simultaneous decoding / presentation. Of course, as those skilled in the art will understand and recognize, in some other possible technical contexts, the term “preselection” may also be known (or mentioned) by using other suitable (equivalent) terms, such as (but not limited to) “presentation” as described, for example, in ETSITS 103190-2 or “preset” as described, for example, in ISO / IEC 23008-3, etc. Thus, a preselected associated box may be a specific box among multiple boxes in the media stream that has a specific predefined (or predetermined) type. As those skilled in the art will understand and recognize, such a specific type indicating preselection can be predefined (or predetermined) using any suitable means.

[0156] If it is determined in step S5300 that the media stream includes a preselected related box, then method 5000 may further include: at step S5310, analyzing metadata information corresponding to the preselected related box, the metadata information indicating the characteristics of the preselection; at step S5320, identifying one or more tracks in the grouped media stream that contribute to the preselection based on the metadata information; and finally at step S5330, providing one or more tracks for downstream processing according to the given preselection. As those skilled in the art will understand and recognize, according to various embodiments, the metadata information described above may be (directly) included in the media stream (or more specifically, multiple boxes of the (grouped) media stream) or may be (indirectly) obtained from the media stream using any suitable means. For example, the metadata information may be included in a head box, which may be associated with or linked to the preselected related box in some way (e.g., as its sub-box). As mentioned above, a preselection generally refers to a set of media content components intended for use by, for example, one or more suitable downstream devices (e.g., media decoders, media players, etc.). In some possible cases, the downstream device may also be simply referred to as a "sink point". Therefore, depending on the implementation method and / or requirements, downstream processing may include, but is not limited to, multiplexing (or re-multiplexing), sorting, merging, decoding, or rendering of those contributing tracks, as described above.

[0157] As configured as described above, the proposed method 5000 can generally provide an efficient and flexible way to identify / identify tracks within a media stream configured to contribute to a specific preselected track, and subsequently signal these tracks, thereby enabling (e.g., by one or more suitable downstream devices) further downstream processing of such contributing tracks. Therefore, broadly speaking, the proposed method can be viewed as providing the possibility and capability to signal information indicating preselection (and possibly processing of the preselection) in a uniform manner within a transport layer file (e.g., ISOBMFF), which can be considered beneficial in various use cases or scenarios, as referenced above. Figure 1A , 2 Those described in detail in 3B and 4B.

[0158] It is worth noting that, as those skilled in the art can understand and recognize, the International Organization for Standardization (ISO) and the International Electron Committee (IEC) have jointly published a functional document for implementing a certain technology. The functional document is titled: ISO / IEC 14496-12, Information technology—Encoding and decoding of audiovisual objects—Part 12: ISO Basic Media File Format (the latest version released in 2020 and available at https: / / www.iso.org / standard / 74428.html). Originally drafted by the Animation Experts Group (MPEG), the document specifies the ISO Basic Media File Format (or ISOBMFF for short), which is the general format upon which other more specific file formats are based.

[0159] As explained above, media information within ISOBMFF is typically structured in a hierarchical, object-oriented manner using building blocks called "boxes." These data structures are defined by unique type identifiers and lengths and can be nested or connected to form the overall file structure.

[0160] Individual media data (such as a single video or audio component) within a file format can be called a "track," which represents a basic stream pre-generated by the media encoder.

[0161] The file format contains timing, structure, and media information for the timing sequence of media data (such as audiovisual presentation). ISO / IEC 14496-12 already includes several means of describing the properties of media components. Such descriptive elements or dedicated boxes are assigned to separate tracks and subtracks within the 'moov' box. Such available boxes include 'extended language tag' boxes or 'kind' boxes.

[0162] To transmit the properties of a combination of tracks or subtracks using signals, as referenced above. Figure 5 In detail, as some possible implementations, this disclosure generally proposes to provide new data structures for use in parallel with, for example, the definition of tracks within a movie head (or any other suitable box). However, it should be noted that while as part of this proposal it is to associate each of these so-called preselected tracks with their constituent (contributing) media tracks, existing means for transmitting their respective properties by signaling should be reused as much as possible.

[0163] For example, in some possible implementations, two new boxes can be introduced, where one container box can be considered to form the counterpart of the DASH preselected element and treat all descriptive and structural metadata as sub-boxes; while the other box can be a (mandatory) header box that uses structural metadata to create the preselection. Of course, as detailed above, any further suitable boxes (e.g., processing related boxes) can also be included if desired. In this case, linking the preselection to the track can be achieved, for example, by utilizing a unique track identifier (ID) assignment in the track header of each track. Since the preselection is typically required to be referenced by external applications, a unique identifier should be assigned accordingly. In particular, the preselection identifier should be unique across the entire bundle, i.e., unique across all ISOBMFF files available for media rendering. Furthermore, since the encoded media format can use different assignments, it may be necessary to define and / or set (multiple) additional elements accordingly (e.g., using corresponding tags to map the corresponding identifiers used in the media format, etc.).

[0164] In view of this, in order to implement the above regarding Figure 5 The proposed method, particularly in order to support pre-selection within the context of ISOBMFF, may generally be suggested in some possible implementations by modifying (creating or replacing) Section 8 of ISO 14496-12 by the following text:

[0165] 3.1 Terms and Definitions

[0166] 3.1.x Pre-selection

[0167] This indicates a set of one or more media components that can be selected by the user for a version of media presentation that is simultaneously decoded / presented.

[0168] 3.1.y bundle

[0169] A set of multiple tracks that provide the overall accessible experience of media presentation, not necessarily contained in a single ISOBMFF file.

[0170] 8.18 Pre-selected structure

[0171] 8.18.1 Introduction

[0172] 8.18.2 Backward Compatibility

[0173] 8.18.3 Pre-selection box

[0174] definition

[0175] Box type: 'pres'

[0176] Containers: Movie Box ('moov'), Movie Clip Box ('moof')

[0177] Mandatory: No

[0178] Quantity: zero or more

[0179] This is a container box for a single preselection. A preselection provides a description of a specific end-user experience and its technical composition. Each available preselection in the media presentation should have its own preselection box.

[0180] The mandatory preselection head box contains the necessary references to all tracks that constitute the preselection. Additionally, the box also provides identifiers that can be used during the selection process.

[0181] The nature of preselected items should be indicated using supplementary boxes (such as extended language boxes or category boxes). When multiple preselected items share the same common nature, content authors should provide an appropriate textual description via tag boxes.

[0182] grammar

[0183] aligned(8)class PreselectionBox extends Box('pres'){

[0184] }

[0185] 8.18.3 Pre-selection head box

[0186] definition

[0187] Box type: 'prhd'

[0188] Container: Preselection box ('pres')

[0189] Mandatory: Yes

[0190] Quantity: Exactly one

[0191] A box specifies the characteristics of a single preselection. A preselection contains exactly one preselection head box.

[0192] `preselection_id` and `preselection_tag` provide preselection identifiers to the application or media codec, respectively. These can be used to select which elements to render.

[0193] Unless otherwise specified, `selection_priority` should be used to guide any automatic selection process. Evaluating the order of pre-selected elements to determine their priority should be avoided.

[0194] grammar

[0195]

[0196] Semantics

[0197] `version` is an integer specifying the version of the box (0 in this specification).

[0198] flags is a 24-bit integer with flags; no values ​​have been defined yet.

[0199] preselection_id is an integer that declares a unique identifier to external applications.

[0200] The preselection_tag is an integer that declares an identifier for the media codec used.

[0201] `selection_priority` is an integer that declares the pre-selected priority when other distinctions are not possible, such as through media language. The lower the number, the higher the priority.

[0202] n_tracks is an integer that declares the number of tracks required to form the preselection.

[0203] track_ID is an array of integers that uniquely identifies each track. For pre-selection, a list of all tracks in the array, referenced by track_ID values, is required.

[0204] 8.18.4 Labels and Group Label Boxes

[0205] definition

[0206] Box type: 'labl'

[0207] Containers: Preselected boxes ('pres') and other optional containers

[0208] Mandatory: No

[0209] Quantity: zero or more

[0210] Boxes can be used to annotate contained structures with text descriptions. The descriptions are intended to be presented to the user as text in some form, not for use in any automatic selection process.

[0211] Multiple boxes can be used to provide text descriptions in different languages.

[0212] grammar

[0213]

[0214] Semantics

[0215] label_id is an integer containing an identifier that can be used by external applications.

[0216] language is a NULL-terminated C string that contains language tag strings that conform to RFC 4646 (BCP 47), such as "en-US", "fr-FR", or "zh-CN".

[0217] The label is a NULL-terminated C string that contains a text description.

[0218] Therefore, it can be further suggested that Table 1 be updated by the rows highlighted in gray as shown below (in the same document ISO 14496-12).

[0219] Table 1 - Box Types, Structure, and Cross-Referencing (Informative)

[0220]

[0221]

[0222] Additionally, it can be further suggested that the new preselected head box be introduced as a container into some existing boxes. For example, it can be suggested to modify the existing box definition (in the same document ISO 14496-12) as follows:

[0223] 8.4.6 Extended Language Tags

[0224] 8.4.6.1 Definition

[0225] Box type: 'elng'

[0226] Containers: Media Box ('mdia'), Preselect Header Box ('prhd')

[0227] Mandatory: No

[0228] Quantity: zero or one

[0229] [...]

[0230] As already described, the modifications proposed above should be understood as merely one possible implementation example, and certainly not the only one. While some specific names have been given / proposed for the proposed boxes related to preselection above, these boxes may also be named with different names. Similarly, as those skilled in the art will understand and recognize, even though the preselection-related boxes proposed above appear to be nested under 'moov' boxes, these boxes may of course be implemented elsewhere. In one possible example (but certainly not any kind of limitation), these preselection-related boxes may be associated with what is called 'EntityToGroupBox' in ISOBMFF or any other suitable place (e.g., nested below it). Likewise, as those skilled in the art will understand and recognize, as the standard progresses and / or evolves, the above-described example tables (or elements therein) and example sections may have different names and / or (hierarchical) locations. In some possible cases, the above-described example tables (or portions thereof) and / or sections may even be partially or completely obsolete, making it possible that the corresponding preselection-related information may have to be defined elsewhere, which would then be considered appropriate.

[0231] Now for reference Figure 6 The diagram schematically illustrates a flowchart of another possible example of a method 6000 for processing a media stream according to an embodiment of the present invention. As described above, Figure 6 The method can generally be considered as described above. Figure 5 An alternative to the method (or, in some possible cases, as an additional option).

[0232] Specifically, method 6000 can begin from step S6100 by receiving a media stream packetized according to a predefined transmission format. Similar to the above, the media stream can be an audio stream, a video stream, or a combination thereof, as described in the reference. Figure 3B The media stream 3401 prepared by the packetizer 3301. Method 6000 can also be performed on the user side, or in other words, in a user (decoding) side environment. Depending on the implementation, the environment may include, but is not limited to, a TV, soundbar, web browser, media player, plugin, etc. The predefined transmission format may be the basic media file format (or simply ISOBMFF) specified by ISO through ISO / IEC 14496-12 MPEG-4 Part 12, or any other suitable (transmission) format. Specifically, the grouped media stream may include multiple layered boxes, each associated with a corresponding box type identifier. Among other possibilities, the multiple boxes may include one or more track boxes that reference (or in other words, indicate) the corresponding track of the media (content) component of the media stream.

[0233] Subsequently, in step S6200, method 6000 may include examining (e.g., accessing, looping, etc.) trackboxes in the media stream to determine a complete (or total) set of preselections present in the media stream. Specifically, determining the complete set of preselections may include: identifying a unique pair consisting of a track group identifier and a track group type; and addressing the preselection by the corresponding track group identifier. As described above, each preselection is associated with a corresponding track group, which itself is identified by a corresponding pair consisting of a corresponding track group identifier and a corresponding track group type. Therefore, a preselection can be addressed (or identified) by the corresponding track group identifier associated with / linked to it.

[0234] Method 6000 may further include selecting a preselection from the complete set of preselections at step S6300. In particular, the preselection may be based on the attributes (e.g., represented as metadata or other suitable form) of the corresponding preselection included in the orbital group box having the same orbital group identifier.

[0235] Subsequently, at step S6400, method 6000 may include determining a group of one or more track boxes that contribute to the selected preselection. Specifically, a group of one or more track boxes that contribute to the (same) preselection can be identified by the presence of (corresponding) track group boxes having the same track group identifier.

[0236] Additionally, at step S6500, method 6000 may include determining the track referenced in each member (element) of a set of one or more track boxes as defined above as contributing to one or more preselected tracks.

[0237] Finally, at step S6600, method 6000 may include providing one or more tracks for downstream processing according to a preselection. As mentioned above, a preselection typically refers to a set of media content components intended for use by one or more suitable downstream devices (or, in some possible cases, a sink) such as a media decoder, media player, etc. Therefore, depending on various implementations and / or requirements, downstream processing may include, but is not limited to, multiplexing (or re-multiplexing), sorting, merging, decoding, or rendering those contributing tracks, as will be described in more detail below.

[0238] As configured as described above, the proposed method 6000 typically provides an efficient and flexible way to identify / identify tracks within a media stream configured to contribute to a specific preselected track, and subsequently signal these tracks, thereby enabling (e.g., by one or more suitable downstream devices) further downstream processing of such contributing tracks. In particular, it should be noted that the method described above in the first aspect typically seeks to provide relevant information in a preselected correlation box about (all) tracks configured to contribute to a specific preselected track, thereby enabling efficient indexing (or identification) of all contributing tracks. In this sense, as Figure 5 The indexing of orbits described in method 5000 above can be considered a kind of forward (direct) indexing. In contrast, in Figure 6 In the method 6000 shown, tracks contributing to a specific preselection can be determined by a combination of track group type and track group identifier. More specifically, tracks having (e.g., containing) track group boxes with a specific (e.g., predefined or predetermined) track group type can generally indicate that the track contributes to the preselection. Further, tracks having the same track group identifier can generally indicate that those tracks belong to (contribute to) the same preselection. In this sense, with Figure 5 Compared to the one proposed in the middle, the reference Figure 6 The described indexing of the orbits can be viewed as a kind of inverted index. In any case, similar to... Figure 5 , Figure 6 The method 6000 proposed in [the document] can also provide the possibility and capability to transmit information indicating preselection (and possibly processing of preselection) in a uniform manner within transport layer files (e.g., ISOBMFF), which can be considered beneficial in various use cases or scenarios, as referenced above. Figure 1A , 2 Those described in detail in 3B and 4B.

[0239] It is worth noting that, with Figure 5 Similarly, in order to implement the above regarding Figure 6 The proposed method 6000, particularly in order to support pre-selection within the context of ISOBMFF, in some possible implementations, can generally be suggested to modify (create or replace) ISO 14496-12 (as referenced) by means of the following text. Figure 5 (The proposed text is an alternative or supplement):

[0240] 3.1 Terms and Definitions

[0241] 3.1.x Pre-selection

[0242] This indicates a set of one or more media components that can be selected by the user for a version of media presentation that is simultaneously decoded / presented.

[0243] 3.1.y Media Components

[0244] In general, it is recommended to modify section 8.3.4.3 of ISO 14496-12 using the following underlined line:

[0245] track_group_type indicates the grouping_type and should be set to one of the following values, or a registered value, a derived specification, or a value in the registration:

[0246] 'msrc' indicates that the track belongs to multi-source rendering. It is specified in 8.3.4.4.1.

[0247] The 'ster' indicator track is the left or right view of a stereoscopic image pair suitable for playback on a stereoscopic display. It is specified in 02.

[0248] 'pres' indicates the orbital contribution to the preselection. Specified in 8.3.4.4.3.

[0249] The track_group_id and track_group_type pairs identify track groups within a file. Tracks containing a specific TrackGroupTypeBox with the same track_group_id and track_group_type values ​​belong to the same track group.

[0250] In general, it is recommended to modify (or create) section 8.3.4.4.3 of ISO 14496-12 using the following lines:

[0251] 8.3.4.1.3 Pre-selection box

[0252] 8.3.4.1.3.1 Definition

[0253] The TrackGroupTypeBox with track_group_type equal to 'pres' indicates the track contribution to the preselection.

[0254] Tracks with the same track_group_id value within a PreselectionGroupBox are part of the same preselection.

[0255] Preselection can be limited by language, genre, or media-specific attributes (such as audio rendering instructions or channel layout). Attributes signaled in the preselection box should take precedence over attributes signaled in contributing tracks.

[0256] All uniquely qualified preselected attributes should exist in at least one preselected box. If they exist in more than one preselected box, the boxes should be identical.

[0257] Note: Pre-selection only groups tracks of the same media type.

[0258] Tracks that do not contain at least one of the required preselected media components should have the `track_in_movie` flag set to '0' in their track header box. This prevents players unfamiliar with the preselected boxes from playing tracks that would result in an incomplete experience.

[0259] Note: It is good practice to set the track_in_movie flag of a track to one. This means that the track provides at least one complete experience.

[0260] 8.3.4.1.3.2 Syntax

[0261]

[0262] 8.3.4.1.3.3 Semantics

[0263] `selection_priority` is an integer that declares the pre-selected priority when other distinctions are not possible, such as through media language. The lower the number, the higher the priority.

[0264] The order specifies the consistency rules for representations in the preselected adapter set according to [MPEG-DASH] from the following enumerated set:

[0265] 0: Undefined

[0266] 1: Sort by time

[0267] 2: Complete sorting

[0268] 8.3.4.1.3.4 Pre-selected Information Box

[0269] 8.3.4.1.3.4.1 Definition

[0270] Box type: 'prsi'

[0271] Container: Pre-selection box

[0272] Mandatory: Yes

[0273] Quantity: Exactly one

[0274] The box gathers all the semantic information about the preselection.

[0275] 8.3.4.1.3.4.2 Syntax

[0276]

[0277] 8.3.4.1.3.4.3 Semantics

[0278] 8.3.4.1.3.4.5 Pre-selection processing box

[0279] 8.3.4.1.3.4.5.1 Definition

[0280] Box type: 'prsp'

[0281] Container: Pre-selection box

[0282] Mandatory: Yes

[0283] Quantity: Exactly one

[0284] The box contains information about how contributions to the pre-selected tracks should be processed. Media type-specific boxes can be used to describe further processing.

[0285] 8.3.4.1.3.4.5.2 Syntax

[0286]

[0287] 8.3.4.1.3.4.5.3 Semantics

[0288] `preselection_tag` is an integer containing the tag's identifier. Tags with the same value belong to the same tag group. The default value of zero indicates that the tag does not belong to any tag group.

[0289] `track_order` defines the order in which tracks should be provided to the decoder. Tracks with lower `track_order` values ​​should be provided to the decoder first. If multiple tracks have the same `track_order` value, the order is irrelevant.

[0290] sample_merge_flag: If the flag is set to '1', each sample of a track should be appended to the sample of the next track with a lower track_order value. If it is set to '0', the track should be provided to a separate decoder instance.

[0291] Further suggestions could be made to create (or modify or replace) section 8.18.4 of ISO 14496-12 as follows:

[0292] 8.18.4 Labels and Group Label Boxes

[0293] definition

[0294] Box type: 'labl'

[0295] Containers: User data box ('udta') and preselection box ('pres') in the orbit.

[0296] Mandatory: No

[0297] Quantity: zero or more

[0298] Tags provide the ability to annotate data structures in ISOBMFF files, offering a description of the context to which the tag is assigned. Such tags can, for example, be used by playback clients to provide selection options to the user. Tags can also be used for simple annotations within another context.

[0299] Additionally, a GroupLabel element can be added at a higher level to provide a summary or title of the labels collected in the group. An example could be its use in a menu to provide context for the label's menu item.

[0300] Multiple tags can be used to provide text descriptions. To annotate preselected languages ​​for a multilingual audience, annotations can be provided in languages ​​different from the preselected languages.

[0301] If is_group_label is set to a value other than zero, the label text in the box will specify a summary or title for all labels with the same label_id. This can be used as a title on a selection menu that contains a set of labels.

[0302] grammar

[0303]

[0304] Semantics

[0305] is_group_label specifies whether the label includes a summary label for the label group.

[0306] `label_id` is an integer containing the label's identifier. Labels with the same value belong to the same label group. The default value of zero indicates that the label does not belong to any label group.

[0307] `language` is a NULL-terminated C string that contains language tag strings conforming to RFC 4646 (BCP 47), such as "en-US", "fr-FR", or "zh-CN". `language` is the target language of the tag.

[0308] The label is a NULL-terminated C string that contains a text description.

[0309] Further suggestions could be made to create (or modify or replace) section 8.18.4 of ISO 14496-12 as follows:

[0310] 8.18.5 Audio Rendering Indicator Box

[0311] definition

[0312] Box type: 'ardi'

[0313] Container: Preselection box ('pres')

[0314] Mandatory: No

[0315] Quantity: zero or one

[0316] The audio rendering indicator box contains prompts for optimizing the playback channel layout.

[0317] grammar

[0318]

[0319] Semantics

[0320] The audio_rendering_indication contains hints about the preferred playback channel layout based on the encoding / decoding in Table 2.

[0321] Table 1 - Encoding and Decoding of Audio Rendering Instructions

[0322] 0 No preference for the reproduction channel layout is given. 1 The preferred reproduction channel layout is three-dimensional. 2 The preferred reproduction channel layout is two-dimensional (e.g., 5.1 multi-channel). 3 The preferred reproduction channel layout is three-dimensional. 4 Pre-render the content for use with headphones. 5 to 255 Reserved for future use

[0323] It is worth noting that, as those skilled in the art will understand and recognize, the newly proposed pre-selected related boxes, as described above, can be further suggested for reference. Figure 5 The description in Table 1 above is for the sake of brevity, so that its details are not repeated.

[0324] Additionally, it could be suggested that new preselected head boxes be introduced as containers into some existing boxes. For example, it could be suggested to modify existing box definitions by highlighting text through underlining or deletion:

[0325] 8.4.6 Extended Language Tags

[0326] 8.4.6.1 Definition

[0327] Box type: 'elng'

[0328] Container: Media Box ('mdia') Preselection box ('pres') )

[0329] Mandatory: No

[0330] Quantity: zero or one

[0331] [...]

[0332] 8.10.1 User Data Box

[0333] 8.10.4.1 Definition

[0334] Box type: 'udta'

[0335] Containers: Movie boxes, track boxes, movie clip boxes, or track clip boxes or pre-selection box

[0336] Mandatory: No

[0337] Quantity: zero or one

[0338] [...]

[0339] 8.10.4 Types of Tracks

[0340] 8.10.4.1 Definition

[0341] Box type: 'Category'

[0342] Container: The container user data box ('udta') in the orbit or Preselection box ('pres')

[0343] Mandatory: No

[0344] Quantity: zero or one

[0345] 12.2.4 Channel Layout

[0346] 8.10.4.1 Definition

[0347] Box type: 'chnl'

[0348] Container: Audio sample input or pre-selection box

[0349] Mandatory: No

[0350] Quantity: zero or one

[0351] [...]

[0352] Similarly, as already described, refer to Figure 6 The modifications described above should be understood as merely one possible implementation example, and certainly not the only one. Although some specific names have been given / proposed for the proposed boxes related to the preselection above, these boxes can also be named with different names. Similarly, as those skilled in the art will understand and recognize, even though the preselection-related boxes proposed above may appear to be associated with a particular box, these boxes can certainly be implemented elsewhere.

[0353] Now for reference Figure 7 The diagram schematically illustrates an example flowchart of a method 7000 for processing a media stream according to an embodiment of the present invention. Similar to the above, the media stream can be an audio stream, a video stream, or a combination thereof. It is worth noting that method 7000 can be executed on an encoding-side environment (e.g., a media encoder). In some scenarios (or use cases), such an encoder may also be referred to as a (media) packer (i.e., configured to pack / group media input), as illustrated in the example. Figure 2 2300-A–C or Figure 3B 3301 in the middle.

[0354] Specifically, method 7000 may include, at step S7100, encapsulating one or more basic streams according to a predefined transport format to generate a packetized media stream, wherein the packetized media stream comprises multiple layered boxes, each layered box being associated with a corresponding box type identifier. Similar to the above, the predefined transport format may be a basic media file format (or ISOBMFF for short) specified by ISO through ISO / IEC 14496-12 MPEG-4 Part 12, or any other suitable (transport) format.

[0355] More specifically, step S7100 of encapsulating one or more elementary streams may include: at step S7110, grouping media data of one or more elementary streams according to a transmission format to generate one or more track boxes that reference (or indicate) corresponding tracks of one or more elementary streams; and at step S7120, generating one or more preselected related boxes of a predefined type based on header information of one or more elementary streams, wherein each of the one or more preselected related boxes indicates a corresponding preselection corresponding to media presentation to the user.

[0356] As configured as described above, the proposed method generally provides an efficient and flexible way to group media inputs (e.g., elementary streams) according to a predefined transport format (e.g., ISOBMFF). More specifically, by generating and including one or more preselection-related boxes (each preselection-related box indicating a corresponding preselection) alongside the grouped media streams, the proposed method also enables the representation of preselections in a codec-independent uniform manner, thereby further enabling appropriate downstream processing of tracks contributing to the corresponding preselections (e.g., according to the methods proposed in the first and second aspects above). Additionally, as mentioned above, this uniform representation of preselections also enables format-independent implementations of preselection data processing in manifest generators (e.g., MPEG HTTP-based dynamically adaptive streaming DASH format files or HTTP real-time streaming HLS format files), thereby avoiding the need for computationally more expensive operations on binary data, while simultaneously reducing implementation effort and improving reliability.

[0357] Figure 8 This is a schematic flowchart illustrating yet another example of a method 8000 for processing a media stream according to an embodiment of the present invention. The media stream may be an audio stream, a video stream, or a combination thereof. Method 8000 may be generated by a manifest generator (e.g., as exemplified). Figure 2 2500 or Figure 4B The manifest generator (4501 in the document) is executed.

[0358] Specifically, method 8000 may include receiving a media stream packetized according to a predefined transmission format at step S8100. More specifically, the packetized media stream may include a plurality of layered boxes, each layered box being associated with a corresponding box type identifier, wherein the plurality of boxes may include one or more track boxes and one or more preselected associated boxes of a predefined type, the one or more track boxes referencing (e.g., indicating) a corresponding track of a media component of the media stream, and wherein each preselected associated box indicates a corresponding preselection corresponding to media presentation to the user.

[0359] Furthermore, method 8000 may also include generating a manifest file based on one or more pre-selected related boxes at step S8200.

[0360] As configured as described above, the proposed method can typically provide an efficient and flexible approach to generating manifest files by also considering pre-selected relevant information (e.g., descriptive information and / or processing-related information). More specifically, in addition to information related to the media stream(s), the proposed method can further include information associated with pre-selection. Pre-selected relevant information can include, but is not limited to, metadata information, processing information, etc. Compared to conventional (manifest generation) techniques, the proposed method for generating manifest files can typically provide a format-independent implementation of pre-selected relevant data processing, thereby avoiding computationally more expensive operations (e.g., operations that must be performed on binary data in conventional techniques), reducing implementation and / or testing workload, and improving reliability.

[0361] In summary, the methods presented and described above with reference to the accompanying drawings generally relate to techniques for processing media streams by considering "pre-selection." Such "pre-selection" awareness techniques can enable a variety of potential use cases. As mentioned above, one possible use case could include selecting from several languages ​​(subtitles).

[0362] Another possible use case could involve narrative importance. Specifically, to better meet the needs of hearing-impaired individuals or to adapt audio works to different hearing conditions, techniques such as dialogue enhancement may be necessary. Generally, these techniques can be used to increase the ratio of dialogue level to background level. As an extension of such dialogue enhancement techniques, additional measures have been proposed to selectively and progressively remove entire audio elements from a work to improve dialogue intelligibility. In this case, the continuum of the "complete material" can be mapped to multiple preselected elements using "dialogue only" and the methods disclosed herein can be employed.

[0363] Further conceivable use cases could involve audience targeting. More specifically, there might sometimes be a focus on a specific audience beyond language. As an example (but not in any way), two sports commentators, each biased against either team, could be provided. In this case, different pre-selections could include or reference different sports commentators.

[0364] Further relevant use cases for pre-selection could involve playback environment adaptation. That is, content creators can generate dedicated versions for different playback environments (such as home theater settings) compared to the version provided by the TV's built-in speakers or headphones. In this case, different pre-selections could involve different audio for different playback environments.

[0365] Finally, the present invention also relates to (a number of) means for performing the methods and techniques described throughout the disclosure. Figure 9 An example of such a device 9000 is generally shown. Specifically, the device 9000 includes a processor 9100 and a memory 9200 coupled to the processor 9100. The memory 9200 may store instructions for the processor 9100. Depending on various use cases and / or implementations, the processor 9100 may also, in particular, receive input data (e.g., media input, packetized media streams, etc.). Depending on various use cases and / or implementations, the processor 9100 may be adapted to perform the methods / techniques described throughout this disclosure (e.g., methods 5000, 6000, 7000, and 8000 as described above) and correspondingly generate output data 9400 (e.g., packetized media streams, manifest files, etc.). For example, according to embodiments of the invention, the device 9000 may implement a packager as needed, the packager being configured to perform the above-described... Figure 7 The described method for processing media streams 7000; or implementing a manifest generator, which is configured to perform the above-described methods. Figure 8 The method for processing media streams described is 8000.

[0366] explain

[0367] Computing devices implementing the techniques described above can have the following example architectures. Other architectures are also possible, including those with more or fewer components. In some implementations, the example architectures include one or more processors (e.g., dual-core). The components include a processor, one or more output devices (e.g., an LCD), one or more network interfaces, one or more input devices (e.g., a mouse, keyboard, touchscreen display), and one or more computer-readable media (e.g., RAM, ROM, SDRAM, hard disk, optical disk, flash memory, etc.). These components can exchange communication and data via one or more communication channels (e.g., buses), which can utilize various hardware and software to facilitate the transfer of data and control signals between the components.

[0368] The term "computer-readable medium" refers to a medium that participates in providing instructions to a processor for execution, including but not limited to non-volatile media (e.g., optical discs or magnetic disks), volatile media (e.g., memory), and transmission media. Transmission media include, but are not limited to, coaxial cables, copper wires, and optical fibers.

[0369] Computer-readable media may further include an operating system (e.g., The system comprises an operating system, a network communication module, an audio interface manager, an audio processing manager, and a real-time content distributor. The operating system can be multi-user, multi-processor, multi-tasking, multi-threaded, real-time, etc. The operating system performs basic tasks, including but not limited to: identifying input from network interfaces and / or devices and providing output to network interfaces and / or devices; recording and managing files and directories on computer-readable media (e.g., memory or storage devices); controlling peripheral devices; and managing traffic on one or more communication channels. The network communication module includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols such as TCP / IP and HTTP).

[0370] The architecture can be implemented in parallel processing or peer-to-peer infrastructure, or on a single device with one or more processors. The software can include multiple software components or can be a single code body.

[0371] The described features can be advantageously implemented in one or more computer programs that can execute on a programmable system including at least one programmable processor, the programmable processor being coupled to receive and transfer data and instructions from a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used directly or indirectly in a computer to perform an activity or produce a result. Computer programs can be written in any form of programming language (e.g., Objective-C, Java), including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, browser-based web application, or other unit suitable for use in a computing environment.

[0372] For example, suitable processors for executing instruction programs include both general-purpose processors and special-purpose processors, as well as a single processor or one or more processors or cores in any type of computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data files, or operatively coupled to and communicating with them; such devices include disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly representing computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM discs. The processor and memory can be supplemented by or incorporated into an ASIC (Application-Specific Integrated Circuit).

[0373] To provide interaction with the user, features can be implemented on a computer that has a display device for showing information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or retina display device. The computer may have touch surface input devices (e.g., touchscreens) or a keyboard and pointing devices such as a mouse or trackball, through which the user can provide input to the computer. The computer may also have a voice input device for receiving voice commands from the user.

[0374] Features can be implemented in computer systems, which may include back-end components such as data servers, or middleware components such as application servers or internet servers, or front-end components such as client computers with graphical user interfaces or internet browsers, or any combination thereof. The components of the system can be connected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include, for example, LANs, WANs, and computers and networks that form the internet.

[0375] A computing system may include clients and servers. Clients and servers are typically located far apart and interact via a communication network. The client-server relationship arises from computer programs running on respective computers and having a client-server relationship relative to each other. In some embodiments, the server transmits data (e.g., HTML pages) to the client device (e.g., to display data to a user interacting with the client device and to receive user input from the user). Data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0376] A system of one or more computers may be configured to perform specific actions by means of software, firmware, hardware, or a combination thereof installed on the system that will cause the system to perform actions in operation. One or more computer programs may be configured to perform specific actions by means of instructions that, when executed by a data processing device, cause the device to perform those actions.

[0377] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or claim, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Specific features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in a particular combination and even initially claimed, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0378] Similarly, although the operations are depicted in a specific order in the accompanying figures, this should not be construed as requiring the operations to be performed in the specific order shown or sequentially, or to perform all the illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

[0379] Furthermore, the separation of various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0380] Unless otherwise specifically stated, it is obvious from the following discussion that, throughout the discussion of the invention, terms such as “processing,” “computing,” “calculating,” “determining,” and “analyzing” are used to refer to the actions and / or processes by which a computer or computing system or similar electronic computing device manipulates and / or transforms data represented as physical (e.g., electronic) quantities into other data similarly represented as physical quantities.

[0381] Throughout the invention, references to "one exemplary embodiment," "some exemplary embodiments," or "exemplary embodiment" mean that a particular feature, structure, or characteristic described in connection with an exemplary embodiment is included in at least one exemplary embodiment of the invention. Therefore, the phrases "in one exemplary embodiment," "in some exemplary embodiments," or "in an exemplary embodiment" appearing throughout the invention do not necessarily refer to the same exemplary embodiment. Furthermore, in one or more exemplary embodiments, specific features, structures, or characteristics can be combined in any suitable manner, as will be apparent to those skilled in the art according to the invention.

[0382] As used herein, unless otherwise specified, ordinal adjectives such as “first,” “second,” “third,” etc., are used to describe common objects only to indicate different instances of similar objects and are not intended to imply that the objects described must be in a given order in time, space, hierarchy, or any other way.

[0383] Similarly, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of “including,” “comprising,” or “having,” and variations thereof, is intended to cover the items listed thereafter and their equivalents, as well as additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “coupled,” and variations thereof, are used extensively and cover direct and indirect installation, connection, support, and coupling.

[0384] In the claims below and in the description herein, the terms *comprising*, *comprised of*, or *which comprises* are open-ended terms meaning that at least the following element / feature is included, but not excluding other elements / features. Therefore, when the term *comprising* is used in a claim, the term should not be construed as limited to the devices, elements, or steps listed thereafter. For example, the scope of the expression *devices including A and B* should not be limited to devices that only include elements A and B. As used herein, the terms *including*, *which includes*, or *that includes* are also open-ended terms meaning that at least the element / feature following the term is included, but not excluding other elements / features. Therefore, *including* is synonymous with *comprising* and means *comprising*.

[0385] It should be understood that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes combined together in a single exemplary embodiment / figure or its description in order to simplify the invention and aid in understanding one or more of the inventive aspects. However, the method of the invention should not be construed as reflecting an intention in the claims to require more features than those expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in fewer than all the features of a single foregoingly disclosed exemplary embodiment. Therefore, the claims accompanying this specification are hereby expressly incorporated, wherein each claim is independently considered as a separate exemplary embodiment of the invention.

[0386] Furthermore, while some exemplary embodiments described herein include features that are included in other exemplary embodiments but not others, as those skilled in the art will understand, combinations of features from different exemplary embodiments are intended to be within the scope of the invention and to form different exemplary embodiments. For example, any exemplary embodiment of the claimed embodiments in the appended claims can be used in any combination.

[0387] Numerous specific details are set forth in the description provided herein. However, it should be understood that exemplary embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail in order to avoid obscuring the understanding of this specification.

[0388] Therefore, although the mode considered to be the best mode of the invention has been described, those skilled in the art will recognize that other and further modifications can be made thereto without departing from the spirit of the invention, and all such changes and modifications falling within the scope of the invention are intended to be claimed. For example, any formulas given above merely indicate procedures that can be used. Functions can be added or removed from the block diagrams, and operations can be interchanged within functional blocks. Steps can be added or removed from the methods described within the scope of this disclosure.

[0389] The foregoing has described exemplary embodiments (“EEE”) of this disclosure with respect to methods and systems for determining an indication of the audio quality of an audio input. Therefore, embodiments of the invention may relate to one or more of the following enumerated examples:

[0390] EEE 1. A method for decoding an encoded media bitstream in a multiplexed format, the multiplexing format including preselection boxes for a version list, the version list having the properties of the version, the preselection boxes being independent of the encoded media format:

[0391] Determine the list of versions with the properties of the selection method from the preselection box;

[0392] The encoded media bitstream is decoded based on the selection method to output a playable audio version.

[0393] EEE 2. The method according to EEE 1, wherein the selection method is a UI exposed to the end user, the UI visualizing the version and property for performing an on-the-spot selection.

[0394] EEE 3. The method according to EEE 1, wherein the selection method is an automatic process based on user preference settings.

[0395] EEE 4. The method according to EEE 1, wherein the selection method is an automated process based on information about the terminal device, the playback geographic area, or other data characteristics.

[0396] EEE 5. The method according to any one of EEE 1 to 4, wherein the multiplexing format is ISOBMFF, transport stream, or MXF.

[0397] EEE 6. The method according to EEE 5, wherein the media bitstream includes an encrypted media payload, and wherein the encoding format is plaintext.

[0398] EEE 7. The method according to any one of EEE 1 to 4, wherein the decoding method is used for a specialized playback architecture, a specialized type of media, or a specialized pre-selection for audio, video, and virtual reality.

[0399] EEE 8. The method according to any one of EEE 1 to 4, wherein the multiplexing format includes information about user selections for pre-downloading.

[0400] EE 9. A method for packaging coded media assets for transmission, wherein the transmission format encompasses a manifest file listing the assets, the method comprising:

[0401] The manifest file is packaged, wherein the manifest file contains a list of available versions and their properties to be obtained from a single-stream or multi-stream asset, and wherein the list can be obtained from the asset without accessing codec-specific information.

[0402] EEE 10. The method according to EEE 9, wherein the manifest file is in MPEG DASH, HLS or SDP format.

[0403] EE 11. A method for processing a media stream, the method comprising:

[0404] Receive the media stream packetized according to a predefined transmission format, wherein the packetized media stream includes a plurality of layered boxes, each of which is associated with a corresponding box type identifier, wherein the plurality of boxes includes one or more track boxes and one or more track group boxes, the one or more track boxes referencing tracks that indicate media components of the media stream, each of the one or more track group boxes being associated with a corresponding track group identifier and a corresponding track group type that jointly identify a corresponding track group within the media stream, and wherein tracks having the same track group identifier and the same track group type belong to the same track group; and wherein each such track group determines a preselection;

[0405] Access all trackboxes in the media stream to determine the pre-selected total set;

[0406] The determination is performed by identifying a set of all unique pairs consisting of a track group identifier and a track group type, and addressing the preselection by the track group identifier.

[0407] The preselection is selected based on the preselected attributes found in all track group boxes with the same track group identifier;

[0408] Determine a group of one or more track boxes that contribute to the preselection, the group of one or more track boxes being identified by the presence of track group boxes having the same track group identifier; and

[0409] Identify the contributing tracks referenced in each member of the set of track boxes.

Claims

1. A method for processing media streams, comprising: Receive the media stream packetized according to a predefined transmission format, wherein the packetized media stream includes a plurality of layered boxes, each of which is associated with a corresponding box type identifier, and wherein the plurality of boxes includes one or more track boxes that reference a corresponding track indicating a media component of the media stream; Determine whether the media stream includes a preselected related box indicating a predefined type, wherein the preselection corresponds to media presentation to the user; and If it is determined that the media stream includes the pre-selected relevant box, then: Analyze the metadata information corresponding to the preselected related boxes, the metadata information indicating the characteristics of the preselection; The metadata information identifies one or more tracks in the media stream of the group that contribute to the pre-selected tracks; and One or more tracks are provided for downstream processing according to a given preselection. The media stream further includes processing information indicating how to process contributions to the preselected track. The processing information includes sorting information, which indicates the orbital order for processing the one or more orbits; The processing information includes merging information, which indicates whether one or more orbits should be merged with one or more other orbits for joint processing; The method further includes: Merge one or more tracks according to the merging information and the sorting information; Wherein, for each track contributing to the preselection, the sorting information includes a corresponding track order value for defining the track order; The merging information includes a corresponding merging flag for contributing to each of the pre-selected tracks, wherein a first setting value of the merging flag indicates that the corresponding track will be merged with adjacent tracks in the track order, and a second setting value of the merging flag indicates that the corresponding track will be processed separately; and Merging the one or more tracks according to the merging information and the sorting information includes: The orbits are scanned sequentially according to the orbital order; and Merge tracks according to the corresponding merge flags.

2. The method according to claim 1, further comprising: The one or more tracks are decoded according to the media presentation indicated by the preselection for playback of the media stream.

3. The method according to claim 2, wherein, One or more tracks are decoded by downstream devices.

4. The method according to claim 2, wherein, The merging of one or more tracks and the decoding of one or more tracks are performed by a single device.

5. The method according to any one of claims 1 to 4, wherein, The media stream includes a plurality of preselected related boxes of the predefined type, and the method further includes selecting the preselected related box from the plurality of preselected related boxes.

6. The method according to claim 5, wherein, The pre-selected related boxes are selected by the application.

7. The method according to claim 5, wherein, The media stream includes one or more tab boxes, each of which includes descriptive information about the media presentation for the user corresponding to a corresponding preselection; and The selection of the pre-selected related boxes is based on the user's input.

8. The method according to any one of claims 1 to 4, wherein, The preselected correlation box is independent of the media codec used to encode the media stream before it is grouped.

9. The method according to any one of claims 1 to 4, wherein, The metadata information corresponding to the preselected related box includes track identification information indicating one or more track identifiers, each of which is associated with a corresponding track, wherein the track associated with the one or more track identifiers in the metadata information is related to the media presentation.

10. The method according to any one of claims 1 to 4, wherein, The metadata information corresponding to the preselected related box includes preselection identification information, which indicates a preselection identifier used to identify the preselection.

11. The method according to any one of claims 1 to 4, wherein, The metadata information corresponding to the preselected related box includes unique preselected specific data for configuring downstream devices to decode the track according to the preselection.

12. The method according to any one of claims 1 to 4, wherein, The media stream includes at least one of an audio stream or a video stream.

13. The method according to any one of claims 1 to 4, wherein, The media presentation to the user is characterized by a corresponding configuration associated with the language, type, and / or one or more media-specific attributes of the media stream.

14. The method according to any one of claims 1 to 4, wherein, The predefined transmission format is the ISO Basic Media File Format (ISOBMFF).

15. A method for processing a media stream, comprising: The media stream is received in packets according to a predefined transmission format, wherein the packetized media stream includes a plurality of layered boxes, each of which is associated with a corresponding box type identifier, wherein the plurality of boxes includes one or more track boxes and one or more track group boxes, the one or more track boxes referencing corresponding tracks indicating media components of the media stream, and each of the one or more track group boxes is associated with a corresponding pair consisting of a track group identifier and a track group type that jointly identifies a corresponding track group within the media stream, wherein tracks having the same track group identifier and the same track group type belong to the same track group; and wherein each such track group determines a preselection corresponding to media presentation to the user. Examining the trackboxes in the media stream to determine a complete set of preselections present in the media stream, wherein determining the complete set of preselections includes: identifying a unique pair consisting of a track group identifier and a track group type; and addressing the preselections by the corresponding track group identifier; Select a preselection from the complete set of preselections, wherein the preselection is based on the attributes of the corresponding preselections included in the track group box with the same track group identifier; Determine contributions to a selected preselected group of one or more track boxes, wherein the contributions to the preselected group of one or more track boxes are identified by the presence of track box groups with the same track group identifier; The track referenced in each member of the set of one or more track boxes is determined as contributing to the preselected one or more tracks; and One or more tracks are provided for downstream processing according to the preselection. Each preselection is associated with a corresponding preselection-related box of a predefined type, and wherein the preselection-related box is instantiated as a track group box having a predefined track group type associated with the preselection; The preselection related box is associated with a preselection processing box, which includes processing information indicating how to process the orbits that contribute to the preselection. The processing information includes sorting information, which indicates the track order used to sort the tracks. The processing information includes merging information, which indicates whether the track should be merged with one or more other tracks; The method further includes: The tracks are merged according to the merging information and the sorting information; The sorting information includes a track order value used to define the track order; The merging information includes a merging flag, wherein a first setting value of the merging flag indicates that the track will be merged with adjacent tracks in the track order, and a second setting value of the merging flag indicates that the track will be processed individually; and Merging the one or more tracks according to the merging information and the sorting information includes: The orbits are scanned sequentially according to the orbital order; and Merge tracks according to the corresponding merge flags.

16. The method according to claim 15, wherein, The preselected related box is associated with a preselected information box that includes semantic information indicating the preselection.

17. The method according to claim 15 or 16, wherein, The processing information includes unique preselected specific data for configuring downstream devices to decode the track according to the preselection.

18. The method according to claim 15 or 16, further comprising: The track is decoded based on the media presentation indicated by the preselection for playback of the media stream.

19. The method according to claim 18, wherein, One or more tracks are decoded by downstream devices.

20. The method according to claim 18, wherein, The merging and decoding of the tracks are performed by a single device.

21. The method according to claim 15 or 16, wherein, The pre-selection is determined by the application.

22. The method according to claim 15 or 16, wherein, The media stream includes one or more tab boxes, each tab box being linked to a corresponding pre-selected information box, and wherein each tab box includes descriptive information about the corresponding media presentation for the user; and The determination of the pre-selection is based on the user's input.

23. The method according to claim 15 or 16, wherein, The media stream includes at least one of an audio stream or a video stream.

24. The method according to claim 15 or 16, wherein, The media presentation to the user is characterized by a corresponding configuration associated with the language, type, and / or one or more media-specific attributes of the media stream.

25. The method according to claim 15 or 16, wherein, The predefined transmission format is the ISO Basic Media File Format (ISOBMFF).

26. A method for processing a media stream, comprising: One or more basic streams are encapsulated according to a predefined transport format to generate packetized media streams. The media stream of the group comprises multiple layered boxes, each associated with a corresponding box type identifier; and Encapsulating the one or more basic streams includes: The media data of the one or more basic streams are grouped according to the transmission format to generate one or more trackboxes that reference the corresponding tracks of the one or more basic streams; One or more preselected relevant boxes of predefined types are generated based on the header information of the one or more basic streams, wherein each of the one or more preselected relevant boxes indicates a corresponding preselection corresponding to the media presentation to the user; and One or more pre-selected processing boxes are generated, each pre-selected processing box including processing information indicating how to process the processing that contributes to the corresponding pre-selected orbit. The processing information includes sorting information, which indicates the orbital order for processing the one or more orbits; The processing information includes merging information, which indicates whether one or more orbits should be merged with one or more other orbits for joint processing; Wherein, for each track contributing to the preselection, the sorting information includes a corresponding track order value for defining the track order; The merging information includes a corresponding merging flag for contributing to each of the preselected tracks, wherein a first setting value of the merging flag indicates that the corresponding track will be merged with adjacent tracks in the track order, and a second setting value of the merging flag indicates that the corresponding track will be processed separately.

27. The method according to claim 26, wherein, Each of the one or more preselected related boxes includes metadata information indicating the corresponding preselected characteristic; and The metadata information includes information indicating the contribution of the media stream to one or more of the corresponding pre-selected tracks.

28. The method of claim 27, wherein, The metadata information corresponding to the corresponding pre-selected related box further includes at least one of the following: Indicates preselection identification information used to identify the corresponding preselected identifier; or A unique preselection specificity used to decode the track based on the preselection.

29. The method according to claim 26, wherein, Encapsulating the one or more basic streams further includes generating one or more track group boxes, each of the one or more track group boxes being associated with a corresponding track group identifier and a corresponding track group type within a media stream that jointly identifies the group, wherein tracks having the same track group identifier and the same track group type belong to the same track group; and Generating the one or more pre-selected related boxes includes: Assign a first unique identifier to each preselected candidate; and For each track contributing to the corresponding preselection, a corresponding preselection-related box is generated and associated with the corresponding preselection, and the track group identifier is set as the first unique identifier, wherein the preselection-related box is instantiated as a track group box having a predefined track group type associated with the preselection.

30. The method according to claim 29, wherein, The track boxes are generated by grouping the tracks based on the corresponding media types that contribute to a pre-selected track.

31. The method according to claim 30, wherein, The media type includes at least one of the following: audio, video, and subtitles.

32. The method according to any one of claims 26 to 31, further comprising: Receive at least one input medium; as well as The input media is processed to generate the one or more elementary streams, wherein the one or more elementary streams include media data of the input media and corresponding header information.

33. The method according to any one of claims 26 to 31, wherein, The method further includes: A manifest file is generated based on one or more pre-selected relevant boxes.

34. The method according to claim 33, wherein, The manifest file is either an MPEG-based HTTP-based dynamic adaptive streaming DASH format file or an HTTP real-time streaming HLS format file.

35. A method for processing media streams, comprising: Receive the media stream packetized according to a predefined transmission format, wherein the packetized media stream includes a plurality of layered boxes, each of which is associated with a corresponding box type identifier, wherein the plurality of boxes includes one or more track boxes and one or more preselected associated boxes of a predefined type, the one or more track boxes referencing corresponding tracks indicating media components of the media stream, and wherein each preselected associated box indicates a corresponding preselection corresponding to media presentation to the user; and A manifest file is generated based on the one or more pre-selected relevant boxes. The media stream further includes processing information indicating how to process contributions to the preselected track. The processing information includes sorting information, which indicates the orbital order for processing the one or more orbits; The processing information includes merging information, which indicates whether one or more orbits should be merged with one or more other orbits for joint processing; Wherein, for each track contributing to the preselection, the sorting information includes a corresponding track order value for defining the track order; The merging information includes a corresponding merging flag for contributing to each of the preselected tracks, wherein a first setting value of the merging flag indicates that the corresponding track will be merged with adjacent tracks in the track order, and a second setting value of the merging flag indicates that the corresponding track will be processed separately.

36. The method according to claim 35, wherein, The manifest file is either an MPEG-based HTTP-based dynamic adaptive streaming DASH format file or an HTTP real-time streaming HLS format file.

37. The method according to claim 35 or 36, wherein, The media presentation to the user is characterized by a corresponding configuration associated with the language, type, and / or one or more media-specific attributes of the media stream.

38. The method according to claim 35 or 36, wherein, The predefined transmission format is the ISO Basic Media File Format (ISOBMFF).

39. A media streaming processing apparatus, comprising a processor and a memory coupled to the processor, wherein, The processor is adapted to cause the media streaming processing device to perform the method according to any one of claims 1 to 38.

40. A computer program product comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 38.

41. A computer-readable storage medium storing a program, the program including instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 38.

Images