Method for bandwidth switching by CMAF and DASH clients using Addressable Resource Index tracks and Addressable Resource Index events

Abstract

A method, apparatus and readable storage medium for processing a media stream, the media stream may be according to the DASH standard or the CMAF standard, the method may include receiving media stream data comprising a plurality of media chunks including a first media chunk and a second media chunk and Addressable Resource Index (ARI) information associated with the first media chunk, determining track change information based on the ARI information, determining based on the track change information that a switch to a different media track in the second media chunk is required, and receiving the first media chunk and the second media chunk via respective media tracks, each of the first media chunk and the second media chunk being delivered to a streaming client device with a delivery delay of no more than one chunk.

Description

[Technical field]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. Provisional Application No. 63 / 388,577, filed July 12, 2022, which claims the benefit of priority to this provisional application, and U.S. Nonprovisional Application No. 18 / 342,230, filed June 27, 2023, which is incorporated by reference in its entirety into this disclosure.

[0002] The present disclosure relates generally to media streaming technologies, including Dynamic Adaptive Streaming over Hypertext transfer protocol (DASH) and Common Media Application Format (CMAF). In particular, the disclosed technology relates to methods and apparatus for switching bandwidth (i.e., media tracks) based on information provided in Addressable Resource Index (ARI) tracks and / or ARI events. [Background technology]

[0003] The Background of the Invention described herein is intended to provide a general overview of the background of the disclosure. To the extent described in the Background of the Invention section, the work of the currently named inventors and other aspects of the description that may not have attained the status of prior art at the time of filing of this application are not admitted, expressly or impliedly, as prior art to the present disclosure.

[0004] The Moving Picture Expert Group (MPEG) dynamic adaptive streaming over hypertext transfer protocol (DASH) provides a standard for streaming multimedia content over IP networks. In the DASH standard, a media presentation description (MPD) is used to provide information to DASH clients to adaptively stream media content by downloading media segments from a DASH server. The DASH standard enables streaming of multi-rate content. One aspect of the DASH standard includes the transport of MPD and in-band events, and the client processing model used to handle the events.

[0005] Common Media Application Format (CMAF) is a standard used to package and deliver various forms of Hypertext transfer protocol (HTTP) based media. It works with, for example, HTTP Live Streaming (HLS) and DASH protocols to simplify the delivery of media to playback devices by packaging data in a uniform transport container file. It also uses chunked encoding and chunked transfer encoding to reduce latency, which reduces storage requirements and therefore costs. Summary of the Invention [Means for solving the problem]

[0006] In an aspect of the present disclosure, a method and apparatus for media stream processing is provided, in particular a method and apparatus for switching bandwidth (i.e., media tracks) based on information provided in an ARI track and / or an ARI event. In some implementations, a method for processing a media stream is disclosed. The media stream may include at least two media tracks, and may be in accordance with the Dynamic Adaptive Streaming over HTTP (DASH) standard or the Common Media Application Format (CMAF). The method may be performed, for example, by a streaming client device, and may include receiving media stream data comprising a plurality of media chunks including a first media chunk and a second media chunk, and Addressable Resource Index (ARI) information associated with the first media chunk, determining track change information based on the ARI information, determining based on the track change information that switching to a different media track in the second media chunk is necessary, and receiving the first media chunk and the second media chunk via the respective media tracks. In some implementations, each of the first media chunk and the second media chunk is delivered to the streaming client device with a delivery delay of one chunk or less.

[0007] In some implementations, another method of processing a media stream is disclosed. A media stream having at least two media tracks and conforming to the Dynamic Adaptive Streaming over HTTP (DASH) standard or the Common Media Application Format (CMAF) is executed by a streaming client, the method comprising: receiving one of an Addressable Resource Index (ARI) sample obtained from an ARI track related to a first media slice in a first media track of the media stream, or an ARI event related to the media stream embedded in the first media slice in the first media track of the media stream, wherein the ARI event provides characteristic information of at least one of a first media slice in the first media track and a first parallel media slice in another media track of the media stream, the position of which is aligned with the first media slice, or a second media slice in the first media track that follows the first media slice, and a second parallel media slice in the other media track of the media stream, the time position of which is aligned with the second media slice, and determining based on the characteristic information that a switch to one of the other media tracks is necessary.

[0008] Aspects of the present disclosure also provide a media stream processing device or apparatus that includes circuitry configured to perform any of the above method implementations.

[0009] An aspect of the present disclosure also provides a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for media stream processing for decoding and / or encoding video.

[0010] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, which are described below. [Brief description of the drawings]

[0011] [Figure 1] 1 illustrates a system according to an embodiment of the present disclosure. [Diagram 2] 1 illustrates a Dynamic Adaptive Streaming over HTTP (DASH) system according to an embodiment of the present disclosure. [Diagram 3] 1 illustrates a DASH client architecture according to an embodiment of the present disclosure. [Figure 4] 1 illustrates an example of a DASH data model according to an embodiment of the present disclosure. [Diagram 5] 1 illustrates an example of a CMAF data model according to an embodiment of the present disclosure. [Figure 6] 1 illustrates an example of switching media tracks at segment / chunk level according to an embodiment of the present disclosure. [Figure 7] 1 illustrates exemplary extrapolation switching and interpolation switching based on ARI tracks / ARI samples carrying switching assistance information. [Figure 8] Shows a typical extrapolated switch based on an ARI event carrying switch auxiliary information. [Figure 9] 1 shows exemplary extrapolated and interpolated switching based on an ARI event carrying switching assistance information with lag. [Figure 10] 1 shows a flowchart of a method according to an embodiment of the present disclosure. [Figure 11] FIG. 1 shows a schematic diagram of a computer system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Dynamic Adaptive Streaming Over Hypertext Transfer Protocol (DASH) and Media Presentation Description (MPD) One well-known media streaming format includes Dynamic adaptive streaming over hypertext transfer protocol (DASH), defined in ISO / IEC 23009-1. DASH is an adaptive bitrate streaming technique that enables streaming of media content using hypertext transfer protocol (HTTP) infrastructures such as web servers, content delivery networks (CDNs), and various proxies and caches. DASH supports both on-demand and live streaming from DASH servers to DASH clients, allowing DASH clients to control streaming sessions, so that DASH servers do not have to deal with the increased load of adaptive streaming management in large-scale deployments. Using DASH also allows DASH clients to select streaming from various DASH servers, thus achieving further network load balancing and benefiting DASH clients. Using DASH, for example, it is possible to dynamically switch between different media tracks by changing the bitrate to adapt to network conditions.

[0013] In DASH, a media presentation description (MPD) file provides DASH clients with information to adaptively stream media content by downloading media segments from a DASH server. The MPD may be in the form of an Extensible Markup Language (XML) document. The MPD file can be fragmented and delivered in parts to mitigate session start delays. The MPD file can also be updated during a streaming session. In some examples, the MPD file supports the representation of content accessibility features, ratings, and camera views. DASH also supports the delivery of content using multi-view scalable coding.

[0014] An MPD file may contain a sequence of one or more periods. Each of the one or more periods may be defined, for example, by a period element in the MPD file. An MPD file may contain an availableStartTime attribute of the MPD and a start attribute of each period. For media presentations of a dynamic type (e.g., a type used for live services), the sum of the start attribute of the period and the MPD attribute availableStartTime, together with the duration of the media segments, may indicate the validity time of the period in Coordinated Universal Time (UTC) format, and in particular the first media segment of each representation of the corresponding period. For media presentations of a static type (e.g., a type used for on-demand services), the start attribute of the first period may be 0. For any other period, the start attribute may specify an offset time of the start time of the corresponding period relative to the start time of the first period. Each period may extend to the beginning of the next period or, in the case of the last period, to the end of the media presentation. Period start times can be highly accurate and can represent the actual timing that results from playing all previous periods' media. In implementations, an MPD is provided such that the next period is contiguous with the content of the previous period, or possibly with the content of the immediately following period, or with the content of a later period (e.g., the period after an advertising period has been inserted).

[0015] Each period may contain one or more adaptation sets, each of which may contain one or more representations of the same media content. A representation may be one of several alternative encodings of the audio or video data. Representations may differ by encoding type, for example by bitrate, resolution and / or codec used for the video data and bitrate and / or codec used for the audio data. The term representation may be used to refer to a section of encoded audio or video data that corresponds to a particular period of multimedia content and is encoded in a particular way.

[0016] Adaptation sets for a particular period can be assigned to a group, indicated by the group attribute in the MPD file. Adaptation sets in the same group are generally considered as alternatives to each other. For example, each adaptation set for video data for a particular period can be assigned to the same group, such that any adaptation set can be selected for decoding to display video data of the multimedia content of the corresponding period. The media content in a period can be represented by one adaptation set from group 0 (if group 0 exists). If group 0 does not exist, the media content in a period can be represented by a combination of at most one adaptation set from each non-zero group (several examples exist). Timing data for each representation of a period can be expressed relative to the start time of the period.

[0017] A representation may contain one or more segments. Each representation may contain an initialization segment, or each segment of a representation may be self-initializing. If present, the initialization segment may contain initialization information for accessing the representation. In some cases, the initialization segment does not contain media data. A segment may be unambiguously referenced by an identifier, such as a uniform resource locator (URL), uniform resource name (URN), or uniform resource identifier (URI).

[0018] In the implementation, URLs are written according to IETF RFC 3986. <absolute-uri>as a fixed convention, e.g. "http" or "https", possibly constrained by a byte range (if a range attribute is provided with the URL). A byte range may be expressed as a byte-range-spec, e.g. as specified in IETF RFC 2616. This may be constrained to a single expression that identifies a contiguous range of bytes. In one embodiment, the segment may be included in the MPD along with the data URL, e.g. as specified in IETF RFC 2397.

[0019] MPD files may provide identifiers for each segment, and in some cases may also provide byte ranges in the form of range attributes, which may correspond to a segment's data within a file that is accessible by a URL, URN, or URI.

[0020] A sub-representation may be embedded in a canonical representation (i.e., the canonical representation contains a sub-representation) and described by a sub-representation element (e.g., SubRepresentation). Properties of one or several media content components embedded in the representation may be described by a sub-representation element. For example, properties of an embedded audio component (e.g., codec, sampling rate, etc.), properties of embedded subtitles (e.g., codec) may be described by a sub-representation element, and several embedded lower-quality video layers (e.g., several low frame rates, etc.) may be described by a sub-representation element. Sub-representation and representation elements may share some common attributes and elements.

[0021] Each representation may also contain one or more media components, where each media component may correspond to an encoded instance of one of the particular media types such as audio, video or timed text (e.g. timed text used for closed captioning). Media components may be contiguous in time across boundaries of consecutive media segments within a representation.

[0022] In some implementations, a DASH client can access and download an MPD file from a DASH server, i.e., the DASH client can obtain an MPD file for use when starting a live session. Based on the MPD file, the DASH client can make several decisions for each selected representation, including determining what the latest segment is available on the server, determining the segment valid start time of the next segment and possibly also of future segments, determining when to start playing a segment and from which timeline within the segment, and when to acquire / fetch a new MPD file. After the service is played, the client can keep track of any drift between the live service and its own playback, which needs to be detected and compensated for.

[0023] Common Media Application Format (CMAF) The Segmented Media Common Media Application Format (CMAF) is an extensible standard used to encode and package segmented media objects for delivery and decoding to end-user devices in adaptive multimedia presentations. The CMAF specification defines a number of logical media objects, which are described below.

[0024] A CMAF track may contain encoded media samples, including audio, video, and subtitles. The media samples are stored in a CMAF-specified container that derives from the ISO Base Media File Format (ISO_BMFF). The media samples may be optionally protected by MPEG Common Encryption. A track may contain a CMAF Header and one or more CMAF Fragments.

[0025] A CMAF switching set may contain alternative tracks that can be switched between and spliced ​​together at CMAF fragment boundaries in order to adaptively stream the same content at different bit rates and resolutions. An Aligned CMAF Switching Set is two or more CMAF Switching Sets that are encoded from the same source with different encoding (e.g., different codecs) and are temporally aligned.

[0026] A CMAF selection set is a group of switching sets of the same media type that can contain alternative content (e.g., different languages) or alternative encoding schemes (e.g., different codecs).

[0027] A CMAF presentation may contain one or more presentation time synchronization selection sets.

[0028] To enable media content to be distributed across different platforms, CMAF supports Addressable Objects. A CMAF Addressable Object may contain the following: · CMAF Header: The Header contains information for initializing the track. · CMAF Segment: A series of one or more contiguous fragments from the same track. · CMAF Chunk: A chunk contains a contiguous subset of samples from a fragment. · CMAF Track File: A complete track in one ISO_BMFF file.

[0029] DASH and CMAF events In DASH and CMAF, events provide a means to signal additional information to a DASH / CMAF client and one or more associated applications. In an implementation, events are timed, so they have a start time and a duration. Event information may include metadata describing the content of the media presentation. Additionally or alternatively, event information may include control messages for the media player that are related to a particular time during playback of the media presentation, such as ad insertion cues. Events may be implemented, for example, as MPD events or in-band events. They can be part of a manifest file (e.g., MPD) or embedded in an ISOBMFF-based media file, such as in an event message (emsg) box.

[0030] A media presentation description (MPD) event is an event that can be signaled in the MPD. A set of events that are assigned to media presentation times can be attached to the MPD at the period level. Homogeneous events can be specified by event stream elements (e.g. EventStream) in a period element. An event ends at the end of the period even if its start time is after the period boundary or its duration extends beyond the period boundary. An event stream element contains a message scheme identification (e.g. @schemeIdUri) and an optional optional value (e.g. @value) for the event stream element. Additionally, if the event stream contains timed events, a time scale attribute (e.g. @timescale) may be provided to assign the event to a particular media presentation time within the period. The timed events themselves can be described by event elements contained in the event stream element.

[0031] Inband event streams can be multiplexed with a representation by including an event message as part of a media segment. An event stream may be present in selected representations, only in one or a few selected adaptation sets, or in all representations. For example, one possible configuration is where only the audio adaptation set contains inband events, or only the video adaptation set contains inband events. Inband event streams present in a representation can be indicated by inband event stream elements (e.g., InbandEventStream) at various levels, such as at the adaptation set level or at the representation level. Furthermore, a representation can contain multiple inband event streams, each indicated by a separate inband event stream element.

[0032] 1 shows a system (100) according to an embodiment of the present disclosure. The system (100) includes a content server (110) and an information processing device (120). The content server (110) can provide a content stream including a primary content (e.g., a main program) and one or more time-scheduled metadata tracks.

[0033] The information processing device (120) can be connected to the content server (110) via an interface. For example, the information processing device (120) can play content received from the content server (110). The content can be played based on a manifest file (e.g., MPD) received by the information processing device (120) (e.g., from the content server (110)). The manifest file can further include signaling of one or more timed metadata tracks.

[0034] A typical DASH / CMAF system is shown in Figure 2. The DASH system (200) may include a content server (210), an advertisement server (220), and an information processing device (230), which are connected to a network (250). The DASH system (200) may also include one or more supplemental content servers.

[0035] The content server (210) can provide the primary content (e.g., a main program) and a manifest file (e.g., an MPD) to the information processing device (230). For example, the manifest file can be generated by an MPD generator (214). In other embodiments, the primary content and the manifest file can be provided by different servers.

[0036] The information processing device (230) can receive the MPD and obtain the primary content from the HTTP server (212) of the content server (210) based on the MPD. The MPD can be processed by a DASH client (232) running on the information processing device (230). Furthermore, the DASH client (232) can obtain advertisement content from the advertisement server (220) and other content (e.g., interactive content) from one or more supplemental content servers. The primary content and advertisement content can be processed by the DASH client (232) and output for display on a display device (236). The display device (236) can be integrated into the information processing device (230) or can be external to the information processing device (230). Furthermore, the DASH client (232) can extract event information from one or more timed metadata tracks and send the extracted event information to an application (234) for further processing. For example, the application (234) can be configured to display supplemental content based on the event information.

[0037] The advertisement server (220) can store advertisement content in an advertisement storage (e.g., a memory). The stored advertisement content can be requested by the information processing device (230) based on the event information.

[0038] 3 illustrates an example DASH / CMAF client architecture for processing DASH and CMAF events according to an embodiment of the disclosure. The DASH / CMAF client (i.e., a DASH / CMAF player) can be configured to communicate with an application (390) and process various types of events, including (i) MPD events, (ii) inband events, and (iii) timed metadata events.

[0039] The manifest parser (305) parses the manifest (e.g., MPD). The manifest is provided, for example, by the content server (110, 210). The manifest parser (305) extracts event information for MPD events, inband events, and timed metadata events embedded in timed metadata tracks. The extracted event information can be provided to the DASH logic (310) (e.g., DASH player control logic, selection logic, and heuristic logic). Based on the event information, the DASH logic (310) can inform the application (390) of the event scheme signaled in the manifest.

[0040] The event information can include event scheme information for distinguishing between different event streams. The event scheme information can be used by an application (390) to periodically retrieve event schemes of interest. Additionally, the application (390) can signal a desired dispatch mode for each of the schemes to be periodically retrieved through one or more subscription APIs. For example, the application (390) can send a subscription request to a DASH client that identifies one or more event schemes of interest and any desired corresponding dispatch modes.

[0041] If an application (390) periodically obtains one or more event schemes delivered as part of one or more timed metadata tracks, the inband event 'moof' parser (325) can stream the one or more timed metadata tracks to the timed metadata track parser (330). For example, the inband event 'moof' parser (325) parses the movie fragment box ("moof") and then parses the timed metadata tracks based on control information obtained from the DASH logic (310).

[0042] The timed metadata track parser (330) can extract event messages embedded in the timed metadata tracks. The extracted event messages can be stored in an event timed metadata buffer (335). A synchronizer / dispatcher module (340) (e.g., an event and timed metadata synchronizer and dispatcher) can dispatch the periodically captured events to an application (390) (i.e., the synchronizer / dispatcher module (340) sends the periodically captured events to the application (390)).

[0043] The MPD events described in the MPD can be parsed by the manifest parser (305) and stored in a buffer (335). For example, the manifest parser (305) parses each event stream element of the MPD and parses each event described in each event stream element. For each event signaled in the MPD, event information such as presentation time and event duration can be associated with the event and stored in the buffer (335).

[0044] The media segments can be parsed to extract inband event messages by an inband event 'moof' parser (325). The identified inband events and their associated presentation times and durations can be stored in a buffer (335).

[0045] Thus, the buffer (335) can store MPD events, inband events, and / or timed metadata events. The buffer (335) can be, for example, a first-in-first-out (FIFO) buffer. The buffer (335) can be managed in association with a media buffer (350). For example, any events or timed metadata corresponding to a media segment can be stored in the buffer (335) for as long as that media segment is present in the media buffer (350).

[0046] A DASH Access Application Programming Interface (API) (315) can manage the fetching and receiving of content streams (i.e., data flows) including media content and various metadata through an HTTP protocol stack (320). The DASH Access API (315) can separate the received content streams into different data flows. The data flows provided to the inband event moof parser can include media segments, one or more timed metadata tracks, and inband event signaling included in the media segments. In one embodiment, the data flows provided to the manifest parser 305 can include an MPD.

[0047] The DASH Access API (315) can forward the manifest to the manifest parser (305). In addition to describing events, the manifest can also provide information about media segments to the DASH logic (310), which can communicate with the application (390) and the inband event moof parser (325). The application (390) can be associated with the media content processed by the DASH client. Control / sync signals exchanged between the application (390), the DASH logic (310), the manifest parser (305), and the DASH Access API (315) can be used to control the fetching of media segments from the HTTP stack (320) based on the information about the media segments provided in the manifest.

[0048] An inband event moof parser (325) parses the media data flow to obtain media segments containing the media content, timed metadata in the timed metadata tracks, and any inband events signaled in the media segments. The media segments containing the media content can be parsed by a file format parser (345) and stored in a media buffer (350).

[0049] The events stored in the buffer (335) allow the synchronizer / dispatcher (340) to communicate available events related to the application (i.e., events of interest) through an event / metadata API to the application. The application can be configured to process available events (e.g., MPD events, inband events, and timed metadata events) and periodically retrieve specific events or timed metadata by notifying the synchronizer / dispatcher (340). Any events stored in the buffer (335) that are not related to the application, but instead related to the DASH client itself, can be forwarded by the synchronizer / dispatcher (340) to the DASH logic (310) for further processing.

[0050] In response to an application (390) periodically retrieving a particular event, the synchronizer / dispatcher (340) can communicate application event instances (i.e., timed metadata samples) corresponding to the event schemes periodically retrieved by the application. The event instances can be communicated according to a dispatch mode indicated by the subscription request (e.g., a subscription request for a particular event scheme) or a default dispatch mode. For example, in a dispatch-on-receive mode, the event instances may be sent to the application (390) as they are received in the buffer (335), whereas in a dispatch-on-start mode, the event instances may be sent to the application (390) at their associated presentation time, e.g., synchronized with a timing signal from the media decoder (355).

[0051] DASH / CMAF media track switching DASH and CMAF support multiple switchable media tracks, which may be provided for the same media content, but with different bit rates to support different resolutions and different transmission bandwidth conditions. In a media streaming session, a client (e.g., a DASH client or a CMAF client) may choose to switch from one track to another, for example, to adapt to a particular bandwidth condition, the bandwidth resources allocated to the client, etc.

[0052] In some implementations, a media track may also be described as a media representation.

[0053] Figure 4 shows an example of a DASH data model. As shown in Figure 4, adaptation set 3 contains four representations, each representing a different track with a different bitrate. The bitrate of representation 2 is 2Mbps (megabits per second), and representation 2 is formed by one or more media segments. Typically, in the DASH data model shown in Figure 4, the smallest media slice unit is a "segment".

[0054] Figure 5 shows an example of a CMAF data model. As shown in Figure 5, switching set 3 includes four CMAF tracks, each representing a different bitrate. CMAF track 2 has a bitrate of 2 Mbps and is formed by one or more chunks. Typically, in the CMAF data model shown in Figure 5, the smallest media slice unit is a "chunk."

[0055] DASH / CMAF Addressable Resource Index In some implementations, it is desirable for an adaptive streaming client (e.g., a DASH client or a CMAF client) to have precise knowledge of Addressable Resource Index (ARI) information describing all the details of the addressable resources and the subset of the CMAF Switching Set, e.g., as defined in ISO / IEC 23000-19, in the metadata track. The ARI information may also describe the full detailed subset of the DASH adaptation set. The ARI information may include offsets, sizes, durations, and quality of timed aligned segments or chunks that are in the same adaptation set / switching set. Such ARI information together with, e.g., relative information about the nearest chunks or segments, may be used by the DASH / CMAF client to facilitate client heuristics. Addressable resources may include track files, segments, or chunks in a CMAF context. In case of an on-demand service, a precise map of such information may be provided by a segment index. Note that similar concepts and implementations may also be applied to DASH contexts.

[0056] In some implementations, the ARI information may be carried in ARI samples in an ARI track or in ARI events.

[0057] In some implementations, the Addressable Resource Index (ARI) may be defined as follows: Sample entry type: 'cari' Container: Sample Description Box(stsd) Required: Not required Quantity: 0 or 1

[0058] The metadata describes all the details of the addressable resources, for example a subset of the CMAF Switching Set defined in ISO / IEC 23000-19, in one index track.

[0059] Table 1 below shows a typical sample entry in CMAF Addressable Resource Index metadata.

[0060] [Table 1]

[0061] Table 2 below shows a typical syntax for an ARI sample.

[0062] [Table 2]

[0063] Typically, the semantics of the above syntax are as described below. switching_set_identifier specifies a unique identifier for the switching set in the application context. ·num_tracks indicates the track numbers assigned in the ARI track. · track_ID - Uses the track_ID to select and order among samples of a track. ·num_quality_indicators specifies the number of quality indicators used to identify the quality of the chunk. quality_identifier specifies an identifier that indicates how the quality values ​​in the sample are to be interpreted. This is a 4CC code that can be registered. · segment_start_flag indicates whether the chunk is the start of a segment. · marker specifies whether this chunk contains at least one styp box. · SAP_type specifies the SAP type of the chunk. emsg_flag indicates whether this chunk provides at least one emsg box. · prft_flag indicates whether this chunk contains at least one prft box. offset specifies the offset of the chunk from the beginning of the segment. · size gives the size of the chunk in octets. quality gives the quality of the chunk according to a given quality scheme identifier. The data type of the quality value (integer or floating point) is defined by the quality scheme. If the quality scheme identifier is the empty string, the quality is an unsigned integer, interpreted proportionally with increasing values ​​indicating increasing quality. ·loss indicates that media data of the chunk was lost. num_prediction_pairs gives how many pairs of expected predictions are provided. · prediction_min_windows provides the minbuffer time value that is equal to the MPD value. predicted_max_bitrate provides a bandwidth value identical to the MPD semantic that is held for the duration of the prediction_min_windows value.

[0064] Delivery of ARI using events In an implementation example in the case of DASH / CMAF, a dedicated metadata track, i.e., ARI track, is created to carry ARI-related information such as offset, size, quality, etc. of timed aligned segments or chunks that are in the same adaptation set / switching set, so that the client has relative information about the nearest chunks or segments to facilitate client heuristics, e.g., which can be used by the client when dynamically switching media tracks or representations.

[0065] Note that one disadvantage of using metadata tracks to carry ARI information (e.g. ARI samples) is the excessive signaling overhead: for example, an extra HTTP GET Request is required by the client for each segment that requires ARI information.

[0066] Embodiments of the present disclosure include a method of carrying ARI (i.e., ARI information, ARI sample) without an ARI metadata track. That is, rather than using a metadata track to carry the ARI (which would result in an extra HTTP GET request (as the ARI sample is sent separately with the media segment / chunk)), the present disclosure may send the ARI sample via an event, such as an inband event or an MPD event. This approach to carrying the ARI sample is considered to be a "media segment / chunk associated ARI transmission" since the ARI sample is sent with the media segment / chunk. An event carrying the ARI is referred to as an ARI event. Using an ARI event may provide at least the following advantages:

[0067] 1. No extra metadata track is needed, resulting in one less HTTP GET request by the CMAF / DASH client for each segment / chunk for which additional ARI information is needed. For example, a CMAF / DASH client may need additional ARI information to facilitate processing a segment / chunk. In this case, the ARI information can be obtained directly from the ARI event carried with the segment / chunk.

[0068] 2. The event processing model allows for the processing of event messages and dispatching them to DASH / CMAF clients. The processing model allows for the timing of ARI samples to be carried as part of the event timing model.

[0069] 3. Flexibility: ARI information may be carried by events in one, some, or all representations in a DASH adaptation set or a CMAF switching set, depending on the needs of the inband events, for example.

[0070] 4. Adaptability and portability: ARI events can be parsed by a packager (eg, derived from inband events or ARI tracks received from an encoder) and added to the MPD as MPD events.

[0071] In some implementations, the ARI information for a chunk / segment may be included in the same chunk / segment.

[0072] In some implementations, the ARI information of a chunk / segment may be included in subsequent chunks / segments arranged in time.

[0073] In some implementations, rather than using in-band events to carry the ARI information, MPD events may be used to carry the ARI information, which may be particularly suitable for on-demand content.

[0074] In some implementations, ARI information may be carried in emsg boxes, each of which may belong to an event scheme defined by or associated with a scheme URN identifier.

[0075] Table 3 below shows example parameters for an ARI event in an MPD.

[0076] [Table 3]

[0077] As shown in Table 3, two elements may be used to describe an ARI event: EventStream and InbandEventStream. Both streams may contain a value attribute. The value attribute may carry the CmafAriMetaDataSampleEntry fields described in Table 1. For example, a CmafAriMetaDataSampleEntry field may contain the following fields: switching_set_identifier num_tracks num_quality_indicators · A numbered list of track_ids List of quality_identifiers

[0078] In some implementations, an event element may include a presentationTime attribute (eg, Event@presentationTime), which indicates the chunk offset from the beginning of the period to which the event's ARI information applies.

[0079] In some implementations, the event element may include a duration attribute (e.g., Event@duration), which indicates the duration for which the ARI information should be used. For example, this may include the duration of a chunk or the duration of a segment.

[0080] In some implementations, an event may contain an event body, which may share the same structure as a CmafAriFormatStruct, as defined in Table 2.

[0081] Table 4 below shows an example of the emsg parameter for an inband ARI event.

[0082] [Table 4] Note that the event body in an MPD event and the message_data in an inband event share the same CMAF ARI sample structure, CmafAriFormatStruct. Therefore, the parsing and processing of the ARI sample after receiving the event from the event dispatcher is the same. In other words, the same parsing and processing logic is shared for MPD events and inband events.

[0083] In some embodiments, ARI events may be processed and dispatched, for example, according to Section A.13 of ISO / IEC 23009-1, for example, in an exemplary DASH / CMAF client architecture shown in FIG.

[0084] In some embodiments, after an ARI event is dispatched, the ARI event is post-processed, which may rely on the parameters shown in Table 5.

[0085] [Table 5] In this disclosure, various embodiments are described to enhance media track switching by using ARI events or ARI track samples. For example, an ARI event (or ARI track sample) may carry size, quality and offset information of some or all aligned chunks (parallel chunks) of some or all tracks in the same switching / adaptation set. The information carried in the ARI track sample or ARI event may be used by a CMAF / DASH client to switch to another track / representation at the relevant chunk boundary.

[0086] FIG. 6 shows an example of media track switching at segment / chunk level. As shown in FIG. 6, there are three tracks / representations, which represent, for example, different bit rates. The time positions of chunks in each track are aligned with the time positions of corresponding chunks in other tracks. For example, the time positions of chunks C1 in each track are aligned. Such time-aligned chunks in different tracks may be called parallel chunks. For example, all C1 chunks are parallel chunks and all C2 chunks are parallel chunks. In FIG. 6, a typical chunk-level switching is performed in the following manner:

[0087] After C1 610 it switches to C2 612, after C3 614 it switches to C4 616, after C4 616 it switches to C5 618.

[0088] Note that in Figure 6, track switching occurs at the level of the smallest media data unit supported by DASH or CMAF. For example, the unit may be a CMAF chunk or a DASH segment. Switching at other levels, such as the DASH representation level or the CMAF track level, may also be supported.

[0089] When switching tracks / representations, there are two important points in time with respect to the decision to switch. The first point is when the client makes the decision to switch, and the second point is when the switch occurs (e.g., from which chunk). For example, the decision to switch may be made at the beginning of a chunk (such as the C1 chunk in Figure 6), at the end of a chunk, or in the middle of a chunk. The decision may be, for example, to switch to track 2 starting from the C2 chunk (so the C2 chunk is the switch), or to switch to track 3 starting from the C1 chunk (so the C1 chunk is the switch), or to switch to track 2 starting from the C3 chunk (so the C3 chunk is the switch).

[0090] In some embodiments, a decision to switch made at chunk i is a decision to switch at chunk i+n, where i and n are non-negative integers.

[0091] In some embodiments, the switching auxiliary information may be carried in the ARI event or ARI track sample, and the DASH / CMAF client may use the latest available auxiliary information to make a decision about switching to a different track / representation.

[0092] In some embodiments, the auxiliary information for each track / representation may include: The offset and size of the current chunk (or the next chunk). The quality of the current chunk (or next chunk). Quality may include media resolution.

[0093] In some embodiments, the auxiliary information may be implicit and may be used by a DASH / CMAF client to derive the switch location.

[0094] In some embodiments, the auxiliary information may be explicit, for example, it may carry explicit notification of switch locations (e.g., chunks and their corresponding tracks), and the DASH / CMAF client can simply follow the auxiliary information to switch.

[0095] In this disclosure, a DASH / CMAF client may use several different approaches for switching tracks / representations when in a streaming session. In one example, the client may determine that a switch is needed at the next chunk / segment (e.g., the current chunk is C1 and switch at C2), or the client may determine that a switch is needed at the next n-th chunk / segment (e.g., the current chunk is C1 and switch at C4). In another example, the client may determine that an immediate switch is needed for the current chunk. An early switch decision may be beneficial in that the client may start requesting / receiving / buffering media data early. An immediate switch decision may be desirable, however, if there is a need to quickly adapt to the current bandwidth conditions. The client may use the appropriate approach as needed.

[0096] In some embodiments, if a client is streaming a representation with chunk transitions, the client is not only getting chunks / segments during streaming, but also getting ARI tracks or ARI events. For example, a DASH / CMAF client may receive ARI samples via a track separate from the media track, or a DASH / CMAF client may receive ARI events multiplexed with or embedded in the media chunks.

[0097] 7 to 9 show examples of timings when a DASH / CMAF client receives ARI information. The ARI information may carry only switching auxiliary information, or may carry other information together with the switching auxiliary information. For ease of explanation and illustration, these figures assume zero transition delay between the encoder and the client. Under this assumption, the packager output (i.e. chunks / segments, their ARI samples, and chunks / segments with events embedded therein) is available to the client as soon as it is prepared by the packager. Note that the same basic principles will apply even when transition delays are taken into account. In the following embodiments, a media unit "chunk" is used for illustration purposes. The embodiments also apply to other media units such as segments.

[0098] In Figures 7-9, vertical dotted lines at each time point, T1, T2, etc., indicate where information is available.

[0099] In some embodiments, the ARI information supporting bandwidth switching is carried in an ARI track, e.g., via ARI samples, as shown in Figure 7. Note that an ARI track is a different track than a media track that contains media chunks or media segments.

[0100] As shown in FIG. 7, ARI information (e.g., location, size, and quality) of chunk C1 carried by ARI sample 1 is available at T1 (received by the DASH / CMAF client). The ARI information of C1 (or ARI information related to C1) can then be used by the client to make a decision to switch at C2. That is, based on the C1 ARI information, the client can make a decision to switch before receiving the C2 chunk (i.e., the next chunk of C1) at T1. Since the ARI information is the ARI information for the received chunk C1 or the ARI information related to the received chunk C1, and the switching decision is a decision for the next chunk, this method is called extrapolated switching. That is, the switching decision is based on the ARI information of the past chunks for switching at the next chunk, so the switching decision is based on estimation / prediction. The ARI information is used to decide on switching at future chunks, and the characteristics of the future chunks are estimated or predicted. For example, the ARI information of C1 is used to estimate / predict the characteristics of chunk C2, and a switching decision is made based on the ARI information so that the client switches to another track for the C2 chunk.

[0101] In some implementations, once the decision to switch is made, the client may begin receiving and / or buffering the desired chunk, the C2 chunk.

[0102] In some implementations, ARI sample 1 of Figure 7 may carry switching assistance information (position, size and quality) for some or all of the parallel chunks (e.g., the C1 chunks of all tracks shown in Figure 6). For example, the ARI information may carry switching assistance information for all parallel C1 chunks.

[0103] In addition to the above-mentioned extrapolation switching solution, the embodiment of the present disclosure further provides an interpolation switching solution when an ARI track is used. As shown in FIG. 7, the ARI information carried by ARI sample 2 for the C2 chunk becomes available to the client at T2. That is, the client needs to wait until T2 to obtain the ARI information of C2. Based on the C2 ARI information, the client may then make a decision to switch during C2 (i.e., the switch point is the C2 chunk, i.e., switch to a parallel C2 chunk of another track). In this case, the ARI information of the C2 chunk itself is used for the client to make a decision on whether to switch at the C2 chunk, and the ARI information is accurate. In this case, the switching decision is based on accurate information, rather than the extrapolation solution, where the switching decision is based on an estimate from one or more previous chunks.

[0104] Typically, the ARI information carried by ARI sample 2 may be for one or more parallel C2 chunks, or may be for all parallel C2 chunks.

[0105] In comparison, in both the extrapolation switching method and the interpolation switching method described above, the choice is made to switch in the C2 chunk. In the extrapolation switching, the decision to switch is made in T1 based on estimation / prediction from the ARI information in the C1 chunk. The estimation / prediction may be made with further reference to at least one of the following: Current network conditions, such as the bandwidth available to the DASH / CMAF client, The current playback requirements, such as media quality, media size and media offset. Using inference / prediction from the ARI information of the C1 chunk, the DASH / CMAF client can make a decision as to whether a switch is necessary and, if so, which C2 chunk to select when switching.

[0106] On the other hand, in interpolative switching, the switching decision is made at T2 based on the ARI information in the C2 chunk, which is accurate to use for the switching decision. Note that the switching decision may also be made based on the ARI information with reference to the current network conditions and / or the current playback requirements.

[0107] Thus, extrapolated switching can achieve an earlier switch decision (i.e., decision at T1 before T2), but the decision is based on estimation / prediction. Interpolated switching can achieve a more accurate switch decision, but the decision is later, for example by one chunk (i.e., decision at T2 one chunk later than T1). Once the switch decision is made, the client may start requesting / receiving / buffering the selected chunk.

[0108] In some implementations, according to an interpolation switching method, a client may first receive an ARI sample (eg, an ARI sample of chunk C2) and then receive the corresponding chunk with a delay of one chunk.

[0109] In some embodiments, bandwidth switching / track switching is performed based on ARI events (no lag). As shown in Figure 8, ARI information supporting bandwidth switching (i.e., switching auxiliary information) is carried in one or more ARI events that are multiplexed with or embedded in one or more chunks.

[0110] As shown in FIG. 8, the switching auxiliary information (e.g., position, size and quality) for chunk C1 carried by ARI event 1 (indicated at 810) is available to the client (received by the client) at T1. Note that since the event is "inband" with chunk C1, both event 1 and chunk C1 are received by the DASH / CMAF client at T1. In this case, the extrapolation switching method described above may be used. That is, based on the ARI information of chunk C1, the client may make a switching decision using prediction / estimation with further reference to the current network conditions and / or current playback requirements. If switching is desired at C2, the client may further select a specific C2 chunk among the parallel C2 chunks.

[0111] Note that the interpolation switching method is not suitable in this scenario: in the case of interpolation switching, an ARI event in the current chunk is supposed to be used to indicate (directly or indirectly) a switch in the current chunk, but since the chunk has already been received, it is too late to perform the switch. For example, when receiving ARI event 1, chunk C1 is also received, so the earliest time to perform the switch is T2.

[0112] In some embodiments, bandwidth switching / track switching is performed based on ARI events with a lag. As shown in Fig. 9, ARI information supporting bandwidth switching (i.e., switching auxiliary information) is carried in one or more ARI events with a lag of one chunk. In this case, chunk i has an ARI event for chunk i+1, where i is an integer. An example is described below with reference to Fig. 9.

[0113] In FIG. 9, ARI event 2 (indicated at 910) carries the switching auxiliary information for the C2 chunk, and the ARI event 2 can be compiled only after the C2 chunk is ready from the encoder side. Therefore, on the packager side, there is a delay of one chunk. That is, instead of packaging chunk C1 in parallel (i.e., synchronously) with the encoder output, the packager must wait one chunk duration until the C2 chunk is generated by the encoder and the ARI event 2 is compiled based on the C2 chunk, after which the ARI event 2 is multiplexed with the C1 chunk. Note that in this case, the current chunk, such as C1, carries the ARI event of the subsequent chunk (e.g., chunk i carries the ARI event of chunk i+n, where i and n are integers).

[0114] Figure 9 illustrates the delay pattern described above. As shown in Figure 9, for the same chunk, the packager's timeline has shifted by one chunk to the right.

[0115] In FIG. 9, in addition to the C1 chunk, switching auxiliary information in the C2 chunk can be used at T2.

[0116] In some implementations, an extrapolated switching method may be used to make a switching decision. For example, at T2, based on the switching assistance information in the C2 chunk carried in the C1 chunk, the client may use prediction / estimation to make a switching decision on whether to switch at the C3 chunk or not, further referring to the current network conditions and / or current playback requirements. If switching is desired, the client may further select a specific C3 chunk among the parallel C3 chunks. When using the extrapolated switching method, the switching decision is based on past chunks to switch at the next chunk (i.e., a future chunk), e.g., based on the C1 chunk to switch at the C3 chunk.

[0117] In some implementations, an interpolation switching method may be used to make the switching decision. For example, at T2, a switching decision on whether to switch at the C2 chunk may be made based on the switching auxiliary information of the C2 chunk carried in the C1 chunk, with further reference to the current network conditions and / or current playback requirements. If switching is desired, the client may further select a specific C2 chunk among the parallel C2 chunks. When an interpolation switching method is used, the switching decision is based on the switching auxiliary information that is accurate for the chunk to switch to. For example, if the switching auxiliary information carried in the C1 chunk is the information of the C2 chunk, then the switching auxiliary information is accurate for making the switching decision on switching at the C2 chunk. Note that if the switching decision is made at T2, then the C2 chunk has not yet been fetched.

[0118] 10 illustrates an exemplary method 1000 for processing a media stream. The media stream may include, for example, a 4G media stream (for a media stream delivered in a 4G network) or a 5G media stream (for a media stream delivered in a 5G network). The method may be implemented, for example, by a computer system (described below), a client device (which may be part of or integrated in an encoder and / or decoder). The media stream may comply with the DASH standard or the CMAF standard. The method 1000 may include some or all of the following steps: receiving 1010 media stream data comprising a plurality of media chunks including a first media chunk and a second media chunk and Addressable Resource Index (ARI) information associated with the first media chunk; determining 1020 track change information based on the ARI information; determining 1030 that a switch to a different media track in the second media chunk is required based on the track change information; and receiving 1040 the first media chunk and the second media chunk via their respective media tracks.

[0119] In some implementations, in step 1040, when the first media chunk and the second media chunk are received via their respective media tracks, each of the first media chunk and the second media chunk is delivered to the streaming client device with a delivery delay of one chunk or less.

[0120] In some implementations, the ARI information may include or be conveyed via one of one or more ARI samples or one or more ARI events. Method 1000 may further include the ARI information comprising at least one of an ARI sample obtained from an ARI track associated with a first media slice in a first media track of the media stream, or an ARI event associated with a media stream embedded in a first media slice in the first media track of the media stream.

[0121] In some implementations, the method 1000 may further include receiving one of an Addressable Resource Index (ARI) sample obtained from an ARI track associated with a first media slice in a first media track of the media stream or an ARI event associated with the media stream embedded in the first media slice in the first media track of the media stream, wherein the ARI event provides characteristic information of at least one of the first media slice in the first media track and a first parallel media slice in another media track of the media stream that is aligned in position to the first media slice, or a second media slice in the first media track that follows the first media slice and a second parallel media slice in the other media track of the media stream that is aligned in time to the second media slice.

[0122] The embodiments of the present disclosure not only apply to both DASH and CMAF, but also to other media streaming technologies by applying similar basic principles.

[0123] The embodiments of the present disclosure may be used individually or combined in any order. The methods of the present disclosure, such as the method 1000 above, may include all or only some of the recited steps. Furthermore, each of the methods (i.e., embodiments), the DASH client, and the CMAF client may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to the DASH and / or CMAF technologies / standards. Typically, each of the methods (or embodiments) may be executed by a DASH / CMAF client, which may be executed on a computing device having a processing circuitry. For example, the client may be executed on an encoder and / or a decoder.

[0124] The techniques described above may be implemented using computer readable instructions and as computer software physically stored on one or more computer readable media. For example, Figure 11 illustrates a computer system (1800) suitable for implementing certain embodiments of the disclosed subject matter.

[0125] Computer software may be encoded using any suitable machine or computer language, which may be processed by assembly, compilation, linking, or similar mechanisms to generate code comprising instructions which may be executed directly, interpreted, executed by microcode, or the like, by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0126] The instructions may be executed in various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0127] 11 of the computer system (1800) are exemplary of course and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. Furthermore, the configuration of components should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of the computer system (1800).

[0128] The computer system (1800) may include several human interface input devices that may be responsive to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture media that is not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images captured from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0129] The input human interface devices may include one or more (only one of each is shown) of a keyboard (1801), a mouse (1802), a trackpad (1803), a touch screen (1810), a data glove (not shown), a joystick (1805), a microphone (1806), a scanner (1807), and a camera (1808).

[0130] The computer system (1800) may also include a number of human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1810), data gloves (not shown), or joystick (1805), although there may also be haptic feedback devices that are not used as input devices), audio output devices (such as speakers (1809), headphones (not shown)), visual output devices (such as screens (1810), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability and with or without haptic feedback capability. Some of the above screens may be capable of outputting two-dimensional visual output, or three or more dimensional output through means such as stereoscopic graphic output. Virtual reality glasses (not shown), holographic displays, smoke generator tanks (not shown), and printers (not shown).

[0131] The computer system (1800) may also include storage devices and associated media that may be directly operated by a human, such as optical media (1821), including a CD / DVD ROM / RW (1820) using CD / DVD or similar media, a thumb-drive (1822), a removable hard drive or solid-state drive (1823), legacy magnetic media such as tape or floppy disks (not shown), and specialized ROM / ASIC / PLD based devices (not shown) such as security dongles.

[0132] Those skilled in the art will also appreciate that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0133] The computer system (1800) may also include an interface (1854) to one or more communications networks (1855). The networks may be, for example, wireless, wired, optical, etc. Furthermore, the networks may be local, wide area, metropolitan, vehicular, industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable, satellite, and terrestrial television, vehicular and industrial networks including CAN bus, etc. Some networks require external network interface adapters that are typically attached to some general purpose data port or peripheral bus (1849) (e.g., a USB port of the computer system (1800)), while other networks are typically built into the core of the computer system (1800) by attaching to the system bus as described below (e.g., an Ethernet interface for a PC computer system or a cellular network interface for a smartphone computer system). Using any of these networks, the computer system (1800) can communicate with its counterpart. Such communication can be one-way, receive-only (e.g., television broadcast), one-way transmit-only (e.g., a CANbus to a specific CANbus device), or two-way, e.g., two-way communication to another computer system using a local or wide area digital network. Several protocols and protocol stacks can be used for each of these networks and network interfaces described above.

[0134] The above-mentioned human interface devices, memory devices that can be directly operated by humans, and network interfaces can be attached to the core (1840) of the computer system (1800).

[0135] The cores (1840) may include one or more central processing units (CPUs) (1841), graphics processing units (GPUs) (1842), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (1843), hardware accelerators for specific tasks (1844), graphics adapters (1850), and the like. Such devices may be connected through a system bus (1848) along with read only memory (ROM) (1845), random access memory (1846), and internal mass storage (1847) such as hard drives, SSDs, etc. that are internal and not directly accessible to the user. In some computer systems, the system bus (1848) may be directly accessible in the form of one or more physical plugs that allow expansion with additional CPUs, GPUs, etc. Peripherals may be attached directly to the core's system bus (1848) or through a peripheral bus (1849). In one example, a screen (1810) may be connected to a graphics adapter (1850). Peripheral bus architectures include PCI, USB, etc.

[0136] The CPU (1841), GPU (1842), FPGA (1843) and accelerator (1844) may execute a number of instructions that may be combined to make up the computer code described above. The computer code may be stored in a ROM (1845) or a RAM (1846). Transient data may also be stored in the RAM (1846), while immutable data may be stored, for example, in an internal mass storage (1847). A cache memory, which may be closely associated with one or more of the CPU (1841), GPU (1842), mass storage (1847), ROM (1845), RAM (1846), etc., may be used to enable fast storage and retrieval from any of the memory devices.

[0137] The computer-readable medium may bear computer code for performing various computer-implemented operations, and the medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those having skill in the computer software arts.

[0138] As a non-limiting example, a computer system having the architecture (1800), and in particular the core (1840), can provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with mass storage that can be directly manipulated by a user, as described above, or even specific storage of the core (1840), such as the core's internal mass storage (1847) or non-transitory storage, such as ROM (1845). Software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (1840). The computer-readable media can include one or more memory devices or chips, depending on particular requirements. The software enables the cores (1840), and in particular the processors (including CPUs, GPUs, FPGAs, etc.) in the cores (1840), to execute certain processes or certain parts of certain processes described herein, including the definition of data structures stored in RAM (1846) and the modification of such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic being hardwired or otherwise implemented in circuitry (e.g., accelerators (1844)) that may act in place of or cooperate with the software to execute certain processes or certain parts of certain processes described herein. Where appropriate, references to "software" may include logic, and vice versa. Where appropriate, references to "computer-readable medium" may include circuitry (such as integrated circuits (ICs)) that store the software to be executed, circuitry that implements the logic to be executed, or both. Any suitable combination of hardware and software is included in the present disclosure.

[0139] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents, which are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods that embody the principles of this disclosure and are therefore within its spirit and scope, even if not explicitly shown or described in this application. [Explanation of symbols]

[0140] 100 system, 110 content server, 120 information processing device, 200 DASH system, 210 content server, 212 HTTP server, 214 MPD generator, 220 advertisement server, 230 information processing device, 232 DASH client, 234 application, 236 display device, 250 network, 305 manifest parser, 310 DASH logic, 315 DASH access API, 320 HTTP protocol stack, 325 inband event 'moof' parser, 330 timed metadata track parser, 335 event timed metadata buffer, 340 dispatcher module, 345 file format parser, 350 media buffer, 355 media decoder, 390 application, 610 C1 chunk, 612 C2 chunk, 614 C3 chunk, 616 C4 chunk, 618 C5 chunk, 810 switching auxiliary information, 910 ARI event 2, 1000 Method, 1800 Computer system, 1801 Keyboard, 1802 Mouse, 1803 Trackpad, 1805 Joystick, 1806 Microphone, 1807 Scanner, 1808 Camera, 1809 Audio output device speaker, 1810 Touch screen, 1820 CD / DVD ROM / RW, 1821 Optical medium, 1822 Thumb-drive, 1823 Solid state drive, 1840 Core, 1841 CPU, 1842 GPU, 1843 FPGA, 1844 Hardware accelerator, 1845 ROM, 1846 RAM, 1847 Mass storage, 1848 System bus, 1849 Peripheral bus, 1850 Graphics adapter, 1854 Interface, 1855 Communication network

Claims

1. 1. A method for processing a media stream, executed by a streaming client device, the media stream comprising at least two media tracks and conforming to the Dynamic Adaptive Streaming over HTTP (DASH) standard or the Common Media Application Format (CMAF), the method comprising: receiving media stream data comprising a plurality of media chunks including a first media chunk and a second media chunk, and Addressable Resource Index (ARI) information associated with the first media chunk; determining track switch information based on the ARI information; determining based on the track change information that a switch to a different media track in the second media chunk is necessary; receiving the first media chunk and the second media chunk via respective media tracks, each of the first media chunk and the second media chunk being delivered to the streaming client device with a delivery delay of no more than one chunk.

2. The ARI information is an ARI sample from an ARI track associated with a first media slice in a first media track of the media stream; or 2. The method of claim 1, comprising at least one ARI event associated with the media stream embedded in the first media slice in the first media track of the media stream.

3. the media stream conforms to the DASH standard and the first media slice is a segment in the media stream; or the media stream conforms to a CMAF format, and the first media slice is a chunk in the media stream; The method of claim 2.

4. the first media slice in the first media track and a first parallel media slice in another media track of the media stream, the first media slice being aligned in position to the first media slice; or a second media slice in the first media track that follows the first media slice, and a second parallel media slice in the other media track of the media stream that is aligned in time with the second media slice. The method according to claim 2 or 3, wherein the ARI information provides at least one of the characteristic information.

5. The characteristic information is the offset of the first media slice of the first media track; a size of the first media slice of the first media track; an offset for each of the first parallel media slices; a size of each of the first concurrent media slices; the quality of the first media slice of the first media track, or The method of claim 4 , comprising at least one of the respective qualities of the first parallel media slice.

6. determining that the switch to the different media track in the second media chunk is necessary based on the track switch information, 5. The method of claim 4, further comprising determining that the switching to the one of the other media tracks is necessary based on the characteristic information and a bandwidth available to the streaming client device.

7. receiving the ARI event, the ARI event providing characteristic information about the first media slice in the first media track and the first parallel media slice in another media track of the media stream that is aligned in time with the first media slice; The step of determining whether the switching to the one of the other media tracks is necessary comprises: selecting one of the second concurrent media slices based on an estimation using the characteristic information; and switching to the one of the second concurrent media slices after the first media slice. The method according to claim 4.

8. 8. The method of claim 7, wherein the media stream is encoded and transmitted in real time, and the ARI event is constructed without consideration of the second media slice and the second parallel media slice.

9. receiving the ARI event, the ARI event providing characteristic information about the second media slice in the first media track that follows the first media slice and a second parallel media slice in the other media track of the media stream that is aligned in time with the second media slice; The step of determining whether the switching to the one of the other media tracks is necessary comprises: selecting one of the second concurrent media slices based on the characteristic information; and switching to the one of the second concurrent media slices after the first media slice. The method according to claim 4.

10. receiving the ARI event, the ARI event providing characteristic information about the second media slice in the first media track that follows the first media slice and a second parallel media slice in the other media track of the media stream that is aligned in time with the second media slice; The step of determining whether the switching to the one of the other media tracks is necessary comprises: selecting a target media slice next to one of the second parallel media slices based on an estimation using the characteristic information; and switching to the target media slice after the second media slice. The method according to claim 4.

11. The method of claim 10 , wherein the media stream is encoded and transmitted in real time, and the ARI event is constructed taking into account the second media slice and the second parallel media slice.

12. receiving an ARI sample, the ARI sample providing characteristic information about the first media slice in the first media track and the first parallel media slice in another media track of the media stream, the first parallel media slice being aligned in time with the first media slice; The step of determining whether the switching to the one of the other media tracks is necessary comprises: selecting one of the second concurrent media slices based on an estimation using the characteristic information; and switching to the one of the second concurrent media slices after the first media slice. The method according to claim 4.

13. receiving the ARI sample, the ARI sample providing characteristic information about the first media slice in the first media track and the first parallel media slice in another media track of the media stream, the first parallel media slice being aligned in time with the first media slice; The step of determining whether the switching to the one of the other media tracks is necessary comprises: selecting one of the first concurrent media slices based on the characteristic information; and switching to the one of the first concurrent media slices. The method according to claim 4.

14. 1. A device comprising a memory for storing computer instructions and a processor in communication with the memory, the processor executing the computer instructions causing the device to: receiving media stream data for a media stream, the media stream comprising at least two media tracks, the media stream conforming to the Dynamic Adaptive Streaming over HTTP (DASH) standard or the Common Media Application Format (CMAF), the media stream data comprising a plurality of media chunks including a first media chunk and a second media chunk, and Addressable Resource Index (ARI) information associated with the first media chunk; determining track switch information based on the ARI information; determining based on the track change information that a switch to a different media track in the second media chunk is necessary; receiving the first media chunk and the second media chunk via respective media tracks, each of the first media chunk and the second media chunk being delivered to the device with a delivery delay of one chunk or less; A device configured to cause

15. The ARI information is an ARI sample from an ARI track associated with a first media slice in a first media track of the media stream; or 15. The device of claim 14, comprising at least one ARI event associated with the media stream embedded in the first media slice in the first media track of the media stream.

16. the media stream conforms to the DASH standard and the first media slice is a segment in the media stream; or the media stream conforms to a CMAF format, and the first media slice is a chunk in the media stream; 16. The device of claim 15.

17. the first media slice in the first media track and a first parallel media slice in another media track of the media stream, the first media slice being aligned in position to the first media slice; or a second media slice in the first media track that follows the first media slice, and a second parallel media slice in the other media track of the media stream that is aligned in time with the second media slice. The device according to claim 15 or 16, wherein the ARI information provides at least one characteristic information of:

18. The characteristic information is the offset of the first media slice of the first media track; a size of the first media slice of the first media track; an offset for each of the first parallel media slices; a size of each of the first concurrent media slices; the quality of the first media slice of the first media track, or The device of claim 17 , comprising at least one of the respective qualities of the first concurrent media slice.

19. When the processor is configured to cause the device to determine that the switch to the different media track in the second media chunk is necessary based on the track switch information, the processor may cause the device to:

20. The device of claim 17, configured to determine that the switching to the one of the other media tracks is necessary based on the characteristic information and a bandwidth available to the device.

20. A non-transitory storage medium storing computer readable instructions, the computer readable instructions, when executed by a processor, causing the processor to: receiving media stream data for a media stream, the media stream comprising at least two media tracks, the media stream conforming to the Dynamic Adaptive Streaming over HTTP (DASH) standard or the Common Media Application Format (CMAF), the media stream data comprising a plurality of media chunks including a first media chunk and a second media chunk, and Addressable Resource Index (ARI) information associated with the first media chunk; determining track switch information based on the ARI information; determining based on the track change information that a switch to a different media track in the second media chunk is necessary; and receiving the first media chunk and the second media chunk via respective media tracks, each of the first media chunk and the second media chunk being delivered to the device with a delivery delay of one chunk or less.

Images