ADDRESSABLE RESOURCE INDEX Event for CMAF and DASH Multimedia Streaming

By delivering ARI information through inband or MPD events with media segments, the inefficiencies in existing media streaming technologies are addressed, improving adaptability and reducing signaling overhead, thus optimizing media streaming performance.

JP2024542553AActive Publication Date: 2024-11-15TENCENT AMERICA LLC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024531145
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-03-27
Filing Date
2023-04-14
Publication Date
2024-11-15
Estimated Expiration
2043-04-14

AI Technical Summary

Technical Problem

Existing media streaming technologies, such as DASH and CMAF, face inefficiencies in conveying Addressable Resource Index (ARI) information, leading to excessive signaling overhead and additional HTTP requests, which can impact the adaptability and performance of media streaming systems.

Method used

The method involves delivering ARI information using inband events or MPD events alongside media segments, eliminating the need for separate metadata tracks and reducing the number of HTTP requests, thereby optimizing the processing and delivery of media streams.

Benefits of technology

This approach reduces signaling overhead and improves the adaptability and efficiency of media streaming by providing ARI information directly with media segments, enhancing client heuristics and reducing latency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542553000001_ABST
    Figure 2024542553000001_ABST
Patent Text Reader

Abstract

A method, an apparatus, and a readable storage medium for processing a media stream. The media stream may conform to the DASH standard or the CMAF standard. The method may include processing an Addressable Resource Index (ARI) event associated with the 5G media stream, the ARI event including at least one of an inband event sent with a first media slice in a content set, the content set including one or more media slices, or a Media Presentation Description (MPD) event, the ARI event carrying configuration information of the one or more media slices in the content set.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Non-provisional Application No. 18 / 190,713, filed March 27, 2023, which claims the benefit of priority to U.S. Provisional Application No. 63 / 388,574, filed July 12, 2022, U.S. Provisional Application No. 63 / 388,568, filed July 12, 2022, and U.S. Provisional Application No. 63 / 332,585, filed April 19, 2022, each of which is incorporated by reference in its entirety herein.

[0002] The present disclosure generally relates to media streaming technologies, including Dynamic Adaptive Streaming over Hypertext transfer protocol (DASH) and Common Media Application Format (CMAF). More specifically, the disclosed technologies include methods and apparatus for delivering Addressable Resource Index (ARI) information using DASH / CMAF events. [Background technology]

[0003] The discussion of the background art provided herein is intended to generally present the context of the present disclosure. The inventors' work is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that that work is described in this background section, together with aspects of the description that may not otherwise be admitted as prior art at the time of filing of this application.

[0004] MPEG (Moving picture expert group) dynamic adaptive streaming over hypertext transfer protocol (DASH) provides a standard for streaming multimedia content over IP networks. In the DASH standard, a media presentation description (MPD) provides information for a DASH client to adaptively stream media content by downloading media segments from a DASH server. The DASH standard enables streaming of multi-rate content. One aspect of the DASH standard includes the transport of MPD and inband events, and the client processing model used for the handling of the events.

[0005] Common Media Application Format (CMAF) is a standard used to package and deliver various forms of hypertext transfer protocol (HTTP) based media. It works with, for example, HTTP Live Streaming (HLS) and DASH protocols to simplify the delivery of media to playback devices by packaging data in a standardized transport container file. It also uses chunked encoding and chunked transfer encoding to reduce latency, which reduces storage requirements and therefore costs. Summary of the Invention [Means for solving the problem]

[0006] Aspects of the present disclosure provide methods and apparatuses for media stream processing, and more specifically, for delivering ARI (Addressable Resource Index) information using DASH / CMAF events. In some example implementations, a method for processing a media stream is disclosed. The method can include processing an ARI (Addressable Resource Index) event associated with the media stream, the ARI event comprising at least one of an inband event sent with a first media slice in a content set, the content set including one or more media slices, or a media presentation description (MPD) event, the ARI event carrying configuration information of the one or more media slices in the content set.

[0007] Aspects of the present disclosure also provide a media stream processing device or apparatus that includes circuitry configured to perform any of the above method implementations.

[0008] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for media stream processing for video decoding and / or encoding.

[0009] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a system according to one embodiment of the present disclosure. [Diagram 2] FIG. 1 illustrates a Dynamic Adaptive Streaming over HTTP (DASH) system according to one embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates a DASH client architecture according to one embodiment of the present disclosure. [Figure 4] A diagram showing an exemplary inband ARI event carried with a media segment or chunk. [Diagram 5] FIG. 13 is a flow diagram for post-processing an ARI event. [Figure 6] FIG. 2 is a flow chart diagram of a method according to an exemplary embodiment of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram of a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DASH (Dynamic Adaptive Streaming Over Hypertext Transfer Protocol) and MPD (Media Presentation Description) One popular format for media streaming includes Dynamic adaptive streaming over hypertext transfer protocol (DASH), as defined in ISO / IEC 23009-1. DASH is an adaptive bitrate streaming technology that enables streaming of media content using hypertext transfer protocol (HTTP) infrastructures such as web servers, content delivery networks (CDNs), various proxies and caches. DASH supports both on-demand and live streaming from DASH servers to DASH clients, and allows DASH clients to control the streaming session, so that DASH servers do not have to deal with the additional burden of stream adaptation management in large-scale deployments. DASH also allows DASH clients to select streaming from various DASH servers, thus achieving further load balancing of the network for the benefit of DASH clients. DASH provides dynamic switching between different media tracks, for example, by varying the bitrate to adapt to network conditions.

[0012] In DASH, a media presentation description (MPD) file provides information for a DASH client to adaptively stream media content by downloading media segments from a DASH server. The MPD may be in the form of an extensible markup language (XML) document. The MPD file can be fragmented and delivered in parts to reduce session start delays. The MPD file can also be updated during a streaming session. In some examples, the MPD file supports the representation of accessibility features, ratings, and camera views of the content. DASH also supports the delivery of multi-view and scalable encoded content.

[0013] An MPD file may contain a sequence of one or more periods. Each of the one or more periods may be defined, for example, by a period element in the MPD file. The MPD file may contain an MPD availableStartTime attribute and a per-period start attribute. For a media presentation of dynamic type (e.g., used for live services), the sum of the period's start attribute and the MPD attribute availableStartTime and the media segment's duration may indicate the availability time of the period in Coordinated Universal Time (UTC) format, in particular of the first media segment of each representation in the corresponding period. For a media presentation of static type (e.g., used for on-demand services), the start attribute of the first period may be 0. For any other period, the start attribute may specify a time offset between the start time of the corresponding period relative to the start time of the first period. Each period may extend until the start of the next period or, in the case of the last period, until the end of the media presentation. The period start times may be precise and may reflect the actual timing resulting from the playback of the media of all previous periods. In an example implementation, an MPD is provided such that the next period is a continuation of the content in the previous period, possibly the immediately following or later period (e.g., after an advertising period has been inserted).

[0014] Each period may include one or more adaptation sets, each of which may include one or more representations of the same media content. A representation may be one of several alternative encoding versions of the audio or video data. Representations may vary by encoding type, e.g., by bitrate, resolution, and / or codec of the video data and bitrate and / or codec of the audio data. The term representation may be used to refer to a section of encoded audio or video data that corresponds to a particular period of multimedia content and that has been encoded in a particular way.

[0015] An adaptation set for a particular time period may be assigned to a group indicated by a group attribute in the MPD file. Adaptation sets within the same group are generally considered alternatives to each other. For example, each adaptation set for video data for a particular time period may be assigned to the same group such that any adaptation set may be selected for decoding to display video data of the multimedia content for the corresponding time period. In some examples, media content within a time period may be represented by either one adaptation set from group 0, if present, or a combination of at most one adaptation set from each non-zero group. Timing data for each representation of a time period may be expressed relative to the start time of the time period.

[0016] A representation may contain one or more segments. Each representation may contain an initialization segment, or each segment of a representation may be self-initializing. If present, the initialization segment may contain initialization information for accessing the representation. In some cases, the initialization segment does not contain media data. A segment may be uniquely referenced by an identifier, such as a uniform resource locator (URL), uniform resource name (URN), or uniform resource identifier (URI).

[0017] In an exemplary implementation, the URL is a fixed scheme, e.g., "http" or "https", and is formatted according to IETF RFC 3986. <absolute-uri>A segment may be defined as a range of byte-ranges, possibly constrained by a byte range if a range attribute is provided with the URL. The byte range may be expressed as a byte-range-spec, for example as defined in IETF RFC 2616. This may be constrained to a single expression identifying a contiguous byte range. In one embodiment, the segment may be included in an MPD with a data URL, for example as defined in IETF RFC 2397.

[0018] The MPD file may provide an identifier for each segment, and in some examples, the MPD file may also provide byte ranges in the form of range attributes that may correspond to data for a segment within a file that is accessible by a URL, URN, or URI.

[0019] A sub-representation may be embedded (or contained) in a regular representation and described by a sub-representation element (e.g., SubRepresentation). A sub-representation element may describe properties of one or several media content components embedded in the representation. For example, a sub-representation element may describe properties of an embedded audio component (e.g., codec, sampling rate, etc.), an embedded subtitle (e.g., codec), or a sub-representation element may describe some embedded lower quality video layers (e.g., some lower frame rate, etc.). Sub-representations and representation elements may share some common attributes and elements.

[0020] Each representation may also include one or more media components, each of which may correspond to an encoded version of one particular media type, such as audio, video, or timed text (e.g., for closed captioning). Media components may be contiguous in time across boundaries of consecutive media segments within one representation.

[0021] In some example implementations, the DASH client can access and download MPD files from the DASH server. That is, the DASH client can obtain the MPD file used for the start of the live session. Based on the MPD file and for each selected representation, the DASH client can make several decisions, including determining what the latest segment is available on the server, determining the segment availability start time of the next segment and possibly future segments, determining from which timeline within the segment and when to start playing the segment, as well as determining when to obtain / fetch a new MPD file. As the service plays, the client can keep track of drift between the live service and its own playback, which needs to be detected and compensated for.

[0022] Common Media Application Format (CMAF) The Common Media Application Format (CMAF) for segmented media is an extensible standard for encoding and packaging segmented media objects for delivery and decoding to end-user devices in adaptive multimedia presentations. The CMAF specification defines several logical media objects, which are described below.

[0023] A CMAF track may contain encoded media samples including audio, video and subtitles. The media samples are stored in a CMAF specified container that derives from the ISO Base Media File Format (ISO_BMFF). The media samples may be optionally protected by MPEG Common Encryption. A track may contain a CMAF header and one or more CMAF fragments.

[0024] A CMAF Switching Set may contain alternative tracks that can be switched between and spliced ​​together at CMAF fragment boundaries in order to adaptively stream the same content at different bit rates and resolutions. An aligned CMAF Switching Set is two or more CMAF Switching Sets that are encoded from the same source with different encoding (e.g., different codecs) and are temporally aligned.

[0025] A CMAF selection set is a group of switching sets of the same media type that can contain alternative content (eg, different languages) or alternative encodings (eg, different codecs).

[0026] A CMAF presentation may include one or more presentation time-synchronized selection sets.

[0027] CMAF supports addressable objects so that media content can be distributed across different platforms. CMAF addressable objects may include: · CMAF Header: The header contains information including information for initializing the track. · CMAF Segment: A series of one or more contiguous fragments from the same track. · CMAF chunk: A chunk contains a contiguous subset of samples from a fragment. · CMAF track file: A complete track in one ISO_BMFF file.

[0028] DASH and CMAF Events In DASH and CMAF, events provide a means of signaling additional information to DASH / CMAF clients, and their associated applications. In an example implementation, events are timed, so they have a start time and a duration. Event information can include metadata describing the content of the media presentation. Additionally or alternatively, event information can include control messages for the media player associated with a particular time during playback of the media presentation, such as an ad insertion cue. Events may be realized, for example, as MPD events or inband events. They may be part of a manifest file (e.g., MPD) or embedded in an ISOBMFF-based media file, such as an event message (emsg) box.

[0029] A media presentation description (MPD) event is an event that can be signaled in the MPD. A set of events assigned to a media presentation time can be provided to the MPD at the period level. Events of the same type can be specified in an event stream element (e.g., EventStream) within a period element. An event ends at the end of the period even if its start time is after the period boundary or the duration of the event extends beyond the period boundary. An event stream element contains a message scheme identification (e.g., @schemeIdUri) and an optional value for the event stream element (e.g., @value). Furthermore, since event streams contain timed events, a timescale attribute (e.g., @timescale) can be provided to assign an event to a specific media presentation time within a period. The timed event itself can be described by an event element contained in the event stream element.

[0030] Inband event streams can be multiplexed with a representation by adding event messages as part of a media segment. An event stream can be present in selected representations, only in one or a few selected adaptation sets, or in all representations. For example, one possible configuration is where only the audio adaptation set contains inband events, or only the video adaptation set contains inband events. Inband event streams present in a representation can be indicated by inband event stream elements (e.g., InbandEventStream) at various levels, such as at the adaptation set level or at the representation level. Furthermore, one representation can contain multiple inband event streams, each indicated by a separate inband event stream element.

[0031] 1 illustrates a system (100) according to one embodiment of the present disclosure. The system (100) includes a content server (110) and an information processing device (120). The content server (110) can provide a content stream that includes a primary content (e.g., a main program) and one or more timed metadata tracks.

[0032] The information processing device (120) can interface with the content server (110). For example, the information processing device (120) can play content received from the content server (110). The content playback can be performed based on a manifest file (e.g., MPD) received by the information processing device (120) (e.g., from the content server (110)). The manifest file can further include signaling for one or more timed metadata tracks.

[0033] An exemplary DASH / CMAF system is shown in Figure 2. The DASH system (200) may include a content server (210), an advertisement server (220), and an information processing device (230), which are connected to a network (250). The DASH system (200) may also include one or more supplemental content servers.

[0034] The content server (210) can provide primary content (e.g., a main program) and a manifest file (e.g., an MPD) to the information processing device (230). The manifest file can be generated, for example, by an MPD generator (214). The primary content and the manifest file can be provided by different servers in other embodiments.

[0035] The information processing device (230) can receive the MPD and obtain the main content from the HTTP server (212) of the content server (210) based on the MPD. The MPD can be processed by a DASH client (232) running on the information processing device (230). Furthermore, the DASH client (232) can obtain advertisement content from the advertisement server (220) and other content (e.g., interactive content) from one or more supplemental content servers. The main content and advertisement content can be processed by the DASH client (232) and output for display on a display device (236). The display device (236) can be integrated into the information processing device (230) or can be external to the information processing device (230). Furthermore, the DASH client (232) can extract event information from one or more timed metadata tracks and send the extracted event information to an application (234) for further processing. The application (234) can be configured to display the supplemental content based on the event information, for example.

[0036] The advertisement server (220) can store advertisement content in an advertisement storage unit such as a memory, etc. The information processing device (230) can request the stored advertisement content based on the event information.

[0037] 3 illustrates an example of a DASH / CMAF client architecture for processing DASH and CMAF events, according to an embodiment of the present disclosure. A DASH / CMAF client (or DASH / CMAF player) communicates with an application (390) and can be configured to process various types of events, including (i) MPD events, (ii) inband events, and (iii) timed metadata events.

[0038] The manifest parser (305) parses the manifest (e.g., MPD). The manifest is provided, for example, by the content server (110, 210). The manifest parser (305) extracts event information for MPD events, inband events, and timed metadata events embedded in timed metadata tracks. The extracted event information can be provided to the DASH logic (310) (e.g., control, selection, and heuristic logic of a DASH player). Based on the event information, the DASH logic (310) can inform the application (390) of the event scheme signaled in the manifest.

[0039] The event information may include event scheme information for distinguishing between different event streams. An application (390) may use the event scheme information to subscribe to event schemes of interest. The application (390) may further indicate a desired delivery mode for each subscribed scheme via one or more subscription APIs. For example, the application (390) may send a subscription request to a DASH client that identifies one or more event schemes of interest and any desired corresponding delivery modes.

[0040] If an application (390) subscribes to one or more event schemes that are delivered as part of one or more timed metadata tracks, the inband event and "moof" parser (325) can stream one or more timed metadata tracks to the timed metadata track parser (330). For example, the inband event and "moof" parser (325) parses a movie fragment box ("moof") and subsequently parses the timed metadata tracks based on control information from the DASH logic (310).

[0041] The timed metadata track parser (330) can extract the event messages embedded in the timed metadata tracks. The extracted event messages can be stored in an event buffer (335) (e.g., an event buffer). The synchronizer / dispatcher module (340) (e.g., an event and timed metadata synchronizer and dispatcher) can emit (or send) the subscribed events to the application (390).

[0042] MPD events described in the MPD can be parsed by the manifest parser (305) and stored in a buffer (335). For example, the manifest parser (305) parses each event stream element of the MPD and each event described in each event stream element. For each event signaled in the MPD, event information such as presentation time and event duration can be stored in a buffer (335) associated with the event.

[0043] An inband event and "moof" parser (325) can parse the media segments to extract inband event messages. Any such identified inband events and associated presentation times and durations can be stored in a buffer (335).

[0044] Thus, the buffer (335) may store MPD events, inband events, and / or timed metadata events therein. The buffer (335) may be, for example, a first-in, first-out (FIFO) buffer. The buffer (335) may be managed in correspondence with the media buffer (350). For example, as long as a media segment is present in the media buffer (350), any events or timed metadata corresponding to that media segment may be stored in the buffer (335).

[0045] The DASH Access Application Programming Interface (API) (315) can manage the fetching and reception of content streams (or data flows) including media content and various metadata via the HTTP protocol stack (320). The DASH Access API (315) can separate the received content stream into different data flows. The data flows provided to the inband event and moof parser can include media segments, one or more timed metadata tracks, and inband event signaling contained in the media segments. In one embodiment, the data flows provided to the manifest parser 305 can include an MPD.

[0046] The DASH Access API (315) can forward the manifest to the manifest parser (305). In addition to describing events, the manifest can also provide information about media segments to the DASH logic (310), which can communicate with the application (390) and the inband event and moof parser (325). The application (390) can be associated with media content processed by the DASH client. Control / sync signals exchanged between the application (390), the DASH logic (310), the manifest parser (305), and the DASH Access API (315) can control the fetching of media segments from the HTTP stack (320) based on information about the media segments described in the manifest.

[0047] The inband event and moof parser (325) can parse the media data flow into media segments that contain the media content, the timed metadata in the timed metadata tracks, and any signaled inband events in the media segments. The media segments that contain the media content can be parsed by the file format parser (345) and stored in the media buffer (350).

[0048] The events stored in the buffer (335) allow the synchronizer / dispatcher (340) to communicate available events related to (or of interest to) the application via an event / metadata API. Applications can be configured to process available events (e.g., MPD events, inband events, or timed metadata events) and subscribe to specific events or timed metadata by notifying the synchronizer / dispatcher (340). Any events stored in the buffer (335) that are not related to an application, but instead related to the DASH client itself, can be forwarded by the synchronizer / dispatcher (340) to the DASH logic (310) for further processing.

[0049] In response to an application (390) subscribing to a particular event, the synchronizer / dispatcher (340) can communicate application event instances (or samples of timed metadata) corresponding to the event scheme to which the application subscribes. The event instances can be communicated according to a dispatch mode indicated by the subscription request or a default dispatch mode (e.g., for a particular event scheme). For example, in a dispatch-on-receive mode, the event instances may be sent to the application (390) upon receipt in the buffer (335), whereas in a dispatch-on-start mode, the event instances may be sent to the application (390) at their associated presentation times, for example, in synchronization with a timing signal from the media decoder (355).

[0050] DASH / CMAF Addressable Resource Index In some example implementations, it is desirable for an adaptive streaming client (e.g., a DASH or CMAF client) to have precise knowledge of ARI (Addressable Resource Index) information, such as offset, size, duration, and quality, of timed aligned segments or chunks that are present in the same adaptation / switching set. Such ARI information, together with relative information about, for example, immediately adjacent chunks or segments, may be used by the DASH / CMAF client to facilitate client heuristics. Addressable Resources may include track files, segments, and chunks in a CMAF context. In case of an on-demand service, a precise map of such information may be provided by a segment index. Note that similar concepts and implementations may also be applied to a DASH context.

[0051] In some example implementations, an Addressable Resource Index (ARI) may be defined as follows: Sample entry type: 'cari' Container: Sample Description Box ("STSD") Required: No Quantity: 0 or 1

[0052] The metadata describes all the details of the addressable resources and, in one index track, for example, a subset of the CMAF switching set defined in ISO / IEC 23000-19.

[0053] Table 1 below shows an illustrative sample entry of CMAF Addressable Resource Index metadata.

[0054] [Table 1]

[0055] Table 2 below shows an example syntax for the ARI sample.

[0056] [Table 2]

[0057] For example, the semantics of the above syntax are explained below: · switching_set_identifier specifies a unique identifier for the switching set in the context of the application. ·num_tracks indicates the track numbers assigned in the ARI track. · track_ID - Uses the track_ID to select and sequence among samples of a track. · num_quality_indicators specifies the number of quality indicators used to identify the quality of the chunk. quality_identifier specifies an identifier that indicates how the quality values ​​in the sample are to be interpreted. This is a 4CC code that can be registered. · segment_start_flag indicates whether the chunk is the start of a segment. · marker specifies whether this chunk contains at least one styp box. · SAP_type specifies the SAP type of the chunk. emsg_flag indicates whether this chunk provides at least one emsg box. · prft_flag indicates whether this chunk contains at least one prft box. offset specifies the offset of the chunk from the beginning of the segment. · size gives the size of the chunk in octets. quality gives the quality of the chunk according to a given quality scheme identifier. The data type of the quality value (integer or floating point) is defined by the quality scheme. If the quality scheme identifier is the empty string, the quality is interpreted proportionally using an unsigned integer, with increasing quality corresponding to increasing values. ·loss indicates that media data of the chunk was lost. num_prediction_pairs gives how many pairs of expected predictions are provided. · prediction_min_windows provides a minbuffer time value that is equal to the MPD value. predicted_max_bitrate provides a bandwidth value identical to the MPD semantic that is held for the duration of the prediction_min_windows value.

[0058] Delivery of ARI using events In an exemplary implementation in the case of DASH / CMAF, a dedicated metadata track, i.e., ARI track, is created to carry ARI-related information such as offset, size, quality, etc. of timed aligned segments or chunks that are in the same adaptation / switching set, so that the client has relative information about the nearest chunks or segments to facilitate client heuristics, e.g., which can be used by the client when dynamically switching media tracks or representations.

[0059] Note that one disadvantage of using metadata tracks to carry ARI information (e.g. ARI samples) is the excessive signaling overhead: for every segment that requires ARI information, an extra HTTP GET request is required by the client.

[0060] Embodiments of the present disclosure include a method of carrying ARI (i.e., ARI information, ARI sample) without using an ARI metadata track. That is, rather than using a metadata track to carry the ARI (which would then result in an extra HTTP GET request (as the ARI sample is sent separately with the media segment / chunk)), the present disclosure may send the ARI sample via an event, such as an inband event or an MPD event. This approach to carrying ARI samples is considered to be a "media segment / chunk associated ARI transmission" since the ARI sample is sent with the media segment / chunk. An event carrying ARI is referred to as an ARI event. Using an ARI event may provide at least the following advantages:

[0061] 1. No extra metadata track is needed, resulting in one less HTTP GET request by the CMAF / DASH client for each segment / chunk for which additional ARI information is needed. For example, a CMAF / DASH client may need additional ARI information to facilitate segment / chunk processing. In this case, the ARI information can be obtained directly from the ARI event carried with the segment / chunk.

[0062] 2. The event processing model allows for the processing of event messages and dispatching them to DASH / CMAF clients. The processing model allows for the timing of ARI samples to be carried as part of the event timing model.

[0063] 3. Flexibility: ARI information may be carried by events in one, some, or all of the presentations in the DASH adaptation set or CMAF switching set, depending on the inband events required, for example.

[0064] 4. Adaptability and portability: ARI events can be parsed by a packager (eg, derived from inband events or ARI tracks received from an encoder) and added to the MPD as MPD events.

[0065] In some example implementations, the ARI information for a chunk / segment may be included in the same chunk / segment.

[0066] In some example implementations, the ARI information of a chunk / segment may be included in subsequent chunks / segments arranged along the time axis.

[0067] In some example implementations, rather than using inband events to carry the ARI information, MPD events may be used to carry the ARI information, which may be particularly suitable for on-demand content.

[0068] In this embodiment, the ARI information may be carried in an emsg box. Each emsg box may belong to an event scheme defined by or associated with a scheme URN identifier.

[0069] For example, referring to Figure 4, ARI event 410 is carried with segment / chunk c(n) in media 1. The ARI information from ARI event 410 applies to the same segment / chunk c(n). When the current chunk ARI information is included in the current chunk event, the scheme URN identifier may be defined as "urn:mpeg:dash:event:ari:2022".

[0070] In another example, referring to Figure 4, an ARI event 412 is carried with segment / chunk c(n+1) in media 2. The ARI information from the ARI event 412 applies to the next segment / chunk c(n+2). If the current chunk event contains next chunk ARI information, the scheme URN identifier may be defined as "urn:mpeg:dash:event:ari-next:2022". For example, the dispatch mode of the event may be set to on receipt.

[0071] Table 3 below shows example parameters for an ARI event in an MPD.

[0072] [Table 3]

[0073] As shown in Table 3, two elements may be used to describe an ARI event: EventStream and InbandEventStream. Both streams may contain a value attribute. The value attribute may carry the CmafAriMetaDataSampleEntry fields described in Table 1. For example, a CmafAriMetaDataSampleEntry field may contain the following fields: switching_set_identifier num_tracks num_quality_indicators · A numbered list of track_ids List of quality_identifiers

[0074] In some example implementations, an Event element may include a presentationTime attribute (eg, Event@presentationTime), which indicates a chunk offset from the beginning of the period to which the event's ARI information applies.

[0075] In some example implementations, the Event element may include a duration attribute (e.g., Event@duration), which indicates the duration for which the ARI information should be used. For example, this may include the duration of a chunk, or the duration of a segment.

[0076] In some example implementations, an event may include an event body, which may share the same structure as a CmafAriFormatStruct, as defined in Table 2.

[0077] Table 4 below shows an example of the emsg parameter for an inband ARI event.

[0078] [Table 4]

[0079] Note that the event body in an MPD event and the message_data in an inband event share the same CMAF ARI sample structure, CmafAriFormatStruct. Therefore, the parsing and processing of the ARI sample after receiving the event from the event dispatcher is the same. That is, the same parsing and processing logic is shared for MPD events and inband events.

[0080] In some embodiments, ARI events may be processed and dispatched, for example, according to clause A.13 of ISO / IEC 23009-1. For example, ARI events may be processed and dispatched in the exemplary DASH / CMAF client architecture shown in FIG.

[0081] In some embodiments, after a call is made for an ARI event, post-processing of the ARI event is performed. For post-processing, the parameters shown in Table 5 may be relied upon.

[0082] [Table 5]

[0083] FIG. 5 shows an example flow 500 for post-processing of an ARI event, including the following steps:

[0084] Step 510: The value field in the ARI event is parsed to extract general information about the event. As previously mentioned, the general information may include a switching_set_identifier, num_tracks, num_quality_indicators, an ordered list of track_ids, and a list of quality_identifiers.

[0085] Step 520: Use presentation_time to identify the chunk or segment to which the information applies.

[0086] Step 530: Parse the event payload (e.g., event body or message_data) and construct a CmafAriFormatStruct.

[0087] Step 540: Process the value and the CmafAriFormatStruct with its heuristics to determine whether to switch to a new representation or stay with the same representation.

[0088] FIG. 6 illustrates an exemplary method 600 for processing a media stream. The media stream may include, for example, a 4G media stream (for a media stream distributed in a 4G network) or a 5G media stream (for a media stream distributed in a 5G network). The method may be implemented, for example, by a computer system described below. The media stream may conform to the DASH standard or the CMAF standard. The method 600 may include some or all of the steps 610 of processing an Addressable Resource Index (ARI) event associated with the media stream, the ARI event including at least one of an inband event transmitted with a first media slice in a content set, the content set comprising one or more media slices, or a Media Presentation Description (MPD) event, the ARI event carrying configuration information of one or more media slices in the content set. The media stream may be supported, for example, by a cloud computing network, a wireless network (e.g., a 4G Long-Term Evolution (LTE) network or a 5G New Radio (NR) network), a Wi-Fi network. The media streams may include, but are not limited to, 4G media streams, 5G media streams, Wi-Fi media streams, and mixtures thereof.

[0089] In some example implementations, the inband events and the MPD events are each identified by or associated with a scheme identifier, which includes a scheme identifier Uniform Resource Identifier (URI).

[0090] In some example implementations, the configuration information of the one or more media slices in method 600 may include at least one of configuration information of a first media slice or configuration information of a second media slice following the first media slice.

[0091] The embodiments of the present disclosure apply to both DASH and CMAF. The content set in the method 600 may include at least one of an adaptation set when the media stream complies with Dynamic Adaptive Streaming over Hypertext transfer protocol (DASH) or a switching set when the media stream complies with Common Media Application Format (CMAF), and the first media slice includes at least one of a media segment in a first representation of the adaptation set or a media chunk in a first track of the switching set.

[0092] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the method (or embodiment), the DASH client, and the CMAF client may be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to the DASH and / or CMAF technologies / standards.

[0093] The techniques described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 7 illustrates a computer system (1800) suitable for implementing certain embodiments of the disclosed subject matter.

[0094] Computer software can be encoded using any suitable machine or computer language that can be assembled, compiled, linked, or similar mechanisms to create code including instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., directly, or via interpretation, microcode execution, etc.

[0095] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0096] 7 for computer system (1800) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (1800).

[0097] The computer system (1800) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users through, for example, tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown), etc. The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still camera, etc.), and video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0098] The input human interface devices may include one or more of a keyboard (1801), a mouse (1802), a trackpad (1803), a touch screen (1810), a data glove (not shown), a joystick (1805), a microphone (1806), a scanner (1807), and a camera (1808) (only one of each shown).

[0099] Additionally, the computer system (1800) may include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses through, for example, haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1810), data gloves (not shown), or joystick (1805), although there may also be haptic feedback devices that do not function as input devices), audio output devices (such as speakers (1809), headphones (not shown)), visual output devices (such as screens (1810) including CRT, LCD, plasma, or OLED screens, each of which may or may not have touch screen input capability, each of which may or may not have haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, or output in excess of three dimensions by means of stereoscopic graphic output, virtual reality glasses (not shown), holographic displays, or smoke tanks (not shown), etc.), and printers (not shown).

[0100] The computer system (1800) may also include human-accessible storage devices and their associated media, such as optical media (1820), including CD / DVD ROM / RW (1821) using CD / DVD or similar media, thumb drives (1822), removable hard drives or solid state drives (1823), legacy magnetic media such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD based devices (not shown) such as security dongles, and the like.

[0101] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0102] The computer system (1800) may also include an interface (1854) to one or more communication networks (1855). The network may be, for example, a wireless network, a wired network, or an optical network. The network may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television and terrestrial television, vehicular and industrial including CAN bus, etc. Any network requires an external network interface adapter that is typically attached to some general purpose data port or peripheral bus (1849) (e.g., a USB port of the computer system (1800)), while other networks are typically built into the core of the computer system (1800) by attaching to the system bus as described below (e.g., an Ethernet interface for a PC computer system or a cellular network interface for a smartphone computer system). Any of these networks may be used by the computer system (1800) to communicate with other entities. Such communications may be one-way receive-only (e.g., television broadcast), one-way transmit-only (e.g., from the CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces described above.

[0103] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core (1840) of the computer system (1800).

[0104] The cores (1840) may include one or more central processing units (CPUs) (1841), graphics processing units (GPUs) (1842), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (1843), hardware accelerators for specific tasks (1844), graphics adapters (1850), and the like. These devices may be connected via a system bus (1848), along with read-only memory (ROM) (1845), random access memory (1846), and internal mass storage (1847) such as internal hard drives and SSDs that are not user accessible. In some computer systems, the system bus (1848) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus (1848) or via a peripheral bus (1849). In one example, a screen (1810) may be connected to the graphics adapter (1850). Architectures for peripheral buses include PCI, USB, etc.

[0105] The CPU (1841), GPU (1842), FPGA (1843), and accelerator (1844) can execute a number of instructions that can be constructed in combination as described above as computer code. This computer code can be stored in ROM (1845) or RAM (1846). Permanent data can be stored, for example, in internal mass storage (1847), while transient data can also be stored in RAM (1846). Rapid storage and retrieval of any memory device can be made possible by using cache memory that can be closely associated with one or more of the CPU (1841), GPU (1842), mass storage (1847), ROM (1845), RAM (1846), etc.

[0106] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0107] As a non-limiting example, a computer system having the architecture (1800), and in particular the core (1840), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with mass storage that a user can directly operate on, as described above, or even specific storage of the core (1840), such as the core internal mass storage (1847) or non-transitory storage, such as ROM (1845). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1840). The computer-readable media can include one or more memory devices or chips, depending on the particular need. The software enables the cores (1840), and in particular the processors (including CPUs, GPUs, FPGAs, etc.) in the cores (1840), to execute certain processes or certain portions of certain processes described in the present application, including the definition of data structures stored in RAM (1846) and the modification of such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic being hardwired or otherwise implemented in circuitry (e.g., accelerators (1844)) that may act in place of or cooperate with the software to execute certain processes or certain portions of certain processes described in the present application. Where appropriate, references to "software" may encompass logic, and vice versa. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0108] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0109] 100 Systems 110 Content Server 120 Information processing device 200 DASH System 210 Content Server 212 HTTP Server 214 MPD Generator 220 Ad Server 230 Information processing equipment 232 DASH client 234 Applications 236 Display Devices 250 Network 305 Manifest Parser 310 DASH Logic 315 DASH Access API 320 HTTP protocol stack 325 inband events and the 'moof' parser 330 Timed Metadata Track Parser 335 Event Buffer 340 Synchronizer / Dispatcher Module 345 File Format Parser 350 Media Buffer 355 Media Decoder 390 Applications 410 ARI Event 412 ARI Event 500 Flow 600 ways 1800 Computer Systems, Architecture 1801 Keyboard 1802 Mouse 1803 Trackpad 1805 Joystick 1806 Mike 1807 Scanner 1808 Camera 1809 Speaker 1810 Touch Screen 1820 Optical media 1821 CD / DVD ROM / RW 1822 Thumb Drive 1823 Removable Hard Drive or Solid State Drive 1840 Core 1841 Central Processing Unit (CPU) 1842 Graphics Processing Unit (GPU) 1843 Field Programmable Gate Area (FPGA) 1844 Hardware Accelerator 1845 Read-Only Memory (ROM) 1846 Random Access Memory (RAM) 1847 Internal Mass Storage 1848 System Bus 1849 General Purpose Data Port or Peripheral Bus 1850 Graphics Adapter 1854 Interface 1855 Communication Network

Claims

1. 1. A method for processing a 5G media stream, the 5G media stream conforming to the Dynamic Adaptive Streaming over HTTP (DASH) standard or the Common Media Application Format (CMAF), the method comprising: Processing an Addressable Resource Index (ARI) event associated with the 5G media stream, The ARI event is an inband event sent with a first media slice in a content set, the content set including one or more media slices; or MPD (Media Presentation description) Event At least one of the ARI event conveys configuration information of the one or more media slices in the content set. A method comprising:

2. The method of claim 1 , wherein the inband events and the MPD events are each identified by or associated with a scheme identifier, the scheme identifier comprising a scheme identifier Uniform Resource Identifier (URI).

3. The configuration information of the one or more media slices comprises: Configuration information of the first media slice; or Configuration information of a second media slice following the first media slice; The method of claim 1 , comprising at least one of:

4. The method of claim 3 , wherein a scheme identifier of the ARI event or the MPD event indicates whether the configuration information applies to the first media slice or the second media slice.

5. The content set: an adaptation set when the 5G media stream complies with the Dynamic Adaptive Streaming over Hypertext transfer protocol (DASH); or A switching set when the 5G media stream complies with CMAF (Common Media Application Format); At least one of The first media slice comprises: a segment in a first representation of the adaptation set; or a chunk in a first track of said switching set; At least one of 4. The method according to any one of claims 1 to 3.

6. The configuration information of the one or more media slices comprises: an offset of each of the one or more media slices relative to a start time of a container that includes the content set, the container including at least one of a CMAF presentation, or a DASH period; the size of each of the one or more media slices; or a quality of each of the one or more media slices; The method of claim 5, comprising at least one of:

7. 6. The method of claim 5, wherein the ARI event carries timing information related to at least one chunk in the switching set or at least one segment in the adaptation set, and the timing information indicates a start time of each of the at least one chunk or each of the at least one segment.

8. determining whether to switch to a representation different from the first representation or a track different from the first track based on the configuration information of the one or more media slices in the content set; 6. The method of claim 5, further comprising:

9. determining whether to switch to a representation different from the first representation or a track different from the first track based on the configuration information of the one or more media slices in the content set and on information extracted from the first media slice; 6. The method of claim 5, further comprising:

10. 6. The method of claim 5, further comprising receiving one or more additional ARI events from a representation different from the first representation or a track different from the first track.

11. The ARI event includes a message portion, the message portion comprising: a message_data field when the ARI event is an inband event; or an event body field when the ARI event is an MPD event; It is one of the the message_data field and the event body field each carry one or more ARI samples; the one or more ARI samples in the message_data field and the one or more ARI samples in the event body field share the same data structure; The method of claim 5.

12. parsing the ARI event to obtain a presentation time associated with the ARI event; determining a target media slice to which the ARI event applies based on the presentation time; constructing the one or more ARI samples based on the data structure; determining whether to switch to a representation different from the first representation or a track different from the first track based on the one or more ARI samples; 12. The method of claim 11, further comprising:

13. A device for processing a 5G media stream, the 5G media stream conforming to the Dynamic Adaptive Streaming over HTTP (DASH) standard or the Common Media Application Format (CMAF), the device comprising: a memory for storing computer instructions; and a processor in communication with the memory, the processor executing the computer instructions causing the device to: and configured to process an Addressable Resource Index (ARI) event associated with the 5G media stream; The ARI event is an inband event sent with a first media slice in a content set, the content set including one or more media slices; or MPD (Media Presentation description) events, At least one of The device, wherein the ARI event conveys configuration information of the one or more media slices in the content set.

14. The device of claim 13 , wherein the inband events and the MPD events are each identified by or associated with a scheme identifier, the scheme identifier comprising a scheme identifier Uniform Resource Identifier (URI).

15. The content set: an adaptation set when the 5G media stream complies with the Dynamic Adaptive Streaming over Hypertext transfer protocol (DASH); or A switching set when the 5G media stream complies with CMAF (Common Media Application Format); At least one of The first media slice comprises: a segment in a first representation of the adaptation set; or a chunk in a first track of said switching set; At least one of A device according to any one of claims 13 to 14.

16. The configuration information of the one or more media slices comprises: an offset of each of the one or more media slices relative to a start time of a container that includes the content set, the container including at least one of a CMAF presentation, or a DASH period; the size of each of the one or more media slices; or a quality of each of the one or more media slices; The device of claim 15, comprising at least one of:

17. When the processor executes the computer instructions, the processor causes the device to: determining whether to switch to a representation different from the first representation or a track different from the first track based on the configuration information of the one or more media slices in the content set. The device of claim 15, further configured to:

18. A non-transitory storage medium for storing computer-readable instructions, the computer-readable instructions, when executed by a processor in a device for processing 5G media streams according to the Dynamic Adaptive Streaming over HTTP (DASH) standard or the Common Media Application Format (CMAF), causing the processor to: Process an Addressable Resource Index (ARI) event associated with the 5G media stream; The ARI event is an inband event sent with a first media slice in a content set, the content set including one or more media slices; or MPD (Media Presentation description) events, At least one of the ARI event conveys configuration information of the one or more media slices in the content set; Non-transitory storage medium.

19. 20. The non-transitory storage medium of claim 18, wherein the inband events and the MPD events are each identified by or associated with a scheme identifier, the scheme identifier comprising a scheme identifier Uniform Resource Identifier (URI).

20. The content set: an adaptation set when the 5G media stream complies with the Dynamic Adaptive Streaming over Hypertext transfer protocol (DASH); or A switching set when the 5G media stream complies with CMAF (Common Media Application Format); At least one of The first media slice comprises: a segment in a first representation of the adaptation set; or a chunk in a first track of said switching set; At least one of A non-transitory storage medium according to any one of claims 18 to 19.

Citation Information

Patent Citations

  • Targeted ad insertion for streaming media data

    JP2017517167A

  • Method and apparatus for controlled selection of viewing point and viewing orientation of audiovisual content

    JP2019526994A

  • Content distribution device, terminal, and program

    JP2021117755A

  • Segment types as delimiters and addressable resource identifiers

    US20180288500A1

  • Streaming media data including an addressable resource index track

    US20220007086A1