ADDRESSABLE RESOURCE INDEX event for CMAF and DASH multimedia streaming

Abstract

A method, an apparatus, and a readable storage medium for processing a media stream. The media stream may conform to the DASH standard or the CMAF standard. The method may include processing an Addressable Resource Index (ARI) event associated with the 5G media stream, the ARI event including at least one of an inband event sent with a first media slice in a content set, the content set including one or more media slices, or a Media Presentation Description (MPD) event, the ARI event carrying configuration information of the one or more media slices in the content set.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Non-provisional Application No. 18 / 190,713, filed March 27, 2023, which claims the benefit of priority to U.S. Provisional Application No. 63 / 388,574, filed July 12, 2022, U.S. Provisional Application No. 63 / 388,568, filed July 12, 2022, and U.S. Provisional Patent Application No. 63 / 332,585, filed April 19, 2022, each of which is incorporated by reference herein in its entirety.

[0002] This disclosure generally relates to media streaming technologies, including Dynamic Adaptive Streaming over Hypertext transfer protocol (DASH) and Common Media Application Format (CMAF). More specifically, the technologies of this disclosure include methods and apparatus for delivering Addressable Resource Index (ARI) information using DASH / CMAF events. [Background technology]

[0003] The discussion of the background art provided herein is intended to generally present the context for the present disclosure. The inventors' work is not admitted expressly or implicitly as prior art to the present disclosure to the extent that that work is described in this background section, along with aspects of the description that may not otherwise be admitted as prior art at the time of filing of this application.

[0004] The Moving Picture Expert Group (MPEG) dynamic adaptive streaming over hypertext transfer protocol (DASH) provides a standard for streaming multimedia content over IP networks. In the DASH standard, a media presentation description (MPD) provides information for a DASH client to adaptively stream media content by downloading media segments from a DASH server. The DASH standard enables multi-rate content streaming. One aspect of the DASH standard includes the transport of MPD events and in-band events, and the client processing model used to handle the events.

[0005] The Common Media Application Format (CMAF) is a standard used to package and deliver various forms of HTTP-based (Hypertext transfer protocol) media. The standard works with protocols such as HTTP Live Streaming (HLS) and DASH to simplify delivery of media to playback devices by packaging data in a standardized transport container file. The standard also uses chunked encoding and chunked transfer encoding to reduce latency, which reduces storage requirements and therefore costs. Summary of the Invention [Means for solving the problem]

[0006] Aspects of the present disclosure provide methods and apparatuses for media stream processing, and more particularly, for delivering Addressable Resource Index (ARI) information using DASH / CMAF events. In some example implementations, a method for processing a media stream is disclosed. The method can include processing an Addressable Resource Index (ARI) event associated with the media stream, the ARI event comprising at least one of an in-band event sent with a first media slice in a content set, the content set including one or more media slices, or a Media Presentation Description (MPD) event, the ARI event carrying configuration information of the one or more media slices in the content set.

[0007] Aspects of the present disclosure also provide a media stream processing device or apparatus that includes circuitry configured to perform any of the above method implementations.

[0008] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for media stream processing for video decoding and / or encoding.

[0009] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a system according to one embodiment of the present disclosure. [Figure 2] FIG. 1 illustrates a Dynamic Adaptive Streaming over HTTP (DASH) system according to one embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates a DASH client architecture according to one embodiment of the present disclosure. [Figure 4] FIG. 10 illustrates an exemplary inband ARI event carried with a media segment or chunk. [Figure 5] FIG. 10 is a flowchart for post-processing an ARI event. [Figure 6] FIG. 2 is a flowchart diagram of a method according to an exemplary embodiment of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram of a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] DASH (Dynamic Adaptive Streaming Over Hypertext Transfer Protocol) and MPD (Media Presentation Description) One common format for media streaming includes Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH), as defined in ISO / IEC 23009-1. DASH is an adaptive bitrate streaming technology that enables streaming of media content using Hypertext Transfer Protocol (HTTP) infrastructure, such as web servers, content delivery networks (CDNs), and various proxies and caches. DASH supports both on-demand and live streaming from DASH servers to DASH clients and allows DASH clients to control the streaming session, so that DASH servers do not need to deal with the additional burden of stream adaptation management in large-scale deployments. DASH also allows DASH clients to select streaming from various DASH servers, thus achieving further network load balancing for the benefit of DASH clients. DASH provides dynamic switching between different media tracks, for example, by varying the bitrate to adapt to network conditions.

[0012] In DASH, a media presentation description (MPD) file provides information for a DASH client to adaptively stream media content by downloading media segments from a DASH server. The MPD may be in the form of an Extensible Markup Language (XML) document. The MPD file can be fragmented and delivered in parts to reduce session start delays. The MPD file can also be updated during a streaming session. In some examples, the MPD file supports content accessibility features, ratings, and camera view representation. DASH also supports the delivery of multi-view and scalable encoded content.

[0013] An MPD file may contain a sequence of one or more periods. Each of the one or more periods may be defined, for example, by a period element in the MPD file. The MPD file may contain an MPD availableStartTime attribute and a per-period start attribute. For a media presentation of dynamic type (e.g., used for live services), the sum of a period's start attribute, the MPD attribute availableStartTime, and the media segment's duration may indicate the availability time of the period in Coordinated Universal Time (UTC) format, in particular of the first media segment of each representation in the corresponding period. For a media presentation of static type (e.g., used for on-demand services), the start attribute of the first period may be 0. For any other periods, the start attribute may specify a time offset between the start time of the corresponding period relative to the start time of the first period. Each period may extend until the start of the next period, or, in the case of the last period, until the end of the media presentation. The period start time may be precise and may reflect the actual timing resulting from the playback of the media of all previous periods. In an example implementation, an MPD is provided such that the next period is a continuation of the content in the previous period, possibly the immediately following period or a later period (e.g., after an advertising period has been inserted).

[0014] Each period can contain one or more adaptation sets, each of which can contain one or more representations of the same media content. A representation can be one of several alternative encoding versions of the audio or video data. Representations can vary by encoding type, for example, by bitrate, resolution, and / or codec of the video data and bitrate, and / or codec of the audio data. The term representation can be used to refer to a section of encoded audio or video data that corresponds to a particular period of multimedia content and is encoded in a particular way.

[0015] An adaptation set for a particular time period can be assigned to a group indicated by a group attribute in the MPD file. Adaptation sets within the same group are generally considered alternatives to each other. For example, each adaptation set for video data for a particular time period can be assigned to the same group so that any adaptation set can be selected for decoding to display the video data of the multimedia content for the corresponding time period. In some examples, media content within a time period can be represented by either one adaptation set from group 0, if present, or a combination of at most one adaptation set from each non-zero group. Timing data for each representation of a time period can be expressed relative to the start time of the time period.

[0016] A representation can contain one or more segments. Each representation can contain an initialization segment, or each segment of a representation can be self-initializing. If present, the initialization segment can contain initialization information for accessing the representation. In some cases, the initialization segment does not contain media data. A segment can be uniquely referenced by an identifier such as a uniform resource locator (URL), uniform resource name (URN), or uniform resource identifier (URI).

[0017] In an exemplary implementation, the URL is defined according to IETF RFC 3986, with a fixed scheme, e.g., "http" or "https." <absolute-uri>, possibly constrained by a byte range if a range attribute is provided with the URL. The byte range may be expressed as a byte-range-spec, for example as defined in IETF RFC 2616. This may be constrained to a single expression identifying a contiguous byte range. In one embodiment, the segment may be included in an MPD with a data URL, for example as defined in IETF RFC 2397.

[0018] An MPD file can provide an identifier for each segment. In some examples, an MPD file can also provide byte ranges in the form of range attributes that can correspond to data for a segment within a file that is accessible by a URL, URN, or URI.

[0019] A sub-representation can be embedded (or contained) in a regular representation and described by a sub-representation element (e.g., SubRepresentation). A sub-representation element can describe the properties of one or several media content components embedded in the representation. For example, a sub-representation element can describe the properties of an embedded audio component (e.g., codec, sampling rate, etc.), an embedded subtitle (e.g., codec), or a sub-representation element can describe several embedded lower-quality video layers (e.g., several lower frame rates, etc.). Sub-representations and representation elements can share some common attributes and elements.

[0020] Each representation may also include one or more media components, each of which may correspond to an encoded version of one particular media type, such as audio, video, or timed text (e.g., for closed captioning). Media components may be temporally contiguous across boundaries of consecutive media segments within a representation.

[0021] In some example implementations, a DASH client can access and download an MPD file from a DASH server. That is, the DASH client can obtain an MPD file used to start a live session. Based on the MPD file and for each selected representation, the DASH client can make several decisions, including determining what the latest segment is available on the server, determining the segment availability start time of the next segment and possibly future segments, determining from which timeline within the segment to start playing the segment, and determining when to obtain / fetch a new MPD file. As the service plays, the client can keep track of any drift between the live service and its own playback, which needs to be detected and compensated for.

[0022] Common Media Application Format (CMAF) The Common Media Application Format (CMAF) for segmented media is an extensible standard for encoding and packaging segmented media objects for delivery and decoding to end-user devices in adaptive multimedia presentations. The CMAF specification defines several logical media objects, which are described below.

[0023] A CMAF track may contain encoded media samples, including audio, video, and subtitles. The media samples are stored in a CMAF-specified container derived from the ISO Base Media File Format (ISO_BMFF). The media samples may be optionally protected by MPEG Common Encryption. A track may contain a CMAF header and one or more CMAF fragments.

[0024] A CMAF Switching Set may contain alternative tracks that can be switched between and spliced ​​together at CMAF fragment boundaries to adaptively stream the same content at different bit rates and resolutions. An aligned CMAF Switching Set is two or more temporally aligned CMAF Switching Sets encoded from the same source using different encodings (e.g., different codecs).

[0025] A CMAF selection set is a group of switching sets of the same media type that can contain alternative content (e.g., different languages) or alternative encodings (e.g., different codecs).

[0026] A CMAF presentation may include one or more presentation time-synchronized selection sets.

[0027] CMAF supports addressable objects so that media content can be distributed across different platforms. CMAF addressable objects may include: · CMAF Header: The header contains information including information for initializing the track. · CMAF segment: A series of one or more contiguous fragments from the same track. · CMAF chunk: A chunk contains a contiguous subset of samples from a fragment. · CMAF track file: A complete track in one ISO_BMFF file.

[0028] DASH and CMAF Events In DASH and CMAF, events provide a means of signaling additional information to DASH / CMAF clients and their associated applications. In an example implementation, events are timed, so they have a start time and a duration. Event information can include metadata that describes the content of the media presentation. Additionally or alternatively, event information can include control messages for the media player associated with specific times during playback of the media presentation, such as ad insertion cues. Events may be realized, for example, as MPD events or in-band events. They can be part of a manifest file (e.g., MPD) or embedded in an ISOBMFF-based media file such as an event message (emsg) box.

[0029] An MPD (media presentation description) event is an event that can be signaled in the MPD. A set of events assigned to a media presentation time can be provided in the MPD at the period level. Events of the same type can be specified by an event stream element (e.g., EventStream) within a period element. An event ends at the end of the period, even if its start time is after the period boundary or its duration extends beyond the period boundary. An event stream element contains a message scheme identification (e.g., @schemeIdUri) and an optional value for the event stream element (e.g., @value). Furthermore, since event streams contain timed events, a timescale attribute (e.g., @timescale) can be provided to assign the event to a specific media presentation time within the period. The timed event itself can be described by an event element contained in the event stream element.

[0030] Inband event streams can be multiplexed with representations by adding event messages as part of media segments. An event stream can be present in selected representations, only one or a few selected adaptation sets, or in all representations. For example, one possible configuration is where only audio adaptation sets contain inband events, or only video adaptation sets contain inband events. Inband event streams present in a representation can be indicated by inband event stream elements (e.g., InbandEventStream) at various levels, such as at the adaptation set level or at the representation level. Furthermore, a representation can contain multiple inband event streams, each indicated by a separate inband event stream element.

[0031] 1 illustrates a system 100 according to one embodiment of the present disclosure. The system 100 includes a content server 110 and an information processing device 120. The content server 110 can provide a content stream including primary content (e.g., a main program) and one or more timed metadata tracks.

[0032] The information processing device (120) can interface with the content server (110). For example, the information processing device (120) can play content received from the content server (110). The content playback can be performed based on a manifest file (e.g., an MPD) received by the information processing device (120) (e.g., from the content server (110)). The manifest file can further include signaling for one or more timed metadata tracks.

[0033] An exemplary DASH / CMAF system is shown in Figure 2. The DASH system (200) may include a content server (210), an advertising server (220), and an information processing device (230), which are connected to a network (250). The DASH system (200) may also include one or more supplemental content servers.

[0034] The content server (210) can provide primary content (e.g., a main program) and a manifest file (e.g., an MPD) to the information processing device (230). The manifest file can be generated, for example, by an MPD generator (214). In other embodiments, the primary content and the manifest file can be provided by different servers.

[0035] The information processing device (230) can receive the MPD and retrieve primary content from the HTTP server (212) of the content server (210) based on the MPD. The MPD can be processed by a DASH client (232) running on the information processing device (230). The DASH client (232) can also retrieve advertising content from the advertising server (220) and other content (e.g., interactive content) from one or more supplemental content servers. The main content and advertising content can be processed by the DASH client (232) and output for display on a display device (236). The display device (236) can be integrated into the information processing device (230) or external to the information processing device (230). The DASH client (232) can also extract event information from one or more timed metadata tracks and send the extracted event information to an application (234) for further processing. The application (234) can be configured to, for example, display supplemental content based on the event information.

[0036] The advertisement server (220) can store advertisement content in an advertisement storage unit such as a memory, etc. The information processing device (230) can request the stored advertisement content based on the event information.

[0037] 3 illustrates an example DASH / CMAF client architecture for processing DASH and CMAF events, according to an embodiment of the present disclosure. A DASH / CMAF client (or DASH / CMAF player) communicates with an application (390) and can be configured to process various types of events, including (i) MPD events, (ii) in-band events, and (iii) timed metadata events.

[0038] The manifest parser (305) parses a manifest (e.g., an MPD). The manifest is provided, for example, by a content server (110, 210). The manifest parser (305) extracts event information about MPD events, in-band events, and timed metadata events embedded in timed metadata tracks. The extracted event information can be provided to the DASH logic (310) (e.g., the control, selection, and heuristic logic of a DASH player). Based on the event information, the DASH logic (310) can notify the application (390) of the event scheme signaled in the manifest.

[0039] The event information can include event scheme information for distinguishing between different event streams. An application (390) can use the event scheme information to subscribe to event schemes of interest. The application (390) can further indicate a desired delivery mode for each subscribed scheme via one or more subscription APIs. For example, the application (390) can send a subscription request to a DASH client that identifies one or more event schemes of interest and any desired corresponding delivery modes.

[0040] If an application (390) subscribes to one or more event schemes that are delivered as part of one or more timed metadata tracks, the inband event and "moof" parser (325) can stream one or more timed metadata tracks to the timed metadata track parser (330). For example, the inband event and "moof" parser (325) parses the movie fragment box ("moof") and subsequently parses the timed metadata tracks based on control information from the DASH logic (310).

[0041] The timed metadata track parser (330) can extract event messages embedded in the timed metadata tracks. The extracted event messages can be stored in an event buffer (335) (e.g., an event buffer). The synchronizer / dispatcher module (340) (e.g., an event and timed metadata synchronizer and dispatcher) can emit (or send) subscribed events to the application (390).

[0042] The MPD events described in the MPD can be parsed by the manifest parser (305) and stored in a buffer (335). For example, the manifest parser (305) parses each event stream element of the MPD and each event described in each event stream element. For each event signaled in the MPD, event information such as presentation time and event duration can be stored in a buffer (335) associated with the event.

[0043] The inband event and "moof" parser (325) can parse the media segments to extract inband event messages. Any such identified inband events and their associated presentation times and durations can be stored in a buffer (335).

[0044] Thus, the buffer (335) can store MPD events, in-band events, and / or timed metadata events therein. The buffer (335) can be, for example, a first-in, first-out (FIFO) buffer. The buffer (335) can be managed in correspondence with the media buffer (350). For example, as long as a media segment is present in the media buffer (350), any events or timed metadata corresponding to that media segment can be stored in the buffer (335).

[0045] The DASH Access Application Programming Interface (API) 315 can manage the fetching and reception of content streams (or data flows) containing media content and various metadata via the HTTP protocol stack 320. The DASH Access API 315 can separate the received content stream into different data flows. The data flows provided to the inband event and moof parser can include media segments, one or more timed metadata tracks, and inband event signaling contained in the media segments. In one embodiment, the data flows provided to the manifest parser 305 can include an MPD.

[0046] The DASH Access API (315) can forward the manifest to the manifest parser (305). In addition to describing events, the manifest can also provide information about media segments to the DASH logic (310), which can communicate with the application (390) and the inband event and moof parser (325). The application (390) can be associated with media content processed by the DASH client. Control / sync signals exchanged between the application (390), the DASH logic (310), the manifest parser (305), and the DASH Access API (315) can control the fetching of media segments from the HTTP stack (320) based on the information about the media segments described in the manifest.

[0047] The inband event and moof parser (325) can parse the media data flow into media segments containing media content, timed metadata in timed metadata tracks, and any signaled inband events within the media segments. The media segments containing the media content can be parsed by the file format parser (345) and stored in the media buffer (350).

[0048] The events stored in the buffer (335) allow the synchronizer / dispatcher (340) to communicate available events relevant to (or of interest to) the application via an event / metadata API. Applications can be configured to process available events (e.g., MPD events, in-band events, or timed metadata events) and subscribe to specific events or timed metadata by notifying the synchronizer / dispatcher (340). Any events stored in the buffer (335) that are not relevant to an application but instead are relevant to the DASH client itself can be forwarded by the synchronizer / dispatcher (340) to the DASH logic (310) for further processing.

[0049] In response to an application (390) subscribing to a particular event, the synchronizer / dispatcher (340) can communicate application event instances (or samples of timed metadata) corresponding to the event scheme to which the application subscribes. The event instances can be communicated according to a dispatch mode indicated by the subscription request or a default dispatch mode (e.g., for a particular event scheme). For example, in a dispatch-on-receive mode, event instances may be sent to the application (390) upon receipt in the buffer (335). Meanwhile, in a dispatch-on-start mode, event instances may be sent to the application (390) at their associated presentation times, for example, synchronized with a timing signal from the media decoder (355).

[0050] DASH / CMAF Addressable Resource Index In some example implementations, it is desirable for an adaptive streaming client (e.g., a DASH or CMAF client) to have precise knowledge of Addressable Resource Index (ARI) information, such as offset, size, duration, and quality, of timed aligned segments or chunks that reside within the same adaptation set / switching set. Along with such ARI information, relative information about, for example, immediately adjacent chunks or segments, may be used by the DASH / CMAF client to facilitate client heuristics. Addressable Resources may include track files, segments, and chunks in a CMAF context. For on-demand services, a precise map of such information may be provided by a segment index. Note that similar concepts and implementations may also be applied to DASH contexts.

[0051] In some example implementations, an Addressable Resource Index (ARI) may be defined as follows: Sample entry type: 'cari' Container: Sample Description Box ("STSD") Required: No Quantity: 0 or 1

[0052] The metadata describes all details of the addressable resources and, for example, a subset of the CMAF switching set defined in ISO / IEC 23000-19 in one index track.

[0053] Table 1 below shows an exemplary sample entry for CMAF Addressable Resource Index metadata.

[0054] [Table 1]

[0055] Table 2 below shows an example syntax for an ARI sample.

[0056] [Table 2]

[0057] For example, the semantics of the above syntax are explained below: · switching_set_identifier specifies the unique identifier of the switching set in the context of the application. ·num_tracks indicates the track number assigned in the ARI track. · track_ID: Uses the track_ID to select and order among the samples of a track. num_quality_indicators specifies the number of quality indicators used to identify the quality of the chunk. quality_identifier specifies the identifier that indicates how the quality values ​​in the sample are to be interpreted. This is a 4CC code that can be registered. · segment_start_flag indicates whether the chunk is the start of a segment. ·marker specifies whether this chunk contains at least one styp box. · SAP_type specifies the SAP type of the chunk. emsg_flag indicates whether this chunk provides at least one emsg box. · prft_flag indicates whether this chunk contains at least one prft box. offset specifies the offset of the chunk from the beginning of the segment. ·size gives the size of the chunk in octets. quality gives the quality of the chunk according to a given quality scheme identifier. The data type of the quality value (integer or floating point) is defined by the quality scheme. If the quality scheme identifier is an empty string, the quality is interpreted proportionally using unsigned integers, with increasing values ​​increasing the quality. ·loss indicates that the media data of the chunk has been lost. num_prediction_pairs provides how many pairs of expected predictions are provided. prediction_min_windows provides the minbuffer time value that is the same as the MPD value. predicted_max_bitrate provides a bandwidth value identical to the MPD semantic that is held for the duration of the prediction_min_windows value.

[0058] Transporting ARI using events In an exemplary implementation in the case of DASH / CMAF, a dedicated metadata track, i.e., an ARI track, is created to carry ARI-related information such as offset, size, quality, etc. of timed aligned segments or chunks that are in the same adaptation set / switching set, allowing the client to have relative information about the nearest chunks or segments to facilitate client heuristics, e.g., that can be used by the client when dynamically switching media tracks or representations.

[0059] Note that one of the disadvantages of using metadata tracks to carry ARI information (e.g., ARI samples) is the excessive signaling overhead: for example, an extra HTTP GET request is required by the client for each segment that requires ARI information.

[0060] Embodiments of the present disclosure include a method for carrying ARI (i.e., ARI information, ARI samples) without using an ARI metadata track. That is, rather than using a metadata track to carry ARI (which would incur an extra HTTP GET request (as the ARI sample is sent separately with the media segment / chunk)), the present disclosure may send ARI samples via events, such as in-band events or MPD events. This approach for carrying ARI samples is considered to be a "media segment / chunk associated ARI transmission" because the ARI sample is sent together with the media segment / chunk. An event carrying ARI is referred to as an ARI event. Using an ARI event can provide at least the following advantages:

[0061] 1. No extra metadata track is required, resulting in one less HTTP GET request by the CMAF / DASH client for each segment / chunk for which additional ARI information is needed. For example, a CMAF / DASH client may need additional ARI information to facilitate segment / chunk processing. In this case, the ARI information can be obtained directly from the ARI event carried with the segment / chunk.

[0062] 2. The event processing model allows for the processing of event messages and dispatching them to DASH / CMAF clients. The processing model allows for the timing of ARI samples to be carried as part of the event timing model.

[0063] 3. Flexibility: ARI information may be carried by events in one, some, or all of the representations in a DASH adaptation set or a CMAF switching set, depending on the in-band events required, for example.

[0064] 4. Adaptability and portability: ARI events can be parsed by a packager (eg, obtained from in-band events or ARI tracks received from an encoder) and added to the MPD as MPD events.

[0065] In some example implementations, the ARI information for a chunk / segment may be included in the same chunk / segment.

[0066] In some example implementations, the ARI information of a chunk / segment may be included in subsequent chunks / segments arranged in time.

[0067] In some example implementations, rather than using in-band events to carry ARI information, MPD events may be used to carry ARI information, which may be particularly well-suited for on-demand content.

[0068] In this embodiment, the ARI information may be carried in emsg boxes, each of which may belong to an event scheme defined by or associated with a scheme URN identifier.

[0069] For example, referring to Figure 4, ARI event 410 is carried with segment / chunk c(n) in media 1. The ARI information from ARI event 410 applies to the same segment / chunk c(n). If the current chunk ARI information is included in the current chunk event, the scheme URN identifier may be defined as "urn:mpeg:dash:event:ari:2022".

[0070] 4, an ARI event 412 is carried with segment / chunk c(n+1) in media2. The ARI information from the ARI event 412 applies to the next segment / chunk c(n+2). If the current chunk event contains the next chunk ARI information, the scheme URN identifier may be defined as "urn:mpeg:dash:event:ari-next:2022". For example, the emission mode of the event may be set on receipt.

[0071] Table 3 below shows example parameters for an ARI event in an MPD.

[0072] [Table 3]

[0073] As shown in Table 3, two elements, EventStream and InbandEventStream, may be used to describe an ARI event. Both streams may contain a value attribute. The value attribute may carry the CmafAriMetaDataSampleEntry fields described in Table 1. For example, a CmafAriMetaDataSampleEntry field may contain the following fields: switching_set_identifier num_tracks num_quality_indicators · A numbered list of track_ids List of quality_identifiers

[0074] In some example implementations, an Event element may include a presentationTime attribute (eg, Event@presentationTime), which indicates the chunk offset from the beginning of the period to which the event's ARI information applies.

[0075] In some example implementations, the Event element may include a duration attribute (e.g., Event@duration), which indicates the duration for which the ARI information should be used. For example, this may include the duration of a chunk or the duration of a segment.

[0076] In some example implementations, an event may include an event body, which may share the same structure as a CmafAriFormatStruct, as defined in Table 2.

[0077] Table 4 below shows an example of the emsg parameter for an inband ARI event.

[0078] [Table 4]

[0079] Note that the event body in an MPD event and the message_data in an inband event share the same CMAF ARI sample structure, CmafAriFormatStruct. Therefore, the parsing and processing of the ARI sample after receiving the event from the event dispatcher is the same. In other words, the same parsing and processing logic is shared for MPD events and inband events.

[0080] In some embodiments, ARI events may be processed and dispatched, for example, according to Section A.13 of ISO / IEC 23009-1, for example, in the exemplary DASH / CMAF client architecture shown in FIG.

[0081] In some embodiments, after an ARI event is dispatched, post-processing of the ARI event is performed, which may rely on the parameters shown in Table 5.

[0082] [Table 5]

[0083] FIG. 5 shows an example flow 500 for post-processing of an ARI event, including the following steps:

[0084] Step 510: The value field within the ARI event is parsed to extract general information about the event. As previously mentioned, the general information may include a switching_set_identifier, num_tracks, num_quality_indicators, an ordered list of track_ids, and a list of quality_identifiers.

[0085] Step 520: Use presentation_time to identify the chunk or segment to which the information applies.

[0086] Step 530: Parse the event payload (e.g., event body or message_data) to construct a CmafAriFormatStruct.

[0087] Step 540: Process the value and CmafAriFormatStruct with its heuristics to decide whether to switch to a new representation or stay with the same representation.

[0088] 6 illustrates an exemplary method 600 for processing a media stream. The media stream may include, for example, a 4G media stream (for a media stream distributed over a 4G network) or a 5G media stream (for a media stream distributed over a 5G network). The method may be implemented, for example, by a computer system described below. The media stream may conform to the DASH standard or the CMAF standard. The method 600 may include part or all of step 610 of processing an Addressable Resource Index (ARI) event associated with the media stream, where the ARI event includes at least one of an in-band event transmitted with a first media slice in a content set, the content set comprising one or more media slices, or a Media Presentation Description (MPD) event, and the ARI event carries configuration information of the one or more media slices in the content set. The media stream may be supported, for example, by a cloud computing network, a wireless network (e.g., a 4G Long-Term Evolution (LTE) network or a 5G New Radio (NR) network), or a Wi-Fi network. The media streams may include, but are not limited to, 4G media streams, 5G media streams, Wi-Fi media streams, and mixtures thereof.

[0089] In some example implementations, the inband events and the MPD events are each identified by or associated with a scheme identifier, which includes a scheme identifier Uniform Resource Identifier (URI).

[0090] In some example implementations, the configuration information of the one or more media slices in method 600 may include at least one of configuration information of a first media slice or configuration information of a second media slice following the first media slice.

[0091] Embodiments of the present disclosure apply to both DASH and CMAF. The content set in method 600 may include at least one of an adaptation set when the media stream complies with Dynamic Adaptive Streaming over Hypertext transfer protocol (DASH) or a switching set when the media stream complies with Common Media Application Format (CMAF), and the first media slice includes at least one of a media segment in a first representation of the adaptation set or a media chunk in a first track of the switching set.

[0092] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the method (or embodiment), the DASH client, and the CMAF client may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to the DASH and / or CMAF technologies / standards.

[0093] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 7 illustrates a computer system (1800) suitable for implementing certain embodiments of the disclosed subject matter.

[0094] Computer software can be encoded using any suitable machine or computer language that can be assembled, compiled, linked, or similar mechanisms to create code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., directly, or via interpretation, microcode execution, etc.

[0095] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0096] 7 for computer system (1800) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (1800).

[0097] The computer system (1800) may include certain human interface input devices that may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0098] The input human interface devices may include one or more of a keyboard (1801), a mouse (1802), a trackpad (1803), a touch screen (1810), a data glove (not shown), a joystick (1805), a microphone (1806), a scanner (1807), and a camera (1808) (only one of each is shown).

[0099] Additionally, the computer system (1800) may include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1810), data gloves (not shown), or joystick (1805), although there may also be haptic feedback devices that do not function as input devices), audio output devices (such as speakers (1809), headphones (not shown)), visual output devices (such as screens (1810) including CRT, LCD, plasma, or OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or output in more than three dimensions by means of stereoscopic graphic output, virtual reality glasses (not shown), holographic displays, or smoke tanks (not shown)), and printers (not shown).

[0100] The computer system (1800) may also include human-accessible storage devices and their associated media, such as optical media (1820) including CD / DVD ROM / RW (1821) using CD / DVD or similar media, thumb drives (1822), removable hard drives or solid state drives (1823), legacy magnetic media (not shown) such as tape and floppy disks, and specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles.

[0101] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0102] The computer system 1800 may also include an interface 1854 to one or more communications networks 1855. The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicular, industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet; cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc.; television wired or wireless wide-area digital networks including cable, satellite, and terrestrial television; and vehicular and industrial networks including CAN bus. Some networks require an external network interface adapter that attaches in a standard manner to some general-purpose data port or peripheral bus 1849 (e.g., a USB port on the computer system 1800); other networks are typically integrated into the core of the computer system 1800 by attaching to a system bus, as described below (e.g., an Ethernet interface for a PC computer system or a cellular network interface for a smartphone computer system). Any of these networks may be used by the computer system 1800 to communicate with other entities. Such communication may be one-way receive-only (e.g., television broadcast), one-way transmit-only (e.g., from the CANbus to a specific CANbus device), or bidirectional, e.g., to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces described above.

[0103] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (1840) of the computer system (1800).

[0104] The cores (1840) may include one or more central processing units (CPUs) (1841), graphics processing units (GPUs) (1842), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (1843), task-specific hardware accelerators (1844), graphics adapters (1850), etc. These devices may be connected via a system bus (1848), along with read-only memory (ROM) (1845), random access memory (1846), and internal mass storage (1847) such as internal hard drives and SSDs that are not user-accessible. In some computer systems, the system bus (1848) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1848) or via a peripheral bus (1849). In one example, a screen (1810) may be connected to the graphics adapter (1850). Architectures for peripheral buses include PCI, USB, etc.

[0105] The CPU (1841), GPU (1842), FPGA (1843), and accelerator (1844) can execute several instructions, which in combination can be constructed as the aforementioned computer code. This computer code can be stored in ROM (1845) or RAM (1846). Transient data can also be stored in RAM (1846), while permanent data can be stored, for example, in internal mass storage (1847). Fast storage and retrieval of any memory device can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU (1841), GPU (1842), mass storage (1847), ROM (1845), RAM (1846), etc.

[0106] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0107] As a non-limiting example, a computer system having architecture (1800), and in particular core (1840), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with mass storage directly operable by a user, as described above, or even specific storage of the core (1840), such as core internal mass storage (1847) or non-transitory storage, such as ROM (1845). Software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (1840). The computer-readable media can include one or more memory devices or chips, depending on particular needs. Software enables the cores (1840), and in particular the processors (including CPUs, GPUs, FPGAs, etc.) in the cores (1840), to execute the particular processes or particular portions of the particular processes described herein, including defining data structures stored in RAM (1846) and modifying such data structures in accordance with the processes defined by the software. Additionally or alternatively, logic may be hardwired into circuitry (e.g., accelerators (1844)) that can act in place of or cooperate with software to execute the particular processes or particular portions of the particular processes described herein, or the computer system may provide functionality as a result of logic otherwise embodied. Where appropriate, references to "software" may encompass logic, and vice versa. References to computer-readable media may encompass circuitry (e.g., integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0108] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0109] 100 systems 110 Content Server 120 Information processing equipment 200 DASH System 210 Content Server 212 HTTP Server 214 MPD Generator 220 Ad Server 230 Information processing equipment 232 DASH client 234 Applications 236 Display Devices 250 Network 305 Manifest Parser 310 DASH Logic 315 DASH Access API 320 HTTP protocol stack 325 inband events and the 'moof' parser 330 Timed Metadata Track Parser 335 Event Buffer 340 Synchronizer / Dispatcher Module 345 File Format Parser 350 Media Buffer 355 Media Decoder 390 Applications 410 ARI Event 412 ARI Event 500 Flow 600 ways 1800 Computer Systems and Architecture 1801 keyboard 1802 Mouse 1803 Trackpad 1805 Joystick 1806 Mike 1807 Scanner 1808 Camera 1809 Speaker 1810 touch screen 1820 Optical media 1821 CD / DVD ROM / RW 1822 thumb drive 1823 Removable Hard Drive or Solid State Drive 1840 Core 1841 Central Processing Unit (CPU) 1842 Graphics Processing Unit (GPU) 1843 Field Programmable Gate Area (FPGA) 1844 Hardware Accelerator 1845 Read-Only Memory (ROM) 1846 Random Access Memory (RAM) 1847 Internal Mass Storage 1848 System Bus 1849 General Purpose Data Port or Peripheral Bus 1850 graphics adapter 1854 Interface 1855 Communication Network

Claims

1. 1. A method for processing a 5G media stream, executed by one or more processors, wherein the 5G media stream complies with the Dynamic Adaptive Streaming over HTTP (DASH) standard or the Common Media Application Format (CMAF), the method comprising: Processing an Addressable Resource Index (ARI) event associated with the 5G media stream, The ARI event is an inband event sent with a first media slice in a content set, the content set including one or more media slices; or MPD (Media Presentation description) event and the ARI event conveys configuration information of the one or more media slices within the content set. A method comprising:

2. The method of claim 1 , wherein the in-band event and the MPD event are each identified by or associated with a scheme identifier, the scheme identifier comprising a scheme identifier Uniform Resource Identifier (URI).

3. The configuration information of the one or more media slices comprises: Configuration information of the first media slice; or Configuration information of a second media slice following the first media slice; The method of claim 1 , comprising at least one of:

4. The method of claim 3 , wherein a scheme identifier of the ARI event or the MPD event indicates whether the configuration information applies to the first media slice or the second media slice.

5. The content set: an adaptation set when the 5G media stream complies with the Dynamic Adaptive Streaming over Hypertext transfer protocol (DASH); or A switching set when the 5G media stream conforms to CMAF (Common Media Application Format); and The first media slice a segment in a first representation of said adaptation set; or a chunk in a first track of said switching set; including at least one of The method of claim 1.

6. The configuration information of the one or more media slices comprises: an offset of each of the one or more media slices relative to a start time of a container including the content set, the container including at least one of a CMAF presentation or a DASH period; the size of each of said one or more media slices; or a quality of each of the one or more media slices; The method of claim 5, comprising at least one of:

7. 6. The method of claim 5, wherein the ARI event carries timing information associated with at least one chunk in the switching set or at least one segment in the adaptation set, and the timing information indicates a start time of each of the at least one chunk or each of the at least one segment.

8. determining whether to switch to a representation different from the first representation or a track different from the first track based on the configuration information of the one or more media slices in the content set; 6. The method of claim 5, further comprising:

9. determining whether to switch to a representation different from the first representation or a track different from the first track based on the configuration information of the one or more media slices in the content set and information extracted from the first media slice; 6. The method of claim 5, further comprising:

10. 6. The method of claim 5, further comprising receiving one or more additional ARI events from a representation different from the first representation or a track different from the first track.

11. The ARI event includes a message portion, the message portion comprising: a message_data field when the ARI event is an inband event, or an event body field when the ARI event is an MPD event; It is one of the the message_data field and the event body field each carry one or more ARI samples; the one or more ARI samples in the message_data field and the one or more ARI samples in the event body field share the same data structure; The method of claim 5.

12. parsing the ARI event to obtain a presentation time associated with the ARI event; determining a target media slice to which the ARI event applies based on the presentation time; constructing the one or more ARI samples based on the data structure; determining whether to switch to a representation different from the first representation or a track different from the first track based on the one or more ARI samples; 12. The method of claim 11, further comprising:

13. A device for processing 5G media streams, configured to perform the method according to any one of claims 1 to 12.

14. A computer program for causing a computer to carry out the method according to any one of claims 1 to 12.

Images