Method, device and storage medium for processing a media stream

Abstract

Methods, devices, and computer-readable storage media for processing media streams. A method can include receiving a DASH event associated with a DASH media stream; determining that the DASH event is an update to a previously received DASH event; and processing the DASH event based on a scheduling mode of the DASH event; wherein the DASH event comprises at least one of: an in-band event transmitted with a first media segment in a content set, the content set comprising one or more media segments; a media presentation description (MPD) event; or a timed metadata sample; wherein the scheduling mode of the DASH event comprises a reception mode and a launch mode.

Description

[0001] By incorporating via reference

[0002] This application is based on and claims priority to U.S. Non-Provisional Application No. 18 / 301,596, filed April 17, 2023, and U.S. Provisional Application No. 63 / 332,596, filed April 19, 2022, which is based on and claims priority to U.S. Provisional Application No. 63 / 332,599, filed April 19, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to media streaming technologies, including Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH). More specifically, this disclosure relates to methods, apparatus, and computer-readable storage media for processing DASH media streams. Background Technology

[0004] The background description provided herein is for the purpose of presenting the overall context of this disclosure. To the extent that the work described in this background section is intended, neither the work of the currently identified inventors nor any aspects of the description which may not be otherwise defined as prior art at the time of filing are expressly or implicitly acknowledged as prior art to this disclosure.

[0005] The Moving Picture Experts Group (MPEG) provides a standard for streaming multimedia content over IP networks using Dynamic Adaptive Streaming (DASH) based on the Hypertext Transfer Protocol. In the DASH standard, Media Presentation Descriptions (MPDs) are used to provide information to DASH clients to adaptively stream media content by downloading media segments from the DASH server. The DASH standard allows for streaming of multi-rate content. One aspect of the DASH standard includes the carrying of MPD events and in-band events, and a client-side processing model for handling these events. Summary of the Invention

[0006] This disclosure provides methods and apparatus for media stream processing, and more specifically, methods and apparatus for processing DASH event updates under a DASH client processing model. In some example implementations, a method for processing media streams, such as DASH media streams, is disclosed. This method may include: receiving a DASH event associated with a DASH media stream; determining that the DASH event is an update to a previously received DASH event; and processing the DASH event based on a scheduling mode of the DASH event; wherein the DASH event includes at least one of the following: an in-band event transmitted with a first media slice in a content set, the content set including one or more media slices; a Media Presentation Description (MPD) event; or a timing metadata sample; wherein the scheduling mode of the DASH event includes a receive mode and a launch mode.

[0007] Various aspects of this disclosure also provide an apparatus for processing media streams, such as DASH media streams. The apparatus may include a memory for storing computer instructions and a processor for communicating with the memory. When the processor executes the computer instructions, the processor is configured to cause the apparatus to perform the methods described above for processing DASH media streams.

[0008] This disclosure also provides a non-transitory computer-readable medium storing instructions that, when executed by a computer for media streaming processing, cause the computer to perform the methods described above for processing DASH media streams. Attached Figure Description

[0009] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0010] Figure 1 A system according to an embodiment of this disclosure is shown.

[0011] Figure 2 An HTTP-based Dynamic Adaptive Streaming (DASH) system according to an embodiment of this disclosure is shown.

[0012] Figure 3 A DASH client architecture according to an embodiment of this disclosure is shown.

[0013] Figure 4 Example activity event tables and scheduling event tables are shown.

[0014] Figure 5a An example common process for updating events is shown in the DASH event handling model.

[0015] Figure 5bAn example of receiving and processing an event for updating is shown in the DASH event handling model.

[0016] Figure 5c An example initiation process for an event used for updating is shown in the DASH event handling model.

[0017] Figure 6 A flowchart illustrating an example implementation of a method according to this disclosure is shown.

[0018] Figure 7 A schematic illustration of a computer system according to an example embodiment of the present disclosure is shown. Detailed Implementation

[0019] Dynamic Adaptive Streaming (DASH) and Media Presentation Description (MPD) based on Hypertext Transfer Protocol

[0020] A popular format for media streaming includes Dynamic Adaptive Streaming (DASH) based on the Hypertext Transfer Protocol, as defined in ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission) 23009-1. DASH is an adaptive bitrate streaming technology that enables the streaming of media content using Hypertext Transfer Protocol (HTTP) infrastructure such as web servers, Content Delivery Networks (CDNs), various proxies, and caches. DASH supports both on-demand and live streaming from DASH servers to DASH clients and allows DASH clients to control the streaming session, freeing DASH servers from the additional load of streaming adaptation management in large-scale deployments. DASH also allows DASH clients to select streaming from various DASH servers, thus further achieving network load balancing for the benefit of DASH clients. DASH provides dynamic switching between different media tracks, for example, by changing the bitrate to adapt to network conditions.

[0021] In DASH, Media Presentation Description (MPD) files provide DASH clients with information to adaptively stream media content by downloading media segments from the DASH server. MPDs can be in the form of Extensible Markup Language (XML) documents. MPD files can be fragmented and delivered in batches to reduce session initiation latency. MPD files can also be updated during a streaming session. In some examples, MPD files support expressions for content accessibility features, ratings, and camera device views. DASH also supports the delivery of multi-view and scalable encoded content.

[0022] An MPD file can contain a sequence of one or more time slots. Each of the one or more time slots can be defined by, for example, a time slot element in the MPD file. The MPD file can include the MPD's availableStartTime attribute and a start attribute for each time slot. For media presentations with a dynamic type (e.g., for live streaming services), the sum of the time slot's start attribute and the MPD's availableStartTime attribute, along with the duration of the media segment, can indicate the available time of that time slot in Coordinated Universal Time (UTC) format, specifically the first media segment of each presentation within the corresponding time slot. For media presentations with a static type (e.g., for video-on-demand services), the start attribute of the first time slot can be 0. For any other time slot, the start attribute can specify the time offset between the start time of the corresponding time slot and the start time of the first time slot. Each time slot can extend until the start of the next time slot, or, in the case of the last time slot, until the end of the media presentation. The time slot start time can be precise and reflects the actual time generated by playing media from all previous time slots. In the example implementation, the MPD is provided such that the next time slot is a continuation of the content in the previous time slot; the next time slot may be the immediately following time slot or a later time slot (e.g., after an inserted advertising time slot).

[0023] Each time slot may contain one or more adaptive sets, and each of the adaptive sets may contain one or more presentations of the same media content. A presentation may be one of several alternative coded versions of audio or video data. Presentations may vary depending on the encoding type, such as the bitrate, resolution, and / or codec of the video data, and the bitrate and / or codec of the audio data. The term presentation may be used to refer to a portion of encoded audio or video data that corresponds to a specific time slot of multimedia content and is encoded in a specific manner.

[0024] Adaptive sets for a specific time period can be assigned to groups indicated by group attributes in the MPD file. Adaptive sets within the same group are generally considered to be alternatives to each other. For example, each adaptive set of video data for a specific time period can be assigned to the same group, allowing any adaptive set to be selected for decoding to display the video data containing multimedia content for that time period. In some examples, the media content within a time period can be represented by any adaptive set from group 0 (if present) or a combination of at most one adaptive set from each non-zero group. The timing data for each presentation of a time period can be represented relative to the start time of that time period.

[0025] A presentation may include one or more fragments. Each presentation may include an initialization fragment, or each fragment of a presentation may be self-initialized. When present, the initialization fragment may contain initialization information for accessing the presentation. In some cases, the initialization fragment does not contain media data. Fragments may be uniquely referenced by identifiers such as Uniform Resource Locators (URLs), Uniform Resource Names (URNs), or Uniform Resource Identifiers (URIs).

[0026] In the example implementation, according to IETF (Internet Engineering Task Force) RFC (Request For Comments) 3986, the URL can be limited to... <absolute-uri>For example, in a fixed scheme with "http" or "https", the URL may be limited to a byte range if the range attribute is provided along with the URL. For instance, a byte range can be represented as a byte-range-spec as defined in IETF RFC 2616. A byte range can be limited to a single expression identifying a range of consecutive bytes. In implementations, fragments can be included in the MPD along with the data URL, as defined in IETF RFC 2397, for example.

[0027] MPD files can provide an identifier for each fragment. In some examples, MPD files can also provide byte ranges as a range attribute, which can correspond to data within fragments of a file accessible via a URL, URN, or URI.

[0028] Sub-representations can be embedded (or contained) within regular representations and are described by sub-representation elements (e.g., SubRepresentation). Sub-representation elements can describe the properties of one or more media content components embedded within the representation. For example, a sub-representation element can describe the properties of embedded audio components (e.g., codec, sample rate, etc.), embedded subheadings (e.g., codec), or embedded lower-quality video layers (e.g., lower frame rates, etc.). Sub-representation elements and representation elements can share some common properties and elements.

[0029] Each presentation may also include one or more media components, where each media component may correspond to an encoded version of a single media type, such as audio, video, or timed text (e.g., for closed captions). Media components may be temporally continuous across the boundaries of consecutive media segments within a presentation.

[0030] In some example implementations, the DASH client can access and download MPD files from the DASH server. That is, the DASH client can retrieve MPD files for use when initiating a live session. Based on the MPD files, and for each selected presentation, the DASH client can make several decisions, including: determining what the latest available segment is on the server; determining the available start time for the next segment and possible future segments; determining when to start playing the segment and from which timeline within the segment; and determining when to obtain / retrieve a new MPD file. Once the service is broadcast, the client can maintain a sense of discrepancy between its own broadcast and the current broadcast, which needs to be detected and compensated for.

[0031] DASH incident

[0032] In the DASH system, events provide a means of signaling additional information to DASH clients and their associated applications. In the example implementation, events are timed and therefore have a start time and duration. Event information may include metadata describing the content presented by the media. Additionally or alternatively, event information may include media player control messages, such as ad insertion prompts, associated with a specific time during playback of the media presentation. Events can be implemented as, for example, MPD events or in-band events. Events may be part of a manifest file (e.g., MPD) or may be embedded in an ISOBMFF (ISO Base Media File Format, ISOBMFF)-based media file, such as an event message (emsg) box.

[0033] Media Presentation Description (MPD) events are events that can be signaled within an MPD. An MPD can provide a sequence of events assigned to media presentation times at the time-segment level. Events of the same type can be specified by event stream elements (e.g., EventStream) within a time-segment element. Events terminate at the end of the time-segment, even if the start time is after the time-segment boundary or the event's duration extends beyond the time-segment boundary. Event stream elements include message scheme identification information (e.g., @schemeIdUri) and optional values ​​for the event stream element (e.g., @value). Furthermore, because event streams contain timed events, timescale attributes (e.g., @timescale) can be provided to assign events to specific media presentation times within the time-segment. Timed events themselves can be described by event elements included within the event stream element.

[0034] In-band event streams can be reused with presentations by adding event messages as part of media segments. Event streams can exist in selected presentations, only in one or more selected adaptive sets, or in all presentations. For example, one possible configuration is where only the audio adaptive set contains in-band events or only the video adaptive set contains in-band events. In-band event streams existing in a presentation can be indicated by in-band event stream elements (e.g., InbandEventStream) at various levels, such as the adaptive set level or the presentation level. Furthermore, a presentation can contain multiple in-band event streams, each indicated by a separate in-band event stream element.

[0035] Figure 1 A system (100) according to an embodiment of the present disclosure is shown. The system (100) includes a content server (110) and an information processing device (120). The content server (110) can provide a content stream, including main content (e.g., a main program) and one or more timed metadata tracks.

[0036] The information processing device (120) can interface with the content server (110). For example, the information processing device (120) can play back content received from the content server (110). The playback of the content can be performed based on a manifest file (e.g., MPD) received by the information processing device (120) (e.g., from the content server (110)). The manifest file may also include signaling for one or more timed metadata tracks.

[0037] Figure 2 An exemplary DASH system is shown. The DASH system (200) may include a content server (210), an advertising server (220), and an information processing device (230), which are connected to a network (250). The DASH system (200) may also include one or more supplementary content servers.

[0038] The content server (210) can provide the information processing device (230) with main content (e.g., the main program) and manifest files (e.g., MPDs). For example, the manifest file can be generated by an MPD generator (214). In other embodiments, the main content and the manifest file can be provided by different servers.

[0039] The information processing device (230) receives the MPD and can retrieve the main content from the HTTP server (212) of the content server (210) based on the MPD. The MPD can be processed by a DASH client (232) running on the information processing device (230). Furthermore, the DASH client (232) can retrieve advertising content from the advertising server (220) or other content (e.g., interactive content) from one or more supplementary content servers. The main content and advertising content can be processed by the DASH client (232) and output for display on a display device (236). The display device (236) can be integrated into the information processing device (230) or external to the information processing device (230). Additionally, the DASH client (232) can extract event information from one or more timed metadata tracks and send the extracted event information to an application (234) for further processing. The application (234) can be configured to display supplementary content, for example, based on the event information.

[0040] The advertising server (220) can store advertising content in an advertising storage device, such as a memory. The information processing device (230) can request the stored advertising content based on event information.

[0041] Figure 3 An example DASH client architecture for handling DASH events according to an embodiment of this disclosure is shown. The DASH client (or DASH player) can be configured to communicate with the application (390) and handle various types of events, including (i) MPD events, (ii) in-band events, and (iii) timing metadata events.

[0042] A manifest parser (305) parses a manifest (e.g., an MPD). For example, the manifest is provided by a content server (110, 210). The manifest parser (305) extracts event information about MPD events, in-band events, and timing metadata events embedded in the timing metadata track. The extracted event information can be provided to DASH logic (310) (e.g., DASH player control, selection, and heuristic logic). The DASH logic (310) can then signal the event scheme in the manifest to the application (390) based on the event information.

[0043] Event information may include event scheme information used to distinguish between different event streams. An application (390) can use the event scheme information to subscribe to event schemes of interest. The application (390) can also indicate the desired scheduling mode for each of the subscribed schemes via one or more subscription APIs (Application Programming Interfaces). For example, the application (390) can send a subscription request to a DASH client identifying one or more event schemes of interest and any desired corresponding scheduling modes.

[0044] If an application (390) subscribes to one or more event schemes delivered as part of one or more timing metadata tracks, the in-band event and "moof" parser (325) can stream one or more timing metadata tracks to the timing metadata track parser (330). For example, the in-band event and "moof" parser (325) parses movie clip frames ("moof") and then parses timing metadata tracks based on control information from the DASH logic (310).

[0045] A timing metadata track parser (330) can extract event messages embedded in the timing metadata track. The extracted event messages can be stored in an event and timing metadata buffer (335). A synchronizer / scheduler module (340) (e.g., an event and timing metadata synchronizer and scheduler) can schedule (or send) subscribed events to the application (390).

[0046] The MPD events described in the MPD can be parsed by the list parser (305) and stored in the buffer (335). For example, the list parser (305) parses each event stream element of the MPD and parses each event described in each event stream element. For each event in the MPD that is signaled, event information such as presentation time and event duration can be stored in the buffer (335) associated with said event.

[0047] The in-band event and "moof" parser (325) can parse media segments to extract in-band event messages. Any such identified in-band event, along with its associated rendering time and duration, can be stored in a buffer (335).

[0048] Therefore, the buffer (335) can store MPD events, in-band events, and / or timing metadata events. For example, the buffer (335) can be a First-In-First-Out (FIFO) buffer. The buffer (335) can be managed in correspondence with the media buffer (350). For example, as long as a media segment exists in the media buffer (350), any event or timing metadata corresponding to that media segment can be stored in the buffer (335).

[0049] The DASH Access Application Programming Interface (API) (315) can manage the acquisition and reception of content streams (or data streams) including media content and various metadata via the HTTP (Hypertext Transfer Protocol, HTTP) protocol stack (320). The DASH Access API (315) can segment the received content stream into different data streams. The data streams provided to the in-band event and moof parsers may include media segments, one or more timing metadata tracks, and in-band event signaling included within the media segments. In an implementation, the data stream provided to the manifest parser 305 may include MPDs.

[0050] The DASH Access API (315) can forward the manifest to the manifest parser (305). In addition to describing events, the manifest can also provide information about media segments to the DASH logic (310), which can communicate with the application (390) and the in-band event and moof parser (325). The application (390) can associate media content processed by the DASH client. Control / synchronization signals exchanged between the application (390), the DASH logic (310), the manifest parser (305), and the DASH Access API (315) can control the retrieval of media segments from the HTTP stack (320) based on information about the media segments provided in the manifest.

[0051] The in-band event and moof parser (325) can parse the media data stream into media segments that include media content, timing metadata in the timing metadata track, and any in-band events notified by signals within the media segments. The media segments that include media content can be parsed by the file format parser (345) and stored in the media buffer (350).

[0052] Events stored in the buffer (335) allow the synchronizer / scheduler (340) to transmit available events (or events of interest) related to the application to the application via the event / metadata API. The application can be configured to handle available events (e.g., MPD events, in-band events, or timing metadata events) and subscribe to specific events or timing metadata by notifying the synchronizer / scheduler (340). Any application-independent events stored in the buffer (335) that are related to the DASH client itself can be forwarded by the synchronizer / scheduler (340) to the DASH logic (310) for further processing.

[0053] In response to an application (390) subscribing to a specific event, the synchronizer / scheduler (340) can send an event instance (or timing metadata sample) corresponding to the event scheme that the application has subscribed to. The event instance can be sent according to the scheduling mode indicated by the subscription request (e.g., for a specific event scheme) or the default scheduling mode. For example, in the on-receive scheduling mode, the event instance can be sent to the application (390) when it is received in the buffer (335). On the other hand, in the on-start scheduling mode, the event instance can be sent to the application (390) at a presentation time associated with the event instance, for example, synchronized with a timing signal from the media decoder (355).

[0054] In some example implementations, the DASH player processes the received MPD. For each time period, the MPD may include one or more event streams. Each event stream may have a range determined by a scheme / value. Each time period may include one or more adaptive sets carrying the rendering / tracking of timing metadata. Some or all of these event streams / timing metadata tracks may be suitable for the application's use.

[0055] Events specific to the DASH player are dispatched to the DASH player's control, selection, and heuristic logic. Figure 3 (DASH logic 310), while application-related event and timing metadata track samples are scheduled to the application, as described below. If the application subscribes to a specific event stream or timing metadata stream, the corresponding event instance or timing metadata sample can be scheduled to the application according to the scheduling mode:

[0056] For receive scheduling mode, the DASH client (or DASH player) should schedule the entire event or timing metadata information at or before the Latest Arrival Time (LAT), or as early as possible. The receiving application can prepare for the event from the event's LAT to its Start Time (ST). In some example implementations, the LAT is the latest time the event should be delivered to the application and is used to ensure the application has sufficient time to prepare for the event. In some example implementations, the DASH client can schedule the event or timing metadata information once it has been appended to the event and timing metadata buffers.

[0057] In start-up scheduling mode, the DASH client (or DASH player) should schedule the event exactly at the ST, which is the start / presentation time of the event / metadata sample. The DASH client (or DASH player) should schedule the event to the application at the presentation time of the corresponding media sample. If the start time of the event has passed, but the current time is within the event duration, the DASH client can schedule the event at the earliest time within the event duration.

[0058] exist Figure 3 In this configuration, the file format parser 345, media buffer 350, and media decoder 355 can work together to form an MSE buffer.

[0059] MPD incident

[0060] Events can be signaled in the MPD. A sequence of events assigned to media presentation times can be provided in the MPD at the time-period level. Events of the same type can be aggregated into an event stream specified by the EventStream element in the time-period element. For example, an event can terminate at the end of a time-period, even if the start time is after a time-period boundary or the event's duration extends beyond the time-period boundary.

[0061] In some example implementations, all events of the same type can be aggregated into a single event stream within a time period. A time period can have multiple event streams, and each event stream can have different combinations of the following: the value of the @schemeIdUri attribute and the value of the @value attribute.

[0062] Table 1 provides example event flow semantics.

[0063] Table 1: Event Flow Semantics

[0064]

[0065]

[0066] Table 2 provides example event semantics for implementations according to this disclosure.

[0067] Table 2: Event Semantics

[0068]

[0069]

[0070] In-band event signaling and event message boxes

[0071] In some example implementations, event streams can be reused with the rendering by adding event messages as part of a fragment. Event streams can exist in selected renderings, only in one or more selected adaptive sets, or in all renderings.

[0072] In some example implementations, the in-band event stream can exist within the rendering. If it is expected to be handled by the DASH client, the in-band event stream can be indicated by an InbandEventStream element at the adaptive set level or the rendering level.

[0073] InbandEventStream can be limited based on the semantics of EventStream, as shown in Table 1. Additional restrictions can also be added to specific semantics of InbandEventStream.

[0074] Event message boxes ('emsg') can be used to provide signaling for general events related to, for example, media presentation time. Semantics identical to those defined in Tables 1 and 2 can also be applied.

[0075] If based on an ISO BMFF container, a media clip can contain one or more event message ('emsg') boxes. If present, the 'emsg' boxes can be positioned as follows:

[0076] 'emsg' can be placed before the first 'moof' box of the clip; or

[0077] 'emsg' can be placed between any 'mdat' (media data) and 'moof' boxes. In this case, an equivalent 'emsg' with the same id value should exist before the first 'moof' box of any fragment.

[0078] In some example implementations, 'emsg' can be limited as follows:

[0079] Box type: 'emsg'

[0080] Container: Fragment

[0081] Mandatory: No

[0082] Quantity: zero or more

[0083] Table 3 shows an example 'emsg' syntax.

[0084]

[0085] For example, the semantics of the above syntax are described below:

[0086] `scheme_id_uri` is a string that identifies the message scheme. The semantics and syntax of `message_data[]` are qualified by, for example, the owner of the scheme being identified. The string can use either URN or URL syntax. A URL can be resolved to an Internet location, and the location that is resolved to can store the specification of the message scheme.

[0087] `value` is a string that specifies the value for the event. The value space and semantics can be limited by the owner of the scheme identified in the `scheme_id_uri` field.

[0088] `timescale` provides a time scale (in ticks per second) for the event duration and the `presentation_time_delta` or `presentation_time` field. This value should be the same as the time scale of the tracks contained in the carry segment. In some implementations, this value should be the same for all events in an event stream.

[0089] The `presentation_time_delta` parameter provides the difference between the media presentation time of the event and the earliest presentation time within the segment. If a segment index exists, the earliest presentation time is determined by the `earliest_presentation_time` field in the first 'sidx' box. If no segment index exists, the earliest presentation time is determined to be the earliest presentation time of any access unit within the media segment. The timescale is provided in the `timescale` field.

[0090] The presentation_time provides the media presentation time of an event measured on the movie timeline, in units of the timescale field provided, and is adjusted by InbandEventStream@presentationTimeOffset in units of the timescale provided by InbandEventStream@timescale; this value should not be less than the earliest presentation time of the carrying segment.

[0091] `event_duration` provides the duration of the event during media rendering. The timescale is indicated in the `timescale` field. In some example implementations, a value of 0xFFFFFFFF indicates an unknown duration. The interpretation of this value must be determined by the owner of the event scheme.

[0092] `id`: A field that identifies the instance of this message. The scope of this identifier for each event is the same as the scope of the `scheme_id_uri` and `value` pairs. Within the same `scheme_id_uri` and `value` pair scope, messages with the same `id` are equivalent; that is, it is sufficient to process any event message box with the same `id`.

[0093] message_data: The message body, which fills the remainder of the message box. Depending on the information above, message_data may be empty. The syntax and semantics of this field must be qualified by the owner of the scheme identified in the scheme_id_uri field.

[0094] `flags` can be variables with multiple bit fields (e.g., a character with 8 bits). Each bit can be used to indicate the status, condition, or type of the esmg. In the example implementation, `(flags & 1)` equals 1 to indicate that the esmg is an update of another esmg with the same values ​​for the `scheme_id_uri`, `value`, and `id` fields. The `&` operator is the bitwise OR operator. Additional bits in `flags` can also be selected to indicate that the esmg is an update.

[0095] In a DASH system, in addition to signaling events, the ability to dynamically update earlier signaled events would be beneficial. For example, the start time, duration, and / or media content specified by the event could be updated. In some scenarios, there are breaking news events that can be signaled. News may sometimes need to be updated based on the latest developments. Therefore, it may be necessary to update the event accordingly. In some scenarios, it may be necessary to remove scheduled events due to, for example, scheduling updates or conflicts.

[0096] In current DASH systems, update events are not explicitly defined. Because DASH clients can be implemented by many different vendors on many different platforms (e.g., operating systems, browsers, etc.), cross-vendor or cross-platform compatibility is not possible. Update events notified by signals may not be identifiable or recognizable by a specific DASH client. Furthermore, the behavior used to handle update events is not defined, leading to further differences in how vendor-dependent DASH client implementations behave.

[0097] This disclosure describes various implementations for precisely defining and identifying update events. Regarding update event support, several implementations related to the DASH client processing model are described, including: monitoring and identifying update events, update event buffers, and scheduling update events only if they have not been scheduled before a previous event.

[0098] In some example implementations, MPD update events are defined. An event with `@status='update'` is an updated instance of an event that may have been previously processed by the DASH client and has the same `@schemeIdUri`, `@value`, and `@id` attributes. If the previous event has not yet been scheduled, the DASH client can replace the previous event with the updated instance. An event with `@status='update'` can differ from the previous event except for the following attributes: `@schemeIdUri`, `@value`, and `@id`.

[0099] In some example implementations, in-band update events are defined. An emsg box with (flags &1) = 1 (& is the bitwise AND operator) is an updated instance of an emsg box that may have been previously processed by the DASH client and has the same scheme_id_uri, value, and id fields. If the previous event has not yet been scheduled, the DASH client can replace the previous event with the updated instance. The updated emsg can differ from the previous emsg except for the following fields: scheme_id_uri, value, and id.

[0100] In some example implementations, when an event is replaced, the old copy of the event stored in the event and timing metadata buffer can be updated using the updated instance of the event.

[0101] In some example implementations, when replacing an event, the render time of the updated event (which can be updated) can be checked. Updated events can be added to the event and timing metadata buffers based on their render time, and older events can be removed from the event and timing metadata buffers. Note that because the render time may be updated, updated events may have different positions in the event and timing metadata buffers.

[0102] In this disclosure, the DASH client event handling model is improved to support updated events. For example, when one or more updated versions of an event are received, the DASH client only schedules the latest event update and can avoid scheduling older / obsolete events.

[0103] As described above, the event handling model (also referred to as the handling model below) can handle events in two scheduling modes: on_receive and on_start. The handling model will share common processing applicable to both scheduling modes, and then use separate processing for each scheduling mode.

[0104] Reference Figure 3 Application 390 subscribes to a specific event stream identified by a (scheme_uri / value) pair under a specific scheduling mode (start or receive). For example, the scheme can be indicated by the @schemeIdUri attribute (or field) or the scheme_id_uri field of the EventStream qualified as 'emsg'. Common processing and scheduling mode-dependent processing are described below.

[0105] In some example implementations, EventStream is used as a container for events of the same type. EventStream can be identified by (scheme ID / value) pairs or by scheme ID alone. Scheme ID can include, for example, scheme_id_uri.

[0106] Public processing

[0107] In some example implementations, the DASH client can build an active event table for each subscribed EventStream identified by, for example, (scheme_uri / value). Exemplarily, the active event table may only need to be set for events in the initial scheduling mode. The active event table maintains a single list of event IDs waiting to be scheduled. Figure 4 An example activity event table setup is shown. Figure 4 In this context, k activity event tables are created, where each table corresponds to a unique (scheme_uri / value) pair. Each table can be implemented as, for example, a list. Each list can hold a list of event identifiers (e.g., Event@id, 'emsg' id) indicating events waiting to be scheduled.

[0108] In some example implementations, the DASH client can also set up a scheduling event table for each subscribed EventStream identified by, for example, a (scheme_uri / value) pair. The scheduling event table maintains a single list of event IDs for events that have been scheduled. Figure 4 An example scheduling event table setup is shown. Figure 4 In this context, x scheduling event tables are created, where each table corresponds to a unique (scheme_uri / value) pair. Each table can be implemented as, for example, a list. Each list can maintain a list of event identifiers (e.g., Event@id, 'emsg' id) that identify events that have been scheduled.

[0109] The DASH client parses events (e.g., 'emsg' / timed metadata samples) and retrieves scheme_uri / (value).

[0110] If the application is not subscribed to the scheme_uri / (value) pair, the DASH client can terminate processing of the event. The DASH client then proceeds to derive / retrieve the start time (ST, or rendering time) of the event instance / metadata sample and uses the equation ET = ST + DU to derive the end time (End Time, ET) of the event instance / metadata sample, where DU is the event duration that can be signaled in the event.

[0111] Figure 5a Example common processing logic is shown.

[0112] Receive processing

[0113] In some example implementations, when the scheduling mode is receive, the DASH client can perform the following processing.

[0114] Step 1: If the current presentation time value is greater than the ET of the update event, then end the process.

[0115] Step 2: Check whether the newly received update event has been scheduled. The DASH client can compare the event's ID with the entry in the scheduled event table identified by the same scheme_uri / (value) as the update event. If an entry with the same ID value exists, the process ends.

[0116] Step 3: Schedule the event / scheduled metadata (i.e., the updated version in the update event) – including ST, id, DU, timescale, and message_data, and add the event to the corresponding scheduled event table identified by the same scheme_uri / (value) pair as the update event.

[0117] Figure 5b An example of the receive processing logic is shown.

[0118] Startup Processing

[0119] In some example implementations, when the scheduling mode is on, the DASH client can perform the following example processing.

[0120] Step 1: If the event is an update, remove the expired event (which will be replaced by the update event) with the same id from the corresponding active event table identified by the same scheme_uri / (value) pair as the update event (if it exists).

[0121] Step 2: Obtain / retrieve the ST of the event instance / metadata sample.

[0122] Step 3: If the current media presentation time value is less than ST, proceed to step 6.

[0123] Step 4: Determine the event end time: ET = ST + DU.

[0124] Step 5: If the current presentation time value is greater than ET, then end the process.

[0125] Step 6: Compare the event ID with the entries in the corresponding active event table and the corresponding scheduled event table (identified by the same scheme_uri / (value) pairs as the update event):

[0126] If an entry with the same event ID value exists in either table, the process ends; or if an entry with the same event ID value exists in the corresponding scheduled event table, the process ends.

[0127] Otherwise, add the event ID (e.g., Event@id, 'emsg' id) that identifies the update event instance / metadata sample to the corresponding active event table. In some example implementations, the update event can be added to the event and time metadata buffer, and expired events can be removed from the same buffer.

[0128] Step 7: At time ST, or if the current presentation time is greater than ST, immediately schedule the updated event / metadata (e.g., message_data).

[0129] Once an update event is scheduled, the DASH client can add the update event (e.g., Event@id, 'emsg' id) to the corresponding scheduled event table.

[0130] Optionally, the DASH client can also check whether an entry for an update event exists in the corresponding active event table. If an entry exists, the DASH client can delete that entry from the corresponding active event table.

[0131] Figure 5c An example startup processing logic is shown.

[0132] The steps described above for public processing, receiving processing, and initiating processing are for illustrative purposes only. Other implementations may include, for example, subsets of these steps.

[0133] In this disclosure, the terms "field" or "attribute" may be used interchangeably for in-band events, MPD events, and timing metadata samples. Unless otherwise stated, an event generally refers to an MPD event, an in-band event, and a timing metadata sample.

[0134] In some example implementations, in-band events and MPD events are identified by or associated with a scheme identifier, which may include, for example, a scheme identifier Uniform Resource Identifier (URI).

[0135] The implementation methods described in this disclosure are applicable to DASH and other media streaming technologies / standards. Figure 6 An exemplary method 600 for processing media streams, such as DASH media streams, is shown. This method can be performed by a DASH client hosted by a media streaming device in a media streaming system, and method 600 may include some or all of the following steps: step 610: receiving a DASH event associated with the DASH media stream; step 620: determining that the DASH event is an update to a previously received DASH event; and step 630: processing the DASH event based on a scheduling mode of the DASH event; wherein the DASH event includes at least one of the following: an in-band event transmitted with a first media slice in a content set, the content set including one or more media slices; a Media Presentation Description (MPD) event; or a timing metadata sample; wherein the scheduling mode of the DASH event includes a receive mode and a launch mode.

[0136] The embodiments described in this disclosure can be used individually or in any combination in any order. Furthermore, each of the methods (or embodiments) and the DASH client can be implemented using a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of this disclosure are applicable to DASH and / or other media streaming technologies / standards.

[0137] The techniques described above can be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media. For example, Figure 7 A computer system (1800) suitable for implementing a particular embodiment of the disclosed subject matter is shown.

[0138] Computer software can be encoded using any suitable machine code or computer language. Machine code or computer language can be assembled, compiled, linked, or otherwise used to create code containing instructions. These instructions can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or through interpretation, microcode execution, or other means.

[0139] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0140] Figure 7 The components shown for the computer system (1800) are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any component or combination of components shown in the exemplary embodiments of the computer system (1800).

[0141] The computer system (1800) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., speech, tapping), visual input (e.g., gestures), or olfactory input (not shown). The human-machine interface device may also be used to capture specific media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image capturing devices), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0142] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard (1801), mouse (1802), touchpad (1803), touch screen (1810), data glove (not shown), joystick (1805), microphone (1806), scanner (1807), and camera device (1808).

[0143] The computer system (1800) may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: haptic output devices (e.g., haptic feedback via a touchscreen (1810), data gloves (not shown), or joystick (1805), but haptic feedback devices that are not used as input devices may also exist); audio output devices (e.g., speakers (1809), headphones (not depicted)); visual output devices (e.g., screens (1810), including CRT (Cathode Ray Tube) screens, LCD (Liquid Crystal Display) screens, plasma screens, OLED (Organic Light Emitting Diode) screens, each screen may or may not have touchscreen input capability, each screen may or may not have haptic feedback capability—some of which may be able to output two-dimensional visual output or more than three-dimensional output in a manner such as stereoscopic output; virtual reality glasses (not depicted); holographic displays and smoke generators (not depicted)); and printers (not depicted).

[0144] The computer system (1800) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVDROM / RW (1820) having media such as CD (Compact Disc, CD) / DVD (Digital Video Disk, DVD) (1821), thumb drives (1822), removable hard disk drives or solid-state drives (1823), conventional magnetic media such as magnetic tape and floppy disks (not depicted), devices based on dedicated ROM (Read Only Memory, ROM) / ASIC (Application Specific Integrated Circuit, ASIC) / PLD (Programable Logic Device, PLD) such as security dongles (not depicted), etc.

[0145] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0146] The computer system (1800) may also include interfaces (1854) to one or more communication networks (1855). Networks may be, for example, wireless networks, wired networks, or optical networks. Networks may also be local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), vehicular and industrial networks, real-time networks, latency-tolerant networks, etc. Examples of networks include: LANs such as Ethernet; wireless LANs; cellular networks including GSM (Global System for Mobile Communications), 3G (the Third Generation), 4G (the Fourth Generation), 5G (the Fifth Generation), LTE (Long Term Evolution), etc.; wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial networks including CAN buses, etc. Some networks typically require external network interface adapters that attach to certain general-purpose data ports or peripheral buses (1849) (e.g., the USB (Universal Serial Bus, USB) port of the computer system (1800); others are typically integrated into the core of the computer system (1800) by attaching to the system bus as described below (e.g., to an Ethernet interface of a PC computer system or a cellular network interface of a smartphone computer system). The computer system (1800) can communicate with other entities through any of these networks. Such communication can be one-way receive-only (e.g., broadcasting TV), one-way transmit-only (e.g., to a CAN bus of a certain CAN bus device), or bidirectional, such as to other computer systems using local area digital networks or wide area digital networks. Specific protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0147] The human-machine interface device, human-accessible storage device and network interface mentioned above can be attached to the core (1840) of the computer system (1800).

[0148] The core (1840) may include one or more central processing units (CPUs) (1841), graphics processing units (GPUs) (1842), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (1843), hardware accelerators (1844) for certain tasks, graphics adapters (1850), etc. These devices, as well as read-only memory (ROM) (1845), random access memory (1846), and internal mass storage devices (e.g., internal non-user-accessible hard disk drives, SSDs, etc.) (1847), may be connected via a system bus (1848). In some computer systems, the system bus (1848) may be accessed in the form of one or more physical plugs to allow for expansion via additional CPUs, GPUs, etc. Peripheral devices may be attached directly or via a peripheral bus (1849) to the core's system bus (1848). In the example, a screen (1810) may be connected to a graphics adapter (1850). Peripheral bus architectures include PCI (Peripheral Component Interconnect / Interface) and USB (Universal Serial Bus).

[0149] The CPU (1841), GPU (1842), FPGA (1843), and accelerator (1844) can execute certain instructions, which, when combined, can form the aforementioned computer code. This computer code can be stored in ROM (1845) or RAM (Random Access Memory) (1846). Transient data can also be stored in RAM (1846), while permanent data can be stored, for example, in an internal mass storage device (1847). Fast storage and retrieval of any memory device in the memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1841), GPUs (1842), mass storage devices (1847), ROMs (1845), RAMs (1846), etc.

[0150] Computer-readable media may have computer code thereon for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0151] As a non-limiting example, a computer system (1800) with an architecture—particularly a core (1840)—can be functionalized by processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented in one or more tangible computer-readable media. Such computer-readable media can be associated with user-accessible mass storage devices as described above, and with certain storage devices of the core (1840) having non-transitory characteristics, such as internal mass storage devices (1847) or ROM (1845). Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core (1840). Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause the core (1840) and, in particular, the processors therein (including CPUs, GPUs, FPGAs, etc.) to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (1846) and modifying such data structures according to the processes defined by the software. Alternatively or as an alternative, a computer system may be provided with functionality by means of logic hardwired or otherwise implemented in circuitry (e.g., an accelerator (1844)), which may replace or operate with software to perform the specific processing or a specific portion of the specific processing described herein. Where appropriate, references to software may include logic, and references to logic may also include software. Where appropriate, references to a computer-readable medium may include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry implementing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0152] While this disclosure has described several exemplary embodiments, there are variations, substitutions, and various equivalents that fall within the scope of this disclosure. Therefore, it should be recognized that those skilled in the art will be able to conceive of many systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. A method for processing HTTP-based dynamically adaptive streaming DASH media streams, the method being executed by a DASH client hosted by a media streaming device in a media streaming system, characterized in that, The method includes: Receive DASH events associated with the DASH media stream; Determine that the DASH event is an update to a previously received DASH event; and The DASH event is processed based on the scheduling mode of the DASH event; The DASH event includes at least one of the following: In-band events transmitted along with the first media slice in a content set, the content set comprising one or more media slices; Media presentations describing the MPD incident; or Timed metadata samples; The scheduling modes of the DASH event include a receive mode and a start mode; When the scheduling mode of the DASH event is the startup mode, processing the DASH event based on the scheduling mode of the DASH event includes: in response to the previously received DASH event, deleting the previously received DASH event from the active event list associated with the same scheme identifier and value pair as the DASH event, the active event list including a list of DASH events to be scheduled.

2. The method according to claim 1, characterized in that, Determining that the DASH event is an update to the previously received DASH event includes: The DASH event is determined to be a newer version based on either its status field or a flag associated with the DASH event; and In response to a combination of fields in the DASH event matching a combination of fields in a previously received DASH event, it is determined that the DASH event is an update of the previously received DASH event, wherein the combination of fields includes at least one of the following: a scheme identifier field and a value field.

3. The method according to claim 2, characterized in that, Determining that the DASH event is an updated version includes: In response to the DASH event being an MPD event, the system determines whether the DASH event is an updated version based on the status field of the DASH event; or In response to the DASH event being an in-band event, the DASH event is determined to be an updated version based on the flags associated with the DASH event.

4. The method according to claim 1, characterized in that, The current media rendering time of the media stream is less than the start time of the DASH event, and the method further includes: In response to the previously received DASH event not being in the list of scheduled events associated with the same scheme identifier and value pair as the DASH event, the DASH event is added to the list of active events, wherein the list of scheduled events includes a list of DASH events that have already been scheduled.

5. The method according to claim 1, characterized in that, The method further includes: In response to the previously received DASH event, processing of the DASH event is stopped in the list of scheduled events associated with the same scheme identifier and value pair as the DASH event.

6. The method according to claim 4, characterized in that, The method further includes: Schedule the DASH event at the start time of the DASH event; and Add the DASH event to the list of scheduling events associated with the same scheme identifier and value pair as the DASH event.

7. The method according to claim 6, characterized in that, After scheduling the DASH event at the start time of the DASH event, the method further includes: Remove the DASH event from the list of active events.

8. The method according to claim 1, characterized in that, The current media rendering time of the media stream is greater than the start time of the DASH event, and the method further includes: The end time of the DASH event is determined based on the start time and duration of the DASH event. In response to the current media rendering time of the media stream being less than the end time of the DASH event and the previously received DASH event not being in the list of scheduled events associated with the same scheme identifier and value pair as the DASH event: At the start time of the DASH event, schedule the DASH event to an application that has subscribed to the event type to which the DASH event belongs; and Add the DASH event to the list of scheduling events associated with the same scheme identifier and value pair as the DASH event.

9. The method according to claim 8, characterized in that: Before scheduling the DASH event at the start time of the DASH event, the method further includes: Add the DASH event to the activity event list; and After scheduling the DASH event at the start time of the DASH event, the method further includes: Remove the DASH event from the list of active events.

10. The method according to any one of claims 1 to 2, characterized in that: The scheduling mode for the DASH event is receive mode; and In response to the previously received DASH event not being in the list of scheduling events associated with the same scheme identifier and value pair as the DASH event: The DASH event is scheduled to an application that has subscribed to the event type to which the DASH event belongs; as well as Add the DASH event to the list of scheduling events associated with the same scheme identifier and value pair as the DASH event.

11. An apparatus for processing HTTP-based dynamically adaptive streaming DASH media streams, the apparatus comprising a memory for storing computer instructions and a processor communicating with the memory, characterized in that, When the processor executes the computer instructions, the processor is configured to cause the device to perform the method according to any one of claims 1 to 10.

12. A system for processing HTTP-based dynamically adaptive streaming DASH media streams, characterized in that, The system includes: Memory, which stores instructions; and A processor that communicates with the memory, wherein, when the processor executes the instructions, the processor is configured to cause the device to perform the method according to any one of claims 1 to 10.

13. A non-transitory storage medium for storing computer-readable instructions, characterized in that, The computer-readable instructions, when executed by a processor in an apparatus for processing DASH media streams, cause the processor to perform the method according to any one of claims 1 to 10.

Images